From 96ced59d6facfbb3e89bc6af49131d1ca54b867c Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:01:43 +0800 Subject: [PATCH 01/83] feat: define ACP primary event rollout policy --- .env.example | 11 ++++ README.md | 18 ++++++ docs/acp-migration.md | 121 +++++++++++++++++++++++++++++++++++++++++ src/tendwire/config.py | 79 +++++++++++++++++++++++++++ tests/test_config.py | 83 ++++++++++++++++++++++++++++ 5 files changed, 312 insertions(+) create mode 100644 docs/acp-migration.md diff --git a/.env.example b/.env.example index a643906..8c60a0c 100644 --- a/.env.example +++ b/.env.example @@ -93,6 +93,17 @@ TENDWIRE_TURN_REFRESH_WORKERS=4 # Compatibility flag: legacy|dual|shadow|observed all use the observed model. TENDWIRE_TURN_MODEL=observed +# Structured agent-event source policy. ACP is preferred when a compatible +# session is bound; existing Herdr turn adapters remain the lossless fallback. +# acp_shadow records ACP without projecting it, while acp_required fails closed +# instead of using legacy content. Thought content is private by default and is +# never eligible for the connector outbox solely because ACP emitted it. +TENDWIRE_AGENT_EVENT_SOURCE=acp_preferred +TENDWIRE_ACP_THOUGHT_POLICY=private_summary +TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS=30 +TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS=5 +TENDWIRE_ACP_MAX_FRAME_BYTES=8388608 + # Process-local bounded LRU for repeated public-text sanitization. Set to 0 to # disable; values above 65536 are capped. The default covers typical retained # turn and working-card reuse without sharing sanitized data between processes. diff --git a/README.md b/README.md index da92142..62dbbb7 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,11 @@ through the Herdr socket/event backend. Both paths normalize Herdr state into neutral Tendwire spaces, workers, attention, turns, pending interactions, command results, connector jobs, and backend health. +ACP is the preferred semantic source for compatible, authenticated worker +sessions while Herdr remains the process and identity authority. Source +precedence, privacy, fallback behavior, and cross-repository rollout are +defined in [docs/acp-migration.md](docs/acp-migration.md). + Herdres can use Tendwire as its source/control plane while Herdres remains the Telegram connector. Tendwire owns Herdr observation, private bindings, turns/pending interactions, attention, command routing, receipts, backend @@ -548,6 +553,11 @@ variables: | `turn_refresh_interval_seconds` | `TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS` | `2.0` | finite positive float | | `turn_refresh_workers` | `TENDWIRE_TURN_REFRESH_WORKERS` | `4` | integer from 1 through 32 and no greater than `max_workers` | | `turn_model` | `TENDWIRE_TURN_MODEL` | `observed` | `observed`; `legacy`, `dual`, and `shadow` are deprecated aliases with identical observed behavior | +| `agent_event_source` | `TENDWIRE_AGENT_EVENT_SOURCE` | `acp_preferred` | `legacy`, `acp_shadow`, `acp_preferred`, or `acp_required` | +| `acp_thought_policy` | `TENDWIRE_ACP_THOUGHT_POLICY` | `private_summary` | `disabled`, `private_summary`, or `private_all`; never a public-delivery grant | +| `acp_request_timeout_seconds` | `TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS` | `30.0` | finite positive float | +| `acp_shutdown_timeout_seconds` | `TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS` | `5.0` | finite positive float | +| `acp_max_frame_bytes` | `TENDWIRE_ACP_MAX_FRAME_BYTES` | `8388608` | integer from 1 through 67108864 | The socket/event backend uses `event_debounce_seconds` for event batching and `reconcile_interval_seconds` for bounded periodic full reconciles. Set @@ -560,6 +570,14 @@ snapshot/projections instead of publishing a truncated authoritative snapshot. Incremental events that would add workers over the cap are ignored with the same public-safe degraded evidence. +`acp_preferred` means ACP is the primary semantic source only for workers with +an authenticated ACP session binding; workers without one continue through the +existing Herdr turn adapters. `acp_shadow` persists and compares ACP events but +does not project them into turns. `acp_required` fails closed for an unbound or +unhealthy ACP worker. None of these modes makes agent thoughts public: thought +events remain private diagnostic data unless a separate, explicit sanitized +projection is introduced. + Snapshot history defaults are sized for a five-minute observation rhythm: $14 \times 24 \times 12 = 4032$ observations, while the 4096-row count ceiling (including the latest row) leaves 64 rows of headroom. This is a diff --git a/docs/acp-migration.md b/docs/acp-migration.md new file mode 100644 index 0000000..a508938 --- /dev/null +++ b/docs/acp-migration.md @@ -0,0 +1,121 @@ +# ACP primary-event migration + +This document defines the migration from backend-specific transcript readers to +Agent Client Protocol (ACP) as Tendwire's preferred semantic event source. +Herdr remains authoritative for workspace, pane, worker identity, process +liveness, and command routing until the ACP control path is proven separately. +Tendwire remains authoritative for persistence, reconciliation, public safety, +command receipts, and connector delivery. + +## Source policy + +`TENDWIRE_AGENT_EVENT_SOURCE` controls projection precedence: + +- `legacy`: use the existing Herdr/Codex/OMP turn readers only. +- `acp_shadow`: ingest ACP events durably, compare them with legacy turns, and + keep legacy turns authoritative. +- `acp_preferred`: use ACP for an authenticated, healthy ACP-bound worker and + fall back to the legacy reader for every other worker. +- `acp_required`: use ACP only and fail closed when the binding or stream is not + healthy. This mode is intended for conformance testing, not initial rollout. + +The default is `acp_preferred`. The default does not invent an ACP session or +silently replace a worker: without a proven binding, legacy observation remains +authoritative. + +## Authority split + +| Concern | Authority | +| --- | --- | +| Workspace and logical pane identity | Herdr | +| Public stable worker identity | Tendwire's authenticated Herdr projection | +| ACP session and message identity | ACP agent, stored privately by Tendwire | +| Messages, thoughts, tools, plans, and usage | ACP when preferred and healthy | +| Turn finality and connector eligibility | Tendwire durable projection | +| Telegram presentation and delivery state | Herdres | +| Command idempotency and uncertain outcomes | Tendwire command receipts | + +An ACP `sessionId` is never a public worker identity. Tendwire must bind it to +the current private `WorkerBinding` generation and reject events after that +binding expires, moves, or is replaced. Replayed ACP events must deduplicate on +their producer identity without changing the public worker identity. + +## Canonical events + +The structured event journal accepts these semantic kinds: + +- user message +- agent message +- thought +- tool call +- tool call update +- plan +- usage +- session information + +Producer IDs, raw inputs, raw outputs, session IDs, terminal IDs, paths, and +reasoning are private. Public turn projection is deliberately narrower: +`user_text`, `assistant_stream_text`, `assistant_final_text`, completion state, +and existing safe metadata. Tool and plan presentation requires its own +sanitizing projection and must not reuse raw ACP payloads. + +## Thought policy + +`TENDWIRE_ACP_THOUGHT_POLICY` has three values: + +- `disabled`: discard thought chunks before persistence. +- `private_summary`: retain readable reasoning summaries privately and discard + raw-reasoning chunks. +- `private_all`: retain every thought chunk privately for explicit local + diagnostics. + +The default is `private_summary`. No thought policy grants connector delivery. +Herdres must never receive a raw thought event. A future public summary feature +requires a separate schema, sanitizer, explicit operator opt-in, and tests that +prove raw reasoning cannot cross the boundary. + +## Runtime lifecycle + +For ACP v1 stdio, the component that owns the adapter process also owns framing, +initialization, request correlation, stderr handling, cancellation, and bounded +shutdown. Tendwire must not claim an ACP worker healthy until initialization, +capability negotiation, session creation/load/resume, and private worker binding +all succeed. + +Disconnect handling is conservative: + +1. Stop accepting events from the disconnected generation. +2. Persist stream health without publishing private adapter details. +3. In `acp_preferred`, allow the next legacy refresh to become authoritative. +4. Reinitialize and rebind before accepting ACP events again. +5. Reconcile replayed messages and tool calls by producer identity. + +## Cross-repository requirements + +Herdr needs an ACP-aware launch or proxy surface that exposes enough private +metadata for Tendwire to bind an ACP session to an existing logical pane. The +binding must survive terminal/session churn without making ACP identity a +public continuity input. + +Herdres needs optional presentations for sanitized tool and plan progress. It +does not ingest ACP directly: it continues polling Tendwire's neutral outbox so +delivery retries, topic binding, rate limits, and Telegram state remain outside +the agent protocol. + +## Rollout gates + +Promotion proceeds `legacy` -> `acp_shadow` -> `acp_preferred`. The following +must pass before `acp_required` is considered: + +- no missing or duplicated user/final messages across adapter restarts; +- deterministic replay deduplication; +- correct open-to-final turn identity; +- tool lifecycle completion after cancellation and permission denial; +- plan replacement without stale entries; +- thought and raw tool payloads absent from every public API/outbox surface; +- fallback after adapter failure without regressing existing final delivery; +- exact worker continuity across Herdr pane moves and agent-session recreation. + +ACP prompt submission, cancellation, and permission handling are a later +control-path migration. They must preserve Tendwire's existing request receipts +and uncertain-outcome rules before replacing Herdr command routing. diff --git a/src/tendwire/config.py b/src/tendwire/config.py index 64ee04c..9ab162b 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -16,7 +16,16 @@ HERDR_BACKENDS = frozenset({"cli", "socket"}) TURN_MODELS = frozenset({"legacy", "dual", "shadow", "observed"}) +AGENT_EVENT_SOURCES = frozenset( + {"legacy", "acp_shadow", "acp_preferred", "acp_required"} +) +ACP_THOUGHT_POLICIES = frozenset({"disabled", "private_summary", "private_all"}) DEFAULT_TURN_MODEL = "observed" +DEFAULT_AGENT_EVENT_SOURCE = "acp_preferred" +DEFAULT_ACP_THOUGHT_POLICY = "private_summary" +DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS = 30.0 +DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS = 5.0 +DEFAULT_ACP_MAX_FRAME_BYTES = 8 * 1024 * 1024 DEFAULT_EVENT_DEBOUNCE_SECONDS = 0.05 DEFAULT_RECONCILE_INTERVAL_SECONDS = 300.0 DEFAULT_EVENT_RETENTION_DAYS = 7 @@ -64,6 +73,11 @@ class Config: herdr_timeout_seconds: float = 5.0 herdr_backend: str = "cli" turn_model: str = DEFAULT_TURN_MODEL + agent_event_source: str = DEFAULT_AGENT_EVENT_SOURCE + acp_thought_policy: str = DEFAULT_ACP_THOUGHT_POLICY + acp_request_timeout_seconds: float = DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS + acp_shutdown_timeout_seconds: float = DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS + acp_max_frame_bytes: int = DEFAULT_ACP_MAX_FRAME_BYTES event_debounce_seconds: float = DEFAULT_EVENT_DEBOUNCE_SECONDS reconcile_interval_seconds: float = DEFAULT_RECONCILE_INTERVAL_SECONDS event_retention_days: int = DEFAULT_EVENT_RETENTION_DAYS @@ -126,6 +140,41 @@ def __post_init__(self) -> None: "turn_model=%s is a compatibility alias and behaves as observed", turn_model, ) + agent_event_source = str(self.agent_event_source or "").strip().lower() + if agent_event_source not in AGENT_EVENT_SOURCES: + allowed = ", ".join(sorted(AGENT_EVENT_SOURCES)) + raise ValueError(f"agent_event_source must be one of: {allowed}") + object.__setattr__(self, "agent_event_source", agent_event_source) + acp_thought_policy = str(self.acp_thought_policy or "").strip().lower() + if acp_thought_policy not in ACP_THOUGHT_POLICIES: + allowed = ", ".join(sorted(ACP_THOUGHT_POLICIES)) + raise ValueError(f"acp_thought_policy must be one of: {allowed}") + object.__setattr__(self, "acp_thought_policy", acp_thought_policy) + object.__setattr__( + self, + "acp_request_timeout_seconds", + _positive_finite_float( + self.acp_request_timeout_seconds, + "acp_request_timeout_seconds", + ), + ) + object.__setattr__( + self, + "acp_shutdown_timeout_seconds", + _positive_finite_float( + self.acp_shutdown_timeout_seconds, + "acp_shutdown_timeout_seconds", + ), + ) + object.__setattr__( + self, + "acp_max_frame_bytes", + _bounded_positive_int( + self.acp_max_frame_bytes, + "acp_max_frame_bytes", + maximum=64 * 1024 * 1024, + ), + ) object.__setattr__( self, "event_debounce_seconds", @@ -443,6 +492,11 @@ def load_config( herdr_timeout_seconds: float | str | None = None, herdr_backend: str | None = None, turn_model: str | None = None, + agent_event_source: str | None = None, + acp_thought_policy: str | None = None, + acp_request_timeout_seconds: float | str | None = None, + acp_shutdown_timeout_seconds: float | str | None = None, + acp_max_frame_bytes: int | str | None = None, event_debounce_seconds: float | str | None = None, reconcile_interval_seconds: float | str | None = None, event_retention_days: int | str | None = None, @@ -538,6 +592,31 @@ def load_config( "TENDWIRE_TURN_MODEL", DEFAULT_TURN_MODEL, ), + agent_event_source=_resolve_value( + agent_event_source, + "TENDWIRE_AGENT_EVENT_SOURCE", + DEFAULT_AGENT_EVENT_SOURCE, + ), + acp_thought_policy=_resolve_value( + acp_thought_policy, + "TENDWIRE_ACP_THOUGHT_POLICY", + DEFAULT_ACP_THOUGHT_POLICY, + ), + acp_request_timeout_seconds=_resolve_value( + acp_request_timeout_seconds, + "TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS", + DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS, + ), + acp_shutdown_timeout_seconds=_resolve_value( + acp_shutdown_timeout_seconds, + "TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS", + DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS, + ), + acp_max_frame_bytes=_resolve_value( + acp_max_frame_bytes, + "TENDWIRE_ACP_MAX_FRAME_BYTES", + DEFAULT_ACP_MAX_FRAME_BYTES, + ), event_debounce_seconds=_resolve_value( event_debounce_seconds, "TENDWIRE_EVENT_DEBOUNCE_SECONDS", diff --git a/tests/test_config.py b/tests/test_config.py index bcf414d..68ace5b 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -9,6 +9,11 @@ import pytest from tendwire.config import ( + DEFAULT_ACP_MAX_FRAME_BYTES, + DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS, + DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS, + DEFAULT_ACP_THOUGHT_POLICY, + DEFAULT_AGENT_EVENT_SOURCE, DEFAULT_COMMAND_RECEIPT_RETENTION_COUNT, DEFAULT_COMMAND_RECEIPT_RETENTION_SECONDS, DEFAULT_COMMAND_RETRY_HORIZON_SECONDS, @@ -27,6 +32,84 @@ ) +def test_acp_event_source_defaults_to_preferred_with_private_summaries( + monkeypatch, +) -> None: + for name in ( + "TENDWIRE_AGENT_EVENT_SOURCE", + "TENDWIRE_ACP_THOUGHT_POLICY", + "TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS", + "TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS", + "TENDWIRE_ACP_MAX_FRAME_BYTES", + ): + monkeypatch.delenv(name, raising=False) + + config = load_config() + + assert config.agent_event_source == DEFAULT_AGENT_EVENT_SOURCE == "acp_preferred" + assert config.acp_thought_policy == DEFAULT_ACP_THOUGHT_POLICY == "private_summary" + assert config.acp_request_timeout_seconds == DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS == 30.0 + assert config.acp_shutdown_timeout_seconds == DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS == 5.0 + assert config.acp_max_frame_bytes == DEFAULT_ACP_MAX_FRAME_BYTES == 8 * 1024 * 1024 + + +def test_acp_configuration_uses_explicit_before_environment(monkeypatch) -> None: + monkeypatch.setenv("TENDWIRE_AGENT_EVENT_SOURCE", "acp_shadow") + monkeypatch.setenv("TENDWIRE_ACP_THOUGHT_POLICY", "private_all") + monkeypatch.setenv("TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS", "11") + monkeypatch.setenv("TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS", "3") + monkeypatch.setenv("TENDWIRE_ACP_MAX_FRAME_BYTES", "4096") + + environment = load_config() + explicit = load_config( + agent_event_source="acp_required", + acp_thought_policy="disabled", + acp_request_timeout_seconds="7.5", + acp_shutdown_timeout_seconds="2.5", + acp_max_frame_bytes="8192", + ) + + assert environment.agent_event_source == "acp_shadow" + assert environment.acp_thought_policy == "private_all" + assert environment.acp_request_timeout_seconds == 11.0 + assert environment.acp_shutdown_timeout_seconds == 3.0 + assert environment.acp_max_frame_bytes == 4096 + assert explicit.agent_event_source == "acp_required" + assert explicit.acp_thought_policy == "disabled" + assert explicit.acp_request_timeout_seconds == 7.5 + assert explicit.acp_shutdown_timeout_seconds == 2.5 + assert explicit.acp_max_frame_bytes == 8192 + + +@pytest.mark.parametrize("value", ["", "acp", "preferred", "future"]) +def test_acp_event_source_rejects_unknown_values(value: str) -> None: + with pytest.raises(ValueError, match="agent_event_source must be one of"): + Config(agent_event_source=value) + + +@pytest.mark.parametrize("value", ["", "public", "summary", "future"]) +def test_acp_thought_policy_rejects_unknown_values(value: str) -> None: + with pytest.raises(ValueError, match="acp_thought_policy must be one of"): + Config(acp_thought_policy=value) + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("acp_request_timeout_seconds", 0), + ("acp_request_timeout_seconds", float("inf")), + ("acp_shutdown_timeout_seconds", -1), + ("acp_shutdown_timeout_seconds", "invalid"), + ("acp_max_frame_bytes", 0), + ("acp_max_frame_bytes", True), + ("acp_max_frame_bytes", 64 * 1024 * 1024 + 1), + ], +) +def test_acp_bounds_reject_invalid_values(field: str, value: object) -> None: + with pytest.raises(ValueError): + Config(**{field: value}) + + def test_turn_model_defaults_to_observed_and_accepts_compatibility_aliases( monkeypatch, caplog, From d30993d12acbe2d84ed868624c9692357acafbd3 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:02:13 +0800 Subject: [PATCH 02/83] Add ACP event projection layer --- src/tendwire/backends/acp_projection.py | 480 ++++++++++++++++++++++++ tests/test_acp_projection.py | 317 ++++++++++++++++ 2 files changed, 797 insertions(+) create mode 100644 src/tendwire/backends/acp_projection.py create mode 100644 tests/test_acp_projection.py diff --git a/src/tendwire/backends/acp_projection.py b/src/tendwire/backends/acp_projection.py new file mode 100644 index 0000000..5588e66 --- /dev/null +++ b/src/tendwire/backends/acp_projection.py @@ -0,0 +1,480 @@ +"""Stateful, transport-independent projection of ACP session events. + +The projector deliberately does not own an ACP connection. It accepts the +JSON-shaped values carried by ``session/update`` notifications and +``session/request_permission`` requests, then emits stable mappings suitable +for a durable event journal. This keeps protocol transport, persistence, and +Tendwire's public-content boundary separate. + +ACP streams reasoning and tool data independently from assistant messages. +That distinction is preserved here: the compatibility turn projection only +contains user and assistant text and can never expose thought or tool payloads. +""" + +from __future__ import annotations + +import hashlib +import json +from collections.abc import Mapping +from copy import deepcopy +from dataclasses import dataclass, field +from typing import Any, Final + + +SUPPORTED_EVENT_KINDS: Final[frozenset[str]] = frozenset( + { + "user_message", + "agent_message", + "thought", + "tool_call", + "tool_call_update", + "plan", + "usage", + "session_info", + } +) + +_UPDATE_KIND_MAP: Final[dict[str, str]] = { + "user_message_chunk": "user_message", + "agent_message_chunk": "agent_message", + "agent_thought_chunk": "thought", + "tool_call": "tool_call", + "tool_call_update": "tool_call_update", + "plan": "plan", + "usage_update": "usage", + "session_info_update": "session_info", +} +_RAW_TOOL_FIELDS: Final[tuple[str, str]] = ("rawInput", "rawOutput") +_MESSAGE_KINDS: Final[frozenset[str]] = frozenset( + {"user_message", "agent_message", "thought"} +) +_LEGACY_EMPTY: Final[dict[str, Any]] = { + "user_text": "", + "assistant_stream_text": "", + "assistant_final_text": "", + "complete": False, + "has_open_turn": False, +} + + +class AcpProjectionError(ValueError): + """Raised when an ACP value cannot be safely normalized.""" + + +@dataclass +class _MessageAssembly: + message_id: str + text: str = "" + + +@dataclass +class _SessionState: + sequence: int = 0 + messages: dict[str, list[_MessageAssembly]] = field( + default_factory=lambda: {kind: [] for kind in _MESSAGE_KINDS} + ) + tools: dict[str, dict[str, Any]] = field(default_factory=dict) + plan: list[dict[str, Any]] = field(default_factory=list) + usage: dict[str, Any] = field(default_factory=dict) + info: dict[str, Any] = field(default_factory=dict) + seen_source_events: set[str] = field(default_factory=set) + complete: bool = False + + +class AcpEventProjector: + """Normalize ACP notifications while retaining per-session assembly state. + + The instance is intentionally in-memory. ``dedupe_hint`` is emitted on + every event so a durable caller can deduplicate replays across process + restarts. Within one instance, events with an explicit protocol/transport + identifier (``source_event_id`` or a recognized ``_meta`` key) are dropped + when repeated. Content hashes are hints only: identical adjacent chunks + can be legitimate and are therefore never blindly discarded. + """ + + def __init__(self) -> None: + self._sessions: dict[str, _SessionState] = {} + + def normalize_session_update( + self, + notification: Mapping[str, Any], + *, + source_event_id: str | None = None, + replay: bool = False, + ) -> dict[str, Any] | None: + """Normalize one ACP ``session/update`` notification or its params. + + ``notification`` may be a full JSON-RPC notification, its ``params`` + object, or a direct object containing ``sessionId`` and ``update``. + Unknown ACP update variants are ignored for forward compatibility. + """ + + params = _unwrap_params(notification) + session_id = _required_string(params, "sessionId") + update = params.get("update") + if not isinstance(update, Mapping): + raise AcpProjectionError("ACP session/update is missing an object update") + + update_name = update.get("sessionUpdate") + if not isinstance(update_name, str): + raise AcpProjectionError("ACP update is missing sessionUpdate") + kind = _UPDATE_KIND_MAP.get(update_name) + if kind is None: + return None + + state = self._sessions.setdefault(session_id, _SessionState()) + explicit_id = source_event_id or _source_event_id(notification, params, update) + if explicit_id is not None: + scoped_id = f"{session_id}:{explicit_id}" + if scoped_id in state.seen_source_events: + return None + + if kind in _MESSAGE_KINDS: + payload = self._normalize_message(state, kind, update) + elif kind in {"tool_call", "tool_call_update"}: + payload = self._normalize_tool(state, kind, update) + elif kind == "plan": + payload = self._normalize_plan(state, update) + elif kind == "usage": + payload = self._normalize_usage(state, update) + else: + payload = self._normalize_session_info(state, update) + + state.sequence += 1 + if explicit_id is not None: + state.seen_source_events.add(f"{session_id}:{explicit_id}") + state.complete = False + return _canonical_event( + session_id=session_id, + sequence=state.sequence, + kind=kind, + payload=payload, + source_event_id=explicit_id, + replay=replay, + original_update=update, + ) + + def normalize_permission_request( + self, + request: Mapping[str, Any], + *, + source_event_id: str | None = None, + replay: bool = False, + ) -> dict[str, Any] | None: + """Project ``session/request_permission`` as a tool lifecycle update. + + ACP attaches permission requests to tool calls, so no synthetic ninth + event kind is introduced. Options are retained in the canonical tool + payload for a decision layer to consume. + """ + + params = _unwrap_params(request) + session_id = _required_string(params, "sessionId") + tool_call = params.get("toolCall") + if not isinstance(tool_call, Mapping): + raise AcpProjectionError("ACP permission request is missing toolCall") + tool_call_id = _required_string(tool_call, "toolCallId") + + state = self._sessions.setdefault(session_id, _SessionState()) + explicit_id = source_event_id or _jsonrpc_request_id(request) or _source_event_id( + request, params, tool_call + ) + if explicit_id is not None: + scoped_id = f"{session_id}:{explicit_id}" + if scoped_id in state.seen_source_events: + return None + + snapshot = _merge_tool_snapshot(state.tools.get(tool_call_id), tool_call) + options = params.get("options", []) + if not isinstance(options, list): + options = [] + snapshot["permission"] = { + "required": True, + "options": [deepcopy(option) for option in options if isinstance(option, Mapping)], + } + state.tools[tool_call_id] = snapshot + state.sequence += 1 + if explicit_id is not None: + state.seen_source_events.add(f"{session_id}:{explicit_id}") + payload = { + "tool_call_id": tool_call_id, + "changes": _without_discriminator(tool_call), + "snapshot": deepcopy(snapshot), + "permission": deepcopy(snapshot["permission"]), + } + return _canonical_event( + session_id=session_id, + sequence=state.sequence, + kind="tool_call_update", + payload=payload, + source_event_id=explicit_id, + replay=replay, + original_update={"sessionUpdate": "tool_call_update", **dict(tool_call)}, + ) + + def project_turn_content( + self, + session_id: str, + *, + complete: bool | None = None, + ) -> dict[str, Any]: + """Return Tendwire's legacy text-only turn shape for one ACP session. + + Thought text, plans, tool content, raw inputs, raw outputs, and + permission details are structurally unreachable from this projection. + ``complete=True`` moves assembled assistant text from stream to final. + Callers should only set it after the ACP ``session/prompt`` response. + """ + + state = self._sessions.get(session_id) + if state is None: + return dict(_LEGACY_EMPTY) + is_complete = state.complete if complete is None else bool(complete) + user_text = _joined_messages(state.messages["user_message"]) + assistant_text = _joined_messages(state.messages["agent_message"]) + return { + "user_text": user_text, + "assistant_stream_text": "" if is_complete else assistant_text, + "assistant_final_text": assistant_text if is_complete else "", + "complete": is_complete, + "has_open_turn": bool(user_text or assistant_text) and not is_complete, + } + + def mark_turn_complete(self, session_id: str) -> dict[str, Any]: + """Mark the current ACP prompt turn complete and return legacy content.""" + + state = self._sessions.setdefault(session_id, _SessionState()) + state.complete = True + return self.project_turn_content(session_id) + + def reset_turn(self, session_id: str) -> None: + """Start a fresh prompt turn while preserving session-level ACP state.""" + + state = self._sessions.setdefault(session_id, _SessionState()) + state.messages = {kind: [] for kind in _MESSAGE_KINDS} + state.complete = False + + def session_snapshot(self, session_id: str) -> dict[str, Any] | None: + """Return a defensive snapshot for persistence or diagnostics.""" + + state = self._sessions.get(session_id) + if state is None: + return None + return { + "session_id": session_id, + "sequence": state.sequence, + "messages": { + kind: [ + {"message_id": message.message_id, "text": message.text} + for message in messages + ] + for kind, messages in state.messages.items() + }, + "tools": deepcopy(state.tools), + "plan": deepcopy(state.plan), + "usage": deepcopy(state.usage), + "session_info": deepcopy(state.info), + "complete": state.complete, + } + + @staticmethod + def _normalize_message( + state: _SessionState, + kind: str, + update: Mapping[str, Any], + ) -> dict[str, Any]: + content = update.get("content") + if not isinstance(content, Mapping): + raise AcpProjectionError(f"ACP {kind} update is missing content") + message_id_value = update.get("messageId") + assemblies = state.messages[kind] + message_id = ( + message_id_value + if isinstance(message_id_value, str) and message_id_value + else assemblies[-1].message_id + if assemblies + else f"implicit-{kind}-1" + ) + if not assemblies or assemblies[-1].message_id != message_id: + assemblies.append(_MessageAssembly(message_id=message_id)) + text_delta = content.get("text") if content.get("type") == "text" else None + if not isinstance(text_delta, str): + text_delta = "" + assemblies[-1].text += text_delta + return { + "message_id": message_id, + "content": deepcopy(dict(content)), + "text_delta": text_delta, + "assembled_text": assemblies[-1].text, + "message_index": len(assemblies) - 1, + } + + @staticmethod + def _normalize_tool( + state: _SessionState, + kind: str, + update: Mapping[str, Any], + ) -> dict[str, Any]: + tool_call_id = _required_string(update, "toolCallId") + previous = state.tools.get(tool_call_id) + snapshot = _merge_tool_snapshot(previous, update) + state.tools[tool_call_id] = snapshot + payload: dict[str, Any] = { + "tool_call_id": tool_call_id, + "snapshot": deepcopy(snapshot), + } + if kind == "tool_call_update": + payload["changes"] = _without_discriminator(update) + return payload + + @staticmethod + def _normalize_plan( + state: _SessionState, update: Mapping[str, Any] + ) -> dict[str, Any]: + entries = update.get("entries", []) + if not isinstance(entries, list): + entries = [] + state.plan = [deepcopy(dict(entry)) for entry in entries if isinstance(entry, Mapping)] + return {"entries": deepcopy(state.plan), "snapshot": True} + + @staticmethod + def _normalize_usage( + state: _SessionState, update: Mapping[str, Any] + ) -> dict[str, Any]: + state.usage.update(_without_discriminator(update)) + return deepcopy(state.usage) + + @staticmethod + def _normalize_session_info( + state: _SessionState, update: Mapping[str, Any] + ) -> dict[str, Any]: + # Presence is meaningful: explicit null clears an existing property. + state.info.update(_without_discriminator(update)) + return deepcopy(state.info) + + +def _unwrap_params(value: Mapping[str, Any]) -> Mapping[str, Any]: + params = value.get("params") + if isinstance(params, Mapping): + return params + return value + + +def _required_string(value: Mapping[str, Any], key: str) -> str: + item = value.get(key) + if not isinstance(item, str) or not item: + raise AcpProjectionError(f"ACP value is missing non-empty {key}") + return item + + +def _source_event_id(*values: Mapping[str, Any]) -> str | None: + for value in values: + for key in ("eventId", "event_id", "notificationId", "notification_id"): + candidate = value.get(key) + if isinstance(candidate, (str, int)) and str(candidate): + return str(candidate) + meta = value.get("_meta") + if isinstance(meta, Mapping): + for key in ("eventId", "event_id", "notificationId", "notification_id"): + candidate = meta.get(key) + if isinstance(candidate, (str, int)) and str(candidate): + return str(candidate) + return None + + +def _jsonrpc_request_id(value: Mapping[str, Any]) -> str | None: + """Return a request ID, but never mistake a notification field for one.""" + + if value.get("method") != "session/request_permission": + return None + candidate = value.get("id") + if isinstance(candidate, (str, int)) and str(candidate): + return str(candidate) + return None + + +def _without_discriminator(value: Mapping[str, Any]) -> dict[str, Any]: + return { + key: deepcopy(item) + for key, item in value.items() + if key not in {"sessionUpdate", "_meta"} + } + + +def _merge_tool_snapshot( + previous: Mapping[str, Any] | None, update: Mapping[str, Any] +) -> dict[str, Any]: + snapshot = deepcopy(dict(previous)) if previous is not None else {} + snapshot.update(_without_discriminator(update)) + return snapshot + + +def _canonical_event( + *, + session_id: str, + sequence: int, + kind: str, + payload: Mapping[str, Any], + source_event_id: str | None, + replay: bool, + original_update: Mapping[str, Any], +) -> dict[str, Any]: + if kind not in SUPPORTED_EVENT_KINDS: + raise AcpProjectionError(f"unsupported canonical event kind: {kind}") + dedupe_material = { + "session_id": session_id, + "kind": kind, + "source_event_id": source_event_id, + "update": original_update, + } + encoded = json.dumps( + dedupe_material, sort_keys=True, separators=(",", ":"), default=str + ).encode("utf-8") + privacy = "session" + private_fields: list[str] = [] + if kind == "thought": + privacy = "private" + private_fields = ["payload"] + elif kind in {"tool_call", "tool_call_update"}: + privacy = "mixed" + for field_name in _RAW_TOOL_FIELDS: + snake_name = "raw_input" if field_name == "rawInput" else "raw_output" + private_fields.extend( + [ + f"payload.snapshot.{field_name}", + f"payload.snapshot.{snake_name}", + f"payload.changes.{field_name}", + f"payload.changes.{snake_name}", + ] + ) + event_id = ( + f"acp:{session_id}:{source_event_id}" + if source_event_id is not None + else f"acp:{session_id}:local:{sequence}" + ) + return { + "schema_version": 1, + "event_id": event_id, + "source": "acp", + "session_id": session_id, + "sequence": sequence, + "kind": kind, + "payload": deepcopy(dict(payload)), + "privacy": privacy, + "private_fields": private_fields, + "source_event_id": source_event_id, + "replay": bool(replay), + "dedupe_hint": hashlib.sha256(encoded).hexdigest(), + "dedupe_safe": source_event_id is not None, + } + + +def _joined_messages(messages: list[_MessageAssembly]) -> str: + return "\n\n".join(message.text for message in messages if message.text) + + +__all__ = [ + "AcpEventProjector", + "AcpProjectionError", + "SUPPORTED_EVENT_KINDS", +] diff --git a/tests/test_acp_projection.py b/tests/test_acp_projection.py new file mode 100644 index 0000000..d41d836 --- /dev/null +++ b/tests/test_acp_projection.py @@ -0,0 +1,317 @@ +from __future__ import annotations + +import pytest + +from tendwire.backends.acp_projection import AcpEventProjector, AcpProjectionError + + +def _update(session_update: str, **fields: object) -> dict[str, object]: + return { + "jsonrpc": "2.0", + "method": "session/update", + "params": { + "sessionId": "session-1", + "update": {"sessionUpdate": session_update, **fields}, + }, + } + + +def test_message_chunks_are_assembled_by_session_kind_and_message_id() -> None: + projector = AcpEventProjector() + + first = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "hello "}, + ) + ) + second = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "world"}, + ) + ) + third = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-2", + content={"type": "text", "text": "follow-up"}, + ) + ) + + assert first is not None and first["payload"]["assembled_text"] == "hello " + assert second is not None and second["payload"]["assembled_text"] == "hello world" + assert third is not None and third["payload"]["message_index"] == 1 + assert [first["sequence"], second["sequence"], third["sequence"]] == [1, 2, 3] + assert projector.project_turn_content("session-1") == { + "user_text": "", + "assistant_stream_text": "hello world\n\nfollow-up", + "assistant_final_text": "", + "complete": False, + "has_open_turn": True, + } + + +def test_thoughts_are_private_and_never_enter_legacy_turn_content() -> None: + projector = AcpEventProjector() + thought = projector.normalize_session_update( + _update( + "agent_thought_chunk", + messageId="reasoning-1", + content={"type": "text", "text": "private reasoning"}, + ) + ) + projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "safe answer"}, + ) + ) + + assert thought is not None + assert thought["kind"] == "thought" + assert thought["privacy"] == "private" + assert thought["private_fields"] == ["payload"] + legacy = projector.mark_turn_complete("session-1") + assert legacy["assistant_final_text"] == "safe answer" + assert legacy["assistant_stream_text"] == "" + assert "reasoning" not in repr(legacy) + + +def test_non_text_content_is_preserved_without_becoming_turn_text() -> None: + projector = AcpEventProjector() + event = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="image-1", + content={"type": "image", "data": "sensitive-base64", "mimeType": "image/png"}, + ) + ) + + assert event is not None + assert event["payload"]["content"]["type"] == "image" + assert event["payload"]["text_delta"] == "" + assert projector.project_turn_content("session-1")["assistant_stream_text"] == "" + + +def test_user_and_agent_text_remain_separate_and_reset_starts_new_turn() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "user_message_chunk", + messageId="prompt-1", + content={"type": "text", "text": "question"}, + ) + ) + projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "answer"}, + ) + ) + assert projector.mark_turn_complete("session-1") == { + "user_text": "question", + "assistant_stream_text": "", + "assistant_final_text": "answer", + "complete": True, + "has_open_turn": False, + } + + projector.reset_turn("session-1") + assert projector.project_turn_content("session-1")["user_text"] == "" + assert projector.session_snapshot("session-1")["sequence"] == 2 + + +def test_tool_lifecycle_merges_partial_updates_and_marks_raw_fields_private() -> None: + projector = AcpEventProjector() + started = projector.normalize_session_update( + _update( + "tool_call", + toolCallId="tool-1", + title="Read configuration", + kind="read", + status="pending", + rawInput={"path": "/private/file"}, + ) + ) + updated = projector.normalize_session_update( + _update( + "tool_call_update", + toolCallId="tool-1", + status="completed", + content=[{"type": "content", "content": {"type": "text", "text": "done"}}], + rawOutput={"secret": "value"}, + ) + ) + + assert started is not None and started["kind"] == "tool_call" + assert updated is not None and updated["kind"] == "tool_call_update" + snapshot = updated["payload"]["snapshot"] + assert snapshot["title"] == "Read configuration" + assert snapshot["status"] == "completed" + assert snapshot["rawInput"] == {"path": "/private/file"} + assert snapshot["rawOutput"] == {"secret": "value"} + assert updated["privacy"] == "mixed" + assert "payload.snapshot.rawInput" in updated["private_fields"] + assert "payload.snapshot.rawOutput" in updated["private_fields"] + assert projector.project_turn_content("session-1")["assistant_stream_text"] == "" + + +def test_permission_request_updates_tool_and_keeps_options() -> None: + projector = AcpEventProjector() + event = projector.normalize_permission_request( + { + "jsonrpc": "2.0", + "id": 42, + "method": "session/request_permission", + "params": { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-9", "status": "pending"}, + "options": [ + {"optionId": "yes", "name": "Allow", "kind": "allow_once"}, + {"optionId": "no", "name": "Reject", "kind": "reject_once"}, + ], + }, + }, + ) + + assert event is not None + assert event["kind"] == "tool_call_update" + assert event["payload"]["permission"]["required"] is True + assert event["payload"]["permission"]["options"][1]["optionId"] == "no" + assert event["source_event_id"] == "42" + assert event["event_id"] == "acp:session-1:42" + + assert ( + projector.normalize_permission_request( + { + "jsonrpc": "2.0", + "id": 42, + "method": "session/request_permission", + "params": { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-9", "status": "pending"}, + "options": [], + }, + } + ) + is None + ) + + +def test_plan_usage_and_session_info_are_full_or_merged_snapshots() -> None: + projector = AcpEventProjector() + plan = projector.normalize_session_update( + _update( + "plan", + entries=[ + {"content": "Implement", "priority": "high", "status": "in_progress"} + ], + ) + ) + usage = projector.normalize_session_update( + _update("usage_update", used=25, size=100, cost={"amount": 0.1, "currency": "USD"}) + ) + title = projector.normalize_session_update( + _update("session_info_update", title="ACP migration") + ) + cleared = projector.normalize_session_update( + _update("session_info_update", title=None, updatedAt="2026-07-31T12:00:00Z") + ) + + assert plan is not None and plan["payload"]["snapshot"] is True + assert usage is not None and usage["payload"]["used"] == 25 + assert title is not None and title["payload"]["title"] == "ACP migration" + assert cleared is not None and cleared["payload"]["title"] is None + assert cleared["payload"]["updatedAt"] == "2026-07-31T12:00:00Z" + + +def test_explicit_event_ids_dedupe_replay_but_identical_unidentified_chunks_do_not() -> None: + projector = AcpEventProjector() + notification = _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "ha"}, + ) + + first = projector.normalize_session_update( + notification, source_event_id="transport-sequence-7", replay=True + ) + duplicate = projector.normalize_session_update( + notification, source_event_id="transport-sequence-7", replay=True + ) + repeated_text = projector.normalize_session_update(notification) + + assert first is not None and first["replay"] is True + assert duplicate is None + assert repeated_text is not None + assert projector.project_turn_content("session-1")["assistant_stream_text"] == "haha" + assert len(first["dedupe_hint"]) == 64 + + +def test_meta_event_id_is_used_as_replay_key() -> None: + projector = AcpEventProjector() + notification = _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "once"}, + _meta={"eventId": "adapter-99"}, + ) + assert projector.normalize_session_update(notification) is not None + assert projector.normalize_session_update(notification) is None + + +def test_unknown_update_is_forward_compatible_and_malformed_input_is_rejected() -> None: + projector = AcpEventProjector() + assert projector.normalize_session_update(_update("future_update", value=1)) is None + with pytest.raises(AcpProjectionError, match="sessionId"): + projector.normalize_session_update( + {"update": {"sessionUpdate": "agent_message_chunk", "content": {}}} + ) + with pytest.raises(AcpProjectionError, match="toolCallId"): + projector.normalize_permission_request( + {"sessionId": "session-1", "toolCall": {}, "options": []} + ) + + +def test_failed_normalization_does_not_consume_sequence_or_replay_id() -> None: + projector = AcpEventProjector() + malformed = _update( + "agent_message_chunk", messageId="answer-1", content="not-an-object" + ) + with pytest.raises(AcpProjectionError, match="missing content"): + projector.normalize_session_update(malformed, source_event_id="event-1") + + valid = _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "recovered"}, + ) + event = projector.normalize_session_update(valid, source_event_id="event-1") + assert event is not None + assert event["sequence"] == 1 + assert event["dedupe_safe"] is True + + +def test_sessions_have_independent_ordering_and_defensive_snapshots() -> None: + projector = AcpEventProjector() + first = projector.normalize_session_update( + _update("usage_update", used=1, size=10) + ) + other = projector.normalize_session_update( + { + "sessionId": "session-2", + "update": {"sessionUpdate": "usage_update", "used": 2, "size": 20}, + } + ) + assert first is not None and first["sequence"] == 1 + assert other is not None and other["sequence"] == 1 + + snapshot = projector.session_snapshot("session-1") + assert snapshot is not None + snapshot["usage"]["used"] = 999 + assert projector.session_snapshot("session-1")["usage"]["used"] == 1 From 39c621820e5b1e9114c01da3a00da9420399cd57 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:05:10 +0800 Subject: [PATCH 03/83] Add durable structured agent event journal --- src/tendwire/core/agent_events.py | 295 ++++++++++++++++++++++++++++ src/tendwire/store/sqlite.py | 310 +++++++++++++++++++++++++++++- tests/test_agent_events.py | 216 +++++++++++++++++++++ 3 files changed, 820 insertions(+), 1 deletion(-) create mode 100644 src/tendwire/core/agent_events.py create mode 100644 tests/test_agent_events.py diff --git a/src/tendwire/core/agent_events.py b/src/tendwire/core/agent_events.py new file mode 100644 index 0000000..5037178 --- /dev/null +++ b/src/tendwire/core/agent_events.py @@ -0,0 +1,295 @@ +"""Canonical structured events emitted by agent-protocol backends. + +The event contract deliberately keeps source identifiers and the unsanitized +payload on the private side of Tendwire's trust boundary. A separate public +projection is produced at construction time so connector code never has to +guess which source fields are safe to expose. +""" + +from __future__ import annotations + +import hashlib +import json +import math +import unicodedata +from collections.abc import Mapping +from dataclasses import dataclass +from datetime import datetime +from typing import Any, Literal + +from .models import sanitize_public_mapping, utc_timestamp + +AgentEventKind = Literal[ + "user_message", + "agent_message", + "thought", + "tool_call", + "tool_call_update", + "plan", + "usage", + "session_info", +] +AgentEventVisibility = Literal["private", "public"] + +AGENT_EVENT_KINDS = frozenset( + { + "user_message", + "agent_message", + "thought", + "tool_call", + "tool_call_update", + "plan", + "usage", + "session_info", + } +) +AGENT_EVENT_VISIBILITIES = frozenset({"private", "public"}) +AGENT_EVENT_MAX_PAYLOAD_BYTES = 64 * 1024 +AGENT_EVENT_MAX_PUBLIC_PAYLOAD_BYTES = 64 * 1024 +AGENT_EVENT_MAX_TEXT_CHARS = 32 * 1024 +AGENT_EVENT_MAX_COLLECTION_ITEMS = 256 +AGENT_EVENT_MAX_DEPTH = 12 +AGENT_EVENT_MAX_IDENTIFIER_CHARS = 2048 +AGENT_EVENT_QUERY_DEFAULT_LIMIT = 100 +AGENT_EVENT_QUERY_MAX_LIMIT = 1000 +AGENT_EVENT_SCHEMA_VERSION = 1 + + +def _canonical_json(value: Any) -> str: + return json.dumps( + value, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ) + + +def _fingerprint(value: Any) -> str: + return hashlib.sha256(_canonical_json(value).encode("utf-8")).hexdigest() + + +def _identifier(value: Any, field: str, *, required: bool = False) -> str | None: + if value is None: + if required: + raise ValueError(f"{field} must not be empty") + return None + if not isinstance(value, str): + raise ValueError(f"{field} must be text or None") + normalized = unicodedata.normalize("NFKC", value).replace("\x00", "").strip() + if not normalized: + if required: + raise ValueError(f"{field} must not be empty") + return None + if len(normalized) > AGENT_EVENT_MAX_IDENTIFIER_CHARS: + raise ValueError(f"{field} is too long") + return normalized + + +def _normalize_payload_value(value: Any, *, depth: int = 0) -> Any: + if depth > AGENT_EVENT_MAX_DEPTH: + raise ValueError("agent event payload is nested too deeply") + if value is None or isinstance(value, bool | int): + return value + if isinstance(value, float): + if not math.isfinite(value): + raise ValueError("agent event payload contains a non-finite number") + return value + if isinstance(value, datetime): + return utc_timestamp(value) + if isinstance(value, str): + normalized = unicodedata.normalize("NFKC", value).replace("\x00", "") + if len(normalized) > AGENT_EVENT_MAX_TEXT_CHARS: + raise ValueError("agent event payload text is too long") + return normalized + if isinstance(value, Mapping): + if len(value) > AGENT_EVENT_MAX_COLLECTION_ITEMS: + raise ValueError("agent event payload mapping has too many entries") + result: dict[str, Any] = {} + for raw_key, item in value.items(): + if not isinstance(raw_key, str): + raise ValueError("agent event payload keys must be text") + key = unicodedata.normalize("NFKC", raw_key).replace("\x00", "") + if not key or len(key) > 256: + raise ValueError("agent event payload contains an invalid key") + if key in result: + raise ValueError( + "agent event payload keys collide after normalization" + ) + result[key] = _normalize_payload_value(item, depth=depth + 1) + return result + if isinstance(value, tuple | list): + if len(value) > AGENT_EVENT_MAX_COLLECTION_ITEMS: + raise ValueError("agent event payload sequence has too many entries") + return [ + _normalize_payload_value(item, depth=depth + 1) for item in value + ] + raise ValueError("agent event payload must contain only JSON-safe values") + + +def normalize_agent_event_payload(payload: Mapping[str, Any]) -> dict[str, Any]: + """Return a bounded, deterministic, JSON-safe private payload.""" + if not isinstance(payload, Mapping): + raise ValueError("agent event payload must be a mapping") + normalized = _normalize_payload_value(payload) + if not isinstance(normalized, dict): # Defensive; mappings normalize to dicts. + raise ValueError("agent event payload must be a mapping") + payload_size = len(_canonical_json(normalized).encode("utf-8")) + if payload_size > AGENT_EVENT_MAX_PAYLOAD_BYTES: + raise ValueError("agent event payload is too large") + return normalized + + +def public_agent_event_payload( + payload: Mapping[str, Any], + *, + visibility: AgentEventVisibility, +) -> dict[str, Any]: + """Build the bounded public projection for an event payload.""" + if visibility == "private": + return {} + public = sanitize_public_mapping(payload, backend_neutral=True) + if ( + len(_canonical_json(public).encode("utf-8")) + > AGENT_EVENT_MAX_PUBLIC_PAYLOAD_BYTES + ): + raise ValueError("public agent event payload is too large") + return public + + +@dataclass(frozen=True) +class AgentEvent: + """One immutable structured source event before durable sequencing.""" + + event_id: str + kind: AgentEventKind + source: str + worker_id: str + visibility: AgentEventVisibility + observed_at: str + payload: dict[str, Any] + public_payload: dict[str, Any] + payload_fingerprint: str + source_session_id: str | None = None + source_turn_id: str | None = None + source_item_id: str | None = None + source_message_id: str | None = None + source_event_id: str | None = None + source_sequence: int | None = None + + def public_dict(self, *, sequence: int | None = None) -> dict[str, Any]: + """Return a connector-safe view with all source identifiers omitted.""" + result: dict[str, Any] = { + "schema_version": AGENT_EVENT_SCHEMA_VERSION, + "event_id": self.event_id, + "kind": self.kind, + "worker_id": self.worker_id, + "visibility": self.visibility, + "observed_at": self.observed_at, + "payload": dict(self.public_payload), + } + if sequence is not None: + result["sequence"] = int(sequence) + return result + + +def agent_event( + *, + kind: AgentEventKind | str, + source: str, + worker_id: str, + payload: Mapping[str, Any], + source_session_id: str | None = None, + source_turn_id: str | None = None, + source_item_id: str | None = None, + source_message_id: str | None = None, + source_event_id: str | None = None, + source_sequence: int | None = None, + visibility: AgentEventVisibility | str = "private", + observed_at: str | None = None, +) -> AgentEvent: + """Validate and construct an event with deterministic retry identity. + + Sources must provide either their event identifier or a session-local + monotonically assigned sequence. The source timestamp is intentionally + excluded from identity so replaying the same notification deduplicates. + """ + normalized_kind = str(kind).strip().lower() + if normalized_kind not in AGENT_EVENT_KINDS: + raise ValueError("unsupported agent event kind") + normalized_visibility = str(visibility).strip().lower() + if normalized_visibility not in AGENT_EVENT_VISIBILITIES: + raise ValueError("visibility must be private or public") + if normalized_kind == "thought" and normalized_visibility != "private": + raise ValueError("thought events must remain private") + normalized_source = _identifier(source, "source", required=True) + normalized_worker = _identifier(worker_id, "worker_id", required=True) + session_id = _identifier(source_session_id, "source_session_id") + turn_id = _identifier(source_turn_id, "source_turn_id") + item_id = _identifier(source_item_id, "source_item_id") + message_id = _identifier(source_message_id, "source_message_id") + event_id = _identifier(source_event_id, "source_event_id") + if source_sequence is not None and ( + isinstance(source_sequence, bool) + or not isinstance(source_sequence, int) + or source_sequence < 0 + or source_sequence > (1 << 63) - 1 + ): + raise ValueError("source_sequence must be a nonnegative SQLite integer") + if event_id is None and source_sequence is None: + raise ValueError("source_event_id or source_sequence is required") + if event_id is None and session_id is None: + raise ValueError("source_session_id is required with source_sequence") + normalized_payload = normalize_agent_event_payload(payload) + public_payload = public_agent_event_payload( + normalized_payload, + visibility=normalized_visibility, # type: ignore[arg-type] + ) + identity = { + "schema_version": AGENT_EVENT_SCHEMA_VERSION, + "source": normalized_source, + "session_id": session_id, + "event_id": event_id, + "sequence": source_sequence, + "kind": normalized_kind, + } + return AgentEvent( + event_id=_fingerprint(identity), + kind=normalized_kind, # type: ignore[arg-type] + source=normalized_source or "", + worker_id=normalized_worker or "", + visibility=normalized_visibility, # type: ignore[arg-type] + observed_at=_identifier(observed_at, "observed_at") or utc_timestamp(), + payload=normalized_payload, + public_payload=public_payload, + payload_fingerprint=_fingerprint(normalized_payload), + source_session_id=session_id, + source_turn_id=turn_id, + source_item_id=item_id, + source_message_id=message_id, + source_event_id=event_id, + source_sequence=source_sequence, + ) + + +@dataclass(frozen=True) +class StoredAgentEvent: + """One durably sequenced structured agent event.""" + + sequence: int + host_id: str + event: AgentEvent + + def public_dict(self) -> dict[str, Any]: + return self.event.public_dict(sequence=self.sequence) + + +@dataclass(frozen=True) +class AppendAgentEventResult: + sequence: int + event_id: str + inserted: bool + + +class AgentEventIdentityConflict(RuntimeError): + """The same deterministic source identity was reused for other content.""" diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index af45530..d78d4e9 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -63,6 +63,16 @@ verify_created_private_sqlite_replacement_at, verify_entry_identity, ) +from ..core.agent_events import ( + AGENT_EVENT_KINDS, + AGENT_EVENT_QUERY_DEFAULT_LIMIT, + AGENT_EVENT_QUERY_MAX_LIMIT, + AgentEvent, + AgentEventIdentityConflict, + AppendAgentEventResult, + StoredAgentEvent, + agent_event, +) from ..core.commands import ( CommandEnvelope, instruction_fingerprint, @@ -133,7 +143,7 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS -STORE_SCHEMA_VERSION = 21 +STORE_SCHEMA_VERSION = 22 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS _LEGACY_TURN_CLAIM_HARD_TTL_SECONDS = 86_400 TURN_CHANGE_RETENTION_DAYS = 7 @@ -1518,6 +1528,55 @@ def _record_response_size( ), ) +CREATE_AGENT_EVENTS_TABLE = """ +CREATE TABLE IF NOT EXISTS agent_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + host_id TEXT NOT NULL, + event_id TEXT NOT NULL, + kind TEXT NOT NULL CHECK ( + kind IN ( + 'user_message', 'agent_message', 'thought', 'tool_call', + 'tool_call_update', 'plan', 'usage', 'session_info' + ) + ), + source TEXT NOT NULL, + worker_id TEXT NOT NULL, + visibility TEXT NOT NULL CHECK (visibility IN ('private', 'public')), + source_session_id TEXT, + source_turn_id TEXT, + source_item_id TEXT, + source_message_id TEXT, + source_event_id TEXT, + source_sequence INTEGER CHECK (source_sequence >= 0), + observed_at TEXT NOT NULL, + payload_fingerprint TEXT NOT NULL, + private_payload_json TEXT NOT NULL, + public_payload_json TEXT NOT NULL, + UNIQUE (host_id, event_id), + CHECK (source_event_id IS NOT NULL OR source_sequence IS NOT NULL), + CHECK (kind != 'thought' OR visibility = 'private') +); +""" + +CREATE_AGENT_EVENT_INDEXES = ( + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_worker_sequence " + "ON agent_events(host_id, worker_id, sequence)" + ), + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_session_sequence " + "ON agent_events(host_id, source_session_id, sequence)" + ), + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_turn_sequence " + "ON agent_events(host_id, source_turn_id, sequence)" + ), + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_source_sequence " + "ON agent_events(host_id, source, sequence)" + ), +) + CREATE_PR6_TABLES = ( CREATE_EVENTS_TABLE, CREATE_SPACES_TABLE, @@ -13124,6 +13183,13 @@ def _migrate_v20_to_v21_conn(conn: sqlite3.Connection) -> None: conn.execute(statement) +def _migrate_v21_to_v22_conn(conn: sqlite3.Connection) -> None: + """Add the append-only structured agent-event journal.""" + conn.execute(CREATE_AGENT_EVENTS_TABLE) + for statement in CREATE_AGENT_EVENT_INDEXES: + conn.execute(statement) + + MIGRATIONS: tuple[Migration, ...] = ( Migration(0, 1, _migrate_v0_to_v1_conn), Migration(1, 2, _migrate_v1_to_v2_conn), @@ -13146,6 +13212,7 @@ def _migrate_v20_to_v21_conn(conn: sqlite3.Connection) -> None: Migration(18, 19, _migrate_v18_to_v19_conn), Migration(19, 20, _migrate_v19_to_v20_conn), Migration(20, 21, _migrate_v20_to_v21_conn), + Migration(21, 22, _migrate_v21_to_v22_conn), ) @@ -13199,6 +13266,7 @@ def _create_current_schema_conn(conn: sqlite3.Connection) -> None: conn.execute(CREATE_TURN_SUPERSESSIONS_TABLE) conn.execute(CREATE_HERDR_TURN_WATERMARKS_TABLE) conn.execute(CREATE_HERDR_TURN_COMPLETIONS_TABLE) + conn.execute(CREATE_AGENT_EVENTS_TABLE) for statement in CREATE_COMMAND_RECEIPT_INDEXES: conn.execute(statement) for statement in CREATE_WORKER_BINDING_INDEXES: @@ -13220,6 +13288,8 @@ def _create_current_schema_conn(conn: sqlite3.Connection) -> None: conn.execute(statement) for statement in CREATE_HERDR_TURN_INDEXES: conn.execute(statement) + for statement in CREATE_AGENT_EVENT_INDEXES: + conn.execute(statement) for statement in CREATE_ATTENTION_LIFECYCLE_INDEXES: conn.execute(statement) for statement in CREATE_TURN_CONTENT_REVISION_INDEXES: @@ -13348,6 +13418,244 @@ def init_store( ) +def _agent_event_from_row(row: tuple[Any, ...]) -> StoredAgentEvent: + private_payload = _json_object(row[15]) + public_payload = _json_object(row[16]) + kind = str(row[3]) + if kind not in AGENT_EVENT_KINDS: + raise StoreSchemaError("invalid_agent_event_kind") + return StoredAgentEvent( + sequence=int(row[0]), + host_id=str(row[1]), + event=AgentEvent( + event_id=str(row[2]), + kind=kind, # type: ignore[arg-type] + source=str(row[4]), + worker_id=str(row[5]), + visibility=str(row[6]), # type: ignore[arg-type] + source_session_id=str(row[7]) if row[7] is not None else None, + source_turn_id=str(row[8]) if row[8] is not None else None, + source_item_id=str(row[9]) if row[9] is not None else None, + source_message_id=str(row[10]) if row[10] is not None else None, + source_event_id=str(row[11]) if row[11] is not None else None, + source_sequence=int(row[12]) if row[12] is not None else None, + observed_at=str(row[13]), + payload_fingerprint=str(row[14]), + payload=private_payload, + public_payload=public_payload, + ), + ) + + +_AGENT_EVENT_SELECT = """ +SELECT + sequence, host_id, event_id, kind, source, worker_id, visibility, + source_session_id, source_turn_id, source_item_id, source_message_id, + source_event_id, source_sequence, observed_at, payload_fingerprint, + private_payload_json, public_payload_json +FROM agent_events +""" + + +def _agent_event_conflicts(existing: StoredAgentEvent, incoming: AgentEvent) -> bool: + stored = existing.event + return ( + stored.kind != incoming.kind + or stored.source != incoming.source + or stored.worker_id != incoming.worker_id + or stored.visibility != incoming.visibility + or stored.source_session_id != incoming.source_session_id + or stored.source_turn_id != incoming.source_turn_id + or stored.source_item_id != incoming.source_item_id + or stored.source_message_id != incoming.source_message_id + or stored.source_event_id != incoming.source_event_id + or stored.source_sequence != incoming.source_sequence + or stored.payload_fingerprint != incoming.payload_fingerprint + or stored.payload != incoming.payload + or stored.public_payload != incoming.public_payload + ) + + +def append_agent_event( + db_path: Path | str, + host_id: str, + event: AgentEvent, +) -> AppendAgentEventResult: + """Append one structured event, or return its existing replay sequence. + + Reusing a deterministic event identity with different content is rejected + instead of silently mutating the journal or accepting source corruption. + """ + normalized_host = str(host_id).strip() + if not normalized_host: + raise ValueError("host_id must not be empty") + if not isinstance(event, AgentEvent): + raise ValueError("event must be an AgentEvent") + canonical_event = agent_event( + kind=event.kind, + source=event.source, + worker_id=event.worker_id, + payload=event.payload, + source_session_id=event.source_session_id, + source_turn_id=event.source_turn_id, + source_item_id=event.source_item_id, + source_message_id=event.source_message_id, + source_event_id=event.source_event_id, + source_sequence=event.source_sequence, + visibility=event.visibility, + observed_at=event.observed_at, + ) + if canonical_event != event: + raise ValueError("event must use the canonical agent event contract") + private_json = _canonical_json(event.payload) + public_json = _canonical_json(event.public_payload) + with _connect(db_path, prepare=True) as conn: + _ensure_schema(conn) + conn.execute("BEGIN IMMEDIATE") + try: + cursor = conn.execute( + """ + INSERT INTO agent_events ( + host_id, event_id, kind, source, worker_id, visibility, + source_session_id, source_turn_id, source_item_id, + source_message_id, source_event_id, source_sequence, + observed_at, payload_fingerprint, private_payload_json, + public_payload_json + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(host_id, event_id) DO NOTHING + """, + ( + normalized_host, + event.event_id, + event.kind, + event.source, + event.worker_id, + event.visibility, + event.source_session_id, + event.source_turn_id, + event.source_item_id, + event.source_message_id, + event.source_event_id, + event.source_sequence, + event.observed_at, + event.payload_fingerprint, + private_json, + public_json, + ), + ) + inserted = cursor.rowcount == 1 + row = conn.execute( + _AGENT_EVENT_SELECT + + " WHERE host_id = ? AND event_id = ?", + (normalized_host, event.event_id), + ).fetchone() + if row is None: + raise StoreSchemaError("agent_event_append_failed") + stored = _agent_event_from_row(row) + if _agent_event_conflicts(stored, event): + raise AgentEventIdentityConflict(event.event_id) + conn.commit() + except Exception: + conn.rollback() + raise + return AppendAgentEventResult( + sequence=stored.sequence, + event_id=event.event_id, + inserted=inserted, + ) + + +def record_agent_event( + db_path: Path | str, + host_id: str, + **event_fields: Any, +) -> AppendAgentEventResult: + """Validate, normalize, and durably append one source event.""" + return append_agent_event(db_path, host_id, agent_event(**event_fields)) + + +def list_agent_events( + db_path: Path | str, + host_id: str, + *, + worker_id: str | None = None, + source: str | None = None, + session_id: str | None = None, + turn_id: str | None = None, + visibility: Literal["private", "public"] | None = None, + after_sequence: int = 0, + limit: int = AGENT_EVENT_QUERY_DEFAULT_LIMIT, +) -> tuple[StoredAgentEvent, ...]: + """List private structured events in append order with bounded work.""" + if ( + isinstance(after_sequence, bool) + or not isinstance(after_sequence, int) + or after_sequence < 0 + ): + raise ValueError("after_sequence must be a nonnegative integer") + if ( + isinstance(limit, bool) + or not isinstance(limit, int) + or not 1 <= limit <= AGENT_EVENT_QUERY_MAX_LIMIT + ): + raise ValueError( + f"limit must be between 1 and {AGENT_EVENT_QUERY_MAX_LIMIT}" + ) + if not _sqlite_store_exists(db_path): + return () + clauses = ["host_id = ?", "sequence > ?"] + parameters: list[Any] = [str(host_id), int(after_sequence)] + for column, value in ( + ("worker_id", worker_id), + ("source", source), + ("source_session_id", session_id), + ("source_turn_id", turn_id), + ("visibility", visibility), + ): + if value is not None: + clauses.append(f"{column} = ?") + parameters.append(str(value)) + parameters.append(int(limit)) + with _connect(db_path) as conn: + _ensure_schema(conn) + rows = conn.execute( + _AGENT_EVENT_SELECT + + " WHERE " + + " AND ".join(clauses) + + " ORDER BY sequence ASC LIMIT ?", + parameters, + ).fetchall() + return tuple(_agent_event_from_row(row) for row in rows) + + +def list_public_agent_events( + db_path: Path | str, + host_id: str, + *, + worker_id: str | None = None, + source: str | None = None, + session_id: str | None = None, + turn_id: str | None = None, + after_sequence: int = 0, + limit: int = AGENT_EVENT_QUERY_DEFAULT_LIMIT, +) -> tuple[dict[str, Any], ...]: + """List connector-safe projections without private source identifiers.""" + return tuple( + stored.public_dict() + for stored in list_agent_events( + db_path, + host_id, + worker_id=worker_id, + source=source, + session_id=session_id, + turn_id=turn_id, + visibility="public", + after_sequence=after_sequence, + limit=limit, + ) + ) + + def _herdr_turn_counter(value: Any, field: str) -> int: if ( not isinstance(value, int) diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py new file mode 100644 index 0000000..f5b2608 --- /dev/null +++ b/tests/test_agent_events.py @@ -0,0 +1,216 @@ +from __future__ import annotations + +import sqlite3 +from dataclasses import replace +from pathlib import Path + +import pytest + +from tendwire.core.agent_events import ( + AGENT_EVENT_KINDS, + AGENT_EVENT_MAX_PAYLOAD_BYTES, + AgentEventIdentityConflict, + agent_event, +) +from tendwire.store import sqlite as store_sqlite + + +def _message_event( + *, + sequence: int, + text: str = "hello", + visibility: str = "public", +): + return agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_turn_id="private-turn-1", + source_item_id="private-item-1", + source_message_id="private-message-1", + source_sequence=sequence, + visibility=visibility, + payload={"text": text}, + observed_at="2026-07-31T00:00:00+00:00", + ) + + +def test_agent_event_contract_covers_acp_primary_kinds() -> None: + assert AGENT_EVENT_KINDS == { + "user_message", + "agent_message", + "thought", + "tool_call", + "tool_call_update", + "plan", + "usage", + "session_info", + } + + +def test_append_is_ordered_and_replay_is_idempotent(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + first = _message_event(sequence=10) + second = _message_event(sequence=11, text="world") + + inserted = store_sqlite.append_agent_event(db_path, "host-1", first) + replayed = store_sqlite.append_agent_event(db_path, "host-1", first) + later = store_sqlite.append_agent_event(db_path, "host-1", second) + + assert inserted.inserted is True + assert replayed.inserted is False + assert replayed.sequence == inserted.sequence + assert later.sequence > inserted.sequence + events = store_sqlite.list_agent_events(db_path, "host-1") + assert [stored.event.payload["text"] for stored in events] == [ + "hello", + "world", + ] + + +def test_deterministic_identity_rejects_changed_replay(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + original = _message_event(sequence=4, text="original") + corrupt_replay = _message_event(sequence=4, text="changed") + assert original.event_id == corrupt_replay.event_id + store_sqlite.append_agent_event(db_path, "host-1", original) + + with pytest.raises(AgentEventIdentityConflict): + store_sqlite.append_agent_event(db_path, "host-1", corrupt_replay) + + stored = store_sqlite.list_agent_events(db_path, "host-1") + assert len(stored) == 1 + assert stored[0].event.payload == {"text": "original"} + + +def test_private_ids_and_payload_never_enter_public_projection(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + event = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_session_id="session-secret", + source_item_id="item-secret", + source_message_id="message-secret", + source_event_id="event-secret", + visibility="public", + payload={ + "text": "safe status", + "session_id": "payload-session-secret", + "cwd": "/home/smith/private-repository", + }, + ) + store_sqlite.append_agent_event(db_path, "host-1", event) + + private = store_sqlite.list_agent_events( + db_path, + "host-1", + session_id="session-secret", + ) + assert private[0].event.source_item_id == "item-secret" + assert private[0].event.payload["cwd"] == "/home/smith/private-repository" + public = store_sqlite.list_public_agent_events(db_path, "host-1") + assert public[0]["payload"] == {"text": "safe status"} + encoded = repr(public[0]) + assert "session-secret" not in encoded + assert "item-secret" not in encoded + assert "message-secret" not in encoded + assert "event-secret" not in encoded + assert "/home/smith" not in encoded + + +def test_thought_events_are_private_and_not_publicly_listed(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="thought events must remain private"): + agent_event( + kind="thought", + source="acp", + worker_id="worker-1", + source_session_id="session-1", + source_sequence=1, + visibility="public", + payload={"text": "reasoning summary"}, + ) + + db_path = tmp_path / "store.db" + thought = agent_event( + kind="thought", + source="acp", + worker_id="worker-1", + source_session_id="session-1", + source_sequence=1, + payload={"text": "reasoning summary"}, + ) + store_sqlite.append_agent_event(db_path, "host-1", thought) + assert store_sqlite.list_public_agent_events(db_path, "host-1") == () + assert store_sqlite.list_agent_events(db_path, "host-1")[0].event.payload == { + "text": "reasoning summary" + } + + +def test_queries_filter_worker_session_turn_and_cursor(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + first = _message_event(sequence=1) + second = agent_event( + kind="plan", + source="acp", + worker_id="worker-2", + source_session_id="session-2", + source_turn_id="turn-2", + source_sequence=2, + payload={"entries": [{"content": "test", "status": "pending"}]}, + ) + first_result = store_sqlite.append_agent_event(db_path, "host-1", first) + store_sqlite.append_agent_event(db_path, "host-1", second) + + assert len( + store_sqlite.list_agent_events(db_path, "host-1", worker_id="worker-1") + ) == 1 + assert len( + store_sqlite.list_agent_events( + db_path, "host-1", session_id="session-2", turn_id="turn-2" + ) + ) == 1 + after = store_sqlite.list_agent_events( + db_path, "host-1", after_sequence=first_result.sequence + ) + assert [stored.event.kind for stored in after] == ["plan"] + + +def test_payload_is_bounded_and_json_safe() -> None: + with pytest.raises(ValueError, match="payload text is too long"): + _message_event(sequence=1, text="x" * (AGENT_EVENT_MAX_PAYLOAD_BYTES + 1)) + with pytest.raises(ValueError, match="JSON-safe"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={"opaque": object()}, + ) + + +def test_store_rejects_noncanonical_public_projection(tmp_path: Path) -> None: + event = _message_event(sequence=1) + tampered = replace(event, public_payload={"session_id": "private-session"}) + with pytest.raises(ValueError, match="canonical agent event contract"): + store_sqlite.append_agent_event(tmp_path / "store.db", "host-1", tampered) + + +def test_v21_migration_is_idempotent_and_preserves_existing_store( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + store_sqlite.init_store(db_path) + with sqlite3.connect(db_path) as conn: + conn.execute("DROP TABLE agent_events") + conn.execute("PRAGMA user_version = 21") + + store_sqlite.init_store(db_path) + store_sqlite.init_store(db_path) + with sqlite3.connect(db_path) as conn: + assert conn.execute("PRAGMA user_version").fetchone() == (22,) + columns = { + str(row[1]) for row in conn.execute("PRAGMA table_info(agent_events)") + } + assert {"sequence", "event_id", "private_payload_json"} <= columns From ecd03dc64fae666a939e22b0733b9f62232506eb Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:07:46 +0800 Subject: [PATCH 04/83] Add strict ACP v1 subprocess transport --- src/tendwire/backends/acp_client.py | 931 ++++++++++++++++++++++++++ src/tendwire/backends/acp_protocol.py | 500 ++++++++++++++ tests/fixtures/acp_fake_agent.py | 150 +++++ tests/test_acp_client.py | 140 ++++ tests/test_acp_protocol.py | 117 ++++ 5 files changed, 1838 insertions(+) create mode 100644 src/tendwire/backends/acp_client.py create mode 100644 src/tendwire/backends/acp_protocol.py create mode 100644 tests/fixtures/acp_fake_agent.py create mode 100644 tests/test_acp_client.py create mode 100644 tests/test_acp_protocol.py diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py new file mode 100644 index 0000000..ddf6dc1 --- /dev/null +++ b/src/tendwire/backends/acp_client.py @@ -0,0 +1,931 @@ +"""Synchronous ACP v1 client over a supervised subprocess's stdio. + +The public client is intentionally independent from Tendwire's daemon and +persistence layers. A single reader thread owns stdout so concurrent requests, +streamed ``session/update`` notifications, and agent permission requests cannot +steal each other's frames. +""" + +from __future__ import annotations + +import math +import os +import queue +import subprocess +import threading +from collections import deque +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from enum import Enum +from pathlib import Path +from types import MappingProxyType +from typing import Any, TypeVar + +from tendwire import __version__ + +from .acp_protocol import ( + ACP_PROTOCOL_VERSION, + DEFAULT_MAX_FRAME_BYTES, + AgentCapabilities, + AcpEnvelopeError, + AcpFramingError, + AcpProtocolError, + InitializeResult, + JsonRpcNotification, + JsonRpcRequest, + JsonRpcResponse, + PermissionRequest, + PromptResult, + RequestId, + SessionInfo, + SessionPage, + SessionResult, + SessionUpdate, + StopReason, + decode_json_line, + encode_message, + error_envelope, + notification_envelope, + parse_permission_request, + parse_session_update, + request_envelope, + result_envelope, +) + +_DEFAULT_REQUEST_TIMEOUT = 30.0 +_DEFAULT_PROMPT_TIMEOUT = 60.0 * 60.0 +_DEFAULT_CLOSE_TIMEOUT = 3.0 +_DEFAULT_QUEUE_SIZE = 4096 +_DEFAULT_STDERR_LIMIT = 64 * 1024 +_METHOD_NOT_FOUND = -32601 +_INTERNAL_ERROR = -32603 + + +class AcpClientError(AcpProtocolError): + """Base error for ACP client lifecycle and transport failures.""" + + +class AcpClientStateError(AcpClientError, RuntimeError): + """An operation is invalid in the client's current state.""" + + +class AcpTransportError(AcpClientError, ConnectionError): + """The ACP subprocess or stdio stream failed.""" + + +class AcpRequestTimeoutError(AcpClientError, TimeoutError): + """A request did not receive a response before its deadline.""" + + +class AcpCapabilityError(AcpClientError, RuntimeError): + """The requested optional method was not advertised by the agent.""" + + +class AcpProtocolVersionError(AcpClientError): + """The agent selected an ACP version this client does not support.""" + + +class AcpEventQueueFullError(AcpTransportError): + """The consumer fell behind the bounded lossless event queues.""" + + +class ClientState(str, Enum): + NEW = "new" + RUNNING = "running" + INITIALIZED = "initialized" + CLOSING = "closing" + CLOSED = "closed" + FAILED = "failed" + + +@dataclass(frozen=True, slots=True) +class RawNotification: + method: str + params: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class InboundRequest: + request_id: RequestId + method: str + params: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class ProcessExit: + returncode: int + stderr_tail: str + + +_T = TypeVar("_T") +_END = object() + + +class AcpClient: + """Thread-safe, blocking ACP v1 client for one agent subprocess.""" + + def __init__( + self, + argv: Sequence[str | os.PathLike[str]], + *, + cwd: str | os.PathLike[str] | None = None, + env: Mapping[str, str] | None = None, + request_timeout: float = _DEFAULT_REQUEST_TIMEOUT, + prompt_timeout: float = _DEFAULT_PROMPT_TIMEOUT, + close_timeout: float = _DEFAULT_CLOSE_TIMEOUT, + max_frame_bytes: int = DEFAULT_MAX_FRAME_BYTES, + max_pending_events: int = _DEFAULT_QUEUE_SIZE, + stderr_limit_bytes: int = _DEFAULT_STDERR_LIMIT, + ) -> None: + command = tuple(os.fspath(item) for item in argv) + if not command or any(not item for item in command): + raise ValueError("argv must contain at least one non-empty argument") + self.argv = command + self.cwd = os.fspath(cwd) if cwd is not None else None + self.env = dict(env) if env is not None else None + self.request_timeout = _positive_timeout(request_timeout, "request_timeout") + self.prompt_timeout = _positive_timeout(prompt_timeout, "prompt_timeout") + self.close_timeout = _positive_timeout(close_timeout, "close_timeout") + if max_frame_bytes <= 0: + raise ValueError("max_frame_bytes must be positive") + if max_pending_events <= 0: + raise ValueError("max_pending_events must be positive") + if stderr_limit_bytes <= 0: + raise ValueError("stderr_limit_bytes must be positive") + self.max_frame_bytes = max_frame_bytes + self.max_pending_events = max_pending_events + self.stderr_limit_bytes = stderr_limit_bytes + + self._state = ClientState.NEW + self._process: subprocess.Popen[bytes] | None = None + self._state_lock = threading.RLock() + self._write_lock = threading.Lock() + self._request_id_lock = threading.Lock() + self._next_id = 1 + self._pending: dict[RequestId, queue.Queue[JsonRpcResponse | BaseException]] = {} + self._pending_lock = threading.Lock() + self._pending_permissions: dict[RequestId, PermissionRequest] = {} + self._permission_lock = threading.Lock() + self._updates: queue.Queue[SessionUpdate | object] = queue.Queue(max_pending_events) + self._permissions: queue.Queue[PermissionRequest | object] = queue.Queue( + max_pending_events + ) + self._notifications: queue.Queue[RawNotification | object] = queue.Queue( + max_pending_events + ) + self._inbound_requests: queue.Queue[InboundRequest | object] = queue.Queue( + max_pending_events + ) + self._reader_thread: threading.Thread | None = None + self._stderr_thread: threading.Thread | None = None + self._stop = threading.Event() + self._stderr_chunks: deque[bytes] = deque() + self._stderr_size = 0 + self._stderr_lock = threading.Lock() + self._failure: BaseException | None = None + self._exit: ProcessExit | None = None + self._initialize_result: InitializeResult | None = None + + def __enter__(self) -> "AcpClient": + self.start() + return self + + def __exit__(self, exc_type: object, exc: object, tb: object) -> None: + self.close() + + @property + def state(self) -> ClientState: + with self._state_lock: + return self._state + + @property + def process(self) -> subprocess.Popen[bytes] | None: + return self._process + + @property + def capabilities(self) -> AgentCapabilities | None: + result = self._initialize_result + return result.capabilities if result is not None else None + + @property + def initialize_result(self) -> InitializeResult | None: + return self._initialize_result + + @property + def failure(self) -> BaseException | None: + return self._failure + + @property + def exit(self) -> ProcessExit | None: + process = self._process + if process is not None: + returncode = process.poll() + if returncode is not None: + self._exit = ProcessExit(returncode, self.stderr_tail()) + return self._exit + + def stderr_tail(self) -> str: + with self._stderr_lock: + data = b"".join(self._stderr_chunks) + return data.decode("utf-8", errors="replace") + + def start(self) -> "AcpClient": + with self._state_lock: + if self._state in {ClientState.RUNNING, ClientState.INITIALIZED}: + return self + if self._state is not ClientState.NEW: + raise AcpClientStateError(f"cannot start ACP client in state {self._state.value}") + try: + process = subprocess.Popen( + self.argv, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + cwd=self.cwd, + env=self.env, + bufsize=0, + close_fds=True, + ) + except OSError as exc: + self._state = ClientState.FAILED + self._failure = AcpTransportError( + f"could not start ACP agent {self.argv[0]!r}" + ) + raise self._failure from exc + self._process = process + self._state = ClientState.RUNNING + self._reader_thread = threading.Thread( + target=self._reader_main, + name=f"acp-reader-{process.pid}", + daemon=True, + ) + self._stderr_thread = threading.Thread( + target=self._stderr_main, + name=f"acp-stderr-{process.pid}", + daemon=True, + ) + self._reader_thread.start() + self._stderr_thread.start() + return self + + def initialize( + self, + *, + client_capabilities: Mapping[str, Any] | None = None, + client_name: str = "tendwire", + client_version: str = __version__, + client_title: str = "Tendwire", + timeout: float | None = None, + ) -> InitializeResult: + """Perform ACP v1 capability negotiation. + + ACP v1 does not define a post-response ``initialized`` notification; + :meth:`initialized` is available only for adapters that explicitly + require that compatibility extension. + """ + self.start() + if not client_name or not client_version: + raise ValueError("client_name and client_version must be non-empty") + with self._state_lock: + if self._state is ClientState.INITIALIZED: + assert self._initialize_result is not None + return self._initialize_result + if self._state is not ClientState.RUNNING: + self._raise_unusable() + client_info: dict[str, Any] = { + "name": client_name, + "version": client_version, + } + if client_title: + client_info["title"] = client_title + result = self.request( + "initialize", + { + "protocolVersion": ACP_PROTOCOL_VERSION, + "clientCapabilities": dict(client_capabilities or {}), + "clientInfo": client_info, + }, + timeout=timeout, + require_initialized=False, + ) + raw = _require_mapping(result, "initialize result") + version = raw.get("protocolVersion") + if version != ACP_PROTOCOL_VERSION: + raise AcpProtocolVersionError( + f"agent selected unsupported ACP protocol version {version!r}" + ) + capabilities_value = raw.get("agentCapabilities", {}) + if not isinstance(capabilities_value, Mapping): + capabilities_value = {} + agent_info_value = raw.get("agentInfo") + agent_info = ( + MappingProxyType(dict(agent_info_value)) + if isinstance(agent_info_value, Mapping) + else None + ) + auth_methods_value = raw.get("authMethods", []) + auth_methods = tuple( + MappingProxyType(dict(item)) + for item in auth_methods_value + if isinstance(item, Mapping) + ) if isinstance(auth_methods_value, list) else () + parsed = InitializeResult( + protocol_version=version, + capabilities=AgentCapabilities.from_mapping(capabilities_value), + agent_info=agent_info, + auth_methods=auth_methods, + raw=MappingProxyType(dict(raw)), + ) + with self._state_lock: + if self._state is not ClientState.RUNNING: + self._raise_unusable() + self._initialize_result = parsed + self._state = ClientState.INITIALIZED + return parsed + + def initialized(self) -> None: + """Send the non-standard ``initialized`` compatibility notification.""" + self._require_initialized() + self.notify("initialized", {}) + + def request( + self, + method: str, + params: Mapping[str, Any] | None = None, + *, + timeout: float | None = None, + require_initialized: bool = True, + ) -> Any: + if require_initialized: + self._require_initialized() + else: + self._require_running() + wait_timeout = self.request_timeout if timeout is None else _positive_timeout( + timeout, "timeout" + ) + request_id = self._new_request_id() + waiter: queue.Queue[JsonRpcResponse | BaseException] = queue.Queue(maxsize=1) + with self._pending_lock: + self._pending[request_id] = waiter + try: + self._write(request_envelope(request_id, method, params)) + except BaseException: + with self._pending_lock: + self._pending.pop(request_id, None) + raise + try: + response = waiter.get(timeout=wait_timeout) + except queue.Empty as exc: + with self._pending_lock: + self._pending.pop(request_id, None) + raise AcpRequestTimeoutError( + f"ACP request {method!r} timed out after {wait_timeout:g}s" + ) from exc + if isinstance(response, BaseException): + raise response + return response.result_or_raise() + + def notify(self, method: str, params: Mapping[str, Any] | None = None) -> None: + self._require_running() + self._write(notification_envelope(method, params)) + + def new_session( + self, + cwd: str | os.PathLike[str], + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[str | os.PathLike[str]] = (), + timeout: float | None = None, + ) -> SessionResult: + params = self._session_setup_params( + cwd, + mcp_servers=mcp_servers, + additional_directories=additional_directories, + ) + result = self.request("session/new", params, timeout=timeout) + raw = _require_mapping(result, "session/new result") + return _parse_session_result(raw, require_session_id=True) + + def load_session( + self, + session_id: str, + cwd: str | os.PathLike[str], + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[str | os.PathLike[str]] = (), + timeout: float | None = None, + ) -> SessionResult: + self._require_capability("loadSession") + params = self._session_setup_params( + cwd, + mcp_servers=mcp_servers, + additional_directories=additional_directories, + ) + params["sessionId"] = _nonempty(session_id, "session_id") + result = self.request("session/load", params, timeout=timeout) + raw = _require_mapping(result, "session/load result") + parsed = _parse_session_result(raw, require_session_id=False) + return SessionResult(session_id, parsed.modes, parsed.config_options, parsed.raw) + + def resume_session( + self, + session_id: str, + cwd: str | os.PathLike[str], + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[str | os.PathLike[str]] = (), + timeout: float | None = None, + ) -> SessionResult: + self._require_capability("sessionResume") + params = self._session_setup_params( + cwd, + mcp_servers=mcp_servers, + additional_directories=additional_directories, + ) + params["sessionId"] = _nonempty(session_id, "session_id") + result = self.request("session/resume", params, timeout=timeout) + raw = _require_mapping(result, "session/resume result") + parsed = _parse_session_result(raw, require_session_id=False) + return SessionResult(session_id, parsed.modes, parsed.config_options, parsed.raw) + + def list_sessions( + self, + *, + cwd: str | os.PathLike[str] | None = None, + cursor: str | None = None, + timeout: float | None = None, + ) -> SessionPage: + self._require_capability("sessionList") + params: dict[str, Any] = {} + if cwd is not None: + params["cwd"] = _absolute_path(cwd, "cwd") + if cursor is not None: + params["cursor"] = _nonempty(cursor, "cursor") + result = self.request("session/list", params, timeout=timeout) + raw = _require_mapping(result, "session/list result") + raw_sessions = raw.get("sessions") + if not isinstance(raw_sessions, list): + raise AcpEnvelopeError("session/list result.sessions must be an array") + sessions = tuple(_parse_session_info(item) for item in raw_sessions) + next_cursor = raw.get("nextCursor") + if next_cursor is not None and not isinstance(next_cursor, str): + next_cursor = None + return SessionPage(sessions, next_cursor, MappingProxyType(dict(raw))) + + def prompt( + self, + session_id: str, + prompt: str | Sequence[Mapping[str, Any]], + *, + timeout: float | None = None, + ) -> PromptResult: + if isinstance(prompt, str): + content: list[Mapping[str, Any]] = [{"type": "text", "text": prompt}] + else: + content = list(prompt) + if not content: + raise ValueError("prompt must contain at least one content block") + for block in content: + if not isinstance(block, Mapping) or not isinstance(block.get("type"), str): + raise ValueError("each prompt content block must have a string type") + result = self.request( + "session/prompt", + {"sessionId": _nonempty(session_id, "session_id"), "prompt": content}, + timeout=self.prompt_timeout if timeout is None else timeout, + ) + raw = _require_mapping(result, "session/prompt result") + stop_reason = raw.get("stopReason") + try: + parsed_reason = StopReason(stop_reason) + except (ValueError, TypeError) as exc: + raise AcpEnvelopeError("session/prompt returned an invalid stopReason") from exc + return PromptResult(parsed_reason, MappingProxyType(dict(raw))) + + def cancel(self, session_id: str) -> None: + """Cancel a turn and cancel all outstanding permissions for the session.""" + session_id = _nonempty(session_id, "session_id") + self.notify("session/cancel", {"sessionId": session_id}) + with self._permission_lock: + pending_ids = [ + request_id + for request_id, request in self._pending_permissions.items() + if request.session_id == session_id + ] + for request_id in pending_ids: + self.respond_permission(request_id, cancelled=True) + + def respond_permission( + self, + request_id: RequestId, + *, + option_id: str | None = None, + cancelled: bool = False, + ) -> None: + if cancelled == (option_id is not None): + raise ValueError("select exactly one of option_id or cancelled=True") + with self._permission_lock: + request = self._pending_permissions.get(request_id) + if request is None: + raise AcpClientStateError("permission request is not pending") + if option_id is not None and option_id not in { + option.option_id for option in request.options + }: + raise ValueError("option_id was not offered by this permission request") + del self._pending_permissions[request_id] + if cancelled: + outcome = {"outcome": "cancelled"} + else: + outcome = {"outcome": "selected", "optionId": option_id} + try: + self._write(result_envelope(request_id, {"outcome": outcome})) + except BaseException: + # Preserve retryability when nothing was written successfully. + with self._permission_lock: + self._pending_permissions[request_id] = request + raise + + def next_update(self, *, timeout: float | None = None) -> SessionUpdate: + return self._queue_get(self._updates, timeout, "session update") + + def next_permission_request( + self, *, timeout: float | None = None + ) -> PermissionRequest: + return self._queue_get(self._permissions, timeout, "permission request") + + def next_notification(self, *, timeout: float | None = None) -> RawNotification: + return self._queue_get(self._notifications, timeout, "notification") + + def next_inbound_request(self, *, timeout: float | None = None) -> InboundRequest: + return self._queue_get(self._inbound_requests, timeout, "inbound request") + + def reject_inbound_request( + self, + request_id: RequestId, + *, + code: int = _METHOD_NOT_FOUND, + message: str = "Method not supported by Tendwire ACP client", + ) -> None: + self._write(error_envelope(request_id, code, message)) + + def close(self) -> None: + with self._state_lock: + if self._state in {ClientState.CLOSED, ClientState.NEW}: + self._state = ClientState.CLOSED + return + if self._state is ClientState.CLOSING: + return + was_failed = self._state is ClientState.FAILED + self._state = ClientState.CLOSING + self._stop.set() + process = self._process + if process is not None: + with self._write_lock: + if process.stdin is not None: + try: + process.stdin.close() + except OSError: + pass + try: + process.wait(timeout=self.close_timeout) + except subprocess.TimeoutExpired: + process.terminate() + try: + process.wait(timeout=self.close_timeout) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=self.close_timeout) + self._exit = ProcessExit(process.returncode, self.stderr_tail()) + for thread in (self._reader_thread, self._stderr_thread): + if thread is not None and thread is not threading.current_thread(): + thread.join(timeout=self.close_timeout) + self._fail_pending(AcpTransportError("ACP client closed")) + self._signal_queues() + with self._state_lock: + self._state = ClientState.FAILED if was_failed else ClientState.CLOSED + + def _session_setup_params( + self, + cwd: str | os.PathLike[str], + *, + mcp_servers: Sequence[Mapping[str, Any]], + additional_directories: Sequence[str | os.PathLike[str]], + ) -> dict[str, Any]: + self._require_initialized() + params: dict[str, Any] = { + "cwd": _absolute_path(cwd, "cwd"), + "mcpServers": [dict(server) for server in mcp_servers], + } + directories = [ + _absolute_path(directory, "additional directory") + for directory in additional_directories + ] + if directories: + self._require_capability("additionalDirectories") + params["additionalDirectories"] = directories + return params + + def _require_capability(self, name: str) -> None: + self._require_initialized() + assert self.capabilities is not None + supported = { + "loadSession": self.capabilities.load_session, + "sessionList": self.capabilities.session_list, + "sessionResume": self.capabilities.session_resume, + "additionalDirectories": self.capabilities.additional_directories, + }.get(name, False) + if not supported: + raise AcpCapabilityError(f"agent did not advertise {name} capability") + + def _new_request_id(self) -> int: + with self._request_id_lock: + request_id = self._next_id + self._next_id += 1 + return request_id + + def _write(self, envelope: Mapping[str, Any]) -> None: + payload = encode_message(envelope, max_frame_bytes=self.max_frame_bytes) + with self._write_lock: + self._require_running() + process = self._process + if process is None or process.stdin is None: + raise AcpTransportError("ACP agent stdin is unavailable") + try: + remaining = memoryview(payload) + while remaining: + written = os.write(process.stdin.fileno(), remaining) + if written <= 0: + raise BrokenPipeError("zero-byte write to ACP agent stdin") + remaining = remaining[written:] + except (BrokenPipeError, OSError) as exc: + failure = AcpTransportError("ACP agent stdin disconnected") + self._set_failed(failure) + raise failure from exc + + def _reader_main(self) -> None: + process = self._process + assert process is not None and process.stdout is not None + buffer = bytearray() + try: + while not self._stop.is_set(): + newline = buffer.find(b"\n") + if newline < 0: + room = self.max_frame_bytes + 1 - len(buffer) + chunk = os.read(process.stdout.fileno(), min(64 * 1024, room)) + if not chunk: + if self._stop.is_set(): + return + if buffer: + raise AcpFramingError("ACP stdout ended during a JSON frame") + returncode = process.poll() + suffix = f" (exit {returncode})" if returncode is not None else "" + raise AcpTransportError(f"ACP agent stdout disconnected{suffix}") + buffer.extend(chunk) + if len(buffer) > self.max_frame_bytes and b"\n" not in buffer: + raise AcpFramingError( + "ACP stdout frame exceeds configured size limit" + ) + continue + line = bytes(buffer[: newline + 1]) + del buffer[: newline + 1] + if len(line) > self.max_frame_bytes: + raise AcpFramingError("ACP stdout frame exceeds configured size limit") + message = decode_json_line( + line, + max_frame_bytes=self.max_frame_bytes, + ) + self._dispatch(message) + except BaseException as exc: + if not self._stop.is_set(): + self._set_failed(exc) + + def _stderr_main(self) -> None: + process = self._process + assert process is not None and process.stderr is not None + try: + while not self._stop.is_set(): + chunk = os.read(process.stderr.fileno(), 4096) + if not chunk: + return + with self._stderr_lock: + self._stderr_chunks.append(chunk) + self._stderr_size += len(chunk) + while self._stderr_size > self.stderr_limit_bytes: + excess = self._stderr_size - self.stderr_limit_bytes + first = self._stderr_chunks[0] + if len(first) <= excess: + self._stderr_size -= len(self._stderr_chunks.popleft()) + else: + self._stderr_chunks[0] = first[excess:] + self._stderr_size -= excess + except OSError: + return + + def _dispatch( + self, message: JsonRpcRequest | JsonRpcNotification | JsonRpcResponse + ) -> None: + if isinstance(message, JsonRpcResponse): + if message.request_id is None: + raise AcpEnvelopeError("uncorrelated ACP response with null id") + with self._pending_lock: + waiter = self._pending.pop(message.request_id, None) + if waiter is None: + # A late response after timeout cannot be safely correlated to a + # live operation. Keep the transport usable and surface it as a + # raw diagnostic notification. + self._put_lossless( + self._notifications, + RawNotification( + "$/orphan_response", + MappingProxyType({"id": message.request_id}), + ), + ) + return + waiter.put_nowait(message) + return + if isinstance(message, JsonRpcNotification): + if message.method == "session/update": + self._put_lossless(self._updates, parse_session_update(message.params)) + else: + self._put_lossless( + self._notifications, + RawNotification(message.method, message.params), + ) + return + + if message.method == "session/request_permission": + try: + parsed = parse_permission_request(message) + except AcpProtocolError as exc: + self._write( + error_envelope( + message.request_id, + -32602, + "Invalid permission request", + data=str(exc), + ) + ) + return + with self._permission_lock: + if message.request_id in self._pending_permissions: + raise AcpEnvelopeError("duplicate pending permission request id") + self._pending_permissions[message.request_id] = parsed + self._put_lossless(self._permissions, parsed) + else: + self._put_lossless( + self._inbound_requests, + InboundRequest(message.request_id, message.method, message.params), + ) + + def _put_lossless(self, target: queue.Queue[Any], value: Any) -> None: + try: + target.put_nowait(value) + except queue.Full as exc: + raise AcpEventQueueFullError( + "ACP event queue is full; refusing to drop protocol data" + ) from exc + + def _queue_get( + self, + source: queue.Queue[_T | object], + timeout: float | None, + description: str, + ) -> _T: + if timeout is not None: + timeout = _positive_timeout(timeout, "timeout") + try: + item = source.get(timeout=timeout) + except queue.Empty as exc: + raise AcpRequestTimeoutError(f"timed out waiting for ACP {description}") from exc + if item is _END: + self._raise_unusable() + raise AcpTransportError("ACP event stream ended") + return item # type: ignore[return-value] + + def _set_failed(self, failure: BaseException) -> None: + if not isinstance(failure, AcpClientError | AcpProtocolError): + failure = AcpTransportError(str(failure)) + with self._state_lock: + if self._state in {ClientState.CLOSING, ClientState.CLOSED}: + return + self._failure = failure + self._state = ClientState.FAILED + self._stop.set() + self._fail_pending(failure) + self._signal_queues() + + def _fail_pending(self, failure: BaseException) -> None: + with self._pending_lock: + waiters = tuple(self._pending.values()) + self._pending.clear() + for waiter in waiters: + try: + waiter.put_nowait(failure) + except queue.Full: + pass + + def _signal_queues(self) -> None: + for target in ( + self._updates, + self._permissions, + self._notifications, + self._inbound_requests, + ): + try: + target.put_nowait(_END) + except queue.Full: + # A full queue already has readable data; the recorded client + # state reports terminal failure once consumers drain it. + pass + + def _require_running(self) -> None: + with self._state_lock: + if self._state not in {ClientState.RUNNING, ClientState.INITIALIZED}: + self._raise_unusable() + + def _require_initialized(self) -> None: + with self._state_lock: + if self._state is not ClientState.INITIALIZED: + if self._state in {ClientState.FAILED, ClientState.CLOSED, ClientState.CLOSING}: + self._raise_unusable() + raise AcpClientStateError("ACP client has not been initialized") + + def _raise_unusable(self) -> None: + if self._failure is not None: + raise AcpTransportError(f"ACP client failed: {self._failure}") from self._failure + raise AcpClientStateError(f"ACP client is {self._state.value}") + + +def _positive_timeout(value: float | int, name: str) -> float: + result = float(value) + if not math.isfinite(result) or result <= 0: + raise ValueError(f"{name} must be finite and positive") + return result + + +def _nonempty(value: str, name: str) -> str: + if not isinstance(value, str) or not value: + raise ValueError(f"{name} must be a non-empty string") + return value + + +def _absolute_path(value: str | os.PathLike[str], name: str) -> str: + result = os.fspath(value) + if not result or not Path(result).is_absolute(): + raise ValueError(f"{name} must be an absolute path") + return result + + +def _require_mapping(value: Any, name: str) -> Mapping[str, Any]: + if not isinstance(value, Mapping): + raise AcpEnvelopeError(f"{name} must be an object") + return value + + +def _parse_session_result( + raw: Mapping[str, Any], *, require_session_id: bool +) -> SessionResult: + session_id = raw.get("sessionId", "") + if require_session_id and (not isinstance(session_id, str) or not session_id): + raise AcpEnvelopeError("session/new result.sessionId must be a non-empty string") + modes_value = raw.get("modes") + modes = ( + MappingProxyType(dict(modes_value)) if isinstance(modes_value, Mapping) else None + ) + config_value = raw.get("configOptions", []) + config_options = tuple( + MappingProxyType(dict(item)) + for item in config_value + if isinstance(item, Mapping) + ) if isinstance(config_value, list) else () + return SessionResult( + session_id=session_id if isinstance(session_id, str) else "", + modes=modes, + config_options=config_options, + raw=MappingProxyType(dict(raw)), + ) + + +def _parse_session_info(value: Any) -> SessionInfo: + raw = _require_mapping(value, "session info") + session_id = raw.get("sessionId") + cwd = raw.get("cwd") + if not isinstance(session_id, str) or not session_id: + raise AcpEnvelopeError("session info.sessionId must be a non-empty string") + if not isinstance(cwd, str) or not Path(cwd).is_absolute(): + raise AcpEnvelopeError("session info.cwd must be an absolute path") + directories_value = raw.get("additionalDirectories", []) + directories = tuple( + item + for item in directories_value + if isinstance(item, str) and Path(item).is_absolute() + ) if isinstance(directories_value, list) else () + title = raw.get("title") + updated_at = raw.get("updatedAt") + return SessionInfo( + session_id=session_id, + cwd=cwd, + additional_directories=directories, + title=title if isinstance(title, str) else None, + updated_at=updated_at if isinstance(updated_at, str) else None, + raw=MappingProxyType(dict(raw)), + ) diff --git a/src/tendwire/backends/acp_protocol.py b/src/tendwire/backends/acp_protocol.py new file mode 100644 index 0000000..b01776a --- /dev/null +++ b/src/tendwire/backends/acp_protocol.py @@ -0,0 +1,500 @@ +"""Strict, stdlib-only primitives for the ACP v1 stdio protocol. + +ACP uses JSON-RPC 2.0 messages delimited by newlines. This module deliberately +does not know about processes or threads; :mod:`acp_client` owns that transport +lifecycle and uses the validated envelopes defined here. +""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from dataclasses import dataclass +from enum import Enum +from types import MappingProxyType +from typing import Any, TypeAlias + +JSONRPC_VERSION = "2.0" +ACP_PROTOCOL_VERSION = 1 +DEFAULT_MAX_FRAME_BYTES = 8 * 1024 * 1024 + +RequestId: TypeAlias = str | int + + +class AcpProtocolError(Exception): + """Base class for ACP framing and envelope errors.""" + + +class AcpFramingError(AcpProtocolError, ValueError): + """A stdio frame is incomplete, oversized, or not valid JSON.""" + + +class AcpEnvelopeError(AcpProtocolError, ValueError): + """A decoded value is not a valid JSON-RPC 2.0 envelope.""" + + +class AcpRemoteError(AcpProtocolError): + """A correlated JSON-RPC error returned by the ACP agent.""" + + def __init__( + self, + code: int, + message: str, + *, + request_id: RequestId | None, + data: Any = None, + ) -> None: + self.code = code + self.message = message + self.request_id = request_id + self.data = data + super().__init__(f"ACP error {code}: {message}") + + +class MessageKind(str, Enum): + REQUEST = "request" + RESPONSE = "response" + NOTIFICATION = "notification" + + +class SessionUpdateKind(str, Enum): + USER_MESSAGE_CHUNK = "user_message_chunk" + AGENT_MESSAGE_CHUNK = "agent_message_chunk" + AGENT_THOUGHT_CHUNK = "agent_thought_chunk" + TOOL_CALL = "tool_call" + TOOL_CALL_UPDATE = "tool_call_update" + PLAN = "plan" + AVAILABLE_COMMANDS_UPDATE = "available_commands_update" + CURRENT_MODE_UPDATE = "current_mode_update" + CONFIG_OPTION_UPDATE = "config_option_update" + SESSION_INFO_UPDATE = "session_info_update" + USAGE_UPDATE = "usage_update" + + +class StopReason(str, Enum): + END_TURN = "end_turn" + MAX_TOKENS = "max_tokens" + MAX_TURN_REQUESTS = "max_turn_requests" + REFUSAL = "refusal" + CANCELLED = "cancelled" + + +class PermissionOptionKind(str, Enum): + ALLOW_ONCE = "allow_once" + ALLOW_ALWAYS = "allow_always" + REJECT_ONCE = "reject_once" + REJECT_ALWAYS = "reject_always" + + +@dataclass(frozen=True, slots=True) +class JsonRpcRequest: + request_id: RequestId + method: str + params: Mapping[str, Any] + + @property + def kind(self) -> MessageKind: + return MessageKind.REQUEST + + +@dataclass(frozen=True, slots=True) +class JsonRpcNotification: + method: str + params: Mapping[str, Any] + + @property + def kind(self) -> MessageKind: + return MessageKind.NOTIFICATION + + +@dataclass(frozen=True, slots=True) +class JsonRpcResponse: + request_id: RequestId | None + result: Any = None + error: Mapping[str, Any] | None = None + + @property + def kind(self) -> MessageKind: + return MessageKind.RESPONSE + + def result_or_raise(self) -> Any: + if self.error is None: + return self.result + raise AcpRemoteError( + self.error["code"], + self.error["message"], + request_id=self.request_id, + data=self.error.get("data"), + ) + + +JsonRpcMessage: TypeAlias = JsonRpcRequest | JsonRpcNotification | JsonRpcResponse + + +@dataclass(frozen=True, slots=True) +class AgentCapabilities: + """Captured ACP capabilities with convenient stable-v1 feature checks.""" + + raw: Mapping[str, Any] + + @classmethod + def from_mapping(cls, value: Mapping[str, Any] | None) -> "AgentCapabilities": + return cls(_freeze_mapping(value or {})) + + @property + def load_session(self) -> bool: + return self.raw.get("loadSession") is True + + @property + def session_list(self) -> bool: + return _is_capability_object(self._session_capabilities().get("list")) + + @property + def session_resume(self) -> bool: + return _is_capability_object(self._session_capabilities().get("resume")) + + @property + def additional_directories(self) -> bool: + return _is_capability_object( + self._session_capabilities().get("additionalDirectories") + ) + + def _session_capabilities(self) -> Mapping[str, Any]: + value = self.raw.get("sessionCapabilities") + return value if isinstance(value, Mapping) else {} + + +@dataclass(frozen=True, slots=True) +class InitializeResult: + protocol_version: int + capabilities: AgentCapabilities + agent_info: Mapping[str, Any] | None + auth_methods: tuple[Mapping[str, Any], ...] + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class SessionResult: + session_id: str + modes: Mapping[str, Any] | None + config_options: tuple[Mapping[str, Any], ...] + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class SessionInfo: + session_id: str + cwd: str + additional_directories: tuple[str, ...] + title: str | None + updated_at: str | None + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class SessionPage: + sessions: tuple[SessionInfo, ...] + next_cursor: str | None + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class PromptResult: + stop_reason: StopReason + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class SessionUpdate: + session_id: str + update_kind: SessionUpdateKind | str + update: Mapping[str, Any] + meta: Mapping[str, Any] | None + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class PermissionOption: + option_id: str + name: str + kind: PermissionOptionKind | str + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class PermissionRequest: + request_id: RequestId + session_id: str + tool_call: Mapping[str, Any] + options: tuple[PermissionOption, ...] + meta: Mapping[str, Any] | None + raw: Mapping[str, Any] + + +def _freeze_mapping(value: Mapping[str, Any]) -> Mapping[str, Any]: + # A shallow immutable copy is intentional: arbitrary extension payloads are + # retained verbatim and callers should treat all raw values as read-only. + return MappingProxyType(dict(value)) + + +def _is_capability_object(value: Any) -> bool: + # ACP advertises optional capabilities with an object (often simply {}). + return isinstance(value, Mapping) + + +def _valid_request_id(value: Any) -> bool: + return (isinstance(value, str) and bool(value)) or ( + isinstance(value, int) and not isinstance(value, bool) + ) + + +def _reject_constant(value: str) -> None: + raise ValueError(f"non-finite JSON number {value}") + + +def _object_without_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate JSON object key {key!r}") + result[key] = value + return result + + +def decode_json_line( + line: bytes | bytearray | memoryview | str, + *, + max_frame_bytes: int = DEFAULT_MAX_FRAME_BYTES, + require_newline: bool = True, +) -> JsonRpcMessage: + """Decode and validate exactly one newline-delimited JSON-RPC message.""" + if max_frame_bytes <= 0: + raise ValueError("max_frame_bytes must be positive") + if isinstance(line, str): + try: + raw = line.encode("utf-8") + except UnicodeEncodeError as exc: + raise AcpFramingError("ACP frame is not valid UTF-8") from exc + else: + raw = bytes(line) + if not raw: + raise AcpFramingError("empty ACP frame") + if len(raw) > max_frame_bytes: + raise AcpFramingError("ACP frame exceeds configured size limit") + if require_newline and not raw.endswith(b"\n"): + raise AcpFramingError("ACP frame is not newline terminated") + payload = raw[:-1] if raw.endswith(b"\n") else raw + if payload.endswith(b"\r"): + payload = payload[:-1] + if b"\n" in payload or b"\r" in payload: + raise AcpFramingError("ACP frame contains an embedded line break") + if not payload: + raise AcpFramingError("empty ACP JSON payload") + try: + text = payload.decode("utf-8", errors="strict") + value = json.loads( + text, + parse_constant=_reject_constant, + object_pairs_hook=_object_without_duplicates, + ) + except (UnicodeDecodeError, json.JSONDecodeError, ValueError) as exc: + raise AcpFramingError("ACP frame is not strict UTF-8 JSON") from exc + return validate_envelope(value) + + +def validate_envelope(value: Any) -> JsonRpcMessage: + """Validate a decoded JSON-RPC 2.0 object and return a typed envelope.""" + if not isinstance(value, Mapping): + raise AcpEnvelopeError("ACP JSON-RPC envelope must be an object") + if value.get("jsonrpc") != JSONRPC_VERSION: + raise AcpEnvelopeError("ACP envelope must declare jsonrpc '2.0'") + + has_method = "method" in value + has_id = "id" in value + has_result = "result" in value + has_error = "error" in value + + if has_method: + if has_result or has_error: + raise AcpEnvelopeError("JSON-RPC call cannot contain result or error") + method = value["method"] + if not isinstance(method, str) or not method: + raise AcpEnvelopeError("JSON-RPC method must be a non-empty string") + params = value.get("params", {}) + if not isinstance(params, Mapping): + raise AcpEnvelopeError("ACP method params must be an object") + frozen_params = _freeze_mapping(params) + if not has_id: + return JsonRpcNotification(method=method, params=frozen_params) + request_id = value["id"] + if not _valid_request_id(request_id): + raise AcpEnvelopeError("JSON-RPC request id must be a non-empty string or integer") + return JsonRpcRequest( + request_id=request_id, + method=method, + params=frozen_params, + ) + + if not has_id: + raise AcpEnvelopeError("JSON-RPC response must contain an id") + request_id = value["id"] + if request_id is not None and not _valid_request_id(request_id): + raise AcpEnvelopeError("JSON-RPC response id is invalid") + if has_result == has_error: + raise AcpEnvelopeError("JSON-RPC response must contain exactly one of result or error") + if has_error: + error = value["error"] + if not isinstance(error, Mapping): + raise AcpEnvelopeError("JSON-RPC error must be an object") + code = error.get("code") + message = error.get("message") + if not isinstance(code, int) or isinstance(code, bool): + raise AcpEnvelopeError("JSON-RPC error code must be an integer") + if not isinstance(message, str): + raise AcpEnvelopeError("JSON-RPC error message must be a string") + return JsonRpcResponse( + request_id=request_id, + error=_freeze_mapping(error), + ) + return JsonRpcResponse(request_id=request_id, result=value["result"]) + + +def encode_message( + value: Mapping[str, Any], + *, + max_frame_bytes: int = DEFAULT_MAX_FRAME_BYTES, +) -> bytes: + """Validate and encode a JSON-RPC object as one bounded UTF-8 line.""" + validate_envelope(value) + try: + payload = json.dumps( + value, + ensure_ascii=False, + allow_nan=False, + separators=(",", ":"), + ).encode("utf-8") + b"\n" + except (TypeError, ValueError, UnicodeEncodeError) as exc: + raise AcpEnvelopeError("ACP envelope is not strict JSON serializable") from exc + if len(payload) > max_frame_bytes: + raise AcpFramingError("ACP frame exceeds configured size limit") + return payload + + +def request_envelope( + request_id: RequestId, + method: str, + params: Mapping[str, Any] | None = None, +) -> dict[str, Any]: + value = { + "jsonrpc": JSONRPC_VERSION, + "id": request_id, + "method": method, + "params": dict(params or {}), + } + validate_envelope(value) + return value + + +def notification_envelope( + method: str, + params: Mapping[str, Any] | None = None, +) -> dict[str, Any]: + value = { + "jsonrpc": JSONRPC_VERSION, + "method": method, + "params": dict(params or {}), + } + validate_envelope(value) + return value + + +def result_envelope(request_id: RequestId, result: Any) -> dict[str, Any]: + value = {"jsonrpc": JSONRPC_VERSION, "id": request_id, "result": result} + validate_envelope(value) + return value + + +def error_envelope( + request_id: RequestId | None, + code: int, + message: str, + *, + data: Any = None, +) -> dict[str, Any]: + error: dict[str, Any] = {"code": code, "message": message} + if data is not None: + error["data"] = data + value = {"jsonrpc": JSONRPC_VERSION, "id": request_id, "error": error} + validate_envelope(value) + return value + + +def parse_session_update(params: Mapping[str, Any]) -> SessionUpdate: + session_id = _required_string(params, "sessionId") + update = params.get("update") + if not isinstance(update, Mapping): + raise AcpEnvelopeError("session/update params.update must be an object") + kind_value = _required_string(update, "sessionUpdate") + try: + kind: SessionUpdateKind | str = SessionUpdateKind(kind_value) + except ValueError: + # ACP extensions and future stable revisions remain observable. + kind = kind_value + meta = params.get("_meta") + if meta is not None and not isinstance(meta, Mapping): + meta = None + return SessionUpdate( + session_id=session_id, + update_kind=kind, + update=_freeze_mapping(update), + meta=_freeze_mapping(meta) if isinstance(meta, Mapping) else None, + raw=_freeze_mapping(params), + ) + + +def parse_permission_request(request: JsonRpcRequest) -> PermissionRequest: + if request.method != "session/request_permission": + raise AcpEnvelopeError("request is not session/request_permission") + params = request.params + session_id = _required_string(params, "sessionId") + tool_call = params.get("toolCall") + if not isinstance(tool_call, Mapping): + raise AcpEnvelopeError("permission request toolCall must be an object") + raw_options = params.get("options") + if not isinstance(raw_options, list) or not raw_options: + raise AcpEnvelopeError("permission request options must be a non-empty array") + options: list[PermissionOption] = [] + for raw in raw_options: + if not isinstance(raw, Mapping): + raise AcpEnvelopeError("permission option must be an object") + option_id = _required_string(raw, "optionId") + name = _required_string(raw, "name") + kind_value = _required_string(raw, "kind") + try: + kind: PermissionOptionKind | str = PermissionOptionKind(kind_value) + except ValueError: + kind = kind_value + options.append( + PermissionOption( + option_id=option_id, + name=name, + kind=kind, + raw=_freeze_mapping(raw), + ) + ) + meta = params.get("_meta") + return PermissionRequest( + request_id=request.request_id, + session_id=session_id, + tool_call=_freeze_mapping(tool_call), + options=tuple(options), + meta=_freeze_mapping(meta) if isinstance(meta, Mapping) else None, + raw=_freeze_mapping(params), + ) + + +def _required_string(value: Mapping[str, Any], key: str) -> str: + result = value.get(key) + if not isinstance(result, str) or not result: + raise AcpEnvelopeError(f"{key} must be a non-empty string") + return result diff --git a/tests/fixtures/acp_fake_agent.py b/tests/fixtures/acp_fake_agent.py new file mode 100644 index 0000000..1d31a7f --- /dev/null +++ b/tests/fixtures/acp_fake_agent.py @@ -0,0 +1,150 @@ +"""Deterministic newline-delimited JSON-RPC peer used by ACP client tests.""" + +from __future__ import annotations + +import json +import sys +import time + + +MODE = sys.argv[1] if len(sys.argv) > 1 else "normal" + + +def send(value: object) -> None: + sys.stdout.write(json.dumps(value, separators=(",", ":")) + "\n") + sys.stdout.flush() + + +def response(request_id: object, result: object) -> None: + send({"jsonrpc": "2.0", "id": request_id, "result": result}) + + +def update(session_id: str, kind: str, **values: object) -> None: + send( + { + "jsonrpc": "2.0", + "method": "session/update", + "params": { + "sessionId": session_id, + "update": {"sessionUpdate": kind, **values}, + }, + } + ) + + +pending_prompt_id: object | None = None +pending_prompt_session = "" + +for line in sys.stdin: + message = json.loads(line) + method = message.get("method") + request_id = message.get("id") + params = message.get("params", {}) + + if method == "initialize": + if MODE == "malformed": + sys.stdout.write("not-json\n") + sys.stdout.flush() + continue + if MODE == "oversize": + response(request_id, {"protocolVersion": 1, "padding": "x" * 10000}) + continue + response( + request_id, + { + "protocolVersion": 1, + "agentCapabilities": ( + {} + if MODE == "baseline" + else { + "loadSession": True, + "sessionCapabilities": { + "list": {}, + "resume": {}, + "additionalDirectories": {}, + }, + } + ), + "agentInfo": {"name": "fake", "version": "1.0"}, + }, + ) + elif method == "initialized": + send( + { + "jsonrpc": "2.0", + "method": "fake/initialized_seen", + "params": {}, + } + ) + elif method == "session/new": + update("s-new", "agent_message_chunk", content={"type": "text", "text": "hi"}) + response( + request_id, + {"sessionId": "s-new", "modes": {"currentModeId": "default"}}, + ) + elif method == "session/load" or method == "session/resume": + response(request_id, {"configOptions": [{"id": "model", "currentValue": "x"}]}) + elif method == "session/list": + if MODE == "slow": + time.sleep(2) + continue + cursor = params.get("cursor") + response( + request_id, + { + "sessions": [ + { + "sessionId": "s2" if cursor else "s1", + "cwd": "/tmp/project", + "title": "second" if cursor else "first", + } + ], + **({} if cursor else {"nextCursor": "page-2"}), + }, + ) + elif method == "session/prompt": + pending_prompt_id = request_id + pending_prompt_session = params["sessionId"] + update( + pending_prompt_session, + "agent_thought_chunk", + content={"type": "text", "text": "reasoning summary"}, + ) + send( + { + "jsonrpc": "2.0", + "id": 900, + "method": "session/request_permission", + "params": { + "sessionId": pending_prompt_session, + "toolCall": {"toolCallId": "tool-1", "status": "pending"}, + "options": [ + { + "optionId": "allow", + "name": "Allow once", + "kind": "allow_once", + }, + { + "optionId": "reject", + "name": "Reject once", + "kind": "reject_once", + }, + ], + }, + } + ) + elif method == "session/cancel": + # The client must additionally resolve permission request 900 as cancelled. + pass + elif request_id == 900 and pending_prompt_id is not None: + outcome = message["result"]["outcome"]["outcome"] + update( + pending_prompt_session, + "plan", + entries=[{"content": "done", "status": "completed"}], + ) + response( + pending_prompt_id, + {"stopReason": "cancelled" if outcome == "cancelled" else "end_turn"}, + ) + pending_prompt_id = None diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py new file mode 100644 index 0000000..d1c8638 --- /dev/null +++ b/tests/test_acp_client.py @@ -0,0 +1,140 @@ +from __future__ import annotations + +import sys +import threading +from pathlib import Path + +import pytest + +from tendwire.backends.acp_client import ( + AcpCapabilityError, + AcpClient, + AcpRequestTimeoutError, + ClientState, +) +from tendwire.backends.acp_protocol import AcpProtocolError, SessionUpdateKind, StopReason + + +FAKE_AGENT = Path(__file__).parent / "fixtures" / "acp_fake_agent.py" + + +def client(mode: str = "normal", **kwargs: object) -> AcpClient: + return AcpClient([sys.executable, "-u", str(FAKE_AGENT), mode], **kwargs) + + +def test_initialize_capabilities_and_session_lifecycle() -> None: + with client() as acp: + initialized = acp.initialize() + assert initialized.protocol_version == 1 + assert initialized.agent_info == {"name": "fake", "version": "1.0"} + assert initialized.capabilities.load_session + assert initialized.capabilities.session_list + assert initialized.capabilities.session_resume + assert acp.state is ClientState.INITIALIZED + + created = acp.new_session( + "/tmp/project", + additional_directories=["/tmp/other"], + ) + assert created.session_id == "s-new" + assert created.modes == {"currentModeId": "default"} + streamed = acp.next_update(timeout=1) + assert streamed.update_kind is SessionUpdateKind.AGENT_MESSAGE_CHUNK + + loaded = acp.load_session("s1", "/tmp/project") + resumed = acp.resume_session("s1", "/tmp/project") + assert loaded.session_id == "s1" + assert resumed.config_options[0]["id"] == "model" + + first = acp.list_sessions(cwd="/tmp/project") + assert first.sessions[0].session_id == "s1" + assert first.next_cursor == "page-2" + second = acp.list_sessions(cursor=first.next_cursor) + assert second.sessions[0].title == "second" + assert second.next_cursor is None + + acp.initialized() + assert acp.next_notification(timeout=1).method == "fake/initialized_seen" + + assert acp.state is ClientState.CLOSED + assert acp.exit is not None + assert acp.exit.returncode == 0 + + +def test_prompt_stream_and_permission_response_can_run_concurrently() -> None: + with client() as acp: + acp.initialize() + acp.new_session("/tmp/project") + # Drain the new-session message update. + acp.next_update(timeout=1) + + outcome: list[object] = [] + failure: list[BaseException] = [] + + def run_prompt() -> None: + try: + outcome.append(acp.prompt("s-new", "please inspect")) + except BaseException as exc: # pragma: no cover - diagnostic path + failure.append(exc) + + thread = threading.Thread(target=run_prompt) + thread.start() + thought = acp.next_update(timeout=1) + assert thought.update_kind is SessionUpdateKind.AGENT_THOUGHT_CHUNK + permission = acp.next_permission_request(timeout=1) + assert permission.options[0].option_id == "allow" + acp.respond_permission(permission.request_id, option_id="allow") + plan = acp.next_update(timeout=1) + assert plan.update_kind is SessionUpdateKind.PLAN + thread.join(timeout=2) + + assert not thread.is_alive() + assert not failure + assert outcome[0].stop_reason is StopReason.END_TURN + + +def test_cancel_resolves_pending_permissions_as_cancelled() -> None: + with client() as acp: + acp.initialize() + result: list[object] = [] + thread = threading.Thread(target=lambda: result.append(acp.prompt("s1", "wait"))) + thread.start() + acp.next_update(timeout=1) + acp.next_permission_request(timeout=1) + acp.cancel("s1") + acp.next_update(timeout=1) + thread.join(timeout=2) + assert not thread.is_alive() + assert result[0].stop_reason is StopReason.CANCELLED + + +def test_optional_methods_require_advertised_capabilities() -> None: + with client("baseline") as acp: + acp.initialize() + with pytest.raises(AcpCapabilityError): + acp.list_sessions() + with pytest.raises(AcpCapabilityError): + acp.load_session("s1", "/tmp") + + +def test_request_timeout_does_not_poison_transport() -> None: + with client("slow", request_timeout=0.05) as acp: + acp.initialize(timeout=1) + with pytest.raises(AcpRequestTimeoutError): + acp.list_sessions() + assert acp.state is ClientState.INITIALIZED + + +@pytest.mark.parametrize("mode", ["malformed", "oversize"]) +def test_malformed_or_oversized_stdout_fails_connection(mode: str) -> None: + with client(mode, max_frame_bytes=1024) as acp: + with pytest.raises(AcpProtocolError): + acp.initialize(timeout=1) + assert acp.state is ClientState.FAILED + + +def test_absolute_session_paths_are_enforced_before_write() -> None: + with client() as acp: + acp.initialize() + with pytest.raises(ValueError, match="absolute"): + acp.new_session("relative/path") diff --git a/tests/test_acp_protocol.py b/tests/test_acp_protocol.py new file mode 100644 index 0000000..7dc567f --- /dev/null +++ b/tests/test_acp_protocol.py @@ -0,0 +1,117 @@ +from __future__ import annotations + +import pytest + +from tendwire.backends.acp_protocol import ( + AgentCapabilities, + AcpEnvelopeError, + AcpFramingError, + AcpRemoteError, + JsonRpcNotification, + JsonRpcRequest, + JsonRpcResponse, + PermissionOptionKind, + SessionUpdateKind, + decode_json_line, + encode_message, + parse_permission_request, + parse_session_update, + request_envelope, +) + + +def test_strict_json_line_round_trip() -> None: + envelope = request_envelope(7, "session/new", {"cwd": "/tmp", "mcpServers": []}) + encoded = encode_message(envelope) + assert encoded.endswith(b"\n") + assert decode_json_line(encoded) == JsonRpcRequest( + 7, "session/new", {"cwd": "/tmp", "mcpServers": []} + ) + + +@pytest.mark.parametrize( + "line", + [ + b'{"jsonrpc":"2.0","method":"x"}', + b'{"jsonrpc":"2.0","method":"x"}\n{}\n', + b'{"jsonrpc":"2.0","method":"x","method":"y"}\n', + b'{"jsonrpc":"2.0","method":"x","params":[],"id":1}\n', + b'{"jsonrpc":"2.0","method":"x","params":{"n":NaN}}\n', + b"\xff\n", + ], +) +def test_rejects_invalid_or_ambiguous_frames(line: bytes) -> None: + with pytest.raises((AcpFramingError, AcpEnvelopeError)): + decode_json_line(line) + + +def test_rejects_oversized_inbound_and_outbound_frames() -> None: + with pytest.raises(AcpFramingError): + decode_json_line(b'{"jsonrpc":"2.0","method":"long"}\n', max_frame_bytes=10) + with pytest.raises(AcpFramingError): + encode_message( + {"jsonrpc": "2.0", "method": "long", "params": {}}, + max_frame_bytes=10, + ) + + +def test_response_error_is_typed() -> None: + response = decode_json_line( + b'{"jsonrpc":"2.0","id":"r1","error":{"code":-32000,"message":"nope","data":{"retry":false}}}\n' + ) + assert isinstance(response, JsonRpcResponse) + with pytest.raises(AcpRemoteError) as raised: + response.result_or_raise() + assert raised.value.request_id == "r1" + assert raised.value.code == -32000 + assert raised.value.data == {"retry": False} + + +def test_notification_and_session_update_are_typed_but_extension_safe() -> None: + message = decode_json_line( + b'{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"s1","update":{"sessionUpdate":"agent_thought_chunk","content":{"type":"text","text":"summary"}}}}\n' + ) + assert isinstance(message, JsonRpcNotification) + update = parse_session_update(message.params) + assert update.session_id == "s1" + assert update.update_kind is SessionUpdateKind.AGENT_THOUGHT_CHUNK + assert update.update["content"]["text"] == "summary" + + extension = parse_session_update( + {"sessionId": "s1", "update": {"sessionUpdate": "vendor_progress"}} + ) + assert extension.update_kind == "vendor_progress" + + +def test_permission_request_validation_and_typed_options() -> None: + request = JsonRpcRequest( + 42, + "session/request_permission", + { + "sessionId": "s1", + "toolCall": {"toolCallId": "tool-1", "status": "pending"}, + "options": [ + {"optionId": "yes", "name": "Allow", "kind": "allow_once"} + ], + }, + ) + parsed = parse_permission_request(request) + assert parsed.request_id == 42 + assert parsed.options[0].kind is PermissionOptionKind.ALLOW_ONCE + + +def test_capability_presence_uses_acp_object_semantics() -> None: + capabilities = AgentCapabilities.from_mapping( + { + "loadSession": True, + "sessionCapabilities": { + "list": {}, + "resume": {}, + "additionalDirectories": {}, + }, + } + ) + assert capabilities.load_session + assert capabilities.session_list + assert capabilities.session_resume + assert capabilities.additional_directories From 7f62dff5aa2887e132f2091f17db38445d2ea0e2 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:09:17 +0800 Subject: [PATCH 05/83] test: expect ACP event journal schema version --- tests/test_backend_pending.py | 2 +- tests/test_connector_outbox.py | 2 +- tests/test_delivery_retention_migration.py | 2 +- tests/test_delivery_retention_recovery.py | 2 +- tests/test_store.py | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/test_backend_pending.py b/tests/test_backend_pending.py index 537b0b7..91c69a8 100644 --- a/tests/test_backend_pending.py +++ b/tests/test_backend_pending.py @@ -1449,7 +1449,7 @@ def test_current_schema_creation_has_exact_binding_and_claim_state(tmp_path: Pat db = tmp_path / "current-schema.db" init_store(db) with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 21 + assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 22 columns = { str(row[1]) for row in conn.execute("PRAGMA table_info(backend_pending)").fetchall() diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index 8a75330..c5c3f80 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -1754,7 +1754,7 @@ def test_v6_to_current_plan_migration_is_bounded_atomic_and_preserves_jobs( ).fetchall() } foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert version == store_sqlite.STORE_SCHEMA_VERSION == 21 + assert version == store_sqlite.STORE_SCHEMA_VERSION == 22 assert plan_row == (plan["plan_token"], 1, None, "active") assert job_count == 2 assert outbox_count == 3 diff --git a/tests/test_delivery_retention_migration.py b/tests/test_delivery_retention_migration.py index 6f1e468..33ce0ed 100644 --- a/tests/test_delivery_retention_migration.py +++ b/tests/test_delivery_retention_migration.py @@ -875,7 +875,7 @@ def test_v10_to_v12_migration_retains_finals_without_reposting_or_leaking( finals = _seed_v10_finals(db_path) init_store(db_path) - assert store_sqlite.STORE_SCHEMA_VERSION == 21 + assert store_sqlite.STORE_SCHEMA_VERSION == 22 delivered_key = _final_key(*finals["delivered"]) hold_keys = {_final_key(*finals[label]) for label in ("hold-a", "hold-b")} diff --git a/tests/test_delivery_retention_recovery.py b/tests/test_delivery_retention_recovery.py index 2497924..3923d3c 100644 --- a/tests/test_delivery_retention_recovery.py +++ b/tests/test_delivery_retention_recovery.py @@ -973,7 +973,7 @@ def test_v12_migration_uses_effective_recovery_lineage_without_repost_or_hold( api = ConnectorOutboxAPI(db_path, HOST_ID) assert api.poll({"name": FINAL_NAME, "limit": 100})["items"] == [] with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 21 + assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 22 anchor = conn.execute( """ SELECT delivery_kind, status diff --git a/tests/test_store.py b/tests/test_store.py index 3286984..09c7796 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -10278,7 +10278,7 @@ def test_v13_migration_repairs_nonpositive_turn_sequences_and_blocks_recurrence( init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 21 + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 22 assert conn.execute( """ SELECT turn_id, list_sequence From 7a5b09966d8b9ca35bd9abf5d5c500974d7983e9 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:13:20 +0800 Subject: [PATCH 06/83] Add guarded ACP event ingestion --- src/tendwire/backends/acp_ingestion.py | 437 +++++++++++++++++++++ tests/test_acp_ingestion.py | 502 +++++++++++++++++++++++++ 2 files changed, 939 insertions(+) create mode 100644 src/tendwire/backends/acp_ingestion.py create mode 100644 tests/test_acp_ingestion.py diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py new file mode 100644 index 0000000..3f9ce20 --- /dev/null +++ b/src/tendwire/backends/acp_ingestion.py @@ -0,0 +1,437 @@ +"""Durable ACP ingestion bound to one authenticated Tendwire worker. + +The transport, semantic projector, and SQLite journal are deliberately separate. +This module is the narrow authority bridge: it binds one ACP session generation +to one private Herdr worker binding, records every accepted semantic event, and +projects only user/assistant text into the existing turn model. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from ..config import Config +from ..core.agent_events import AgentEvent, agent_event +from ..core.models import WorkerBinding, stable_fingerprint +from ..store.sqlite import ( + AppendAgentEventResult, + TurnRefreshApplyResult, + append_agent_event, + apply_turn_refresh, + list_worker_bindings, +) +from .acp_projection import AcpEventProjector + + +AppendEvent = Callable[[Path | str, str, AgentEvent], AppendAgentEventResult] +ApplyTurn = Callable[..., TurnRefreshApplyResult] +BindingIsCurrent = Callable[[Path | str, str, WorkerBinding], bool] + + +@dataclass(frozen=True) +class AcpIngestionResult: + """Outcome of accepting, ignoring, or projecting one ACP event.""" + + kind: str | None + event: AppendAgentEventResult | None = None + turn: TurnRefreshApplyResult | None = None + ignored_reason: str | None = None + + +class AcpSessionIngestor: + """Ingest one ACP session generation for one private worker binding. + + ``stream_generation`` must change whenever a transport is recreated. ACP v1 + does not require a stable event ID for notifications, so generation-scoped + synthetic IDs avoid corrupting the append-only journal. Notifications with + authoritative source IDs still deduplicate across reconnects. + """ + + def __init__( + self, + config: Config, + *, + session_id: str, + stream_generation: str, + binding: WorkerBinding, + projector: AcpEventProjector | None = None, + append_event: AppendEvent = append_agent_event, + apply_turn: ApplyTurn = apply_turn_refresh, + binding_is_current: BindingIsCurrent | None = None, + ) -> None: + if config.db_path is None: + raise ValueError("ACP ingestion requires a sqlite db path") + if not isinstance(session_id, str) or not session_id.strip(): + raise ValueError("ACP session and stream generation are required") + if not isinstance(stream_generation, str) or not stream_generation.strip(): + raise ValueError("ACP session and stream generation are required") + if binding.host_id != config.host_id: + raise ValueError("ACP binding host does not match configuration") + if not binding.private_fingerprint: + raise ValueError("ACP ingestion requires an authenticated private binding") + if ( + binding.turn_target_kind != "acp_session_id" + or binding.turn_target_value != session_id.strip() + ): + raise ValueError("ACP session does not match the private worker binding") + if config.agent_event_source == "legacy": + raise ValueError("ACP ingestion is disabled by agent_event_source=legacy") + self.config = config + self.session_id = session_id.strip() + self.stream_generation = stream_generation.strip() + self.binding = binding + self.projector = projector or AcpEventProjector() + self._append_event = append_event + self._apply_turn = apply_turn + self._binding_is_current = binding_is_current or _binding_is_current + self._turn_ordinal = 0 + self._source_turn_id: str | None = None + self._turn_complete = False + + @property + def source_turn_id(self) -> str | None: + """Return the opaque public-safe identity of the active ACP turn.""" + + return self._source_turn_id + + def start_turn(self, *, producer_turn_id: str | None = None) -> str: + """Reset message assembly and allocate one opaque turn identity.""" + + if producer_turn_id is not None and ( + not isinstance(producer_turn_id, str) or not producer_turn_id.strip() + ): + raise ValueError("producer_turn_id must be non-empty text or None") + self._turn_ordinal += 1 + self.projector.reset_turn(self.session_id) + # An authoritative producer turn ID must retain identity across ACP + # transport recreation. Generation only scopes locally synthesized + # ordinals, whose meaning cannot survive a reconnect. + identity = ( + { + "source": "acp", + "session": self.session_id, + "producer_turn": producer_turn_id.strip(), + } + if producer_turn_id is not None + else { + "source": "acp", + "session": self.session_id, + "generation": self.stream_generation, + "turn": self._turn_ordinal, + } + ) + self._source_turn_id = f"acpt_{stable_fingerprint(identity)}" + self._turn_complete = False + return self._source_turn_id + + def ingest_update( + self, + notification: Mapping[str, Any], + *, + source_event_id: str | None = None, + replay: bool = False, + ) -> AcpIngestionResult: + """Normalize, journal, and conditionally project ``session/update``.""" + + mismatch = _notification_mismatch( + notification, + method="session/update", + session_id=self.session_id, + ) + if mismatch is not None: + return AcpIngestionResult(None, ignored_reason=mismatch) + update_kind = _session_update_kind(notification) + if self._turn_complete and update_kind in _TURN_SCOPED_UPDATES: + return AcpIngestionResult(None, ignored_reason="turn_already_complete") + thought_rejection = _thought_rejection_reason( + notification, + policy=self.config.acp_thought_policy, + ) + if thought_rejection is not None: + return AcpIngestionResult("thought", ignored_reason=thought_rejection) + if not self._current_binding_is_valid(): + return AcpIngestionResult(None, ignored_reason="stale_binding") + canonical = self.projector.normalize_session_update( + notification, + source_event_id=source_event_id, + replay=replay, + ) + if canonical is None: + return AcpIngestionResult(None, ignored_reason="unsupported_or_duplicate") + return self._accept(canonical) + + def ingest_permission_request( + self, + request: Mapping[str, Any], + *, + source_event_id: str | None = None, + replay: bool = False, + ) -> AcpIngestionResult: + """Journal a permission request as a private tool lifecycle update.""" + + mismatch = _notification_mismatch( + request, + method="session/request_permission", + session_id=self.session_id, + ) + if mismatch is not None: + return AcpIngestionResult(None, ignored_reason=mismatch) + if self._turn_complete: + return AcpIngestionResult(None, ignored_reason="turn_already_complete") + if not self._current_binding_is_valid(): + return AcpIngestionResult(None, ignored_reason="stale_binding") + canonical = self.projector.normalize_permission_request( + request, + source_event_id=source_event_id, + replay=replay, + ) + if canonical is None: + return AcpIngestionResult(None, ignored_reason="duplicate") + return self._accept(canonical) + + def mark_prompt_complete(self) -> AcpIngestionResult: + """Finalize the current text projection after ``session/prompt`` returns.""" + + if self._source_turn_id is None: + return AcpIngestionResult(None, ignored_reason="no_active_turn") + if self._turn_complete: + return AcpIngestionResult(None, ignored_reason="turn_already_complete") + if not self._current_binding_is_valid(): + return AcpIngestionResult(None, ignored_reason="stale_binding") + content = self.projector.mark_turn_complete(self.session_id) + content["source_turn_id"] = self._source_turn_id + if self.config.agent_event_source == "acp_shadow": + self._turn_complete = True + return AcpIngestionResult("agent_message") + turn = self._project_turn(content) + self._turn_complete = True + return AcpIngestionResult( + "agent_message", + turn=turn, + ignored_reason="stale_binding" if turn.stale_binding else None, + ) + + def _accept(self, canonical: Mapping[str, Any]) -> AcpIngestionResult: + kind = str(canonical.get("kind") or "") + if kind == "thought" and self.config.acp_thought_policy == "disabled": + return AcpIngestionResult(kind, ignored_reason="thought_policy_disabled") + if self._source_turn_id is None and kind in { + "user_message", + "agent_message", + "thought", + "tool_call", + "tool_call_update", + "plan", + }: + self.start_turn() + + payload = canonical.get("payload") + if not isinstance(payload, Mapping): + raise ValueError("canonical ACP event payload must be a mapping") + sequence = canonical.get("sequence") + if isinstance(sequence, bool) or not isinstance(sequence, int) or sequence < 0: + raise ValueError("canonical ACP event sequence must be nonnegative") + explicit_event_id = canonical.get("source_event_id") + source_id = ( + str(explicit_event_id) + if explicit_event_id is not None and str(explicit_event_id) + else f"stream:{self.stream_generation}:{sequence}" + ) + event = agent_event( + kind=kind, + source="acp", + worker_id=self.binding.worker_id, + payload=payload, + source_session_id=self.session_id, + source_turn_id=self._source_turn_id, + source_item_id=_source_item_id(kind, payload), + source_message_id=_source_message_id(kind, payload), + source_event_id=source_id, + source_sequence=sequence, + # The complete structured journal is private initially. Public and + # connector views require a separate explicit sanitizing projection. + visibility="private", + ) + appended = self._append_event( + Path(self.config.db_path), + self.config.host_id, + event, + ) + + turn: TurnRefreshApplyResult | None = None + if ( + kind in {"user_message", "agent_message"} + and self.config.agent_event_source != "acp_shadow" + and appended.inserted + ): + content = self.projector.project_turn_content(self.session_id) + if self._source_turn_id is not None: + content["source_turn_id"] = self._source_turn_id + turn = self._project_turn(content) + return AcpIngestionResult( + kind, + event=appended, + turn=turn, + ignored_reason=( + "stale_binding" + if turn is not None and turn.stale_binding + else "duplicate_event" + if not appended.inserted + else None + ), + ) + + def _current_binding_is_valid(self) -> bool: + try: + return bool( + self._binding_is_current( + Path(self.config.db_path), + self.config.host_id, + self.binding, + ) + ) + except Exception: + # Binding lookup is an authority check. Any lookup failure must + # fail closed rather than accepting an unauthenticated event. + return False + + def _project_turn(self, content: Mapping[str, Any]) -> TurnRefreshApplyResult: + return self._apply_turn( + Path(self.config.db_path), + self.config.host_id, + self.binding.worker_id, + content, + expected_binding=self.binding, + pending_stale_grace_seconds=self.config.pending_stale_grace_seconds, + turn_model=self.config.turn_model, + ) + + +def _source_message_id(kind: str, payload: Mapping[str, Any]) -> str | None: + if kind not in {"user_message", "agent_message", "thought"}: + return None + value = payload.get("message_id") + return str(value) if value is not None and str(value) else None + + +def _source_item_id(kind: str, payload: Mapping[str, Any]) -> str | None: + if kind not in {"tool_call", "tool_call_update"}: + return None + value = payload.get("tool_call_id") + return str(value) if value is not None and str(value) else None + + +_TURN_SCOPED_UPDATES = frozenset( + { + "user_message_chunk", + "agent_message_chunk", + "agent_thought_chunk", + "tool_call", + "tool_call_update", + "plan", + } +) +_THOUGHT_RAW_LABELS = frozenset( + {"raw", "reasoning", "raw_reasoning", "raw-reasoning", "chain_of_thought"} +) + + +def _params(value: Mapping[str, Any]) -> Mapping[str, Any]: + params = value.get("params") + return params if isinstance(params, Mapping) else value + + +def _notification_mismatch( + value: Mapping[str, Any], + *, + method: str, + session_id: str, +) -> str | None: + supplied_method = value.get("method") + if supplied_method is not None and supplied_method != method: + return "method_mismatch" + supplied_session = _params(value).get("sessionId") + if supplied_session != session_id: + return "session_mismatch" + return None + + +def _session_update(value: Mapping[str, Any]) -> Mapping[str, Any] | None: + update = _params(value).get("update") + return update if isinstance(update, Mapping) else None + + +def _session_update_kind(value: Mapping[str, Any]) -> str | None: + update = _session_update(value) + kind = update.get("sessionUpdate") if update is not None else None + return kind if isinstance(kind, str) else None + + +def _thought_classification(value: Mapping[str, Any]) -> str | None: + update = _session_update(value) + if update is None or update.get("sessionUpdate") != "agent_thought_chunk": + return None + candidates: list[Mapping[str, Any]] = [] + for container in (update, update.get("content")): + if not isinstance(container, Mapping): + continue + meta = container.get("_meta") + if not isinstance(meta, Mapping): + continue + candidates.append(meta) + tendwire = meta.get("tendwire") + if isinstance(tendwire, Mapping): + candidates.insert(0, tendwire) + for meta in candidates: + for key in ("thought_kind", "thoughtKind", "reasoning_kind", "reasoningKind"): + label = meta.get(key) + if isinstance(label, str) and label.strip(): + return label.strip().lower() + return "unclassified" + + +def _thought_rejection_reason( + value: Mapping[str, Any], + *, + policy: str, +) -> str | None: + classification = _thought_classification(value) + if classification is None: + return None + if policy == "disabled": + return "thought_policy_disabled" + if policy == "private_summary" and classification in _THOUGHT_RAW_LABELS: + return "thought_policy_requires_summary" + return None + + +def _binding_is_current( + db_path: Path | str, + host_id: str, + expected: WorkerBinding, +) -> bool: + """Check the durable private authority immediately before accepting data.""" + + for current in list_worker_bindings( + Path(db_path), + str(host_id), + backend=expected.backend, + ): + if ( + current.worker_id == expected.worker_id + and current.worker_fingerprint == expected.worker_fingerprint + and current.backend == expected.backend + and current.target_kind == expected.target_kind + and current.target_value == expected.target_value + and current.turn_target_kind == expected.turn_target_kind + and current.turn_target_value == expected.turn_target_value + and current.private_fingerprint == expected.private_fingerprint + ): + return True + return False + + +__all__ = ["AcpIngestionResult", "AcpSessionIngestor"] diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py new file mode 100644 index 0000000..33b7ca4 --- /dev/null +++ b/tests/test_acp_ingestion.py @@ -0,0 +1,502 @@ +"""Integration-boundary tests for durable ACP event ingestion.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tendwire.backends.acp_ingestion import AcpSessionIngestor +from tendwire.config import Config +from tendwire.core.agent_events import AgentEvent, AppendAgentEventResult +from tendwire.core.models import WorkerBinding +from tendwire.store.sqlite import TurnRefreshApplyResult, upsert_worker_bindings + + +def _binding() -> WorkerBinding: + return WorkerBinding( + host_id="host-a", + worker_id="worker-a", + worker_fingerprint="worker-fingerprint", + backend="herdr", + target_kind="pane_id", + target_value="private-pane", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + private_fingerprint="binding-fingerprint", + ) + + +def _update(kind: str, **fields: object) -> dict[str, object]: + return { + "method": "session/update", + "params": { + "sessionId": "session-a", + "update": {"sessionUpdate": kind, **fields}, + }, + } + + +def test_messages_are_journaled_privately_and_projected_without_thoughts( + tmp_path: Path, +) -> None: + events: list[AgentEvent] = [] + turns: list[dict[str, object]] = [] + + def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + events.append(event) + return AppendAgentEventResult(len(events), event.event_id, True) + + def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): + turns.append(dict(content)) + return TurnRefreshApplyResult(1, False) + + ingestor = AcpSessionIngestor( + Config(host_id="host-a", db_path=tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=append, + apply_turn=apply, + binding_is_current=lambda *_args: True, + ) + turn_id = ingestor.start_turn(producer_turn_id="private-turn") + ingestor.ingest_update( + _update( + "user_message_chunk", + messageId="user-1", + content={"type": "text", "text": "question"}, + ) + ) + ingestor.ingest_update( + _update( + "agent_thought_chunk", + messageId="reasoning-1", + content={"type": "text", "text": "private reasoning"}, + _meta={"tendwire": {"thought_kind": "summary"}}, + ) + ) + ingestor.ingest_update( + _update( + "agent_message_chunk", + messageId="assistant-1", + content={"type": "text", "text": "answer"}, + ) + ) + ingestor.mark_prompt_complete() + + assert turn_id.startswith("acpt_") + assert [event.kind for event in events] == [ + "user_message", + "thought", + "agent_message", + ] + assert all(event.visibility == "private" for event in events) + assert turns[-1]["assistant_final_text"] == "answer" + assert turns[-1]["user_text"] == "question" + assert "private reasoning" not in repr(turns) + assert turns[-1]["source_turn_id"] == turn_id + + +def test_shadow_mode_journals_without_turn_projection(tmp_path: Path) -> None: + events: list[AgentEvent] = [] + + def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + events.append(event) + return AppendAgentEventResult(1, event.event_id, True) + + def unexpected_turn(*_args, **_kwargs): + raise AssertionError("shadow mode must not project turns") + + ingestor = AcpSessionIngestor( + Config( + host_id="host-a", + db_path=tmp_path / "events.db", + agent_event_source="acp_shadow", + ), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=append, + apply_turn=unexpected_turn, + binding_is_current=lambda *_args: True, + ) + result = ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "shadow"}, + ) + ) + + assert result.event is not None + assert result.turn is None + assert len(events) == 1 + + +def test_disabled_thought_policy_discards_before_persistence(tmp_path: Path) -> None: + def unexpected_append(*_args, **_kwargs): + raise AssertionError("disabled thoughts must not be persisted") + + ingestor = AcpSessionIngestor( + Config( + host_id="host-a", + db_path=tmp_path / "events.db", + acp_thought_policy="disabled", + ), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=unexpected_append, + ) + result = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "discard me"}, + ) + ) + + assert result.ignored_reason == "thought_policy_disabled" + + +def test_synthetic_event_identity_is_scoped_to_stream_generation(tmp_path: Path) -> None: + seen: list[str] = [] + + def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + assert event.source_event_id is not None + seen.append(event.source_event_id) + return AppendAgentEventResult(1, event.event_id, True) + + for generation in ("generation-a", "generation-b"): + ingestor = AcpSessionIngestor( + Config(host_id="host-a", db_path=tmp_path / "events.db"), + session_id="session-a", + stream_generation=generation, + binding=_binding(), + append_event=append, + apply_turn=lambda *_args, **_kwargs: TurnRefreshApplyResult(0, False), + binding_is_current=lambda *_args: True, + ) + ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "same chunk"}, + ) + ) + + assert seen == ["stream:generation-a:1", "stream:generation-b:1"] + + +def test_constructor_rejects_binding_for_another_acp_session(tmp_path: Path) -> None: + binding = _binding() + mismatched = WorkerBinding( + **{**binding.__dict__, "turn_target_value": "another-session"} + ) + + with pytest.raises(ValueError, match="does not match"): + AcpSessionIngestor( + Config(host_id="host-a", db_path=tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=mismatched, + ) + + +def test_notification_session_mismatch_is_rejected_before_state_or_persistence( + tmp_path: Path, +) -> None: + def unexpected(*_args, **_kwargs): + raise AssertionError("mismatched session must not cross the authority boundary") + + ingestor = AcpSessionIngestor( + Config(host_id="host-a", db_path=tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=unexpected, + apply_turn=unexpected, + binding_is_current=unexpected, + ) + notification = _update( + "agent_message_chunk", + content={"type": "text", "text": "wrong worker"}, + ) + params = notification["params"] + assert isinstance(params, dict) + params["sessionId"] = "session-b" + + result = ingestor.ingest_update(notification) + + assert result.ignored_reason == "session_mismatch" + assert ingestor.source_turn_id is None + assert ingestor.projector.session_snapshot("session-b") is None + + +def test_required_mode_fails_closed_when_durable_binding_is_stale( + tmp_path: Path, +) -> None: + def unexpected(*_args, **_kwargs): + raise AssertionError("stale ACP events must not be journaled or projected") + + ingestor = AcpSessionIngestor( + Config( + host_id="host-a", + db_path=tmp_path / "events.db", + agent_event_source="acp_required", + ), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=unexpected, + apply_turn=unexpected, + binding_is_current=lambda *_args: False, + ) + + result = ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "must not publish"}, + ) + ) + + assert result.ignored_reason == "stale_binding" + assert result.event is None + assert result.turn is None + assert ingestor.source_turn_id is None + + +def test_default_authority_check_accepts_the_current_durable_binding( + tmp_path: Path, +) -> None: + db_path = tmp_path / "events.db" + binding = _binding() + upsert_worker_bindings(db_path, [binding]) + ingestor = AcpSessionIngestor( + Config(host_id="host-a", db_path=db_path, agent_event_source="acp_required"), + session_id="session-a", + stream_generation="generation-a", + binding=binding, + ) + + result = ingestor.ingest_update(_update("usage_update", used=1, size=100)) + + assert result.event is not None + assert result.event.inserted + assert result.ignored_reason is None + + +def test_shadow_completion_never_projects_and_finality_is_idempotent( + tmp_path: Path, +) -> None: + events: list[AgentEvent] = [] + + def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + events.append(event) + return AppendAgentEventResult(len(events), event.event_id, True) + + def unexpected_turn(*_args, **_kwargs): + raise AssertionError("shadow mode must never project, including completion") + + ingestor = AcpSessionIngestor( + Config( + host_id="host-a", + db_path=tmp_path / "events.db", + agent_event_source="acp_shadow", + ), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=append, + apply_turn=unexpected_turn, + binding_is_current=lambda *_args: True, + ) + ingestor.start_turn(producer_turn_id="turn-1") + ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "shadow final"}, + ) + ) + + completed = ingestor.mark_prompt_complete() + repeated = ingestor.mark_prompt_complete() + late = ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "late mutation"}, + ) + ) + + assert completed.turn is None + assert repeated.ignored_reason == "turn_already_complete" + assert late.ignored_reason == "turn_already_complete" + assert len(events) == 1 + + +def test_required_mode_projects_messages_and_final_exactly_once(tmp_path: Path) -> None: + events: list[AgentEvent] = [] + turns: list[dict[str, object]] = [] + + def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + events.append(event) + return AppendAgentEventResult(len(events), event.event_id, True) + + def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): + turns.append(dict(content)) + return TurnRefreshApplyResult(1, False) + + ingestor = AcpSessionIngestor( + Config( + host_id="host-a", + db_path=tmp_path / "events.db", + agent_event_source="acp_required", + ), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=append, + apply_turn=apply, + binding_is_current=lambda *_args: True, + ) + ingestor.start_turn(producer_turn_id="turn-1") + streamed = ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "answer"}, + ) + ) + completed = ingestor.mark_prompt_complete() + + assert streamed.turn is not None + assert completed.turn is not None + assert [turn["complete"] for turn in turns] == [False, True] + assert turns[-1]["assistant_final_text"] == "answer" + assert turns[-1]["assistant_stream_text"] == "" + + +def test_duplicate_durable_event_is_not_reprojected(tmp_path: Path) -> None: + projected = False + + def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + return AppendAgentEventResult(9, event.event_id, False) + + def apply(*_args, **_kwargs): + nonlocal projected + projected = True + return TurnRefreshApplyResult(1, False) + + ingestor = AcpSessionIngestor( + Config(host_id="host-a", db_path=tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=append, + apply_turn=apply, + binding_is_current=lambda *_args: True, + ) + result = ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "replayed"}, + ), + source_event_id="event-1", + replay=True, + ) + + assert result.ignored_reason == "duplicate_event" + assert not projected + + +def test_producer_turn_identity_survives_transport_recreation(tmp_path: Path) -> None: + identities: list[str] = [] + for generation in ("generation-a", "generation-b"): + ingestor = AcpSessionIngestor( + Config(host_id="host-a", db_path=tmp_path / "events.db"), + session_id="session-a", + stream_generation=generation, + binding=_binding(), + binding_is_current=lambda *_args: True, + ) + identities.append(ingestor.start_turn(producer_turn_id="producer-turn-7")) + + assert identities[0] == identities[1] + + +def test_private_summary_policy_retains_display_chunks_but_rejects_marked_raw_thoughts( + tmp_path: Path, +) -> None: + events: list[AgentEvent] = [] + + def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + events.append(event) + return AppendAgentEventResult(len(events), event.event_id, True) + + ingestor = AcpSessionIngestor( + Config(host_id="host-a", db_path=tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=append, + binding_is_current=lambda *_args: True, + ) + unclassified = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "adapter display summary"}, + ) + ) + raw = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "raw secret"}, + _meta={"tendwire": {"thought_kind": "raw"}}, + ) + ) + summary = ingestor.ingest_update( + _update( + "agent_thought_chunk", + messageId="summary-1", + content={"type": "text", "text": "readable summary"}, + _meta={"tendwire": {"thought_kind": "summary"}}, + ) + ) + + assert unclassified.event is not None + assert raw.ignored_reason == "thought_policy_requires_summary" + assert summary.event is not None + assert len(events) == 2 + assert all(event.visibility == "private" for event in events) + assert all(event.public_payload == {} for event in events) + assert "raw secret" not in repr(events) + + +def test_private_all_policy_retains_marked_raw_thought_privately(tmp_path: Path) -> None: + events: list[AgentEvent] = [] + + def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + events.append(event) + return AppendAgentEventResult(1, event.event_id, True) + + ingestor = AcpSessionIngestor( + Config( + host_id="host-a", + db_path=tmp_path / "events.db", + acp_thought_policy="private_all", + ), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=append, + binding_is_current=lambda *_args: True, + ) + result = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "raw local diagnostic"}, + _meta={"tendwire": {"thought_kind": "raw"}}, + ) + ) + + assert result.event is not None + assert len(events) == 1 + assert events[0].payload["text_delta"] == "raw local diagnostic" + assert events[0].public_payload == {} From e19c8c0c2a22b2e93aee9c49c04db9603dead9d1 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:13:45 +0800 Subject: [PATCH 07/83] Add supervised ACP session runtime --- src/tendwire/backends/acp_runtime.py | 550 +++++++++++++++++++++++++++ tests/test_acp_runtime.py | 443 +++++++++++++++++++++ 2 files changed, 993 insertions(+) create mode 100644 src/tendwire/backends/acp_runtime.py create mode 100644 tests/test_acp_runtime.py diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py new file mode 100644 index 0000000..98491a4 --- /dev/null +++ b/src/tendwire/backends/acp_runtime.py @@ -0,0 +1,550 @@ +"""Supervised ACP session runtime bound to one Tendwire worker. + +The lower-level :mod:`acp_client` owns subprocess framing and JSON-RPC request +correlation. This module connects that transport to durable ACP ingestion, +keeps both inbound event queues drained, and exposes a deliberately redacted +health surface suitable for operator APIs. +""" + +from __future__ import annotations + +import threading +import time +import uuid +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from enum import Enum +from pathlib import Path +from typing import Any + +from ..config import Config +from ..core.models import WorkerBinding +from .acp_client import AcpClient +from .acp_ingestion import AcpSessionIngestor +from .acp_protocol import PermissionRequest, PromptResult, SessionResult + + +class AcpRuntimeError(RuntimeError): + """Base error raised by the supervised ACP runtime.""" + + +class AcpRuntimeStateError(AcpRuntimeError): + """An operation is invalid for the runtime's current lifecycle state.""" + + +class AcpRuntimeProtocolError(AcpRuntimeError): + """The ACP client returned data that cannot be safely bound or finalized.""" + + +class AcpRuntimeStopTimeout(AcpRuntimeError, TimeoutError): + """The runtime could not stop all supervised work within its deadline.""" + + +class SessionOpenMode(str, Enum): + NEW = "new" + LOAD = "load" + RESUME = "resume" + + +class RuntimeState(str, Enum): + NEW = "new" + STARTING = "starting" + RUNNING = "running" + STOPPING = "stopping" + STOPPED = "stopped" + FAILED = "failed" + + +@dataclass(frozen=True, slots=True) +class AcpRuntimeStatus: + """Public-safe runtime health and counters. + + This type intentionally has no command, process, session, worker, target, + or exception-message fields. Those values are private routing material. + """ + + state: RuntimeState + healthy: bool + updates_ingested: int + permissions_ingested: int + permissions_selected: int + permissions_cancelled: int + invalid_permission_selections: int + prompts_started: int + prompts_completed: int + prompts_failed: int + cancellation_requests: int + failure_type: str | None + + +PermissionCallback = Callable[[PermissionRequest], str | None] +IngestorFactory = Callable[..., AcpSessionIngestor] + + +class AcpRuntime: + """Run and durably ingest exactly one ACP session for one worker binding. + + Permission requests fail closed: the default response is ``cancelled``. + A callback can authorize an action only by returning the ID of an option + present in that exact request. + """ + + def __init__( + self, + client: AcpClient, + *, + config: Config, + binding: WorkerBinding, + cwd: str | Path, + session_mode: SessionOpenMode | str = SessionOpenMode.NEW, + session_id: str | None = None, + stream_generation: str | None = None, + client_capabilities: Mapping[str, Any] | None = None, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[str | Path] = (), + permission_callback: PermissionCallback | None = None, + ingestor: AcpSessionIngestor | None = None, + ingestor_factory: IngestorFactory = AcpSessionIngestor, + poll_timeout: float = 0.05, + stop_timeout: float = 3.0, + ) -> None: + try: + mode = SessionOpenMode(session_mode) + except ValueError as exc: + raise ValueError(f"unsupported ACP session mode {session_mode!r}") from exc + if mode is not SessionOpenMode.NEW and not session_id: + raise ValueError(f"session_id is required for ACP {mode.value}") + if mode is SessionOpenMode.NEW and session_id is not None: + raise ValueError("session_id must be omitted when creating an ACP session") + if binding.host_id != config.host_id: + raise ValueError("ACP runtime binding host does not match configuration") + if not binding.private_fingerprint: + raise ValueError("ACP runtime requires an authenticated private binding") + resolved_cwd = Path(cwd) + if not resolved_cwd.is_absolute(): + raise ValueError("ACP runtime cwd must be absolute") + if poll_timeout <= 0 or stop_timeout <= 0: + raise ValueError("ACP runtime timeouts must be positive") + + self._client = client + self._config = config + self._binding = binding + self._cwd = resolved_cwd + self._session_mode = mode + self._requested_session_id = session_id + self._stream_generation = stream_generation or uuid.uuid4().hex + self._client_capabilities = dict(client_capabilities or {}) + self._mcp_servers = tuple(dict(server) for server in mcp_servers) + self._additional_directories = tuple(Path(path) for path in additional_directories) + self._permission_callback = permission_callback + self._provided_ingestor = ingestor + self._ingestor_factory = ingestor_factory + self._poll_timeout = float(poll_timeout) + self._stop_timeout = float(stop_timeout) + + self._state = RuntimeState.NEW + self._session_id: str | None = None + self._ingestor: AcpSessionIngestor | None = None + self._failure: BaseException | None = None + self._state_lock = threading.RLock() + self._ingest_lock = threading.Lock() + self._prompt_lock = threading.Lock() + self._idle_condition = threading.Condition(self._state_lock) + self._stop_event = threading.Event() + self._threads: tuple[threading.Thread, ...] = () + self._update_idle_epoch = 0 + + self._updates_ingested = 0 + self._permissions_ingested = 0 + self._permissions_selected = 0 + self._permissions_cancelled = 0 + self._invalid_permission_selections = 0 + self._prompts_started = 0 + self._prompts_completed = 0 + self._prompts_failed = 0 + self._cancellation_requests = 0 + + def __enter__(self) -> "AcpRuntime": + return self.start() + + def __exit__(self, exc_type: object, exc: object, tb: object) -> None: + try: + self.stop() + except BaseException: + if exc is None: + raise + + def start(self) -> "AcpRuntime": + """Initialize capabilities, open one session, and start consumers.""" + + with self._state_lock: + if self._state is RuntimeState.RUNNING: + return self + if self._state is not RuntimeState.NEW: + raise AcpRuntimeStateError( + f"cannot start ACP runtime in state {self._state.value}" + ) + self._state = RuntimeState.STARTING + try: + self._client.initialize(client_capabilities=self._client_capabilities) + session = self._open_session() + if not isinstance(session, SessionResult) or not session.session_id: + raise AcpRuntimeProtocolError( + "ACP session setup returned an invalid response" + ) + self._session_id = session.session_id + self._ingestor = self._make_ingestor(session.session_id) + threads = ( + threading.Thread( + target=self._consume_updates, + name="tendwire-acp-updates", + daemon=True, + ), + threading.Thread( + target=self._consume_permissions, + name="tendwire-acp-permissions", + daemon=True, + ), + ) + self._threads = threads + with self._state_lock: + self._state = RuntimeState.RUNNING + for thread in threads: + thread.start() + except BaseException as exc: + self._record_failure(exc) + raise + return self + + def prompt( + self, + prompt: str | Sequence[Mapping[str, Any]], + *, + producer_turn_id: str | None = None, + timeout: float | None = None, + drain_timeout: float | None = None, + ) -> PromptResult: + """Submit one prompt and finalize only after its prior updates drain.""" + + wait_limit = self._stop_timeout if drain_timeout is None else float(drain_timeout) + if wait_limit <= 0: + raise ValueError("drain_timeout must be positive") + with self._prompt_lock: + self.raise_if_failed() + session_id, ingestor = self._running_components() + with self._state_lock: + self._prompts_started += 1 + try: + ingestor.start_turn(producer_turn_id=producer_turn_id) + except BaseException as exc: + with self._state_lock: + self._prompts_failed += 1 + self._record_failure(exc) + raise + try: + result = self._client.prompt(session_id, prompt, timeout=timeout) + except BaseException: + with self._state_lock: + self._prompts_failed += 1 + raise + if not isinstance(result, PromptResult): + error = AcpRuntimeProtocolError( + "ACP prompt returned an invalid response" + ) + with self._state_lock: + self._prompts_failed += 1 + self._record_failure(error) + raise error + + # The transport dispatches updates before the prompt response, but + # the consumer runs on another thread. Requiring a queue timeout + # after the response is a barrier: every earlier queued update has + # been durably ingested before the turn is marked complete. + try: + self._wait_for_post_response_idle(wait_limit) + with self._ingest_lock: + ingestor.mark_prompt_complete() + except BaseException as exc: + with self._state_lock: + self._prompts_failed += 1 + self._record_failure(exc) + raise + with self._state_lock: + self._prompts_completed += 1 + return result + + def cancel(self) -> None: + """Cancel the active session and any permission requests pending in it.""" + + self.raise_if_failed() + session_id, _ = self._running_components() + self._client.cancel(session_id) + with self._state_lock: + self._cancellation_requests += 1 + + def status(self) -> AcpRuntimeStatus: + """Return redacted health and counters safe for a public status API.""" + + with self._state_lock: + return AcpRuntimeStatus( + state=self._state, + healthy=self._state is RuntimeState.RUNNING and self._failure is None, + updates_ingested=self._updates_ingested, + permissions_ingested=self._permissions_ingested, + permissions_selected=self._permissions_selected, + permissions_cancelled=self._permissions_cancelled, + invalid_permission_selections=self._invalid_permission_selections, + prompts_started=self._prompts_started, + prompts_completed=self._prompts_completed, + prompts_failed=self._prompts_failed, + cancellation_requests=self._cancellation_requests, + failure_type=( + type(self._failure).__name__ if self._failure is not None else None + ), + ) + + def raise_if_failed(self) -> None: + """Raise the original background/runtime failure without redaction.""" + + with self._state_lock: + failure = self._failure + if failure is not None: + raise failure + + def join(self, timeout: float | None = None) -> bool: + """Wait a bounded interval for consumer threads; return whether all exited.""" + + wait_limit = self._stop_timeout if timeout is None else float(timeout) + if wait_limit <= 0: + raise ValueError("join timeout must be positive") + deadline = time.monotonic() + wait_limit + for thread in self._threads: + if thread is threading.current_thread(): + continue + thread.join(timeout=max(0.0, deadline - time.monotonic())) + return all( + thread is threading.current_thread() or not thread.is_alive() + for thread in self._threads + ) + + def stop(self, *, timeout: float | None = None) -> None: + """Close transport and consumers without waiting beyond one deadline.""" + + wait_limit = self._stop_timeout if timeout is None else float(timeout) + if wait_limit <= 0: + raise ValueError("stop timeout must be positive") + with self._state_lock: + if self._state is RuntimeState.STOPPED: + return + if self._state is RuntimeState.NEW: + self._state = RuntimeState.STOPPED + return + if self._state is not RuntimeState.FAILED: + self._state = RuntimeState.STOPPING + self._stop_event.set() + close_failures: list[BaseException] = [] + + def close_client() -> None: + try: + self._client.close() + except BaseException as exc: + close_failures.append(exc) + + closer = threading.Thread( + target=close_client, + name="tendwire-acp-close", + daemon=True, + ) + deadline = time.monotonic() + wait_limit + closer.start() + closer.join(timeout=max(0.0, deadline - time.monotonic())) + remaining = max(0.0, deadline - time.monotonic()) + joined = ( + self.join(timeout=remaining) + if remaining > 0 + else all(not thread.is_alive() for thread in self._threads) + ) + if closer.is_alive() or not joined: + error = AcpRuntimeStopTimeout( + "ACP runtime did not stop within the configured deadline" + ) + self._record_failure(error) + raise error + if close_failures: + self._record_failure(close_failures[0]) + raise close_failures[0] + with self._state_lock: + if self._failure is None: + self._state = RuntimeState.STOPPED + failure = self._failure + if failure is not None: + raise failure + + def _open_session(self) -> SessionResult: + options = { + "mcp_servers": self._mcp_servers, + "additional_directories": self._additional_directories, + } + if self._session_mode is SessionOpenMode.NEW: + return self._client.new_session(self._cwd, **options) + assert self._requested_session_id is not None + if self._session_mode is SessionOpenMode.LOAD: + return self._client.load_session( + self._requested_session_id, self._cwd, **options + ) + return self._client.resume_session( + self._requested_session_id, self._cwd, **options + ) + + def _make_ingestor(self, session_id: str) -> AcpSessionIngestor: + if self._provided_ingestor is not None: + existing_session = getattr(self._provided_ingestor, "session_id", session_id) + if existing_session != session_id: + raise ValueError("provided ACP ingestor is bound to another session") + return self._provided_ingestor + return self._ingestor_factory( + self._config, + session_id=session_id, + stream_generation=self._stream_generation, + binding=self._binding, + ) + + def _consume_updates(self) -> None: + try: + while True: + try: + update = self._client.next_update(timeout=self._poll_timeout) + except TimeoutError: + with self._idle_condition: + self._update_idle_epoch += 1 + self._idle_condition.notify_all() + if self._stop_event.is_set(): + return + continue + if update.session_id != self._session_id: + raise AcpRuntimeProtocolError( + "ACP update belongs to a different session" + ) + ingestor = self._require_ingestor() + with self._ingest_lock: + ingestor.ingest_update(update.raw) + with self._state_lock: + self._updates_ingested += 1 + except BaseException as exc: + if not self._stop_event.is_set(): + self._record_failure(exc) + + def _consume_permissions(self) -> None: + try: + while True: + try: + request = self._client.next_permission_request( + timeout=self._poll_timeout + ) + except TimeoutError: + if self._stop_event.is_set(): + return + continue + if request.session_id != self._session_id: + raise AcpRuntimeProtocolError( + "ACP permission belongs to a different session" + ) + ingestor = self._require_ingestor() + with self._ingest_lock: + ingestor.ingest_permission_request( + request.raw, + source_event_id=f"permission:{request.request_id}", + ) + with self._state_lock: + self._permissions_ingested += 1 + + selected: str | None = None + callback_failure: BaseException | None = None + if self._permission_callback is not None: + try: + candidate = self._permission_callback(request) + if candidate is not None and candidate in { + option.option_id for option in request.options + }: + selected = candidate + elif candidate is not None: + with self._state_lock: + self._invalid_permission_selections += 1 + except BaseException as exc: + callback_failure = exc + if selected is None: + self._client.respond_permission( + request.request_id, + cancelled=True, + ) + with self._state_lock: + self._permissions_cancelled += 1 + else: + self._client.respond_permission( + request.request_id, + option_id=selected, + ) + with self._state_lock: + self._permissions_selected += 1 + if callback_failure is not None: + raise callback_failure + except BaseException as exc: + if not self._stop_event.is_set(): + self._record_failure(exc) + + def _wait_for_post_response_idle(self, timeout: float) -> None: + deadline = time.monotonic() + timeout + with self._idle_condition: + epoch = self._update_idle_epoch + while self._update_idle_epoch <= epoch: + if self._failure is not None: + raise self._failure + if self._state is not RuntimeState.RUNNING: + raise AcpRuntimeStateError( + "ACP runtime stopped before prompt updates drained" + ) + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AcpRuntimeStopTimeout( + "ACP prompt updates did not drain before the deadline" + ) + self._idle_condition.wait(timeout=remaining) + + def _running_components(self) -> tuple[str, AcpSessionIngestor]: + with self._state_lock: + if self._state is not RuntimeState.RUNNING: + raise AcpRuntimeStateError( + f"ACP runtime is not running ({self._state.value})" + ) + session_id = self._session_id + ingestor = self._ingestor + if session_id is None or ingestor is None: # pragma: no cover - invariant guard + raise AcpRuntimeStateError("ACP runtime has no bound session") + return session_id, ingestor + + def _require_ingestor(self) -> AcpSessionIngestor: + ingestor = self._ingestor + if ingestor is None: # pragma: no cover - consumers start after binding + raise AcpRuntimeStateError("ACP runtime has no ingestor") + return ingestor + + def _record_failure(self, failure: BaseException) -> None: + with self._idle_condition: + if self._failure is None: + self._failure = failure + self._state = RuntimeState.FAILED + self._stop_event.set() + self._idle_condition.notify_all() + + +__all__ = [ + "AcpRuntime", + "AcpRuntimeError", + "AcpRuntimeProtocolError", + "AcpRuntimeStateError", + "AcpRuntimeStatus", + "AcpRuntimeStopTimeout", + "PermissionCallback", + "RuntimeState", + "SessionOpenMode", +] diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py new file mode 100644 index 0000000..b40432a --- /dev/null +++ b/tests/test_acp_runtime.py @@ -0,0 +1,443 @@ +from __future__ import annotations + +import queue +import threading +import time +from pathlib import Path +from typing import Any + +import pytest + +from tendwire.backends.acp_client import AcpRequestTimeoutError +from tendwire.backends.acp_protocol import ( + PermissionOption, + PermissionOptionKind, + PermissionRequest, + PromptResult, + SessionResult, + SessionUpdate, + SessionUpdateKind, + StopReason, +) +from tendwire.backends.acp_runtime import ( + AcpRuntime, + AcpRuntimeProtocolError, + AcpRuntimeStopTimeout, + RuntimeState, + SessionOpenMode, +) +from tendwire.config import Config +from tendwire.core.models import WorkerBinding + + +_END = object() + + +class FakeClient: + def __init__(self) -> None: + self.updates: queue.Queue[SessionUpdate | object] = queue.Queue() + self.permissions: queue.Queue[PermissionRequest | object] = queue.Queue() + self.calls: list[tuple[str, tuple[Any, ...], dict[str, Any]]] = [] + self.permission_responses: list[tuple[object, str | None, bool]] = [] + self.prompt_result: object = PromptResult(StopReason.END_TURN, {}) + self.prompt_failure: BaseException | None = None + self.closed = False + + def initialize(self, **kwargs: Any) -> object: + self.calls.append(("initialize", (), kwargs)) + return object() + + def new_session(self, cwd: Path, **kwargs: Any) -> SessionResult: + self.calls.append(("new", (cwd,), kwargs)) + return SessionResult("session-private", None, (), {}) + + def load_session( + self, session_id: str, cwd: Path, **kwargs: Any + ) -> SessionResult: + self.calls.append(("load", (session_id, cwd), kwargs)) + return SessionResult(session_id, None, (), {}) + + def resume_session( + self, session_id: str, cwd: Path, **kwargs: Any + ) -> SessionResult: + self.calls.append(("resume", (session_id, cwd), kwargs)) + return SessionResult(session_id, None, (), {}) + + def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: + self.calls.append(("prompt", (session_id, prompt), kwargs)) + if self.prompt_failure is not None: + raise self.prompt_failure + return self.prompt_result + + def cancel(self, session_id: str) -> None: + self.calls.append(("cancel", (session_id,), {})) + + def next_update(self, *, timeout: float) -> SessionUpdate: + try: + value = self.updates.get(timeout=timeout) + except queue.Empty as exc: + raise AcpRequestTimeoutError("idle") from exc + if value is _END: + raise EOFError("closed") + assert isinstance(value, SessionUpdate) + return value + + def next_permission_request(self, *, timeout: float) -> PermissionRequest: + try: + value = self.permissions.get(timeout=timeout) + except queue.Empty as exc: + raise AcpRequestTimeoutError("idle") from exc + if value is _END: + raise EOFError("closed") + assert isinstance(value, PermissionRequest) + return value + + def respond_permission( + self, + request_id: object, + *, + option_id: str | None = None, + cancelled: bool = False, + ) -> None: + self.permission_responses.append((request_id, option_id, cancelled)) + + def close(self) -> None: + self.closed = True + self.updates.put(_END) + self.permissions.put(_END) + + +class FakeIngestor: + def __init__(self, session_id: str = "session-private") -> None: + self.session_id = session_id + self.started: list[str | None] = [] + self.updates: list[object] = [] + self.permissions: list[tuple[object, str | None]] = [] + self.completions = 0 + self.update_failure: BaseException | None = None + + def start_turn(self, *, producer_turn_id: str | None = None) -> str: + self.started.append(producer_turn_id) + return "opaque-turn" + + def ingest_update(self, raw: object) -> None: + if self.update_failure is not None: + raise self.update_failure + self.updates.append(raw) + + def ingest_permission_request( + self, raw: object, *, source_event_id: str | None = None + ) -> None: + self.permissions.append((raw, source_event_id)) + + def mark_prompt_complete(self) -> None: + self.completions += 1 + + +def binding() -> WorkerBinding: + return WorkerBinding( + host_id="host-a", + worker_id="worker-public", + worker_fingerprint="worker-fingerprint", + backend="herdr", + target_kind="pane_id", + target_value="pane-private-secret", + turn_target_kind="acp_session_id", + turn_target_value="session-private", + private_fingerprint="binding-private-secret", + ) + + +def runtime( + tmp_path: Path, + client: FakeClient, + ingestor: FakeIngestor | None = None, + **kwargs: Any, +) -> AcpRuntime: + return AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=tmp_path / "events.db"), + binding=binding(), + cwd=tmp_path, + stream_generation="generation-private-secret", + ingestor=ingestor or FakeIngestor(), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + **kwargs, + ) + + +def update(session_id: str = "session-private") -> SessionUpdate: + raw = { + "sessionId": session_id, + "update": { + "sessionUpdate": "agent_message_chunk", + "content": {"type": "text", "text": "answer"}, + }, + } + return SessionUpdate( + session_id, + SessionUpdateKind.AGENT_MESSAGE_CHUNK, + raw["update"], + None, + raw, + ) + + +def permission( + request_id: object = 7, session_id: str = "session-private" +) -> PermissionRequest: + options = ( + PermissionOption( + "allow-once", + "Allow once", + PermissionOptionKind.ALLOW_ONCE, + {"optionId": "allow-once", "name": "Allow once", "kind": "allow_once"}, + ), + PermissionOption( + "reject-once", + "Reject once", + PermissionOptionKind.REJECT_ONCE, + {"optionId": "reject-once", "name": "Reject once", "kind": "reject_once"}, + ), + ) + raw = { + "sessionId": session_id, + "toolCall": {"toolCallId": "tool-private"}, + "options": [dict(option.raw) for option in options], + } + return PermissionRequest( + request_id, + session_id, + raw["toolCall"], + options, + None, + raw, + ) + + +def wait_until(predicate, timeout: float = 1.0) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return + time.sleep(0.005) + raise AssertionError("condition did not become true") + + +def test_start_negotiates_opens_one_session_and_binds_factory(tmp_path: Path) -> None: + client = FakeClient() + captured: dict[str, object] = {} + ingestor = FakeIngestor() + + def factory(config: Config, **kwargs: object) -> FakeIngestor: + captured.update(kwargs) + captured["config"] = config + return ingestor + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=tmp_path / "events.db"), + binding=binding(), + cwd=tmp_path, + stream_generation="generation-private-secret", + client_capabilities={"fs": {"readTextFile": True}}, + ingestor_factory=factory, # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + try: + assert [call[0] for call in client.calls[:2]] == ["initialize", "new"] + assert client.calls[0][2]["client_capabilities"] == { + "fs": {"readTextFile": True} + } + assert captured["session_id"] == "session-private" + assert captured["binding"] is not None + assert captured["stream_generation"] == "generation-private-secret" + assert service.status().healthy + finally: + service.stop() + + +@pytest.mark.parametrize( + ("mode", "method"), + [(SessionOpenMode.LOAD, "load"), (SessionOpenMode.RESUME, "resume")], +) +def test_load_and_resume_use_requested_session( + tmp_path: Path, mode: SessionOpenMode, method: str +) -> None: + client = FakeClient() + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=tmp_path / "events.db"), + binding=binding(), + cwd=tmp_path, + session_mode=mode, + session_id="existing-private", + ingestor=FakeIngestor("existing-private"), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + try: + assert client.calls[1][0] == method + assert client.calls[1][1][0] == "existing-private" + finally: + service.stop() + + +def test_background_consumers_ingest_losslessly_and_permissions_fail_closed( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + client.updates.put(update()) + client.permissions.put(permission()) + wait_until(lambda: service.status().permissions_ingested == 1) + wait_until(lambda: service.status().updates_ingested == 1) + + assert len(ingestor.updates) == 1 + assert ingestor.permissions[0][1] == "permission:7" + assert client.permission_responses == [(7, None, True)] + assert service.status().permissions_cancelled == 1 + finally: + service.stop() + + +def test_callback_can_select_only_an_offered_permission(tmp_path: Path) -> None: + client = FakeClient() + decisions = iter(["not-offered", "allow-once"]) + service = runtime( + tmp_path, + client, + permission_callback=lambda _request: next(decisions), + ).start() + try: + client.permissions.put(permission(1)) + client.permissions.put(permission(2)) + wait_until(lambda: service.status().permissions_ingested == 2) + wait_until(lambda: len(client.permission_responses) == 2) + + assert client.permission_responses == [ + (1, None, True), + (2, "allow-once", False), + ] + status = service.status() + assert status.invalid_permission_selections == 1 + assert status.permissions_cancelled == 1 + assert status.permissions_selected == 1 + finally: + service.stop() + + +def test_callback_failure_cancels_permission_before_propagating(tmp_path: Path) -> None: + client = FakeClient() + callback_failure = LookupError("decision failed") + + def fail(_request: PermissionRequest) -> str: + raise callback_failure + + service = runtime(tmp_path, client, permission_callback=fail).start() + client.permissions.put(permission()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + assert client.permission_responses == [(7, None, True)] + assert service.status().permissions_cancelled == 1 + with pytest.raises(LookupError) as raised: + service.stop() + assert raised.value is callback_failure + + +def test_prompt_finalizes_only_after_valid_response_and_update_drain( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + client.updates.put(update()) + result = service.prompt("question", producer_turn_id="producer-private") + + assert result.stop_reason is StopReason.END_TURN + assert ingestor.started == ["producer-private"] + assert len(ingestor.updates) == 1 + assert ingestor.completions == 1 + status = service.status() + assert status.prompts_started == 1 + assert status.prompts_completed == 1 + assert status.prompts_failed == 0 + finally: + service.stop() + + +def test_invalid_prompt_response_never_marks_complete_and_propagates( + tmp_path: Path, +) -> None: + client = FakeClient() + client.prompt_result = {"stopReason": "end_turn"} + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRuntimeProtocolError, match="invalid response"): + service.prompt("question") + assert ingestor.completions == 0 + assert service.status().state is RuntimeState.FAILED + with pytest.raises(AcpRuntimeProtocolError): + service.stop() + + +def test_background_ingestion_failure_is_propagated_and_status_is_redacted( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + failure = OSError("session-private pane-private-secret") + ingestor.update_failure = failure + service = runtime(tmp_path, client, ingestor).start() + client.updates.put(update()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + status = service.status() + assert not status.healthy + assert status.failure_type == "OSError" + rendered = repr(status) + assert "session-private" not in rendered + assert "pane-private-secret" not in rendered + assert "generation-private-secret" not in rendered + with pytest.raises(OSError) as raised: + service.raise_if_failed() + assert raised.value is failure + with pytest.raises(OSError): + service.stop() + + +def test_cancel_targets_bound_session_and_stop_joins_consumers(tmp_path: Path) -> None: + client = FakeClient() + service = runtime(tmp_path, client).start() + service.cancel() + assert ("cancel", ("session-private",), {}) in client.calls + assert service.status().cancellation_requests == 1 + + service.stop() + assert client.closed + assert service.join(timeout=0.1) + assert service.status().state is RuntimeState.STOPPED + + +def test_stop_deadline_is_bounded_even_when_client_close_hangs(tmp_path: Path) -> None: + client = FakeClient() + release_close = threading.Event() + + def hanging_close() -> None: + release_close.wait(timeout=1) + + client.close = hanging_close # type: ignore[method-assign] + service = runtime(tmp_path, client).start() + started = time.monotonic() + try: + with pytest.raises(AcpRuntimeStopTimeout): + service.stop(timeout=0.05) + assert time.monotonic() - started < 0.25 + finally: + release_close.set() From b26f7d1ef4cf2b07c17f46454b79934bd3478824 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:16:43 +0800 Subject: [PATCH 08/83] docs: define rebasing-free ACP adapter upgrades --- docs/acp-migration.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/acp-migration.md b/docs/acp-migration.md index a508938..d1eff91 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -74,6 +74,28 @@ Herdres must never receive a raw thought event. A future public summary feature requires a separate schema, sanitizer, explicit operator opt-in, and tests that prove raw reasoning cannot cross the boundary. +## Upstream upgrade boundary + +Tendwire integrates with the stable ACP wire protocol, not an adapter's source +tree. Official adapters such as `codex-acp` and `claude-agent-acp` remain +separately installed executables and must be replaceable without vendoring, +rebasing, or resolving Tendwire source conflicts. + +The boundary has four rules: + +- negotiate protocol version and capabilities at every process start; +- never import adapter implementation modules or depend on their repository + layout, generated internal types, commits, or private event handlers; +- ignore unknown standard update variants conservatively and retain explicitly + namespaced extension metadata only on Tendwire's private side; +- verify adapter releases with black-box ACP compatibility fixtures before + promotion, while keeping the previously proven executable for rollback. + +An adapter upgrade therefore restarts only its owned process/session; it does +not require a Tendwire rebase. A session may resume when the new adapter +advertises that capability. Otherwise Tendwire opens a new transport generation +and reconciles it through the durable semantic journal. + ## Runtime lifecycle For ACP v1 stdio, the component that owns the adapter process also owns framing, From ac5c9c8174dcde8e33238d9bdf1d3e17b8c9a887 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:20:21 +0800 Subject: [PATCH 09/83] Harden ACP event journal invariants --- src/tendwire/core/agent_events.py | 164 +++++++--- src/tendwire/store/sqlite.py | 480 +++++++++++++++++++++++++----- tests/test_agent_events.py | 381 +++++++++++++++++++++++- 3 files changed, 915 insertions(+), 110 deletions(-) diff --git a/src/tendwire/core/agent_events.py b/src/tendwire/core/agent_events.py index 5037178..cef01c0 100644 --- a/src/tendwire/core/agent_events.py +++ b/src/tendwire/core/agent_events.py @@ -11,10 +11,10 @@ import hashlib import json import math -import unicodedata from collections.abc import Mapping +from copy import deepcopy from dataclasses import dataclass -from datetime import datetime +from datetime import datetime, timezone from typing import Any, Literal from .models import sanitize_public_mapping, utc_timestamp @@ -28,6 +28,7 @@ "plan", "usage", "session_info", + "extension", ] AgentEventVisibility = Literal["private", "public"] @@ -41,6 +42,7 @@ "plan", "usage", "session_info", + "extension", } ) AGENT_EVENT_VISIBILITIES = frozenset({"private", "public"}) @@ -48,6 +50,7 @@ AGENT_EVENT_MAX_PUBLIC_PAYLOAD_BYTES = 64 * 1024 AGENT_EVENT_MAX_TEXT_CHARS = 32 * 1024 AGENT_EVENT_MAX_COLLECTION_ITEMS = 256 +AGENT_EVENT_MAX_TOTAL_ITEMS = 4096 AGENT_EVENT_MAX_DEPTH = 12 AGENT_EVENT_MAX_IDENTIFIER_CHARS = 2048 AGENT_EVENT_QUERY_DEFAULT_LIMIT = 100 @@ -69,24 +72,79 @@ def _fingerprint(value: Any) -> str: return hashlib.sha256(_canonical_json(value).encode("utf-8")).hexdigest() -def _identifier(value: Any, field: str, *, required: bool = False) -> str | None: +def normalize_agent_event_identifier( + value: Any, + field: str, + *, + required: bool = False, +) -> str | None: + """Validate an opaque identifier without changing its source identity. + + Protocol identifiers are byte-significant. Compatibility normalization or + trimming would make distinct source IDs collide (for example ``"1"`` and + ``"①"`` under NFKC), defeating the journal's replay-conflict checks. + """ if value is None: if required: raise ValueError(f"{field} must not be empty") return None if not isinstance(value, str): raise ValueError(f"{field} must be text or None") - normalized = unicodedata.normalize("NFKC", value).replace("\x00", "").strip() - if not normalized: - if required: - raise ValueError(f"{field} must not be empty") - return None - if len(normalized) > AGENT_EVENT_MAX_IDENTIFIER_CHARS: + if not value.strip(): + raise ValueError(f"{field} must not be empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain NUL") + if len(value) > AGENT_EVENT_MAX_IDENTIFIER_CHARS: raise ValueError(f"{field} is too long") - return normalized + try: + value.encode("utf-8") + except UnicodeEncodeError as exc: + raise ValueError(f"{field} must contain valid Unicode") from exc + return value -def _normalize_payload_value(value: Any, *, depth: int = 0) -> Any: +def _timestamp(value: str | datetime | None) -> str: + if value is None: + return utc_timestamp() + if isinstance(value, datetime): + parsed = value + elif isinstance(value, str): + raw = value.strip() + if not raw or len(raw) > 64: + raise ValueError("observed_at must be an aware ISO-8601 timestamp") + if raw.endswith(("Z", "z")): + raw = raw[:-1] + "+00:00" + try: + parsed = datetime.fromisoformat(raw) + except ValueError as exc: + raise ValueError( + "observed_at must be an aware ISO-8601 timestamp" + ) from exc + else: + raise ValueError("observed_at must be an aware ISO-8601 timestamp") + if parsed.tzinfo is None or parsed.utcoffset() is None: + raise ValueError("observed_at must be an aware ISO-8601 timestamp") + return parsed.astimezone(timezone.utc).isoformat() + + +def _valid_unicode(value: str) -> str: + try: + value.encode("utf-8") + except UnicodeEncodeError as exc: + raise ValueError( + "agent event payload contains invalid Unicode" + ) from exc + return value + + +def _normalize_payload_value( + value: Any, + *, + depth: int = 0, + item_budget: list[int] | None = None, +) -> Any: + if item_budget is None: + item_budget = [AGENT_EVENT_MAX_TOTAL_ITEMS] if depth > AGENT_EVENT_MAX_DEPTH: raise ValueError("agent event payload is nested too deeply") if value is None or isinstance(value, bool | int): @@ -96,33 +154,45 @@ def _normalize_payload_value(value: Any, *, depth: int = 0) -> Any: raise ValueError("agent event payload contains a non-finite number") return value if isinstance(value, datetime): + if value.tzinfo is None or value.utcoffset() is None: + raise ValueError("agent event payload contains a naive datetime") return utc_timestamp(value) if isinstance(value, str): - normalized = unicodedata.normalize("NFKC", value).replace("\x00", "") - if len(normalized) > AGENT_EVENT_MAX_TEXT_CHARS: + if len(value) > AGENT_EVENT_MAX_TEXT_CHARS: raise ValueError("agent event payload text is too long") - return normalized + return _valid_unicode(value) if isinstance(value, Mapping): if len(value) > AGENT_EVENT_MAX_COLLECTION_ITEMS: raise ValueError("agent event payload mapping has too many entries") + item_budget[0] -= len(value) + if item_budget[0] < 0: + raise ValueError("agent event payload has too many total items") result: dict[str, Any] = {} for raw_key, item in value.items(): if not isinstance(raw_key, str): raise ValueError("agent event payload keys must be text") - key = unicodedata.normalize("NFKC", raw_key).replace("\x00", "") + key = _valid_unicode(raw_key) if not key or len(key) > 256: raise ValueError("agent event payload contains an invalid key") - if key in result: - raise ValueError( - "agent event payload keys collide after normalization" - ) - result[key] = _normalize_payload_value(item, depth=depth + 1) + result[key] = _normalize_payload_value( + item, + depth=depth + 1, + item_budget=item_budget, + ) return result if isinstance(value, tuple | list): if len(value) > AGENT_EVENT_MAX_COLLECTION_ITEMS: raise ValueError("agent event payload sequence has too many entries") + item_budget[0] -= len(value) + if item_budget[0] < 0: + raise ValueError("agent event payload has too many total items") return [ - _normalize_payload_value(item, depth=depth + 1) for item in value + _normalize_payload_value( + item, + depth=depth + 1, + item_budget=item_budget, + ) + for item in value ] raise ValueError("agent event payload must contain only JSON-safe values") @@ -186,7 +256,7 @@ def public_dict(self, *, sequence: int | None = None) -> dict[str, Any]: "worker_id": self.worker_id, "visibility": self.visibility, "observed_at": self.observed_at, - "payload": dict(self.public_payload), + "payload": deepcopy(self.public_payload), } if sequence is not None: result["sequence"] = int(sequence) @@ -206,7 +276,7 @@ def agent_event( source_event_id: str | None = None, source_sequence: int | None = None, visibility: AgentEventVisibility | str = "private", - observed_at: str | None = None, + observed_at: str | datetime | None = None, ) -> AgentEvent: """Validate and construct an event with deterministic retry identity. @@ -222,13 +292,23 @@ def agent_event( raise ValueError("visibility must be private or public") if normalized_kind == "thought" and normalized_visibility != "private": raise ValueError("thought events must remain private") - normalized_source = _identifier(source, "source", required=True) - normalized_worker = _identifier(worker_id, "worker_id", required=True) - session_id = _identifier(source_session_id, "source_session_id") - turn_id = _identifier(source_turn_id, "source_turn_id") - item_id = _identifier(source_item_id, "source_item_id") - message_id = _identifier(source_message_id, "source_message_id") - event_id = _identifier(source_event_id, "source_event_id") + if normalized_kind == "extension" and normalized_visibility != "private": + raise ValueError("extension events must remain private") + normalized_source = normalize_agent_event_identifier( + source, "source", required=True + ) + normalized_worker = normalize_agent_event_identifier( + worker_id, "worker_id", required=True + ) + session_id = normalize_agent_event_identifier( + source_session_id, "source_session_id" + ) + turn_id = normalize_agent_event_identifier(source_turn_id, "source_turn_id") + item_id = normalize_agent_event_identifier(source_item_id, "source_item_id") + message_id = normalize_agent_event_identifier( + source_message_id, "source_message_id" + ) + event_id = normalize_agent_event_identifier(source_event_id, "source_event_id") if source_sequence is not None and ( isinstance(source_sequence, bool) or not isinstance(source_sequence, int) @@ -245,21 +325,22 @@ def agent_event( normalized_payload, visibility=normalized_visibility, # type: ignore[arg-type] ) - identity = { + identity: dict[str, Any] = { "schema_version": AGENT_EVENT_SCHEMA_VERSION, "source": normalized_source, "session_id": session_id, - "event_id": event_id, - "sequence": source_sequence, - "kind": normalized_kind, } + if event_id is not None: + identity.update({"identity": "event_id", "value": event_id}) + else: + identity.update({"identity": "sequence", "value": source_sequence}) return AgentEvent( event_id=_fingerprint(identity), kind=normalized_kind, # type: ignore[arg-type] source=normalized_source or "", worker_id=normalized_worker or "", visibility=normalized_visibility, # type: ignore[arg-type] - observed_at=_identifier(observed_at, "observed_at") or utc_timestamp(), + observed_at=_timestamp(observed_at), payload=normalized_payload, public_payload=public_payload, payload_fingerprint=_fingerprint(normalized_payload), @@ -291,5 +372,18 @@ class AppendAgentEventResult: inserted: bool +@dataclass(frozen=True) +class AppendBoundAgentEventResult: + """Atomic binding check and journal append outcome.""" + + status: Literal["inserted", "replayed", "binding_changed"] + event_id: str + sequence: int | None = None + + @property + def inserted(self) -> bool: + return self.status == "inserted" + + class AgentEventIdentityConflict(RuntimeError): """The same deterministic source identity was reused for other content.""" diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index d78d4e9..d3b26eb 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -70,8 +70,10 @@ AgentEvent, AgentEventIdentityConflict, AppendAgentEventResult, + AppendBoundAgentEventResult, StoredAgentEvent, agent_event, + normalize_agent_event_identifier, ) from ..core.commands import ( CommandEnvelope, @@ -143,7 +145,7 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS -STORE_SCHEMA_VERSION = 22 +STORE_SCHEMA_VERSION = 23 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS _LEGACY_TURN_CLAIM_HARD_TTL_SECONDS = 86_400 TURN_CHANGE_RETENTION_DAYS = 7 @@ -1536,7 +1538,7 @@ def _record_response_size( kind TEXT NOT NULL CHECK ( kind IN ( 'user_message', 'agent_message', 'thought', 'tool_call', - 'tool_call_update', 'plan', 'usage', 'session_info' + 'tool_call_update', 'plan', 'usage', 'session_info', 'extension' ) ), source TEXT NOT NULL, @@ -1553,8 +1555,66 @@ def _record_response_size( private_payload_json TEXT NOT NULL, public_payload_json TEXT NOT NULL, UNIQUE (host_id, event_id), + CHECK (length(host_id) BETWEEN 1 AND 2048), + CHECK (instr(host_id, char(0)) = 0), + CHECK (length(event_id) = 64), + CHECK (length(source) BETWEEN 1 AND 2048), + CHECK (instr(source, char(0)) = 0), + CHECK (length(worker_id) BETWEEN 1 AND 2048), + CHECK (instr(worker_id, char(0)) = 0), + CHECK ( + source_session_id IS NULL OR ( + length(source_session_id) BETWEEN 1 AND 2048 + AND instr(source_session_id, char(0)) = 0 + ) + ), + CHECK ( + source_turn_id IS NULL OR ( + length(source_turn_id) BETWEEN 1 AND 2048 + AND instr(source_turn_id, char(0)) = 0 + ) + ), + CHECK ( + source_item_id IS NULL OR ( + length(source_item_id) BETWEEN 1 AND 2048 + AND instr(source_item_id, char(0)) = 0 + ) + ), + CHECK ( + source_message_id IS NULL OR ( + length(source_message_id) BETWEEN 1 AND 2048 + AND instr(source_message_id, char(0)) = 0 + ) + ), + CHECK ( + source_event_id IS NULL OR ( + length(source_event_id) BETWEEN 1 AND 2048 + AND instr(source_event_id, char(0)) = 0 + ) + ), + CHECK (length(observed_at) BETWEEN 20 AND 40), + CHECK (length(payload_fingerprint) = 64), + CHECK ( + CASE WHEN json_valid(private_payload_json) + THEN json_type(private_payload_json) = 'object' ELSE 0 END + ), + CHECK ( + length(CAST(private_payload_json AS BLOB)) <= 65536 + ), + CHECK ( + CASE WHEN json_valid(public_payload_json) + THEN json_type(public_payload_json) = 'object' ELSE 0 END + ), + CHECK ( + length(CAST(public_payload_json AS BLOB)) <= 65536 + ), + CHECK (visibility != 'private' OR public_payload_json = '{}'), CHECK (source_event_id IS NOT NULL OR source_sequence IS NOT NULL), - CHECK (kind != 'thought' OR visibility = 'private') + CHECK ( + source_event_id IS NOT NULL + OR (source_session_id IS NOT NULL AND source_sequence IS NOT NULL) + ), + CHECK (kind NOT IN ('thought', 'extension') OR visibility = 'private') ); """ @@ -1575,6 +1635,27 @@ def _record_response_size( "CREATE INDEX IF NOT EXISTS idx_agent_events_host_source_sequence " "ON agent_events(host_id, source, sequence)" ), + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_visibility_sequence " + "ON agent_events(host_id, visibility, sequence)" + ), + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_observed_sequence " + "ON agent_events(host_id, observed_at, sequence)" + ), + ( + "CREATE UNIQUE INDEX IF NOT EXISTS " + "idx_agent_events_source_event_identity " + "ON agent_events(" + "host_id, source, COALESCE(source_session_id, ''), source_event_id" + ") WHERE source_event_id IS NOT NULL" + ), + ( + "CREATE UNIQUE INDEX IF NOT EXISTS " + "idx_agent_events_source_sequence_identity " + "ON agent_events(host_id, source, source_session_id, source_sequence) " + "WHERE source_event_id IS NULL" + ), ) CREATE_PR6_TABLES = ( @@ -13190,6 +13271,114 @@ def _migrate_v21_to_v22_conn(conn: sqlite3.Connection) -> None: conn.execute(statement) +def _migrate_v22_to_v23_conn(conn: sqlite3.Connection) -> None: + """Harden event identity and rebuild v22 rows under the canonical contract.""" + columns = [ + "sequence", + "host_id", + "event_id", + "kind", + "source", + "worker_id", + "visibility", + "source_session_id", + "source_turn_id", + "source_item_id", + "source_message_id", + "source_event_id", + "source_sequence", + "observed_at", + "payload_fingerprint", + "private_payload_json", + "public_payload_json", + ] + rows = conn.execute( + "SELECT " + ", ".join(columns) + " FROM agent_events ORDER BY sequence" + ).fetchall() + conn.execute("ALTER TABLE agent_events RENAME TO agent_events_v22") + conn.execute(CREATE_AGENT_EVENTS_TABLE) + try: + for row in rows: + try: + private_payload = _json_object(row[15]) + public_payload = _json_object(row[16]) + if _canonical_json(private_payload) != row[15]: + raise StoreSchemaError("invalid_v22_agent_event_payload") + if _canonical_json(public_payload) != row[16]: + raise StoreSchemaError("invalid_v22_agent_event_projection") + host_id = normalize_agent_event_identifier( + row[1], "host_id", required=True + ) + canonical = agent_event( + kind=row[3], + source=row[4], + worker_id=row[5], + visibility=row[6], + source_session_id=row[7], + source_turn_id=row[8], + source_item_id=row[9], + source_message_id=row[10], + source_event_id=row[11], + source_sequence=row[12], + observed_at=row[13], + payload=private_payload, + ) + legacy_identity = { + "schema_version": 1, + "source": canonical.source, + "session_id": canonical.source_session_id, + "event_id": canonical.source_event_id, + "sequence": canonical.source_sequence, + "kind": canonical.kind, + } + legacy_event_id = hashlib.sha256( + _canonical_json(legacy_identity).encode("utf-8") + ).hexdigest() + if str(row[2]) != legacy_event_id: + raise StoreSchemaError("invalid_v22_agent_event_identity") + if str(row[14]) != canonical.payload_fingerprint: + raise StoreSchemaError("invalid_v22_agent_event_fingerprint") + if public_payload != canonical.public_payload: + raise StoreSchemaError("invalid_v22_agent_event_projection") + except (TypeError, ValueError, OverflowError) as exc: + raise StoreSchemaError("invalid_v22_agent_event_row") from exc + conn.execute( + """ + INSERT INTO agent_events ( + sequence, host_id, event_id, kind, source, worker_id, + visibility, source_session_id, source_turn_id, + source_item_id, source_message_id, source_event_id, + source_sequence, observed_at, payload_fingerprint, + private_payload_json, public_payload_json + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + int(row[0]), + host_id, + canonical.event_id, + canonical.kind, + canonical.source, + canonical.worker_id, + canonical.visibility, + canonical.source_session_id, + canonical.source_turn_id, + canonical.source_item_id, + canonical.source_message_id, + canonical.source_event_id, + canonical.source_sequence, + canonical.observed_at, + canonical.payload_fingerprint, + _canonical_json(canonical.payload), + _canonical_json(canonical.public_payload), + ), + ) + except sqlite3.IntegrityError as exc: + raise StoreSchemaError("conflicting_v22_agent_event_identity") from exc + conn.execute("DROP TABLE agent_events_v22") + for statement in CREATE_AGENT_EVENT_INDEXES: + conn.execute(statement) + + MIGRATIONS: tuple[Migration, ...] = ( Migration(0, 1, _migrate_v0_to_v1_conn), Migration(1, 2, _migrate_v1_to_v2_conn), @@ -13213,6 +13402,7 @@ def _migrate_v21_to_v22_conn(conn: sqlite3.Connection) -> None: Migration(19, 20, _migrate_v19_to_v20_conn), Migration(20, 21, _migrate_v20_to_v21_conn), Migration(21, 22, _migrate_v21_to_v22_conn), + Migration(22, 23, _migrate_v22_to_v23_conn), ) @@ -13424,10 +13614,11 @@ def _agent_event_from_row(row: tuple[Any, ...]) -> StoredAgentEvent: kind = str(row[3]) if kind not in AGENT_EVENT_KINDS: raise StoreSchemaError("invalid_agent_event_kind") - return StoredAgentEvent( - sequence=int(row[0]), - host_id=str(row[1]), - event=AgentEvent( + try: + normalized_host = normalize_agent_event_identifier( + row[1], "host_id", required=True + ) + stored_event = AgentEvent( event_id=str(row[2]), kind=kind, # type: ignore[arg-type] source=str(row[4]), @@ -13443,7 +13634,29 @@ def _agent_event_from_row(row: tuple[Any, ...]) -> StoredAgentEvent: payload_fingerprint=str(row[14]), payload=private_payload, public_payload=public_payload, - ), + ) + canonical = agent_event( + kind=stored_event.kind, + source=stored_event.source, + worker_id=stored_event.worker_id, + payload=stored_event.payload, + source_session_id=stored_event.source_session_id, + source_turn_id=stored_event.source_turn_id, + source_item_id=stored_event.source_item_id, + source_message_id=stored_event.source_message_id, + source_event_id=stored_event.source_event_id, + source_sequence=stored_event.source_sequence, + visibility=stored_event.visibility, + observed_at=stored_event.observed_at, + ) + except (TypeError, ValueError, OverflowError) as exc: + raise StoreSchemaError("invalid_agent_event_row") from exc + if canonical != stored_event: + raise StoreSchemaError("invalid_agent_event_row") + return StoredAgentEvent( + sequence=int(row[0]), + host_id=normalized_host or "", + event=stored_event, ) @@ -13476,19 +13689,7 @@ def _agent_event_conflicts(existing: StoredAgentEvent, incoming: AgentEvent) -> ) -def append_agent_event( - db_path: Path | str, - host_id: str, - event: AgentEvent, -) -> AppendAgentEventResult: - """Append one structured event, or return its existing replay sequence. - - Reusing a deterministic event identity with different content is rejected - instead of silently mutating the journal or accepting source corruption. - """ - normalized_host = str(host_id).strip() - if not normalized_host: - raise ValueError("host_id must not be empty") +def _canonical_agent_event_for_append(event: AgentEvent) -> AgentEvent: if not isinstance(event, AgentEvent): raise ValueError("event must be an AgentEvent") canonical_event = agent_event( @@ -13507,61 +13708,136 @@ def append_agent_event( ) if canonical_event != event: raise ValueError("event must use the canonical agent event contract") + return canonical_event + + +def _append_agent_event_conn( + conn: sqlite3.Connection, + host_id: str, + event: AgentEvent, +) -> AppendAgentEventResult: private_json = _canonical_json(event.payload) public_json = _canonical_json(event.public_payload) + cursor = conn.execute( + """ + INSERT INTO agent_events ( + host_id, event_id, kind, source, worker_id, visibility, + source_session_id, source_turn_id, source_item_id, + source_message_id, source_event_id, source_sequence, + observed_at, payload_fingerprint, private_payload_json, + public_payload_json + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(host_id, event_id) DO NOTHING + """, + ( + host_id, + event.event_id, + event.kind, + event.source, + event.worker_id, + event.visibility, + event.source_session_id, + event.source_turn_id, + event.source_item_id, + event.source_message_id, + event.source_event_id, + event.source_sequence, + event.observed_at, + event.payload_fingerprint, + private_json, + public_json, + ), + ) + inserted = cursor.rowcount == 1 + row = conn.execute( + _AGENT_EVENT_SELECT + " WHERE host_id = ? AND event_id = ?", + (host_id, event.event_id), + ).fetchone() + if row is None: + raise StoreSchemaError("agent_event_append_failed") + stored = _agent_event_from_row(row) + if _agent_event_conflicts(stored, event): + raise AgentEventIdentityConflict(event.event_id) + return AppendAgentEventResult( + sequence=stored.sequence, + event_id=event.event_id, + inserted=inserted, + ) + + +def append_agent_event( + db_path: Path | str, + host_id: str, + event: AgentEvent, +) -> AppendAgentEventResult: + """Append one structured event, or return its existing replay sequence. + + Reusing a deterministic event identity with different content is rejected + instead of silently mutating the journal or accepting source corruption. + """ + normalized_host = normalize_agent_event_identifier( + host_id, "host_id", required=True + ) + _canonical_agent_event_for_append(event) with _connect(db_path, prepare=True) as conn: _ensure_schema(conn) conn.execute("BEGIN IMMEDIATE") try: - cursor = conn.execute( - """ - INSERT INTO agent_events ( - host_id, event_id, kind, source, worker_id, visibility, - source_session_id, source_turn_id, source_item_id, - source_message_id, source_event_id, source_sequence, - observed_at, payload_fingerprint, private_payload_json, - public_payload_json - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(host_id, event_id) DO NOTHING - """, - ( - normalized_host, - event.event_id, - event.kind, - event.source, - event.worker_id, - event.visibility, - event.source_session_id, - event.source_turn_id, - event.source_item_id, - event.source_message_id, - event.source_event_id, - event.source_sequence, - event.observed_at, - event.payload_fingerprint, - private_json, - public_json, - ), + result = _append_agent_event_conn(conn, normalized_host or "", event) + conn.commit() + except Exception: + conn.rollback() + raise + return result + + +def append_agent_event_for_binding( + db_path: Path | str, + host_id: str, + event: AgentEvent, + *, + expected_binding: WorkerBinding, +) -> AppendBoundAgentEventResult: + """Append only while the expected active worker binding remains current. + + The binding check and insert share one ``BEGIN IMMEDIATE`` transaction, so + a concurrent inventory refresh cannot invalidate the binding between the + check and journal mutation. + """ + normalized_host = normalize_agent_event_identifier( + host_id, "host_id", required=True + ) + _canonical_agent_event_for_append(event) + if not isinstance(expected_binding, WorkerBinding): + raise ValueError("expected_binding must be a WorkerBinding") + with _connect(db_path, prepare=True) as conn: + _ensure_schema(conn) + conn.execute("BEGIN IMMEDIATE") + try: + if not _agent_event_binding_matches_conn( + conn, + normalized_host or "", + event.worker_id, + expected_binding, + ): + conn.rollback() + return AppendBoundAgentEventResult( + status="binding_changed", + event_id=event.event_id, + ) + result = _append_agent_event_conn( + conn, + normalized_host or "", + event, ) - inserted = cursor.rowcount == 1 - row = conn.execute( - _AGENT_EVENT_SELECT - + " WHERE host_id = ? AND event_id = ?", - (normalized_host, event.event_id), - ).fetchone() - if row is None: - raise StoreSchemaError("agent_event_append_failed") - stored = _agent_event_from_row(row) - if _agent_event_conflicts(stored, event): - raise AgentEventIdentityConflict(event.event_id) conn.commit() except Exception: conn.rollback() raise - return AppendAgentEventResult( - sequence=stored.sequence, - event_id=event.event_id, - inserted=inserted, + return AppendBoundAgentEventResult( + status="inserted" if result.inserted else "replayed", + event_id=result.event_id, + sequence=result.sequence, ) @@ -13591,6 +13867,7 @@ def list_agent_events( isinstance(after_sequence, bool) or not isinstance(after_sequence, int) or after_sequence < 0 + or after_sequence > (1 << 63) - 1 ): raise ValueError("after_sequence must be a nonnegative integer") if ( @@ -13601,20 +13878,29 @@ def list_agent_events( raise ValueError( f"limit must be between 1 and {AGENT_EVENT_QUERY_MAX_LIMIT}" ) + normalized_host = normalize_agent_event_identifier( + host_id, "host_id", required=True + ) + normalized_filters: list[tuple[str, str | None]] = [] + for column, value, field in ( + ("worker_id", worker_id, "worker_id"), + ("source", source, "source"), + ("source_session_id", session_id, "source_session_id"), + ("source_turn_id", turn_id, "source_turn_id"), + ): + normalized_filters.append( + (column, normalize_agent_event_identifier(value, field)) + ) + if visibility is not None and visibility not in {"private", "public"}: + raise ValueError("visibility must be private, public, or None") if not _sqlite_store_exists(db_path): return () clauses = ["host_id = ?", "sequence > ?"] - parameters: list[Any] = [str(host_id), int(after_sequence)] - for column, value in ( - ("worker_id", worker_id), - ("source", source), - ("source_session_id", session_id), - ("source_turn_id", turn_id), - ("visibility", visibility), - ): + parameters: list[Any] = [normalized_host, int(after_sequence)] + for column, value in (*normalized_filters, ("visibility", visibility)): if value is not None: clauses.append(f"{column} = ?") - parameters.append(str(value)) + parameters.append(value) parameters.append(int(limit)) with _connect(db_path) as conn: _ensure_schema(conn) @@ -22525,6 +22811,52 @@ def _turn_refresh_binding_matches_conn( ) +def _agent_event_binding_matches_conn( + conn: sqlite3.Connection, + host_id: str, + worker_id: str, + expected: WorkerBinding, +) -> bool: + """Match the complete routing generation used by an agent event stream.""" + if ( + str(expected.host_id) != str(host_id) + or str(expected.worker_id) != str(worker_id) + ): + return False + row = conn.execute( + """ + SELECT + worker_id, + worker_fingerprint, + backend, + target_kind, + target_value, + turn_target_kind, + turn_target_value + FROM worker_bindings + WHERE host_id = ? + AND backend = ? + AND private_fingerprint = ? + AND expires_at > ? + """, + ( + str(host_id), + str(expected.backend), + str(expected.private_fingerprint), + utc_timestamp(), + ), + ).fetchone() + return row is not None and tuple(row) == ( + str(expected.worker_id), + str(expected.worker_fingerprint), + str(expected.backend), + str(expected.target_kind), + str(expected.target_value), + expected.turn_target_kind, + expected.turn_target_value, + ) + + def _turn_refresh_is_cancelled( *, deadline_monotonic: float | None, diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py index f5b2608..1083f8c 100644 --- a/tests/test_agent_events.py +++ b/tests/test_agent_events.py @@ -1,7 +1,10 @@ from __future__ import annotations +import hashlib import sqlite3 +from concurrent.futures import ThreadPoolExecutor from dataclasses import replace +from datetime import datetime, timezone from pathlib import Path import pytest @@ -9,9 +12,11 @@ from tendwire.core.agent_events import ( AGENT_EVENT_KINDS, AGENT_EVENT_MAX_PAYLOAD_BYTES, + AGENT_EVENT_MAX_TOTAL_ITEMS, AgentEventIdentityConflict, agent_event, ) +from tendwire.core.models import WorkerBinding from tendwire.store import sqlite as store_sqlite @@ -46,6 +51,7 @@ def test_agent_event_contract_covers_acp_primary_kinds() -> None: "plan", "usage", "session_info", + "extension", } @@ -84,6 +90,48 @@ def test_deterministic_identity_rejects_changed_replay(tmp_path: Path) -> None: assert stored[0].event.payload == {"text": "original"} +def test_source_identity_reuse_cannot_evade_conflict_by_changing_kind_or_sequence( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + original = _message_event(sequence=4) + changed_kind = agent_event( + kind="plan", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_event_id="stable-source-id", + source_sequence=5, + payload={"entries": []}, + ) + original_with_id = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_event_id="stable-source-id", + source_sequence=4, + payload={"text": "hello"}, + ) + assert original_with_id.event_id == changed_kind.event_id + store_sqlite.append_agent_event(db_path, "host-1", original_with_id) + with pytest.raises(AgentEventIdentityConflict): + store_sqlite.append_agent_event(db_path, "host-1", changed_kind) + + changed_sequence_kind = agent_event( + kind="plan", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_sequence=original.source_sequence, + payload={"entries": []}, + ) + assert original.event_id == changed_sequence_kind.event_id + store_sqlite.append_agent_event(db_path, "host-2", original) + with pytest.raises(AgentEventIdentityConflict): + store_sqlite.append_agent_event(db_path, "host-2", changed_sequence_kind) + + def test_private_ids_and_payload_never_enter_public_projection(tmp_path: Path) -> None: db_path = tmp_path / "store.db" event = agent_event( @@ -188,6 +236,122 @@ def test_payload_is_bounded_and_json_safe() -> None: source_event_id="event-1", payload={"opaque": object()}, ) + for value in (float("nan"), float("inf"), float("-inf")): + with pytest.raises(ValueError, match="non-finite"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={"tokens": value}, + ) + with pytest.raises(ValueError, match="too many total items"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={ + str(index): [0] * 256 + for index in range((AGENT_EVENT_MAX_TOTAL_ITEMS // 256) + 1) + }, + ) + + +def test_payload_and_opaque_ids_preserve_valid_unicode_exactly() -> None: + circled = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_event_id="①", + visibility="public", + payload={"text": "① and fullwidth ｅ and NUL \x00 remain private-exact"}, + ) + ascii_event = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_event_id="1", + payload={"text": "1"}, + ) + assert circled.event_id != ascii_event.event_id + assert circled.source_event_id == "①" + assert circled.payload["text"] == "① and fullwidth ｅ and NUL \x00 remain private-exact" + assert "\x00" not in str(circled.public_payload.get("text", "")) + + with pytest.raises(ValueError, match="valid Unicode"): + _message_event(sequence=9, text="\ud800") + with pytest.raises(ValueError, match="must not contain NUL"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="bad\x00id", + payload={}, + ) + + +def test_journal_payload_is_adapter_neutral_and_preserves_namespaced_extensions( + tmp_path: Path, +) -> None: + payload = { + "text": "portable message", + "org.example.agent/experimental-v2": { + "futureField": [1, "two", {"enabled": True}], + "_meta": {"opaque": "retained privately"}, + }, + } + event = agent_event( + kind="extension", + source="org.example.agent/v2", + worker_id="worker-1", + source_event_id="extension-event", + payload=payload, + ) + store_sqlite.append_agent_event(tmp_path / "store.db", "host-1", event) + stored = store_sqlite.list_agent_events(tmp_path / "store.db", "host-1") + assert stored[0].event.source == "org.example.agent/v2" + assert stored[0].event.payload == payload + with pytest.raises(ValueError, match="extension events must remain private"): + agent_event( + kind="extension", + source="org.example.agent/v2", + worker_id="worker-1", + source_event_id="unsafe-public-extension", + visibility="public", + payload=payload, + ) + + +def test_observed_at_is_strict_aware_and_canonical_utc() -> None: + event = agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={"at": datetime(2026, 7, 31, tzinfo=timezone.utc)}, + observed_at="2026-08-01T08:00:00+08:00", + ) + assert event.observed_at == "2026-08-01T00:00:00+00:00" + assert event.payload["at"] == "2026-07-31T00:00:00+00:00" + for invalid in ("not-a-time", "/home/private", "2026-07-31T00:00:00"): + with pytest.raises(ValueError, match="aware ISO-8601"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={}, + observed_at=invalid, + ) + with pytest.raises(ValueError, match="naive datetime"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={"at": datetime(2026, 7, 31)}, + ) def test_store_rejects_noncanonical_public_projection(tmp_path: Path) -> None: @@ -197,6 +361,160 @@ def test_store_rejects_noncanonical_public_projection(tmp_path: Path) -> None: store_sqlite.append_agent_event(tmp_path / "store.db", "host-1", tampered) +def test_store_fails_closed_when_public_projection_is_corrupted( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + store_sqlite.append_agent_event(db_path, "host-1", _message_event(sequence=1)) + with sqlite3.connect(db_path) as conn: + conn.execute( + "UPDATE agent_events SET public_payload_json = ?", + ('{"cwd":"/home/private","text":"safe"}',), + ) + with pytest.raises(store_sqlite.StoreSchemaError, match="invalid_agent_event_row"): + store_sqlite.list_public_agent_events(db_path, "host-1") + + +def test_host_scoping_and_concurrent_replay_are_isolated(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + event = _message_event(sequence=1) + store_sqlite.init_store(db_path) + + with ThreadPoolExecutor(max_workers=8) as executor: + results = list( + executor.map( + lambda _: store_sqlite.append_agent_event(db_path, "host-1", event), + range(24), + ) + ) + assert sum(result.inserted for result in results) == 1 + assert len({result.sequence for result in results}) == 1 + + other = store_sqlite.append_agent_event(db_path, "host-2", event) + assert other.inserted is True + assert len(store_sqlite.list_agent_events(db_path, "host-1")) == 1 + assert len(store_sqlite.list_agent_events(db_path, "host-2")) == 1 + assert store_sqlite.list_agent_events(db_path, "host-3") == () + with pytest.raises(ValueError, match="host_id must not be empty"): + store_sqlite.list_agent_events(db_path, " ") + + +def test_binding_guard_and_event_append_are_one_atomic_operation( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + binding = WorkerBinding( + host_id="host-1", + worker_id="worker-1", + worker_fingerprint="worker-fingerprint-1", + backend="herdr", + target_kind="pane", + target_value="pane-1", + turn_target_kind="pane", + turn_target_value="pane-1", + sendable=True, + observed_at="2026-07-31T00:00:00+00:00", + expires_at="9999-12-31T23:59:59+00:00", + private_fingerprint="private-binding-generation-1", + ) + store_sqlite.upsert_worker_bindings(db_path, [binding]) + event = _message_event(sequence=1) + + inserted = store_sqlite.append_agent_event_for_binding( + db_path, + "host-1", + event, + expected_binding=binding, + ) + replayed = store_sqlite.append_agent_event_for_binding( + db_path, + "host-1", + event, + expected_binding=binding, + ) + assert (inserted.status, inserted.inserted, inserted.sequence) == ( + "inserted", + True, + replayed.sequence, + ) + assert (replayed.status, replayed.inserted) == ("replayed", False) + + replacement = replace( + binding, + worker_id="worker-replacement", + worker_fingerprint="worker-fingerprint-2", + observed_at="2026-07-31T00:00:01+00:00", + ) + store_sqlite.upsert_worker_bindings(db_path, [replacement]) + rejected_event = _message_event(sequence=2, text="must not persist") + rejected = store_sqlite.append_agent_event_for_binding( + db_path, + "host-1", + rejected_event, + expected_binding=binding, + ) + assert (rejected.status, rejected.sequence, rejected.inserted) == ( + "binding_changed", + None, + False, + ) + assert [ + stored.event.payload["text"] + for stored in store_sqlite.list_agent_events(db_path, "host-1") + ] == ["hello"] + + +def test_database_constraints_and_indexes_cover_public_and_source_identity( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + store_sqlite.append_agent_event(db_path, "host-1", _message_event(sequence=1)) + with sqlite3.connect(db_path) as conn: + indexes = { + str(row[1]) for row in conn.execute("PRAGMA index_list(agent_events)") + } + assert { + "idx_agent_events_host_visibility_sequence", + "idx_agent_events_source_event_identity", + "idx_agent_events_source_sequence_identity", + } <= indexes + with pytest.raises(sqlite3.IntegrityError): + conn.execute( + "UPDATE agent_events SET kind = 'thought' WHERE host_id = 'host-1'" + ) + conn.rollback() + with pytest.raises(sqlite3.IntegrityError): + conn.execute( + "UPDATE agent_events SET public_payload_json = '[]' " + "WHERE host_id = 'host-1'" + ) + + +@pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) +def test_agent_event_schema_migrates_from_every_prior_version( + tmp_path: Path, + source_version: int, +) -> None: + db_path = tmp_path / f"v{source_version}.db" + with sqlite3.connect(db_path) as conn: + conn.execute("CREATE TABLE durable_sentinel (value TEXT NOT NULL)") + conn.execute("INSERT INTO durable_sentinel VALUES ('preserved')") + conn.commit() + store_sqlite._run_migrations(conn, target_version=source_version) + store_sqlite._run_migrations(conn) + assert conn.execute("PRAGMA user_version").fetchone() == ( + store_sqlite.STORE_SCHEMA_VERSION, + ) + assert conn.execute("SELECT value FROM durable_sentinel").fetchone() == ( + "preserved", + ) + assert conn.execute( + "SELECT COUNT(*) FROM sqlite_master " + "WHERE type = 'table' AND name = 'agent_events'" + ).fetchone() == (1,) + assert conn.execute("PRAGMA integrity_check").fetchone() == ("ok",) + + def test_v21_migration_is_idempotent_and_preserves_existing_store( tmp_path: Path, ) -> None: @@ -209,8 +527,69 @@ def test_v21_migration_is_idempotent_and_preserves_existing_store( store_sqlite.init_store(db_path) store_sqlite.init_store(db_path) with sqlite3.connect(db_path) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == (22,) + assert conn.execute("PRAGMA user_version").fetchone() == (23,) columns = { str(row[1]) for row in conn.execute("PRAGMA table_info(agent_events)") } assert {"sequence", "event_id", "private_payload_json"} <= columns + + +def test_v22_migration_rekeys_legacy_event_identity_without_losing_sequence( + tmp_path: Path, +) -> None: + db_path = tmp_path / "v22-event.db" + event = _message_event(sequence=7) + legacy_identity = { + "schema_version": 1, + "source": event.source, + "session_id": event.source_session_id, + "event_id": event.source_event_id, + "sequence": event.source_sequence, + "kind": event.kind, + } + legacy_event_id = hashlib.sha256( + store_sqlite._canonical_json(legacy_identity).encode("utf-8") + ).hexdigest() + with sqlite3.connect(db_path) as conn: + store_sqlite._run_migrations(conn, target_version=22) + conn.execute( + """ + INSERT INTO agent_events ( + sequence, host_id, event_id, kind, source, worker_id, + visibility, source_session_id, source_turn_id, + source_item_id, source_message_id, source_event_id, + source_sequence, observed_at, payload_fingerprint, + private_payload_json, public_payload_json + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + 19, + "host-1", + legacy_event_id, + event.kind, + event.source, + event.worker_id, + event.visibility, + event.source_session_id, + event.source_turn_id, + event.source_item_id, + event.source_message_id, + event.source_event_id, + event.source_sequence, + event.observed_at, + event.payload_fingerprint, + store_sqlite._canonical_json(event.payload), + store_sqlite._canonical_json(event.public_payload), + ), + ) + conn.commit() + store_sqlite._run_migrations(conn) + row = conn.execute( + "SELECT sequence, event_id FROM agent_events" + ).fetchone() + assert row == (19, event.event_id) + assert conn.execute("PRAGMA user_version").fetchone() == (23,) + + replay = store_sqlite.append_agent_event(db_path, "host-1", event) + assert replay.inserted is False + assert replay.sequence == 19 From d3634f12fa633ff02461334f7bf7a383ce9fd62f Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:22:00 +0800 Subject: [PATCH 10/83] test: expect hardened ACP journal schema --- tests/test_backend_pending.py | 2 +- tests/test_connector_outbox.py | 2 +- tests/test_delivery_retention_migration.py | 2 +- tests/test_delivery_retention_recovery.py | 2 +- tests/test_store.py | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/test_backend_pending.py b/tests/test_backend_pending.py index 91c69a8..2676486 100644 --- a/tests/test_backend_pending.py +++ b/tests/test_backend_pending.py @@ -1449,7 +1449,7 @@ def test_current_schema_creation_has_exact_binding_and_claim_state(tmp_path: Pat db = tmp_path / "current-schema.db" init_store(db) with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 22 + assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 23 columns = { str(row[1]) for row in conn.execute("PRAGMA table_info(backend_pending)").fetchall() diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index c5c3f80..6841af3 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -1754,7 +1754,7 @@ def test_v6_to_current_plan_migration_is_bounded_atomic_and_preserves_jobs( ).fetchall() } foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert version == store_sqlite.STORE_SCHEMA_VERSION == 22 + assert version == store_sqlite.STORE_SCHEMA_VERSION == 23 assert plan_row == (plan["plan_token"], 1, None, "active") assert job_count == 2 assert outbox_count == 3 diff --git a/tests/test_delivery_retention_migration.py b/tests/test_delivery_retention_migration.py index 33ce0ed..7fcb34c 100644 --- a/tests/test_delivery_retention_migration.py +++ b/tests/test_delivery_retention_migration.py @@ -875,7 +875,7 @@ def test_v10_to_v12_migration_retains_finals_without_reposting_or_leaking( finals = _seed_v10_finals(db_path) init_store(db_path) - assert store_sqlite.STORE_SCHEMA_VERSION == 22 + assert store_sqlite.STORE_SCHEMA_VERSION == 23 delivered_key = _final_key(*finals["delivered"]) hold_keys = {_final_key(*finals[label]) for label in ("hold-a", "hold-b")} diff --git a/tests/test_delivery_retention_recovery.py b/tests/test_delivery_retention_recovery.py index 3923d3c..6f6670e 100644 --- a/tests/test_delivery_retention_recovery.py +++ b/tests/test_delivery_retention_recovery.py @@ -973,7 +973,7 @@ def test_v12_migration_uses_effective_recovery_lineage_without_repost_or_hold( api = ConnectorOutboxAPI(db_path, HOST_ID) assert api.poll({"name": FINAL_NAME, "limit": 100})["items"] == [] with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 22 + assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 23 anchor = conn.execute( """ SELECT delivery_kind, status diff --git a/tests/test_store.py b/tests/test_store.py index 09c7796..a7a0a4d 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -10278,7 +10278,7 @@ def test_v13_migration_repairs_nonpositive_turn_sequences_and_blocks_recurrence( init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 22 + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 23 assert conn.execute( """ SELECT turn_id, list_sequence From d881ca334ed2f1fc1bdfc4f6f7e6d0a41be94d40 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:23:09 +0800 Subject: [PATCH 11/83] Integrate ACP runtime into daemon lifecycle --- src/tendwire/daemon.py | 156 +++++++++++++++ tests/test_daemon_acp.py | 403 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 559 insertions(+) create mode 100644 tests/test_daemon_acp.py diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index a92af95..f1b4f86 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -47,6 +47,15 @@ def _nonnegative_float(value: Any) -> float | None: return converted +def _public_failure_type(value: Any) -> str | None: + """Return a bounded exception type label, never arbitrary failure text.""" + if not isinstance(value, str) or not value or len(value) > 128: + return None + if any(not (character.isalnum() or character in "._") for character in value): + return None + return value + + _STORE_COUNT_FIELDS = ( "snapshots", "events", @@ -517,6 +526,7 @@ class DaemonHooks: submit_command: Callable[[Config, str], CommandEnvelope | Mapping[str, Any]] = _default_submit_command event_backend_factory: Callable[[Config, threading.Event], Any] | None = None turn_scheduler_factory: Callable[[Config], Any] = _default_turn_scheduler_factory + acp_runtime_factory: Callable[[Config, threading.Event], Any | None] | None = None class TendwireDaemon: @@ -540,6 +550,8 @@ def __init__( self._server: UnixSocketJSONServer | None = None self._event_backend: Any | None = None self._turn_scheduler: Any | None = None + self._acp_runtime: Any | None = None + self._acp_startup_failure_type: str | None = None self._stop_lock = threading.Lock() self._automatic_maintenance_status: dict[str, Any] | None = None @@ -590,6 +602,8 @@ def start(self) -> None: self._snapshot = self.hooks.observe_initial_snapshot(self.config) self._after_snapshot_saved() + self._start_acp_runtime() + scheduler = self.hooks.turn_scheduler_factory(self.config) self._turn_scheduler = scheduler @@ -651,6 +665,9 @@ def start(self) -> None: ) except Exception: pass + runtime = self._acp_runtime + self._acp_runtime = None + self._stop_acp_runtime(runtime) self._event_backend = None if backend is not None: try: @@ -692,9 +709,11 @@ def stop(self) -> None: server = self._server backend = self._event_backend scheduler = self._turn_scheduler + runtime = self._acp_runtime self._server = None self._event_backend = None self._turn_scheduler = None + self._acp_runtime = None if server is not None: try: @@ -702,6 +721,8 @@ def stop(self) -> None: except Exception: pass + self._stop_acp_runtime(runtime) + if backend is not None: flush = getattr(backend, "flush", None) if callable(flush): @@ -730,6 +751,135 @@ def stop(self) -> None: except Exception: pass + def _start_acp_runtime(self) -> None: + """Start an injected ACP runtime according to the configured policy.""" + policy = self.config.agent_event_source + self._acp_startup_failure_type = None + if policy == "legacy": + return + + factory = self.hooks.acp_runtime_factory + if factory is None: + if policy == "acp_required": + raise RuntimeError("ACP runtime is required but unavailable") + return + + runtime: Any | None = None + try: + runtime = factory(self.config, self.stop_event) + if runtime is None: + if policy == "acp_required": + raise RuntimeError("ACP runtime is required but unavailable") + return + self._acp_runtime = runtime + runtime.start() + health = self._acp_runtime_health() + if health["healthy"] is not True: + failure_type = health.get("failure_type") + self._acp_startup_failure_type = _public_failure_type(failure_type) + raise RuntimeError("ACP runtime did not become healthy") + except Exception as exc: + self._acp_startup_failure_type = ( + self._acp_startup_failure_type or type(exc).__name__ + ) + if runtime is not None: + self._stop_acp_runtime(runtime) + self._acp_runtime = None + if policy == "acp_required": + raise RuntimeError( + "ACP runtime is required but failed to start " + f"({self._acp_startup_failure_type})" + ) from None + + def _stop_acp_runtime(self, runtime: Any | None) -> None: + """Best-effort bounded shutdown for an injected ACP runtime.""" + if runtime is None: + return + timeout = self.config.acp_shutdown_timeout_seconds + stop = getattr(runtime, "stop", None) + if callable(stop): + try: + stop(timeout=timeout) + except Exception: + pass + join = getattr(runtime, "join", None) + if callable(join): + try: + join(timeout=timeout) + except Exception: + pass + + def _acp_runtime_health(self) -> dict[str, Any]: + """Return a fixed, public-safe ACP lifecycle aggregate.""" + counters = { + "updates_ingested": 0, + "permissions_ingested": 0, + "permissions_selected": 0, + "permissions_cancelled": 0, + "invalid_permission_selections": 0, + "prompts_started": 0, + "prompts_completed": 0, + "prompts_failed": 0, + "cancellation_requests": 0, + } + policy = self.config.agent_event_source + if policy == "legacy": + return { + "policy": policy, + "status": "disabled", + "healthy": False, + "state": "disabled", + "failure_type": None, + "counters": counters, + } + + runtime = self._acp_runtime + if runtime is None: + return { + "policy": policy, + "status": "unavailable", + "healthy": False, + "state": "unavailable", + "failure_type": self._acp_startup_failure_type, + "counters": counters, + } + + status_method = getattr(runtime, "status", None) + try: + raw = status_method() if callable(status_method) else None + except Exception as exc: + return { + "policy": policy, + "status": "degraded", + "healthy": False, + "state": "failed", + "failure_type": type(exc).__name__, + "counters": counters, + } + + def field(name: str) -> Any: + if isinstance(raw, Mapping): + return raw.get(name) + return getattr(raw, name, None) + + state_value = field("state") + state = getattr(state_value, "value", state_value) + if state not in {"new", "starting", "running", "stopping", "stopped", "failed"}: + state = "unknown" + healthy = field("healthy") is True and state == "running" + for key in counters: + counters[key] = _nonnegative_int(field(key)) + failure_type_value = field("failure_type") + failure_type = _public_failure_type(failure_type_value) + return { + "policy": policy, + "status": "healthy" if healthy else "degraded", + "healthy": healthy, + "state": state, + "failure_type": failure_type, + "counters": counters, + } + def _after_snapshot_saved(self) -> None: if self.config.db_path is None: return @@ -999,6 +1149,7 @@ def get_health(self) -> dict[str, Any]: or stored_last_snapshot_at or snapshot.updated_at ) + acp_health = self._acp_runtime_health() payload = { "schema_version": 1, "status": ( @@ -1006,6 +1157,10 @@ def get_health(self) -> dict[str, Any]: if store_ok and not maintenance_degraded and pending_ingestion["status"] == "healthy" + and ( + self.config.agent_event_source != "acp_required" + or acp_health["healthy"] is True + ) else "degraded" ), "host_id": self.config.host_id, @@ -1053,6 +1208,7 @@ def get_health(self) -> dict[str, Any]: self.config, self._turn_scheduler, ), + "acp": acp_health, "pending_ingestion": pending_ingestion, "limits": { "event_debounce_seconds": self.config.event_debounce_seconds, diff --git a/tests/test_daemon_acp.py b/tests/test_daemon_acp.py new file mode 100644 index 0000000..f615c0b --- /dev/null +++ b/tests/test_daemon_acp.py @@ -0,0 +1,403 @@ +"""ACP lifecycle policy tests for the Tendwire daemon.""" + +from __future__ import annotations + +import json +import os +import stat +import threading +from pathlib import Path +from typing import Any + +import pytest + +from tendwire.config import Config +from tendwire.core.models import BackendHealth, Snapshot +from tendwire.daemon import DaemonHooks, TendwireDaemon +from tendwire.store.sqlite import init_store, save_snapshot + + +def _snapshot() -> Snapshot: + return Snapshot( + host_id="daemon-host", + updated_at="2026-01-01T00:00:00+00:00", + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="empty_healthy", + observed_at="2026-01-01T00:00:00+00:00", + ) + ], + ) + + +class _Scheduler: + def __init__(self, calls: list[str]) -> None: + self.calls = calls + + def start(self) -> None: + self.calls.append("scheduler_start") + + def request_refresh(self) -> None: + self.calls.append("scheduler_request") + + def stop(self, *, flush_timeout_seconds: float | None = None) -> None: + self.calls.append(f"scheduler_stop:{flush_timeout_seconds}") + + def operational_status(self) -> dict[str, Any]: + return {"status": "healthy"} + + +class _Runtime: + def __init__( + self, + calls: list[str], + *, + healthy: bool = True, + start_failure: BaseException | None = None, + ) -> None: + self.calls = calls + self.healthy = healthy + self.start_failure = start_failure + + def start(self) -> None: + self.calls.append("acp_start") + if self.start_failure is not None: + raise self.start_failure + + def status(self) -> dict[str, Any]: + return { + "state": "running" if self.healthy else "failed", + "healthy": self.healthy, + "updates_ingested": 7, + "permissions_ingested": 2, + "permissions_selected": 1, + "permissions_cancelled": 1, + "invalid_permission_selections": 0, + "prompts_started": 3, + "prompts_completed": 2, + "prompts_failed": 1, + "cancellation_requests": 1, + "failure_type": None if self.healthy else "AcpTransportError", + # Deliberately private transport material must never be projected. + "argv": ["sentinel-private-command"], + "session_id": "sentinel-private-session", + "binding_id": "sentinel-private-binding", + } + + def stop(self, *, timeout: float) -> None: + self.calls.append(f"acp_stop:{timeout}") + + def join(self, *, timeout: float) -> bool: + self.calls.append(f"acp_join:{timeout}") + return True + + +def _hooks( + tmp_path: Path, + calls: list[str], + *, + acp_runtime_factory: Any = None, + scheduler_factory: Any = None, +) -> DaemonHooks: + db_path = tmp_path / "daemon.db" + + def initialize(path: Path) -> None: + calls.append("init_store") + init_store(path) + + def observe(_config: Config) -> Snapshot: + calls.append("observe") + snapshot = _snapshot() + save_snapshot(db_path, snapshot) + return snapshot + + def make_scheduler(_config: Config) -> _Scheduler: + calls.append("scheduler_factory") + return _Scheduler(calls) + + return DaemonHooks( + init_store=initialize, + observe_initial_snapshot=observe, + turn_scheduler_factory=scheduler_factory or make_scheduler, + acp_runtime_factory=acp_runtime_factory, + ) + + +def _config(tmp_path: Path, policy: str) -> Config: + return Config( + host_id="daemon-host", + data_dir=tmp_path, + db_path=tmp_path / "daemon.db", + socket_path=tmp_path / "daemon.sock", + agent_event_source=policy, + acp_shutdown_timeout_seconds=1.25, + ) + + +def test_legacy_policy_never_calls_acp_factory(tmp_path: Path) -> None: + calls: list[str] = [] + + def forbidden_factory(_config: Config, _stop_event: threading.Event) -> Any: + raise AssertionError("legacy mode must never discover or start ACP") + + daemon = TendwireDaemon( + _config(tmp_path, "legacy"), + hooks=_hooks(tmp_path, calls, acp_runtime_factory=forbidden_factory), + ) + daemon.start() + try: + assert daemon.get_health()["acp"] == { + "policy": "legacy", + "status": "disabled", + "healthy": False, + "state": "disabled", + "failure_type": None, + "counters": { + "updates_ingested": 0, + "permissions_ingested": 0, + "permissions_selected": 0, + "permissions_cancelled": 0, + "invalid_permission_selections": 0, + "prompts_started": 0, + "prompts_completed": 0, + "prompts_failed": 0, + "cancellation_requests": 0, + }, + } + finally: + daemon.stop() + + assert calls[-1] == "scheduler_stop:6.0" + + +@pytest.mark.parametrize("policy", ["acp_shadow", "acp_preferred"]) +def test_optional_acp_policy_tolerates_unavailable_runtime( + tmp_path: Path, + policy: str, +) -> None: + calls: list[str] = [] + + def unavailable(_config: Config, _stop_event: threading.Event) -> None: + calls.append("acp_factory") + return None + + daemon = TendwireDaemon( + _config(tmp_path, policy), + hooks=_hooks(tmp_path, calls, acp_runtime_factory=unavailable), + ) + daemon.start() + try: + assert calls[-2:] == ["scheduler_start", "scheduler_request"] + assert daemon.get_health()["acp"]["status"] == "unavailable" + finally: + daemon.stop() + + +def test_required_acp_without_factory_fails_before_socket_or_scheduler( + tmp_path: Path, +) -> None: + calls: list[str] = [] + socket_path = tmp_path / "daemon.sock" + daemon = TendwireDaemon( + _config(tmp_path, "acp_required"), + hooks=_hooks(tmp_path, calls), + ) + + with pytest.raises(RuntimeError, match="ACP runtime is required"): + daemon.start() + + assert calls == ["init_store", "observe"] + assert not os.path.lexists(socket_path) + assert daemon.server is None + + +def test_required_acp_starts_before_socket_and_exposes_only_redacted_health( + tmp_path: Path, +) -> None: + calls: list[str] = [] + socket_path = tmp_path / "daemon.sock" + runtime = _Runtime(calls) + + def runtime_factory(config: Config, stop_event: threading.Event) -> _Runtime: + assert config.agent_event_source == "acp_required" + assert stop_event.is_set() is False + assert not os.path.lexists(socket_path) + calls.append("acp_factory") + return runtime + + daemon = TendwireDaemon( + _config(tmp_path, "acp_required"), + hooks=_hooks(tmp_path, calls, acp_runtime_factory=runtime_factory), + ) + daemon.start() + try: + assert calls == [ + "init_store", + "observe", + "acp_factory", + "acp_start", + "scheduler_factory", + "scheduler_start", + "scheduler_request", + ] + assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) + health = daemon.get_health() + assert health["status"] == "ok" + acp = health["acp"] + assert acp == { + "policy": "acp_required", + "status": "healthy", + "healthy": True, + "state": "running", + "failure_type": None, + "counters": { + "updates_ingested": 7, + "permissions_ingested": 2, + "permissions_selected": 1, + "permissions_cancelled": 1, + "invalid_permission_selections": 0, + "prompts_started": 3, + "prompts_completed": 2, + "prompts_failed": 1, + "cancellation_requests": 1, + }, + } + encoded = json.dumps(acp) + assert "sentinel-private" not in encoded + assert "argv" not in encoded + assert "session" not in encoded + assert "binding" not in encoded + runtime.healthy = False + degraded = daemon.get_health() + assert degraded["status"] == "degraded" + assert degraded["acp"]["status"] == "degraded" + finally: + daemon.stop() + + assert calls[-3:] == [ + "acp_stop:1.25", + "acp_join:1.25", + "scheduler_stop:6.0", + ] + + +@pytest.mark.parametrize("policy", ["acp_shadow", "acp_preferred"]) +def test_optional_unhealthy_acp_is_stopped_and_legacy_scheduler_continues( + tmp_path: Path, + policy: str, +) -> None: + calls: list[str] = [] + runtime = _Runtime(calls, healthy=False) + daemon = TendwireDaemon( + _config(tmp_path, policy), + hooks=_hooks( + tmp_path, + calls, + acp_runtime_factory=lambda _config, _stop_event: runtime, + ), + ) + + daemon.start() + try: + assert calls == [ + "init_store", + "observe", + "acp_start", + "acp_stop:1.25", + "acp_join:1.25", + "scheduler_factory", + "scheduler_start", + "scheduler_request", + ] + acp = daemon.get_health()["acp"] + assert acp["status"] == "unavailable" + assert acp["failure_type"] == "AcpTransportError" + finally: + daemon.stop() + + +def test_required_unhealthy_acp_stops_and_fails_closed(tmp_path: Path) -> None: + calls: list[str] = [] + runtime = _Runtime(calls, healthy=False) + daemon = TendwireDaemon( + _config(tmp_path, "acp_required"), + hooks=_hooks( + tmp_path, + calls, + acp_runtime_factory=lambda _config, _stop_event: runtime, + ), + ) + + with pytest.raises(RuntimeError, match=r"failed to start \(AcpTransportError\)"): + daemon.start() + + assert calls == [ + "init_store", + "observe", + "acp_start", + "acp_stop:1.25", + "acp_join:1.25", + ] + assert not os.path.lexists(tmp_path / "daemon.sock") + + +def test_optional_acp_start_failure_is_cleaned_up_before_legacy_fallback( + tmp_path: Path, +) -> None: + calls: list[str] = [] + runtime = _Runtime( + calls, + start_failure=RuntimeError("sentinel-private-command --session secret"), + ) + daemon = TendwireDaemon( + _config(tmp_path, "acp_preferred"), + hooks=_hooks( + tmp_path, + calls, + acp_runtime_factory=lambda _config, _stop_event: runtime, + ), + ) + + daemon.start() + try: + assert calls[2:5] == ["acp_start", "acp_stop:1.25", "acp_join:1.25"] + acp = daemon.get_health()["acp"] + assert acp["status"] == "unavailable" + assert acp["failure_type"] == "RuntimeError" + assert "sentinel-private" not in json.dumps(acp) + assert calls[-2:] == ["scheduler_start", "scheduler_request"] + finally: + daemon.stop() + + +def test_scheduler_start_failure_also_stops_and_joins_acp(tmp_path: Path) -> None: + calls: list[str] = [] + runtime = _Runtime(calls) + + class FailingScheduler(_Scheduler): + def start(self) -> None: + self.calls.append("scheduler_start") + raise RuntimeError("sentinel scheduler failure") + + daemon = TendwireDaemon( + _config(tmp_path, "acp_required"), + hooks=_hooks( + tmp_path, + calls, + acp_runtime_factory=lambda _config, _stop_event: runtime, + scheduler_factory=lambda _config: FailingScheduler(calls), + ), + ) + + with pytest.raises(RuntimeError, match="sentinel scheduler failure"): + daemon.start() + + assert calls[-3:] == [ + "scheduler_stop:6.0", + "acp_stop:1.25", + "acp_join:1.25", + ] + assert daemon._acp_runtime is None + assert not os.path.lexists(tmp_path / "daemon.sock") From 314794198a43760b381c45075541eaf3d14c713d Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:25:02 +0800 Subject: [PATCH 12/83] Make ACP event ingestion binding-atomic --- src/tendwire/backends/acp_ingestion.py | 68 ++-------- tests/test_acp_ingestion.py | 173 +++++++++++++++++++------ 2 files changed, 146 insertions(+), 95 deletions(-) diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py index 3f9ce20..11761ff 100644 --- a/src/tendwire/backends/acp_ingestion.py +++ b/src/tendwire/backends/acp_ingestion.py @@ -17,18 +17,16 @@ from ..core.agent_events import AgentEvent, agent_event from ..core.models import WorkerBinding, stable_fingerprint from ..store.sqlite import ( - AppendAgentEventResult, + AppendBoundAgentEventResult, TurnRefreshApplyResult, - append_agent_event, + append_agent_event_for_binding, apply_turn_refresh, - list_worker_bindings, ) from .acp_projection import AcpEventProjector -AppendEvent = Callable[[Path | str, str, AgentEvent], AppendAgentEventResult] +AppendEvent = Callable[..., AppendBoundAgentEventResult] ApplyTurn = Callable[..., TurnRefreshApplyResult] -BindingIsCurrent = Callable[[Path | str, str, WorkerBinding], bool] @dataclass(frozen=True) @@ -36,7 +34,7 @@ class AcpIngestionResult: """Outcome of accepting, ignoring, or projecting one ACP event.""" kind: str | None - event: AppendAgentEventResult | None = None + event: AppendBoundAgentEventResult | None = None turn: TurnRefreshApplyResult | None = None ignored_reason: str | None = None @@ -58,9 +56,8 @@ def __init__( stream_generation: str, binding: WorkerBinding, projector: AcpEventProjector | None = None, - append_event: AppendEvent = append_agent_event, + append_event: AppendEvent = append_agent_event_for_binding, apply_turn: ApplyTurn = apply_turn_refresh, - binding_is_current: BindingIsCurrent | None = None, ) -> None: if config.db_path is None: raise ValueError("ACP ingestion requires a sqlite db path") @@ -86,7 +83,6 @@ def __init__( self.projector = projector or AcpEventProjector() self._append_event = append_event self._apply_turn = apply_turn - self._binding_is_current = binding_is_current or _binding_is_current self._turn_ordinal = 0 self._source_turn_id: str | None = None self._turn_complete = False @@ -152,8 +148,6 @@ def ingest_update( ) if thought_rejection is not None: return AcpIngestionResult("thought", ignored_reason=thought_rejection) - if not self._current_binding_is_valid(): - return AcpIngestionResult(None, ignored_reason="stale_binding") canonical = self.projector.normalize_session_update( notification, source_event_id=source_event_id, @@ -181,8 +175,6 @@ def ingest_permission_request( return AcpIngestionResult(None, ignored_reason=mismatch) if self._turn_complete: return AcpIngestionResult(None, ignored_reason="turn_already_complete") - if not self._current_binding_is_valid(): - return AcpIngestionResult(None, ignored_reason="stale_binding") canonical = self.projector.normalize_permission_request( request, source_event_id=source_event_id, @@ -199,8 +191,6 @@ def mark_prompt_complete(self) -> AcpIngestionResult: return AcpIngestionResult(None, ignored_reason="no_active_turn") if self._turn_complete: return AcpIngestionResult(None, ignored_reason="turn_already_complete") - if not self._current_binding_is_valid(): - return AcpIngestionResult(None, ignored_reason="stale_binding") content = self.projector.mark_turn_complete(self.session_id) content["source_turn_id"] = self._source_turn_id if self.config.agent_event_source == "acp_shadow": @@ -259,13 +249,14 @@ def _accept(self, canonical: Mapping[str, Any]) -> AcpIngestionResult: Path(self.config.db_path), self.config.host_id, event, + expected_binding=self.binding, ) turn: TurnRefreshApplyResult | None = None if ( kind in {"user_message", "agent_message"} and self.config.agent_event_source != "acp_shadow" - and appended.inserted + and appended.status == "inserted" ): content = self.projector.project_turn_content(self.session_id) if self._source_turn_id is not None: @@ -277,27 +268,14 @@ def _accept(self, canonical: Mapping[str, Any]) -> AcpIngestionResult: turn=turn, ignored_reason=( "stale_binding" - if turn is not None and turn.stale_binding + if appended.status == "binding_changed" + or (turn is not None and turn.stale_binding) else "duplicate_event" - if not appended.inserted + if appended.status == "replayed" else None ), ) - def _current_binding_is_valid(self) -> bool: - try: - return bool( - self._binding_is_current( - Path(self.config.db_path), - self.config.host_id, - self.binding, - ) - ) - except Exception: - # Binding lookup is an authority check. Any lookup failure must - # fail closed rather than accepting an unauthenticated event. - return False - def _project_turn(self, content: Mapping[str, Any]) -> TurnRefreshApplyResult: return self._apply_turn( Path(self.config.db_path), @@ -408,30 +386,4 @@ def _thought_rejection_reason( return None -def _binding_is_current( - db_path: Path | str, - host_id: str, - expected: WorkerBinding, -) -> bool: - """Check the durable private authority immediately before accepting data.""" - - for current in list_worker_bindings( - Path(db_path), - str(host_id), - backend=expected.backend, - ): - if ( - current.worker_id == expected.worker_id - and current.worker_fingerprint == expected.worker_fingerprint - and current.backend == expected.backend - and current.target_kind == expected.target_kind - and current.target_value == expected.target_value - and current.turn_target_kind == expected.turn_target_kind - and current.turn_target_value == expected.turn_target_value - and current.private_fingerprint == expected.private_fingerprint - ): - return True - return False - - __all__ = ["AcpIngestionResult", "AcpSessionIngestor"] diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py index 33b7ca4..f6858f2 100644 --- a/tests/test_acp_ingestion.py +++ b/tests/test_acp_ingestion.py @@ -2,15 +2,20 @@ from __future__ import annotations +from dataclasses import replace from pathlib import Path import pytest from tendwire.backends.acp_ingestion import AcpSessionIngestor from tendwire.config import Config -from tendwire.core.agent_events import AgentEvent, AppendAgentEventResult +from tendwire.core.agent_events import AgentEvent, AppendBoundAgentEventResult from tendwire.core.models import WorkerBinding -from tendwire.store.sqlite import TurnRefreshApplyResult, upsert_worker_bindings +from tendwire.store.sqlite import ( + TurnRefreshApplyResult, + list_agent_events, + upsert_worker_bindings, +) def _binding() -> WorkerBinding: @@ -37,15 +42,27 @@ def _update(kind: str, **fields: object) -> dict[str, object]: } +def _appended(sequence: int, event: AgentEvent) -> AppendBoundAgentEventResult: + return AppendBoundAgentEventResult("inserted", event.event_id, sequence) + + def test_messages_are_journaled_privately_and_projected_without_thoughts( tmp_path: Path, ) -> None: events: list[AgentEvent] = [] turns: list[dict[str, object]] = [] - def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + *, + expected_binding: WorkerBinding, + ) -> AppendBoundAgentEventResult: + assert expected_binding.worker_id == "worker-a" + assert expected_binding.private_fingerprint == "binding-fingerprint" events.append(event) - return AppendAgentEventResult(len(events), event.event_id, True) + return _appended(len(events), event) def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): turns.append(dict(content)) @@ -58,7 +75,6 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): binding=_binding(), append_event=append, apply_turn=apply, - binding_is_current=lambda *_args: True, ) turn_id = ingestor.start_turn(producer_turn_id="private-turn") ingestor.ingest_update( @@ -101,9 +117,14 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): def test_shadow_mode_journals_without_turn_projection(tmp_path: Path) -> None: events: list[AgentEvent] = [] - def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: events.append(event) - return AppendAgentEventResult(1, event.event_id, True) + return _appended(1, event) def unexpected_turn(*_args, **_kwargs): raise AssertionError("shadow mode must not project turns") @@ -119,7 +140,6 @@ def unexpected_turn(*_args, **_kwargs): binding=_binding(), append_event=append, apply_turn=unexpected_turn, - binding_is_current=lambda *_args: True, ) result = ingestor.ingest_update( _update( @@ -161,10 +181,15 @@ def unexpected_append(*_args, **_kwargs): def test_synthetic_event_identity_is_scoped_to_stream_generation(tmp_path: Path) -> None: seen: list[str] = [] - def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: assert event.source_event_id is not None seen.append(event.source_event_id) - return AppendAgentEventResult(1, event.event_id, True) + return _appended(1, event) for generation in ("generation-a", "generation-b"): ingestor = AcpSessionIngestor( @@ -174,7 +199,6 @@ def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEvent binding=_binding(), append_event=append, apply_turn=lambda *_args, **_kwargs: TurnRefreshApplyResult(0, False), - binding_is_current=lambda *_args: True, ) ingestor.ingest_update( _update( @@ -214,7 +238,6 @@ def unexpected(*_args, **_kwargs): binding=_binding(), append_event=unexpected, apply_turn=unexpected, - binding_is_current=unexpected, ) notification = _update( "agent_message_chunk", @@ -234,21 +257,29 @@ def unexpected(*_args, **_kwargs): def test_required_mode_fails_closed_when_durable_binding_is_stale( tmp_path: Path, ) -> None: - def unexpected(*_args, **_kwargs): - raise AssertionError("stale ACP events must not be journaled or projected") + db_path = tmp_path / "events.db" + binding = _binding() + upsert_worker_bindings(db_path, [binding]) + replacement = replace( + binding, + worker_id="replacement-worker", + worker_fingerprint="replacement-fingerprint", + ) + upsert_worker_bindings(db_path, [replacement]) + + def unexpected_projection(*_args, **_kwargs): + raise AssertionError("stale ACP events must not be projected") ingestor = AcpSessionIngestor( Config( host_id="host-a", - db_path=tmp_path / "events.db", + db_path=db_path, agent_event_source="acp_required", ), session_id="session-a", stream_generation="generation-a", - binding=_binding(), - append_event=unexpected, - apply_turn=unexpected, - binding_is_current=lambda *_args: False, + binding=binding, + apply_turn=unexpected_projection, ) result = ingestor.ingest_update( @@ -259,9 +290,11 @@ def unexpected(*_args, **_kwargs): ) assert result.ignored_reason == "stale_binding" - assert result.event is None + assert result.event is not None + assert result.event.status == "binding_changed" + assert result.event.sequence is None assert result.turn is None - assert ingestor.source_turn_id is None + assert list_agent_events(db_path, "host-a") == () def test_default_authority_check_accepts_the_current_durable_binding( @@ -289,9 +322,14 @@ def test_shadow_completion_never_projects_and_finality_is_idempotent( ) -> None: events: list[AgentEvent] = [] - def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: events.append(event) - return AppendAgentEventResult(len(events), event.event_id, True) + return _appended(len(events), event) def unexpected_turn(*_args, **_kwargs): raise AssertionError("shadow mode must never project, including completion") @@ -307,7 +345,6 @@ def unexpected_turn(*_args, **_kwargs): binding=_binding(), append_event=append, apply_turn=unexpected_turn, - binding_is_current=lambda *_args: True, ) ingestor.start_turn(producer_turn_id="turn-1") ingestor.ingest_update( @@ -336,9 +373,14 @@ def test_required_mode_projects_messages_and_final_exactly_once(tmp_path: Path) events: list[AgentEvent] = [] turns: list[dict[str, object]] = [] - def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: events.append(event) - return AppendAgentEventResult(len(events), event.event_id, True) + return _appended(len(events), event) def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): turns.append(dict(content)) @@ -355,7 +397,6 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): binding=_binding(), append_event=append, apply_turn=apply, - binding_is_current=lambda *_args: True, ) ingestor.start_turn(producer_turn_id="turn-1") streamed = ingestor.ingest_update( @@ -376,8 +417,13 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): def test_duplicate_durable_event_is_not_reprojected(tmp_path: Path) -> None: projected = False - def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: - return AppendAgentEventResult(9, event.event_id, False) + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: + return AppendBoundAgentEventResult("replayed", event.event_id, 9) def apply(*_args, **_kwargs): nonlocal projected @@ -391,7 +437,6 @@ def apply(*_args, **_kwargs): binding=_binding(), append_event=append, apply_turn=apply, - binding_is_current=lambda *_args: True, ) result = ingestor.ingest_update( _update( @@ -406,6 +451,53 @@ def apply(*_args, **_kwargs): assert not projected +def test_atomic_durable_replay_is_reported_without_second_projection( + tmp_path: Path, +) -> None: + db_path = tmp_path / "events.db" + binding = _binding() + upsert_worker_bindings(db_path, [binding]) + turns: list[dict[str, object]] = [] + + def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): + turns.append(dict(content)) + return TurnRefreshApplyResult(len(turns), False) + + def ingestor() -> AcpSessionIngestor: + return AcpSessionIngestor( + Config(host_id="host-a", db_path=db_path), + session_id="session-a", + stream_generation="generation-a", + binding=binding, + apply_turn=apply, + ) + + notification = _update( + "agent_message_chunk", + messageId="assistant-1", + content={"type": "text", "text": "durable once"}, + ) + inserted = ingestor().ingest_update( + notification, + source_event_id="event-1", + ) + replayed = ingestor().ingest_update( + notification, + source_event_id="event-1", + replay=True, + ) + + assert inserted.event is not None + assert inserted.event.status == "inserted" + assert inserted.turn is not None + assert replayed.event is not None + assert replayed.event.status == "replayed" + assert replayed.ignored_reason == "duplicate_event" + assert replayed.turn is None + assert len(turns) == 1 + assert len(list_agent_events(db_path, "host-a")) == 1 + + def test_producer_turn_identity_survives_transport_recreation(tmp_path: Path) -> None: identities: list[str] = [] for generation in ("generation-a", "generation-b"): @@ -414,7 +506,6 @@ def test_producer_turn_identity_survives_transport_recreation(tmp_path: Path) -> session_id="session-a", stream_generation=generation, binding=_binding(), - binding_is_current=lambda *_args: True, ) identities.append(ingestor.start_turn(producer_turn_id="producer-turn-7")) @@ -426,9 +517,14 @@ def test_private_summary_policy_retains_display_chunks_but_rejects_marked_raw_th ) -> None: events: list[AgentEvent] = [] - def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: events.append(event) - return AppendAgentEventResult(len(events), event.event_id, True) + return _appended(len(events), event) ingestor = AcpSessionIngestor( Config(host_id="host-a", db_path=tmp_path / "events.db"), @@ -436,7 +532,6 @@ def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEvent stream_generation="generation-a", binding=_binding(), append_event=append, - binding_is_current=lambda *_args: True, ) unclassified = ingestor.ingest_update( _update( @@ -472,9 +567,14 @@ def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEvent def test_private_all_policy_retains_marked_raw_thought_privately(tmp_path: Path) -> None: events: list[AgentEvent] = [] - def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEventResult: + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: events.append(event) - return AppendAgentEventResult(1, event.event_id, True) + return _appended(1, event) ingestor = AcpSessionIngestor( Config( @@ -486,7 +586,6 @@ def append(_path: Path | str, _host: str, event: AgentEvent) -> AppendAgentEvent stream_generation="generation-a", binding=_binding(), append_event=append, - binding_is_current=lambda *_args: True, ) result = ingestor.ingest_update( _update( From 8b39b8927abf591d49a12dcb1ecea78b871c424f Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:29:36 +0800 Subject: [PATCH 13/83] Harden ACP transport lifecycle --- src/tendwire/backends/acp_client.py | 596 +++++++++++++++++++------- src/tendwire/backends/acp_protocol.py | 33 +- tests/fixtures/acp_fake_agent.py | 109 ++++- tests/test_acp_client.py | 165 ++++++- tests/test_acp_protocol.py | 63 +++ 5 files changed, 789 insertions(+), 177 deletions(-) diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index ddf6dc1..69e596e 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -11,8 +11,11 @@ import math import os import queue +import select +import signal import subprocess import threading +import time from collections import deque from collections.abc import Mapping, Sequence from dataclasses import dataclass @@ -117,6 +120,13 @@ class ProcessExit: stderr_tail: str +@dataclass(slots=True) +class _PendingRequest: + waiter: queue.Queue[JsonRpcResponse | BaseException] + method: str + session_id: str | None + + _T = TypeVar("_T") _END = object() @@ -138,11 +148,13 @@ def __init__( stderr_limit_bytes: int = _DEFAULT_STDERR_LIMIT, ) -> None: command = tuple(os.fspath(item) for item in argv) - if not command or any(not item for item in command): + if not command or any( + not isinstance(item, str) or not item or "\x00" in item for item in command + ): raise ValueError("argv must contain at least one non-empty argument") self.argv = command - self.cwd = os.fspath(cwd) if cwd is not None else None - self.env = dict(env) if env is not None else None + self.cwd = _absolute_path(cwd, "process cwd") if cwd is not None else None + self.env = _validated_env(env) self.request_timeout = _positive_timeout(request_timeout, "request_timeout") self.prompt_timeout = _positive_timeout(prompt_timeout, "prompt_timeout") self.close_timeout = _positive_timeout(close_timeout, "close_timeout") @@ -160,11 +172,14 @@ def __init__( self._process: subprocess.Popen[bytes] | None = None self._state_lock = threading.RLock() self._write_lock = threading.Lock() + self._initialize_lock = threading.Lock() self._request_id_lock = threading.Lock() self._next_id = 1 - self._pending: dict[RequestId, queue.Queue[JsonRpcResponse | BaseException]] = {} + self._pending: dict[RequestId, _PendingRequest] = {} self._pending_lock = threading.Lock() self._pending_permissions: dict[RequestId, PermissionRequest] = {} + self._cancelled_sessions: set[str] = set() + self._active_prompts: dict[str, int] = {} self._permission_lock = threading.Lock() self._updates: queue.Queue[SessionUpdate | object] = queue.Queue(max_pending_events) self._permissions: queue.Queue[PermissionRequest | object] = queue.Queue( @@ -179,6 +194,7 @@ def __init__( self._reader_thread: threading.Thread | None = None self._stderr_thread: threading.Thread | None = None self._stop = threading.Event() + self._closed = threading.Event() self._stderr_chunks: deque[bytes] = deque() self._stderr_size = 0 self._stderr_lock = threading.Lock() @@ -245,6 +261,7 @@ def start(self) -> "AcpClient": env=self.env, bufsize=0, close_fds=True, + start_new_session=os.name == "posix", ) except OSError as exc: self._state = ClientState.FAILED @@ -253,6 +270,17 @@ def start(self) -> "AcpClient": ) raise self._failure from exc self._process = process + assert process.stdin is not None + try: + os.set_blocking(process.stdin.fileno(), False) + except OSError as exc: + process.kill() + process.wait() + self._state = ClientState.FAILED + self._failure = AcpTransportError( + "could not configure ACP agent stdin" + ) + raise self._failure from exc self._state = ClientState.RUNNING self._reader_thread = threading.Thread( target=self._reader_main, @@ -279,74 +307,78 @@ def initialize( ) -> InitializeResult: """Perform ACP v1 capability negotiation. - ACP v1 does not define a post-response ``initialized`` notification; - :meth:`initialized` is available only for adapters that explicitly - require that compatibility extension. + ACP v1 does not define a post-response ``initialized`` notification. """ self.start() if not client_name or not client_version: raise ValueError("client_name and client_version must be non-empty") - with self._state_lock: - if self._state is ClientState.INITIALIZED: - assert self._initialize_result is not None - return self._initialize_result - if self._state is not ClientState.RUNNING: - self._raise_unusable() - client_info: dict[str, Any] = { - "name": client_name, - "version": client_version, - } - if client_title: - client_info["title"] = client_title - result = self.request( - "initialize", - { - "protocolVersion": ACP_PROTOCOL_VERSION, - "clientCapabilities": dict(client_capabilities or {}), - "clientInfo": client_info, - }, - timeout=timeout, - require_initialized=False, - ) - raw = _require_mapping(result, "initialize result") - version = raw.get("protocolVersion") - if version != ACP_PROTOCOL_VERSION: - raise AcpProtocolVersionError( - f"agent selected unsupported ACP protocol version {version!r}" + with self._initialize_lock: + with self._state_lock: + if self._state is ClientState.INITIALIZED: + assert self._initialize_result is not None + return self._initialize_result + if self._state is not ClientState.RUNNING: + self._raise_unusable() + client_info: dict[str, Any] = { + "name": client_name, + "version": client_version, + } + if client_title: + client_info["title"] = client_title + result = self.request( + "initialize", + { + "protocolVersion": ACP_PROTOCOL_VERSION, + "clientCapabilities": dict(client_capabilities or {}), + "clientInfo": client_info, + }, + timeout=timeout, + require_initialized=False, ) - capabilities_value = raw.get("agentCapabilities", {}) - if not isinstance(capabilities_value, Mapping): - capabilities_value = {} - agent_info_value = raw.get("agentInfo") - agent_info = ( - MappingProxyType(dict(agent_info_value)) - if isinstance(agent_info_value, Mapping) - else None - ) - auth_methods_value = raw.get("authMethods", []) - auth_methods = tuple( - MappingProxyType(dict(item)) - for item in auth_methods_value - if isinstance(item, Mapping) - ) if isinstance(auth_methods_value, list) else () - parsed = InitializeResult( - protocol_version=version, - capabilities=AgentCapabilities.from_mapping(capabilities_value), - agent_info=agent_info, - auth_methods=auth_methods, - raw=MappingProxyType(dict(raw)), - ) - with self._state_lock: - if self._state is not ClientState.RUNNING: - self._raise_unusable() - self._initialize_result = parsed - self._state = ClientState.INITIALIZED - return parsed - - def initialized(self) -> None: - """Send the non-standard ``initialized`` compatibility notification.""" - self._require_initialized() - self.notify("initialized", {}) + raw = _require_mapping(result, "initialize result") + version = raw.get("protocolVersion") + if ( + not isinstance(version, int) + or isinstance(version, bool) + or version != ACP_PROTOCOL_VERSION + ): + failure = AcpProtocolVersionError( + f"agent selected unsupported ACP protocol version {version!r}" + ) + self._set_failed(failure) + raise failure + capabilities_value = raw.get("agentCapabilities", {}) + if not isinstance(capabilities_value, Mapping): + capabilities_value = {} + agent_info_value = raw.get("agentInfo") + agent_info = ( + MappingProxyType(dict(agent_info_value)) + if isinstance(agent_info_value, Mapping) + else None + ) + auth_methods_value = raw.get("authMethods", []) + auth_methods = ( + tuple( + MappingProxyType(dict(item)) + for item in auth_methods_value + if isinstance(item, Mapping) + ) + if isinstance(auth_methods_value, list) + else () + ) + parsed = InitializeResult( + protocol_version=version, + capabilities=AgentCapabilities.from_mapping(capabilities_value), + agent_info=agent_info, + auth_methods=auth_methods, + raw=MappingProxyType(dict(raw)), + ) + with self._state_lock: + if self._state is not ClientState.RUNNING: + self._raise_unusable() + self._initialize_result = parsed + self._state = ClientState.INITIALIZED + return parsed def request( self, @@ -363,24 +395,51 @@ def request( wait_timeout = self.request_timeout if timeout is None else _positive_timeout( timeout, "timeout" ) + deadline = time.monotonic() + wait_timeout request_id = self._new_request_id() waiter: queue.Queue[JsonRpcResponse | BaseException] = queue.Queue(maxsize=1) + session_id_value = (params or {}).get("sessionId") + pending = _PendingRequest( + waiter=waiter, + method=method, + session_id=( + session_id_value if isinstance(session_id_value, str) else None + ), + ) with self._pending_lock: - self._pending[request_id] = waiter + self._pending[request_id] = pending try: - self._write(request_envelope(request_id, method, params)) + self._write( + request_envelope(request_id, method, params), + deadline=deadline, + ) except BaseException: with self._pending_lock: - self._pending.pop(request_id, None) + if self._pending.get(request_id) is pending: + self._pending.pop(request_id, None) raise try: - response = waiter.get(timeout=wait_timeout) + response = waiter.get(timeout=max(0.0, deadline - time.monotonic())) except queue.Empty as exc: with self._pending_lock: - self._pending.pop(request_id, None) + if self._pending.get(request_id) is pending: + self._pending.pop(request_id, None) + # A response dispatcher that won the lock must enqueue before + # releasing it. Recheck while no dispatcher can still claim us. + try: + response = waiter.get_nowait() + except queue.Empty: + response = None + if response is not None: + if isinstance(response, BaseException): + raise response + return response.result_or_raise() raise AcpRequestTimeoutError( f"ACP request {method!r} timed out after {wait_timeout:g}s" ) from exc + with self._pending_lock: + if self._pending.get(request_id) is pending: + self._pending.pop(request_id, None) if isinstance(response, BaseException): raise response return response.result_or_raise() @@ -472,6 +531,30 @@ def list_sessions( next_cursor = None return SessionPage(sessions, next_cursor, MappingProxyType(dict(raw))) + def close_session( + self, session_id: str, *, timeout: float | None = None + ) -> Mapping[str, Any]: + """Close an active session when advertised by the agent.""" + self._require_capability("sessionClose") + result = self.request( + "session/close", + {"sessionId": _nonempty(session_id, "session_id")}, + timeout=timeout, + ) + return _require_mapping(result, "session/close result") + + def delete_session( + self, session_id: str, *, timeout: float | None = None + ) -> Mapping[str, Any]: + """Delete a listed session when advertised by the agent.""" + self._require_capability("sessionDelete") + result = self.request( + "session/delete", + {"sessionId": _nonempty(session_id, "session_id")}, + timeout=timeout, + ) + return _require_mapping(result, "session/delete result") + def prompt( self, session_id: str, @@ -488,11 +571,30 @@ def prompt( for block in content: if not isinstance(block, Mapping) or not isinstance(block.get("type"), str): raise ValueError("each prompt content block must have a string type") - result = self.request( - "session/prompt", - {"sessionId": _nonempty(session_id, "session_id"), "prompt": content}, - timeout=self.prompt_timeout if timeout is None else timeout, - ) + session_id = _nonempty(session_id, "session_id") + with self._permission_lock: + if self._active_prompts.get(session_id, 0) == 0: + # A new turn supersedes an unconfirmed cancellation from a + # prior timed-out turn. + self._cancelled_sessions.discard(session_id) + self._active_prompts[session_id] = self._active_prompts.get(session_id, 0) + 1 + response_received = False + try: + result = self.request( + "session/prompt", + {"sessionId": session_id, "prompt": content}, + timeout=self.prompt_timeout if timeout is None else timeout, + ) + response_received = True + finally: + with self._permission_lock: + active = self._active_prompts.get(session_id, 0) - 1 + if active > 0: + self._active_prompts[session_id] = active + else: + self._active_prompts.pop(session_id, None) + if response_received: + self._cancelled_sessions.discard(session_id) raw = _require_mapping(result, "session/prompt result") stop_reason = raw.get("stopReason") try: @@ -504,15 +606,32 @@ def prompt( def cancel(self, session_id: str) -> None: """Cancel a turn and cancel all outstanding permissions for the session.""" session_id = _nonempty(session_id, "session_id") - self.notify("session/cancel", {"sessionId": session_id}) with self._permission_lock: - pending_ids = [ - request_id - for request_id, request in self._pending_permissions.items() - if request.session_id == session_id - ] - for request_id in pending_ids: - self.respond_permission(request_id, cancelled=True) + self._cancelled_sessions.add(session_id) + try: + self.notify("session/cancel", {"sessionId": session_id}) + except BaseException: + with self._permission_lock: + self._cancelled_sessions.discard(session_id) + raise + while True: + with self._permission_lock: + pending_ids = [ + request_id + for request_id, request in self._pending_permissions.items() + if request.session_id == session_id + ] + if not pending_ids: + break + for request_id in pending_ids: + try: + self.respond_permission(request_id, cancelled=True) + except AcpClientStateError: + # A consumer may have resolved it concurrently. + continue + with self._permission_lock: + if self._active_prompts.get(session_id, 0) == 0: + self._cancelled_sessions.discard(session_id) def respond_permission( self, @@ -539,9 +658,8 @@ def respond_permission( try: self._write(result_envelope(request_id, {"outcome": outcome})) except BaseException: - # Preserve retryability when nothing was written successfully. - with self._permission_lock: - self._pending_permissions[request_id] = request + # A partial frame may have reached the peer. Retrying the same + # JSON-RPC response is unsafe; transport failure is terminal. raise def next_update(self, *, timeout: float | None = None) -> SessionUpdate: @@ -550,7 +668,23 @@ def next_update(self, *, timeout: float | None = None) -> SessionUpdate: def next_permission_request( self, *, timeout: float | None = None ) -> PermissionRequest: - return self._queue_get(self._permissions, timeout, "permission request") + deadline = None if timeout is None else time.monotonic() + _positive_timeout( + timeout, "timeout" + ) + while True: + remaining = None if deadline is None else deadline - time.monotonic() + if remaining is not None and remaining <= 0: + raise AcpRequestTimeoutError( + "timed out waiting for ACP permission request" + ) + request = self._queue_get( + self._permissions, + remaining, + "permission request", + ) + with self._permission_lock: + if self._pending_permissions.get(request.request_id) is request: + return request def next_notification(self, *, timeout: float | None = None) -> RawNotification: return self._queue_get(self._notifications, timeout, "notification") @@ -568,40 +702,94 @@ def reject_inbound_request( self._write(error_envelope(request_id, code, message)) def close(self) -> None: + wait_for_other_close = False with self._state_lock: if self._state in {ClientState.CLOSED, ClientState.NEW}: self._state = ClientState.CLOSED + self._closed.set() return if self._state is ClientState.CLOSING: - return - was_failed = self._state is ClientState.FAILED - self._state = ClientState.CLOSING - self._stop.set() + wait_for_other_close = True + else: + was_failed = self._state is ClientState.FAILED + self._state = ClientState.CLOSING + if wait_for_other_close: + self._closed.wait(timeout=self.close_timeout * 3) + return process = self._process - if process is not None: - with self._write_lock: - if process.stdin is not None: - try: - process.stdin.close() - except OSError: - pass - try: - process.wait(timeout=self.close_timeout) - except subprocess.TimeoutExpired: - process.terminate() + try: + if process is not None: + acquired_write = self._write_lock.acquire(timeout=self.close_timeout) + if not acquired_write: + # A blocked frame cannot be completed safely during close. + # Terminating our private process group wakes the writer. + self._signal_process(process, signal.SIGTERM) + acquired_write = self._write_lock.acquire( + timeout=self.close_timeout + ) + if not acquired_write: + self._signal_process(process, signal.SIGKILL) + acquired_write = self._write_lock.acquire( + timeout=self.close_timeout + ) + if not acquired_write: + raise AcpTransportError("timed out closing ACP agent stdin") + try: + if process.stdin is not None: + try: + process.stdin.close() + except OSError: + pass + finally: + self._write_lock.release() try: process.wait(timeout=self.close_timeout) except subprocess.TimeoutExpired: - process.kill() - process.wait(timeout=self.close_timeout) - self._exit = ProcessExit(process.returncode, self.stderr_tail()) - for thread in (self._reader_thread, self._stderr_thread): - if thread is not None and thread is not threading.current_thread(): - thread.join(timeout=self.close_timeout) - self._fail_pending(AcpTransportError("ACP client closed")) - self._signal_queues() - with self._state_lock: - self._state = ClientState.FAILED if was_failed else ClientState.CLOSED + self._signal_process(process, signal.SIGTERM) + try: + process.wait(timeout=self.close_timeout) + except subprocess.TimeoutExpired: + self._signal_process(process, signal.SIGKILL) + process.wait(timeout=self.close_timeout) + # The adapter can exit while leaving a spawned agent process + # holding its inherited stdio descriptors. The private process + # group makes those descendants safe to terminate as one unit. + self._signal_process(process, signal.SIGTERM) + self._stop.set() + for thread in (self._reader_thread, self._stderr_thread): + if thread is not None and thread is not threading.current_thread(): + thread.join(timeout=self.close_timeout) + if process is not None and any( + thread is not None and thread.is_alive() + for thread in (self._reader_thread, self._stderr_thread) + ): + self._signal_process(process, signal.SIGKILL) + for thread in (self._reader_thread, self._stderr_thread): + if thread is not None and thread is not threading.current_thread(): + thread.join(timeout=self.close_timeout) + if process is not None: + self._exit = ProcessExit(process.returncode, self.stderr_tail()) + self._fail_pending(AcpTransportError("ACP client closed")) + with self._permission_lock: + self._pending_permissions.clear() + self._cancelled_sessions.clear() + self._active_prompts.clear() + self._signal_queues() + with self._state_lock: + self._state = ClientState.FAILED if was_failed else ClientState.CLOSED + except BaseException as exc: + with self._state_lock: + if self._failure is None: + self._failure = AcpTransportError( + f"failed to stop ACP agent: {type(exc).__name__}" + ) + self._state = ClientState.FAILED + self._fail_pending(self._failure) + self._signal_queues() + raise + finally: + self._stop.set() + self._closed.set() def _session_setup_params( self, @@ -631,6 +819,8 @@ def _require_capability(self, name: str) -> None: "loadSession": self.capabilities.load_session, "sessionList": self.capabilities.session_list, "sessionResume": self.capabilities.session_resume, + "sessionClose": self.capabilities.session_close, + "sessionDelete": self.capabilities.session_delete, "additionalDirectories": self.capabilities.additional_directories, }.get(name, False) if not supported: @@ -639,27 +829,85 @@ def _require_capability(self, name: str) -> None: def _new_request_id(self) -> int: with self._request_id_lock: request_id = self._next_id + if request_id > 2**63 - 1: + raise AcpClientStateError("ACP request ID space exhausted") self._next_id += 1 return request_id - def _write(self, envelope: Mapping[str, Any]) -> None: + def _write( + self, + envelope: Mapping[str, Any], + *, + deadline: float | None = None, + ) -> None: payload = encode_message(envelope, max_frame_bytes=self.max_frame_bytes) - with self._write_lock: + if deadline is None: + deadline = time.monotonic() + self.request_timeout + remaining_timeout = max(0.0, deadline - time.monotonic()) + if not self._write_lock.acquire(timeout=remaining_timeout): + raise AcpRequestTimeoutError("timed out waiting to write ACP frame") + try: self._require_running() process = self._process if process is None or process.stdin is None: raise AcpTransportError("ACP agent stdin is unavailable") try: + fd = process.stdin.fileno() remaining = memoryview(payload) + bytes_written = 0 while remaining: - written = os.write(process.stdin.fileno(), remaining) + wait = deadline - time.monotonic() + if wait <= 0: + raise AcpRequestTimeoutError( + "timed out writing ACP frame to agent" + ) + if not self._wait_writable(fd, wait): + raise AcpRequestTimeoutError( + "timed out writing ACP frame to agent" + ) + try: + written = self._write_chunk(fd, remaining) + except BlockingIOError: + continue if written <= 0: raise BrokenPipeError("zero-byte write to ACP agent stdin") + bytes_written += written remaining = remaining[written:] + except AcpRequestTimeoutError as exc: + if bytes_written: + failure = AcpTransportError( + "ACP frame write timed out after a partial write" + ) + self._set_failed(failure) + raise failure from exc + raise except (BrokenPipeError, OSError) as exc: failure = AcpTransportError("ACP agent stdin disconnected") self._set_failed(failure) raise failure from exc + finally: + self._write_lock.release() + + @staticmethod + def _wait_writable(fd: int, timeout: float) -> bool: + _, writable, _ = select.select([], [fd], [], timeout) + return bool(writable) + + @staticmethod + def _write_chunk(fd: int, data: memoryview) -> int: + return os.write(fd, data) + + @staticmethod + def _signal_process(process: subprocess.Popen[bytes], signum: int) -> None: + try: + if os.name == "posix": + os.killpg(process.pid, signum) + elif signum == signal.SIGTERM: + process.terminate() + else: + process.kill() + except ProcessLookupError: + pass def _reader_main(self) -> None: process = self._process @@ -672,7 +920,7 @@ def _reader_main(self) -> None: room = self.max_frame_bytes + 1 - len(buffer) chunk = os.read(process.stdout.fileno(), min(64 * 1024, room)) if not chunk: - if self._stop.is_set(): + if self._stop.is_set() or self.state is ClientState.CLOSING: return if buffer: raise AcpFramingError("ACP stdout ended during a JSON frame") @@ -702,7 +950,7 @@ def _stderr_main(self) -> None: process = self._process assert process is not None and process.stderr is not None try: - while not self._stop.is_set(): + while True: chunk = os.read(process.stderr.fileno(), 4096) if not chunk: return @@ -725,22 +973,23 @@ def _dispatch( ) -> None: if isinstance(message, JsonRpcResponse): if message.request_id is None: - raise AcpEnvelopeError("uncorrelated ACP response with null id") + # JSON-RPC uses null for errors whose request ID could not be + # recovered. We never emit null request IDs, so this cannot + # correlate to local work and is safe to ignore. + return with self._pending_lock: - waiter = self._pending.pop(message.request_id, None) - if waiter is None: - # A late response after timeout cannot be safely correlated to a - # live operation. Keep the transport usable and surface it as a - # raw diagnostic notification. - self._put_lossless( - self._notifications, - RawNotification( - "$/orphan_response", - MappingProxyType({"id": message.request_id}), - ), - ) + pending = self._pending.get(message.request_id) + if pending is not None: + try: + pending.waiter.put_nowait(message) + except queue.Full as exc: + raise AcpEnvelopeError( + "duplicate ACP response for one request ID" + ) from exc + if pending is None: + # Late responses are expected after a local timeout. They have + # no consumer and must not be copied into a bounded event queue. return - waiter.put_nowait(message) return if isinstance(message, JsonRpcNotification): if message.method == "session/update": @@ -768,8 +1017,18 @@ def _dispatch( with self._permission_lock: if message.request_id in self._pending_permissions: raise AcpEnvelopeError("duplicate pending permission request id") - self._pending_permissions[message.request_id] = parsed - self._put_lossless(self._permissions, parsed) + cancelled = parsed.session_id in self._cancelled_sessions + if not cancelled: + self._pending_permissions[message.request_id] = parsed + if cancelled: + self._write( + result_envelope( + message.request_id, + {"outcome": {"outcome": "cancelled"}}, + ) + ) + else: + self._put_lossless(self._permissions, parsed) else: self._put_lossless( self._inbound_requests, @@ -790,16 +1049,37 @@ def _queue_get( timeout: float | None, description: str, ) -> _T: + deadline = None if timeout is not None: - timeout = _positive_timeout(timeout, "timeout") - try: - item = source.get(timeout=timeout) - except queue.Empty as exc: - raise AcpRequestTimeoutError(f"timed out waiting for ACP {description}") from exc - if item is _END: - self._raise_unusable() - raise AcpTransportError("ACP event stream ended") - return item # type: ignore[return-value] + deadline = time.monotonic() + _positive_timeout(timeout, "timeout") + while True: + if source.empty() and self.state in { + ClientState.FAILED, + ClientState.CLOSING, + ClientState.CLOSED, + }: + self._raise_unusable() + wait = 0.1 + if deadline is not None: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AcpRequestTimeoutError( + f"timed out waiting for ACP {description}" + ) + wait = min(wait, remaining) + try: + item = source.get(timeout=wait) + except queue.Empty: + continue + if item is _END: + # Preserve terminal visibility for all future consumers. + try: + source.put_nowait(_END) + except queue.Full: + pass + self._raise_unusable() + raise AcpTransportError("ACP event stream ended") + return item # type: ignore[return-value] def _set_failed(self, failure: BaseException) -> None: if not isinstance(failure, AcpClientError | AcpProtocolError): @@ -815,11 +1095,11 @@ def _set_failed(self, failure: BaseException) -> None: def _fail_pending(self, failure: BaseException) -> None: with self._pending_lock: - waiters = tuple(self._pending.values()) + pending_requests = tuple(self._pending.values()) self._pending.clear() - for waiter in waiters: + for pending in pending_requests: try: - waiter.put_nowait(failure) + pending.waiter.put_nowait(failure) except queue.Full: pass @@ -870,11 +1150,31 @@ def _nonempty(value: str, name: str) -> str: def _absolute_path(value: str | os.PathLike[str], name: str) -> str: result = os.fspath(value) - if not result or not Path(result).is_absolute(): + if not isinstance(result, str) or not result or "\x00" in result: + raise ValueError(f"{name} must be a non-empty text path without NUL bytes") + if not Path(result).is_absolute(): raise ValueError(f"{name} must be an absolute path") return result +def _validated_env(env: Mapping[str, str] | None) -> dict[str, str] | None: + if env is None: + return None + result: dict[str, str] = {} + for key, value in env.items(): + if ( + not isinstance(key, str) + or not key + or "=" in key + or "\x00" in key + or not isinstance(value, str) + or "\x00" in value + ): + raise ValueError("env must contain valid string names and values") + result[key] = value + return result + + def _require_mapping(value: Any, name: str) -> Mapping[str, Any]: if not isinstance(value, Mapping): raise AcpEnvelopeError(f"{name} must be an object") diff --git a/src/tendwire/backends/acp_protocol.py b/src/tendwire/backends/acp_protocol.py index b01776a..9bc8a77 100644 --- a/src/tendwire/backends/acp_protocol.py +++ b/src/tendwire/backends/acp_protocol.py @@ -18,7 +18,10 @@ ACP_PROTOCOL_VERSION = 1 DEFAULT_MAX_FRAME_BYTES = 8 * 1024 * 1024 -RequestId: TypeAlias = str | int +RequestId: TypeAlias = str | int | None + +_MIN_REQUEST_NUMBER = -(2**63) +_MAX_REQUEST_NUMBER = 2**63 - 1 class AcpProtocolError(Exception): @@ -153,6 +156,14 @@ def session_list(self) -> bool: def session_resume(self) -> bool: return _is_capability_object(self._session_capabilities().get("resume")) + @property + def session_close(self) -> bool: + return _is_capability_object(self._session_capabilities().get("close")) + + @property + def session_delete(self) -> bool: + return _is_capability_object(self._session_capabilities().get("delete")) + @property def additional_directories(self) -> bool: return _is_capability_object( @@ -243,8 +254,16 @@ def _is_capability_object(value: Any) -> bool: def _valid_request_id(value: Any) -> bool: - return (isinstance(value, str) and bool(value)) or ( - isinstance(value, int) and not isinstance(value, bool) + # ACP v1 inherits JSON-RPC's String, integral Number, or Null request IDs. + # The official schema represents Number as a signed 64-bit integer. + return ( + value is None + or isinstance(value, str) + or ( + isinstance(value, int) + and not isinstance(value, bool) + and _MIN_REQUEST_NUMBER <= value <= _MAX_REQUEST_NUMBER + ) ) @@ -328,7 +347,9 @@ def validate_envelope(value: Any) -> JsonRpcMessage: return JsonRpcNotification(method=method, params=frozen_params) request_id = value["id"] if not _valid_request_id(request_id): - raise AcpEnvelopeError("JSON-RPC request id must be a non-empty string or integer") + raise AcpEnvelopeError( + "JSON-RPC request id must be a string, signed 64-bit integer, or null" + ) return JsonRpcRequest( request_id=request_id, method=method, @@ -464,10 +485,14 @@ def parse_permission_request(request: JsonRpcRequest) -> PermissionRequest: if not isinstance(raw_options, list) or not raw_options: raise AcpEnvelopeError("permission request options must be a non-empty array") options: list[PermissionOption] = [] + seen_option_ids: set[str] = set() for raw in raw_options: if not isinstance(raw, Mapping): raise AcpEnvelopeError("permission option must be an object") option_id = _required_string(raw, "optionId") + if option_id in seen_option_ids: + raise AcpEnvelopeError("permission option IDs must be unique") + seen_option_ids.add(option_id) name = _required_string(raw, "name") kind_value = _required_string(raw, "kind") try: diff --git a/tests/fixtures/acp_fake_agent.py b/tests/fixtures/acp_fake_agent.py index 1d31a7f..c513cd3 100644 --- a/tests/fixtures/acp_fake_agent.py +++ b/tests/fixtures/acp_fake_agent.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +import signal import sys import time @@ -34,6 +35,13 @@ def update(session_id: str, kind: str, **values: object) -> None: pending_prompt_id: object | None = None pending_prompt_session = "" +pending_permission_ids: set[object] = set() + +if MODE == "no_read": + time.sleep(60) + +if MODE == "stubborn": + signal.signal(signal.SIGTERM, signal.SIG_IGN) for line in sys.stdin: message = json.loads(line) @@ -46,13 +54,17 @@ def update(session_id: str, kind: str, **values: object) -> None: sys.stdout.write("not-json\n") sys.stdout.flush() continue + if MODE == "partial_eof": + sys.stdout.write('{"jsonrpc":"2.0","id":') + sys.stdout.flush() + raise SystemExit(0) if MODE == "oversize": response(request_id, {"protocolVersion": 1, "padding": "x" * 10000}) continue response( request_id, { - "protocolVersion": 1, + "protocolVersion": True if MODE == "bool_version" else 1, "agentCapabilities": ( {} if MODE == "baseline" @@ -60,22 +72,48 @@ def update(session_id: str, kind: str, **values: object) -> None: "loadSession": True, "sessionCapabilities": { "list": {}, + "delete": {}, "resume": {}, + "close": {}, "additionalDirectories": {}, }, + "vendorFutureCapability": {"level": 2}, } ), "agentInfo": {"name": "fake", "version": "1.0"}, }, ) - elif method == "initialized": - send( - { - "jsonrpc": "2.0", - "method": "fake/initialized_seen", - "params": {}, - } - ) + if MODE == "extensions": + send( + { + "jsonrpc": "2.0", + "method": "vendor/future_notification", + "params": {"opaque": {"revision": 9}}, + } + ) + if MODE == "null_response": + send( + { + "jsonrpc": "2.0", + "id": None, + "error": {"code": -32600, "message": "unrelated invalid request"}, + } + ) + if MODE == "stderr_tail": + sys.stderr.write("prefix-" + "x" * 500 + "-TAIL") + sys.stderr.flush() + if MODE == "exit_after_init": + time.sleep(0.05) + raise SystemExit(0) + if MODE == "flood": + time.sleep(0.05) + for index in range(4): + update( + "s-flood", + "vendor_progress", + sequence=index, + vendor={"opaque": True}, + ) elif method == "session/new": update("s-new", "agent_message_chunk", content={"type": "text", "text": "hi"}) response( @@ -84,6 +122,8 @@ def update(session_id: str, kind: str, **values: object) -> None: ) elif method == "session/load" or method == "session/resume": response(request_id, {"configOptions": [{"id": "model", "currentValue": "x"}]}) + elif method == "session/close" or method == "session/delete": + response(request_id, {"vendorReceipt": method}) elif method == "session/list": if MODE == "slow": time.sleep(2) @@ -133,18 +173,43 @@ def update(session_id: str, kind: str, **values: object) -> None: }, } ) + pending_permission_ids.add(900) elif method == "session/cancel": - # The client must additionally resolve permission request 900 as cancelled. - pass - elif request_id == 900 and pending_prompt_id is not None: + if MODE == "cancel_race" and pending_prompt_id is not None: + send( + { + "jsonrpc": "2.0", + "id": 901, + "method": "session/request_permission", + "params": { + "sessionId": pending_prompt_session, + "toolCall": {"toolCallId": "tool-race", "status": "pending"}, + "options": [ + { + "optionId": "allow-race", + "name": "Allow once", + "kind": "allow_once", + } + ], + }, + } + ) + pending_permission_ids.add(901) + elif request_id in pending_permission_ids and pending_prompt_id is not None: outcome = message["result"]["outcome"]["outcome"] - update( - pending_prompt_session, - "plan", - entries=[{"content": "done", "status": "completed"}], - ) - response( - pending_prompt_id, - {"stopReason": "cancelled" if outcome == "cancelled" else "end_turn"}, - ) - pending_prompt_id = None + pending_permission_ids.remove(request_id) + if not pending_permission_ids: + update( + pending_prompt_session, + "plan", + entries=[{"content": "done", "status": "completed"}], + ) + response( + pending_prompt_id, + {"stopReason": "cancelled" if outcome == "cancelled" else "end_turn"}, + ) + pending_prompt_id = None + +if MODE == "stubborn": + while True: + time.sleep(60) diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py index d1c8638..974d277 100644 --- a/tests/test_acp_client.py +++ b/tests/test_acp_client.py @@ -2,6 +2,7 @@ import sys import threading +import time from pathlib import Path import pytest @@ -9,7 +10,9 @@ from tendwire.backends.acp_client import ( AcpCapabilityError, AcpClient, + AcpEventQueueFullError, AcpRequestTimeoutError, + AcpTransportError, ClientState, ) from tendwire.backends.acp_protocol import AcpProtocolError, SessionUpdateKind, StopReason @@ -30,6 +33,9 @@ def test_initialize_capabilities_and_session_lifecycle() -> None: assert initialized.capabilities.load_session assert initialized.capabilities.session_list assert initialized.capabilities.session_resume + assert initialized.capabilities.session_close + assert initialized.capabilities.session_delete + assert initialized.capabilities.raw["vendorFutureCapability"] == {"level": 2} assert acp.state is ClientState.INITIALIZED created = acp.new_session( @@ -53,8 +59,8 @@ def test_initialize_capabilities_and_session_lifecycle() -> None: assert second.sessions[0].title == "second" assert second.next_cursor is None - acp.initialized() - assert acp.next_notification(timeout=1).method == "fake/initialized_seen" + assert acp.close_session("s1")["vendorReceipt"] == "session/close" + assert acp.delete_session("s2")["vendorReceipt"] == "session/delete" assert acp.state is ClientState.CLOSED assert acp.exit is not None @@ -108,6 +114,23 @@ def test_cancel_resolves_pending_permissions_as_cancelled() -> None: assert result[0].stop_reason is StopReason.CANCELLED +def test_cancel_also_resolves_permission_that_races_after_notification() -> None: + with client("cancel_race") as acp: + acp.initialize() + result: list[object] = [] + thread = threading.Thread(target=lambda: result.append(acp.prompt("s1", "wait"))) + thread.start() + acp.next_update(timeout=1) + acp.next_permission_request(timeout=1) + acp.cancel("s1") + acp.next_update(timeout=1) + thread.join(timeout=2) + assert not thread.is_alive() + assert result[0].stop_reason is StopReason.CANCELLED + with pytest.raises(AcpRequestTimeoutError): + acp.next_permission_request(timeout=0.05) + + def test_optional_methods_require_advertised_capabilities() -> None: with client("baseline") as acp: acp.initialize() @@ -115,6 +138,10 @@ def test_optional_methods_require_advertised_capabilities() -> None: acp.list_sessions() with pytest.raises(AcpCapabilityError): acp.load_session("s1", "/tmp") + with pytest.raises(AcpCapabilityError): + acp.close_session("s1") + with pytest.raises(AcpCapabilityError): + acp.delete_session("s1") def test_request_timeout_does_not_poison_transport() -> None: @@ -125,7 +152,7 @@ def test_request_timeout_does_not_poison_transport() -> None: assert acp.state is ClientState.INITIALIZED -@pytest.mark.parametrize("mode", ["malformed", "oversize"]) +@pytest.mark.parametrize("mode", ["malformed", "oversize", "partial_eof"]) def test_malformed_or_oversized_stdout_fails_connection(mode: str) -> None: with client(mode, max_frame_bytes=1024) as acp: with pytest.raises(AcpProtocolError): @@ -138,3 +165,135 @@ def test_absolute_session_paths_are_enforced_before_write() -> None: acp.initialize() with pytest.raises(ValueError, match="absolute"): acp.new_session("relative/path") + + +def test_concurrent_initialize_is_exactly_once_and_returns_same_result() -> None: + with client() as acp: + results: list[object] = [] + failures: list[BaseException] = [] + + def initialize() -> None: + try: + results.append(acp.initialize()) + except BaseException as exc: # pragma: no cover - diagnostic path + failures.append(exc) + + threads = [threading.Thread(target=initialize) for _ in range(6)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=2) + assert not failures + assert len(results) == 6 + assert all(result is results[0] for result in results) + + +def test_backpressure_failure_remains_visible_after_full_queue_drains() -> None: + with client("flood", max_pending_events=1) as acp: + acp.initialize() + deadline = time.monotonic() + 1 + while acp.state is not ClientState.FAILED and time.monotonic() < deadline: + time.sleep(0.01) + assert acp.state is ClientState.FAILED + assert isinstance(acp.failure, AcpEventQueueFullError) + first = acp.next_update(timeout=1) + assert first.update_kind == "vendor_progress" + with pytest.raises(AcpTransportError): + acp.next_update(timeout=1) + + +def test_blocked_or_partial_stdin_write_is_bounded_and_terminal( + monkeypatch: pytest.MonkeyPatch, +) -> None: + real_write = AcpClient._write_chunk + select_calls = 0 + + def partial_write(fd: int, data: memoryview) -> int: + return real_write(fd, data[:8]) + + def writable_once(fd: int, timeout: float) -> bool: + nonlocal select_calls + select_calls += 1 + return select_calls == 1 + + monkeypatch.setattr(AcpClient, "_write_chunk", staticmethod(partial_write)) + monkeypatch.setattr(AcpClient, "_wait_writable", staticmethod(writable_once)) + acp = client("no_read", request_timeout=0.2, close_timeout=0.05) + try: + started = time.monotonic() + with pytest.raises(AcpTransportError, match="partial write"): + acp.initialize( + client_capabilities={"vendor/padding": "small"}, + timeout=0.2, + ) + assert time.monotonic() - started < 1 + assert acp.state is ClientState.FAILED + finally: + acp.close() + + +def test_close_escalates_to_kill_for_stubborn_adapter() -> None: + acp = client("stubborn", close_timeout=0.05) + acp.initialize() + acp.close() + assert acp.state is ClientState.CLOSED + assert acp.exit is not None + assert acp.exit.returncode != 0 + + +def test_stderr_tail_is_bounded_and_keeps_suffix() -> None: + acp = client("stderr_tail", stderr_limit_bytes=64) + acp.initialize() + acp.close() + tail = acp.stderr_tail() + assert len(tail.encode()) <= 64 + assert tail.endswith("-TAIL") + + +def test_unknown_adapter_extensions_remain_observable() -> None: + with client("extensions") as acp: + initialized = acp.initialize() + assert initialized.capabilities.raw["vendorFutureCapability"] == {"level": 2} + notification = acp.next_notification(timeout=1) + assert notification.method == "vendor/future_notification" + assert notification.params["opaque"] == {"revision": 9} + + +def test_uncorrelated_null_error_response_does_not_poison_transport() -> None: + with client("null_response") as acp: + acp.initialize() + time.sleep(0.05) + assert acp.state is ClientState.INITIALIZED + + +def test_unexpected_clean_stdout_eof_is_transport_failure() -> None: + with client("exit_after_init") as acp: + acp.initialize() + deadline = time.monotonic() + 1 + while acp.state is not ClientState.FAILED and time.monotonic() < deadline: + time.sleep(0.01) + assert acp.state is ClientState.FAILED + assert isinstance(acp.failure, AcpTransportError) + + +def test_boolean_protocol_version_is_not_accepted_as_integer_one() -> None: + with client("bool_version") as acp: + with pytest.raises(AcpProtocolError, match="protocol version"): + acp.initialize() + assert acp.state is ClientState.FAILED + + +@pytest.mark.parametrize( + "kwargs", + [ + {"cwd": "relative"}, + {"cwd": "/tmp/bad\x00path"}, + {"env": {"BAD=NAME": "value"}}, + {"env": {"NAME": "bad\x00value"}}, + ], +) +def test_process_paths_and_environment_are_validated( + kwargs: dict[str, object], +) -> None: + with pytest.raises(ValueError): + client(**kwargs) diff --git a/tests/test_acp_protocol.py b/tests/test_acp_protocol.py index 7dc567f..41407fd 100644 --- a/tests/test_acp_protocol.py +++ b/tests/test_acp_protocol.py @@ -17,6 +17,7 @@ parse_permission_request, parse_session_update, request_envelope, + validate_envelope, ) @@ -106,12 +107,74 @@ def test_capability_presence_uses_acp_object_semantics() -> None: "loadSession": True, "sessionCapabilities": { "list": {}, + "delete": {}, "resume": {}, + "close": {}, "additionalDirectories": {}, }, + "vendorFutureCapability": {"revision": 3}, } ) assert capabilities.load_session assert capabilities.session_list assert capabilities.session_resume + assert capabilities.session_close + assert capabilities.session_delete assert capabilities.additional_directories + assert capabilities.raw["vendorFutureCapability"] == {"revision": 3} + + +def test_request_ids_follow_acp_json_rpc_domain() -> None: + assert isinstance( + validate_envelope({"jsonrpc": "2.0", "id": "", "method": "vendor/x"}), + JsonRpcRequest, + ) + null_request = validate_envelope( + {"jsonrpc": "2.0", "id": None, "method": "vendor/x"} + ) + assert isinstance(null_request, JsonRpcRequest) + assert null_request.request_id is None + assert isinstance( + validate_envelope( + {"jsonrpc": "2.0", "id": 2**63 - 1, "result": {}} + ), + JsonRpcResponse, + ) + for invalid_id in (True, 2**63, -(2**63) - 1, 1.5): + with pytest.raises(AcpEnvelopeError): + validate_envelope( + {"jsonrpc": "2.0", "id": invalid_id, "method": "vendor/x"} + ) + + +def test_permission_option_ids_must_be_unambiguous() -> None: + request = JsonRpcRequest( + 42, + "session/request_permission", + { + "sessionId": "s1", + "toolCall": {"toolCallId": "tool-1"}, + "options": [ + {"optionId": "same", "name": "Allow", "kind": "allow_once"}, + {"optionId": "same", "name": "Reject", "kind": "reject_once"}, + ], + }, + ) + with pytest.raises(AcpEnvelopeError, match="unique"): + parse_permission_request(request) + + +def test_unknown_update_and_nested_extension_payload_are_preserved() -> None: + extension = parse_session_update( + { + "sessionId": "s1", + "update": { + "sessionUpdate": "vendor/future_progress", + "opaque": {"revision": 7, "items": [1, 2]}, + }, + "_meta": {"vendor.example/trace": "abc"}, + } + ) + assert extension.update_kind == "vendor/future_progress" + assert extension.update["opaque"] == {"revision": 7, "items": [1, 2]} + assert extension.meta == {"vendor.example/trace": "abc"} From d6daf651e0e2311280b9d9aa5cadbc9417cad161 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:33:29 +0800 Subject: [PATCH 14/83] Harden ACP event projection boundaries --- src/tendwire/backends/acp_projection.py | 449 ++++++++++++++++++++---- tests/test_acp_projection.py | 320 ++++++++++++++++- 2 files changed, 704 insertions(+), 65 deletions(-) diff --git a/src/tendwire/backends/acp_projection.py b/src/tendwire/backends/acp_projection.py index 5588e66..333bba6 100644 --- a/src/tendwire/backends/acp_projection.py +++ b/src/tendwire/backends/acp_projection.py @@ -55,6 +55,8 @@ "complete": False, "has_open_turn": False, } +_MAX_IDENTIFIER_CHARS: Final[int] = 2048 +_MAX_SOURCE_ID_CHARS: Final[int] = 512 class AcpProjectionError(ValueError): @@ -77,7 +79,10 @@ class _SessionState: plan: list[dict[str, Any]] = field(default_factory=list) usage: dict[str, Any] = field(default_factory=dict) info: dict[str, Any] = field(default_factory=dict) - seen_source_events: set[str] = field(default_factory=set) + # The digest lets us distinguish a harmless replay from a producer reusing + # one supposedly authoritative ID for different content. + seen_source_events: dict[str, str] = field(default_factory=dict) + retained_bytes: int = 0 complete: bool = False @@ -92,8 +97,46 @@ class AcpEventProjector: can be legitimate and are therefore never blindly discarded. """ - def __init__(self) -> None: + def __init__( + self, + *, + max_sessions: int = 64, + max_source_events_per_session: int = 4096, + max_messages_per_kind: int = 1024, + max_tool_calls_per_session: int = 4096, + max_state_fields: int = 1024, + max_plan_entries: int = 4096, + max_text_chars_per_message: int = 4 * 1024 * 1024, + max_event_bytes: int = 8 * 1024 * 1024, + max_session_state_bytes: int = 8 * 1024 * 1024, + max_total_state_bytes: int = 128 * 1024 * 1024, + ) -> None: + limits = { + "max_sessions": max_sessions, + "max_source_events_per_session": max_source_events_per_session, + "max_messages_per_kind": max_messages_per_kind, + "max_tool_calls_per_session": max_tool_calls_per_session, + "max_state_fields": max_state_fields, + "max_plan_entries": max_plan_entries, + "max_text_chars_per_message": max_text_chars_per_message, + "max_event_bytes": max_event_bytes, + "max_session_state_bytes": max_session_state_bytes, + "max_total_state_bytes": max_total_state_bytes, + } + for name, value in limits.items(): + if isinstance(value, bool) or not isinstance(value, int) or value < 1: + raise ValueError(f"{name} must be a positive integer") self._sessions: dict[str, _SessionState] = {} + self._max_sessions = max_sessions + self._max_source_events_per_session = max_source_events_per_session + self._max_messages_per_kind = max_messages_per_kind + self._max_tool_calls_per_session = max_tool_calls_per_session + self._max_state_fields = max_state_fields + self._max_plan_entries = max_plan_entries + self._max_text_chars_per_message = max_text_chars_per_message + self._max_event_bytes = max_event_bytes + self._max_session_state_bytes = max_session_state_bytes + self._max_total_state_bytes = max_total_state_bytes def normalize_session_update( self, @@ -122,14 +165,32 @@ def normalize_session_update( if kind is None: return None - state = self._sessions.setdefault(session_id, _SessionState()) - explicit_id = source_event_id or _source_event_id(notification, params, update) + _bounded_json( + {"update": update, "_meta": params.get("_meta")}, + label="ACP session update", + max_bytes=self._max_event_bytes, + ) + state, is_new_session = self._pending_session(session_id) + explicit_id = _explicit_source_event_id(source_event_id) + if explicit_id is None: + explicit_id = _source_event_id(notification, params, update) + replay_digest = _event_digest(kind, update) if explicit_id is not None: - scoped_id = f"{session_id}:{explicit_id}" - if scoped_id in state.seen_source_events: + previous_digest = state.seen_source_events.get(explicit_id) + if previous_digest == replay_digest: return None + if previous_digest is not None: + raise AcpProjectionError( + "ACP source event ID was reused for different content" + ) + if len(state.seen_source_events) >= self._max_source_events_per_session: + raise AcpProjectionError("ACP source event replay window is full") if kind in _MESSAGE_KINDS: + if state.complete: + raise AcpProjectionError( + "ACP turn is complete; reset_turn is required before new message chunks" + ) payload = self._normalize_message(state, kind, update) elif kind in {"tool_call", "tool_call_update"}: payload = self._normalize_tool(state, kind, update) @@ -139,11 +200,18 @@ def normalize_session_update( payload = self._normalize_usage(state, update) else: payload = self._normalize_session_info(state, update) + extension_meta = { + **_extension_metadata(params), + **_extension_metadata(update), + } + if extension_meta: + payload["extensions"] = extension_meta state.sequence += 1 if explicit_id is not None: - state.seen_source_events.add(f"{session_id}:{explicit_id}") - state.complete = False + state.seen_source_events[explicit_id] = replay_digest + if is_new_session: + self._sessions[session_id] = state return _canonical_event( session_id=session_id, sequence=state.sequence, @@ -175,33 +243,69 @@ def normalize_permission_request( raise AcpProjectionError("ACP permission request is missing toolCall") tool_call_id = _required_string(tool_call, "toolCallId") - state = self._sessions.setdefault(session_id, _SessionState()) - explicit_id = source_event_id or _jsonrpc_request_id(request) or _source_event_id( - request, params, tool_call + _bounded_json( + params, + label="ACP permission request", + max_bytes=self._max_event_bytes, ) + state, is_new_session = self._pending_session(session_id) + explicit_id = _explicit_source_event_id(source_event_id) + if explicit_id is None: + explicit_id = _jsonrpc_request_id(request) or _source_event_id( + request, params, tool_call + ) + options = params.get("options") + if not isinstance(options, list) or not options: + raise AcpProjectionError("ACP permission request options must be non-empty") + normalized_options = _permission_options(options) + request_material = {"toolCall": tool_call, "options": normalized_options} + replay_digest = _event_digest("tool_call_update", request_material) if explicit_id is not None: - scoped_id = f"{session_id}:{explicit_id}" - if scoped_id in state.seen_source_events: + previous_digest = state.seen_source_events.get(explicit_id) + if previous_digest == replay_digest: return None - - snapshot = _merge_tool_snapshot(state.tools.get(tool_call_id), tool_call) - options = params.get("options", []) - if not isinstance(options, list): - options = [] + if previous_digest is not None: + raise AcpProjectionError( + "ACP source event ID was reused for different content" + ) + if len(state.seen_source_events) >= self._max_source_events_per_session: + raise AcpProjectionError("ACP source event replay window is full") + + if ( + tool_call_id not in state.tools + and len(state.tools) >= self._max_tool_calls_per_session + ): + raise AcpProjectionError("ACP tool call state limit exceeded") + previous_tool = state.tools.get(tool_call_id) + snapshot = _merge_tool_snapshot(previous_tool, tool_call) snapshot["permission"] = { "required": True, - "options": [deepcopy(option) for option in options if isinstance(option, Mapping)], + "options": deepcopy(normalized_options), } + if len(snapshot) > self._max_state_fields: + raise AcpProjectionError("ACP tool snapshot field limit exceeded") + self._reserve_state( + state, + _json_size(snapshot) - (_json_size(previous_tool) if previous_tool else 0), + ) state.tools[tool_call_id] = snapshot state.sequence += 1 if explicit_id is not None: - state.seen_source_events.add(f"{session_id}:{explicit_id}") + state.seen_source_events[explicit_id] = replay_digest + if is_new_session: + self._sessions[session_id] = state payload = { "tool_call_id": tool_call_id, "changes": _without_discriminator(tool_call), "snapshot": deepcopy(snapshot), "permission": deepcopy(snapshot["permission"]), } + extension_meta = { + **_extension_metadata(params), + **_extension_metadata(tool_call), + } + if extension_meta: + payload["extensions"] = extension_meta return _canonical_event( session_id=session_id, sequence=state.sequence, @@ -209,7 +313,11 @@ def normalize_permission_request( payload=payload, source_event_id=explicit_id, replay=replay, - original_update={"sessionUpdate": "tool_call_update", **dict(tool_call)}, + original_update={ + "sessionUpdate": "tool_call_update", + "toolCall": dict(tool_call), + "options": normalized_options, + }, ) def project_turn_content( @@ -229,7 +337,11 @@ def project_turn_content( state = self._sessions.get(session_id) if state is None: return dict(_LEGACY_EMPTY) - is_complete = state.complete if complete is None else bool(complete) + # A read-only projection may promote a snapshot to final, but must + # never demote already-final state. Reopening requires reset_turn(). + is_complete = state.complete or ( + bool(complete) if complete is not None else False + ) user_text = _joined_messages(state.messages["user_message"]) assistant_text = _joined_messages(state.messages["agent_message"]) return { @@ -243,17 +355,25 @@ def project_turn_content( def mark_turn_complete(self, session_id: str) -> dict[str, Any]: """Mark the current ACP prompt turn complete and return legacy content.""" - state = self._sessions.setdefault(session_id, _SessionState()) + state = self._session(session_id) state.complete = True return self.project_turn_content(session_id) def reset_turn(self, session_id: str) -> None: """Start a fresh prompt turn while preserving session-level ACP state.""" - state = self._sessions.setdefault(session_id, _SessionState()) + state = self._session(session_id) + state.retained_bytes = max( + 0, state.retained_bytes - _messages_state_bytes(state.messages) + ) state.messages = {kind: [] for kind in _MESSAGE_KINDS} state.complete = False + def drop_session(self, session_id: str) -> bool: + """Release all in-memory state after the owning ACP session is closed.""" + + return self._sessions.pop(session_id, None) is not None + def session_snapshot(self, session_id: str) -> dict[str, Any] | None: """Return a defensive snapshot for persistence or diagnostics.""" @@ -277,8 +397,8 @@ def session_snapshot(self, session_id: str) -> dict[str, Any] | None: "complete": state.complete, } - @staticmethod def _normalize_message( + self, state: _SessionState, kind: str, update: Mapping[str, Any], @@ -288,69 +408,151 @@ def _normalize_message( raise AcpProjectionError(f"ACP {kind} update is missing content") message_id_value = update.get("messageId") assemblies = state.messages[kind] - message_id = ( - message_id_value - if isinstance(message_id_value, str) and message_id_value - else assemblies[-1].message_id - if assemblies - else f"implicit-{kind}-1" + if message_id_value is not None: + message_id = _identifier(message_id_value, "messageId") + else: + # ACP stable v1 chunks do not require message IDs. Keep their + # assembly separate from adapter extensions that do provide IDs. + message_id = f"implicit-{kind}-1" + assembly = next( + (item for item in assemblies if item.message_id == message_id), None ) - if not assemblies or assemblies[-1].message_id != message_id: - assemblies.append(_MessageAssembly(message_id=message_id)) text_delta = content.get("text") if content.get("type") == "text" else None if not isinstance(text_delta, str): text_delta = "" - assemblies[-1].text += text_delta + content_copy = _content_payload(content) + extension_meta = _extension_metadata(update) + previous_text = assembly.text if assembly is not None else "" + assembled_text = previous_text + text_delta + if len(assembled_text) > self._max_text_chars_per_message: + raise AcpProjectionError("ACP assembled message text limit exceeded") + if assembly is None: + if len(assemblies) >= self._max_messages_per_kind: + raise AcpProjectionError("ACP message assembly limit exceeded") + self._reserve_state( + state, + len(message_id.encode("utf-8")) + len(text_delta.encode("utf-8")), + ) + assembly = _MessageAssembly(message_id=message_id, text=assembled_text) + assemblies.append(assembly) + else: + self._reserve_state(state, len(text_delta.encode("utf-8"))) + assembly.text = assembled_text return { "message_id": message_id, - "content": deepcopy(dict(content)), + "content": content_copy, "text_delta": text_delta, - "assembled_text": assemblies[-1].text, - "message_index": len(assemblies) - 1, + "assembled_text": assembled_text, + "message_index": assemblies.index(assembly), + **({"extensions": extension_meta} if extension_meta else {}), } - @staticmethod def _normalize_tool( + self, state: _SessionState, kind: str, update: Mapping[str, Any], ) -> dict[str, Any]: tool_call_id = _required_string(update, "toolCallId") previous = state.tools.get(tool_call_id) + if previous is None and len(state.tools) >= self._max_tool_calls_per_session: + raise AcpProjectionError("ACP tool call state limit exceeded") snapshot = _merge_tool_snapshot(previous, update) - state.tools[tool_call_id] = snapshot + if len(snapshot) > self._max_state_fields: + raise AcpProjectionError("ACP tool snapshot field limit exceeded") payload: dict[str, Any] = { "tool_call_id": tool_call_id, "snapshot": deepcopy(snapshot), } if kind == "tool_call_update": payload["changes"] = _without_discriminator(update) + extension_meta = _extension_metadata(update) + if extension_meta: + payload["extensions"] = extension_meta + self._reserve_state( + state, _json_size(snapshot) - (_json_size(previous) if previous else 0) + ) + state.tools[tool_call_id] = snapshot return payload - @staticmethod def _normalize_plan( - state: _SessionState, update: Mapping[str, Any] + self, state: _SessionState, update: Mapping[str, Any] ) -> dict[str, Any]: - entries = update.get("entries", []) + entries = update.get("entries") if not isinstance(entries, list): - entries = [] - state.plan = [deepcopy(dict(entry)) for entry in entries if isinstance(entry, Mapping)] - return {"entries": deepcopy(state.plan), "snapshot": True} + raise AcpProjectionError("ACP plan update entries must be an array") + if len(entries) > self._max_plan_entries: + raise AcpProjectionError("ACP plan entry limit exceeded") + if any(not isinstance(entry, Mapping) for entry in entries): + raise AcpProjectionError("ACP plan entry must be an object") + replacement = [deepcopy(dict(entry)) for entry in entries] + payload: dict[str, Any] = {"entries": deepcopy(replacement), "snapshot": True} + extension_meta = _extension_metadata(update) + if extension_meta: + payload["extensions"] = extension_meta + self._reserve_state(state, _json_size(replacement) - _json_size(state.plan)) + state.plan = replacement + return payload - @staticmethod def _normalize_usage( - state: _SessionState, update: Mapping[str, Any] + self, state: _SessionState, update: Mapping[str, Any] ) -> dict[str, Any]: - state.usage.update(_without_discriminator(update)) - return deepcopy(state.usage) + replacement = {**state.usage, **_without_discriminator(update)} + if len(replacement) > self._max_state_fields: + raise AcpProjectionError("ACP usage state field limit exceeded") + self._reserve_state(state, _json_size(replacement) - _json_size(state.usage)) + state.usage = replacement + payload = deepcopy(state.usage) + extension_meta = _extension_metadata(update) + if extension_meta: + payload["extensions"] = extension_meta + return payload - @staticmethod def _normalize_session_info( - state: _SessionState, update: Mapping[str, Any] + self, state: _SessionState, update: Mapping[str, Any] ) -> dict[str, Any]: # Presence is meaningful: explicit null clears an existing property. - state.info.update(_without_discriminator(update)) - return deepcopy(state.info) + replacement = {**state.info, **_without_discriminator(update)} + if len(replacement) > self._max_state_fields: + raise AcpProjectionError("ACP session info state field limit exceeded") + self._reserve_state(state, _json_size(replacement) - _json_size(state.info)) + state.info = replacement + payload = deepcopy(state.info) + extension_meta = _extension_metadata(update) + if extension_meta: + payload["extensions"] = extension_meta + return payload + + def _session(self, session_id: str) -> _SessionState: + state = self._sessions.get(session_id) + if state is not None: + return state + if len(self._sessions) >= self._max_sessions: + raise AcpProjectionError("ACP projector session limit exceeded") + state = _SessionState() + self._sessions[session_id] = state + return state + + def _pending_session(self, session_id: str) -> tuple[_SessionState, bool]: + state = self._sessions.get(session_id) + if state is not None: + return state, False + if len(self._sessions) >= self._max_sessions: + raise AcpProjectionError("ACP projector session limit exceeded") + return _SessionState(), True + + def _reserve_state(self, state: _SessionState, retained_delta: int) -> None: + retained = max(0, state.retained_bytes + retained_delta) + if retained > self._max_session_state_bytes: + raise AcpProjectionError("ACP retained session state limit exceeded") + other_retained = sum( + item.retained_bytes + for item in self._sessions.values() + if item is not state + ) + if other_retained + retained > self._max_total_state_bytes: + raise AcpProjectionError("ACP total retained state limit exceeded") + state.retained_bytes = retained def _unwrap_params(value: Mapping[str, Any]) -> Mapping[str, Any]: @@ -362,23 +564,42 @@ def _unwrap_params(value: Mapping[str, Any]) -> Mapping[str, Any]: def _required_string(value: Mapping[str, Any], key: str) -> str: item = value.get(key) - if not isinstance(item, str) or not item: - raise AcpProjectionError(f"ACP value is missing non-empty {key}") - return item + try: + return _identifier(item, key) + except AcpProjectionError: + raise AcpProjectionError(f"ACP value is missing non-empty {key}") from None + + +def _identifier(value: Any, label: str) -> str: + if not isinstance(value, str): + raise AcpProjectionError(f"ACP value has invalid {label}") + try: + value.encode("utf-8") + except UnicodeEncodeError as exc: + raise AcpProjectionError(f"ACP value has invalid {label}") from exc + if not value or len(value) > _MAX_IDENTIFIER_CHARS or "\x00" in value: + raise AcpProjectionError(f"ACP value has invalid {label}") + return value + + +def _explicit_source_event_id(value: Any) -> str | None: + if value is None: + return None + return _source_identifier(value, "source_event_id") def _source_event_id(*values: Mapping[str, Any]) -> str | None: for value in values: for key in ("eventId", "event_id", "notificationId", "notification_id"): candidate = value.get(key) - if isinstance(candidate, (str, int)) and str(candidate): - return str(candidate) + if _valid_wire_id(candidate): + return _source_identifier(str(candidate), "source event ID") meta = value.get("_meta") if isinstance(meta, Mapping): for key in ("eventId", "event_id", "notificationId", "notification_id"): candidate = meta.get(key) - if isinstance(candidate, (str, int)) and str(candidate): - return str(candidate) + if _valid_wire_id(candidate): + return _source_identifier(str(candidate), "source event ID") return None @@ -388,11 +609,26 @@ def _jsonrpc_request_id(value: Mapping[str, Any]) -> str | None: if value.get("method") != "session/request_permission": return None candidate = value.get("id") - if isinstance(candidate, (str, int)) and str(candidate): - return str(candidate) + if _valid_wire_id(candidate): + # JSON-RPC request IDs and producer notification IDs are separate + # namespaces and commonly both start at small integers. + return _source_identifier(f"request:{candidate}", "JSON-RPC request ID") return None +def _valid_wire_id(value: Any) -> bool: + return not isinstance(value, bool) and isinstance(value, (str, int)) and bool( + str(value) + ) + + +def _source_identifier(value: Any, label: str) -> str: + identifier = _identifier(value, label) + if len(identifier) > _MAX_SOURCE_ID_CHARS: + raise AcpProjectionError(f"ACP value has invalid {label}") + return identifier + + def _without_discriminator(value: Mapping[str, Any]) -> dict[str, Any]: return { key: deepcopy(item) @@ -409,6 +645,91 @@ def _merge_tool_snapshot( return snapshot +def _content_payload(content: Mapping[str, Any]) -> dict[str, Any]: + """Retain content plus only explicitly namespaced private metadata.""" + + copied = { + key: deepcopy(item) for key, item in content.items() if key != "_meta" + } + meta = _extension_metadata(content) + if meta: + copied["_meta"] = meta + return copied + + +def _extension_metadata(value: Mapping[str, Any]) -> dict[str, Any]: + meta = value.get("_meta") + if not isinstance(meta, Mapping): + return {} + return { + str(key): deepcopy(item) + for key, item in meta.items() + if isinstance(key, str) and "/" in key + } + + +def _permission_options(options: list[Any]) -> list[dict[str, Any]]: + normalized: list[dict[str, Any]] = [] + seen: set[str] = set() + for option in options: + if not isinstance(option, Mapping): + raise AcpProjectionError("ACP permission request option must be an object") + option_id = _required_string(option, "optionId") + _required_string(option, "name") + _required_string(option, "kind") + if option_id in seen: + raise AcpProjectionError("ACP permission option IDs must be unique") + seen.add(option_id) + normalized.append(deepcopy(dict(option))) + return normalized + + +def _bounded_json(value: Any, *, label: str, max_bytes: int) -> bytes: + try: + encoded = json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + ).encode("utf-8") + except (TypeError, ValueError, RecursionError) as exc: + raise AcpProjectionError(f"{label} must be bounded JSON data") from exc + if len(encoded) > max_bytes: + raise AcpProjectionError(f"{label} exceeds the size limit") + return encoded + + +def _json_size(value: Any) -> int: + return len( + json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + ).encode("utf-8") + ) + + +def _messages_state_bytes( + messages: Mapping[str, list[_MessageAssembly]], +) -> int: + return sum( + len(message.message_id.encode("utf-8")) + len(message.text.encode("utf-8")) + for assemblies in messages.values() + for message in assemblies + ) + + +def _event_digest(kind: str, value: Mapping[str, Any]) -> str: + encoded = json.dumps( + {"kind": kind, "value": value}, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + def _canonical_event( *, session_id: str, @@ -447,6 +768,10 @@ def _canonical_event( f"payload.changes.{snake_name}", ] ) + if "permission" in payload: + private_fields.extend( + ["payload.permission", "payload.snapshot.permission"] + ) event_id = ( f"acp:{session_id}:{source_event_id}" if source_event_id is not None diff --git a/tests/test_acp_projection.py b/tests/test_acp_projection.py index d41d836..cf22a87 100644 --- a/tests/test_acp_projection.py +++ b/tests/test_acp_projection.py @@ -183,8 +183,9 @@ def test_permission_request_updates_tool_and_keeps_options() -> None: assert event["kind"] == "tool_call_update" assert event["payload"]["permission"]["required"] is True assert event["payload"]["permission"]["options"][1]["optionId"] == "no" - assert event["source_event_id"] == "42" - assert event["event_id"] == "acp:session-1:42" + assert event["source_event_id"] == "request:42" + assert event["event_id"] == "acp:session-1:request:42" + assert "payload.permission" in event["private_fields"] assert ( projector.normalize_permission_request( @@ -195,7 +196,18 @@ def test_permission_request_updates_tool_and_keeps_options() -> None: "params": { "sessionId": "session-1", "toolCall": {"toolCallId": "tool-9", "status": "pending"}, - "options": [], + "options": [ + { + "optionId": "yes", + "name": "Allow", + "kind": "allow_once", + }, + { + "optionId": "no", + "name": "Reject", + "kind": "reject_once", + }, + ], }, } ) @@ -315,3 +327,305 @@ def test_sessions_have_independent_ordering_and_defensive_snapshots() -> None: assert snapshot is not None snapshot["usage"]["used"] = 999 assert projector.session_snapshot("session-1")["usage"]["used"] == 1 + + +def test_interleaved_explicit_messages_and_implicit_v1_chunks_do_not_alias() -> None: + projector = AcpEventProjector() + + for message_id, text in (("a", "A1"), ("b", "B"), ("a", "A2")): + projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId=message_id, + content={"type": "text", "text": text}, + ) + ) + implicit = projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "stable-v1"}, + ) + ) + + assert implicit is not None + assert implicit["payload"]["message_id"] == "implicit-agent_message-1" + assert projector.project_turn_content("session-1")["assistant_stream_text"] == ( + "A1A2\n\nB\n\nstable-v1" + ) + + +def test_replay_identity_collision_is_rejected_and_sessions_are_isolated() -> None: + projector = AcpEventProjector() + first = _update( + "agent_message_chunk", + content={"type": "text", "text": "one"}, + ) + assert projector.normalize_session_update(first, source_event_id="event-7") + assert projector.normalize_session_update(first, source_event_id="event-7") is None + + with pytest.raises(AcpProjectionError, match="reused for different content"): + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "different"}, + ), + source_event_id="event-7", + ) + + other = { + "sessionId": "session-2", + "update": first["params"]["update"], + } + isolated = projector.normalize_session_update(other, source_event_id="event-7") + assert isolated is not None and isolated["sequence"] == 1 + + +def test_permission_request_ids_do_not_collide_with_notification_event_ids() -> None: + projector = AcpEventProjector() + notification = _update( + "tool_call", + toolCallId="tool-1", + status="pending", + _meta={"eventId": 42}, + ) + assert projector.normalize_session_update(notification) is not None + permission = projector.normalize_permission_request( + { + "jsonrpc": "2.0", + "id": 42, + "method": "session/request_permission", + "params": { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-1"}, + "options": [ + {"optionId": "yes", "name": "Allow", "kind": "allow_once"} + ], + }, + } + ) + assert permission is not None + assert permission["source_event_id"] == "request:42" + + +def test_namespaced_extension_metadata_is_private_and_adapter_neutral() -> None: + projector = AcpEventProjector(max_sessions=1) + # Unknown variants neither allocate a session nor consume ordering state. + assert projector.normalize_session_update(_update("vendor/future", value=1)) is None + event = projector.normalize_session_update( + { + "params": { + "sessionId": "session-2", + "_meta": {"vendor.example/params": {"trace": "abc"}}, + "update": { + "sessionUpdate": "agent_message_chunk", + "content": { + "type": "text", + "text": "safe", + "_meta": { + "vendor.example/content": {"revision": 1}, + "unscoped": "discard", + }, + }, + "_meta": { + "vendor.example/update": {"opaque": True}, + "adapterInternal": "discard", + }, + }, + }, + } + ) + assert event is not None + assert event["sequence"] == 1 + assert event["payload"]["extensions"] == { + "vendor.example/params": {"trace": "abc"}, + "vendor.example/update": {"opaque": True} + } + assert event["payload"]["content"]["_meta"] == { + "vendor.example/content": {"revision": 1} + } + assert "adapterInternal" not in repr(event["payload"]) + assert "unscoped" not in repr(event["payload"]) + + +def test_plan_is_a_validated_full_replacement() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update("plan", entries=[{"content": "old", "status": "pending"}]) + ) + replacement = projector.normalize_session_update( + _update("plan", entries=[{"content": "new", "status": "completed"}]) + ) + assert replacement is not None + assert replacement["payload"]["entries"] == [ + {"content": "new", "status": "completed"} + ] + with pytest.raises(AcpProjectionError, match="entries must be an array"): + projector.normalize_session_update(_update("plan", entries="bad")) + assert projector.session_snapshot("session-1")["plan"] == [ + {"content": "new", "status": "completed"} + ] + + +def test_completion_is_not_reopened_by_session_updates_and_requires_reset() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "final"}, + ) + ) + projector.mark_turn_complete("session-1") + projector.normalize_session_update(_update("usage_update", used=9, size=10)) + projector.normalize_session_update( + _update("session_info_update", title="still complete") + ) + assert projector.project_turn_content("session-1")["complete"] is True + assert projector.project_turn_content("session-1", complete=False)["complete"] is True + with pytest.raises(AcpProjectionError, match="reset_turn"): + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "late"}, + ) + ) + + projector.reset_turn("session-1") + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "next"}, + ) + ) + assert projector.project_turn_content("session-1")["assistant_stream_text"] == "next" + + +def test_bounded_state_fails_closed_and_drop_session_releases_capacity() -> None: + projector = AcpEventProjector( + max_sessions=1, + max_source_events_per_session=1, + max_messages_per_kind=1, + max_tool_calls_per_session=1, + max_state_fields=1, + max_plan_entries=1, + max_text_chars_per_message=3, + max_event_bytes=1024, + ) + assert projector.normalize_session_update( + _update("usage_update", used=1), source_event_id="one" + ) + with pytest.raises(AcpProjectionError, match="replay window"): + projector.normalize_session_update( + _update("usage_update", used=2), source_event_id="two" + ) + with pytest.raises(AcpProjectionError, match="session limit"): + projector.normalize_session_update( + { + "sessionId": "session-2", + "update": {"sessionUpdate": "usage_update", "used": 1}, + } + ) + assert projector.drop_session("session-1") is True + assert projector.drop_session("session-1") is False + assert projector.normalize_session_update( + { + "sessionId": "session-2", + "update": {"sessionUpdate": "usage_update", "used": 1}, + } + ) + + +def test_failed_or_oversized_input_does_not_allocate_or_mutate_session() -> None: + projector = AcpEventProjector(max_sessions=1, max_event_bytes=128) + with pytest.raises(AcpProjectionError, match="bounded JSON"): + projector.normalize_session_update( + _update("session_info_update", invalid={1, 2, 3}) + ) + assert projector.session_snapshot("session-1") is None + + with pytest.raises(AcpProjectionError, match="size limit"): + projector.normalize_session_update( + _update("session_info_update", value="x" * 200) + ) + assert projector.session_snapshot("session-1") is None + accepted = projector.normalize_session_update( + { + "sessionId": "session-2", + "update": {"sessionUpdate": "usage_update", "used": 1}, + } + ) + assert accepted is not None and accepted["sequence"] == 1 + + +def test_aggregate_retained_session_state_has_a_hard_budget() -> None: + projector = AcpEventProjector(max_session_state_bytes=12) + with pytest.raises(AcpProjectionError, match="retained session state"): + projector.normalize_session_update( + _update("session_info_update", title="far too large") + ) + assert projector.session_snapshot("session-1") is None + + +def test_total_retained_state_is_bounded_across_isolated_sessions() -> None: + projector = AcpEventProjector( + max_session_state_bytes=100, + max_total_state_bytes=18, + ) + assert projector.normalize_session_update( + { + "sessionId": "session-a", + "update": {"sessionUpdate": "session_info_update", "value": "1234"}, + } + ) + with pytest.raises(AcpProjectionError, match="total retained state"): + projector.normalize_session_update( + { + "sessionId": "session-b", + "update": { + "sessionUpdate": "session_info_update", + "value": "1234", + }, + } + ) + assert projector.session_snapshot("session-b") is None + + +def test_all_non_message_events_remain_unreachable_from_legacy_turns() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "do not leak thought"}, + ) + ) + projector.normalize_session_update( + _update( + "tool_call", + toolCallId="tool-secret", + rawInput={"secret": "do not leak raw input"}, + ) + ) + projector.normalize_session_update( + _update("plan", entries=[{"content": "do not leak plan"}]) + ) + projector.normalize_permission_request( + { + "jsonrpc": "2.0", + "id": "permission-secret", + "method": "session/request_permission", + "params": { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-secret"}, + "options": [ + {"optionId": "secret", "name": "Private", "kind": "reject_once"} + ], + }, + } + ) + legacy = projector.project_turn_content("session-1") + assert legacy == { + "user_text": "", + "assistant_stream_text": "", + "assistant_final_text": "", + "complete": False, + "has_open_turn": False, + } From 7a15fbdc23631669a9dc707c04c4e40f6bd66301 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:33:36 +0800 Subject: [PATCH 15/83] Harden ACP runtime lifecycle and finality --- src/tendwire/backends/acp_runtime.py | 428 +++++++++++++++++++-------- tests/test_acp_runtime.py | 268 ++++++++++++++++- 2 files changed, 562 insertions(+), 134 deletions(-) diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 98491a4..d22e12f 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -15,13 +15,18 @@ from dataclasses import dataclass from enum import Enum from pathlib import Path -from typing import Any +from typing import Any, Protocol from ..config import Config -from ..core.models import WorkerBinding -from .acp_client import AcpClient +from ..core.models import WorkerBinding, stable_fingerprint from .acp_ingestion import AcpSessionIngestor -from .acp_protocol import PermissionRequest, PromptResult, SessionResult +from .acp_protocol import ( + PermissionRequest, + PromptResult, + RequestId, + SessionResult, + SessionUpdate, +) class AcpRuntimeError(RuntimeError): @@ -81,6 +86,66 @@ class AcpRuntimeStatus: IngestorFactory = Callable[..., AcpSessionIngestor] +class AcpRuntimeClient(Protocol): + """Adapter-neutral client surface required by :class:`AcpRuntime`.""" + + def initialize( + self, + *, + client_capabilities: Mapping[str, Any] | None = None, + ) -> object: ... + + def new_session( + self, + cwd: Path, + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[Path] = (), + ) -> SessionResult: ... + + def load_session( + self, + session_id: str, + cwd: Path, + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[Path] = (), + ) -> SessionResult: ... + + def resume_session( + self, + session_id: str, + cwd: Path, + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[Path] = (), + ) -> SessionResult: ... + + def prompt( + self, + session_id: str, + prompt: str | Sequence[Mapping[str, Any]], + *, + timeout: float | None = None, + ) -> PromptResult: ... + + def cancel(self, session_id: str) -> None: ... + + def next_update(self, *, timeout: float) -> SessionUpdate: ... + + def next_permission_request(self, *, timeout: float) -> PermissionRequest: ... + + def respond_permission( + self, + request_id: RequestId, + *, + option_id: str | None = None, + cancelled: bool = False, + ) -> None: ... + + def close(self) -> None: ... + + class AcpRuntime: """Run and durably ingest exactly one ACP session for one worker binding. @@ -91,7 +156,7 @@ class AcpRuntime: def __init__( self, - client: AcpClient, + client: AcpRuntimeClient, *, config: Config, binding: WorkerBinding, @@ -120,6 +185,13 @@ def __init__( raise ValueError("ACP runtime binding host does not match configuration") if not binding.private_fingerprint: raise ValueError("ACP runtime requires an authenticated private binding") + if binding.turn_target_kind != "acp_session_id": + raise ValueError("ACP runtime requires an ACP session worker binding") + if ( + mode is not SessionOpenMode.NEW + and binding.turn_target_value != session_id + ): + raise ValueError("ACP runtime session does not match the worker binding") resolved_cwd = Path(cwd) if not resolved_cwd.is_absolute(): raise ValueError("ACP runtime cwd must be absolute") @@ -147,12 +219,16 @@ def __init__( self._ingestor: AcpSessionIngestor | None = None self._failure: BaseException | None = None self._state_lock = threading.RLock() + self._lifecycle_lock = threading.Lock() self._ingest_lock = threading.Lock() self._prompt_lock = threading.Lock() self._idle_condition = threading.Condition(self._state_lock) self._stop_event = threading.Event() self._threads: tuple[threading.Thread, ...] = () self._update_idle_epoch = 0 + self._permission_idle_epoch = 0 + self._close_thread: threading.Thread | None = None + self._close_failures: list[BaseException] = [] self._updates_ingested = 0 self._permissions_ingested = 0 @@ -177,43 +253,61 @@ def __exit__(self, exc_type: object, exc: object, tb: object) -> None: def start(self) -> "AcpRuntime": """Initialize capabilities, open one session, and start consumers.""" - with self._state_lock: - if self._state is RuntimeState.RUNNING: - return self - if self._state is not RuntimeState.NEW: - raise AcpRuntimeStateError( - f"cannot start ACP runtime in state {self._state.value}" - ) - self._state = RuntimeState.STARTING - try: - self._client.initialize(client_capabilities=self._client_capabilities) - session = self._open_session() - if not isinstance(session, SessionResult) or not session.session_id: - raise AcpRuntimeProtocolError( - "ACP session setup returned an invalid response" - ) - self._session_id = session.session_id - self._ingestor = self._make_ingestor(session.session_id) - threads = ( - threading.Thread( - target=self._consume_updates, - name="tendwire-acp-updates", - daemon=True, - ), - threading.Thread( - target=self._consume_permissions, - name="tendwire-acp-permissions", - daemon=True, - ), - ) - self._threads = threads + with self._lifecycle_lock: with self._state_lock: - self._state = RuntimeState.RUNNING - for thread in threads: - thread.start() - except BaseException as exc: - self._record_failure(exc) - raise + if self._state is RuntimeState.RUNNING: + return self + if self._state is not RuntimeState.NEW: + raise AcpRuntimeStateError( + f"cannot start ACP runtime in state {self._state.value}" + ) + self._state = RuntimeState.STARTING + try: + self._client.initialize(client_capabilities=self._client_capabilities) + session = self._open_session() + if not isinstance(session, SessionResult) or not session.session_id: + raise AcpRuntimeProtocolError( + "ACP session setup returned an invalid response" + ) + if session.session_id != self._binding.turn_target_value: + raise AcpRuntimeProtocolError( + "ACP session setup did not return the bound session" + ) + if ( + self._requested_session_id is not None + and session.session_id != self._requested_session_id + ): + raise AcpRuntimeProtocolError( + "ACP session setup returned an unexpected session" + ) + self._session_id = session.session_id + self._ingestor = self._make_ingestor(session.session_id) + threads = ( + threading.Thread( + target=self._consume_updates, + name="tendwire-acp-updates", + daemon=True, + ), + threading.Thread( + target=self._consume_permissions, + name="tendwire-acp-permissions", + daemon=True, + ), + ) + self._threads = threads + for thread in threads: + thread.start() + with self._state_lock: + if self._failure is not None: + raise self._failure + self._state = RuntimeState.RUNNING + except BaseException as exc: + self._record_failure(exc) + # ``__enter__`` is never completed when start fails, so no + # caller cleanup can be assumed. Bound shutdown prevents an + # initialized adapter or a partially started consumer leaking. + self._shutdown_transport(self._stop_timeout) + raise return self def prompt( @@ -243,9 +337,18 @@ def prompt( raise try: result = self._client.prompt(session_id, prompt, timeout=timeout) - except BaseException: + except BaseException as exc: with self._state_lock: self._prompts_failed += 1 + # Once a turn has been opened locally, a timed-out or failed + # prompt cannot be retried safely: late updates would otherwise + # be attributed to the next turn. Best-effort cancellation + # contains the remote work and the runtime becomes terminal. + try: + self._cancel_session(session_id) + except BaseException: + pass + self._record_failure(exc) raise if not isinstance(result, PromptResult): error = AcpRuntimeProtocolError( @@ -278,9 +381,7 @@ def cancel(self) -> None: self.raise_if_failed() session_id, _ = self._running_components() - self._client.cancel(session_id) - with self._state_lock: - self._cancellation_requests += 1 + self._cancel_session(session_id) def status(self) -> AcpRuntimeStatus: """Return redacted health and counters safe for a public status API.""" @@ -319,11 +420,13 @@ def join(self, timeout: float | None = None) -> bool: raise ValueError("join timeout must be positive") deadline = time.monotonic() + wait_limit for thread in self._threads: - if thread is threading.current_thread(): + if thread is threading.current_thread() or thread.ident is None: continue thread.join(timeout=max(0.0, deadline - time.monotonic())) return all( - thread is threading.current_thread() or not thread.is_alive() + thread is threading.current_thread() + or thread.ident is None + or not thread.is_alive() for thread in self._threads ) @@ -333,52 +436,76 @@ def stop(self, *, timeout: float | None = None) -> None: wait_limit = self._stop_timeout if timeout is None else float(timeout) if wait_limit <= 0: raise ValueError("stop timeout must be positive") - with self._state_lock: - if self._state is RuntimeState.STOPPED: - return - if self._state is RuntimeState.NEW: - self._state = RuntimeState.STOPPED - return - if self._state is not RuntimeState.FAILED: - self._state = RuntimeState.STOPPING - self._stop_event.set() - close_failures: list[BaseException] = [] - - def close_client() -> None: - try: - self._client.close() - except BaseException as exc: - close_failures.append(exc) - - closer = threading.Thread( - target=close_client, - name="tendwire-acp-close", - daemon=True, - ) deadline = time.monotonic() + wait_limit - closer.start() - closer.join(timeout=max(0.0, deadline - time.monotonic())) - remaining = max(0.0, deadline - time.monotonic()) - joined = ( - self.join(timeout=remaining) - if remaining > 0 - else all(not thread.is_alive() for thread in self._threads) - ) - if closer.is_alive() or not joined: + if not self._lifecycle_lock.acquire(timeout=wait_limit): error = AcpRuntimeStopTimeout( - "ACP runtime did not stop within the configured deadline" + "ACP runtime lifecycle did not stop within the configured deadline" ) self._record_failure(error) raise error - if close_failures: - self._record_failure(close_failures[0]) - raise close_failures[0] - with self._state_lock: - if self._failure is None: - self._state = RuntimeState.STOPPED - failure = self._failure - if failure is not None: - raise failure + try: + with self._idle_condition: + if self._state is RuntimeState.STOPPED: + return + if self._state is RuntimeState.NEW: + self._state = RuntimeState.STOPPED + return + if self._state is not RuntimeState.FAILED: + self._state = RuntimeState.STOPPING + self._stop_event.set() + self._idle_condition.notify_all() + + remaining = max(0.0, deadline - time.monotonic()) + if remaining <= 0 or not self._shutdown_transport(remaining): + error = AcpRuntimeStopTimeout( + "ACP runtime did not stop within the configured deadline" + ) + self._record_failure(error) + raise error + + with self._state_lock: + failure = self._failure + if self._close_failures and failure is None: + failure = self._close_failures[0] + self._record_failure(failure) + with self._state_lock: + if failure is None: + self._state = RuntimeState.STOPPED + if failure is not None: + raise failure + finally: + self._lifecycle_lock.release() + + def _shutdown_transport(self, timeout: float) -> bool: + """Start adapter close at most once and join all supervised work.""" + + if self._close_thread is None: + + def close_client() -> None: + try: + self._client.close() + except BaseException as exc: + self._close_failures.append(exc) + + self._close_thread = threading.Thread( + target=close_client, + name="tendwire-acp-close", + daemon=True, + ) + self._close_thread.start() + + deadline = time.monotonic() + timeout + self._close_thread.join(timeout=max(0.0, deadline - time.monotonic())) + remaining = max(0.0, deadline - time.monotonic()) + consumers_joined = ( + self.join(timeout=remaining) + if remaining > 0 + else all( + thread.ident is None or not thread.is_alive() + for thread in self._threads + ) + ) + return not self._close_thread.is_alive() and consumers_joined def _open_session(self) -> SessionResult: options = { @@ -442,61 +569,92 @@ def _consume_permissions(self) -> None: timeout=self._poll_timeout ) except TimeoutError: + with self._idle_condition: + self._permission_idle_epoch += 1 + self._idle_condition.notify_all() if self._stop_event.is_set(): return continue - if request.session_id != self._session_id: - raise AcpRuntimeProtocolError( - "ACP permission belongs to a different session" - ) - ingestor = self._require_ingestor() - with self._ingest_lock: - ingestor.ingest_permission_request( - request.raw, - source_event_id=f"permission:{request.request_id}", - ) + self._handle_permission(request) + except BaseException as exc: + if not self._stop_event.is_set(): + self._record_failure(exc) + + def _handle_permission(self, request: PermissionRequest) -> None: + """Journal then resolve one permission, failing closed before response.""" + + response_attempted = False + try: + if request.session_id != self._session_id: + raise AcpRuntimeProtocolError( + "ACP permission belongs to a different session" + ) + ingestor = self._require_ingestor() + with self._ingest_lock: + ingestor.ingest_permission_request( + request.raw, + source_event_id=_permission_source_event_id(request.request_id), + ) + with self._state_lock: + self._permissions_ingested += 1 + + selected: str | None = None + callback_failure: BaseException | None = None + if self._permission_callback is not None: + try: + candidate = self._permission_callback(request) + if candidate is not None and candidate in { + option.option_id for option in request.options + }: + selected = candidate + elif candidate is not None: + with self._state_lock: + self._invalid_permission_selections += 1 + except BaseException as exc: + callback_failure = exc + response_attempted = True + if selected is None: + self._client.respond_permission( + request.request_id, + cancelled=True, + ) with self._state_lock: - self._permissions_ingested += 1 - - selected: str | None = None - callback_failure: BaseException | None = None - if self._permission_callback is not None: - try: - candidate = self._permission_callback(request) - if candidate is not None and candidate in { - option.option_id for option in request.options - }: - selected = candidate - elif candidate is not None: - with self._state_lock: - self._invalid_permission_selections += 1 - except BaseException as exc: - callback_failure = exc - if selected is None: + self._permissions_cancelled += 1 + else: + self._client.respond_permission( + request.request_id, + option_id=selected, + ) + with self._state_lock: + self._permissions_selected += 1 + if callback_failure is not None: + raise callback_failure + except BaseException: + if not response_attempted: + # No response bytes have been attempted yet, so cancellation + # is safe. Never retry after respond_permission itself fails: + # a partial JSON-RPC frame may already have reached the agent. + try: self._client.respond_permission( request.request_id, cancelled=True, ) - with self._state_lock: - self._permissions_cancelled += 1 + except BaseException: + pass else: - self._client.respond_permission( - request.request_id, - option_id=selected, - ) with self._state_lock: - self._permissions_selected += 1 - if callback_failure is not None: - raise callback_failure - except BaseException as exc: - if not self._stop_event.is_set(): - self._record_failure(exc) + self._permissions_cancelled += 1 + raise def _wait_for_post_response_idle(self, timeout: float) -> None: deadline = time.monotonic() + timeout with self._idle_condition: - epoch = self._update_idle_epoch - while self._update_idle_epoch <= epoch: + update_epoch = self._update_idle_epoch + permission_epoch = self._permission_idle_epoch + while ( + self._update_idle_epoch <= update_epoch + or self._permission_idle_epoch <= permission_epoch + ): if self._failure is not None: raise self._failure if self._state is not RuntimeState.RUNNING: @@ -528,6 +686,11 @@ def _require_ingestor(self) -> AcpSessionIngestor: raise AcpRuntimeStateError("ACP runtime has no ingestor") return ingestor + def _cancel_session(self, session_id: str) -> None: + self._client.cancel(session_id) + with self._state_lock: + self._cancellation_requests += 1 + def _record_failure(self, failure: BaseException) -> None: with self._idle_condition: if self._failure is None: @@ -537,8 +700,15 @@ def _record_failure(self, failure: BaseException) -> None: self._idle_condition.notify_all() +def _permission_source_event_id(request_id: RequestId) -> str: + """Return a bounded opaque ID while preserving JSON-RPC ID types.""" + + return f"permission:{stable_fingerprint({'request_id': request_id})}" + + __all__ = [ "AcpRuntime", + "AcpRuntimeClient", "AcpRuntimeError", "AcpRuntimeProtocolError", "AcpRuntimeStateError", diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index b40432a..291d13a 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -41,15 +41,22 @@ def __init__(self) -> None: self.permission_responses: list[tuple[object, str | None, bool]] = [] self.prompt_result: object = PromptResult(StopReason.END_TURN, {}) self.prompt_failure: BaseException | None = None + self.initialize_failure: BaseException | None = None + self.new_session_result: SessionResult | None = None self.closed = False + self.close_calls = 0 def initialize(self, **kwargs: Any) -> object: self.calls.append(("initialize", (), kwargs)) + if self.initialize_failure is not None: + raise self.initialize_failure return object() def new_session(self, cwd: Path, **kwargs: Any) -> SessionResult: self.calls.append(("new", (cwd,), kwargs)) - return SessionResult("session-private", None, (), {}) + return self.new_session_result or SessionResult( + "session-private", None, (), {} + ) def load_session( self, session_id: str, cwd: Path, **kwargs: Any @@ -102,6 +109,7 @@ def respond_permission( self.permission_responses.append((request_id, option_id, cancelled)) def close(self) -> None: + self.close_calls += 1 self.closed = True self.updates.put(_END) self.permissions.put(_END) @@ -115,6 +123,7 @@ def __init__(self, session_id: str = "session-private") -> None: self.permissions: list[tuple[object, str | None]] = [] self.completions = 0 self.update_failure: BaseException | None = None + self.permission_failure: BaseException | None = None def start_turn(self, *, producer_turn_id: str | None = None) -> str: self.started.append(producer_turn_id) @@ -128,13 +137,15 @@ def ingest_update(self, raw: object) -> None: def ingest_permission_request( self, raw: object, *, source_event_id: str | None = None ) -> None: + if self.permission_failure is not None: + raise self.permission_failure self.permissions.append((raw, source_event_id)) def mark_prompt_complete(self) -> None: self.completions += 1 -def binding() -> WorkerBinding: +def binding(session_id: str = "session-private") -> WorkerBinding: return WorkerBinding( host_id="host-a", worker_id="worker-public", @@ -143,7 +154,7 @@ def binding() -> WorkerBinding: target_kind="pane_id", target_value="pane-private-secret", turn_target_kind="acp_session_id", - turn_target_value="session-private", + turn_target_value=session_id, private_fingerprint="binding-private-secret", ) @@ -270,7 +281,7 @@ def test_load_and_resume_use_requested_session( service = AcpRuntime( client, # type: ignore[arg-type] config=Config(host_id="host-a", db_path=tmp_path / "events.db"), - binding=binding(), + binding=binding("existing-private"), cwd=tmp_path, session_mode=mode, session_id="existing-private", @@ -285,6 +296,35 @@ def test_load_and_resume_use_requested_session( service.stop() +def test_start_rejects_unbound_session_and_closes_adapter(tmp_path: Path) -> None: + client = FakeClient() + client.new_session_result = SessionResult("other-private", None, (), {}) + service = runtime(tmp_path, client) + + with pytest.raises(AcpRuntimeProtocolError, match="bound session"): + service.start() + + assert client.closed + assert client.close_calls == 1 + assert service.join(timeout=0.1) + assert service.status().state is RuntimeState.FAILED + + +def test_initialize_failure_is_cleaned_up_without_caller_stop(tmp_path: Path) -> None: + client = FakeClient() + failure = OSError("initialize failed") + client.initialize_failure = failure + service = runtime(tmp_path, client) + + with pytest.raises(OSError) as raised: + service.start() + + assert raised.value is failure + assert client.closed + assert client.close_calls == 1 + assert service.status().failure_type == "OSError" + + def test_background_consumers_ingest_losslessly_and_permissions_fail_closed( tmp_path: Path, ) -> None: @@ -298,7 +338,10 @@ def test_background_consumers_ingest_losslessly_and_permissions_fail_closed( wait_until(lambda: service.status().updates_ingested == 1) assert len(ingestor.updates) == 1 - assert ingestor.permissions[0][1] == "permission:7" + permission_event_id = ingestor.permissions[0][1] + assert permission_event_id is not None + assert permission_event_id.startswith("permission:") + assert permission_event_id != "permission:7" assert client.permission_responses == [(7, None, True)] assert service.status().permissions_cancelled == 1 finally: @@ -349,6 +392,46 @@ def fail(_request: PermissionRequest) -> str: assert raised.value is callback_failure +def test_permission_ingestion_failure_cancels_before_runtime_fails( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + failure = OSError("journal unavailable") + ingestor.permission_failure = failure + service = runtime(tmp_path, client, ingestor).start() + client.permissions.put(permission()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + assert client.permission_responses == [(7, None, True)] + assert service.status().permissions_cancelled == 1 + assert service.status().permissions_ingested == 0 + with pytest.raises(OSError) as raised: + service.stop() + assert raised.value is failure + + +def test_permission_source_identity_distinguishes_jsonrpc_id_types( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + client.permissions.put(permission(1)) + client.permissions.put(permission("1")) + wait_until(lambda: service.status().permissions_ingested == 2) + + source_ids = [source_id for _raw, source_id in ingestor.permissions] + assert len(set(source_ids)) == 2 + assert all( + source_id is not None and source_id.startswith("permission:") + for source_id in source_ids + ) + finally: + service.stop() + + def test_prompt_finalizes_only_after_valid_response_and_update_drain( tmp_path: Path, ) -> None: @@ -371,6 +454,118 @@ def test_prompt_finalizes_only_after_valid_response_and_update_drain( service.stop() +def test_prompt_finality_waits_for_permission_resolution(tmp_path: Path) -> None: + client = FakeClient() + ingestor = FakeIngestor() + callback_entered = threading.Event() + release_callback = threading.Event() + + def decide(_request: PermissionRequest) -> str: + callback_entered.set() + assert release_callback.wait(timeout=1) + return "allow-once" + + service = runtime( + tmp_path, + client, + ingestor, + permission_callback=decide, + ).start() + result: list[PromptResult] = [] + failure: list[BaseException] = [] + + def run_prompt() -> None: + try: + result.append(service.prompt("question", drain_timeout=0.5)) + except BaseException as exc: + failure.append(exc) + + try: + client.permissions.put(permission()) + assert callback_entered.wait(timeout=1) + prompt_thread = threading.Thread(target=run_prompt) + prompt_thread.start() + time.sleep(0.04) + assert prompt_thread.is_alive() + assert ingestor.completions == 0 + + release_callback.set() + prompt_thread.join(timeout=1) + assert not prompt_thread.is_alive() + assert failure == [] + assert len(result) == 1 + assert ingestor.completions == 1 + assert client.permission_responses == [(7, "allow-once", False)] + finally: + release_callback.set() + service.stop() + + +def test_cross_kind_ingestion_cannot_overtake_an_active_update( + tmp_path: Path, +) -> None: + client = FakeClient() + update_entered = threading.Event() + release_update = threading.Event() + order: list[str] = [] + + class OrderedIngestor(FakeIngestor): + def ingest_update(self, raw: object) -> None: + order.append("update-start") + update_entered.set() + assert release_update.wait(timeout=1) + super().ingest_update(raw) + order.append("update-end") + + def ingest_permission_request( + self, raw: object, *, source_event_id: str | None = None + ) -> None: + order.append("permission") + super().ingest_permission_request( + raw, + source_event_id=source_event_id, + ) + + service = runtime(tmp_path, client, OrderedIngestor()).start() + try: + client.updates.put(update()) + assert update_entered.wait(timeout=1) + client.permissions.put(permission()) + time.sleep(0.04) + assert order == ["update-start"] + + release_update.set() + wait_until(lambda: service.status().permissions_ingested == 1) + assert order == ["update-start", "update-end", "permission"] + finally: + release_update.set() + service.stop() + + +def test_prompt_transport_failure_cancels_and_makes_runtime_terminal( + tmp_path: Path, +) -> None: + client = FakeClient() + failure = AcpRequestTimeoutError("prompt timed out") + client.prompt_failure = failure + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRequestTimeoutError) as raised: + service.prompt("question") + + assert raised.value is failure + assert ingestor.started == [None] + assert ingestor.completions == 0 + assert ("cancel", ("session-private",), {}) in client.calls + assert service.status().cancellation_requests == 1 + assert service.status().state is RuntimeState.FAILED + with pytest.raises(AcpRequestTimeoutError): + service.prompt("unsafe retry") + with pytest.raises(AcpRequestTimeoutError): + service.stop() + + def test_invalid_prompt_response_never_marks_complete_and_propagates( tmp_path: Path, ) -> None: @@ -425,6 +620,29 @@ def test_cancel_targets_bound_session_and_stop_joins_consumers(tmp_path: Path) - assert service.status().state is RuntimeState.STOPPED +def test_concurrent_stop_closes_adapter_exactly_once(tmp_path: Path) -> None: + client = FakeClient() + service = runtime(tmp_path, client).start() + failures: list[BaseException] = [] + + def stop() -> None: + try: + service.stop() + except BaseException as exc: + failures.append(exc) + + callers = [threading.Thread(target=stop) for _ in range(2)] + for caller in callers: + caller.start() + for caller in callers: + caller.join(timeout=1) + + assert failures == [] + assert all(not caller.is_alive() for caller in callers) + assert client.close_calls == 1 + assert service.status().state is RuntimeState.STOPPED + + def test_stop_deadline_is_bounded_even_when_client_close_hangs(tmp_path: Path) -> None: client = FakeClient() release_close = threading.Event() @@ -441,3 +659,43 @@ def hanging_close() -> None: assert time.monotonic() - started < 0.25 finally: release_close.set() + + +def test_concurrent_stop_wait_for_lifecycle_is_also_deadline_bounded( + tmp_path: Path, +) -> None: + client = FakeClient() + close_entered = threading.Event() + release_close = threading.Event() + original_close = client.close + + def hanging_close() -> None: + close_entered.set() + release_close.wait(timeout=1) + original_close() + + client.close = hanging_close # type: ignore[method-assign] + service = runtime(tmp_path, client).start() + first_failures: list[BaseException] = [] + + def first_stop() -> None: + try: + service.stop(timeout=0.2) + except BaseException as exc: + first_failures.append(exc) + + caller = threading.Thread(target=first_stop) + caller.start() + assert close_entered.wait(timeout=1) + started = time.monotonic() + try: + with pytest.raises(AcpRuntimeStopTimeout, match="lifecycle"): + service.stop(timeout=0.03) + assert time.monotonic() - started < 0.15 + finally: + release_close.set() + caller.join(timeout=1) + + assert len(first_failures) == 1 + assert isinstance(first_failures[0], AcpRuntimeStopTimeout) + assert client.close_calls == 1 From 23ebe2bf123deed4e59c239ac2b3abdaa0eee1a8 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:34:33 +0800 Subject: [PATCH 16/83] Add black-box ACP adapter probe --- docs/acp-adapter-probe.md | 65 +++++ src/tendwire/backends/acp_probe.py | 365 +++++++++++++++++++++++++++++ tests/test_acp_probe.py | 159 +++++++++++++ 3 files changed, 589 insertions(+) create mode 100644 docs/acp-adapter-probe.md create mode 100644 src/tendwire/backends/acp_probe.py create mode 100644 tests/test_acp_probe.py diff --git a/docs/acp-adapter-probe.md b/docs/acp-adapter-probe.md new file mode 100644 index 0000000..f662b72 --- /dev/null +++ b/docs/acp-adapter-probe.md @@ -0,0 +1,65 @@ +# Black-box ACP adapter compatibility probe + +Tendwire treats an ACP adapter as a separately installed executable. The +adapter is not vendored, imported, rebased, or coupled to a repository layout. +Run the compatibility probe after installing or upgrading any adapter and +before promoting that executable into the Tendwire runtime: + +```console +python -m tendwire.backends.acp_probe -- /absolute/path/to/adapter adapter-arg +``` + +The command starts the supplied argv directly without a shell, negotiates ACP +v1 capabilities on a fresh process, closes the process, prints one bounded JSON +object, and exits zero only when negotiation and shutdown succeed. Use +`--timeout`, `--close-timeout`, and an absolute `--cwd` before the `--` marker +when needed. Probe timeouts have hard upper bounds. + +The output is intentionally narrow. It contains fixed boolean capabilities, +bounded counts for authentication methods and unknown capability extensions, a +process-reaped flag, and a fixed failure category. It never contains: + +- adapter argv or executable paths; +- working directories or environment values; +- stderr, exception text, or raw JSON-RPC payloads; +- agent-provided names, versions, extension names, or extension values; +- session IDs, messages, thoughts, tool data, plans, or authentication values. + +Example successful shape: + +```json +{ + "authentication": {"method_count": 0, "method_count_capped": false}, + "capabilities": { + "additional_directories": true, + "auth_logout": false, + "mcp_http": false, + "mcp_sse": false, + "prompt_audio": false, + "prompt_embedded_context": false, + "prompt_image": false, + "session_cancel": true, + "session_close": true, + "session_delete": true, + "session_list": true, + "session_load": true, + "session_new": true, + "session_prompt": true, + "session_resume": true, + "session_update": true + }, + "compatible": true, + "extensions": {"capability_count": 0, "capability_count_capped": false}, + "failure": null, + "process_reaped": true, + "protocol_version": 1, + "schema_version": 1 +} +``` + +An incompatible result exits with status 1 and reports only one of these stable +categories: `invalid_configuration`, `launch_failed`, `timeout`, +`protocol_version`, `protocol_error`, `transport_error`, `shutdown_failed`, or +`internal_error`. Keep the previously proven executable available for rollback; +the probe validates initialization and capability negotiation, not account +authentication or a stateful agent session. diff --git a/src/tendwire/backends/acp_probe.py b/src/tendwire/backends/acp_probe.py new file mode 100644 index 0000000..9569a33 --- /dev/null +++ b/src/tendwire/backends/acp_probe.py @@ -0,0 +1,365 @@ +"""Bounded black-box compatibility probe for external ACP v1 adapters. + +The probe deliberately knows only the stable ACP wire protocol. It launches a +caller-supplied executable without a shell, negotiates a fresh connection, and +returns a fixed public-safe capability summary. Adapter output, argv, paths, +environment values, stderr, exception text, and extension names never enter the +report. + +Operator entry point:: + + python -m tendwire.backends.acp_probe -- adapter-command arg ... +""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import sys +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from enum import Enum +from types import MappingProxyType +from typing import Any + +from .acp_client import ( + AcpClient, + AcpProtocolVersionError, + AcpRequestTimeoutError, + AcpTransportError, +) +from .acp_protocol import ( + ACP_PROTOCOL_VERSION, + AcpEnvelopeError, + AcpFramingError, + AcpRemoteError, + InitializeResult, +) + +PROBE_SCHEMA_VERSION = 1 +DEFAULT_PROBE_TIMEOUT_SECONDS = 5.0 +DEFAULT_PROBE_CLOSE_TIMEOUT_SECONDS = 1.0 +MAX_PROBE_TIMEOUT_SECONDS = 30.0 +MAX_PROBE_CLOSE_TIMEOUT_SECONDS = 5.0 +PROBE_MAX_FRAME_BYTES = 1024 * 1024 +PROBE_MAX_PENDING_EVENTS = 16 +PROBE_STDERR_LIMIT_BYTES = 4096 +_MAX_REPORTED_COUNT = 1000 + +_CAPABILITY_KEYS = ( + "session_new", + "session_prompt", + "session_cancel", + "session_update", + "session_load", + "session_list", + "session_delete", + "session_resume", + "session_close", + "additional_directories", + "prompt_image", + "prompt_audio", + "prompt_embedded_context", + "mcp_http", + "mcp_sse", + "auth_logout", +) +_KNOWN_TOP_LEVEL_CAPABILITIES = { + "loadSession", + "promptCapabilities", + "mcpCapabilities", + "sessionCapabilities", + "auth", + "_meta", +} +_KNOWN_NESTED_CAPABILITIES = { + "promptCapabilities": {"image", "audio", "embeddedContext", "_meta"}, + "mcpCapabilities": {"http", "sse", "_meta"}, + "sessionCapabilities": { + "list", + "delete", + "additionalDirectories", + "resume", + "close", + "_meta", + }, + "auth": {"logout", "_meta"}, +} + + +class ProbeFailure(str, Enum): + """Fixed failure categories safe to expose to operators and automation.""" + + INVALID_CONFIGURATION = "invalid_configuration" + LAUNCH_FAILED = "launch_failed" + TIMEOUT = "timeout" + PROTOCOL_VERSION = "protocol_version" + PROTOCOL = "protocol_error" + TRANSPORT = "transport_error" + SHUTDOWN = "shutdown_failed" + INTERNAL = "internal_error" + + +@dataclass(frozen=True, slots=True) +class AcpAdapterProbeReport: + """Fixed-shape, bounded result that contains no adapter-controlled text.""" + + compatible: bool + protocol_version: int | None + capabilities: Mapping[str, bool] + authentication_method_count: int + authentication_method_count_capped: bool + extension_capability_count: int + extension_capability_count_capped: bool + process_reaped: bool + failure: ProbeFailure | None + schema_version: int = PROBE_SCHEMA_VERSION + + def to_payload(self) -> dict[str, Any]: + return { + "schema_version": self.schema_version, + "compatible": self.compatible, + "protocol_version": self.protocol_version, + "capabilities": { + key: bool(self.capabilities.get(key, False)) + for key in _CAPABILITY_KEYS + }, + "authentication": { + "method_count": self.authentication_method_count, + "method_count_capped": self.authentication_method_count_capped, + }, + "extensions": { + "capability_count": self.extension_capability_count, + "capability_count_capped": self.extension_capability_count_capped, + }, + "process_reaped": self.process_reaped, + "failure": self.failure.value if self.failure is not None else None, + } + + +def probe_adapter( + argv: Sequence[str | os.PathLike[str]], + *, + cwd: str | os.PathLike[str] | None = None, + env: Mapping[str, str] | None = None, + timeout_seconds: float = DEFAULT_PROBE_TIMEOUT_SECONDS, + close_timeout_seconds: float = DEFAULT_PROBE_CLOSE_TIMEOUT_SECONDS, +) -> AcpAdapterProbeReport: + """Negotiate ACP v1 with one separately installed adapter executable. + + A new subprocess and initialization exchange are used for every call. No + session is created and no adapter source package is imported. All expected + failures become a fixed-category incompatible report; no raw diagnostic text + crosses this black-box boundary. + """ + + try: + timeout = _bounded_timeout( + timeout_seconds, + "timeout_seconds", + maximum=MAX_PROBE_TIMEOUT_SECONDS, + ) + close_timeout = _bounded_timeout( + close_timeout_seconds, + "close_timeout_seconds", + maximum=MAX_PROBE_CLOSE_TIMEOUT_SECONDS, + ) + client = AcpClient( + argv, + cwd=cwd, + env=env, + request_timeout=timeout, + close_timeout=close_timeout, + max_frame_bytes=PROBE_MAX_FRAME_BYTES, + max_pending_events=PROBE_MAX_PENDING_EVENTS, + stderr_limit_bytes=PROBE_STDERR_LIMIT_BYTES, + ) + except Exception: + return _failure_report(ProbeFailure.INVALID_CONFIGURATION) + + initialized: InitializeResult | None = None + failure: ProbeFailure | None = None + started = False + try: + client.start() + started = True + initialized = client.initialize(timeout=timeout) + except Exception as exc: # exception text is intentionally never returned + failure = _failure_category(exc, started=started) + + process_reaped = False + try: + client.close() + except Exception: + if failure is None: + failure = ProbeFailure.SHUTDOWN + finally: + process = client.process + process_reaped = process is None or process.poll() is not None + if not process_reaped and failure is None: + failure = ProbeFailure.SHUTDOWN + + if initialized is None: + return _failure_report(failure or ProbeFailure.INTERNAL, process_reaped) + + capabilities = _capability_summary(initialized) + auth_count, auth_capped = _bounded_count(len(initialized.auth_methods)) + extension_count, extension_capped = _bounded_count( + _extension_capability_count(initialized.capabilities.raw) + ) + compatible = ( + failure is None + and process_reaped + and initialized.protocol_version == ACP_PROTOCOL_VERSION + ) + return AcpAdapterProbeReport( + compatible=compatible, + protocol_version=initialized.protocol_version, + capabilities=MappingProxyType(capabilities), + authentication_method_count=auth_count, + authentication_method_count_capped=auth_capped, + extension_capability_count=extension_count, + extension_capability_count_capped=extension_capped, + process_reaped=process_reaped, + failure=failure, + ) + + +def _capability_summary(initialized: InitializeResult) -> dict[str, bool]: + raw = initialized.capabilities.raw + prompt = _mapping(raw.get("promptCapabilities")) + mcp = _mapping(raw.get("mcpCapabilities")) + session = _mapping(raw.get("sessionCapabilities")) + auth = _mapping(raw.get("auth")) + return { + # ACP v1 baseline methods are guaranteed by a successful negotiation. + "session_new": True, + "session_prompt": True, + "session_cancel": True, + "session_update": True, + "session_load": initialized.capabilities.load_session, + "session_list": initialized.capabilities.session_list, + "session_delete": initialized.capabilities.session_delete, + "session_resume": initialized.capabilities.session_resume, + "session_close": initialized.capabilities.session_close, + "additional_directories": initialized.capabilities.additional_directories, + "prompt_image": prompt.get("image") is True, + "prompt_audio": prompt.get("audio") is True, + "prompt_embedded_context": prompt.get("embeddedContext") is True, + "mcp_http": mcp.get("http") is True, + "mcp_sse": mcp.get("sse") is True, + "auth_logout": isinstance(auth.get("logout"), Mapping), + } + + +def _extension_capability_count(raw: Mapping[str, Any]) -> int: + count = sum(key not in _KNOWN_TOP_LEVEL_CAPABILITIES for key in raw) + for section, known_keys in _KNOWN_NESTED_CAPABILITIES.items(): + nested = raw.get(section) + if isinstance(nested, Mapping): + count += sum(key not in known_keys for key in nested) + return count + + +def _mapping(value: Any) -> Mapping[str, Any]: + return value if isinstance(value, Mapping) else {} + + +def _bounded_count(value: int) -> tuple[int, bool]: + return min(value, _MAX_REPORTED_COUNT), value > _MAX_REPORTED_COUNT + + +def _failure_report( + failure: ProbeFailure, + process_reaped: bool = True, +) -> AcpAdapterProbeReport: + return AcpAdapterProbeReport( + compatible=False, + protocol_version=None, + capabilities=MappingProxyType({key: False for key in _CAPABILITY_KEYS}), + authentication_method_count=0, + authentication_method_count_capped=False, + extension_capability_count=0, + extension_capability_count_capped=False, + process_reaped=process_reaped, + failure=failure, + ) + + +def _failure_category(exc: Exception, *, started: bool) -> ProbeFailure: + if isinstance(exc, AcpRequestTimeoutError): + return ProbeFailure.TIMEOUT + if isinstance(exc, AcpProtocolVersionError): + return ProbeFailure.PROTOCOL_VERSION + if isinstance(exc, (AcpFramingError, AcpEnvelopeError, AcpRemoteError)): + return ProbeFailure.PROTOCOL + if isinstance(exc, AcpTransportError): + return ProbeFailure.TRANSPORT if started else ProbeFailure.LAUNCH_FAILED + if isinstance(exc, (TypeError, ValueError, OSError)): + return ProbeFailure.INVALID_CONFIGURATION + return ProbeFailure.INTERNAL + + +def _bounded_timeout(value: float, name: str, *, maximum: float) -> float: + result = float(value) + if not math.isfinite(result) or not 0 < result <= maximum: + raise ValueError(f"{name} must be positive and at most {maximum:g}") + return result + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="python -m tendwire.backends.acp_probe", + description="Probe a separately installed ACP v1 adapter executable.", + ) + parser.add_argument( + "--timeout", + type=float, + default=DEFAULT_PROBE_TIMEOUT_SECONDS, + help=f"initialization timeout in seconds (maximum {MAX_PROBE_TIMEOUT_SECONDS:g})", + ) + parser.add_argument( + "--close-timeout", + type=float, + default=DEFAULT_PROBE_CLOSE_TIMEOUT_SECONDS, + help=( + "shutdown stage timeout in seconds " + f"(maximum {MAX_PROBE_CLOSE_TIMEOUT_SECONDS:g})" + ), + ) + parser.add_argument( + "--cwd", + default=None, + help="absolute process working directory (never included in output)", + ) + parser.add_argument( + "adapter_argv", + nargs=argparse.REMAINDER, + help="adapter executable and arguments, conventionally after --", + ) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + parser = _build_parser() + args = parser.parse_args(argv) + adapter_argv = list(args.adapter_argv) + if adapter_argv and adapter_argv[0] == "--": + adapter_argv.pop(0) + if not adapter_argv: + parser.error("an adapter executable is required after --") + report = probe_adapter( + adapter_argv, + cwd=args.cwd, + timeout_seconds=args.timeout, + close_timeout_seconds=args.close_timeout, + ) + sys.stdout.write(json.dumps(report.to_payload(), sort_keys=True, separators=(",", ":"))) + sys.stdout.write("\n") + return 0 if report.compatible else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_acp_probe.py b/tests/test_acp_probe.py new file mode 100644 index 0000000..a474a0b --- /dev/null +++ b/tests/test_acp_probe.py @@ -0,0 +1,159 @@ +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +from tendwire.backends.acp_probe import ( + MAX_PROBE_CLOSE_TIMEOUT_SECONDS, + MAX_PROBE_TIMEOUT_SECONDS, + ProbeFailure, + main, + probe_adapter, +) + + +FAKE_AGENT = Path(__file__).parent / "fixtures" / "acp_fake_agent.py" + + +def adapter_argv(mode: str = "normal") -> list[str]: + return [sys.executable, "-u", str(FAKE_AGENT), mode] + + +def test_probe_negotiates_fresh_stable_v1_capabilities_and_reaps_process() -> None: + first = probe_adapter(adapter_argv()) + second = probe_adapter(adapter_argv()) + + for report in (first, second): + payload = report.to_payload() + assert payload["schema_version"] == 1 + assert payload["compatible"] is True + assert payload["protocol_version"] == 1 + assert payload["process_reaped"] is True + assert payload["failure"] is None + assert payload["capabilities"]["session_new"] is True + assert payload["capabilities"]["session_load"] is True + assert payload["capabilities"]["session_close"] is True + assert payload["capabilities"]["session_delete"] is True + assert payload["extensions"] == { + "capability_count": 1, + "capability_count_capped": False, + } + + +def test_baseline_adapter_reports_only_baseline_capabilities() -> None: + payload = probe_adapter(adapter_argv("baseline")).to_payload() + assert payload["compatible"] is True + assert payload["capabilities"]["session_prompt"] is True + assert payload["capabilities"]["session_load"] is False + assert payload["capabilities"]["session_list"] is False + assert payload["extensions"]["capability_count"] == 0 + + +@pytest.mark.parametrize( + ("mode", "failure", "timeout"), + [ + ("malformed", ProbeFailure.PROTOCOL, 0.5), + ("partial_eof", ProbeFailure.PROTOCOL, 0.5), + ("bool_version", ProbeFailure.PROTOCOL_VERSION, 0.5), + ("no_read", ProbeFailure.TIMEOUT, 0.05), + ], +) +def test_probe_fails_closed_with_fixed_failure_categories( + mode: str, + failure: ProbeFailure, + timeout: float, +) -> None: + report = probe_adapter( + adapter_argv(mode), + timeout_seconds=timeout, + close_timeout_seconds=0.05, + ) + assert report.compatible is False + assert report.failure is failure + assert report.process_reaped is True + assert not any(report.capabilities.values()) + + +def test_missing_executable_does_not_expose_argv_or_exception_text() -> None: + secret = "TOP_SECRET_ADAPTER_ARGUMENT" + payload = probe_adapter(["/definitely/not/an/acp-adapter", secret]).to_payload() + encoded = json.dumps(payload) + assert payload["compatible"] is False + assert payload["failure"] == ProbeFailure.LAUNCH_FAILED.value + assert secret not in encoded + assert "/definitely/not" not in encoded + + +def test_report_never_exposes_agent_or_extension_controlled_text() -> None: + payload = probe_adapter(adapter_argv("extensions")).to_payload() + encoded = json.dumps(payload) + assert len(encoded.encode("utf-8")) < 2048 + assert "fake" not in encoded + assert "vendorFutureCapability" not in encoded + assert "vendor/future_notification" not in encoded + assert "level" not in encoded + assert payload["extensions"]["capability_count"] == 1 + + +def test_timeout_configuration_is_strictly_bounded() -> None: + for timeout in (0, -1, float("inf"), MAX_PROBE_TIMEOUT_SECONDS + 1): + report = probe_adapter(adapter_argv(), timeout_seconds=timeout) + assert report.failure is ProbeFailure.INVALID_CONFIGURATION + report = probe_adapter( + adapter_argv(), + close_timeout_seconds=MAX_PROBE_CLOSE_TIMEOUT_SECONDS + 1, + ) + assert report.failure is ProbeFailure.INVALID_CONFIGURATION + + +def test_stubborn_adapter_is_reaped_without_becoming_source_dependency() -> None: + report = probe_adapter( + adapter_argv("stubborn"), + timeout_seconds=1, + close_timeout_seconds=0.05, + ) + assert report.compatible is True + assert report.process_reaped is True + + +def test_module_cli_outputs_one_bounded_json_object(capsys: pytest.CaptureFixture[str]) -> None: + exit_code = main(["--timeout", "1", "--", *adapter_argv("baseline")]) + captured = capsys.readouterr() + payload = json.loads(captured.out) + assert exit_code == 0 + assert captured.err == "" + assert payload["compatible"] is True + assert len(captured.out.encode("utf-8")) < 2048 + + +def test_module_runs_as_black_box_without_importing_adapter_source() -> None: + source_root = str(Path(__file__).parents[1] / "src") + env = dict(os.environ) + env["PYTHONPATH"] = source_root + completed = subprocess.run( + [ + sys.executable, + "-m", + "tendwire.backends.acp_probe", + "--timeout", + "1", + "--", + *adapter_argv("baseline"), + ], + cwd=Path(__file__).parents[1], + env=env, + capture_output=True, + text=True, + timeout=5, + check=False, + ) + payload = json.loads(completed.stdout) + assert completed.returncode == 0 + assert completed.stderr == "" + assert payload["compatible"] is True + assert payload["process_reaped"] is True From 1df7bb157ecfe5208c779fb84d9903da2f507ad6 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 20:50:18 +0800 Subject: [PATCH 17/83] test: expect ACP journal schema --- tests/test_delivery_retention_projection.py | 2 +- tests/test_store.py | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/test_delivery_retention_projection.py b/tests/test_delivery_retention_projection.py index a941aec..47fb289 100644 --- a/tests/test_delivery_retention_projection.py +++ b/tests/test_delivery_retention_projection.py @@ -147,7 +147,7 @@ def _assert_continuity_integrity(db_path: Path) -> None: with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA user_version").fetchone() == ( store_sqlite.STORE_SCHEMA_VERSION, - ) == (21,) + ) == (23,) assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] current_counts = conn.execute( """ diff --git a/tests/test_store.py b/tests/test_store.py index a7a0a4d..e6c733e 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -14149,7 +14149,9 @@ def test_v20_to_v21_adds_herdr_turn_watermark_and_provenance_tables( init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == (21,) + assert conn.execute("PRAGMA user_version").fetchone() == ( + store_sqlite.STORE_SCHEMA_VERSION, + ) == (23,) assert { str(row[0]) for row in conn.execute( From 820fa4d86ac0bb53d3ee97dba491a53b1548f7d1 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:21:43 +0800 Subject: [PATCH 18/83] Harden ACP v1 transport conformance --- docs/acp-adapter-probe.md | 36 ++-- src/tendwire/backends/acp_client.py | 267 +++++++++++++++++++++++++- src/tendwire/backends/acp_probe.py | 103 +++++----- src/tendwire/backends/acp_protocol.py | 42 +++- tests/fixtures/acp_fake_agent.py | 72 ++++++- tests/test_acp_client.py | 154 ++++++++++++++- tests/test_acp_probe.py | 75 ++++++-- tests/test_acp_protocol.py | 60 +++++- 8 files changed, 691 insertions(+), 118 deletions(-) diff --git a/docs/acp-adapter-probe.md b/docs/acp-adapter-probe.md index f662b72..a717dce 100644 --- a/docs/acp-adapter-probe.md +++ b/docs/acp-adapter-probe.md @@ -1,8 +1,8 @@ -# Black-box ACP adapter compatibility probe +# Black-box ACP adapter initialization probe Tendwire treats an ACP adapter as a separately installed executable. The adapter is not vendored, imported, rebased, or coupled to a repository layout. -Run the compatibility probe after installing or upgrading any adapter and +Run the initialization probe after installing or upgrading any adapter and before promoting that executable into the Tendwire runtime: ```console @@ -15,9 +15,13 @@ object, and exits zero only when negotiation and shutdown succeed. Use `--timeout`, `--close-timeout`, and an absolute `--cwd` before the `--` marker when needed. Probe timeouts have hard upper bounds. -The output is intentionally narrow. It contains fixed boolean capabilities, -bounded counts for authentication methods and unknown capability extensions, a -process-reaped flag, and a fixed failure category. It never contains: +The probe scope is deliberately limited to `initialize`. It does not create a +session, send a prompt, spend model tokens, or claim that the adapter actually +implements ACP's mandatory baseline session methods. The output contains fixed +booleans for optional capabilities advertised by the agent, bounded counts for +schema-valid authentication methods and `_meta` capability-extension +namespaces, a process-reaped flag, and a fixed failure category. It never +contains: - adapter argv or executable paths; - working directories or environment values; @@ -30,7 +34,7 @@ Example successful shape: ```json { "authentication": {"method_count": 0, "method_count_capped": false}, - "capabilities": { + "advertised_capabilities": { "additional_directories": true, "auth_logout": false, "mcp_http": false, @@ -38,28 +42,26 @@ Example successful shape: "prompt_audio": false, "prompt_embedded_context": false, "prompt_image": false, - "session_cancel": true, "session_close": true, "session_delete": true, "session_list": true, "session_load": true, - "session_new": true, - "session_prompt": true, - "session_resume": true, - "session_update": true + "session_resume": true }, - "compatible": true, + "initialization_compatible": true, "extensions": {"capability_count": 0, "capability_count_capped": false}, "failure": null, + "probe_scope": "initialize", "process_reaped": true, "protocol_version": 1, - "schema_version": 1 + "schema_version": 2 } ``` -An incompatible result exits with status 1 and reports only one of these stable -categories: `invalid_configuration`, `launch_failed`, `timeout`, +An initialization-incompatible result exits with status 1 and reports only one +of these stable categories: `invalid_configuration`, `launch_failed`, `timeout`, `protocol_version`, `protocol_error`, `transport_error`, `shutdown_failed`, or `internal_error`. Keep the previously proven executable available for rollback; -the probe validates initialization and capability negotiation, not account -authentication or a stateful agent session. +the probe validates initialization and advertised capability parsing, not +account authentication, mandatory baseline method behavior, or a stateful agent +session. diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index 69e596e..4b9491c 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -146,6 +146,8 @@ def __init__( max_frame_bytes: int = DEFAULT_MAX_FRAME_BYTES, max_pending_events: int = _DEFAULT_QUEUE_SIZE, stderr_limit_bytes: int = _DEFAULT_STDERR_LIMIT, + supported_extension_notifications: Sequence[str] = (), + supported_extension_requests: Sequence[str] = (), ) -> None: command = tuple(os.fspath(item) for item in argv) if not command or any( @@ -167,6 +169,14 @@ def __init__( self.max_frame_bytes = max_frame_bytes self.max_pending_events = max_pending_events self.stderr_limit_bytes = stderr_limit_bytes + self.supported_extension_notifications = _extension_method_set( + supported_extension_notifications, + "supported_extension_notifications", + ) + self.supported_extension_requests = _extension_method_set( + supported_extension_requests, + "supported_extension_requests", + ) self._state = ClientState.NEW self._process: subprocess.Popen[bytes] | None = None @@ -346,6 +356,13 @@ def initialize( f"agent selected unsupported ACP protocol version {version!r}" ) self._set_failed(failure) + # ACP v1 says clients should close a connection after an + # unsupported selection. Do the bounded cleanup here so direct + # callers cannot accidentally leave the adapter process alive. + try: + self.close() + except BaseException: + pass raise failure capabilities_value = raw.get("agentCapabilities", {}) if not isinstance(capabilities_value, Mapping): @@ -568,9 +585,12 @@ def prompt( content = list(prompt) if not content: raise ValueError("prompt must contain at least one content block") - for block in content: - if not isinstance(block, Mapping) or not isinstance(block.get("type"), str): - raise ValueError("each prompt content block must have a string type") + self._require_initialized() + assert self.capabilities is not None + content = [ + _validated_prompt_content_block(block, self.capabilities) + for block in content + ] session_id = _nonempty(session_id, "session_id") with self._permission_lock: if self._active_prompts.get(session_id, 0) == 0: @@ -801,7 +821,7 @@ def _session_setup_params( self._require_initialized() params: dict[str, Any] = { "cwd": _absolute_path(cwd, "cwd"), - "mcpServers": [dict(server) for server in mcp_servers], + "mcpServers": [self._validated_mcp_server(server) for server in mcp_servers], } directories = [ _absolute_path(directory, "additional directory") @@ -812,6 +832,10 @@ def _session_setup_params( params["additionalDirectories"] = directories return params + def _validated_mcp_server(self, server: Mapping[str, Any]) -> dict[str, Any]: + assert self.capabilities is not None + return _validated_mcp_server(server, self.capabilities) + def _require_capability(self, name: str) -> None: self._require_initialized() assert self.capabilities is not None @@ -994,11 +1018,14 @@ def _dispatch( if isinstance(message, JsonRpcNotification): if message.method == "session/update": self._put_lossless(self._updates, parse_session_update(message.params)) - else: + elif message.method in self.supported_extension_notifications: self._put_lossless( self._notifications, RawNotification(message.method, message.params), ) + # ACP extension notifications are one-way and unrecognized ones + # should be ignored. Protocol-level $/ notifications are also + # explicitly optional, so they take the same bounded path. return if message.method == "session/request_permission": @@ -1029,11 +1056,19 @@ def _dispatch( ) else: self._put_lossless(self._permissions, parsed) - else: + elif message.method in self.supported_extension_requests: self._put_lossless( self._inbound_requests, InboundRequest(message.request_id, message.method, message.params), ) + else: + self._write( + error_envelope( + message.request_id, + _METHOD_NOT_FOUND, + "Method not found", + ) + ) def _put_lossless(self, target: queue.Queue[Any], value: Any) -> None: try: @@ -1157,6 +1192,226 @@ def _absolute_path(value: str | os.PathLike[str], name: str) -> str: return result +def _extension_method_set(values: Sequence[str], name: str) -> frozenset[str]: + if isinstance(values, str): + raise ValueError(f"{name} must be a sequence of extension method names") + result: set[str] = set() + for value in values: + if not isinstance(value, str) or not value.startswith("_") or len(value) == 1: + raise ValueError(f"{name} entries must be ACP methods starting with '_'") + result.add(value) + return frozenset(result) + + +def _validated_prompt_content_block( + value: Mapping[str, Any], + capabilities: AgentCapabilities, +) -> dict[str, Any]: + if not isinstance(value, Mapping): + raise ValueError("each prompt content block must be an object") + kind = value.get("type") + if kind == "text": + _reject_unknown_fields(value, {"type", "text", "annotations", "_meta"}, "text content") + _string_field(value, "text", "text content") + elif kind == "image": + if not capabilities.prompt_image: + raise AcpCapabilityError("agent did not advertise prompt image capability") + _reject_unknown_fields( + value, + {"type", "data", "mimeType", "uri", "annotations", "_meta"}, + "image content", + ) + _string_field(value, "data", "image content") + _string_field(value, "mimeType", "image content") + _optional_string_field(value, "uri", "image content") + elif kind == "audio": + if not capabilities.prompt_audio: + raise AcpCapabilityError("agent did not advertise prompt audio capability") + _reject_unknown_fields( + value, + {"type", "data", "mimeType", "annotations", "_meta"}, + "audio content", + ) + _string_field(value, "data", "audio content") + _string_field(value, "mimeType", "audio content") + elif kind == "resource_link": + _reject_unknown_fields( + value, + { + "type", + "name", + "uri", + "description", + "mimeType", + "size", + "title", + "annotations", + "_meta", + }, + "resource link", + ) + _string_field(value, "name", "resource link") + _string_field(value, "uri", "resource link") + for field in ("description", "mimeType", "title"): + _optional_string_field(value, field, "resource link") + size = value.get("size") + if size is not None and ( + not isinstance(size, int) + or isinstance(size, bool) + or not -(2**63) <= size <= 2**63 - 1 + ): + raise ValueError("resource link.size must be a signed 64-bit integer") + elif kind == "resource": + if not capabilities.prompt_embedded_context: + raise AcpCapabilityError( + "agent did not advertise prompt embeddedContext capability" + ) + _reject_unknown_fields( + value, + {"type", "resource", "annotations", "_meta"}, + "embedded resource", + ) + resource = value.get("resource") + if not isinstance(resource, Mapping): + raise ValueError("embedded resource.resource must be an object") + _string_field(resource, "uri", "embedded resource payload") + _optional_string_field(resource, "mimeType", "embedded resource payload") + has_text = "text" in resource + has_blob = "blob" in resource + if has_text == has_blob: + raise ValueError( + "embedded resource payload must contain exactly one of text or blob" + ) + payload_field = "text" if has_text else "blob" + _string_field(resource, payload_field, "embedded resource payload") + _reject_unknown_fields( + resource, + {"uri", "mimeType", payload_field, "_meta"}, + "embedded resource payload", + ) + _optional_mapping_field(resource, "_meta", "embedded resource payload") + else: + raise ValueError("prompt content block type is not valid ACP v1") + + _optional_mapping_field(value, "annotations", f"{kind} content") + _optional_mapping_field(value, "_meta", f"{kind} content") + return dict(value) + + +def _validated_mcp_server( + value: Mapping[str, Any], + capabilities: AgentCapabilities, +) -> dict[str, Any]: + if not isinstance(value, Mapping): + raise ValueError("each MCP server must be an object") + transport = value.get("type") + if transport is None: + _reject_unknown_fields( + value, + {"name", "command", "args", "env", "_meta"}, + "stdio MCP server", + ) + name = _string_field(value, "name", "stdio MCP server") + command = _absolute_path( + _string_field(value, "command", "stdio MCP server"), + "stdio MCP server.command", + ) + args = _string_array_field(value, "args", "stdio MCP server") + env = _name_value_array_field(value, "env", "stdio MCP server") + result: dict[str, Any] = { + "name": name, + "command": command, + "args": args, + "env": env, + } + elif transport in {"http", "sse"}: + if transport == "http" and not capabilities.mcp_http: + raise AcpCapabilityError("agent did not advertise MCP HTTP capability") + if transport == "sse" and not capabilities.mcp_sse: + raise AcpCapabilityError("agent did not advertise MCP SSE capability") + _reject_unknown_fields( + value, + {"type", "name", "url", "headers", "_meta"}, + f"{transport} MCP server", + ) + result = { + "type": transport, + "name": _string_field(value, "name", f"{transport} MCP server"), + "url": _string_field(value, "url", f"{transport} MCP server"), + "headers": _name_value_array_field( + value, + "headers", + f"{transport} MCP server", + ), + } + else: + raise ValueError("MCP server type is not valid stable ACP v1") + meta = _optional_mapping_field(value, "_meta", "MCP server") + if meta is not None: + result["_meta"] = dict(meta) + return result + + +def _reject_unknown_fields( + value: Mapping[str, Any], allowed: set[str], label: str +) -> None: + if any(key not in allowed for key in value): + raise ValueError(f"{label} contains fields outside the ACP v1 schema") + + +def _string_field(value: Mapping[str, Any], field: str, label: str) -> str: + result = value.get(field) + if not isinstance(result, str): + raise ValueError(f"{label}.{field} must be a string") + return result + + +def _optional_string_field(value: Mapping[str, Any], field: str, label: str) -> None: + result = value.get(field) + if result is not None and not isinstance(result, str): + raise ValueError(f"{label}.{field} must be a string or null") + + +def _optional_mapping_field( + value: Mapping[str, Any], field: str, label: str +) -> Mapping[str, Any] | None: + result = value.get(field) + if result is not None and not isinstance(result, Mapping): + raise ValueError(f"{label}.{field} must be an object or null") + return result + + +def _string_array_field( + value: Mapping[str, Any], field: str, label: str +) -> list[str]: + result = value.get(field) + if not isinstance(result, list) or any(not isinstance(item, str) for item in result): + raise ValueError(f"{label}.{field} must be an array of strings") + return list(result) + + +def _name_value_array_field( + value: Mapping[str, Any], field: str, label: str +) -> list[dict[str, Any]]: + result = value.get(field) + if not isinstance(result, list): + raise ValueError(f"{label}.{field} must be an array") + normalized: list[dict[str, Any]] = [] + for item in result: + if not isinstance(item, Mapping): + raise ValueError(f"{label}.{field} entries must be objects") + _reject_unknown_fields(item, {"name", "value", "_meta"}, f"{label}.{field} entry") + normalized_item: dict[str, Any] = { + "name": _string_field(item, "name", f"{label}.{field} entry"), + "value": _string_field(item, "value", f"{label}.{field} entry"), + } + meta = _optional_mapping_field(item, "_meta", f"{label}.{field} entry") + if meta is not None: + normalized_item["_meta"] = dict(meta) + normalized.append(normalized_item) + return normalized + + def _validated_env(env: Mapping[str, str] | None) -> dict[str, str] | None: if env is None: return None diff --git a/src/tendwire/backends/acp_probe.py b/src/tendwire/backends/acp_probe.py index 9569a33..3e63222 100644 --- a/src/tendwire/backends/acp_probe.py +++ b/src/tendwire/backends/acp_probe.py @@ -38,7 +38,8 @@ InitializeResult, ) -PROBE_SCHEMA_VERSION = 1 +PROBE_SCHEMA_VERSION = 2 +PROBE_SCOPE = "initialize" DEFAULT_PROBE_TIMEOUT_SECONDS = 5.0 DEFAULT_PROBE_CLOSE_TIMEOUT_SECONDS = 1.0 MAX_PROBE_TIMEOUT_SECONDS = 30.0 @@ -48,11 +49,7 @@ PROBE_STDERR_LIMIT_BYTES = 4096 _MAX_REPORTED_COUNT = 1000 -_CAPABILITY_KEYS = ( - "session_new", - "session_prompt", - "session_cancel", - "session_update", +_ADVERTISED_CAPABILITY_KEYS = ( "session_load", "session_list", "session_delete", @@ -66,27 +63,6 @@ "mcp_sse", "auth_logout", ) -_KNOWN_TOP_LEVEL_CAPABILITIES = { - "loadSession", - "promptCapabilities", - "mcpCapabilities", - "sessionCapabilities", - "auth", - "_meta", -} -_KNOWN_NESTED_CAPABILITIES = { - "promptCapabilities": {"image", "audio", "embeddedContext", "_meta"}, - "mcpCapabilities": {"http", "sse", "_meta"}, - "sessionCapabilities": { - "list", - "delete", - "additionalDirectories", - "resume", - "close", - "_meta", - }, - "auth": {"logout", "_meta"}, -} class ProbeFailure(str, Enum): @@ -106,9 +82,9 @@ class ProbeFailure(str, Enum): class AcpAdapterProbeReport: """Fixed-shape, bounded result that contains no adapter-controlled text.""" - compatible: bool + initialization_compatible: bool protocol_version: int | None - capabilities: Mapping[str, bool] + advertised_capabilities: Mapping[str, bool] authentication_method_count: int authentication_method_count_capped: bool extension_capability_count: int @@ -120,11 +96,12 @@ class AcpAdapterProbeReport: def to_payload(self) -> dict[str, Any]: return { "schema_version": self.schema_version, - "compatible": self.compatible, + "probe_scope": PROBE_SCOPE, + "initialization_compatible": self.initialization_compatible, "protocol_version": self.protocol_version, - "capabilities": { - key: bool(self.capabilities.get(key, False)) - for key in _CAPABILITY_KEYS + "advertised_capabilities": { + key: bool(self.advertised_capabilities.get(key, False)) + for key in _ADVERTISED_CAPABILITY_KEYS }, "authentication": { "method_count": self.authentication_method_count, @@ -151,8 +128,8 @@ def probe_adapter( A new subprocess and initialization exchange are used for every call. No session is created and no adapter source package is imported. All expected - failures become a fixed-category incompatible report; no raw diagnostic text - crosses this black-box boundary. + failures become a fixed-category initialization-incompatible report; no raw + diagnostic text crosses this black-box boundary. """ try: @@ -205,19 +182,22 @@ def probe_adapter( return _failure_report(failure or ProbeFailure.INTERNAL, process_reaped) capabilities = _capability_summary(initialized) - auth_count, auth_capped = _bounded_count(len(initialized.auth_methods)) + auth_count, auth_capped = _bounded_count( + sum(_valid_stable_auth_method(item) for item in initialized.auth_methods) + ) extension_count, extension_capped = _bounded_count( _extension_capability_count(initialized.capabilities.raw) ) - compatible = ( + initialization_compatible = ( failure is None and process_reaped and initialized.protocol_version == ACP_PROTOCOL_VERSION + and client.failure is None ) return AcpAdapterProbeReport( - compatible=compatible, + initialization_compatible=initialization_compatible, protocol_version=initialized.protocol_version, - capabilities=MappingProxyType(capabilities), + advertised_capabilities=MappingProxyType(capabilities), authentication_method_count=auth_count, authentication_method_count_capped=auth_capped, extension_capability_count=extension_count, @@ -232,13 +212,7 @@ def _capability_summary(initialized: InitializeResult) -> dict[str, bool]: prompt = _mapping(raw.get("promptCapabilities")) mcp = _mapping(raw.get("mcpCapabilities")) session = _mapping(raw.get("sessionCapabilities")) - auth = _mapping(raw.get("auth")) return { - # ACP v1 baseline methods are guaranteed by a successful negotiation. - "session_new": True, - "session_prompt": True, - "session_cancel": True, - "session_update": True, "session_load": initialized.capabilities.load_session, "session_list": initialized.capabilities.session_list, "session_delete": initialized.capabilities.session_delete, @@ -250,19 +224,38 @@ def _capability_summary(initialized: InitializeResult) -> dict[str, bool]: "prompt_embedded_context": prompt.get("embeddedContext") is True, "mcp_http": mcp.get("http") is True, "mcp_sse": mcp.get("sse") is True, - "auth_logout": isinstance(auth.get("logout"), Mapping), + "auth_logout": initialized.capabilities.auth_logout, } def _extension_capability_count(raw: Mapping[str, Any]) -> int: - count = sum(key not in _KNOWN_TOP_LEVEL_CAPABILITIES for key in raw) - for section, known_keys in _KNOWN_NESTED_CAPABILITIES.items(): - nested = raw.get(section) - if isinstance(nested, Mapping): - count += sum(key not in known_keys for key in nested) + """Count extension namespaces only where ACP v1 permits them: `_meta`.""" + count = _meta_entry_count(raw) + for section in ("promptCapabilities", "mcpCapabilities", "sessionCapabilities", "auth"): + nested = _mapping(raw.get(section)) + count += _meta_entry_count(nested) + session = _mapping(raw.get("sessionCapabilities")) + for capability in ("list", "delete", "additionalDirectories", "resume", "close"): + count += _meta_entry_count(_mapping(session.get(capability))) + auth = _mapping(raw.get("auth")) + count += _meta_entry_count(_mapping(auth.get("logout"))) return count +def _meta_entry_count(value: Mapping[str, Any]) -> int: + meta = value.get("_meta") + return len(meta) if isinstance(meta, Mapping) else 0 + + +def _valid_stable_auth_method(value: Mapping[str, Any]) -> bool: + method_type = value.get("type", "agent") + return ( + method_type == "agent" + and isinstance(value.get("id"), str) + and isinstance(value.get("name"), str) + ) + + def _mapping(value: Any) -> Mapping[str, Any]: return value if isinstance(value, Mapping) else {} @@ -276,9 +269,11 @@ def _failure_report( process_reaped: bool = True, ) -> AcpAdapterProbeReport: return AcpAdapterProbeReport( - compatible=False, + initialization_compatible=False, protocol_version=None, - capabilities=MappingProxyType({key: False for key in _CAPABILITY_KEYS}), + advertised_capabilities=MappingProxyType( + {key: False for key in _ADVERTISED_CAPABILITY_KEYS} + ), authentication_method_count=0, authentication_method_count_capped=False, extension_capability_count=0, @@ -358,7 +353,7 @@ def main(argv: Sequence[str] | None = None) -> int: ) sys.stdout.write(json.dumps(report.to_payload(), sort_keys=True, separators=(",", ":"))) sys.stdout.write("\n") - return 0 if report.compatible else 1 + return 0 if report.initialization_compatible else 1 if __name__ == "__main__": diff --git a/src/tendwire/backends/acp_protocol.py b/src/tendwire/backends/acp_protocol.py index 9bc8a77..1143b63 100644 --- a/src/tendwire/backends/acp_protocol.py +++ b/src/tendwire/backends/acp_protocol.py @@ -170,10 +170,43 @@ def additional_directories(self) -> bool: self._session_capabilities().get("additionalDirectories") ) + @property + def prompt_image(self) -> bool: + return self._prompt_capabilities().get("image") is True + + @property + def prompt_audio(self) -> bool: + return self._prompt_capabilities().get("audio") is True + + @property + def prompt_embedded_context(self) -> bool: + return self._prompt_capabilities().get("embeddedContext") is True + + @property + def mcp_http(self) -> bool: + return self._mcp_capabilities().get("http") is True + + @property + def mcp_sse(self) -> bool: + return self._mcp_capabilities().get("sse") is True + + @property + def auth_logout(self) -> bool: + auth = self.raw.get("auth") + return isinstance(auth, Mapping) and _is_capability_object(auth.get("logout")) + def _session_capabilities(self) -> Mapping[str, Any]: value = self.raw.get("sessionCapabilities") return value if isinstance(value, Mapping) else {} + def _prompt_capabilities(self) -> Mapping[str, Any]: + value = self.raw.get("promptCapabilities") + return value if isinstance(value, Mapping) else {} + + def _mcp_capabilities(self) -> Mapping[str, Any]: + value = self.raw.get("mcpCapabilities") + return value if isinstance(value, Mapping) else {} + @dataclass(frozen=True, slots=True) class InitializeResult: @@ -481,9 +514,10 @@ def parse_permission_request(request: JsonRpcRequest) -> PermissionRequest: tool_call = params.get("toolCall") if not isinstance(tool_call, Mapping): raise AcpEnvelopeError("permission request toolCall must be an object") + _required_string(tool_call, "toolCallId") raw_options = params.get("options") - if not isinstance(raw_options, list) or not raw_options: - raise AcpEnvelopeError("permission request options must be a non-empty array") + if not isinstance(raw_options, list): + raise AcpEnvelopeError("permission request options must be an array") options: list[PermissionOption] = [] seen_option_ids: set[str] = set() for raw in raw_options: @@ -497,8 +531,8 @@ def parse_permission_request(request: JsonRpcRequest) -> PermissionRequest: kind_value = _required_string(raw, "kind") try: kind: PermissionOptionKind | str = PermissionOptionKind(kind_value) - except ValueError: - kind = kind_value + except ValueError as exc: + raise AcpEnvelopeError("permission option kind is not valid ACP v1") from exc options.append( PermissionOption( option_id=option_id, diff --git a/tests/fixtures/acp_fake_agent.py b/tests/fixtures/acp_fake_agent.py index c513cd3..c38ba40 100644 --- a/tests/fixtures/acp_fake_agent.py +++ b/tests/fixtures/acp_fake_agent.py @@ -49,6 +49,16 @@ def update(session_id: str, kind: str, **values: object) -> None: request_id = message.get("id") params = message.get("params", {}) + if MODE == "initialize_only" and method != "initialize": + send( + { + "jsonrpc": "2.0", + "id": request_id, + "error": {"code": -32601, "message": "Method not found"}, + } + ) + continue + if method == "initialize": if MODE == "malformed": sys.stdout.write("not-json\n") @@ -67,7 +77,7 @@ def update(session_id: str, kind: str, **values: object) -> None: "protocolVersion": True if MODE == "bool_version" else 1, "agentCapabilities": ( {} - if MODE == "baseline" + if MODE in {"baseline", "initialize_only"} else { "loadSession": True, "sessionCapabilities": { @@ -77,20 +87,56 @@ def update(session_id: str, kind: str, **values: object) -> None: "close": {}, "additionalDirectories": {}, }, - "vendorFutureCapability": {"level": 2}, + "promptCapabilities": { + "image": True, + "audio": True, + "embeddedContext": True, + }, + "mcpCapabilities": {"http": True, "sse": True}, + "_meta": {"vendor.example": {"level": 2}}, } ), "agentInfo": {"name": "fake", "version": "1.0"}, + **( + { + "authMethods": [ + {}, + {"id": "missing-name"}, + {"id": "valid", "name": "Valid agent login"}, + {"type": "future", "id": "x", "name": "Future"}, + ] + } + if MODE == "auth_shapes" + else {} + ), }, ) if MODE == "extensions": send( { "jsonrpc": "2.0", - "method": "vendor/future_notification", + "method": "_vendor.example/future_notification", "params": {"opaque": {"revision": 9}}, } ) + if MODE == "extension_flood": + for index in range(32): + send( + { + "jsonrpc": "2.0", + "method": "_vendor.example/progress", + "params": {"sequence": index}, + } + ) + if MODE in {"unknown_request", "supported_request"}: + send( + { + "jsonrpc": "2.0", + "id": 777 if MODE == "unknown_request" else 778, + "method": "_vendor.example/request", + "params": {"opaque": True}, + } + ) if MODE == "null_response": send( { @@ -110,9 +156,8 @@ def update(session_id: str, kind: str, **values: object) -> None: for index in range(4): update( "s-flood", - "vendor_progress", - sequence=index, - vendor={"opaque": True}, + "agent_message_chunk", + content={"type": "text", "text": str(index)}, ) elif method == "session/new": update("s-new", "agent_message_chunk", content={"type": "text", "text": "hi"}) @@ -123,7 +168,7 @@ def update(session_id: str, kind: str, **values: object) -> None: elif method == "session/load" or method == "session/resume": response(request_id, {"configOptions": [{"id": "model", "currentValue": "x"}]}) elif method == "session/close" or method == "session/delete": - response(request_id, {"vendorReceipt": method}) + response(request_id, {"_meta": {"vendor.example": {"receipt": method}}}) elif method == "session/list": if MODE == "slow": time.sleep(2) @@ -143,6 +188,9 @@ def update(session_id: str, kind: str, **values: object) -> None: }, ) elif method == "session/prompt": + if MODE == "echo_prompt": + response(request_id, {"stopReason": "end_turn"}) + continue pending_prompt_id = request_id pending_prompt_session = params["sessionId"] update( @@ -195,6 +243,16 @@ def update(session_id: str, kind: str, **values: object) -> None: } ) pending_permission_ids.add(901) + elif request_id in {777, 778}: + error = message.get("error", {}) + update( + "s-extension", + "agent_message_chunk", + content={ + "type": "text", + "text": "method-not-found" if error.get("code") == -32601 else "unexpected", + }, + ) elif request_id in pending_permission_ids and pending_prompt_id is not None: outcome = message["result"]["outcome"]["outcome"] pending_permission_ids.remove(request_id) diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py index 974d277..7712834 100644 --- a/tests/test_acp_client.py +++ b/tests/test_acp_client.py @@ -35,7 +35,9 @@ def test_initialize_capabilities_and_session_lifecycle() -> None: assert initialized.capabilities.session_resume assert initialized.capabilities.session_close assert initialized.capabilities.session_delete - assert initialized.capabilities.raw["vendorFutureCapability"] == {"level": 2} + assert initialized.capabilities.raw["_meta"] == { + "vendor.example": {"level": 2} + } assert acp.state is ClientState.INITIALIZED created = acp.new_session( @@ -59,8 +61,8 @@ def test_initialize_capabilities_and_session_lifecycle() -> None: assert second.sessions[0].title == "second" assert second.next_cursor is None - assert acp.close_session("s1")["vendorReceipt"] == "session/close" - assert acp.delete_session("s2")["vendorReceipt"] == "session/delete" + assert acp.close_session("s1")["_meta"]["vendor.example"]["receipt"] == "session/close" + assert acp.delete_session("s2")["_meta"]["vendor.example"]["receipt"] == "session/delete" assert acp.state is ClientState.CLOSED assert acp.exit is not None @@ -167,6 +169,105 @@ def test_absolute_session_paths_are_enforced_before_write() -> None: acp.new_session("relative/path") +def test_prompt_content_is_validated_and_gated_by_negotiated_capabilities() -> None: + with client("baseline") as acp: + acp.initialize() + with pytest.raises(AcpCapabilityError, match="image"): + acp.prompt( + "s1", + [{"type": "image", "data": "AA==", "mimeType": "image/png"}], + ) + with pytest.raises(ValueError, match="text"): + acp.prompt("s1", [{"type": "text"}]) + with pytest.raises(ValueError, match="outside"): + acp.prompt("s1", [{"type": "text", "text": "ok", "vendor": True}]) + + +def test_prompt_content_matches_stable_v1_generated_shapes() -> None: + blocks = [ + {"type": "text", "text": "hello"}, + {"type": "image", "data": "AA==", "mimeType": "image/png"}, + {"type": "audio", "data": "AA==", "mimeType": "audio/wav"}, + {"type": "resource_link", "name": "main.py", "uri": "file:///tmp/main.py"}, + { + "type": "resource", + "resource": { + "uri": "file:///tmp/main.py", + "mimeType": "text/x-python", + "text": "print('ok')", + }, + }, + ] + with client("echo_prompt") as acp: + acp.initialize() + result = acp.prompt("s1", blocks) + assert result.stop_reason is StopReason.END_TURN + + +def test_mcp_servers_match_stable_v1_generated_shapes_and_capabilities() -> None: + with client() as acp: + acp.initialize() + created = acp.new_session( + "/tmp/project", + mcp_servers=[ + { + "name": "stdio-tools", + "command": "/usr/bin/tools", + "args": ["--stdio"], + "env": [{"name": "MODE", "value": "test"}], + }, + { + "type": "http", + "name": "http-tools", + "url": "https://example.invalid/mcp", + "headers": [{"name": "Authorization", "value": "opaque"}], + }, + { + "type": "sse", + "name": "legacy-tools", + "url": "https://example.invalid/sse", + "headers": [], + }, + ], + ) + assert created.session_id == "s-new" + + with client("baseline") as acp: + acp.initialize() + with pytest.raises(AcpCapabilityError, match="HTTP"): + acp.new_session( + "/tmp/project", + mcp_servers=[ + { + "type": "http", + "name": "http-tools", + "url": "https://example.invalid/mcp", + "headers": [], + } + ], + ) + with pytest.raises(ValueError, match="absolute"): + acp.new_session( + "/tmp/project", + mcp_servers=[ + {"name": "stdio-tools", "command": "tools", "args": [], "env": []} + ], + ) + with pytest.raises(ValueError, match="stable ACP v1"): + acp.new_session( + "/tmp/project", + mcp_servers=[ + { + "type": "stdio", + "name": "stdio-tools", + "command": "/usr/bin/tools", + "args": [], + "env": [], + } + ], + ) + + def test_concurrent_initialize_is_exactly_once_and_returns_same_result() -> None: with client() as acp: results: list[object] = [] @@ -197,7 +298,7 @@ def test_backpressure_failure_remains_visible_after_full_queue_drains() -> None: assert acp.state is ClientState.FAILED assert isinstance(acp.failure, AcpEventQueueFullError) first = acp.next_update(timeout=1) - assert first.update_kind == "vendor_progress" + assert first.update_kind is SessionUpdateKind.AGENT_MESSAGE_CHUNK with pytest.raises(AcpTransportError): acp.next_update(timeout=1) @@ -251,14 +352,45 @@ def test_stderr_tail_is_bounded_and_keeps_suffix() -> None: def test_unknown_adapter_extensions_remain_observable() -> None: - with client("extensions") as acp: + method = "_vendor.example/future_notification" + with client("extensions", supported_extension_notifications=(method,)) as acp: initialized = acp.initialize() - assert initialized.capabilities.raw["vendorFutureCapability"] == {"level": 2} + assert initialized.capabilities.raw["_meta"] == { + "vendor.example": {"level": 2} + } notification = acp.next_notification(timeout=1) - assert notification.method == "vendor/future_notification" + assert notification.method == method assert notification.params["opaque"] == {"revision": 9} +def test_unrecognized_extension_notifications_are_ignored_without_backpressure() -> None: + with client("extension_flood", max_pending_events=1) as acp: + acp.initialize() + time.sleep(0.1) + assert acp.state is ClientState.INITIALIZED + with pytest.raises(AcpRequestTimeoutError): + acp.next_notification(timeout=0.05) + + +def test_unsupported_inbound_request_gets_automatic_method_not_found() -> None: + with client("unknown_request") as acp: + acp.initialize() + confirmation = acp.next_update(timeout=1) + assert confirmation.session_id == "s-extension" + assert confirmation.update["content"]["text"] == "method-not-found" + + +def test_explicitly_supported_extension_request_remains_observable() -> None: + method = "_vendor.example/request" + with client("supported_request", supported_extension_requests=(method,)) as acp: + acp.initialize() + request = acp.next_inbound_request(timeout=1) + assert request.method == method + acp.reject_inbound_request(request.request_id) + confirmation = acp.next_update(timeout=1) + assert confirmation.update["content"]["text"] == "method-not-found" + + def test_uncorrelated_null_error_response_does_not_poison_transport() -> None: with client("null_response") as acp: acp.initialize() @@ -283,6 +415,14 @@ def test_boolean_protocol_version_is_not_accepted_as_integer_one() -> None: assert acp.state is ClientState.FAILED +def test_unsupported_protocol_version_is_reaped_during_initialize() -> None: + acp = client("bool_version", close_timeout=0.1) + with pytest.raises(AcpProtocolError, match="protocol version"): + acp.initialize() + assert acp.process is not None + assert acp.process.poll() is not None + + @pytest.mark.parametrize( "kwargs", [ diff --git a/tests/test_acp_probe.py b/tests/test_acp_probe.py index a474a0b..816f504 100644 --- a/tests/test_acp_probe.py +++ b/tests/test_acp_probe.py @@ -12,6 +12,7 @@ MAX_PROBE_CLOSE_TIMEOUT_SECONDS, MAX_PROBE_TIMEOUT_SECONDS, ProbeFailure, + _extension_capability_count, main, probe_adapter, ) @@ -30,15 +31,15 @@ def test_probe_negotiates_fresh_stable_v1_capabilities_and_reaps_process() -> No for report in (first, second): payload = report.to_payload() - assert payload["schema_version"] == 1 - assert payload["compatible"] is True + assert payload["schema_version"] == 2 + assert payload["probe_scope"] == "initialize" + assert payload["initialization_compatible"] is True assert payload["protocol_version"] == 1 assert payload["process_reaped"] is True assert payload["failure"] is None - assert payload["capabilities"]["session_new"] is True - assert payload["capabilities"]["session_load"] is True - assert payload["capabilities"]["session_close"] is True - assert payload["capabilities"]["session_delete"] is True + assert payload["advertised_capabilities"]["session_load"] is True + assert payload["advertised_capabilities"]["session_close"] is True + assert payload["advertised_capabilities"]["session_delete"] is True assert payload["extensions"] == { "capability_count": 1, "capability_count_capped": False, @@ -47,13 +48,26 @@ def test_probe_negotiates_fresh_stable_v1_capabilities_and_reaps_process() -> No def test_baseline_adapter_reports_only_baseline_capabilities() -> None: payload = probe_adapter(adapter_argv("baseline")).to_payload() - assert payload["compatible"] is True - assert payload["capabilities"]["session_prompt"] is True - assert payload["capabilities"]["session_load"] is False - assert payload["capabilities"]["session_list"] is False + assert payload["initialization_compatible"] is True + assert "session_prompt" not in payload["advertised_capabilities"] + assert "session_new" not in payload["advertised_capabilities"] + assert payload["advertised_capabilities"]["session_load"] is False + assert payload["advertised_capabilities"]["session_list"] is False assert payload["extensions"]["capability_count"] == 0 +def test_initialize_only_agent_does_not_gain_untested_baseline_claims() -> None: + payload = probe_adapter(adapter_argv("initialize_only")).to_payload() + assert payload["initialization_compatible"] is True + assert payload["probe_scope"] == "initialize" + assert not { + "session_new", + "session_prompt", + "session_cancel", + "session_update", + }.intersection(payload["advertised_capabilities"]) + + @pytest.mark.parametrize( ("mode", "failure", "timeout"), [ @@ -73,17 +87,17 @@ def test_probe_fails_closed_with_fixed_failure_categories( timeout_seconds=timeout, close_timeout_seconds=0.05, ) - assert report.compatible is False + assert report.initialization_compatible is False assert report.failure is failure assert report.process_reaped is True - assert not any(report.capabilities.values()) + assert not any(report.advertised_capabilities.values()) def test_missing_executable_does_not_expose_argv_or_exception_text() -> None: secret = "TOP_SECRET_ADAPTER_ARGUMENT" payload = probe_adapter(["/definitely/not/an/acp-adapter", secret]).to_payload() encoded = json.dumps(payload) - assert payload["compatible"] is False + assert payload["initialization_compatible"] is False assert payload["failure"] == ProbeFailure.LAUNCH_FAILED.value assert secret not in encoded assert "/definitely/not" not in encoded @@ -94,8 +108,8 @@ def test_report_never_exposes_agent_or_extension_controlled_text() -> None: encoded = json.dumps(payload) assert len(encoded.encode("utf-8")) < 2048 assert "fake" not in encoded - assert "vendorFutureCapability" not in encoded - assert "vendor/future_notification" not in encoded + assert "vendor.example" not in encoded + assert "_vendor.example/future_notification" not in encoded assert "level" not in encoded assert payload["extensions"]["capability_count"] == 1 @@ -117,7 +131,7 @@ def test_stubborn_adapter_is_reaped_without_becoming_source_dependency() -> None timeout_seconds=1, close_timeout_seconds=0.05, ) - assert report.compatible is True + assert report.initialization_compatible is True assert report.process_reaped is True @@ -127,7 +141,7 @@ def test_module_cli_outputs_one_bounded_json_object(capsys: pytest.CaptureFixtur payload = json.loads(captured.out) assert exit_code == 0 assert captured.err == "" - assert payload["compatible"] is True + assert payload["initialization_compatible"] is True assert len(captured.out.encode("utf-8")) < 2048 @@ -155,5 +169,30 @@ def test_module_runs_as_black_box_without_importing_adapter_source() -> None: payload = json.loads(completed.stdout) assert completed.returncode == 0 assert completed.stderr == "" - assert payload["compatible"] is True + assert payload["initialization_compatible"] is True assert payload["process_reaped"] is True + + +def test_extension_count_uses_only_spec_reserved_meta_locations() -> None: + assert ( + _extension_capability_count( + { + "_meta": {"vendor.one": {}}, + "promptCapabilities": {"_meta": {"vendor.two": {}}}, + "sessionCapabilities": { + "list": {"_meta": {"vendor.three": {}}} + }, + } + ) + == 3 + ) + assert _extension_capability_count({"forbiddenRootExtension": {}}) == 0 + + +def test_authentication_count_skips_invalid_stable_schema_items() -> None: + payload = probe_adapter(adapter_argv("auth_shapes")).to_payload() + assert payload["initialization_compatible"] is True + assert payload["authentication"] == { + "method_count": 1, + "method_count_capped": False, + } diff --git a/tests/test_acp_protocol.py b/tests/test_acp_protocol.py index 41407fd..c7caaa0 100644 --- a/tests/test_acp_protocol.py +++ b/tests/test_acp_protocol.py @@ -112,7 +112,14 @@ def test_capability_presence_uses_acp_object_semantics() -> None: "close": {}, "additionalDirectories": {}, }, - "vendorFutureCapability": {"revision": 3}, + "promptCapabilities": { + "image": True, + "audio": True, + "embeddedContext": True, + }, + "mcpCapabilities": {"http": True, "sse": True}, + "auth": {"logout": {}}, + "_meta": {"vendor.example": {"revision": 3}}, } ) assert capabilities.load_session @@ -121,16 +128,22 @@ def test_capability_presence_uses_acp_object_semantics() -> None: assert capabilities.session_close assert capabilities.session_delete assert capabilities.additional_directories - assert capabilities.raw["vendorFutureCapability"] == {"revision": 3} + assert capabilities.prompt_image + assert capabilities.prompt_audio + assert capabilities.prompt_embedded_context + assert capabilities.mcp_http + assert capabilities.mcp_sse + assert capabilities.auth_logout + assert capabilities.raw["_meta"] == {"vendor.example": {"revision": 3}} def test_request_ids_follow_acp_json_rpc_domain() -> None: assert isinstance( - validate_envelope({"jsonrpc": "2.0", "id": "", "method": "vendor/x"}), + validate_envelope({"jsonrpc": "2.0", "id": "", "method": "_vendor.example/x"}), JsonRpcRequest, ) null_request = validate_envelope( - {"jsonrpc": "2.0", "id": None, "method": "vendor/x"} + {"jsonrpc": "2.0", "id": None, "method": "_vendor.example/x"} ) assert isinstance(null_request, JsonRpcRequest) assert null_request.request_id is None @@ -143,7 +156,7 @@ def test_request_ids_follow_acp_json_rpc_domain() -> None: for invalid_id in (True, 2**63, -(2**63) - 1, 1.5): with pytest.raises(AcpEnvelopeError): validate_envelope( - {"jsonrpc": "2.0", "id": invalid_id, "method": "vendor/x"} + {"jsonrpc": "2.0", "id": invalid_id, "method": "_vendor.example/x"} ) @@ -164,6 +177,43 @@ def test_permission_option_ids_must_be_unambiguous() -> None: parse_permission_request(request) +def test_permission_request_matches_stable_v1_required_fields_and_enum() -> None: + empty_options = JsonRpcRequest( + 42, + "session/request_permission", + {"sessionId": "s1", "toolCall": {"toolCallId": "tool-1"}, "options": []}, + ) + assert parse_permission_request(empty_options).options == () + + missing_tool_call_id = JsonRpcRequest( + 43, + "session/request_permission", + { + "sessionId": "s1", + "toolCall": {}, + "options": [ + {"optionId": "allow", "name": "Allow", "kind": "allow_once"} + ], + }, + ) + with pytest.raises(AcpEnvelopeError, match="toolCallId"): + parse_permission_request(missing_tool_call_id) + + unknown_kind = JsonRpcRequest( + 44, + "session/request_permission", + { + "sessionId": "s1", + "toolCall": {"toolCallId": "tool-1"}, + "options": [ + {"optionId": "future", "name": "Future", "kind": "future_kind"} + ], + }, + ) + with pytest.raises(AcpEnvelopeError, match="ACP v1"): + parse_permission_request(unknown_kind) + + def test_unknown_update_and_nested_extension_payload_are_preserved() -> None: extension = parse_session_update( { From 2eda9dbc3b466b4c1e9fd88ddcc8fc1323b66bfb Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:21:47 +0800 Subject: [PATCH 19/83] fix: keep ACP migration fail closed --- .env.example | 12 +- README.md | 25 ++-- docs/acp-migration.md | 67 ++++++--- src/tendwire/config.py | 2 +- src/tendwire/core/agent_events.py | 8 +- src/tendwire/daemon.py | 13 +- src/tendwire/store/sqlite.py | 230 +++++++++++++++++++++++++++++- tests/test_agent_events.py | 55 ++++++- tests/test_config.py | 2 +- tests/test_daemon_acp.py | 31 ++-- tests/test_store.py | 2 +- 11 files changed, 380 insertions(+), 67 deletions(-) diff --git a/.env.example b/.env.example index 8c60a0c..9b55f61 100644 --- a/.env.example +++ b/.env.example @@ -93,12 +93,12 @@ TENDWIRE_TURN_REFRESH_WORKERS=4 # Compatibility flag: legacy|dual|shadow|observed all use the observed model. TENDWIRE_TURN_MODEL=observed -# Structured agent-event source policy. ACP is preferred when a compatible -# session is bound; existing Herdr turn adapters remain the lossless fallback. -# acp_shadow records ACP without projecting it, while acp_required fails closed -# instead of using legacy content. Thought content is private by default and is -# never eligible for the connector outbox solely because ACP emitted it. -TENDWIRE_AGENT_EVENT_SOURCE=acp_preferred +# Structured agent-event source policy. ACP runtime discovery and per-worker +# authority are experimental and are not wired into the stock daemon yet. +# acp_shadow requires an explicitly injected runtime and records ACP without +# projecting it. acp_required refuses legacy turn ingestion and fails closed +# unless that explicit runtime starts healthy. +TENDWIRE_AGENT_EVENT_SOURCE=legacy TENDWIRE_ACP_THOUGHT_POLICY=private_summary TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS=30 TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS=5 diff --git a/README.md b/README.md index 62dbbb7..ac69c56 100644 --- a/README.md +++ b/README.md @@ -553,7 +553,7 @@ variables: | `turn_refresh_interval_seconds` | `TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS` | `2.0` | finite positive float | | `turn_refresh_workers` | `TENDWIRE_TURN_REFRESH_WORKERS` | `4` | integer from 1 through 32 and no greater than `max_workers` | | `turn_model` | `TENDWIRE_TURN_MODEL` | `observed` | `observed`; `legacy`, `dual`, and `shadow` are deprecated aliases with identical observed behavior | -| `agent_event_source` | `TENDWIRE_AGENT_EVENT_SOURCE` | `acp_preferred` | `legacy`, `acp_shadow`, `acp_preferred`, or `acp_required` | +| `agent_event_source` | `TENDWIRE_AGENT_EVENT_SOURCE` | `legacy` | `legacy`, `acp_shadow`, `acp_preferred`, or `acp_required`; ACP modes are experimental | | `acp_thought_policy` | `TENDWIRE_ACP_THOUGHT_POLICY` | `private_summary` | `disabled`, `private_summary`, or `private_all`; never a public-delivery grant | | `acp_request_timeout_seconds` | `TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS` | `30.0` | finite positive float | | `acp_shutdown_timeout_seconds` | `TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS` | `5.0` | finite positive float | @@ -570,13 +570,22 @@ snapshot/projections instead of publishing a truncated authoritative snapshot. Incremental events that would add workers over the cap are ignored with the same public-safe degraded evidence. -`acp_preferred` means ACP is the primary semantic source only for workers with -an authenticated ACP session binding; workers without one continue through the -existing Herdr turn adapters. `acp_shadow` persists and compares ACP events but -does not project them into turns. `acp_required` fails closed for an unbound or -unhealthy ACP worker. None of these modes makes agent thoughts public: thought -events remain private diagnostic data unless a separate, explicit sanitized -projection is introduced. +The stock daemon currently defaults to `legacy`. ACP runtime discovery, +per-worker authority selection, and automatic reconnect are not production +wired yet; ACP modes are integration/test surfaces that require an explicitly +supplied runtime factory. `acp_shadow` persists ACP events without projecting +them, but no automated shadow comparator is implemented. `acp_preferred` must +not be treated as a production authority promise until that coordinator exists. +`acp_required` fails startup without an explicit healthy runtime and never +starts the legacy turn scheduler. None of these modes makes agent thoughts +public: thought events remain private diagnostic data unless a separate, +explicit sanitized projection is introduced. + +Store maintenance retires expired structured agent-event payloads in bounded +batches using `event_retention_days`. Compact identity tombstones remain so a +replayed source event cannot be reinserted or silently change content after its +private payload has expired. Tombstones intentionally retain hashes and opaque +identity only; they are not a recoverable copy of the removed payload. Snapshot history defaults are sized for a five-minute observation rhythm: $14 \times 24 \times 12 = 4032$ observations, while the 4096-row count diff --git a/docs/acp-migration.md b/docs/acp-migration.md index d1eff91..e22eff1 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -1,7 +1,8 @@ # ACP primary-event migration -This document defines the migration from backend-specific transcript readers to -Agent Client Protocol (ACP) as Tendwire's preferred semantic event source. +This document defines the experimental migration from backend-specific +transcript readers to Agent Client Protocol (ACP). ACP is not yet Tendwire's +default or a production-wired semantic authority. Herdr remains authoritative for workspace, pane, worker identity, process liveness, and command routing until the ACP control path is proven separately. Tendwire remains authoritative for persistence, reconciliation, public safety, @@ -12,16 +13,21 @@ command receipts, and connector delivery. `TENDWIRE_AGENT_EVENT_SOURCE` controls projection precedence: - `legacy`: use the existing Herdr/Codex/OMP turn readers only. -- `acp_shadow`: ingest ACP events durably, compare them with legacy turns, and - keep legacy turns authoritative. -- `acp_preferred`: use ACP for an authenticated, healthy ACP-bound worker and - fall back to the legacy reader for every other worker. +- `acp_shadow`: with an explicitly supplied ACP runtime, ingest ACP events + durably without projecting them; legacy turns remain authoritative. Automated + comparison is not implemented yet. +- `acp_preferred`: an experimental integration surface for future per-worker + ACP authority and legacy fallback. The stock daemon does not discover or + construct an ACP runtime. - `acp_required`: use ACP only and fail closed when the binding or stream is not - healthy. This mode is intended for conformance testing, not initial rollout. + healthy. The daemon does not start its legacy turn scheduler in this mode. + This mode requires an explicitly supplied healthy authority runtime and is + intended for conformance testing, not rollout. -The default is `acp_preferred`. The default does not invent an ACP session or -silently replace a worker: without a proven binding, legacy observation remains -authoritative. +The default is `legacy`. Selecting an ACP mode does not discover an adapter, +invent an ACP session, or bind a worker. Until a per-worker authority +coordinator exists, operators must not interpret `acp_preferred` as proof that +ACP is authoritative. ## Authority split @@ -30,7 +36,7 @@ authoritative. | Workspace and logical pane identity | Herdr | | Public stable worker identity | Tendwire's authenticated Herdr projection | | ACP session and message identity | ACP agent, stored privately by Tendwire | -| Messages, thoughts, tools, plans, and usage | ACP when preferred and healthy | +| Messages, thoughts, tools, plans, and usage | Future ACP authority coordinator; currently experimental | | Turn finality and connector eligibility | Tendwire durable projection | | Telegram presentation and delivery state | Herdres | | Command idempotency and uncertain outcomes | Tendwire command receipts | @@ -91,10 +97,12 @@ The boundary has four rules: - verify adapter releases with black-box ACP compatibility fixtures before promotion, while keeping the previously proven executable for rollback. -An adapter upgrade therefore restarts only its owned process/session; it does -not require a Tendwire rebase. A session may resume when the new adapter -advertises that capability. Otherwise Tendwire opens a new transport generation -and reconciles it through the durable semantic journal. +The wire-process boundary is designed so an adapter upgrade does not require a +Tendwire rebase. The current initialization-only probe is not a promotion gate: +it does not authenticate, create/load a session, prompt, validate updates, +exercise permissions/cancellation, or pin an executable digest. Stateful +conformance fixtures and an immutable rollback manifest are still required. +Session resume/replay reconciliation is likewise incomplete. ## Runtime lifecycle @@ -104,6 +112,11 @@ shutdown. Tendwire must not claim an ACP worker healthy until initialization, capability negotiation, session creation/load/resume, and private worker binding all succeed. +The stock daemon currently has no production ACP runtime factory or multi-worker +supervisor. Herdr must first provide authenticated per-worker launch/session +metadata, and Tendwire must add per-worker health, authority, reconnect, and +durable projection recovery before ACP can become the default. + Disconnect handling is conservative: 1. Stop accepting events from the disconnected generation. @@ -112,12 +125,30 @@ Disconnect handling is conservative: 4. Reinitialize and rebind before accepting ACP events again. 5. Reconcile replayed messages and tool calls by producer identity. +These disconnect steps are requirements, not a description of the current +implementation. + +## Retention + +`event_retention_days` also bounds raw structured ACP journal payloads. Online +maintenance replaces expired payload rows with compact identity tombstones in +bounded batches. A tombstone retains the original sequence and a replay-contract +fingerprint, allowing exact retries to remain idempotent and conflicting reuse +to fail closed without retaining messages, thoughts, raw tool input/output, or +other source payloads. Tombstones are intentionally not deleted automatically: +removing them would make a late replay indistinguishable from a new event. +SQLite secure deletion is enabled for this bounded cleanup transaction, but WAL +files, filesystem snapshots, and backups retain their own operator-managed +lifecycle and are not a cryptographic erasure guarantee. + ## Cross-repository requirements Herdr needs an ACP-aware launch or proxy surface that exposes enough private metadata for Tendwire to bind an ACP session to an existing logical pane. The binding must survive terminal/session churn without making ACP identity a -public continuity input. +public continuity input. It must also identify adapter executable/version, +session-open mode, working directory, and binding generation without exposing +those values on public APIs. Herdres needs optional presentations for sanitized tool and plan progress. It does not ingest ACP directly: it continues polling Tendwire's neutral outbox so @@ -126,7 +157,9 @@ the agent protocol. ## Rollout gates -Promotion proceeds `legacy` -> `acp_shadow` -> `acp_preferred`. The following +Promotion remains blocked at the default `legacy` posture. When the missing +runtime and Herdr prerequisites exist, it may proceed `legacy` -> `acp_shadow` +-> `acp_preferred`. The following must pass before `acp_required` is considered: - no missing or duplicated user/final messages across adapter restarts; diff --git a/src/tendwire/config.py b/src/tendwire/config.py index 9ab162b..1c62b37 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -21,7 +21,7 @@ ) ACP_THOUGHT_POLICIES = frozenset({"disabled", "private_summary", "private_all"}) DEFAULT_TURN_MODEL = "observed" -DEFAULT_AGENT_EVENT_SOURCE = "acp_preferred" +DEFAULT_AGENT_EVENT_SOURCE = "legacy" DEFAULT_ACP_THOUGHT_POLICY = "private_summary" DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS = 30.0 DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS = 5.0 diff --git a/src/tendwire/core/agent_events.py b/src/tendwire/core/agent_events.py index cef01c0..cc27d4f 100644 --- a/src/tendwire/core/agent_events.py +++ b/src/tendwire/core/agent_events.py @@ -46,11 +46,11 @@ } ) AGENT_EVENT_VISIBILITIES = frozenset({"private", "public"}) -AGENT_EVENT_MAX_PAYLOAD_BYTES = 64 * 1024 +AGENT_EVENT_MAX_PAYLOAD_BYTES = 16 * 1024 * 1024 AGENT_EVENT_MAX_PUBLIC_PAYLOAD_BYTES = 64 * 1024 -AGENT_EVENT_MAX_TEXT_CHARS = 32 * 1024 -AGENT_EVENT_MAX_COLLECTION_ITEMS = 256 -AGENT_EVENT_MAX_TOTAL_ITEMS = 4096 +AGENT_EVENT_MAX_TEXT_CHARS = 4 * 1024 * 1024 +AGENT_EVENT_MAX_COLLECTION_ITEMS = 4096 +AGENT_EVENT_MAX_TOTAL_ITEMS = 32768 AGENT_EVENT_MAX_DEPTH = 12 AGENT_EVENT_MAX_IDENTIFIER_CHARS = 2048 AGENT_EVENT_QUERY_DEFAULT_LIMIT = 100 diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index f1b4f86..f2f433b 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -604,8 +604,10 @@ def start(self) -> None: self._start_acp_runtime() - scheduler = self.hooks.turn_scheduler_factory(self.config) - self._turn_scheduler = scheduler + scheduler = None + if self.config.agent_event_source != "acp_required": + scheduler = self.hooks.turn_scheduler_factory(self.config) + self._turn_scheduler = scheduler api = TendwireDaemonAPI( get_snapshot=self.get_snapshot, @@ -639,10 +641,11 @@ def start(self) -> None: if backend is not None else None ) - if callable(callback_setter): + if callable(callback_setter) and scheduler is not None: callback_setter(scheduler.request_refresh) - scheduler.start() - scheduler.request_refresh() + if scheduler is not None: + scheduler.start() + scheduler.request_refresh() except Exception: self.stop_event.set() backend = self._event_backend diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index d3b26eb..d80bac7 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -145,7 +145,7 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS -STORE_SCHEMA_VERSION = 23 +STORE_SCHEMA_VERSION = 24 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS _LEGACY_TURN_CLAIM_HARD_TTL_SECONDS = 86_400 TURN_CHANGE_RETENTION_DAYS = 7 @@ -1599,7 +1599,7 @@ def _record_response_size( THEN json_type(private_payload_json) = 'object' ELSE 0 END ), CHECK ( - length(CAST(private_payload_json AS BLOB)) <= 65536 + length(CAST(private_payload_json AS BLOB)) <= 16777216 ), CHECK ( CASE WHEN json_valid(public_payload_json) @@ -1658,6 +1658,27 @@ def _record_response_size( ), ) +CREATE_AGENT_EVENT_TOMBSTONES_TABLE = """ +CREATE TABLE IF NOT EXISTS agent_event_tombstones ( + host_id TEXT NOT NULL, + event_id TEXT NOT NULL, + sequence INTEGER NOT NULL CHECK (sequence >= 1), + replay_fingerprint TEXT NOT NULL CHECK (length(replay_fingerprint) = 64), + retired_at TEXT NOT NULL CHECK (length(retired_at) BETWEEN 20 AND 40), + PRIMARY KEY (host_id, event_id), + CHECK (length(host_id) BETWEEN 1 AND 2048), + CHECK (instr(host_id, char(0)) = 0), + CHECK (length(event_id) = 64) +); +""" + +CREATE_AGENT_EVENT_TOMBSTONE_INDEXES = ( + ( + "CREATE INDEX IF NOT EXISTS idx_agent_event_tombstones_host_sequence " + "ON agent_event_tombstones(host_id, sequence)" + ), +) + CREATE_PR6_TABLES = ( CREATE_EVENTS_TABLE, CREATE_SPACES_TABLE, @@ -13379,6 +13400,35 @@ def _migrate_v22_to_v23_conn(conn: sqlite3.Connection) -> None: conn.execute(statement) +def _migrate_v23_to_v24_conn(conn: sqlite3.Connection) -> None: + """Raise private event bounds and add replay-preserving retention tombstones.""" + conn.execute("ALTER TABLE agent_events RENAME TO agent_events_v23") + conn.execute(CREATE_AGENT_EVENTS_TABLE) + conn.execute( + """ + INSERT INTO agent_events ( + sequence, host_id, event_id, kind, source, worker_id, visibility, + source_session_id, source_turn_id, source_item_id, + source_message_id, source_event_id, source_sequence, observed_at, + payload_fingerprint, private_payload_json, public_payload_json + ) + SELECT + sequence, host_id, event_id, kind, source, worker_id, visibility, + source_session_id, source_turn_id, source_item_id, + source_message_id, source_event_id, source_sequence, observed_at, + payload_fingerprint, private_payload_json, public_payload_json + FROM agent_events_v23 + ORDER BY sequence + """ + ) + conn.execute("DROP TABLE agent_events_v23") + for statement in CREATE_AGENT_EVENT_INDEXES: + conn.execute(statement) + conn.execute(CREATE_AGENT_EVENT_TOMBSTONES_TABLE) + for statement in CREATE_AGENT_EVENT_TOMBSTONE_INDEXES: + conn.execute(statement) + + MIGRATIONS: tuple[Migration, ...] = ( Migration(0, 1, _migrate_v0_to_v1_conn), Migration(1, 2, _migrate_v1_to_v2_conn), @@ -13403,6 +13453,7 @@ def _migrate_v22_to_v23_conn(conn: sqlite3.Connection) -> None: Migration(20, 21, _migrate_v20_to_v21_conn), Migration(21, 22, _migrate_v21_to_v22_conn), Migration(22, 23, _migrate_v22_to_v23_conn), + Migration(23, 24, _migrate_v23_to_v24_conn), ) @@ -13457,6 +13508,7 @@ def _create_current_schema_conn(conn: sqlite3.Connection) -> None: conn.execute(CREATE_HERDR_TURN_WATERMARKS_TABLE) conn.execute(CREATE_HERDR_TURN_COMPLETIONS_TABLE) conn.execute(CREATE_AGENT_EVENTS_TABLE) + conn.execute(CREATE_AGENT_EVENT_TOMBSTONES_TABLE) for statement in CREATE_COMMAND_RECEIPT_INDEXES: conn.execute(statement) for statement in CREATE_WORKER_BINDING_INDEXES: @@ -13480,6 +13532,8 @@ def _create_current_schema_conn(conn: sqlite3.Connection) -> None: conn.execute(statement) for statement in CREATE_AGENT_EVENT_INDEXES: conn.execute(statement) + for statement in CREATE_AGENT_EVENT_TOMBSTONE_INDEXES: + conn.execute(statement) for statement in CREATE_ATTENTION_LIFECYCLE_INDEXES: conn.execute(statement) for statement in CREATE_TURN_CONTENT_REVISION_INDEXES: @@ -13689,6 +13743,28 @@ def _agent_event_conflicts(existing: StoredAgentEvent, incoming: AgentEvent) -> ) +def _agent_event_replay_fingerprint(event: AgentEvent) -> str: + """Fingerprint the replay contract while excluding source observation time.""" + contract = { + "event_id": event.event_id, + "kind": event.kind, + "source": event.source, + "worker_id": event.worker_id, + "visibility": event.visibility, + "source_session_id": event.source_session_id, + "source_turn_id": event.source_turn_id, + "source_item_id": event.source_item_id, + "source_message_id": event.source_message_id, + "source_event_id": event.source_event_id, + "source_sequence": event.source_sequence, + "payload_fingerprint": event.payload_fingerprint, + "public_payload_fingerprint": hashlib.sha256( + _canonical_json(event.public_payload).encode("utf-8") + ).hexdigest(), + } + return hashlib.sha256(_canonical_json(contract).encode("utf-8")).hexdigest() + + def _canonical_agent_event_for_append(event: AgentEvent) -> AgentEvent: if not isinstance(event, AgentEvent): raise ValueError("event must be an AgentEvent") @@ -13716,6 +13792,22 @@ def _append_agent_event_conn( host_id: str, event: AgentEvent, ) -> AppendAgentEventResult: + tombstone = conn.execute( + """ + SELECT sequence, replay_fingerprint + FROM agent_event_tombstones + WHERE host_id = ? AND event_id = ? + """, + (host_id, event.event_id), + ).fetchone() + if tombstone is not None: + if str(tombstone[1]) != _agent_event_replay_fingerprint(event): + raise AgentEventIdentityConflict(event.event_id) + return AppendAgentEventResult( + sequence=int(tombstone[0]), + event_id=event.event_id, + inserted=False, + ) private_json = _canonical_json(event.payload) public_json = _canonical_json(event.public_payload) cursor = conn.execute( @@ -17256,6 +17348,115 @@ def cleanup_event_retention( })) +def cleanup_agent_event_retention( + db_path: Path, + host_id: str, + *, + retention_days: int, + now: str | None = None, + dry_run: bool = False, + batch_size: int = 100, +) -> dict[str, Any]: + """Retire private event payloads while retaining compact replay identities.""" + days = max(1, int(retention_days)) + bounded_batch = max(1, min(int(batch_size), 1_000)) + cutoff_at = _utc_cutoff(retention_days=days, now=now) + base = { + "schema_version": 1, + "host_id": str(host_id), + "dry_run": bool(dry_run), + "retention_days": days, + "cutoff_at": cutoff_at, + "batch_size": bounded_batch, + } + if not _sqlite_store_exists(db_path): + return dict(sanitize_public_value({ + **base, + "ok": False, + "status": "store_unavailable", + "examined": 0, + "deleted": 0, + "tombstoned": 0, + "remaining_candidates": False, + })) + with _connect(db_path, isolation_level=None) as conn: + _ensure_schema(conn) + # Retention is a privacy boundary. Overwrite retired cells in the main + # database where SQLite can do so; WAL files and backups retain their + # independent operator-managed lifecycle. + conn.execute("PRAGMA secure_delete=ON") + conn.execute("BEGIN IMMEDIATE") + try: + rows = conn.execute( + _AGENT_EVENT_SELECT + + " WHERE host_id = ? AND observed_at < ?" + + " ORDER BY observed_at, sequence LIMIT ?", + (str(host_id), cutoff_at, bounded_batch + 1), + ).fetchall() + candidates = rows[:bounded_batch] + deleted = 0 + if candidates and not dry_run: + retired_at = utc_timestamp() + for row in candidates: + stored = _agent_event_from_row(row) + conn.execute( + """ + INSERT INTO agent_event_tombstones ( + host_id, event_id, sequence, + replay_fingerprint, retired_at + ) VALUES (?, ?, ?, ?, ?) + """, + ( + stored.host_id, + stored.event.event_id, + stored.sequence, + _agent_event_replay_fingerprint(stored.event), + retired_at, + ), + ) + sequences = [int(row[0]) for row in candidates] + placeholders = ",".join("?" for _ in sequences) + deleted = int( + conn.execute( + f"DELETE FROM agent_events WHERE host_id = ? " + f"AND sequence IN ({placeholders})", + (str(host_id), *sequences), + ).rowcount + or 0 + ) + if deleted != len(candidates): + raise StoreSchemaError("agent_event_retention_delete_mismatch") + if dry_run: + remaining = len(rows) > bounded_batch + conn.rollback() + else: + remaining = bool( + conn.execute( + """ + SELECT 1 FROM agent_events + WHERE host_id = ? AND observed_at < ? + LIMIT 1 + """, + (str(host_id), cutoff_at), + ).fetchone() + ) + conn.commit() + except Exception: + conn.rollback() + raise + examined = len(candidates) + retired = examined if dry_run else deleted + return dict(sanitize_public_value({ + **base, + "ok": True, + "status": "ok", + "examined": examined, + "deleted": retired, + "tombstoned": retired, + "remaining_candidates": remaining, + })) + + def _turn_content_retention_candidates_conn( conn: sqlite3.Connection, *, @@ -18765,6 +18966,14 @@ def run_store_maintenance( dry_run=dry_run, batch_size=event_batch_size, ) + agent_events = cleanup_agent_event_retention( + db_path, + host_id, + retention_days=retention_days, + now=now, + dry_run=dry_run, + batch_size=event_batch_size, + ) snapshots = cleanup_snapshot_retention( db_path, retention_days=snapshot_retention_days, @@ -18835,6 +19044,7 @@ def run_store_maintenance( ) ok = ( bool(retention.get("ok")) + and bool(agent_events.get("ok")) and bool(snapshots.get("ok")) and bool(outbox.get("ok")) and bool(final_retention.get("ok")) @@ -18859,6 +19069,22 @@ def run_store_maintenance( retention.get("remaining_candidates") ), }, + "agent_events": { + "retention_days": int( + agent_events.get("retention_days") or retention_days + ), + "cutoff_at": agent_events.get("cutoff_at"), + "batch_size": int( + agent_events.get("batch_size") or event_batch_size + ), + "examined": int(agent_events.get("examined") or 0), + "deleted": int(agent_events.get("deleted") or 0), + "tombstoned": int(agent_events.get("tombstoned") or 0), + "remaining_candidates": bool( + agent_events.get("remaining_candidates") + ), + "replay_identity_retained": True, + }, "snapshots": { "scope": "database", "retention_days": int( diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py index 1083f8c..62aefcb 100644 --- a/tests/test_agent_events.py +++ b/tests/test_agent_events.py @@ -490,6 +490,57 @@ def test_database_constraints_and_indexes_cover_public_and_source_identity( ) +def test_retention_removes_private_payload_but_preserves_replay_identity( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + old = replace( + _message_event(sequence=1, text="private historical payload", visibility="private"), + observed_at="2026-06-01T00:00:00+00:00", + ) + recent = replace( + _message_event(sequence=2, text="recent", visibility="private"), + observed_at="2026-07-30T00:00:00+00:00", + ) + inserted = store_sqlite.append_agent_event(db_path, "host-1", old) + store_sqlite.append_agent_event(db_path, "host-1", recent) + + result = store_sqlite.cleanup_agent_event_retention( + db_path, + "host-1", + retention_days=7, + now="2026-07-31T00:00:00+00:00", + ) + + assert result["deleted"] == result["tombstoned"] == 1 + assert [item.event.payload["text"] for item in store_sqlite.list_agent_events(db_path, "host-1")] == ["recent"] + with sqlite3.connect(db_path) as conn: + tombstone = conn.execute( + "SELECT sequence, length(replay_fingerprint) " + "FROM agent_event_tombstones WHERE host_id = ? AND event_id = ?", + ("host-1", old.event_id), + ).fetchone() + encoded = "\n".join(conn.iterdump()) + assert tombstone == (inserted.sequence, 64) + assert "private historical payload" not in encoded + + replay = store_sqlite.append_agent_event(db_path, "host-1", old) + assert replay.inserted is False + assert replay.sequence == inserted.sequence + with pytest.raises(AgentEventIdentityConflict): + store_sqlite.append_agent_event( + db_path, + "host-1", + _message_event(sequence=1, text="changed", visibility="private"), + ) + + +def test_journal_accepts_acp_sized_private_text(tmp_path: Path) -> None: + event = _message_event(sequence=1, text="x" * (64 * 1024), visibility="private") + result = store_sqlite.append_agent_event(tmp_path / "store.db", "host-1", event) + assert result.inserted is True + + @pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) def test_agent_event_schema_migrates_from_every_prior_version( tmp_path: Path, @@ -527,7 +578,7 @@ def test_v21_migration_is_idempotent_and_preserves_existing_store( store_sqlite.init_store(db_path) store_sqlite.init_store(db_path) with sqlite3.connect(db_path) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == (23,) + assert conn.execute("PRAGMA user_version").fetchone() == (24,) columns = { str(row[1]) for row in conn.execute("PRAGMA table_info(agent_events)") } @@ -588,7 +639,7 @@ def test_v22_migration_rekeys_legacy_event_identity_without_losing_sequence( "SELECT sequence, event_id FROM agent_events" ).fetchone() assert row == (19, event.event_id) - assert conn.execute("PRAGMA user_version").fetchone() == (23,) + assert conn.execute("PRAGMA user_version").fetchone() == (24,) replay = store_sqlite.append_agent_event(db_path, "host-1", event) assert replay.inserted is False diff --git a/tests/test_config.py b/tests/test_config.py index 68ace5b..da87472 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -46,7 +46,7 @@ def test_acp_event_source_defaults_to_preferred_with_private_summaries( config = load_config() - assert config.agent_event_source == DEFAULT_AGENT_EVENT_SOURCE == "acp_preferred" + assert config.agent_event_source == DEFAULT_AGENT_EVENT_SOURCE == "legacy" assert config.acp_thought_policy == DEFAULT_ACP_THOUGHT_POLICY == "private_summary" assert config.acp_request_timeout_seconds == DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS == 30.0 assert config.acp_shutdown_timeout_seconds == DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS == 5.0 diff --git a/tests/test_daemon_acp.py b/tests/test_daemon_acp.py index f615c0b..a2262ac 100644 --- a/tests/test_daemon_acp.py +++ b/tests/test_daemon_acp.py @@ -238,9 +238,6 @@ def runtime_factory(config: Config, stop_event: threading.Event) -> _Runtime: "observe", "acp_factory", "acp_start", - "scheduler_factory", - "scheduler_start", - "scheduler_request", ] assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) health = daemon.get_health() @@ -276,10 +273,9 @@ def runtime_factory(config: Config, stop_event: threading.Event) -> _Runtime: finally: daemon.stop() - assert calls[-3:] == [ + assert calls[-2:] == [ "acp_stop:1.25", "acp_join:1.25", - "scheduler_stop:6.0", ] @@ -372,14 +368,12 @@ def test_optional_acp_start_failure_is_cleaned_up_before_legacy_fallback( daemon.stop() -def test_scheduler_start_failure_also_stops_and_joins_acp(tmp_path: Path) -> None: +def test_required_acp_never_constructs_legacy_scheduler(tmp_path: Path) -> None: calls: list[str] = [] runtime = _Runtime(calls) - class FailingScheduler(_Scheduler): - def start(self) -> None: - self.calls.append("scheduler_start") - raise RuntimeError("sentinel scheduler failure") + def forbidden_scheduler(_config: Config) -> _Scheduler: + raise AssertionError("acp_required must never construct legacy ingestion") daemon = TendwireDaemon( _config(tmp_path, "acp_required"), @@ -387,17 +381,14 @@ def start(self) -> None: tmp_path, calls, acp_runtime_factory=lambda _config, _stop_event: runtime, - scheduler_factory=lambda _config: FailingScheduler(calls), + scheduler_factory=forbidden_scheduler, ), ) - with pytest.raises(RuntimeError, match="sentinel scheduler failure"): - daemon.start() + daemon.start() + try: + assert calls == ["init_store", "observe", "acp_start"] + finally: + daemon.stop() - assert calls[-3:] == [ - "scheduler_stop:6.0", - "acp_stop:1.25", - "acp_join:1.25", - ] - assert daemon._acp_runtime is None - assert not os.path.lexists(tmp_path / "daemon.sock") + assert calls[-2:] == ["acp_stop:1.25", "acp_join:1.25"] diff --git a/tests/test_store.py b/tests/test_store.py index e6c733e..167d4a5 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -10278,7 +10278,7 @@ def test_v13_migration_repairs_nonpositive_turn_sequences_and_blocks_recurrence( init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 23 + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 24 assert conn.execute( """ SELECT turn_id, list_sequence From eebab98b223799b31161e347ab8b1c09f6b1008b Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:24:20 +0800 Subject: [PATCH 20/83] fix(acp): make projection ingestion transactional --- src/tendwire/backends/acp_ingestion.py | 154 ++++--- src/tendwire/backends/acp_projection.py | 578 +++++++++++++++++++----- tests/test_acp_ingestion.py | 138 ++++-- tests/test_acp_projection.py | 299 +++++++++--- 4 files changed, 893 insertions(+), 276 deletions(-) diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py index 11761ff..a51884d 100644 --- a/src/tendwire/backends/acp_ingestion.py +++ b/src/tendwire/backends/acp_ingestion.py @@ -22,7 +22,7 @@ append_agent_event_for_binding, apply_turn_refresh, ) -from .acp_projection import AcpEventProjector +from .acp_projection import AcpEventProjector, AcpProjectionCheckpoint AppendEvent = Callable[..., AppendBoundAgentEventResult] @@ -148,14 +148,27 @@ def ingest_update( ) if thought_rejection is not None: return AcpIngestionResult("thought", ignored_reason=thought_rejection) - canonical = self.projector.normalize_session_update( - notification, - source_event_id=source_event_id, - replay=replay, - ) + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + if self._source_turn_id is None and update_kind in _TURN_SCOPED_UPDATES: + self.start_turn() + try: + canonical = self.projector.normalize_session_update( + notification, + source_event_id=source_event_id, + replay=replay, + ) + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise if canonical is None: + self._restore_speculation(checkpoint, prior_turn_state) return AcpIngestionResult(None, ignored_reason="unsupported_or_duplicate") - return self._accept(canonical) + return self._accept( + canonical, + checkpoint=checkpoint, + prior_turn_state=prior_turn_state, + ) def ingest_permission_request( self, @@ -175,14 +188,27 @@ def ingest_permission_request( return AcpIngestionResult(None, ignored_reason=mismatch) if self._turn_complete: return AcpIngestionResult(None, ignored_reason="turn_already_complete") - canonical = self.projector.normalize_permission_request( - request, - source_event_id=source_event_id, - replay=replay, - ) + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + if self._source_turn_id is None: + self.start_turn() + try: + canonical = self.projector.normalize_permission_request( + request, + source_event_id=source_event_id, + replay=replay, + ) + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise if canonical is None: + self._restore_speculation(checkpoint, prior_turn_state) return AcpIngestionResult(None, ignored_reason="duplicate") - return self._accept(canonical) + return self._accept( + canonical, + checkpoint=checkpoint, + prior_turn_state=prior_turn_state, + ) def mark_prompt_complete(self) -> AcpIngestionResult: """Finalize the current text projection after ``session/prompt`` returns.""" @@ -204,53 +230,56 @@ def mark_prompt_complete(self) -> AcpIngestionResult: ignored_reason="stale_binding" if turn.stale_binding else None, ) - def _accept(self, canonical: Mapping[str, Any]) -> AcpIngestionResult: + def _accept( + self, + canonical: Mapping[str, Any], + *, + checkpoint: AcpProjectionCheckpoint, + prior_turn_state: tuple[int, str | None, bool], + ) -> AcpIngestionResult: kind = str(canonical.get("kind") or "") if kind == "thought" and self.config.acp_thought_policy == "disabled": + self._restore_speculation(checkpoint, prior_turn_state) return AcpIngestionResult(kind, ignored_reason="thought_policy_disabled") - if self._source_turn_id is None and kind in { - "user_message", - "agent_message", - "thought", - "tool_call", - "tool_call_update", - "plan", - }: - self.start_turn() - - payload = canonical.get("payload") - if not isinstance(payload, Mapping): - raise ValueError("canonical ACP event payload must be a mapping") - sequence = canonical.get("sequence") - if isinstance(sequence, bool) or not isinstance(sequence, int) or sequence < 0: - raise ValueError("canonical ACP event sequence must be nonnegative") - explicit_event_id = canonical.get("source_event_id") - source_id = ( - str(explicit_event_id) - if explicit_event_id is not None and str(explicit_event_id) - else f"stream:{self.stream_generation}:{sequence}" - ) - event = agent_event( - kind=kind, - source="acp", - worker_id=self.binding.worker_id, - payload=payload, - source_session_id=self.session_id, - source_turn_id=self._source_turn_id, - source_item_id=_source_item_id(kind, payload), - source_message_id=_source_message_id(kind, payload), - source_event_id=source_id, - source_sequence=sequence, - # The complete structured journal is private initially. Public and - # connector views require a separate explicit sanitizing projection. - visibility="private", - ) - appended = self._append_event( - Path(self.config.db_path), - self.config.host_id, - event, - expected_binding=self.binding, - ) + try: + payload = canonical.get("payload") + if not isinstance(payload, Mapping): + raise ValueError("canonical ACP event payload must be a mapping") + sequence = canonical.get("sequence") + if isinstance(sequence, bool) or not isinstance(sequence, int) or sequence < 0: + raise ValueError("canonical ACP event sequence must be nonnegative") + explicit_event_id = canonical.get("source_event_id") + source_id = ( + str(explicit_event_id) + if explicit_event_id is not None and str(explicit_event_id) + else f"stream:{self.stream_generation}:{sequence}" + ) + event = agent_event( + kind=kind, + source="acp", + worker_id=self.binding.worker_id, + payload=payload, + source_session_id=self.session_id, + source_turn_id=self._source_turn_id, + source_item_id=_source_item_id(kind, payload), + source_message_id=_source_message_id(kind, payload), + source_event_id=source_id, + source_sequence=sequence, + # The complete structured journal is private initially. Public and + # connector views require a separate explicit sanitizing projection. + visibility="private", + ) + appended = self._append_event( + Path(self.config.db_path), + self.config.host_id, + event, + expected_binding=self.binding, + ) + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + if appended.status != "inserted": + self._restore_speculation(checkpoint, prior_turn_state) turn: TurnRefreshApplyResult | None = None if ( @@ -276,6 +305,17 @@ def _accept(self, canonical: Mapping[str, Any]) -> AcpIngestionResult: ), ) + def _turn_state(self) -> tuple[int, str | None, bool]: + return self._turn_ordinal, self._source_turn_id, self._turn_complete + + def _restore_speculation( + self, + checkpoint: AcpProjectionCheckpoint, + prior_turn_state: tuple[int, str | None, bool], + ) -> None: + self.projector.restore_session(checkpoint) + self._turn_ordinal, self._source_turn_id, self._turn_complete = prior_turn_state + def _project_turn(self, content: Mapping[str, Any]) -> TurnRefreshApplyResult: return self._apply_turn( Path(self.config.db_path), diff --git a/src/tendwire/backends/acp_projection.py b/src/tendwire/backends/acp_projection.py index 333bba6..a6ec4f5 100644 --- a/src/tendwire/backends/acp_projection.py +++ b/src/tendwire/backends/acp_projection.py @@ -15,6 +15,8 @@ import hashlib import json +import math +import os from collections.abc import Mapping from copy import deepcopy from dataclasses import dataclass, field @@ -57,6 +59,33 @@ } _MAX_IDENTIFIER_CHARS: Final[int] = 2048 _MAX_SOURCE_ID_CHARS: Final[int] = 512 +_TOOL_KINDS: Final[frozenset[str]] = frozenset( + { + "read", + "edit", + "delete", + "move", + "search", + "execute", + "think", + "fetch", + "switch_mode", + "other", + } +) +_TOOL_STATUSES: Final[frozenset[str]] = frozenset( + {"pending", "in_progress", "completed", "failed"} +) +_PLAN_PRIORITIES: Final[frozenset[str]] = frozenset({"high", "medium", "low"}) +_PLAN_STATUSES: Final[frozenset[str]] = frozenset( + {"pending", "in_progress", "completed"} +) +_PERMISSION_KINDS: Final[frozenset[str]] = frozenset( + {"allow_once", "allow_always", "reject_once", "reject_always"} +) +_CONTENT_TYPES: Final[frozenset[str]] = frozenset( + {"text", "image", "audio", "resource_link", "resource"} +) class AcpProjectionError(ValueError): @@ -67,6 +96,7 @@ class AcpProjectionError(ValueError): class _MessageAssembly: message_id: str text: str = "" + explicit: bool = False @dataclass @@ -84,6 +114,18 @@ class _SessionState: seen_source_events: dict[str, str] = field(default_factory=dict) retained_bytes: int = 0 complete: bool = False + active_message: tuple[str, str] | None = None + last_update_name: str | None = None + implicit_message_ordinals: dict[str, int] = field(default_factory=dict) + replaced_state: _SessionState | None = field(default=None, repr=False) + + +@dataclass(frozen=True) +class AcpProjectionCheckpoint: + """Opaque rollback point for one session's in-memory projection state.""" + + session_id: str + state: _SessionState | None class AcpEventProjector: @@ -92,9 +134,10 @@ class AcpEventProjector: The instance is intentionally in-memory. ``dedupe_hint`` is emitted on every event so a durable caller can deduplicate replays across process restarts. Within one instance, events with an explicit protocol/transport - identifier (``source_event_id`` or a recognized ``_meta`` key) are dropped - when repeated. Content hashes are hints only: identical adjacent chunks - can be legitimate and are therefore never blindly discarded. + identifier supplied by the transport as ``source_event_id`` are dropped + when repeated. ACP ``_meta`` is opaque and is never trusted as identity. + Content hashes are hints only: identical adjacent chunks can be legitimate + and are therefore never blindly discarded. """ def __init__( @@ -170,10 +213,10 @@ def normalize_session_update( label="ACP session update", max_bytes=self._max_event_bytes, ) - state, is_new_session = self._pending_session(session_id) + _extension_metadata(params) + _validate_supported_update(update_name, update) + state, _is_new_session = self._pending_session(session_id) explicit_id = _explicit_source_event_id(source_event_id) - if explicit_id is None: - explicit_id = _source_event_id(notification, params, update) replay_digest = _event_digest(kind, update) if explicit_id is not None: previous_digest = state.seen_source_events.get(explicit_id) @@ -200,18 +243,28 @@ def normalize_session_update( payload = self._normalize_usage(state, update) else: payload = self._normalize_session_info(state, update) - extension_meta = { - **_extension_metadata(params), - **_extension_metadata(update), - } + extension_meta = _scoped_metadata(params=params, update=update) if extension_meta: payload["extensions"] = extension_meta state.sequence += 1 + state.last_update_name = update_name + if kind not in _MESSAGE_KINDS and state.active_message is not None: + active_kind, active_id = state.active_message + active = next( + ( + item + for item in state.messages[active_kind] + if item.message_id == active_id + ), + None, + ) + if active is not None and not active.explicit: + state.active_message = None if explicit_id is not None: state.seen_source_events[explicit_id] = replay_digest - if is_new_session: - self._sessions[session_id] = state + state.replaced_state = None + self._sessions[session_id] = state return _canonical_event( session_id=session_id, sequence=state.sequence, @@ -248,12 +301,10 @@ def normalize_permission_request( label="ACP permission request", max_bytes=self._max_event_bytes, ) - state, is_new_session = self._pending_session(session_id) + _extension_metadata(params) + _validate_tool_update(tool_call, label="ACP permission toolCall") + state, _is_new_session = self._pending_session(session_id) explicit_id = _explicit_source_event_id(source_event_id) - if explicit_id is None: - explicit_id = _jsonrpc_request_id(request) or _source_event_id( - request, params, tool_call - ) options = params.get("options") if not isinstance(options, list) or not options: raise AcpProjectionError("ACP permission request options must be non-empty") @@ -290,20 +341,30 @@ def normalize_permission_request( ) state.tools[tool_call_id] = snapshot state.sequence += 1 + state.last_update_name = "tool_call_update" + if state.active_message is not None: + active_kind, active_id = state.active_message + active = next( + ( + item + for item in state.messages[active_kind] + if item.message_id == active_id + ), + None, + ) + if active is not None and not active.explicit: + state.active_message = None if explicit_id is not None: state.seen_source_events[explicit_id] = replay_digest - if is_new_session: - self._sessions[session_id] = state + state.replaced_state = None + self._sessions[session_id] = state payload = { "tool_call_id": tool_call_id, "changes": _without_discriminator(tool_call), "snapshot": deepcopy(snapshot), "permission": deepcopy(snapshot["permission"]), } - extension_meta = { - **_extension_metadata(params), - **_extension_metadata(tool_call), - } + extension_meta = _scoped_metadata(params=params, toolCall=tool_call) if extension_meta: payload["extensions"] = extension_meta return _canonical_event( @@ -355,19 +416,44 @@ def project_turn_content( def mark_turn_complete(self, session_id: str) -> dict[str, Any]: """Mark the current ACP prompt turn complete and return legacy content.""" - state = self._session(session_id) + state, _is_new_session = self._pending_session(session_id) state.complete = True + state.replaced_state = None + self._sessions[session_id] = state return self.project_turn_content(session_id) def reset_turn(self, session_id: str) -> None: """Start a fresh prompt turn while preserving session-level ACP state.""" - state = self._session(session_id) + state, _is_new_session = self._pending_session(session_id) state.retained_bytes = max( 0, state.retained_bytes - _messages_state_bytes(state.messages) ) state.messages = {kind: [] for kind in _MESSAGE_KINDS} state.complete = False + state.active_message = None + state.last_update_name = None + state.implicit_message_ordinals = {} + state.replaced_state = None + self._sessions[session_id] = state + + def checkpoint_session(self, session_id: str) -> AcpProjectionCheckpoint: + """Capture one session so a failed durable append can be rolled back.""" + + return AcpProjectionCheckpoint( + session_id=session_id, + state=self._sessions.get(session_id), + ) + + def restore_session(self, checkpoint: AcpProjectionCheckpoint) -> None: + """Restore a checkpoint created before a speculative normalization.""" + + if not isinstance(checkpoint, AcpProjectionCheckpoint): + raise TypeError("checkpoint must be an AcpProjectionCheckpoint") + if checkpoint.state is None: + self._sessions.pop(checkpoint.session_id, None) + else: + self._sessions[checkpoint.session_id] = checkpoint.state def drop_session(self, session_id: str) -> bool: """Release all in-memory state after the owning ACP session is closed.""" @@ -403,25 +489,47 @@ def _normalize_message( kind: str, update: Mapping[str, Any], ) -> dict[str, Any]: - content = update.get("content") - if not isinstance(content, Mapping): - raise AcpProjectionError(f"ACP {kind} update is missing content") + content = update["content"] + assert isinstance(content, Mapping) message_id_value = update.get("messageId") assemblies = state.messages[kind] if message_id_value is not None: message_id = _identifier(message_id_value, "messageId") + explicit = True else: - # ACP stable v1 chunks do not require message IDs. Keep their - # assembly separate from adapter extensions that do provide IDs. - message_id = f"implicit-{kind}-1" - assembly = next( - (item for item in assemblies if item.message_id == message_id), None - ) + explicit = False + active = state.active_message + if active is not None and active[0] == kind: + candidate = next( + ( + item + for item in assemblies + if item.message_id == active[1] and not item.explicit + ), + None, + ) + else: + candidate = None + if state.last_update_name == update.get("sessionUpdate") and candidate is not None: + message_id = candidate.message_id + else: + ordinal = state.implicit_message_ordinals.get(kind, 0) + 1 + message_id = f"implicit-{kind}-{ordinal}" + active_key = state.active_message + assembly = None + if active_key == (kind, message_id): + assembly = next( + (item for item in assemblies if item.message_id == message_id), None + ) + elif explicit and any( + item.explicit and item.message_id == message_id + for items in state.messages.values() + for item in items + ): + raise AcpProjectionError("ACP messageId was reused after a message boundary") text_delta = content.get("text") if content.get("type") == "text" else None - if not isinstance(text_delta, str): - text_delta = "" + text_delta = text_delta if isinstance(text_delta, str) else "" content_copy = _content_payload(content) - extension_meta = _extension_metadata(update) previous_text = assembly.text if assembly is not None else "" assembled_text = previous_text + text_delta if len(assembled_text) > self._max_text_chars_per_message: @@ -433,18 +541,24 @@ def _normalize_message( state, len(message_id.encode("utf-8")) + len(text_delta.encode("utf-8")), ) - assembly = _MessageAssembly(message_id=message_id, text=assembled_text) + assembly = _MessageAssembly( + message_id=message_id, + text=assembled_text, + explicit=explicit, + ) assemblies.append(assembly) + if not explicit: + state.implicit_message_ordinals[kind] = int(message_id.rsplit("-", 1)[1]) else: self._reserve_state(state, len(text_delta.encode("utf-8"))) assembly.text = assembled_text + state.active_message = (kind, message_id) return { "message_id": message_id, "content": content_copy, "text_delta": text_delta, "assembled_text": assembled_text, "message_index": assemblies.index(assembly), - **({"extensions": extension_meta} if extension_meta else {}), } def _normalize_tool( @@ -466,9 +580,6 @@ def _normalize_tool( } if kind == "tool_call_update": payload["changes"] = _without_discriminator(update) - extension_meta = _extension_metadata(update) - if extension_meta: - payload["extensions"] = extension_meta self._reserve_state( state, _json_size(snapshot) - (_json_size(previous) if previous else 0) ) @@ -483,13 +594,8 @@ def _normalize_plan( raise AcpProjectionError("ACP plan update entries must be an array") if len(entries) > self._max_plan_entries: raise AcpProjectionError("ACP plan entry limit exceeded") - if any(not isinstance(entry, Mapping) for entry in entries): - raise AcpProjectionError("ACP plan entry must be an object") - replacement = [deepcopy(dict(entry)) for entry in entries] + replacement = [_normalized_plan_entry(entry) for entry in entries] payload: dict[str, Any] = {"entries": deepcopy(replacement), "snapshot": True} - extension_meta = _extension_metadata(update) - if extension_meta: - payload["extensions"] = extension_meta self._reserve_state(state, _json_size(replacement) - _json_size(state.plan)) state.plan = replacement return payload @@ -497,46 +603,37 @@ def _normalize_plan( def _normalize_usage( self, state: _SessionState, update: Mapping[str, Any] ) -> dict[str, Any]: - replacement = {**state.usage, **_without_discriminator(update)} + replacement = { + "used": update["used"], + "size": update["size"], + **({"cost": deepcopy(update["cost"])} if update.get("cost") is not None else {}), + } if len(replacement) > self._max_state_fields: raise AcpProjectionError("ACP usage state field limit exceeded") self._reserve_state(state, _json_size(replacement) - _json_size(state.usage)) state.usage = replacement payload = deepcopy(state.usage) - extension_meta = _extension_metadata(update) - if extension_meta: - payload["extensions"] = extension_meta return payload def _normalize_session_info( self, state: _SessionState, update: Mapping[str, Any] ) -> dict[str, Any]: # Presence is meaningful: explicit null clears an existing property. - replacement = {**state.info, **_without_discriminator(update)} + changes = { + key: deepcopy(update[key]) for key in ("title", "updatedAt") if key in update + } + replacement = {**state.info, **changes} if len(replacement) > self._max_state_fields: raise AcpProjectionError("ACP session info state field limit exceeded") self._reserve_state(state, _json_size(replacement) - _json_size(state.info)) state.info = replacement payload = deepcopy(state.info) - extension_meta = _extension_metadata(update) - if extension_meta: - payload["extensions"] = extension_meta return payload - def _session(self, session_id: str) -> _SessionState: - state = self._sessions.get(session_id) - if state is not None: - return state - if len(self._sessions) >= self._max_sessions: - raise AcpProjectionError("ACP projector session limit exceeded") - state = _SessionState() - self._sessions[session_id] = state - return state - def _pending_session(self, session_id: str) -> tuple[_SessionState, bool]: state = self._sessions.get(session_id) if state is not None: - return state, False + return _copy_session_state(state), False if len(self._sessions) >= self._max_sessions: raise AcpProjectionError("ACP projector session limit exceeded") return _SessionState(), True @@ -548,7 +645,7 @@ def _reserve_state(self, state: _SessionState, retained_delta: int) -> None: other_retained = sum( item.retained_bytes for item in self._sessions.values() - if item is not state + if item is not state and item is not state.replaced_state ) if other_retained + retained > self._max_total_state_bytes: raise AcpProjectionError("ACP total retained state limit exceeded") @@ -562,6 +659,36 @@ def _unwrap_params(value: Mapping[str, Any]) -> Mapping[str, Any]: return value +def _copy_session_state(state: _SessionState) -> _SessionState: + """Copy mutable indexes while sharing immutable text and untouched snapshots.""" + + return _SessionState( + sequence=state.sequence, + messages={ + kind: [ + _MessageAssembly( + message_id=message.message_id, + text=message.text, + explicit=message.explicit, + ) + for message in messages + ] + for kind, messages in state.messages.items() + }, + tools=dict(state.tools), + plan=state.plan, + usage=state.usage, + info=state.info, + seen_source_events=dict(state.seen_source_events), + retained_bytes=state.retained_bytes, + complete=state.complete, + active_message=state.active_message, + last_update_name=state.last_update_name, + implicit_message_ordinals=dict(state.implicit_message_ordinals), + replaced_state=state, + ) + + def _required_string(value: Mapping[str, Any], key: str) -> str: item = value.get(key) try: @@ -588,40 +715,6 @@ def _explicit_source_event_id(value: Any) -> str | None: return _source_identifier(value, "source_event_id") -def _source_event_id(*values: Mapping[str, Any]) -> str | None: - for value in values: - for key in ("eventId", "event_id", "notificationId", "notification_id"): - candidate = value.get(key) - if _valid_wire_id(candidate): - return _source_identifier(str(candidate), "source event ID") - meta = value.get("_meta") - if isinstance(meta, Mapping): - for key in ("eventId", "event_id", "notificationId", "notification_id"): - candidate = meta.get(key) - if _valid_wire_id(candidate): - return _source_identifier(str(candidate), "source event ID") - return None - - -def _jsonrpc_request_id(value: Mapping[str, Any]) -> str | None: - """Return a request ID, but never mistake a notification field for one.""" - - if value.get("method") != "session/request_permission": - return None - candidate = value.get("id") - if _valid_wire_id(candidate): - # JSON-RPC request IDs and producer notification IDs are separate - # namespaces and commonly both start at small integers. - return _source_identifier(f"request:{candidate}", "JSON-RPC request ID") - return None - - -def _valid_wire_id(value: Any) -> bool: - return not isinstance(value, bool) and isinstance(value, (str, int)) and bool( - str(value) - ) - - def _source_identifier(value: Any, label: str) -> str: identifier = _identifier(value, label) if len(identifier) > _MAX_SOURCE_ID_CHARS: @@ -641,30 +734,48 @@ def _merge_tool_snapshot( previous: Mapping[str, Any] | None, update: Mapping[str, Any] ) -> dict[str, Any]: snapshot = deepcopy(dict(previous)) if previous is not None else {} - snapshot.update(_without_discriminator(update)) + if update.get("sessionUpdate") == "tool_call": + snapshot.setdefault("kind", "other") + snapshot.setdefault("status", "pending") + snapshot.setdefault("content", []) + snapshot.setdefault("locations", []) + for key in ( + "toolCallId", + "title", + "kind", + "status", + "content", + "locations", + "rawInput", + "rawOutput", + ): + if key in update and update[key] is not None: + snapshot[key] = deepcopy(update[key]) return snapshot def _content_payload(content: Mapping[str, Any]) -> dict[str, Any]: - """Retain content plus only explicitly namespaced private metadata.""" + """Retain a validated content block, including opaque standard ``_meta``.""" - copied = { - key: deepcopy(item) for key, item in content.items() if key != "_meta" - } - meta = _extension_metadata(content) - if meta: - copied["_meta"] = meta - return copied + return deepcopy(dict(content)) def _extension_metadata(value: Mapping[str, Any]) -> dict[str, Any]: meta = value.get("_meta") - if not isinstance(meta, Mapping): + if meta is None: return {} + if not isinstance(meta, Mapping) or any(not isinstance(key, str) for key in meta): + raise AcpProjectionError("ACP _meta must be an object with string keys") + return deepcopy(dict(meta)) + + +def _scoped_metadata(**values: Mapping[str, Any]) -> dict[str, Any]: + """Preserve each protocol object's opaque metadata without key collisions.""" + return { - str(key): deepcopy(item) - for key, item in meta.items() - if isinstance(key, str) and "/" in key + scope: metadata + for scope, value in values.items() + if (metadata := _extension_metadata(value)) } @@ -676,7 +787,10 @@ def _permission_options(options: list[Any]) -> list[dict[str, Any]]: raise AcpProjectionError("ACP permission request option must be an object") option_id = _required_string(option, "optionId") _required_string(option, "name") - _required_string(option, "kind") + kind = _required_string(option, "kind") + if kind not in _PERMISSION_KINDS: + raise AcpProjectionError("ACP permission option has invalid kind") + _extension_metadata(option) if option_id in seen: raise AcpProjectionError("ACP permission option IDs must be unique") seen.add(option_id) @@ -684,6 +798,228 @@ def _permission_options(options: list[Any]) -> list[dict[str, Any]]: return normalized +def _validate_supported_update(update_name: str, update: Mapping[str, Any]) -> None: + _extension_metadata(update) + if update_name in { + "user_message_chunk", + "agent_message_chunk", + "agent_thought_chunk", + }: + content = update.get("content") + if not isinstance(content, Mapping): + raise AcpProjectionError("ACP message update is missing object content") + _validate_content_block(content, label="ACP message content") + if "messageId" in update and update["messageId"] is not None: + _identifier(update["messageId"], "messageId") + return + if update_name == "tool_call": + _validate_tool_call(update) + return + if update_name == "tool_call_update": + _validate_tool_update(update) + return + if update_name == "plan": + entries = update.get("entries") + if not isinstance(entries, list): + raise AcpProjectionError("ACP plan update entries must be an array") + for entry in entries: + _normalized_plan_entry(entry) + return + if update_name == "usage_update": + _validate_usage(update) + return + if update_name == "session_info_update": + for key in ("title", "updatedAt"): + if key in update and update[key] is not None and not isinstance(update[key], str): + raise AcpProjectionError(f"ACP session info {key} must be text or null") + + +def _validate_content_block(content: Mapping[str, Any], *, label: str) -> None: + content_type = content.get("type") + if content_type not in _CONTENT_TYPES: + raise AcpProjectionError(f"{label} has unsupported type") + _extension_metadata(content) + if content_type == "text": + _required_text(content, "text", label=label) + elif content_type in {"image", "audio"}: + _required_text(content, "data", label=label) + _required_text(content, "mimeType", label=label) + if content_type == "image": + _optional_text(content, "uri", label=label) + elif content_type == "resource_link": + _required_text(content, "name", label=label) + _required_text(content, "uri", label=label) + for key in ("description", "mimeType", "title"): + _optional_text(content, key, label=label) + if "size" in content and content["size"] is not None: + size = content["size"] + if ( + isinstance(size, bool) + or not isinstance(size, int) + or not -(2**63) <= size <= 2**63 - 1 + ): + raise AcpProjectionError(f"{label} size must be an integer or null") + else: + resource = content.get("resource") + if not isinstance(resource, Mapping): + raise AcpProjectionError(f"{label} resource must be an object") + _extension_metadata(resource) + _required_text(resource, "uri", label=f"{label} resource") + has_text = "text" in resource + has_blob = "blob" in resource + if has_text == has_blob: + raise AcpProjectionError( + f"{label} resource must contain exactly one of text or blob" + ) + _required_text( + resource, + "text" if has_text else "blob", + label=f"{label} resource", + ) + _optional_text(resource, "mimeType", label=f"{label} resource") + annotations = content.get("annotations") + if annotations is not None and not isinstance(annotations, Mapping): + raise AcpProjectionError(f"{label} annotations must be an object or null") + + +def _validate_tool_call(update: Mapping[str, Any]) -> None: + _required_string(update, "toolCallId") + if not isinstance(update.get("title"), str): + raise AcpProjectionError("ACP tool_call is missing string title") + _validate_tool_fields(update, creation=True) + + +def _validate_tool_update( + update: Mapping[str, Any], *, label: str = "ACP tool_call_update" +) -> None: + _required_string(update, "toolCallId") + _validate_tool_fields(update, creation=False, label=label) + + +def _validate_tool_fields( + update: Mapping[str, Any], + *, + creation: bool, + label: str = "ACP tool call", +) -> None: + _extension_metadata(update) + kind = update.get("kind") + if kind is not None and (not isinstance(kind, str) or kind not in _TOOL_KINDS): + raise AcpProjectionError(f"{label} has invalid kind") + status = update.get("status") + if status is not None and ( + not isinstance(status, str) or status not in _TOOL_STATUSES + ): + raise AcpProjectionError(f"{label} has invalid status") + if "title" in update and not creation: + title = update["title"] + if title is not None and not isinstance(title, str): + raise AcpProjectionError(f"{label} title must be text or null") + for key in ("content", "locations"): + value = update.get(key) + if value is not None and not isinstance(value, list): + raise AcpProjectionError(f"{label} {key} must be an array or null") + if isinstance(update.get("content"), list): + for item in update["content"]: + _validate_tool_content(item) + if isinstance(update.get("locations"), list): + for location in update["locations"]: + _validate_tool_location(location) + + +def _validate_tool_content(value: Any) -> None: + if not isinstance(value, Mapping): + raise AcpProjectionError("ACP tool content item must be an object") + _extension_metadata(value) + item_type = value.get("type") + if item_type == "content": + content = value.get("content") + if not isinstance(content, Mapping): + raise AcpProjectionError("ACP tool content is missing content block") + _validate_content_block(content, label="ACP tool content block") + elif item_type == "diff": + path = _required_text(value, "path", label="ACP tool diff") + if not os.path.isabs(path): + raise AcpProjectionError("ACP tool diff path must be absolute") + _required_text(value, "newText", label="ACP tool diff") + _optional_text(value, "oldText", label="ACP tool diff") + elif item_type == "terminal": + _required_string(value, "terminalId") + else: + raise AcpProjectionError("ACP tool content item has unsupported type") + + +def _validate_tool_location(value: Any) -> None: + if not isinstance(value, Mapping): + raise AcpProjectionError("ACP tool location must be an object") + _extension_metadata(value) + path = _required_text(value, "path", label="ACP tool location") + if not os.path.isabs(path): + raise AcpProjectionError("ACP tool location path must be absolute") + line = value.get("line") + if line is not None and ( + isinstance(line, bool) or not isinstance(line, int) or not 0 <= line <= 2**32 - 1 + ): + raise AcpProjectionError("ACP tool location line must be a u32 or null") + + +def _normalized_plan_entry(value: Any) -> dict[str, Any]: + if not isinstance(value, Mapping): + raise AcpProjectionError("ACP plan entry must be an object") + content = value.get("content") + priority = value.get("priority") + status = value.get("status") + if not isinstance(content, str): + raise AcpProjectionError("ACP plan entry is missing string content") + if priority not in _PLAN_PRIORITIES: + raise AcpProjectionError("ACP plan entry has invalid priority") + if status not in _PLAN_STATUSES: + raise AcpProjectionError("ACP plan entry has invalid status") + normalized = {"content": content, "priority": priority, "status": status} + meta = _extension_metadata(value) + if meta: + normalized["_meta"] = meta + return normalized + + +def _validate_usage(update: Mapping[str, Any]) -> None: + for key in ("used", "size"): + value = update.get(key) + if ( + isinstance(value, bool) + or not isinstance(value, int) + or not 0 <= value <= 2**64 - 1 + ): + raise AcpProjectionError(f"ACP usage {key} must be a u64") + cost = update.get("cost") + if cost is None: + return + if not isinstance(cost, Mapping): + raise AcpProjectionError("ACP usage cost must be an object or null") + amount = cost.get("amount") + if ( + isinstance(amount, bool) + or not isinstance(amount, (int, float)) + or not math.isfinite(float(amount)) + ): + raise AcpProjectionError("ACP usage cost amount must be a finite number") + if not isinstance(cost.get("currency"), str): + raise AcpProjectionError("ACP usage cost currency must be text") + _extension_metadata(cost) + + +def _required_text(value: Mapping[str, Any], key: str, *, label: str) -> str: + item = value.get(key) + if not isinstance(item, str): + raise AcpProjectionError(f"{label} is missing string {key}") + return item + + +def _optional_text(value: Mapping[str, Any], key: str, *, label: str) -> None: + if key in value and value[key] is not None and not isinstance(value[key], str): + raise AcpProjectionError(f"{label} {key} must be text or null") + + def _bounded_json(value: Any, *, label: str, max_bytes: int) -> bytes: try: encoded = json.dumps( @@ -691,6 +1027,7 @@ def _bounded_json(value: Any, *, label: str, max_bytes: int) -> bytes: sort_keys=True, separators=(",", ":"), ensure_ascii=False, + allow_nan=False, ).encode("utf-8") except (TypeError, ValueError, RecursionError) as exc: raise AcpProjectionError(f"{label} must be bounded JSON data") from exc @@ -706,6 +1043,7 @@ def _json_size(value: Any) -> int: sort_keys=True, separators=(",", ":"), ensure_ascii=False, + allow_nan=False, ).encode("utf-8") ) @@ -726,6 +1064,7 @@ def _event_digest(kind: str, value: Mapping[str, Any]) -> str: sort_keys=True, separators=(",", ":"), ensure_ascii=False, + allow_nan=False, ).encode("utf-8") return hashlib.sha256(encoded).hexdigest() @@ -749,7 +1088,11 @@ def _canonical_event( "update": original_update, } encoded = json.dumps( - dedupe_material, sort_keys=True, separators=(",", ":"), default=str + dedupe_material, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, ).encode("utf-8") privacy = "session" private_fields: list[str] = [] @@ -800,6 +1143,7 @@ def _joined_messages(messages: list[_MessageAssembly]) -> str: __all__ = [ "AcpEventProjector", + "AcpProjectionCheckpoint", "AcpProjectionError", "SUPPORTED_EVENT_KINDS", ] diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py index f6858f2..23e4372 100644 --- a/tests/test_acp_ingestion.py +++ b/tests/test_acp_ingestion.py @@ -46,6 +46,16 @@ def _appended(sequence: int, event: AgentEvent) -> AppendBoundAgentEventResult: return AppendBoundAgentEventResult("inserted", event.event_id, sequence) +def _config(db_path: Path, **kwargs: object) -> Config: + agent_event_source = str(kwargs.pop("agent_event_source", "acp_preferred")) + return Config( + host_id="host-a", + db_path=db_path, + agent_event_source=agent_event_source, + **kwargs, + ) + + def test_messages_are_journaled_privately_and_projected_without_thoughts( tmp_path: Path, ) -> None: @@ -69,7 +79,7 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): return TurnRefreshApplyResult(1, False) ingestor = AcpSessionIngestor( - Config(host_id="host-a", db_path=tmp_path / "events.db"), + _config(tmp_path / "events.db"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -130,11 +140,7 @@ def unexpected_turn(*_args, **_kwargs): raise AssertionError("shadow mode must not project turns") ingestor = AcpSessionIngestor( - Config( - host_id="host-a", - db_path=tmp_path / "events.db", - agent_event_source="acp_shadow", - ), + _config(tmp_path / "events.db", agent_event_source="acp_shadow"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -151,6 +157,10 @@ def unexpected_turn(*_args, **_kwargs): assert result.event is not None assert result.turn is None assert len(events) == 1 + assert ingestor.source_turn_id is not None + assert ingestor.projector.project_turn_content("session-a")[ + "assistant_stream_text" + ] == "shadow" def test_disabled_thought_policy_discards_before_persistence(tmp_path: Path) -> None: @@ -158,11 +168,7 @@ def unexpected_append(*_args, **_kwargs): raise AssertionError("disabled thoughts must not be persisted") ingestor = AcpSessionIngestor( - Config( - host_id="host-a", - db_path=tmp_path / "events.db", - acp_thought_policy="disabled", - ), + _config(tmp_path / "events.db", acp_thought_policy="disabled"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -193,7 +199,7 @@ def append( for generation in ("generation-a", "generation-b"): ingestor = AcpSessionIngestor( - Config(host_id="host-a", db_path=tmp_path / "events.db"), + _config(tmp_path / "events.db"), session_id="session-a", stream_generation=generation, binding=_binding(), @@ -218,7 +224,7 @@ def test_constructor_rejects_binding_for_another_acp_session(tmp_path: Path) -> with pytest.raises(ValueError, match="does not match"): AcpSessionIngestor( - Config(host_id="host-a", db_path=tmp_path / "events.db"), + _config(tmp_path / "events.db"), session_id="session-a", stream_generation="generation-a", binding=mismatched, @@ -232,7 +238,7 @@ def unexpected(*_args, **_kwargs): raise AssertionError("mismatched session must not cross the authority boundary") ingestor = AcpSessionIngestor( - Config(host_id="host-a", db_path=tmp_path / "events.db"), + _config(tmp_path / "events.db"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -271,11 +277,7 @@ def unexpected_projection(*_args, **_kwargs): raise AssertionError("stale ACP events must not be projected") ingestor = AcpSessionIngestor( - Config( - host_id="host-a", - db_path=db_path, - agent_event_source="acp_required", - ), + _config(db_path, agent_event_source="acp_required"), session_id="session-a", stream_generation="generation-a", binding=binding, @@ -295,6 +297,8 @@ def unexpected_projection(*_args, **_kwargs): assert result.event.sequence is None assert result.turn is None assert list_agent_events(db_path, "host-a") == () + assert ingestor.source_turn_id is None + assert ingestor.projector.session_snapshot("session-a") is None def test_default_authority_check_accepts_the_current_durable_binding( @@ -304,7 +308,7 @@ def test_default_authority_check_accepts_the_current_durable_binding( binding = _binding() upsert_worker_bindings(db_path, [binding]) ingestor = AcpSessionIngestor( - Config(host_id="host-a", db_path=db_path, agent_event_source="acp_required"), + _config(db_path, agent_event_source="acp_required"), session_id="session-a", stream_generation="generation-a", binding=binding, @@ -335,11 +339,7 @@ def unexpected_turn(*_args, **_kwargs): raise AssertionError("shadow mode must never project, including completion") ingestor = AcpSessionIngestor( - Config( - host_id="host-a", - db_path=tmp_path / "events.db", - agent_event_source="acp_shadow", - ), + _config(tmp_path / "events.db", agent_event_source="acp_shadow"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -387,11 +387,7 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): return TurnRefreshApplyResult(1, False) ingestor = AcpSessionIngestor( - Config( - host_id="host-a", - db_path=tmp_path / "events.db", - agent_event_source="acp_required", - ), + _config(tmp_path / "events.db", agent_event_source="acp_required"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -431,7 +427,7 @@ def apply(*_args, **_kwargs): return TurnRefreshApplyResult(1, False) ingestor = AcpSessionIngestor( - Config(host_id="host-a", db_path=tmp_path / "events.db"), + _config(tmp_path / "events.db"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -449,6 +445,74 @@ def apply(*_args, **_kwargs): assert result.ignored_reason == "duplicate_event" assert not projected + assert ingestor.source_turn_id is None + assert ingestor.projector.session_snapshot("session-a") is None + + +def test_append_exception_rolls_back_turn_identity_sequence_and_message( + tmp_path: Path, +) -> None: + attempts = 0 + + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: + nonlocal attempts + attempts += 1 + if attempts == 1: + raise RuntimeError("durable append failed") + return _appended(1, event) + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + append_event=append, + apply_turn=lambda *_args, **_kwargs: TurnRefreshApplyResult(1, False), + ) + notification = _update( + "agent_message_chunk", + content={"type": "text", "text": "exactly once"}, + ) + + with pytest.raises(RuntimeError, match="durable append failed"): + ingestor.ingest_update(notification) + assert ingestor.source_turn_id is None + assert ingestor.projector.session_snapshot("session-a") is None + + accepted = ingestor.ingest_update(notification) + assert accepted.event is not None and accepted.event.status == "inserted" + snapshot = ingestor.projector.session_snapshot("session-a") + assert snapshot is not None and snapshot["sequence"] == 1 + assert ingestor.projector.project_turn_content("session-a")[ + "assistant_stream_text" + ] == "exactly once" + + +def test_oversized_first_chunk_does_not_leave_an_implicit_turn(tmp_path: Path) -> None: + from tendwire.backends.acp_projection import AcpEventProjector, AcpProjectionError + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + projector=AcpEventProjector(max_event_bytes=128), + ) + + with pytest.raises(AcpProjectionError, match="size limit"): + ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "x" * 300}, + ) + ) + assert ingestor.source_turn_id is None + assert ingestor.projector.session_snapshot("session-a") is None def test_atomic_durable_replay_is_reported_without_second_projection( @@ -465,7 +529,7 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): def ingestor() -> AcpSessionIngestor: return AcpSessionIngestor( - Config(host_id="host-a", db_path=db_path), + _config(db_path), session_id="session-a", stream_generation="generation-a", binding=binding, @@ -502,7 +566,7 @@ def test_producer_turn_identity_survives_transport_recreation(tmp_path: Path) -> identities: list[str] = [] for generation in ("generation-a", "generation-b"): ingestor = AcpSessionIngestor( - Config(host_id="host-a", db_path=tmp_path / "events.db"), + _config(tmp_path / "events.db"), session_id="session-a", stream_generation=generation, binding=_binding(), @@ -527,7 +591,7 @@ def append( return _appended(len(events), event) ingestor = AcpSessionIngestor( - Config(host_id="host-a", db_path=tmp_path / "events.db"), + _config(tmp_path / "events.db"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -577,11 +641,7 @@ def append( return _appended(1, event) ingestor = AcpSessionIngestor( - Config( - host_id="host-a", - db_path=tmp_path / "events.db", - acp_thought_policy="private_all", - ), + _config(tmp_path / "events.db", acp_thought_policy="private_all"), session_id="session-a", stream_generation="generation-a", binding=_binding(), diff --git a/tests/test_acp_projection.py b/tests/test_acp_projection.py index cf22a87..5ed9ae5 100644 --- a/tests/test_acp_projection.py +++ b/tests/test_acp_projection.py @@ -163,53 +163,32 @@ def test_tool_lifecycle_merges_partial_updates_and_marks_raw_fields_private() -> def test_permission_request_updates_tool_and_keeps_options() -> None: projector = AcpEventProjector() - event = projector.normalize_permission_request( - { - "jsonrpc": "2.0", - "id": 42, - "method": "session/request_permission", - "params": { - "sessionId": "session-1", - "toolCall": {"toolCallId": "tool-9", "status": "pending"}, - "options": [ - {"optionId": "yes", "name": "Allow", "kind": "allow_once"}, - {"optionId": "no", "name": "Reject", "kind": "reject_once"}, - ], - }, + request = { + "jsonrpc": "2.0", + "id": 42, + "method": "session/request_permission", + "params": { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-9", "status": "pending"}, + "options": [ + {"optionId": "yes", "name": "Allow", "kind": "allow_once"}, + {"optionId": "no", "name": "Reject", "kind": "reject_once"}, + ], }, - ) + } + event = projector.normalize_permission_request(request, source_event_id="permission-42") assert event is not None assert event["kind"] == "tool_call_update" assert event["payload"]["permission"]["required"] is True assert event["payload"]["permission"]["options"][1]["optionId"] == "no" - assert event["source_event_id"] == "request:42" - assert event["event_id"] == "acp:session-1:request:42" + assert event["source_event_id"] == "permission-42" + assert event["event_id"] == "acp:session-1:permission-42" assert "payload.permission" in event["private_fields"] assert ( projector.normalize_permission_request( - { - "jsonrpc": "2.0", - "id": 42, - "method": "session/request_permission", - "params": { - "sessionId": "session-1", - "toolCall": {"toolCallId": "tool-9", "status": "pending"}, - "options": [ - { - "optionId": "yes", - "name": "Allow", - "kind": "allow_once", - }, - { - "optionId": "no", - "name": "Reject", - "kind": "reject_once", - }, - ], - }, - } + request, source_event_id="permission-42" ) is None ) @@ -242,6 +221,156 @@ def test_plan_usage_and_session_info_are_full_or_merged_snapshots() -> None: assert cleared["payload"]["updatedAt"] == "2026-07-31T12:00:00Z" +@pytest.mark.parametrize( + "content", + [ + {"type": "text", "text": "hello", "_meta": {"traceparent": "00-abc"}}, + {"type": "image", "data": "aW1hZ2U=", "mimeType": "image/png"}, + {"type": "audio", "data": "YXVkaW8=", "mimeType": "audio/wav"}, + { + "type": "resource_link", + "name": "source.py", + "uri": "file:///workspace/source.py", + "mimeType": "text/x-python", + "size": 42, + }, + { + "type": "resource", + "resource": { + "uri": "file:///workspace/source.py", + "mimeType": "text/x-python", + "text": "print('ok')", + "_meta": {"messageCount": 1}, + }, + }, + ], +) +def test_official_v1_content_block_shapes_are_accepted(content: dict[str, object]) -> None: + projector = AcpEventProjector() + event = projector.normalize_session_update( + _update("agent_message_chunk", messageId="message-1", content=content) + ) + + assert event is not None + assert event["payload"]["content"] == content + + +def test_official_v1_tool_shapes_are_validated_and_preserved() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "tool_call", + toolCallId="tool-1", + title="Edit source", + kind="edit", + status="in_progress", + locations=[{"path": "/workspace/source.py", "line": 7}], + ) + ) + event = projector.normalize_session_update( + _update( + "tool_call_update", + toolCallId="tool-1", + status="completed", + content=[ + { + "type": "diff", + "path": "/workspace/source.py", + "oldText": "old", + "newText": "new", + "_meta": {"traceparent": "00-tool"}, + }, + {"type": "terminal", "terminalId": "terminal-1"}, + { + "type": "content", + "content": {"type": "text", "text": "done"}, + }, + ], + ) + ) + + assert event is not None + assert event["payload"]["snapshot"]["status"] == "completed" + assert event["payload"]["snapshot"]["locations"] == [ + {"path": "/workspace/source.py", "line": 7} + ] + + +@pytest.mark.parametrize( + "notification,match", + [ + (_update("agent_message_chunk", content={"type": "text"}), "string text"), + (_update("agent_message_chunk", content={"type": "future"}), "unsupported type"), + (_update("tool_call", toolCallId="tool-1"), "string title"), + ( + _update( + "tool_call", + toolCallId="tool-1", + title="Bad status", + status="cancelled", + ), + "invalid status", + ), + ( + _update("plan", entries=[{"content": "missing fields"}]), + "invalid priority", + ), + (_update("usage_update", used=1), "usage size"), + (_update("usage_update", used=True, size=10), "usage used"), + (_update("session_info_update", title=7), "title must be text"), + ], +) +def test_malformed_supported_v1_updates_fail_without_allocating_state( + notification: dict[str, object], match: str +) -> None: + projector = AcpEventProjector() + with pytest.raises(AcpProjectionError, match=match): + projector.normalize_session_update(notification) + assert projector.session_snapshot("session-1") is None + + +def test_usage_update_is_a_complete_snapshot_and_omission_clears_cost() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "usage_update", + used=5, + size=100, + cost={"amount": 0.25, "currency": "USD"}, + ) + ) + event = projector.normalize_session_update( + _update("usage_update", used=7, size=100) + ) + + assert event is not None + assert event["payload"] == {"used": 7, "size": 100} + assert projector.session_snapshot("session-1")["usage"] == { + "used": 7, + "size": 100, + } + + +def test_implicit_v1_message_is_split_after_an_update_type_boundary() -> None: + projector = AcpEventProjector() + first = projector.normalize_session_update( + _update("agent_message_chunk", content={"type": "text", "text": "before"}) + ) + projector.normalize_session_update( + _update("tool_call", toolCallId="tool-1", title="Boundary") + ) + second = projector.normalize_session_update( + _update("agent_message_chunk", content={"type": "text", "text": "after"}) + ) + + assert first is not None and second is not None + assert first["payload"]["message_id"] == "implicit-agent_message-1" + assert second["payload"]["message_id"] == "implicit-agent_message-2" + assert projector.project_turn_content("session-1")["assistant_stream_text"] == ( + "before\n\nafter" + ) + + def test_explicit_event_ids_dedupe_replay_but_identical_unidentified_chunks_do_not() -> None: projector = AcpEventProjector() notification = _update( @@ -265,7 +394,7 @@ def test_explicit_event_ids_dedupe_replay_but_identical_unidentified_chunks_do_n assert len(first["dedupe_hint"]) == 64 -def test_meta_event_id_is_used_as_replay_key() -> None: +def test_meta_event_id_is_opaque_and_not_used_as_replay_key() -> None: projector = AcpEventProjector() notification = _update( "agent_message_chunk", @@ -273,8 +402,11 @@ def test_meta_event_id_is_used_as_replay_key() -> None: content={"type": "text", "text": "once"}, _meta={"eventId": "adapter-99"}, ) - assert projector.normalize_session_update(notification) is not None - assert projector.normalize_session_update(notification) is None + first = projector.normalize_session_update(notification) + second = projector.normalize_session_update(notification) + assert first is not None and second is not None + assert first["source_event_id"] is None + assert first["payload"]["extensions"]["update"] == {"eventId": "adapter-99"} def test_unknown_update_is_forward_compatible_and_malformed_input_is_rejected() -> None: @@ -288,6 +420,16 @@ def test_unknown_update_is_forward_compatible_and_malformed_input_is_rejected() projector.normalize_permission_request( {"sessionId": "session-1", "toolCall": {}, "options": []} ) + with pytest.raises(AcpProjectionError, match="invalid kind"): + projector.normalize_permission_request( + { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-1"}, + "options": [ + {"optionId": "maybe", "name": "Maybe", "kind": "sometimes"} + ], + } + ) def test_failed_normalization_does_not_consume_sequence_or_replay_id() -> None: @@ -295,7 +437,7 @@ def test_failed_normalization_does_not_consume_sequence_or_replay_id() -> None: malformed = _update( "agent_message_chunk", messageId="answer-1", content="not-an-object" ) - with pytest.raises(AcpProjectionError, match="missing content"): + with pytest.raises(AcpProjectionError, match="missing object content"): projector.normalize_session_update(malformed, source_event_id="event-1") valid = _update( @@ -332,7 +474,7 @@ def test_sessions_have_independent_ordering_and_defensive_snapshots() -> None: def test_interleaved_explicit_messages_and_implicit_v1_chunks_do_not_alias() -> None: projector = AcpEventProjector() - for message_id, text in (("a", "A1"), ("b", "B"), ("a", "A2")): + for message_id, text in (("a", "A1"), ("b", "B")): projector.normalize_session_update( _update( "agent_message_chunk", @@ -340,6 +482,14 @@ def test_interleaved_explicit_messages_and_implicit_v1_chunks_do_not_alias() -> content={"type": "text", "text": text}, ) ) + with pytest.raises(AcpProjectionError, match="reused after a message boundary"): + projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="a", + content={"type": "text", "text": "A2"}, + ) + ) implicit = projector.normalize_session_update( _update( "agent_message_chunk", @@ -350,7 +500,7 @@ def test_interleaved_explicit_messages_and_implicit_v1_chunks_do_not_alias() -> assert implicit is not None assert implicit["payload"]["message_id"] == "implicit-agent_message-1" assert projector.project_turn_content("session-1")["assistant_stream_text"] == ( - "A1A2\n\nB\n\nstable-v1" + "A1\n\nB\n\nstable-v1" ) @@ -385,10 +535,13 @@ def test_permission_request_ids_do_not_collide_with_notification_event_ids() -> notification = _update( "tool_call", toolCallId="tool-1", + title="Tracked tool", status="pending", _meta={"eventId": 42}, ) - assert projector.normalize_session_update(notification) is not None + assert projector.normalize_session_update( + notification, source_event_id="notification-42" + ) is not None permission = projector.normalize_permission_request( { "jsonrpc": "2.0", @@ -404,7 +557,7 @@ def test_permission_request_ids_do_not_collide_with_notification_event_ids() -> } ) assert permission is not None - assert permission["source_event_id"] == "request:42" + assert permission["source_event_id"] is None def test_namespaced_extension_metadata_is_private_and_adapter_neutral() -> None: @@ -437,32 +590,42 @@ def test_namespaced_extension_metadata_is_private_and_adapter_neutral() -> None: assert event is not None assert event["sequence"] == 1 assert event["payload"]["extensions"] == { - "vendor.example/params": {"trace": "abc"}, - "vendor.example/update": {"opaque": True} + "params": {"vendor.example/params": {"trace": "abc"}}, + "update": { + "vendor.example/update": {"opaque": True}, + "adapterInternal": "discard", + }, } assert event["payload"]["content"]["_meta"] == { - "vendor.example/content": {"revision": 1} + "vendor.example/content": {"revision": 1}, + "unscoped": "discard", } - assert "adapterInternal" not in repr(event["payload"]) - assert "unscoped" not in repr(event["payload"]) + assert "adapterInternal" in repr(event["payload"]) + assert "unscoped" in repr(event["payload"]) def test_plan_is_a_validated_full_replacement() -> None: projector = AcpEventProjector() projector.normalize_session_update( - _update("plan", entries=[{"content": "old", "status": "pending"}]) + _update( + "plan", + entries=[{"content": "old", "priority": "medium", "status": "pending"}], + ) ) replacement = projector.normalize_session_update( - _update("plan", entries=[{"content": "new", "status": "completed"}]) + _update( + "plan", + entries=[{"content": "new", "priority": "high", "status": "completed"}], + ) ) assert replacement is not None assert replacement["payload"]["entries"] == [ - {"content": "new", "status": "completed"} + {"content": "new", "priority": "high", "status": "completed"} ] with pytest.raises(AcpProjectionError, match="entries must be an array"): projector.normalize_session_update(_update("plan", entries="bad")) assert projector.session_snapshot("session-1")["plan"] == [ - {"content": "new", "status": "completed"} + {"content": "new", "priority": "high", "status": "completed"} ] @@ -505,23 +668,23 @@ def test_bounded_state_fails_closed_and_drop_session_releases_capacity() -> None max_source_events_per_session=1, max_messages_per_kind=1, max_tool_calls_per_session=1, - max_state_fields=1, + max_state_fields=2, max_plan_entries=1, max_text_chars_per_message=3, max_event_bytes=1024, ) assert projector.normalize_session_update( - _update("usage_update", used=1), source_event_id="one" + _update("usage_update", used=1, size=10), source_event_id="one" ) with pytest.raises(AcpProjectionError, match="replay window"): projector.normalize_session_update( - _update("usage_update", used=2), source_event_id="two" + _update("usage_update", used=2, size=10), source_event_id="two" ) with pytest.raises(AcpProjectionError, match="session limit"): projector.normalize_session_update( { "sessionId": "session-2", - "update": {"sessionUpdate": "usage_update", "used": 1}, + "update": {"sessionUpdate": "usage_update", "used": 1, "size": 10}, } ) assert projector.drop_session("session-1") is True @@ -529,7 +692,7 @@ def test_bounded_state_fails_closed_and_drop_session_releases_capacity() -> None assert projector.normalize_session_update( { "sessionId": "session-2", - "update": {"sessionUpdate": "usage_update", "used": 1}, + "update": {"sessionUpdate": "usage_update", "used": 1, "size": 10}, } ) @@ -550,7 +713,7 @@ def test_failed_or_oversized_input_does_not_allocate_or_mutate_session() -> None accepted = projector.normalize_session_update( { "sessionId": "session-2", - "update": {"sessionUpdate": "usage_update", "used": 1}, + "update": {"sessionUpdate": "usage_update", "used": 1, "size": 10}, } ) assert accepted is not None and accepted["sequence"] == 1 @@ -573,7 +736,7 @@ def test_total_retained_state_is_bounded_across_isolated_sessions() -> None: assert projector.normalize_session_update( { "sessionId": "session-a", - "update": {"sessionUpdate": "session_info_update", "value": "1234"}, + "update": {"sessionUpdate": "session_info_update", "title": "1234"}, } ) with pytest.raises(AcpProjectionError, match="total retained state"): @@ -582,7 +745,7 @@ def test_total_retained_state_is_bounded_across_isolated_sessions() -> None: "sessionId": "session-b", "update": { "sessionUpdate": "session_info_update", - "value": "1234", + "title": "1234", }, } ) @@ -601,11 +764,21 @@ def test_all_non_message_events_remain_unreachable_from_legacy_turns() -> None: _update( "tool_call", toolCallId="tool-secret", + title="Private tool", rawInput={"secret": "do not leak raw input"}, ) ) projector.normalize_session_update( - _update("plan", entries=[{"content": "do not leak plan"}]) + _update( + "plan", + entries=[ + { + "content": "do not leak plan", + "priority": "low", + "status": "pending", + } + ], + ) ) projector.normalize_permission_request( { From d5bc05b7e807512fd2e9fbdad7f0ea90659a0139 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:27:53 +0800 Subject: [PATCH 21/83] fix: fail ACP runtime closed on stale bindings --- src/tendwire/backends/acp_runtime.py | 58 ++++++++- tests/test_acp_runtime.py | 173 ++++++++++++++++++++++++++- 2 files changed, 221 insertions(+), 10 deletions(-) diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index d22e12f..5050d5c 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -41,6 +41,10 @@ class AcpRuntimeProtocolError(AcpRuntimeError): """The ACP client returned data that cannot be safely bound or finalized.""" +class AcpRuntimeBindingError(AcpRuntimeProtocolError): + """The runtime's authenticated worker binding is no longer current.""" + + class AcpRuntimeStopTimeout(AcpRuntimeError, TimeoutError): """The runtime could not stop all supervised work within its deadline.""" @@ -205,7 +209,9 @@ def __init__( self._session_mode = mode self._requested_session_id = session_id self._stream_generation = stream_generation or uuid.uuid4().hex - self._client_capabilities = dict(client_capabilities or {}) + self._client_capabilities = _runtime_client_capabilities( + client_capabilities + ) self._mcp_servers = tuple(dict(server) for server in mcp_servers) self._additional_directories = tuple(Path(path) for path in additional_directories) self._permission_callback = permission_callback @@ -366,7 +372,8 @@ def prompt( try: self._wait_for_post_response_idle(wait_limit) with self._ingest_lock: - ingestor.mark_prompt_complete() + completion = ingestor.mark_prompt_complete() + _raise_for_binding_rejection(completion) except BaseException as exc: with self._state_lock: self._prompts_failed += 1 @@ -554,7 +561,8 @@ def _consume_updates(self) -> None: ) ingestor = self._require_ingestor() with self._ingest_lock: - ingestor.ingest_update(update.raw) + outcome = ingestor.ingest_update(update.raw) + _raise_for_binding_rejection(outcome) with self._state_lock: self._updates_ingested += 1 except BaseException as exc: @@ -591,10 +599,11 @@ def _handle_permission(self, request: PermissionRequest) -> None: ) ingestor = self._require_ingestor() with self._ingest_lock: - ingestor.ingest_permission_request( + outcome = ingestor.ingest_permission_request( request.raw, source_event_id=_permission_source_event_id(request.request_id), ) + _raise_for_binding_rejection(outcome) with self._state_lock: self._permissions_ingested += 1 @@ -706,8 +715,49 @@ def _permission_source_event_id(request_id: RequestId) -> str: return f"permission:{stable_fingerprint({'request_id': request_id})}" +def _runtime_client_capabilities( + capabilities: Mapping[str, Any] | None, +) -> dict[str, Any]: + """Return only client capabilities this runtime can actually service. + + The runtime handles ACP session updates and permission requests, neither of + which is advertised through ``clientCapabilities``. It has no handlers for + filesystem, terminal, elicitation, or session-config requests. Known keys + are therefore stripped even when supplied by an embedding caller. Unknown + top-level keys could advertise extension methods and are rejected. + """ + if capabilities is None: + return {} + if not isinstance(capabilities, Mapping): + raise ValueError("client_capabilities must be a mapping or None") + known = {"fs", "terminal", "session", "elicitation", "_meta"} + if any(not isinstance(key, str) or key not in known for key in capabilities): + raise ValueError( + "ACP runtime cannot advertise unsupported client capabilities" + ) + return {} + + +def _raise_for_binding_rejection(outcome: object) -> None: + """Make every stale durable-binding outcome terminal and public-safe.""" + if outcome is None: + return + reason = getattr(outcome, "ignored_reason", None) + event = getattr(outcome, "event", None) + turn = getattr(outcome, "turn", None) + event_status = getattr(event, "status", None) + turn_stale = getattr(turn, "stale_binding", False) + if ( + reason in {"stale_binding", "binding_changed"} + or event_status == "binding_changed" + or turn_stale is True + ): + raise AcpRuntimeBindingError("ACP worker binding is no longer current") + + __all__ = [ "AcpRuntime", + "AcpRuntimeBindingError", "AcpRuntimeClient", "AcpRuntimeError", "AcpRuntimeProtocolError", diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index 291d13a..3588ded 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -1,9 +1,12 @@ from __future__ import annotations import queue +import sqlite3 import threading import time +from dataclasses import replace from pathlib import Path +from types import SimpleNamespace from typing import Any import pytest @@ -21,6 +24,7 @@ ) from tendwire.backends.acp_runtime import ( AcpRuntime, + AcpRuntimeBindingError, AcpRuntimeProtocolError, AcpRuntimeStopTimeout, RuntimeState, @@ -28,6 +32,7 @@ ) from tendwire.config import Config from tendwire.core.models import WorkerBinding +from tendwire.store.sqlite import list_agent_events, upsert_worker_bindings _END = object() @@ -124,25 +129,31 @@ def __init__(self, session_id: str = "session-private") -> None: self.completions = 0 self.update_failure: BaseException | None = None self.permission_failure: BaseException | None = None + self.update_result: object = None + self.permission_result: object = None + self.completion_result: object = None def start_turn(self, *, producer_turn_id: str | None = None) -> str: self.started.append(producer_turn_id) return "opaque-turn" - def ingest_update(self, raw: object) -> None: + def ingest_update(self, raw: object) -> object: if self.update_failure is not None: raise self.update_failure self.updates.append(raw) + return self.update_result def ingest_permission_request( self, raw: object, *, source_event_id: str | None = None - ) -> None: + ) -> object: if self.permission_failure is not None: raise self.permission_failure self.permissions.append((raw, source_event_id)) + return self.permission_result - def mark_prompt_complete(self) -> None: + def mark_prompt_complete(self) -> object: self.completions += 1 + return self.completion_result def binding(session_id: str = "session-private") -> WorkerBinding: @@ -178,6 +189,30 @@ def runtime( ) +def bound_runtime( + tmp_path: Path, + client: FakeClient, + current_binding: WorkerBinding, + **kwargs: Any, +) -> AcpRuntime: + db_path = tmp_path / "bound-events.db" + upsert_worker_bindings(db_path, [current_binding]) + return AcpRuntime( + client, # type: ignore[arg-type] + config=Config( + host_id="host-a", + db_path=db_path, + agent_event_source="acp_required", + ), + binding=current_binding, + cwd=tmp_path, + stream_generation="generation-private-secret", + poll_timeout=0.01, + stop_timeout=0.5, + **kwargs, + ) + + def update(session_id: str = "session-private") -> SessionUpdate: raw = { "sessionId": session_id, @@ -259,9 +294,7 @@ def factory(config: Config, **kwargs: object) -> FakeIngestor: ).start() try: assert [call[0] for call in client.calls[:2]] == ["initialize", "new"] - assert client.calls[0][2]["client_capabilities"] == { - "fs": {"readTextFile": True} - } + assert client.calls[0][2]["client_capabilities"] == {} assert captured["session_id"] == "session-private" assert captured["binding"] is not None assert captured["stream_generation"] == "generation-private-secret" @@ -270,6 +303,37 @@ def factory(config: Config, **kwargs: object) -> FakeIngestor: service.stop() +@pytest.mark.parametrize( + "claims", + [ + {"fs": {"readTextFile": True, "writeTextFile": True}}, + {"terminal": True}, + {"elicitation": {"form": {}, "url": {}}}, + {"session": {"configOptions": {"boolean": {}}}}, + {"_meta": {"example.test/capability": True}}, + ], +) +def test_runtime_strips_client_capabilities_it_cannot_serve( + tmp_path: Path, + claims: dict[str, object], +) -> None: + client = FakeClient() + service = runtime(tmp_path, client, client_capabilities=claims).start() + try: + assert client.calls[0][2]["client_capabilities"] == {} + finally: + service.stop() + + +def test_runtime_rejects_unknown_extension_capability_claims(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="unsupported client capabilities"): + runtime( + tmp_path, + FakeClient(), + client_capabilities={"example.test/custom": {}}, + ) + + @pytest.mark.parametrize( ("mode", "method"), [(SessionOpenMode.LOAD, "load"), (SessionOpenMode.RESUME, "resume")], @@ -411,6 +475,78 @@ def test_permission_ingestion_failure_cancels_before_runtime_fails( assert raised.value is failure +def test_replaced_binding_cancels_permission_before_callback_and_is_terminal( + tmp_path: Path, +) -> None: + client = FakeClient() + current = binding() + callback_called = threading.Event() + + def unsafe_allow(_request: PermissionRequest) -> str: + callback_called.set() + return "allow-once" + + service = bound_runtime( + tmp_path, + client, + current, + permission_callback=unsafe_allow, + ).start() + replacement = replace( + current, + worker_id="replacement-worker-private", + worker_fingerprint="replacement-fingerprint-private", + observed_at="2026-08-01T00:00:00+00:00", + ) + upsert_worker_bindings(tmp_path / "bound-events.db", [replacement]) + client.permissions.put(permission()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + status = service.status() + assert callback_called.is_set() is False + assert client.permission_responses == [(7, None, True)] + assert status.permissions_ingested == 0 + assert status.permissions_selected == 0 + assert status.permissions_cancelled == 1 + assert status.failure_type == "AcpRuntimeBindingError" + assert list_agent_events(tmp_path / "bound-events.db", "host-a") == () + rendered = repr(status) + assert "replacement-worker-private" not in rendered + assert "binding-private-secret" not in rendered + with pytest.raises(AcpRuntimeBindingError): + service.stop() + + +def test_binding_expiry_after_start_rejects_update_and_is_terminal( + tmp_path: Path, +) -> None: + client = FakeClient() + current = binding() + service = bound_runtime(tmp_path, client, current).start() + with sqlite3.connect(tmp_path / "bound-events.db") as conn: + conn.execute( + "UPDATE worker_bindings SET expires_at = ? " + "WHERE host_id = ? AND private_fingerprint = ?", + ( + "2000-01-01T00:00:00+00:00", + current.host_id, + current.private_fingerprint, + ), + ) + client.updates.put(update()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + status = service.status() + assert status.updates_ingested == 0 + assert status.failure_type == "AcpRuntimeBindingError" + assert list_agent_events(tmp_path / "bound-events.db", "host-a") == () + rendered = repr(status) + assert "session-private" not in rendered + assert "binding-private-secret" not in rendered + with pytest.raises(AcpRuntimeBindingError): + service.stop() + + def test_permission_source_identity_distinguishes_jsonrpc_id_types( tmp_path: Path, ) -> None: @@ -454,6 +590,31 @@ def test_prompt_finalizes_only_after_valid_response_and_update_drain( service.stop() +def test_stale_binding_completion_is_terminal_and_not_counted_complete( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + ingestor.completion_result = SimpleNamespace( + ignored_reason="stale_binding", + event=None, + turn=None, + ) + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRuntimeBindingError): + service.prompt("question") + + status = service.status() + assert status.state is RuntimeState.FAILED + assert status.failure_type == "AcpRuntimeBindingError" + assert status.prompts_started == 1 + assert status.prompts_completed == 0 + assert status.prompts_failed == 1 + with pytest.raises(AcpRuntimeBindingError): + service.stop() + + def test_prompt_finality_waits_for_permission_resolution(tmp_path: Path) -> None: client = FakeClient() ingestor = FakeIngestor() From e1d943a2b57e6b64931c627aaf309bf04ae77738 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:34:42 +0800 Subject: [PATCH 22/83] fix: bind new ACP sessions in two phases --- src/tendwire/backends/acp_runtime.py | 109 ++++++++-- tests/test_acp_runtime.py | 308 +++++++++++++++++++++++++-- 2 files changed, 388 insertions(+), 29 deletions(-) diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 5050d5c..81b661d 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -19,6 +19,7 @@ from ..config import Config from ..core.models import WorkerBinding, stable_fingerprint +from ..store.sqlite import list_worker_bindings from .acp_ingestion import AcpSessionIngestor from .acp_protocol import ( PermissionRequest, @@ -90,6 +91,16 @@ class AcpRuntimeStatus: IngestorFactory = Callable[..., AcpSessionIngestor] +class SessionBindingCallback(Protocol): + """Atomically persist an ACP binding derived from one continuity anchor.""" + + def __call__( + self, + session_id: str, + continuity_binding: WorkerBinding, + ) -> WorkerBinding: ... + + class AcpRuntimeClient(Protocol): """Adapter-neutral client surface required by :class:`AcpRuntime`.""" @@ -172,6 +183,7 @@ def __init__( mcp_servers: Sequence[Mapping[str, Any]] = (), additional_directories: Sequence[str | Path] = (), permission_callback: PermissionCallback | None = None, + session_binding_callback: SessionBindingCallback | None = None, ingestor: AcpSessionIngestor | None = None, ingestor_factory: IngestorFactory = AcpSessionIngestor, poll_timeout: float = 0.05, @@ -185,17 +197,30 @@ def __init__( raise ValueError(f"session_id is required for ACP {mode.value}") if mode is SessionOpenMode.NEW and session_id is not None: raise ValueError("session_id must be omitted when creating an ACP session") + if mode is SessionOpenMode.NEW and session_binding_callback is None: + raise ValueError("session_binding_callback is required for ACP new") + if mode is not SessionOpenMode.NEW and session_binding_callback is not None: + raise ValueError( + "session_binding_callback is only valid when creating an ACP session" + ) if binding.host_id != config.host_id: raise ValueError("ACP runtime binding host does not match configuration") if not binding.private_fingerprint: raise ValueError("ACP runtime requires an authenticated private binding") - if binding.turn_target_kind != "acp_session_id": - raise ValueError("ACP runtime requires an ACP session worker binding") if ( - mode is not SessionOpenMode.NEW - and binding.turn_target_value != session_id + mode is SessionOpenMode.NEW + and binding.turn_target_kind == "acp_session_id" ): - raise ValueError("ACP runtime session does not match the worker binding") + raise ValueError( + "ACP new requires a non-ACP worker continuity binding" + ) + if mode is not SessionOpenMode.NEW: + if binding.turn_target_kind != "acp_session_id": + raise ValueError("ACP runtime requires an ACP session worker binding") + if binding.turn_target_value != session_id: + raise ValueError("ACP runtime session does not match the worker binding") + if config.db_path is None: + raise ValueError("ACP runtime requires a sqlite db path") resolved_cwd = Path(cwd) if not resolved_cwd.is_absolute(): raise ValueError("ACP runtime cwd must be absolute") @@ -215,6 +240,7 @@ def __init__( self._mcp_servers = tuple(dict(server) for server in mcp_servers) self._additional_directories = tuple(Path(path) for path in additional_directories) self._permission_callback = permission_callback + self._session_binding_callback = session_binding_callback self._provided_ingestor = ingestor self._ingestor_factory = ingestor_factory self._poll_timeout = float(poll_timeout) @@ -269,23 +295,25 @@ def start(self) -> "AcpRuntime": ) self._state = RuntimeState.STARTING try: + self._require_current_binding(self._binding) self._client.initialize(client_capabilities=self._client_capabilities) session = self._open_session() if not isinstance(session, SessionResult) or not session.session_id: raise AcpRuntimeProtocolError( "ACP session setup returned an invalid response" ) - if session.session_id != self._binding.turn_target_value: - raise AcpRuntimeProtocolError( - "ACP session setup did not return the bound session" - ) - if ( - self._requested_session_id is not None - and session.session_id != self._requested_session_id - ): - raise AcpRuntimeProtocolError( - "ACP session setup returned an unexpected session" - ) + if self._session_mode is SessionOpenMode.NEW: + self._binding = self._bind_new_session(session.session_id) + else: + if session.session_id != self._binding.turn_target_value: + raise AcpRuntimeProtocolError( + "ACP session setup did not return the bound session" + ) + if session.session_id != self._requested_session_id: + raise AcpRuntimeProtocolError( + "ACP session setup returned an unexpected session" + ) + self._require_current_binding(self._binding) self._session_id = session.session_id self._ingestor = self._make_ingestor(session.session_id) threads = ( @@ -543,6 +571,54 @@ def _make_ingestor(self, session_id: str) -> AcpSessionIngestor: binding=self._binding, ) + def _bind_new_session(self, session_id: str) -> WorkerBinding: + callback = self._session_binding_callback + if callback is None: # pragma: no cover - constructor invariant + raise AcpRuntimeBindingError("ACP session binding is unavailable") + continuity = self._binding + self._require_current_binding(continuity) + bound = callback(session_id, continuity) + if not isinstance(bound, WorkerBinding): + raise AcpRuntimeBindingError("ACP session binder returned an invalid binding") + if ( + bound.host_id != continuity.host_id + or bound.worker_id != continuity.worker_id + or bound.worker_fingerprint != continuity.worker_fingerprint + or bound.backend != continuity.backend + or bound.target_kind != continuity.target_kind + or bound.target_value != continuity.target_value + ): + raise AcpRuntimeBindingError("ACP session binder changed worker continuity") + if ( + bound.turn_target_kind != "acp_session_id" + or bound.turn_target_value != session_id + ): + raise AcpRuntimeBindingError("ACP session binder returned the wrong session") + if ( + not bound.private_fingerprint + or bound.private_fingerprint == continuity.private_fingerprint + ): + raise AcpRuntimeBindingError( + "ACP session binder did not establish a distinct private binding" + ) + # The callback must add a distinct ACP binding. It must not repurpose or + # overwrite the Herdr continuity row it was given. + self._require_current_binding(continuity) + self._require_current_binding(bound) + return bound + + def _require_current_binding(self, expected: WorkerBinding) -> None: + db_path = self._config.db_path + if db_path is None: # pragma: no cover - constructor invariant + raise AcpRuntimeBindingError("ACP binding store is unavailable") + current = list_worker_bindings( + Path(db_path), + expected.host_id, + backend=expected.backend, + ) + if expected not in current: + raise AcpRuntimeBindingError("ACP worker binding is not current") + def _consume_updates(self) -> None: try: while True: @@ -766,5 +842,6 @@ def _raise_for_binding_rejection(outcome: object) -> None: "AcpRuntimeStopTimeout", "PermissionCallback", "RuntimeState", + "SessionBindingCallback", "SessionOpenMode", ] diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index 3588ded..7bcc560 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -32,7 +32,11 @@ ) from tendwire.config import Config from tendwire.core.models import WorkerBinding -from tendwire.store.sqlite import list_agent_events, upsert_worker_bindings +from tendwire.store.sqlite import ( + list_agent_events, + list_worker_bindings, + upsert_worker_bindings, +) _END = object() @@ -48,6 +52,7 @@ def __init__(self) -> None: self.prompt_failure: BaseException | None = None self.initialize_failure: BaseException | None = None self.new_session_result: SessionResult | None = None + self.restored_session_result: SessionResult | None = None self.closed = False self.close_calls = 0 @@ -67,13 +72,17 @@ def load_session( self, session_id: str, cwd: Path, **kwargs: Any ) -> SessionResult: self.calls.append(("load", (session_id, cwd), kwargs)) - return SessionResult(session_id, None, (), {}) + return self.restored_session_result or SessionResult( + session_id, None, (), {} + ) def resume_session( self, session_id: str, cwd: Path, **kwargs: Any ) -> SessionResult: self.calls.append(("resume", (session_id, cwd), kwargs)) - return SessionResult(session_id, None, (), {}) + return self.restored_session_result or SessionResult( + session_id, None, (), {} + ) def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: self.calls.append(("prompt", (session_id, prompt), kwargs)) @@ -170,18 +179,53 @@ def binding(session_id: str = "session-private") -> WorkerBinding: ) +def continuity_binding() -> WorkerBinding: + return WorkerBinding( + host_id="host-a", + worker_id="worker-public", + worker_fingerprint="worker-fingerprint", + backend="herdr", + target_kind="pane_id", + target_value="pane-private-secret", + turn_target_kind="pane_id", + turn_target_value="pane-private-secret", + private_fingerprint="continuity-binding-private-secret", + ) + + +def binding_callback(db_path: Path): + def establish( + session_id: str, + continuity: WorkerBinding, + ) -> WorkerBinding: + bound = replace( + continuity, + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="", + ) + upsert_worker_bindings(db_path, [bound]) + return bound + + return establish + + def runtime( tmp_path: Path, client: FakeClient, ingestor: FakeIngestor | None = None, **kwargs: Any, ) -> AcpRuntime: + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) return AcpRuntime( client, # type: ignore[arg-type] - config=Config(host_id="host-a", db_path=tmp_path / "events.db"), - binding=binding(), + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, cwd=tmp_path, stream_generation="generation-private-secret", + session_binding_callback=binding_callback(db_path), ingestor=ingestor or FakeIngestor(), # type: ignore[arg-type] poll_timeout=0.01, stop_timeout=0.5, @@ -206,6 +250,8 @@ def bound_runtime( ), binding=current_binding, cwd=tmp_path, + session_mode=SessionOpenMode.LOAD, + session_id=current_binding.turn_target_value, stream_generation="generation-private-secret", poll_timeout=0.01, stop_timeout=0.5, @@ -275,6 +321,9 @@ def test_start_negotiates_opens_one_session_and_binds_factory(tmp_path: Path) -> client = FakeClient() captured: dict[str, object] = {} ingestor = FakeIngestor() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) def factory(config: Config, **kwargs: object) -> FakeIngestor: captured.update(kwargs) @@ -283,11 +332,12 @@ def factory(config: Config, **kwargs: object) -> FakeIngestor: service = AcpRuntime( client, # type: ignore[arg-type] - config=Config(host_id="host-a", db_path=tmp_path / "events.db"), - binding=binding(), + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, cwd=tmp_path, stream_generation="generation-private-secret", client_capabilities={"fs": {"readTextFile": True}}, + session_binding_callback=binding_callback(db_path), ingestor_factory=factory, # type: ignore[arg-type] poll_timeout=0.01, stop_timeout=0.5, @@ -342,10 +392,13 @@ def test_load_and_resume_use_requested_session( tmp_path: Path, mode: SessionOpenMode, method: str ) -> None: client = FakeClient() + db_path = tmp_path / "events.db" + existing = binding("existing-private") + upsert_worker_bindings(db_path, [existing]) service = AcpRuntime( client, # type: ignore[arg-type] - config=Config(host_id="host-a", db_path=tmp_path / "events.db"), - binding=binding("existing-private"), + config=Config(host_id="host-a", db_path=db_path), + binding=existing, cwd=tmp_path, session_mode=mode, session_id="existing-private", @@ -360,17 +413,246 @@ def test_load_and_resume_use_requested_session( service.stop() -def test_start_rejects_unbound_session_and_closes_adapter(tmp_path: Path) -> None: +@pytest.mark.parametrize("mode", [SessionOpenMode.LOAD, SessionOpenMode.RESUME]) +def test_load_and_resume_reject_agent_session_mismatch_and_close( + tmp_path: Path, + mode: SessionOpenMode, +) -> None: client = FakeClient() - client.new_session_result = SessionResult("other-private", None, (), {}) - service = runtime(tmp_path, client) + client.restored_session_result = SessionResult( + "attacker-session-private", + None, + (), + {}, + ) + db_path = tmp_path / "events.db" + existing = binding("existing-private") + upsert_worker_bindings(db_path, [existing]) + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=existing, + cwd=tmp_path, + session_mode=mode, + session_id="existing-private", + poll_timeout=0.01, + stop_timeout=0.5, + ) with pytest.raises(AcpRuntimeProtocolError, match="bound session"): service.start() assert client.closed assert client.close_calls == 1 - assert service.join(timeout=0.1) + assert service.status().state is RuntimeState.FAILED + + +def test_new_accepts_unpredictable_agent_generated_session_id(tmp_path: Path) -> None: + client = FakeClient() + generated = "agent-generated-unpredictable-7f94" + client.new_session_result = SessionResult(generated, None, (), {}) + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + seen: list[tuple[str, WorkerBinding]] = [] + + def establish(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + seen.append((session_id, anchor)) + return binding_callback(db_path)(session_id, anchor) + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=establish, + ingestor=FakeIngestor(generated), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + try: + assert seen == [(generated, continuity)] + assert service.status().healthy + finally: + service.stop() + + +def test_new_requires_explicit_session_binder_before_launch(tmp_path: Path) -> None: + client = FakeClient() + + with pytest.raises(ValueError, match="session_binding_callback is required"): + AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=tmp_path / "events.db"), + binding=continuity_binding(), + cwd=tmp_path, + ) + + assert client.calls == [] + assert client.close_calls == 0 + + +@pytest.mark.parametrize("mismatch", ["session", "worker"]) +def test_new_rejects_malicious_binder_return_and_closes_adapter( + tmp_path: Path, + mismatch: str, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def malicious(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + return replace( + anchor, + worker_id=( + "attacker-worker-private" + if mismatch == "worker" + else anchor.worker_id + ), + turn_target_kind="acp_session_id", + turn_target_value=( + "attacker-session-private" + if mismatch == "session" + else session_id + ), + private_fingerprint="", + ) + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=malicious, + poll_timeout=0.01, + stop_timeout=0.5, + ) + + with pytest.raises(AcpRuntimeBindingError): + service.start() + + assert client.closed + assert client.close_calls == 1 + assert service.status().state is RuntimeState.FAILED + assert service.status().failure_type == "AcpRuntimeBindingError" + assert list_worker_bindings(db_path, "host-a", backend="herdr") == [ + continuity + ] + + +def test_new_binder_exception_closes_adapter_and_fails_terminally( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + failure = RuntimeError("binder-private-failure") + + def fail(_session_id: str, _anchor: WorkerBinding) -> WorkerBinding: + raise failure + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=fail, + poll_timeout=0.01, + stop_timeout=0.5, + ) + + with pytest.raises(RuntimeError) as raised: + service.start() + + assert raised.value is failure + assert client.closed + assert client.close_calls == 1 + status = service.status() + assert status.state is RuntimeState.FAILED + assert status.failure_type == "RuntimeError" + assert "binder-private-failure" not in repr(status) + + +def test_new_rejects_valid_shaped_binding_that_was_not_persisted( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def dishonest(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + return replace( + anchor, + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="", + ) + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=dishonest, + poll_timeout=0.01, + stop_timeout=0.5, + ) + + with pytest.raises(AcpRuntimeBindingError, match="not current"): + service.start() + + assert client.closed + assert client.close_calls == 1 + assert service.status().state is RuntimeState.FAILED + + +def test_new_rejects_binder_that_overwrites_herdr_continuity( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def destructive(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + upsert_worker_bindings( + db_path, + [ + replace( + anchor, + worker_id="replacement-worker-private", + worker_fingerprint="replacement-fingerprint-private", + observed_at="2026-08-01T00:00:00+00:00", + ) + ], + ) + bound = replace( + anchor, + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="", + ) + upsert_worker_bindings(db_path, [bound]) + return bound + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=destructive, + poll_timeout=0.01, + stop_timeout=0.5, + ) + + with pytest.raises(AcpRuntimeBindingError, match="not current"): + service.start() + + assert client.closed + assert client.close_calls == 1 assert service.status().state is RuntimeState.FAILED From 155de16197ec111b8693f7a08408fa3f033e8eac Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:36:25 +0800 Subject: [PATCH 23/83] fix(acp): atomically journal projected events --- src/tendwire/backends/acp_ingestion.py | 120 ++++++---- src/tendwire/store/sqlite.py | 128 +++++++++++ tests/test_acp_atomic_ingestion.py | 307 +++++++++++++++++++++++++ tests/test_acp_ingestion.py | 101 +++++--- 4 files changed, 579 insertions(+), 77 deletions(-) create mode 100644 tests/test_acp_atomic_ingestion.py diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py index a51884d..7028fc1 100644 --- a/src/tendwire/backends/acp_ingestion.py +++ b/src/tendwire/backends/acp_ingestion.py @@ -18,15 +18,14 @@ from ..core.models import WorkerBinding, stable_fingerprint from ..store.sqlite import ( AppendBoundAgentEventResult, + AppendProjectedAgentEventResult, TurnRefreshApplyResult, - append_agent_event_for_binding, - apply_turn_refresh, + append_agent_event_and_apply_turn_for_binding, ) from .acp_projection import AcpEventProjector, AcpProjectionCheckpoint -AppendEvent = Callable[..., AppendBoundAgentEventResult] -ApplyTurn = Callable[..., TurnRefreshApplyResult] +PersistEvent = Callable[..., AppendProjectedAgentEventResult] @dataclass(frozen=True) @@ -56,8 +55,7 @@ def __init__( stream_generation: str, binding: WorkerBinding, projector: AcpEventProjector | None = None, - append_event: AppendEvent = append_agent_event_for_binding, - apply_turn: ApplyTurn = apply_turn_refresh, + persist_event: PersistEvent = append_agent_event_and_apply_turn_for_binding, ) -> None: if config.db_path is None: raise ValueError("ACP ingestion requires a sqlite db path") @@ -81,8 +79,7 @@ def __init__( self.stream_generation = stream_generation.strip() self.binding = binding self.projector = projector or AcpEventProjector() - self._append_event = append_event - self._apply_turn = apply_turn + self._persist_event = persist_event self._turn_ordinal = 0 self._source_turn_id: str | None = None self._turn_complete = False @@ -211,23 +208,65 @@ def ingest_permission_request( ) def mark_prompt_complete(self) -> AcpIngestionResult: - """Finalize the current text projection after ``session/prompt`` returns.""" + """Durably finalize the current turn after ``session/prompt`` returns.""" if self._source_turn_id is None: return AcpIngestionResult(None, ignored_reason="no_active_turn") if self._turn_complete: return AcpIngestionResult(None, ignored_reason="turn_already_complete") - content = self.projector.mark_turn_complete(self.session_id) - content["source_turn_id"] = self._source_turn_id - if self.config.agent_event_source == "acp_shadow": - self._turn_complete = True - return AcpIngestionResult("agent_message") - turn = self._project_turn(content) + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + try: + content = self.projector.mark_turn_complete(self.session_id) + content["source_turn_id"] = self._source_turn_id + marker = agent_event( + kind="extension", + source="acp", + worker_id=self.binding.worker_id, + payload={ + "schema_version": 1, + "extension": "tendwire.acp.prompt_completion", + "complete": True, + "projection": content, + }, + source_session_id=self.session_id, + source_turn_id=self._source_turn_id, + source_event_id=f"prompt-complete:{self._source_turn_id}", + visibility="private", + ) + persisted = self._persist_event( + Path(self.config.db_path), + self.config.host_id, + marker, + expected_binding=self.binding, + content=( + None + if self.config.agent_event_source == "acp_shadow" + else content + ), + observed_at=marker.observed_at, + turn_model=self.config.turn_model, + ) + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + if persisted.event.status == "binding_changed": + self._restore_speculation(checkpoint, prior_turn_state) + return AcpIngestionResult( + "extension", + event=persisted.event, + ignored_reason="stale_binding", + ) self._turn_complete = True return AcpIngestionResult( - "agent_message", - turn=turn, - ignored_reason="stale_binding" if turn.stale_binding else None, + "extension", + event=persisted.event, + turn=persisted.turn, + ignored_reason=( + "duplicate_event" + if persisted.event.status == "replayed" + else None + ), ) def _accept( @@ -269,28 +308,35 @@ def _accept( # connector views require a separate explicit sanitizing projection. visibility="private", ) - appended = self._append_event( + projection: Mapping[str, Any] | None = None + if ( + kind in {"user_message", "agent_message"} + and self.config.agent_event_source != "acp_shadow" + ): + content = self.projector.project_turn_content(self.session_id) + if self._source_turn_id is not None: + content["source_turn_id"] = self._source_turn_id + projection = content + persisted = self._persist_event( Path(self.config.db_path), self.config.host_id, event, expected_binding=self.binding, + content=projection, + observed_at=event.observed_at, + turn_model=self.config.turn_model, ) except BaseException: self._restore_speculation(checkpoint, prior_turn_state) raise - if appended.status != "inserted": + appended = persisted.event + # A durable replay on a newly constructed ingestor is also the + # reconstruction path for its in-memory projector. Keep that state so + # prompt completion can finalize the recovered text. Only a stale + # binding invalidates the speculative normalization. + if appended.status == "binding_changed": self._restore_speculation(checkpoint, prior_turn_state) - - turn: TurnRefreshApplyResult | None = None - if ( - kind in {"user_message", "agent_message"} - and self.config.agent_event_source != "acp_shadow" - and appended.status == "inserted" - ): - content = self.projector.project_turn_content(self.session_id) - if self._source_turn_id is not None: - content["source_turn_id"] = self._source_turn_id - turn = self._project_turn(content) + turn = persisted.turn return AcpIngestionResult( kind, event=appended, @@ -316,18 +362,6 @@ def _restore_speculation( self.projector.restore_session(checkpoint) self._turn_ordinal, self._source_turn_id, self._turn_complete = prior_turn_state - def _project_turn(self, content: Mapping[str, Any]) -> TurnRefreshApplyResult: - return self._apply_turn( - Path(self.config.db_path), - self.config.host_id, - self.binding.worker_id, - content, - expected_binding=self.binding, - pending_stale_grace_seconds=self.config.pending_stale_grace_seconds, - turn_model=self.config.turn_model, - ) - - def _source_message_id(kind: str, payload: Mapping[str, Any]) -> str | None: if kind not in {"user_message", "agent_message", "thought"}: return None diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index d80bac7..e7b2c52 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -499,6 +499,14 @@ class TurnRefreshApplyResult: cancelled: bool = False +@dataclass(frozen=True) +class AppendProjectedAgentEventResult: + """One binding-fenced journal append and optional turn projection outcome.""" + + event: AppendBoundAgentEventResult + turn: TurnRefreshApplyResult | None = None + + @dataclass(frozen=True) class HerdrTurnWatermark: """Durable replay position and retained completeness-break evidence.""" @@ -23125,6 +23133,126 @@ def _begin_turn_refresh_transaction( raise +def append_agent_event_and_apply_turn_for_binding( + db_path: Path | str, + host_id: str, + event: AgentEvent, + *, + expected_binding: WorkerBinding, + content: Mapping[str, Any] | None = None, + observed_at: str | None = None, + turn_model: str = DEFAULT_TURN_MODEL, + _fault_inject: Callable[[str], None] | None = None, +) -> AppendProjectedAgentEventResult: + """Atomically journal an agent event and apply its text-only turn projection. + + A replay still runs the idempotent projection merge. This lets a retry + repair a projection that was independently removed without duplicating the + journal event or its revision-keyed connector delivery. Retention + tombstones participate in the same replay contract as live event rows. + """ + + normalized_host = normalize_agent_event_identifier( + host_id, "host_id", required=True + ) + _canonical_agent_event_for_append(event) + if not isinstance(expected_binding, WorkerBinding): + raise ValueError("expected_binding must be a WorkerBinding") + if content is not None: + if not isinstance(content, Mapping): + raise ValueError("content must be a mapping or None") + if not str(content.get("source_turn_id") or "").strip(): + raise ValueError("projected agent content requires source_turn_id") + normalized_turn_model = str(turn_model or "").strip().lower() + if normalized_turn_model not in TURN_MODELS: + allowed = ", ".join(sorted(TURN_MODELS)) + raise ValueError(f"turn_model must be one of: {allowed}") + if _fault_inject is not None and not callable(_fault_inject): + raise TypeError("_fault_inject must be callable or None") + current_time, _ = _pending_observed_time(observed_at or event.observed_at) + rearm_key: tuple[str, str] | None = None + + def fault(boundary: str) -> None: + if _fault_inject is not None: + _fault_inject(boundary) + + with _connect(db_path, prepare=True, isolation_level=None) as conn: + _ensure_schema(conn) + conn.execute("BEGIN IMMEDIATE") + try: + if not _agent_event_binding_matches_conn( + conn, + normalized_host or "", + event.worker_id, + expected_binding, + ): + conn.rollback() + return AppendProjectedAgentEventResult( + event=AppendBoundAgentEventResult( + status="binding_changed", + event_id=event.event_id, + ) + ) + fault("after_binding_check") + appended = _append_agent_event_conn( + conn, + normalized_host or "", + event, + ) + fault("after_event_append") + turn: TurnRefreshApplyResult | None = None + if content is not None: + worker_exists = conn.execute( + "SELECT 1 FROM workers WHERE host_id = ? AND worker_id = ?", + (normalized_host or "", event.worker_id), + ).fetchone() + if worker_exists is None: + raise StoreSchemaError("agent_event_projection_worker_missing") + merge_result = _merge_turn_content_conn( + conn, + normalized_host or "", + event.worker_id, + content, + observed_at=current_time, + turn_model=normalized_turn_model, + ) + rearm_key = ( + merge_result.submission_link_rearm + or merge_result.submission_link + ) + if merge_result.submission_link is not None: + owner_key, fingerprint = merge_result.submission_link + settle_submission_links_conn( + conn, + normalized_host or "", + owner_key, + fingerprint, + now=current_time, + ) + turn = TurnRefreshApplyResult(merge_result.updated, False) + fault("after_turn_projection") + fault("before_commit") + conn.commit() + except Exception: + conn.rollback() + raise + if rearm_key is not None: + _rearm_submission_link_component( + db_path, + normalized_host or "", + rearm_key[0], + rearm_key[1], + ) + return AppendProjectedAgentEventResult( + event=AppendBoundAgentEventResult( + status="inserted" if appended.inserted else "replayed", + event_id=appended.event_id, + sequence=appended.sequence, + ), + turn=turn, + ) + + def apply_turn_refresh( db_path: Path | str, host_id: str, diff --git a/tests/test_acp_atomic_ingestion.py b/tests/test_acp_atomic_ingestion.py new file mode 100644 index 0000000..66bbf7d --- /dev/null +++ b/tests/test_acp_atomic_ingestion.py @@ -0,0 +1,307 @@ +"""Atomicity and recovery tests for ACP journal-to-turn ingestion.""" + +from __future__ import annotations + +import sqlite3 +from dataclasses import replace +from pathlib import Path + +import pytest + +from tendwire.backends.acp_ingestion import AcpSessionIngestor +from tendwire.config import Config +from tendwire.core.agent_events import AgentEvent, agent_event +from tendwire.core.models import WorkerBinding +from tendwire.core.projector import project_from_raw +from tendwire.store.sqlite import ( + append_agent_event_and_apply_turn_for_binding, + cleanup_agent_event_retention, + init_store, + list_agent_events, + save_snapshot, + turns_payload_from_store, + upsert_worker_bindings, +) + + +def _store( + tmp_path: Path, + *, + source: str = "acp_preferred", +) -> tuple[Config, WorkerBinding]: + db_path = tmp_path / "events.db" + config = Config(host_id="host-a", db_path=db_path, agent_event_source=source) + snapshot = project_from_raw( + config, + workers=[{"id": "worker-a", "name": "Worker A"}], + ) + init_store(db_path) + save_snapshot(db_path, snapshot) + worker = snapshot.workers[0] + binding = WorkerBinding( + host_id=config.host_id, + worker_id=worker.id, + worker_fingerprint=worker.fingerprint, + backend="herdr", + target_kind="pane_id", + target_value="pane-a", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + sendable=True, + private_fingerprint="private-a", + ) + upsert_worker_bindings(db_path, [binding]) + return config, binding + + +def _event( + binding: WorkerBinding, + *, + observed_at: str = "2026-07-31T00:00:00+00:00", +) -> AgentEvent: + return agent_event( + kind="agent_message", + source="acp", + worker_id=binding.worker_id, + payload={"schema_version": 1, "message_id": "message-a", "text": "answer"}, + source_session_id="session-a", + source_turn_id="turn-a", + source_message_id="message-a", + source_event_id="event-a", + observed_at=observed_at, + ) + + +def _content(*, complete: bool = False) -> dict[str, object]: + return { + "source_turn_id": "turn-a", + "user_text": "", + "assistant_stream_text": "" if complete else "answer", + "assistant_final_text": "answer" if complete else "", + "complete": complete, + "has_open_turn": not complete, + } + + +def _counts(db_path: Path) -> tuple[int, int]: + with sqlite3.connect(db_path) as conn: + events = int(conn.execute("SELECT COUNT(*) FROM agent_events").fetchone()[0]) + turns = int(conn.execute("SELECT COUNT(*) FROM turns").fetchone()[0]) + return events, turns + + +@pytest.mark.parametrize( + "boundary", + ( + "after_binding_check", + "after_event_append", + "after_turn_projection", + "before_commit", + ), +) +def test_every_atomic_boundary_rolls_back_and_retry_succeeds( + tmp_path: Path, + boundary: str, +) -> None: + config, binding = _store(tmp_path) + event = _event(binding) + + def fail(current: str) -> None: + if current == boundary: + raise RuntimeError(boundary) + + with pytest.raises(RuntimeError, match=boundary): + append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(), + _fault_inject=fail, + ) + + assert _counts(config.db_path) == (0, 0) + retried = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(), + ) + assert retried.event.status == "inserted" + assert retried.turn is not None + assert _counts(config.db_path) == (1, 1) + + +def test_stale_binding_writes_neither_journal_nor_projection(tmp_path: Path) -> None: + config, binding = _store(tmp_path) + stale = replace(binding, target_value="old-pane") + result = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + _event(binding), + expected_binding=stale, + content=_content(), + ) + + assert result.event.status == "binding_changed" + assert result.turn is None + assert _counts(config.db_path) == (0, 0) + + +def test_tombstoned_replay_can_repair_projection_without_reinserting_event( + tmp_path: Path, +) -> None: + config, binding = _store(tmp_path) + event = _event(binding, observed_at="2020-01-01T00:00:00+00:00") + inserted = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + ) + assert inserted.event.status == "inserted" + cleanup = cleanup_agent_event_retention( + config.db_path, + config.host_id, + retention_days=1, + now="2026-07-31T00:00:00+00:00", + ) + assert cleanup["tombstoned"] == 1 + assert len(list_agent_events(config.db_path, config.host_id)) == 0 + + repaired = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(), + ) + assert repaired.event.status == "replayed" + assert repaired.turn is not None + assert _counts(config.db_path) == (0, 1) + + +def _update(text: str) -> dict[str, object]: + return { + "method": "session/update", + "params": { + "sessionId": "session-a", + "update": { + "sessionUpdate": "agent_message_chunk", + "messageId": "message-a", + "content": {"type": "text", "text": text}, + }, + }, + } + + +def test_process_reconstruction_replays_text_then_completes_once( + tmp_path: Path, +) -> None: + config, binding = _store(tmp_path) + first = AcpSessionIngestor( + config, + session_id="session-a", + stream_generation="generation-a", + binding=binding, + ) + first.start_turn(producer_turn_id="producer-a") + first.ingest_update(_update("answer"), source_event_id="message-event-a") + + reconstructed = AcpSessionIngestor( + config, + session_id="session-a", + stream_generation="generation-b", + binding=binding, + ) + reconstructed.start_turn(producer_turn_id="producer-a") + replayed = reconstructed.ingest_update( + _update("answer"), source_event_id="message-event-a", replay=True + ) + completed = reconstructed.mark_prompt_complete() + + assert replayed.event is not None and replayed.event.status == "replayed" + assert completed.event is not None and completed.event.status == "inserted" + turns = turns_payload_from_store(config.db_path, config.host_id)["turns"] + assert len(turns) == 1 + assert turns[0]["assistant_final_text"] == "answer" + with sqlite3.connect(config.db_path) as conn: + before = int( + conn.execute("SELECT COUNT(*) FROM connector_outbox").fetchone()[0] + ) + + second_reconstruction = AcpSessionIngestor( + config, + session_id="session-a", + stream_generation="generation-c", + binding=binding, + ) + second_reconstruction.start_turn(producer_turn_id="producer-a") + second_reconstruction.ingest_update( + _update("answer"), source_event_id="message-event-a", replay=True + ) + completion_replay = second_reconstruction.mark_prompt_complete() + assert completion_replay.event is not None + assert completion_replay.event.status == "replayed" + with sqlite3.connect(config.db_path) as conn: + after = int(conn.execute("SELECT COUNT(*) FROM connector_outbox").fetchone()[0]) + assert after == before + + +def test_completion_failure_rolls_back_marker_and_final_then_retries( + tmp_path: Path, +) -> None: + config, binding = _store(tmp_path) + fail_completion = True + + def persist(*args, **kwargs): + def fault(boundary: str) -> None: + if ( + fail_completion + and args[2].kind == "extension" + and boundary == "before_commit" + ): + raise RuntimeError("completion fault") + + kwargs["_fault_inject"] = fault + return append_agent_event_and_apply_turn_for_binding(*args, **kwargs) + + ingestor = AcpSessionIngestor( + config, + session_id="session-a", + stream_generation="generation-a", + binding=binding, + persist_event=persist, + ) + ingestor.start_turn(producer_turn_id="producer-a") + ingestor.ingest_update(_update("answer"), source_event_id="message-event-a") + with pytest.raises(RuntimeError, match="completion fault"): + ingestor.mark_prompt_complete() + assert len(list_agent_events(config.db_path, config.host_id)) == 1 + + fail_completion = False + completed = ingestor.mark_prompt_complete() + assert completed.event is not None and completed.event.status == "inserted" + assert len(list_agent_events(config.db_path, config.host_id)) == 2 + turn = turns_payload_from_store(config.db_path, config.host_id)["turns"][0] + assert turn["assistant_final_text"] == "answer" + + +def test_shadow_mode_journals_completion_without_projecting_turn( + tmp_path: Path, +) -> None: + config, binding = _store(tmp_path, source="acp_shadow") + ingestor = AcpSessionIngestor( + config, + session_id="session-a", + stream_generation="generation-a", + binding=binding, + ) + ingestor.start_turn(producer_turn_id="producer-a") + ingestor.ingest_update(_update("answer"), source_event_id="message-event-a") + ingestor.mark_prompt_complete() + + events = list_agent_events(config.db_path, config.host_id) + assert [item.event.kind for item in events] == ["agent_message", "extension"] + assert turns_payload_from_store(config.db_path, config.host_id)["turns"] == [] diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py index 23e4372..cbe6ee8 100644 --- a/tests/test_acp_ingestion.py +++ b/tests/test_acp_ingestion.py @@ -12,6 +12,7 @@ from tendwire.core.agent_events import AgentEvent, AppendBoundAgentEventResult from tendwire.core.models import WorkerBinding from tendwire.store.sqlite import ( + AppendProjectedAgentEventResult, TurnRefreshApplyResult, list_agent_events, upsert_worker_bindings, @@ -46,6 +47,37 @@ def _appended(sequence: int, event: AgentEvent) -> AppendBoundAgentEventResult: return AppendBoundAgentEventResult("inserted", event.event_id, sequence) +def _persist( + append, + apply=None, +): + def persist( + path: Path | str, + host_id: str, + event: AgentEvent, + *, + expected_binding: WorkerBinding, + content=None, + **_kwargs, + ) -> AppendProjectedAgentEventResult: + appended = append( + path, + host_id, + event, + expected_binding=expected_binding, + ) + turn = None + if content is not None and appended.status != "binding_changed": + turn = ( + apply(path, host_id, event.worker_id, content) + if apply is not None + else TurnRefreshApplyResult(0, False) + ) + return AppendProjectedAgentEventResult(appended, turn) + + return persist + + def _config(db_path: Path, **kwargs: object) -> Config: agent_event_source = str(kwargs.pop("agent_event_source", "acp_preferred")) return Config( @@ -83,8 +115,7 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=append, - apply_turn=apply, + persist_event=_persist(append, apply), ) turn_id = ingestor.start_turn(producer_turn_id="private-turn") ingestor.ingest_update( @@ -116,6 +147,7 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): "user_message", "thought", "agent_message", + "extension", ] assert all(event.visibility == "private" for event in events) assert turns[-1]["assistant_final_text"] == "answer" @@ -144,8 +176,7 @@ def unexpected_turn(*_args, **_kwargs): session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=append, - apply_turn=unexpected_turn, + persist_event=_persist(append, unexpected_turn), ) result = ingestor.ingest_update( _update( @@ -172,7 +203,7 @@ def unexpected_append(*_args, **_kwargs): session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=unexpected_append, + persist_event=_persist(unexpected_append), ) result = ingestor.ingest_update( _update( @@ -203,8 +234,7 @@ def append( session_id="session-a", stream_generation=generation, binding=_binding(), - append_event=append, - apply_turn=lambda *_args, **_kwargs: TurnRefreshApplyResult(0, False), + persist_event=_persist(append), ) ingestor.ingest_update( _update( @@ -242,8 +272,7 @@ def unexpected(*_args, **_kwargs): session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=unexpected, - apply_turn=unexpected, + persist_event=_persist(unexpected, unexpected), ) notification = _update( "agent_message_chunk", @@ -273,15 +302,11 @@ def test_required_mode_fails_closed_when_durable_binding_is_stale( ) upsert_worker_bindings(db_path, [replacement]) - def unexpected_projection(*_args, **_kwargs): - raise AssertionError("stale ACP events must not be projected") - ingestor = AcpSessionIngestor( _config(db_path, agent_event_source="acp_required"), session_id="session-a", stream_generation="generation-a", binding=binding, - apply_turn=unexpected_projection, ) result = ingestor.ingest_update( @@ -343,8 +368,7 @@ def unexpected_turn(*_args, **_kwargs): session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=append, - apply_turn=unexpected_turn, + persist_event=_persist(append, unexpected_turn), ) ingestor.start_turn(producer_turn_id="turn-1") ingestor.ingest_update( @@ -366,7 +390,8 @@ def unexpected_turn(*_args, **_kwargs): assert completed.turn is None assert repeated.ignored_reason == "turn_already_complete" assert late.ignored_reason == "turn_already_complete" - assert len(events) == 1 + assert len(events) == 2 + assert events[-1].kind == "extension" def test_required_mode_projects_messages_and_final_exactly_once(tmp_path: Path) -> None: @@ -391,8 +416,7 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=append, - apply_turn=apply, + persist_event=_persist(append, apply), ) ingestor.start_turn(producer_turn_id="turn-1") streamed = ingestor.ingest_update( @@ -410,7 +434,7 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): assert turns[-1]["assistant_stream_text"] == "" -def test_duplicate_durable_event_is_not_reprojected(tmp_path: Path) -> None: +def test_duplicate_durable_event_can_idempotently_repair_projection(tmp_path: Path) -> None: projected = False def append( @@ -431,8 +455,7 @@ def apply(*_args, **_kwargs): session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=append, - apply_turn=apply, + persist_event=_persist(append, apply), ) result = ingestor.ingest_update( _update( @@ -444,9 +467,9 @@ def apply(*_args, **_kwargs): ) assert result.ignored_reason == "duplicate_event" - assert not projected - assert ingestor.source_turn_id is None - assert ingestor.projector.session_snapshot("session-a") is None + assert projected + assert ingestor.source_turn_id is not None + assert ingestor.projector.session_snapshot("session-a") is not None def test_append_exception_rolls_back_turn_identity_sequence_and_message( @@ -471,8 +494,10 @@ def append( session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=append, - apply_turn=lambda *_args, **_kwargs: TurnRefreshApplyResult(1, False), + persist_event=_persist( + append, + lambda *_args, **_kwargs: TurnRefreshApplyResult(1, False), + ), ) notification = _update( "agent_message_chunk", @@ -515,7 +540,7 @@ def test_oversized_first_chunk_does_not_leave_an_implicit_turn(tmp_path: Path) - assert ingestor.projector.session_snapshot("session-a") is None -def test_atomic_durable_replay_is_reported_without_second_projection( +def test_atomic_durable_replay_can_repair_projection( tmp_path: Path, ) -> None: db_path = tmp_path / "events.db" @@ -529,11 +554,20 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): def ingestor() -> AcpSessionIngestor: return AcpSessionIngestor( - _config(db_path), + _config(db_path), session_id="session-a", stream_generation="generation-a", binding=binding, - apply_turn=apply, + persist_event=_persist( + lambda _path, _host, event, **_kwargs: ( + AppendBoundAgentEventResult( + "inserted" if not turns else "replayed", + event.event_id, + 1, + ) + ), + apply, + ), ) notification = _update( @@ -557,9 +591,8 @@ def ingestor() -> AcpSessionIngestor: assert replayed.event is not None assert replayed.event.status == "replayed" assert replayed.ignored_reason == "duplicate_event" - assert replayed.turn is None - assert len(turns) == 1 - assert len(list_agent_events(db_path, "host-a")) == 1 + assert replayed.turn is not None + assert len(turns) == 2 def test_producer_turn_identity_survives_transport_recreation(tmp_path: Path) -> None: @@ -595,7 +628,7 @@ def append( session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=append, + persist_event=_persist(append), ) unclassified = ingestor.ingest_update( _update( @@ -645,7 +678,7 @@ def append( session_id="session-a", stream_generation="generation-a", binding=_binding(), - append_event=append, + persist_event=_persist(append), ) result = ingestor.ingest_update( _update( From b4ee6b531e56aa3978d5c888249cdb53fb862c44 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:47:28 +0800 Subject: [PATCH 24/83] fix: bound ACP event retention automatically --- README.md | 9 +- docs/acp-migration.md | 26 +-- src/tendwire/daemon.py | 20 +++ src/tendwire/store/sqlite.py | 296 +++++++++++++++++++++++-------- tests/test_agent_events.py | 335 +++++++++++++++++++++++++++++++++++ tests/test_daemon.py | 106 ++++++++++- 6 files changed, 702 insertions(+), 90 deletions(-) diff --git a/README.md b/README.md index ac69c56..9714bb1 100644 --- a/README.md +++ b/README.md @@ -584,8 +584,13 @@ explicit sanitized projection is introduced. Store maintenance retires expired structured agent-event payloads in bounded batches using `event_retention_days`. Compact identity tombstones remain so a replayed source event cannot be reinserted or silently change content after its -private payload has expired. Tombstones intentionally retain hashes and opaque -identity only; they are not a recoverable copy of the removed payload. +private payload has expired. Due automatic daemon maintenance performs the same +host-scoped retirement as explicit `store cleanup`, using a metadata-only scan +that does not load private payloads. Tombstones intentionally retain bounded +per-event hashes and opaque identity only; they are not a recoverable copy of the +removed payload. Their count is permanent and grows with distinct source-event +identities. SQLite secure deletion is best-effort page hygiene, not a promise of +immediate physical erasure across WAL/checkpoints, snapshots, or backups. Snapshot history defaults are sized for a five-minute observation rhythm: $14 \times 24 \times 12 = 4032$ observations, while the 4096-row count diff --git a/docs/acp-migration.md b/docs/acp-migration.md index e22eff1..d4a1975 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -130,16 +130,22 @@ implementation. ## Retention -`event_retention_days` also bounds raw structured ACP journal payloads. Online -maintenance replaces expired payload rows with compact identity tombstones in -bounded batches. A tombstone retains the original sequence and a replay-contract -fingerprint, allowing exact retries to remain idempotent and conflicting reuse -to fail closed without retaining messages, thoughts, raw tool input/output, or -other source payloads. Tombstones are intentionally not deleted automatically: -removing them would make a late replay indistinguishable from a new event. -SQLite secure deletion is enabled for this bounded cleanup transaction, but WAL -files, filesystem snapshots, and backups retain their own operator-managed -lifecycle and are not a cryptographic erasure guarantee. +`event_retention_days` also bounds raw structured ACP journal payloads. Due +automatic maintenance and explicit online cleanup replace expired payload rows +with compact identity tombstones in bounded batches. Candidate scanning reads +only bounded identity metadata and the existing payload digest; it does not load +the retired private payload into maintenance memory. A tombstone retains the +original sequence and a replay-contract fingerprint, allowing exact retries to +remain idempotent and conflicting reuse to fail closed without retaining +messages, thoughts, raw tool input/output, or other source payloads. + +Each tombstone has bounded per-event identity metadata, but tombstone count is +permanent and therefore grows with the number of distinct source events. +Tombstones are intentionally not deleted automatically: removing them would make +a late replay indistinguishable from a new event. Cleanup asks SQLite to scrub +deleted cells in modified pages, but WAL/checkpoint timing, filesystem snapshots, +and backups have independent operator-managed lifecycles. Logical retention is +not an immediate physical-erasure or cryptographic-erasure guarantee. ## Cross-repository requirements diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index f2f433b..b798ab0 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -901,6 +901,8 @@ def _after_snapshot_saved(self) -> None: result = maybe_run_automatic_store_maintenance( Path(self.config.db_path), policy=policy, + agent_event_host_id=self.config.host_id, + agent_event_retention_days=self.config.event_retention_days, turn_model=self.config.turn_model, acknowledged_final_retention_days=( self.config.acknowledged_final_retention_days @@ -930,6 +932,12 @@ def _after_snapshot_saved(self) -> None: ) snapshot_result = result.get("snapshot") snapshot_counts = snapshot_result if isinstance(snapshot_result, Mapping) else {} + agent_event_result = result.get("agent_events") + agent_event_counts = ( + agent_event_result + if isinstance(agent_event_result, Mapping) + else {} + ) maintenance_status = { "ok": bool(result.get("ok")) and bool(turn_change_result.get("ok")), "status": ( @@ -941,6 +949,15 @@ def _after_snapshot_saved(self) -> None: "examined": int(snapshot_counts.get("examined") or 0), "deleted": int(snapshot_counts.get("deleted") or 0), "remaining_candidates": bool(snapshot_counts.get("remaining_candidates")), + "agent_events_examined": int( + agent_event_counts.get("examined") or 0 + ), + "agent_events_deleted": int( + agent_event_counts.get("deleted") or 0 + ), + "agent_events_remaining_candidates": bool( + agent_event_counts.get("remaining_candidates") + ), } except Exception: self._automatic_maintenance_status = { @@ -950,6 +967,9 @@ def _after_snapshot_saved(self) -> None: "examined": 0, "deleted": 0, "remaining_candidates": False, + "agent_events_examined": 0, + "agent_events_deleted": 0, + "agent_events_remaining_candidates": False, } else: self._automatic_maintenance_status = maintenance_status diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index e7b2c52..45b96e9 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -13751,28 +13751,62 @@ def _agent_event_conflicts(existing: StoredAgentEvent, incoming: AgentEvent) -> ) -def _agent_event_replay_fingerprint(event: AgentEvent) -> str: - """Fingerprint the replay contract while excluding source observation time.""" +def _agent_event_replay_contract_fingerprint( + *, + event_id: str, + kind: str, + source: str, + worker_id: str, + visibility: str, + source_session_id: str | None, + source_turn_id: str | None, + source_item_id: str | None, + source_message_id: str | None, + source_event_id: str | None, + source_sequence: int | None, + payload_fingerprint: str, + public_payload_json: str, +) -> str: + """Fingerprint compact replay metadata without reading private payload text.""" contract = { - "event_id": event.event_id, - "kind": event.kind, - "source": event.source, - "worker_id": event.worker_id, - "visibility": event.visibility, - "source_session_id": event.source_session_id, - "source_turn_id": event.source_turn_id, - "source_item_id": event.source_item_id, - "source_message_id": event.source_message_id, - "source_event_id": event.source_event_id, - "source_sequence": event.source_sequence, - "payload_fingerprint": event.payload_fingerprint, + "event_id": event_id, + "kind": kind, + "source": source, + "worker_id": worker_id, + "visibility": visibility, + "source_session_id": source_session_id, + "source_turn_id": source_turn_id, + "source_item_id": source_item_id, + "source_message_id": source_message_id, + "source_event_id": source_event_id, + "source_sequence": source_sequence, + "payload_fingerprint": payload_fingerprint, "public_payload_fingerprint": hashlib.sha256( - _canonical_json(event.public_payload).encode("utf-8") + public_payload_json.encode("utf-8") ).hexdigest(), } return hashlib.sha256(_canonical_json(contract).encode("utf-8")).hexdigest() +def _agent_event_replay_fingerprint(event: AgentEvent) -> str: + """Fingerprint the replay contract while excluding source observation time.""" + return _agent_event_replay_contract_fingerprint( + event_id=event.event_id, + kind=event.kind, + source=event.source, + worker_id=event.worker_id, + visibility=event.visibility, + source_session_id=event.source_session_id, + source_turn_id=event.source_turn_id, + source_item_id=event.source_item_id, + source_message_id=event.source_message_id, + source_event_id=event.source_event_id, + source_sequence=event.source_sequence, + payload_fingerprint=event.payload_fingerprint, + public_payload_json=_canonical_json(event.public_payload), + ) + + def _canonical_agent_event_for_append(event: AgentEvent) -> AgentEvent: if not isinstance(event, AgentEvent): raise ValueError("event must be an AgentEvent") @@ -16957,6 +16991,8 @@ def maybe_run_automatic_store_maintenance( db_path: Path, *, policy: SnapshotRetentionPolicy, + agent_event_host_id: str | None = None, + agent_event_retention_days: int | None = None, turn_model: str = DEFAULT_TURN_MODEL, acknowledged_final_retention_days: int = ACKNOWLEDGED_FINAL_RETENTION_DAYS, acknowledged_final_retention_count: int = ACKNOWLEDGED_FINAL_RETENTION_COUNT, @@ -16989,7 +17025,47 @@ def maybe_run_automatic_store_maintenance( or acknowledged_final_retention_count > _SQLITE_MAX_INTEGER ): raise ValueError("acknowledged final retention values are too large") + if agent_event_host_id is None and agent_event_retention_days is not None: + raise ValueError("agent_event_host_id is required for event retention") + normalized_agent_event_host = ( + normalize_agent_event_identifier( + agent_event_host_id, + "agent_event_host_id", + required=True, + ) + if agent_event_host_id is not None + else None + ) + agent_event_days = ( + policy.retention_days + if agent_event_retention_days is None + else agent_event_retention_days + ) + if ( + normalized_agent_event_host is not None + and ( + isinstance(agent_event_days, bool) + or not isinstance(agent_event_days, int) + or not 1 <= agent_event_days <= _MAX_RETENTION_DAYS + ) + ): + raise ValueError("agent_event_retention_days must be positive") current_at = _connector_now(now) + agent_event_cutoff_at = _utc_cutoff( + retention_days=agent_event_days, + now=current_at, + ) + empty_agent_events = { + "host_id": normalized_agent_event_host, + "retention_days": agent_event_days, + "cutoff_at": agent_event_cutoff_at, + "batch_size": policy.batch_size, + "examined": 0, + "deleted": 0, + "tombstoned": 0, + "remaining_candidates": False, + "replay_identity_retained": True, + } empty_command_requests = _command_request_maintenance_summary( None, retry_horizon_seconds=command_retry_horizon_seconds, @@ -17020,6 +17096,7 @@ def maybe_run_automatic_store_maintenance( "deleted": 0, "remaining_candidates": False, }, + "agent_events": empty_agent_events, "final_retention": { "examined": 0, "deleted": 0, @@ -17042,6 +17119,8 @@ def maybe_run_automatic_store_maintenance( ) with _connect(db_path, isolation_level=None) as conn: _ensure_schema(conn) + if normalized_agent_event_host is not None: + conn.execute("PRAGMA secure_delete=ON") conn.execute("BEGIN IMMEDIATE") try: state = conn.execute( @@ -17077,6 +17156,7 @@ def maybe_run_automatic_store_maintenance( "deleted": 0, "remaining_candidates": False, }, + "agent_events": empty_agent_events, "final_retention": { "examined": 0, "deleted": 0, @@ -17102,6 +17182,18 @@ def maybe_run_automatic_store_maintenance( conn, current=current_at, ) + agent_events = dict(empty_agent_events) + if normalized_agent_event_host is not None: + agent_events.update( + _cleanup_agent_event_retention_conn( + conn, + normalized_agent_event_host, + cutoff_at=agent_event_cutoff_at, + batch_size=policy.batch_size, + dry_run=False, + retired_at=current_at, + ) + ) candidates, _ = _snapshot_retention_candidates_conn( conn, cutoff_at=cutoff_at, @@ -17250,6 +17342,7 @@ def maybe_run_automatic_store_maintenance( "deleted": deleted, "remaining_candidates": bool(remaining_ids), }, + "agent_events": agent_events, "final_retention": { "examined": final_examined, "deleted": final_deleted, @@ -17356,6 +17449,106 @@ def cleanup_event_retention( })) +_AGENT_EVENT_RETENTION_SELECT = """ +SELECT + sequence, host_id, event_id, kind, source, worker_id, visibility, + source_session_id, source_turn_id, source_item_id, source_message_id, + source_event_id, source_sequence, payload_fingerprint, public_payload_json +FROM agent_events +""" + + +def _agent_event_retention_candidate( + row: tuple[Any, ...], +) -> tuple[str, str, int, str]: + """Reduce one bounded metadata row to its durable tombstone fields.""" + public_payload_json = str(row[14]) + public_payload = _json_object(public_payload_json) + if _canonical_json(public_payload) != public_payload_json: + raise StoreSchemaError("invalid_agent_event_projection") + return ( + str(row[1]), + str(row[2]), + int(row[0]), + _agent_event_replay_contract_fingerprint( + event_id=str(row[2]), + kind=str(row[3]), + source=str(row[4]), + worker_id=str(row[5]), + visibility=str(row[6]), + source_session_id=str(row[7]) if row[7] is not None else None, + source_turn_id=str(row[8]) if row[8] is not None else None, + source_item_id=str(row[9]) if row[9] is not None else None, + source_message_id=str(row[10]) if row[10] is not None else None, + source_event_id=str(row[11]) if row[11] is not None else None, + source_sequence=int(row[12]) if row[12] is not None else None, + payload_fingerprint=str(row[13]), + public_payload_json=public_payload_json, + ), + ) + + +def _cleanup_agent_event_retention_conn( + conn: sqlite3.Connection, + host_id: str, + *, + cutoff_at: str, + batch_size: int, + dry_run: bool, + retired_at: str, +) -> dict[str, Any]: + """Retire one batch using streamed metadata, never private payload values.""" + cursor = conn.execute( + _AGENT_EVENT_RETENTION_SELECT + + " WHERE host_id = ? AND observed_at < ?" + + " ORDER BY observed_at, sequence LIMIT ?", + (str(host_id), cutoff_at, int(batch_size) + 1), + ) + candidates: list[tuple[str, str, int, str]] = [] + remaining = False + for row in cursor: + if len(candidates) >= batch_size: + remaining = True + break + if dry_run: + candidates.append((str(row[1]), str(row[2]), int(row[0]), "")) + else: + candidates.append(_agent_event_retention_candidate(row)) + + deleted = 0 + if candidates and not dry_run: + conn.executemany( + """ + INSERT INTO agent_event_tombstones ( + host_id, event_id, sequence, replay_fingerprint, retired_at + ) VALUES (?, ?, ?, ?, ?) + """, + ( + (candidate_host, event_id, sequence, fingerprint, retired_at) + for candidate_host, event_id, sequence, fingerprint in candidates + ), + ) + sequences = [candidate[2] for candidate in candidates] + placeholders = ",".join("?" for _ in sequences) + deleted = int( + conn.execute( + f"DELETE FROM agent_events WHERE host_id = ? " + f"AND sequence IN ({placeholders})", + (str(host_id), *sequences), + ).rowcount + or 0 + ) + if deleted != len(candidates): + raise StoreSchemaError("agent_event_retention_delete_mismatch") + retired = len(candidates) if dry_run else deleted + return { + "examined": len(candidates), + "deleted": retired, + "tombstoned": retired, + "remaining_candidates": remaining, + } + + def cleanup_agent_event_retention( db_path: Path, host_id: str, @@ -17365,7 +17558,7 @@ def cleanup_agent_event_retention( dry_run: bool = False, batch_size: int = 100, ) -> dict[str, Any]: - """Retire private event payloads while retaining compact replay identities.""" + """Retire event payloads while retaining compact replay identities.""" days = max(1, int(retention_days)) bounded_batch = max(1, min(int(batch_size), 1_000)) cutoff_at = _utc_cutoff(retention_days=days, now=now) @@ -17389,79 +17582,32 @@ def cleanup_agent_event_retention( })) with _connect(db_path, isolation_level=None) as conn: _ensure_schema(conn) - # Retention is a privacy boundary. Overwrite retired cells in the main - # database where SQLite can do so; WAL files and backups retain their - # independent operator-managed lifecycle. + # Ask SQLite to scrub deleted cells in modified pages. WAL frames, + # checkpoints, filesystem snapshots, and backups have independent + # lifecycles, so this is not an immediate physical-erasure guarantee. conn.execute("PRAGMA secure_delete=ON") conn.execute("BEGIN IMMEDIATE") try: - rows = conn.execute( - _AGENT_EVENT_SELECT - + " WHERE host_id = ? AND observed_at < ?" - + " ORDER BY observed_at, sequence LIMIT ?", - (str(host_id), cutoff_at, bounded_batch + 1), - ).fetchall() - candidates = rows[:bounded_batch] - deleted = 0 - if candidates and not dry_run: - retired_at = utc_timestamp() - for row in candidates: - stored = _agent_event_from_row(row) - conn.execute( - """ - INSERT INTO agent_event_tombstones ( - host_id, event_id, sequence, - replay_fingerprint, retired_at - ) VALUES (?, ?, ?, ?, ?) - """, - ( - stored.host_id, - stored.event.event_id, - stored.sequence, - _agent_event_replay_fingerprint(stored.event), - retired_at, - ), - ) - sequences = [int(row[0]) for row in candidates] - placeholders = ",".join("?" for _ in sequences) - deleted = int( - conn.execute( - f"DELETE FROM agent_events WHERE host_id = ? " - f"AND sequence IN ({placeholders})", - (str(host_id), *sequences), - ).rowcount - or 0 - ) - if deleted != len(candidates): - raise StoreSchemaError("agent_event_retention_delete_mismatch") + result = _cleanup_agent_event_retention_conn( + conn, + str(host_id), + cutoff_at=cutoff_at, + batch_size=bounded_batch, + dry_run=bool(dry_run), + retired_at=utc_timestamp(), + ) if dry_run: - remaining = len(rows) > bounded_batch conn.rollback() else: - remaining = bool( - conn.execute( - """ - SELECT 1 FROM agent_events - WHERE host_id = ? AND observed_at < ? - LIMIT 1 - """, - (str(host_id), cutoff_at), - ).fetchone() - ) conn.commit() except Exception: conn.rollback() raise - examined = len(candidates) - retired = examined if dry_run else deleted return dict(sanitize_public_value({ **base, "ok": True, "status": "ok", - "examined": examined, - "deleted": retired, - "tombstoned": retired, - "remaining_candidates": remaining, + **result, })) diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py index 62aefcb..de37f0a 100644 --- a/tests/test_agent_events.py +++ b/tests/test_agent_events.py @@ -2,6 +2,9 @@ import hashlib import sqlite3 +import threading +import time +import tracemalloc from concurrent.futures import ThreadPoolExecutor from dataclasses import replace from datetime import datetime, timezone @@ -535,12 +538,344 @@ def test_retention_removes_private_payload_but_preserves_replay_identity( ) +def test_retention_streams_metadata_for_exact_16_mib_private_row( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "large-retention.db" + text_limit = 4 * 1024 * 1024 + empty_overhead = len( + store_sqlite._canonical_json( + {"part_a": "", "part_b": "", "part_c": "", "part_d": ""} + ).encode("utf-8") + ) + payload = { + "part_a": "a" * text_limit, + "part_b": "b" * text_limit, + "part_c": "c" * text_limit, + "part_d": "d" * (text_limit - empty_overhead), + } + assert len(store_sqlite._canonical_json(payload).encode("utf-8")) == ( + AGENT_EVENT_MAX_PAYLOAD_BYTES + ) + event = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_sequence=91, + visibility="private", + payload=payload, + observed_at="2026-06-01T00:00:00+00:00", + ) + store_sqlite.append_agent_event(db_path, "host-1", event) + replay_contract = { + "event_id": event.event_id, + "kind": event.kind, + "source": event.source, + "worker_id": event.worker_id, + "visibility": event.visibility, + "source_session_id": event.source_session_id, + "source_turn_id": event.source_turn_id, + "source_item_id": event.source_item_id, + "source_message_id": event.source_message_id, + "source_event_id": event.source_event_id, + "source_sequence": event.source_sequence, + "payload_fingerprint": event.payload_fingerprint, + "public_payload_fingerprint": hashlib.sha256( + store_sqlite._canonical_json(event.public_payload).encode("utf-8") + ).hexdigest(), + } + expected_fingerprint = hashlib.sha256( + store_sqlite._canonical_json(replay_contract).encode("utf-8") + ).hexdigest() + + assert "private_payload_json" not in ( + store_sqlite._AGENT_EVENT_RETENTION_SELECT.lower() + ) + + def forbidden_full_row(_row: object) -> object: + raise AssertionError("retention must not materialize a private event row") + + monkeypatch.setattr(store_sqlite, "_agent_event_from_row", forbidden_full_row) + tracemalloc.start() + result = store_sqlite.cleanup_agent_event_retention( + db_path, + "host-1", + retention_days=7, + now="2026-07-31T00:00:00+00:00", + ) + _current_bytes, peak_bytes = tracemalloc.get_traced_memory() + tracemalloc.stop() + + assert result["deleted"] == result["tombstoned"] == 1 + assert peak_bytes < 8 * 1024 * 1024 + with sqlite3.connect(db_path) as conn: + assert conn.execute( + "SELECT replay_fingerprint FROM agent_event_tombstones " + "WHERE host_id = ? AND event_id = ?", + ("host-1", event.event_id), + ).fetchone() == (expected_fingerprint,) + + +def test_automatic_maintenance_retires_agent_events_only_when_due( + tmp_path: Path, +) -> None: + db_path = tmp_path / "automatic-agent-retention.db" + old = replace( + _message_event(sequence=1, text="old", visibility="private"), + observed_at="2026-01-01T00:00:00+00:00", + ) + recent = replace( + _message_event(sequence=2, text="recent", visibility="private"), + observed_at="2026-01-31T23:00:00+00:00", + ) + store_sqlite.append_agent_event(db_path, "host-1", old) + store_sqlite.append_agent_event(db_path, "host-1", recent) + policy = store_sqlite.SnapshotRetentionPolicy( + retention_days=30, + retention_count=100, + batch_size=10, + ) + + first = store_sqlite.maybe_run_automatic_store_maintenance( + db_path, + policy=policy, + agent_event_host_id="host-1", + agent_event_retention_days=7, + cadence_seconds=3600, + now="2026-02-01T00:00:00+00:00", + ) + late_old = replace( + _message_event(sequence=3, text="late old", visibility="private"), + observed_at="2026-01-02T00:00:00+00:00", + ) + store_sqlite.append_agent_event(db_path, "host-1", late_old) + not_due = store_sqlite.maybe_run_automatic_store_maintenance( + db_path, + policy=policy, + agent_event_host_id="host-1", + agent_event_retention_days=7, + cadence_seconds=3600, + now="2026-02-01T00:10:00+00:00", + ) + second = store_sqlite.maybe_run_automatic_store_maintenance( + db_path, + policy=policy, + agent_event_host_id="host-1", + agent_event_retention_days=7, + cadence_seconds=3600, + now="2026-02-01T01:00:00+00:00", + ) + + assert first["agent_events"]["deleted"] == 1 + assert not_due["status"] == "not_due" + assert not_due["agent_events"]["deleted"] == 0 + assert second["agent_events"]["deleted"] == 1 + assert [ + item.event.payload["text"] + for item in store_sqlite.list_agent_events(db_path, "host-1") + ] == ["recent"] + replay = store_sqlite.append_agent_event(db_path, "host-1", old) + assert replay.inserted is False + with pytest.raises(AgentEventIdentityConflict): + store_sqlite.append_agent_event( + db_path, + "host-1", + _message_event(sequence=1, text="changed", visibility="private"), + ) + + +def test_automatic_agent_retention_failure_does_not_advance_cadence( + tmp_path: Path, +) -> None: + db_path = tmp_path / "automatic-agent-rollback.db" + old = replace( + _message_event(sequence=1, visibility="private"), + observed_at="2026-01-01T00:00:00+00:00", + ) + inserted = store_sqlite.append_agent_event(db_path, "host-1", old) + with sqlite3.connect(db_path) as conn: + conn.execute( + """ + INSERT INTO agent_event_tombstones ( + host_id, event_id, sequence, replay_fingerprint, retired_at + ) VALUES (?, ?, ?, ?, ?) + """, + ( + "host-1", + old.event_id, + inserted.sequence, + "0" * 64, + "2026-01-02T00:00:00+00:00", + ), + ) + + with pytest.raises(sqlite3.IntegrityError): + store_sqlite.maybe_run_automatic_store_maintenance( + db_path, + policy=store_sqlite.SnapshotRetentionPolicy( + retention_days=30, + retention_count=100, + batch_size=10, + ), + agent_event_host_id="host-1", + agent_event_retention_days=7, + now="2026-02-01T00:00:00+00:00", + ) + + with sqlite3.connect(db_path) as conn: + assert conn.execute( + "SELECT last_completed_at FROM store_maintenance_state " + "WHERE scope = 'automatic'" + ).fetchone() == (None,) + assert conn.execute("SELECT COUNT(*) FROM agent_events").fetchone() == (1,) + + +def test_retention_conflict_rolls_back_and_serializes_concurrent_append( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "retention-concurrency.db" + old = replace( + _message_event(sequence=1, visibility="private"), + observed_at="2026-01-01T00:00:00+00:00", + ) + store_sqlite.append_agent_event(db_path, "host-1", old) + entered = threading.Event() + release = threading.Event() + original = store_sqlite._agent_event_retention_candidate + + def blocking_candidate( + row: tuple[object, ...], + ) -> tuple[str, str, int, str]: + entered.set() + assert release.wait(timeout=5) + return original(row) + + monkeypatch.setattr( + store_sqlite, + "_agent_event_retention_candidate", + blocking_candidate, + ) + with ThreadPoolExecutor(max_workers=2) as executor: + cleanup = executor.submit( + store_sqlite.cleanup_agent_event_retention, + db_path, + "host-1", + retention_days=7, + now="2026-02-01T00:00:00+00:00", + ) + assert entered.wait(timeout=5) + append = executor.submit( + store_sqlite.append_agent_event, + db_path, + "host-1", + _message_event(sequence=2, text="new", visibility="private"), + ) + time.sleep(0.05) + assert append.done() is False + release.set() + assert cleanup.result(timeout=5)["deleted"] == 1 + assert append.result(timeout=5).inserted is True + + +def test_retention_tombstone_conflict_rolls_back_active_event(tmp_path: Path) -> None: + db_path = tmp_path / "retention-rollback.db" + old = replace( + _message_event(sequence=1, visibility="private"), + observed_at="2026-01-01T00:00:00+00:00", + ) + inserted = store_sqlite.append_agent_event(db_path, "host-1", old) + with sqlite3.connect(db_path) as conn: + conn.execute( + """ + INSERT INTO agent_event_tombstones ( + host_id, event_id, sequence, replay_fingerprint, retired_at + ) VALUES (?, ?, ?, ?, ?) + """, + ( + "host-1", + old.event_id, + inserted.sequence, + "0" * 64, + "2026-01-02T00:00:00+00:00", + ), + ) + + with pytest.raises(sqlite3.IntegrityError): + store_sqlite.cleanup_agent_event_retention( + db_path, + "host-1", + retention_days=7, + now="2026-02-01T00:00:00+00:00", + ) + + assert store_sqlite.list_agent_events(db_path, "host-1")[0].event == old + with sqlite3.connect(db_path) as conn: + assert conn.execute( + "SELECT replay_fingerprint FROM agent_event_tombstones " + "WHERE host_id = ? AND event_id = ?", + ("host-1", old.event_id), + ).fetchone() == ("0" * 64,) + + def test_journal_accepts_acp_sized_private_text(tmp_path: Path) -> None: event = _message_event(sequence=1, text="x" * (64 * 1024), visibility="private") result = store_sqlite.append_agent_event(tmp_path / "store.db", "host-1", event) assert result.inserted is True +def test_v23_to_v24_preserves_populated_journal_sequence(tmp_path: Path) -> None: + db_path = tmp_path / "v23-populated.db" + event = _message_event(sequence=81, text="x" * (60 * 1024), visibility="private") + with sqlite3.connect(db_path) as conn: + store_sqlite._run_migrations(conn, target_version=23) + conn.execute( + """ + INSERT INTO agent_events ( + sequence, host_id, event_id, kind, source, worker_id, + visibility, source_session_id, source_turn_id, + source_item_id, source_message_id, source_event_id, + source_sequence, observed_at, payload_fingerprint, + private_payload_json, public_payload_json + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + 123, + "host-1", + event.event_id, + event.kind, + event.source, + event.worker_id, + event.visibility, + event.source_session_id, + event.source_turn_id, + event.source_item_id, + event.source_message_id, + event.source_event_id, + event.source_sequence, + event.observed_at, + event.payload_fingerprint, + store_sqlite._canonical_json(event.payload), + store_sqlite._canonical_json(event.public_payload), + ), + ) + conn.commit() + store_sqlite._run_migrations(conn) + assert conn.execute("PRAGMA user_version").fetchone() == (24,) + assert conn.execute( + "SELECT sequence FROM agent_events WHERE event_id = ?", + (event.event_id,), + ).fetchone() == (123,) + assert conn.execute( + "SELECT COUNT(*) FROM agent_event_tombstones" + ).fetchone() == (0,) + + later = _message_event(sequence=82, text="later", visibility="private") + assert store_sqlite.append_agent_event(db_path, "host-1", later).sequence == 124 + + @pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) def test_agent_event_schema_migrates_from_every_prior_version( tmp_path: Path, diff --git a/tests/test_daemon.py b/tests/test_daemon.py index 5028fca..7663f23 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -2213,6 +2213,64 @@ def _assert_private_daemon_failure( assert value not in rendered +def test_snapshot_maintenance_wires_agent_event_retention_without_socket( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "maintenance-wiring.db" + config = Config( + host_id="daemon-host", + data_dir=tmp_path, + db_path=db_path, + event_retention_days=9, + snapshot_maintenance_batch_size=13, + ) + init_store(db_path) + captured: dict[str, Any] = {} + + def maintenance(path: Path, **kwargs: Any) -> dict[str, Any]: + captured.update({"path": path, **kwargs}) + return { + "schema_version": 1, + "ok": True, + "status": "ok", + "due": False, + "snapshot": { + "examined": 0, + "deleted": 0, + "remaining_candidates": False, + }, + "agent_events": { + "examined": 5, + "deleted": 4, + "remaining_candidates": True, + }, + } + + monkeypatch.setattr( + "tendwire.store.sqlite.maybe_run_automatic_store_maintenance", + maintenance, + ) + daemon = TendwireDaemon(config) + daemon._after_snapshot_saved() + + assert captured["path"] == db_path + assert captured["agent_event_host_id"] == "daemon-host" + assert captured["agent_event_retention_days"] == 9 + assert captured["policy"].batch_size == 13 + assert daemon._automatic_maintenance_status == { + "ok": True, + "status": "ok", + "due": False, + "examined": 0, + "deleted": 0, + "remaining_candidates": False, + "agent_events_examined": 5, + "agent_events_deleted": 4, + "agent_events_remaining_candidates": True, + } + + @_UNIX_SOCKET_TEST def test_cli_snapshot_barrier_checks_maintenance_once_and_reads_do_not( tmp_path: Path, @@ -2234,7 +2292,9 @@ def test_cli_snapshot_barrier_checks_maintenance_once_and_reads_do_not( command_receipt_retention_seconds=691_200, command_receipt_retention_count=77, ) - calls: list[tuple[Path, Any, int, int, int, int, int, int]] = [] + calls: list[ + tuple[Path, Any, str | None, int | None, int, int, int, int, int, int] + ] = [] def observe(_config: Config) -> Snapshot: snapshot = _public_snapshot() @@ -2245,6 +2305,8 @@ def maintenance( path: Path, *, policy: Any, + agent_event_host_id: str | None = None, + agent_event_retention_days: int | None = None, turn_model: str = "legacy", acknowledged_final_retention_days: int = 30, acknowledged_final_retention_count: int = 4096, @@ -2260,6 +2322,8 @@ def maintenance( ( path, policy, + agent_event_host_id, + agent_event_retention_days, acknowledged_final_retention_days, acknowledged_final_retention_count, command_retry_horizon_seconds, @@ -2280,6 +2344,11 @@ def maintenance( "deleted": 0, "remaining_candidates": False, }, + "agent_events": { + "examined": 2, + "deleted": 1, + "remaining_candidates": True, + }, "batch_size": policy.batch_size, } @@ -2302,19 +2371,44 @@ def maintenance( daemon.stop() assert len(calls) == 1 - path, policy, final_days, final_count, retry_horizon, retention_seconds, retention_count, cadence = calls[0] + ( + path, + policy, + agent_host, + agent_days, + final_days, + final_count, + retry_horizon, + retention_seconds, + retention_count, + cadence, + ) = calls[0] assert path == db_path assert ( policy.retention_days, policy.retention_count, policy.batch_size, + agent_host, + agent_days, final_days, final_count, retry_horizon, retention_seconds, retention_count, cadence, - ) == (21, 123, 17, 33, 456, 120, 691_200, 77, 91) + ) == ( + 21, + 123, + 17, + "daemon-host", + config.event_retention_days, + 33, + 456, + 120, + 691_200, + 77, + 91, + ) assert health["store"]["maintenance"]["last_check"] == { "ok": True, "status": "not_due", @@ -2322,6 +2416,9 @@ def maintenance( "examined": 0, "deleted": 0, "remaining_candidates": False, + "agent_events_examined": 2, + "agent_events_deleted": 1, + "agent_events_remaining_candidates": True, } @@ -2373,6 +2470,9 @@ def maintenance_failure(*_args: Any, **_kwargs: Any) -> dict[str, Any]: "examined": 0, "deleted": 0, "remaining_candidates": False, + "agent_events_examined": 0, + "agent_events_deleted": 0, + "agent_events_remaining_candidates": False, } assert str(tmp_path) not in encoded assert "secret.db" not in encoded From a9c0448f8ce7fcb5125ae663274b1c1a10fba5fc Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:47:51 +0800 Subject: [PATCH 25/83] fix(acp): harden v1 projection decoding --- src/tendwire/backends/acp_projection.py | 423 ++++++++++++++++-------- tests/test_acp_projection.py | 140 +++++++- 2 files changed, 402 insertions(+), 161 deletions(-) diff --git a/src/tendwire/backends/acp_projection.py b/src/tendwire/backends/acp_projection.py index a6ec4f5..312d295 100644 --- a/src/tendwire/backends/acp_projection.py +++ b/src/tendwire/backends/acp_projection.py @@ -86,6 +86,7 @@ _CONTENT_TYPES: Final[frozenset[str]] = frozenset( {"text", "image", "audio", "resource_link", "resource"} ) +_ANNOTATION_ROLES: Final[frozenset[str]] = frozenset({"assistant", "user"}) class AcpProjectionError(ValueError): @@ -151,6 +152,7 @@ def __init__( max_plan_entries: int = 4096, max_text_chars_per_message: int = 4 * 1024 * 1024, max_event_bytes: int = 8 * 1024 * 1024, + max_json_depth: int = 128, max_session_state_bytes: int = 8 * 1024 * 1024, max_total_state_bytes: int = 128 * 1024 * 1024, ) -> None: @@ -163,6 +165,7 @@ def __init__( "max_plan_entries": max_plan_entries, "max_text_chars_per_message": max_text_chars_per_message, "max_event_bytes": max_event_bytes, + "max_json_depth": max_json_depth, "max_session_state_bytes": max_session_state_bytes, "max_total_state_bytes": max_total_state_bytes, } @@ -178,6 +181,7 @@ def __init__( self._max_plan_entries = max_plan_entries self._max_text_chars_per_message = max_text_chars_per_message self._max_event_bytes = max_event_bytes + self._max_json_depth = max_json_depth self._max_session_state_bytes = max_session_state_bytes self._max_total_state_bytes = max_total_state_bytes @@ -212,9 +216,9 @@ def normalize_session_update( {"update": update, "_meta": params.get("_meta")}, label="ACP session update", max_bytes=self._max_event_bytes, + max_depth=self._max_json_depth, ) - _extension_metadata(params) - _validate_supported_update(update_name, update) + update = _normalized_supported_update(update_name, update) state, _is_new_session = self._pending_session(session_id) explicit_id = _explicit_source_event_id(source_event_id) replay_digest = _event_digest(kind, update) @@ -262,6 +266,10 @@ def normalize_session_update( if active is not None and not active.explicit: state.active_message = None if explicit_id is not None: + self._reserve_state( + state, + len(explicit_id.encode("utf-8")) + len(replay_digest.encode("ascii")), + ) state.seen_source_events[explicit_id] = replay_digest state.replaced_state = None self._sessions[session_id] = state @@ -300,9 +308,11 @@ def normalize_permission_request( params, label="ACP permission request", max_bytes=self._max_event_bytes, + max_depth=self._max_json_depth, + ) + tool_call = _normalized_tool_update( + tool_call, label="ACP permission toolCall" ) - _extension_metadata(params) - _validate_tool_update(tool_call, label="ACP permission toolCall") state, _is_new_session = self._pending_session(session_id) explicit_id = _explicit_source_event_id(source_event_id) options = params.get("options") @@ -355,6 +365,10 @@ def normalize_permission_request( if active is not None and not active.explicit: state.active_message = None if explicit_id is not None: + self._reserve_state( + state, + len(explicit_id.encode("utf-8")) + len(replay_digest.encode("ascii")), + ) state.seen_source_events[explicit_id] = replay_digest state.replaced_state = None self._sessions[session_id] = state @@ -529,9 +543,10 @@ def _normalize_message( raise AcpProjectionError("ACP messageId was reused after a message boundary") text_delta = content.get("text") if content.get("type") == "text" else None text_delta = text_delta if isinstance(text_delta, str) else "" + public_text_delta = text_delta if _content_is_user_visible(content) else "" content_copy = _content_payload(content) previous_text = assembly.text if assembly is not None else "" - assembled_text = previous_text + text_delta + assembled_text = previous_text + public_text_delta if len(assembled_text) > self._max_text_chars_per_message: raise AcpProjectionError("ACP assembled message text limit exceeded") if assembly is None: @@ -539,7 +554,8 @@ def _normalize_message( raise AcpProjectionError("ACP message assembly limit exceeded") self._reserve_state( state, - len(message_id.encode("utf-8")) + len(text_delta.encode("utf-8")), + len(message_id.encode("utf-8")) + + len(public_text_delta.encode("utf-8")), ) assembly = _MessageAssembly( message_id=message_id, @@ -550,7 +566,7 @@ def _normalize_message( if not explicit: state.implicit_message_ordinals[kind] = int(message_id.rsplit("-", 1)[1]) else: - self._reserve_state(state, len(text_delta.encode("utf-8"))) + self._reserve_state(state, len(public_text_delta.encode("utf-8"))) assembly.text = assembled_text state.active_message = (kind, message_id) return { @@ -760,13 +776,23 @@ def _content_payload(content: Mapping[str, Any]) -> dict[str, Any]: return deepcopy(dict(content)) +def _content_is_user_visible(content: Mapping[str, Any]) -> bool: + annotations = content.get("annotations") + if not isinstance(annotations, Mapping) or "audience" not in annotations: + return True + audience = annotations.get("audience") + if audience is None: + return True + return isinstance(audience, list) and "user" in audience + + def _extension_metadata(value: Mapping[str, Any]) -> dict[str, Any]: meta = value.get("_meta") - if meta is None: + if meta is None or not isinstance(meta, Mapping) or any( + not isinstance(key, str) for key in meta + ): return {} - if not isinstance(meta, Mapping) or any(not isinstance(key, str) for key in meta): - raise AcpProjectionError("ACP _meta must be an object with string keys") - return deepcopy(dict(meta)) + return _safe_deepcopy(dict(meta), label="ACP _meta") def _scoped_metadata(**values: Mapping[str, Any]) -> dict[str, Any]: @@ -779,188 +805,272 @@ def _scoped_metadata(**values: Mapping[str, Any]) -> dict[str, Any]: } -def _permission_options(options: list[Any]) -> list[dict[str, Any]]: - normalized: list[dict[str, Any]] = [] - seen: set[str] = set() - for option in options: - if not isinstance(option, Mapping): - raise AcpProjectionError("ACP permission request option must be an object") - option_id = _required_string(option, "optionId") - _required_string(option, "name") - kind = _required_string(option, "kind") - if kind not in _PERMISSION_KINDS: - raise AcpProjectionError("ACP permission option has invalid kind") - _extension_metadata(option) - if option_id in seen: - raise AcpProjectionError("ACP permission option IDs must be unique") - seen.add(option_id) - normalized.append(deepcopy(dict(option))) - return normalized - +def _normalized_supported_update( + update_name: str, update: Mapping[str, Any] +) -> dict[str, Any]: + """Apply ACP v1's tolerant decoding rules to optional/list fields.""" -def _validate_supported_update(update_name: str, update: Mapping[str, Any]) -> None: - _extension_metadata(update) + normalized = _safe_deepcopy(dict(update), label="ACP session update") + _salvage_meta(normalized) if update_name in { "user_message_chunk", "agent_message_chunk", "agent_thought_chunk", }: - content = update.get("content") + content = normalized.get("content") if not isinstance(content, Mapping): raise AcpProjectionError("ACP message update is missing object content") - _validate_content_block(content, label="ACP message content") - if "messageId" in update and update["messageId"] is not None: - _identifier(update["messageId"], "messageId") - return - if update_name == "tool_call": - _validate_tool_call(update) - return - if update_name == "tool_call_update": - _validate_tool_update(update) - return - if update_name == "plan": - entries = update.get("entries") - if not isinstance(entries, list): + normalized["content"] = _normalized_content_block( + content, label="ACP message content" + ) + if "messageId" in normalized and normalized["messageId"] is not None: + _identifier(normalized["messageId"], "messageId") + elif update_name == "tool_call": + normalized = _normalized_tool_call(normalized) + elif update_name == "tool_call_update": + normalized = _normalized_tool_update(normalized) + elif update_name == "plan": + if "entries" not in normalized: raise AcpProjectionError("ACP plan update entries must be an array") - for entry in entries: - _normalized_plan_entry(entry) - return - if update_name == "usage_update": - _validate_usage(update) - return - if update_name == "session_info_update": + entries = normalized["entries"] + if not isinstance(entries, list): + normalized["entries"] = [] + else: + accepted: list[dict[str, Any]] = [] + for entry in entries: + try: + accepted.append(_normalized_plan_entry(entry)) + except AcpProjectionError: + continue + normalized["entries"] = accepted + elif update_name == "usage_update": + _validate_usage(normalized) + if normalized.get("cost") is not None: + try: + cost = normalized["cost"] + if not isinstance(cost, Mapping): + raise AcpProjectionError("ACP usage cost must be an object or null") + cost_copy = _safe_deepcopy(dict(cost), label="ACP usage cost") + _salvage_meta(cost_copy) + _validate_cost(cost_copy) + normalized["cost"] = cost_copy + except (AcpProjectionError, OverflowError): + normalized.pop("cost", None) + elif update_name == "session_info_update": for key in ("title", "updatedAt"): - if key in update and update[key] is not None and not isinstance(update[key], str): - raise AcpProjectionError(f"ACP session info {key} must be text or null") + if key in normalized and normalized[key] is not None and not isinstance( + normalized[key], str + ): + normalized.pop(key) + return normalized + + +def _salvage_meta(value: dict[str, Any]) -> None: + meta = value.get("_meta") + if meta is not None and ( + not isinstance(meta, Mapping) or any(not isinstance(key, str) for key in meta) + ): + value.pop("_meta", None) -def _validate_content_block(content: Mapping[str, Any], *, label: str) -> None: - content_type = content.get("type") +def _normalized_annotations(value: Any) -> dict[str, Any] | None: + if not isinstance(value, Mapping): + return None + normalized = _safe_deepcopy(dict(value), label="ACP annotations") + _salvage_meta(normalized) + audience = normalized.get("audience") + if isinstance(audience, list): + normalized["audience"] = [item for item in audience if item in _ANNOTATION_ROLES] + elif audience is not None: + normalized.pop("audience", None) + if normalized.get("lastModified") is not None and not isinstance( + normalized.get("lastModified"), str + ): + normalized.pop("lastModified", None) + priority = normalized.get("priority") + if priority is not None and not _is_finite_number(priority): + normalized.pop("priority", None) + return normalized + + +def _normalized_content_block( + content: Mapping[str, Any], *, label: str +) -> dict[str, Any]: + normalized = _safe_deepcopy(dict(content), label=label) + content_type = normalized.get("type") if content_type not in _CONTENT_TYPES: raise AcpProjectionError(f"{label} has unsupported type") - _extension_metadata(content) + _salvage_meta(normalized) + annotations = _normalized_annotations(normalized.get("annotations")) + if annotations is None: + normalized.pop("annotations", None) + else: + normalized["annotations"] = annotations if content_type == "text": - _required_text(content, "text", label=label) + _required_text(normalized, "text", label=label) elif content_type in {"image", "audio"}: - _required_text(content, "data", label=label) - _required_text(content, "mimeType", label=label) - if content_type == "image": - _optional_text(content, "uri", label=label) + _required_text(normalized, "data", label=label) + _required_text(normalized, "mimeType", label=label) + if content_type == "image" and normalized.get("uri") is not None and not isinstance( + normalized.get("uri"), str + ): + normalized.pop("uri", None) elif content_type == "resource_link": - _required_text(content, "name", label=label) - _required_text(content, "uri", label=label) + _required_text(normalized, "name", label=label) + _required_text(normalized, "uri", label=label) for key in ("description", "mimeType", "title"): - _optional_text(content, key, label=label) - if "size" in content and content["size"] is not None: - size = content["size"] - if ( - isinstance(size, bool) - or not isinstance(size, int) - or not -(2**63) <= size <= 2**63 - 1 - ): - raise AcpProjectionError(f"{label} size must be an integer or null") + if normalized.get(key) is not None and not isinstance(normalized.get(key), str): + normalized.pop(key, None) + size = normalized.get("size") + if size is not None and ( + isinstance(size, bool) + or not isinstance(size, int) + or not -(2**63) <= size <= 2**63 - 1 + ): + normalized.pop("size", None) else: - resource = content.get("resource") + resource = normalized.get("resource") if not isinstance(resource, Mapping): raise AcpProjectionError(f"{label} resource must be an object") - _extension_metadata(resource) - _required_text(resource, "uri", label=f"{label} resource") - has_text = "text" in resource - has_blob = "blob" in resource + resource_copy = _safe_deepcopy(dict(resource), label=f"{label} resource") + _salvage_meta(resource_copy) + _required_text(resource_copy, "uri", label=f"{label} resource") + has_text = "text" in resource_copy + has_blob = "blob" in resource_copy if has_text == has_blob: raise AcpProjectionError( f"{label} resource must contain exactly one of text or blob" ) _required_text( - resource, + resource_copy, "text" if has_text else "blob", label=f"{label} resource", ) - _optional_text(resource, "mimeType", label=f"{label} resource") - annotations = content.get("annotations") - if annotations is not None and not isinstance(annotations, Mapping): - raise AcpProjectionError(f"{label} annotations must be an object or null") + if resource_copy.get("mimeType") is not None and not isinstance( + resource_copy.get("mimeType"), str + ): + resource_copy.pop("mimeType", None) + normalized["resource"] = resource_copy + return normalized + + +def _permission_options(options: list[Any]) -> list[dict[str, Any]]: + normalized: list[dict[str, Any]] = [] + seen: set[str] = set() + for option in options: + if not isinstance(option, Mapping): + raise AcpProjectionError("ACP permission request option must be an object") + option_copy = _safe_deepcopy(dict(option), label="ACP permission option") + _salvage_meta(option_copy) + option_id = _required_string(option_copy, "optionId") + _required_string(option_copy, "name") + kind = _required_string(option_copy, "kind") + if kind not in _PERMISSION_KINDS: + raise AcpProjectionError("ACP permission option has invalid kind") + if option_id in seen: + raise AcpProjectionError("ACP permission option IDs must be unique") + seen.add(option_id) + normalized.append(option_copy) + return normalized -def _validate_tool_call(update: Mapping[str, Any]) -> None: - _required_string(update, "toolCallId") - if not isinstance(update.get("title"), str): +def _normalized_tool_call(update: Mapping[str, Any]) -> dict[str, Any]: + normalized = _safe_deepcopy(dict(update), label="ACP tool call") + _required_string(normalized, "toolCallId") + if not isinstance(normalized.get("title"), str): raise AcpProjectionError("ACP tool_call is missing string title") - _validate_tool_fields(update, creation=True) + _salvage_tool_fields(normalized, creation=True, label="ACP tool call") + return normalized -def _validate_tool_update( +def _normalized_tool_update( update: Mapping[str, Any], *, label: str = "ACP tool_call_update" -) -> None: - _required_string(update, "toolCallId") - _validate_tool_fields(update, creation=False, label=label) +) -> dict[str, Any]: + normalized = _safe_deepcopy(dict(update), label=label) + _required_string(normalized, "toolCallId") + _salvage_tool_fields(normalized, creation=False, label=label) + return normalized -def _validate_tool_fields( - update: Mapping[str, Any], - *, - creation: bool, - label: str = "ACP tool call", +def _salvage_tool_fields( + update: dict[str, Any], *, creation: bool, label: str ) -> None: - _extension_metadata(update) + _salvage_meta(update) kind = update.get("kind") if kind is not None and (not isinstance(kind, str) or kind not in _TOOL_KINDS): - raise AcpProjectionError(f"{label} has invalid kind") + update.pop("kind", None) status = update.get("status") if status is not None and ( not isinstance(status, str) or status not in _TOOL_STATUSES ): - raise AcpProjectionError(f"{label} has invalid status") - if "title" in update and not creation: - title = update["title"] - if title is not None and not isinstance(title, str): - raise AcpProjectionError(f"{label} title must be text or null") - for key in ("content", "locations"): + update.pop("status", None) + if not creation and update.get("title") is not None and not isinstance( + update.get("title"), str + ): + update.pop("title", None) + for key, normalizer in ( + ("content", _normalized_tool_content), + ("locations", _normalized_tool_location), + ): value = update.get(key) - if value is not None and not isinstance(value, list): - raise AcpProjectionError(f"{label} {key} must be an array or null") - if isinstance(update.get("content"), list): - for item in update["content"]: - _validate_tool_content(item) - if isinstance(update.get("locations"), list): - for location in update["locations"]: - _validate_tool_location(location) - - -def _validate_tool_content(value: Any) -> None: + if not isinstance(value, list): + if value is not None: + update.pop(key, None) + continue + accepted: list[dict[str, Any]] = [] + for item in value: + try: + accepted.append(normalizer(item)) + except AcpProjectionError: + continue + update[key] = accepted + + +def _normalized_tool_content(value: Any) -> dict[str, Any]: if not isinstance(value, Mapping): raise AcpProjectionError("ACP tool content item must be an object") - _extension_metadata(value) - item_type = value.get("type") + normalized = _safe_deepcopy(dict(value), label="ACP tool content item") + _salvage_meta(normalized) + item_type = normalized.get("type") if item_type == "content": - content = value.get("content") + content = normalized.get("content") if not isinstance(content, Mapping): raise AcpProjectionError("ACP tool content is missing content block") - _validate_content_block(content, label="ACP tool content block") + normalized["content"] = _normalized_content_block( + content, label="ACP tool content block" + ) elif item_type == "diff": - path = _required_text(value, "path", label="ACP tool diff") + path = _required_text(normalized, "path", label="ACP tool diff") if not os.path.isabs(path): raise AcpProjectionError("ACP tool diff path must be absolute") - _required_text(value, "newText", label="ACP tool diff") - _optional_text(value, "oldText", label="ACP tool diff") + _required_text(normalized, "newText", label="ACP tool diff") + if normalized.get("oldText") is not None and not isinstance( + normalized.get("oldText"), str + ): + normalized.pop("oldText", None) elif item_type == "terminal": - _required_string(value, "terminalId") + _required_string(normalized, "terminalId") else: raise AcpProjectionError("ACP tool content item has unsupported type") + return normalized -def _validate_tool_location(value: Any) -> None: +def _normalized_tool_location(value: Any) -> dict[str, Any]: if not isinstance(value, Mapping): raise AcpProjectionError("ACP tool location must be an object") - _extension_metadata(value) - path = _required_text(value, "path", label="ACP tool location") + normalized = _safe_deepcopy(dict(value), label="ACP tool location") + _salvage_meta(normalized) + path = _required_text(normalized, "path", label="ACP tool location") if not os.path.isabs(path): raise AcpProjectionError("ACP tool location path must be absolute") - line = value.get("line") + line = normalized.get("line") if line is not None and ( - isinstance(line, bool) or not isinstance(line, int) or not 0 <= line <= 2**32 - 1 + isinstance(line, bool) + or not isinstance(line, int) + or not 0 <= line <= 2**32 - 1 ): - raise AcpProjectionError("ACP tool location line must be a u32 or null") + normalized.pop("line", None) + return normalized def _normalized_plan_entry(value: Any) -> dict[str, Any]: @@ -991,23 +1101,28 @@ def _validate_usage(update: Mapping[str, Any]) -> None: or not 0 <= value <= 2**64 - 1 ): raise AcpProjectionError(f"ACP usage {key} must be a u64") - cost = update.get("cost") - if cost is None: - return + + +def _validate_cost(cost: Any) -> None: if not isinstance(cost, Mapping): raise AcpProjectionError("ACP usage cost must be an object or null") amount = cost.get("amount") - if ( - isinstance(amount, bool) - or not isinstance(amount, (int, float)) - or not math.isfinite(float(amount)) - ): + if not _is_finite_number(amount): raise AcpProjectionError("ACP usage cost amount must be a finite number") if not isinstance(cost.get("currency"), str): raise AcpProjectionError("ACP usage cost currency must be text") _extension_metadata(cost) +def _is_finite_number(value: Any) -> bool: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return False + try: + return math.isfinite(float(value)) + except (ValueError, OverflowError): + return False + + def _required_text(value: Mapping[str, Any], key: str, *, label: str) -> str: item = value.get(key) if not isinstance(item, str): @@ -1015,12 +1130,10 @@ def _required_text(value: Mapping[str, Any], key: str, *, label: str) -> str: return item -def _optional_text(value: Mapping[str, Any], key: str, *, label: str) -> None: - if key in value and value[key] is not None and not isinstance(value[key], str): - raise AcpProjectionError(f"{label} {key} must be text or null") - - -def _bounded_json(value: Any, *, label: str, max_bytes: int) -> bytes: +def _bounded_json( + value: Any, *, label: str, max_bytes: int, max_depth: int +) -> bytes: + _validate_json_depth(value, label=label, max_depth=max_depth) try: encoded = json.dumps( value, @@ -1036,6 +1149,30 @@ def _bounded_json(value: Any, *, label: str, max_bytes: int) -> bytes: return encoded +def _validate_json_depth(value: Any, *, label: str, max_depth: int) -> None: + stack: list[tuple[Any, int]] = [(value, 0)] + visited: set[int] = set() + while stack: + item, depth = stack.pop() + if depth > max_depth: + raise AcpProjectionError(f"{label} exceeds the nesting limit") + if not isinstance(item, (Mapping, list, tuple)): + continue + identity = id(item) + if identity in visited: + continue + visited.add(identity) + children = item.values() if isinstance(item, Mapping) else item + stack.extend((child, depth + 1) for child in children) + + +def _safe_deepcopy(value: Any, *, label: str) -> Any: + try: + return deepcopy(value) + except (TypeError, ValueError, RecursionError, OverflowError) as exc: + raise AcpProjectionError(f"{label} could not be copied safely") from exc + + def _json_size(value: Any) -> int: return len( json.dumps( diff --git a/tests/test_acp_projection.py b/tests/test_acp_projection.py index 5ed9ae5..17232fc 100644 --- a/tests/test_acp_projection.py +++ b/tests/test_acp_projection.py @@ -97,6 +97,41 @@ def test_non_text_content_is_preserved_without_becoming_turn_text() -> None: assert projector.project_turn_content("session-1")["assistant_stream_text"] == "" +def test_assistant_only_text_stays_in_private_event_but_not_legacy_turn() -> None: + projector = AcpEventProjector() + private = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="private-1", + content={ + "type": "text", + "text": "assistant-only context", + "annotations": {"audience": ["assistant"]}, + }, + ) + ) + public = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="public-1", + content={ + "type": "text", + "text": "safe answer", + "annotations": {"audience": ["user", "invalid-role"]}, + }, + ) + ) + + assert private is not None + assert private["payload"]["content"]["text"] == "assistant-only context" + assert private["payload"]["assembled_text"] == "" + assert public is not None + assert public["payload"]["content"]["annotations"]["audience"] == ["user"] + assert projector.project_turn_content("session-1")["assistant_stream_text"] == ( + "safe answer" + ) + + def test_user_and_agent_text_remain_separate_and_reset_starts_new_turn() -> None: projector = AcpEventProjector() projector.normalize_session_update( @@ -302,22 +337,8 @@ def test_official_v1_tool_shapes_are_validated_and_preserved() -> None: (_update("agent_message_chunk", content={"type": "text"}), "string text"), (_update("agent_message_chunk", content={"type": "future"}), "unsupported type"), (_update("tool_call", toolCallId="tool-1"), "string title"), - ( - _update( - "tool_call", - toolCallId="tool-1", - title="Bad status", - status="cancelled", - ), - "invalid status", - ), - ( - _update("plan", entries=[{"content": "missing fields"}]), - "invalid priority", - ), (_update("usage_update", used=1), "usage size"), (_update("usage_update", used=True, size=10), "usage used"), - (_update("session_info_update", title=7), "title must be text"), ], ) def test_malformed_supported_v1_updates_fail_without_allocating_state( @@ -351,6 +372,56 @@ def test_usage_update_is_a_complete_snapshot_and_omission_clears_cost() -> None: } +def test_optional_fields_default_and_invalid_collection_items_are_skipped() -> None: + projector = AcpEventProjector() + tool = projector.normalize_session_update( + _update( + "tool_call", + toolCallId="tool-1", + title="Forward compatible", + kind="future_kind", + status="cancelled", + content=[ + {"type": "future"}, + {"type": "content", "content": {"type": "text", "text": "ok"}}, + ], + locations=[{"path": "/valid"}, {"path": 7}], + _meta="invalid optional metadata", + ) + ) + plan = projector.normalize_session_update( + _update( + "plan", + entries=[ + {"content": "keep", "priority": "high", "status": "pending"}, + {"content": "skip", "priority": "future", "status": "pending"}, + ], + ) + ) + info = projector.normalize_session_update( + _update("session_info_update", title=7, updatedAt="valid") + ) + usage = projector.normalize_session_update( + _update( + "usage_update", + used=1, + size=2, + cost={"amount": 10**400, "currency": "USD"}, + ) + ) + + assert tool is not None + assert tool["payload"]["snapshot"]["kind"] == "other" + assert tool["payload"]["snapshot"]["status"] == "pending" + assert len(tool["payload"]["snapshot"]["content"]) == 1 + assert tool["payload"]["snapshot"]["locations"] == [{"path": "/valid"}] + assert plan is not None and plan["payload"]["entries"] == [ + {"content": "keep", "priority": "high", "status": "pending"} + ] + assert info is not None and info["payload"] == {"updatedAt": "valid"} + assert usage is not None and usage["payload"] == {"used": 1, "size": 2} + + def test_implicit_v1_message_is_split_after_an_update_type_boundary() -> None: projector = AcpEventProjector() first = projector.normalize_session_update( @@ -622,11 +693,11 @@ def test_plan_is_a_validated_full_replacement() -> None: assert replacement["payload"]["entries"] == [ {"content": "new", "priority": "high", "status": "completed"} ] + defaulted = projector.normalize_session_update(_update("plan", entries="bad")) + assert defaulted is not None and defaulted["payload"]["entries"] == [] with pytest.raises(AcpProjectionError, match="entries must be an array"): - projector.normalize_session_update(_update("plan", entries="bad")) - assert projector.session_snapshot("session-1")["plan"] == [ - {"content": "new", "priority": "high", "status": "completed"} - ] + projector.normalize_session_update(_update("plan")) + assert projector.session_snapshot("session-1")["plan"] == [] def test_completion_is_not_reopened_by_session_updates_and_requires_reset() -> None: @@ -719,6 +790,39 @@ def test_failed_or_oversized_input_does_not_allocate_or_mutate_session() -> None assert accepted is not None and accepted["sequence"] == 1 +def test_deep_json_is_rejected_as_a_projection_error_before_copying() -> None: + nested: object = "leaf" + for _ in range(130): + nested = {"next": nested} + projector = AcpEventProjector(max_json_depth=64) + + with pytest.raises(AcpProjectionError, match="nesting limit"): + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={ + "type": "text", + "text": "safe", + "_meta": {"nested": nested}, + }, + ) + ) + assert projector.session_snapshot("session-1") is None + + +def test_replay_index_bytes_count_toward_retained_state_limit() -> None: + projector = AcpEventProjector( + max_session_state_bytes=100, + max_total_state_bytes=100, + ) + with pytest.raises(AcpProjectionError, match="retained session state"): + projector.normalize_session_update( + _update("session_info_update"), + source_event_id="x" * 50, + ) + assert projector.session_snapshot("session-1") is None + + def test_aggregate_retained_session_state_has_a_hard_budget() -> None: projector = AcpEventProjector(max_session_state_bytes=12) with pytest.raises(AcpProjectionError, match="retained session state"): From 2639bc56b975bd8d19f731be368c8024d07d893a Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:50:24 +0800 Subject: [PATCH 26/83] fix(acp): preserve ordered prompt lifecycle --- src/tendwire/backends/acp_client.py | 128 ++++++++++++----- src/tendwire/backends/acp_ingestion.py | 148 ++++++++++++++++++- src/tendwire/backends/acp_runtime.py | 189 ++++++++++++++++--------- tests/fixtures/acp_fake_agent.py | 8 ++ tests/test_acp_client.py | 20 +++ tests/test_acp_ingestion.py | 58 ++++++++ tests/test_acp_runtime.py | 144 +++++++++++++++++-- 7 files changed, 575 insertions(+), 120 deletions(-) diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index 4b9491c..c683f64 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -129,6 +129,7 @@ class _PendingRequest: _T = TypeVar("_T") _END = object() +SessionEvent = SessionUpdate | PermissionRequest class AcpClient: @@ -191,10 +192,14 @@ def __init__( self._cancelled_sessions: set[str] = set() self._active_prompts: dict[str, int] = {} self._permission_lock = threading.Lock() - self._updates: queue.Queue[SessionUpdate | object] = queue.Queue(max_pending_events) - self._permissions: queue.Queue[PermissionRequest | object] = queue.Queue( + # Updates and permission requests share one reader-ordered queue. A + # pair of duplicate queues can both reorder cross-kind events and fail + # the transport when an embedding consumes only one of them. + self._session_events: queue.Queue[SessionEvent | object] = queue.Queue( max_pending_events ) + self._session_event_backlog: deque[SessionEvent] = deque() + self._session_event_lock = threading.Lock() self._notifications: queue.Queue[RawNotification | object] = queue.Queue( max_pending_events ) @@ -579,18 +584,7 @@ def prompt( *, timeout: float | None = None, ) -> PromptResult: - if isinstance(prompt, str): - content: list[Mapping[str, Any]] = [{"type": "text", "text": prompt}] - else: - content = list(prompt) - if not content: - raise ValueError("prompt must contain at least one content block") - self._require_initialized() - assert self.capabilities is not None - content = [ - _validated_prompt_content_block(block, self.capabilities) - for block in content - ] + content = list(self.prepare_prompt(prompt)) session_id = _nonempty(session_id, "session_id") with self._permission_lock: if self._active_prompts.get(session_id, 0) == 0: @@ -623,6 +617,25 @@ def prompt( raise AcpEnvelopeError("session/prompt returned an invalid stopReason") from exc return PromptResult(parsed_reason, MappingProxyType(dict(raw))) + def prepare_prompt( + self, + prompt: str | Sequence[Mapping[str, Any]], + ) -> tuple[Mapping[str, Any], ...]: + """Validate prompt content without sending it to the agent.""" + + if isinstance(prompt, str): + content: list[Mapping[str, Any]] = [{"type": "text", "text": prompt}] + else: + content = list(prompt) + if not content: + raise ValueError("prompt must contain at least one content block") + self._require_initialized() + assert self.capabilities is not None + return tuple( + _validated_prompt_content_block(block, self.capabilities) + for block in content + ) + def cancel(self, session_id: str) -> None: """Cancel a turn and cancel all outstanding permissions for the session.""" session_id = _nonempty(session_id, "session_id") @@ -683,28 +696,68 @@ def respond_permission( raise def next_update(self, *, timeout: float | None = None) -> SessionUpdate: - return self._queue_get(self._updates, timeout, "session update") + return self._next_typed_session_event(SessionUpdate, timeout, "session update") def next_permission_request( self, *, timeout: float | None = None ) -> PermissionRequest: - deadline = None if timeout is None else time.monotonic() + _positive_timeout( - timeout, "timeout" + return self._next_typed_session_event( + PermissionRequest, + timeout, + "permission request", ) - while True: - remaining = None if deadline is None else deadline - time.monotonic() - if remaining is not None and remaining <= 0: - raise AcpRequestTimeoutError( - "timed out waiting for ACP permission request" + + def next_session_event(self, *, timeout: float | None = None) -> SessionEvent: + """Return the next update or permission request in exact reader order.""" + + return self._next_typed_session_event(SessionEvent, timeout, "session event") + + def _next_typed_session_event( + self, + expected: type[_T] | object, + timeout: float | None, + description: str, + ) -> _T: + deadline = None + if timeout is not None: + deadline = time.monotonic() + _positive_timeout(timeout, "timeout") + with self._session_event_lock: + while True: + for index, candidate in enumerate(self._session_event_backlog): + if self._session_event_matches(candidate, expected): + del self._session_event_backlog[index] + return candidate # type: ignore[return-value] + remaining = None if deadline is None else deadline - time.monotonic() + if remaining is not None and remaining <= 0: + raise AcpRequestTimeoutError( + f"timed out waiting for ACP {description}" + ) + candidate = self._queue_get( + self._session_events, + remaining, + description, ) - request = self._queue_get( - self._permissions, - remaining, - "permission request", - ) - with self._permission_lock: - if self._pending_permissions.get(request.request_id) is request: - return request + if isinstance(candidate, PermissionRequest): + with self._permission_lock: + pending = ( + self._pending_permissions.get(candidate.request_id) + is candidate + ) + if not pending: + continue + if self._session_event_matches(candidate, expected): + return candidate # type: ignore[return-value] + if len(self._session_event_backlog) >= self.max_pending_events: + raise AcpEventQueueFullError( + "ACP typed event backlog is full; consume the ordered stream" + ) + self._session_event_backlog.append(candidate) + + @staticmethod + def _session_event_matches(candidate: SessionEvent, expected: object) -> bool: + if expected is SessionEvent: + return True + return isinstance(candidate, expected) # type: ignore[arg-type] def next_notification(self, *, timeout: float | None = None) -> RawNotification: return self._queue_get(self._notifications, timeout, "notification") @@ -1017,7 +1070,10 @@ def _dispatch( return if isinstance(message, JsonRpcNotification): if message.method == "session/update": - self._put_lossless(self._updates, parse_session_update(message.params)) + self._put_lossless( + self._session_events, + parse_session_update(message.params), + ) elif message.method in self.supported_extension_notifications: self._put_lossless( self._notifications, @@ -1055,7 +1111,7 @@ def _dispatch( ) ) else: - self._put_lossless(self._permissions, parsed) + self._put_lossless(self._session_events, parsed) elif message.method in self.supported_extension_requests: self._put_lossless( self._inbound_requests, @@ -1072,7 +1128,10 @@ def _dispatch( def _put_lossless(self, target: queue.Queue[Any], value: Any) -> None: try: - target.put_nowait(value) + # Brief backpressure lets an active ordered consumer drain LOAD + # replay bursts larger than the queue. A genuinely abandoned + # queue still fails closed within a bounded interval. + target.put(value, timeout=min(self.request_timeout, 0.5)) except queue.Full as exc: raise AcpEventQueueFullError( "ACP event queue is full; refusing to drop protocol data" @@ -1140,8 +1199,7 @@ def _fail_pending(self, failure: BaseException) -> None: def _signal_queues(self) -> None: for target in ( - self._updates, - self._permissions, + self._session_events, self._notifications, self._inbound_requests, ): diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py index 7028fc1..7cad6c4 100644 --- a/src/tendwire/backends/acp_ingestion.py +++ b/src/tendwire/backends/acp_ingestion.py @@ -8,7 +8,8 @@ from __future__ import annotations -from collections.abc import Callable, Mapping +from collections.abc import Callable, Mapping, Sequence +from copy import deepcopy from dataclasses import dataclass from pathlib import Path from typing import Any @@ -23,6 +24,7 @@ append_agent_event_and_apply_turn_for_binding, ) from .acp_projection import AcpEventProjector, AcpProjectionCheckpoint +from .acp_protocol import StopReason PersistEvent = Callable[..., AppendProjectedAgentEventResult] @@ -126,6 +128,7 @@ def ingest_update( *, source_event_id: str | None = None, replay: bool = False, + setup_replay: bool = False, ) -> AcpIngestionResult: """Normalize, journal, and conditionally project ``session/update``.""" @@ -165,14 +168,82 @@ def ingest_update( canonical, checkpoint=checkpoint, prior_turn_state=prior_turn_state, + project_turn=not setup_replay, + replay_namespace="load" if setup_replay else None, ) + def begin_prompt( + self, + prompt: Sequence[Mapping[str, Any]], + *, + producer_turn_id: str | None = None, + ) -> AcpIngestionResult: + """Durably record outgoing prompt content before transport send.""" + + blocks = [dict(block) for block in prompt] + if not blocks: + raise ValueError("prompt must contain at least one content block") + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + source_turn_id = self.start_turn(producer_turn_id=producer_turn_id) + text = "\n".join( + str(block.get("text")) + for block in blocks + if block.get("type") == "text" and isinstance(block.get("text"), str) + ) + source_event_id = f"prompt-input:{source_turn_id}" + try: + canonical = self.projector.normalize_session_update( + { + "method": "session/update", + "params": { + "sessionId": self.session_id, + "update": { + "sessionUpdate": "user_message_chunk", + "messageId": source_event_id, + "content": {"type": "text", "text": text}, + }, + }, + }, + source_event_id=source_event_id, + replay=False, + ) + if canonical is None: # pragma: no cover - fresh turn invariant + raise RuntimeError("outgoing ACP prompt was unexpectedly duplicated") + payload = canonical.get("payload") + if not isinstance(payload, Mapping): # pragma: no cover - projector invariant + raise RuntimeError("outgoing ACP prompt projection is invalid") + canonical = { + **canonical, + "payload": { + **payload, + "prompt_content": deepcopy(blocks), + "outgoing": True, + }, + } + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + return self._accept( + canonical, + checkpoint=checkpoint, + prior_turn_state=prior_turn_state, + ) + + def reset_after_load(self) -> None: + """Drop replay turn assembly before accepting a new active prompt.""" + + self.projector.reset_turn(self.session_id) + self._source_turn_id = None + self._turn_complete = False + def ingest_permission_request( self, request: Mapping[str, Any], *, source_event_id: str | None = None, replay: bool = False, + setup_replay: bool = False, ) -> AcpIngestionResult: """Journal a permission request as a private tool lifecycle update.""" @@ -205,9 +276,14 @@ def ingest_permission_request( canonical, checkpoint=checkpoint, prior_turn_state=prior_turn_state, + project_turn=not setup_replay, + replay_namespace="load" if setup_replay else None, ) - def mark_prompt_complete(self) -> AcpIngestionResult: + def mark_prompt_complete( + self, + stop_reason: StopReason | str = StopReason.END_TURN, + ) -> AcpIngestionResult: """Durably finalize the current turn after ``session/prompt`` returns.""" if self._source_turn_id is None: @@ -217,8 +293,16 @@ def mark_prompt_complete(self) -> AcpIngestionResult: checkpoint = self.projector.checkpoint_session(self.session_id) prior_turn_state = self._turn_state() try: + try: + normalized_reason = StopReason(stop_reason) + except ValueError as exc: + raise ValueError("unsupported ACP prompt stop reason") from exc content = self.projector.mark_turn_complete(self.session_id) content["source_turn_id"] = self._source_turn_id + content["assistant_final_text"] = _final_text_for_stop_reason( + str(content.get("assistant_final_text") or ""), + normalized_reason, + ) marker = agent_event( kind="extension", source="acp", @@ -227,6 +311,8 @@ def mark_prompt_complete(self) -> AcpIngestionResult: "schema_version": 1, "extension": "tendwire.acp.prompt_completion", "complete": True, + "stop_reason": normalized_reason.value, + "outcome": _STOP_REASON_OUTCOMES[normalized_reason], "projection": content, }, source_session_id=self.session_id, @@ -275,6 +361,8 @@ def _accept( *, checkpoint: AcpProjectionCheckpoint, prior_turn_state: tuple[int, str | None, bool], + project_turn: bool = True, + replay_namespace: str | None = None, ) -> AcpIngestionResult: kind = str(canonical.get("kind") or "") if kind == "thought" and self.config.acp_thought_policy == "disabled": @@ -291,7 +379,17 @@ def _accept( source_id = ( str(explicit_event_id) if explicit_event_id is not None and str(explicit_event_id) - else f"stream:{self.stream_generation}:{sequence}" + else ( + _replay_source_event_id( + replay_namespace, + session_id=self.session_id, + sequence=sequence, + kind=kind, + payload=payload, + ) + if replay_namespace is not None + else f"stream:{self.stream_generation}:{sequence}" + ) ) event = agent_event( kind=kind, @@ -312,6 +410,7 @@ def _accept( if ( kind in {"user_message", "agent_message"} and self.config.agent_event_source != "acp_shadow" + and project_turn ): content = self.projector.project_turn_content(self.session_id) if self._source_turn_id is not None: @@ -362,6 +461,49 @@ def _restore_speculation( self.projector.restore_session(checkpoint) self._turn_ordinal, self._source_turn_id, self._turn_complete = prior_turn_state + +_STOP_REASON_OUTCOMES = { + StopReason.END_TURN: "completed", + StopReason.MAX_TOKENS: "truncated_max_tokens", + StopReason.MAX_TURN_REQUESTS: "truncated_max_turn_requests", + StopReason.REFUSAL: "refused", + StopReason.CANCELLED: "cancelled", +} + +_STOP_REASON_NOTICES = { + StopReason.MAX_TOKENS: "[ACP response truncated: token limit reached]", + StopReason.MAX_TURN_REQUESTS: "[ACP response truncated: request limit reached]", + StopReason.REFUSAL: "[ACP agent refused the request]", + StopReason.CANCELLED: "[ACP prompt cancelled]", +} + + +def _final_text_for_stop_reason(text: str, stop_reason: StopReason) -> str: + notice = _STOP_REASON_NOTICES.get(stop_reason) + if notice is None: + return text + return f"{text}\n\n{notice}" if text else notice + + +def _replay_source_event_id( + namespace: str, + *, + session_id: str, + sequence: int, + kind: str, + payload: Mapping[str, Any], +) -> str: + fingerprint = stable_fingerprint( + { + "session": session_id, + "sequence": sequence, + "kind": kind, + "payload": payload, + } + ) + return f"{namespace}:{fingerprint}" + + def _source_message_id(kind: str, payload: Mapping[str, Any]) -> str | None: if kind not in {"user_message", "agent_message", "thought"}: return None diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 81b661d..550f4bc 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -144,11 +144,18 @@ def prompt( timeout: float | None = None, ) -> PromptResult: ... - def cancel(self, session_id: str) -> None: ... + def prepare_prompt( + self, + prompt: str | Sequence[Mapping[str, Any]], + ) -> tuple[Mapping[str, Any], ...]: ... - def next_update(self, *, timeout: float) -> SessionUpdate: ... + def cancel(self, session_id: str) -> None: ... - def next_permission_request(self, *, timeout: float) -> PermissionRequest: ... + def next_session_event( + self, + *, + timeout: float, + ) -> SessionUpdate | PermissionRequest: ... def respond_permission( self, @@ -257,8 +264,8 @@ def __init__( self._idle_condition = threading.Condition(self._state_lock) self._stop_event = threading.Event() self._threads: tuple[threading.Thread, ...] = () - self._update_idle_epoch = 0 - self._permission_idle_epoch = 0 + self._event_idle_epoch = 0 + self._setup_replay = False self._close_thread: threading.Thread | None = None self._close_failures: list[BaseException] = [] @@ -297,6 +304,15 @@ def start(self) -> "AcpRuntime": try: self._require_current_binding(self._binding) self._client.initialize(client_capabilities=self._client_capabilities) + if self._session_mode is SessionOpenMode.LOAD: + assert self._requested_session_id is not None + # ACP load may synchronously replay more updates than the + # bounded client queue can hold before returning. Bind and + # drain the ordered stream before issuing the request. + self._session_id = self._requested_session_id + self._ingestor = self._make_ingestor(self._requested_session_id) + self._setup_replay = True + self._start_consumer() session = self._open_session() if not isinstance(session, SessionResult) or not session.session_id: raise AcpRuntimeProtocolError( @@ -314,23 +330,18 @@ def start(self) -> "AcpRuntime": "ACP session setup returned an unexpected session" ) self._require_current_binding(self._binding) - self._session_id = session.session_id - self._ingestor = self._make_ingestor(session.session_id) - threads = ( - threading.Thread( - target=self._consume_updates, - name="tendwire-acp-updates", - daemon=True, - ), - threading.Thread( - target=self._consume_permissions, - name="tendwire-acp-permissions", - daemon=True, - ), - ) - self._threads = threads - for thread in threads: - thread.start() + if self._session_mode is SessionOpenMode.LOAD: + self._wait_for_event_idle( + self._stop_timeout, + allowed_state=RuntimeState.STARTING, + ) + with self._ingest_lock: + self._require_ingestor().reset_after_load() + self._setup_replay = False + else: + self._session_id = session.session_id + self._ingestor = self._make_ingestor(session.session_id) + self._start_consumer() with self._state_lock: if self._failure is not None: raise self._failure @@ -363,14 +374,23 @@ def prompt( with self._state_lock: self._prompts_started += 1 try: - ingestor.start_turn(producer_turn_id=producer_turn_id) + prepared_prompt = _prepare_prompt_content(self._client, prompt) + prompt_event = ingestor.begin_prompt( + prepared_prompt, + producer_turn_id=producer_turn_id, + ) + _raise_for_binding_rejection(prompt_event) except BaseException as exc: with self._state_lock: self._prompts_failed += 1 self._record_failure(exc) raise try: - result = self._client.prompt(session_id, prompt, timeout=timeout) + result = self._client.prompt( + session_id, + prepared_prompt, + timeout=timeout, + ) except BaseException as exc: with self._state_lock: self._prompts_failed += 1 @@ -398,9 +418,9 @@ def prompt( # after the response is a barrier: every earlier queued update has # been durably ingested before the turn is marked complete. try: - self._wait_for_post_response_idle(wait_limit) + self._wait_for_event_idle(wait_limit) with self._ingest_lock: - completion = ingestor.mark_prompt_complete() + completion = ingestor.mark_prompt_complete(result.stop_reason) _raise_for_binding_rejection(completion) except BaseException as exc: with self._state_lock: @@ -416,7 +436,11 @@ def cancel(self) -> None: self.raise_if_failed() session_id, _ = self._running_components() - self._cancel_session(session_id) + try: + self._cancel_session(session_id) + except BaseException as exc: + self._record_failure(exc) + raise def status(self) -> AcpRuntimeStatus: """Return redacted health and counters safe for a public status API.""" @@ -619,52 +643,62 @@ def _require_current_binding(self, expected: WorkerBinding) -> None: if expected not in current: raise AcpRuntimeBindingError("ACP worker binding is not current") - def _consume_updates(self) -> None: - try: - while True: - try: - update = self._client.next_update(timeout=self._poll_timeout) - except TimeoutError: - with self._idle_condition: - self._update_idle_epoch += 1 - self._idle_condition.notify_all() - if self._stop_event.is_set(): - return - continue - if update.session_id != self._session_id: - raise AcpRuntimeProtocolError( - "ACP update belongs to a different session" - ) - ingestor = self._require_ingestor() - with self._ingest_lock: - outcome = ingestor.ingest_update(update.raw) - _raise_for_binding_rejection(outcome) - with self._state_lock: - self._updates_ingested += 1 - except BaseException as exc: - if not self._stop_event.is_set(): - self._record_failure(exc) + def _start_consumer(self) -> None: + if self._threads: + return + thread = threading.Thread( + target=self._consume_session_events, + name="tendwire-acp-session-events", + daemon=True, + ) + self._threads = (thread,) + thread.start() - def _consume_permissions(self) -> None: + def _consume_session_events(self) -> None: try: while True: try: - request = self._client.next_permission_request( - timeout=self._poll_timeout - ) + event = self._client.next_session_event(timeout=self._poll_timeout) except TimeoutError: with self._idle_condition: - self._permission_idle_epoch += 1 + self._event_idle_epoch += 1 self._idle_condition.notify_all() if self._stop_event.is_set(): return continue - self._handle_permission(request) + setup_replay = self._setup_replay + if isinstance(event, SessionUpdate): + if event.session_id != self._session_id: + raise AcpRuntimeProtocolError( + "ACP update belongs to a different session" + ) + ingestor = self._require_ingestor() + with self._ingest_lock: + outcome = ingestor.ingest_update( + event.raw, + replay=setup_replay, + setup_replay=setup_replay, + ) + _raise_for_binding_rejection(outcome) + with self._state_lock: + self._updates_ingested += 1 + elif isinstance(event, PermissionRequest): + self._handle_permission( + event, + setup_replay=setup_replay, + ) + else: # pragma: no cover - typed protocol invariant + raise AcpRuntimeProtocolError("ACP session event type is invalid") except BaseException as exc: if not self._stop_event.is_set(): self._record_failure(exc) - def _handle_permission(self, request: PermissionRequest) -> None: + def _handle_permission( + self, + request: PermissionRequest, + *, + setup_replay: bool = False, + ) -> None: """Journal then resolve one permission, failing closed before response.""" response_attempted = False @@ -678,6 +712,8 @@ def _handle_permission(self, request: PermissionRequest) -> None: outcome = ingestor.ingest_permission_request( request.raw, source_event_id=_permission_source_event_id(request.request_id), + replay=setup_replay, + setup_replay=setup_replay, ) _raise_for_binding_rejection(outcome) with self._state_lock: @@ -731,18 +767,19 @@ def _handle_permission(self, request: PermissionRequest) -> None: self._permissions_cancelled += 1 raise - def _wait_for_post_response_idle(self, timeout: float) -> None: + def _wait_for_event_idle( + self, + timeout: float, + *, + allowed_state: RuntimeState = RuntimeState.RUNNING, + ) -> None: deadline = time.monotonic() + timeout with self._idle_condition: - update_epoch = self._update_idle_epoch - permission_epoch = self._permission_idle_epoch - while ( - self._update_idle_epoch <= update_epoch - or self._permission_idle_epoch <= permission_epoch - ): + event_epoch = self._event_idle_epoch + while self._event_idle_epoch <= event_epoch: if self._failure is not None: raise self._failure - if self._state is not RuntimeState.RUNNING: + if self._state is not allowed_state: raise AcpRuntimeStateError( "ACP runtime stopped before prompt updates drained" ) @@ -791,6 +828,24 @@ def _permission_source_event_id(request_id: RequestId) -> str: return f"permission:{stable_fingerprint({'request_id': request_id})}" +def _prepare_prompt_content( + client: AcpRuntimeClient, + prompt: str | Sequence[Mapping[str, Any]], +) -> tuple[Mapping[str, Any], ...]: + """Validate before persistence when the transport exposes its validator.""" + + prepare = getattr(client, "prepare_prompt", None) + if callable(prepare): + prepared = tuple(prepare(prompt)) + elif isinstance(prompt, str): + prepared = ({"type": "text", "text": prompt},) + else: + prepared = tuple(dict(block) for block in prompt) + if not prepared or any(not isinstance(block, Mapping) for block in prepared): + raise ValueError("prompt must contain at least one content block") + return tuple(dict(block) for block in prepared) + + def _runtime_client_capabilities( capabilities: Mapping[str, Any] | None, ) -> dict[str, Any]: diff --git a/tests/fixtures/acp_fake_agent.py b/tests/fixtures/acp_fake_agent.py index c38ba40..e9a4ac0 100644 --- a/tests/fixtures/acp_fake_agent.py +++ b/tests/fixtures/acp_fake_agent.py @@ -166,6 +166,14 @@ def update(session_id: str, kind: str, **values: object) -> None: {"sessionId": "s-new", "modes": {"currentModeId": "default"}}, ) elif method == "session/load" or method == "session/resume": + if MODE == "load_replay" and method == "session/load": + for index in range(64): + update( + params["sessionId"], + "user_message_chunk" if index % 2 == 0 else "agent_message_chunk", + messageId=f"replay-{index}", + content={"type": "text", "text": str(index)}, + ) response(request_id, {"configOptions": [{"id": "model", "currentValue": "x"}]}) elif method == "session/close" or method == "session/delete": response(request_id, {"_meta": {"vendor.example": {"receipt": method}}}) diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py index 7712834..51c0c49 100644 --- a/tests/test_acp_client.py +++ b/tests/test_acp_client.py @@ -101,6 +101,26 @@ def run_prompt() -> None: assert outcome[0].stop_reason is StopReason.END_TURN +def test_ordered_session_event_api_preserves_cross_kind_reader_order() -> None: + with client() as acp: + acp.initialize() + outcome: list[object] = [] + thread = threading.Thread( + target=lambda: outcome.append(acp.prompt("s1", "inspect")) + ) + thread.start() + first = acp.next_session_event(timeout=1) + second = acp.next_session_event(timeout=1) + assert first.update_kind is SessionUpdateKind.AGENT_THOUGHT_CHUNK + assert second.options[0].option_id == "allow" + acp.respond_permission(second.request_id, option_id="allow") + third = acp.next_session_event(timeout=1) + assert third.update_kind is SessionUpdateKind.PLAN + thread.join(timeout=2) + assert not thread.is_alive() + assert outcome[0].stop_reason is StopReason.END_TURN + + def test_cancel_resolves_pending_permissions_as_cancelled() -> None: with client() as acp: acp.initialize() diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py index cbe6ee8..f4998e1 100644 --- a/tests/test_acp_ingestion.py +++ b/tests/test_acp_ingestion.py @@ -11,6 +11,7 @@ from tendwire.config import Config from tendwire.core.agent_events import AgentEvent, AppendBoundAgentEventResult from tendwire.core.models import WorkerBinding +from tendwire.backends.acp_protocol import StopReason from tendwire.store.sqlite import ( AppendProjectedAgentEventResult, TurnRefreshApplyResult, @@ -434,6 +435,63 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): assert turns[-1]["assistant_stream_text"] == "" +@pytest.mark.parametrize( + ("stop_reason", "outcome", "notice"), + ( + (StopReason.END_TURN, "completed", None), + (StopReason.MAX_TOKENS, "truncated_max_tokens", "token limit"), + ( + StopReason.MAX_TURN_REQUESTS, + "truncated_max_turn_requests", + "request limit", + ), + (StopReason.REFUSAL, "refused", "refused"), + (StopReason.CANCELLED, "cancelled", "cancelled"), + ), +) +def test_outgoing_prompt_is_durable_before_no_echo_completion_stop_reason( + tmp_path: Path, + stop_reason: StopReason, + outcome: str, + notice: str | None, +) -> None: + events: list[AgentEvent] = [] + turns: list[dict[str, object]] = [] + + def append(_path, _host, event, **_kwargs): + events.append(event) + return _appended(len(events), event) + + def apply(_path, _host, _worker, content, **_kwargs): + turns.append(dict(content)) + return TurnRefreshApplyResult(len(turns), False) + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db", agent_event_source="acp_required"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + persist_event=_persist(append, apply), + ) + begun = ingestor.begin_prompt( + ({"type": "text", "text": "question not echoed"},), + producer_turn_id="turn-a", + ) + completed = ingestor.mark_prompt_complete(stop_reason) + + assert begun.event is not None and begun.event.status == "inserted" + assert events[0].kind == "user_message" + assert events[0].payload["outgoing"] is True + assert turns[0]["user_text"] == "question not echoed" + assert completed.event is not None + assert events[-1].payload["stop_reason"] == stop_reason.value + assert events[-1].payload["outcome"] == outcome + final_text = str(turns[-1]["assistant_final_text"]) + assert (notice is not None and notice in final_text) or ( + notice is None and final_text == "" + ) + + def test_duplicate_durable_event_can_idempotently_repair_projection(tmp_path: Path) -> None: projected = False diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index 7bcc560..18a247a 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -2,6 +2,7 @@ import queue import sqlite3 +import sys import threading import time from dataclasses import replace @@ -11,7 +12,7 @@ import pytest -from tendwire.backends.acp_client import AcpRequestTimeoutError +from tendwire.backends.acp_client import AcpClient, AcpRequestTimeoutError from tendwire.backends.acp_protocol import ( PermissionOption, PermissionOptionKind, @@ -40,12 +41,16 @@ _END = object() +FAKE_AGENT = Path(__file__).parent / "fixtures" / "acp_fake_agent.py" class FakeClient: def __init__(self) -> None: - self.updates: queue.Queue[SessionUpdate | object] = queue.Queue() - self.permissions: queue.Queue[PermissionRequest | object] = queue.Queue() + self.events: queue.Queue[SessionUpdate | PermissionRequest | object] = queue.Queue() + # Existing tests enqueue through the typed names; both feed the one + # reader-ordered stream used by the runtime. + self.updates = self.events + self.permissions = self.events self.calls: list[tuple[str, tuple[Any, ...], dict[str, Any]]] = [] self.permission_responses: list[tuple[object, str | None, bool]] = [] self.prompt_result: object = PromptResult(StopReason.END_TURN, {}) @@ -90,6 +95,11 @@ def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: raise self.prompt_failure return self.prompt_result + def prepare_prompt(self, prompt: object) -> tuple[dict[str, Any], ...]: + if isinstance(prompt, str): + return ({"type": "text", "text": prompt},) + return tuple(dict(block) for block in prompt) # type: ignore[arg-type] + def cancel(self, session_id: str) -> None: self.calls.append(("cancel", (session_id,), {})) @@ -113,6 +123,20 @@ def next_permission_request(self, *, timeout: float) -> PermissionRequest: assert isinstance(value, PermissionRequest) return value + def next_session_event( + self, + *, + timeout: float, + ) -> SessionUpdate | PermissionRequest: + try: + value = self.events.get(timeout=timeout) + except queue.Empty as exc: + raise AcpRequestTimeoutError("idle") from exc + if value is _END: + raise EOFError("closed") + assert isinstance(value, SessionUpdate | PermissionRequest) + return value + def respond_permission( self, request_id: object, @@ -125,8 +149,7 @@ def respond_permission( def close(self) -> None: self.close_calls += 1 self.closed = True - self.updates.put(_END) - self.permissions.put(_END) + self.events.put(_END) class FakeIngestor: @@ -136,6 +159,8 @@ def __init__(self, session_id: str = "session-private") -> None: self.updates: list[object] = [] self.permissions: list[tuple[object, str | None]] = [] self.completions = 0 + self.completion_reasons: list[StopReason] = [] + self.load_resets = 0 self.update_failure: BaseException | None = None self.permission_failure: BaseException | None = None self.update_result: object = None @@ -146,22 +171,41 @@ def start_turn(self, *, producer_turn_id: str | None = None) -> str: self.started.append(producer_turn_id) return "opaque-turn" - def ingest_update(self, raw: object) -> object: + def ingest_update(self, raw: object, **_kwargs: Any) -> object: if self.update_failure is not None: raise self.update_failure self.updates.append(raw) return self.update_result def ingest_permission_request( - self, raw: object, *, source_event_id: str | None = None + self, + raw: object, + *, + source_event_id: str | None = None, + **_kwargs: Any, ) -> object: if self.permission_failure is not None: raise self.permission_failure self.permissions.append((raw, source_event_id)) return self.permission_result - def mark_prompt_complete(self) -> object: + def begin_prompt( + self, + prompt: object, + *, + producer_turn_id: str | None = None, + ) -> object: + return self.start_turn(producer_turn_id=producer_turn_id) + + def reset_after_load(self) -> None: + self.load_resets += 1 + + def mark_prompt_complete( + self, + stop_reason: StopReason = StopReason.END_TURN, + ) -> object: self.completions += 1 + self.completion_reasons.append(stop_reason) return self.completion_result @@ -221,7 +265,11 @@ def runtime( upsert_worker_bindings(db_path, [continuity]) return AcpRuntime( client, # type: ignore[arg-type] - config=Config(host_id="host-a", db_path=db_path), + config=Config( + host_id="host-a", + db_path=db_path, + agent_event_source="acp_required", + ), binding=continuity, cwd=tmp_path, stream_generation="generation-private-secret", @@ -394,21 +442,27 @@ def test_load_and_resume_use_requested_session( client = FakeClient() db_path = tmp_path / "events.db" existing = binding("existing-private") + ingestor = FakeIngestor("existing-private") upsert_worker_bindings(db_path, [existing]) service = AcpRuntime( client, # type: ignore[arg-type] - config=Config(host_id="host-a", db_path=db_path), + config=Config( + host_id="host-a", + db_path=db_path, + agent_event_source="acp_required", + ), binding=existing, cwd=tmp_path, session_mode=mode, session_id="existing-private", - ingestor=FakeIngestor("existing-private"), # type: ignore[arg-type] + ingestor=ingestor, # type: ignore[arg-type] poll_timeout=0.01, stop_timeout=0.5, ).start() try: assert client.calls[1][0] == method assert client.calls[1][1][0] == "existing-private" + assert ingestor.load_resets == (1 if mode is SessionOpenMode.LOAD else 0) finally: service.stop() @@ -430,7 +484,11 @@ def test_load_and_resume_reject_agent_session_mismatch_and_close( upsert_worker_bindings(db_path, [existing]) service = AcpRuntime( client, # type: ignore[arg-type] - config=Config(host_id="host-a", db_path=db_path), + config=Config( + host_id="host-a", + db_path=db_path, + agent_event_source="acp_required", + ), binding=existing, cwd=tmp_path, session_mode=mode, @@ -953,20 +1011,25 @@ def test_cross_kind_ingestion_cannot_overtake_an_active_update( order: list[str] = [] class OrderedIngestor(FakeIngestor): - def ingest_update(self, raw: object) -> None: + def ingest_update(self, raw: object, **kwargs: Any) -> None: order.append("update-start") update_entered.set() assert release_update.wait(timeout=1) - super().ingest_update(raw) + super().ingest_update(raw, **kwargs) order.append("update-end") def ingest_permission_request( - self, raw: object, *, source_event_id: str | None = None + self, + raw: object, + *, + source_event_id: str | None = None, + **kwargs: Any, ) -> None: order.append("permission") super().ingest_permission_request( raw, source_event_id=source_event_id, + **kwargs, ) service = runtime(tmp_path, client, OrderedIngestor()).start() @@ -985,6 +1048,57 @@ def ingest_permission_request( service.stop() +def test_load_drains_replay_larger_than_client_queue_before_response( + tmp_path: Path, +) -> None: + db_path = tmp_path / "load.db" + current = binding("s-load") + upsert_worker_bindings(db_path, [current]) + client = AcpClient( + [sys.executable, "-u", str(FAKE_AGENT), "load_replay"], + max_pending_events=4, + ) + service = AcpRuntime( + client, + config=Config( + host_id="host-a", + db_path=db_path, + agent_event_source="acp_required", + ), + binding=current, + cwd=tmp_path, + session_mode=SessionOpenMode.LOAD, + session_id="s-load", + poll_timeout=0.005, + stop_timeout=2, + ).start() + try: + assert service.status().updates_ingested == 64 + assert len(list_agent_events(db_path, "host-a")) == 64 + with sqlite3.connect(db_path) as conn: + assert conn.execute("SELECT COUNT(*) FROM connector_outbox").fetchone()[0] == 0 + finally: + service.stop(timeout=2) + + +@pytest.mark.parametrize("stop_reason", tuple(StopReason)) +def test_runtime_carries_every_prompt_stop_reason_to_completion( + tmp_path: Path, + stop_reason: StopReason, +) -> None: + client = FakeClient() + client.prompt_result = PromptResult(stop_reason, {"stopReason": stop_reason.value}) + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + result = service.prompt("no echo", producer_turn_id="turn-a") + assert result.stop_reason is stop_reason + assert ingestor.started == ["turn-a"] + assert ingestor.completion_reasons == [stop_reason] + finally: + service.stop() + + def test_prompt_transport_failure_cancels_and_makes_runtime_terminal( tmp_path: Path, ) -> None: From d2b59b6d121baff3aff765e2e5137d04dbc29006 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:56:27 +0800 Subject: [PATCH 27/83] fix(acp): lease new session bindings --- src/tendwire/backends/acp_runtime.py | 137 ++++++++++++++++----- tests/test_acp_runtime.py | 171 +++++++++++++++++++++++++++ 2 files changed, 278 insertions(+), 30 deletions(-) diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 550f4bc..fc9b920 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -19,7 +19,7 @@ from ..config import Config from ..core.models import WorkerBinding, stable_fingerprint -from ..store.sqlite import list_worker_bindings +from ..store.sqlite import expire_worker_bindings, list_worker_bindings from .acp_ingestion import AcpSessionIngestor from .acp_protocol import ( PermissionRequest, @@ -258,7 +258,10 @@ def __init__( self._ingestor: AcpSessionIngestor | None = None self._failure: BaseException | None = None self._state_lock = threading.RLock() - self._lifecycle_lock = threading.Lock() + # Session binders are embedding callbacks and may synchronously call + # stop(). Reentrancy must terminate startup, not deadlock on our own + # lifecycle lock. + self._lifecycle_lock = threading.RLock() self._ingest_lock = threading.Lock() self._prompt_lock = threading.Lock() self._idle_condition = threading.Condition(self._state_lock) @@ -266,6 +269,7 @@ def __init__( self._threads: tuple[threading.Thread, ...] = () self._event_idle_epoch = 0 self._setup_replay = False + self._provisional_binding: WorkerBinding | None = None self._close_thread: threading.Thread | None = None self._close_failures: list[BaseException] = [] @@ -347,6 +351,7 @@ def start(self) -> "AcpRuntime": raise self._failure self._state = RuntimeState.RUNNING except BaseException as exc: + self._release_derived_binding(reason="acp_startup_rollback") self._record_failure(exc) # ``__enter__`` is never completed when start fails, so no # caller cleanup can be assumed. Bound shutdown prevents an @@ -374,6 +379,7 @@ def prompt( with self._state_lock: self._prompts_started += 1 try: + self._require_current_binding(self._binding) prepared_prompt = _prepare_prompt_content(self._client, prompt) prompt_event = ingestor.begin_prompt( prepared_prompt, @@ -522,6 +528,8 @@ def stop(self, *, timeout: float | None = None) -> None: self._record_failure(error) raise error + self._release_derived_binding(reason="acp_runtime_stopped") + with self._state_lock: failure = self._failure if self._close_failures and failure is None: @@ -601,36 +609,104 @@ def _bind_new_session(self, session_id: str) -> WorkerBinding: raise AcpRuntimeBindingError("ACP session binding is unavailable") continuity = self._binding self._require_current_binding(continuity) - bound = callback(session_id, continuity) - if not isinstance(bound, WorkerBinding): - raise AcpRuntimeBindingError("ACP session binder returned an invalid binding") - if ( - bound.host_id != continuity.host_id - or bound.worker_id != continuity.worker_id - or bound.worker_fingerprint != continuity.worker_fingerprint - or bound.backend != continuity.backend - or bound.target_kind != continuity.target_kind - or bound.target_value != continuity.target_value - ): - raise AcpRuntimeBindingError("ACP session binder changed worker continuity") - if ( - bound.turn_target_kind != "acp_session_id" - or bound.turn_target_value != session_id - ): - raise AcpRuntimeBindingError("ACP session binder returned the wrong session") - if ( - not bound.private_fingerprint - or bound.private_fingerprint == continuity.private_fingerprint - ): - raise AcpRuntimeBindingError( - "ACP session binder did not establish a distinct private binding" - ) - # The callback must add a distinct ACP binding. It must not repurpose or - # overwrite the Herdr continuity row it was given. - self._require_current_binding(continuity) - self._require_current_binding(bound) + existing_acp = self._binding_fingerprints(continuity.host_id, backend="acp") + try: + bound = callback(session_id, continuity) + with self._state_lock: + if self._state is not RuntimeState.STARTING: + raise AcpRuntimeStateError( + "ACP runtime stopped during session binding" + ) + if not isinstance(bound, WorkerBinding): + raise AcpRuntimeBindingError( + "ACP session binder returned an invalid binding" + ) + if ( + bound.host_id != continuity.host_id + or bound.worker_id != continuity.worker_id + or bound.worker_fingerprint != continuity.worker_fingerprint + or bound.backend != "acp" + or bound.target_kind != continuity.target_kind + or bound.target_value != continuity.target_value + ): + raise AcpRuntimeBindingError( + "ACP session binder changed worker continuity" + ) + if ( + bound.turn_target_kind != "acp_session_id" + or bound.turn_target_value != session_id + ): + raise AcpRuntimeBindingError( + "ACP session binder returned the wrong session" + ) + if ( + not bound.private_fingerprint + or bound.private_fingerprint == continuity.private_fingerprint + ): + raise AcpRuntimeBindingError( + "ACP session binder did not establish a distinct private binding" + ) + # The callback must add a distinct ACP binding. It must not + # repurpose or overwrite the Herdr continuity row it was given. + self._require_current_binding(continuity) + self._require_current_binding(bound) + except BaseException: + self._expire_new_acp_bindings(continuity.host_id, existing_acp) + raise + if bound.private_fingerprint not in existing_acp: + self._provisional_binding = bound return bound + def _binding_fingerprints(self, host_id: str, *, backend: str) -> set[str]: + db_path = self._config.db_path + if db_path is None: # pragma: no cover - constructor invariant + return set() + return { + item.private_fingerprint + for item in list_worker_bindings(Path(db_path), host_id, backend=backend) + } + + def _expire_new_acp_bindings( + self, + host_id: str, + existing_fingerprints: set[str], + ) -> None: + db_path = self._config.db_path + if db_path is None: # pragma: no cover - constructor invariant + return + current = list_worker_bindings(Path(db_path), host_id, backend="acp") + created = [ + item + for item in current + if item.private_fingerprint not in existing_fingerprints + ] + for item in created: + expire_worker_bindings( + Path(db_path), + host_id, + backend="acp", + private_fingerprints=[item.private_fingerprint], + # Callback clocks are untrusted. Using the row's own + # observation instant guarantees a future-dated provisional + # lease is still revocable. + now=item.observed_at, + reason="acp_startup_rollback", + ) + + def _release_derived_binding(self, *, reason: str) -> None: + bound = self._provisional_binding + self._provisional_binding = None + if bound is None or self._config.db_path is None: + return + expire_worker_bindings( + Path(self._config.db_path), + bound.host_id, + backend="acp", + private_fingerprints=[bound.private_fingerprint], + now=bound.observed_at, + reason=reason, + ) + def _require_current_binding(self, expected: WorkerBinding) -> None: db_path = self._config.db_path if db_path is None: # pragma: no cover - constructor invariant @@ -814,6 +890,7 @@ def _cancel_session(self, session_id: str) -> None: self._cancellation_requests += 1 def _record_failure(self, failure: BaseException) -> None: + self._release_derived_binding(reason="acp_runtime_failed") with self._idle_condition: if self._failure is None: self._failure = failure diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index 18a247a..4eb1634 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -27,6 +27,7 @@ AcpRuntime, AcpRuntimeBindingError, AcpRuntimeProtocolError, + AcpRuntimeStateError, AcpRuntimeStopTimeout, RuntimeState, SessionOpenMode, @@ -34,6 +35,8 @@ from tendwire.config import Config from tendwire.core.models import WorkerBinding from tendwire.store.sqlite import ( + expire_stale_worker_bindings, + expire_worker_bindings, list_agent_events, list_worker_bindings, upsert_worker_bindings, @@ -244,6 +247,7 @@ def establish( ) -> WorkerBinding: bound = replace( continuity, + backend="acp", turn_target_kind="acp_session_id", turn_target_value=session_id, private_fingerprint="", @@ -535,6 +539,171 @@ def establish(session_id: str, anchor: WorkerBinding) -> WorkerBinding: service.stop() +def test_new_acp_binding_survives_herdr_refresh_and_normal_stop( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=binding_callback(db_path), + ingestor=FakeIngestor(), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + + derived = list_worker_bindings(db_path, "host-a", backend="acp") + assert len(derived) == 1 + assert derived[0].turn_target_value == "session-private" + expire_stale_worker_bindings( + db_path, + "host-a", + backend="herdr", + current_private_fingerprints=[continuity.private_fingerprint], + ) + assert list_worker_bindings(db_path, "host-a", backend="acp") == derived + service.stop() + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + released = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(released) == 1 + assert released[0].reason == "acp_runtime_stopped" + + +@pytest.mark.parametrize("failure_mode", ("raise", "bad_return")) +def test_new_cleans_binding_persisted_by_failed_callback( + tmp_path: Path, + failure_mode: str, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def fail_after_persist(session_id: str, anchor: WorkerBinding): + bound = binding_callback(db_path)(session_id, anchor) + if failure_mode == "raise": + raise RuntimeError("after persist") + return object() + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=fail_after_persist, # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ) + with pytest.raises((RuntimeError, AcpRuntimeBindingError)): + service.start() + + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + retired = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(retired) == 1 + assert retired[0].reason == "acp_startup_rollback" + assert list_worker_bindings(db_path, "host-a", backend="herdr") == [continuity] + + +def test_new_rolls_back_persisted_binding_when_ingestor_startup_fails( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def fail_factory(*_args: object, **_kwargs: object) -> FakeIngestor: + raise RuntimeError("factory failed") + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=binding_callback(db_path), + ingestor_factory=fail_factory, # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ) + with pytest.raises(RuntimeError, match="factory failed"): + service.start() + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + + +def test_new_callback_can_stop_runtime_without_lifecycle_deadlock( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + service: AcpRuntime + + def stop_during_bind(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + bound = binding_callback(db_path)(session_id, anchor) + service.stop(timeout=0.2) + return bound + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=stop_during_bind, + poll_timeout=0.01, + stop_timeout=0.5, + ) + started = time.monotonic() + with pytest.raises(AcpRuntimeStateError, match="stopped during"): + service.start() + assert time.monotonic() - started < 0.5 + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + + +def test_prompt_rechecks_binding_before_remote_send(tmp_path: Path) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=binding_callback(db_path), + ingestor=FakeIngestor(), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + derived = list_worker_bindings(db_path, "host-a", backend="acp")[0] + expire_worker_bindings( + db_path, + derived.host_id, + backend="acp", + private_fingerprints=[derived.private_fingerprint], + reason="test_expiry", + ) + + with pytest.raises(AcpRuntimeBindingError): + service.prompt("must not send") + assert [call[0] for call in client.calls].count("prompt") == 0 + + def test_new_requires_explicit_session_binder_before_launch(tmp_path: Path) -> None: client = FakeClient() @@ -644,6 +813,7 @@ def test_new_rejects_valid_shaped_binding_that_was_not_persisted( def dishonest(session_id: str, anchor: WorkerBinding) -> WorkerBinding: return replace( anchor, + backend="acp", turn_target_kind="acp_session_id", turn_target_value=session_id, private_fingerprint="", @@ -689,6 +859,7 @@ def destructive(session_id: str, anchor: WorkerBinding) -> WorkerBinding: ) bound = replace( anchor, + backend="acp", turn_target_kind="acp_session_id", turn_target_value=session_id, private_fingerprint="", From 5dd2413993c1396c0116eceabb15ff470f1840d1 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 21:57:00 +0800 Subject: [PATCH 28/83] fix(acp): make private event policy fail closed --- .env.example | 5 +- README.md | 2 +- docs/acp-migration.md | 17 ++-- src/tendwire/backends/acp_ingestion.py | 41 ++++++--- src/tendwire/config.py | 2 +- src/tendwire/core/agent_events.py | 12 ++- src/tendwire/store/sqlite.py | 45 +++++++++- tests/test_acp_ingestion.py | 33 +++++-- tests/test_agent_events.py | 119 ++++++++++++++++++++++++- tests/test_config.py | 4 +- tests/test_store.py | 2 +- 11 files changed, 240 insertions(+), 42 deletions(-) diff --git a/.env.example b/.env.example index 9b55f61..0f7766e 100644 --- a/.env.example +++ b/.env.example @@ -99,7 +99,10 @@ TENDWIRE_TURN_MODEL=observed # projecting it. acp_required refuses legacy turn ingestion and fails closed # unless that explicit runtime starts healthy. TENDWIRE_AGENT_EVENT_SOURCE=legacy -TENDWIRE_ACP_THOUGHT_POLICY=private_summary +# Stable ACP v1 does not distinguish raw reasoning from summaries. The safe +# default discards thought chunks. `private_summary` is an explicit trusted- +# adapter convention; `private_all` retains all thoughts for local diagnostics. +TENDWIRE_ACP_THOUGHT_POLICY=disabled TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS=30 TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS=5 TENDWIRE_ACP_MAX_FRAME_BYTES=8388608 diff --git a/README.md b/README.md index 9714bb1..6d27500 100644 --- a/README.md +++ b/README.md @@ -554,7 +554,7 @@ variables: | `turn_refresh_workers` | `TENDWIRE_TURN_REFRESH_WORKERS` | `4` | integer from 1 through 32 and no greater than `max_workers` | | `turn_model` | `TENDWIRE_TURN_MODEL` | `observed` | `observed`; `legacy`, `dual`, and `shadow` are deprecated aliases with identical observed behavior | | `agent_event_source` | `TENDWIRE_AGENT_EVENT_SOURCE` | `legacy` | `legacy`, `acp_shadow`, `acp_preferred`, or `acp_required`; ACP modes are experimental | -| `acp_thought_policy` | `TENDWIRE_ACP_THOUGHT_POLICY` | `private_summary` | `disabled`, `private_summary`, or `private_all`; never a public-delivery grant | +| `acp_thought_policy` | `TENDWIRE_ACP_THOUGHT_POLICY` | `disabled` | `disabled`, `private_summary`, or `private_all`; never a public-delivery grant | | `acp_request_timeout_seconds` | `TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS` | `30.0` | finite positive float | | `acp_shutdown_timeout_seconds` | `TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS` | `5.0` | finite positive float | | `acp_max_frame_bytes` | `TENDWIRE_ACP_MAX_FRAME_BYTES` | `8388608` | integer from 1 through 67108864 | diff --git a/docs/acp-migration.md b/docs/acp-migration.md index d4a1975..4454fd5 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -70,15 +70,20 @@ sanitizing projection and must not reuse raw ACP payloads. `TENDWIRE_ACP_THOUGHT_POLICY` has three values: - `disabled`: discard thought chunks before persistence. -- `private_summary`: retain readable reasoning summaries privately and discard - raw-reasoning chunks. +- `private_summary`: retain a chunk privately only when a trusted adapter sets + the exact update-level marker + `_meta["tendwire.dev/thought_kind"] = "summary"`; unclassified, unknown, + contradictory, and raw chunks are discarded. - `private_all`: retain every thought chunk privately for explicit local diagnostics. -The default is `private_summary`. No thought policy grants connector delivery. -Herdres must never receive a raw thought event. A future public summary feature -requires a separate schema, sanitizer, explicit operator opt-in, and tests that -prove raw reasoning cannot cross the boundary. +The default is `disabled`. Stable ACP v1 does not define a raw-versus-summary +classification. The `private_summary` marker is only a Tendwire adapter +convention and is not an ACP guarantee; enable it only for a trusted adapter. +No thought policy grants connector delivery. Herdres must never receive a raw +thought event. A future public summary feature requires a separate schema, +sanitizer, explicit operator opt-in, and tests that prove raw reasoning cannot +cross the boundary. ## Upstream upgrade boundary diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py index 7cad6c4..a125b77 100644 --- a/src/tendwire/backends/acp_ingestion.py +++ b/src/tendwire/backends/acp_ingestion.py @@ -528,8 +528,9 @@ def _source_item_id(kind: str, payload: Mapping[str, Any]) -> str | None: "plan", } ) -_THOUGHT_RAW_LABELS = frozenset( - {"raw", "reasoning", "raw_reasoning", "raw-reasoning", "chain_of_thought"} +_TRUSTED_THOUGHT_SUMMARY_KEY = "tendwire.dev/thought_kind" +_LEGACY_THOUGHT_CLASSIFICATION_KEYS = frozenset( + {"thought_kind", "thoughtKind", "reasoning_kind", "reasoningKind"} ) @@ -568,23 +569,35 @@ def _thought_classification(value: Mapping[str, Any]) -> str | None: update = _session_update(value) if update is None or update.get("sessionUpdate") != "agent_thought_chunk": return None - candidates: list[Mapping[str, Any]] = [] + update_meta = update.get("_meta") + trusted = ( + update_meta.get(_TRUSTED_THOUGHT_SUMMARY_KEY) + if isinstance(update_meta, Mapping) + else None + ) + if trusted != "summary": + return "unclassified" if trusted is None else "unknown" + + # The exact update-level marker is a Tendwire adapter convention, not an + # ACP classification guarantee. Contradictory legacy or content metadata + # therefore fails closed even when the trusted marker says "summary". for container in (update, update.get("content")): if not isinstance(container, Mapping): continue meta = container.get("_meta") if not isinstance(meta, Mapping): continue - candidates.append(meta) - tendwire = meta.get("tendwire") - if isinstance(tendwire, Mapping): - candidates.insert(0, tendwire) - for meta in candidates: - for key in ("thought_kind", "thoughtKind", "reasoning_kind", "reasoningKind"): - label = meta.get(key) - if isinstance(label, str) and label.strip(): - return label.strip().lower() - return "unclassified" + marker = meta.get(_TRUSTED_THOUGHT_SUMMARY_KEY) + if marker is not None and marker != "summary": + return "conflicting" + if any(key in meta for key in _LEGACY_THOUGHT_CLASSIFICATION_KEYS): + return "conflicting" + legacy_namespace = meta.get("tendwire") + if isinstance(legacy_namespace, Mapping) and any( + key in legacy_namespace for key in _LEGACY_THOUGHT_CLASSIFICATION_KEYS + ): + return "conflicting" + return "summary" def _thought_rejection_reason( @@ -597,7 +610,7 @@ def _thought_rejection_reason( return None if policy == "disabled": return "thought_policy_disabled" - if policy == "private_summary" and classification in _THOUGHT_RAW_LABELS: + if policy == "private_summary" and classification != "summary": return "thought_policy_requires_summary" return None diff --git a/src/tendwire/config.py b/src/tendwire/config.py index 1c62b37..8e1f8a5 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -22,7 +22,7 @@ ACP_THOUGHT_POLICIES = frozenset({"disabled", "private_summary", "private_all"}) DEFAULT_TURN_MODEL = "observed" DEFAULT_AGENT_EVENT_SOURCE = "legacy" -DEFAULT_ACP_THOUGHT_POLICY = "private_summary" +DEFAULT_ACP_THOUGHT_POLICY = "disabled" DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS = 30.0 DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS = 5.0 DEFAULT_ACP_MAX_FRAME_BYTES = 8 * 1024 * 1024 diff --git a/src/tendwire/core/agent_events.py b/src/tendwire/core/agent_events.py index cc27d4f..f20a789 100644 --- a/src/tendwire/core/agent_events.py +++ b/src/tendwire/core/agent_events.py @@ -46,6 +46,9 @@ } ) AGENT_EVENT_VISIBILITIES = frozenset({"private", "public"}) +AGENT_EVENT_PRIVATE_KINDS = frozenset( + {"thought", "tool_call", "tool_call_update", "plan", "extension"} +) AGENT_EVENT_MAX_PAYLOAD_BYTES = 16 * 1024 * 1024 AGENT_EVENT_MAX_PUBLIC_PAYLOAD_BYTES = 64 * 1024 AGENT_EVENT_MAX_TEXT_CHARS = 4 * 1024 * 1024 @@ -290,10 +293,11 @@ def agent_event( normalized_visibility = str(visibility).strip().lower() if normalized_visibility not in AGENT_EVENT_VISIBILITIES: raise ValueError("visibility must be private or public") - if normalized_kind == "thought" and normalized_visibility != "private": - raise ValueError("thought events must remain private") - if normalized_kind == "extension" and normalized_visibility != "private": - raise ValueError("extension events must remain private") + if ( + normalized_kind in AGENT_EVENT_PRIVATE_KINDS + and normalized_visibility != "private" + ): + raise ValueError(f"{normalized_kind} events must remain private") normalized_source = normalize_agent_event_identifier( source, "source", required=True ) diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index 45b96e9..8fede08 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -145,7 +145,7 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS -STORE_SCHEMA_VERSION = 24 +STORE_SCHEMA_VERSION = 25 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS _LEGACY_TURN_CLAIM_HARD_TTL_SECONDS = 86_400 TURN_CHANGE_RETENTION_DAYS = 7 @@ -1622,7 +1622,11 @@ def _record_response_size( source_event_id IS NOT NULL OR (source_session_id IS NOT NULL AND source_sequence IS NOT NULL) ), - CHECK (kind NOT IN ('thought', 'extension') OR visibility = 'private') + CHECK ( + kind NOT IN ( + 'thought', 'tool_call', 'tool_call_update', 'plan', 'extension' + ) OR visibility = 'private' + ) ); """ @@ -13437,6 +13441,42 @@ def _migrate_v23_to_v24_conn(conn: sqlite3.Connection) -> None: conn.execute(statement) +def _migrate_v24_to_v25_conn(conn: sqlite3.Connection) -> None: + """Make thought, tool, plan, and extension journal rows private-only.""" + conn.execute("ALTER TABLE agent_events RENAME TO agent_events_v24") + conn.execute(CREATE_AGENT_EVENTS_TABLE) + conn.execute( + """ + INSERT INTO agent_events ( + sequence, host_id, event_id, kind, source, worker_id, visibility, + source_session_id, source_turn_id, source_item_id, + source_message_id, source_event_id, source_sequence, observed_at, + payload_fingerprint, private_payload_json, public_payload_json + ) + SELECT + sequence, host_id, event_id, kind, source, worker_id, + CASE + WHEN kind IN ('thought', 'tool_call', 'tool_call_update', 'plan', 'extension') + THEN 'private' + ELSE visibility + END, + source_session_id, source_turn_id, source_item_id, + source_message_id, source_event_id, source_sequence, observed_at, + payload_fingerprint, private_payload_json, + CASE + WHEN kind IN ('thought', 'tool_call', 'tool_call_update', 'plan', 'extension') + THEN '{}' + ELSE public_payload_json + END + FROM agent_events_v24 + ORDER BY sequence + """ + ) + conn.execute("DROP TABLE agent_events_v24") + for statement in CREATE_AGENT_EVENT_INDEXES: + conn.execute(statement) + + MIGRATIONS: tuple[Migration, ...] = ( Migration(0, 1, _migrate_v0_to_v1_conn), Migration(1, 2, _migrate_v1_to_v2_conn), @@ -13462,6 +13502,7 @@ def _migrate_v23_to_v24_conn(conn: sqlite3.Connection) -> None: Migration(21, 22, _migrate_v21_to_v22_conn), Migration(22, 23, _migrate_v22_to_v23_conn), Migration(23, 24, _migrate_v23_to_v24_conn), + Migration(24, 25, _migrate_v24_to_v25_conn), ) diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py index f4998e1..abbe64b 100644 --- a/tests/test_acp_ingestion.py +++ b/tests/test_acp_ingestion.py @@ -131,7 +131,7 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): "agent_thought_chunk", messageId="reasoning-1", content={"type": "text", "text": "private reasoning"}, - _meta={"tendwire": {"thought_kind": "summary"}}, + _meta={"tendwire.dev/thought_kind": "summary"}, ) ) ingestor.ingest_update( @@ -146,7 +146,6 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): assert turn_id.startswith("acpt_") assert [event.kind for event in events] == [ "user_message", - "thought", "agent_message", "extension", ] @@ -667,7 +666,7 @@ def test_producer_turn_identity_survives_transport_recreation(tmp_path: Path) -> assert identities[0] == identities[1] -def test_private_summary_policy_retains_display_chunks_but_rejects_marked_raw_thoughts( +def test_private_summary_requires_exact_trusted_marker_and_rejects_conflicts( tmp_path: Path, ) -> None: events: list[AgentEvent] = [] @@ -682,7 +681,7 @@ def append( return _appended(len(events), event) ingestor = AcpSessionIngestor( - _config(tmp_path / "events.db"), + _config(tmp_path / "events.db", acp_thought_policy="private_summary"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -701,19 +700,39 @@ def append( _meta={"tendwire": {"thought_kind": "raw"}}, ) ) + unknown = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "unknown secret"}, + _meta={"tendwire.dev/thought_kind": "SUMMARY"}, + ) + ) + conflicting = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={ + "type": "text", + "text": "conflicting secret", + "_meta": {"reasoning_kind": "raw"}, + }, + _meta={"tendwire.dev/thought_kind": "summary"}, + ) + ) summary = ingestor.ingest_update( _update( "agent_thought_chunk", messageId="summary-1", content={"type": "text", "text": "readable summary"}, - _meta={"tendwire": {"thought_kind": "summary"}}, + _meta={"tendwire.dev/thought_kind": "summary"}, ) ) - assert unclassified.event is not None + assert unclassified.ignored_reason == "thought_policy_requires_summary" assert raw.ignored_reason == "thought_policy_requires_summary" + assert unknown.ignored_reason == "thought_policy_requires_summary" + assert conflicting.ignored_reason == "thought_policy_requires_summary" assert summary.event is not None - assert len(events) == 2 + assert len(events) == 1 assert all(event.visibility == "private" for event in events) assert all(event.public_payload == {} for event in events) assert "raw secret" not in repr(events) diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py index de37f0a..8a36b1b 100644 --- a/tests/test_agent_events.py +++ b/tests/test_agent_events.py @@ -199,6 +199,40 @@ def test_thought_events_are_private_and_not_publicly_listed(tmp_path: Path) -> N } +@pytest.mark.parametrize("kind", ["tool_call", "tool_call_update", "plan"]) +def test_tool_and_plan_events_are_private_only(kind: str, tmp_path: Path) -> None: + with pytest.raises(ValueError, match=rf"{kind} events must remain private"): + agent_event( + kind=kind, + source="acp", + worker_id="worker-1", + source_session_id="session-1", + source_sequence=1, + visibility="public", + payload={"content": [{"type": "text", "text": "private tool data"}]}, + ) + + event = agent_event( + kind=kind, + source="acp", + worker_id="worker-1", + source_session_id="session-1", + source_sequence=1, + payload={"content": [{"type": "text", "text": "private tool data"}]}, + ) + db_path = tmp_path / f"{kind}.db" + store_sqlite.append_agent_event(db_path, "host-1", event) + assert store_sqlite.list_public_agent_events(db_path, "host-1") == () + assert "private tool data" in repr( + store_sqlite.list_agent_events(db_path, "host-1") + ) + with sqlite3.connect(db_path) as conn, pytest.raises(sqlite3.IntegrityError): + conn.execute( + "UPDATE agent_events SET visibility = 'public' WHERE event_id = ?", + (event.event_id,), + ) + + def test_queries_filter_worker_session_turn_and_cursor(tmp_path: Path) -> None: db_path = tmp_path / "store.db" first = _message_event(sequence=1) @@ -863,7 +897,7 @@ def test_v23_to_v24_preserves_populated_journal_sequence(tmp_path: Path) -> None ) conn.commit() store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == (24,) + assert conn.execute("PRAGMA user_version").fetchone() == (25,) assert conn.execute( "SELECT sequence FROM agent_events WHERE event_id = ?", (event.event_id,), @@ -876,6 +910,85 @@ def test_v23_to_v24_preserves_populated_journal_sequence(tmp_path: Path) -> None assert store_sqlite.append_agent_event(db_path, "host-1", later).sequence == 124 +def test_v24_to_v25_privatises_populated_tool_and_plan_rows(tmp_path: Path) -> None: + db_path = tmp_path / "v24-public-tools.db" + kinds = ("tool_call", "tool_call_update", "plan") + events = [ + agent_event( + kind=kind, + source="acp", + worker_id="worker-1", + source_session_id="private-session", + source_sequence=index, + payload={"content": [{"type": "text", "text": f"private-{kind}"}]}, + observed_at="2026-07-31T00:00:00+00:00", + ) + for index, kind in enumerate(kinds, 1) + ] + with sqlite3.connect(db_path) as conn: + store_sqlite._run_migrations(conn, target_version=24) + conn.execute("DROP TABLE agent_events") + conn.execute( + store_sqlite.CREATE_AGENT_EVENTS_TABLE.replace( + """kind NOT IN ( + 'thought', 'tool_call', 'tool_call_update', 'plan', 'extension' + ) OR visibility = 'private'""", + "kind NOT IN ('thought', 'extension') OR visibility = 'private'", + ) + ) + for sequence, event in enumerate(events, 41): + conn.execute( + """ + INSERT INTO agent_events ( + sequence, host_id, event_id, kind, source, worker_id, + visibility, source_session_id, source_turn_id, + source_item_id, source_message_id, source_event_id, + source_sequence, observed_at, payload_fingerprint, + private_payload_json, public_payload_json + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + sequence, + "host-1", + event.event_id, + event.kind, + event.source, + event.worker_id, + "public", + event.source_session_id, + event.source_turn_id, + event.source_item_id, + event.source_message_id, + event.source_event_id, + event.source_sequence, + event.observed_at, + event.payload_fingerprint, + store_sqlite._canonical_json(event.payload), + store_sqlite._canonical_json(event.payload), + ), + ) + conn.commit() + store_sqlite._run_migrations(conn) + assert conn.execute("PRAGMA user_version").fetchone() == (25,) + assert conn.execute( + "SELECT sequence, kind, visibility, public_payload_json " + "FROM agent_events ORDER BY sequence" + ).fetchall() == [ + (41, "tool_call", "private", "{}"), + (42, "tool_call_update", "private", "{}"), + (43, "plan", "private", "{}"), + ] + assert conn.execute("PRAGMA integrity_check").fetchone() == ("ok",) + + tmp_path.chmod(0o700) + db_path.chmod(0o600) + assert store_sqlite.list_public_agent_events(db_path, "host-1") == () + assert [ + stored.event.kind + for stored in store_sqlite.list_agent_events(db_path, "host-1") + ] == list(kinds) + + @pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) def test_agent_event_schema_migrates_from_every_prior_version( tmp_path: Path, @@ -913,7 +1026,7 @@ def test_v21_migration_is_idempotent_and_preserves_existing_store( store_sqlite.init_store(db_path) store_sqlite.init_store(db_path) with sqlite3.connect(db_path) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == (24,) + assert conn.execute("PRAGMA user_version").fetchone() == (25,) columns = { str(row[1]) for row in conn.execute("PRAGMA table_info(agent_events)") } @@ -974,7 +1087,7 @@ def test_v22_migration_rekeys_legacy_event_identity_without_losing_sequence( "SELECT sequence, event_id FROM agent_events" ).fetchone() assert row == (19, event.event_id) - assert conn.execute("PRAGMA user_version").fetchone() == (24,) + assert conn.execute("PRAGMA user_version").fetchone() == (25,) replay = store_sqlite.append_agent_event(db_path, "host-1", event) assert replay.inserted is False diff --git a/tests/test_config.py b/tests/test_config.py index da87472..cc30c7a 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -32,7 +32,7 @@ ) -def test_acp_event_source_defaults_to_preferred_with_private_summaries( +def test_acp_event_source_defaults_to_legacy_with_thoughts_disabled( monkeypatch, ) -> None: for name in ( @@ -47,7 +47,7 @@ def test_acp_event_source_defaults_to_preferred_with_private_summaries( config = load_config() assert config.agent_event_source == DEFAULT_AGENT_EVENT_SOURCE == "legacy" - assert config.acp_thought_policy == DEFAULT_ACP_THOUGHT_POLICY == "private_summary" + assert config.acp_thought_policy == DEFAULT_ACP_THOUGHT_POLICY == "disabled" assert config.acp_request_timeout_seconds == DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS == 30.0 assert config.acp_shutdown_timeout_seconds == DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS == 5.0 assert config.acp_max_frame_bytes == DEFAULT_ACP_MAX_FRAME_BYTES == 8 * 1024 * 1024 diff --git a/tests/test_store.py b/tests/test_store.py index 167d4a5..5d55f86 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -10278,7 +10278,7 @@ def test_v13_migration_repairs_nonpositive_turn_sequences_and_blocks_recurrence( init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 24 + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 25 assert conn.execute( """ SELECT turn_id, list_sequence From ef150e55ebaee5802aea48572b74e9fc03dac460 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 22:10:01 +0800 Subject: [PATCH 29/83] fix(acp): multiplex typed session events --- src/tendwire/backends/acp_client.py | 99 +++++++++----- tests/test_acp_client.py | 202 +++++++++++++++++++++++++++- 2 files changed, 263 insertions(+), 38 deletions(-) diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index c683f64..f3fe8e5 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -195,11 +195,8 @@ def __init__( # Updates and permission requests share one reader-ordered queue. A # pair of duplicate queues can both reorder cross-kind events and fail # the transport when an embedding consumes only one of them. - self._session_events: queue.Queue[SessionEvent | object] = queue.Queue( - max_pending_events - ) - self._session_event_backlog: deque[SessionEvent] = deque() - self._session_event_lock = threading.Lock() + self._session_events: deque[SessionEvent] = deque() + self._session_event_condition = threading.Condition() self._notifications: queue.Queue[RawNotification | object] = queue.Queue( max_pending_events ) @@ -721,37 +718,50 @@ def _next_typed_session_event( deadline = None if timeout is not None: deadline = time.monotonic() + _positive_timeout(timeout, "timeout") - with self._session_event_lock: + with self._session_event_condition: while True: - for index, candidate in enumerate(self._session_event_backlog): - if self._session_event_matches(candidate, expected): - del self._session_event_backlog[index] - return candidate # type: ignore[return-value] + index = self._matching_session_event_index(expected) + if index is not None: + candidate = self._session_events[index] + del self._session_events[index] + self._session_event_condition.notify_all() + return candidate # type: ignore[return-value] + if self.state in { + ClientState.FAILED, + ClientState.CLOSING, + ClientState.CLOSED, + }: + self._raise_unusable() remaining = None if deadline is None else deadline - time.monotonic() if remaining is not None and remaining <= 0: raise AcpRequestTimeoutError( f"timed out waiting for ACP {description}" ) - candidate = self._queue_get( - self._session_events, - remaining, - description, - ) - if isinstance(candidate, PermissionRequest): - with self._permission_lock: - pending = ( - self._pending_permissions.get(candidate.request_id) - is candidate - ) - if not pending: - continue - if self._session_event_matches(candidate, expected): - return candidate # type: ignore[return-value] - if len(self._session_event_backlog) >= self.max_pending_events: - raise AcpEventQueueFullError( - "ACP typed event backlog is full; consume the ordered stream" + self._session_event_condition.wait(timeout=remaining) + + def _matching_session_event_index(self, expected: object) -> int | None: + index = 0 + removed_stale = False + while index < len(self._session_events): + candidate = self._session_events[index] + if isinstance(candidate, PermissionRequest): + with self._permission_lock: + pending = ( + self._pending_permissions.get(candidate.request_id) + is candidate ) - self._session_event_backlog.append(candidate) + if not pending: + del self._session_events[index] + removed_stale = True + continue + if self._session_event_matches(candidate, expected): + if removed_stale: + self._session_event_condition.notify_all() + return index + index += 1 + if removed_stale: + self._session_event_condition.notify_all() + return None @staticmethod def _session_event_matches(candidate: SessionEvent, expected: object) -> bool: @@ -776,16 +786,20 @@ def reject_inbound_request( def close(self) -> None: wait_for_other_close = False + closed_without_process = False with self._state_lock: if self._state in {ClientState.CLOSED, ClientState.NEW}: self._state = ClientState.CLOSED self._closed.set() - return - if self._state is ClientState.CLOSING: + closed_without_process = True + elif self._state is ClientState.CLOSING: wait_for_other_close = True else: was_failed = self._state is ClientState.FAILED self._state = ClientState.CLOSING + if closed_without_process: + self._signal_queues() + return if wait_for_other_close: self._closed.wait(timeout=self.close_timeout * 3) return @@ -1070,10 +1084,7 @@ def _dispatch( return if isinstance(message, JsonRpcNotification): if message.method == "session/update": - self._put_lossless( - self._session_events, - parse_session_update(message.params), - ) + self._put_session_event(parse_session_update(message.params)) elif message.method in self.supported_extension_notifications: self._put_lossless( self._notifications, @@ -1111,7 +1122,7 @@ def _dispatch( ) ) else: - self._put_lossless(self._session_events, parsed) + self._put_session_event(parsed) elif message.method in self.supported_extension_requests: self._put_lossless( self._inbound_requests, @@ -1137,6 +1148,19 @@ def _put_lossless(self, target: queue.Queue[Any], value: Any) -> None: "ACP event queue is full; refusing to drop protocol data" ) from exc + def _put_session_event(self, value: SessionEvent) -> None: + deadline = time.monotonic() + min(self.request_timeout, 0.5) + with self._session_event_condition: + while len(self._session_events) >= self.max_pending_events: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AcpEventQueueFullError( + "ACP event queue is full; refusing to drop protocol data" + ) + self._session_event_condition.wait(timeout=remaining) + self._session_events.append(value) + self._session_event_condition.notify_all() + def _queue_get( self, source: queue.Queue[_T | object], @@ -1198,8 +1222,9 @@ def _fail_pending(self, failure: BaseException) -> None: pass def _signal_queues(self) -> None: + with self._session_event_condition: + self._session_event_condition.notify_all() for target in ( - self._session_events, self._notifications, self._inbound_requests, ): diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py index 51c0c49..859484a 100644 --- a/tests/test_acp_client.py +++ b/tests/test_acp_client.py @@ -15,7 +15,13 @@ AcpTransportError, ClientState, ) -from tendwire.backends.acp_protocol import AcpProtocolError, SessionUpdateKind, StopReason +from tendwire.backends.acp_protocol import ( + AcpProtocolError, + PermissionRequest, + SessionUpdate, + SessionUpdateKind, + StopReason, +) FAKE_AGENT = Path(__file__).parent / "fixtures" / "acp_fake_agent.py" @@ -25,6 +31,63 @@ def client(mode: str = "normal", **kwargs: object) -> AcpClient: return AcpClient([sys.executable, "-u", str(FAKE_AGENT), mode], **kwargs) +def _typed_update(index: int) -> SessionUpdate: + update = { + "sessionUpdate": "agent_message_chunk", + "messageId": f"message-{index}", + "content": {"type": "text", "text": str(index)}, + } + raw = {"sessionId": "s", "update": update} + return SessionUpdate("s", SessionUpdateKind.AGENT_MESSAGE_CHUNK, update, None, raw) + + +def _typed_permission(index: int) -> PermissionRequest: + raw = { + "sessionId": "s", + "toolCall": {"toolCallId": f"tool-{index}"}, + "options": [], + } + return PermissionRequest( + index, + "s", + raw["toolCall"], + (), + None, + raw, + ) + + +def _install_permission(acp: AcpClient, request: PermissionRequest) -> None: + with acp._permission_lock: + acp._pending_permissions[request.request_id] = request + acp._put_session_event(request) + + +def _waiter_has_released_condition(acp: AcpClient, ready: threading.Event) -> None: + assert ready.wait(timeout=1) + # The routing callback signals just before Condition.wait(). Acquiring the + # condition proves that the consumer has actually released it to sleep. + with acp._session_event_condition: + pass + + +def _wait_for_condition_waiters(acp: AcpClient, count: int) -> None: + deadline = time.monotonic() + 1 + while time.monotonic() < deadline: + with acp._session_event_condition: + if len(acp._session_event_condition._waiters) >= count: + return + time.sleep(0.001) + raise AssertionError("session event consumers did not start waiting") + + +def _capture_failure(method, failures: list[BaseException]) -> None: + try: + method() + except BaseException as exc: + failures.append(exc) + + def test_initialize_capabilities_and_session_lifecycle() -> None: with client() as acp: initialized = acp.initialize() @@ -121,6 +184,143 @@ def test_ordered_session_event_api_preserves_cross_kind_reader_order() -> None: assert outcome[0].stop_reason is StopReason.END_TURN +@pytest.mark.parametrize("first_kind", ("permission", "update")) +def test_concurrent_typed_consumers_route_either_first_kind_without_deadlock( + first_kind: str, +) -> None: + acp = client() + update = _typed_update(1) + permission = _typed_permission(1) + waiting = threading.Event() + original_match = acp._matching_session_event_index + waiting_type = SessionUpdate if first_kind == "permission" else PermissionRequest + + def observe_wait(expected: object) -> int | None: + result = original_match(expected) + if expected is waiting_type and result is None: + waiting.set() + return result + + acp._matching_session_event_index = observe_wait # type: ignore[method-assign] + if first_kind == "permission": + _install_permission(acp, permission) + else: + acp._put_session_event(update) + + updates: list[SessionUpdate] = [] + permissions: list[PermissionRequest] = [] + update_thread = threading.Thread(target=lambda: updates.append(acp.next_update())) + permission_thread = threading.Thread( + target=lambda: permissions.append(acp.next_permission_request()) + ) + blocked_thread = update_thread if first_kind == "permission" else permission_thread + matching_thread = permission_thread if first_kind == "permission" else update_thread + blocked_thread.start() + _waiter_has_released_condition(acp, waiting) + matching_thread.start() + matching_thread.join(timeout=1) + assert not matching_thread.is_alive() + + if first_kind == "permission": + acp._put_session_event(update) + else: + _install_permission(acp, permission) + blocked_thread.join(timeout=1) + assert not blocked_thread.is_alive() + assert updates == [update] + assert permissions == [permission] + assert list(acp._session_events) == [] + + +def test_ordered_consumer_remains_exact_with_mixed_session_events() -> None: + acp = client() + expected: list[SessionUpdate | PermissionRequest] = [] + for index in range(20): + event: SessionUpdate | PermissionRequest + if index % 2: + event = _typed_permission(index) + _install_permission(acp, event) + else: + event = _typed_update(index) + acp._put_session_event(event) + expected.append(event) + + assert [acp.next_session_event() for _ in expected] == expected + assert list(acp._session_events) == [] + + +def test_close_and_failure_wake_all_session_event_waiters() -> None: + acp = client() + failures: list[BaseException] = [] + threads = [ + threading.Thread( + target=lambda method=method: _capture_failure(method, failures) + ) + for method in (acp.next_update, acp.next_permission_request) + ] + for thread in threads: + thread.start() + _wait_for_condition_waiters(acp, 2) + acp.close() + for thread in threads: + thread.join(timeout=1) + assert not thread.is_alive() + assert len(failures) == 2 + + failed = client() + failure: list[BaseException] = [] + thread = threading.Thread( + target=lambda: _capture_failure(failed.next_session_event, failure) + ) + thread.start() + _wait_for_condition_waiters(failed, 1) + failed._set_failed(AcpTransportError("boom")) + thread.join(timeout=1) + assert not thread.is_alive() + assert len(failure) == 1 + assert isinstance(failure[0], AcpTransportError) + + +def test_mixed_typed_consumer_stress_is_bounded_and_exactly_once() -> None: + acp = client(max_pending_events=8) + count = 200 + updates: list[SessionUpdate] = [] + permissions: list[PermissionRequest] = [] + update_thread = threading.Thread( + target=lambda: updates.extend(acp.next_update() for _ in range(count)) + ) + permission_thread = threading.Thread( + target=lambda: permissions.extend( + acp.next_permission_request() for _ in range(count) + ) + ) + update_thread.start() + permission_thread.start() + maximum_depth = 0 + for index in range(count): + update = _typed_update(index) + permission = _typed_permission(index) + if index % 2: + _install_permission(acp, permission) + acp._put_session_event(update) + else: + acp._put_session_event(update) + _install_permission(acp, permission) + with acp._session_event_condition: + maximum_depth = max(maximum_depth, len(acp._session_events)) + update_thread.join(timeout=3) + permission_thread.join(timeout=3) + + assert not update_thread.is_alive() + assert not permission_thread.is_alive() + assert maximum_depth <= acp.max_pending_events + assert [item.update["messageId"] for item in updates] == [ + f"message-{index}" for index in range(count) + ] + assert [item.request_id for item in permissions] == list(range(count)) + assert list(acp._session_events) == [] + + def test_cancel_resolves_pending_permissions_as_cancelled() -> None: with client() as acp: acp.initialize() From 8f3d9216c7cd1f89bbb3e68d36c38955a705f989 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 22:10:57 +0800 Subject: [PATCH 30/83] fix(acp): fence replay projection authority --- docs/acp-migration.md | 7 + src/tendwire/store/sqlite.py | 188 +++++++++++++++++--- tests/test_acp_atomic_ingestion.py | 251 ++++++++++++++++++++++++++- tests/test_agent_events.py | 270 ++++++++++++++++++++++++++++- tests/test_store.py | 2 +- 5 files changed, 680 insertions(+), 38 deletions(-) diff --git a/docs/acp-migration.md b/docs/acp-migration.md index 4454fd5..a8909dc 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -144,6 +144,13 @@ original sequence and a replay-contract fingerprint, allowing exact retries to remain idempotent and conflicting reuse to fail closed without retaining messages, thoughts, raw tool input/output, or other source payloads. +Schema v26 tombstones also retain the original event `observed_at` as the only +authority time for a one-time repair when the matching owned turn projection +is provably absent. Exact replays never re-merge caller timestamp or content +into an existing live or superseded projection, so they cannot reorder final +connector delivery. Tombstones migrated from pre-v26 stores have no retained +authority time; they remain deduplication evidence but cannot repair a turn. + Each tombstone has bounded per-event identity metadata, but tombstone count is permanent and therefore grows with the number of distinct source events. Tombstones are intentionally not deleted automatically: removing them would make diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index 8fede08..c335b79 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -145,7 +145,7 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS -STORE_SCHEMA_VERSION = 25 +STORE_SCHEMA_VERSION = 26 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS _LEGACY_TURN_CLAIM_HARD_TTL_SECONDS = 86_400 TURN_CHANGE_RETENTION_DAYS = 7 @@ -1671,6 +1671,25 @@ def _record_response_size( ) CREATE_AGENT_EVENT_TOMBSTONES_TABLE = """ +CREATE TABLE IF NOT EXISTS agent_event_tombstones ( + host_id TEXT NOT NULL, + event_id TEXT NOT NULL, + sequence INTEGER NOT NULL CHECK (sequence >= 1), + replay_fingerprint TEXT NOT NULL CHECK (length(replay_fingerprint) = 64), + observed_at TEXT, + retired_at TEXT NOT NULL CHECK (length(retired_at) BETWEEN 20 AND 40), + PRIMARY KEY (host_id, event_id), + CHECK (length(host_id) BETWEEN 1 AND 2048), + CHECK (instr(host_id, char(0)) = 0), + CHECK (length(event_id) = 64), + CHECK (observed_at IS NULL OR length(observed_at) BETWEEN 20 AND 40) +); +""" + +# v24/v25 tombstones predate retained replay-authority time. Migration code +# must build the historical shape rather than whatever the current target DDL +# happens to contain. +CREATE_AGENT_EVENT_TOMBSTONES_V25_TABLE = """ CREATE TABLE IF NOT EXISTS agent_event_tombstones ( host_id TEXT NOT NULL, event_id TEXT NOT NULL, @@ -13342,11 +13361,23 @@ def _migrate_v22_to_v23_conn(conn: sqlite3.Connection) -> None: host_id = normalize_agent_event_identifier( row[1], "host_id", required=True ) + # v22 allowed tool/plan rows to be public. The current + # constructor and table correctly reject that historical + # shape, so normalize it while it is still in the private + # migration transaction instead of rebuilding through the + # latest DDL and failing before v24 -> v25 can privatise it. + legacy_sensitive = str(row[3]) in { + "thought", + "tool_call", + "tool_call_update", + "plan", + "extension", + } canonical = agent_event( kind=row[3], source=row[4], worker_id=row[5], - visibility=row[6], + visibility="private" if legacy_sensitive else row[6], source_session_id=row[7], source_turn_id=row[8], source_item_id=row[9], @@ -13371,7 +13402,7 @@ def _migrate_v22_to_v23_conn(conn: sqlite3.Connection) -> None: raise StoreSchemaError("invalid_v22_agent_event_identity") if str(row[14]) != canonical.payload_fingerprint: raise StoreSchemaError("invalid_v22_agent_event_fingerprint") - if public_payload != canonical.public_payload: + if not legacy_sensitive and public_payload != canonical.public_payload: raise StoreSchemaError("invalid_v22_agent_event_projection") except (TypeError, ValueError, OverflowError) as exc: raise StoreSchemaError("invalid_v22_agent_event_row") from exc @@ -13425,10 +13456,20 @@ def _migrate_v23_to_v24_conn(conn: sqlite3.Connection) -> None: payload_fingerprint, private_payload_json, public_payload_json ) SELECT - sequence, host_id, event_id, kind, source, worker_id, visibility, + sequence, host_id, event_id, kind, source, worker_id, + CASE + WHEN kind IN ('thought', 'tool_call', 'tool_call_update', 'plan', 'extension') + THEN 'private' + ELSE visibility + END, source_session_id, source_turn_id, source_item_id, source_message_id, source_event_id, source_sequence, observed_at, - payload_fingerprint, private_payload_json, public_payload_json + payload_fingerprint, private_payload_json, + CASE + WHEN kind IN ('thought', 'tool_call', 'tool_call_update', 'plan', 'extension') + THEN '{}' + ELSE public_payload_json + END FROM agent_events_v23 ORDER BY sequence """ @@ -13436,7 +13477,7 @@ def _migrate_v23_to_v24_conn(conn: sqlite3.Connection) -> None: conn.execute("DROP TABLE agent_events_v23") for statement in CREATE_AGENT_EVENT_INDEXES: conn.execute(statement) - conn.execute(CREATE_AGENT_EVENT_TOMBSTONES_TABLE) + conn.execute(CREATE_AGENT_EVENT_TOMBSTONES_V25_TABLE) for statement in CREATE_AGENT_EVENT_TOMBSTONE_INDEXES: conn.execute(statement) @@ -13477,6 +13518,19 @@ def _migrate_v24_to_v25_conn(conn: sqlite3.Connection) -> None: conn.execute(statement) +def _migrate_v25_to_v26_conn(conn: sqlite3.Connection) -> None: + """Retain original event authority time for safe tombstone repair.""" + if "observed_at" in _table_columns(conn, "agent_event_tombstones"): + return + conn.execute( + """ + ALTER TABLE agent_event_tombstones + ADD COLUMN observed_at TEXT + CHECK (observed_at IS NULL OR length(observed_at) BETWEEN 20 AND 40) + """ + ) + + MIGRATIONS: tuple[Migration, ...] = ( Migration(0, 1, _migrate_v0_to_v1_conn), Migration(1, 2, _migrate_v1_to_v2_conn), @@ -13503,6 +13557,7 @@ def _migrate_v24_to_v25_conn(conn: sqlite3.Connection) -> None: Migration(22, 23, _migrate_v22_to_v23_conn), Migration(23, 24, _migrate_v23_to_v24_conn), Migration(24, 25, _migrate_v24_to_v25_conn), + Migration(25, 26, _migrate_v25_to_v26_conn), ) @@ -17494,14 +17549,15 @@ def cleanup_event_retention( SELECT sequence, host_id, event_id, kind, source, worker_id, visibility, source_session_id, source_turn_id, source_item_id, source_message_id, - source_event_id, source_sequence, payload_fingerprint, public_payload_json + source_event_id, source_sequence, payload_fingerprint, public_payload_json, + observed_at FROM agent_events """ def _agent_event_retention_candidate( row: tuple[Any, ...], -) -> tuple[str, str, int, str]: +) -> tuple[str, str, int, str, str]: """Reduce one bounded metadata row to its durable tombstone fields.""" public_payload_json = str(row[14]) public_payload = _json_object(public_payload_json) @@ -17526,6 +17582,7 @@ def _agent_event_retention_candidate( payload_fingerprint=str(row[13]), public_payload_json=public_payload_json, ), + str(row[15]), ) @@ -17545,14 +17602,16 @@ def _cleanup_agent_event_retention_conn( + " ORDER BY observed_at, sequence LIMIT ?", (str(host_id), cutoff_at, int(batch_size) + 1), ) - candidates: list[tuple[str, str, int, str]] = [] + candidates: list[tuple[str, str, int, str, str]] = [] remaining = False for row in cursor: if len(candidates) >= batch_size: remaining = True break if dry_run: - candidates.append((str(row[1]), str(row[2]), int(row[0]), "")) + candidates.append( + (str(row[1]), str(row[2]), int(row[0]), "", str(row[15])) + ) else: candidates.append(_agent_event_retention_candidate(row)) @@ -17561,12 +17620,26 @@ def _cleanup_agent_event_retention_conn( conn.executemany( """ INSERT INTO agent_event_tombstones ( - host_id, event_id, sequence, replay_fingerprint, retired_at - ) VALUES (?, ?, ?, ?, ?) + host_id, event_id, sequence, replay_fingerprint, + observed_at, retired_at + ) VALUES (?, ?, ?, ?, ?, ?) """, ( - (candidate_host, event_id, sequence, fingerprint, retired_at) - for candidate_host, event_id, sequence, fingerprint in candidates + ( + candidate_host, + event_id, + sequence, + fingerprint, + observed_at, + retired_at, + ) + for ( + candidate_host, + event_id, + sequence, + fingerprint, + observed_at, + ) in candidates ), ) sequences = [candidate[2] for candidate in candidates] @@ -23278,6 +23351,54 @@ def _agent_event_binding_matches_conn( ) +def _agent_event_authority_observed_at_conn( + conn: sqlite3.Connection, + host_id: str, + event_id: str, +) -> str | None: + """Return the immutable event time, including retained replay metadata. + + Tombstones written before schema v26 have no retained authority time. + They remain valid deduplication evidence but cannot authorize a repair. + """ + row = conn.execute( + "SELECT observed_at FROM agent_events WHERE host_id = ? AND event_id = ?", + (str(host_id), str(event_id)), + ).fetchone() + if row is None: + row = conn.execute( + "SELECT observed_at FROM agent_event_tombstones " + "WHERE host_id = ? AND event_id = ?", + (str(host_id), str(event_id)), + ).fetchone() + if row is None or row[0] is None: + return None + observed_at = _strict_utc_timestamp(row[0]) + if observed_at is None: + raise StoreSchemaError("invalid_agent_event_authority_time") + return observed_at + + +def _owned_turn_projection_exists_conn( + conn: sqlite3.Connection, + host_id: str, + worker_id: str, + source_turn_id: str, +) -> bool: + """Fail closed when any live or superseded owned projection still exists.""" + rows = conn.execute( + "SELECT payload_json FROM turns WHERE host_id = ? AND worker_id = ?", + (str(host_id), str(worker_id)), + ).fetchall() + for row in rows: + payload = _json_object(row[0]) + if _owned_source_turn_matches(payload, source_turn_id) or _source_turn_matches( + payload, source_turn_id + ): + return True + return False + + def _turn_refresh_is_cancelled( *, deadline_monotonic: float | None, @@ -23333,10 +23454,9 @@ def append_agent_event_and_apply_turn_for_binding( ) -> AppendProjectedAgentEventResult: """Atomically journal an agent event and apply its text-only turn projection. - A replay still runs the idempotent projection merge. This lets a retry - repair a projection that was independently removed without duplicating the - journal event or its revision-keyed connector delivery. Retention - tombstones participate in the same replay contract as live event rows. + A replay may repair a provably absent projection exactly once, using the + immutable observation time retained with the original event. It never + re-merges caller data into a live or superseded owned projection. """ normalized_host = normalize_agent_event_identifier( @@ -23356,7 +23476,9 @@ def append_agent_event_and_apply_turn_for_binding( raise ValueError(f"turn_model must be one of: {allowed}") if _fault_inject is not None and not callable(_fault_inject): raise TypeError("_fault_inject must be callable or None") - current_time, _ = _pending_observed_time(observed_at or event.observed_at) + # Validate the compatibility argument, but never grant it replay authority. + if observed_at is not None: + _pending_observed_time(observed_at) rearm_key: tuple[str, str] | None = None def fault(boundary: str) -> None: @@ -23388,7 +23510,29 @@ def fault(boundary: str) -> None: ) fault("after_event_append") turn: TurnRefreshApplyResult | None = None - if content is not None: + authority_time = _agent_event_authority_observed_at_conn( + conn, + normalized_host or "", + event.event_id, + ) + projection_allowed = content is not None + if not appended.inserted and content is not None: + durable_source_turn = str(event.source_turn_id or "").strip() + caller_source_turn = str(content.get("source_turn_id") or "").strip() + projection_allowed = bool( + authority_time is not None + and durable_source_turn + and caller_source_turn == durable_source_turn + and not _owned_turn_projection_exists_conn( + conn, + normalized_host or "", + event.worker_id, + durable_source_turn, + ) + ) + if projection_allowed and content is not None: + if authority_time is None: + raise StoreSchemaError("agent_event_authority_time_missing") worker_exists = conn.execute( "SELECT 1 FROM workers WHERE host_id = ? AND worker_id = ?", (normalized_host or "", event.worker_id), @@ -23400,7 +23544,7 @@ def fault(boundary: str) -> None: normalized_host or "", event.worker_id, content, - observed_at=current_time, + observed_at=authority_time, turn_model=normalized_turn_model, ) rearm_key = ( @@ -23414,7 +23558,7 @@ def fault(boundary: str) -> None: normalized_host or "", owner_key, fingerprint, - now=current_time, + now=authority_time, ) turn = TurnRefreshApplyResult(merge_result.updated, False) fault("after_turn_projection") diff --git a/tests/test_acp_atomic_ingestion.py b/tests/test_acp_atomic_ingestion.py index 66bbf7d..bd01ae4 100644 --- a/tests/test_acp_atomic_ingestion.py +++ b/tests/test_acp_atomic_ingestion.py @@ -13,6 +13,7 @@ from tendwire.core.agent_events import AgentEvent, agent_event from tendwire.core.models import WorkerBinding from tendwire.core.projector import project_from_raw +from tendwire.store import sqlite as store_sqlite from tendwire.store.sqlite import ( append_agent_event_and_apply_turn_for_binding, cleanup_agent_event_retention, @@ -28,12 +29,26 @@ def _store( tmp_path: Path, *, source: str = "acp_preferred", + stable_owner: bool = False, ) -> tuple[Config, WorkerBinding]: db_path = tmp_path / "events.db" config = Config(host_id="host-a", db_path=db_path, agent_event_source=source) snapshot = project_from_raw( config, - workers=[{"id": "worker-a", "name": "Worker A"}], + workers=[ + { + "id": "worker-a", + "name": "Worker A", + "meta": ( + { + "stable_key": "wsk1_" + ("a" * 64), + "stable_key_version": 1, + } + if stable_owner + else {} + ), + } + ], ) init_store(db_path) save_snapshot(db_path, snapshot) @@ -58,26 +73,35 @@ def _event( binding: WorkerBinding, *, observed_at: str = "2026-07-31T00:00:00+00:00", + source_turn_id: str = "turn-a", + message_id: str = "message-a", + source_event_id: str = "event-a", + text: str = "answer", ) -> AgentEvent: return agent_event( kind="agent_message", source="acp", worker_id=binding.worker_id, - payload={"schema_version": 1, "message_id": "message-a", "text": "answer"}, + payload={"schema_version": 1, "message_id": message_id, "text": text}, source_session_id="session-a", - source_turn_id="turn-a", - source_message_id="message-a", - source_event_id="event-a", + source_turn_id=source_turn_id, + source_message_id=message_id, + source_event_id=source_event_id, observed_at=observed_at, ) -def _content(*, complete: bool = False) -> dict[str, object]: +def _content( + *, + complete: bool = False, + source_turn_id: str = "turn-a", + text: str = "answer", +) -> dict[str, object]: return { - "source_turn_id": "turn-a", + "source_turn_id": source_turn_id, "user_text": "", - "assistant_stream_text": "" if complete else "answer", - "assistant_final_text": "answer" if complete else "", + "assistant_stream_text": "" if complete else text, + "assistant_final_text": text if complete else "", "complete": complete, "has_open_turn": not complete, } @@ -182,6 +206,215 @@ def test_tombstoned_replay_can_repair_projection_without_reinserting_event( assert _counts(config.db_path) == (0, 1) +@pytest.mark.parametrize("retire_original", (False, True)) +def test_replay_cannot_supersede_newer_turn_or_requeue_connector( + tmp_path: Path, + retire_original: bool, +) -> None: + config, binding = _store(tmp_path, stable_owner=True) + old = _event( + binding, + observed_at="2020-01-01T00:00:00+00:00", + source_turn_id="turn-old", + message_id="message-old", + source_event_id="event-old", + text="old", + ) + first = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + old, + expected_binding=binding, + content=_content(complete=True, source_turn_id="turn-old", text="old"), + ) + assert first.event.status == "inserted" + if retire_original: + assert cleanup_agent_event_retention( + config.db_path, + config.host_id, + retention_days=1, + now="2021-01-01T00:00:00+00:00", + )["tombstoned"] == 1 + + new = _event( + binding, + observed_at="2026-01-01T00:00:00+00:00", + source_turn_id="turn-new", + message_id="message-new", + source_event_id="event-new", + text="new", + ) + second = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + new, + expected_binding=binding, + content=_content(complete=True, source_turn_id="turn-new", text="new"), + ) + assert second.event.status == "inserted" + turns_before = turns_payload_from_store(config.db_path, config.host_id)["turns"] + assert turns_before[0]["assistant_final_text"] == "new" + with sqlite3.connect(config.db_path) as conn: + outbox_before = conn.execute( + "SELECT turn_id, status FROM connector_outbox ORDER BY id" + ).fetchall() + + replayed = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + replace(old, observed_at="2030-01-01T00:00:00+00:00"), + expected_binding=binding, + content=_content(complete=True, source_turn_id="turn-old", text="old"), + observed_at="2040-01-01T00:00:00+00:00", + ) + + assert replayed.event.status == "replayed" + assert replayed.turn is None + turns = turns_payload_from_store(config.db_path, config.host_id)["turns"] + assert turns == turns_before + with sqlite3.connect(config.db_path) as conn: + assert conn.execute( + "SELECT turn_id, status FROM connector_outbox ORDER BY id" + ).fetchall() == outbox_before + + +@pytest.mark.parametrize("retire_original", (False, True)) +def test_replay_repairs_only_absent_projection_with_original_authority_time( + tmp_path: Path, + retire_original: bool, +) -> None: + config, binding = _store(tmp_path, stable_owner=True) + event = _event(binding, observed_at="2020-01-01T00:00:00+00:00") + inserted = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + ) + assert inserted.event.status == "inserted" + if retire_original: + assert cleanup_agent_event_retention( + config.db_path, + config.host_id, + retention_days=1, + now="2021-01-01T00:00:00+00:00", + )["tombstoned"] == 1 + + repaired = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + replace(event, observed_at="2030-01-01T00:00:00+00:00"), + expected_binding=binding, + content=_content(complete=True), + observed_at="2040-01-01T00:00:00+00:00", + ) + assert repaired.event.status == "replayed" + assert repaired.turn is not None and repaired.turn.updated == 1 + with sqlite3.connect(config.db_path) as conn: + turn_before = conn.execute( + "SELECT payload_json, observed_at FROM turns" + ).fetchone() + outbox_before = conn.execute( + "SELECT turn_id, status FROM connector_outbox ORDER BY id" + ).fetchall() + assert turn_before is not None + assert turn_before[1] == "2020-01-01T00:00:00+00:00" + + ignored = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + replace(event, observed_at="2050-01-01T00:00:00+00:00"), + expected_binding=binding, + content=_content(complete=True, text="caller rewrite"), + ) + assert ignored.event.status == "replayed" + assert ignored.turn is None + with sqlite3.connect(config.db_path) as conn: + assert conn.execute( + "SELECT payload_json, observed_at FROM turns" + ).fetchone() == turn_before + assert conn.execute( + "SELECT turn_id, status FROM connector_outbox ORDER BY id" + ).fetchall() == outbox_before + + +def test_legacy_tombstone_without_authority_time_cannot_repair(tmp_path: Path) -> None: + config, binding = _store(tmp_path) + event = _event(binding, observed_at="2020-01-01T00:00:00+00:00") + append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + ) + cleanup_agent_event_retention( + config.db_path, + config.host_id, + retention_days=1, + now="2021-01-01T00:00:00+00:00", + ) + with sqlite3.connect(config.db_path) as conn: + conn.execute("UPDATE agent_event_tombstones SET observed_at = NULL") + + replayed = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(complete=True), + ) + assert replayed.event.status == "replayed" + assert replayed.turn is None + assert _counts(config.db_path) == (0, 0) + + +def test_replay_repair_respects_binding_fence_and_rolls_back(tmp_path: Path) -> None: + config, binding = _store(tmp_path) + event = _event(binding) + append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + ) + stale = replace(binding, target_value="old-pane") + fenced = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=stale, + content=_content(), + ) + assert fenced.event.status == "binding_changed" + assert fenced.turn is None + assert _counts(config.db_path) == (1, 0) + + def fail(boundary: str) -> None: + if boundary == "before_commit": + raise RuntimeError("repair rollback") + + with pytest.raises(RuntimeError, match="repair rollback"): + append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(), + _fault_inject=fail, + ) + assert _counts(config.db_path) == (1, 0) + repaired = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(), + ) + assert repaired.event.status == "replayed" + assert repaired.turn is not None + assert _counts(config.db_path) == (1, 1) + + def _update(text: str) -> dict[str, object]: return { "method": "session/update", diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py index 8a36b1b..8bb905c 100644 --- a/tests/test_agent_events.py +++ b/tests/test_agent_events.py @@ -23,6 +23,124 @@ from tendwire.store import sqlite as store_sqlite +# Exact production table shapes from schema v22 and v23. These fixtures do +# not use the current target DDL, because doing so masks cross-version rebuild +# failures when historical public tool/plan rows are populated. +_HISTORICAL_V22_AGENT_EVENTS_DDL = """ +CREATE TABLE agent_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + host_id TEXT NOT NULL, + event_id TEXT NOT NULL, + kind TEXT NOT NULL CHECK ( + kind IN ( + 'user_message', 'agent_message', 'thought', 'tool_call', + 'tool_call_update', 'plan', 'usage', 'session_info' + ) + ), + source TEXT NOT NULL, + worker_id TEXT NOT NULL, + visibility TEXT NOT NULL CHECK (visibility IN ('private', 'public')), + source_session_id TEXT, + source_turn_id TEXT, + source_item_id TEXT, + source_message_id TEXT, + source_event_id TEXT, + source_sequence INTEGER CHECK (source_sequence >= 0), + observed_at TEXT NOT NULL, + payload_fingerprint TEXT NOT NULL, + private_payload_json TEXT NOT NULL, + public_payload_json TEXT NOT NULL, + UNIQUE (host_id, event_id), + CHECK (source_event_id IS NOT NULL OR source_sequence IS NOT NULL), + CHECK (kind != 'thought' OR visibility = 'private') +) +""" + +_HISTORICAL_V23_AGENT_EVENTS_DDL = """ +CREATE TABLE agent_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + host_id TEXT NOT NULL, + event_id TEXT NOT NULL, + kind TEXT NOT NULL CHECK ( + kind IN ( + 'user_message', 'agent_message', 'thought', 'tool_call', + 'tool_call_update', 'plan', 'usage', 'session_info', 'extension' + ) + ), + source TEXT NOT NULL, + worker_id TEXT NOT NULL, + visibility TEXT NOT NULL CHECK (visibility IN ('private', 'public')), + source_session_id TEXT, + source_turn_id TEXT, + source_item_id TEXT, + source_message_id TEXT, + source_event_id TEXT, + source_sequence INTEGER CHECK (source_sequence >= 0), + observed_at TEXT NOT NULL, + payload_fingerprint TEXT NOT NULL, + private_payload_json TEXT NOT NULL, + public_payload_json TEXT NOT NULL, + UNIQUE (host_id, event_id), + CHECK (length(host_id) BETWEEN 1 AND 2048), + CHECK (instr(host_id, char(0)) = 0), + CHECK (length(event_id) = 64), + CHECK (length(source) BETWEEN 1 AND 2048), + CHECK (instr(source, char(0)) = 0), + CHECK (length(worker_id) BETWEEN 1 AND 2048), + CHECK (instr(worker_id, char(0)) = 0), + CHECK ( + source_session_id IS NULL OR ( + length(source_session_id) BETWEEN 1 AND 2048 + AND instr(source_session_id, char(0)) = 0 + ) + ), + CHECK ( + source_turn_id IS NULL OR ( + length(source_turn_id) BETWEEN 1 AND 2048 + AND instr(source_turn_id, char(0)) = 0 + ) + ), + CHECK ( + source_item_id IS NULL OR ( + length(source_item_id) BETWEEN 1 AND 2048 + AND instr(source_item_id, char(0)) = 0 + ) + ), + CHECK ( + source_message_id IS NULL OR ( + length(source_message_id) BETWEEN 1 AND 2048 + AND instr(source_message_id, char(0)) = 0 + ) + ), + CHECK ( + source_event_id IS NULL OR ( + length(source_event_id) BETWEEN 1 AND 2048 + AND instr(source_event_id, char(0)) = 0 + ) + ), + CHECK (length(observed_at) BETWEEN 20 AND 40), + CHECK (length(payload_fingerprint) = 64), + CHECK ( + CASE WHEN json_valid(private_payload_json) + THEN json_type(private_payload_json) = 'object' ELSE 0 END + ), + CHECK (length(CAST(private_payload_json AS BLOB)) <= 65536), + CHECK ( + CASE WHEN json_valid(public_payload_json) + THEN json_type(public_payload_json) = 'object' ELSE 0 END + ), + CHECK (length(CAST(public_payload_json AS BLOB)) <= 65536), + CHECK (visibility != 'private' OR public_payload_json = '{}'), + CHECK (source_event_id IS NOT NULL OR source_sequence IS NOT NULL), + CHECK ( + source_event_id IS NOT NULL + OR (source_session_id IS NOT NULL AND source_sequence IS NOT NULL) + ), + CHECK (kind NOT IN ('thought', 'extension') OR visibility = 'private') +) +""" + + def _message_event( *, sequence: int, @@ -782,7 +900,7 @@ def test_retention_conflict_rolls_back_and_serializes_concurrent_append( def blocking_candidate( row: tuple[object, ...], - ) -> tuple[str, str, int, str]: + ) -> tuple[str, str, int, str, str]: entered.set() assert release.wait(timeout=5) return original(row) @@ -897,7 +1015,9 @@ def test_v23_to_v24_preserves_populated_journal_sequence(tmp_path: Path) -> None ) conn.commit() store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == (25,) + assert conn.execute("PRAGMA user_version").fetchone() == ( + store_sqlite.STORE_SCHEMA_VERSION, + ) assert conn.execute( "SELECT sequence FROM agent_events WHERE event_id = ?", (event.event_id,), @@ -927,7 +1047,7 @@ def test_v24_to_v25_privatises_populated_tool_and_plan_rows(tmp_path: Path) -> N ] with sqlite3.connect(db_path) as conn: store_sqlite._run_migrations(conn, target_version=24) - conn.execute("DROP TABLE agent_events") + conn.execute("DROP TABLE IF EXISTS agent_events") conn.execute( store_sqlite.CREATE_AGENT_EVENTS_TABLE.replace( """kind NOT IN ( @@ -969,7 +1089,9 @@ def test_v24_to_v25_privatises_populated_tool_and_plan_rows(tmp_path: Path) -> N ) conn.commit() store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == (25,) + assert conn.execute("PRAGMA user_version").fetchone() == ( + store_sqlite.STORE_SCHEMA_VERSION, + ) assert conn.execute( "SELECT sequence, kind, visibility, public_payload_json " "FROM agent_events ORDER BY sequence" @@ -989,6 +1111,138 @@ def test_v24_to_v25_privatises_populated_tool_and_plan_rows(tmp_path: Path) -> N ] == list(kinds) +@pytest.mark.parametrize( + ("source_version", "historical_ddl"), + ( + (22, _HISTORICAL_V22_AGENT_EVENTS_DDL), + (23, _HISTORICAL_V23_AGENT_EVENTS_DDL), + ), +) +def test_authentic_populated_public_tool_migrations_reach_current_private_schema( + tmp_path: Path, + source_version: int, + historical_ddl: str, +) -> None: + db_path = tmp_path / f"authentic-v{source_version}.db" + kinds = ("tool_call", "tool_call_update", "plan") + with sqlite3.connect(db_path) as conn: + store_sqlite._run_migrations(conn, target_version=source_version - 1) + conn.execute("DROP TABLE IF EXISTS agent_events") + conn.execute(historical_ddl) + for sequence, kind in enumerate(kinds, 51): + event = agent_event( + kind=kind, + source="acp", + worker_id="worker-1", + source_session_id="private-session", + source_sequence=sequence, + payload={"content": [{"type": "text", "text": f"private-{kind}"}]}, + observed_at="2026-07-31T00:00:00+00:00", + ) + event_id = event.event_id + if source_version == 22: + legacy_identity = { + "schema_version": 1, + "source": event.source, + "session_id": event.source_session_id, + "event_id": event.source_event_id, + "sequence": event.source_sequence, + "kind": event.kind, + } + event_id = hashlib.sha256( + store_sqlite._canonical_json(legacy_identity).encode("utf-8") + ).hexdigest() + conn.execute( + """ + INSERT INTO agent_events ( + sequence, host_id, event_id, kind, source, worker_id, + visibility, source_session_id, source_turn_id, + source_item_id, source_message_id, source_event_id, + source_sequence, observed_at, payload_fingerprint, + private_payload_json, public_payload_json + ) VALUES (?, ?, ?, ?, ?, ?, 'public', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + sequence, + "host-1", + event_id, + event.kind, + event.source, + event.worker_id, + event.source_session_id, + event.source_turn_id, + event.source_item_id, + event.source_message_id, + event.source_event_id, + event.source_sequence, + event.observed_at, + event.payload_fingerprint, + store_sqlite._canonical_json(event.payload), + store_sqlite._canonical_json(event.payload), + ), + ) + conn.execute(f"PRAGMA user_version = {source_version}") + conn.commit() + + store_sqlite._run_migrations(conn) + + assert conn.execute("PRAGMA user_version").fetchone() == ( + store_sqlite.STORE_SCHEMA_VERSION, + ) + assert conn.execute( + "SELECT sequence, kind, visibility, public_payload_json " + "FROM agent_events ORDER BY sequence" + ).fetchall() == [ + (51, "tool_call", "private", "{}"), + (52, "tool_call_update", "private", "{}"), + (53, "plan", "private", "{}"), + ] + assert conn.execute("PRAGMA integrity_check").fetchone() == ("ok",) + + +def test_v25_to_v26_retains_legacy_tombstones_as_dedup_only(tmp_path: Path) -> None: + db_path = tmp_path / "v25-tombstone.db" + legacy_event_id = "a" * 64 + with sqlite3.connect(db_path) as conn: + store_sqlite._run_migrations(conn, target_version=25) + assert "observed_at" not in { + str(row[1]) + for row in conn.execute("PRAGMA table_info(agent_event_tombstones)") + } + conn.execute( + """ + INSERT INTO agent_event_tombstones ( + host_id, event_id, sequence, replay_fingerprint, retired_at + ) VALUES ('host-1', ?, 1, ?, '2026-01-02T00:00:00+00:00') + """, + (legacy_event_id, "b" * 64), + ) + conn.commit() + store_sqlite._run_migrations(conn) + assert conn.execute("PRAGMA user_version").fetchone() == (26,) + assert conn.execute( + "SELECT observed_at FROM agent_event_tombstones WHERE event_id = ?", + (legacy_event_id,), + ).fetchone() == (None,) + + event = replace( + _message_event(sequence=901, visibility="private"), + observed_at="2020-01-01T00:00:00+00:00", + ) + store_sqlite.append_agent_event(db_path, "host-1", event) + assert store_sqlite.cleanup_agent_event_retention( + db_path, + "host-1", + retention_days=1, + now="2021-01-01T00:00:00+00:00", + )["tombstoned"] == 1 + with sqlite3.connect(db_path) as conn: + assert conn.execute( + "SELECT observed_at FROM agent_event_tombstones WHERE event_id = ?", + (event.event_id,), + ).fetchone() == ("2020-01-01T00:00:00+00:00",) + + @pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) def test_agent_event_schema_migrates_from_every_prior_version( tmp_path: Path, @@ -1026,7 +1280,9 @@ def test_v21_migration_is_idempotent_and_preserves_existing_store( store_sqlite.init_store(db_path) store_sqlite.init_store(db_path) with sqlite3.connect(db_path) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == (25,) + assert conn.execute("PRAGMA user_version").fetchone() == ( + store_sqlite.STORE_SCHEMA_VERSION, + ) columns = { str(row[1]) for row in conn.execute("PRAGMA table_info(agent_events)") } @@ -1087,7 +1343,9 @@ def test_v22_migration_rekeys_legacy_event_identity_without_losing_sequence( "SELECT sequence, event_id FROM agent_events" ).fetchone() assert row == (19, event.event_id) - assert conn.execute("PRAGMA user_version").fetchone() == (25,) + assert conn.execute("PRAGMA user_version").fetchone() == ( + store_sqlite.STORE_SCHEMA_VERSION, + ) replay = store_sqlite.append_agent_event(db_path, "host-1", event) assert replay.inserted is False diff --git a/tests/test_store.py b/tests/test_store.py index 5d55f86..d9fd090 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -10278,7 +10278,7 @@ def test_v13_migration_repairs_nonpositive_turn_sequences_and_blocks_recurrence( init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 25 + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 26 assert conn.execute( """ SELECT turn_id, list_sequence From 6543163abf56e0dbedd21f24479ae611cee6eac1 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 22:15:49 +0800 Subject: [PATCH 31/83] fix(acp): close remaining runtime protocol gaps --- src/tendwire/backends/acp_ingestion.py | 50 +++++++-- src/tendwire/backends/acp_projection.py | 13 +++ src/tendwire/backends/acp_runtime.py | 30 ++++-- tests/test_acp_ingestion.py | 94 +++++++++++++++++ tests/test_acp_projection.py | 44 ++++++++ tests/test_acp_runtime.py | 133 +++++++++++++++++++++--- 6 files changed, 334 insertions(+), 30 deletions(-) diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py index a125b77..4551fad 100644 --- a/src/tendwire/backends/acp_ingestion.py +++ b/src/tendwire/backends/acp_ingestion.py @@ -85,6 +85,7 @@ def __init__( self._turn_ordinal = 0 self._source_turn_id: str | None = None self._turn_complete = False + self._local_prompt_recorded = False @property def source_turn_id(self) -> str | None: @@ -101,9 +102,10 @@ def start_turn(self, *, producer_turn_id: str | None = None) -> str: raise ValueError("producer_turn_id must be non-empty text or None") self._turn_ordinal += 1 self.projector.reset_turn(self.session_id) - # An authoritative producer turn ID must retain identity across ACP - # transport recreation. Generation only scopes locally synthesized - # ordinals, whose meaning cannot survive a reconnect. + # An authoritative producer turn ID retains identity across ACP + # transport recreation. The fallback exists only for unsolicited or + # historical inbound streams; outgoing prompts require producer + # identity before any durable or remote side effect. identity = ( { "source": "acp", @@ -114,12 +116,12 @@ def start_turn(self, *, producer_turn_id: str | None = None) -> str: else { "source": "acp", "session": self.session_id, - "generation": self.stream_generation, "turn": self._turn_ordinal, } ) self._source_turn_id = f"acpt_{stable_fingerprint(identity)}" self._turn_complete = False + self._local_prompt_recorded = False return self._source_turn_id def ingest_update( @@ -142,6 +144,17 @@ def ingest_update( update_kind = _session_update_kind(notification) if self._turn_complete and update_kind in _TURN_SCOPED_UPDATES: return AcpIngestionResult(None, ignored_reason="turn_already_complete") + if ( + update_kind == "user_message_chunk" + and self._local_prompt_recorded + and not replay + ): + # ACP agents commonly echo the prompt as a user-message update. + # begin_prompt() already journaled the complete producer-owned + # input, so accepting the echo would duplicate both the journal + # and the compatibility turn. Load replay has no local producer + # record and must continue to retain historical user messages. + return AcpIngestionResult("user_message", ignored_reason="prompt_echo") thought_rejection = _thought_rejection_reason( notification, policy=self.config.acp_thought_policy, @@ -180,6 +193,8 @@ def begin_prompt( ) -> AcpIngestionResult: """Durably record outgoing prompt content before transport send.""" + if not isinstance(producer_turn_id, str) or not producer_turn_id.strip(): + raise ValueError("producer_turn_id must be non-empty text") blocks = [dict(block) for block in prompt] if not blocks: raise ValueError("prompt must contain at least one content block") @@ -224,11 +239,14 @@ def begin_prompt( except BaseException: self._restore_speculation(checkpoint, prior_turn_state) raise - return self._accept( + result = self._accept( canonical, checkpoint=checkpoint, prior_turn_state=prior_turn_state, ) + if result.event is not None and result.event.status != "binding_changed": + self._local_prompt_recorded = True + return result def reset_after_load(self) -> None: """Drop replay turn assembly before accepting a new active prompt.""" @@ -236,6 +254,7 @@ def reset_after_load(self) -> None: self.projector.reset_turn(self.session_id) self._source_turn_id = None self._turn_complete = False + self._local_prompt_recorded = False def ingest_permission_request( self, @@ -344,6 +363,7 @@ def mark_prompt_complete( ignored_reason="stale_binding", ) self._turn_complete = True + self._local_prompt_recorded = False return AcpIngestionResult( "extension", event=persisted.event, @@ -360,7 +380,7 @@ def _accept( canonical: Mapping[str, Any], *, checkpoint: AcpProjectionCheckpoint, - prior_turn_state: tuple[int, str | None, bool], + prior_turn_state: tuple[int, str | None, bool, bool], project_turn: bool = True, replay_namespace: str | None = None, ) -> AcpIngestionResult: @@ -450,16 +470,26 @@ def _accept( ), ) - def _turn_state(self) -> tuple[int, str | None, bool]: - return self._turn_ordinal, self._source_turn_id, self._turn_complete + def _turn_state(self) -> tuple[int, str | None, bool, bool]: + return ( + self._turn_ordinal, + self._source_turn_id, + self._turn_complete, + self._local_prompt_recorded, + ) def _restore_speculation( self, checkpoint: AcpProjectionCheckpoint, - prior_turn_state: tuple[int, str | None, bool], + prior_turn_state: tuple[int, str | None, bool, bool], ) -> None: self.projector.restore_session(checkpoint) - self._turn_ordinal, self._source_turn_id, self._turn_complete = prior_turn_state + ( + self._turn_ordinal, + self._source_turn_id, + self._turn_complete, + self._local_prompt_recorded, + ) = prior_turn_state _STOP_REASON_OUTCOMES = { diff --git a/src/tendwire/backends/acp_projection.py b/src/tendwire/backends/acp_projection.py index 312d295..231920b 100644 --- a/src/tendwire/backends/acp_projection.py +++ b/src/tendwire/backends/acp_projection.py @@ -33,6 +33,7 @@ "plan", "usage", "session_info", + "extension", } ) @@ -43,6 +44,9 @@ "tool_call": "tool_call", "tool_call_update": "tool_call_update", "plan": "plan", + "available_commands_update": "extension", + "current_mode_update": "extension", + "config_option_update": "extension", "usage_update": "usage", "session_info_update": "session_info", } @@ -245,6 +249,12 @@ def normalize_session_update( payload = self._normalize_plan(state, update) elif kind == "usage": payload = self._normalize_usage(state, update) + elif kind == "extension": + payload = { + "schema_version": 1, + "extension": f"acp.session_update.{update_name}", + "update": _without_discriminator(update), + } else: payload = self._normalize_session_info(state, update) extension_meta = _scoped_metadata(params=params, update=update) @@ -1236,6 +1246,9 @@ def _canonical_event( if kind == "thought": privacy = "private" private_fields = ["payload"] + elif kind == "extension": + privacy = "private" + private_fields = ["payload"] elif kind in {"tool_call", "tool_call_update"}: privacy = "mixed" for field_name in _RAW_TOOL_FIELDS: diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index fc9b920..32e7bca 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -222,6 +222,8 @@ def __init__( "ACP new requires a non-ACP worker continuity binding" ) if mode is not SessionOpenMode.NEW: + if binding.backend != "acp": + raise ValueError("ACP load/resume requires an ACP backend binding") if binding.turn_target_kind != "acp_session_id": raise ValueError("ACP runtime requires an ACP session worker binding") if binding.turn_target_value != session_id: @@ -269,7 +271,12 @@ def __init__( self._threads: tuple[threading.Thread, ...] = () self._event_idle_epoch = 0 self._setup_replay = False - self._provisional_binding: WorkerBinding | None = None + # NEW acquires authority through its binder during start. LOAD/RESUME + # are handed an existing live ACP lease and own releasing it once the + # runtime stops or fails. + self._provisional_binding: WorkerBinding | None = ( + binding if mode is not SessionOpenMode.NEW else None + ) self._close_thread: threading.Thread | None = None self._close_failures: list[BaseException] = [] @@ -376,6 +383,9 @@ def prompt( with self._prompt_lock: self.raise_if_failed() session_id, ingestor = self._running_components() + if not isinstance(producer_turn_id, str) or not producer_turn_id.strip(): + raise ValueError("producer_turn_id must be non-empty text") + stable_producer_turn_id = producer_turn_id.strip() with self._state_lock: self._prompts_started += 1 try: @@ -383,7 +393,7 @@ def prompt( prepared_prompt = _prepare_prompt_content(self._client, prompt) prompt_event = ingestor.begin_prompt( prepared_prompt, - producer_turn_id=producer_turn_id, + producer_turn_id=stable_producer_turn_id, ) _raise_for_binding_rejection(prompt_event) except BaseException as exc: @@ -756,8 +766,9 @@ def _consume_session_events(self) -> None: setup_replay=setup_replay, ) _raise_for_binding_rejection(outcome) - with self._state_lock: - self._updates_ingested += 1 + if _outcome_has_persisted_event(outcome): + with self._state_lock: + self._updates_ingested += 1 elif isinstance(event, PermissionRequest): self._handle_permission( event, @@ -792,8 +803,9 @@ def _handle_permission( setup_replay=setup_replay, ) _raise_for_binding_rejection(outcome) - with self._state_lock: - self._permissions_ingested += 1 + if _outcome_has_persisted_event(outcome): + with self._state_lock: + self._permissions_ingested += 1 selected: str | None = None callback_failure: BaseException | None = None @@ -963,6 +975,12 @@ def _raise_for_binding_rejection(outcome: object) -> None: raise AcpRuntimeBindingError("ACP worker binding is no longer current") +def _outcome_has_persisted_event(outcome: object) -> bool: + """Count only updates that reached the durable event boundary.""" + + return getattr(outcome, "event", None) is not None + + __all__ = [ "AcpRuntime", "AcpRuntimeBindingError", diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py index abbe64b..3bf8af6 100644 --- a/tests/test_acp_ingestion.py +++ b/tests/test_acp_ingestion.py @@ -16,6 +16,7 @@ AppendProjectedAgentEventResult, TurnRefreshApplyResult, list_agent_events, + list_public_agent_events, upsert_worker_bindings, ) @@ -491,6 +492,99 @@ def apply(_path, _host, _worker, content, **_kwargs): ) +def test_live_prompt_echo_is_suppressed_but_load_replay_user_message_is_retained( + tmp_path: Path, +) -> None: + db_path = tmp_path / "events.db" + binding = _binding() + upsert_worker_bindings(db_path, [binding]) + ingestor = AcpSessionIngestor( + _config(db_path, agent_event_source="acp_shadow"), + session_id="session-a", + stream_generation="generation-a", + binding=binding, + ) + ingestor.begin_prompt( + ({"type": "text", "text": "one question"},), + producer_turn_id="producer-turn-a", + ) + + echo = ingestor.ingest_update( + _update( + "user_message_chunk", + messageId="adapter-echo", + content={"type": "text", "text": "one question"}, + ) + ) + historical = ingestor.ingest_update( + _update( + "user_message_chunk", + messageId="historical-user", + content={"type": "text", "text": "historical question"}, + ), + replay=True, + setup_replay=True, + ) + + assert echo.event is None and echo.ignored_reason == "prompt_echo" + assert historical.event is not None + events = list_agent_events(db_path, "host-a") + assert [event.event.kind for event in events] == ["user_message", "user_message"] + assert events[0].event.payload["assembled_text"] == "one question" + assert events[1].event.payload["assembled_text"] == "historical question" + assert list_public_agent_events(db_path, "host-a") == () + + +@pytest.mark.parametrize( + ("update_kind", "fields"), + [ + ("available_commands_update", {"availableCommands": [{"name": "review"}]}), + ("current_mode_update", {"currentModeId": "agent"}), + ( + "config_option_update", + {"configOptions": [{"id": "model", "currentValue": "safe"}]}, + ), + ], +) +def test_stable_control_updates_persist_privately_and_replay_idempotently( + tmp_path: Path, + update_kind: str, + fields: dict[str, object], +) -> None: + db_path = tmp_path / f"{update_kind}.db" + binding = _binding() + upsert_worker_bindings(db_path, [binding]) + notification = _update(update_kind, **fields) + + outcomes = [] + for generation in ("generation-a", "generation-b"): + ingestor = AcpSessionIngestor( + _config(db_path), + session_id="session-a", + stream_generation=generation, + binding=binding, + ) + outcomes.append( + ingestor.ingest_update( + notification, + replay=True, + setup_replay=True, + ) + ) + + assert outcomes[0].event is not None and outcomes[0].event.status == "inserted" + assert outcomes[1].event is not None and outcomes[1].event.status == "replayed" + events = list_agent_events(db_path, "host-a") + assert len(events) == 1 + assert events[0].event.kind == "extension" + assert events[0].event.visibility == "private" + assert events[0].event.public_payload == {} + assert events[0].event.payload["extension"] == ( + f"acp.session_update.{update_kind}" + ) + assert list_public_agent_events(db_path, "host-a") == () + + def test_duplicate_durable_event_can_idempotently_repair_projection(tmp_path: Path) -> None: projected = False diff --git a/tests/test_acp_projection.py b/tests/test_acp_projection.py index 17232fc..79fe025 100644 --- a/tests/test_acp_projection.py +++ b/tests/test_acp_projection.py @@ -16,6 +16,50 @@ def _update(session_update: str, **fields: object) -> dict[str, object]: } +@pytest.mark.parametrize( + ("update_kind", "fields"), + [ + ( + "available_commands_update", + {"availableCommands": [{"name": "review", "description": "Review"}]}, + ), + ("current_mode_update", {"currentModeId": "agent"}), + ( + "config_option_update", + {"configOptions": [{"id": "model", "currentValue": "safe"}]}, + ), + ], +) +def test_stable_session_control_updates_are_private_canonical_extensions( + update_kind: str, + fields: dict[str, object], +) -> None: + projector = AcpEventProjector() + notification = _update(update_kind, **fields) + + event = projector.normalize_session_update( + notification, + source_event_id="stable-update-1", + ) + duplicate = projector.normalize_session_update( + notification, + source_event_id="stable-update-1", + replay=True, + ) + + assert event is not None + assert event["kind"] == "extension" + assert event["privacy"] == "private" + assert event["private_fields"] == ["payload"] + assert event["payload"] == { + "schema_version": 1, + "extension": f"acp.session_update.{update_kind}", + "update": fields, + } + assert duplicate is None + assert projector.project_turn_content("session-1")["has_open_turn"] is False + + def test_message_chunks_are_assembled_by_session_kind_and_message_id() -> None: projector = AcpEventProjector() diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index 4eb1634..02ff784 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -166,8 +166,17 @@ def __init__(self, session_id: str = "session-private") -> None: self.load_resets = 0 self.update_failure: BaseException | None = None self.permission_failure: BaseException | None = None - self.update_result: object = None - self.permission_result: object = None + persisted = SimpleNamespace(status="inserted") + self.update_result: object = SimpleNamespace( + event=persisted, + turn=None, + ignored_reason=None, + ) + self.permission_result: object = SimpleNamespace( + event=persisted, + turn=None, + ignored_reason=None, + ) self.completion_result: object = None def start_turn(self, *, producer_turn_id: str | None = None) -> str: @@ -217,7 +226,7 @@ def binding(session_id: str = "session-private") -> WorkerBinding: host_id="host-a", worker_id="worker-public", worker_fingerprint="worker-fingerprint", - backend="herdr", + backend="acp", target_kind="pane_id", target_value="pane-private-secret", turn_target_kind="acp_session_id", @@ -403,6 +412,15 @@ def factory(config: Config, **kwargs: object) -> FakeIngestor: assert service.status().healthy finally: service.stop() + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + retired = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(retired) == 1 + assert retired[0].reason == "acp_runtime_stopped" @pytest.mark.parametrize( @@ -469,6 +487,15 @@ def test_load_and_resume_use_requested_session( assert ingestor.load_resets == (1 if mode is SessionOpenMode.LOAD else 0) finally: service.stop() + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + retired = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(retired) == 1 + assert retired[0].reason == "acp_runtime_stopped" @pytest.mark.parametrize("mode", [SessionOpenMode.LOAD, SessionOpenMode.RESUME]) @@ -507,6 +534,40 @@ def test_load_and_resume_reject_agent_session_mismatch_and_close( assert client.closed assert client.close_calls == 1 assert service.status().state is RuntimeState.FAILED + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + retired = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(retired) == 1 + assert retired[0].reason == "acp_startup_rollback" + + +@pytest.mark.parametrize("mode", [SessionOpenMode.LOAD, SessionOpenMode.RESUME]) +def test_load_and_resume_reject_non_acp_binding_before_transport( + tmp_path: Path, + mode: SessionOpenMode, +) -> None: + client = FakeClient() + legacy = replace(binding("existing-private"), backend="herdr") + + with pytest.raises(ValueError, match="ACP backend binding"): + AcpRuntime( + client, # type: ignore[arg-type] + config=Config( + host_id="host-a", + db_path=tmp_path / "events.db", + agent_event_source="acp_required", + ), + binding=legacy, + cwd=tmp_path, + session_mode=mode, + session_id="existing-private", + ) + + assert client.calls == [] def test_new_accepts_unpredictable_agent_generated_session_id(tmp_path: Path) -> None: @@ -700,7 +761,7 @@ def test_prompt_rechecks_binding_before_remote_send(tmp_path: Path) -> None: ) with pytest.raises(AcpRuntimeBindingError): - service.prompt("must not send") + service.prompt("must not send", producer_turn_id="producer-private") assert [call[0] for call in client.calls].count("prompt") == 0 @@ -1101,6 +1162,43 @@ def test_prompt_finalizes_only_after_valid_response_and_update_drain( service.stop() +def test_prompt_requires_crash_stable_producer_identity_before_remote_send( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + with pytest.raises(ValueError, match="producer_turn_id"): + service.prompt("question") + with pytest.raises(ValueError, match="producer_turn_id"): + service.prompt("question", producer_turn_id=" ") + + assert [call[0] for call in client.calls].count("prompt") == 0 + assert ingestor.started == [] + assert service.status().prompts_started == 0 + assert service.status().state is RuntimeState.RUNNING + finally: + service.stop() + + +def test_ignored_update_does_not_increment_persisted_counter(tmp_path: Path) -> None: + client = FakeClient() + ingestor = FakeIngestor() + ingestor.update_result = SimpleNamespace( + event=None, + turn=None, + ignored_reason="prompt_echo", + ) + service = runtime(tmp_path, client, ingestor).start() + try: + client.updates.put(update()) + wait_until(lambda: len(ingestor.updates) == 1) + assert service.status().updates_ingested == 0 + finally: + service.stop() + + def test_stale_binding_completion_is_terminal_and_not_counted_complete( tmp_path: Path, ) -> None: @@ -1114,7 +1212,7 @@ def test_stale_binding_completion_is_terminal_and_not_counted_complete( service = runtime(tmp_path, client, ingestor).start() with pytest.raises(AcpRuntimeBindingError): - service.prompt("question") + service.prompt("question", producer_turn_id="producer-private") status = service.status() assert status.state is RuntimeState.FAILED @@ -1148,7 +1246,13 @@ def decide(_request: PermissionRequest) -> str: def run_prompt() -> None: try: - result.append(service.prompt("question", drain_timeout=0.5)) + result.append( + service.prompt( + "question", + producer_turn_id="producer-private", + drain_timeout=0.5, + ) + ) except BaseException as exc: failure.append(exc) @@ -1182,12 +1286,13 @@ def test_cross_kind_ingestion_cannot_overtake_an_active_update( order: list[str] = [] class OrderedIngestor(FakeIngestor): - def ingest_update(self, raw: object, **kwargs: Any) -> None: + def ingest_update(self, raw: object, **kwargs: Any) -> object: order.append("update-start") update_entered.set() assert release_update.wait(timeout=1) - super().ingest_update(raw, **kwargs) + result = super().ingest_update(raw, **kwargs) order.append("update-end") + return result def ingest_permission_request( self, @@ -1195,9 +1300,9 @@ def ingest_permission_request( *, source_event_id: str | None = None, **kwargs: Any, - ) -> None: + ) -> object: order.append("permission") - super().ingest_permission_request( + return super().ingest_permission_request( raw, source_event_id=source_event_id, **kwargs, @@ -1280,16 +1385,16 @@ def test_prompt_transport_failure_cancels_and_makes_runtime_terminal( service = runtime(tmp_path, client, ingestor).start() with pytest.raises(AcpRequestTimeoutError) as raised: - service.prompt("question") + service.prompt("question", producer_turn_id="producer-private") assert raised.value is failure - assert ingestor.started == [None] + assert ingestor.started == ["producer-private"] assert ingestor.completions == 0 assert ("cancel", ("session-private",), {}) in client.calls assert service.status().cancellation_requests == 1 assert service.status().state is RuntimeState.FAILED with pytest.raises(AcpRequestTimeoutError): - service.prompt("unsafe retry") + service.prompt("unsafe retry", producer_turn_id="producer-private-2") with pytest.raises(AcpRequestTimeoutError): service.stop() @@ -1303,7 +1408,7 @@ def test_invalid_prompt_response_never_marks_complete_and_propagates( service = runtime(tmp_path, client, ingestor).start() with pytest.raises(AcpRuntimeProtocolError, match="invalid response"): - service.prompt("question") + service.prompt("question", producer_turn_id="producer-private") assert ingestor.completions == 0 assert service.status().state is RuntimeState.FAILED with pytest.raises(AcpRuntimeProtocolError): From 71665afc6768647ac0e042f8316488b417dcdcb0 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 22:20:33 +0800 Subject: [PATCH 32/83] test: avoid ACP probe scheduling flake --- tests/test_acp_probe.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/test_acp_probe.py b/tests/test_acp_probe.py index 816f504..11e35ac 100644 --- a/tests/test_acp_probe.py +++ b/tests/test_acp_probe.py @@ -71,9 +71,9 @@ def test_initialize_only_agent_does_not_gain_untested_baseline_claims() -> None: @pytest.mark.parametrize( ("mode", "failure", "timeout"), [ - ("malformed", ProbeFailure.PROTOCOL, 0.5), - ("partial_eof", ProbeFailure.PROTOCOL, 0.5), - ("bool_version", ProbeFailure.PROTOCOL_VERSION, 0.5), + ("malformed", ProbeFailure.PROTOCOL, 2.0), + ("partial_eof", ProbeFailure.PROTOCOL, 2.0), + ("bool_version", ProbeFailure.PROTOCOL_VERSION, 2.0), ("no_read", ProbeFailure.TIMEOUT, 0.05), ], ) From 33c556a7fee824f9a14b5658e91782aef2792c4e Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 22:39:06 +0800 Subject: [PATCH 33/83] test: expect ACP hardened schema --- docs/acp-migration.md | 10 +++++++--- tests/test_backend_pending.py | 2 +- tests/test_connector_outbox.py | 2 +- tests/test_delivery_retention_migration.py | 2 +- tests/test_delivery_retention_projection.py | 2 +- tests/test_delivery_retention_recovery.py | 2 +- tests/test_store.py | 2 +- 7 files changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/acp-migration.md b/docs/acp-migration.md index a8909dc..2f08da5 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -58,6 +58,8 @@ The structured event journal accepts these semantic kinds: - plan - usage - session information +- private extension/control state, including available commands, current mode, + and session configuration updates Producer IDs, raw inputs, raw outputs, session IDs, terminal IDs, paths, and reasoning are private. Public turn projection is deliberately narrower: @@ -189,6 +191,8 @@ must pass before `acp_required` is considered: - fallback after adapter failure without regressing existing final delivery; - exact worker continuity across Herdr pane moves and agent-session recreation. -ACP prompt submission, cancellation, and permission handling are a later -control-path migration. They must preserve Tendwire's existing request receipts -and uncertain-outcome rules before replacing Herdr command routing. +The isolated ACP runtime now implements prompt submission, cancellation, and +permission handling. Production daemon routing remains a later control-path +migration: it must connect those primitives to the per-worker authority +coordinator while preserving Tendwire's existing request receipts and +uncertain-outcome rules before replacing Herdr command routing. diff --git a/tests/test_backend_pending.py b/tests/test_backend_pending.py index 2676486..5edb76c 100644 --- a/tests/test_backend_pending.py +++ b/tests/test_backend_pending.py @@ -1449,7 +1449,7 @@ def test_current_schema_creation_has_exact_binding_and_claim_state(tmp_path: Pat db = tmp_path / "current-schema.db" init_store(db) with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 23 + assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 26 columns = { str(row[1]) for row in conn.execute("PRAGMA table_info(backend_pending)").fetchall() diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index 6841af3..a2c698a 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -1754,7 +1754,7 @@ def test_v6_to_current_plan_migration_is_bounded_atomic_and_preserves_jobs( ).fetchall() } foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert version == store_sqlite.STORE_SCHEMA_VERSION == 23 + assert version == store_sqlite.STORE_SCHEMA_VERSION == 26 assert plan_row == (plan["plan_token"], 1, None, "active") assert job_count == 2 assert outbox_count == 3 diff --git a/tests/test_delivery_retention_migration.py b/tests/test_delivery_retention_migration.py index 7fcb34c..675af0f 100644 --- a/tests/test_delivery_retention_migration.py +++ b/tests/test_delivery_retention_migration.py @@ -875,7 +875,7 @@ def test_v10_to_v12_migration_retains_finals_without_reposting_or_leaking( finals = _seed_v10_finals(db_path) init_store(db_path) - assert store_sqlite.STORE_SCHEMA_VERSION == 23 + assert store_sqlite.STORE_SCHEMA_VERSION == 26 delivered_key = _final_key(*finals["delivered"]) hold_keys = {_final_key(*finals[label]) for label in ("hold-a", "hold-b")} diff --git a/tests/test_delivery_retention_projection.py b/tests/test_delivery_retention_projection.py index 47fb289..7e6d35b 100644 --- a/tests/test_delivery_retention_projection.py +++ b/tests/test_delivery_retention_projection.py @@ -147,7 +147,7 @@ def _assert_continuity_integrity(db_path: Path) -> None: with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA user_version").fetchone() == ( store_sqlite.STORE_SCHEMA_VERSION, - ) == (23,) + ) == (26,) assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] current_counts = conn.execute( """ diff --git a/tests/test_delivery_retention_recovery.py b/tests/test_delivery_retention_recovery.py index 6f6670e..48bcac1 100644 --- a/tests/test_delivery_retention_recovery.py +++ b/tests/test_delivery_retention_recovery.py @@ -973,7 +973,7 @@ def test_v12_migration_uses_effective_recovery_lineage_without_repost_or_hold( api = ConnectorOutboxAPI(db_path, HOST_ID) assert api.poll({"name": FINAL_NAME, "limit": 100})["items"] == [] with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 23 + assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 26 anchor = conn.execute( """ SELECT delivery_kind, status diff --git a/tests/test_store.py b/tests/test_store.py index d9fd090..5f9deac 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -14151,7 +14151,7 @@ def test_v20_to_v21_adds_herdr_turn_watermark_and_provenance_tables( with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA user_version").fetchone() == ( store_sqlite.STORE_SCHEMA_VERSION, - ) == (23,) + ) == (26,) assert { str(row[0]) for row in conn.execute( From cb9b948c4f3d9eed721174c1bd1f976e842f80b0 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 23:13:33 +0800 Subject: [PATCH 34/83] test: stabilize final ACP release gate --- tests/test_herdr_events.py | 4 ++-- tests/test_release_readiness.py | 11 +++++++++++ 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/tests/test_herdr_events.py b/tests/test_herdr_events.py index f1a89f3..c8e0a48 100644 --- a/tests/test_herdr_events.py +++ b/tests/test_herdr_events.py @@ -1484,7 +1484,7 @@ def refresh_current() -> None: binding = next(iter(backend._bindings.values())) refreshes.append( herdr_turns.refresh_turn_binding( - backend.config, binding, adapter_timeout_seconds=1 + backend.config, binding, adapter_timeout_seconds=2 ) ) @@ -1527,7 +1527,7 @@ def refresh_current() -> None: binding = next(iter(backend._bindings.values())) refreshes.append( herdr_turns.refresh_turn_binding( - backend.config, binding, adapter_timeout_seconds=1 + backend.config, binding, adapter_timeout_seconds=2 ) ) diff --git a/tests/test_release_readiness.py b/tests/test_release_readiness.py index 00443e7..962f126 100644 --- a/tests/test_release_readiness.py +++ b/tests/test_release_readiness.py @@ -286,6 +286,17 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( "deleted": 0, "remaining_candidates": False, }, + "agent_events": { + "host_id": None, + "retention_days": 36500, + "cutoff_at": "1926-02-04T00:00:00+00:00", + "batch_size": 5, + "examined": 0, + "deleted": 0, + "tombstoned": 0, + "remaining_candidates": False, + "replay_identity_retained": True, + }, "final_retention": { "examined": 0, "deleted": 0, From 91dba776e19f2d2c1efdb690bc4f75b20757e9f3 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Fri, 31 Jul 2026 23:52:02 +0800 Subject: [PATCH 35/83] feat(acp): wire Herdr-owned production runtimes --- README.md | 15 +- docs/acp-migration.md | 76 ++- src/tendwire/backends/acp_client.py | 7 +- src/tendwire/backends/acp_coordinator.py | 733 ++++++++++++++++++++++ src/tendwire/backends/acp_runtime.py | 76 ++- src/tendwire/backends/herdr_socket.py | 26 + src/tendwire/backends/herdr_turns.py | 26 +- src/tendwire/command_submission.py | 227 ++++++- src/tendwire/daemon.py | 31 +- tests/fixtures/herdr_acp_contract_v1.json | 37 ++ tests/test_acp_coordinator.py | 465 ++++++++++++++ tests/test_acp_runtime.py | 32 + 12 files changed, 1706 insertions(+), 45 deletions(-) create mode 100644 src/tendwire/backends/acp_coordinator.py create mode 100644 tests/fixtures/herdr_acp_contract_v1.json create mode 100644 tests/test_acp_coordinator.py diff --git a/README.md b/README.md index 6d27500..7f493a7 100644 --- a/README.md +++ b/README.md @@ -570,14 +570,13 @@ snapshot/projections instead of publishing a truncated authoritative snapshot. Incremental events that would add workers over the cap are ignored with the same public-safe degraded evidence. -The stock daemon currently defaults to `legacy`. ACP runtime discovery, -per-worker authority selection, and automatic reconnect are not production -wired yet; ACP modes are integration/test surfaces that require an explicitly -supplied runtime factory. `acp_shadow` persists ACP events without projecting -them, but no automated shadow comparator is implemented. `acp_preferred` must -not be treated as a production authority promise until that coordinator exists. -`acp_required` fails startup without an explicit healthy runtime and never -starts the legacy turn scheduler. None of these modes makes agent thoughts +The stock daemon currently defaults to `legacy`. ACP modes use Herdr's private +`agent.acp_endpoint` contract and accept only workers explicitly marked +`acp_owned_ready`; ordinary PTY workers are never attached as sidecars. +`acp_shadow` persists ACP events without projecting them, but no automated +shadow comparator is implemented. `acp_preferred` falls back only before an ACP +reservation/send, while `acp_required` fails closed and never starts the legacy +turn scheduler. None of these modes makes agent thoughts public: thought events remain private diagnostic data unless a separate, explicit sanitized projection is introduced. diff --git a/docs/acp-migration.md b/docs/acp-migration.md index 2f08da5..c5a825e 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -2,7 +2,8 @@ This document defines the experimental migration from backend-specific transcript readers to Agent Client Protocol (ACP). ACP is not yet Tendwire's -default or a production-wired semantic authority. +default. The stock daemon now contains the production coordinator and command +path, but activates them only for an explicitly Herdr-owned ACP worker. Herdr remains authoritative for workspace, pane, worker identity, process liveness, and command routing until the ACP control path is proven separately. Tendwire remains authoritative for persistence, reconciliation, public safety, @@ -13,21 +14,22 @@ command receipts, and connector delivery. `TENDWIRE_AGENT_EVENT_SOURCE` controls projection precedence: - `legacy`: use the existing Herdr/Codex/OMP turn readers only. -- `acp_shadow`: with an explicitly supplied ACP runtime, ingest ACP events +- `acp_shadow`: ingest ACP events durably without projecting them; legacy turns remain authoritative. Automated comparison is not implemented yet. -- `acp_preferred`: an experimental integration surface for future per-worker - ACP authority and legacy fallback. The stock daemon does not discover or - construct an ACP runtime. +- `acp_preferred`: use an explicitly Herdr-owned ACP endpoint when available; + fall back to legacy only before any ACP command reservation or observable + send. - `acp_required`: use ACP only and fail closed when the binding or stream is not healthy. The daemon does not start its legacy turn scheduler in this mode. - This mode requires an explicitly supplied healthy authority runtime and is - intended for conformance testing, not rollout. + This mode requires every eligible worker endpoint to be ACP-owned and healthy. + Zero observed workers is a valid idle state; if a later worker appears + without ACP ownership, runtime health immediately becomes degraded. -The default is `legacy`. Selecting an ACP mode does not discover an adapter, -invent an ACP session, or bind a worker. Until a per-worker authority -coordinator exists, operators must not interpret `acp_preferred` as proof that -ACP is authoritative. +The default is `legacy`. In an ACP mode the coordinator asks Herdr for a +one-shot private endpoint, validates its worker generation and explicit +`acp_owned_ready` lifecycle, then creates/loads/resumes the ACP session. An +ordinary live PTY session is never treated as ACP-owned. ## Authority split @@ -36,7 +38,7 @@ ACP is authoritative. | Workspace and logical pane identity | Herdr | | Public stable worker identity | Tendwire's authenticated Herdr projection | | ACP session and message identity | ACP agent, stored privately by Tendwire | -| Messages, thoughts, tools, plans, and usage | Future ACP authority coordinator; currently experimental | +| Messages, thoughts, tools, plans, and usage | ACP coordinator for ACP-owned workers; currently experimental | | Turn finality and connector eligibility | Tendwire durable projection | | Telegram presentation and delivery state | Herdres | | Command idempotency and uncertain outcomes | Tendwire command receipts | @@ -109,7 +111,7 @@ Tendwire rebase. The current initialization-only probe is not a promotion gate: it does not authenticate, create/load a session, prompt, validate updates, exercise permissions/cancellation, or pin an executable digest. Stateful conformance fixtures and an immutable rollback manifest are still required. -Session resume/replay reconciliation is likewise incomplete. +Adapter promotion and rollback remain operator-managed. ## Runtime lifecycle @@ -119,10 +121,23 @@ shutdown. Tendwire must not claim an ACP worker healthy until initialization, capability negotiation, session creation/load/resume, and private worker binding all succeed. -The stock daemon currently has no production ACP runtime factory or multi-worker -supervisor. Herdr must first provide authenticated per-worker launch/session -metadata, and Tendwire must add per-worker health, authority, reconnect, and -durable projection recovery before ACP can become the default. +The stock daemon has a multi-worker runtime factory. It discovers endpoints +through Herdr's private `agent.acp_endpoint` method, validates the fixed stdio +attach shape, and supervises one runtime per worker generation. Endpoint +tickets are one-shot private values: Tendwire uses one only for its immediate +attach and never persists or publishes it. Reconnect always re-resolves Herdr +authority and mints a fresh endpoint. + +While attached, the coordinator uses non-mutating `agent.acp_status` checks +before every prompt and during reconciliation. The reported lifecycle must be +`acp_owned_attached` and its numeric generation must match the attached slot. +A mismatch or unavailable status retires the slot before any prompt frame is +written. Endpoint minting is never used as a status probe. + +In `acp_preferred`, the legacy scheduler remains available only for workers not +currently owned by a healthy ACP slot. It rechecks this exclusion after dequeue +and immediately before a legacy read, preventing queued legacy work from +overwriting or duplicating the active ACP worker projection. Disconnect handling is conservative: @@ -132,8 +147,11 @@ Disconnect handling is conservative: 4. Reinitialize and rebind before accepting ACP events again. 5. Reconcile replayed messages and tool calls by producer identity. -These disconnect steps are requirements, not a description of the current -implementation. +Command acknowledgement occurs after the complete `session/prompt` request +frame is written, not after the agent finishes the turn. End-of-turn response +and update draining continue under runtime supervision. A failure after the +durable `send_started` transition is terminally uncertain and never falls back +to a second transport. ## Retention @@ -163,12 +181,11 @@ not an immediate physical-erasure or cryptographic-erasure guarantee. ## Cross-repository requirements -Herdr needs an ACP-aware launch or proxy surface that exposes enough private -metadata for Tendwire to bind an ACP session to an existing logical pane. The -binding must survive terminal/session churn without making ACP identity a -public continuity input. It must also identify adapter executable/version, -session-open mode, working directory, and binding generation without exposing -those values on public APIs. +Herdr provides a private `agent.acp_endpoint` launch/proxy surface containing +adapter identity/version, session-open mode, cwd, generation, and an explicitly +ACP-owned lifecycle. Tendwire accepts only the configured Herdr executable and +the fixed `agent acp-attach` argument shape; arbitrary executable, environment, +or argument injection is rejected. Herdres needs optional presentations for sanitized tool and plan progress. It does not ingest ACP directly: it continues polling Tendwire's neutral outbox so @@ -191,8 +208,7 @@ must pass before `acp_required` is considered: - fallback after adapter failure without regressing existing final delivery; - exact worker continuity across Herdr pane moves and agent-session recreation. -The isolated ACP runtime now implements prompt submission, cancellation, and -permission handling. Production daemon routing remains a later control-path -migration: it must connect those primitives to the per-worker authority -coordinator while preserving Tendwire's existing request receipts and -uncertain-outcome rules before replacing Herdr command routing. +The ACP runtime implements prompt submission, cancellation, permission handling, +per-worker coordination, reconnect, and receipt-backed command routing. ACP +remains non-default until the cross-repository integration and rollout gates +above pass against real adapters. diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index f3fe8e5..a5e1244 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -22,7 +22,7 @@ from enum import Enum from pathlib import Path from types import MappingProxyType -from typing import Any, TypeVar +from typing import Any, Callable, TypeVar from tendwire import __version__ @@ -406,6 +406,7 @@ def request( *, timeout: float | None = None, require_initialized: bool = True, + on_written: Callable[[], None] | None = None, ) -> Any: if require_initialized: self._require_initialized() @@ -432,6 +433,8 @@ def request( request_envelope(request_id, method, params), deadline=deadline, ) + if on_written is not None: + on_written() except BaseException: with self._pending_lock: if self._pending.get(request_id) is pending: @@ -580,6 +583,7 @@ def prompt( prompt: str | Sequence[Mapping[str, Any]], *, timeout: float | None = None, + on_submitted: Callable[[], None] | None = None, ) -> PromptResult: content = list(self.prepare_prompt(prompt)) session_id = _nonempty(session_id, "session_id") @@ -595,6 +599,7 @@ def prompt( "session/prompt", {"sessionId": session_id, "prompt": content}, timeout=self.prompt_timeout if timeout is None else timeout, + on_written=on_submitted, ) response_received = True finally: diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py new file mode 100644 index 0000000..00b7234 --- /dev/null +++ b/src/tendwire/backends/acp_coordinator.py @@ -0,0 +1,733 @@ +"""Production Herdr-to-ACP worker discovery and runtime coordination. + +Herdr remains the authority for live worker identity and mints one-shot attach +endpoints. This module validates that private contract, supervises one ACP +runtime per worker generation, and exposes opaque prompt routes to the daemon. +No endpoint ticket, process argv, cwd, adapter identity, or ACP session ID is +part of the public health surface. +""" + +from __future__ import annotations + +import threading +import time +from collections.abc import Callable, Mapping +from dataclasses import dataclass, replace +from pathlib import Path +from typing import Any + +from ..config import Config +from ..core.models import Worker, WorkerBinding +from ..store.sqlite import list_worker_bindings, upsert_worker_bindings +from .acp_client import AcpClient +from .acp_runtime import AcpRuntime, RuntimeState, SessionOpenMode +from .herdr_socket import HerdrSocketClient + + +class AcpCoordinatorError(RuntimeError): + """The private Herdr ACP endpoint contract or supervisor failed.""" + + +@dataclass(frozen=True, slots=True) +class HerdrAcpEndpoint: + command: tuple[str, ...] + cwd: Path + generation: str + session_mode: SessionOpenMode + session_id: str | None + + +@dataclass(frozen=True, slots=True) +class HerdrAcpStatus: + generation: str + lifecycle: str + + +@dataclass(slots=True) +class _RuntimeSlot: + continuity: WorkerBinding + generation: str + runtime: AcpRuntime + + +class _PromptRoute: + def __init__(self, owner: "AcpRuntimeCoordinator", worker: Worker) -> None: + self._owner = owner + self._worker = worker + + @property + def binding_fingerprint(self) -> str: + slot = self._owner._current_slot(self._worker) + binding = getattr(slot.runtime, "_binding", None) + return ( + str(binding.private_fingerprint) + if isinstance(binding, WorkerBinding) + else "" + ) + + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + ) -> object: + slot = self._owner._current_slot(self._worker) + self._owner._require_attached_generation(slot) + return slot.runtime.submit_prompt( + text, + producer_turn_id=producer_turn_id, + acknowledgement_timeout=timeout, + ) + + +EndpointClientFactory = Callable[[Config], Any] +RuntimeFactory = Callable[..., AcpRuntime] +ClientFactory = Callable[..., AcpClient] + + +class AcpRuntimeCoordinator: + """Reconcile Herdr worker authority into per-generation ACP runtimes.""" + + def __init__( + self, + config: Config, + stop_event: threading.Event, + *, + endpoint_client_factory: EndpointClientFactory | None = None, + runtime_factory: RuntimeFactory = AcpRuntime, + client_factory: ClientFactory = AcpClient, + reconcile_interval: float | None = None, + ) -> None: + if config.db_path is None: + raise ValueError("ACP coordinator requires a sqlite db path") + self.config = config + self._daemon_stop = stop_event + self._endpoint_client_factory = ( + endpoint_client_factory or _default_endpoint_client_factory + ) + self._runtime_factory = runtime_factory + self._client_factory = client_factory + self._reconcile_interval = max( + 1.0, + float( + config.turn_refresh_interval_seconds + if reconcile_interval is None + else reconcile_interval + ), + ) + self._lock = threading.RLock() + self._stop = threading.Event() + self._slots: dict[str, _RuntimeSlot] = {} + self._thread: threading.Thread | None = None + self._state = RuntimeState.NEW + self._failure_type: str | None = None + self._required_degraded = False + + def start(self) -> "AcpRuntimeCoordinator": + with self._lock: + if self._state is RuntimeState.RUNNING: + return self + if self._state is not RuntimeState.NEW: + raise AcpCoordinatorError("ACP coordinator cannot be restarted") + self._state = RuntimeState.STARTING + try: + self._reconcile(strict=self.config.agent_event_source == "acp_required") + except Exception as exc: + with self._lock: + self._state = RuntimeState.FAILED + self._failure_type = type(exc).__name__ + self._stop_all() + raise + with self._lock: + self._state = RuntimeState.RUNNING + thread = threading.Thread( + target=self._run, + name="tendwire-acp-coordinator", + daemon=True, + ) + self._thread = thread + thread.start() + return self + + def stop(self, *, timeout: float | None = None) -> None: + limit = ( + self.config.acp_shutdown_timeout_seconds + if timeout is None + else float(timeout) + ) + if limit <= 0: + raise ValueError("stop timeout must be positive") + with self._lock: + if self._state is RuntimeState.STOPPED: + return + self._state = RuntimeState.STOPPING + self._stop.set() + self._stop_all(timeout=limit) + thread = self._thread + if thread is not None and thread is not threading.current_thread(): + thread.join(timeout=limit) + with self._lock: + if self._state is not RuntimeState.FAILED: + self._state = RuntimeState.STOPPED + + def join(self, timeout: float | None = None) -> bool: + thread = self._thread + if thread is None or thread is threading.current_thread(): + return True + thread.join(timeout=timeout) + return not thread.is_alive() + + def status(self) -> dict[str, Any]: + counters = { + "updates_ingested": 0, + "permissions_ingested": 0, + "permissions_selected": 0, + "permissions_cancelled": 0, + "invalid_permission_selections": 0, + "prompts_started": 0, + "prompts_completed": 0, + "prompts_failed": 0, + "cancellation_requests": 0, + } + with self._lock: + state = self._state + slots = tuple(self._slots.values()) + failure_type = self._failure_type + required_degraded = self._required_degraded + healthy = state is RuntimeState.RUNNING and not required_degraded + for slot in slots: + try: + status = slot.runtime.status() + except Exception as exc: # noqa: BLE001 + healthy = False + failure_type = failure_type or type(exc).__name__ + continue + healthy = healthy and status.healthy + for field in counters: + counters[field] += max(0, int(getattr(status, field, 0))) + failure_type = failure_type or status.failure_type + return { + "state": state.value, + "healthy": healthy, + "failure_type": failure_type, + **counters, + } + + def prompt_route(self, worker: Worker) -> _PromptRoute | None: + try: + self._current_slot(worker) + except AcpCoordinatorError: + # A just-observed worker may not have reached the periodic pass. + try: + self._reconcile_worker(worker.id, strict=False) + self._current_slot(worker) + except Exception: # noqa: BLE001 + return None + return _PromptRoute(self, worker) + + def owns_worker(self, worker_id: str, worker_fingerprint: str) -> bool: + """Return whether a healthy ACP slot currently owns this exact worker.""" + with self._lock: + slot = self._slots.get(worker_id) + return bool( + slot is not None + and slot.continuity.worker_fingerprint == worker_fingerprint + and slot.runtime.status().healthy + ) + + def _current_slot(self, worker: Worker) -> _RuntimeSlot: + with self._lock: + if self._state is not RuntimeState.RUNNING: + raise AcpCoordinatorError("ACP coordinator is not running") + slot = self._slots.get(worker.id) + if slot is None: + raise AcpCoordinatorError("ACP worker route is unavailable") + if slot.continuity.worker_fingerprint != worker.fingerprint: + raise AcpCoordinatorError("ACP worker authority is stale") + if not slot.runtime.status().healthy: + raise AcpCoordinatorError("ACP worker runtime is unhealthy") + return slot + + def _run(self) -> None: + while not self._stop.wait(self._reconcile_interval): + if self._daemon_stop.is_set(): + return + try: + self._reconcile(strict=False) + except Exception as exc: # noqa: BLE001 + with self._lock: + self._failure_type = type(exc).__name__ + + def _continuity_bindings(self) -> tuple[dict[str, WorkerBinding], int]: + bindings = list_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="herdr", + ) + grouped: dict[str, list[WorkerBinding]] = {} + for binding in bindings: + if binding.sendable and binding.target_kind in { + "agent_id", + "agent", + "name", + "label", + "terminal_id", + "pane_id", + }: + grouped.setdefault(binding.worker_id, []).append(binding) + current = { + worker_id: rows[0] + for worker_id, rows in grouped.items() + if len(rows) == 1 + } + ambiguities = sum(1 for rows in grouped.values() if len(rows) != 1) + return current, ambiguities + + def _reconcile(self, *, strict: bool) -> None: + current, ambiguities = self._continuity_bindings() + with self._lock: + stale = [worker_id for worker_id in self._slots if worker_id not in current] + for worker_id in stale: + self._retire_worker(worker_id) + failures: list[BaseException] = [ + AcpCoordinatorError("worker has ambiguous Herdr authority") + for _ in range(ambiguities) + ] + for worker_id, continuity in current.items(): + try: + self._reconcile_binding(continuity) + except Exception as exc: # noqa: BLE001 + failures.append(exc) + self._retire_worker(worker_id) + with self._lock: + self._required_degraded = bool(failures) and ( + self.config.agent_event_source == "acp_required" + ) + if failures: + self._failure_type = type(failures[0]).__name__ + elif not self._required_degraded: + self._failure_type = None + if strict and failures: + raise AcpCoordinatorError("one or more ACP workers failed to attach") + + def _reconcile_worker(self, worker_id: str, *, strict: bool) -> None: + current, _ambiguities = self._continuity_bindings() + continuity = current.get(worker_id) + if continuity is None: + if strict: + raise AcpCoordinatorError("worker has no unique Herdr authority") + return + self._reconcile_binding(continuity) + + def _reconcile_binding(self, continuity: WorkerBinding) -> None: + with self._lock: + existing = self._slots.get(continuity.worker_id) + if ( + existing is not None + and _same_continuity(existing.continuity, continuity) + and existing.runtime.status().healthy + ): + try: + self._require_attached_generation(existing) + return + except Exception: + self._retire_worker(continuity.worker_id, expected=existing) + existing = None + if existing is not None: + self._retire_worker(continuity.worker_id, expected=existing) + endpoint = self._resolve_endpoint(continuity) + runtime = self._build_runtime(continuity, endpoint) + try: + runtime.start() + except Exception: + try: + runtime.stop(timeout=self.config.acp_shutdown_timeout_seconds) + except Exception: + pass + raise + slot = _RuntimeSlot(continuity, endpoint.generation, runtime) + with self._lock: + displaced = self._slots.get(continuity.worker_id) + self._slots[continuity.worker_id] = slot + if displaced is not None: + self._stop_runtime(displaced.runtime) + + def _resolve_endpoint(self, continuity: WorkerBinding) -> HerdrAcpEndpoint: + client = self._endpoint_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + result = client.agent_acp_endpoint( + continuity.target_value, + timeout=self.config.herdr_timeout_seconds, + ) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + return _parse_endpoint(self.config, continuity, result) + + def _resolve_status(self, continuity: WorkerBinding) -> HerdrAcpStatus: + client = self._endpoint_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + result = client.agent_acp_status( + continuity.target_value, + timeout=self.config.herdr_timeout_seconds, + ) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + return _parse_status(continuity, result) + + def _require_attached_generation(self, slot: _RuntimeSlot) -> None: + try: + status = self._resolve_status(slot.continuity) + except Exception: + self._retire_worker(slot.continuity.worker_id, expected=slot) + raise + if ( + status.lifecycle != "acp_owned_attached" + or status.generation != slot.generation + ): + self._retire_worker(slot.continuity.worker_id, expected=slot) + raise AcpCoordinatorError("ACP worker generation lease is not current") + + def _build_runtime( + self, + continuity: WorkerBinding, + endpoint: HerdrAcpEndpoint, + ) -> AcpRuntime: + client = self._client_factory( + endpoint.command, + cwd=endpoint.cwd, + request_timeout=self.config.acp_request_timeout_seconds, + prompt_timeout=float(self.config.submission_hard_ttl_seconds), + close_timeout=self.config.acp_shutdown_timeout_seconds, + max_frame_bytes=self.config.acp_max_frame_bytes, + ) + if endpoint.session_mode is SessionOpenMode.NEW: + binding = continuity + callback = self._bind_new_session + else: + assert endpoint.session_id is not None + binding = _derived_binding(continuity, endpoint.session_id) + upsert_worker_bindings(Path(self.config.db_path), [binding]) + callback = None + return self._runtime_factory( + client, + config=self.config, + binding=binding, + cwd=endpoint.cwd, + session_mode=endpoint.session_mode, + session_id=endpoint.session_id, + stream_generation=endpoint.generation, + session_binding_callback=callback, + poll_timeout=min(0.25, self.config.acp_request_timeout_seconds), + stop_timeout=self.config.acp_shutdown_timeout_seconds, + ) + + def _bind_new_session( + self, + session_id: str, + continuity: WorkerBinding, + ) -> WorkerBinding: + bound = _derived_binding(continuity, session_id) + upsert_worker_bindings(Path(self.config.db_path), [bound]) + return bound + + def _retire_worker( + self, + worker_id: str, + *, + expected: _RuntimeSlot | None = None, + ) -> None: + with self._lock: + slot = self._slots.get(worker_id) + if slot is None or (expected is not None and slot is not expected): + return + self._slots.pop(worker_id, None) + self._stop_runtime(slot.runtime) + + def _stop_runtime(self, runtime: AcpRuntime, *, timeout: float | None = None) -> None: + try: + runtime.stop( + timeout=( + self.config.acp_shutdown_timeout_seconds + if timeout is None + else timeout + ) + ) + except Exception: + pass + + def _stop_all(self, *, timeout: float | None = None) -> None: + with self._lock: + slots = tuple(self._slots.values()) + self._slots.clear() + if not slots: + return + total = ( + self.config.acp_shutdown_timeout_seconds + if timeout is None + else timeout + ) + deadline = time.monotonic() + total + for slot in slots: + self._stop_runtime( + slot.runtime, + timeout=max(0.001, deadline - time.monotonic()), + ) + + +def _default_endpoint_client_factory(config: Config) -> HerdrSocketClient: + return HerdrSocketClient(timeout=config.herdr_timeout_seconds) + + +def _derived_binding( + continuity: WorkerBinding, + session_id: str, +) -> WorkerBinding: + return replace( + continuity, + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="", + ) + + +def _same_continuity(left: WorkerBinding, right: WorkerBinding) -> bool: + """Compare authority identity while ignoring observation lease refreshes.""" + return ( + left.host_id, + left.worker_id, + left.worker_fingerprint, + left.backend, + left.target_kind, + left.target_value, + left.private_fingerprint, + left.sendable, + ) == ( + right.host_id, + right.worker_id, + right.worker_fingerprint, + right.backend, + right.target_kind, + right.target_value, + right.private_fingerprint, + right.sendable, + ) + + +def _nonempty_text(value: Any, field: str) -> str: + if not isinstance(value, str) or not value or value.strip() != value: + raise AcpCoordinatorError(f"Herdr ACP endpoint {field} is invalid") + if len(value) > 4096 or "\x00" in value: + raise AcpCoordinatorError(f"Herdr ACP endpoint {field} is invalid") + return value + + +def _parse_endpoint( + config: Config, + continuity: WorkerBinding, + value: Any, +) -> HerdrAcpEndpoint: + """Strictly validate the private, one-shot Herdr launch contract.""" + if not isinstance(value, Mapping) or set(value) != { + "type", + "endpoint", + "worker", + "adapter", + "session", + "cwd", + "lifecycle", + }: + raise AcpCoordinatorError("Herdr ACP endpoint response shape is invalid") + if ( + value.get("type") != "agent_acp_endpoint" + or value.get("lifecycle") != "acp_owned_ready" + ): + raise AcpCoordinatorError("Herdr ACP endpoint is not ready") + endpoint = value.get("endpoint") + worker = value.get("worker") + adapter = value.get("adapter") + session = value.get("session") + if not all(isinstance(item, Mapping) for item in (endpoint, worker, adapter, session)): + raise AcpCoordinatorError("Herdr ACP endpoint nested shape is invalid") + assert isinstance(endpoint, Mapping) + assert isinstance(worker, Mapping) + assert isinstance(adapter, Mapping) + assert isinstance(session, Mapping) + if set(endpoint) != {"transport", "command", "args", "protocol_version"}: + raise AcpCoordinatorError("Herdr ACP launch specification is invalid") + if ( + endpoint.get("transport") != "stdio" + or type(endpoint.get("protocol_version")) is not int + or endpoint.get("protocol_version") != 1 + ): + raise AcpCoordinatorError("Herdr ACP launch protocol is unsupported") + command = _nonempty_text(endpoint.get("command"), "command") + if command != "herdr" or Path(config.herdr_bin).name != "herdr": + raise AcpCoordinatorError("Herdr ACP endpoint executable is not configured Herdr") + raw_generation = worker.get("generation") + if ( + type(raw_generation) is not int + or raw_generation < 0 + or raw_generation > (1 << 64) - 1 + ): + raise AcpCoordinatorError("Herdr ACP endpoint generation is invalid") + generation = str(raw_generation) + args = endpoint.get("args") + if ( + not isinstance(args, list) + or len(args) != 7 + or args[0:2] != ["agent", "acp-attach"] + or args[3] != "--generation" + or args[5] != "--ticket" + ): + raise AcpCoordinatorError("Herdr ACP attach arguments are invalid") + target = _nonempty_text(args[2], "target") + if target != continuity.target_value: + raise AcpCoordinatorError("Herdr ACP endpoint target changed") + if _nonempty_text(args[4], "argument generation") != generation: + raise AcpCoordinatorError("Herdr ACP endpoint generation is inconsistent") + _nonempty_text(args[6], "ticket") + if set(worker) != { + "terminal_id", + "workspace_id", + "tab_id", + "pane_id", + "name", + "agent", + "generation", + }: + raise AcpCoordinatorError("Herdr ACP worker identity shape is invalid") + pane_id = _nonempty_text(worker.get("pane_id"), "pane_id") + if continuity.target_kind == "pane_id" and pane_id != continuity.target_value: + raise AcpCoordinatorError("Herdr ACP pane authority changed") + for field in ("terminal_id", "workspace_id", "tab_id", "name", "agent"): + _nonempty_text(worker.get(field), field) + if set(adapter) != {"name", "version"}: + raise AcpCoordinatorError("Herdr ACP adapter identity shape is invalid") + _nonempty_text(adapter.get("name"), "adapter name") + _nonempty_text(adapter.get("version"), "adapter version") + if set(session) not in ({"mode"}, {"id", "mode"}): + raise AcpCoordinatorError("Herdr ACP session shape is invalid") + try: + mode = SessionOpenMode(session.get("mode")) + except (TypeError, ValueError) as exc: + raise AcpCoordinatorError("Herdr ACP session mode is invalid") from exc + session_id_value = session.get("id") + if mode is SessionOpenMode.NEW: + if "id" in session and session_id_value is not None: + raise AcpCoordinatorError("new Herdr ACP endpoint already has a session") + session_id = None + else: + session_id = _nonempty_text(session_id_value, "session id") + cwd = Path(_nonempty_text(value.get("cwd"), "cwd")) + if not cwd.is_absolute(): + raise AcpCoordinatorError("Herdr ACP cwd must be absolute") + return HerdrAcpEndpoint( + command=( + config.herdr_bin, + *(_nonempty_text(item, "argument") for item in args), + ), + cwd=cwd, + generation=generation, + session_mode=mode, + session_id=session_id, + ) + + +def _parse_status( + continuity: WorkerBinding, + value: Any, +) -> HerdrAcpStatus: + """Validate Herdr's non-ticketing ACP generation/lease status.""" + if not isinstance(value, Mapping) or set(value) != { + "type", + "worker", + "adapter", + "session", + "cwd", + "lifecycle", + }: + raise AcpCoordinatorError("Herdr ACP status response shape is invalid") + lifecycle = value.get("lifecycle") + if value.get("type") != "agent_acp_status" or lifecycle not in { + "acp_owned_ready", + "acp_owned_attached", + }: + raise AcpCoordinatorError("Herdr ACP status lifecycle is invalid") + worker = value.get("worker") + adapter = value.get("adapter") + session = value.get("session") + if not all(isinstance(item, Mapping) for item in (worker, adapter, session)): + raise AcpCoordinatorError("Herdr ACP status nested shape is invalid") + assert isinstance(worker, Mapping) + assert isinstance(adapter, Mapping) + assert isinstance(session, Mapping) + if set(worker) != { + "terminal_id", + "workspace_id", + "tab_id", + "pane_id", + "name", + "agent", + "generation", + }: + raise AcpCoordinatorError("Herdr ACP status worker shape is invalid") + raw_generation = worker.get("generation") + if ( + type(raw_generation) is not int + or raw_generation < 0 + or raw_generation > (1 << 64) - 1 + ): + raise AcpCoordinatorError("Herdr ACP status generation is invalid") + for field in ( + "terminal_id", + "workspace_id", + "tab_id", + "pane_id", + "name", + "agent", + ): + _nonempty_text(worker.get(field), field) + if ( + continuity.target_kind == "pane_id" + and worker.get("pane_id") != continuity.target_value + ): + raise AcpCoordinatorError("Herdr ACP status pane authority changed") + if set(adapter) != {"name", "version"}: + raise AcpCoordinatorError("Herdr ACP status adapter shape is invalid") + _nonempty_text(adapter.get("name"), "adapter name") + _nonempty_text(adapter.get("version"), "adapter version") + if set(session) not in ({"mode"}, {"id", "mode"}): + raise AcpCoordinatorError("Herdr ACP status session shape is invalid") + try: + mode = SessionOpenMode(session.get("mode")) + except (TypeError, ValueError) as exc: + raise AcpCoordinatorError("Herdr ACP status session mode is invalid") from exc + if mode is SessionOpenMode.NEW: + if "id" in session and session.get("id") is not None: + raise AcpCoordinatorError("new Herdr ACP status already has a session") + else: + _nonempty_text(session.get("id"), "session id") + cwd = Path(_nonempty_text(value.get("cwd"), "cwd")) + if not cwd.is_absolute(): + raise AcpCoordinatorError("Herdr ACP status cwd must be absolute") + return HerdrAcpStatus(str(raw_generation), str(lifecycle)) + + +def production_acp_runtime_factory( + config: Config, + stop_event: threading.Event, +) -> AcpRuntimeCoordinator: + """Build the stock daemon's Herdr-backed multi-worker ACP coordinator.""" + return AcpRuntimeCoordinator(config, stop_event) diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 32e7bca..763b8a0 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -142,6 +142,7 @@ def prompt( prompt: str | Sequence[Mapping[str, Any]], *, timeout: float | None = None, + on_submitted: Callable[[], None] | None = None, ) -> PromptResult: ... def prepare_prompt( @@ -279,6 +280,7 @@ def __init__( ) self._close_thread: threading.Thread | None = None self._close_failures: list[BaseException] = [] + self._prompt_threads: set[threading.Thread] = set() self._updates_ingested = 0 self._permissions_ingested = 0 @@ -374,6 +376,7 @@ def prompt( producer_turn_id: str | None = None, timeout: float | None = None, drain_timeout: float | None = None, + on_submitted: Callable[[], None] | None = None, ) -> PromptResult: """Submit one prompt and finalize only after its prior updates drain.""" @@ -402,10 +405,13 @@ def prompt( self._record_failure(exc) raise try: + prompt_kwargs: dict[str, Any] = {"timeout": timeout} + if on_submitted is not None: + prompt_kwargs["on_submitted"] = on_submitted result = self._client.prompt( session_id, prepared_prompt, - timeout=timeout, + **prompt_kwargs, ) except BaseException as exc: with self._state_lock: @@ -447,6 +453,68 @@ def prompt( self._prompts_completed += 1 return result + def submit_prompt( + self, + prompt: str | Sequence[Mapping[str, Any]], + *, + producer_turn_id: str, + acknowledgement_timeout: float, + completion_timeout: float | None = None, + ) -> None: + """Start a prompt and return after its complete frame is written. + + End-of-turn completion continues under runtime supervision. A caller + can therefore durably acknowledge delivery without blocking for the + agent's entire turn. If acknowledgement is not observed, delivery is + uncertain and the command layer must never retry it automatically. + """ + + if acknowledgement_timeout <= 0: + raise ValueError("acknowledgement_timeout must be positive") + acknowledged = threading.Event() + finished = threading.Event() + failures: list[BaseException] = [] + + def run_prompt() -> None: + try: + self.prompt( + prompt, + producer_turn_id=producer_turn_id, + timeout=completion_timeout, + on_submitted=acknowledged.set, + ) + except BaseException as exc: + failures.append(exc) + finally: + finished.set() + with self._state_lock: + self._prompt_threads.discard(threading.current_thread()) + + thread = threading.Thread( + target=run_prompt, + name="tendwire-acp-prompt", + daemon=True, + ) + with self._state_lock: + self._prompt_threads.add(thread) + thread.start() + deadline = time.monotonic() + acknowledgement_timeout + while True: + if acknowledged.is_set(): + return + if finished.is_set(): + if failures: + raise failures[0] + raise AcpRuntimeProtocolError( + "ACP prompt completed without a submission acknowledgement" + ) + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AcpRuntimeStopTimeout( + "ACP prompt submission acknowledgement timed out" + ) + acknowledged.wait(min(remaining, 0.01)) + def cancel(self) -> None: """Cancel the active session and any permission requests pending in it.""" @@ -494,7 +562,9 @@ def join(self, timeout: float | None = None) -> bool: if wait_limit <= 0: raise ValueError("join timeout must be positive") deadline = time.monotonic() + wait_limit - for thread in self._threads: + with self._state_lock: + threads = (*self._threads, *self._prompt_threads) + for thread in threads: if thread is threading.current_thread() or thread.ident is None: continue thread.join(timeout=max(0.0, deadline - time.monotonic())) @@ -502,7 +572,7 @@ def join(self, timeout: float | None = None) -> bool: thread is threading.current_thread() or thread.ident is None or not thread.is_alive() - for thread in self._threads + for thread in threads ) def stop(self, *, timeout: float | None = None) -> None: diff --git a/src/tendwire/backends/herdr_socket.py b/src/tendwire/backends/herdr_socket.py index 1ed1b51..5c44c45 100644 --- a/src/tendwire/backends/herdr_socket.py +++ b/src/tendwire/backends/herdr_socket.py @@ -300,6 +300,32 @@ def agent_send( ) -> Any: return self.request("agent.send", params, timeout=timeout) + def agent_acp_endpoint( + self, + target: str, + *, + timeout: float | None = None, + ) -> Any: + """Mint a one-shot, private ACP attach endpoint for one live agent.""" + return self.request( + "agent.acp_endpoint", + {"target": target}, + timeout=timeout, + ) + + def agent_acp_status( + self, + target: str, + *, + timeout: float | None = None, + ) -> Any: + """Read ACP ownership/generation without minting an attach ticket.""" + return self.request( + "agent.acp_status", + {"target": target}, + timeout=timeout, + ) + def _send_request( self, method: str, diff --git a/src/tendwire/backends/herdr_turns.py b/src/tendwire/backends/herdr_turns.py index 268a1d5..570f788 100644 --- a/src/tendwire/backends/herdr_turns.py +++ b/src/tendwire/backends/herdr_turns.py @@ -4429,6 +4429,28 @@ def __init__( TurnRefreshKey, tuple[_TurnRefreshItem, float], ] = {} + self._worker_excluded: Callable[[str, str], bool] | None = None + + def set_worker_exclusion( + self, + callback: Callable[[str, str], bool] | None, + ) -> None: + """Exclude workers currently owned by a stronger semantic source.""" + with self._condition: + self._worker_excluded = callback + self._rescan_requested = True + self._condition.notify_all() + + def _is_worker_excluded(self, binding: WorkerBinding) -> bool: + callback = self._worker_excluded + if callback is None: + return False + try: + return callback(binding.worker_id, binding.worker_fingerprint) is True + except Exception: + # Authority uncertainty must not let the legacy reader overwrite + # a potentially ACP-owned worker. + return True def start(self) -> None: with self._condition: @@ -4517,7 +4539,7 @@ def _scan_bindings(self) -> None: items = [ _TurnRefreshItem.from_binding(binding) for binding in bindings - if _eligible_turn_binding(binding) + if _eligible_turn_binding(binding) and not self._is_worker_excluded(binding) ] with self._condition: self._scan_failed = False @@ -4586,6 +4608,8 @@ def _run_item(self, item: _TurnRefreshItem) -> TurnRefreshResult: return TurnRefreshResult("failed", 0, retry_binding_lookup=True) if binding is None: return TurnRefreshResult("stale_binding", 0) + if self._is_worker_excluded(binding): + return TurnRefreshResult("stale_binding", 0) try: if self._uses_default_reader: result = _refresh_turn_binding( diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index ebdf472..c98d24c 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -6,7 +6,7 @@ from collections.abc import Callable, Mapping from dataclasses import dataclass from enum import Enum -from typing import Any +from typing import Any, Protocol from .config import Config from .core.actions import CommandContext, execute_command @@ -106,6 +106,28 @@ SocketClientFactory = Callable[[Config], Any] +class AcpPromptRoute(Protocol): + """One live, authority-checked ACP prompt route owned by the daemon. + + The route deliberately exposes neither adapter argv nor session identity. + Its binding fingerprint is private durable evidence used only by the + command receipt state machine. + """ + + binding_fingerprint: str + + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + ) -> object: ... + + +AcpPromptRouter = Callable[[Worker], AcpPromptRoute | None] + + @dataclass(frozen=True) class ResolvedCommandTarget: worker: Worker @@ -2608,6 +2630,186 @@ def replay_command_receipt( ) +def submit_acp_command( + config: Config, + params: Mapping[str, Any] | str, + *, + prompt_router: AcpPromptRouter, + required: bool = False, +) -> CommandEnvelope | None: + """Submit ``send_instruction`` through a live ACP worker route. + + ``None`` means the ACP path made no durable change and an optional policy + may safely use the legacy Herdr sender. Once a receipt reaches + ``send_started``, every failure is terminally uncertain and this function + never permits a second transport attempt. + """ + + payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) + request, parse_error = parse_command_request(payload) + if parse_error is not None or request is None: + return None + if validate_request(request) is not None: + return None + if request.dry_run: + return None + if request.action != "send_instruction": + return ( + _backend_unavailable( + request, + "command is not supported by the required ACP control path", + ) + if required and request.action in _MUTATING_ACTIONS + else None + ) + + existing_receipt: Mapping[str, Any] | None = None + if config.db_path is not None: + try: + candidate = get_command_request( + config.db_path, + config.host_id, + request.request_id or "", + ) + except Exception: + candidate = None + if isinstance(candidate, Mapping): + existing_receipt = candidate + + takeover: _ReceiptTakeover | None = None + if existing_receipt is not None: + decided = _receipt_authority(config, request, existing_receipt) + if isinstance(decided, CommandEnvelope): + return _negotiated_submission_envelope(config, request, decided) + takeover = decided + + try: + snapshot = _current_snapshot(config) + except Exception: # noqa: BLE001 + if takeover is not None: + return _request_in_progress(request) + return ( + _backend_unavailable( + request, + "Current worker authority is temporarily unavailable", + ) + if required + else None + ) + + health_error = _backend_health_error(config, request, snapshot) + worker = _resolve_authoritative_worker(request, snapshot) + if isinstance(worker, CommandEnvelope): + if takeover is not None: + return _request_in_progress(request) + if health_error is not None: + return health_error if required else None + return worker + if takeover is not None and worker.id != takeover.public_worker_id: + return _duplicate_request(request) + + permanent_error = _worker_status_error(request, worker) or health_error + if permanent_error is not None: + if required: + canonical = build_canonical_mutation(request, public_worker_id=worker.id) + reservation = _reserve_canonical_request(config, request, canonical) + if isinstance(reservation, CommandEnvelope): + return reservation + return _finish_before_send(config, request, reservation, permanent_error) + return None + + try: + route = prompt_router(worker) + except Exception: # noqa: BLE001 + route = None + if route is None: + if takeover is not None: + return _request_in_progress(request) + return ( + _backend_unavailable(request, "ACP worker route is unavailable") + if required + else None + ) + binding_fingerprint = str( + getattr(route, "binding_fingerprint", "") or "" + ).strip() + if not binding_fingerprint: + if takeover is not None: + return _request_in_progress(request) + return ( + _backend_unavailable(request, "ACP worker route has no durable authority") + if required + else None + ) + + canonical = build_canonical_mutation(request, public_worker_id=worker.id) + reservation = _reserve_canonical_request(config, request, canonical) + if isinstance(reservation, CommandEnvelope): + return reservation + send_started = _mark_request_send_started( + config, + request, + reservation, + binding_fingerprint=binding_fingerprint, + worker=worker, + instruction_text=_instruction_text(request), + ) + if isinstance(send_started, CommandEnvelope): + return send_started + if not isinstance(send_started, Mapping): + return _recover_request(config, request, reservation.canonical) + + try: + route.prompt( + _instruction_text(request), + producer_turn_id=turn_submission_id( + config.host_id, + request.request_id or "", + ), + timeout=config.acp_request_timeout_seconds, + ) + except Exception: # noqa: BLE001 + return _finish_request( + config, + request, + reservation, + _instruction_uncertain_envelope( + request, + worker, + verdict="unknown", + ), + expected_state="send_started", + terminal_state="uncertain", + ) + + observed_turn: Mapping[str, Any] | None = send_started + if config.db_path is not None: + try: + refreshed = linked_turn_for_submission( + config.db_path, + host_id=config.host_id, + request_id=request.request_id or "", + ) + except Exception: # noqa: BLE001 + refreshed = None + if isinstance(refreshed, Mapping): + observed_turn = refreshed + accepted = _accepted_send_envelope( + request, + worker, + observed_turn, + submission_verdict="submitted", + ) + return _finish_request( + config, + request, + reservation, + accepted, + expected_state="send_started", + terminal_state="accepted", + ) + + def _submit_command_v2( config: Config, params: Mapping[str, Any] | str, @@ -2889,8 +3091,31 @@ def submit_command( params: Mapping[str, Any] | str, *, socket_client_factory: SocketClientFactory | None = None, + acp_prompt_router: AcpPromptRouter | None = None, + acp_required: bool = False, ) -> CommandEnvelope: """Submit one command and apply optional response-envelope negotiation.""" + if acp_prompt_router is not None: + acp_envelope = submit_acp_command( + config, + params, + prompt_router=acp_prompt_router, + required=acp_required, + ) + if acp_envelope is not None: + payload = ( + params + if isinstance(params, str) + else _raw_payload_from_mapping(params) + ) + request, parse_error = parse_command_request(payload) + if parse_error is None and request is not None: + return _negotiated_submission_envelope( + config, + request, + acp_envelope, + ) + return acp_envelope envelope = _submit_command_v2( config, params, diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index b798ab0..b607c40 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -517,6 +517,12 @@ def _default_turn_scheduler_factory(config: Config) -> Any: return TurnIngestionScheduler(config) +def _default_acp_runtime_factory(config: Config, stop_event: threading.Event) -> Any: + from .backends.acp_coordinator import production_acp_runtime_factory + + return production_acp_runtime_factory(config, stop_event) + + @dataclass(frozen=True) class DaemonHooks: """Dependency injection points for deterministic daemon tests.""" @@ -526,7 +532,9 @@ class DaemonHooks: submit_command: Callable[[Config, str], CommandEnvelope | Mapping[str, Any]] = _default_submit_command event_backend_factory: Callable[[Config, threading.Event], Any] | None = None turn_scheduler_factory: Callable[[Config], Any] = _default_turn_scheduler_factory - acp_runtime_factory: Callable[[Config, threading.Event], Any | None] | None = None + acp_runtime_factory: Callable[[Config, threading.Event], Any | None] | None = ( + _default_acp_runtime_factory + ) class TendwireDaemon: @@ -608,6 +616,11 @@ def start(self) -> None: if self.config.agent_event_source != "acp_required": scheduler = self.hooks.turn_scheduler_factory(self.config) self._turn_scheduler = scheduler + if self.config.agent_event_source == "acp_preferred": + owns_worker = getattr(self._acp_runtime, "owns_worker", None) + set_exclusion = getattr(scheduler, "set_worker_exclusion", None) + if callable(owns_worker) and callable(set_exclusion): + set_exclusion(owns_worker) api = TendwireDaemonAPI( get_snapshot=self.get_snapshot, @@ -1433,6 +1446,22 @@ def submit_command(self, params: Mapping[str, Any]) -> CommandEnvelope | Mapping sort_keys=True, separators=(",", ":"), ) + policy = self.config.agent_event_source + runtime = self._acp_runtime + route = getattr(runtime, "prompt_route", None) + if policy == "acp_required" or ( + policy == "acp_preferred" and callable(route) + ): + from .command_submission import submit_command + + return submit_command( + self.config, + payload, + acp_prompt_router=( + route if callable(route) else lambda _worker: None + ), + acp_required=policy == "acp_required", + ) return self.hooks.submit_command(self.config, payload) diff --git a/tests/fixtures/herdr_acp_contract_v1.json b/tests/fixtures/herdr_acp_contract_v1.json new file mode 100644 index 0000000..1aa2627 --- /dev/null +++ b/tests/fixtures/herdr_acp_contract_v1.json @@ -0,0 +1,37 @@ +{ + "request": { + "id": "tw:acp:endpoint", + "method": "agent.acp_endpoint", + "params": {"target": "term_abc"} + }, + "result": { + "type": "agent_acp_endpoint", + "endpoint": { + "transport": "stdio", + "command": "herdr", + "args": [ + "agent", + "acp-attach", + "term_abc", + "--generation", + "42", + "--ticket", + "1234567890123456789012345678901234567890123" + ], + "protocol_version": 1 + }, + "worker": { + "terminal_id": "term_abc", + "workspace_id": "w1", + "tab_id": "w1:t1", + "pane_id": "w1:p1", + "name": "reviewer", + "agent": "codex", + "generation": 42 + }, + "adapter": {"name": "codex-acp", "version": "0.13.3"}, + "session": {"mode": "new"}, + "cwd": "/work", + "lifecycle": "acp_owned_ready" + } +} diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py new file mode 100644 index 0000000..a9e6a5d --- /dev/null +++ b/tests/test_acp_coordinator.py @@ -0,0 +1,465 @@ +"""Production ACP coordinator and command-path contract tests.""" + +from __future__ import annotations + +import json +import threading +from concurrent.futures import ThreadPoolExecutor +from dataclasses import replace +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest + +from tendwire.backends.acp_coordinator import ( + AcpCoordinatorError, + AcpRuntimeCoordinator, + _parse_endpoint, +) +from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult +from tendwire.command_submission import submit_command +from tendwire.config import Config +from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding +from tendwire.store.sqlite import ( + get_command_request, + init_store, + save_snapshot, + upsert_worker_bindings, +) + + +def _config(tmp_path: Path, *, policy: str = "acp_preferred") -> Config: + return Config( + host_id="acp-host", + data_dir=tmp_path, + db_path=tmp_path / "tendwire.db", + herdr_backend="socket", + herdr_bin="herdr", + agent_event_source=policy, + ) + + +def _binding() -> WorkerBinding: + return WorkerBinding( + host_id="acp-host", + worker_id="worker-1", + worker_fingerprint="worker-fingerprint", + backend="herdr", + target_kind="pane_id", + target_value="pane-private", + turn_target_kind="pane_id", + turn_target_value="pane-private", + sendable=True, + private_fingerprint="herdr-private-binding", + ) + + +def _endpoint(*, generation: int = 42, lifecycle: str = "acp_owned_ready") -> dict[str, Any]: + return { + "type": "agent_acp_endpoint", + "endpoint": { + "transport": "stdio", + "command": "herdr", + "args": [ + "agent", + "acp-attach", + "pane-private", + "--generation", + str(generation), + "--ticket", + "one-shot-private-ticket", + ], + "protocol_version": 1, + }, + "worker": { + "terminal_id": "pane-private", + "workspace_id": "workspace-private", + "tab_id": "tab-private", + "pane_id": "pane-private", + "name": "agent-name", + "agent": "codex", + "generation": generation, + }, + "adapter": {"name": "codex-acp", "version": "1.2.3"}, + "session": {"mode": "new"}, + "cwd": "/tmp/project", + "lifecycle": lifecycle, + } + + +def _status(*, generation: int = 42, lifecycle: str = "acp_owned_attached") -> dict[str, Any]: + endpoint = _endpoint(generation=generation) + return { + "type": "agent_acp_status", + "worker": endpoint["worker"], + "adapter": endpoint["adapter"], + "session": endpoint["session"], + "cwd": endpoint["cwd"], + "lifecycle": lifecycle, + } + + +def test_endpoint_requires_explicit_acp_ownership_and_strict_attach_shape(tmp_path: Path) -> None: + config = _config(tmp_path) + parsed = _parse_endpoint(config, _binding(), _endpoint()) + assert parsed.generation == "42" + assert parsed.command[0:3] == ("herdr", "agent", "acp-attach") + + with pytest.raises(AcpCoordinatorError, match="not ready"): + _parse_endpoint(config, _binding(), _endpoint(lifecycle="ready")) + injected = _endpoint() + injected["endpoint"]["command"] = "/tmp/evil" + with pytest.raises(AcpCoordinatorError, match="configured Herdr"): + _parse_endpoint(config, _binding(), injected) + replayed = _endpoint(generation=43) + replayed["endpoint"]["args"][4] = "42" + with pytest.raises(AcpCoordinatorError, match="inconsistent"): + _parse_endpoint(config, _binding(), replayed) + + +def test_canonical_herdr_acp_contract_fixture_executes_configured_binary( + tmp_path: Path, +) -> None: + fixture = json.loads( + (Path(__file__).parent / "fixtures" / "herdr_acp_contract_v1.json").read_text( + encoding="utf-8" + ) + ) + assert fixture["request"] == { + "id": "tw:acp:endpoint", + "method": "agent.acp_endpoint", + "params": {"target": "term_abc"}, + } + config = replace(_config(tmp_path), herdr_bin="/opt/herdr/bin/herdr") + continuity = replace( + _binding(), + target_kind="terminal_id", + target_value="term_abc", + turn_target_kind="terminal_id", + turn_target_value="term_abc", + private_fingerprint="fixture-binding", + ) + parsed = _parse_endpoint(config, continuity, fixture["result"]) + assert parsed.command[0] == "/opt/herdr/bin/herdr" + assert parsed.command[1:] == tuple(fixture["result"]["endpoint"]["args"]) + assert parsed.generation == "42" + + +class _Route: + binding_fingerprint = "acp-private-binding" + + def __init__(self, failure: BaseException | None = None) -> None: + self.failure = failure + self.calls: list[tuple[str, str, float]] = [] + + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + ) -> None: + self.calls.append((text, producer_turn_id, timeout)) + if self.failure is not None: + raise self.failure + + +def _seed(config: Config) -> Worker: + assert config.db_path is not None + init_store(config.db_path) + worker = Worker( + id="worker-1", + name="worker", + status="idle", + fingerprint="worker-fingerprint", + ) + save_snapshot( + config.db_path, + Snapshot( + host_id=config.host_id, + updated_at="2026-07-31T00:00:00+00:00", + workers=[worker], + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty", + ) + ], + ), + ) + upsert_worker_bindings(config.db_path, [_binding()]) + return worker + + +def _request(request_id: str = "request-1") -> dict[str, Any]: + return { + "schema_version": 1, + "action": "send_instruction", + "request_id": request_id, + "dry_run": False, + "target": {"worker_id": "worker-1"}, + "instruction": {"text": "do the work"}, + } + + +def test_acp_command_uses_durable_receipt_and_duplicate_does_not_resend(tmp_path: Path) -> None: + config = _config(tmp_path) + worker = _seed(config) + route = _Route() + first = submit_command( + config, + _request(), + acp_prompt_router=lambda routed: route if routed == worker else None, + ) + second = submit_command( + config, + _request(), + acp_prompt_router=lambda _worker: route, + ) + assert first.status == "accepted" + assert second.status == "accepted" + assert len(route.calls) == 1 + receipt = get_command_request(config.db_path, config.host_id, "request-1") + assert receipt is not None and receipt["state"] == "accepted" + assert "acp-private-binding" not in json.dumps(first.to_dict()) + + +def test_acp_failure_after_send_started_is_uncertain_and_never_falls_back(tmp_path: Path) -> None: + config = _config(tmp_path) + _seed(config) + route = _Route(RuntimeError("private ticket and argv")) + legacy_calls: list[str] = [] + envelope = submit_command( + config, + _request("request-uncertain"), + acp_prompt_router=lambda _worker: route, + ) + assert envelope.status == "request_state_uncertain" + assert envelope.disposition == "terminal_uncertain" + assert len(route.calls) == 1 + assert "private ticket" not in json.dumps(envelope.to_dict()) + receipt = get_command_request( + config.db_path, + config.host_id, + "request-uncertain", + ) + assert receipt is not None and receipt["state"] == "uncertain" + + +def test_concurrent_duplicate_acp_command_has_one_external_send(tmp_path: Path) -> None: + config = _config(tmp_path) + _seed(config) + entered = threading.Event() + release = threading.Event() + + class BlockingRoute(_Route): + def prompt(self, text: str, *, producer_turn_id: str, timeout: float) -> None: + self.calls.append((text, producer_turn_id, timeout)) + entered.set() + assert release.wait(1.0) + + route = BlockingRoute() + with ThreadPoolExecutor(max_workers=2) as pool: + first = pool.submit( + submit_command, + config, + _request("request-concurrent"), + acp_prompt_router=lambda _worker: route, + ) + assert entered.wait(1.0) + second = pool.submit( + submit_command, + config, + _request("request-concurrent"), + acp_prompt_router=lambda _worker: route, + ) + second_result = second.result(timeout=1.0) + release.set() + first_result = first.result(timeout=1.0) + assert first_result.status == "accepted" + assert second_result.status == "pending" + assert len(route.calls) == 1 + + +def test_required_has_no_legacy_fallback_when_route_is_absent(tmp_path: Path) -> None: + config = _config(tmp_path, policy="acp_required") + _seed(config) + envelope = submit_command( + config, + _request("request-required"), + acp_prompt_router=lambda _worker: None, + acp_required=True, + ) + assert envelope.status == "backend_unavailable" + assert get_command_request( + config.db_path, + config.host_id, + "request-required", + ) is None + + +def test_new_and_resumed_endpoint_session_invariants(tmp_path: Path) -> None: + config = _config(tmp_path) + invalid_new = _endpoint() + invalid_new["session"] = {"mode": "new", "id": "already-bound"} + with pytest.raises(AcpCoordinatorError, match="already has"): + _parse_endpoint(config, _binding(), invalid_new) + resumed = _endpoint() + resumed["session"] = {"mode": "resume", "id": "session-private"} + assert _parse_endpoint(config, _binding(), resumed).session_id == "session-private" + missing = _endpoint() + missing["session"] = {"mode": "resume"} + with pytest.raises(AcpCoordinatorError, match="session id"): + _parse_endpoint(config, _binding(), missing) + + +def test_reconnect_remints_endpoint_instead_of_replaying_attach_ticket(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + minted: list[str] = [] + generation = [42] + runtimes: list[Any] = [] + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + value = _endpoint(generation=generation[0]) + ticket = f"one-shot-private-ticket-{len(minted) + 1}" + value["endpoint"]["args"][6] = ticket + minted.append(ticket) + return value + + def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: + return _status(generation=generation[0]) + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self.stopped = False + self.prompt_calls = 0 + runtimes.append(self) + + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + self.stopped = True + + def status(self) -> Any: + return SimpleNamespace( + healthy=not self.stopped, + failure_type=None, + updates_ingested=0, + permissions_ingested=0, + permissions_selected=0, + permissions_cancelled=0, + invalid_permission_selections=0, + prompts_started=0, + prompts_completed=0, + prompts_failed=0, + cancellation_requests=0, + ) + + def submit_prompt(self, *_args: Any, **_kwargs: Any) -> None: + self.prompt_calls += 1 + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + client_factory=lambda *_args, **_kwargs: object(), + runtime_factory=Runtime, + reconcile_interval=60.0, + ).start() + try: + assert minted == ["one-shot-private-ticket-1"] + coordinator._reconcile_worker("worker-1", strict=True) + worker = Worker( + id="worker-1", + name="worker", + status="idle", + fingerprint="worker-fingerprint", + ) + route = coordinator.prompt_route(worker) + assert route is not None + route.prompt("hello", producer_turn_id="producer", timeout=1.0) + assert minted == ["one-shot-private-ticket-1"] + assert runtimes[0].prompt_calls == 1 + generation[0] = 43 + with pytest.raises(AcpCoordinatorError, match="generation lease"): + route.prompt("stale", producer_turn_id="producer-2", timeout=1.0) + assert runtimes[0].prompt_calls == 1 + coordinator._reconcile_worker("worker-1", strict=True) + assert minted == [ + "one-shot-private-ticket-1", + "one-shot-private-ticket-2", + ] + finally: + coordinator.stop() + + +def test_preferred_legacy_scheduler_excludes_acp_owned_worker(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + read = threading.Event() + + def reader(*_args: Any, **_kwargs: Any) -> TurnRefreshResult: + read.set() + return TurnRefreshResult("updated", 1) + + scheduler = TurnIngestionScheduler( + config, + refresh_interval_seconds=0.05, + max_workers=1, + reader=reader, + ) + scheduler.set_worker_exclusion( + lambda worker_id, fingerprint: ( + worker_id == "worker-1" and fingerprint == "worker-fingerprint" + ) + ) + scheduler.start() + try: + assert not read.wait(0.2) + finally: + scheduler.stop(flush_timeout_seconds=1.0) + + +def test_required_zero_workers_is_idle_healthy_then_new_unowned_worker_degrades( + tmp_path: Path, +) -> None: + config = _config(tmp_path, policy="acp_required") + assert config.db_path is not None + init_store(config.db_path) + + class RejectingClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + raise AcpCoordinatorError("ACP ownership required") + + def close(self) -> None: + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: RejectingClient(), + reconcile_interval=60.0, + ).start() + try: + assert coordinator.status()["healthy"] is True + upsert_worker_bindings(config.db_path, [_binding()]) + coordinator._reconcile(strict=False) + health = coordinator.status() + assert health["healthy"] is False + assert health["failure_type"] == "AcpCoordinatorError" + finally: + coordinator.stop() diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index 02ff784..6a65756 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -96,6 +96,9 @@ def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: self.calls.append(("prompt", (session_id, prompt), kwargs)) if self.prompt_failure is not None: raise self.prompt_failure + on_submitted = kwargs.get("on_submitted") + if callable(on_submitted): + on_submitted() return self.prompt_result def prepare_prompt(self, prompt: object) -> tuple[dict[str, Any], ...]: @@ -1182,6 +1185,35 @@ def test_prompt_requires_crash_stable_producer_identity_before_remote_send( service.stop() +def test_submit_prompt_acknowledges_frame_before_end_of_turn(tmp_path: Path) -> None: + class BlockingPromptClient(FakeClient): + def __init__(self) -> None: + super().__init__() + self.release = threading.Event() + + def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: + self.calls.append(("prompt", (session_id, prompt), kwargs)) + on_submitted = kwargs.get("on_submitted") + assert callable(on_submitted) + on_submitted() + assert self.release.wait(1.0) + return self.prompt_result + + client = BlockingPromptClient() + service = runtime(tmp_path, client).start() + started = time.monotonic() + service.submit_prompt( + "long turn", + producer_turn_id="producer-turn-ack", + acknowledgement_timeout=0.25, + ) + assert time.monotonic() - started < 0.2 + assert service.status().prompts_completed == 0 + client.release.set() + wait_until(lambda: service.status().prompts_completed == 1) + service.stop() + + def test_ignored_update_does_not_increment_persisted_counter(tmp_path: Path) -> None: client = FakeClient() ingestor = FakeIngestor() From c8de8180803583a8f36040d8d67df0221022a594 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 11:17:12 +0800 Subject: [PATCH 36/83] fix(acp): fence production runtime lifecycle --- README.md | 6 + docs/acp-migration.md | 25 +- src/tendwire/backends/acp_client.py | 10 + src/tendwire/backends/acp_coordinator.py | 311 ++++++++++++++--- src/tendwire/command_submission.py | 9 +- tests/test_acp_client.py | 14 + tests/test_acp_coordinator.py | 422 ++++++++++++++++++++++- 7 files changed, 740 insertions(+), 57 deletions(-) diff --git a/README.md b/README.md index 7f493a7..c5f808a 100644 --- a/README.md +++ b/README.md @@ -573,6 +573,12 @@ same public-safe degraded evidence. The stock daemon currently defaults to `legacy`. ACP modes use Herdr's private `agent.acp_endpoint` contract and accept only workers explicitly marked `acp_owned_ready`; ordinary PTY workers are never attached as sidecars. +Production ACP startup is currently fail-closed with +`AcpPermissionBridgeUnavailable`: Tendwire does not yet have a durable, +worker/session-correlated bridge from `answer_decision` to the exact synchronous +ACP permission request. The daemon must not silently auto-cancel tool +permissions while reporting ACP healthy. The generic coordinator callback is +an embedding/test hook, not a production authorization path. `acp_shadow` persists ACP events without projecting them, but no automated shadow comparator is implemented. `acp_preferred` falls back only before an ACP reservation/send, while `acp_required` fails closed and never starts the legacy diff --git a/docs/acp-migration.md b/docs/acp-migration.md index c5a825e..d3ccb4e 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -2,8 +2,10 @@ This document defines the experimental migration from backend-specific transcript readers to Agent Client Protocol (ACP). ACP is not yet Tendwire's -default. The stock daemon now contains the production coordinator and command -path, but activates them only for an explicitly Herdr-owned ACP worker. +default. The stock daemon contains the coordinator and command path, but +production ACP activation is currently fail-closed until a durable permission +decision bridge is configured. The coordinator can otherwise attach only an +explicitly Herdr-owned ACP worker. Herdr remains authoritative for workspace, pane, worker identity, process liveness, and command routing until the ACP control path is proven separately. Tendwire remains authoritative for persistence, reconciliation, public safety, @@ -31,6 +33,15 @@ one-shot private endpoint, validates its worker generation and explicit `acp_owned_ready` lifecycle, then creates/loads/resumes the ACP session. An ordinary live PTY session is never treated as ACP-owned. +ACP `session/request_permission` is synchronous and can authorize destructive +tools. Tendwire does not yet have the required durable worker/session-correlated +bridge from a public `answer_decision` command back to the exact request and +offered `optionId`. The stock production factory therefore raises the redacted +`AcpPermissionBridgeUnavailable` startup failure for every ACP mode. It must not +silently cancel permissions while reporting the runtime healthy. Tests and +embedders may inject an explicit callback into the generic coordinator, but +that callback is not a production authorization surface. + ## Authority split | Concern | Authority | @@ -208,7 +219,9 @@ must pass before `acp_required` is considered: - fallback after adapter failure without regressing existing final delivery; - exact worker continuity across Herdr pane moves and agent-session recreation. -The ACP runtime implements prompt submission, cancellation, permission handling, -per-worker coordination, reconnect, and receipt-backed command routing. ACP -remains non-default until the cross-repository integration and rollout gates -above pass against real adapters. +The ACP runtime implements prompt submission, cancellation, fail-closed +permission handling, per-worker coordination, reconnect, and receipt-backed +instruction routing. Interactive permission approval remains a runtime blocker: +until its durable bridge exists, the stock production factory refuses ACP +startup. ACP also remains non-default until the cross-repository integration +and rollout gates above pass against real adapters. diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index a5e1244..c14a0aa 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -504,6 +504,16 @@ def load_session( ) params["sessionId"] = _nonempty(session_id, "session_id") result = self.request("session/load", params, timeout=timeout) + # ACP v1 defines the successful load response as null after all replay + # updates have been sent. Accept a mapping as a compatibility + # extension for agents that also return initial session state. + if result is None: + return SessionResult( + session_id, + None, + (), + MappingProxyType({}), + ) raw = _require_mapping(result, "session/load result") parsed = _parse_session_result(raw, require_session_id=False) return SessionResult(session_id, parsed.modes, parsed.config_options, parsed.raw) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 00b7234..9dbaa21 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -17,10 +17,19 @@ from typing import Any from ..config import Config -from ..core.models import Worker, WorkerBinding -from ..store.sqlite import list_worker_bindings, upsert_worker_bindings +from ..core.models import Worker, WorkerBinding, utc_timestamp +from ..store.sqlite import ( + expire_worker_bindings, + list_worker_bindings, + upsert_worker_bindings, +) from .acp_client import AcpClient -from .acp_runtime import AcpRuntime, RuntimeState, SessionOpenMode +from .acp_runtime import ( + AcpRuntime, + PermissionCallback, + RuntimeState, + SessionOpenMode, +) from .herdr_socket import HerdrSocketClient @@ -28,6 +37,10 @@ class AcpCoordinatorError(RuntimeError): """The private Herdr ACP endpoint contract or supervisor failed.""" +class AcpPermissionBridgeUnavailable(AcpCoordinatorError): + """Production ACP cannot authorize tools without a durable user decision.""" + + @dataclass(frozen=True, slots=True) class HerdrAcpEndpoint: command: tuple[str, ...] @@ -51,19 +64,19 @@ class _RuntimeSlot: class _PromptRoute: - def __init__(self, owner: "AcpRuntimeCoordinator", worker: Worker) -> None: + def __init__( + self, + owner: "AcpRuntimeCoordinator", + worker: Worker, + slot: _RuntimeSlot, + ) -> None: self._owner = owner self._worker = worker + self._slot = slot @property def binding_fingerprint(self) -> str: - slot = self._owner._current_slot(self._worker) - binding = getattr(slot.runtime, "_binding", None) - return ( - str(binding.private_fingerprint) - if isinstance(binding, WorkerBinding) - else "" - ) + return self._owner._route_binding_fingerprint(self._worker, self._slot) def prompt( self, @@ -72,9 +85,9 @@ def prompt( producer_turn_id: str, timeout: float, ) -> object: - slot = self._owner._current_slot(self._worker) - self._owner._require_attached_generation(slot) - return slot.runtime.submit_prompt( + return self._owner._submit_prompt( + self._worker, + self._slot, text, producer_turn_id=producer_turn_id, acknowledgement_timeout=timeout, @@ -98,6 +111,8 @@ def __init__( runtime_factory: RuntimeFactory = AcpRuntime, client_factory: ClientFactory = AcpClient, reconcile_interval: float | None = None, + permission_callback: PermissionCallback | None = None, + require_permission_bridge: bool = False, ) -> None: if config.db_path is None: raise ValueError("ACP coordinator requires a sqlite db path") @@ -108,6 +123,8 @@ def __init__( ) self._runtime_factory = runtime_factory self._client_factory = client_factory + self._permission_callback = permission_callback + self._require_permission_bridge = bool(require_permission_bridge) self._reconcile_interval = max( 1.0, float( @@ -117,6 +134,11 @@ def __init__( ), ) self._lock = threading.RLock() + # Endpoint minting, runtime publication, prompt lease validation, and + # shutdown are one private generation transaction. Herdr + # tickets are one-shot, so overlapping reconciles cannot be repaired + # after the fact by selecting whichever runtime happened to attach. + self._reconcile_lock = threading.RLock() self._stop = threading.Event() self._slots: dict[str, _RuntimeSlot] = {} self._thread: threading.Thread | None = None @@ -130,8 +152,18 @@ def start(self) -> "AcpRuntimeCoordinator": return self if self._state is not RuntimeState.NEW: raise AcpCoordinatorError("ACP coordinator cannot be restarted") + if self._require_permission_bridge and self._permission_callback is None: + self._state = RuntimeState.FAILED + self._failure_type = "AcpPermissionBridgeUnavailable" + raise AcpPermissionBridgeUnavailable( + "durable ACP permission decisions are not configured" + ) self._state = RuntimeState.STARTING try: + # ACP adapter processes cannot survive this coordinator process. + # Revoke any process-owned rows left by an unclean prior exit before + # a fresh Herdr generation is allowed to attach. + self._expire_orphaned_bindings() self._reconcile(strict=self.config.agent_event_source == "acp_required") except Exception as exc: with self._lock: @@ -140,6 +172,9 @@ def start(self) -> "AcpRuntimeCoordinator": self._stop_all() raise with self._lock: + if self._state is not RuntimeState.STARTING or self._stop.is_set(): + self._state = RuntimeState.STOPPED + raise AcpCoordinatorError("ACP coordinator is stopping") self._state = RuntimeState.RUNNING thread = threading.Thread( target=self._run, @@ -163,10 +198,24 @@ def stop(self, *, timeout: float | None = None) -> None: return self._state = RuntimeState.STOPPING self._stop.set() - self._stop_all(timeout=limit) + deadline = time.monotonic() + limit + # Wait for endpoint mint/start or prompt lease validation to leave its + # critical section before clearing slots. A reconcile that observed + # STOPPING stops its provisional runtime instead of publishing it. + acquired = self._reconcile_lock.acquire(timeout=limit) + if not acquired: + with self._lock: + self._failure_type = "AcpCoordinatorError" + raise AcpCoordinatorError( + "ACP coordinator reconciliation did not stop within the deadline" + ) + try: + self._stop_all(timeout=max(0.001, deadline - time.monotonic())) + finally: + self._reconcile_lock.release() thread = self._thread if thread is not None and thread is not threading.current_thread(): - thread.join(timeout=limit) + thread.join(timeout=max(0.0, deadline - time.monotonic())) with self._lock: if self._state is not RuntimeState.FAILED: self._state = RuntimeState.STOPPED @@ -216,15 +265,15 @@ def status(self) -> dict[str, Any]: def prompt_route(self, worker: Worker) -> _PromptRoute | None: try: - self._current_slot(worker) + slot = self._current_slot(worker) except AcpCoordinatorError: # A just-observed worker may not have reached the periodic pass. try: self._reconcile_worker(worker.id, strict=False) - self._current_slot(worker) + slot = self._current_slot(worker) except Exception: # noqa: BLE001 return None - return _PromptRoute(self, worker) + return _PromptRoute(self, worker, slot) def owns_worker(self, worker_id: str, worker_fingerprint: str) -> bool: """Return whether a healthy ACP slot currently owns this exact worker.""" @@ -259,6 +308,14 @@ def _run(self) -> None: with self._lock: self._failure_type = type(exc).__name__ + def _require_reconcile_state(self, *, allow_starting: bool) -> None: + with self._lock: + allowed = {RuntimeState.RUNNING} + if allow_starting: + allowed.add(RuntimeState.STARTING) + if self._state not in allowed or self._stop.is_set(): + raise AcpCoordinatorError("ACP coordinator is stopping") + def _continuity_bindings(self) -> tuple[dict[str, WorkerBinding], int]: bindings = list_worker_bindings( Path(self.config.db_path), @@ -285,6 +342,15 @@ def _continuity_bindings(self) -> tuple[dict[str, WorkerBinding], int]: return current, ambiguities def _reconcile(self, *, strict: bool) -> None: + with self._reconcile_lock: + try: + self._require_reconcile_state(allow_starting=True) + self._reconcile_locked(strict=strict) + finally: + if self._stop.is_set(): + self._stop_all() + + def _reconcile_locked(self, *, strict: bool) -> None: current, ambiguities = self._continuity_bindings() with self._lock: stale = [worker_id for worker_id in self._slots if worker_id not in current] @@ -295,6 +361,7 @@ def _reconcile(self, *, strict: bool) -> None: for _ in range(ambiguities) ] for worker_id, continuity in current.items(): + self._require_reconcile_state(allow_starting=True) try: self._reconcile_binding(continuity) except Exception as exc: # noqa: BLE001 @@ -312,13 +379,21 @@ def _reconcile(self, *, strict: bool) -> None: raise AcpCoordinatorError("one or more ACP workers failed to attach") def _reconcile_worker(self, worker_id: str, *, strict: bool) -> None: - current, _ambiguities = self._continuity_bindings() - continuity = current.get(worker_id) - if continuity is None: - if strict: - raise AcpCoordinatorError("worker has no unique Herdr authority") - return - self._reconcile_binding(continuity) + with self._reconcile_lock: + try: + self._require_reconcile_state(allow_starting=False) + current, _ambiguities = self._continuity_bindings() + continuity = current.get(worker_id) + if continuity is None: + if strict: + raise AcpCoordinatorError( + "worker has no unique Herdr authority" + ) + return + self._reconcile_binding(continuity) + finally: + if self._stop.is_set(): + self._stop_all() def _reconcile_binding(self, continuity: WorkerBinding) -> None: with self._lock: @@ -346,6 +421,11 @@ def _reconcile_binding(self, continuity: WorkerBinding) -> None: except Exception: pass raise + try: + self._require_reconcile_state(allow_starting=True) + except Exception: + self._stop_runtime(runtime) + raise slot = _RuntimeSlot(continuity, endpoint.generation, runtime) with self._lock: displaced = self._slots.get(continuity.worker_id) @@ -398,6 +478,53 @@ def _require_attached_generation(self, slot: _RuntimeSlot) -> None: self._retire_worker(slot.continuity.worker_id, expected=slot) raise AcpCoordinatorError("ACP worker generation lease is not current") + def _submit_prompt( + self, + worker: Worker, + slot: _RuntimeSlot, + text: str, + *, + producer_turn_id: str, + acknowledgement_timeout: float, + ) -> object: + """Write through the exact route generation used by the receipt.""" + + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + current = self._current_slot(worker) + if current is not slot: + raise AcpCoordinatorError("ACP worker route is stale") + self._require_attached_generation(slot) + if self._current_slot(worker) is not slot: + raise AcpCoordinatorError("ACP worker route is stale") + # The route lease covers the complete JSON-RPC request frame, not + # just status validation. Retirement can proceed as soon as the + # runtime acknowledges that the frame is written; turn completion + # remains supervised asynchronously by the runtime. + return slot.runtime.submit_prompt( + text, + producer_turn_id=producer_turn_id, + acknowledgement_timeout=acknowledgement_timeout, + ) + + def _route_binding_fingerprint( + self, + worker: Worker, + slot: _RuntimeSlot, + ) -> str: + """Return authority only while this exact route remains current.""" + + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + if self._current_slot(worker) is not slot: + raise AcpCoordinatorError("ACP worker route is stale") + binding = getattr(slot.runtime, "_binding", None) + return ( + str(binding.private_fingerprint) + if isinstance(binding, WorkerBinding) + else "" + ) + def _build_runtime( self, continuity: WorkerBinding, @@ -419,18 +546,32 @@ def _build_runtime( binding = _derived_binding(continuity, endpoint.session_id) upsert_worker_bindings(Path(self.config.db_path), [binding]) callback = None - return self._runtime_factory( - client, - config=self.config, - binding=binding, - cwd=endpoint.cwd, - session_mode=endpoint.session_mode, - session_id=endpoint.session_id, - stream_generation=endpoint.generation, - session_binding_callback=callback, - poll_timeout=min(0.25, self.config.acp_request_timeout_seconds), - stop_timeout=self.config.acp_shutdown_timeout_seconds, - ) + try: + return self._runtime_factory( + client, + config=self.config, + binding=binding, + cwd=endpoint.cwd, + session_mode=endpoint.session_mode, + session_id=endpoint.session_id, + stream_generation=endpoint.generation, + session_binding_callback=callback, + permission_callback=self._permission_callback, + poll_timeout=min(0.25, self.config.acp_request_timeout_seconds), + stop_timeout=self.config.acp_shutdown_timeout_seconds, + ) + except Exception: + try: + client.close() + except Exception: + pass + if endpoint.session_mode is not SessionOpenMode.NEW: + _expire_derived_binding( + self.config, + binding, + reason="acp_runtime_construction_failed", + ) + raise def _bind_new_session( self, @@ -455,6 +596,7 @@ def _retire_worker( self._stop_runtime(slot.runtime) def _stop_runtime(self, runtime: AcpRuntime, *, timeout: float | None = None) -> None: + binding = getattr(runtime, "_binding", None) try: runtime.stop( timeout=( @@ -465,6 +607,24 @@ def _stop_runtime(self, runtime: AcpRuntime, *, timeout: float | None = None) -> ) except Exception: pass + finally: + if isinstance(binding, WorkerBinding) and binding.backend == "acp": + _expire_derived_binding( + self.config, + binding, + reason="acp_runtime_retired", + ) + + def _expire_orphaned_bindings(self) -> None: + """Revoke ACP leases that no runtime in this process can own.""" + + expire_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="acp", + now=utc_timestamp(), + reason="acp_coordinator_restarted", + ) def _stop_all(self, *, timeout: float | None = None) -> None: with self._lock: @@ -498,10 +658,32 @@ def _derived_binding( backend="acp", turn_target_kind="acp_session_id", turn_target_value=session_id, + # The ACP runtime owns this private lease until explicit stop/failure. + # Inheriting the observer's short Herdr lease would strand a healthy + # attached runtime after the next observation-expiry boundary. + expires_at=None, private_fingerprint="", ) +def _expire_derived_binding( + config: Config, + binding: WorkerBinding, + *, + reason: str, +) -> None: + if config.db_path is None: # pragma: no cover - coordinator invariant + return + expire_worker_bindings( + Path(config.db_path), + binding.host_id, + backend="acp", + private_fingerprints=[binding.private_fingerprint], + now=binding.observed_at, + reason=reason, + ) + + def _same_continuity(left: WorkerBinding, right: WorkerBinding) -> bool: """Compare authority identity while ignoring observation lease refreshes.""" return ( @@ -533,6 +715,38 @@ def _nonempty_text(value: Any, field: str) -> str: return value +def _require_target_identity( + continuity: WorkerBinding, + worker: Mapping[str, Any], + *, + response: str, +) -> None: + """Prove the returned worker still owns the requested Herdr target.""" + + direct_fields = { + "terminal_id": "terminal_id", + "pane_id": "pane_id", + "name": "name", + "label": "name", + "agent": "agent", + } + field = direct_fields.get(continuity.target_kind) + if field is not None: + matched = worker.get(field) == continuity.target_value + else: + # Older Herdr projections can call the terminal identity `agent_id`. + # The v1 endpoint contract has no agent_id member, so require that the + # authority token still matches one of its immutable identity fields. + matched = continuity.target_value in { + worker.get("terminal_id"), + worker.get("pane_id"), + worker.get("name"), + worker.get("agent"), + } + if not matched: + raise AcpCoordinatorError(f"Herdr ACP {response} target changed") + + def _parse_endpoint( config: Config, continuity: WorkerBinding, @@ -608,11 +822,10 @@ def _parse_endpoint( "generation", }: raise AcpCoordinatorError("Herdr ACP worker identity shape is invalid") - pane_id = _nonempty_text(worker.get("pane_id"), "pane_id") - if continuity.target_kind == "pane_id" and pane_id != continuity.target_value: - raise AcpCoordinatorError("Herdr ACP pane authority changed") + _require_target_identity(continuity, worker, response="endpoint") for field in ("terminal_id", "workspace_id", "tab_id", "name", "agent"): _nonempty_text(worker.get(field), field) + _nonempty_text(worker.get("pane_id"), "pane_id") if set(adapter) != {"name", "version"}: raise AcpCoordinatorError("Herdr ACP adapter identity shape is invalid") _nonempty_text(adapter.get("name"), "adapter name") @@ -699,11 +912,7 @@ def _parse_status( "agent", ): _nonempty_text(worker.get(field), field) - if ( - continuity.target_kind == "pane_id" - and worker.get("pane_id") != continuity.target_value - ): - raise AcpCoordinatorError("Herdr ACP status pane authority changed") + _require_target_identity(continuity, worker, response="status") if set(adapter) != {"name", "version"}: raise AcpCoordinatorError("Herdr ACP status adapter shape is invalid") _nonempty_text(adapter.get("name"), "adapter name") @@ -730,4 +939,12 @@ def production_acp_runtime_factory( stop_event: threading.Event, ) -> AcpRuntimeCoordinator: """Build the stock daemon's Herdr-backed multi-worker ACP coordinator.""" - return AcpRuntimeCoordinator(config, stop_event) + # ACP permission requests are synchronous and can authorize destructive + # tools. Until the daemon has a durable, worker/session-correlated decision + # broker, production must refuse ACP modes instead of silently cancelling + # requests while reporting the worker healthy. + return AcpRuntimeCoordinator( + config, + stop_event, + require_permission_bridge=True, + ) diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index c98d24c..40a79a3 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -2730,9 +2730,12 @@ def submit_acp_command( if required else None ) - binding_fingerprint = str( - getattr(route, "binding_fingerprint", "") or "" - ).strip() + try: + binding_fingerprint = str( + getattr(route, "binding_fingerprint", "") or "" + ).strip() + except Exception: # noqa: BLE001 + binding_fingerprint = "" if not binding_fingerprint: if takeover is not None: return _request_in_progress(request) diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py index 859484a..adf12ec 100644 --- a/tests/test_acp_client.py +++ b/tests/test_acp_client.py @@ -132,6 +132,20 @@ def test_initialize_capabilities_and_session_lifecycle() -> None: assert acp.exit.returncode == 0 +def test_load_session_accepts_standard_null_result(monkeypatch: pytest.MonkeyPatch) -> None: + """ACP v1 completes load replay with a JSON-RPC null result.""" + + with client() as acp: + acp.initialize() + monkeypatch.setattr(acp, "request", lambda *_args, **_kwargs: None) + loaded = acp.load_session("s1", "/tmp/project") + + assert loaded.session_id == "s1" + assert loaded.modes is None + assert loaded.config_options == () + assert loaded.raw == {} + + def test_prompt_stream_and_permission_response_can_run_concurrently() -> None: with client() as acp: acp.initialize() diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index a9e6a5d..498c5d2 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -14,16 +14,21 @@ from tendwire.backends.acp_coordinator import ( AcpCoordinatorError, + AcpPermissionBridgeUnavailable, AcpRuntimeCoordinator, + _derived_binding, _parse_endpoint, + _parse_status, + production_acp_runtime_factory, ) from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult -from tendwire.command_submission import submit_command +from tendwire.command_submission import submit_acp_command, submit_command from tendwire.config import Config from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding from tendwire.store.sqlite import ( get_command_request, init_store, + list_worker_bindings, save_snapshot, upsert_worker_bindings, ) @@ -117,6 +122,21 @@ def test_endpoint_requires_explicit_acp_ownership_and_strict_attach_shape(tmp_pa with pytest.raises(AcpCoordinatorError, match="inconsistent"): _parse_endpoint(config, _binding(), replayed) + wrong_terminal = _endpoint() + wrong_terminal["worker"]["terminal_id"] = "other-terminal" + terminal_binding = replace( + _binding(), + target_kind="terminal_id", + target_value="pane-private", + ) + with pytest.raises(AcpCoordinatorError, match="target changed"): + _parse_endpoint(config, terminal_binding, wrong_terminal) + + wrong_status = _status() + wrong_status["worker"]["terminal_id"] = "other-terminal" + with pytest.raises(AcpCoordinatorError, match="target changed"): + _parse_status(terminal_binding, wrong_status) + def test_canonical_herdr_acp_contract_fixture_executes_configured_binary( tmp_path: Path, @@ -248,6 +268,41 @@ def test_acp_failure_after_send_started_is_uncertain_and_never_falls_back(tmp_pa assert receipt is not None and receipt["state"] == "uncertain" +def test_route_authority_failure_is_safe_before_receipt_reservation(tmp_path: Path) -> None: + config = _config(tmp_path) + _seed(config) + + class VanishedRoute: + @property + def binding_fingerprint(self) -> str: + raise AcpCoordinatorError("slot changed") + + def prompt(self, *_args: Any, **_kwargs: Any) -> None: + raise AssertionError("a route without authority must not send") + + assert ( + submit_acp_command( + config, + _request("route-race-preferred"), + prompt_router=lambda _worker: VanishedRoute(), + ) + is None + ) + required = submit_acp_command( + config, + _request("route-race-required"), + prompt_router=lambda _worker: VanishedRoute(), + required=True, + ) + assert required is not None + assert required.status == "backend_unavailable" + assert get_command_request( + config.db_path, + config.host_id, + "route-race-required", + ) is None + + def test_concurrent_duplicate_acp_command_has_one_external_send(tmp_path: Path) -> None: config = _config(tmp_path) _seed(config) @@ -315,6 +370,17 @@ def test_new_and_resumed_endpoint_session_invariants(tmp_path: Path) -> None: _parse_endpoint(config, _binding(), missing) +def test_derived_acp_binding_outlives_observation_lease() -> None: + continuity = replace( + _binding(), + observed_at="2026-07-31T00:00:00+00:00", + expires_at="2026-07-31T00:00:05+00:00", + ) + derived = _derived_binding(continuity, "session-private") + assert derived.expires_at.startswith("9999-") + assert derived.private_fingerprint != continuity.private_fingerprint + + def test_reconnect_remints_endpoint_instead_of_replaying_attach_ticket(tmp_path: Path) -> None: config = _config(tmp_path) assert config.db_path is not None @@ -401,6 +467,360 @@ def submit_prompt(self, *_args: Any, **_kwargs: Any) -> None: "one-shot-private-ticket-1", "one-shot-private-ticket-2", ] + replaced_route = coordinator.prompt_route(worker) + assert replaced_route is not None + generation[0] = 44 + coordinator._reconcile_worker("worker-1", strict=True) + with pytest.raises(AcpCoordinatorError, match="stale"): + _ = replaced_route.binding_fingerprint + with pytest.raises(AcpCoordinatorError, match="stale"): + replaced_route.prompt( + "must-not-cross-generations", + producer_turn_id="producer-3", + timeout=1.0, + ) + assert runtimes[-1].prompt_calls == 0 + finally: + coordinator.stop() + + +def test_concurrent_reconcile_mints_only_one_endpoint(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + entered = threading.Event() + release = threading.Event() + minted = 0 + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + nonlocal minted + minted += 1 + entered.set() + assert release.wait(2.0) + return _endpoint() + + def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: + return _status() + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self.stopped = False + + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + self.stopped = True + + def status(self) -> Any: + return SimpleNamespace(healthy=not self.stopped, failure_type=None) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + client_factory=lambda *_args, **_kwargs: object(), + runtime_factory=Runtime, + reconcile_interval=60.0, + ).start() + upsert_worker_bindings(config.db_path, [_binding()]) + try: + with ThreadPoolExecutor(max_workers=2) as pool: + first = pool.submit(coordinator._reconcile_worker, "worker-1", strict=True) + assert entered.wait(1.0) + second = pool.submit(coordinator._reconcile_worker, "worker-1", strict=True) + release.set() + first.result(timeout=2.0) + second.result(timeout=2.0) + assert minted == 1 + finally: + release.set() + coordinator.stop() + + +def test_stop_cannot_leave_inflight_reconcile_runtime_published(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + entered = threading.Event() + release = threading.Event() + stop_done = threading.Event() + runtimes: list[Any] = [] + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + entered.set() + assert release.wait(2.0) + return _endpoint() + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self.stopped = False + runtimes.append(self) + + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + self.stopped = True + + def status(self) -> Any: + return SimpleNamespace(healthy=not self.stopped, failure_type=None) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + client_factory=lambda *_args, **_kwargs: object(), + runtime_factory=Runtime, + reconcile_interval=60.0, + ).start() + upsert_worker_bindings(config.db_path, [_binding()]) + with ThreadPoolExecutor(max_workers=2) as pool: + reconcile = pool.submit(coordinator._reconcile_worker, "worker-1", strict=True) + assert entered.wait(1.0) + + def stop() -> None: + coordinator.stop() + stop_done.set() + + stopping = pool.submit(stop) + assert not stop_done.wait(0.1) + release.set() + with pytest.raises(AcpCoordinatorError, match="stopping"): + reconcile.result(timeout=2.0) + stopping.result(timeout=2.0) + assert coordinator._slots == {} + assert runtimes and all(runtime.stopped for runtime in runtimes) + + +def test_prompt_frame_acknowledgement_fences_generation_retirement(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + generation = [42] + endpoint_calls = 0 + prompt_entered = threading.Event() + prompt_release = threading.Event() + runtimes: list[Any] = [] + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + nonlocal endpoint_calls + endpoint_calls += 1 + return _endpoint(generation=generation[0]) + + def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: + return _status(generation=generation[0]) + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self._binder = kwargs["session_binding_callback"] + self.stopped = False + runtimes.append(self) + + def start(self) -> None: + if self._binder is not None: + self._binding = self._binder( + f"session-private-{len(runtimes)}", self._binding + ) + + def submit_prompt(self, *_args: Any, **_kwargs: Any) -> None: + prompt_entered.set() + assert prompt_release.wait(2.0) + + def stop(self, *, timeout: float) -> None: + self.stopped = True + + def status(self) -> Any: + return SimpleNamespace(healthy=not self.stopped, failure_type=None) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + client_factory=lambda *_args, **_kwargs: object(), + runtime_factory=Runtime, + reconcile_interval=60.0, + ).start() + worker = Worker( + id="worker-1", + name="worker", + status="working", + fingerprint="worker-fingerprint", + ) + route = coordinator.prompt_route(worker) + assert route is not None + try: + with ThreadPoolExecutor(max_workers=2) as pool: + prompt = pool.submit( + route.prompt, + "one frame", + producer_turn_id="producer-1", + timeout=1.0, + ) + assert prompt_entered.wait(1.0) + generation[0] = 43 + reconcile = pool.submit( + coordinator._reconcile_worker, "worker-1", strict=True + ) + # A new endpoint cannot be minted and the old transport cannot be + # stopped while its request frame is still being acknowledged. + assert not reconcile.done() + assert endpoint_calls == 1 + assert not runtimes[0].stopped + prompt_release.set() + prompt.result(timeout=2.0) + reconcile.result(timeout=2.0) + assert endpoint_calls == 2 + assert runtimes[0].stopped + assert len(runtimes) == 2 + finally: + prompt_release.set() + coordinator.stop() + + +def test_runtime_factory_failure_rolls_back_resumed_binding(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + endpoint = _endpoint() + endpoint["session"] = {"mode": "resume", "id": "session-private"} + return endpoint + + def close(self) -> None: + return None + + class Client: + def __init__(self) -> None: + self.closed = False + + def close(self) -> None: + self.closed = True + + client = Client() + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + client_factory=lambda *_args, **_kwargs: client, + runtime_factory=lambda *_args, **_kwargs: (_ for _ in ()).throw( + RuntimeError("constructor failed with --ticket private") + ), + reconcile_interval=60.0, + ).start() + upsert_worker_bindings(config.db_path, [_binding()]) + try: + with pytest.raises(RuntimeError, match="constructor failed"): + coordinator._reconcile_worker("worker-1", strict=True) + assert client.closed + assert list_worker_bindings(config.db_path, config.host_id, backend="acp") == [] + finally: + coordinator.stop() + + +def test_coordinator_start_revokes_orphaned_process_binding(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + orphaned = _derived_binding(_binding(), "orphaned-session") + upsert_worker_bindings(config.db_path, [orphaned]) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: object(), + client_factory=lambda *_args, **_kwargs: object(), + reconcile_interval=60.0, + ).start() + try: + assert list_worker_bindings( + config.db_path, config.host_id, backend="acp" + ) == [] + finally: + coordinator.stop() + + +def test_production_coordinator_fails_closed_without_permission_bridge( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + coordinator = production_acp_runtime_factory(config, threading.Event()) + + with pytest.raises( + AcpPermissionBridgeUnavailable, + match="durable ACP permission decisions", + ): + coordinator.start() + + status = coordinator.status() + assert status["healthy"] is False + assert status["state"] == "failed" + assert status["failure_type"] == "AcpPermissionBridgeUnavailable" + + +def test_coordinator_forwards_explicit_permission_bridge(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + seen: list[Any] = [] + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + return _endpoint() + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + seen.append(kwargs["permission_callback"]) + self._binding = kwargs["binding"] + + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + return None + + def status(self) -> Any: + return SimpleNamespace(healthy=True, failure_type=None) + + def permission_bridge(_request: Any) -> str | None: + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + client_factory=lambda *_args, **_kwargs: object(), + runtime_factory=Runtime, + permission_callback=permission_bridge, + require_permission_bridge=True, + reconcile_interval=60.0, + ).start() + try: + assert seen == [permission_bridge] finally: coordinator.stop() From fbbf923283b32c81779fc68d18c3fb4cbc3ece4d Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 12:02:40 +0800 Subject: [PATCH 37/83] feat(acp): bridge durable permission decisions --- README.md | 14 +- src/tendwire/backends/acp_coordinator.py | 82 ++++- src/tendwire/backends/acp_permissions.py | 275 ++++++++++++++++ src/tendwire/backends/acp_runtime.py | 47 ++- src/tendwire/command_submission.py | 96 +++++- src/tendwire/daemon.py | 13 +- src/tendwire/store/sqlite.py | 91 +++++- tests/test_acp_coordinator.py | 19 +- tests/test_acp_permissions.py | 330 ++++++++++++++++++++ tests/test_acp_runtime.py | 6 +- tests/test_agent_events.py | 2 +- tests/test_backend_pending.py | 2 +- tests/test_connector_outbox.py | 2 +- tests/test_delivery_retention_migration.py | 2 +- tests/test_delivery_retention_projection.py | 2 +- tests/test_delivery_retention_recovery.py | 2 +- tests/test_store.py | 4 +- 17 files changed, 919 insertions(+), 70 deletions(-) create mode 100644 src/tendwire/backends/acp_permissions.py create mode 100644 tests/test_acp_permissions.py diff --git a/README.md b/README.md index c5f808a..6489e6c 100644 --- a/README.md +++ b/README.md @@ -573,12 +573,14 @@ same public-safe degraded evidence. The stock daemon currently defaults to `legacy`. ACP modes use Herdr's private `agent.acp_endpoint` contract and accept only workers explicitly marked `acp_owned_ready`; ordinary PTY workers are never attached as sidecars. -Production ACP startup is currently fail-closed with -`AcpPermissionBridgeUnavailable`: Tendwire does not yet have a durable, -worker/session-correlated bridge from `answer_decision` to the exact synchronous -ACP permission request. The daemon must not silently auto-cancel tool -permissions while reporting ACP healthy. The generic coordinator callback is -an embedding/test hook, not a production authorization path. +Production ACP uses a bounded per-worker permission broker. It publishes only +the sanitized tool title and numeric choices (option label and kind) through +the durable pending-decision surface; ACP option IDs, arguments, session IDs, +and adapter metadata remain private. `answer_decision` is fenced to the exact +worker binding, ACP session, and Herdr generation. A command is accepted only +after the complete JSON-RPC permission-response frame is written. Missing or +retired ACP authority fails closed without falling back to PTY input, and +concurrent answers can produce at most one response. `acp_shadow` persists ACP events without projecting them, but no automated shadow comparator is implemented. `acp_preferred` falls back only before an ACP reservation/send, while `acp_required` fails closed and never starts the legacy diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 9dbaa21..011aca1 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -24,6 +24,7 @@ upsert_worker_bindings, ) from .acp_client import AcpClient +from .acp_permissions import AcpPermissionBroker from .acp_runtime import ( AcpRuntime, PermissionCallback, @@ -61,6 +62,7 @@ class _RuntimeSlot: continuity: WorkerBinding generation: str runtime: AcpRuntime + permission_broker: AcpPermissionBroker | None = None class _PromptRoute: @@ -113,9 +115,14 @@ def __init__( reconcile_interval: float | None = None, permission_callback: PermissionCallback | None = None, require_permission_bridge: bool = False, + durable_permission_bridge: bool = False, ) -> None: if config.db_path is None: raise ValueError("ACP coordinator requires a sqlite db path") + if durable_permission_bridge and permission_callback is not None: + raise ValueError( + "durable ACP permission bridge cannot use an embedding callback" + ) self.config = config self._daemon_stop = stop_event self._endpoint_client_factory = ( @@ -125,6 +132,7 @@ def __init__( self._client_factory = client_factory self._permission_callback = permission_callback self._require_permission_bridge = bool(require_permission_bridge) + self._durable_permission_bridge = bool(durable_permission_bridge) self._reconcile_interval = max( 1.0, float( @@ -152,7 +160,11 @@ def start(self) -> "AcpRuntimeCoordinator": return self if self._state is not RuntimeState.NEW: raise AcpCoordinatorError("ACP coordinator cannot be restarted") - if self._require_permission_bridge and self._permission_callback is None: + if ( + self._require_permission_bridge + and self._permission_callback is None + and not self._durable_permission_bridge + ): self._state = RuntimeState.FAILED self._failure_type = "AcpPermissionBridgeUnavailable" raise AcpPermissionBridgeUnavailable( @@ -285,6 +297,35 @@ def owns_worker(self, worker_id: str, worker_fingerprint: str) -> bool: and slot.runtime.status().healthy ) + def owns_permission_decision(self, decision: Any) -> bool: + """Return whether one pending decision belongs to an exact live slot.""" + worker_id = str(getattr(decision, "worker_id", "") or "") + with self._lock: + slot = self._slots.get(worker_id) + if slot is None or slot.permission_broker is None: + return False + try: + self._require_attached_generation(slot) + return slot.permission_broker.owns(decision) + except Exception: + return False + + def answer_permission_decision(self, decision: Any, *, timeout: float) -> None: + """Select an offered option and wait for its response-frame write.""" + worker_id = str(getattr(decision, "worker_id", "") or "") + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + with self._lock: + slot = self._slots.get(worker_id) + if slot is None or slot.permission_broker is None: + raise AcpCoordinatorError("ACP permission route is unavailable") + self._require_attached_generation(slot) + if not slot.permission_broker.owns(decision): + raise AcpCoordinatorError("ACP permission authority changed") + # Keep retirement fenced until respond_permission has acknowledged + # writing the complete JSON-RPC response frame. + slot.permission_broker.answer(decision, timeout=timeout) + def _current_slot(self, worker: Worker) -> _RuntimeSlot: with self._lock: if self._state is not RuntimeState.RUNNING: @@ -412,10 +453,12 @@ def _reconcile_binding(self, continuity: WorkerBinding) -> None: if existing is not None: self._retire_worker(continuity.worker_id, expected=existing) endpoint = self._resolve_endpoint(continuity) - runtime = self._build_runtime(continuity, endpoint) + runtime, permission_broker = self._build_runtime(continuity, endpoint) try: runtime.start() except Exception: + if permission_broker is not None: + permission_broker.close() try: runtime.stop(timeout=self.config.acp_shutdown_timeout_seconds) except Exception: @@ -424,9 +467,13 @@ def _reconcile_binding(self, continuity: WorkerBinding) -> None: try: self._require_reconcile_state(allow_starting=True) except Exception: + if permission_broker is not None: + permission_broker.close() self._stop_runtime(runtime) raise - slot = _RuntimeSlot(continuity, endpoint.generation, runtime) + slot = _RuntimeSlot( + continuity, endpoint.generation, runtime, permission_broker + ) with self._lock: displaced = self._slots.get(continuity.worker_id) self._slots[continuity.worker_id] = slot @@ -529,7 +576,7 @@ def _build_runtime( self, continuity: WorkerBinding, endpoint: HerdrAcpEndpoint, - ) -> AcpRuntime: + ) -> tuple[AcpRuntime, AcpPermissionBroker | None]: client = self._client_factory( endpoint.command, cwd=endpoint.cwd, @@ -546,8 +593,19 @@ def _build_runtime( binding = _derived_binding(continuity, endpoint.session_id) upsert_worker_bindings(Path(self.config.db_path), [binding]) callback = None + permission_broker = ( + AcpPermissionBroker( + self.config, + worker_id=continuity.worker_id, + worker_fingerprint=continuity.worker_fingerprint, + generation=endpoint.generation, + timeout=float(self.config.submission_hard_ttl_seconds), + ) + if self._durable_permission_bridge + else None + ) try: - return self._runtime_factory( + runtime = self._runtime_factory( client, config=self.config, binding=binding, @@ -556,11 +614,14 @@ def _build_runtime( session_id=endpoint.session_id, stream_generation=endpoint.generation, session_binding_callback=callback, - permission_callback=self._permission_callback, + permission_callback=(permission_broker or self._permission_callback), poll_timeout=min(0.25, self.config.acp_request_timeout_seconds), stop_timeout=self.config.acp_shutdown_timeout_seconds, ) + return runtime, permission_broker except Exception: + if permission_broker is not None: + permission_broker.close() try: client.close() except Exception: @@ -593,6 +654,8 @@ def _retire_worker( if slot is None or (expected is not None and slot is not expected): return self._slots.pop(worker_id, None) + if slot.permission_broker is not None: + slot.permission_broker.close() self._stop_runtime(slot.runtime) def _stop_runtime(self, runtime: AcpRuntime, *, timeout: float | None = None) -> None: @@ -639,6 +702,8 @@ def _stop_all(self, *, timeout: float | None = None) -> None: ) deadline = time.monotonic() + total for slot in slots: + if slot.permission_broker is not None: + slot.permission_broker.close() self._stop_runtime( slot.runtime, timeout=max(0.001, deadline - time.monotonic()), @@ -939,12 +1004,9 @@ def production_acp_runtime_factory( stop_event: threading.Event, ) -> AcpRuntimeCoordinator: """Build the stock daemon's Herdr-backed multi-worker ACP coordinator.""" - # ACP permission requests are synchronous and can authorize destructive - # tools. Until the daemon has a durable, worker/session-correlated decision - # broker, production must refuse ACP modes instead of silently cancelling - # requests while reporting the worker healthy. return AcpRuntimeCoordinator( config, stop_event, require_permission_bridge=True, + durable_permission_bridge=True, ) diff --git a/src/tendwire/backends/acp_permissions.py b/src/tendwire/backends/acp_permissions.py new file mode 100644 index 0000000..83cf82f --- /dev/null +++ b/src/tendwire/backends/acp_permissions.py @@ -0,0 +1,275 @@ +"""Durable, privacy-preserving bridge for ACP permission decisions.""" + +from __future__ import annotations + +import secrets +import threading +import time +from collections.abc import Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from ..config import Config +from ..core.models import WorkerBinding, sanitize_public_text, stable_fingerprint +from ..core.turns import PendingObservation, PendingObservedChoice +from ..store.sqlite import ( + apply_backend_pending_observation, + list_worker_bindings, +) +from .acp_protocol import PermissionRequest +from .acp_runtime import PermissionSelection + + +class AcpPermissionBrokerError(RuntimeError): + """A permission could not be safely correlated or acknowledged.""" + + +_MAX_PERMISSION_OPTIONS = 64 + + +@dataclass(slots=True) +class _Offer: + decision_ref: str + binding: WorkerBinding + generation: str + option_ids: tuple[str, ...] + condition: threading.Condition + selected: int | None = None + response_state: str = "pending" + response_error: BaseException | None = None + + +class AcpPermissionBroker: + """One bounded permission rendezvous for one worker generation.""" + + def __init__( + self, + config: Config, + *, + worker_id: str, + worker_fingerprint: str, + generation: str, + timeout: float, + ) -> None: + if config.db_path is None: + raise ValueError("ACP permission broker requires a sqlite db path") + if timeout <= 0: + raise ValueError("ACP permission broker timeout must be positive") + self.config = config + self.worker_id = worker_id + self.worker_fingerprint = worker_fingerprint + self.generation = generation + self.timeout = float(timeout) + self._lock = threading.RLock() + self._offer: _Offer | None = None + self._closed = False + + def __call__(self, request: PermissionRequest) -> PermissionSelection | None: + if not request.options or len(request.options) > _MAX_PERMISSION_OPTIONS: + return None + binding = self._exact_binding(request.session_id) + title = _tool_title(request.tool_call) + labels = tuple( + _option_label( + option.name, + str( + option.kind.value + if hasattr(option.kind, "value") + else option.kind + ), + ) + for option in request.options + ) + if not labels: + return None + nonce = secrets.token_urlsafe(24) + source_revision = stable_fingerprint( + { + "route": "acp_permission_v1", + "worker_id": self.worker_id, + "worker_fingerprint": self.worker_fingerprint, + "binding": binding.private_fingerprint, + "session": binding.turn_target_value, + "generation": self.generation, + "nonce": nonce, + } + ) + persisted_revision = stable_fingerprint( + { + "decision_revision": source_revision, + "binding_private_fingerprint": binding.private_fingerprint, + "observed_turn_target_value": binding.turn_target_value, + } + ) + offer = _Offer( + decision_ref=f"decision-{persisted_revision}", + binding=binding, + generation=self.generation, + option_ids=tuple(option.option_id for option in request.options), + condition=threading.Condition(self._lock), + ) + observation = PendingObservation( + "open_prompt", + question=title, + pending_kind="approval", + choices=tuple( + PendingObservedChoice( + choice_id="choice-" + + stable_fingerprint( + { + "revision": source_revision, + "ordinal": index, + "label": label, + } + ), + label=label, + picker_ordinal=index, + ) + for index, label in enumerate(labels, 1) + ), + revision_digest=source_revision, + decision_kind="single", + decision_options=labels, + decision_multi_select=False, + decision_question_count=1, + ) + try: + with self._lock: + if self._closed or self._offer is not None: + return None + self._offer = offer + changed = apply_backend_pending_observation( + Path(self.config.db_path), + self.config.host_id, + self.worker_id, + observation, + binding_private_fingerprint=binding.private_fingerprint, + observed_turn_target_value=binding.turn_target_value, + binding_authoritative=True, + route_kind="acp_permission", + ) + if not changed: + raise AcpPermissionBrokerError("permission overlay was not published") + deadline = time.monotonic() + self.timeout + with offer.condition: + while offer.selected is None and not self._closed: + remaining = deadline - time.monotonic() + if remaining <= 0: + break + offer.condition.wait(remaining) + if offer.selected is None: + self._clear_offer(offer) + return None + option_id = offer.option_ids[offer.selected - 1] + return PermissionSelection( + option_id, + response_written=lambda: self._response_complete(offer, None), + response_failed=lambda exc: self._response_complete(offer, exc), + ) + except BaseException: + self._clear_offer(offer) + raise + + def owns(self, decision: Any) -> bool: + with self._lock: + offer = self._offer + return bool( + not self._closed + and offer is not None + and decision.worker_id == self.worker_id + and decision.worker_fingerprint == self.worker_fingerprint + and decision.binding_private_fingerprint == offer.binding.private_fingerprint + and decision.turn_target_value == offer.binding.turn_target_value + and decision.decision_ref == offer.decision_ref + and offer.generation == self.generation + ) + + def answer(self, decision: Any, *, timeout: float) -> None: + with self._lock: + offer = self._offer + if offer is None or not self.owns(decision): + raise AcpPermissionBrokerError("ACP permission authority changed") + if decision.text is not None or len(decision.option_refs) != 1: + raise AcpPermissionBrokerError("ACP permission selection is invalid") + ordinal = int(decision.option_refs[0]) + if ordinal < 1 or ordinal > len(offer.option_ids) or offer.selected is not None: + raise AcpPermissionBrokerError("ACP permission was already answered") + offer.selected = ordinal + offer.condition.notify_all() + deadline = time.monotonic() + max(0.1, float(timeout)) + while offer.response_state == "pending" and not self._closed: + remaining = deadline - time.monotonic() + if remaining <= 0: + break + offer.condition.wait(remaining) + if offer.response_state != "written": + raise AcpPermissionBrokerError("ACP permission response state is uncertain") + self._offer = None + + def close(self) -> None: + with self._lock: + self._closed = True + offer = self._offer + if offer is not None: + offer.condition.notify_all() + if offer is not None: + self._clear_offer(offer) + + def _response_complete(self, offer: _Offer, error: BaseException | None) -> None: + with offer.condition: + if self._offer is offer: + offer.response_state = "failed" if error is not None else "written" + offer.response_error = error + offer.condition.notify_all() + + def _clear_offer(self, offer: _Offer) -> None: + with self._lock: + if self._offer is not offer: + return + self._offer = None + try: + apply_backend_pending_observation( + Path(self.config.db_path), + self.config.host_id, + self.worker_id, + PendingObservation("read_succeeded_no_prompt"), + binding_private_fingerprint=offer.binding.private_fingerprint, + observed_turn_target_value=offer.binding.turn_target_value, + route_kind="acp_permission", + ) + except Exception: + pass + + def _exact_binding(self, session_id: str) -> WorkerBinding: + rows = [ + row + for row in list_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="acp", + ) + if row.worker_id == self.worker_id + and row.worker_fingerprint == self.worker_fingerprint + and row.turn_target_kind == "acp_session_id" + and row.turn_target_value == session_id + and row.sendable + ] + if len(rows) != 1: + raise AcpPermissionBrokerError("ACP permission binding is not current") + return rows[0] + + +def _tool_title(tool_call: Any) -> str: + if isinstance(tool_call, Mapping): + value = tool_call.get("title") + if isinstance(value, str) and value.strip(): + clean = sanitize_public_text(value.strip()[:500]) + return clean or "Tool permission" + return "Tool permission" + + +def _option_label(name: str, kind: str) -> str: + clean_name = sanitize_public_text(name.strip()[:300]) or "Option" + clean_kind = sanitize_public_text(kind.strip()[:80]) or "unknown" + return f"{clean_name} ({clean_kind})" diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 763b8a0..84fb7a6 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -87,7 +87,20 @@ class AcpRuntimeStatus: failure_type: str | None -PermissionCallback = Callable[[PermissionRequest], str | None] +@dataclass(frozen=True, slots=True) +class PermissionSelection: + """One selected option plus transport acknowledgement hooks. + + The callback that selected an option must not report its durable command as + accepted until the JSON-RPC response frame has actually been written. + """ + + option_id: str + response_written: Callable[[], None] + response_failed: Callable[[BaseException], None] + + +PermissionCallback = Callable[[PermissionRequest], str | PermissionSelection | None] IngestorFactory = Callable[..., AcpSessionIngestor] @@ -878,15 +891,26 @@ def _handle_permission( self._permissions_ingested += 1 selected: str | None = None + selection: PermissionSelection | None = None callback_failure: BaseException | None = None if self._permission_callback is not None: try: candidate = self._permission_callback(request) - if candidate is not None and candidate in { + candidate_id = ( + candidate.option_id + if isinstance(candidate, PermissionSelection) + else candidate + ) + if candidate_id is not None and candidate_id in { option.option_id for option in request.options }: - selected = candidate - elif candidate is not None: + selected = candidate_id + selection = ( + candidate + if isinstance(candidate, PermissionSelection) + else None + ) + elif candidate_id is not None: with self._state_lock: self._invalid_permission_selections += 1 except BaseException as exc: @@ -900,10 +924,17 @@ def _handle_permission( with self._state_lock: self._permissions_cancelled += 1 else: - self._client.respond_permission( - request.request_id, - option_id=selected, - ) + try: + self._client.respond_permission( + request.request_id, + option_id=selected, + ) + except BaseException as exc: + if selection is not None: + selection.response_failed(exc) + raise + if selection is not None: + selection.response_written() with self._state_lock: self._permissions_selected += 1 if callback_failure is not None: diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 40a79a3..5a55108 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -128,6 +128,14 @@ def prompt( AcpPromptRouter = Callable[[Worker], AcpPromptRoute | None] +class AcpPermissionDecisionRouter(Protocol): + """Private daemon-owned bridge for one exact ACP permission decision.""" + + def owns_permission_decision(self, decision: Any) -> bool: ... + + def answer_permission_decision(self, decision: Any, *, timeout: float) -> None: ... + + @dataclass(frozen=True) class ResolvedCommandTarget: worker: Worker @@ -641,6 +649,7 @@ def _decision_claim_has_exact_route(claim: Any) -> bool: and not isinstance(claim.option_count, bool) and claim.option_count >= 1 and isinstance(getattr(claim, "option_refs", None), tuple) + and getattr(claim, "route_kind", None) in {"legacy", "acp_permission"} and ( (claim.text is None and bool(claim.option_refs)) or ( @@ -666,6 +675,7 @@ def _same_decision_route(left: Any, right: Any) -> bool: left.option_count, left.option_refs, left.text, + left.route_kind, ) == ( right.worker_id, @@ -677,10 +687,16 @@ def _same_decision_route(left: Any, right: Any) -> bool: right.option_count, right.option_refs, right.text, + right.route_kind, ) ) +def _decision_uses_acp_binding(_config: Config, decision: Any) -> bool: + """Classify from durable provenance, independent of current liveness.""" + return getattr(decision, "route_kind", "legacy") == "acp_permission" + + class PreSendCertainty(Enum): """How a pre-send failure must be classified before any external mutation. @@ -2043,6 +2059,13 @@ def _validate_pending_decision( ) if validated.status == "validated" and _decision_claim_has_exact_route(validated): return validated + if validated.status == "acp_authority_unavailable": + return _safe_transient_pre_send( + _backend_unavailable( + request, + "ACP permission authority is temporarily unavailable", + ) + ) status = { "already_claimed": STATUS_ANSWER_IN_PROGRESS, "unknown_worker": STATUS_UNKNOWN_WORKER, @@ -2080,6 +2103,11 @@ def _claim_pending_decision( ): if _same_decision_route(validated, claim): return claim + if claim.status == "acp_authority_unavailable": + return _backend_unavailable( + request, + "ACP permission authority is temporarily unavailable", + ) status = { "already_claimed": STATUS_ANSWER_IN_PROGRESS, "unknown_worker": STATUS_UNKNOWN_WORKER, @@ -2145,6 +2173,9 @@ def _answer_decision( validated: Any, reservation: ReservedCommandMutation, client: Any, + *, + acp_permission_router: AcpPermissionDecisionRouter | None = None, + acp_handoff: bool = False, ) -> CommandEnvelope: assert config.db_path is not None claim = _claim_pending_decision(config, request, validated) @@ -2153,6 +2184,18 @@ def _answer_decision( if claim.status == STATUS_ANSWER_IN_PROGRESS: _abandon_request_reservation(config, request, reservation) return _answer_in_progress(request, receipt_reserved=True) + if claim.status == STATUS_BACKEND_UNAVAILABLE: + if _abandon_request_reservation(config, request, reservation): + return claim + return _finish_before_send( + config, + request, + reservation, + _backend_uncertain( + request, + "ACP permission reservation state is uncertain", + ), + ) return _finish_before_send(config, request, reservation, claim) claim_token = claim.claim_token @@ -2212,12 +2255,20 @@ def _answer_decision( ) try: - _submit_decision_calibration( - client, - started.turn_target_value.strip(), - started, - timeout=config.herdr_timeout_seconds, - ) + if acp_handoff: + if acp_permission_router is None: + raise RuntimeError("ACP permission bridge is unavailable") + acp_permission_router.answer_permission_decision( + started, + timeout=config.acp_request_timeout_seconds, + ) + else: + _submit_decision_calibration( + client, + started.turn_target_value.strip(), + started, + timeout=config.herdr_timeout_seconds, + ) except Exception: # noqa: BLE001 return _finish_request( config, @@ -2225,7 +2276,7 @@ def _answer_decision( reservation, _backend_uncertain( request, - "Herdr decision input state is uncertain after send start", + "permission decision state is uncertain after send start", ), expected_state="send_started", terminal_state="uncertain", @@ -2818,6 +2869,7 @@ def _submit_command_v2( params: Mapping[str, Any] | str, *, socket_client_factory: SocketClientFactory | None = None, + acp_permission_router: AcpPermissionDecisionRouter | None = None, ) -> CommandEnvelope: """Submit one command through the authoritative daemon/socket path.""" payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) @@ -2983,9 +3035,33 @@ def _submit_command_v2( # claim is released or expires. return _answer_in_progress(request, receipt_reserved=True) + acp_handoff = False + if answer_pre_send is None and request.action == "answer_decision": + acp_binding = _decision_uses_acp_binding(config, validated) + if acp_binding: + try: + acp_handoff = bool( + acp_permission_router is not None + and acp_permission_router.owns_permission_decision(validated) + ) + except Exception: + acp_handoff = False + if not acp_handoff: + unavailable = _backend_unavailable( + request, + "ACP permission authority is temporarily unavailable", + ) + if takeover is not None: + return _request_in_progress(request) + return unavailable + client_or_error: Any | CommandEnvelope | None = None if answer_pre_send is None and health_error is None: - client_or_error = _connect_socket(config, request, socket_client_factory) + client_or_error = ( + object() + if acp_handoff + else _connect_socket(config, request, socket_client_factory) + ) if isinstance(client_or_error, CommandEnvelope): # The socket could not be reached before any transmission -> safe # transient. Stay retryable rather than reserving a durable rejection. @@ -3020,6 +3096,8 @@ def _submit_command_v2( validated, reservation, client_or_error, + acp_permission_router=acp_permission_router, + acp_handoff=acp_handoff, ) return _answer_pending( config, @@ -3096,6 +3174,7 @@ def submit_command( socket_client_factory: SocketClientFactory | None = None, acp_prompt_router: AcpPromptRouter | None = None, acp_required: bool = False, + acp_permission_router: AcpPermissionDecisionRouter | None = None, ) -> CommandEnvelope: """Submit one command and apply optional response-envelope negotiation.""" if acp_prompt_router is not None: @@ -3123,6 +3202,7 @@ def submit_command( config, params, socket_client_factory=socket_client_factory, + acp_permission_router=acp_permission_router, ) payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) request, parse_error = parse_command_request(payload) diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index b607c40..8ef99d1 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -1449,18 +1449,27 @@ def submit_command(self, params: Mapping[str, Any]) -> CommandEnvelope | Mapping policy = self.config.agent_event_source runtime = self._acp_runtime route = getattr(runtime, "prompt_route", None) + permission_router = ( + runtime + if callable(getattr(runtime, "answer_permission_decision", None)) + else None + ) if policy == "acp_required" or ( policy == "acp_preferred" and callable(route) - ): + ) or permission_router is not None: from .command_submission import submit_command return submit_command( self.config, payload, acp_prompt_router=( - route if callable(route) else lambda _worker: None + route + if policy in {"acp_required", "acp_preferred"} + and callable(route) + else None ), acp_required=policy == "acp_required", + acp_permission_router=permission_router, ) return self.hooks.submit_command(self.config, payload) diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index c335b79..ee9b932 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -145,7 +145,7 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS -STORE_SCHEMA_VERSION = 26 +STORE_SCHEMA_VERSION = 27 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS _LEGACY_TURN_CLAIM_HARD_TTL_SECONDS = 86_400 TURN_CHANGE_RETENTION_DAYS = 7 @@ -408,6 +408,7 @@ class BackendPendingDecisionClaim: "decision_not_pending", "invalid_selection", "unsupported_decision", + "acp_authority_unavailable", "already_claimed", ] claim_token: str | None = None @@ -420,6 +421,7 @@ class BackendPendingDecisionClaim: option_count: int | None = None option_refs: tuple[str, ...] = () text: str | None = None + route_kind: Literal["legacy", "acp_permission"] = "legacy" @dataclass(frozen=True) @@ -441,6 +443,7 @@ class BackendPendingDecisionSend: option_count: int | None = None option_refs: tuple[str, ...] = () text: str | None = None + route_kind: Literal["legacy", "acp_permission"] = "legacy" @dataclass(frozen=True) @@ -13531,6 +13534,39 @@ def _migrate_v25_to_v26_conn(conn: sqlite3.Connection) -> None: ) +def _migrate_v26_to_v27_conn(conn: sqlite3.Connection) -> None: + """Persist private pending-decision transport provenance.""" + pending_columns = _table_columns(conn, "backend_pending") + required_pending_columns = { + "revision_digest", + "choice_routes_json", + "binding_private_fingerprint", + "observed_turn_target_value", + "observation_state", + "freshness", + "updated_at", + } + if not required_pending_columns.issubset(pending_columns): + # Some old fixture databases intentionally contain only the v11 + # command tables. Reconstruct this unrelated family defensively. + _migrate_v9_to_v10_conn(conn) + pending_columns = _table_columns(conn, "backend_pending") + if not _table_columns(conn, "backend_pending_claims"): + conn.execute(CREATE_BACKEND_PENDING_CLAIMS_TABLE) + if "route_kind" not in pending_columns: + conn.execute( + "ALTER TABLE backend_pending ADD COLUMN route_kind " + "TEXT NOT NULL DEFAULT 'legacy' " + "CHECK (route_kind IN ('legacy', 'acp_permission'))" + ) + if "route_kind" not in _table_columns(conn, "backend_pending_claims"): + conn.execute( + "ALTER TABLE backend_pending_claims ADD COLUMN route_kind " + "TEXT NOT NULL DEFAULT 'legacy' " + "CHECK (route_kind IN ('legacy', 'acp_permission'))" + ) + + MIGRATIONS: tuple[Migration, ...] = ( Migration(0, 1, _migrate_v0_to_v1_conn), Migration(1, 2, _migrate_v1_to_v2_conn), @@ -13558,6 +13594,7 @@ def _migrate_v25_to_v26_conn(conn: sqlite3.Connection) -> None: Migration(23, 24, _migrate_v23_to_v24_conn), Migration(24, 25, _migrate_v24_to_v25_conn), Migration(25, 26, _migrate_v25_to_v26_conn), + Migration(26, 27, _migrate_v26_to_v27_conn), ) @@ -13594,6 +13631,7 @@ def _create_current_schema_conn(conn: sqlite3.Connection) -> None: conn.execute(statement) conn.execute(CREATE_LEGACY_BACKEND_PENDING_TABLE) _migrate_v9_to_v10_conn(conn) + _migrate_v26_to_v27_conn(conn) conn.execute(CREATE_ATTENTION_LIFECYCLES_TABLE) conn.execute(CREATE_TURN_CONTENT_REVISIONS_TABLE) conn.execute(CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE) @@ -21037,10 +21075,14 @@ def apply_backend_pending_observation( stale_grace_seconds: float = DEFAULT_PENDING_STALE_GRACE_SECONDS, binding_private_fingerprint: str | None = None, observed_turn_target_value: str | None = None, + binding_authoritative: bool = False, + route_kind: Literal["legacy", "acp_permission"] = "legacy", ) -> bool: """Apply one explicit observation in a short writer transaction.""" if not _sqlite_store_exists(db_path): return False + if route_kind not in {"legacy", "acp_permission"}: + raise ValueError("invalid backend pending route kind") current_time, _ = _pending_observed_time(observed_at) with _connect(db_path, isolation_level=None) as conn: _ensure_schema(conn) @@ -21059,6 +21101,20 @@ def apply_backend_pending_observation( observed_turn_target_value=str( observed_turn_target_value or "" ), + binding_authoritative=bool(binding_authoritative), + ) + conn.execute( + "UPDATE backend_pending SET route_kind = ? " + "WHERE host_id = ? AND worker_id = ? " + "AND binding_private_fingerprint = ? " + "AND observed_turn_target_value = ?", + ( + route_kind, + str(host_id), + str(worker_id), + str(binding_private_fingerprint or ""), + str(observed_turn_target_value or ""), + ), ) conn.commit() return changed @@ -21441,9 +21497,10 @@ def _backend_pending_claim_context_conn( SELECT private_fingerprint, worker_fingerprint, turn_target_value FROM worker_bindings WHERE host_id = ? AND worker_id = ? AND worker_fingerprint = ? - AND backend = 'herdr' AND turn_target_kind = 'pane_id' + AND ((backend = 'herdr' AND turn_target_kind = 'pane_id') + OR (backend = 'acp' AND turn_target_kind = 'acp_session_id')) AND private_fingerprint = ? AND turn_target_value = ? - AND sendable = 1 AND expires_at > ? + AND sendable = 1 AND (expires_at IS NULL OR expires_at > ?) """, ( str(host_id), @@ -21756,7 +21813,8 @@ def claim_backend_pending_decision( """ SELECT payload_json, choice_routes_json, revision_digest, freshness, binding_private_fingerprint, - observed_turn_target_value, observation_state + observed_turn_target_value, observation_state, + route_kind FROM backend_pending WHERE host_id = ? AND worker_id = ? """, @@ -21785,13 +21843,16 @@ def claim_backend_pending_decision( observed_at=current_time, ) decision = _decision_from_pending_row(row[0], row[1], row[2]) - if ( - context is None - or decision is None - or decision.get("decision_ref") != str(decision_ref) - ): + if decision is None or decision.get("decision_ref") != str(decision_ref): conn.rollback() return BackendPendingDecisionClaim("decision_not_pending") + if context is None: + conn.rollback() + return BackendPendingDecisionClaim( + "acp_authority_unavailable" + if str(row[7]) == "acp_permission" + else "decision_not_pending" + ) if int(decision["question_count"]) > 1: conn.rollback() return BackendPendingDecisionClaim("unsupported_decision") @@ -21839,6 +21900,7 @@ def claim_backend_pending_decision( "option_count": option_count, "option_refs": option_refs, "text": text, + "route_kind": str(row[7]), } if not claim: conn.rollback() @@ -21850,13 +21912,14 @@ def claim_backend_pending_decision( host_id, worker_id, claim_token, revision_digest, choice_id, picker_ordinal, worker_fingerprint, binding_private_fingerprint, turn_target_value, state, - claimed_at, send_started_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'claimed', ?, NULL) + claimed_at, send_started_at, route_kind + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'claimed', ?, NULL, ?) """, ( str(host_id), str(worker_id), token, str(row[2]), _encode_decision_claim_selection(option_refs, text), picker_ordinal, context[2], context[1], context[3], current_time, + str(row[7]), ), ) conn.commit() @@ -21888,7 +21951,7 @@ def start_backend_pending_decision_send( """ SELECT worker_id, revision_digest, choice_id, worker_fingerprint, binding_private_fingerprint, - turn_target_value, state, claimed_at + turn_target_value, state, claimed_at, route_kind FROM backend_pending_claims WHERE host_id = ? AND claim_token = ? """, @@ -21916,7 +21979,7 @@ def start_backend_pending_decision_send( """ SELECT payload_json, choice_routes_json, revision_digest, freshness, binding_private_fingerprint, - observed_turn_target_value + observed_turn_target_value, route_kind FROM backend_pending WHERE host_id = ? AND worker_id = ? AND observation_state = 'open' @@ -21940,6 +22003,7 @@ def start_backend_pending_decision_send( str(current[2]) != str(row[1]) or str(current[4]) != str(row[4]) or str(current[5]) != str(row[5]) + or str(current[6]) != str(row[8]) or decision is None or validated_selection is None ): @@ -21956,6 +22020,7 @@ def start_backend_pending_decision_send( "option_count": len(decision["options"]), "option_refs": option_refs, "text": text, + "route_kind": str(row[8]), } if str(row[6]) == "send_started": conn.rollback() diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 498c5d2..f036b8e 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -14,7 +14,6 @@ from tendwire.backends.acp_coordinator import ( AcpCoordinatorError, - AcpPermissionBridgeUnavailable, AcpRuntimeCoordinator, _derived_binding, _parse_endpoint, @@ -758,7 +757,7 @@ def test_coordinator_start_revokes_orphaned_process_binding(tmp_path: Path) -> N coordinator.stop() -def test_production_coordinator_fails_closed_without_permission_bridge( +def test_production_coordinator_installs_durable_permission_bridge( tmp_path: Path, ) -> None: config = _config(tmp_path) @@ -766,16 +765,12 @@ def test_production_coordinator_fails_closed_without_permission_bridge( init_store(config.db_path) coordinator = production_acp_runtime_factory(config, threading.Event()) - with pytest.raises( - AcpPermissionBridgeUnavailable, - match="durable ACP permission decisions", - ): - coordinator.start() - - status = coordinator.status() - assert status["healthy"] is False - assert status["state"] == "failed" - assert status["failure_type"] == "AcpPermissionBridgeUnavailable" + coordinator.start() + try: + assert coordinator.status()["state"] == "running" + assert coordinator._durable_permission_bridge is True + finally: + coordinator.stop() def test_coordinator_forwards_explicit_permission_bridge(tmp_path: Path) -> None: diff --git a/tests/test_acp_permissions.py b/tests/test_acp_permissions.py new file mode 100644 index 0000000..4ec50a7 --- /dev/null +++ b/tests/test_acp_permissions.py @@ -0,0 +1,330 @@ +"""End-to-end durable ACP permission decision bridge tests.""" + +from __future__ import annotations + +import json +import sqlite3 +import threading +import time +from concurrent.futures import ThreadPoolExecutor +from dataclasses import replace +from pathlib import Path +from typing import Any + +import pytest +import tendwire.store.sqlite as store_sqlite +import tendwire.command_submission as command_submission +from tendwire.backends.acp_permissions import AcpPermissionBroker +from tendwire.backends.acp_protocol import ( + PermissionOption, + PermissionOptionKind, + PermissionRequest, + SessionResult, +) +from tendwire.backends.acp_runtime import AcpRuntime, SessionOpenMode +from tendwire.command_submission import submit_command +from tendwire.core.models import Worker +from tendwire.daemon import TendwireDaemon +from tendwire.store.sqlite import ( + expire_worker_bindings, + get_command_request, + list_worker_bindings, + pending_payload_from_store, + upsert_worker_bindings, +) + +from tests.test_answer_decision import _answer_request +from tests.test_command_submission import _binding, _config, _seed +from tests.test_acp_runtime import FakeClient, FakeIngestor + + +class _Router: + def __init__(self, broker: AcpPermissionBroker) -> None: + self.broker = broker + + def owns_permission_decision(self, decision: Any) -> bool: + return self.broker.owns(decision) + + def answer_permission_decision(self, decision: Any, *, timeout: float) -> None: + self.broker.answer(decision, timeout=timeout) + + +def _permission(session_id: str) -> PermissionRequest: + options = ( + PermissionOption( + "private-allow-id", + "Allow once", + PermissionOptionKind.ALLOW_ONCE, + {}, + ), + PermissionOption( + "private-reject-id", + "Reject", + PermissionOptionKind.REJECT_ONCE, + {}, + ), + ) + return PermissionRequest( + 71, + session_id, + { + "toolCallId": "private-tool-id", + "title": "Run tests", + "rawInput": "/secret", + }, + options, + {"private": "metadata"}, + {}, + ) + + +def _wait_pending(config: Any, worker_id: str) -> dict[str, Any]: + assert config.db_path is not None + deadline = time.monotonic() + 2 + while time.monotonic() < deadline: + payload = pending_payload_from_store(config.db_path, config.host_id) + rows = [ + row + for row in payload["pending_interactions"] + if row["worker_id"] == worker_id + ] + if rows: + return rows[0] + time.sleep(0.01) + raise AssertionError("permission overlay was not published") + + +def _setup(tmp_path: Path) -> tuple[Any, Worker, str, AcpPermissionBroker]: + config = _config(tmp_path) + worker = Worker(id="w-1", name="Alpha", status="active") + continuity = _binding(worker) + _seed(config, [worker], [continuity]) + session_id = "private-session-id" + acp_binding = replace( + continuity, + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="", + ) + assert config.db_path is not None + upsert_worker_bindings(config.db_path, [acp_binding]) + return ( + config, + worker, + session_id, + AcpPermissionBroker( + config, + worker_id=worker.id, + worker_fingerprint=worker.fingerprint, + generation="42", + timeout=2, + ), + ) + + +def test_permission_bridge_is_private_durable_and_frame_acknowledged( + tmp_path: Path, +) -> None: + config, worker, session_id, broker = _setup(tmp_path) + assert config.db_path is not None + acp_binding = list_worker_bindings( + config.db_path, config.host_id, backend="acp" + )[0] + client = FakeClient() + client.restored_session_result = SessionResult(session_id, None, (), {}) + runtime = AcpRuntime( + client, + config=config, + binding=acp_binding, + cwd=tmp_path, + session_mode=SessionOpenMode.LOAD, + session_id=session_id, + stream_generation="42", + permission_callback=broker, + ingestor=FakeIngestor(session_id), + poll_timeout=0.01, + stop_timeout=1, + ) + runtime.start() + try: + client.permissions.put(_permission(session_id)) + pending = _wait_pending(config, worker.id) + serialized = json.dumps(pending, sort_keys=True) + assert pending["question"] == "Run tests" + assert pending["meta"]["decision"]["options"] == [ + {"ref": "1", "label": "Allow once (allow_once)"}, + {"ref": "2", "label": "Reject (reject_once)"}, + ] + for secret in ( + "private-allow-id", + "private-reject-id", + "private-tool-id", + session_id, + "/secret", + ): + assert secret not in serialized + + result = submit_command( + config, + _answer_request( + pending["meta"]["decision"]["decision_ref"], + selection={"option_refs": ["1"]}, + ), + acp_permission_router=_Router(broker), + ) + assert result.ok is True + assert client.permission_responses == [(71, "private-allow-id", False)] + receipt = get_command_request( + config.db_path, config.host_id, "decision-request-1" + ) + assert receipt is not None and receipt["state"] == "accepted" + finally: + broker.close() + runtime.stop() + + +def test_acp_permission_never_falls_back_to_legacy_socket(tmp_path: Path) -> None: + config, worker, session_id, broker = _setup(tmp_path) + thread = threading.Thread(target=lambda: broker(_permission(session_id))) + thread.start() + pending = _wait_pending(config, worker.id) + assert config.db_path is not None + assert expire_worker_bindings( + config.db_path, + config.host_id, + backend="acp", + private_fingerprints=[ + row.private_fingerprint + for row in list_worker_bindings( + config.db_path, config.host_id, backend="acp" + ) + ], + reason="test_retired_before_answer", + ) == 1 + socket_calls: list[bool] = [] + result = submit_command( + config, + _answer_request(pending["meta"]["decision"]["decision_ref"]), + socket_client_factory=lambda _config: socket_calls.append(True), + acp_permission_router=None, + ) + assert result.ok is False + assert result.disposition == "no_receipt" + assert socket_calls == [] + broker.close() + thread.join(timeout=2) + + +def test_concurrent_answers_write_exactly_one_permission_response( + tmp_path: Path, +) -> None: + config, worker, session_id, broker = _setup(tmp_path) + selections: list[Any] = [] + + def adapter_side() -> None: + selected = broker(_permission(session_id)) + selections.append(selected) + assert selected is not None + selected.response_written() + + adapter = threading.Thread(target=adapter_side) + adapter.start() + pending = _wait_pending(config, worker.id) + decision_ref = pending["meta"]["decision"]["decision_ref"] + router = _Router(broker) + requests = [ + _answer_request(decision_ref, request_id=f"concurrent-{index}") + for index in range(2) + ] + with ThreadPoolExecutor(max_workers=2) as pool: + results = list( + pool.map( + lambda request: submit_command( + config, + request, + acp_permission_router=router, + ), + requests, + ) + ) + adapter.join(timeout=2) + assert not adapter.is_alive() + assert sum(result.ok is True for result in results) == 1 + assert len(selections) == 1 + assert selections[0].option_id in { + "private-allow-id", + "private-reject-id", + } + + +def test_broker_stop_cancels_waiter_and_closes_public_overlay(tmp_path: Path) -> None: + config, worker, session_id, broker = _setup(tmp_path) + outcomes: list[Any] = [] + thread = threading.Thread( + target=lambda: outcomes.append(broker(_permission(session_id))) + ) + thread.start() + _wait_pending(config, worker.id) + broker.close() + thread.join(timeout=2) + assert not thread.is_alive() + assert outcomes == [None] + assert config.db_path is not None + payload = pending_payload_from_store(config.db_path, config.host_id) + assert all( + row["worker_id"] != worker.id + for row in payload["pending_interactions"] + ) + + +def test_v27_provenance_migration_preserves_stale_pending_state( + tmp_path: Path, +) -> None: + db_path = tmp_path / "v26.db" + with sqlite3.connect(db_path) as conn: + store_sqlite._run_migrations(conn, target_version=26) + conn.execute( + """ + INSERT INTO backend_pending ( + host_id, worker_id, payload_json, observed_at, + revision_digest, choice_routes_json, + binding_private_fingerprint, observed_turn_target_value, + observation_state, freshness, updated_at + ) VALUES ('host', 'worker', '{}', '2026-01-01T00:00:00+00:00', + '', '{}', '', '', 'failed', 'stale', + '2026-01-01T00:00:00+00:00') + """ + ) + conn.commit() + store_sqlite.init_store(db_path) + with sqlite3.connect(db_path) as conn: + assert conn.execute("PRAGMA user_version").fetchone() == (27,) + assert conn.execute( + "SELECT freshness, route_kind FROM backend_pending" + ).fetchone() == ("stale", "legacy") + + +def test_shadow_daemon_routes_permission_answers_without_enabling_acp_prompts( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = replace(_config(tmp_path), agent_event_source="acp_shadow") + + class Runtime: + def answer_permission_decision(self, _decision: Any, *, timeout: float) -> None: + del timeout + + runtime = Runtime() + daemon = TendwireDaemon(config) + daemon._acp_runtime = runtime + captured: dict[str, Any] = {} + + def submit(_config: Any, _payload: Any, **kwargs: Any) -> str: + captured.update(kwargs) + return "routed" + + monkeypatch.setattr(command_submission, "submit_command", submit) + assert daemon.submit_command({"schema_version": 1, "action": "noop"}) == "routed" + assert captured["acp_permission_router"] is runtime + assert captured["acp_prompt_router"] is None diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index 6a65756..d38c5df 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -33,7 +33,7 @@ SessionOpenMode, ) from tendwire.config import Config -from tendwire.core.models import WorkerBinding +from tendwire.core.models import WorkerBinding, utc_timestamp from tendwire.store.sqlite import ( expire_stale_worker_bindings, expire_worker_bindings, @@ -917,7 +917,7 @@ def destructive(session_id: str, anchor: WorkerBinding) -> WorkerBinding: anchor, worker_id="replacement-worker-private", worker_fingerprint="replacement-fingerprint-private", - observed_at="2026-08-01T00:00:00+00:00", + observed_at=utc_timestamp(), ) ], ) @@ -1071,7 +1071,7 @@ def unsafe_allow(_request: PermissionRequest) -> str: current, worker_id="replacement-worker-private", worker_fingerprint="replacement-fingerprint-private", - observed_at="2026-08-01T00:00:00+00:00", + observed_at=utc_timestamp(), ) upsert_worker_bindings(tmp_path / "bound-events.db", [replacement]) client.permissions.put(permission()) diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py index 8bb905c..254ef85 100644 --- a/tests/test_agent_events.py +++ b/tests/test_agent_events.py @@ -1219,7 +1219,7 @@ def test_v25_to_v26_retains_legacy_tombstones_as_dedup_only(tmp_path: Path) -> N ) conn.commit() store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == (26,) + assert conn.execute("PRAGMA user_version").fetchone() == (27,) assert conn.execute( "SELECT observed_at FROM agent_event_tombstones WHERE event_id = ?", (legacy_event_id,), diff --git a/tests/test_backend_pending.py b/tests/test_backend_pending.py index 5edb76c..245b12d 100644 --- a/tests/test_backend_pending.py +++ b/tests/test_backend_pending.py @@ -1449,7 +1449,7 @@ def test_current_schema_creation_has_exact_binding_and_claim_state(tmp_path: Pat db = tmp_path / "current-schema.db" init_store(db) with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 26 + assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 27 columns = { str(row[1]) for row in conn.execute("PRAGMA table_info(backend_pending)").fetchall() diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index a2c698a..5a9f1c7 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -1754,7 +1754,7 @@ def test_v6_to_current_plan_migration_is_bounded_atomic_and_preserves_jobs( ).fetchall() } foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert version == store_sqlite.STORE_SCHEMA_VERSION == 26 + assert version == store_sqlite.STORE_SCHEMA_VERSION == 27 assert plan_row == (plan["plan_token"], 1, None, "active") assert job_count == 2 assert outbox_count == 3 diff --git a/tests/test_delivery_retention_migration.py b/tests/test_delivery_retention_migration.py index 675af0f..0a52d02 100644 --- a/tests/test_delivery_retention_migration.py +++ b/tests/test_delivery_retention_migration.py @@ -875,7 +875,7 @@ def test_v10_to_v12_migration_retains_finals_without_reposting_or_leaking( finals = _seed_v10_finals(db_path) init_store(db_path) - assert store_sqlite.STORE_SCHEMA_VERSION == 26 + assert store_sqlite.STORE_SCHEMA_VERSION == 27 delivered_key = _final_key(*finals["delivered"]) hold_keys = {_final_key(*finals[label]) for label in ("hold-a", "hold-b")} diff --git a/tests/test_delivery_retention_projection.py b/tests/test_delivery_retention_projection.py index 7e6d35b..102fa24 100644 --- a/tests/test_delivery_retention_projection.py +++ b/tests/test_delivery_retention_projection.py @@ -147,7 +147,7 @@ def _assert_continuity_integrity(db_path: Path) -> None: with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA user_version").fetchone() == ( store_sqlite.STORE_SCHEMA_VERSION, - ) == (26,) + ) == (27,) assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] current_counts = conn.execute( """ diff --git a/tests/test_delivery_retention_recovery.py b/tests/test_delivery_retention_recovery.py index 48bcac1..15f430a 100644 --- a/tests/test_delivery_retention_recovery.py +++ b/tests/test_delivery_retention_recovery.py @@ -973,7 +973,7 @@ def test_v12_migration_uses_effective_recovery_lineage_without_repost_or_hold( api = ConnectorOutboxAPI(db_path, HOST_ID) assert api.poll({"name": FINAL_NAME, "limit": 100})["items"] == [] with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 26 + assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 27 anchor = conn.execute( """ SELECT delivery_kind, status diff --git a/tests/test_store.py b/tests/test_store.py index 5f9deac..d38db29 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -10278,7 +10278,7 @@ def test_v13_migration_repairs_nonpositive_turn_sequences_and_blocks_recurrence( init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 26 + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 27 assert conn.execute( """ SELECT turn_id, list_sequence @@ -14151,7 +14151,7 @@ def test_v20_to_v21_adds_herdr_turn_watermark_and_provenance_tables( with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA user_version").fetchone() == ( store_sqlite.STORE_SCHEMA_VERSION, - ) == (26,) + ) == (27,) assert { str(row[0]) for row in conn.execute( From b1eaebebde2d67575bca4bafc7e8569762fd1084 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 12:25:13 +0800 Subject: [PATCH 38/83] fix(acp): bound permission shutdown cleanup --- src/tendwire/backends/acp_coordinator.py | 12 ++++ src/tendwire/backends/acp_permissions.py | 24 ++++++- tests/test_acp_coordinator.py | 86 ++++++++++++++++++++++++ tests/test_acp_permissions.py | 74 ++++++++++++++++++++ 4 files changed, 194 insertions(+), 2 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 011aca1..3baa7de 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -209,7 +209,19 @@ def stop(self, *, timeout: float | None = None) -> None: if self._state is RuntimeState.STOPPED: return self._state = RuntimeState.STOPPING + slots = tuple(self._slots.values()) self._stop.set() + # A durable permission answer keeps the generation fence until the + # complete JSON-RPC response frame is written. Wake any broker waiters + # before waiting for that fence: otherwise a slow/stuck adapter write + # can hold ``_reconcile_lock`` for the request timeout (normally much + # longer than the coordinator's bounded shutdown deadline). Closing a + # broker is fail-closed; an answer that has not observed a complete + # frame becomes uncertain and releases the fence without selecting a + # second route. + for slot in slots: + if slot.permission_broker is not None: + slot.permission_broker.close() deadline = time.monotonic() + limit # Wait for endpoint mint/start or prompt lease validation to leave its # critical section before clearing slots. A reconcile that observed diff --git a/src/tendwire/backends/acp_permissions.py b/src/tendwire/backends/acp_permissions.py index 83cf82f..568eb91 100644 --- a/src/tendwire/backends/acp_permissions.py +++ b/src/tendwire/backends/acp_permissions.py @@ -38,6 +38,7 @@ class _Offer: selected: int | None = None response_state: str = "pending" response_error: BaseException | None = None + answer_abandoned: bool = False class AcpPermissionBroker: @@ -186,6 +187,8 @@ def owns(self, decision: Any) -> bool: ) def answer(self, decision: Any, *, timeout: float) -> None: + clear_failed = False + uncertain = False with self._lock: offer = self._offer if offer is None or not self.owns(decision): @@ -204,8 +207,21 @@ def answer(self, decision: Any, *, timeout: float) -> None: break offer.condition.wait(remaining) if offer.response_state != "written": - raise AcpPermissionBrokerError("ACP permission response state is uncertain") - self._offer = None + # The command receipt is now terminally uncertain. A late + # transport callback must retire the stale public overlay and + # its send_started claim, but only after it proves that the + # response frame was written (or definitively failed). Until + # then the offer remains fail-closed and cannot be answered a + # second time. + offer.answer_abandoned = True + clear_failed = offer.response_state == "failed" + uncertain = True + else: + self._offer = None + if clear_failed: + self._clear_offer(offer) + if uncertain: + raise AcpPermissionBrokerError("ACP permission response state is uncertain") def close(self) -> None: with self._lock: @@ -217,11 +233,15 @@ def close(self) -> None: self._clear_offer(offer) def _response_complete(self, offer: _Offer, error: BaseException | None) -> None: + clear = False with offer.condition: if self._offer is offer: offer.response_state = "failed" if error is not None else "written" offer.response_error = error + clear = error is not None or offer.answer_abandoned offer.condition.notify_all() + if clear: + self._clear_offer(offer) def _clear_offer(self, offer: _Offer) -> None: with self._lock: diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index f036b8e..66accd4 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -4,6 +4,7 @@ import json import threading +import time from concurrent.futures import ThreadPoolExecutor from dataclasses import replace from pathlib import Path @@ -602,6 +603,91 @@ def stop() -> None: assert runtimes and all(runtime.stopped for runtime in runtimes) +def test_stop_closes_permission_waiter_before_generation_fence_deadline( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + answer_entered = threading.Event() + broker_closed = threading.Event() + runtime_stopped = threading.Event() + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + return _endpoint() + + def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: + return _status() + + def close(self) -> None: + return None + + class Broker: + def owns(self, _decision: Any) -> bool: + return True + + def answer(self, _decision: Any, *, timeout: float) -> None: + del timeout + answer_entered.set() + assert broker_closed.wait(2.0) + raise AcpCoordinatorError("permission bridge closed") + + def close(self) -> None: + broker_closed.set() + + broker = Broker() + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self._binder = kwargs["session_binding_callback"] + + def start(self) -> None: + if self._binder is not None: + self._binding = self._binder("session-private", self._binding) + + def stop(self, *, timeout: float) -> None: + del timeout + runtime_stopped.set() + + def status(self) -> Any: + return SimpleNamespace( + healthy=not runtime_stopped.is_set(), + failure_type=None, + ) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + client_factory=lambda *_args, **_kwargs: object(), + runtime_factory=Runtime, + durable_permission_bridge=True, + reconcile_interval=60.0, + ).start() + slot = coordinator._slots["worker-1"] + slot.permission_broker = broker # type: ignore[assignment] + decision = SimpleNamespace(worker_id="worker-1") + + with ThreadPoolExecutor(max_workers=2) as pool: + answer = pool.submit( + coordinator.answer_permission_decision, + decision, + timeout=30.0, + ) + assert answer_entered.wait(1.0) + started = time.monotonic() + coordinator.stop(timeout=0.5) + assert time.monotonic() - started < 0.5 + with pytest.raises(AcpCoordinatorError, match="closed"): + answer.result(timeout=1.0) + + assert broker_closed.is_set() + assert runtime_stopped.is_set() + + def test_prompt_frame_acknowledgement_fences_generation_retirement(tmp_path: Path) -> None: config = _config(tmp_path) assert config.db_path is not None diff --git a/tests/test_acp_permissions.py b/tests/test_acp_permissions.py index 4ec50a7..dfc844a 100644 --- a/tests/test_acp_permissions.py +++ b/tests/test_acp_permissions.py @@ -278,6 +278,80 @@ def test_broker_stop_cancels_waiter_and_closes_public_overlay(tmp_path: Path) -> ) +def test_late_permission_frame_completion_retires_uncertain_overlay( + tmp_path: Path, +) -> None: + config, worker, session_id, broker = _setup(tmp_path) + selections: list[Any] = [] + + def adapter_side() -> None: + selections.append(broker(_permission(session_id))) + + adapter = threading.Thread(target=adapter_side) + adapter.start() + pending = _wait_pending(config, worker.id) + router = _Router(broker) + result = submit_command( + replace(config, acp_request_timeout_seconds=0.01), + _answer_request(pending["meta"]["decision"]["decision_ref"]), + acp_permission_router=router, + ) + adapter.join(timeout=2) + assert not adapter.is_alive() + assert len(selections) == 1 and selections[0] is not None + assert result.status == "request_state_uncertain" + + # The JSON-RPC writer completes after the durable command deadline. The + # original receipt remains uncertain and cannot be replayed, while the + # now-resolved permission prompt is removed instead of becoming a forever + # visible, forever-unanswerable overlay. + selections[0].response_written() + assert config.db_path is not None + payload = pending_payload_from_store(config.db_path, config.host_id) + assert all( + row["worker_id"] != worker.id + for row in payload["pending_interactions"] + ) + receipt = get_command_request( + config.db_path, + config.host_id, + "decision-request-1", + ) + assert receipt is not None and receipt["state"] == "uncertain" + broker.close() + + +def test_failed_permission_frame_retires_overlay_without_retry( + tmp_path: Path, +) -> None: + config, worker, session_id, broker = _setup(tmp_path) + + def adapter_side() -> None: + selected = broker(_permission(session_id)) + assert selected is not None + selected.response_failed(OSError("partial private frame")) + + adapter = threading.Thread(target=adapter_side) + adapter.start() + pending = _wait_pending(config, worker.id) + result = submit_command( + config, + _answer_request(pending["meta"]["decision"]["decision_ref"]), + acp_permission_router=_Router(broker), + ) + adapter.join(timeout=2) + assert not adapter.is_alive() + assert result.status == "request_state_uncertain" + assert "private frame" not in json.dumps(result.to_dict()) + assert config.db_path is not None + payload = pending_payload_from_store(config.db_path, config.host_id) + assert all( + row["worker_id"] != worker.id + for row in payload["pending_interactions"] + ) + broker.close() + + def test_v27_provenance_migration_preserves_stale_pending_state( tmp_path: Path, ) -> None: From 2692e2dfb3a9d9a86fa5dc50d388d65d73dd36e7 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 12:43:50 +0800 Subject: [PATCH 39/83] docs(acp): update production rollout gates --- docs/acp-migration.md | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/docs/acp-migration.md b/docs/acp-migration.md index d3ccb4e..f43529a 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -3,9 +3,9 @@ This document defines the experimental migration from backend-specific transcript readers to Agent Client Protocol (ACP). ACP is not yet Tendwire's default. The stock daemon contains the coordinator and command path, but -production ACP activation is currently fail-closed until a durable permission -decision bridge is configured. The coordinator can otherwise attach only an -explicitly Herdr-owned ACP worker. +production ACP activation remains operator-gated on installing a supported ACP +adapter and explicitly registering shell-only panes as Herdr ACP-owned workers. +The coordinator never attaches an ordinary PTY worker as a sidecar. Herdr remains authoritative for workspace, pane, worker identity, process liveness, and command routing until the ACP control path is proven separately. Tendwire remains authoritative for persistence, reconciliation, public safety, @@ -34,13 +34,16 @@ one-shot private endpoint, validates its worker generation and explicit ordinary live PTY session is never treated as ACP-owned. ACP `session/request_permission` is synchronous and can authorize destructive -tools. Tendwire does not yet have the required durable worker/session-correlated +tools. The stock production factory enables a durable, worker/session-correlated bridge from a public `answer_decision` command back to the exact request and -offered `optionId`. The stock production factory therefore raises the redacted -`AcpPermissionBridgeUnavailable` startup failure for every ACP mode. It must not -silently cancel permissions while reporting the runtime healthy. Tests and -embedders may inject an explicit callback into the generic coordinator, but -that callback is not a production authorization surface. +offered `optionId`. It publishes only a sanitized tool title and numbered +choices; option IDs, arguments, ACP session IDs, adapter metadata, and raw tool +payloads remain private. Selection is fenced to the exact worker binding and +Herdr generation, and is accepted only after the complete JSON-RPC response +frame is written. Missing, stale, timed-out, or uncertain authority fails +closed without a second transport attempt. Embedders may instead inject an +explicit callback into the generic coordinator, but the stock daemon does not +depend on such a callback. ## Authority split @@ -205,9 +208,11 @@ the agent protocol. ## Rollout gates -Promotion remains blocked at the default `legacy` posture. When the missing -runtime and Herdr prerequisites exist, it may proceed `legacy` -> `acp_shadow` --> `acp_preferred`. The following +Promotion remains blocked at the default `legacy` posture until a supported ACP +adapter is installed, target panes are explicitly registered through Herdr's +ACP-owned lifecycle, and the integration is exercised against those real +adapters. Rollout may then proceed `legacy` -> `acp_shadow` -> `acp_preferred`. +The following must pass before `acp_required` is considered: - no missing or duplicated user/final messages across adapter restarts; @@ -221,7 +226,7 @@ must pass before `acp_required` is considered: The ACP runtime implements prompt submission, cancellation, fail-closed permission handling, per-worker coordination, reconnect, and receipt-backed -instruction routing. Interactive permission approval remains a runtime blocker: -until its durable bridge exists, the stock production factory refuses ACP -startup. ACP also remains non-default until the cross-repository integration -and rollout gates above pass against real adapters. +instruction routing, including durable interactive permission approval. ACP +remains non-default until supported adapters are installed, workers are +explicitly registered as ACP-owned, and the cross-repository rollout gates +above pass against real adapters. From d1c9f52cd1caafa3f2bba93876076d3e2e8a6286 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 13:04:48 +0800 Subject: [PATCH 40/83] fix(acp): make shadow ownership fail closed --- README.md | 9 +- docs/acp-migration.md | 21 ++-- src/tendwire/command_submission.py | 32 ++++- src/tendwire/daemon.py | 7 +- tests/test_acp_coordinator.py | 194 ++++++++++++++++++++++++++++- 5 files changed, 243 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 6489e6c..0230053 100644 --- a/README.md +++ b/README.md @@ -582,9 +582,12 @@ after the complete JSON-RPC permission-response frame is written. Missing or retired ACP authority fails closed without falling back to PTY input, and concurrent answers can produce at most one response. `acp_shadow` persists ACP events without projecting them, but no automated -shadow comparator is implemented. `acp_preferred` falls back only before an ACP -reservation/send, while `acp_required` fails closed and never starts the legacy -turn scheduler. None of these modes makes agent thoughts +shadow comparator is implemented. For ACP-owned workers it is observation-only: +legacy turn ingestion is excluded and commands fail `backend_unavailable` +without sending on either transport. Validate real adapter execution with an +isolated `acp_preferred` or `acp_required` canary. `acp_preferred` falls back +only before an ACP reservation/send, while `acp_required` fails closed and +never starts the legacy turn scheduler. None of these modes makes agent thoughts public: thought events remain private diagnostic data unless a separate, explicit sanitized projection is introduced. diff --git a/docs/acp-migration.md b/docs/acp-migration.md index f43529a..bfd7d86 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -16,9 +16,12 @@ command receipts, and connector delivery. `TENDWIRE_AGENT_EVENT_SOURCE` controls projection precedence: - `legacy`: use the existing Herdr/Codex/OMP turn readers only. -- `acp_shadow`: ingest ACP events - durably without projecting them; legacy turns remain authoritative. Automated - comparison is not implemented yet. +- `acp_shadow`: ingest ACP events durably without projecting them. Ordinary + legacy workers remain legacy-authoritative. ACP-owned workers are excluded + from legacy turn ingestion, and command submission to them fails closed with + `backend_unavailable`: shadow is observation-only and does not execute an + equivalent prompt on both transports. Automated comparison is not + implemented yet. - `acp_preferred`: use an explicitly Herdr-owned ACP endpoint when available; fall back to legacy only before any ACP command reservation or observable send. @@ -148,10 +151,10 @@ before every prompt and during reconciliation. The reported lifecycle must be A mismatch or unavailable status retires the slot before any prompt frame is written. Endpoint minting is never used as a status probe. -In `acp_preferred`, the legacy scheduler remains available only for workers not -currently owned by a healthy ACP slot. It rechecks this exclusion after dequeue -and immediately before a legacy read, preventing queued legacy work from -overwriting or duplicating the active ACP worker projection. +In `acp_shadow` and `acp_preferred`, the legacy scheduler remains available only +for workers not currently owned by a healthy ACP slot. It rechecks this +exclusion after dequeue and immediately before a legacy read, preventing queued +legacy work from overwriting or duplicating the active ACP worker projection. Disconnect handling is conservative: @@ -212,7 +215,9 @@ Promotion remains blocked at the default `legacy` posture until a supported ACP adapter is installed, target panes are explicitly registered through Herdr's ACP-owned lifecycle, and the integration is exercised against those real adapters. Rollout may then proceed `legacy` -> `acp_shadow` -> `acp_preferred`. -The following +Use an isolated `acp_preferred` (or stricter `acp_required`) canary for real +adapter prompt validation; `acp_shadow` intentionally does not pretend to +compare equivalent executed traffic. The following must pass before `acp_required` is considered: - no missing or duplicated user/final messages across adapter restarts; diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 5a55108..eadcfec 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -2687,6 +2687,7 @@ def submit_acp_command( *, prompt_router: AcpPromptRouter, required: bool = False, + observation_only: bool = False, ) -> CommandEnvelope | None: """Submit ``send_instruction`` through a live ACP worker route. @@ -2694,6 +2695,10 @@ def submit_acp_command( may safely use the legacy Herdr sender. Once a receipt reaches ``send_started``, every failure is terminally uncertain and this function never permits a second transport attempt. + + In observation-only shadow mode a live ACP route is ownership evidence, + not a transport. Such a target fails closed before receipt reservation; + returning ``None`` would incorrectly fall through to its legacy route. """ payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) @@ -2759,6 +2764,22 @@ def submit_acp_command( if takeover is not None and worker.id != takeover.public_worker_id: return _duplicate_request(request) + route: AcpPromptRoute | None = None + route_resolved = False + if observation_only: + route_resolved = True + try: + route = prompt_router(worker) + except Exception: # noqa: BLE001 + route = None + if route is not None: + return _backend_unavailable( + request, + "ACP shadow is observation-only for ACP-owned workers; use an " + "isolated ACP preferred or required canary to validate prompt " + "execution", + ) + permanent_error = _worker_status_error(request, worker) or health_error if permanent_error is not None: if required: @@ -2769,10 +2790,11 @@ def submit_acp_command( return _finish_before_send(config, request, reservation, permanent_error) return None - try: - route = prompt_router(worker) - except Exception: # noqa: BLE001 - route = None + if not route_resolved: + try: + route = prompt_router(worker) + except Exception: # noqa: BLE001 + route = None if route is None: if takeover is not None: return _request_in_progress(request) @@ -3174,6 +3196,7 @@ def submit_command( socket_client_factory: SocketClientFactory | None = None, acp_prompt_router: AcpPromptRouter | None = None, acp_required: bool = False, + acp_observation_only: bool = False, acp_permission_router: AcpPermissionDecisionRouter | None = None, ) -> CommandEnvelope: """Submit one command and apply optional response-envelope negotiation.""" @@ -3183,6 +3206,7 @@ def submit_command( params, prompt_router=acp_prompt_router, required=acp_required, + observation_only=acp_observation_only, ) if acp_envelope is not None: payload = ( diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index 8ef99d1..8a401f3 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -616,7 +616,7 @@ def start(self) -> None: if self.config.agent_event_source != "acp_required": scheduler = self.hooks.turn_scheduler_factory(self.config) self._turn_scheduler = scheduler - if self.config.agent_event_source == "acp_preferred": + if self.config.agent_event_source in {"acp_shadow", "acp_preferred"}: owns_worker = getattr(self._acp_runtime, "owns_worker", None) set_exclusion = getattr(scheduler, "set_worker_exclusion", None) if callable(owns_worker) and callable(set_exclusion): @@ -1455,7 +1455,7 @@ def submit_command(self, params: Mapping[str, Any]) -> CommandEnvelope | Mapping else None ) if policy == "acp_required" or ( - policy == "acp_preferred" and callable(route) + policy in {"acp_shadow", "acp_preferred"} and callable(route) ) or permission_router is not None: from .command_submission import submit_command @@ -1464,11 +1464,12 @@ def submit_command(self, params: Mapping[str, Any]) -> CommandEnvelope | Mapping payload, acp_prompt_router=( route - if policy in {"acp_required", "acp_preferred"} + if policy in {"acp_shadow", "acp_required", "acp_preferred"} and callable(route) else None ), acp_required=policy == "acp_required", + acp_observation_only=policy == "acp_shadow", acp_permission_router=permission_router, ) return self.hooks.submit_command(self.config, payload) diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 66accd4..dbeaf5f 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -25,6 +25,7 @@ from tendwire.command_submission import submit_acp_command, submit_command from tendwire.config import Config from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding +from tendwire.daemon import DaemonHooks, TendwireDaemon from tendwire.store.sqlite import ( get_command_request, init_store, @@ -268,6 +269,191 @@ def test_acp_failure_after_send_started_is_uncertain_and_never_falls_back(tmp_pa assert receipt is not None and receipt["state"] == "uncertain" +def test_shadow_owned_command_is_observation_only_before_receipt(tmp_path: Path) -> None: + config = _config(tmp_path, policy="acp_shadow") + worker = replace(_seed(config), status="working") + assert config.db_path is not None + save_snapshot( + config.db_path, + Snapshot( + host_id=config.host_id, + updated_at="2026-07-31T00:00:01+00:00", + workers=[worker], + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty", + ) + ], + ), + ) + route = _Route() + + envelope = submit_acp_command( + config, + _request("shadow-observation-only"), + prompt_router=lambda _worker: route, + observation_only=True, + ) + + assert envelope is not None + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert route.calls == [] + assert get_command_request( + config.db_path, + config.host_id, + "shadow-observation-only", + ) is None + + +def test_daemon_shadow_owned_command_never_reaches_legacy_sender(tmp_path: Path) -> None: + config = _config(tmp_path, policy="acp_shadow") + worker = _seed(config) + route = _Route() + legacy_calls: list[str] = [] + + class Runtime: + def prompt_route(self, routed: Worker) -> _Route | None: + return route if routed == worker else None + + def legacy_sender(_config: Config, _payload: str) -> Any: + legacy_calls.append("calibrate-or-write") + raise AssertionError("ACP-owned shadow target must not use legacy PTY I/O") + + daemon = TendwireDaemon( + config, + hooks=DaemonHooks(submit_command=legacy_sender), + ) + daemon._acp_runtime = Runtime() + + envelope = daemon.submit_command(_request("shadow-daemon-fence")) + + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert route.calls == [] + assert legacy_calls == [] + + +def test_daemon_shadow_preserves_ordinary_legacy_worker_submission( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = _config(tmp_path, policy="acp_shadow") + _seed(config) + legacy_calls: list[str] = [] + + class Runtime: + def prompt_route(self, _worker: Worker) -> None: + return None + + class LegacyClient: + def connect(self) -> "LegacyClient": + return self + + def request( + self, + method: str, + params: dict[str, Any], + *, + timeout: float | None = None, + ) -> dict[str, Any]: + del timeout + legacy_calls.append(method) + if method == "agent.get": + return {"result": {"agent": {"pane_id": "pane-private"}}} + if method == "agent.prompt": + return { + "type": "agent_prompted", + "agent": {"pane_id": "pane-private"}, + "delivery": "submitted", + } + return {"accepted": True, "params": params} + + def close(self) -> None: + return None + + monkeypatch.setattr( + "tendwire.command_submission._default_socket_client_factory", + lambda _config: LegacyClient(), + ) + daemon = TendwireDaemon(config) + daemon._acp_runtime = Runtime() + + envelope = daemon.submit_command(_request("shadow-legacy-worker")) + + assert envelope.status == "accepted" + assert legacy_calls[-1] == "agent.prompt" + + +def test_daemon_wires_shadow_ownership_fence_into_legacy_scheduler( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = replace( + _config(tmp_path, policy="acp_shadow"), + herdr_backend="cli", + socket_path=tmp_path / "shadow.sock", + ) + callback: Any | None = None + + class Runtime: + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + return None + + def status(self) -> dict[str, Any]: + return {"state": "running", "healthy": True} + + def owns_worker(self, worker_id: str, fingerprint: str) -> bool: + return worker_id == "worker-1" and fingerprint == "worker-fingerprint" + + class Scheduler: + def set_worker_exclusion(self, value: Any) -> None: + nonlocal callback + callback = value + + def start(self) -> None: + return None + + def request_refresh(self) -> None: + return None + + def stop(self, *, flush_timeout_seconds: float) -> None: + return None + + def observe(_config: Config) -> Snapshot: + snapshot = Snapshot( + host_id=config.host_id, + updated_at="2026-07-31T00:00:00+00:00", + workers=[], + backend_health=[], + ) + assert config.db_path is not None + save_snapshot(config.db_path, snapshot) + return snapshot + + daemon = TendwireDaemon( + config, + hooks=DaemonHooks( + observe_initial_snapshot=observe, + turn_scheduler_factory=lambda _config: Scheduler(), + acp_runtime_factory=lambda _config, _stop: Runtime(), + ), + ) + monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", lambda _self: None) + try: + daemon.start() + assert callable(callback) + assert callback("worker-1", "worker-fingerprint") is True + assert callback("legacy-worker", "legacy-fingerprint") is False + finally: + daemon.stop() + + def test_route_authority_failure_is_safe_before_receipt_reservation(tmp_path: Path) -> None: config = _config(tmp_path) _seed(config) @@ -906,8 +1092,12 @@ def permission_bridge(_request: Any) -> str | None: coordinator.stop() -def test_preferred_legacy_scheduler_excludes_acp_owned_worker(tmp_path: Path) -> None: - config = _config(tmp_path) +@pytest.mark.parametrize("policy", ["acp_shadow", "acp_preferred"]) +def test_acp_owned_worker_is_excluded_from_legacy_scheduler( + tmp_path: Path, + policy: str, +) -> None: + config = _config(tmp_path, policy=policy) assert config.db_path is not None init_store(config.db_path) upsert_worker_bindings(config.db_path, [_binding()]) From e7bd6d24e40e967fb96600c7e411af3143447d52 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 14:20:48 +0800 Subject: [PATCH 41/83] test: stabilize descriptor leak assertion --- tests/test_local_state_permissions.py | 31 +++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/tests/test_local_state_permissions.py b/tests/test_local_state_permissions.py index 5e4650a..7480eea 100644 --- a/tests/test_local_state_permissions.py +++ b/tests/test_local_state_permissions.py @@ -83,6 +83,29 @@ def _assert_path_free(value: object, *paths: Path) -> None: rendered = repr(value) for path in paths: assert str(path) not in rendered + + +def _open_fd_identities() -> dict[int, tuple[int, int, int, int]]: + identities: dict[int, tuple[int, int, int, int]] = {} + for raw_fd in os.listdir("/proc/self/fd"): + fd = int(raw_fd) + try: + current = os.fstat(fd) + except OSError as exc: + # Listing /proc/self/fd briefly exposes the directory descriptor + # used by listdir itself; it is closed before fstat can inspect it. + if exc.errno == errno.EBADF: + continue + raise + identities[fd] = ( + current.st_dev, + current.st_ino, + current.st_mode, + current.st_rdev, + ) + return identities + + def _sqlite_replacement_source( tmp_path: Path, *, mode: int = 0o600 ) -> tuple[Path, Path, int]: @@ -806,7 +829,7 @@ def test_sqlite_family_sidecar_replacement_fails_without_target_mutation( unexpected_uid = os.geteuid() + 100_000 replacement_active = False replacement_identity = None - before_fds = set(os.listdir("/proc/self/fd")) + before_fds = _open_fd_identities() preflight_calls = 0 def owner_aware_lstat_at(dir_fd: int, name: str): @@ -876,7 +899,11 @@ def replace_at_preflight(phase: str, selected_kind: LocalStateKind) -> None: else: assert sidecar.read_bytes() == b"hostile-replacement" assert _mode(sidecar) == 0o644 - assert set(os.listdir("/proc/self/fd")) == before_fds + # The full suite can finish background subprocess cleanup while this + # test runs, legitimately closing descriptors owned by pytest. Require + # every remaining descriptor to retain its original identity, which + # still catches both new descriptors and reuse of a closed fd number. + assert _open_fd_identities().items() <= before_fds.items() finally: os.close(parent_fd) From 826addef855e4f6c76a1ca2693c2bb76f60c8669 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 17:27:24 +0800 Subject: [PATCH 42/83] Preserve opaque revisions across connector transport --- src/tendwire/cli.py | 12 ++++++++++- src/tendwire/connectors/outbox.py | 13 ++++++++---- src/tendwire/daemon_api.py | 16 +++++++++++++- src/tendwire/store/sqlite.py | 13 ++++++++---- tests/test_cli.py | 8 ++++--- tests/test_connector_daemon_cli.py | 27 ++++++++++++++++++++++++ tests/test_connector_outbox.py | 34 ++++++++++++++++++++++++++++++ tests/test_daemon.py | 8 ++++--- 8 files changed, 115 insertions(+), 16 deletions(-) diff --git a/src/tendwire/cli.py b/src/tendwire/cli.py index 6e1d64e..7b26b2a 100644 --- a/src/tendwire/cli.py +++ b/src/tendwire/cli.py @@ -984,6 +984,11 @@ def _restore_cli_content_text( and original.get("availability") == "complete" ): sanitized["text"] = text + content_revision = original.get("content_revision") + if isinstance(content_revision, str) and re.fullmatch( + r"twrev1\.[A-Za-z0-9_-]+", content_revision + ): + sanitized["content_revision"] = content_revision def _restore_cli_plan_token( @@ -1007,12 +1012,17 @@ def _restore_cli_plan_token( r"twfinal1\.[A-Za-z0-9_-]+", final_identity ): sanitized["final_identity"] = final_identity + content_revision = original.get("content_revision") + if isinstance(content_revision, str) and re.fullmatch( + r"twrev1\.[A-Za-z0-9_-]+", content_revision + ): + sanitized["content_revision"] = content_revision delivery_key = original.get("key") if isinstance(delivery_key, str) and re.fullmatch( r"turn-final:revision:twfinal1\.[A-Za-z0-9_-]+", delivery_key ): sanitized["key"] = delivery_key - for nested_key in ("turn", "final", "payload"): + for nested_key in ("turn", "final", "payload", "content"): nested_original = original.get(nested_key) nested_sanitized = sanitized.get(nested_key) if isinstance(nested_original, dict) and isinstance(nested_sanitized, dict): diff --git a/src/tendwire/connectors/outbox.py b/src/tendwire/connectors/outbox.py index fbb6888..802cac0 100644 --- a/src/tendwire/connectors/outbox.py +++ b/src/tendwire/connectors/outbox.py @@ -124,6 +124,9 @@ def _restore_plan_tokens(clean: dict[str, Any], original: Mapping[str, Any]) -> ) if final_identity: clean["final_identity"] = final_identity + content_revision = _revision(original.get("content_revision")) + if content_revision: + clean["content_revision"] = content_revision turn_id = _text(original.get("turn_id")) if ( turn_id.startswith("turn-") @@ -131,15 +134,17 @@ def _restore_plan_tokens(clean: dict[str, Any], original: Mapping[str, Any]) -> and all(char in _CONNECTOR_NAME_CHARS for char in turn_id) ): clean["turn_id"] = turn_id - nested_turn = original.get("turn") - if isinstance(nested_turn, Mapping): - clean_nested = clean.get("turn") + for nested_key in ("turn", "content"): + nested_turn = original.get(nested_key) + if not isinstance(nested_turn, Mapping): + continue + clean_nested = clean.get(nested_key) if not isinstance(clean_nested, dict): clean_nested = sanitize_public_mapping( nested_turn, backend_neutral=True, ) - clean["turn"] = clean_nested + clean[nested_key] = clean_nested _restore_plan_tokens(clean_nested, nested_turn) return clean diff --git a/src/tendwire/daemon_api.py b/src/tendwire/daemon_api.py index 7096e07..123b4f5 100644 --- a/src/tendwire/daemon_api.py +++ b/src/tendwire/daemon_api.py @@ -727,6 +727,13 @@ def _restore_content_page_text( result = response.get("result") if isinstance(result, dict): result["text"] = text + content_revision = original_result.get("content_revision") + if ( + isinstance(content_revision, str) + and re.fullmatch(r"twrev1\.[A-Za-z0-9_-]+", content_revision) + is not None + ): + result["content_revision"] = content_revision def _restore_turn_delta_text( @@ -784,6 +791,13 @@ def restore(target: dict[str, Any], original: Mapping[str, Any]) -> None: is not None ): target["final_identity"] = final_identity + content_revision = original.get("content_revision") + if ( + isinstance(content_revision, str) + and re.fullmatch(r"twrev1\.[A-Za-z0-9_-]+", content_revision) + is not None + ): + target["content_revision"] = content_revision delivery_key = original.get("key") if ( isinstance(delivery_key, str) @@ -794,7 +808,7 @@ def restore(target: dict[str, Any], original: Mapping[str, Any]) -> None: is not None ): target["key"] = delivery_key - for nested_key in ("turn", "final", "payload"): + for nested_key in ("turn", "final", "payload", "content"): nested_original = original.get(nested_key) nested_target = target.get(nested_key) if isinstance(nested_original, Mapping) and isinstance(nested_target, dict): diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index ee9b932..4cd80e6 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -4359,12 +4359,17 @@ def _restore_presentation_tokens( final_identity = original.get("final_identity") if _valid_presentation_opaque(final_identity, "twfinal1."): sanitized["final_identity"] = str(final_identity) + content_revision_value = original.get("content_revision") + if _valid_presentation_opaque(content_revision_value, "twrev1."): + sanitized["content_revision"] = str(content_revision_value) turn_id = original.get("turn_id") if _valid_presentation_label(turn_id, prefix="turn-"): sanitized["turn_id"] = str(turn_id) - nested_turn = original.get("turn") - if isinstance(nested_turn, Mapping): - clean_nested = sanitized.get("turn") + for nested_key in ("turn", "content"): + nested_turn = original.get(nested_key) + if not isinstance(nested_turn, Mapping): + continue + clean_nested = sanitized.get(nested_key) if not isinstance(clean_nested, dict): clean_nested = dict( sanitize_public_mapping( @@ -4372,7 +4377,7 @@ def _restore_presentation_tokens( backend_neutral=True, ) ) - sanitized["turn"] = clean_nested + sanitized[nested_key] = clean_nested _restore_presentation_tokens(clean_nested, nested_turn) return sanitized diff --git a/tests/test_cli.py b/tests/test_cli.py index 38827ab..4575c21 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -909,6 +909,7 @@ def test_cli_turn_content_get_preserves_exact_page_and_params( monkeypatch, ) -> None: page_text = "\n " + ("界" * 20_000) + "\r\n " + revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" calls: list[tuple[str, dict[str, Any]]] = [] class FakeDaemonAPIClient: @@ -924,7 +925,7 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str "ok": True, "status": "ok", "turn_id": "turn-public", - "content_revision": "twrev1.public", + "content_revision": revision, "field": "assistant_final_text", "availability": "complete", "segment_id": "twseg1.public", @@ -953,7 +954,7 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str "--turn-id", "turn-public", "--revision", - "twrev1.public", + revision, "--field", "assistant_final_text", "--cursor", @@ -966,6 +967,7 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str assert code == 0 assert captured.err == "" assert payload["turn_id"] == "turn-public" + assert payload["content_revision"] == revision assert payload["segment_id"] == "twseg1.public" assert payload["text"] == page_text assert calls == [ @@ -974,7 +976,7 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str { "schema_version": 1, "turn_id": "turn-public", - "content_revision": "twrev1.public", + "content_revision": revision, "field": "assistant_final_text", "cursor": "twcur1.public", }, diff --git a/tests/test_connector_daemon_cli.py b/tests/test_connector_daemon_cli.py index 96b1c65..09ef2e2 100644 --- a/tests/test_connector_daemon_cli.py +++ b/tests/test_connector_daemon_cli.py @@ -415,6 +415,8 @@ def test_cli_daemon_connector_result_is_sanitized_before_printing( capsys, monkeypatch, ) -> None: + revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" + class FakeDaemonAPIClient: def __init__(self, socket_path: Any, *, timeout_seconds: float, max_response_bytes: int = 1024 * 1024): pass @@ -437,6 +439,11 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str "safe": "kept", "turn_id": "turn-public-final", "plan_token": "twplan1.publicPlan", + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + }, "chat_id": "sentinel-private-chat", "raw_payload": "sentinel-private-raw", }, @@ -470,6 +477,11 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str "safe": "kept", "turn_id": "turn-public-final", "plan_token": "twplan1.publicPlan", + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + }, } assert "sentinel-private" not in encoded assert "raw_payload" not in encoded @@ -477,6 +489,7 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str def test_daemon_connector_preserves_public_turn_id_for_final_ready() -> None: + revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" api = TendwireDaemonAPI( get_snapshot=lambda: Snapshot(host_id="host-a"), get_health=lambda: {}, @@ -493,6 +506,13 @@ def test_daemon_connector_preserves_public_turn_id_for_final_ready() -> None: "schema_version": 2, "operation": "final_ready", "turn_id": "turn-public-final", + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + "known_incomplete": False, + "fields": {}, + }, "pane_id": "sentinel-private-pane", "session_id": "sentinel-private-session", "terminal_id": "sentinel-private-terminal", @@ -512,6 +532,13 @@ def test_daemon_connector_preserves_public_turn_id_for_final_ready() -> None: "schema_version": 2, "operation": "final_ready", "turn_id": "turn-public-final", + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + "known_incomplete": False, + "fields": {}, + }, } _assert_json_only_and_safe(response) diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index 5a9f1c7..e7f5d51 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -176,6 +176,40 @@ def test_poll_leases_sanitized_item_and_skips_duplicate_live_lease(tmp_path: Pat _assert_no_forbidden(first) +def test_poll_preserves_strict_content_revision_tokens(tmp_path: Path) -> None: + db_path = tmp_path / "revision-token.db" + revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" + _enqueue_final_root( + db_path, + key_suffix="public", + ordering_key="wsk1_public", + ) + with sqlite3.connect(str(db_path)) as conn: + conn.execute( + "UPDATE connector_outbox SET payload_json = ?", + ( + json.dumps( + { + "schema_version": 2, + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + }, + } + ), + ), + ) + + item = ConnectorOutboxAPI(db_path, "host-a").poll( + {"name": "turn-final", "limit": 1} + )["items"][0] + + assert item["payload"]["content_revision"] == revision + assert item["payload"]["content"]["content_revision"] == revision + _assert_no_forbidden(item) + + def test_poll_uses_configured_default_lease_and_explicit_lease_wins(tmp_path: Path) -> None: db_path = tmp_path / "lease-default.db" _enqueue(db_path, key="default-lease") diff --git a/tests/test_daemon.py b/tests/test_daemon.py index 7663f23..7fae773 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -735,6 +735,7 @@ def test_daemon_api_versions_turn_list_and_preserves_exact_content_page() -> Non turn_calls: list[dict[str, Any]] = [] page_calls: list[dict[str, Any]] = [] page_text = "\n " + ("α" * 20_000) + " \r\n" + revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" def get_turns(**params: Any) -> dict[str, Any]: turn_calls.append(dict(params)) @@ -790,7 +791,7 @@ def get_turns(**params: Any) -> dict[str, Any]: "ok": True, "status": "ok", "turn_id": "turn-public", - "content_revision": "twrev1.public", + "content_revision": revision, "field": "assistant_final_text", "availability": "complete", "segment_id": "twseg1.public", @@ -821,7 +822,7 @@ def get_turns(**params: Any) -> dict[str, Any]: "params": { "schema_version": 1, "turn_id": "turn-public", - "content_revision": "twrev1.public", + "content_revision": revision, "field": "assistant_final_text", }, } @@ -861,11 +862,12 @@ def get_turns(**params: Any) -> dict[str, Any]: "since": None, } assert page["result"]["text"] == page_text + assert page["result"]["content_revision"] == revision assert page_calls == [ { "schema_version": 1, "turn_id": "turn-public", - "content_revision": "twrev1.public", + "content_revision": revision, "field": "assistant_final_text", } ] From e0f88664b56e41e63919a4c97060ae31a795e324 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 19:22:51 +0800 Subject: [PATCH 43/83] feat: bridge ACP sessions into visible panes --- src/tendwire/backends/acp_coordinator.py | 985 +++++++++++++++++++++- src/tendwire/backends/herdr_socket.py | 25 + src/tendwire/core/commands.py | 46 +- tests/fixtures/herdr_acp_contract_v1.json | 4 + tests/test_acp_coordinator.py | 410 +++++++++ tests/test_command_replay_authority.py | 1 + tests/test_commands.py | 118 ++- 7 files changed, 1566 insertions(+), 23 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 3baa7de..e6ee11e 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -9,18 +9,26 @@ from __future__ import annotations +import json import threading import time from collections.abc import Callable, Mapping -from dataclasses import dataclass, replace +from concurrent.futures import Future, ThreadPoolExecutor +from dataclasses import dataclass, field, replace from pathlib import Path from typing import Any from ..config import Config from ..core.models import Worker, WorkerBinding, utc_timestamp +from ..core.commands import turn_submission_id +from ..core.models import stable_fingerprint from ..store.sqlite import ( expire_worker_bindings, + latest_snapshot, + list_agent_events, list_worker_bindings, + pending_payload_from_store, + record_agent_event, upsert_worker_bindings, ) from .acp_client import AcpClient @@ -42,6 +50,10 @@ class AcpPermissionBridgeUnavailable(AcpCoordinatorError): """Production ACP cannot authorize tools without a durable user decision.""" +class AcpConsoleInputGap(AcpCoordinatorError): + """The bounded Herdr console queue lost unconsumed pane input.""" + + @dataclass(frozen=True, slots=True) class HerdrAcpEndpoint: command: tuple[str, ...] @@ -49,12 +61,20 @@ class HerdrAcpEndpoint: generation: str session_mode: SessionOpenMode session_id: str | None + console: HerdrAcpConsoleEndpoint | None = None @dataclass(frozen=True, slots=True) class HerdrAcpStatus: generation: str lifecycle: str + console_lifecycle: str + + +@dataclass(frozen=True, slots=True) +class HerdrAcpConsoleEndpoint: + generation: int + lease: str @dataclass(slots=True) @@ -63,6 +83,17 @@ class _RuntimeSlot: generation: str runtime: AcpRuntime permission_broker: AcpPermissionBroker | None = None + console: HerdrAcpConsoleEndpoint | None = None + console_input_sequence: int = 0 + console_event_sequence: int = 0 + console_cursor_loaded: bool = False + console_local_turns: set[str] | None = None + console_executor: ThreadPoolExecutor | None = None + console_submissions: dict[int, Future[Any]] | None = None + console_failures: int = 0 + lock: threading.RLock = field(default_factory=threading.RLock, repr=False) + retired: bool = False + console_bridge_thread: threading.Thread | None = None class _PromptRoute: @@ -149,10 +180,15 @@ def __init__( self._reconcile_lock = threading.RLock() self._stop = threading.Event() self._slots: dict[str, _RuntimeSlot] = {} + self._retired_slots: list[_RuntimeSlot] = [] self._thread: threading.Thread | None = None + self._console_thread: threading.Thread | None = None self._state = RuntimeState.NEW self._failure_type: str | None = None self._required_degraded = False + self._console_degraded = False + self._console_failure_type: str | None = None + self._console_failed_workers: set[str] = set() def start(self) -> "AcpRuntimeCoordinator": with self._lock: @@ -195,6 +231,13 @@ def start(self) -> "AcpRuntimeCoordinator": ) self._thread = thread thread.start() + console_thread = threading.Thread( + target=self._run_console_bridge, + name="tendwire-acp-console-bridge", + daemon=True, + ) + self._console_thread = console_thread + console_thread.start() return self def stop(self, *, timeout: float | None = None) -> None: @@ -209,7 +252,7 @@ def stop(self, *, timeout: float | None = None) -> None: if self._state is RuntimeState.STOPPED: return self._state = RuntimeState.STOPPING - slots = tuple(self._slots.values()) + slots = tuple(self._slots.values()) + tuple(self._retired_slots) self._stop.set() # A durable permission answer keeps the generation fence until the # complete JSON-RPC response frame is written. Wake any broker waiters @@ -240,16 +283,61 @@ def stop(self, *, timeout: float | None = None) -> None: thread = self._thread if thread is not None and thread is not threading.current_thread(): thread.join(timeout=max(0.0, deadline - time.monotonic())) + console_thread = self._console_thread + if console_thread is not None and console_thread is not threading.current_thread(): + console_thread.join(timeout=max(0.0, deadline - time.monotonic())) + unfinished = bool( + (thread is not None and thread.is_alive()) + or (console_thread is not None and console_thread.is_alive()) + ) + for slot in slots: + with slot.lock: + bridge_thread = slot.console_bridge_thread + futures = tuple((slot.console_submissions or {}).values()) + executor = slot.console_executor + if bridge_thread is not None and bridge_thread is not threading.current_thread(): + bridge_thread.join(timeout=max(0.0, deadline - time.monotonic())) + unfinished = unfinished or bridge_thread.is_alive() + slot_unfinished = False + for future in futures: + if future.done(): + continue + try: + future.result(timeout=max(0.0, deadline - time.monotonic())) + except Exception: + pass + slot_unfinished = slot_unfinished or not future.done() + unfinished = unfinished or slot_unfinished + if not slot_unfinished and executor is not None: + # The executor was already asked to shut down by retirement; + # once every task is terminal this join is immediate and + # ensures no non-daemon worker survives a successful stop. + executor.shutdown(wait=True, cancel_futures=True) with self._lock: - if self._state is not RuntimeState.FAILED: + if unfinished: + self._state = RuntimeState.FAILED + self._failure_type = "AcpRuntimeStopTimeout" + elif self._state is not RuntimeState.FAILED: self._state = RuntimeState.STOPPED def join(self, timeout: float | None = None) -> bool: - thread = self._thread - if thread is None or thread is threading.current_thread(): - return True - thread.join(timeout=timeout) - return not thread.is_alive() + deadline = None if timeout is None else time.monotonic() + timeout + threads = (self._thread, self._console_thread) + for thread in threads: + if thread is None or thread is threading.current_thread(): + continue + remaining = ( + None + if deadline is None + else max(0.0, deadline - time.monotonic()) + ) + thread.join(timeout=remaining) + return all( + thread is None + or thread is threading.current_thread() + or not thread.is_alive() + for thread in threads + ) def status(self) -> dict[str, Any]: counters = { @@ -268,7 +356,14 @@ def status(self) -> dict[str, Any]: slots = tuple(self._slots.values()) failure_type = self._failure_type required_degraded = self._required_degraded - healthy = state is RuntimeState.RUNNING and not required_degraded + console_degraded = self._console_degraded + console_failure_type = self._console_failure_type + failure_type = failure_type or console_failure_type + healthy = ( + state is RuntimeState.RUNNING + and not required_degraded + and not console_degraded + ) for slot in slots: try: status = slot.runtime.status() @@ -361,6 +456,431 @@ def _run(self) -> None: with self._lock: self._failure_type = type(exc).__name__ + def _run_console_bridge(self) -> None: + while not self._stop.wait(0.1): + if self._daemon_stop.is_set(): + return + self._bridge_console_slots() + + def _bridge_console_slots(self) -> None: + """Dispatch independent visible pane bridges without head-of-line blocking.""" + with self._lock: + self._retired_slots = [ + slot for slot in self._retired_slots if _slot_has_live_work(slot) + ] + slots = tuple(self._slots.values()) + for slot in slots: + if slot.console is None: + continue + with slot.lock: + if slot.retired: + continue + thread = slot.console_bridge_thread + if thread is not None and thread.is_alive(): + continue + thread = threading.Thread( + target=self._bridge_console_slot_supervised, + args=(slot,), + name=f"tendwire-acp-console-bridge-{slot.continuity.worker_id}", + daemon=True, + ) + slot.console_bridge_thread = thread + thread.start() + + def _bridge_console_slot_supervised(self, slot: _RuntimeSlot) -> None: + worker_id = slot.continuity.worker_id + try: + self._bridge_console_slot(slot) + with slot.lock: + if slot.retired: + return + slot.console_failures = 0 + with self._lock: + self._console_failed_workers.discard(worker_id) + self._console_degraded = bool(self._console_failed_workers) + if not self._console_degraded: + self._console_failure_type = None + except Exception as exc: + with slot.lock: + if slot.retired: + return + slot.console_failures += 1 + failure_count = slot.console_failures + # Console availability is supervised independently of the ACP + # runtime. A later pass replays inputs and idempotent outputs. + with self._lock: + self._console_failed_workers.add(worker_id) + self._console_degraded = True + self._console_failure_type = type(exc).__name__ + if failure_count >= 3: + try: + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + self._retire_worker( + worker_id, + expected=slot, + preserve_console_failure=True, + ) + current, _ambiguities = self._continuity_bindings() + continuity = current.get(worker_id) + if continuity is None: + raise AcpCoordinatorError( + "worker has no unique Herdr authority" + ) + self._reconcile_binding(continuity) + except Exception: + # Keep the worker in the persistent failure set until a + # replacement slot completes a successful console pass. + pass + + def _bridge_console_slot(self, slot: _RuntimeSlot) -> None: + with slot.lock: + if slot.retired: + return + console = slot.console + if console is None or self.config.db_path is None: + return + binding = getattr(slot.runtime, "_binding", None) + if not isinstance(binding, WorkerBinding): + raise AcpCoordinatorError("ACP console session binding is unavailable") + with slot.lock: + cursor_loaded = slot.console_cursor_loaded + if not cursor_loaded: + persisted_cursor = _load_console_event_cursor( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + binding.turn_target_value, + ) + loaded_cursor = ( + persisted_cursor + if persisted_cursor is not None + else _initial_console_event_cursor( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + binding.turn_target_value, + ) + ) + with slot.lock: + if slot.retired: + return + slot.console_event_sequence = loaded_cursor + slot.console_cursor_loaded = True + with slot.lock: + event_sequence = slot.console_event_sequence + input_sequence = slot.console_input_sequence + submissions = slot.console_submissions or {} + slot.console_submissions = submissions + events = list_agent_events( + Path(self.config.db_path), + self.config.host_id, + worker_id=slot.continuity.worker_id, + source="acp", + session_id=binding.turn_target_value, + after_sequence=event_sequence, + limit=100, + ) + # Snapshot suppression after the event page. A console submission + # registers its deterministic turn id before it can emit the ACP user + # event, so this ordering closes the race where the bridge could copy + # an old set and then observe the newly emitted event in the same page. + with slot.lock: + if slot.retired: + return + local_turns = set(slot.console_local_turns or ()) + output: list[dict[str, str]] = [] + completed_submissions: list[int] = [] + for sequence, future in tuple(submissions.items()): + if not future.done(): + continue + completed_submissions.append(sequence) + try: + outcome = future.result() + if outcome in {"permission", "cancelled"}: + output.append( + { + "event_id": f"console-{outcome}:{console.generation}:{sequence}", + "stream": "status", + "text": ( + "permission selection accepted" + if outcome == "permission" + else "active turn cancellation requested" + ), + } + ) + except Exception as exc: + output.append( + { + "event_id": f"console-error:{console.generation}:{sequence}", + "stream": "error", + "text": f"instruction failed ({type(exc).__name__})", + } + ) + # A terminal command receipt now exists (accepted or rejected), so + # the next exchange may acknowledge this one Herdr queue item. + record_agent_event( + Path(self.config.db_path), + self.config.host_id, + kind="extension", + source="tendwire-console", + worker_id=slot.continuity.worker_id, + payload={ + "extension": "tendwire.acp.console_input_cursor", + "generation": console.generation, + "input_sequence": sequence, + }, + source_session_id=binding.turn_target_value, + source_event_id=f"input:{console.generation}:{sequence}", + visibility="private", + ) + input_sequence = max(input_sequence, sequence) + pending = _console_pending_decision( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + ) + if pending is not None: + decision_ref, _options, prompt = pending + output.append( + { + "event_id": "permission:" + stable_fingerprint( + { + "worker_id": slot.continuity.worker_id, + "decision_ref": decision_ref, + } + ), + "stream": "status", + "text": prompt, + } + ) + consumed_local_turns: set[str] = set() + for stored in events: + event = stored.event + if event.kind == "user_message" and event.source_turn_id in local_turns: + if event.source_turn_id is not None: + consumed_local_turns.add(event.source_turn_id) + continue + rendered = _console_event_output(event.kind, event.payload) + if rendered is None: + continue + stream, text = rendered + output.append( + {"event_id": event.event_id, "stream": stream, "text": text} + ) + client = self._endpoint_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + result = client.agent_acp_console_exchange( + slot.continuity.target_value, + generation=console.generation, + lease=console.lease, + after_input_sequence=input_sequence, + output=output, + timeout=self.config.herdr_timeout_seconds, + ) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + try: + inputs = _parse_console_exchange(result, input_sequence) + except AcpConsoleInputGap: + gap_output = [{ + "event_id": f"console-gap:{console.generation}:{input_sequence}", + "stream": "error", + "text": "console input backlog overflowed; restart the pane console before retrying", + }] + client = self._endpoint_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + client.agent_acp_console_exchange( + slot.continuity.target_value, + generation=console.generation, + lease=console.lease, + after_input_sequence=input_sequence, + output=gap_output, + timeout=self.config.herdr_timeout_seconds, + ) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + raise + with slot.lock: + if slot.retired: + return + slot.console_input_sequence = input_sequence + for sequence in completed_submissions: + submissions.pop(sequence, None) + if events: + # Herdr deduplicates output event_id values, so replaying a page + # after a crash between exchange and this cursor update is safe. + next_event_sequence = max(item.sequence for item in events) + record_agent_event( + Path(self.config.db_path), + self.config.host_id, + kind="extension", + source="tendwire-console", + worker_id=slot.continuity.worker_id, + payload={ + "extension": "tendwire.acp.console_cursor", + "sequence": next_event_sequence, + }, + source_session_id=binding.turn_target_value, + source_event_id=f"cursor:{next_event_sequence}", + visibility="private", + ) + with slot.lock: + if slot.retired: + return + slot.console_event_sequence = next_event_sequence + if slot.console_local_turns is not None: + slot.console_local_turns.difference_update(consumed_local_turns) + for sequence, text in inputs[:1]: + with slot.lock: + if slot.retired or sequence in submissions: + continue + executor = slot.console_executor + if executor is None: + raise AcpCoordinatorError("ACP console submission worker is unavailable") + submissions[sequence] = executor.submit( + self._submit_console_input, slot, sequence, text + ) + + def _submit_console_input( + self, slot: _RuntimeSlot, sequence: int, text: str + ) -> str: + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + with self._lock: + current = self._slots.get(slot.continuity.worker_id) + with slot.lock: + retired = slot.retired + if current is not slot or retired: + raise AcpCoordinatorError("ACP console input generation is stale") + return self._submit_console_input_fenced(slot, sequence, text) + + def _submit_console_input_fenced( + self, slot: _RuntimeSlot, sequence: int, text: str + ) -> str: + snapshot = latest_snapshot(Path(self.config.db_path), self.config.host_id) + worker = next( + ( + item + for item in (snapshot.workers if snapshot is not None else ()) + if item.id == slot.continuity.worker_id + and item.fingerprint == slot.continuity.worker_fingerprint + ), + None, + ) + stable_key = worker.meta.get("stable_key") if worker is not None else None + stable_version = ( + worker.meta.get("stable_key_version") if worker is not None else None + ) + if not isinstance(stable_key, str) or stable_version != 1: + raise AcpCoordinatorError("ACP console stable worker identity is unavailable") + request_id = "acpc." + stable_fingerprint( + { + "generation": slot.generation, + "input_sequence": sequence, + "worker_id": slot.continuity.worker_id, + } + ) + if text.strip() == "/cancel": + slot.runtime.cancel() + return "cancelled" + request: dict[str, Any] = { + "schema_version": 1, + "response_schema_version": 3, + "action": "send_instruction", + "request_id": request_id, + "dry_run": False, + "target": { + "stable_key": stable_key, + "stable_key_version": 1, + }, + "instruction": {"text": text}, + } + pending = _console_pending_decision( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + ) + if pending is not None: + decision_ref, options, _prompt = pending + selected = _console_permission_selection(text, options) + if selected is None: + raise AcpCoordinatorError( + "console input is not a valid permission selection" + ) + request.pop("instruction", None) + request["action"] = "answer_decision" + request["params"] = { + "decision_ref": decision_ref, + "selection": {"option_refs": [selected]}, + } + from ..command_submission import submit_command + + local_turn_id: str | None = None + if request["action"] == "send_instruction": + submission_id = turn_submission_id(self.config.host_id, request_id) + session_id = str(getattr(slot.runtime, "_session_id", "") or "") + if session_id: + local_turn_id = "acpt_" + stable_fingerprint( + { + "source": "acp", + "session": session_id, + "producer_turn": submission_id, + } + ) + # Register before submit_command can emit the ACP user event; + # the independent bridge thread may observe it immediately. + with slot.lock: + if slot.retired: + raise AcpCoordinatorError( + "ACP console input generation is stale" + ) + local_turns = slot.console_local_turns + if local_turns is None: + local_turns = set() + slot.console_local_turns = local_turns + local_turns.add(local_turn_id) + try: + result = submit_command( + self.config, + json.dumps(request, sort_keys=True, separators=(",", ":")), + acp_prompt_router=self.prompt_route, + acp_required=True, + acp_observation_only=False, + acp_permission_router=self, + ) + except Exception: + if local_turn_id is not None: + with slot.lock: + if slot.console_local_turns is not None: + slot.console_local_turns.discard(local_turn_id) + raise + if result.status not in {"accepted", "duplicate_request"}: + if local_turn_id is not None: + with slot.lock: + if slot.console_local_turns is not None: + slot.console_local_turns.discard(local_turn_id) + raise AcpCoordinatorError("ACP console input was not accepted") + if request["action"] == "answer_decision": + return "permission" + if result.status == "duplicate_request" and local_turn_id is not None: + # A duplicate receipt does not start a new ACP turn, so there is no + # corresponding user event left to suppress in this process. + with slot.lock: + if slot.console_local_turns is not None: + slot.console_local_turns.discard(local_turn_id) + return "instruction" + def _require_reconcile_state(self, *, allow_starting: bool) -> None: with self._lock: allowed = {RuntimeState.RUNNING} @@ -483,8 +1003,44 @@ def _reconcile_binding(self, continuity: WorkerBinding) -> None: permission_broker.close() self._stop_runtime(runtime) raise + runtime_binding = getattr(runtime, "_binding", None) + console_cursor = 0 + console_input_cursor = 0 + console_cursor_loaded = False + if ( + endpoint.console is not None + and isinstance(runtime_binding, WorkerBinding) + and runtime_binding.turn_target_value + ): + console_cursor = _initial_console_event_cursor( + Path(self.config.db_path), + self.config.host_id, + continuity.worker_id, + runtime_binding.turn_target_value, + ) + console_cursor_loaded = True + console_input_cursor = _load_console_input_cursor( + Path(self.config.db_path), + self.config.host_id, + continuity.worker_id, + runtime_binding.turn_target_value, + endpoint.console.generation, + ) slot = _RuntimeSlot( - continuity, endpoint.generation, runtime, permission_broker + continuity, + endpoint.generation, + runtime, + permission_broker, + console=endpoint.console, + console_input_sequence=console_input_cursor, + console_event_sequence=console_cursor, + console_cursor_loaded=console_cursor_loaded, + console_local_turns=set(), + console_executor=ThreadPoolExecutor( + max_workers=1, + thread_name_prefix=f"tendwire-acp-console-{continuity.worker_id}", + ), + console_submissions={}, ) with self._lock: displaced = self._slots.get(continuity.worker_id) @@ -532,6 +1088,7 @@ def _require_attached_generation(self, slot: _RuntimeSlot) -> None: raise if ( status.lifecycle != "acp_owned_attached" + or status.console_lifecycle != "attached" or status.generation != slot.generation ): self._retire_worker(slot.continuity.worker_id, expected=slot) @@ -660,15 +1217,28 @@ def _retire_worker( worker_id: str, *, expected: _RuntimeSlot | None = None, + preserve_console_failure: bool = False, ) -> None: - with self._lock: - slot = self._slots.get(worker_id) - if slot is None or (expected is not None and slot is not expected): - return - self._slots.pop(worker_id, None) - if slot.permission_broker is not None: - slot.permission_broker.close() - self._stop_runtime(slot.runtime) + with self._reconcile_lock: + with self._lock: + slot = self._slots.get(worker_id) + if slot is None or (expected is not None and slot is not expected): + return + self._slots.pop(worker_id, None) + self._retired_slots.append(slot) + if not preserve_console_failure: + self._console_failed_workers.discard(worker_id) + self._console_degraded = bool(self._console_failed_workers) + if not self._console_degraded: + self._console_failure_type = None + with slot.lock: + slot.retired = True + executor = slot.console_executor + if slot.permission_broker is not None: + slot.permission_broker.close() + if executor is not None: + executor.shutdown(wait=False, cancel_futures=True) + self._stop_runtime(slot.runtime) def _stop_runtime(self, runtime: AcpRuntime, *, timeout: float | None = None) -> None: binding = getattr(runtime, "_binding", None) @@ -714,14 +1284,32 @@ def _stop_all(self, *, timeout: float | None = None) -> None: ) deadline = time.monotonic() + total for slot in slots: + # Bridge-held slot critical sections are deliberately local and + # non-blocking; synchronize the retirement flag with every reader + # so shutdown cannot race a last console mutation or submission. + with slot.lock: + slot.retired = True + executor = slot.console_executor if slot.permission_broker is not None: slot.permission_broker.close() + if executor is not None: + executor.shutdown(wait=False, cancel_futures=True) self._stop_runtime( slot.runtime, timeout=max(0.001, deadline - time.monotonic()), ) +def _slot_has_live_work(slot: _RuntimeSlot) -> bool: + with slot.lock: + thread = slot.console_bridge_thread + futures = tuple((slot.console_submissions or {}).values()) + return bool( + (thread is not None and thread.is_alive()) + or any(not future.done() for future in futures) + ) + + def _default_endpoint_client_factory(config: Config) -> HerdrSocketClient: return HerdrSocketClient(timeout=config.herdr_timeout_seconds) @@ -833,6 +1421,7 @@ def _parse_endpoint( if not isinstance(value, Mapping) or set(value) != { "type", "endpoint", + "console", "worker", "adapter", "session", @@ -846,12 +1435,17 @@ def _parse_endpoint( ): raise AcpCoordinatorError("Herdr ACP endpoint is not ready") endpoint = value.get("endpoint") + console = value.get("console") worker = value.get("worker") adapter = value.get("adapter") session = value.get("session") - if not all(isinstance(item, Mapping) for item in (endpoint, worker, adapter, session)): + if not all( + isinstance(item, Mapping) + for item in (endpoint, console, worker, adapter, session) + ): raise AcpCoordinatorError("Herdr ACP endpoint nested shape is invalid") assert isinstance(endpoint, Mapping) + assert isinstance(console, Mapping) assert isinstance(worker, Mapping) assert isinstance(adapter, Mapping) assert isinstance(session, Mapping) @@ -874,6 +1468,13 @@ def _parse_endpoint( ): raise AcpCoordinatorError("Herdr ACP endpoint generation is invalid") generation = str(raw_generation) + if set(console) != {"generation", "lease"}: + raise AcpCoordinatorError("Herdr ACP console endpoint shape is invalid") + if console.get("generation") != raw_generation: + raise AcpCoordinatorError("Herdr ACP console generation is inconsistent") + console_lease = _nonempty_text(console.get("lease"), "console lease") + if len(console_lease) > 512: + raise AcpCoordinatorError("Herdr ACP console lease is invalid") args = endpoint.get("args") if ( not isinstance(args, list) @@ -932,6 +1533,7 @@ def _parse_endpoint( generation=generation, session_mode=mode, session_id=session_id, + console=HerdrAcpConsoleEndpoint(raw_generation, console_lease), ) @@ -947,14 +1549,18 @@ def _parse_status( "session", "cwd", "lifecycle", + "console_lifecycle", }: raise AcpCoordinatorError("Herdr ACP status response shape is invalid") lifecycle = value.get("lifecycle") + console_lifecycle = value.get("console_lifecycle") if value.get("type") != "agent_acp_status" or lifecycle not in { "acp_owned_ready", "acp_owned_attached", }: raise AcpCoordinatorError("Herdr ACP status lifecycle is invalid") + if console_lifecycle not in {"starting", "attached", "missing"}: + raise AcpCoordinatorError("Herdr ACP console lifecycle is invalid") worker = value.get("worker") adapter = value.get("adapter") session = value.get("session") @@ -1008,7 +1614,346 @@ def _parse_status( cwd = Path(_nonempty_text(value.get("cwd"), "cwd")) if not cwd.is_absolute(): raise AcpCoordinatorError("Herdr ACP status cwd must be absolute") - return HerdrAcpStatus(str(raw_generation), str(lifecycle)) + return HerdrAcpStatus( + str(raw_generation), str(lifecycle), str(console_lifecycle) + ) + + +def _parse_console_exchange( + value: Any, after_sequence: int +) -> tuple[tuple[int, str], ...]: + if not isinstance(value, Mapping) or set(value) != { + "type", + "inputs", + "outputs", + "input_floor_sequence", + "output_floor_sequence", + "next_input_sequence", + "next_output_sequence", + }: + raise AcpCoordinatorError("Herdr ACP console exchange shape is invalid") + if value.get("type") != "agent_acp_console_exchange": + raise AcpCoordinatorError("Herdr ACP console exchange type is invalid") + raw_inputs = value.get("inputs") + raw_outputs = value.get("outputs") + if not isinstance(raw_inputs, list) or not isinstance(raw_outputs, list): + raise AcpCoordinatorError("Herdr ACP console exchange lists are invalid") + input_floor = value.get("input_floor_sequence") + output_floor = value.get("output_floor_sequence") + next_input = value.get("next_input_sequence") + next_output = value.get("next_output_sequence") + if ( + type(after_sequence) is not int + or after_sequence < 0 + or after_sequence > (1 << 64) - 1 + or type(input_floor) is not int + or type(output_floor) is not int + or input_floor < 0 + or output_floor < 0 + or type(next_input) is not int + or type(next_output) is not int + or next_input < input_floor + or next_output < output_floor + or next_input > (1 << 64) - 1 + or next_output > (1 << 64) - 1 + ): + raise AcpCoordinatorError("Herdr ACP console exchange floors are invalid") + if input_floor > after_sequence + 1: + raise AcpConsoleInputGap("Herdr ACP console input floor has a gap") + parsed: list[tuple[int, str]] = [] + expected = after_sequence + 1 + for item in raw_inputs: + if not isinstance(item, Mapping) or set(item) != {"sequence", "text"}: + raise AcpCoordinatorError("Herdr ACP console input shape is invalid") + sequence = item.get("sequence") + text = item.get("text") + if ( + type(sequence) is not int + or sequence != expected + or not isinstance(text, str) + or not text.strip() + ): + if type(sequence) is int and sequence > expected: + raise AcpConsoleInputGap("Herdr ACP console input sequence has a gap") + raise AcpCoordinatorError("Herdr ACP console input sequence is invalid") + parsed.append((sequence, text)) + expected += 1 + if next_input != expected: + if next_input > expected: + raise AcpConsoleInputGap( + "Herdr ACP console input response is incomplete" + ) + raise AcpCoordinatorError( + "Herdr ACP console next input sequence is invalid" + ) + output_expected = output_floor + allowed_streams = { + "user", + "assistant", + "thought", + "tool", + "plan", + "status", + "error", + "turn_end", + } + for item in raw_outputs: + if not isinstance(item, Mapping) or set(item) != { + "sequence", + "event_id", + "stream", + "text", + }: + raise AcpCoordinatorError("Herdr ACP console output shape is invalid") + sequence = item.get("sequence") + event_id = item.get("event_id") + stream = item.get("stream") + text = item.get("text") + if ( + type(sequence) is not int + or sequence != output_expected + or not isinstance(event_id, str) + or not event_id + or len(event_id) > 512 + or stream not in allowed_streams + or not isinstance(text, str) + ): + raise AcpCoordinatorError("Herdr ACP console output item is invalid") + output_expected += 1 + if next_output != output_expected: + raise AcpCoordinatorError( + "Herdr ACP console next output sequence is invalid" + ) + return tuple(parsed) + + +def _console_event_output( + kind: str, payload: Mapping[str, Any] +) -> tuple[str, str] | None: + if kind in {"user_message", "agent_message", "thought"}: + delta = payload.get("text_delta") + if not isinstance(delta, str) or not delta: + return None + return ( + { + "user_message": "user", + "agent_message": "assistant", + "thought": "thought", + }[kind], + delta, + ) + if kind in {"tool_call", "tool_call_update"}: + snapshot = payload.get("snapshot") + if not isinstance(snapshot, Mapping): + return None + title = str(snapshot.get("title") or snapshot.get("kind") or "tool") + status = str(snapshot.get("status") or "updated") + details = [f"{title} [{status}]"] + content = snapshot.get("content") + if isinstance(content, list): + for item in content: + if not isinstance(item, Mapping): + continue + if item.get("type") == "content": + block = item.get("content") + if ( + isinstance(block, Mapping) + and block.get("type") == "text" + and isinstance(block.get("text"), str) + ): + details.append(str(block["text"])) + elif item.get("type") == "diff" and isinstance(item.get("path"), str): + details.append(f"diff {item['path']}") + if isinstance(item.get("oldText"), str): + details.append("- " + str(item["oldText"])) + if isinstance(item.get("newText"), str): + details.append("+ " + str(item["newText"])) + # Herdr also enforces a bounded console item. Bound before transport so + # a single verbose tool result cannot starve all later events. + return "tool", "\n".join(details)[: 256 * 1024] + if kind == "plan": + entries = payload.get("entries") + if not isinstance(entries, list): + return None + lines = [ + f"[{str(item.get('status') or 'pending')}] {str(item.get('content') or '').strip()}" + for item in entries + if isinstance(item, Mapping) and str(item.get("content") or "").strip() + ] + return ("plan", "\n".join(lines)) if lines else None + if kind == "usage": + return "status", "usage " + json.dumps( + dict(payload), sort_keys=True, separators=(",", ":") + ) + if kind == "session_info" and payload.get("title"): + return "status", f"session {payload['title']}" + if ( + kind == "extension" + and payload.get("extension") == "tendwire.acp.prompt_completion" + ): + return "status", f"turn {str(payload.get('outcome') or 'complete')}" + return None + + +def _load_console_event_cursor( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, +) -> int | None: + after = 0 + cursor = 0 + found = False + while True: + page = list_agent_events( + db_path, + host_id, + worker_id=worker_id, + source="tendwire-console", + session_id=session_id, + after_sequence=after, + limit=1000, + ) + if not page: + return cursor if found else None + for stored in page: + after = max(after, stored.sequence) + value = stored.event.payload.get("sequence") + if type(value) is int and value >= 0: + found = True + cursor = max(cursor, value) + if len(page) < 1000: + return cursor if found else None + + +def _initial_console_event_cursor( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, +) -> int: + persisted = _load_console_event_cursor( + db_path, host_id, worker_id, session_id + ) + if persisted is not None: + return persisted + after = 0 + latest = 0 + while True: + page = list_agent_events( + db_path, + host_id, + worker_id=worker_id, + source="acp", + session_id=session_id, + after_sequence=after, + limit=1000, + ) + if not page: + return latest + latest = max(latest, max(item.sequence for item in page)) + after = latest + if len(page) < 1000: + return latest + + +def _load_console_input_cursor( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, + generation: int, +) -> int: + after = 0 + cursor = 0 + while True: + page = list_agent_events( + db_path, + host_id, + worker_id=worker_id, + source="tendwire-console", + session_id=session_id, + after_sequence=after, + limit=1000, + ) + if not page: + return cursor + for stored in page: + after = max(after, stored.sequence) + payload = stored.event.payload + if ( + payload.get("extension") == "tendwire.acp.console_input_cursor" + and payload.get("generation") == generation + and type(payload.get("input_sequence")) is int + ): + cursor = max(cursor, int(payload["input_sequence"])) + if len(page) < 1000: + return cursor + + +def _console_pending_decision( + db_path: Path, + host_id: str, + worker_id: str, +) -> tuple[str, tuple[tuple[str, str], ...], str] | None: + payload = pending_payload_from_store(db_path, host_id) + interactions = payload.get("pending_interactions") + if not isinstance(interactions, list): + return None + matches = [ + item + for item in interactions + if isinstance(item, Mapping) + and item.get("worker_id") == worker_id + and item.get("status") not in {"resolved", "closed", "stale"} + and isinstance(item.get("meta"), Mapping) + and isinstance(item["meta"].get("decision"), Mapping) + ] + if len(matches) != 1: + return None + item = matches[0] + decision = item["meta"]["decision"] + decision_ref = decision.get("decision_ref") + raw_options = decision.get("options") + if not isinstance(decision_ref, str) or not isinstance(raw_options, list): + return None + options = tuple( + (str(option.get("ref") or ""), str(option.get("label") or "")) + for option in raw_options + if isinstance(option, Mapping) + and str(option.get("ref") or "") + and str(option.get("label") or "") + ) + if not options: + return None + question = str(item.get("question") or "Permission required") + choices = " ".join(f"{ref}={label}" for ref, label in options) + return decision_ref, options, f"permission> {question} ({choices}); reply with a number or /cancel" + + +def _console_permission_selection( + text: str, options: tuple[tuple[str, str], ...] +) -> str | None: + value = text.strip().lower() + for ref, label in options: + if value == ref.lower() or value == label.lower(): + return ref + aliases = { + "allow": ("allow",), + "yes": ("allow",), + "deny": ("deny", "reject"), + "reject": ("deny", "reject"), + "no": ("deny", "reject"), + } + terms = aliases.get(value) + if terms is None: + return None + matches = [ + ref + for ref, label in options + if any(term in label.lower() for term in terms) + ] + return matches[0] if len(matches) == 1 else None def production_acp_runtime_factory( diff --git a/src/tendwire/backends/herdr_socket.py b/src/tendwire/backends/herdr_socket.py index 5c44c45..e3aeca6 100644 --- a/src/tendwire/backends/herdr_socket.py +++ b/src/tendwire/backends/herdr_socket.py @@ -326,6 +326,31 @@ def agent_acp_status( timeout=timeout, ) + def agent_acp_console_exchange( + self, + target: str, + *, + generation: int, + lease: str, + after_input_sequence: int = 0, + output: Iterable[Mapping[str, Any]] = (), + timeout: float | None = None, + ) -> Any: + """Exchange pane input and idempotent ACP output as coordinator.""" + return self.request( + "agent.acp_console_exchange", + { + "target": target, + "generation": generation, + "lease": lease, + "role": "coordinator", + "output": [dict(item) for item in output], + "after_input_sequence": after_input_sequence, + "after_output_sequence": 0, + }, + timeout=timeout, + ) + def _send_request( self, method: str, diff --git a/src/tendwire/core/commands.py b/src/tendwire/core/commands.py index 1c9a3a0..ff145f0 100644 --- a/src/tendwire/core/commands.py +++ b/src/tendwire/core/commands.py @@ -189,7 +189,16 @@ ) # Neutral target fields permitted in command requests. -TARGET_ALLOWED_FIELDS = frozenset({"worker_id", "worker_fingerprint", "space_id", "name"}) +TARGET_ALLOWED_FIELDS = frozenset( + { + "worker_id", + "worker_fingerprint", + "space_id", + "name", + "stable_key", + "stable_key_version", + } +) # Selectors that name a worker durably. A worker_fingerprint is a mutable # observation precondition -- "proceed only if the worker still looks like this" @@ -197,7 +206,9 @@ # fingerprints would then be indistinguishable to any identity-based idempotency # key, letting one request ID claim another worker's stored result. Every target # must carry at least one of these. -TARGET_STABLE_SELECTOR_FIELDS = frozenset({"worker_id", "space_id", "name"}) +TARGET_STABLE_SELECTOR_FIELDS = frozenset( + {"worker_id", "space_id", "name", "stable_key"} +) INSTRUCTION_ALLOWED_FIELDS = frozenset({"text"}) ANSWER_PENDING_PARAM_FIELDS = frozenset( {"pending_id", "pending_fingerprint", "choice_id"} @@ -212,6 +223,7 @@ MAX_INSTRUCTION_LENGTH = 4096 _REQUEST_ID_RE = re.compile(r"[A-Za-z0-9._-]{1,128}", re.ASCII) +_STABLE_WORKER_KEY_RE = re.compile(r"wsk1_[0-9a-f]{64}", re.ASCII) _TURN_SUBMISSION_ID_RE = re.compile(r"twsub1\.[0-9a-f]{64}", re.ASCII) _INSTRUCTION_FINGERPRINT_DOMAIN = b"tendwire.instruction-fingerprint.v1" _TURN_SUBMISSION_ID_DOMAIN = b"tendwire.turn-submission-id.v1" @@ -422,6 +434,27 @@ def _validate_target_shape(target: dict[str, Any] | None) -> dict[str, Any] | No f"target contains disallowed fields: {sorted(extra)}", details={"field": "target", "disallowed": sorted(extra)}, ) + stable_key_present = "stable_key" in target + stable_version_present = "stable_key_version" in target + stable_key = target.get("stable_key") + stable_version = target.get("stable_key_version") + if stable_key_present != stable_version_present: + return error_value( + STATUS_INVALID_REQUEST, + "target stable_key and stable_key_version must be supplied together", + details={"field": "target"}, + ) + if stable_key_present and ( + not isinstance(stable_key, str) + or _STABLE_WORKER_KEY_RE.fullmatch(stable_key) is None + or type(stable_version) is not int + or stable_version != 1 + ): + return error_value( + STATUS_INVALID_REQUEST, + "target stable worker identity is invalid or unsupported", + details={"field": "target"}, + ) if _string_value(target.get("worker_fingerprint")) and not _target_has_stable_selector( target ): @@ -684,6 +717,8 @@ def build_selector_proof(request: CommandRequest) -> str: "worker_id": _string_value(target.get("worker_id")), "name": _string_value(target.get("name")), "space_id": _optional_string(target.get("space_id")), + "stable_key": _string_value(target.get("stable_key")), + "stable_key_version": target.get("stable_key_version"), } payload = { "proof_version": SELECTOR_PROOF_VERSION, @@ -1296,6 +1331,8 @@ def resolve_target( name = _string_value(target.get("name")) space_id = _optional_string(target.get("space_id")) fingerprint = _optional_string(target.get("worker_fingerprint")) + stable_key = _string_value(target.get("stable_key")) + stable_key_version = target.get("stable_key_version") # First match by identity/name/space, excluding fingerprint. identity_matches: list[Worker] = [] @@ -1306,6 +1343,11 @@ def resolve_target( continue if space_id is not None and worker.space_id != space_id: continue + if stable_key and ( + worker.meta.get("stable_key") != stable_key + or worker.meta.get("stable_key_version") != stable_key_version + ): + continue identity_matches.append(worker) # If a fingerprint was supplied, filter further. A non-empty identity match diff --git a/tests/fixtures/herdr_acp_contract_v1.json b/tests/fixtures/herdr_acp_contract_v1.json index 1aa2627..f32ee41 100644 --- a/tests/fixtures/herdr_acp_contract_v1.json +++ b/tests/fixtures/herdr_acp_contract_v1.json @@ -20,6 +20,10 @@ ], "protocol_version": 1 }, + "console": { + "generation": 42, + "lease": "console-coordinator-private-lease" + }, "worker": { "terminal_id": "term_abc", "workspace_id": "w1", diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index dbeaf5f..c30cb4e 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -16,12 +16,20 @@ from tendwire.backends.acp_coordinator import ( AcpCoordinatorError, AcpRuntimeCoordinator, + HerdrAcpConsoleEndpoint, + _RuntimeSlot, _derived_binding, + _console_event_output, + _console_permission_selection, + _load_console_event_cursor, + _load_console_input_cursor, + _parse_console_exchange, _parse_endpoint, _parse_status, production_acp_runtime_factory, ) from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult +from tendwire.backends.acp_runtime import RuntimeState from tendwire.command_submission import submit_acp_command, submit_command from tendwire.config import Config from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding @@ -30,6 +38,7 @@ get_command_request, init_store, list_worker_bindings, + record_agent_event, save_snapshot, upsert_worker_bindings, ) @@ -78,6 +87,10 @@ def _endpoint(*, generation: int = 42, lifecycle: str = "acp_owned_ready") -> di ], "protocol_version": 1, }, + "console": { + "generation": generation, + "lease": "console-coordinator-private-lease", + }, "worker": { "terminal_id": "pane-private", "workspace_id": "workspace-private", @@ -103,6 +116,7 @@ def _status(*, generation: int = 42, lifecycle: str = "acp_owned_attached") -> d "session": endpoint["session"], "cwd": endpoint["cwd"], "lifecycle": lifecycle, + "console_lifecycle": "attached", } @@ -139,6 +153,402 @@ def test_endpoint_requires_explicit_acp_ownership_and_strict_attach_shape(tmp_pa _parse_status(terminal_binding, wrong_status) +def test_console_exchange_requires_floor_and_next_sequence_contract() -> None: + result = { + "type": "agent_acp_console_exchange", + "inputs": [{"sequence": 3, "text": "continue"}], + "outputs": [], + "input_floor_sequence": 3, + "output_floor_sequence": 1, + "next_input_sequence": 4, + "next_output_sequence": 1, + } + assert _parse_console_exchange(result, 2) == ((3, "continue"),) + missing_next = dict(result) + missing_next.pop("next_input_sequence") + with pytest.raises(AcpCoordinatorError, match="shape"): + _parse_console_exchange(missing_next, 2) + lost = dict(result, input_floor_sequence=4) + with pytest.raises(AcpCoordinatorError, match="gap"): + _parse_console_exchange(lost, 2) + + incomplete = dict(result, inputs=[], next_input_sequence=4) + with pytest.raises(AcpCoordinatorError, match="incomplete"): + _parse_console_exchange(incomplete, 2) + + output = dict( + result, + outputs=[ + { + "sequence": 7, + "event_id": "event-7", + "stream": "assistant", + "text": "done", + } + ], + output_floor_sequence=7, + next_output_sequence=8, + ) + assert _parse_console_exchange(output, 2) == ((3, "continue"),) + malformed_output = dict(output, outputs=[{"sequence": 7, "text": "done"}]) + with pytest.raises(AcpCoordinatorError, match="output shape"): + _parse_console_exchange(malformed_output, 2) + wrong_output_next = dict(output, next_output_sequence=9) + with pytest.raises(AcpCoordinatorError, match="next output"): + _parse_console_exchange(wrong_output_next, 2) + + +def test_console_event_projection_covers_messages_thought_tools_and_plan() -> None: + assert _console_event_output("agent_message", {"text_delta": "done"}) == ( + "assistant", + "done", + ) + assert _console_event_output("thought", {"text_delta": "reason"}) == ( + "thought", + "reason", + ) + assert _console_event_output( + "tool_call_update", {"snapshot": {"title": "pytest", "status": "completed"}} + ) == ("tool", "pytest [completed]") + assert _console_event_output( + "tool_call_update", + { + "snapshot": { + "title": "Edit source", + "status": "completed", + "rawInput": {"secret": "not rendered"}, + "rawOutput": "not rendered", + "content": [ + {"type": "content", "content": {"type": "text", "text": "done"}}, + { + "type": "diff", + "path": "/workspace/source.py", + "oldText": "old", + "newText": "new", + }, + ], + } + }, + ) == ( + "tool", + "Edit source [completed]\ndone\ndiff /workspace/source.py\n- old\n+ new", + ) + assert _console_event_output( + "plan", {"entries": [{"content": "verify", "status": "in_progress"}]} + ) == ("plan", "[in_progress] verify") + + +def test_console_permission_selection_is_explicit_and_fail_closed() -> None: + options = (("1", "Allow once (allow_once)"), ("2", "Reject (reject_once)")) + assert _console_permission_selection("1", options) == "1" + assert _console_permission_selection("allow", options) == "1" + assert _console_permission_selection("deny", options) == "2" + assert _console_permission_selection("do something else", options) is None + + +def test_console_cursors_survive_restart_crash_boundaries(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + record_agent_event( + config.db_path, + config.host_id, + kind="extension", + source="tendwire-console", + worker_id="worker-1", + payload={"extension": "tendwire.acp.console_cursor", "sequence": 41}, + source_session_id="session-a", + source_event_id="cursor:41", + visibility="private", + ) + record_agent_event( + config.db_path, + config.host_id, + kind="extension", + source="tendwire-console", + worker_id="worker-1", + payload={ + "extension": "tendwire.acp.console_input_cursor", + "generation": 42, + "input_sequence": 7, + }, + source_session_id="session-a", + source_event_id="input:42:7", + visibility="private", + ) + assert _load_console_event_cursor( + config.db_path, config.host_id, "worker-1", "session-a" + ) == 41 + assert _load_console_input_cursor( + config.db_path, config.host_id, "worker-1", "session-a", 42 + ) == 7 + assert _load_console_input_cursor( + config.db_path, config.host_id, "worker-1", "session-a", 43 + ) == 0 + + +def test_console_bridge_polls_independently_of_slow_reconcile_interval( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), threading.Event(), reconcile_interval=60.0 + ) + ticks: list[float] = [] + + def tick() -> None: + ticks.append(time.monotonic()) + if len(ticks) == 3: + coordinator._stop.set() + + coordinator._bridge_console_slots = tick # type: ignore[method-assign] + started = time.monotonic() + coordinator._run_console_bridge() + assert len(ticks) == 3 + assert ticks[-1] - started < 0.5 + + +def test_console_bridge_dispatches_slow_workers_independently(tmp_path: Path) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), threading.Event(), reconcile_interval=60.0 + ) + slow_entered = threading.Event() + fast_entered = threading.Event() + release = threading.Event() + + slow_slot = _RuntimeSlot( + _binding(), + "42", + SimpleNamespace(), + console=HerdrAcpConsoleEndpoint(42, "slow-lease"), + ) + fast_slot = _RuntimeSlot( + replace(_binding(), worker_id="worker-2"), + "43", + SimpleNamespace(), + console=HerdrAcpConsoleEndpoint(43, "fast-lease"), + ) + coordinator._slots = {"worker-1": slow_slot, "worker-2": fast_slot} + + def bridge(slot: _RuntimeSlot) -> None: + if slot is slow_slot: + slow_entered.set() + assert release.wait(2.0) + else: + fast_entered.set() + + coordinator._bridge_console_slot = bridge # type: ignore[method-assign] + coordinator._bridge_console_slots() + try: + assert slow_entered.wait(1.0) + assert fast_entered.wait(1.0) + finally: + release.set() + for slot in (slow_slot, fast_slot): + thread = slot.console_bridge_thread + if thread is not None: + thread.join(timeout=1.0) + + +def test_console_failure_remains_degraded_after_slot_disappears(tmp_path: Path) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path, policy="acp_required"), + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + slot = _RuntimeSlot( + _binding(), + "42", + SimpleNamespace(), + console=HerdrAcpConsoleEndpoint(42, "console-lease"), + ) + + def fail(_slot: _RuntimeSlot) -> None: + raise OSError("console unavailable") + + coordinator._bridge_console_slot = fail # type: ignore[method-assign] + coordinator._bridge_console_slot_supervised(slot) + assert coordinator.status()["healthy"] is False + coordinator._slots.clear() + coordinator._bridge_console_slots() + status = coordinator.status() + assert status["healthy"] is False + assert status["failure_type"] == "OSError" + + +def test_console_submission_rejects_a_retired_generation_before_store_access( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), threading.Event(), reconcile_interval=60.0 + ) + coordinator._state = RuntimeState.RUNNING + stale = _RuntimeSlot(_binding(), "42", SimpleNamespace()) + replacement = _RuntimeSlot(_binding(), "43", SimpleNamespace()) + coordinator._slots["worker-1"] = replacement + with pytest.raises(AcpCoordinatorError, match="generation is stale"): + coordinator._submit_console_input(stale, 1, "must not cross sessions") + + +def test_console_local_turn_is_suppressed_before_acp_submission_emits( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + worker = Worker( + id="worker-1", + name="Agent", + status="active", + meta={"stable_key": "wsk1_" + "a" * 64, "stable_key_version": 1}, + ) + snapshot = Snapshot( + host_id=config.host_id, + updated_at="2026-01-01T00:00:00+00:00", + workers=[worker], + ) + monkeypatch.setattr( + "tendwire.backends.acp_coordinator.latest_snapshot", + lambda _path, _host: snapshot, + ) + observed: list[set[str]] = [] + + def submit_while_observing(*_args: Any, **_kwargs: Any) -> Any: + observed.append(set(slot.console_local_turns or ())) + return SimpleNamespace(status="accepted") + + monkeypatch.setattr( + "tendwire.command_submission.submit_command", submit_while_observing + ) + continuity = replace(_binding(), worker_fingerprint=worker.fingerprint) + runtime = SimpleNamespace(_session_id="session-a") + slot = _RuntimeSlot( + continuity, + "42", + runtime, + console_local_turns=set(), + ) + coordinator = AcpRuntimeCoordinator( + config, threading.Event(), reconcile_interval=60.0 + ) + + assert coordinator._submit_console_input_fenced(slot, 1, "hello") == "instruction" + assert len(observed) == 1 + assert len(observed[0]) == 1 + assert slot.console_local_turns == observed[0] + + +def test_stop_reports_failed_while_console_submission_thread_is_still_running( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), threading.Event(), reconcile_interval=60.0 + ) + coordinator._state = RuntimeState.RUNNING + entered = threading.Event() + release = threading.Event() + executor = ThreadPoolExecutor(max_workers=1) + + def block() -> None: + entered.set() + assert release.wait(2.0) + + future = executor.submit(block) + assert entered.wait(1.0) + runtime = SimpleNamespace(stop=lambda *, timeout: None) + slot = _RuntimeSlot( + _binding(), + "42", + runtime, + console_executor=executor, + console_submissions={1: future}, + ) + coordinator._slots["worker-1"] = slot + try: + coordinator.stop(timeout=0.05) + status = coordinator.status() + assert status["state"] == "failed" + assert status["failure_type"] == "AcpRuntimeStopTimeout" + finally: + release.set() + future.result(timeout=1.0) + + +def test_console_input_submission_worker_does_not_block_event_exchange( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + entered = threading.Event() + release = threading.Event() + exchanges: list[int] = [] + + class EndpointClient: + def connect(self) -> None: + return None + + def close(self) -> None: + return None + + def agent_acp_console_exchange(self, _target: str, **params: Any) -> Any: + exchanges.append(int(params["after_input_sequence"])) + return { + "type": "agent_acp_console_exchange", + "inputs": [{"sequence": 1, "text": "stream while I run"}], + "outputs": [], + "input_floor_sequence": 1, + "output_floor_sequence": 1, + "next_input_sequence": 2, + "next_output_sequence": 1, + } + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + reconcile_interval=60.0, + ) + + def blocking_submit(_slot: Any, _sequence: int, _text: str) -> str: + entered.set() + assert release.wait(2.0) + return "instruction" + + coordinator._submit_console_input = blocking_submit # type: ignore[method-assign] + runtime = SimpleNamespace( + _binding=replace( + _binding(), + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + ) + ) + executor = ThreadPoolExecutor(max_workers=1) + slot = _RuntimeSlot( + continuity=_binding(), + generation="42", + runtime=runtime, + console=HerdrAcpConsoleEndpoint(42, "coordinator-lease"), + console_cursor_loaded=True, + console_executor=executor, + console_submissions={}, + console_local_turns=set(), + ) + try: + coordinator._bridge_console_slot(slot) + assert entered.wait(1.0) + started = time.monotonic() + coordinator._bridge_console_slot(slot) + assert time.monotonic() - started < 0.5 + assert exchanges == [0, 0] + assert slot.console_input_sequence == 0 + finally: + release.set() + executor.shutdown(wait=True, cancel_futures=True) + + def test_canonical_herdr_acp_contract_fixture_executes_configured_binary( tmp_path: Path, ) -> None: diff --git a/tests/test_command_replay_authority.py b/tests/test_command_replay_authority.py index 0a3d31d..d76022d 100644 --- a/tests/test_command_replay_authority.py +++ b/tests/test_command_replay_authority.py @@ -483,6 +483,7 @@ def test_fingerprint_only_target_is_rejected_by_daemon_and_read_only_replay( assert response["result"]["error"]["details"]["allowed"] == [ "name", "space_id", + "stable_key", "worker_id", ] # The read-only response-loss path cannot resolve it either, and must never diff --git a/tests/test_commands.py b/tests/test_commands.py index 7ce2299..6a341bb 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -1108,7 +1108,12 @@ def test_validate_rejects_a_fingerprint_only_target(action: str) -> None: assert error is not None assert error["code"] == STATUS_INVALID_REQUEST assert "worker_fingerprint" in error["message"] - assert error["details"]["allowed"] == ["name", "space_id", "worker_id"] + assert error["details"]["allowed"] == [ + "name", + "space_id", + "stable_key", + "worker_id", + ] @pytest.mark.parametrize( @@ -1134,6 +1139,117 @@ def test_validate_accepts_a_fingerprint_beside_a_stable_selector( assert validate_request(request) is None +def test_stable_worker_key_is_a_strict_unique_target_selector() -> None: + stable_key = "wsk1_" + ("a" * 64) + workers = [ + Worker( + id="old-public-id", + name="Coda", + status="idle", + meta={"stable_key": stable_key, "stable_key_version": 1}, + ), + Worker( + id="other-worker", + name="Coda", + status="idle", + meta={"stable_key": "wsk1_" + ("b" * 64), "stable_key_version": 1}, + ), + ] + + resolved, candidates, status = resolve_target( + {"stable_key": stable_key, "stable_key_version": 1}, + workers, + ) + + assert status == STATUS_RESOLVED + assert resolved is not None and resolved["worker_id"] == "old-public-id" + assert [candidate["worker_id"] for candidate in candidates] == ["old-public-id"] + + +def test_stable_worker_key_selector_fails_closed_on_duplicate_live_owner() -> None: + stable_key = "wsk1_" + ("c" * 64) + workers = [ + Worker( + id=f"worker-{index}", + name=f"Coda {index}", + status="idle", + meta={"stable_key": stable_key, "stable_key_version": 1}, + ) + for index in (1, 2) + ] + resolved, candidates, status = resolve_target( + {"stable_key": stable_key, "stable_key_version": 1}, workers + ) + assert resolved is None + assert status == STATUS_AMBIGUOUS_TARGET + assert {item["worker_id"] for item in candidates} == {"worker-1", "worker-2"} + + +def test_stable_worker_key_and_worker_id_must_select_the_same_live_worker() -> None: + stable_key = "wsk1_" + ("d" * 64) + workers = [ + Worker( + id="worker-a", + name="A", + status="idle", + meta={"stable_key": stable_key, "stable_key_version": 1}, + ), + Worker(id="worker-b", name="B", status="idle"), + ] + resolved, candidates, status = resolve_target( + { + "worker_id": "worker-b", + "stable_key": stable_key, + "stable_key_version": 1, + }, + workers, + ) + assert (resolved, candidates, status) == (None, [], STATUS_NOT_FOUND) + + +def test_stable_worker_key_selector_never_trusts_non_meta_observation_fields() -> None: + stable_key = "wsk1_" + ("e" * 64) + worker = Worker(id=stable_key, name=stable_key, status="idle", meta={}) + resolved, candidates, status = resolve_target( + {"stable_key": stable_key, "stable_key_version": 1}, [worker] + ) + assert (resolved, candidates, status) == (None, [], STATUS_NOT_FOUND) + + +@pytest.mark.parametrize( + "target", + [ + {"stable_key": "wsk1_" + ("a" * 64)}, + {"stable_key_version": 1}, + {"stable_key": "wsk1_short", "stable_key_version": 1}, + {"stable_key": "wsk1_" + ("a" * 64), "stable_key_version": 2}, + {"stable_key": "wsk1_" + ("a" * 64), "stable_key_version": True}, + ], +) +def test_stable_worker_key_target_rejects_partial_or_unsupported_identity( + target: dict[str, Any], +) -> None: + request = CommandRequest(action="resolve_target", target=target) + error = validate_request(request) + assert error is not None + assert error["code"] == STATUS_INVALID_REQUEST + + +def test_selector_proof_fences_stable_worker_key_version() -> None: + def proof(key: str) -> str: + return build_selector_proof( + CommandRequest( + action="send_instruction", + request_id="stable-key-proof", + dry_run=False, + target={"stable_key": key, "stable_key_version": 1}, + instruction={"text": "hello"}, + ) + ) + + assert proof("wsk1_" + ("a" * 64)) != proof("wsk1_" + ("b" * 64)) + + @pytest.mark.parametrize( "stable", [ From 8a0ade6e0878055fa0fc042414e50415a684d9c0 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 19:41:46 +0800 Subject: [PATCH 44/83] fix: make ACP pane bridge crash and frame safe --- src/tendwire/backends/acp_coordinator.py | 285 +++++++++++++++++----- tests/fixtures/herdr_acp_contract_v1.json | 2 +- tests/test_acp_coordinator.py | 207 +++++++++++++++- 3 files changed, 436 insertions(+), 58 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index e6ee11e..b8fb3a1 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -54,6 +54,13 @@ class AcpConsoleInputGap(AcpCoordinatorError): """The bounded Herdr console queue lost unconsumed pane input.""" +# Herdr's aggregate console queue admission is 768 KiB and its normal control +# frame is 2 MiB. Keep Tendwire's retained output contribution below 512 KiB +# so queued pane input and the echoed response retain explicit headroom. +_CONSOLE_OUTPUT_QUEUE_BUDGET_BYTES = 512 * 1024 +_CONSOLE_OUTPUT_ITEM_TEXT_BYTES = 128 * 1024 + + @dataclass(frozen=True, slots=True) class HerdrAcpEndpoint: command: tuple[str, ...] @@ -597,26 +604,20 @@ def _bridge_console_slot(self, slot: _RuntimeSlot) -> None: completed_submissions.append(sequence) try: outcome = future.result() - if outcome in {"permission", "cancelled"}: - output.append( - { - "event_id": f"console-{outcome}:{console.generation}:{sequence}", - "stream": "status", - "text": ( - "permission selection accepted" - if outcome == "permission" - else "active turn cancellation requested" - ), - } - ) - except Exception as exc: - output.append( - { - "event_id": f"console-error:{console.generation}:{sequence}", - "stream": "error", - "text": f"instruction failed ({type(exc).__name__})", - } - ) + durable_outcome = str(outcome) + except Exception: + # Keep the durable payload deterministic across a retry of the + # same request id; exception types can differ across a crash. + durable_outcome = "error" + _record_console_submission_outcome( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + binding.turn_target_value, + generation=console.generation, + input_sequence=sequence, + outcome=durable_outcome, + ) # A terminal command receipt now exists (accepted or rejected), so # the next exchange may acknowledge this one Herdr queue item. record_agent_event( @@ -643,51 +644,98 @@ def _bridge_console_slot(self, slot: _RuntimeSlot) -> None: if pending is not None: decision_ref, _options, prompt = pending output.append( - { - "event_id": "permission:" + stable_fingerprint( + _bounded_console_output( + "permission:" + stable_fingerprint( { "worker_id": slot.continuity.worker_id, "decision_ref": decision_ref, } ), - "stream": "status", - "text": prompt, - } + "status", + prompt, + ) ) consumed_local_turns: set[str] = set() + processed_event_sequence = event_sequence for stored in events: event = stored.event if event.kind == "user_message" and event.source_turn_id in local_turns: if event.source_turn_id is not None: consumed_local_turns.add(event.source_turn_id) + processed_event_sequence = stored.sequence continue rendered = _console_event_output(event.kind, event.payload) if rendered is None: + processed_event_sequence = stored.sequence continue stream, text = rendered - output.append( - {"event_id": event.event_id, "stream": stream, "text": text} - ) + item = _bounded_console_output(event.event_id, stream, text) + if not _console_output_fits(output, item, budget=_CONSOLE_OUTPUT_QUEUE_BUDGET_BYTES): + break + output.append(item) + processed_event_sequence = stored.sequence client = self._endpoint_client_factory(self.config) + gap_error: AcpConsoleInputGap | None = None + inputs: tuple[tuple[int, str], ...] = () try: connect = getattr(client, "connect", None) if callable(connect): connect() + # Probe the retained Herdr output queue before publishing. Herdr + # echoes that queue in its response, so publishing blind can create + # a response larger than its 2 MiB control-frame limit. result = client.agent_acp_console_exchange( slot.continuity.target_value, generation=console.generation, lease=console.lease, after_input_sequence=input_sequence, - output=output, + output=(), timeout=self.config.herdr_timeout_seconds, ) + inputs = _parse_console_exchange(result, input_sequence) + retained_bytes = _console_exchange_output_bytes(result) + publish = _fit_console_output_batch( + output, + budget=max(0, _CONSOLE_OUTPUT_QUEUE_BUDGET_BYTES - retained_bytes), + ) + if len(publish) < len(output): + # Only ACP events after this prefix must be replayed. Synthetic + # permission output is first and idempotent, so a full queue may + # delay it without advancing any ACP event cursor. + published_ids = {item["event_id"] for item in publish} + processed_event_sequence = event_sequence + consumed_local_turns.clear() + for stored in events: + event = stored.event + if event.kind == "user_message" and event.source_turn_id in local_turns: + if event.source_turn_id is not None: + consumed_local_turns.add(event.source_turn_id) + processed_event_sequence = stored.sequence + continue + rendered = _console_event_output(event.kind, event.payload) + if rendered is None: + processed_event_sequence = stored.sequence + continue + if event.event_id not in published_ids: + break + processed_event_sequence = stored.sequence + if publish: + result = client.agent_acp_console_exchange( + slot.continuity.target_value, + generation=console.generation, + lease=console.lease, + after_input_sequence=input_sequence, + output=publish, + timeout=self.config.herdr_timeout_seconds, + ) + inputs = _parse_console_exchange(result, input_sequence) + except AcpConsoleInputGap as exc: + gap_error = exc finally: close = getattr(client, "close", None) if callable(close): close() - try: - inputs = _parse_console_exchange(result, input_sequence) - except AcpConsoleInputGap: + if gap_error is not None: gap_output = [{ "event_id": f"console-gap:{console.generation}:{input_sequence}", "stream": "error", @@ -710,17 +758,17 @@ def _bridge_console_slot(self, slot: _RuntimeSlot) -> None: close = getattr(client, "close", None) if callable(close): close() - raise + raise gap_error with slot.lock: if slot.retired: return slot.console_input_sequence = input_sequence for sequence in completed_submissions: submissions.pop(sequence, None) - if events: + if processed_event_sequence > event_sequence: # Herdr deduplicates output event_id values, so replaying a page # after a crash between exchange and this cursor update is safe. - next_event_sequence = max(item.sequence for item in events) + next_event_sequence = processed_event_sequence record_agent_event( Path(self.config.db_path), self.config.host_id, @@ -1012,11 +1060,13 @@ def _reconcile_binding(self, continuity: WorkerBinding) -> None: and isinstance(runtime_binding, WorkerBinding) and runtime_binding.turn_target_value ): - console_cursor = _initial_console_event_cursor( + console_cursor = _prepare_console_event_cursor( Path(self.config.db_path), self.config.host_id, continuity.worker_id, runtime_binding.turn_target_value, + session_mode=endpoint.session_mode, + generation=endpoint.console.generation, ) console_cursor_loaded = True console_input_cursor = _load_console_input_cursor( @@ -1792,9 +1842,66 @@ def _console_event_output( and payload.get("extension") == "tendwire.acp.prompt_completion" ): return "status", f"turn {str(payload.get('outcome') or 'complete')}" + if ( + kind == "extension" + and payload.get("extension") == "tendwire.acp.console_submission_outcome" + ): + outcome = payload.get("outcome") + if outcome == "permission": + return "status", "permission selection accepted" + if outcome == "cancelled": + return "status", "active turn cancellation requested" + if outcome == "error": + return "error", "instruction failed" return None +def _bounded_console_output(event_id: str, stream: str, text: str) -> dict[str, str]: + encoded = text.encode("utf-8") + if len(encoded) > _CONSOLE_OUTPUT_ITEM_TEXT_BYTES: + encoded = encoded[:_CONSOLE_OUTPUT_ITEM_TEXT_BYTES] + while True: + try: + text = encoded.decode("utf-8") + break + except UnicodeDecodeError as exc: + encoded = encoded[: exc.start] + text += "\n[console output truncated]" + return {"event_id": event_id, "stream": stream, "text": text} + + +def _console_output_wire_bytes(output: list[dict[str, str]]) -> int: + return len( + json.dumps(output, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + ) + + +def _console_output_fits( + output: list[dict[str, str]], item: dict[str, str], *, budget: int +) -> bool: + return _console_output_wire_bytes([*output, item]) <= budget + + +def _fit_console_output_batch( + output: list[dict[str, str]], *, budget: int +) -> list[dict[str, str]]: + fitted: list[dict[str, str]] = [] + for item in output: + if not _console_output_fits(fitted, item, budget=budget): + break + fitted.append(item) + return fitted + + +def _console_exchange_output_bytes(value: Any) -> int: + assert isinstance(value, Mapping) + outputs = value.get("outputs") + assert isinstance(outputs, list) + return len( + json.dumps(outputs, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + ) + + def _load_console_event_cursor( db_path: Path, host_id: str, @@ -1818,8 +1925,13 @@ def _load_console_event_cursor( return cursor if found else None for stored in page: after = max(after, stored.sequence) - value = stored.event.payload.get("sequence") - if type(value) is int and value >= 0: + payload = stored.event.payload + value = payload.get("sequence") + if ( + payload.get("extension") == "tendwire.acp.console_cursor" + and type(value) is int + and value >= 0 + ): found = True cursor = max(cursor, value) if len(page) < 1000: @@ -1837,24 +1949,85 @@ def _initial_console_event_cursor( ) if persisted is not None: return persisted - after = 0 - latest = 0 - while True: - page = list_agent_events( - db_path, - host_id, - worker_id=worker_id, - source="acp", - session_id=session_id, - after_sequence=after, - limit=1000, - ) - if not page: - return latest - latest = max(latest, max(item.sequence for item in page)) - after = latest - if len(page) < 1000: - return latest + return 0 + + +def _record_console_event_cursor( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, + sequence: int, + *, + source_event_id: str, +) -> None: + record_agent_event( + db_path, + host_id, + kind="extension", + source="tendwire-console", + worker_id=worker_id, + payload={"extension": "tendwire.acp.console_cursor", "sequence": sequence}, + source_session_id=session_id, + source_event_id=source_event_id, + visibility="private", + ) + + +def _prepare_console_event_cursor( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, + *, + session_mode: SessionOpenMode, + generation: int, +) -> int: + persisted = _load_console_event_cursor(db_path, host_id, worker_id, session_id) + if persisted is not None: + return persisted + # A missing checkpoint is never evidence that any stored ACP update was + # displayed. Replay from zero for NEW setup updates as well as LOAD/RESUME; + # Herdr's stable event ids make the conservative replay idempotent. + baseline = 0 + _record_console_event_cursor( + db_path, + host_id, + worker_id, + session_id, + baseline, + source_event_id=f"baseline:{generation}:{baseline}", + ) + return baseline + + +def _record_console_submission_outcome( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, + *, + generation: int, + input_sequence: int, + outcome: str, +) -> None: + record_agent_event( + db_path, + host_id, + kind="extension", + source="acp", + worker_id=worker_id, + payload={ + "schema_version": 1, + "extension": "tendwire.acp.console_submission_outcome", + "generation": generation, + "input_sequence": input_sequence, + "outcome": outcome, + }, + source_session_id=session_id, + source_event_id=f"console-outcome:{generation}:{input_sequence}", + visibility="private", + ) def _load_console_input_cursor( diff --git a/tests/fixtures/herdr_acp_contract_v1.json b/tests/fixtures/herdr_acp_contract_v1.json index f32ee41..37c3a73 100644 --- a/tests/fixtures/herdr_acp_contract_v1.json +++ b/tests/fixtures/herdr_acp_contract_v1.json @@ -22,7 +22,7 @@ }, "console": { "generation": 42, - "lease": "console-coordinator-private-lease" + "lease": "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQ" }, "worker": { "terminal_id": "term_abc", diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index c30cb4e..5b252b4 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -20,16 +20,21 @@ _RuntimeSlot, _derived_binding, _console_event_output, + _bounded_console_output, + _console_output_wire_bytes, + _fit_console_output_batch, _console_permission_selection, _load_console_event_cursor, _load_console_input_cursor, _parse_console_exchange, _parse_endpoint, _parse_status, + _prepare_console_event_cursor, + _record_console_submission_outcome, production_acp_runtime_factory, ) from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult -from tendwire.backends.acp_runtime import RuntimeState +from tendwire.backends.acp_runtime import RuntimeState, SessionOpenMode from tendwire.command_submission import submit_acp_command, submit_command from tendwire.config import Config from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding @@ -37,6 +42,7 @@ from tendwire.store.sqlite import ( get_command_request, init_store, + list_agent_events, list_worker_bindings, record_agent_event, save_snapshot, @@ -236,6 +242,11 @@ def test_console_event_projection_covers_messages_thought_tools_and_plan() -> No assert _console_event_output( "plan", {"entries": [{"content": "verify", "status": "in_progress"}]} ) == ("plan", "[in_progress] verify") + chunks = [ + _console_event_output("agent_message", {"text_delta": "hello"}), + _console_event_output("agent_message", {"text_delta": " world"}), + ] + assert "".join(item[1] for item in chunks if item is not None) == "hello world" def test_console_permission_selection_is_explicit_and_fail_closed() -> None: @@ -287,6 +298,197 @@ def test_console_cursors_survive_restart_crash_boundaries(tmp_path: Path) -> Non ) == 0 +def test_missing_checkpoint_replays_from_zero_including_new_session_start_updates( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + record_agent_event( + config.db_path, + config.host_id, + kind="agent_message", + source="acp", + worker_id="worker-1", + payload={"text_delta": "stored before coordinator checkpoint"}, + source_session_id="session-resume", + source_event_id="agent-before-crash", + visibility="private", + ) + assert _prepare_console_event_cursor( + config.db_path, + config.host_id, + "worker-1", + "session-resume", + session_mode=SessionOpenMode.RESUME, + generation=42, + ) == 0 + assert _load_console_event_cursor( + config.db_path, config.host_id, "worker-1", "session-resume" + ) == 0 + + record_agent_event( + config.db_path, + config.host_id, + kind="agent_message", + source="acp", + worker_id="worker-1", + payload={"text_delta": "setup replay"}, + source_session_id="session-new", + source_event_id="agent-during-setup", + visibility="private", + ) + baseline = _prepare_console_event_cursor( + config.db_path, + config.host_id, + "worker-1", + "session-new", + session_mode=SessionOpenMode.NEW, + generation=43, + ) + assert baseline == 0 + assert _load_console_event_cursor( + config.db_path, config.host_id, "worker-1", "session-new" + ) == baseline + + +def test_console_failure_outcome_is_durable_before_input_ack(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + _record_console_submission_outcome( + config.db_path, + config.host_id, + "worker-1", + "session-a", + generation=42, + input_sequence=7, + outcome="error", + ) + # Simulate a crash before the following input-cursor write. The error is + # replayable even though Herdr input 7 was not acknowledged yet. + assert _load_console_input_cursor( + config.db_path, config.host_id, "worker-1", "session-a", 42 + ) == 0 + stored = list_agent_events( + config.db_path, + config.host_id, + worker_id="worker-1", + source="acp", + session_id="session-a", + ) + assert len(stored) == 1 + assert _console_event_output(stored[0].event.kind, stored[0].event.payload) == ( + "error", + "instruction failed", + ) + + +def test_console_output_batch_is_utf8_bounded_and_replay_deterministic() -> None: + first = _bounded_console_output("event-a", "tool", "😀" * 100_000) + second = _bounded_console_output("event-b", "assistant", "β" * 100_000) + assert first["text"].encode("utf-8").decode("utf-8") == first["text"] + assert "[console output truncated]" in first["text"] + budget = _console_output_wire_bytes([first]) + assert _fit_console_output_batch([first, second], budget=budget) == [first] + assert _fit_console_output_batch([first, second], budget=budget) == [first] + assert _console_output_wire_bytes([first]) <= budget + + +def test_console_bridge_byte_batches_and_advances_only_published_prefix( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + for index in range(10): + record_agent_event( + config.db_path, + config.host_id, + kind="agent_message", + source="acp", + worker_id="worker-1", + payload={"text_delta": "😀" * 40_000}, + source_session_id="session-a", + source_event_id=f"large-agent-chunk-{index}", + visibility="private", + ) + + class EndpointClient: + retained: list[dict[str, Any]] = [] + next_output = 1 + + def connect(self) -> None: + return None + + def close(self) -> None: + return None + + def agent_acp_console_exchange( + self, _target: str, **params: Any + ) -> dict[str, Any]: + for item in params["output"]: + EndpointClient.retained.append( + {"sequence": EndpointClient.next_output, **dict(item)} + ) + EndpointClient.next_output += 1 + floor = ( + EndpointClient.retained[0]["sequence"] + if EndpointClient.retained + else EndpointClient.next_output + ) + return { + "type": "agent_acp_console_exchange", + "inputs": [], + "outputs": list(EndpointClient.retained), + "input_floor_sequence": 1, + "output_floor_sequence": floor, + "next_input_sequence": 1, + "next_output_sequence": EndpointClient.next_output, + } + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + reconcile_interval=60.0, + ) + runtime = SimpleNamespace( + _binding=replace( + _binding(), + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + ) + ) + executor = ThreadPoolExecutor(max_workers=1) + slot = _RuntimeSlot( + continuity=_binding(), + generation="42", + runtime=runtime, + console=HerdrAcpConsoleEndpoint(42, "coordinator-lease"), + console_cursor_loaded=True, + console_executor=executor, + console_submissions={}, + console_local_turns=set(), + ) + try: + coordinator._bridge_console_slot(slot) + first_cursor = slot.console_event_sequence + assert 0 < first_cursor + assert len(EndpointClient.retained) < 10 + assert _console_output_wire_bytes(EndpointClient.retained) <= 512 * 1024 + # A full retained queue cannot make Tendwire acknowledge the remainder. + coordinator._bridge_console_slot(slot) + assert slot.console_event_sequence == first_cursor + # Once the pane drains, the exact unacknowledged suffix is published. + EndpointClient.retained.clear() + coordinator._bridge_console_slot(slot) + assert slot.console_event_sequence > first_cursor + finally: + executor.shutdown(wait=True, cancel_futures=True) + + def test_console_bridge_polls_independently_of_slow_reconcile_interval( tmp_path: Path, ) -> None: @@ -575,6 +777,9 @@ def test_canonical_herdr_acp_contract_fixture_executes_configured_binary( assert parsed.command[0] == "/opt/herdr/bin/herdr" assert parsed.command[1:] == tuple(fixture["result"]["endpoint"]["args"]) assert parsed.generation == "42" + assert parsed.console == HerdrAcpConsoleEndpoint( + 42, "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQ" + ) class _Route: From 4bf6dc149d08f95dec188180117ca7a0735d7fec Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 20:21:13 +0800 Subject: [PATCH 45/83] fix: fence ACP prompts on visible console loss --- src/tendwire/backends/acp_coordinator.py | 101 +++++++ src/tendwire/command_submission.py | 24 +- src/tendwire/daemon.py | 10 +- tests/test_acp_coordinator.py | 324 +++++++++++++++++++++++ 4 files changed, 455 insertions(+), 4 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index b8fb3a1..8f53075 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -54,6 +54,10 @@ class AcpConsoleInputGap(AcpCoordinatorError): """The bounded Herdr console queue lost unconsumed pane input.""" +class AcpVisibleConsoleUnavailable(AcpCoordinatorError): + """A worker cannot accept new prompts while its pane bridge is lost.""" + + # Herdr's aggregate console queue admission is 768 KiB and its normal control # frame is 2 MiB. Keep Tendwire's retained output contribution below 512 KiB # so queued pane input and the echoed response retain explicit headroom. @@ -196,6 +200,7 @@ def __init__( self._console_degraded = False self._console_failure_type: str | None = None self._console_failed_workers: set[str] = set() + self._console_failed_claims: dict[str, str] = {} def start(self) -> "AcpRuntimeCoordinator": with self._lock: @@ -392,11 +397,17 @@ def status(self) -> dict[str, Any]: def prompt_route(self, worker: Worker) -> _PromptRoute | None: try: slot = self._current_slot(worker) + except AcpVisibleConsoleUnavailable: + # Console loss is a deliberate hard fence, not a stale runtime + # that reconciliation should remint synchronously on this command. + return None except AcpCoordinatorError: # A just-observed worker may not have reached the periodic pass. try: self._reconcile_worker(worker.id, strict=False) slot = self._current_slot(worker) + except AcpVisibleConsoleUnavailable: + return None except Exception: # noqa: BLE001 return None return _PromptRoute(self, worker, slot) @@ -411,6 +422,23 @@ def owns_worker(self, worker_id: str, worker_fingerprint: str) -> bool: and slot.runtime.status().healthy ) + def claims_worker(self, worker_id: str, worker_fingerprint: str) -> bool: + """Return whether ACP has published authority for this exact worker. + + Unlike ``owns_worker``, this remains true across a console/runtime + outage so preferred mode cannot fall through to legacy pane I/O. + """ + + with self._lock: + slot = self._slots.get(worker_id) + return bool( + ( + slot is not None + and slot.continuity.worker_fingerprint == worker_fingerprint + ) + or self._console_failed_claims.get(worker_id) == worker_fingerprint + ) + def owns_permission_decision(self, decision: Any) -> bool: """Return whether one pending decision belongs to an exact live slot.""" worker_id = str(getattr(decision, "worker_id", "") or "") @@ -445,8 +473,18 @@ def _current_slot(self, worker: Worker) -> _RuntimeSlot: if self._state is not RuntimeState.RUNNING: raise AcpCoordinatorError("ACP coordinator is not running") slot = self._slots.get(worker.id) + console_failed = worker.id in self._console_failed_workers if slot is None: raise AcpCoordinatorError("ACP worker route is unavailable") + if console_failed: + # The visible pane is part of the required transport, not an + # optional observer. Fence every newly resolved route (and every + # use of a route resolved before the failure) on the first bridge + # loss. The in-flight ACP turn itself may continue to drain, but + # pane and Telegram callers cannot start another headless turn. + raise AcpVisibleConsoleUnavailable( + "ACP worker visible console is unavailable" + ) if slot.continuity.worker_fingerprint != worker.fingerprint: raise AcpCoordinatorError("ACP worker authority is stale") if not slot.runtime.status().healthy: @@ -503,7 +541,13 @@ def _bridge_console_slot_supervised(self, slot: _RuntimeSlot) -> None: return slot.console_failures = 0 with self._lock: + # A superseded bridge must not clear the fence for its + # replacement. Recovery requires a successful pass by the + # exact slot currently published for this worker. + if self._slots.get(worker_id) is not slot: + return self._console_failed_workers.discard(worker_id) + self._console_failed_claims.pop(worker_id, None) self._console_degraded = bool(self._console_failed_workers) if not self._console_degraded: self._console_failure_type = None @@ -517,6 +561,9 @@ def _bridge_console_slot_supervised(self, slot: _RuntimeSlot) -> None: # runtime. A later pass replays inputs and idempotent outputs. with self._lock: self._console_failed_workers.add(worker_id) + self._console_failed_claims[worker_id] = ( + slot.continuity.worker_fingerprint + ) self._console_degraded = True self._console_failure_type = type(exc).__name__ if failure_count >= 3: @@ -903,6 +950,7 @@ def _submit_console_input_fenced( self.config, json.dumps(request, sort_keys=True, separators=(",", ":")), acp_prompt_router=self.prompt_route, + acp_worker_owner=self.claims_worker, acp_required=True, acp_observation_only=False, acp_permission_router=self, @@ -962,6 +1010,19 @@ def _continuity_bindings(self) -> tuple[dict[str, WorkerBinding], int]: ambiguities = sum(1 for rows in grouped.values() if len(rows) != 1) return current, ambiguities + def _herdr_authority_claims(self) -> set[tuple[str, str]]: + """Return exact sendable identities without requiring unique routing.""" + + return { + (binding.worker_id, binding.worker_fingerprint) + for binding in list_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="herdr", + ) + if binding.sendable + } + def _reconcile(self, *, strict: bool) -> None: with self._reconcile_lock: try: @@ -973,8 +1034,25 @@ def _reconcile(self, *, strict: bool) -> None: def _reconcile_locked(self, *, strict: bool) -> None: current, ambiguities = self._continuity_bindings() + with self._lock: + failed_claims = tuple(self._console_failed_claims.items()) + exact_authorities = ( + self._herdr_authority_claims() if failed_claims else set() + ) with self._lock: stale = [worker_id for worker_id in self._slots if worker_id not in current] + disappeared_failures = [ + worker_id + for worker_id, fingerprint in failed_claims + if (worker_id, fingerprint) not in exact_authorities + and self._console_failed_claims.get(worker_id) == fingerprint + ] + for worker_id in disappeared_failures: + self._console_failed_workers.discard(worker_id) + self._console_failed_claims.pop(worker_id, None) + self._console_degraded = bool(self._console_failed_workers) + if not self._console_degraded: + self._console_failure_type = None for worker_id in stale: self._retire_worker(worker_id) failures: list[BaseException] = [ @@ -1006,6 +1084,28 @@ def _reconcile_worker(self, worker_id: str, *, strict: bool) -> None: current, _ambiguities = self._continuity_bindings() continuity = current.get(worker_id) if continuity is None: + with self._lock: + failed_fingerprint = self._console_failed_claims.get(worker_id) + authority_remains = True + if failed_fingerprint is not None: + try: + authority_remains = ( + worker_id, + failed_fingerprint, + ) in self._herdr_authority_claims() + except Exception: + # A failed ownership check cannot safely reopen PTY + # fallback; the periodic reconcile can retry it. + authority_remains = True + with self._lock: + if worker_id not in self._slots and not authority_remains: + self._console_failed_workers.discard(worker_id) + self._console_failed_claims.pop(worker_id, None) + self._console_degraded = bool( + self._console_failed_workers + ) + if not self._console_degraded: + self._console_failure_type = None if strict: raise AcpCoordinatorError( "worker has no unique Herdr authority" @@ -1278,6 +1378,7 @@ def _retire_worker( self._retired_slots.append(slot) if not preserve_console_failure: self._console_failed_workers.discard(worker_id) + self._console_failed_claims.pop(worker_id, None) self._console_degraded = bool(self._console_failed_workers) if not self._console_degraded: self._console_failure_type = None diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index eadcfec..3411ac4 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -126,6 +126,7 @@ def prompt( AcpPromptRouter = Callable[[Worker], AcpPromptRoute | None] +AcpWorkerOwner = Callable[[str, str], bool] class AcpPermissionDecisionRouter(Protocol): @@ -2686,6 +2687,7 @@ def submit_acp_command( params: Mapping[str, Any] | str, *, prompt_router: AcpPromptRouter, + worker_owner: AcpWorkerOwner | None = None, required: bool = False, observation_only: bool = False, ) -> CommandEnvelope | None: @@ -2764,6 +2766,20 @@ def submit_acp_command( if takeover is not None and worker.id != takeover.public_worker_id: return _duplicate_request(request) + # Preferred mode may fall back only for workers that ACP has never + # claimed. Once the coordinator publishes an exact worker generation, + # losing its visible console or runtime is an ACP outage, not permission + # to inject keys through the legacy PTY path. + owned_by_acp = False + if worker_owner is not None: + try: + owned_by_acp = bool(worker_owner(worker.id, worker.fingerprint)) + except Exception: # noqa: BLE001 + # An ownership oracle failure cannot prove that legacy pane I/O is + # safe. Prefer a retryable no-send result over crossing transports. + owned_by_acp = True + route_required = required or owned_by_acp + route: AcpPromptRoute | None = None route_resolved = False if observation_only: @@ -2782,7 +2798,7 @@ def submit_acp_command( permanent_error = _worker_status_error(request, worker) or health_error if permanent_error is not None: - if required: + if route_required: canonical = build_canonical_mutation(request, public_worker_id=worker.id) reservation = _reserve_canonical_request(config, request, canonical) if isinstance(reservation, CommandEnvelope): @@ -2800,7 +2816,7 @@ def submit_acp_command( return _request_in_progress(request) return ( _backend_unavailable(request, "ACP worker route is unavailable") - if required + if route_required else None ) try: @@ -2814,7 +2830,7 @@ def submit_acp_command( return _request_in_progress(request) return ( _backend_unavailable(request, "ACP worker route has no durable authority") - if required + if route_required else None ) @@ -3195,6 +3211,7 @@ def submit_command( *, socket_client_factory: SocketClientFactory | None = None, acp_prompt_router: AcpPromptRouter | None = None, + acp_worker_owner: AcpWorkerOwner | None = None, acp_required: bool = False, acp_observation_only: bool = False, acp_permission_router: AcpPermissionDecisionRouter | None = None, @@ -3205,6 +3222,7 @@ def submit_command( config, params, prompt_router=acp_prompt_router, + worker_owner=acp_worker_owner, required=acp_required, observation_only=acp_observation_only, ) diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index 8a401f3..c3697ae 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -617,7 +617,9 @@ def start(self) -> None: scheduler = self.hooks.turn_scheduler_factory(self.config) self._turn_scheduler = scheduler if self.config.agent_event_source in {"acp_shadow", "acp_preferred"}: - owns_worker = getattr(self._acp_runtime, "owns_worker", None) + owns_worker = getattr(self._acp_runtime, "claims_worker", None) + if not callable(owns_worker): + owns_worker = getattr(self._acp_runtime, "owns_worker", None) set_exclusion = getattr(scheduler, "set_worker_exclusion", None) if callable(owns_worker) and callable(set_exclusion): set_exclusion(owns_worker) @@ -1449,6 +1451,9 @@ def submit_command(self, params: Mapping[str, Any]) -> CommandEnvelope | Mapping policy = self.config.agent_event_source runtime = self._acp_runtime route = getattr(runtime, "prompt_route", None) + worker_owner = getattr(runtime, "claims_worker", None) + if not callable(worker_owner): + worker_owner = getattr(runtime, "owns_worker", None) permission_router = ( runtime if callable(getattr(runtime, "answer_permission_decision", None)) @@ -1468,6 +1473,9 @@ def submit_command(self, params: Mapping[str, Any]) -> CommandEnvelope | Mapping and callable(route) else None ), + acp_worker_owner=( + worker_owner if callable(worker_owner) else None + ), acp_required=policy == "acp_required", acp_observation_only=policy == "acp_shadow", acp_permission_router=permission_router, diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 5b252b4..2443ac7 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -578,6 +578,224 @@ def fail(_slot: _RuntimeSlot) -> None: assert status["failure_type"] == "OSError" +def test_first_console_failure_immediately_fences_prompt_route_until_success( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path, policy="acp_required"), + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + runtime = SimpleNamespace( + status=lambda: SimpleNamespace(healthy=True, failure_type=None), + _binding=_binding(), + ) + slot = _RuntimeSlot( + _binding(), + "42", + runtime, + console=HerdrAcpConsoleEndpoint(42, "console-lease"), + ) + coordinator._slots["worker-1"] = slot + worker = Worker( + id="worker-1", + name="worker", + status="idle", + fingerprint="worker-fingerprint", + ) + issued_before_loss = coordinator.prompt_route(worker) + assert issued_before_loss is not None + reconcile_calls = 0 + + def unexpected_reconcile(_worker_id: str, *, strict: bool) -> None: + nonlocal reconcile_calls + reconcile_calls += 1 + + coordinator._reconcile_worker = unexpected_reconcile # type: ignore[method-assign] + + failure_entered = threading.Event() + release_failure = threading.Event() + + def fail(_slot: _RuntimeSlot) -> None: + failure_entered.set() + assert release_failure.wait(2.0) + raise OSError("console unavailable") + + coordinator._bridge_console_slot = fail # type: ignore[method-assign] + failed_pass = threading.Thread( + target=coordinator._bridge_console_slot_supervised, + args=(slot,), + ) + failed_pass.start() + assert failure_entered.wait(1.0) + release_failure.set() + failed_pass.join(timeout=1.0) + assert not failed_pass.is_alive() + + assert coordinator.prompt_route(worker) is None + assert reconcile_calls == 0 + with pytest.raises(AcpCoordinatorError, match="visible console"): + _ = issued_before_loss.binding_fingerprint + assert coordinator.status()["healthy"] is False + + success_entered = threading.Event() + release_success = threading.Event() + + def succeed(_slot: _RuntimeSlot) -> None: + success_entered.set() + assert release_success.wait(2.0) + + coordinator._bridge_console_slot = succeed # type: ignore[method-assign] + successful_pass = threading.Thread( + target=coordinator._bridge_console_slot_supervised, + args=(slot,), + ) + successful_pass.start() + assert success_entered.wait(1.0) + # Starting a recovery bridge is not recovery; it must complete one + # visible-console exchange before either command ingress can route again. + assert coordinator.prompt_route(worker) is None + release_success.set() + successful_pass.join(timeout=1.0) + assert not successful_pass.is_alive() + + assert coordinator.prompt_route(worker) is not None + assert reconcile_calls == 0 + assert coordinator.status()["healthy"] is True + + +def test_superseded_console_success_cannot_clear_replacement_fence( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path, policy="acp_required"), + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + runtime = SimpleNamespace( + status=lambda: SimpleNamespace(healthy=True, failure_type=None), + _binding=_binding(), + ) + old = _RuntimeSlot( + _binding(), + "42", + runtime, + console=HerdrAcpConsoleEndpoint(42, "old-lease"), + ) + replacement = _RuntimeSlot( + _binding(), + "43", + runtime, + console=HerdrAcpConsoleEndpoint(43, "replacement-lease"), + ) + coordinator._slots["worker-1"] = replacement + coordinator._console_failed_workers.add("worker-1") + coordinator._console_degraded = True + coordinator._bridge_console_slot = lambda _slot: None # type: ignore[method-assign] + + coordinator._bridge_console_slot_supervised(old) + assert "worker-1" in coordinator._console_failed_workers + assert coordinator.status()["healthy"] is False + + coordinator._bridge_console_slot_supervised(replacement) + assert "worker-1" not in coordinator._console_failed_workers + assert coordinator.status()["healthy"] is True + + +def test_failed_remint_retains_exact_acp_claim_and_blocks_preferred_fallback( + tmp_path: Path, +) -> None: + config = _config(tmp_path, policy="acp_preferred") + worker = _seed(config) + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + runtime = SimpleNamespace( + status=lambda: SimpleNamespace(healthy=True, failure_type=None), + stop=lambda *, timeout: None, + _binding=_binding(), + ) + slot = _RuntimeSlot( + _binding(), + "42", + runtime, + console=HerdrAcpConsoleEndpoint(42, "console-lease"), + ) + coordinator._slots[worker.id] = slot + + def fail_console(_slot: _RuntimeSlot) -> None: + raise OSError("console unavailable") + + coordinator._bridge_console_slot = fail_console # type: ignore[method-assign] + coordinator._continuity_bindings = ( # type: ignore[method-assign] + lambda: ({worker.id: _binding()}, 0) + ) + + def fail_remint(_continuity: WorkerBinding) -> None: + raise AcpCoordinatorError("replacement attach failed") + + coordinator._reconcile_binding = fail_remint # type: ignore[method-assign] + + for _attempt in range(3): + coordinator._bridge_console_slot_supervised(slot) + + assert worker.id not in coordinator._slots + assert coordinator.claims_worker(worker.id, worker.fingerprint) is True + assert coordinator.prompt_route(worker) is None + + def forbidden_legacy(_config: Config) -> Any: + raise AssertionError("retired ACP claim must not reopen legacy pane I/O") + + envelope = submit_command( + config, + _request("failed-remint-no-fallback"), + socket_client_factory=forbidden_legacy, + acp_prompt_router=coordinator.prompt_route, + acp_worker_owner=coordinator.claims_worker, + ) + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert get_command_request( + config.db_path, + config.host_id, + "failed-remint-no-fallback", + ) is None + + +def test_failed_claim_clears_only_after_exact_herdr_authority_disappears( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path, policy="acp_preferred"), + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + coordinator._console_failed_workers.add("worker-1") + coordinator._console_failed_claims["worker-1"] = "worker-fingerprint" + coordinator._console_degraded = True + coordinator._continuity_bindings = ( # type: ignore[method-assign] + lambda: ({}, 1) + ) + coordinator._herdr_authority_claims = ( # type: ignore[method-assign] + lambda: {("worker-1", "worker-fingerprint")} + ) + + coordinator._reconcile_locked(strict=False) + assert coordinator.claims_worker("worker-1", "worker-fingerprint") is True + assert coordinator.status()["healthy"] is False + + coordinator._herdr_authority_claims = lambda: set() # type: ignore[method-assign] + coordinator._reconcile_locked(strict=False) + assert coordinator.claims_worker("worker-1", "worker-fingerprint") is False + assert coordinator._console_degraded is False + + def test_console_submission_rejects_a_retired_generation_before_store_access( tmp_path: Path, ) -> None: @@ -884,6 +1102,77 @@ def test_acp_failure_after_send_started_is_uncertain_and_never_falls_back(tmp_pa assert receipt is not None and receipt["state"] == "uncertain" +def test_preferred_acp_owned_route_loss_fails_closed_without_receipt_or_legacy( + tmp_path: Path, +) -> None: + config = _config(tmp_path, policy="acp_preferred") + worker = _seed(config) + + def forbidden_legacy(_config: Config) -> Any: + raise AssertionError("ACP-owned console loss must not reach legacy pane I/O") + + for _attempt in range(2): + envelope = submit_command( + config, + _request("preferred-owned-console-loss"), + socket_client_factory=forbidden_legacy, + acp_prompt_router=lambda _worker: None, + acp_worker_owner=lambda worker_id, fingerprint: ( + worker_id == worker.id and fingerprint == worker.fingerprint + ), + ) + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert get_command_request( + config.db_path, + config.host_id, + "preferred-owned-console-loss", + ) is None + + +def test_preferred_non_acp_worker_still_uses_legacy_sender(tmp_path: Path) -> None: + config = _config(tmp_path, policy="acp_preferred") + _seed(config) + legacy_calls: list[str] = [] + + class LegacyClient: + def connect(self) -> "LegacyClient": + return self + + def request( + self, + method: str, + params: dict[str, Any], + *, + timeout: float | None = None, + ) -> dict[str, Any]: + del timeout + legacy_calls.append(method) + if method == "agent.get": + return {"result": {"agent": {"pane_id": "pane-private"}}} + if method == "agent.prompt": + return { + "type": "agent_prompted", + "agent": {"pane_id": "pane-private"}, + "delivery": "submitted", + } + return {"accepted": True, "params": params} + + def close(self) -> None: + return None + + envelope = submit_command( + config, + _request("preferred-non-acp-worker"), + socket_client_factory=lambda _config: LegacyClient(), + acp_prompt_router=lambda _worker: None, + acp_worker_owner=lambda _worker_id, _fingerprint: False, + ) + + assert envelope.status == "accepted" + assert legacy_calls[-1] == "agent.prompt" + + def test_shadow_owned_command_is_observation_only_before_receipt(tmp_path: Path) -> None: config = _config(tmp_path, policy="acp_shadow") worker = replace(_seed(config), status="working") @@ -951,6 +1240,41 @@ def legacy_sender(_config: Config, _payload: str) -> Any: assert legacy_calls == [] +def test_daemon_preferred_console_loss_uses_claim_to_block_legacy_sender( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = _config(tmp_path, policy="acp_preferred") + worker = _seed(config) + + class Runtime: + def prompt_route(self, _worker: Worker) -> None: + return None + + def claims_worker(self, worker_id: str, fingerprint: str) -> bool: + return worker_id == worker.id and fingerprint == worker.fingerprint + + def forbidden_legacy(_config: Config) -> Any: + raise AssertionError("ACP-owned console loss must not use legacy pane I/O") + + monkeypatch.setattr( + "tendwire.command_submission._default_socket_client_factory", + forbidden_legacy, + ) + daemon = TendwireDaemon(config) + daemon._acp_runtime = Runtime() + + envelope = daemon.submit_command(_request("daemon-preferred-console-loss")) + + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert get_command_request( + config.db_path, + config.host_id, + "daemon-preferred-console-loss", + ) is None + + def test_daemon_shadow_preserves_ordinary_legacy_worker_submission( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, From cec39364e2e469fe5efbf2ffedc72ff214c341bf Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 20:27:05 +0800 Subject: [PATCH 46/83] fix: preserve ACP ownership across fallback hazards --- src/tendwire/backends/acp_coordinator.py | 11 +++- src/tendwire/command_submission.py | 6 +- tests/test_acp_coordinator.py | 73 ++++++++++++++++++++++++ 3 files changed, 87 insertions(+), 3 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 8f53075..4182cb3 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -1376,9 +1376,16 @@ def _retire_worker( return self._slots.pop(worker_id, None) self._retired_slots.append(slot) - if not preserve_console_failure: + # A visible-console failure is an exact sticky ownership + # claim. Retirement, ambiguity, and failed reminting must not + # reopen legacy PTY fallback while Herdr still publishes that + # identity. Only a successful current console pass or the + # positive-disappearance path in reconciliation may remove it. + if ( + not preserve_console_failure + and worker_id not in self._console_failed_claims + ): self._console_failed_workers.discard(worker_id) - self._console_failed_claims.pop(worker_id, None) self._console_degraded = bool(self._console_failed_workers) if not self._console_degraded: self._console_failure_type = None diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 3411ac4..cca55ff 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -2751,7 +2751,11 @@ def submit_acp_command( request, "Current worker authority is temporarily unavailable", ) - if required + # A configured ownership oracle means preferred mode can route + # both ACP and never-ACP workers. Without the authoritative + # snapshot there is no exact worker identity to ask it about, so + # falling through would treat "unknown" as proof of never-ACP. + if required or worker_owner is not None else None ) diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 2443ac7..80d4890 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -796,6 +796,46 @@ def test_failed_claim_clears_only_after_exact_herdr_authority_disappears( assert coordinator._console_degraded is False +def test_first_console_failure_survives_unique_route_ambiguity_and_retirement( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path, policy="acp_preferred"), + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + runtime = SimpleNamespace( + status=lambda: SimpleNamespace(healthy=True, failure_type=None), + stop=lambda *, timeout: None, + _binding=_binding(), + ) + slot = _RuntimeSlot( + _binding(), + "42", + runtime, + console=HerdrAcpConsoleEndpoint(42, "console-lease"), + ) + coordinator._slots["worker-1"] = slot + coordinator._bridge_console_slot = ( # type: ignore[method-assign] + lambda _slot: (_ for _ in ()).throw(OSError("console unavailable")) + ) + coordinator._bridge_console_slot_supervised(slot) + assert coordinator.claims_worker("worker-1", "worker-fingerprint") is True + + # Two sendable routes make the worker non-unique. The old slot is stale, + # but exact Herdr authority remains and therefore so must the ACP claim. + coordinator._continuity_bindings = lambda: ({}, 1) # type: ignore[method-assign] + coordinator._herdr_authority_claims = ( # type: ignore[method-assign] + lambda: {("worker-1", "worker-fingerprint")} + ) + coordinator._reconcile_locked(strict=False) + + assert "worker-1" not in coordinator._slots + assert coordinator.claims_worker("worker-1", "worker-fingerprint") is True + assert coordinator.status()["healthy"] is False + + def test_console_submission_rejects_a_retired_generation_before_store_access( tmp_path: Path, ) -> None: @@ -1173,6 +1213,39 @@ def close(self) -> None: assert legacy_calls[-1] == "agent.prompt" +def test_preferred_snapshot_failure_with_owner_oracle_never_falls_back( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = _config(tmp_path, policy="acp_preferred") + _seed(config) + + def unavailable_snapshot(_config: Config) -> Snapshot: + raise OSError("authority store temporarily unavailable") + + def forbidden_legacy(_config: Config) -> Any: + raise AssertionError("unknown ACP ownership must not reach legacy pane I/O") + + monkeypatch.setattr( + "tendwire.command_submission._current_snapshot", unavailable_snapshot + ) + envelope = submit_command( + config, + _request("preferred-authority-read-failure"), + socket_client_factory=forbidden_legacy, + acp_prompt_router=lambda _worker: None, + acp_worker_owner=lambda _worker_id, _fingerprint: False, + ) + + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert get_command_request( + config.db_path, + config.host_id, + "preferred-authority-read-failure", + ) is None + + def test_shadow_owned_command_is_observation_only_before_receipt(tmp_path: Path) -> None: config = _config(tmp_path, policy="acp_shadow") worker = replace(_seed(config), status="working") From 6c21629cd855f02223e747046d1c792bd1e8856a Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sat, 1 Aug 2026 20:32:09 +0800 Subject: [PATCH 47/83] fix: retain published ACP ownership through remint failure --- src/tendwire/backends/acp_coordinator.py | 39 +++++++++++++-- tests/test_acp_coordinator.py | 60 ++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 4 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 4182cb3..65e565d 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -201,6 +201,10 @@ def __init__( self._console_failure_type: str | None = None self._console_failed_workers: set[str] = set() self._console_failed_claims: dict[str, str] = {} + # Exact ACP ownership survives runtime retirement. Preferred mode may + # use legacy PTY I/O only after Herdr positively stops publishing this + # exact worker identity, never merely because reminting failed. + self._published_acp_claims: dict[str, str] = {} def start(self) -> "AcpRuntimeCoordinator": with self._lock: @@ -437,6 +441,7 @@ def claims_worker(self, worker_id: str, worker_fingerprint: str) -> bool: and slot.continuity.worker_fingerprint == worker_fingerprint ) or self._console_failed_claims.get(worker_id) == worker_fingerprint + or self._published_acp_claims.get(worker_id) == worker_fingerprint ) def owns_permission_decision(self, decision: Any) -> bool: @@ -564,6 +569,9 @@ def _bridge_console_slot_supervised(self, slot: _RuntimeSlot) -> None: self._console_failed_claims[worker_id] = ( slot.continuity.worker_fingerprint ) + self._published_acp_claims[worker_id] = ( + slot.continuity.worker_fingerprint + ) self._console_degraded = True self._console_failure_type = type(exc).__name__ if failure_count >= 3: @@ -1036,8 +1044,11 @@ def _reconcile_locked(self, *, strict: bool) -> None: current, ambiguities = self._continuity_bindings() with self._lock: failed_claims = tuple(self._console_failed_claims.items()) + published_claims = tuple(self._published_acp_claims.items()) exact_authorities = ( - self._herdr_authority_claims() if failed_claims else set() + self._herdr_authority_claims() + if failed_claims or published_claims + else set() ) with self._lock: stale = [worker_id for worker_id in self._slots if worker_id not in current] @@ -1045,11 +1056,23 @@ def _reconcile_locked(self, *, strict: bool) -> None: worker_id for worker_id, fingerprint in failed_claims if (worker_id, fingerprint) not in exact_authorities + and worker_id not in self._slots and self._console_failed_claims.get(worker_id) == fingerprint ] for worker_id in disappeared_failures: self._console_failed_workers.discard(worker_id) - self._console_failed_claims.pop(worker_id, None) + failed_fingerprint = self._console_failed_claims.pop(worker_id, None) + if self._published_acp_claims.get(worker_id) == failed_fingerprint: + self._published_acp_claims.pop(worker_id, None) + disappeared_published = [ + worker_id + for worker_id, fingerprint in published_claims + if (worker_id, fingerprint) not in exact_authorities + and worker_id not in self._slots + and self._published_acp_claims.get(worker_id) == fingerprint + ] + for worker_id in disappeared_published: + self._published_acp_claims.pop(worker_id, None) self._console_degraded = bool(self._console_failed_workers) if not self._console_degraded: self._console_failure_type = None @@ -1086,12 +1109,16 @@ def _reconcile_worker(self, worker_id: str, *, strict: bool) -> None: if continuity is None: with self._lock: failed_fingerprint = self._console_failed_claims.get(worker_id) + published_fingerprint = self._published_acp_claims.get( + worker_id + ) + claimed_fingerprint = failed_fingerprint or published_fingerprint authority_remains = True - if failed_fingerprint is not None: + if claimed_fingerprint is not None: try: authority_remains = ( worker_id, - failed_fingerprint, + claimed_fingerprint, ) in self._herdr_authority_claims() except Exception: # A failed ownership check cannot safely reopen PTY @@ -1101,6 +1128,7 @@ def _reconcile_worker(self, worker_id: str, *, strict: bool) -> None: if worker_id not in self._slots and not authority_remains: self._console_failed_workers.discard(worker_id) self._console_failed_claims.pop(worker_id, None) + self._published_acp_claims.pop(worker_id, None) self._console_degraded = bool( self._console_failed_workers ) @@ -1195,6 +1223,9 @@ def _reconcile_binding(self, continuity: WorkerBinding) -> None: with self._lock: displaced = self._slots.get(continuity.worker_id) self._slots[continuity.worker_id] = slot + self._published_acp_claims[continuity.worker_id] = ( + continuity.worker_fingerprint + ) if displaced is not None: self._stop_runtime(displaced.runtime) diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 80d4890..82ea8e9 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -836,6 +836,63 @@ def test_first_console_failure_survives_unique_route_ambiguity_and_retirement( assert coordinator.status()["healthy"] is False +def test_published_claim_survives_unhealthy_runtime_retire_and_failed_remint( + tmp_path: Path, +) -> None: + config = _config(tmp_path, policy="acp_preferred") + worker = _seed(config) + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + runtime = SimpleNamespace( + status=lambda: SimpleNamespace(healthy=False, failure_type="runtime_failed"), + stop=lambda *, timeout: None, + _binding=_binding(), + ) + slot = _RuntimeSlot( + _binding(), + "42", + runtime, + console=HerdrAcpConsoleEndpoint(42, "console-lease"), + ) + coordinator._slots[worker.id] = slot + coordinator._published_acp_claims[worker.id] = worker.fingerprint + coordinator._continuity_bindings = ( # type: ignore[method-assign] + lambda: ({worker.id: _binding()}, 0) + ) + coordinator._resolve_endpoint = ( # type: ignore[method-assign] + lambda _continuity: (_ for _ in ()).throw( + AcpCoordinatorError("replacement attach failed") + ) + ) + + coordinator._reconcile_locked(strict=False) + + assert worker.id not in coordinator._slots + assert coordinator.claims_worker(worker.id, worker.fingerprint) is True + + def forbidden_legacy(_config: Config) -> Any: + raise AssertionError("published ACP ownership must survive failed remint") + + envelope = submit_command( + config, + _request("unhealthy-runtime-failed-remint"), + socket_client_factory=forbidden_legacy, + acp_prompt_router=coordinator.prompt_route, + acp_worker_owner=coordinator.claims_worker, + ) + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert get_command_request( + config.db_path, + config.host_id, + "unhealthy-runtime-failed-remint", + ) is None + + def test_console_submission_rejects_a_retired_generation_before_store_access( tmp_path: Path, ) -> None: @@ -1644,6 +1701,9 @@ def submit_prompt(self, *_args: Any, **_kwargs: Any) -> None: ).start() try: assert minted == ["one-shot-private-ticket-1"] + assert coordinator._published_acp_claims == { + "worker-1": "worker-fingerprint" + } coordinator._reconcile_worker("worker-1", strict=True) worker = Worker( id="worker-1", From 19fe4e302dfd90089ee2bcd4591747be99072649 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 06:05:58 +0800 Subject: [PATCH 48/83] fix(acp): fail closed on Herdr readiness timeout --- src/tendwire/backends/herdr_events.py | 3 +- tests/test_daemon.py | 48 +++++++++++++++++++++++++++ tests/test_herdr_events.py | 37 +++++++++++++++++++++ 3 files changed, 87 insertions(+), 1 deletion(-) diff --git a/src/tendwire/backends/herdr_events.py b/src/tendwire/backends/herdr_events.py index b207a18..da4d60b 100644 --- a/src/tendwire/backends/herdr_events.py +++ b/src/tendwire/backends/herdr_events.py @@ -1008,7 +1008,8 @@ def start(self, *, wait_for_reconcile: bool = True, timeout_seconds: float | Non thread.start() if wait_for_reconcile: timeout = self.config.herdr_timeout_seconds if timeout_seconds is None else timeout_seconds - self._ready.wait(max(0.001, float(timeout))) + if not self._ready.wait(max(0.001, float(timeout))): + raise HerdrSocketTimeoutError("initial Herdr reconciliation timed out") def stop(self) -> None: self.stop_event.set() diff --git a/tests/test_daemon.py b/tests/test_daemon.py index 7fae773..eff01b7 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -22,6 +22,7 @@ import pytest from tendwire import __version__ +from tendwire.backends.herdr_socket import HerdrSocketTimeoutError from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult from tendwire.cli import main from tendwire.config import Config @@ -3964,6 +3965,53 @@ def event_backend_factory(_config: Config, _stop_event: threading.Event) -> Any: daemon.stop() +@_UNIX_SOCKET_TEST +def test_daemon_backend_timeout_never_reaches_acp_startup(tmp_path: Path) -> None: + calls: list[str] = [] + + class TimedOutEventBackend: + def start(self, *, wait_for_reconcile: bool) -> None: + assert wait_for_reconcile is True + calls.append("backend_start") + raise HerdrSocketTimeoutError("initial Herdr reconciliation timed out") + + def stop(self) -> None: + calls.append("backend_stop") + + def forbidden_acp_factory(_config: Config, _stop_event: threading.Event) -> Any: + calls.append("acp_factory") + raise AssertionError("ACP startup must not follow a Herdr readiness timeout") + + config = Config( + host_id="daemon-host", + data_dir=tmp_path, + db_path=tmp_path / "backend-timeout.db", + socket_path=tmp_path / "backend-timeout.sock", + herdr_backend="socket", + agent_event_source="acp_preferred", + ) + daemon = TendwireDaemon( + config, + hooks=DaemonHooks( + event_backend_factory=lambda _config, _stop_event: TimedOutEventBackend(), + acp_runtime_factory=forbidden_acp_factory, + ), + ) + + try: + with pytest.raises( + HerdrSocketTimeoutError, + match="initial Herdr reconciliation timed out", + ): + daemon.start() + + assert calls == ["backend_start", "backend_stop"] + assert daemon.server is None + assert not os.path.lexists(config.socket_path) + finally: + daemon.stop() + + @_UNIX_SOCKET_TEST def test_daemon_scheduler_start_failure_detaches_callback_and_cleans_components( tmp_path: Path, diff --git a/tests/test_herdr_events.py b/tests/test_herdr_events.py index c8e0a48..7340c52 100644 --- a/tests/test_herdr_events.py +++ b/tests/test_herdr_events.py @@ -2774,6 +2774,43 @@ def read_event(self, subscription_id: str, *, timeout: float | None = None) -> d assert time.monotonic() - started < 2.0 +def test_start_raises_instead_of_proceeding_with_stale_state_when_not_ready( + tmp_path: Path, + monkeypatch: Any, +) -> None: + config = _config(tmp_path, "initial-reconcile-timeout") + init_store(Path(config.db_path)) + save_snapshot( + Path(config.db_path), + project_from_observations( + config, + workers=[Worker(id="stale-worker", name="Stale Worker", status="waiting")], + ), + ) + backend = HerdrEventBackend( + config, + debounce_seconds=0, + reconnect_delay_seconds=0, + ) + assert latest_snapshot(backend.db_path, backend.config.host_id) is not None + + def wait_until_stopped() -> None: + backend.stop_event.wait() + + monkeypatch.setattr(backend, "run_forever", wait_until_stopped) + + try: + with pytest.raises( + HerdrSocketTimeoutError, + match="initial Herdr reconciliation timed out", + ): + backend.start(wait_for_reconcile=True, timeout_seconds=0.01) + + assert backend.ready is False + finally: + backend.stop() + + @pytest.mark.parametrize("batched", [False, True], ids=["one-flush-per-event", "one-batch"]) def test_real_idless_working_idle_working_preserves_every_transition( tmp_path: Path, From e7fc86f1b9d44b87028e88eed9b2a68ff4adbf9d Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 09:23:08 +0800 Subject: [PATCH 49/83] fix(herdr): separate initial reconcile timeout --- .env.example | 5 +- README.md | 5 ++ src/tendwire/backends/herdr_events.py | 18 +++- src/tendwire/config.py | 28 ++++++- tests/test_config.py | 41 +++++++++ tests/test_daemon.py | 40 +++++++++ tests/test_herdr_events.py | 116 ++++++++++++++++++++++++++ 7 files changed, 247 insertions(+), 6 deletions(-) diff --git a/.env.example b/.env.example index 0f7766e..6ef3179 100644 --- a/.env.example +++ b/.env.example @@ -81,9 +81,12 @@ TENDWIRE_STORE_MAINTENANCE_CADENCE_SECONDS=3600 # TENDWIRE_HOST_ID=my-host # Optional Herdr binary and timeout overrides. Turn-adapter reads use the same -# timeout. +# per-RPC timeout. Initial socket reconciliation has a separate whole-startup +# budget so a multi-call inventory can exceed one RPC without weakening normal +# operation deadlines. # TENDWIRE_HERDR_BIN=herdr TENDWIRE_HERDR_TIMEOUT_SECONDS=1.0 +TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS=120.0 # Daemon-owned turn-ingestion cadence and dedicated worker pool. Defaults are # 2.0 seconds and 4 workers; the worker count cannot exceed diff --git a/README.md b/README.md index 0230053..b0885e5 100644 --- a/README.md +++ b/README.md @@ -180,6 +180,11 @@ not expose raw backend argv. The Herdr binary path, data directory, and database path expand `~`; each Herdr probe uses `TENDWIRE_HERDR_TIMEOUT_SECONDS` or `--herdr-timeout` when set, defaulting to 5.0 seconds. +Socket-daemon startup waits up to +`TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS` (default `120.0`) for the +entire initial reconciliation. This startup-only budget does not change the +normal per-RPC `TENDWIRE_HERDR_TIMEOUT_SECONDS` deadline. + When Herdr 0.7.0 is present, the adapter first tries the no-flag JSON envelopes (`herdr workspace list`, `herdr agent list`) that wrap records under `result.workspaces` and `result.agents`, then keeps `--json` list variants as a diff --git a/src/tendwire/backends/herdr_events.py b/src/tendwire/backends/herdr_events.py index da4d60b..e16fddb 100644 --- a/src/tendwire/backends/herdr_events.py +++ b/src/tendwire/backends/herdr_events.py @@ -1007,8 +1007,13 @@ def start(self, *, wait_for_reconcile: bool = True, timeout_seconds: float | Non self._thread = thread thread.start() if wait_for_reconcile: - timeout = self.config.herdr_timeout_seconds if timeout_seconds is None else timeout_seconds + timeout = ( + self.config.herdr_initial_reconcile_timeout_seconds + if timeout_seconds is None + else timeout_seconds + ) if not self._ready.wait(max(0.001, float(timeout))): + self.stop() raise HerdrSocketTimeoutError("initial Herdr reconciliation timed out") def stop(self) -> None: @@ -1016,8 +1021,9 @@ def stop(self) -> None: self.flush() thread = self._thread if thread is not None and thread is not threading.current_thread(): - thread.join(timeout=max(1.0, self.config.herdr_timeout_seconds)) - self._thread = None + thread.join(timeout=max(1.0, self.config.herdr_timeout_seconds + 1.0)) + if thread is None or not thread.is_alive(): + self._thread = None def run_forever(self) -> None: while not self.stop_event.is_set(): @@ -1640,6 +1646,8 @@ def _consume_pane_replay( def _replay_turns_after_reconcile(self, client: Any) -> None: """Probe pane.turns once, then replay each pane independently.""" + if self.stop_event.is_set(): + return pane_ids = tuple(self._subscription_pane_ids) if not pane_ids: self._turn_api_probed = True @@ -1657,6 +1665,8 @@ def _replay_turns_after_reconcile(self, client: Any) -> None: probe_replay: HerdrPaneTurnsReplay | None = None probe_error: HerdrErrorResponse | AttributeError | None = None if not self._turn_api_probed: + if self.stop_event.is_set(): + return probe_pane_id = pane_ids[0] supported, probe_replay, probe_error = self._probe_turn_api( client, @@ -1668,6 +1678,8 @@ def _replay_turns_after_reconcile(self, client: Any) -> None: if not self._turn_api_supported: return for pane_id in pane_ids: + if self.stop_event.is_set(): + return self._consume_pane_replay( client, pane_id, diff --git a/src/tendwire/config.py b/src/tendwire/config.py index 8e1f8a5..02dbdce 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -26,6 +26,7 @@ DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS = 30.0 DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS = 5.0 DEFAULT_ACP_MAX_FRAME_BYTES = 8 * 1024 * 1024 +DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS = 120.0 DEFAULT_EVENT_DEBOUNCE_SECONDS = 0.05 DEFAULT_RECONCILE_INTERVAL_SECONDS = 300.0 DEFAULT_EVENT_RETENTION_DAYS = 7 @@ -71,6 +72,9 @@ class Config: db_path: Path | None = None socket_path: Path | None = None herdr_timeout_seconds: float = 5.0 + herdr_initial_reconcile_timeout_seconds: float = ( + DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS + ) herdr_backend: str = "cli" turn_model: str = DEFAULT_TURN_MODEL agent_event_source: str = DEFAULT_AGENT_EVENT_SOURCE @@ -123,8 +127,22 @@ def __post_init__(self) -> None: normalized_socket_group = str(self.socket_group).strip() object.__setattr__(self, "socket_group", normalized_socket_group or None) - if self.herdr_timeout_seconds <= 0: - raise ValueError("herdr_timeout_seconds must be positive") + object.__setattr__( + self, + "herdr_timeout_seconds", + _positive_finite_float( + self.herdr_timeout_seconds, + "herdr_timeout_seconds", + ), + ) + object.__setattr__( + self, + "herdr_initial_reconcile_timeout_seconds", + _positive_finite_float( + self.herdr_initial_reconcile_timeout_seconds, + "herdr_initial_reconcile_timeout_seconds", + ), + ) backend = str(self.herdr_backend or "").strip().lower() if backend not in HERDR_BACKENDS: allowed = ", ".join(sorted(HERDR_BACKENDS)) @@ -490,6 +508,7 @@ def load_config( socket_path: str | Path | None = None, socket_group: str | None = None, herdr_timeout_seconds: float | str | None = None, + herdr_initial_reconcile_timeout_seconds: float | str | None = None, herdr_backend: str | None = None, turn_model: str | None = None, agent_event_source: str | None = None, @@ -586,6 +605,11 @@ def load_config( db_path=resolved_db_path, socket_path=resolved_socket_path, herdr_timeout_seconds=resolved_herdr_timeout_seconds, + herdr_initial_reconcile_timeout_seconds=_resolve_value( + herdr_initial_reconcile_timeout_seconds, + "TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS", + DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS, + ), herdr_backend=resolved_herdr_backend, turn_model=_resolve_value( turn_model, diff --git a/tests/test_config.py b/tests/test_config.py index cc30c7a..528c81d 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -17,6 +17,7 @@ DEFAULT_COMMAND_RECEIPT_RETENTION_COUNT, DEFAULT_COMMAND_RECEIPT_RETENTION_SECONDS, DEFAULT_COMMAND_RETRY_HORIZON_SECONDS, + DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS, DEFAULT_SUBMISSION_HARD_TTL_SECONDS, DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS, DEFAULT_TURN_MODEL, @@ -32,6 +33,46 @@ ) +def test_initial_reconcile_timeout_is_distinct_and_configurable(monkeypatch) -> None: + monkeypatch.delenv( + "TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS", + raising=False, + ) + defaults = load_config(herdr_timeout_seconds="2.5") + assert defaults.herdr_timeout_seconds == 2.5 + assert ( + defaults.herdr_initial_reconcile_timeout_seconds + == DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS + == 120.0 + ) + + monkeypatch.setenv("TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS", "45") + environment = load_config(herdr_timeout_seconds="1.5") + explicit = load_config( + herdr_timeout_seconds="0.75", + herdr_initial_reconcile_timeout_seconds="90", + ) + + assert environment.herdr_timeout_seconds == 1.5 + assert environment.herdr_initial_reconcile_timeout_seconds == 45.0 + assert explicit.herdr_timeout_seconds == 0.75 + assert explicit.herdr_initial_reconcile_timeout_seconds == 90.0 + + +@pytest.mark.parametrize("value", ["", "invalid", "0", "-1", "inf", "nan"]) +def test_initial_reconcile_timeout_rejects_invalid_environment( + monkeypatch, + value: str, +) -> None: + monkeypatch.setenv("TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS", value) + + with pytest.raises( + ValueError, + match="herdr_initial_reconcile_timeout_seconds must be a finite positive number", + ): + load_config() + + def test_acp_event_source_defaults_to_legacy_with_thoughts_disabled( monkeypatch, ) -> None: diff --git a/tests/test_daemon.py b/tests/test_daemon.py index eff01b7..0fcfa61 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -4012,6 +4012,46 @@ def forbidden_acp_factory(_config: Config, _stop_event: threading.Event) -> Any: daemon.stop() +@_UNIX_SOCKET_TEST +def test_daemon_default_backend_keeps_startup_and_rpc_timeouts_distinct( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured: list[tuple[float, float]] = [] + + def time_out_start(self: Any, *, wait_for_reconcile: bool) -> None: + assert wait_for_reconcile is True + captured.append( + ( + self.config.herdr_timeout_seconds, + self.config.herdr_initial_reconcile_timeout_seconds, + ) + ) + raise HerdrSocketTimeoutError("initial Herdr reconciliation timed out") + + monkeypatch.setattr( + "tendwire.backends.herdr_events.HerdrEventBackend.start", + time_out_start, + ) + config = Config( + host_id="daemon-distinct-timeouts", + data_dir=tmp_path, + db_path=tmp_path / "daemon-distinct-timeouts.db", + socket_path=tmp_path / "daemon-distinct-timeouts.sock", + herdr_backend="socket", + herdr_timeout_seconds=0.25, + herdr_initial_reconcile_timeout_seconds=17, + ) + daemon = TendwireDaemon(config) + + try: + with pytest.raises(HerdrSocketTimeoutError): + daemon.start() + assert captured == [(0.25, 17.0)] + finally: + daemon.stop() + + @_UNIX_SOCKET_TEST def test_daemon_scheduler_start_failure_detaches_callback_and_cleans_components( tmp_path: Path, diff --git a/tests/test_herdr_events.py b/tests/test_herdr_events.py index 7340c52..0bd68e6 100644 --- a/tests/test_herdr_events.py +++ b/tests/test_herdr_events.py @@ -2807,10 +2807,126 @@ def wait_until_stopped() -> None: backend.start(wait_for_reconcile=True, timeout_seconds=0.01) assert backend.ready is False + assert backend.running is False + assert backend._thread is None finally: backend.stop() +def test_start_defaults_to_dedicated_initial_reconcile_timeout( + tmp_path: Path, + monkeypatch: Any, +) -> None: + config = Config( + host_id="initial-reconcile-budget", + data_dir=tmp_path, + db_path=tmp_path / "initial-reconcile-budget.db", + herdr_backend="socket", + herdr_timeout_seconds=0.25, + herdr_initial_reconcile_timeout_seconds=42, + ) + backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) + waited: list[float] = [] + + class NeverReady: + def clear(self) -> None: + return None + + def is_set(self) -> bool: + return False + + def wait(self, timeout: float | None = None) -> bool: + assert timeout is not None + waited.append(timeout) + return False + + backend._ready = NeverReady() # type: ignore[assignment] + monkeypatch.setattr(backend, "run_forever", backend.stop_event.wait) + + try: + with pytest.raises(HerdrSocketTimeoutError): + backend.start(wait_for_reconcile=True) + assert waited == [42.0] + assert config.herdr_timeout_seconds == 0.25 + finally: + backend.stop() + + +def test_start_timeout_cancels_remaining_per_pane_replay_and_joins_worker( + tmp_path: Path, + monkeypatch: Any, +) -> None: + config = Config( + host_id="initial-reconcile-cancel", + data_dir=tmp_path, + db_path=tmp_path / "initial-reconcile-cancel.db", + herdr_backend="socket", + herdr_timeout_seconds=0.1, + herdr_initial_reconcile_timeout_seconds=0.01, + ) + init_store(Path(config.db_path)) + + class Client: + def close(self) -> None: + return None + + backend = HerdrEventBackend( + config, + client_factory=lambda _config: Client(), + debounce_seconds=0, + reconnect_delay_seconds=0, + ) + consumed: list[str] = [] + replay_started = threading.Event() + + class ReadinessGate: + def clear(self) -> None: + return None + + def is_set(self) -> bool: + return False + + def set(self) -> None: + return None + + def wait(self, timeout: float | None = None) -> bool: + assert replay_started.wait(1.0) + return False + + backend._ready = ReadinessGate() # type: ignore[assignment] + + def reconcile_once(*, client: Any) -> None: + backend._subscription_pane_ids = ["pane-1", "pane-2"] + + def probe_turn_api( + client: Any, + pane_id: str, + watermark: Any, + ) -> tuple[bool, None, None]: + return True, None, None + + def consume_pane_replay( + client: Any, + pane_id: str, + watermark: Any, + **kwargs: Any, + ) -> None: + consumed.append(pane_id) + replay_started.set() + time.sleep(0.05) + + monkeypatch.setattr(backend, "reconcile_once", reconcile_once) + monkeypatch.setattr(backend, "_probe_turn_api", probe_turn_api) + monkeypatch.setattr(backend, "_consume_pane_replay", consume_pane_replay) + + with pytest.raises(HerdrSocketTimeoutError): + backend.start(wait_for_reconcile=True) + + assert consumed == ["pane-1"] + assert backend.running is False + assert backend._thread is None + + @pytest.mark.parametrize("batched", [False, True], ids=["one-flush-per-event", "one-batch"]) def test_real_idless_working_idle_working_preserves_every_transition( tmp_path: Path, From 51e0fe7debad86a7a74e1d2ff7f3eea5ea5270b5 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 10:23:38 +0800 Subject: [PATCH 50/83] fix(herdr): isolate turn replay sockets --- src/tendwire/backends/herdr_events.py | 79 ++++++--- tests/test_herdr_events.py | 225 ++++++++++++++++++++++++++ 2 files changed, 286 insertions(+), 18 deletions(-) diff --git a/src/tendwire/backends/herdr_events.py b/src/tendwire/backends/herdr_events.py index e16fddb..80d3523 100644 --- a/src/tendwire/backends/herdr_events.py +++ b/src/tendwire/backends/herdr_events.py @@ -1035,7 +1035,16 @@ def run_forever(self) -> None: self._turn_api_supported = False self.reconcile_once(client=client) reconciled = True - self._replay_turns_after_reconcile(client) + # Herdr ordinary RPC connections are one-shot. Keep this + # future subscription client out of pane.turns entirely; + # each replay request uses its own short-lived client. + if callable(getattr(client, "pane_turns", None)) or callable( + getattr(client, "request", None) + ): + self._replay_turns_after_reconcile() + else: + self._turn_api_probed = True + self._turn_api_supported = False if self.stop_event.is_set(): break if hasattr(client, "connect"): @@ -1048,7 +1057,7 @@ def run_forever(self) -> None: # Events concurrent with it are buffered by the socket # client and become harmless watermark-deduped duplicates. if self._turn_api_supported: - self._replay_turns_after_reconcile(client) + self._replay_turns_after_reconcile() self._ready.set() self._read_event_stream(client, stream.subscription_id) finally: @@ -1318,6 +1327,30 @@ def _call_pane_turns( ) return _pane_turns_replay(value, pane_id) + def _call_pane_turns_isolated( + self, + pane_id: str, + *, + since: int, + expected_epoch: int | None, + allow_uncorrelated: bool = False, + ) -> HerdrPaneTurnsReplay: + """Call one pane.turns RPC on a fresh, always-closed client.""" + client = self.client_factory(self.config) + try: + if hasattr(client, "connect"): + client.connect() + return self._call_pane_turns( + client, + pane_id, + since=since, + expected_epoch=expected_epoch, + allow_uncorrelated=allow_uncorrelated, + ) + finally: + if hasattr(client, "close"): + client.close() + def _record_turn_diagnostic( self, code: str, @@ -1534,7 +1567,7 @@ def _turn_api_method_unsupported(cls, exc: HerdrErrorResponse) -> bool: def _probe_turn_api( self, - client: Any, + client: Any | None, pane_id: str, watermark: HerdrTurnWatermark | None, ) -> tuple[ @@ -1543,13 +1576,19 @@ def _probe_turn_api( HerdrErrorResponse | AttributeError | None, ]: """Probe pane.turns once without treating pane-scoped errors as absence.""" - if not callable(getattr(client, "pane_turns", None)) and not callable( - getattr(client, "request", None) - ): + if client is not None and not callable( + getattr(client, "pane_turns", None) + ) and not callable(getattr(client, "request", None)): return False, None, None try: - replay = self._call_pane_turns( - client, + call = self._call_pane_turns_isolated if client is None else ( + lambda current_pane_id, **kwargs: self._call_pane_turns( + client, + current_pane_id, + **kwargs, + ) + ) + replay = call( pane_id, since=watermark.last_turn if watermark is not None else 0, expected_epoch=watermark.turn_epoch if watermark is not None else None, @@ -1559,24 +1598,30 @@ def _probe_turn_api( if self._turn_api_method_unsupported(exc): return False, None, None return True, None, exc - except AttributeError as exc: - return True, None, exc + except AttributeError: + return False, None, None return True, replay, None def _consume_pane_replay( self, - client: Any, + client: Any | None, pane_id: str, watermark: HerdrTurnWatermark | None, *, replay: HerdrPaneTurnsReplay | None = None, error: HerdrErrorResponse | AttributeError | None = None, ) -> None: + call = self._call_pane_turns_isolated if client is None else ( + lambda current_pane_id, **kwargs: self._call_pane_turns( + client, + current_pane_id, + **kwargs, + ) + ) try: if error is not None: raise error - current_replay = replay or self._call_pane_turns( - client, + current_replay = replay or call( pane_id, since=watermark.last_turn if watermark is not None else 0, expected_epoch=watermark.turn_epoch if watermark is not None else None, @@ -1586,8 +1631,7 @@ def _consume_pane_replay( message = self._herdr_error_message(exc) if code == "turn_epoch_mismatch": try: - current_replay = self._call_pane_turns( - client, + current_replay = call( pane_id, since=0, expected_epoch=None, @@ -1610,8 +1654,7 @@ def _consume_pane_replay( return if code == "invalid_params" and "newer than current turn" in message: try: - current_replay = self._call_pane_turns( - client, + current_replay = call( pane_id, since=0, expected_epoch=None, @@ -1644,7 +1687,7 @@ def _consume_pane_replay( return self._consume_replay(current_replay, watermark) - def _replay_turns_after_reconcile(self, client: Any) -> None: + def _replay_turns_after_reconcile(self, client: Any | None = None) -> None: """Probe pane.turns once, then replay each pane independently.""" if self.stop_event.is_set(): return diff --git a/tests/test_herdr_events.py b/tests/test_herdr_events.py index 0bd68e6..4580774 100644 --- a/tests/test_herdr_events.py +++ b/tests/test_herdr_events.py @@ -2870,6 +2870,9 @@ class Client: def close(self) -> None: return None + def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: + raise AssertionError("the patched replay hooks own this test") + backend = HerdrEventBackend( config, client_factory=lambda _config: Client(), @@ -6387,6 +6390,228 @@ def subscribe( assert (watermark.turn_epoch, watermark.last_turn) == (7, 1) +def test_turn_api_run_loop_keeps_stream_client_exclusive_and_replay_calls_one_shot( + tmp_path: Path, +) -> None: + config = _config(tmp_path, "turn-api-one-shot-connections") + init_store(Path(config.db_path)) + first_pane = _turn_api_pane() + second_pane = { + **_turn_api_pane(), + "pane_id": "w123456789abcde:pB", + "terminal_id": "terminal-turn-api-b", + } + operations: list[str] = [] + subscribed = threading.Event() + second_post_replay_started = threading.Event() + release_second_post_replay = threading.Event() + per_pane_calls = {first_pane["pane_id"]: 0, second_pane["pane_id"]: 0} + replay_clients: list[Any] = [] + + class SubscriptionClient(_StaticClient): + def __init__(self) -> None: + super().__init__( + workspaces=[{"id": "w123456789abcde", "name": "Build"}], + panes=[first_pane, second_pane], + ) + self.pane_turn_calls = 0 + + def connect(self) -> None: + operations.append("stream.connect") + + def close(self) -> None: + operations.append("stream.close") + + def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: + self.pane_turn_calls += 1 + raise AssertionError("the subscription client must never carry pane.turns") + + def subscribe( + self, + _method: str, + _params: Mapping[str, Any], + **_kwargs: Any, + ) -> Any: + operations.append("subscribe") + subscribed.set() + return SimpleNamespace(subscription_id="one-shot-stream") + + def read_event( + self, + _subscription_id: str, + *, + timeout: float | None = None, + ) -> dict[str, Any]: + assert backend.ready is True + backend.stop_event.set() + raise HerdrSocketTimeoutError("idle") + + class OneShotReplayClient: + def __init__(self) -> None: + self.connected = False + self.closed = False + self.calls = 0 + + def connect(self) -> None: + assert self.connected is False + self.connected = True + + def close(self) -> None: + self.closed = True + + def pane_turns( + self, + params: Mapping[str, Any], + **_kwargs: Any, + ) -> Any: + assert self.connected is True + assert self.closed is False + self.calls += 1 + assert self.calls == 1, "ordinary Herdr RPC connections are one-shot" + pane_id = str(params["pane_id"]) + per_pane_calls[pane_id] += 1 + phase = "pre" if per_pane_calls[pane_id] == 1 else "post" + operations.append(f"{phase}:{pane_id}") + if phase == "post" and pane_id == second_pane["pane_id"]: + second_post_replay_started.set() + assert release_second_post_replay.wait(1) + records = [_turn_record(1)] if phase == "post" else [] + return { + "turns": { + "pane_id": pane_id, + "turn_epoch": 7, + "records": records, + "truncated": False, + "oldest_available": 1 if records else None, + } + } + + stream_client = SubscriptionClient() + + def client_factory(_config: Config) -> Any: + if not replay_clients and not operations: + return stream_client + client = OneShotReplayClient() + replay_clients.append(client) + return client + + backend = HerdrEventBackend( + config, + client_factory=client_factory, + debounce_seconds=0, + reconnect_delay_seconds=0, + turn_completion_processor=lambda *_args, **_kwargs: SimpleNamespace( + status="unchanged" + ), + ) + thread = threading.Thread(target=backend.run_forever, daemon=True) + thread.start() + + assert subscribed.wait(1) + assert second_post_replay_started.wait(1) + assert backend.ready is False + assert stream_client.pane_turn_calls == 0 + + release_second_post_replay.set() + thread.join(timeout=2) + + assert thread.is_alive() is False + assert backend.ready is True + assert len(replay_clients) == 4 + assert all(client.calls == 1 for client in replay_clients) + assert all(client.connected is True and client.closed is True for client in replay_clients) + replay_order = [ + operation + for operation in operations + if operation == "subscribe" or operation.startswith(("pre:", "post:")) + ] + assert replay_order == [ + f"pre:{first_pane['pane_id']}", + f"pre:{second_pane['pane_id']}", + "subscribe", + f"post:{first_pane['pane_id']}", + f"post:{second_pane['pane_id']}", + ] + + +def test_turn_api_production_retry_uses_another_short_lived_client( + tmp_path: Path, +) -> None: + backend = _turn_api_backend( + tmp_path, + "turn-api-isolated-retry", + lambda *_args, **_kwargs: SimpleNamespace(status="unchanged"), + ) + pane_id = _turn_api_pane()["pane_id"] + set_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + turn_epoch=7, + last_turn=2, + ) + backend._turn_api_probed = True + backend._turn_api_supported = True + clients: list[Any] = [] + + class RetryClient: + def __init__(self, ordinal: int) -> None: + self.ordinal = ordinal + self.connected = False + self.closed = False + self.calls = 0 + + def connect(self) -> None: + self.connected = True + + def close(self) -> None: + self.closed = True + + def pane_turns( + self, + params: Mapping[str, Any], + **_kwargs: Any, + ) -> Any: + self.calls += 1 + assert self.calls == 1 + if self.ordinal == 1: + assert params["expected_epoch"] == 7 + raise HerdrErrorResponse( + {"code": "turn_epoch_mismatch", "message": "epoch changed"}, + "isolated-retry", + ) + assert params == {"pane_id": pane_id, "since": 0} + return { + "turns": { + "pane_id": pane_id, + "turn_epoch": 8, + "records": [_turn_record(1, epoch=8)], + "truncated": False, + "oldest_available": 1, + } + } + + def client_factory(_config: Config) -> RetryClient: + client = RetryClient(len(clients) + 1) + clients.append(client) + return client + + backend.client_factory = client_factory + backend._replay_turns_after_reconcile() + + assert len(clients) == 2 + assert all(client.calls == 1 for client in clients) + assert all(client.connected is True and client.closed is True for client in clients) + watermark = get_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + ) + assert watermark is not None + assert (watermark.turn_epoch, watermark.last_turn) == (8, 1) + assert watermark.last_completeness_break_reason == "turn_epoch_mismatch" + + def test_turn_api_restart_replays_exactly_the_missed_turn(tmp_path: Path) -> None: pane_id = _turn_api_pane()["pane_id"] first = _turn_api_backend( From f6cd44746ebca44c170efe8349583269de7739c8 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 13:43:15 +0800 Subject: [PATCH 51/83] fix(herdr): survive health persistence contention --- src/tendwire/backends/herdr_events.py | 8 ++++++ tests/test_herdr_events.py | 40 ++++++++++++++++++++++++--- 2 files changed, 44 insertions(+), 4 deletions(-) diff --git a/src/tendwire/backends/herdr_events.py b/src/tendwire/backends/herdr_events.py index 80d3523..1decb80 100644 --- a/src/tendwire/backends/herdr_events.py +++ b/src/tendwire/backends/herdr_events.py @@ -3129,5 +3129,13 @@ def _mark_unhealthy(self, outcome: str) -> Snapshot: def _mark_unhealthy_safe(self, outcome: str) -> Snapshot | None: try: return self._mark_unhealthy(outcome) + except Exception: + # Health persistence is secondary to keeping the long-running + # observation loop alive. In particular, another store operation + # can briefly hold the secure SQLite parent lock and make this + # best-effort write fail closed. The next loop iteration performs + # a complete reconciliation, so retain the in-memory unhealthy + # state and let that authoritative retry recover the backend. + return None finally: self._ready.set() diff --git a/tests/test_herdr_events.py b/tests/test_herdr_events.py index 4580774..6cc4777 100644 --- a/tests/test_herdr_events.py +++ b/tests/test_herdr_events.py @@ -47,6 +47,7 @@ from tendwire.core.projector import project_from_observations from tendwire.core.turns import PendingObservation from tendwire.daemon import DaemonHooks, TendwireDaemon +from tendwire.local_state import LocalStateErrorCode, local_state_error from tendwire.store.sqlite import ( SnapshotObservationContext, SnapshotRetentionPolicy, @@ -4474,12 +4475,43 @@ def boom(*_args: Any, **_kwargs: Any) -> None: monkeypatch.setattr("tendwire.backends.herdr_events.save_snapshot", boom) - try: - backend._mark_unhealthy_safe("protocol_error") - except RuntimeError: - pass + assert backend._mark_unhealthy_safe("protocol_error") is None + assert backend.ready is True + + +def test_run_forever_retries_when_unhealthy_persistence_fails( + tmp_path: Path, + monkeypatch: Any, +) -> None: + backend = _backend(tmp_path, "retry-after-health-persist-error") + reconcile_calls = 0 + clients_created = 0 + + def client_factory(_config: Config) -> object: + nonlocal clients_created + clients_created += 1 + return object() + + def reconcile_once(*, client: object) -> None: + nonlocal reconcile_calls + reconcile_calls += 1 + if reconcile_calls == 1: + raise RuntimeError("observation failed") + backend.stop_event.set() + + def persist_failure(*_args: Any, **_kwargs: Any) -> None: + raise local_state_error(LocalStateErrorCode.OPERATION_FAILED) + + backend.client_factory = client_factory + monkeypatch.setattr(backend, "reconcile_once", reconcile_once) + monkeypatch.setattr("tendwire.backends.herdr_events.save_snapshot", persist_failure) + + backend.run_forever() + assert reconcile_calls == 2 + assert clients_created == 2 assert backend.ready is True + assert backend.health.outcome == "unknown" def test_protocol_error_health_is_degraded_and_specific(tmp_path: Path) -> None: From 6d0d9a05b37c3998fda98f468044c084266ca1e1 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 17:05:18 +0800 Subject: [PATCH 52/83] fix(acp): close failed turns and replay completion races --- src/tendwire/backends/acp_runtime.py | 42 +- src/tendwire/backends/herdr_events.py | 154 ++++- src/tendwire/store/sqlite.py | 309 ++++++++++- tests/test_acp_permissions.py | 2 +- tests/test_acp_runtime.py | 117 +++- tests/test_agent_events.py | 2 +- tests/test_backend_pending.py | 2 +- tests/test_connector_outbox.py | 2 +- tests/test_delivery_retention_migration.py | 2 +- tests/test_delivery_retention_projection.py | 2 +- tests/test_delivery_retention_recovery.py | 2 +- tests/test_herdr_events.py | 586 +++++++++++++++++++- tests/test_store.py | 4 +- 13 files changed, 1192 insertions(+), 34 deletions(-) diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 84fb7a6..7b05478 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -27,6 +27,7 @@ RequestId, SessionResult, SessionUpdate, + StopReason, ) @@ -433,10 +434,7 @@ def prompt( # prompt cannot be retried safely: late updates would otherwise # be attributed to the next turn. Best-effort cancellation # contains the remote work and the runtime becomes terminal. - try: - self._cancel_session(session_id) - except BaseException: - pass + self._close_failed_prompt(session_id, ingestor) self._record_failure(exc) raise if not isinstance(result, PromptResult): @@ -445,6 +443,7 @@ def prompt( ) with self._state_lock: self._prompts_failed += 1 + self._close_failed_prompt(session_id, ingestor) self._record_failure(error) raise error @@ -1002,6 +1001,41 @@ def _cancel_session(self, session_id: str) -> None: with self._state_lock: self._cancellation_requests += 1 + def _close_failed_prompt( + self, + session_id: str, + ingestor: AcpSessionIngestor, + ) -> None: + """Best-effort cancel and durable closure after ``begin_prompt``. + + The original prompt exception remains authoritative even if either + cleanup operation fails. ``mark_prompt_complete`` is idempotent, so + this method cannot create a second completion for an already closed + turn. + """ + + try: + self._cancel_session(session_id) + except BaseException: + pass + try: + # The prompt response (including an invalid one) can overtake the + # consumer thread after earlier session/update frames were queued. + # Preserve those updates in the failed turn before writing its + # terminal marker. A late post-cancel update remains harmless: + # the ingestor rejects turn-scoped updates after completion. + self._wait_for_event_idle(self._stop_timeout) + except BaseException: + pass + try: + with self._ingest_lock: + completion = ingestor.mark_prompt_complete( + StopReason.CANCELLED + ) + _raise_for_binding_rejection(completion) + except BaseException: + pass + def _record_failure(self, failure: BaseException) -> None: self._release_derived_binding(reason="acp_runtime_failed") with self._idle_condition: diff --git a/src/tendwire/backends/herdr_events.py b/src/tendwire/backends/herdr_events.py index 1decb80..326e508 100644 --- a/src/tendwire/backends/herdr_events.py +++ b/src/tendwire/backends/herdr_events.py @@ -40,12 +40,15 @@ SnapshotRetentionPolicy, expire_stale_worker_bindings, expire_worker_bindings, + get_herdr_turn_refresh_retry, get_herdr_turn_watermark, + herdr_turn_refresh_retry_due, latest_snapshot, list_worker_bindings, maybe_run_automatic_store_maintenance, record_herdr_turn_completeness_break, record_herdr_turn_completion, + record_herdr_turn_refresh_retry, save_snapshot, set_herdr_turn_watermark, upsert_worker_bindings, @@ -153,6 +156,20 @@ } ) _COMPLETED_TURN_REFRESH_STATUSES = frozenset({"updated", "unchanged", "missing"}) +_RETRYABLE_COMPLETED_TURN_REFRESH_STATUSES = frozenset( + { + "binding_ambiguous", + "binding_missing", + "failed", + "stale_binding", + "store_unavailable", + "timeout", + } +) +_COMPLETED_TURN_REFRESH_MAX_RETRY_AGE_SECONDS = 5 * 60 +_COMPLETED_TURN_REFRESH_MAX_ATTEMPTS = 8 +_COMPLETED_TURN_REFRESH_RETRY_BASE_DELAY_SECONDS = 1 +_COMPLETED_TURN_REFRESH_RETRY_MAX_DELAY_SECONDS = 30 class HerdrEventBackendError(Exception): @@ -806,6 +823,7 @@ def __init__( self._last_cap_status_at: str | None = None self._automatic_maintenance_status: dict[str, Any] | None = None self._next_reconcile_monotonic: float | None = None + self._next_turn_replay_monotonic: float | None = None self._subscription_pane_ids: list[str] = [] self._turn_api_probed = False self._turn_api_supported = False @@ -1128,15 +1146,37 @@ def _schedule_next_reconcile(self) -> None: self._next_reconcile_monotonic = time.monotonic() + self.reconcile_interval_seconds def _run_periodic_reconcile_if_due(self, client: Any | None = None) -> None: - if self.reconcile_interval_seconds <= 0: - return + current = time.monotonic() due_at = self._next_reconcile_monotonic - if due_at is None: + reconcile_due = ( + self.reconcile_interval_seconds > 0 + and due_at is not None + and current >= due_at + ) + turn_replay_due = ( + self._next_turn_replay_monotonic is not None + and current >= self._next_turn_replay_monotonic + ) + if self.reconcile_interval_seconds > 0 and due_at is None: self._schedule_next_reconcile() + if not reconcile_due and not turn_replay_due: return - if time.monotonic() < due_at: - return - self.reconcile_once(client=client) + if reconcile_due: + self.reconcile_once(client=client) + if self._turn_api_supported: + # Completion refreshes intentionally leave their watermark behind + # while a worker binding is still settling. Periodic snapshot + # reconciliation must therefore also replay the durable Herdr turn + # ledger; otherwise a retryable completion would not be revisited + # until the subscription happened to disconnect. + self._next_turn_replay_monotonic = None + self._replay_turns_after_reconcile() + + def _schedule_turn_replay(self, delay_seconds: float) -> None: + candidate = time.monotonic() + max(0.0, float(delay_seconds)) + due_at = self._next_turn_replay_monotonic + if due_at is None or candidate < due_at: + self._next_turn_replay_monotonic = candidate def _pending_event_count(self) -> int: with self._lock: @@ -1474,6 +1514,21 @@ def _process_turn_record( if record.turn <= watermark.last_turn: return if record.turn != watermark.last_turn + 1: + # A later live notification is not evidence of ledger loss while + # the exact next completion is durably waiting for refresh. Leave + # ordering intact; pane.turns replay will revisit the blocker. + pending_predecessor = get_herdr_turn_refresh_retry( + self.db_path, + self.config.host_id, + record.pane_id, + turn_epoch=record.turn_epoch, + turn=watermark.last_turn + 1, + ) + if ( + pending_predecessor is not None + and pending_predecessor.status == "pending" + ): + return self._record_completeness_break( HerdrPaneTurnsReplay( pane_id=record.pane_id, @@ -1485,9 +1540,53 @@ def _process_turn_record( "live_gap", ) return + existing_retry = get_herdr_turn_refresh_retry( + self.db_path, + self.config.host_id, + record.pane_id, + turn_epoch=record.turn_epoch, + turn=record.turn, + ) + if existing_retry is not None and existing_retry.status == "escalated": + # Escalation and watermark advancement are separate durable writes. + # If the process stopped between them, finalize provenance without + # rerunning the known-poison refresh or losing the terminal marker. + self._record_turn_diagnostic( + "herdr_turn_completion_refresh_escalated_recovered", + record.pane_id, + status=existing_retry.refresh_status, + ) + record_herdr_turn_completion( + self.db_path, + self.config.host_id, + record.pane_id, + turn_epoch=record.turn_epoch, + turn=record.turn, + outcome=record.outcome, + completed_unix_ms=record.completed_unix_ms, + message=record.message, + message_truncated=record.message_truncated, + agent_session_path=record.agent_session_path, + worker_id=None, + refreshed_turn_id=None, + preserve_refresh_retry=True, + ) + return + if not herdr_turn_refresh_retry_due( + self.db_path, + self.config.host_id, + record.pane_id, + turn_epoch=record.turn_epoch, + turn=record.turn, + ): + self._schedule_turn_replay( + _COMPLETED_TURN_REFRESH_RETRY_BASE_DELAY_SECONDS + ) + return status, worker_id, refreshed_turn_id = self._completion_processor_result( record ) + preserve_refresh_retry = False if status not in _COMPLETED_TURN_REFRESH_STATUSES: self._record_turn_diagnostic( "herdr_turn_completion_refresh_skipped", @@ -1495,6 +1594,39 @@ def _process_turn_record( status=status or "unknown", ) refreshed_turn_id = None + if status in _RETRYABLE_COMPLETED_TURN_REFRESH_STATUSES: + retry = record_herdr_turn_refresh_retry( + self.db_path, + self.config.host_id, + record.pane_id, + turn_epoch=record.turn_epoch, + turn=record.turn, + refresh_status=status, + base_delay_seconds=( + _COMPLETED_TURN_REFRESH_RETRY_BASE_DELAY_SECONDS + ), + max_delay_seconds=( + _COMPLETED_TURN_REFRESH_RETRY_MAX_DELAY_SECONDS + ), + max_retry_age_seconds=( + _COMPLETED_TURN_REFRESH_MAX_RETRY_AGE_SECONDS + ), + max_attempts=_COMPLETED_TURN_REFRESH_MAX_ATTEMPTS, + ) + if retry.status == "pending": + retry_delay = min( + _COMPLETED_TURN_REFRESH_RETRY_MAX_DELAY_SECONDS, + _COMPLETED_TURN_REFRESH_RETRY_BASE_DELAY_SECONDS + * (2 ** min(retry.attempt_count - 1, 30)), + ) + self._schedule_turn_replay(retry_delay) + return + preserve_refresh_retry = True + self._record_turn_diagnostic( + "herdr_turn_completion_refresh_escalated", + record.pane_id, + status=status, + ) record_herdr_turn_completion( self.db_path, self.config.host_id, @@ -1508,6 +1640,7 @@ def _process_turn_record( agent_session_path=record.agent_session_path, worker_id=worker_id, refreshed_turn_id=refreshed_turn_id, + preserve_refresh_retry=preserve_refresh_retry, ) def _consume_replay( @@ -1538,6 +1671,15 @@ def _consume_replay( return for record in replay.records: self._process_turn_record(record) + current = get_herdr_turn_watermark( + self.db_path, + self.config.host_id, + replay.pane_id, + ) + if current is None or current.last_turn < record.turn: + # Preserve strict same-pane ordering: a pending refresh blocks + # later records in this replay, but never another pane. + return @classmethod def _turn_api_method_unsupported(cls, exc: HerdrErrorResponse) -> bool: diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index 4cd80e6..b8484b0 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -145,7 +145,7 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS -STORE_SCHEMA_VERSION = 27 +STORE_SCHEMA_VERSION = 28 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS _LEGACY_TURN_CLAIM_HARD_TTL_SECONDS = 86_400 TURN_CHANGE_RETENTION_DAYS = 7 @@ -524,6 +524,23 @@ class HerdrTurnWatermark: updated_at: str +@dataclass(frozen=True) +class HerdrTurnRefreshRetry: + """Durable local retry state for one Herdr completion refresh.""" + + host_id: str + pane_id: str + turn_epoch: int + turn: int + status: Literal["pending", "escalated"] + refresh_status: str + first_seen_at: str + last_attempt_at: str + next_attempt_at: str | None + attempt_count: int + escalated_at: str | None + + @dataclass(frozen=True) class _TurnContentMergeResult: """Observation merge outcome and optional shadow-link settlement key.""" @@ -1534,6 +1551,36 @@ def _record_response_size( ); """ +CREATE_HERDR_TURN_REFRESH_RETRIES_TABLE = """ +CREATE TABLE IF NOT EXISTS herdr_turn_refresh_retries ( + host_id TEXT NOT NULL, + pane_id TEXT NOT NULL, + turn_epoch INTEGER NOT NULL CHECK (turn_epoch >= 0), + turn INTEGER NOT NULL CHECK (turn >= 0), + status TEXT NOT NULL CHECK (status IN ('pending', 'escalated')), + refresh_status TEXT NOT NULL, + first_seen_at TEXT NOT NULL, + last_attempt_at TEXT NOT NULL, + next_attempt_at TEXT, + attempt_count INTEGER NOT NULL CHECK (attempt_count >= 1), + escalated_at TEXT, + PRIMARY KEY (host_id, pane_id, turn_epoch, turn), + CHECK ( + ( + status = 'pending' + AND next_attempt_at IS NOT NULL + AND escalated_at IS NULL + ) + OR + ( + status = 'escalated' + AND next_attempt_at IS NULL + AND escalated_at IS NOT NULL + ) + ) +); +""" + CREATE_HERDR_TURN_INDEXES = ( ( "CREATE INDEX IF NOT EXISTS idx_herdr_turn_completions_worker " @@ -1541,6 +1588,13 @@ def _record_response_size( ), ) +CREATE_HERDR_TURN_REFRESH_RETRY_INDEXES = ( + ( + "CREATE INDEX IF NOT EXISTS idx_herdr_turn_refresh_retries_due " + "ON herdr_turn_refresh_retries(host_id, status, next_attempt_at)" + ), +) + CREATE_AGENT_EVENTS_TABLE = """ CREATE TABLE IF NOT EXISTS agent_events ( sequence INTEGER PRIMARY KEY AUTOINCREMENT, @@ -13572,6 +13626,13 @@ def _migrate_v26_to_v27_conn(conn: sqlite3.Connection) -> None: ) +def _migrate_v27_to_v28_conn(conn: sqlite3.Connection) -> None: + """Persist bounded Herdr completion-refresh retries and escalation.""" + conn.execute(CREATE_HERDR_TURN_REFRESH_RETRIES_TABLE) + for statement in CREATE_HERDR_TURN_REFRESH_RETRY_INDEXES: + conn.execute(statement) + + MIGRATIONS: tuple[Migration, ...] = ( Migration(0, 1, _migrate_v0_to_v1_conn), Migration(1, 2, _migrate_v1_to_v2_conn), @@ -13600,6 +13661,7 @@ def _migrate_v26_to_v27_conn(conn: sqlite3.Connection) -> None: Migration(24, 25, _migrate_v24_to_v25_conn), Migration(25, 26, _migrate_v25_to_v26_conn), Migration(26, 27, _migrate_v26_to_v27_conn), + Migration(27, 28, _migrate_v27_to_v28_conn), ) @@ -13654,6 +13716,7 @@ def _create_current_schema_conn(conn: sqlite3.Connection) -> None: conn.execute(CREATE_TURN_SUPERSESSIONS_TABLE) conn.execute(CREATE_HERDR_TURN_WATERMARKS_TABLE) conn.execute(CREATE_HERDR_TURN_COMPLETIONS_TABLE) + conn.execute(CREATE_HERDR_TURN_REFRESH_RETRIES_TABLE) conn.execute(CREATE_AGENT_EVENTS_TABLE) conn.execute(CREATE_AGENT_EVENT_TOMBSTONES_TABLE) for statement in CREATE_COMMAND_RECEIPT_INDEXES: @@ -13677,6 +13740,8 @@ def _create_current_schema_conn(conn: sqlite3.Connection) -> None: conn.execute(statement) for statement in CREATE_HERDR_TURN_INDEXES: conn.execute(statement) + for statement in CREATE_HERDR_TURN_REFRESH_RETRY_INDEXES: + conn.execute(statement) for statement in CREATE_AGENT_EVENT_INDEXES: conn.execute(statement) for statement in CREATE_AGENT_EVENT_TOMBSTONE_INDEXES: @@ -14377,6 +14442,210 @@ def latest_turn_id_for_worker( return str(row[0]) if row is not None else None +def _herdr_turn_refresh_retry_from_row( + row: tuple[Any, ...], +) -> HerdrTurnRefreshRetry: + return HerdrTurnRefreshRetry( + host_id=str(row[0]), + pane_id=str(row[1]), + turn_epoch=int(row[2]), + turn=int(row[3]), + status=str(row[4]), # type: ignore[arg-type] + refresh_status=str(row[5]), + first_seen_at=str(row[6]), + last_attempt_at=str(row[7]), + next_attempt_at=str(row[8]) if row[8] is not None else None, + attempt_count=int(row[9]), + escalated_at=str(row[10]) if row[10] is not None else None, + ) + + +def get_herdr_turn_refresh_retry( + db_path: Path | str, + host_id: str, + pane_id: str, + *, + turn_epoch: int, + turn: int, +) -> HerdrTurnRefreshRetry | None: + """Return durable completion-refresh retry state, if any.""" + epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") + turn_number = _herdr_turn_counter(turn, "turn") + if not _sqlite_store_exists(db_path): + return None + with _connect(db_path) as conn: + _ensure_schema(conn) + row = conn.execute( + """ + SELECT + host_id, pane_id, turn_epoch, turn, status, refresh_status, + first_seen_at, last_attempt_at, next_attempt_at, + attempt_count, escalated_at + FROM herdr_turn_refresh_retries + WHERE host_id = ? AND pane_id = ? AND turn_epoch = ? AND turn = ? + """, + (str(host_id), str(pane_id), epoch, turn_number), + ).fetchone() + return _herdr_turn_refresh_retry_from_row(row) if row is not None else None + + +def herdr_turn_refresh_retry_due( + db_path: Path | str, + host_id: str, + pane_id: str, + *, + turn_epoch: int, + turn: int, + now: str | None = None, +) -> bool: + """Return whether an absent/pending retry may run under the local clock. + + A backwards local-clock jump makes the retry immediately due. Combined + with the durable attempt ceiling this cannot leave a pane wedged waiting + for a wall clock to catch up. + """ + retry = get_herdr_turn_refresh_retry( + db_path, + host_id, + pane_id, + turn_epoch=turn_epoch, + turn=turn, + ) + if retry is None: + return True + if retry.status == "escalated" or retry.next_attempt_at is None: + return False + current = _connector_datetime(now or utc_timestamp()) + last_attempt = _connector_datetime(retry.last_attempt_at) + next_attempt = _connector_datetime(retry.next_attempt_at) + return current < last_attempt or current >= next_attempt + + +def record_herdr_turn_refresh_retry( + db_path: Path | str, + host_id: str, + pane_id: str, + *, + turn_epoch: int, + turn: int, + refresh_status: str, + now: str | None = None, + base_delay_seconds: int = 1, + max_delay_seconds: int = 30, + max_retry_age_seconds: int = 300, + max_attempts: int = 8, +) -> HerdrTurnRefreshRetry: + """Record one failed refresh attempt with bounded durable backoff.""" + epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") + turn_number = _herdr_turn_counter(turn, "turn") + normalized_status = str(refresh_status).strip() + if not normalized_status or len(normalized_status) > 128: + raise ValueError("refresh_status must contain at most 128 characters") + bounds = (base_delay_seconds, max_delay_seconds, max_retry_age_seconds) + if any( + isinstance(value, bool) or not isinstance(value, int) or value < 0 + for value in bounds + ): + raise ValueError("retry delays and age must be nonnegative integers") + if max_delay_seconds < base_delay_seconds: + raise ValueError("max_delay_seconds must be at least base_delay_seconds") + if ( + isinstance(max_attempts, bool) + or not isinstance(max_attempts, int) + or max_attempts < 1 + ): + raise ValueError("max_attempts must be a positive integer") + current_dt = _connector_datetime(now or utc_timestamp()) + current = current_dt.isoformat() + with _connect(db_path) as conn: + _ensure_schema(conn) + conn.execute("BEGIN IMMEDIATE") + try: + row = conn.execute( + """ + SELECT + host_id, pane_id, turn_epoch, turn, status, + refresh_status, first_seen_at, last_attempt_at, + next_attempt_at, attempt_count, escalated_at + FROM herdr_turn_refresh_retries + WHERE host_id = ? AND pane_id = ? AND turn_epoch = ? AND turn = ? + """, + (str(host_id), str(pane_id), epoch, turn_number), + ).fetchone() + if row is not None and str(row[4]) == "escalated": + conn.commit() + return _herdr_turn_refresh_retry_from_row(row) + first_seen = str(row[6]) if row is not None else current + attempt_count = (int(row[9]) if row is not None else 0) + 1 + age_seconds = max( + 0.0, + (current_dt - _connector_datetime(first_seen)).total_seconds(), + ) + escalated = ( + attempt_count >= max_attempts + or age_seconds >= max_retry_age_seconds + ) + # Anchor backoff at the later of the current and last local sample. + # The due predicate explicitly detects rollback, while this avoids + # persisting decreasing attempt timestamps. + attempt_dt = current_dt + if row is not None: + attempt_dt = max(attempt_dt, _connector_datetime(str(row[7]))) + delay = min( + max_delay_seconds, + base_delay_seconds * (2 ** min(attempt_count - 1, 30)), + ) + next_attempt = ( + None + if escalated + else (attempt_dt + timedelta(seconds=delay)).isoformat() + ) + escalated_at = attempt_dt.isoformat() if escalated else None + conn.execute( + """ + INSERT INTO herdr_turn_refresh_retries ( + host_id, pane_id, turn_epoch, turn, status, + refresh_status, first_seen_at, last_attempt_at, + next_attempt_at, attempt_count, escalated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(host_id, pane_id, turn_epoch, turn) DO UPDATE SET + status = excluded.status, + refresh_status = excluded.refresh_status, + last_attempt_at = excluded.last_attempt_at, + next_attempt_at = excluded.next_attempt_at, + attempt_count = excluded.attempt_count, + escalated_at = excluded.escalated_at + """, + ( + str(host_id), + str(pane_id), + epoch, + turn_number, + "escalated" if escalated else "pending", + normalized_status, + first_seen, + attempt_dt.isoformat(), + next_attempt, + attempt_count, + escalated_at, + ), + ) + conn.commit() + except Exception: + conn.rollback() + raise + retry = get_herdr_turn_refresh_retry( + db_path, + host_id, + pane_id, + turn_epoch=epoch, + turn=turn_number, + ) + if retry is None: + raise StoreSchemaError("herdr_turn_refresh_retry_unavailable") + return retry + + def record_herdr_turn_completion( db_path: Path | str, host_id: str, @@ -14392,6 +14661,7 @@ def record_herdr_turn_completion( worker_id: str | None, refreshed_turn_id: str | None, observed_at: str | None = None, + preserve_refresh_retry: bool = False, ) -> HerdrTurnWatermark: """Store completion provenance and advance its replay watermark atomically.""" epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") @@ -14411,6 +14681,8 @@ def record_herdr_turn_completion( str, ): raise ValueError("agent_session_path must be text or None") + if not isinstance(preserve_refresh_retry, bool): + raise ValueError("preserve_refresh_retry must be a boolean") current = observed_at or utc_timestamp() with _connect(db_path) as conn: _ensure_schema(conn) @@ -14476,6 +14748,15 @@ def record_herdr_turn_completion( """, (str(host_id), str(pane_id), epoch, turn_number, current), ) + if not preserve_refresh_retry: + conn.execute( + """ + DELETE FROM herdr_turn_refresh_retries + WHERE host_id = ? AND pane_id = ? + AND turn_epoch = ? AND turn = ? AND status = 'pending' + """, + (str(host_id), str(pane_id), epoch, turn_number), + ) conn.commit() except Exception: conn.rollback() @@ -19162,6 +19443,17 @@ def cleanup_herdr_turn_retention( ).fetchall() ] completion_ids = completion_pool[:bounded_batch] + completion_retry_keys = [] + if completion_ids: + placeholders = ",".join("?" for _ in completion_ids) + completion_retry_keys = conn.execute( + f""" + SELECT host_id, pane_id, turn_epoch, turn + FROM herdr_turn_completions + WHERE rowid IN ({placeholders}) + """, + completion_ids, + ).fetchall() remaining_budget = bounded_batch - len(completion_ids) watermark_params = { **params, @@ -19212,6 +19504,14 @@ def cleanup_herdr_turn_retention( """, completion_ids, ) + conn.executemany( + """ + DELETE FROM herdr_turn_refresh_retries + WHERE host_id = ? AND pane_id = ? + AND turn_epoch = ? AND turn = ? + """, + completion_retry_keys, + ) if watermark_keys: conn.executemany( """ @@ -19220,6 +19520,13 @@ def cleanup_herdr_turn_retention( """, watermark_keys, ) + conn.executemany( + """ + DELETE FROM herdr_turn_refresh_retries + WHERE host_id = ? AND pane_id = ? + """, + watermark_keys, + ) conn.commit() else: conn.rollback() diff --git a/tests/test_acp_permissions.py b/tests/test_acp_permissions.py index dfc844a..99cd2a8 100644 --- a/tests/test_acp_permissions.py +++ b/tests/test_acp_permissions.py @@ -373,7 +373,7 @@ def test_v27_provenance_migration_preserves_stale_pending_state( conn.commit() store_sqlite.init_store(db_path) with sqlite3.connect(db_path) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == (27,) + assert conn.execute("PRAGMA user_version").fetchone() == (28,) assert conn.execute( "SELECT freshness, route_kind FROM backend_pending" ).fetchone() == ("stale", "legacy") diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index d38c5df..e3003ef 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -33,12 +33,13 @@ SessionOpenMode, ) from tendwire.config import Config -from tendwire.core.models import WorkerBinding, utc_timestamp +from tendwire.core.models import Snapshot, Worker, WorkerBinding, utc_timestamp from tendwire.store.sqlite import ( expire_stale_worker_bindings, expire_worker_bindings, list_agent_events, list_worker_bindings, + save_snapshot, upsert_worker_bindings, ) @@ -169,6 +170,7 @@ def __init__(self, session_id: str = "session-private") -> None: self.load_resets = 0 self.update_failure: BaseException | None = None self.permission_failure: BaseException | None = None + self.completion_failure: BaseException | None = None persisted = SimpleNamespace(status="inserted") self.update_result: object = SimpleNamespace( event=persisted, @@ -221,6 +223,8 @@ def mark_prompt_complete( ) -> object: self.completions += 1 self.completion_reasons.append(stop_reason) + if self.completion_failure is not None: + raise self.completion_failure return self.completion_result @@ -741,6 +745,7 @@ def stop_during_bind(session_id: str, anchor: WorkerBinding) -> WorkerBinding: def test_prompt_rechecks_binding_before_remote_send(tmp_path: Path) -> None: client = FakeClient() + ingestor = FakeIngestor() db_path = tmp_path / "events.db" continuity = continuity_binding() upsert_worker_bindings(db_path, [continuity]) @@ -750,7 +755,7 @@ def test_prompt_rechecks_binding_before_remote_send(tmp_path: Path) -> None: binding=continuity, cwd=tmp_path, session_binding_callback=binding_callback(db_path), - ingestor=FakeIngestor(), # type: ignore[arg-type] + ingestor=ingestor, # type: ignore[arg-type] poll_timeout=0.01, stop_timeout=0.5, ).start() @@ -766,6 +771,8 @@ def test_prompt_rechecks_binding_before_remote_send(tmp_path: Path) -> None: with pytest.raises(AcpRuntimeBindingError): service.prompt("must not send", producer_turn_id="producer-private") assert [call[0] for call in client.calls].count("prompt") == 0 + assert ingestor.started == [] + assert ingestor.completions == 0 def test_new_requires_explicit_session_binder_before_launch(tmp_path: Path) -> None: @@ -1421,7 +1428,8 @@ def test_prompt_transport_failure_cancels_and_makes_runtime_terminal( assert raised.value is failure assert ingestor.started == ["producer-private"] - assert ingestor.completions == 0 + assert ingestor.completions == 1 + assert ingestor.completion_reasons == [StopReason.CANCELLED] assert ("cancel", ("session-private",), {}) in client.calls assert service.status().cancellation_requests == 1 assert service.status().state is RuntimeState.FAILED @@ -1431,7 +1439,104 @@ def test_prompt_transport_failure_cancels_and_makes_runtime_terminal( service.stop() -def test_invalid_prompt_response_never_marks_complete_and_propagates( +def test_prompt_failure_preserves_original_when_cancel_finalization_fails( + tmp_path: Path, +) -> None: + client = FakeClient() + original = AcpRequestTimeoutError("prompt timed out") + client.prompt_failure = original + ingestor = FakeIngestor() + ingestor.completion_failure = OSError("completion unavailable") + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRequestTimeoutError) as raised: + service.prompt("question", producer_turn_id="producer-private") + + assert raised.value is original + assert ingestor.completions == 1 + assert ingestor.completion_reasons == [StopReason.CANCELLED] + assert service.status().failure_type == "AcpRequestTimeoutError" + + +def test_prompt_failure_drains_queued_updates_before_cancelled_completion( + tmp_path: Path, +) -> None: + actions: list[str] = [] + + class UpdatingFailureClient(FakeClient): + def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: + self.calls.append(("prompt", (session_id, prompt), kwargs)) + self.events.put(update()) + raise AcpRequestTimeoutError("prompt timed out") + + class OrderedIngestor(FakeIngestor): + def ingest_update(self, raw: object, **kwargs: Any) -> object: + outcome = super().ingest_update(raw, **kwargs) + actions.append("update") + return outcome + + def mark_prompt_complete( + self, + stop_reason: StopReason = StopReason.END_TURN, + ) -> object: + actions.append("complete") + return super().mark_prompt_complete(stop_reason) + + client = UpdatingFailureClient() + ingestor = OrderedIngestor() + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRequestTimeoutError): + service.prompt("question", producer_turn_id="producer-private") + + assert actions == ["update", "complete"] + assert ingestor.completion_reasons == [StopReason.CANCELLED] + + +def test_prompt_transport_failure_materializes_one_cancelled_final_projection( + tmp_path: Path, +) -> None: + client = FakeClient() + failure = AcpRequestTimeoutError("prompt timed out") + client.prompt_failure = failure + current = binding() + service = bound_runtime(tmp_path, client, current) + save_snapshot( + tmp_path / "bound-events.db", + Snapshot( + host_id="host-a", + updated_at=utc_timestamp(), + workers=[ + Worker( + id=current.worker_id, + name="worker-public", + status="active", + fingerprint=current.worker_fingerprint, + ) + ], + ), + ) + service.start() + + with pytest.raises(AcpRequestTimeoutError) as raised: + service.prompt("question", producer_turn_id="producer-private") + + assert raised.value is failure + with sqlite3.connect(tmp_path / "bound-events.db") as conn: + turns = conn.execute( + "SELECT payload_json FROM turns WHERE host_id = ?", + ("host-a",), + ).fetchall() + revisions = conn.execute( + "SELECT final_state, assistant_final_text " + "FROM turn_content_revisions WHERE host_id = ? AND is_current = 1", + ("host-a",), + ).fetchall() + assert len(turns) == 1 + assert revisions == [("complete", "[ACP prompt cancelled]")] + + +def test_invalid_prompt_response_cancels_and_finalizes_open_turn( tmp_path: Path, ) -> None: client = FakeClient() @@ -1441,7 +1546,9 @@ def test_invalid_prompt_response_never_marks_complete_and_propagates( with pytest.raises(AcpRuntimeProtocolError, match="invalid response"): service.prompt("question", producer_turn_id="producer-private") - assert ingestor.completions == 0 + assert ("cancel", ("session-private",), {}) in client.calls + assert ingestor.completions == 1 + assert ingestor.completion_reasons == [StopReason.CANCELLED] assert service.status().state is RuntimeState.FAILED with pytest.raises(AcpRuntimeProtocolError): service.stop() diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py index 254ef85..74b47b6 100644 --- a/tests/test_agent_events.py +++ b/tests/test_agent_events.py @@ -1219,7 +1219,7 @@ def test_v25_to_v26_retains_legacy_tombstones_as_dedup_only(tmp_path: Path) -> N ) conn.commit() store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == (27,) + assert conn.execute("PRAGMA user_version").fetchone() == (28,) assert conn.execute( "SELECT observed_at FROM agent_event_tombstones WHERE event_id = ?", (legacy_event_id,), diff --git a/tests/test_backend_pending.py b/tests/test_backend_pending.py index 245b12d..ffc9afa 100644 --- a/tests/test_backend_pending.py +++ b/tests/test_backend_pending.py @@ -1449,7 +1449,7 @@ def test_current_schema_creation_has_exact_binding_and_claim_state(tmp_path: Pat db = tmp_path / "current-schema.db" init_store(db) with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 27 + assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 28 columns = { str(row[1]) for row in conn.execute("PRAGMA table_info(backend_pending)").fetchall() diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index e7f5d51..8a92c05 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -1788,7 +1788,7 @@ def test_v6_to_current_plan_migration_is_bounded_atomic_and_preserves_jobs( ).fetchall() } foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert version == store_sqlite.STORE_SCHEMA_VERSION == 27 + assert version == store_sqlite.STORE_SCHEMA_VERSION == 28 assert plan_row == (plan["plan_token"], 1, None, "active") assert job_count == 2 assert outbox_count == 3 diff --git a/tests/test_delivery_retention_migration.py b/tests/test_delivery_retention_migration.py index 0a52d02..b7d92a0 100644 --- a/tests/test_delivery_retention_migration.py +++ b/tests/test_delivery_retention_migration.py @@ -875,7 +875,7 @@ def test_v10_to_v12_migration_retains_finals_without_reposting_or_leaking( finals = _seed_v10_finals(db_path) init_store(db_path) - assert store_sqlite.STORE_SCHEMA_VERSION == 27 + assert store_sqlite.STORE_SCHEMA_VERSION == 28 delivered_key = _final_key(*finals["delivered"]) hold_keys = {_final_key(*finals[label]) for label in ("hold-a", "hold-b")} diff --git a/tests/test_delivery_retention_projection.py b/tests/test_delivery_retention_projection.py index 102fa24..253b70f 100644 --- a/tests/test_delivery_retention_projection.py +++ b/tests/test_delivery_retention_projection.py @@ -147,7 +147,7 @@ def _assert_continuity_integrity(db_path: Path) -> None: with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA user_version").fetchone() == ( store_sqlite.STORE_SCHEMA_VERSION, - ) == (27,) + ) == (28,) assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] current_counts = conn.execute( """ diff --git a/tests/test_delivery_retention_recovery.py b/tests/test_delivery_retention_recovery.py index 15f430a..653b2f5 100644 --- a/tests/test_delivery_retention_recovery.py +++ b/tests/test_delivery_retention_recovery.py @@ -973,7 +973,7 @@ def test_v12_migration_uses_effective_recovery_lineage_without_repost_or_hold( api = ConnectorOutboxAPI(db_path, HOST_ID) assert api.poll({"name": FINAL_NAME, "limit": 100})["items"] == [] with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 27 + assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 28 anchor = conn.execute( """ SELECT delivery_kind, status diff --git a/tests/test_herdr_events.py b/tests/test_herdr_events.py index 6cc4777..95a8fe2 100644 --- a/tests/test_herdr_events.py +++ b/tests/test_herdr_events.py @@ -52,7 +52,9 @@ SnapshotObservationContext, SnapshotRetentionPolicy, apply_backend_pending_observation, + get_herdr_turn_refresh_retry, get_herdr_turn_watermark, + herdr_turn_refresh_retry_due, init_store, latest_snapshot, list_attention_items, @@ -61,6 +63,7 @@ maybe_run_automatic_store_maintenance, merge_turn_content, pending_payload_from_store, + record_herdr_turn_refresh_retry, save_snapshot, set_herdr_turn_watermark, turns_payload_from_store, @@ -95,6 +98,22 @@ _PUBLIC_JSON_FORBIDDEN_COMPACT = {key.replace("_", "") for key in _PUBLIC_JSON_FORBIDDEN_KEYS} +def _force_herdr_turn_refresh_retry_due( + backend: HerdrEventBackend, + pane_id: str, + turn: int, +) -> None: + with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: + conn.execute( + """ + UPDATE herdr_turn_refresh_retries + SET next_attempt_at = '1970-01-01T00:00:00+00:00' + WHERE host_id = ? AND pane_id = ? AND turn = ? + """, + (backend.config.host_id, pane_id, turn), + ) + + def _assert_no_public_json_forbidden(value: Any, path: str = "$") -> None: if isinstance(value, dict): for key, item in value.items(): @@ -4420,6 +4439,163 @@ def test_periodic_reconcile_uses_config_and_zero_disables_it(tmp_path: Path) -> assert enabled.operational_status["last_reconcile_at"] is not None +def test_periodic_reconcile_replays_supported_turn_ledger( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = Config( + host_id="periodic-turn-replay", + data_dir=tmp_path, + db_path=tmp_path / "periodic-turn-replay.db", + herdr_backend="socket", + reconcile_interval_seconds=0.001, + ) + init_store(Path(config.db_path)) + backend = HerdrEventBackend(config, debounce_seconds=0) + backend._turn_api_supported = True + replays: list[bool] = [] + monkeypatch.setattr( + backend, + "_replay_turns_after_reconcile", + lambda: replays.append(True), + ) + backend._next_reconcile_monotonic = time.monotonic() - 1 + + backend._run_periodic_reconcile_if_due(_initial_pane_client()) + + assert replays == [True] + + +def test_periodic_reconcile_recovers_retry_without_stream_disconnect( + tmp_path: Path, +) -> None: + calls = 0 + + def process(_config: Config, _pane_id: str, **_kwargs: Any) -> Any: + nonlocal calls + calls += 1 + return SimpleNamespace( + status="binding_missing" if calls == 1 else "updated", + worker_id="claude", + refreshed_turn_id=None if calls == 1 else "periodic-public-turn", + ) + + config = Config( + host_id="periodic-live-turn-retry", + data_dir=tmp_path, + db_path=tmp_path / "periodic-live-turn-retry.db", + herdr_backend="socket", + reconcile_interval_seconds=300, + ) + init_store(Path(config.db_path)) + backend = HerdrEventBackend( + config, + debounce_seconds=0, + reconnect_delay_seconds=0, + turn_completion_processor=process, + ) + pane = _turn_api_pane() + pane_id = pane["pane_id"] + reconcile_client = _StaticClient( + workspaces=[{"id": "w123456789abcde", "name": "Build"}], + panes=[pane], + ) + backend.reconcile_once(client=reconcile_client) + set_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + turn_epoch=7, + last_turn=0, + ) + record = herdr_events._turn_completion_record( + {"pane": pane, **_turn_record(1)} + ) + backend._process_turn_record(record) + assert calls == 1 + assert backend._next_turn_replay_monotonic is not None + _force_herdr_turn_refresh_retry_due(backend, pane_id, 1) + backend._turn_api_probed = True + backend._turn_api_supported = True + + class ReplayClient: + def connect(self) -> None: + return None + + def close(self) -> None: + return None + + def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: + assert params == { + "pane_id": pane_id, + "since": 0, + "expected_epoch": 7, + } + return { + "turns": { + "pane_id": pane_id, + "turn_epoch": 7, + "records": [_turn_record(1)], + "truncated": False, + "oldest_available": 1, + } + } + + backend.client_factory = lambda _config: ReplayClient() + backend._next_reconcile_monotonic = time.monotonic() + 300 + backend._next_turn_replay_monotonic = time.monotonic() - 1 + backend._run_periodic_reconcile_if_due(reconcile_client) + + watermark = get_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + ) + assert calls == 2 + assert watermark is not None and watermark.last_turn == 1 + assert reconcile_client.calls == [ + "workspace.list", + "tab.list", + "pane.list", + "agent.list", + ] + + +def test_refresh_retry_ignores_producer_time_and_survives_clock_rollback( + tmp_path: Path, +) -> None: + db_path = tmp_path / "local-retry-clock.db" + init_store(db_path) + retry = record_herdr_turn_refresh_retry( + db_path, + "local-clock-host", + "pane-clock", + turn_epoch=1, + turn=1, + refresh_status="binding_missing", + now="2026-08-02T00:00:10+00:00", + ) + + assert retry.first_seen_at == "2026-08-02T00:00:10+00:00" + assert retry.attempt_count == 1 + assert not herdr_turn_refresh_retry_due( + db_path, + "local-clock-host", + "pane-clock", + turn_epoch=1, + turn=1, + now="2026-08-02T00:00:10.500000+00:00", + ) + assert herdr_turn_refresh_retry_due( + db_path, + "local-clock-host", + "pane-clock", + turn_epoch=1, + turn=1, + now="2026-08-01T23:59:00+00:00", + ) + + def test_debounce_batches_until_flush_and_shutdown_flushes(tmp_path: Path) -> None: backend = _backend(tmp_path, "debounce", debounce_seconds=60) backend.reconcile_once(client=_initial_pane_client()) @@ -5995,11 +6171,12 @@ def fake_run(args: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str "store_unavailable", ], ) -def test_default_completion_processor_nonterminal_status_advances_with_diagnostic( +def test_default_completion_processor_retryable_status_preserves_watermark( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, refresh_status: str, ) -> None: + monkeypatch.setattr(herdr_events.time, "time", lambda: 1_700_000_000.1) backend = _backend(tmp_path, f"default-composition-{refresh_status}") backend.reconcile_once( client=_StaticClient( @@ -6061,10 +6238,179 @@ def unavailable(*_args: Any, **_kwargs: Any) -> Any: backend.config.host_id, pane_id, ) - assert watermark is not None and watermark.last_turn == 1 + assert watermark is not None and watermark.last_turn == 0 assert backend.operational_status["turn_completion_diagnostics"] == { f"herdr_turn_completion_refresh_skipped:{refresh_status}": 1 } + with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: + assert conn.execute( + """ + SELECT turn, refreshed_turn_id + FROM herdr_turn_completions + WHERE host_id = ? AND pane_id = ? + """, + (backend.config.host_id, pane_id), + ).fetchall() == [] + + +def test_completion_binding_race_replays_once_after_binding_appears( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(herdr_events.time, "time", lambda: 1_700_000_000.1) + config = Config( + host_id="turn-api-binding-race", + data_dir=tmp_path, + db_path=tmp_path / "turn-api-binding-race.db", + herdr_backend="socket", + herdr_bin="turn-adapter", + herdr_timeout_seconds=0.5, + ) + init_store(Path(config.db_path)) + backend = HerdrEventBackend( + config, + debounce_seconds=0, + reconnect_delay_seconds=0, + ) + pane = _turn_api_pane() + pane_id = pane["pane_id"] + backend.reconcile_once( + client=_StaticClient( + workspaces=[{"id": "w123456789abcde", "name": "Build"}], + panes=[pane], + ) + ) + set_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + turn_epoch=7, + last_turn=0, + ) + + original_list_worker_bindings = herdr_turns.list_worker_bindings + monkeypatch.setattr( + herdr_turns, + "list_worker_bindings", + lambda *_args, **_kwargs: [], + ) + event_data = {"pane": pane, **_turn_record(1)} + assert backend.queue_event_envelope( + {"event": "pane.turn_completed", "data": event_data} + ) + + watermark = get_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + ) + assert watermark is not None and watermark.last_turn == 0 + + final_payload = { + "result": { + "turn": { + "available": True, + "user_text": "binding race prompt", + "assistant_final_text": "binding race response", + "complete": True, + "has_open_turn": False, + "source_turn_id": "binding-race-turn", + } + } + } + adapter_calls: list[list[str]] = [] + + def fake_run(args: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]: + adapter_calls.append(args) + return subprocess.CompletedProcess( + args=args, + returncode=0, + stdout=json.dumps(final_payload), + stderr="", + ) + + monkeypatch.setattr( + herdr_turns, + "list_worker_bindings", + original_list_worker_bindings, + ) + monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) + record = herdr_events._turn_completion_record(event_data) + _force_herdr_turn_refresh_retry_due(backend, pane_id, 1) + backend._process_turn_record(record) + backend._process_turn_record(record) + + watermark = get_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + ) + assert watermark is not None and watermark.last_turn == 1 + assert len(adapter_calls) == 1 + with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: + completion_rows = conn.execute( + """ + SELECT turn, refreshed_turn_id + FROM herdr_turn_completions + WHERE host_id = ? AND pane_id = ? + """, + (backend.config.host_id, pane_id), + ).fetchall() + final_ready_count = conn.execute( + """ + SELECT COUNT(*) + FROM connector_outbox + WHERE host_id = ? + AND connector = 'turn-final' + AND delivery_kind = 'final_ready' + """, + (backend.config.host_id,), + ).fetchone()[0] + assert len(completion_rows) == 1 + assert completion_rows[0][0] == 1 + assert completion_rows[0][1] + assert final_ready_count == 1 + + +def test_completed_turn_missing_content_remains_terminal( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + backend = _backend(tmp_path, "default-composition-missing") + backend.reconcile_once( + client=_StaticClient( + workspaces=[{"id": "w123456789abcde", "name": "Build"}], + panes=[_turn_api_pane()], + ) + ) + pane_id = _turn_api_pane()["pane_id"] + set_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + turn_epoch=7, + last_turn=0, + ) + monkeypatch.setattr( + herdr_turns, + "_refresh_turn_binding", + lambda *_args, **_kwargs: herdr_turns.TurnRefreshResult("missing", 0), + ) + + assert backend.queue_event_envelope( + { + "event": "pane.turn_completed", + "data": {"pane": _turn_api_pane(), **_turn_record(1)}, + } + ) + + watermark = get_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + ) + assert watermark is not None and watermark.last_turn == 1 + assert backend.operational_status["turn_completion_diagnostics"] == {} with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: assert conn.execute( """ @@ -6076,9 +6422,226 @@ def unavailable(*_args: Any, **_kwargs: Any) -> Any: ).fetchall() == [(1, None)] +def test_poison_completion_escalates_then_later_same_pane_turn_advances( + tmp_path: Path, +) -> None: + calls = 0 + + def process(_config: Config, _pane_id: str, **_kwargs: Any) -> Any: + nonlocal calls + calls += 1 + if calls <= herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS: + return SimpleNamespace( + status="failed", + worker_id="claude", + refreshed_turn_id=None, + ) + return SimpleNamespace( + status="updated", + worker_id="claude", + refreshed_turn_id=f"public-turn-{calls}", + ) + + backend = _turn_api_backend(tmp_path, "turn-api-poison", process) + pane_id = _turn_api_pane()["pane_id"] + set_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + turn_epoch=7, + last_turn=0, + ) + + poison_record = herdr_events._turn_completion_record( + {"pane": _turn_api_pane(), **_turn_record(1)} + ) + for attempt in range(herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS): + if attempt: + _force_herdr_turn_refresh_retry_due(backend, pane_id, 1) + backend._process_turn_record(poison_record) + backend._process_turn_record( + herdr_events._turn_completion_record( + {"pane": _turn_api_pane(), **_turn_record(2)} + ) + ) + + watermark = get_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + ) + assert watermark is not None and watermark.last_turn == 2 + assert backend.operational_status["turn_completion_diagnostics"] == { + "herdr_turn_completion_refresh_skipped:failed": ( + herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS + ), + "herdr_turn_completion_refresh_escalated:failed": 1, + } + retry = get_herdr_turn_refresh_retry( + backend.db_path, + backend.config.host_id, + pane_id, + turn_epoch=7, + turn=1, + ) + assert retry is not None + assert retry.status == "escalated" + assert retry.attempt_count == herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS + assert retry.escalated_at is not None + with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: + assert conn.execute( + """ + SELECT turn, refreshed_turn_id + FROM herdr_turn_completions + WHERE host_id = ? AND pane_id = ? + ORDER BY turn + """, + (backend.config.host_id, pane_id), + ).fetchall() == [ + ( + 1, + None, + ), + ( + 2, + f"public-turn-{herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS + 1}", + ), + ] + + +def test_retryable_completion_watermark_survives_backend_restart( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(herdr_events.time, "time", lambda: 1_700_000_000.1) + first = _turn_api_backend( + tmp_path, + "turn-api-retry-restart", + lambda *_args, **_kwargs: SimpleNamespace( + status="binding_missing", + worker_id=None, + refreshed_turn_id=None, + ), + ) + pane_id = _turn_api_pane()["pane_id"] + set_herdr_turn_watermark( + first.db_path, + first.config.host_id, + pane_id, + turn_epoch=7, + last_turn=0, + ) + record = herdr_events._turn_completion_record( + {"pane": _turn_api_pane(), **_turn_record(1)} + ) + first._process_turn_record(record) + before_restart = get_herdr_turn_watermark( + first.db_path, + first.config.host_id, + pane_id, + ) + assert before_restart is not None and before_restart.last_turn == 0 + + calls = 0 + + def process_after_restart( + _config: Config, + _pane_id: str, + **_kwargs: Any, + ) -> Any: + nonlocal calls + calls += 1 + return SimpleNamespace( + status="updated", + worker_id="claude", + refreshed_turn_id="public-turn-after-restart", + ) + + restarted = HerdrEventBackend( + first.config, + debounce_seconds=0, + reconnect_delay_seconds=0, + turn_completion_processor=process_after_restart, + ) + _force_herdr_turn_refresh_retry_due(restarted, pane_id, 1) + restarted._process_turn_record(record) + restarted._process_turn_record(record) + + after_restart = get_herdr_turn_watermark( + restarted.db_path, + restarted.config.host_id, + pane_id, + ) + assert after_restart is not None and after_restart.last_turn == 1 + assert calls == 1 + with closing(sqlite3.connect(str(restarted.db_path))) as conn, conn: + assert conn.execute( + """ + SELECT turn, refreshed_turn_id + FROM herdr_turn_completions + WHERE host_id = ? AND pane_id = ? + """, + (restarted.config.host_id, pane_id), + ).fetchall() == [(1, "public-turn-after-restart")] + + +def test_restart_finishes_escalation_written_before_completion_watermark( + tmp_path: Path, +) -> None: + processor_calls = 0 + + def process(_config: Config, _pane_id: str, **_kwargs: Any) -> Any: + nonlocal processor_calls + processor_calls += 1 + return SimpleNamespace(status="failed") + + backend = _turn_api_backend( + tmp_path, + "turn-api-escalation-crash-boundary", + process, + ) + pane_id = _turn_api_pane()["pane_id"] + set_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + turn_epoch=7, + last_turn=0, + ) + retry = record_herdr_turn_refresh_retry( + backend.db_path, + backend.config.host_id, + pane_id, + turn_epoch=7, + turn=1, + refresh_status="failed", + max_attempts=1, + ) + assert retry.status == "escalated" + + backend._process_turn_record( + herdr_events._turn_completion_record( + {"pane": _turn_api_pane(), **_turn_record(1)} + ) + ) + + watermark = get_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_id, + ) + assert processor_calls == 0 + assert watermark is not None and watermark.last_turn == 1 + assert backend.operational_status["turn_completion_diagnostics"] == { + "herdr_turn_completion_refresh_escalated_recovered:failed": 1 + } + + def test_replay_timeout_on_one_pane_does_not_block_other_panes_or_subscribe( tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, ) -> None: + monkeypatch.setattr(herdr_events.time, "time", lambda: 1_700_000_000.1) pane_a = _turn_api_pane() pane_b = { **_turn_api_pane(), @@ -6143,13 +6706,18 @@ def subscribe( assert set(processed) == {pane_a["pane_id"], pane_b["pane_id"]} assert len(client.subscriptions) == 1 - for pane in (pane_a, pane_b): - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane["pane_id"], - ) - assert watermark is not None and watermark.last_turn == 1 + timed_out = get_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_a["pane_id"], + ) + completed = get_herdr_turn_watermark( + backend.db_path, + backend.config.host_id, + pane_b["pane_id"], + ) + assert timed_out is not None and timed_out.last_turn == 0 + assert completed is not None and completed.last_turn == 1 def test_first_probe_pane_not_found_skips_only_that_pane(tmp_path: Path) -> None: diff --git a/tests/test_store.py b/tests/test_store.py index d38db29..f0b78c3 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -10278,7 +10278,7 @@ def test_v13_migration_repairs_nonpositive_turn_sequences_and_blocks_recurrence( init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 27 + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 28 assert conn.execute( """ SELECT turn_id, list_sequence @@ -14151,7 +14151,7 @@ def test_v20_to_v21_adds_herdr_turn_watermark_and_provenance_tables( with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA user_version").fetchone() == ( store_sqlite.STORE_SCHEMA_VERSION, - ) == (27,) + ) == (28,) assert { str(row[0]) for row in conn.execute( From 51e39bfa9353a32f4156757370195614ff47e796 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 18:31:11 +0800 Subject: [PATCH 53/83] fix(acp): recover console cursor across session remints --- src/tendwire/backends/acp_coordinator.py | 8 ++++++-- tests/test_acp_coordinator.py | 6 ++++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 65e565d..153b779 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -2173,9 +2173,14 @@ def _load_console_input_cursor( db_path: Path, host_id: str, worker_id: str, - session_id: str, + _session_id: str, generation: int, ) -> int: + # Herdr owns the visible input queue at the worker generation, not at an + # adapter session. A NEW adapter session can be reminted after a bridge + # failure while Herdr retains the same console generation and sequence. + # Recovering only from the reminted session would reset this cursor to zero + # and turn every retained input into a permanent false gap. after = 0 cursor = 0 while True: @@ -2184,7 +2189,6 @@ def _load_console_input_cursor( host_id, worker_id=worker_id, source="tendwire-console", - session_id=session_id, after_sequence=after, limit=1000, ) diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 82ea8e9..c459c04 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -293,6 +293,12 @@ def test_console_cursors_survive_restart_crash_boundaries(tmp_path: Path) -> Non assert _load_console_input_cursor( config.db_path, config.host_id, "worker-1", "session-a", 42 ) == 7 + # The visible Herdr input queue survives an ACP adapter remint. Its cursor + # is owned by worker generation 42, so a replacement session must recover + # the already acknowledged input rather than reporting a false gap. + assert _load_console_input_cursor( + config.db_path, config.host_id, "worker-1", "session-b", 42 + ) == 7 assert _load_console_input_cursor( config.db_path, config.host_id, "worker-1", "session-a", 43 ) == 0 From 011126abfbd432771b2d8bca773b3676ce4df7a3 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 19:46:08 +0800 Subject: [PATCH 54/83] fix(acp): cache optional legacy worker probes --- src/tendwire/backends/acp_coordinator.py | 45 ++++++++++ tests/test_acp_coordinator.py | 105 ++++++++++++++++++++++- 2 files changed, 148 insertions(+), 2 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 153b779..e934e2d 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -39,6 +39,7 @@ RuntimeState, SessionOpenMode, ) +from .herdr_protocol import HerdrErrorResponse from .herdr_socket import HerdrSocketClient @@ -205,6 +206,13 @@ def __init__( # use legacy PTY I/O only after Herdr positively stops publishing this # exact worker identity, never merely because reminting failed. self._published_acp_claims: dict[str, str] = {} + # Optional ACP policies discover workers from the same Herdr binding + # stream as legacy PTY agents. Remember an exact worker generation + # that positively reported it is not ACP-owned so the periodic pass + # does not issue a mutating endpoint-mint request every interval. A + # registration/identity change produces a new private binding + # fingerprint and therefore becomes probeable immediately. + self._optional_endpoint_absences: dict[str, str] = {} def start(self) -> "AcpRuntimeCoordinator": with self._lock: @@ -1045,6 +1053,12 @@ def _reconcile_locked(self, *, strict: bool) -> None: with self._lock: failed_claims = tuple(self._console_failed_claims.items()) published_claims = tuple(self._published_acp_claims.items()) + for worker_id, fingerprint in tuple( + self._optional_endpoint_absences.items() + ): + binding = current.get(worker_id) + if binding is None or binding.private_fingerprint != fingerprint: + self._optional_endpoint_absences.pop(worker_id, None) exact_authorities = ( self._herdr_authority_claims() if failed_claims or published_claims @@ -1084,9 +1098,28 @@ def _reconcile_locked(self, *, strict: bool) -> None: ] for worker_id, continuity in current.items(): self._require_reconcile_state(allow_starting=True) + with self._lock: + existing = self._slots.get(worker_id) + optional_absence = ( + self._optional_endpoint_absences.get(worker_id) + == continuity.private_fingerprint + ) + if existing is None and optional_absence: + continue try: self._reconcile_binding(continuity) except Exception as exc: # noqa: BLE001 + if ( + existing is None + and self.config.agent_event_source + in {"acp_shadow", "acp_preferred"} + and _optional_acp_absence(exc) + ): + with self._lock: + self._optional_endpoint_absences[worker_id] = ( + continuity.private_fingerprint + ) + continue failures.append(exc) self._retire_worker(worker_id) with self._lock: @@ -1561,6 +1594,18 @@ def _same_continuity(left: WorkerBinding, right: WorkerBinding) -> bool: ) +def _optional_acp_absence(exc: BaseException) -> bool: + """Recognize a positive, generation-scoped legacy/non-ACP classification.""" + + if not isinstance(exc, HerdrErrorResponse) or not isinstance(exc.error, Mapping): + return False + return exc.error.get("code") in { + "acp_worker_unauthenticated", + "acp_ownership_required", + "acp_adapter_unsupported", + } + + def _nonempty_text(value: Any, field: str) -> str: if not isinstance(value, str) or not value or value.strip() != value: raise AcpCoordinatorError(f"Herdr ACP endpoint {field} is invalid") diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index c459c04..d0cd70b 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -33,13 +33,21 @@ _record_console_submission_outcome, production_acp_runtime_factory, ) -from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult from tendwire.backends.acp_runtime import RuntimeState, SessionOpenMode +from tendwire.backends.herdr_protocol import HerdrErrorResponse +from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult from tendwire.command_submission import submit_acp_command, submit_command from tendwire.config import Config -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding +from tendwire.core.models import ( + BackendHealth, + Snapshot, + Worker, + WorkerBinding, + utc_timestamp, +) from tendwire.daemon import DaemonHooks, TendwireDaemon from tendwire.store.sqlite import ( + expire_worker_bindings, get_command_request, init_store, list_agent_events, @@ -2232,3 +2240,96 @@ def close(self) -> None: assert health["failure_type"] == "AcpCoordinatorError" finally: coordinator.stop() + + +def test_preferred_caches_exact_non_acp_generation_without_endpoint_churn( + tmp_path: Path, +) -> None: + config = _config(tmp_path, policy="acp_preferred") + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + endpoint_calls = 0 + + class NonAcpClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + nonlocal endpoint_calls + endpoint_calls += 1 + raise HerdrErrorResponse( + { + "code": "acp_worker_unauthenticated", + "message": "worker is not ACP-owned", + }, + "request-private", + ) + + def close(self) -> None: + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: NonAcpClient(), + reconcile_interval=60.0, + ).start() + try: + assert endpoint_calls == 1 + assert coordinator.status()["healthy"] is True + coordinator._reconcile(strict=False) + coordinator._reconcile(strict=False) + assert endpoint_calls == 1 + assert coordinator.status()["failure_type"] is None + + changed_at = utc_timestamp() + expire_worker_bindings( + config.db_path, + config.host_id, + backend="herdr", + private_fingerprints=["herdr-private-binding"], + now=changed_at, + reason="new-generation", + ) + upsert_worker_bindings( + config.db_path, + [ + replace( + _binding(), + private_fingerprint="herdr-private-binding-2", + observed_at=changed_at, + ) + ], + ) + coordinator._reconcile(strict=False) + assert endpoint_calls == 2 + finally: + coordinator.stop() + + +def test_required_does_not_cache_non_acp_endpoint_failure(tmp_path: Path) -> None: + config = _config(tmp_path, policy="acp_required") + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + endpoint_calls = 0 + + class NonAcpClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + nonlocal endpoint_calls + endpoint_calls += 1 + raise HerdrErrorResponse( + {"code": "acp_ownership_required", "message": "PTY-owned"}, + "request-private", + ) + + def close(self) -> None: + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: NonAcpClient(), + reconcile_interval=60.0, + ) + with pytest.raises(AcpCoordinatorError, match="failed to attach"): + coordinator.start() + assert endpoint_calls == 1 From 2d2f8105aae72d3bb7ab8ee8673657b6f5ddca17 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 19:55:24 +0800 Subject: [PATCH 55/83] fix(acp): probe optional workers without minting --- src/tendwire/backends/acp_coordinator.py | 41 +++++++++++------ tests/test_acp_coordinator.py | 58 +++++++++++++++--------- 2 files changed, 65 insertions(+), 34 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index e934e2d..c13a57f 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -209,9 +209,10 @@ def __init__( # Optional ACP policies discover workers from the same Herdr binding # stream as legacy PTY agents. Remember an exact worker generation # that positively reported it is not ACP-owned so the periodic pass - # does not issue a mutating endpoint-mint request every interval. A - # registration/identity change produces a new private binding - # fingerprint and therefore becomes probeable immediately. + # does not issue a mutating endpoint-mint request every interval. + # Cached workers are checked with the non-ticketing status method, so + # a later ACP registration becomes attachable immediately without + # relying on mutable observation fingerprints. self._optional_endpoint_absences: dict[str, str] = {} def start(self) -> "AcpRuntimeCoordinator": @@ -1053,11 +1054,9 @@ def _reconcile_locked(self, *, strict: bool) -> None: with self._lock: failed_claims = tuple(self._console_failed_claims.items()) published_claims = tuple(self._published_acp_claims.items()) - for worker_id, fingerprint in tuple( - self._optional_endpoint_absences.items() - ): + for worker_id in tuple(self._optional_endpoint_absences): binding = current.get(worker_id) - if binding is None or binding.private_fingerprint != fingerprint: + if binding is None: self._optional_endpoint_absences.pop(worker_id, None) exact_authorities = ( self._herdr_authority_claims() @@ -1100,12 +1099,28 @@ def _reconcile_locked(self, *, strict: bool) -> None: self._require_reconcile_state(allow_starting=True) with self._lock: existing = self._slots.get(worker_id) - optional_absence = ( - self._optional_endpoint_absences.get(worker_id) - == continuity.private_fingerprint - ) + optional_absence = worker_id in self._optional_endpoint_absences if existing is None and optional_absence: - continue + try: + status = self._resolve_status(continuity) + except Exception as exc: # noqa: BLE001 + if _optional_acp_absence(exc): + with self._lock: + self._optional_endpoint_absences[worker_id] = ( + continuity.worker_fingerprint + ) + continue + failures.append(exc) + continue + if status.lifecycle != "acp_owned_ready": + failures.append( + AcpCoordinatorError( + "ACP worker is attached without a local runtime" + ) + ) + continue + with self._lock: + self._optional_endpoint_absences.pop(worker_id, None) try: self._reconcile_binding(continuity) except Exception as exc: # noqa: BLE001 @@ -1117,7 +1132,7 @@ def _reconcile_locked(self, *, strict: bool) -> None: ): with self._lock: self._optional_endpoint_absences[worker_id] = ( - continuity.private_fingerprint + continuity.worker_fingerprint ) continue failures.append(exc) diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index d0cd70b..25a3d88 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -43,11 +43,9 @@ Snapshot, Worker, WorkerBinding, - utc_timestamp, ) from tendwire.daemon import DaemonHooks, TendwireDaemon from tendwire.store.sqlite import ( - expire_worker_bindings, get_command_request, init_store, list_agent_events, @@ -2250,11 +2248,28 @@ def test_preferred_caches_exact_non_acp_generation_without_endpoint_churn( init_store(config.db_path) upsert_worker_bindings(config.db_path, [_binding()]) endpoint_calls = 0 + status_calls = 0 + registered = False class NonAcpClient: def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: nonlocal endpoint_calls endpoint_calls += 1 + if registered: + return _endpoint() + raise HerdrErrorResponse( + { + "code": "acp_worker_unauthenticated", + "message": "worker is not ACP-owned", + }, + "request-private", + ) + + def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: + nonlocal status_calls + status_calls += 1 + if registered: + return _status(lifecycle="acp_owned_ready") raise HerdrErrorResponse( { "code": "acp_worker_unauthenticated", @@ -2266,10 +2281,26 @@ def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: def close(self) -> None: return None + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self.stopped = False + + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + self.stopped = True + + def status(self) -> Any: + return SimpleNamespace(healthy=not self.stopped, failure_type=None) + coordinator = AcpRuntimeCoordinator( config, threading.Event(), endpoint_client_factory=lambda _config: NonAcpClient(), + client_factory=lambda *_args, **_kwargs: object(), + runtime_factory=Runtime, reconcile_interval=60.0, ).start() try: @@ -2278,29 +2309,14 @@ def close(self) -> None: coordinator._reconcile(strict=False) coordinator._reconcile(strict=False) assert endpoint_calls == 1 + assert status_calls == 2 assert coordinator.status()["failure_type"] is None - changed_at = utc_timestamp() - expire_worker_bindings( - config.db_path, - config.host_id, - backend="herdr", - private_fingerprints=["herdr-private-binding"], - now=changed_at, - reason="new-generation", - ) - upsert_worker_bindings( - config.db_path, - [ - replace( - _binding(), - private_fingerprint="herdr-private-binding-2", - observed_at=changed_at, - ) - ], - ) + registered = True coordinator._reconcile(strict=False) assert endpoint_calls == 2 + assert status_calls == 3 + assert "worker-1" in coordinator._slots finally: coordinator.stop() From e8eb9f4e732d1236c6fcdb185a22d7e7c23715eb Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 20:08:03 +0800 Subject: [PATCH 56/83] perf(acp): reduce idle console control traffic --- src/tendwire/backends/acp_coordinator.py | 7 ++++++- tests/test_acp_coordinator.py | 5 +++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index c13a57f..4c1cd48 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -64,6 +64,11 @@ class AcpVisibleConsoleUnavailable(AcpCoordinatorError): # so queued pane input and the echoed response retain explicit headroom. _CONSOLE_OUTPUT_QUEUE_BUDGET_BYTES = 512 * 1024 _CONSOLE_OUTPUT_ITEM_TEXT_BYTES = 128 * 1024 +# Three idle ACP panes previously produced roughly 30 control exchanges per +# second on the Raspberry Pi. A 500 ms cadence keeps pane input/output latency +# sub-second while leaving enough server capacity for delivery and health API +# traffic. Active prompts continue independently inside their ACP runtimes. +_CONSOLE_BRIDGE_INTERVAL_SECONDS = 0.5 @dataclass(frozen=True, slots=True) @@ -516,7 +521,7 @@ def _run(self) -> None: self._failure_type = type(exc).__name__ def _run_console_bridge(self) -> None: - while not self._stop.wait(0.1): + while not self._stop.wait(_CONSOLE_BRIDGE_INTERVAL_SECONDS): if self._daemon_stop.is_set(): return self._bridge_console_slots() diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 25a3d88..c9ebcd5 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -18,6 +18,7 @@ AcpRuntimeCoordinator, HerdrAcpConsoleEndpoint, _RuntimeSlot, + _CONSOLE_BRIDGE_INTERVAL_SECONDS, _derived_binding, _console_event_output, _bounded_console_output, @@ -148,7 +149,6 @@ def test_endpoint_requires_explicit_acp_ownership_and_strict_attach_shape(tmp_pa replayed["endpoint"]["args"][4] = "42" with pytest.raises(AcpCoordinatorError, match="inconsistent"): _parse_endpoint(config, _binding(), replayed) - wrong_terminal = _endpoint() wrong_terminal["worker"]["terminal_id"] = "other-terminal" terminal_binding = replace( @@ -518,7 +518,8 @@ def tick() -> None: started = time.monotonic() coordinator._run_console_bridge() assert len(ticks) == 3 - assert ticks[-1] - started < 0.5 + assert 0.25 <= _CONSOLE_BRIDGE_INTERVAL_SECONDS <= 0.5 + assert ticks[-1] - started < 3.5 * _CONSOLE_BRIDGE_INTERVAL_SECONDS def test_console_bridge_dispatches_slow_workers_independently(tmp_path: Path) -> None: From cf8f750cf3784dd3681f1a07b151fc09edbe8b95 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 21:00:41 +0800 Subject: [PATCH 57/83] fix(acp): fence prompt generation before receipt --- src/tendwire/backends/acp_coordinator.py | 27 +++- src/tendwire/command_submission.py | 175 ++++++++++++++--------- tests/test_acp_coordinator.py | 75 ++++++++++ 3 files changed, 207 insertions(+), 70 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 4c1cd48..836a779 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -13,6 +13,7 @@ import threading import time from collections.abc import Callable, Mapping +from contextlib import contextmanager from concurrent.futures import Future, ThreadPoolExecutor from dataclasses import dataclass, field, replace from pathlib import Path @@ -123,6 +124,7 @@ def __init__( self._owner = owner self._worker = worker self._slot = slot + self._prepared = threading.local() @property def binding_fingerprint(self) -> str: @@ -141,8 +143,29 @@ def prompt( text, producer_turn_id=producer_turn_id, acknowledgement_timeout=timeout, + generation_prepared=bool( + getattr(self._prepared, "depth", 0) + ), ) + @contextmanager + def prepare(self): + """Fence one exact generation before its durable send receipt exists.""" + + with self._owner._reconcile_lock: + self._owner._require_reconcile_state(allow_starting=False) + if self._owner._current_slot(self._worker) is not self._slot: + raise AcpCoordinatorError("ACP worker route is stale") + self._owner._require_attached_generation(self._slot) + if self._owner._current_slot(self._worker) is not self._slot: + raise AcpCoordinatorError("ACP worker route is stale") + depth = int(getattr(self._prepared, "depth", 0)) + self._prepared.depth = depth + 1 + try: + yield self + finally: + self._prepared.depth = depth + EndpointClientFactory = Callable[[Config], Any] RuntimeFactory = Callable[..., AcpRuntime] @@ -1336,6 +1359,7 @@ def _submit_prompt( *, producer_turn_id: str, acknowledgement_timeout: float, + generation_prepared: bool = False, ) -> object: """Write through the exact route generation used by the receipt.""" @@ -1344,7 +1368,8 @@ def _submit_prompt( current = self._current_slot(worker) if current is not slot: raise AcpCoordinatorError("ACP worker route is stale") - self._require_attached_generation(slot) + if not generation_prepared: + self._require_attached_generation(slot) if self._current_slot(worker) is not slot: raise AcpCoordinatorError("ACP worker route is stale") # The route lease covers the complete JSON-RPC request frame, not diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index cca55ff..1f7144d 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -124,6 +124,12 @@ def prompt( timeout: float, ) -> object: ... + # Production routes may expose a context manager that fences their exact + # generation from the final authority check through the prompt-frame + # acknowledgement. Test and third-party routes remain compatible without + # it; submit_acp_command probes this method dynamically. + def prepare(self) -> Any: ... + AcpPromptRouter = Callable[[Worker], AcpPromptRoute | None] AcpWorkerOwner = Callable[[str, str], bool] @@ -2823,87 +2829,118 @@ def submit_acp_command( if route_required else None ) - try: - binding_fingerprint = str( - getattr(route, "binding_fingerprint", "") or "" - ).strip() - except Exception: # noqa: BLE001 - binding_fingerprint = "" - if not binding_fingerprint: - if takeover is not None: - return _request_in_progress(request) - return ( - _backend_unavailable(request, "ACP worker route has no durable authority") - if route_required - else None + def submit_through(active_route: AcpPromptRoute) -> CommandEnvelope | None: + try: + binding_fingerprint = str( + getattr(active_route, "binding_fingerprint", "") or "" + ).strip() + except Exception: # noqa: BLE001 + binding_fingerprint = "" + if not binding_fingerprint: + if takeover is not None: + return _request_in_progress(request) + return ( + _backend_unavailable( + request, "ACP worker route has no durable authority" + ) + if route_required + else None + ) + + canonical = build_canonical_mutation(request, public_worker_id=worker.id) + reservation = _reserve_canonical_request(config, request, canonical) + if isinstance(reservation, CommandEnvelope): + return reservation + send_started = _mark_request_send_started( + config, + request, + reservation, + binding_fingerprint=binding_fingerprint, + worker=worker, + instruction_text=_instruction_text(request), ) + if isinstance(send_started, CommandEnvelope): + return send_started + if not isinstance(send_started, Mapping): + return _recover_request(config, request, reservation.canonical) - canonical = build_canonical_mutation(request, public_worker_id=worker.id) - reservation = _reserve_canonical_request(config, request, canonical) - if isinstance(reservation, CommandEnvelope): - return reservation - send_started = _mark_request_send_started( - config, - request, - reservation, - binding_fingerprint=binding_fingerprint, - worker=worker, - instruction_text=_instruction_text(request), - ) - if isinstance(send_started, CommandEnvelope): - return send_started - if not isinstance(send_started, Mapping): - return _recover_request(config, request, reservation.canonical) + try: + active_route.prompt( + _instruction_text(request), + producer_turn_id=turn_submission_id( + config.host_id, + request.request_id or "", + ), + timeout=config.acp_request_timeout_seconds, + ) + except Exception: # noqa: BLE001 + return _finish_request( + config, + request, + reservation, + _instruction_uncertain_envelope( + request, + worker, + verdict="unknown", + ), + expected_state="send_started", + terminal_state="uncertain", + ) - try: - route.prompt( - _instruction_text(request), - producer_turn_id=turn_submission_id( - config.host_id, - request.request_id or "", - ), - timeout=config.acp_request_timeout_seconds, + observed_turn: Mapping[str, Any] | None = send_started + if config.db_path is not None: + try: + refreshed = linked_turn_for_submission( + config.db_path, + host_id=config.host_id, + request_id=request.request_id or "", + ) + except Exception: # noqa: BLE001 + refreshed = None + if isinstance(refreshed, Mapping): + observed_turn = refreshed + accepted = _accepted_send_envelope( + request, + worker, + observed_turn, + submission_verdict="submitted", ) - except Exception: # noqa: BLE001 return _finish_request( config, request, reservation, - _instruction_uncertain_envelope( - request, - worker, - verdict="unknown", - ), + accepted, expected_state="send_started", - terminal_state="uncertain", + terminal_state="accepted", ) - observed_turn: Mapping[str, Any] | None = send_started - if config.db_path is not None: - try: - refreshed = linked_turn_for_submission( - config.db_path, - host_id=config.host_id, - request_id=request.request_id or "", + prepare_route = getattr(route, "prepare", None) + if not callable(prepare_route): + return submit_through(route) + + # Enter the generation fence before reserving a receipt. A failed status + # check is therefore a retryable no-receipt outcome, never a fabricated + # send_started ambiguity. Keep the context active until the ACP client has + # acknowledged writing the complete prompt frame. + try: + preparation = prepare_route() + active_route = preparation.__enter__() + except Exception: # noqa: BLE001 - no receipt or transport exists yet + if takeover is not None: + return _request_in_progress(request) + return ( + _backend_unavailable( + request, "ACP worker route could not be prepared" ) - except Exception: # noqa: BLE001 - refreshed = None - if isinstance(refreshed, Mapping): - observed_turn = refreshed - accepted = _accepted_send_envelope( - request, - worker, - observed_turn, - submission_verdict="submitted", - ) - return _finish_request( - config, - request, - reservation, - accepted, - expected_state="send_started", - terminal_state="accepted", - ) + if route_required + else None + ) + if active_route is None: + active_route = route + try: + return submit_through(active_route) + finally: + preparation.__exit__(None, None, None) def _submit_command_v2( diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index c9ebcd5..12ad368 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -6,6 +6,7 @@ import threading import time from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager from dataclasses import replace from pathlib import Path from types import SimpleNamespace @@ -1129,6 +1130,13 @@ def prompt( raise self.failure +class _PreparationFailureRoute(_Route): + @contextmanager + def prepare(self): + raise AcpCoordinatorError("transient generation status failure") + yield self + + def _seed(config: Config) -> Worker: assert config.db_path is not None init_store(config.db_path) @@ -1212,6 +1220,73 @@ def test_acp_failure_after_send_started_is_uncertain_and_never_falls_back(tmp_pa assert receipt is not None and receipt["state"] == "uncertain" +def test_acp_generation_preflight_failure_is_retryable_before_receipt( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + route = _PreparationFailureRoute() + + envelope = submit_command( + config, + _request("request-preflight-retry"), + acp_prompt_router=lambda routed: route if routed == worker else None, + acp_required=True, + ) + + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert route.calls == [] + assert get_command_request( + config.db_path, + config.host_id, + "request-preflight-retry", + ) is None + + +def test_production_route_checks_generation_before_reserving_receipt( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + runtime = SimpleNamespace( + status=lambda: SimpleNamespace(healthy=True, failure_type=None), + _binding=_binding(), + ) + coordinator._slots[worker.id] = _RuntimeSlot( + _binding(), + "42", + runtime, + ) + coordinator._require_attached_generation = ( # type: ignore[method-assign] + lambda _slot: (_ for _ in ()).throw( + AcpCoordinatorError("transient Herdr status timeout") + ) + ) + + envelope = submit_command( + config, + _request("request-production-preflight"), + acp_prompt_router=coordinator.prompt_route, + acp_worker_owner=coordinator.claims_worker, + acp_required=True, + ) + + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert get_command_request( + config.db_path, + config.host_id, + "request-production-preflight", + ) is None + + def test_preferred_acp_owned_route_loss_fails_closed_without_receipt_or_legacy( tmp_path: Path, ) -> None: From 47bbc247d4ac7422b80ad96cc6ead1a9f91af247 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 22:11:43 +0800 Subject: [PATCH 58/83] fix(acp): persist receipt at transport boundary --- src/tendwire/backends/acp_client.py | 8 +++ src/tendwire/backends/acp_coordinator.py | 4 ++ src/tendwire/backends/acp_runtime.py | 6 ++ src/tendwire/command_submission.py | 69 +++++++++++++++++++---- tests/test_acp_client.py | 25 +++++++++ tests/test_acp_coordinator.py | 70 +++++++++++++++++++++++- 6 files changed, 170 insertions(+), 12 deletions(-) diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index c14a0aa..aacca86 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -406,6 +406,7 @@ def request( *, timeout: float | None = None, require_initialized: bool = True, + on_writing: Callable[[], None] | None = None, on_written: Callable[[], None] | None = None, ) -> Any: if require_initialized: @@ -429,6 +430,11 @@ def request( with self._pending_lock: self._pending[request_id] = pending try: + # The durable command receipt must cross send_started at the last + # definite no-write boundary. A callback failure here removes the + # pending waiter and no ACP frame has touched the transport. + if on_writing is not None: + on_writing() self._write( request_envelope(request_id, method, params), deadline=deadline, @@ -593,6 +599,7 @@ def prompt( prompt: str | Sequence[Mapping[str, Any]], *, timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, on_submitted: Callable[[], None] | None = None, ) -> PromptResult: content = list(self.prepare_prompt(prompt)) @@ -609,6 +616,7 @@ def prompt( "session/prompt", {"sessionId": session_id, "prompt": content}, timeout=self.prompt_timeout if timeout is None else timeout, + on_writing=on_send_start, on_written=on_submitted, ) response_received = True diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 836a779..a4ce5bb 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -136,6 +136,7 @@ def prompt( *, producer_turn_id: str, timeout: float, + on_send_start: Callable[[], None] | None = None, ) -> object: return self._owner._submit_prompt( self._worker, @@ -143,6 +144,7 @@ def prompt( text, producer_turn_id=producer_turn_id, acknowledgement_timeout=timeout, + on_send_start=on_send_start, generation_prepared=bool( getattr(self._prepared, "depth", 0) ), @@ -1359,6 +1361,7 @@ def _submit_prompt( *, producer_turn_id: str, acknowledgement_timeout: float, + on_send_start: Callable[[], None] | None = None, generation_prepared: bool = False, ) -> object: """Write through the exact route generation used by the receipt.""" @@ -1380,6 +1383,7 @@ def _submit_prompt( text, producer_turn_id=producer_turn_id, acknowledgement_timeout=acknowledgement_timeout, + on_send_start=on_send_start, ) def _route_binding_fingerprint( diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 7b05478..1cf2b67 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -156,6 +156,7 @@ def prompt( prompt: str | Sequence[Mapping[str, Any]], *, timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, on_submitted: Callable[[], None] | None = None, ) -> PromptResult: ... @@ -390,6 +391,7 @@ def prompt( producer_turn_id: str | None = None, timeout: float | None = None, drain_timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, on_submitted: Callable[[], None] | None = None, ) -> PromptResult: """Submit one prompt and finalize only after its prior updates drain.""" @@ -420,6 +422,8 @@ def prompt( raise try: prompt_kwargs: dict[str, Any] = {"timeout": timeout} + if on_send_start is not None: + prompt_kwargs["on_send_start"] = on_send_start if on_submitted is not None: prompt_kwargs["on_submitted"] = on_submitted result = self._client.prompt( @@ -472,6 +476,7 @@ def submit_prompt( producer_turn_id: str, acknowledgement_timeout: float, completion_timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, ) -> None: """Start a prompt and return after its complete frame is written. @@ -493,6 +498,7 @@ def run_prompt() -> None: prompt, producer_turn_id=producer_turn_id, timeout=completion_timeout, + on_send_start=on_send_start, on_submitted=acknowledged.set, ) except BaseException as exc: diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 1f7144d..712933c 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -122,6 +122,7 @@ def prompt( *, producer_turn_id: str, timeout: float, + on_send_start: Callable[[], None] | None = None, ) -> object: ... # Production routes may expose a context manager that fences their exact @@ -2851,17 +2852,51 @@ def submit_through(active_route: AcpPromptRoute) -> CommandEnvelope | None: reservation = _reserve_canonical_request(config, request, canonical) if isinstance(reservation, CommandEnvelope): return reservation - send_started = _mark_request_send_started( - config, - request, - reservation, - binding_fingerprint=binding_fingerprint, - worker=worker, - instruction_text=_instruction_text(request), - ) - if isinstance(send_started, CommandEnvelope): - return send_started - if not isinstance(send_started, Mapping): + + send_started: Mapping[str, Any] | None = None + send_start_outcome: CommandEnvelope | None = None + + class SendStartRejected(RuntimeError): + pass + + def mark_send_started_at_transport_boundary() -> None: + nonlocal send_started, send_start_outcome + started = _mark_request_send_started( + config, + request, + reservation, + binding_fingerprint=binding_fingerprint, + worker=worker, + instruction_text=_instruction_text(request), + ) + if isinstance(started, CommandEnvelope): + send_start_outcome = started + raise SendStartRejected + if not isinstance(started, Mapping): + send_start_outcome = _recover_request( + config, request, reservation.canonical + ) + raise SendStartRejected + send_started = started + + def retryable_before_transport() -> CommandEnvelope: + abandoned = False + if config.db_path is not None: + try: + abandoned = abandon_command_request_reservation( + config.db_path, + host_id=config.host_id, + request_id=request.request_id or "", + canonical_fingerprint=reservation.canonical.fingerprint, + owner_token=reservation.owner_token, + ) + except Exception: # noqa: BLE001 + abandoned = False + if abandoned: + return _backend_unavailable( + request, + "ACP prompt did not reach its transport boundary", + ) return _recover_request(config, request, reservation.canonical) try: @@ -2872,8 +2907,15 @@ def submit_through(active_route: AcpPromptRoute) -> CommandEnvelope | None: request.request_id or "", ), timeout=config.acp_request_timeout_seconds, + on_send_start=mark_send_started_at_transport_boundary, + ) + except SendStartRejected: + return send_start_outcome or _recover_request( + config, request, reservation.canonical ) except Exception: # noqa: BLE001 + if send_started is None: + return retryable_before_transport() return _finish_request( config, request, @@ -2887,6 +2929,11 @@ def submit_through(active_route: AcpPromptRoute) -> CommandEnvelope | None: terminal_state="uncertain", ) + if send_started is None: + # A route that claims success without crossing the durable + # transport boundary is not an accepted implementation. + return retryable_before_transport() + observed_turn: Mapping[str, Any] | None = send_started if config.db_path is not None: try: diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py index adf12ec..b3800ed 100644 --- a/tests/test_acp_client.py +++ b/tests/test_acp_client.py @@ -403,6 +403,31 @@ def test_absolute_session_paths_are_enforced_before_write() -> None: acp.new_session("relative/path") +def test_prewrite_callback_failure_emits_no_acp_frame( + monkeypatch: pytest.MonkeyPatch, +) -> None: + class ReceiptUnavailable(RuntimeError): + pass + + with client() as acp: + acp.initialize() + writes: list[object] = [] + + def forbidden_write(*args: object, **kwargs: object) -> None: + writes.append((args, kwargs)) + + monkeypatch.setattr(acp, "_write", forbidden_write) + with pytest.raises(ReceiptUnavailable): + acp.request( + "session/list", + {}, + on_writing=lambda: (_ for _ in ()).throw(ReceiptUnavailable()), + ) + + assert writes == [] + assert acp._pending == {} + + def test_prompt_content_is_validated_and_gated_by_negotiated_capabilities() -> None: with client("baseline") as acp: acp.initialize() diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 12ad368..54a0e39 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -1124,8 +1124,11 @@ def prompt( *, producer_turn_id: str, timeout: float, + on_send_start: Any = None, ) -> None: self.calls.append((text, producer_turn_id, timeout)) + if callable(on_send_start): + on_send_start() if self.failure is not None: raise self.failure @@ -1137,6 +1140,20 @@ def prepare(self): yield self +class _BeforeTransportFailureRoute(_Route): + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Any = None, + ) -> None: + del on_send_start + self.calls.append((text, producer_turn_id, timeout)) + raise AcpCoordinatorError("visible console route changed before write") + + def _seed(config: Config) -> Worker: assert config.db_path is not None init_store(config.db_path) @@ -1287,6 +1304,48 @@ def test_production_route_checks_generation_before_reserving_receipt( ) is None +def test_acp_failure_before_transport_boundary_is_immediately_retryable( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + failed_route = _BeforeTransportFailureRoute() + + first = submit_command( + config, + _request("request-prewrite-retry"), + acp_prompt_router=lambda routed: failed_route if routed == worker else None, + acp_required=True, + ) + assert first.status == "backend_unavailable" + assert first.disposition == "no_receipt" + receipt = get_command_request( + config.db_path, + config.host_id, + "request-prewrite-retry", + ) + assert receipt is not None + assert receipt["state"] == "reserved" + assert receipt["send_started_at"] is None + + good_route = _Route() + second = submit_command( + config, + _request("request-prewrite-retry"), + acp_prompt_router=lambda routed: good_route if routed == worker else None, + acp_required=True, + ) + assert second.status == "accepted" + assert second.disposition == "terminal_accepted" + assert len(good_route.calls) == 1 + receipt = get_command_request( + config.db_path, + config.host_id, + "request-prewrite-retry", + ) + assert receipt is not None and receipt["state"] == "accepted" + + def test_preferred_acp_owned_route_loss_fails_closed_without_receipt_or_legacy( tmp_path: Path, ) -> None: @@ -1653,8 +1712,17 @@ def test_concurrent_duplicate_acp_command_has_one_external_send(tmp_path: Path) release = threading.Event() class BlockingRoute(_Route): - def prompt(self, text: str, *, producer_turn_id: str, timeout: float) -> None: + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Any = None, + ) -> None: self.calls.append((text, producer_turn_id, timeout)) + if callable(on_send_start): + on_send_start() entered.set() assert release.wait(1.0) From 8f7c0fa3a08a8fd7b923068268c391b108cc3196 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Sun, 2 Aug 2026 23:53:27 +0800 Subject: [PATCH 59/83] fix(acp): mark send start only when writable --- src/tendwire/backends/acp_client.py | 18 ++++++++---- tests/test_acp_client.py | 44 +++++++++++++++++++++++++++-- 2 files changed, 55 insertions(+), 7 deletions(-) diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index aacca86..ce0db34 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -430,14 +430,10 @@ def request( with self._pending_lock: self._pending[request_id] = pending try: - # The durable command receipt must cross send_started at the last - # definite no-write boundary. A callback failure here removes the - # pending waiter and no ACP frame has touched the transport. - if on_writing is not None: - on_writing() self._write( request_envelope(request_id, method, params), deadline=deadline, + on_writing=on_writing, ) if on_written is not None: on_written() @@ -953,6 +949,7 @@ def _write( envelope: Mapping[str, Any], *, deadline: float | None = None, + on_writing: Callable[[], None] | None = None, ) -> None: payload = encode_message(envelope, max_frame_bytes=self.max_frame_bytes) if deadline is None: @@ -969,6 +966,7 @@ def _write( fd = process.stdin.fileno() remaining = memoryview(payload) bytes_written = 0 + write_started = False while remaining: wait = deadline - time.monotonic() if wait <= 0: @@ -979,6 +977,16 @@ def _write( raise AcpRequestTimeoutError( "timed out writing ACP frame to agent" ) + # Cross the durable send boundary only after the write + # lock is held and the transport reports writable. A + # timeout waiting for either condition has written zero + # bytes and remains safely retryable. The callback still + # runs before the first write, so a process crash cannot + # leave an externally visible frame without a receipt. + if not write_started: + if on_writing is not None: + on_writing() + write_started = True try: written = self._write_chunk(fd, remaining) except BlockingIOError: diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py index b3800ed..51669ba 100644 --- a/tests/test_acp_client.py +++ b/tests/test_acp_client.py @@ -413,10 +413,11 @@ class ReceiptUnavailable(RuntimeError): acp.initialize() writes: list[object] = [] - def forbidden_write(*args: object, **kwargs: object) -> None: + def forbidden_write(*args: object, **kwargs: object) -> int: writes.append((args, kwargs)) + return 0 - monkeypatch.setattr(acp, "_write", forbidden_write) + monkeypatch.setattr(acp, "_write_chunk", forbidden_write) with pytest.raises(ReceiptUnavailable): acp.request( "session/list", @@ -428,6 +429,45 @@ def forbidden_write(*args: object, **kwargs: object) -> None: assert acp._pending == {} +def test_write_lock_timeout_does_not_cross_prewrite_boundary() -> None: + with client() as acp: + acp.initialize() + callbacks: list[str] = [] + assert acp._write_lock.acquire(timeout=0.1) + try: + with pytest.raises(AcpRequestTimeoutError, match="waiting to write"): + acp.request( + "session/list", + {}, + timeout=0.01, + on_writing=lambda: callbacks.append("started"), + ) + finally: + acp._write_lock.release() + + assert callbacks == [] + assert acp._pending == {} + + +def test_unwritable_transport_does_not_cross_prewrite_boundary( + monkeypatch: pytest.MonkeyPatch, +) -> None: + with client() as acp: + acp.initialize() + callbacks: list[str] = [] + monkeypatch.setattr(acp, "_wait_writable", lambda _fd, _timeout: False) + with pytest.raises(AcpRequestTimeoutError, match="writing ACP frame"): + acp.request( + "session/list", + {}, + timeout=0.01, + on_writing=lambda: callbacks.append("started"), + ) + + assert callbacks == [] + assert acp._pending == {} + + def test_prompt_content_is_validated_and_gated_by_negotiated_capabilities() -> None: with client("baseline") as acp: acp.initialize() From d8f801f12d2a71595e5ea311c61c3955eb955f3b Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Mon, 3 Aug 2026 09:22:02 +0800 Subject: [PATCH 60/83] feat(acp): steer live messages into active turns --- src/tendwire/backends/acp_client.py | 51 ++++++++++ src/tendwire/backends/acp_coordinator.py | 57 ++++++++++++ src/tendwire/backends/acp_ingestion.py | 73 +++++++++++++++ src/tendwire/backends/acp_protocol.py | 14 +++ src/tendwire/backends/acp_runtime.py | 70 ++++++++++++++ src/tendwire/command_submission.py | 53 ++++++++++- tests/fixtures/acp_fake_agent.py | 12 +++ tests/test_acp_client.py | 26 ++++++ tests/test_acp_coordinator.py | 114 +++++++++++++++++++++++ tests/test_acp_ingestion.py | 54 +++++++++++ tests/test_acp_runtime.py | 49 ++++++++++ 11 files changed, 572 insertions(+), 1 deletion(-) diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index ce0db34..1521343 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -45,6 +45,8 @@ SessionResult, SessionUpdate, StopReason, + SteeringOutcome, + SteeringResult, decode_json_line, encode_message, error_envelope, @@ -239,6 +241,17 @@ def capabilities(self) -> AgentCapabilities | None: def initialize_result(self) -> InitializeResult | None: return self._initialize_result + @property + def steering_supported(self) -> bool: + """Whether the agent explicitly advertised the steering extension.""" + + initialized = self._initialize_result + if initialized is None: + return False + meta = initialized.raw.get("_meta") + steering = meta.get("steering") if isinstance(meta, Mapping) else None + return isinstance(steering, Mapping) and steering.get("supported") is True + @property def failure(self) -> BaseException | None: return self._failure @@ -633,6 +646,44 @@ def prompt( raise AcpEnvelopeError("session/prompt returned an invalid stopReason") from exc return PromptResult(parsed_reason, MappingProxyType(dict(raw))) + def steer_session( + self, + session_id: str, + prompt: str | Sequence[Mapping[str, Any]], + *, + timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, + on_submitted: Callable[[], None] | None = None, + ) -> SteeringResult: + """Inject input into an active turn through an advertised extension. + + ``codex-acp`` serializes these requests per session and either injects + into the live turn or starts a new turn after the prior one drains. + The method is never used unless the initialize response opted in. + """ + + if not self.steering_supported: + raise AcpCapabilityError("agent did not advertise steering capability") + content = list(self.prepare_prompt(prompt)) + result = self.request( + "_session/steering", + { + "sessionId": _nonempty(session_id, "session_id"), + "prompt": content, + }, + timeout=timeout, + on_writing=on_send_start, + on_written=on_submitted, + ) + raw = _require_mapping(result, "_session/steering result") + try: + outcome = SteeringOutcome(raw.get("outcome")) + except (TypeError, ValueError) as exc: + raise AcpEnvelopeError( + "_session/steering returned an invalid outcome" + ) from exc + return SteeringResult(outcome, MappingProxyType(dict(raw))) + def prepare_prompt( self, prompt: str | Sequence[Mapping[str, Any]], diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index a4ce5bb..92a0c6d 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -150,6 +150,28 @@ def prompt( ), ) + @property + def supports_steering(self) -> bool: + return self._owner._supports_steering(self._worker, self._slot) + + def steer( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Callable[[], None] | None = None, + ) -> object: + return self._owner._submit_steering( + self._worker, + self._slot, + text, + producer_turn_id=producer_turn_id, + acknowledgement_timeout=timeout, + on_send_start=on_send_start, + generation_prepared=bool(getattr(self._prepared, "depth", 0)), + ) + @contextmanager def prepare(self): """Fence one exact generation before its durable send receipt exists.""" @@ -1386,6 +1408,41 @@ def _submit_prompt( on_send_start=on_send_start, ) + def _supports_steering(self, worker: Worker, slot: _RuntimeSlot) -> bool: + try: + return self._current_slot(worker) is slot and slot.runtime.can_steer() + except Exception: + return False + + def _submit_steering( + self, + worker: Worker, + slot: _RuntimeSlot, + text: str, + *, + producer_turn_id: str, + acknowledgement_timeout: float, + on_send_start: Callable[[], None] | None = None, + generation_prepared: bool = False, + ) -> object: + """Steer the exact attached generation used by the receipt.""" + + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + current = self._current_slot(worker) + if current is not slot: + raise AcpCoordinatorError("ACP worker route is stale") + if not generation_prepared: + self._require_attached_generation(slot) + if self._current_slot(worker) is not slot or not slot.runtime.can_steer(): + raise AcpCoordinatorError("ACP steering route is unavailable") + return slot.runtime.submit_steering( + text, + producer_turn_id=producer_turn_id, + acknowledgement_timeout=acknowledgement_timeout, + on_send_start=on_send_start, + ) + def _route_binding_fingerprint( self, worker: Worker, diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py index 4551fad..7dbd1ab 100644 --- a/src/tendwire/backends/acp_ingestion.py +++ b/src/tendwire/backends/acp_ingestion.py @@ -248,6 +248,79 @@ def begin_prompt( self._local_prompt_recorded = True return result + def can_append_prompt(self) -> bool: + """Return whether a steering input can join the current logical turn.""" + + return self._source_turn_id is not None and not self._turn_complete + + def append_prompt( + self, + prompt: Sequence[Mapping[str, Any]], + *, + producer_turn_id: str, + ) -> AcpIngestionResult: + """Durably append one steering input to the current ACP turn.""" + + if not isinstance(producer_turn_id, str) or not producer_turn_id.strip(): + raise ValueError("producer_turn_id must be non-empty text") + if not self.can_append_prompt(): + raise RuntimeError("ACP steering requires an active turn") + blocks = [dict(block) for block in prompt] + if not blocks: + raise ValueError("prompt must contain at least one content block") + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + text = "\n".join( + str(block.get("text")) + for block in blocks + if block.get("type") == "text" and isinstance(block.get("text"), str) + ) + producer = producer_turn_id.strip() + source_event_id = "steer-input:" + stable_fingerprint( + {"producer_turn": producer} + ) + try: + canonical = self.projector.normalize_session_update( + { + "method": "session/update", + "params": { + "sessionId": self.session_id, + "update": { + "sessionUpdate": "user_message_chunk", + "messageId": source_event_id, + "content": {"type": "text", "text": text}, + }, + }, + }, + source_event_id=source_event_id, + replay=False, + ) + if canonical is None: + raise RuntimeError("outgoing ACP steering input was duplicated") + payload = canonical.get("payload") + if not isinstance(payload, Mapping): + raise RuntimeError("outgoing ACP steering projection is invalid") + canonical = { + **canonical, + "payload": { + **payload, + "prompt_content": deepcopy(blocks), + "outgoing": True, + "steering": True, + }, + } + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + result = self._accept( + canonical, + checkpoint=checkpoint, + prior_turn_state=prior_turn_state, + ) + if result.event is not None and result.event.status != "binding_changed": + self._local_prompt_recorded = True + return result + def reset_after_load(self) -> None: """Drop replay turn assembly before accepting a new active prompt.""" diff --git a/src/tendwire/backends/acp_protocol.py b/src/tendwire/backends/acp_protocol.py index 1143b63..8d6662b 100644 --- a/src/tendwire/backends/acp_protocol.py +++ b/src/tendwire/backends/acp_protocol.py @@ -82,6 +82,14 @@ class StopReason(str, Enum): CANCELLED = "cancelled" +class SteeringOutcome(str, Enum): + """Outcome returned by the capability-gated Codex ACP steering extension.""" + + INJECTED = "injected" + STARTED_NEW_TURN = "startedNewTurn" + FAILED = "failed" + + class PermissionOptionKind(str, Enum): ALLOW_ONCE = "allow_once" ALLOW_ALWAYS = "allow_always" @@ -248,6 +256,12 @@ class PromptResult: raw: Mapping[str, Any] +@dataclass(frozen=True, slots=True) +class SteeringResult: + outcome: SteeringOutcome + raw: Mapping[str, Any] + + @dataclass(frozen=True, slots=True) class SessionUpdate: session_id: str diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 1cf2b67..0a219e2 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -28,6 +28,7 @@ SessionResult, SessionUpdate, StopReason, + SteeringResult, ) @@ -165,6 +166,19 @@ def prepare_prompt( prompt: str | Sequence[Mapping[str, Any]], ) -> tuple[Mapping[str, Any], ...]: ... + @property + def steering_supported(self) -> bool: ... + + def steer_session( + self, + session_id: str, + prompt: str | Sequence[Mapping[str, Any]], + *, + timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, + on_submitted: Callable[[], None] | None = None, + ) -> SteeringResult: ... + def cancel(self, session_id: str) -> None: ... def next_session_event( @@ -282,6 +296,7 @@ def __init__( self._lifecycle_lock = threading.RLock() self._ingest_lock = threading.Lock() self._prompt_lock = threading.Lock() + self._steering_lock = threading.Lock() self._idle_condition = threading.Condition(self._state_lock) self._stop_event = threading.Event() self._threads: tuple[threading.Thread, ...] = () @@ -533,6 +548,61 @@ def run_prompt() -> None: ) acknowledged.wait(min(remaining, 0.01)) + def can_steer(self) -> bool: + """Return whether this runtime can append to a live ACP turn.""" + + try: + supported = getattr(self._client, "steering_supported", False) is True + except Exception: + supported = False + if not supported: + return False + with self._state_lock: + if self._state is not RuntimeState.RUNNING or self._failure is not None: + return False + with self._ingest_lock: + ingestor = self._ingestor + can_append = getattr(ingestor, "can_append_prompt", None) + return bool(callable(can_append) and can_append()) + + def submit_steering( + self, + prompt: str | Sequence[Mapping[str, Any]], + *, + producer_turn_id: str, + acknowledgement_timeout: float, + on_send_start: Callable[[], None] | None = None, + ) -> SteeringResult: + """Inject one input into the current turn through ACP steering.""" + + if acknowledgement_timeout <= 0: + raise ValueError("acknowledgement_timeout must be positive") + if not isinstance(producer_turn_id, str) or not producer_turn_id.strip(): + raise ValueError("producer_turn_id must be non-empty text") + with self._steering_lock: + self.raise_if_failed() + session_id, ingestor = self._running_components() + prepared = _prepare_prompt_content(self._client, prompt) + if not self.can_steer(): + raise AcpRuntimeStateError("ACP steering is unavailable") + + def start_and_record() -> None: + if on_send_start is not None: + on_send_start() + with self._ingest_lock: + result = ingestor.append_prompt( + prepared, + producer_turn_id=producer_turn_id.strip(), + ) + _raise_for_binding_rejection(result) + + return self._client.steer_session( + session_id, + prepared, + timeout=acknowledgement_timeout, + on_send_start=start_and_record, + ) + def cancel(self) -> None: """Cancel the active session and any permission requests pending in it.""" diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 712933c..555c6d2 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -131,6 +131,18 @@ def prompt( # it; submit_acp_command probes this method dynamically. def prepare(self) -> Any: ... + @property + def supports_steering(self) -> bool: ... + + def steer( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Callable[[], None] | None = None, + ) -> object: ... + AcpPromptRouter = Callable[[Worker], AcpPromptRoute | None] AcpWorkerOwner = Callable[[str, str], bool] @@ -2900,7 +2912,17 @@ def retryable_before_transport() -> CommandEnvelope: return _recover_request(config, request, reservation.canonical) try: - active_route.prompt( + use_steering = False + steer = getattr(active_route, "steer", None) + if _target_state_at_send(worker) == "active" and callable(steer): + try: + use_steering = ( + getattr(active_route, "supports_steering", False) is True + ) + except Exception: + use_steering = False + submit = steer if use_steering else active_route.prompt + route_result = submit( _instruction_text(request), producer_turn_id=turn_submission_id( config.host_id, @@ -2909,6 +2931,35 @@ def retryable_before_transport() -> CommandEnvelope: timeout=config.acp_request_timeout_seconds, on_send_start=mark_send_started_at_transport_boundary, ) + if use_steering: + raw_outcome = getattr(route_result, "outcome", None) + steering_outcome = getattr(raw_outcome, "value", raw_outcome) + if steering_outcome == "failed": + return _finish_request( + config, + request, + reservation, + _instruction_rejected_envelope( + request, + worker, + verdict="steering_failed", + ), + expected_state="send_started", + terminal_state="rejected", + ) + if steering_outcome not in {"injected", "startedNewTurn"}: + return _finish_request( + config, + request, + reservation, + _instruction_uncertain_envelope( + request, + worker, + verdict="unknown", + ), + expected_state="send_started", + terminal_state="uncertain", + ) except SendStartRejected: return send_start_outcome or _recover_request( config, request, reservation.canonical diff --git a/tests/fixtures/acp_fake_agent.py b/tests/fixtures/acp_fake_agent.py index e9a4ac0..fe381eb 100644 --- a/tests/fixtures/acp_fake_agent.py +++ b/tests/fixtures/acp_fake_agent.py @@ -97,6 +97,11 @@ def update(session_id: str, kind: str, **values: object) -> None: } ), "agentInfo": {"name": "fake", "version": "1.0"}, + **( + {"_meta": {"steering": {"supported": True}}} + if MODE == "steering" + else {} + ), **( { "authMethods": [ @@ -230,6 +235,13 @@ def update(session_id: str, kind: str, **values: object) -> None: } ) pending_permission_ids.add(900) + elif method == "_session/steering": + update( + params["sessionId"], + "user_message_chunk", + content=params["prompt"][0], + ) + response(request_id, {"outcome": "injected"}) elif method == "session/cancel": if MODE == "cancel_race" and pending_prompt_id is not None: send( diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py index 51669ba..f0a91a4 100644 --- a/tests/test_acp_client.py +++ b/tests/test_acp_client.py @@ -21,6 +21,7 @@ SessionUpdate, SessionUpdateKind, StopReason, + SteeringOutcome, ) @@ -178,6 +179,31 @@ def run_prompt() -> None: assert outcome[0].stop_reason is StopReason.END_TURN +def test_advertised_steering_extension_is_capability_gated() -> None: + with client("steering") as acp: + acp.initialize() + acp.new_session("/tmp/project") + acp.next_update(timeout=1) + assert acp.steering_supported + callbacks: list[str] = [] + result = acp.steer_session( + "s-new", + "live input", + on_send_start=lambda: callbacks.append("started"), + on_submitted=lambda: callbacks.append("submitted"), + ) + assert result.outcome is SteeringOutcome.INJECTED + assert callbacks == ["started", "submitted"] + echoed = acp.next_update(timeout=1) + assert echoed.update_kind is SessionUpdateKind.USER_MESSAGE_CHUNK + + with client() as acp: + acp.initialize() + assert not acp.steering_supported + with pytest.raises(AcpCapabilityError, match="steering"): + acp.steer_session("s1", "no capability") + + def test_ordered_session_event_api_preserves_cross_kind_reader_order() -> None: with client() as acp: acp.initialize() diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 54a0e39..d3205ba 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -1118,6 +1118,10 @@ def __init__(self, failure: BaseException | None = None) -> None: self.failure = failure self.calls: list[tuple[str, str, float]] = [] + @property + def supports_steering(self) -> bool: + return False + def prompt( self, text: str, @@ -1154,6 +1158,30 @@ def prompt( raise AcpCoordinatorError("visible console route changed before write") +class _SteeringRoute(_Route): + def __init__(self, outcome: str = "injected") -> None: + super().__init__() + self.outcome = outcome + self.steering_calls: list[tuple[str, str, float]] = [] + + @property + def supports_steering(self) -> bool: + return True + + def steer( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Any = None, + ) -> object: + self.steering_calls.append((text, producer_turn_id, timeout)) + if callable(on_send_start): + on_send_start() + return SimpleNamespace(outcome=self.outcome) + + def _seed(config: Config) -> Worker: assert config.db_path is not None init_store(config.db_path) @@ -1215,6 +1243,92 @@ def test_acp_command_uses_durable_receipt_and_duplicate_does_not_resend(tmp_path assert "acp-private-binding" not in json.dumps(first.to_dict()) +def test_active_acp_worker_uses_advertised_steering_instead_of_second_prompt( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + assert config.db_path is not None + active = replace(worker, status="active") + save_snapshot( + config.db_path, + Snapshot( + host_id=config.host_id, + updated_at="2026-07-31T00:00:01+00:00", + workers=[active], + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty", + ) + ], + ), + ) + upsert_worker_bindings(config.db_path, [_binding()]) + route = _SteeringRoute() + + envelope = submit_command( + config, + _request("request-active-steer"), + acp_prompt_router=lambda routed: route if routed.id == active.id else None, + acp_required=True, + ) + + assert envelope.status == "accepted" + assert envelope.disposition == "terminal_accepted" + assert route.calls == [] + assert len(route.steering_calls) == 1 + receipt = get_command_request( + config.db_path, + config.host_id, + "request-active-steer", + ) + assert receipt is not None and receipt["state"] == "accepted" + + +def test_definite_acp_steering_failure_is_rejected_not_uncertain( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + assert config.db_path is not None + active = replace(worker, status="active") + save_snapshot( + config.db_path, + Snapshot( + host_id=config.host_id, + updated_at="2026-07-31T00:00:01+00:00", + workers=[active], + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty", + ) + ], + ), + ) + upsert_worker_bindings(config.db_path, [_binding()]) + route = _SteeringRoute("failed") + + envelope = submit_command( + config, + _request("request-active-steer-failed"), + acp_prompt_router=lambda _routed: route, + acp_required=True, + ) + + assert envelope.status == "rejected" + assert envelope.disposition == "terminal_rejected" + receipt = get_command_request( + config.db_path, + config.host_id, + "request-active-steer-failed", + ) + assert receipt is not None and receipt["state"] == "rejected" + + def test_acp_failure_after_send_started_is_uncertain_and_never_falls_back(tmp_path: Path) -> None: config = _config(tmp_path) _seed(config) diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py index 3bf8af6..3d90cc4 100644 --- a/tests/test_acp_ingestion.py +++ b/tests/test_acp_ingestion.py @@ -535,6 +535,60 @@ def test_live_prompt_echo_is_suppressed_but_load_replay_user_message_is_retained assert list_public_agent_events(db_path, "host-a") == () +def test_steering_prompt_appends_to_active_turn_without_resetting_identity( + tmp_path: Path, +) -> None: + db_path = tmp_path / "events.db" + events: list[AgentEvent] = [] + turns: list[dict[str, object]] = [] + + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: + events.append(event) + return _appended(len(events), event) + + def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): + turns.append(dict(content)) + return TurnRefreshApplyResult(len(turns), False) + + binding = _binding() + ingestor = AcpSessionIngestor( + _config(db_path), + session_id="session-a", + stream_generation="generation-a", + binding=binding, + persist_event=_persist(append, apply), + ) + begun = ingestor.begin_prompt( + ({"type": "text", "text": "initial"},), + producer_turn_id="producer-initial", + ) + source_turn_id = ingestor.source_turn_id + steered = ingestor.append_prompt( + ({"type": "text", "text": "live follow-up"},), + producer_turn_id="producer-steer", + ) + + assert begun.event is not None + assert steered.event is not None + assert events[1].payload["steering"] is True + assert ingestor.source_turn_id == source_turn_id + content = ingestor.projector.project_turn_content("session-a") + assert content["user_text"] == "initial\n\nlive follow-up" + + ingestor.mark_prompt_complete() + assert not ingestor.can_append_prompt() + with pytest.raises(RuntimeError, match="active turn"): + ingestor.append_prompt( + ({"type": "text", "text": "too late"},), + producer_turn_id="producer-late", + ) + + @pytest.mark.parametrize( ("update_kind", "fields"), [ diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index e3003ef..ea9424e 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -22,6 +22,8 @@ SessionUpdate, SessionUpdateKind, StopReason, + SteeringOutcome, + SteeringResult, ) from tendwire.backends.acp_runtime import ( AcpRuntime, @@ -64,6 +66,11 @@ def __init__(self) -> None: self.restored_session_result: SessionResult | None = None self.closed = False self.close_calls = 0 + self.steering_supported = False + self.steering_result = SteeringResult( + SteeringOutcome.INJECTED, + {"outcome": "injected"}, + ) def initialize(self, **kwargs: Any) -> object: self.calls.append(("initialize", (), kwargs)) @@ -107,6 +114,16 @@ def prepare_prompt(self, prompt: object) -> tuple[dict[str, Any], ...]: return ({"type": "text", "text": prompt},) return tuple(dict(block) for block in prompt) # type: ignore[arg-type] + def steer_session(self, session_id: str, prompt: object, **kwargs: Any) -> object: + self.calls.append(("steer", (session_id, prompt), kwargs)) + on_send_start = kwargs.get("on_send_start") + if callable(on_send_start): + on_send_start() + on_submitted = kwargs.get("on_submitted") + if callable(on_submitted): + on_submitted() + return self.steering_result + def cancel(self, session_id: str) -> None: self.calls.append(("cancel", (session_id,), {})) @@ -171,6 +188,8 @@ def __init__(self, session_id: str = "session-private") -> None: self.update_failure: BaseException | None = None self.permission_failure: BaseException | None = None self.completion_failure: BaseException | None = None + self.appended: list[tuple[object, str]] = [] + self.appendable = True persisted = SimpleNamespace(status="inserted") self.update_result: object = SimpleNamespace( event=persisted, @@ -214,6 +233,13 @@ def begin_prompt( ) -> object: return self.start_turn(producer_turn_id=producer_turn_id) + def can_append_prompt(self) -> bool: + return self.appendable + + def append_prompt(self, prompt: object, *, producer_turn_id: str) -> object: + self.appended.append((prompt, producer_turn_id)) + return self.update_result + def reset_after_load(self) -> None: self.load_resets += 1 @@ -1221,6 +1247,29 @@ def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: service.stop() +def test_submit_steering_records_input_at_transport_boundary(tmp_path: Path) -> None: + client = FakeClient() + client.steering_supported = True + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + callbacks: list[str] = [] + result = service.submit_steering( + "live follow-up", + producer_turn_id="producer-steer", + acknowledgement_timeout=0.25, + on_send_start=lambda: callbacks.append("started"), + ) + assert result.outcome is SteeringOutcome.INJECTED + assert callbacks == ["started"] + assert len(ingestor.appended) == 1 + assert ingestor.appended[0][1] == "producer-steer" + assert [call[0] for call in client.calls].count("prompt") == 0 + assert [call[0] for call in client.calls].count("steer") == 1 + finally: + service.stop() + + def test_ignored_update_does_not_increment_persisted_counter(tmp_path: Path) -> None: client = FakeClient() ingestor = FakeIngestor() From 85b7f435d6138a39c788aaa46555a7ad168aa4af Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Mon, 3 Aug 2026 09:51:01 +0800 Subject: [PATCH 61/83] fix(acp): preserve bindings across observer refresh --- src/tendwire/backends/acp_coordinator.py | 5 +++ src/tendwire/backends/acp_runtime.py | 44 +++++++++++++++++++++++- tests/test_acp_coordinator.py | 1 + tests/test_acp_runtime.py | 41 ++++++++++++++++++++++ 4 files changed, 90 insertions(+), 1 deletion(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 92a0c6d..b8a7e9e 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -1651,6 +1651,11 @@ def _derived_binding( backend="acp", turn_target_kind="acp_session_id", turn_target_value=session_id, + # This row is established by the current Tendwire process, not by the + # Herdr observation that supplied continuity. Reusing the observer's + # timestamp can lose the upsert to a newer expired row left by the + # previous process and make an otherwise valid restart fail closed. + observed_at=utc_timestamp(), # The ACP runtime owns this private lease until explicit stop/failure. # Inheriting the observer's short Herdr lease would strand a healthy # attached runtime after the next observation-expiry boundary. diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index 0a219e2..c38be80 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -884,7 +884,19 @@ def _require_current_binding(self, expected: WorkerBinding) -> None: expected.host_id, backend=expected.backend, ) - if expected not in current: + # Herdr may refresh only the observation lease while an ACP endpoint + # is being initialized. That does not change routing authority and + # must not invalidate the in-flight generation transaction. Every + # identity and routing field remains exact; process-owned ACP rows are + # still checked byte-for-byte so their revocation cannot be masked. + if expected.backend == "herdr": + present = any( + _same_binding_authority(item, expected) + for item in current + ) + else: + present = expected in current + if not present: raise AcpRuntimeBindingError("ACP worker binding is not current") def _start_consumer(self) -> None: @@ -1169,6 +1181,36 @@ def _runtime_client_capabilities( return {} +def _same_binding_authority(left: WorkerBinding, right: WorkerBinding) -> bool: + """Compare durable route authority while ignoring observer lease refreshes.""" + + return ( + left.host_id, + left.worker_id, + left.worker_fingerprint, + left.backend, + left.target_kind, + left.target_value, + left.turn_target_kind, + left.turn_target_value, + left.sendable, + left.reason, + left.private_fingerprint, + ) == ( + right.host_id, + right.worker_id, + right.worker_fingerprint, + right.backend, + right.target_kind, + right.target_value, + right.turn_target_kind, + right.turn_target_value, + right.sendable, + right.reason, + right.private_fingerprint, + ) + + def _raise_for_binding_rejection(outcome: object) -> None: """Make every stale durable-binding outcome terminal and public-safe.""" if outcome is None: diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index d3205ba..52ffdb6 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -1903,6 +1903,7 @@ def test_derived_acp_binding_outlives_observation_lease() -> None: ) derived = _derived_binding(continuity, "session-private") assert derived.expires_at.startswith("9999-") + assert derived.observed_at > continuity.observed_at assert derived.private_fingerprint != continuity.private_fingerprint diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index ea9424e..a26d0ae 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -673,6 +673,47 @@ def test_new_acp_binding_survives_herdr_refresh_and_normal_stop( assert released[0].reason == "acp_runtime_stopped" +def test_new_session_binding_accepts_concurrent_herdr_lease_refresh( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def bind_after_refresh(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + refreshed = replace( + anchor, + observed_at="2098-01-01T00:00:00+00:00", + expires_at="9999-12-31T23:59:59+00:00", + ) + upsert_worker_bindings(db_path, [refreshed]) + return binding_callback(db_path)(session_id, anchor) + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=bind_after_refresh, + ingestor=FakeIngestor(), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + try: + assert service.status().healthy + current = list_worker_bindings(db_path, "host-a", backend="herdr") + assert current == [ + replace( + continuity, + observed_at="2098-01-01T00:00:00+00:00", + expires_at="9999-12-31T23:59:59+00:00", + ) + ] + finally: + service.stop() + + @pytest.mark.parametrize("failure_mode", ("raise", "bad_return")) def test_new_cleans_binding_persisted_by_failed_callback( tmp_path: Path, From 5e2b70b80a49e35adb5f58cd047b25c5310cabda Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Mon, 3 Aug 2026 09:58:50 +0800 Subject: [PATCH 62/83] fix(acp): scope synthetic events to transport --- src/tendwire/backends/acp_coordinator.py | 7 ++++++- tests/test_acp_coordinator.py | 4 ++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index b8a7e9e..200acb1 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -1501,7 +1501,12 @@ def _build_runtime( cwd=endpoint.cwd, session_mode=endpoint.session_mode, session_id=endpoint.session_id, - stream_generation=endpoint.generation, + # Herdr's generation authenticates the worker lease and can + # remain stable across several freshly minted adapter + # transports. AcpRuntime deliberately creates a new stream + # nonce when this argument is omitted; reusing the Herdr + # generation would make synthetic notification identities + # collide after a Tendwire restart. session_binding_callback=callback, permission_callback=(permission_broker or self._permission_callback), poll_timeout=min(0.25, self.config.acp_request_timeout_seconds), diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 52ffdb6..0186ac8 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -1932,6 +1932,10 @@ def close(self) -> None: class Runtime: def __init__(self, _client: Any, **kwargs: Any) -> None: + # The endpoint generation fences Herdr authority, but must not be + # reused as the ACP transport stream generation. AcpRuntime owns a + # fresh nonce for every adapter process. + assert "stream_generation" not in kwargs self._binding = kwargs["binding"] self.stopped = False self.prompt_calls = 0 From 0a66883201b271f843f7b4c0eb135b4880b4ce3a Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Mon, 3 Aug 2026 10:08:24 +0800 Subject: [PATCH 63/83] feat(acp): support live-only console recovery --- src/tendwire/backends/acp_coordinator.py | 64 ++++++++++++++++++- src/tendwire/config.py | 22 +++++++ tests/test_acp_coordinator.py | 81 +++++++++++++++++++++++- tests/test_config.py | 13 ++++ 4 files changed, 176 insertions(+), 4 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 200acb1..05d40d0 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -55,6 +55,10 @@ class AcpPermissionBridgeUnavailable(AcpCoordinatorError): class AcpConsoleInputGap(AcpCoordinatorError): """The bounded Herdr console queue lost unconsumed pane input.""" + def __init__(self, message: str, *, recovery_after_sequence: int) -> None: + super().__init__(message) + self.recovery_after_sequence = recovery_after_sequence + class AcpVisibleConsoleUnavailable(AcpCoordinatorError): """A worker cannot accept new prompts while its pane bridge is lost.""" @@ -851,6 +855,53 @@ def _bridge_console_slot(self, slot: _RuntimeSlot) -> None: close = getattr(client, "close", None) if callable(close): close() + if gap_error is not None: + if self.config.acp_console_input_policy == "live_only": + # The operator explicitly chose live traffic over recovery of + # a partially discarded backlog. Persist the tail watermark + # before acknowledging it, then make one fresh exchange so an + # input arriving after the probe is still delivered. + input_sequence = max( + input_sequence, + gap_error.recovery_after_sequence, + ) + record_agent_event( + Path(self.config.db_path), + self.config.host_id, + kind="extension", + source="tendwire-console", + worker_id=slot.continuity.worker_id, + payload={ + "extension": "tendwire.acp.console_input_cursor", + "generation": console.generation, + "input_sequence": input_sequence, + "recovery": "live_only", + }, + source_session_id=binding.turn_target_value, + source_event_id=( + f"input-live-baseline:{console.generation}:{input_sequence}" + ), + visibility="private", + ) + client = self._endpoint_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + result = client.agent_acp_console_exchange( + slot.continuity.target_value, + generation=console.generation, + lease=console.lease, + after_input_sequence=input_sequence, + output=(), + timeout=self.config.herdr_timeout_seconds, + ) + inputs = _parse_console_exchange(result, input_sequence) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + gap_error = None if gap_error is not None: gap_output = [{ "event_id": f"console-gap:{console.generation}:{input_sequence}", @@ -2009,7 +2060,10 @@ def _parse_console_exchange( ): raise AcpCoordinatorError("Herdr ACP console exchange floors are invalid") if input_floor > after_sequence + 1: - raise AcpConsoleInputGap("Herdr ACP console input floor has a gap") + raise AcpConsoleInputGap( + "Herdr ACP console input floor has a gap", + recovery_after_sequence=next_input - 1, + ) parsed: list[tuple[int, str]] = [] expected = after_sequence + 1 for item in raw_inputs: @@ -2024,14 +2078,18 @@ def _parse_console_exchange( or not text.strip() ): if type(sequence) is int and sequence > expected: - raise AcpConsoleInputGap("Herdr ACP console input sequence has a gap") + raise AcpConsoleInputGap( + "Herdr ACP console input sequence has a gap", + recovery_after_sequence=next_input - 1, + ) raise AcpCoordinatorError("Herdr ACP console input sequence is invalid") parsed.append((sequence, text)) expected += 1 if next_input != expected: if next_input > expected: raise AcpConsoleInputGap( - "Herdr ACP console input response is incomplete" + "Herdr ACP console input response is incomplete", + recovery_after_sequence=next_input - 1, ) raise AcpCoordinatorError( "Herdr ACP console next input sequence is invalid" diff --git a/src/tendwire/config.py b/src/tendwire/config.py index 02dbdce..0ddc887 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -20,9 +20,11 @@ {"legacy", "acp_shadow", "acp_preferred", "acp_required"} ) ACP_THOUGHT_POLICIES = frozenset({"disabled", "private_summary", "private_all"}) +ACP_CONSOLE_INPUT_POLICIES = frozenset({"preserve", "live_only"}) DEFAULT_TURN_MODEL = "observed" DEFAULT_AGENT_EVENT_SOURCE = "legacy" DEFAULT_ACP_THOUGHT_POLICY = "disabled" +DEFAULT_ACP_CONSOLE_INPUT_POLICY = "preserve" DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS = 30.0 DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS = 5.0 DEFAULT_ACP_MAX_FRAME_BYTES = 8 * 1024 * 1024 @@ -79,6 +81,7 @@ class Config: turn_model: str = DEFAULT_TURN_MODEL agent_event_source: str = DEFAULT_AGENT_EVENT_SOURCE acp_thought_policy: str = DEFAULT_ACP_THOUGHT_POLICY + acp_console_input_policy: str = DEFAULT_ACP_CONSOLE_INPUT_POLICY acp_request_timeout_seconds: float = DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS acp_shutdown_timeout_seconds: float = DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS acp_max_frame_bytes: int = DEFAULT_ACP_MAX_FRAME_BYTES @@ -168,6 +171,19 @@ def __post_init__(self) -> None: allowed = ", ".join(sorted(ACP_THOUGHT_POLICIES)) raise ValueError(f"acp_thought_policy must be one of: {allowed}") object.__setattr__(self, "acp_thought_policy", acp_thought_policy) + acp_console_input_policy = str( + self.acp_console_input_policy or "" + ).strip().lower() + if acp_console_input_policy not in ACP_CONSOLE_INPUT_POLICIES: + allowed = ", ".join(sorted(ACP_CONSOLE_INPUT_POLICIES)) + raise ValueError( + f"acp_console_input_policy must be one of: {allowed}" + ) + object.__setattr__( + self, + "acp_console_input_policy", + acp_console_input_policy, + ) object.__setattr__( self, "acp_request_timeout_seconds", @@ -513,6 +529,7 @@ def load_config( turn_model: str | None = None, agent_event_source: str | None = None, acp_thought_policy: str | None = None, + acp_console_input_policy: str | None = None, acp_request_timeout_seconds: float | str | None = None, acp_shutdown_timeout_seconds: float | str | None = None, acp_max_frame_bytes: int | str | None = None, @@ -626,6 +643,11 @@ def load_config( "TENDWIRE_ACP_THOUGHT_POLICY", DEFAULT_ACP_THOUGHT_POLICY, ), + acp_console_input_policy=_resolve_value( + acp_console_input_policy, + "TENDWIRE_ACP_CONSOLE_INPUT_POLICY", + DEFAULT_ACP_CONSOLE_INPUT_POLICY, + ), acp_request_timeout_seconds=_resolve_value( acp_request_timeout_seconds, "TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS", diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 0186ac8..a19a8bd 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -15,6 +15,7 @@ import pytest from tendwire.backends.acp_coordinator import ( + AcpConsoleInputGap, AcpCoordinatorError, AcpRuntimeCoordinator, HerdrAcpConsoleEndpoint, @@ -182,8 +183,9 @@ def test_console_exchange_requires_floor_and_next_sequence_contract() -> None: with pytest.raises(AcpCoordinatorError, match="shape"): _parse_console_exchange(missing_next, 2) lost = dict(result, input_floor_sequence=4) - with pytest.raises(AcpCoordinatorError, match="gap"): + with pytest.raises(AcpConsoleInputGap, match="gap") as raised: _parse_console_exchange(lost, 2) + assert raised.value.recovery_after_sequence == 3 incomplete = dict(result, inputs=[], next_input_sequence=4) with pytest.raises(AcpCoordinatorError, match="incomplete"): @@ -211,6 +213,83 @@ def test_console_exchange_requires_floor_and_next_sequence_contract() -> None: _parse_console_exchange(wrong_output_next, 2) +def test_live_only_console_policy_skips_lost_backlog_to_current_tail( + tmp_path: Path, +) -> None: + config = replace( + _config(tmp_path), + acp_console_input_policy="live_only", + ) + assert config.db_path is not None + init_store(config.db_path) + exchanges: list[int] = [] + + class EndpointClient: + def agent_acp_console_exchange( + self, + _target: Any, + *, + generation: int, + lease: str, + after_input_sequence: int, + output: Any, + timeout: float, + ) -> dict[str, Any]: + exchanges.append(after_input_sequence) + return { + "type": "agent_acp_console_exchange", + "inputs": ( + [ + {"sequence": 3, "text": "historical one"}, + {"sequence": 4, "text": "historical two"}, + ] + if after_input_sequence == 0 + else [] + ), + "outputs": [], + "input_floor_sequence": 3, + "output_floor_sequence": 1, + "next_input_sequence": 5, + "next_output_sequence": 1, + } + + def close(self) -> None: + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + reconcile_interval=60.0, + ) + binding = replace( + _binding(), + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + private_fingerprint="", + ) + slot = _RuntimeSlot( + continuity=_binding(), + generation="42", + runtime=SimpleNamespace(_binding=binding), + console=HerdrAcpConsoleEndpoint(42, "coordinator-lease"), + console_cursor_loaded=True, + ) + + coordinator._bridge_console_slot(slot) + + assert exchanges == [0, 4] + assert slot.console_input_sequence == 4 + assert _load_console_input_cursor( + config.db_path, + config.host_id, + "worker-1", + "session-a", + 42, + ) == 4 + + def test_console_event_projection_covers_messages_thought_tools_and_plan() -> None: assert _console_event_output("agent_message", {"text_delta": "done"}) == ( "assistant", diff --git a/tests/test_config.py b/tests/test_config.py index 528c81d..8563b58 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -10,6 +10,7 @@ from tendwire.config import ( DEFAULT_ACP_MAX_FRAME_BYTES, + DEFAULT_ACP_CONSOLE_INPUT_POLICY, DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS, DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS, DEFAULT_ACP_THOUGHT_POLICY, @@ -82,6 +83,7 @@ def test_acp_event_source_defaults_to_legacy_with_thoughts_disabled( "TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS", "TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS", "TENDWIRE_ACP_MAX_FRAME_BYTES", + "TENDWIRE_ACP_CONSOLE_INPUT_POLICY", ): monkeypatch.delenv(name, raising=False) @@ -92,6 +94,7 @@ def test_acp_event_source_defaults_to_legacy_with_thoughts_disabled( assert config.acp_request_timeout_seconds == DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS == 30.0 assert config.acp_shutdown_timeout_seconds == DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS == 5.0 assert config.acp_max_frame_bytes == DEFAULT_ACP_MAX_FRAME_BYTES == 8 * 1024 * 1024 + assert config.acp_console_input_policy == DEFAULT_ACP_CONSOLE_INPUT_POLICY == "preserve" def test_acp_configuration_uses_explicit_before_environment(monkeypatch) -> None: @@ -100,6 +103,7 @@ def test_acp_configuration_uses_explicit_before_environment(monkeypatch) -> None monkeypatch.setenv("TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS", "11") monkeypatch.setenv("TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS", "3") monkeypatch.setenv("TENDWIRE_ACP_MAX_FRAME_BYTES", "4096") + monkeypatch.setenv("TENDWIRE_ACP_CONSOLE_INPUT_POLICY", "live_only") environment = load_config() explicit = load_config( @@ -108,6 +112,7 @@ def test_acp_configuration_uses_explicit_before_environment(monkeypatch) -> None acp_request_timeout_seconds="7.5", acp_shutdown_timeout_seconds="2.5", acp_max_frame_bytes="8192", + acp_console_input_policy="preserve", ) assert environment.agent_event_source == "acp_shadow" @@ -115,11 +120,19 @@ def test_acp_configuration_uses_explicit_before_environment(monkeypatch) -> None assert environment.acp_request_timeout_seconds == 11.0 assert environment.acp_shutdown_timeout_seconds == 3.0 assert environment.acp_max_frame_bytes == 4096 + assert environment.acp_console_input_policy == "live_only" assert explicit.agent_event_source == "acp_required" assert explicit.acp_thought_policy == "disabled" assert explicit.acp_request_timeout_seconds == 7.5 assert explicit.acp_shutdown_timeout_seconds == 2.5 assert explicit.acp_max_frame_bytes == 8192 + assert explicit.acp_console_input_policy == "preserve" + + +@pytest.mark.parametrize("value", ["", "drop", "future"]) +def test_acp_console_input_policy_rejects_unknown_values(value: str) -> None: + with pytest.raises(ValueError, match="acp_console_input_policy must be one of"): + Config(acp_console_input_policy=value) @pytest.mark.parametrize("value", ["", "acp", "preferred", "future"]) From c46aa084c22e5d3e24b0fd2bc44426fea721162b Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Mon, 3 Aug 2026 15:52:44 +0800 Subject: [PATCH 64/83] fix(acp): trust live steering capability --- src/tendwire/command_submission.py | 10 +++++++++- tests/test_acp_coordinator.py | 10 ++++++---- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 555c6d2..58a8e7c 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -2914,8 +2914,16 @@ def retryable_before_transport() -> CommandEnvelope: try: use_steering = False steer = getattr(active_route, "steer", None) - if _target_state_at_send(worker) == "active" and callable(steer): + if callable(steer): try: + # The ACP runtime is authoritative about whether this + # exact session currently has an appendable prompt. A + # Herdr snapshot can briefly remain idle after the ACP + # prompt has started; requiring both signals opens a + # second session/prompt that the adapter serializes behind + # the live turn, so its submission acknowledgement can + # never arrive in time. An actually idle runtime reports + # supports_steering=False and keeps the normal prompt path. use_steering = ( getattr(active_route, "supports_steering", False) is True ) diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index a19a8bd..f7de34e 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -1322,19 +1322,21 @@ def test_acp_command_uses_durable_receipt_and_duplicate_does_not_resend(tmp_path assert "acp-private-binding" not in json.dumps(first.to_dict()) -def test_active_acp_worker_uses_advertised_steering_instead_of_second_prompt( +@pytest.mark.parametrize("observed_status", ["idle", "active"]) +def test_live_acp_route_uses_advertised_steering_despite_observer_lag( tmp_path: Path, + observed_status: str, ) -> None: config = _config(tmp_path) worker = _seed(config) assert config.db_path is not None - active = replace(worker, status="active") + observed = replace(worker, status=observed_status) save_snapshot( config.db_path, Snapshot( host_id=config.host_id, updated_at="2026-07-31T00:00:01+00:00", - workers=[active], + workers=[observed], backend_health=[ BackendHealth( name="herdr", @@ -1350,7 +1352,7 @@ def test_active_acp_worker_uses_advertised_steering_instead_of_second_prompt( envelope = submit_command( config, _request("request-active-steer"), - acp_prompt_router=lambda routed: route if routed.id == active.id else None, + acp_prompt_router=lambda routed: route if routed.id == observed.id else None, acp_required=True, ) From d2ce261ff78213d016388837276616e5b58b4de1 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Mon, 3 Aug 2026 16:31:13 +0800 Subject: [PATCH 65/83] fix(acp): retry definite steering misses once --- src/tendwire/backends/acp_runtime.py | 22 ++++++++- src/tendwire/core/models.py | 1 + tests/test_acp_runtime.py | 71 ++++++++++++++++++++++++++++ tests/test_public_content_safety.py | 3 ++ 4 files changed, 96 insertions(+), 1 deletion(-) diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index c38be80..e69dd48 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -28,6 +28,7 @@ SessionResult, SessionUpdate, StopReason, + SteeringOutcome, SteeringResult, ) @@ -596,12 +597,31 @@ def start_and_record() -> None: ) _raise_for_binding_rejection(result) - return self._client.steer_session( + deadline = time.monotonic() + acknowledgement_timeout + result = self._client.steer_session( session_id, prepared, timeout=acknowledgement_timeout, on_send_start=start_and_record, ) + if result.outcome is not SteeringOutcome.FAILED: + return result + + # The steering extension defines ``failed`` as a definite + # non-application outcome. Retrying once is therefore safe and + # prevents a transient adapter/app-server race from turning a + # live Telegram message into a terminal drop. The durable input + # and transport-boundary callback were already recorded by the + # first attempt, so the retry deliberately omits the callback and + # stays inside the caller's original acknowledgement budget. + remaining = deadline - time.monotonic() + if remaining <= 0: + return result + return self._client.steer_session( + session_id, + prepared, + timeout=remaining, + ) def cancel(self) -> None: """Cancel the active session and any permission requests pending in it.""" diff --git a/src/tendwire/core/models.py b/src/tendwire/core/models.py index f1aa12a..32157c3 100644 --- a/src/tendwire/core/models.py +++ b/src/tendwire/core/models.py @@ -489,6 +489,7 @@ "agent_prompt_unsubmitted", "agent_input_pending", "agent_prompt_stalled", + "steering_failed", "unknown", } ) diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index a26d0ae..325d57c 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -1311,6 +1311,77 @@ def test_submit_steering_records_input_at_transport_boundary(tmp_path: Path) -> service.stop() +def test_submit_steering_retries_definite_non_application_once(tmp_path: Path) -> None: + class FailOnceSteeringClient(FakeClient): + def __init__(self) -> None: + super().__init__() + self.steering_supported = True + self.steering_attempts = 0 + + def steer_session( + self, session_id: str, prompt: object, **kwargs: Any + ) -> SteeringResult: + self.calls.append(("steer", (session_id, prompt), kwargs)) + self.steering_attempts += 1 + on_send_start = kwargs.get("on_send_start") + if callable(on_send_start): + on_send_start() + outcome = ( + SteeringOutcome.FAILED + if self.steering_attempts == 1 + else SteeringOutcome.INJECTED + ) + return SteeringResult(outcome, {"outcome": outcome.value}) + + client = FailOnceSteeringClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + callbacks: list[str] = [] + result = service.submit_steering( + "live retry follow-up", + producer_turn_id="producer-steer-retry", + acknowledgement_timeout=0.25, + on_send_start=lambda: callbacks.append("started"), + ) + assert result.outcome is SteeringOutcome.INJECTED + assert callbacks == ["started"] + assert len(ingestor.appended) == 1 + assert ingestor.appended[0][1] == "producer-steer-retry" + assert [call[0] for call in client.calls].count("steer") == 2 + assert client.calls[-1][2].get("on_send_start") is None + assert 0 < client.calls[-1][2]["timeout"] <= 0.25 + finally: + service.stop() + + +def test_submit_steering_returns_second_definite_failure_without_third_attempt( + tmp_path: Path, +) -> None: + client = FakeClient() + client.steering_supported = True + client.steering_result = SteeringResult( + SteeringOutcome.FAILED, + {"outcome": SteeringOutcome.FAILED.value}, + ) + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + callbacks: list[str] = [] + result = service.submit_steering( + "live failed follow-up", + producer_turn_id="producer-steer-failed", + acknowledgement_timeout=0.25, + on_send_start=lambda: callbacks.append("started"), + ) + assert result.outcome is SteeringOutcome.FAILED + assert callbacks == ["started"] + assert len(ingestor.appended) == 1 + assert [call[0] for call in client.calls].count("steer") == 2 + finally: + service.stop() + + def test_ignored_update_does_not_increment_persisted_counter(tmp_path: Path) -> None: client = FakeClient() ingestor = FakeIngestor() diff --git a/tests/test_public_content_safety.py b/tests/test_public_content_safety.py index 2a6b4ae..e94d7a9 100644 --- a/tests/test_public_content_safety.py +++ b/tests/test_public_content_safety.py @@ -375,6 +375,9 @@ def test_public_submission_verdict_is_closed_vocabulary() -> None: assert sanitize_public_value({"submission_verdict": "agent_not_ready"}) == { "submission_verdict": "agent_not_ready" } + assert sanitize_public_value({"submission_verdict": "steering_failed"}) == { + "submission_verdict": "steering_failed" + } assert sanitize_public_value( {"submission_verdict": "agent_target_ambiguous"} ) == {"submission_verdict": "agent_target_ambiguous"} From db7d9c4f3b90324fe3c28ed5b0b16818958d489b Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Mon, 3 Aug 2026 20:12:27 +0800 Subject: [PATCH 66/83] fix(commands): prompt resolved Herdr pane targets --- src/tendwire/command_submission.py | 8 +++++--- tests/test_command_submission.py | 15 ++++++++++++--- tests/test_daemon.py | 2 +- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 58a8e7c..30c4e51 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -1127,7 +1127,6 @@ class ReservedCommandMutation: class PreparedInstructionMutation: client: Any pane_id: str - target_value: str binding_fingerprint: str @@ -1188,7 +1187,6 @@ def _prepare_instruction( return PreparedInstructionMutation( client=client, pane_id=pane_or_error, - target_value=str(resolved.binding.target_value), binding_fingerprint=binding_fingerprint, ) @@ -1749,7 +1747,11 @@ def _submit_instruction( prepared.client, "agent.prompt", { - "target": prepared.target_value, + # Herdr 0.7.5 deliberately restricts agent.prompt to a + # current pane id or a unique live agent name. Private + # bindings may instead be keyed by terminal id, so use the + # pane resolved and validated during the pre-send phase. + "target": prepared.pane_id, "text": _instruction_text(request), "wait": { "until": ["working"], diff --git a/tests/test_command_submission.py b/tests/test_command_submission.py index 60b6584..75fcd54 100644 --- a/tests/test_command_submission.py +++ b/tests/test_command_submission.py @@ -312,6 +312,7 @@ def make_client(config: Config) -> _FakeSocketClient: def _expected_submit_calls( target: str = "agent-secret", *, + resolved_target: str = "pane-secret", text: str = "hello", timeout_ms: int = 5000, ) -> list[dict[str, Any]]: @@ -320,7 +321,7 @@ def _expected_submit_calls( { "method": "agent.prompt", "params": { - "target": target, + "target": resolved_target, "text": text, "wait": {"until": ["working"], "timeout_ms": timeout_ms}, }, @@ -1715,7 +1716,10 @@ def test_submit_command_terminal_binding_resolves_pane_and_submits_input(tmp_pat envelope = submit_command(config, _request(), socket_client_factory=_factory(calls, pane_id="pane-private")) assert envelope.status == STATUS_ACCEPTED - assert calls == _expected_submit_calls("term-secret") + assert calls == _expected_submit_calls( + "term-secret", + resolved_target="pane-private", + ) public_json = json.dumps(envelope.to_dict()) assert "term-secret" not in public_json assert "pane-private" not in public_json @@ -4485,6 +4489,11 @@ def request( "read": {"text": _REALISTIC_VISIBLE_PANE}, } if method == "agent.prompt": + if params.get("target") != "w1:p1": + raise HerdrErrorResponse( + {"code": "agent_not_found", "message": "target not found"}, + "req-2", + ) return { "type": "agent_prompted", "agent": {"pane_id": "w1:p1"}, @@ -4505,7 +4514,7 @@ def request( { "method": "agent.prompt", "params": { - "target": "term-dup", + "target": "w1:p1", "text": "hello", "wait": {"until": ["working"], "timeout_ms": 5000}, }, diff --git a/tests/test_daemon.py b/tests/test_daemon.py index 0fcfa61..92ae7b0 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -5025,7 +5025,7 @@ def submit(index: int) -> None: { "method": "agent.prompt", "params": { - "target": "agent-private", + "target": "pane-private", "text": "hello", "wait": {"until": ["working"], "timeout_ms": 5000}, }, From 2556e1852bb4526584cee4ceee5557981a103567 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Mon, 3 Aug 2026 21:41:08 +0800 Subject: [PATCH 67/83] Expose linked submissions in turn deltas --- src/tendwire/store/sqlite.py | 29 ++++++++++++++++++++++------- tests/test_turn_submissions.py | 12 +++++++++++- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index b8484b0..6dca6fc 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -24204,7 +24204,8 @@ def _update_turn_row( CASE WHEN revisions.final_state != 'absent' AND NOT (revisions.final_state = 'complete' AND revisions.final_char_length BETWEEN 1 AND :text_max) - THEN substr(revisions.assistant_final_text, 1, :preview_max) END + THEN substr(revisions.assistant_final_text, 1, :preview_max) END, + linked_submission.submission_id, linked_submission.state """ @@ -24217,7 +24218,8 @@ def _turn_delta_projection( revision, user_state, user_char_length, user_byte_length, user_page_count, user_inline, user_preview, final_state, final_char_length, final_byte_length, final_page_count, final_inline, final_preview, - ) = row[:18] + submission_id, submission_state, + ) = row[:20] try: loaded = json.loads(str(payload_json or "{}")) except (TypeError, json.JSONDecodeError): @@ -24260,6 +24262,9 @@ def _turn_delta_projection( str(turn_id), legacy_user, legacy_final, user_state=legacy_user_state, final_state=legacy_final_state, )) + if submission_id is not None and submission_state == "linked": + item["submission_id"] = str(submission_id) + item["submission_state"] = "linked" item["schema_version"] = TURN_DELTA_PROJECTION_SCHEMA_VERSION return item, dict(loaded) @@ -24313,6 +24318,8 @@ def _turn_delta_descriptor_only(projected: Mapping[str, Any]) -> dict[str, Any]: "superseded_by_turn_id", "superseded_at", "fingerprint", + "submission_id", + "submission_state", ) bounded = {key: projected[key] for key in descriptor_keys if key in projected} raw_content = projected.get("content") @@ -24526,6 +24533,10 @@ def _turn_delta_payload_from_store( ON revisions.host_id = turns.host_id AND revisions.turn_id = turns.turn_id AND revisions.is_current = 1 + LEFT JOIN turn_submissions AS linked_submission + ON linked_submission.host_id = turns.host_id + AND linked_submission.linked_turn_id = turns.turn_id + AND linked_submission.state = 'linked' WHERE turns.host_id = :host_id AND turns.list_sequence <= :insertion_high AND COALESCE(json_extract(turns.payload_json, '$.superseded_at'), '') = '' @@ -24576,13 +24587,17 @@ def _turn_delta_payload_from_store( ON revisions.host_id = turns.host_id AND revisions.turn_id = turns.turn_id AND revisions.is_current = 1 + LEFT JOIN turn_submissions AS linked_submission + ON linked_submission.host_id = turns.host_id + AND linked_submission.linked_turn_id = turns.turn_id + AND linked_submission.state = 'linked' ORDER BY positioned.seq, positioned.turn_id """, params).fetchall() if work_counters is not None: if mode == "changes": work_counters.journal_queries += 1 work_counters.journal_rows_scanned += ( - int(rows[0][21]) if rows else 0 + int(rows[0][23]) if rows else 0 ) work_counters.projection_queries += 1 work_counters.projection_rows_read += len(rows) @@ -24604,9 +24619,9 @@ def _turn_delta_payload_from_store( worker, sequence, turn_id = str(row[3]), int(row[4]), str(row[2]) changed_at = str(row[1] or utc_timestamp()) else: - worker, sequence, turn_id = "", int(row[18]), str(row[19]) - changed_at = str(row[20] or utc_timestamp()) - projected, raw_payload = _turn_delta_projection(tuple(row[:18])) + worker, sequence, turn_id = "", int(row[20]), str(row[21]) + changed_at = str(row[22] or utc_timestamp()) + projected, raw_payload = _turn_delta_projection(tuple(row[:20])) tombstoned = bool(str(raw_payload.get("superseded_at") or "").strip()) if mode == "changes" and ( row[2] is None or tombstoned or projected is None @@ -24661,7 +24676,7 @@ def _turn_delta_payload_from_store( "checkpoint": checkpoint, "aggregate": { "journal_rows_scanned": ( - int(rows[0][21]) if mode == "changes" and rows else 0 + int(rows[0][23]) if mode == "changes" and rows else 0 ), "projection_rows_read": len(rows), "changes_returned": len(changes), diff --git a/tests/test_turn_submissions.py b/tests/test_turn_submissions.py index 1aa634f..9c1d2dd 100644 --- a/tests/test_turn_submissions.py +++ b/tests/test_turn_submissions.py @@ -1944,7 +1944,7 @@ def record_candidates(*args: object, **kwargs: object): ] candidate_calls_after_observation = candidate_calls - turn_delta_payload_from_store( + linked_page = turn_delta_payload_from_store( db_path, "host-a", now=datetime.fromisoformat( @@ -1956,6 +1956,16 @@ def record_candidates(*args: object, **kwargs: object): assert _submission_rows(db_path) == [ ("observed-live-timeline", "linked", observed_turn_id) ] + linked_turn = next( + change["turn"] + for change in linked_page["changes"] + if change.get("op") == "upsert" + and change.get("turn_id") == observed_turn_id + ) + assert linked_turn["submission_id"] == turn_submission_id( + "host-a", "observed-live-timeline" + ) + assert linked_turn["submission_state"] == "linked" with sqlite3.connect(str(db_path)) as conn: linked_at = conn.execute( """ From 6e73c9f87b122c271f3ad35a071778a6782b68ab Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Mon, 3 Aug 2026 22:21:08 +0800 Subject: [PATCH 68/83] Fail exhausted zero-job final plans --- src/tendwire/store/sqlite.py | 55 +++++++++++++++++++--- tests/test_connector_outbox.py | 86 ++++++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+), 6 deletions(-) diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index 6dca6fc..e125016 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -5621,12 +5621,27 @@ def _mark_exhausted_presentation_plans_conn( SET state = 'failed' WHERE plans.host_id = ? {connector_clause} - AND plans.state IN ('active', 'waiting_predecessor') - AND EXISTS ( - SELECT 1 - FROM turn_presentation_jobs AS jobs - JOIN connector_outbox AS outbox ON outbox.id = jobs.outbox_id - WHERE jobs.plan_id = plans.id AND outbox.status = 'dead_letter' + AND ( + ( + plans.state IN ('active', 'waiting_predecessor') + AND EXISTS ( + SELECT 1 + FROM turn_presentation_jobs AS jobs + JOIN connector_outbox AS outbox + ON outbox.id = jobs.outbox_id + WHERE jobs.plan_id = plans.id + AND outbox.status = 'dead_letter' + ) + ) + OR ( + plans.state = 'preparing' + AND EXISTS ( + SELECT 1 + FROM connector_outbox AS source + WHERE source.id = plans.source_outbox_id + AND source.status = 'dead_letter' + ) + ) ) """, params, @@ -5778,6 +5793,25 @@ def prepare_connector_plan_commit( "generation": int(plan[7]), } ) + if plan_state in {"failed", "superseded"}: + # A source final can exhaust before commit materializes any + # child jobs. Preserve terminal idempotency so a recovering + # connector can abandon that zero-job plan instead of retrying + # a now-stale source reference forever. + conn.commit() + return _presentation_response( + { + "schema_version": _PRESENTATION_SCHEMA_VERSION, + "ok": True, + "status": "ok", + "host_id": str(host_id), + "name": str(name), + "plan_token": str(plan_token), + "state": plan_state, + "job_count": 0, + "generation": int(plan[7]), + } + ) _, early_revision_error = _current_presentation_revision_conn( conn, host_id=str(host_id), @@ -7539,6 +7573,15 @@ def _connector_update_ref( outbox_status=outbox_status, now=current_time, ) + if outbox_status == _CONNECTOR_EXHAUSTED_OUTBOX_STATUS: + # The exhausted row can be either a materialized plan job or + # the final-ready source anchor of a still-preparing plan. + _mark_exhausted_presentation_plans_conn( + conn, + host_id=str(host_id), + name=str(name), + now=current_time, + ) conn.commit() return _connector_response( ok=True, diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index 8a92c05..d46dc4f 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -1960,6 +1960,92 @@ def fail_second(*args: Any, **kwargs: Any) -> int: assert outbox_count == linked_count == 0 +def test_exhausted_source_fails_zero_job_preparing_plan_idempotently( + tmp_path: Path, +) -> None: + db_path = tmp_path / "prepare-source-exhausted.db" + turn_id, revision = _canonical_turn(db_path, final_text="abcdefgh") + with sqlite3.connect(str(db_path)) as conn: + conn.execute( + """ + INSERT INTO connector_outbox ( + host_id, connector, delivery_key, delivery_kind, + turn_id, content_revision, ordering_key, status, + payload_json, private_state_json, created_at, updated_at + ) VALUES ( + 'host-a', 'turn-final', ?, 'final_ready', ?, ?, + 'wsk1_source_exhausted', 'queued', '{}', '{}', ?, ? + ) + """, + ( + "turn-final:revision:twfinal1." + ("s" * 64), + turn_id, + revision, + "2026-01-01T00:00:00+00:00", + "2026-01-01T00:00:00+00:00", + ), + ) + api = ConnectorOutboxAPI(db_path, "host-a", max_attempts=1) + source = api.poll({"name": "turn-final", "limit": 1})["items"][0] + begun = api.prepare( + { + "schema_version": 1, + "action": "begin", + "name": "turn-final", + "turn_id": turn_id, + "content_revision": revision, + "presentation_version": "turn-present-v27", + "part_count": 1, + "source_ref": source["ref"], + } + ) + token = begun["plan_token"] + assert _put_final_part( + api, + plan_token=token, + ordinal=0, + start=0, + end=8, + )["ok"] is True + + exhausted = api.fail( + { + "name": "turn-final", + "ref": source["ref"], + "delay_seconds": 0, + } + ) + repeated_commit = api.prepare( + { + "schema_version": 1, + "action": "commit", + "name": "turn-final", + "plan_token": token, + "source_ref": source["ref"], + } + ) + + assert exhausted["status"] == "attempts_exhausted" + assert repeated_commit["ok"] is True + assert repeated_commit["state"] == "failed" + assert repeated_commit["job_count"] == 0 + with sqlite3.connect(str(db_path)) as conn: + assert conn.execute( + "SELECT state FROM turn_presentation_plans WHERE plan_token = ?", + (token,), + ).fetchone()[0] == "failed" + assert conn.execute( + """ + SELECT COUNT(*) + FROM turn_presentation_jobs + WHERE plan_id = ( + SELECT id FROM turn_presentation_plans WHERE plan_token = ? + ) AND outbox_id IS NOT NULL + """, + (token,), + ).fetchone()[0] == 0 + + def test_prepare_commit_rechecks_current_revision_and_creates_no_jobs_on_conflict( tmp_path: Path, ) -> None: From f28503d7a6618652bbbdf87179d5e3c88c23e779 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 00:12:05 +0800 Subject: [PATCH 69/83] Expose final delivery source age --- src/tendwire/connectors/outbox.py | 22 ++++++++++++---------- src/tendwire/store/sqlite.py | 4 +++- tests/test_connector_outbox.py | 15 +++++++++++++++ 3 files changed, 30 insertions(+), 11 deletions(-) diff --git a/src/tendwire/connectors/outbox.py b/src/tendwire/connectors/outbox.py index 802cac0..8d332e2 100644 --- a/src/tendwire/connectors/outbox.py +++ b/src/tendwire/connectors/outbox.py @@ -437,16 +437,18 @@ def poll(self, params: Mapping[str, Any] | None = None) -> dict[str, Any]: if not ref: continue clean_payload = _clean_mapping(item.get("payload")) - clean_item = sanitize_public_value( - { - "ref": ref, - "key": str(item.get("key") or ""), - "attempt": int(item.get("attempt") or 0), - "leased_until": str(item.get("leased_until") or ""), - "available_at": str(item.get("available_at") or ""), - "payload": clean_payload, - } - ) + public_item = { + "ref": ref, + "key": str(item.get("key") or ""), + "attempt": int(item.get("attempt") or 0), + "leased_until": str(item.get("leased_until") or ""), + "available_at": str(item.get("available_at") or ""), + "payload": clean_payload, + } + created_at = str(item.get("created_at") or "") + if name == _TURN_FINAL_NAME and created_at: + public_item["created_at"] = created_at + clean_item = sanitize_public_value(public_item) if isinstance(clean_item, dict): item_key = str(item.get("key") or "") if item_key.startswith(_FINAL_KEY_PREFIX): diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index e125016..51a2b41 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -6779,7 +6779,8 @@ def poll_connector_outbox( outbox.id, outbox.delivery_key, outbox.payload_json, - outbox.private_state_json + outbox.private_state_json, + outbox.created_at FROM connector_outbox AS outbox WHERE outbox.host_id = ? AND outbox.connector = ? @@ -7037,6 +7038,7 @@ def poll_connector_outbox( "leased_until": lease_expires_at, "ref": public_ref, "available_at": current_time, + "created_at": str(row[4] or ""), "payload": _restore_presentation_tokens( sanitize_public_mapping( _json_object(row[2]), diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index d46dc4f..97f3919 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -176,6 +176,21 @@ def test_poll_leases_sanitized_item_and_skips_duplicate_live_lease(tmp_path: Pat _assert_no_forbidden(first) +def test_final_ready_poll_exposes_durable_source_age(tmp_path: Path) -> None: + db_path = tmp_path / "final-created-at.db" + key = _enqueue_final_root( + db_path, + key_suffix="created_at", + ordering_key="wsk1_created_at", + ) + api = ConnectorOutboxAPI(db_path, "host-a") + + item = api.poll({"name": "turn-final", "limit": 1})["items"][0] + + assert item["key"] == key + assert item["created_at"] == "2026-01-01T00:00:00+00:00" + + def test_poll_preserves_strict_content_revision_tokens(tmp_path: Path) -> None: db_path = tmp_path / "revision-token.db" revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" From 3ea86860b8e9460d18704f5adfbd0cc7ea530670 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 07:33:11 +0800 Subject: [PATCH 70/83] fix(acp): finalize structured backend errors --- src/tendwire/backends/herdr_turns.py | 34 ++++++++++++++++ tests/test_herdr_turns.py | 60 ++++++++++++++++++++++++++++ 2 files changed, 94 insertions(+) diff --git a/src/tendwire/backends/herdr_turns.py b/src/tendwire/backends/herdr_turns.py index 570f788..04524e5 100644 --- a/src/tendwire/backends/herdr_turns.py +++ b/src/tendwire/backends/herdr_turns.py @@ -635,6 +635,21 @@ def _public_turn_pending_projection( } +def _backend_terminal_error_text(turn: Mapping[str, Any]) -> str | None: + """Return a bounded public final for a structured terminal adapter error.""" + + error = turn.get("api_error") + if not isinstance(error, Mapping): + return None + text = redact_private_prompt_text(error.get("text"), max_chars=600) + if text: + return text + code = redact_private_prompt_text(error.get("code"), max_chars=120) + if code: + return f"The agent ended this turn with an API error ({code})." + return "The agent ended this turn with an API error." + + class _TurnReadTimeout(Exception): """Fixed internal timeout signal; never serialized with private details.""" @@ -747,6 +762,15 @@ def _read_private_turn( turn.get("assistant_stream_text"), open_turn_id, ) + terminal_error = _backend_terminal_error_text(turn) + if opened is not None and terminal_error: + opened = { + **dict(opened), + "assistant_stream_text": None, + "assistant_final_text": terminal_error, + "complete": True, + "has_open_turn": False, + } if raise_timeout: opened_data = dict(opened or {}) opened_data.update(pending_projection) @@ -756,6 +780,16 @@ def _read_private_turn( return opened content = {key: turn.get(key) for key in _TURN_CONTENT_KEYS if key in turn} + terminal_error = _backend_terminal_error_text(turn) + if terminal_error: + content.update( + { + "assistant_stream_text": None, + "assistant_final_text": terminal_error, + "complete": True, + "has_open_turn": False, + } + ) content.update(pending_projection) if raise_timeout: content["_backend_pending_observation"] = pending_observation diff --git a/tests/test_herdr_turns.py b/tests/test_herdr_turns.py index cbceb2f..6a32aa6 100644 --- a/tests/test_herdr_turns.py +++ b/tests/test_herdr_turns.py @@ -646,6 +646,66 @@ def test_read_private_turn_emits_open_turn_from_open_fields(monkeypatch) -> None assert content["source_turn_id"] == "prompt-open" +def test_read_private_turn_terminalizes_structured_api_error(monkeypatch) -> None: + config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) + payload = { + "result": { + "turn": { + "available": True, + "turn_id": "prompt-error", + "user_text": "Please answer this.", + "assistant_final_text": "", + "complete": False, + "api_error": { + "code": "rate_limit_error", + "text": "You've hit your weekly limit. Try again after the reset.", + }, + } + } + } + + monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) + content = herdr_turns._read_private_turn(config, "pane-1") + + assert content is not None + assert content["source_turn_id"] == "prompt-error" + assert content["assistant_final_text"] == ( + "You've hit your weekly limit. Try again after the reset." + ) + assert content.get("assistant_stream_text") is None + assert content["complete"] is True + assert content["has_open_turn"] is False + + +def test_read_private_turn_terminalizes_open_fields_api_error(monkeypatch) -> None: + config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) + payload = { + "result": { + "turn": { + "available": True, + "complete": True, + "has_open_turn": True, + "turn_id": "older", + "user_text": "older prompt", + "assistant_final_text": "older answer", + "open_turn_id": "prompt-error", + "open_user_text": "current prompt", + "api_error": {"code": "overloaded_error", "text": "Provider overloaded."}, + } + } + } + + monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) + content = herdr_turns._read_private_turn(config, "pane-1") + + assert content is not None + assert content["source_turn_id"] == "prompt-error" + assert content["user_text"] == "current prompt" + assert content["assistant_final_text"] == "Provider overloaded." + assert content["complete"] is True + assert content["has_open_turn"] is False + + def test_open_turn_and_its_completion_share_source_turn_id(monkeypatch) -> None: """The open turn (prompt-open) and its later completion must share the id so a working card edits into the final instead of duplicating.""" From 91891bfb4aa44a03bea37571911b5c83e0ec977d Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 17:34:19 +0800 Subject: [PATCH 71/83] refactor(acp): require structured agent transport --- .env.example | 27 +- README.md | 46 +- docs/acp-migration.md | 329 +- pyproject.toml | 4 +- scripts/codex_session_reader_benchmark.py | 464 -- scripts/turn_ingestion_benchmark.py | 1549 ------- src/tendwire/backends/acp_client.py | 45 +- src/tendwire/backends/acp_coordinator.py | 191 +- src/tendwire/backends/acp_ingestion.py | 15 +- src/tendwire/backends/acp_probe.py | 4 +- src/tendwire/backends/acp_protocol.py | 40 + src/tendwire/backends/acp_runtime.py | 26 +- src/tendwire/backends/herdr_cli.py | 66 - src/tendwire/backends/herdr_decision.py | 124 - src/tendwire/backends/herdr_events.py | 843 +--- src/tendwire/backends/herdr_protocol.py | 19 +- src/tendwire/backends/herdr_socket.py | 41 +- src/tendwire/backends/herdr_turns.py | 4866 -------------------- src/tendwire/cli.py | 21 +- src/tendwire/command_submission.py | 1378 +----- src/tendwire/config.py | 71 - src/tendwire/daemon.py | 287 +- src/tendwire/store/sqlite.py | 6 +- tests/fixtures/acp_fake_agent.py | 27 +- tests/test_acp_atomic_ingestion.py | 22 +- tests/test_acp_client.py | 7 +- tests/test_acp_coordinator.py | 696 +-- tests/test_acp_ingestion.py | 166 +- tests/test_acp_permissions.py | 93 +- tests/test_acp_probe.py | 7 +- tests/test_acp_runtime.py | 13 +- tests/test_answer_decision.py | 888 ---- tests/test_backend.py | 8 +- tests/test_backend_pending.py | 1560 ------- tests/test_cli.py | 66 +- tests/test_cli_command.py | 11 - tests/test_codex_session_reader.py | 1226 ----- tests/test_command_presend_retryability.py | 1076 ----- tests/test_command_replay_authority.py | 1442 ------ tests/test_command_submission.py | 4542 ------------------ tests/test_config.py | 111 +- tests/test_daemon.py | 882 +--- tests/test_daemon_acp.py | 343 +- tests/test_herdr_events.py | 4271 +---------------- tests/test_herdr_socket.py | 45 - tests/test_herdr_turns.py | 2380 ---------- tests/test_turn_ingestion.py | 2371 ---------- tests/test_turn_ingestion_benchmark.py | 168 - tests/test_worker_label_and_model.py | 30 - tests/test_worker_stable_key.py | 32 +- 50 files changed, 899 insertions(+), 32046 deletions(-) delete mode 100755 scripts/codex_session_reader_benchmark.py delete mode 100755 scripts/turn_ingestion_benchmark.py delete mode 100644 src/tendwire/backends/herdr_decision.py delete mode 100644 src/tendwire/backends/herdr_turns.py delete mode 100644 tests/test_answer_decision.py delete mode 100644 tests/test_backend_pending.py delete mode 100644 tests/test_codex_session_reader.py delete mode 100644 tests/test_command_presend_retryability.py delete mode 100644 tests/test_command_replay_authority.py delete mode 100644 tests/test_command_submission.py delete mode 100644 tests/test_herdr_turns.py delete mode 100644 tests/test_turn_ingestion.py delete mode 100644 tests/test_turn_ingestion_benchmark.py diff --git a/.env.example b/.env.example index 6ef3179..364664d 100644 --- a/.env.example +++ b/.env.example @@ -80,29 +80,18 @@ TENDWIRE_STORE_MAINTENANCE_CADENCE_SECONDS=3600 # Optional stable host label for the local machine. # TENDWIRE_HOST_ID=my-host -# Optional Herdr binary and timeout overrides. Turn-adapter reads use the same -# per-RPC timeout. Initial socket reconciliation has a separate whole-startup -# budget so a multi-call inventory can exceed one RPC without weakening normal -# operation deadlines. +# Optional Herdr binary and timeout overrides. Herdr supplies worker/pane +# lifecycle and owns ACP endpoints; it is not an agent transcript transport. +# Initial socket reconciliation has a separate whole-startup budget so a +# multi-call inventory can exceed one RPC without weakening normal deadlines. # TENDWIRE_HERDR_BIN=herdr TENDWIRE_HERDR_TIMEOUT_SECONDS=1.0 TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS=120.0 -# Daemon-owned turn-ingestion cadence and dedicated worker pool. Defaults are -# 2.0 seconds and 4 workers; the worker count cannot exceed -# TENDWIRE_MAX_WORKERS. The internal refresh queue is fixed at 64. -TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS=2.0 -TENDWIRE_TURN_REFRESH_WORKERS=4 -# Compatibility flag: legacy|dual|shadow|observed all use the observed model. -TENDWIRE_TURN_MODEL=observed - -# Structured agent-event source policy. ACP runtime discovery and per-worker -# authority are experimental and are not wired into the stock daemon yet. -# acp_shadow requires an explicitly injected runtime and records ACP without -# projecting it. acp_required refuses legacy turn ingestion and fails closed -# unless that explicit runtime starts healthy. -TENDWIRE_AGENT_EVENT_SOURCE=legacy -# Stable ACP v1 does not distinguish raw reasoning from summaries. The safe +# ACP is required for every supported agent. The daemon fails closed when a +# worker lacks a healthy Herdr-owned ACP endpoint; there is no transcript, +# shadow, preferred, or fallback mode. Stable ACP v1 does not distinguish raw +# reasoning from summaries. The safe # default discards thought chunks. `private_summary` is an explicit trusted- # adapter convention; `private_all` retains all thoughts for local diagnostics. TENDWIRE_ACP_THOUGHT_POLICY=disabled diff --git a/README.md b/README.md index b0885e5..fd826a2 100644 --- a/README.md +++ b/README.md @@ -29,16 +29,12 @@ background daemon; use [INSTALL.md](INSTALL.md) for persistent service setup. ## Relationship to Herdr, Herdres, and connectors -Herdr is the only concrete runtime backend documented here. Tendwire can observe -Herdr through the conservative CLI one-shot path or, when explicitly enabled, -through the Herdr socket/event backend. Both paths normalize Herdr state into -neutral Tendwire spaces, workers, attention, turns, pending interactions, -command results, connector jobs, and backend health. - -ACP is the preferred semantic source for compatible, authenticated worker -sessions while Herdr remains the process and identity authority. Source -precedence, privacy, fallback behavior, and cross-repository rollout are -defined in [docs/acp-migration.md](docs/acp-migration.md). +Herdr supplies workspace, pane, and worker lifecycle plus ownership of private +ACP endpoints. Tendwire can observe that lifecycle through the conservative CLI +one-shot path or the Herdr socket/event backend. ACP is required for supported +agent semantics and commands; Herdr is not used to read transcripts or inject +pane input. The authority and privacy split is defined in +[docs/acp-migration.md](docs/acp-migration.md). Herdres can use Tendwire as its source/control plane while Herdres remains the Telegram connector. Tendwire owns Herdr observation, private bindings, @@ -555,10 +551,6 @@ variables: | `snapshot_retention_count` | `TENDWIRE_SNAPSHOT_RETENTION_COUNT` | `4096` | positive integer; includes each host's latest row | | `snapshot_maintenance_batch_size` | `TENDWIRE_SNAPSHOT_MAINTENANCE_BATCH_SIZE` | `100` | integer from 1 through 1000 | | `store_maintenance_cadence_seconds` | `TENDWIRE_STORE_MAINTENANCE_CADENCE_SECONDS` | `3600` | positive integer | -| `turn_refresh_interval_seconds` | `TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS` | `2.0` | finite positive float | -| `turn_refresh_workers` | `TENDWIRE_TURN_REFRESH_WORKERS` | `4` | integer from 1 through 32 and no greater than `max_workers` | -| `turn_model` | `TENDWIRE_TURN_MODEL` | `observed` | `observed`; `legacy`, `dual`, and `shadow` are deprecated aliases with identical observed behavior | -| `agent_event_source` | `TENDWIRE_AGENT_EVENT_SOURCE` | `legacy` | `legacy`, `acp_shadow`, `acp_preferred`, or `acp_required`; ACP modes are experimental | | `acp_thought_policy` | `TENDWIRE_ACP_THOUGHT_POLICY` | `disabled` | `disabled`, `private_summary`, or `private_all`; never a public-delivery grant | | `acp_request_timeout_seconds` | `TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS` | `30.0` | finite positive float | | `acp_shutdown_timeout_seconds` | `TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS` | `5.0` | finite positive float | @@ -575,9 +567,12 @@ snapshot/projections instead of publishing a truncated authoritative snapshot. Incremental events that would add workers over the cap are ignored with the same public-safe degraded evidence. -The stock daemon currently defaults to `legacy`. ACP modes use Herdr's private -`agent.acp_endpoint` contract and accept only workers explicitly marked -`acp_owned_ready`; ordinary PTY workers are never attached as sidecars. +The stock daemon requires ACP for supported agents. It uses Herdr's private +`agent.acp_endpoint` contract and accepts only workers explicitly marked +`acp_owned_ready`; ordinary PTY workers are never attached as sidecars. A +missing or unhealthy ACP endpoint degrades daemon health and fails command +submission closed. There is no transcript-reader scheduler, shadow/preferred +mode, or PTY fallback. Production ACP uses a bounded per-worker permission broker. It publishes only the sanitized tool title and numeric choices (option label and kind) through the durable pending-decision surface; ACP option IDs, arguments, session IDs, @@ -585,16 +580,9 @@ and adapter metadata remain private. `answer_decision` is fenced to the exact worker binding, ACP session, and Herdr generation. A command is accepted only after the complete JSON-RPC permission-response frame is written. Missing or retired ACP authority fails closed without falling back to PTY input, and -concurrent answers can produce at most one response. -`acp_shadow` persists ACP events without projecting them, but no automated -shadow comparator is implemented. For ACP-owned workers it is observation-only: -legacy turn ingestion is excluded and commands fail `backend_unavailable` -without sending on either transport. Validate real adapter execution with an -isolated `acp_preferred` or `acp_required` canary. `acp_preferred` falls back -only before an ACP reservation/send, while `acp_required` fails closed and -never starts the legacy turn scheduler. None of these modes makes agent thoughts -public: thought events remain private diagnostic data unless a separate, -explicit sanitized projection is introduced. +concurrent answers can produce at most one response. None of this makes agent +thoughts public: thought events remain private diagnostic data unless a +separate, explicit sanitized projection is introduced. Store maintenance retires expired structured agent-event payloads in bounded batches using `event_retention_days`. Compact identity tombstones remain so a @@ -1513,10 +1501,6 @@ metadata and does not change observation-derived turn identity, turn-list order, Goal 10 delivery, or Herdres consumption. Lazy and periodic settlement cover both submission-first and observation-first arrival order. -`TENDWIRE_TURN_MODEL` remains accepted for rollout compatibility. `legacy`, -`dual`, and `shadow` emit a warning and use the same observation-authoritative -behavior as `observed`. - `disposition`, not `status` alone, is the receipt-authority and finality contract: diff --git a/docs/acp-migration.md b/docs/acp-migration.md index bfd7d86..dde8263 100644 --- a/docs/acp-migration.md +++ b/docs/acp-migration.md @@ -1,237 +1,106 @@ -# ACP primary-event migration - -This document defines the experimental migration from backend-specific -transcript readers to Agent Client Protocol (ACP). ACP is not yet Tendwire's -default. The stock daemon contains the coordinator and command path, but -production ACP activation remains operator-gated on installing a supported ACP -adapter and explicitly registering shell-only panes as Herdr ACP-owned workers. -The coordinator never attaches an ordinary PTY worker as a sidecar. -Herdr remains authoritative for workspace, pane, worker identity, process -liveness, and command routing until the ACP control path is proven separately. -Tendwire remains authoritative for persistence, reconciliation, public safety, -command receipts, and connector delivery. - -## Source policy - -`TENDWIRE_AGENT_EVENT_SOURCE` controls projection precedence: - -- `legacy`: use the existing Herdr/Codex/OMP turn readers only. -- `acp_shadow`: ingest ACP events durably without projecting them. Ordinary - legacy workers remain legacy-authoritative. ACP-owned workers are excluded - from legacy turn ingestion, and command submission to them fails closed with - `backend_unavailable`: shadow is observation-only and does not execute an - equivalent prompt on both transports. Automated comparison is not - implemented yet. -- `acp_preferred`: use an explicitly Herdr-owned ACP endpoint when available; - fall back to legacy only before any ACP command reservation or observable - send. -- `acp_required`: use ACP only and fail closed when the binding or stream is not - healthy. The daemon does not start its legacy turn scheduler in this mode. - This mode requires every eligible worker endpoint to be ACP-owned and healthy. - Zero observed workers is a valid idle state; if a later worker appears - without ACP ownership, runtime health immediately becomes degraded. - -The default is `legacy`. In an ACP mode the coordinator asks Herdr for a -one-shot private endpoint, validates its worker generation and explicit -`acp_owned_ready` lifecycle, then creates/loads/resumes the ACP session. An -ordinary live PTY session is never treated as ACP-owned. - -ACP `session/request_permission` is synchronous and can authorize destructive -tools. The stock production factory enables a durable, worker/session-correlated -bridge from a public `answer_decision` command back to the exact request and -offered `optionId`. It publishes only a sanitized tool title and numbered -choices; option IDs, arguments, ACP session IDs, adapter metadata, and raw tool -payloads remain private. Selection is fenced to the exact worker binding and -Herdr generation, and is accepted only after the complete JSON-RPC response -frame is written. Missing, stale, timed-out, or uncertain authority fails -closed without a second transport attempt. Embedders may instead inject an -explicit callback into the generic coordinator, but the stock daemon does not -depend on such a callback. +# ACP-required architecture + +ACP is Tendwire's required semantic and command protocol for every supported +agent. This is a release boundary, not a runtime rollout switch: rollback means +deploying an earlier release. There are no legacy, shadow, preferred, dual, or +fallback modes in the daemon. ## Authority split | Concern | Authority | | --- | --- | -| Workspace and logical pane identity | Herdr | +| Workspace, pane, worker lifecycle, and ACP endpoint ownership | Herdr | | Public stable worker identity | Tendwire's authenticated Herdr projection | -| ACP session and message identity | ACP agent, stored privately by Tendwire | -| Messages, thoughts, tools, plans, and usage | ACP coordinator for ACP-owned workers; currently experimental | -| Turn finality and connector eligibility | Tendwire durable projection | -| Telegram presentation and delivery state | Herdres | -| Command idempotency and uncertain outcomes | Tendwire command receipts | - -An ACP `sessionId` is never a public worker identity. Tendwire must bind it to -the current private `WorkerBinding` generation and reject events after that -binding expires, moves, or is replaced. Replayed ACP events must deduplicate on -their producer identity without changing the public worker identity. - -## Canonical events - -The structured event journal accepts these semantic kinds: - -- user message -- agent message -- thought -- tool call -- tool call update -- plan -- usage -- session information -- private extension/control state, including available commands, current mode, - and session configuration updates - -Producer IDs, raw inputs, raw outputs, session IDs, terminal IDs, paths, and -reasoning are private. Public turn projection is deliberately narrower: -`user_text`, `assistant_stream_text`, `assistant_final_text`, completion state, -and existing safe metadata. Tool and plan presentation requires its own -sanitizing projection and must not reuse raw ACP payloads. - -## Thought policy - -`TENDWIRE_ACP_THOUGHT_POLICY` has three values: - -- `disabled`: discard thought chunks before persistence. -- `private_summary`: retain a chunk privately only when a trusted adapter sets - the exact update-level marker - `_meta["tendwire.dev/thought_kind"] = "summary"`; unclassified, unknown, - contradictory, and raw chunks are discarded. -- `private_all`: retain every thought chunk privately for explicit local - diagnostics. - -The default is `disabled`. Stable ACP v1 does not define a raw-versus-summary -classification. The `private_summary` marker is only a Tendwire adapter -convention and is not an ACP guarantee; enable it only for a trusted adapter. -No thought policy grants connector delivery. Herdres must never receive a raw -thought event. A future public summary feature requires a separate schema, -sanitizer, explicit operator opt-in, and tests that prove raw reasoning cannot -cross the boundary. - -## Upstream upgrade boundary - -Tendwire integrates with the stable ACP wire protocol, not an adapter's source -tree. Official adapters such as `codex-acp` and `claude-agent-acp` remain -separately installed executables and must be replaceable without vendoring, -rebasing, or resolving Tendwire source conflicts. - -The boundary has four rules: - -- negotiate protocol version and capabilities at every process start; -- never import adapter implementation modules or depend on their repository - layout, generated internal types, commits, or private event handlers; -- ignore unknown standard update variants conservatively and retain explicitly - namespaced extension metadata only on Tendwire's private side; -- verify adapter releases with black-box ACP compatibility fixtures before - promotion, while keeping the previously proven executable for rollback. - -The wire-process boundary is designed so an adapter upgrade does not require a -Tendwire rebase. The current initialization-only probe is not a promotion gate: -it does not authenticate, create/load a session, prompt, validate updates, -exercise permissions/cancellation, or pin an executable digest. Stateful -conformance fixtures and an immutable rollback manifest are still required. -Adapter promotion and rollback remain operator-managed. - -## Runtime lifecycle - -For ACP v1 stdio, the component that owns the adapter process also owns framing, -initialization, request correlation, stderr handling, cancellation, and bounded -shutdown. Tendwire must not claim an ACP worker healthy until initialization, -capability negotiation, session creation/load/resume, and private worker binding -all succeed. - -The stock daemon has a multi-worker runtime factory. It discovers endpoints -through Herdr's private `agent.acp_endpoint` method, validates the fixed stdio -attach shape, and supervises one runtime per worker generation. Endpoint -tickets are one-shot private values: Tendwire uses one only for its immediate -attach and never persists or publishes it. Reconnect always re-resolves Herdr -authority and mints a fresh endpoint. - -While attached, the coordinator uses non-mutating `agent.acp_status` checks -before every prompt and during reconciliation. The reported lifecycle must be -`acp_owned_attached` and its numeric generation must match the attached slot. -A mismatch or unavailable status retires the slot before any prompt frame is -written. Endpoint minting is never used as a status probe. - -In `acp_shadow` and `acp_preferred`, the legacy scheduler remains available only -for workers not currently owned by a healthy ACP slot. It rechecks this -exclusion after dequeue and immediately before a legacy read, preventing queued -legacy work from overwriting or duplicating the active ACP worker projection. - -Disconnect handling is conservative: - -1. Stop accepting events from the disconnected generation. -2. Persist stream health without publishing private adapter details. -3. In `acp_preferred`, allow the next legacy refresh to become authoritative. -4. Reinitialize and rebind before accepting ACP events again. -5. Reconcile replayed messages and tool calls by producer identity. - -Command acknowledgement occurs after the complete `session/prompt` request -frame is written, not after the agent finishes the turn. End-of-turn response -and update draining continue under runtime supervision. A failure after the -durable `send_started` transition is terminally uncertain and never falls back -to a second transport. - -## Retention - -`event_retention_days` also bounds raw structured ACP journal payloads. Due -automatic maintenance and explicit online cleanup replace expired payload rows -with compact identity tombstones in bounded batches. Candidate scanning reads -only bounded identity metadata and the existing payload digest; it does not load -the retired private payload into maintenance memory. A tombstone retains the -original sequence and a replay-contract fingerprint, allowing exact retries to -remain idempotent and conflicting reuse to fail closed without retaining -messages, thoughts, raw tool input/output, or other source payloads. - -Schema v26 tombstones also retain the original event `observed_at` as the only -authority time for a one-time repair when the matching owned turn projection -is provably absent. Exact replays never re-merge caller timestamp or content -into an existing live or superseded projection, so they cannot reorder final -connector delivery. Tombstones migrated from pre-v26 stores have no retained -authority time; they remain deduplication evidence but cannot repair a turn. - -Each tombstone has bounded per-event identity metadata, but tombstone count is -permanent and therefore grows with the number of distinct source events. -Tombstones are intentionally not deleted automatically: removing them would make -a late replay indistinguishable from a new event. Cleanup asks SQLite to scrub -deleted cells in modified pages, but WAL/checkpoint timing, filesystem snapshots, -and backups have independent operator-managed lifecycles. Logical retention is -not an immediate physical-erasure or cryptographic-erasure guarantee. - -## Cross-repository requirements - -Herdr provides a private `agent.acp_endpoint` launch/proxy surface containing -adapter identity/version, session-open mode, cwd, generation, and an explicitly -ACP-owned lifecycle. Tendwire accepts only the configured Herdr executable and -the fixed `agent acp-attach` argument shape; arbitrary executable, environment, -or argument injection is rejected. - -Herdres needs optional presentations for sanitized tool and plan progress. It -does not ingest ACP directly: it continues polling Tendwire's neutral outbox so -delivery retries, topic binding, rate limits, and Telegram state remain outside -the agent protocol. - -## Rollout gates - -Promotion remains blocked at the default `legacy` posture until a supported ACP -adapter is installed, target panes are explicitly registered through Herdr's -ACP-owned lifecycle, and the integration is exercised against those real -adapters. Rollout may then proceed `legacy` -> `acp_shadow` -> `acp_preferred`. -Use an isolated `acp_preferred` (or stricter `acp_required`) canary for real -adapter prompt validation; `acp_shadow` intentionally does not pretend to -compare equivalent executed traffic. The following -must pass before `acp_required` is considered: - +| Agent session, messages, tools, plans, usage, and permissions | ACP | +| Durable journal, public turn projection, privacy, and finality | Tendwire | +| Command idempotency and uncertain outcomes | Tendwire receipts | +| Connector retry and delivery state | Tendwire outbox | +| Telegram presentation | Herdres | + +Herdr is not a transcript or command-input transport. Tendwire consumes its +worker/pane lifecycle events, verifies the worker generation, and asks it to +mint a one-shot private ACP endpoint. A supported worker without an explicitly +Herdr-owned healthy ACP endpoint is unavailable and degrades ACP health. + +An ACP `sessionId`, endpoint ticket, adapter command, terminal ID, and pane ID +are private. Tendwire binds the session to the current private worker binding +and rejects updates or commands after that generation moves, expires, or is +replaced. + +## Runtime and protocol boundary + +The daemon supervises one ACP worker session for each eligible Herdr worker. +The supervisor owns reconciliation, generation fencing, reconnect, console +exchange, and command routing. The worker session owns initialize, session +open/load/resume, update draining, permission handling, cancellation, and +bounded shutdown. The bounded connection owns subprocess I/O, JSON-RPC request +correlation, framing limits, stderr limits, and backpressure. + +Tendwire uses the official `agent-client-protocol` Python package for generated +ACP schemas and validation. Tendwire retains its bounded stdio connection +because its hard frame-size, queue, write-deadline, shutdown, and privacy +requirements are stricter than the upstream convenience transport. Adapter +executables remain separately installed and replaceable; no adapter source tree +is imported or vendored. + +Before every prompt and during reconciliation, Tendwire verifies Herdr's +non-mutating ACP status for the exact worker generation. Reconnect always asks +Herdr for a fresh one-shot endpoint. A missing route fails before receipt +reservation; a failure after the durable `send_started` boundary is terminally +uncertain and is never retried through another transport. + +## Durable projection and privacy + +The structured journal accepts user and agent messages, thoughts, tool calls, +tool updates, plans, usage, session information, and private extension/control +updates. Producer identity and raw ACP payloads are retained only on the +private side. The public turn projection remains deliberately narrower and +continues to drive finality and connector eligibility. + +`TENDWIRE_ACP_THOUGHT_POLICY` controls private thought retention: + +- `disabled` discards thoughts before persistence. +- `private_summary` retains only updates carrying Tendwire's exact trusted + summary marker. +- `private_all` retains raw thought chunks for explicit local diagnostics. + +No thought policy grants public or outbox delivery. Raw reasoning, tool input, +tool output, session IDs, paths, terminal data, and adapter metadata must remain +outside every public API and connector payload. + +ACP permission requests use the durable pending-decision projection. Only a +sanitized title and numbered public choices are exposed. The private option ID, +tool call, arguments, ACP session, and metadata remain behind the boundary. +`answer_decision` is fenced to the exact worker binding and generation and is +accepted only after the full JSON-RPC response frame is written. + +## Retention and replay + +`event_retention_days` bounds private structured-event payloads. Cleanup +replaces expired payloads with compact identity tombstones so exact replays +remain idempotent and conflicting producer-identity reuse fails closed. +Tombstones preserve only bounded replay evidence and authority time; they are +not recoverable event content. + +Tendwire continues to own durable command receipts and its neutral connector +outbox. ACP adapter restarts therefore do not erase accepted-command evidence, +turn finality, acknowledgement state, or delivery retries. + +## Release and conformance gates + +An ACP-required release must pass: + +- initialization and official-schema validation against each supported + adapter; +- new/load/resume, prompt, steering, cancellation, and permission flows; +- generation fencing and reconnect with freshly minted Herdr endpoints; - no missing or duplicated user/final messages across adapter restarts; -- deterministic replay deduplication; -- correct open-to-final turn identity; -- tool lifecycle completion after cancellation and permission denial; -- plan replacement without stale entries; -- thought and raw tool payloads absent from every public API/outbox surface; -- fallback after adapter failure without regressing existing final delivery; -- exact worker continuity across Herdr pane moves and agent-session recreation. - -The ACP runtime implements prompt submission, cancellation, fail-closed -permission handling, per-worker coordination, reconnect, and receipt-backed -instruction routing, including durable interactive permission approval. ACP -remains non-default until supported adapters are installed, workers are -explicitly registered as ACP-owned, and the cross-repository rollout gates -above pass against real adapters. +- deterministic replay deduplication and exactly-once final projection; +- complete tool and plan lifecycle after cancellation or permission denial; +- absence of raw thoughts, tool payloads, session IDs, and terminal data from + public APIs and the outbox; +- durable receipt behavior at every pre-send and post-send failure boundary; +- exact worker continuity across Herdr pane moves and agent recreation. + +If a release fails these gates, roll back the release. Do not reintroduce a +runtime fallback mode. diff --git a/pyproject.toml b/pyproject.toml index 0950545..b959794 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,7 +21,9 @@ classifiers = [ "Programming Language :: Python :: 3.13", "Topic :: Software Development :: Libraries :: Application Frameworks", ] -dependencies = [] +dependencies = [ + "agent-client-protocol>=0.11,<0.12", +] [project.scripts] tendwire = "tendwire.cli:main" diff --git a/scripts/codex_session_reader_benchmark.py b/scripts/codex_session_reader_benchmark.py deleted file mode 100755 index 0b5dc18..0000000 --- a/scripts/codex_session_reader_benchmark.py +++ /dev/null @@ -1,464 +0,0 @@ -#!/usr/bin/env python3 -"""Deterministic synthetic benchmark for the private Codex session reader. - -Run from a source checkout with ``PYTHONPATH=src``. The benchmark creates a -private memory-backed 20,000-file Codex fixture, prints one compact aggregate -JSON object, and never reports generated session identities or paths. Timing -ceilings are broad documented-host evidence gates; bounded work is the stable -contract. -""" - -from __future__ import annotations - -import argparse -import json -import os -import platform -import re -import sys -import tempfile -from collections.abc import Mapping -from pathlib import Path -from time import perf_counter_ns -from typing import Any -from uuid import UUID - -from tendwire.backends import herdr_turns - -REPORT_SCHEMA_VERSION = 1 -FIXTURE_FILE_COUNT = 20_000 -FIXTURE_SPARSE_BYTES = 20 * 1024 * 1024 -FIXTURE_DATE = "2026-07-03" -TARGET_ORDINAL = 10_000 -COLD_LOOKUP_BUDGET_NS = 30_000_000_000 -WARM_LOOKUP_BUDGET_NS = 1_000_000_000 -COLD_PARSE_BUDGET_NS = 2_000_000_000 -INCREMENTAL_POLL_BUDGET_NS = 1_000_000_000 -UNCHANGED_POLL_BUDGET_NS = 1_000_000_000 -_UUID_PATTERN = re.compile( - r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", - re.ASCII, -) - - -class _ArgumentError(Exception): - pass - - -class _Parser(argparse.ArgumentParser): - def error(self, message: str) -> None: - raise _ArgumentError(message) - - -def _canonical_json(value: Any) -> str: - return json.dumps( - value, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - ) - - -def _event(kind: str, turn_id: str, **extra: Any) -> dict[str, Any]: - return { - "type": "event_msg", - "payload": {"type": kind, "turn_id": turn_id, **extra}, - } - - -def _message( - turn_id: str, - role: str, - text: str, - *, - phase: str | None = None, -) -> dict[str, Any]: - payload: dict[str, Any] = { - "type": "message", - "role": role, - "content": [{"type": "output_text", "text": text}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - } - if phase is not None: - payload["phase"] = phase - return {"type": "response_item", "payload": payload} - - -def _jsonl(*records: Mapping[str, Any]) -> bytes: - return b"".join( - _canonical_json(record).encode("utf-8") + b"\n" for record in records - ) - - -def _rollout_name(session_id: str) -> str: - return f"rollout-{FIXTURE_DATE}T00-00-00-{session_id}.jsonl" - - -def _create_private_file(path: Path) -> None: - descriptor = os.open( - path, - os.O_CREAT | os.O_EXCL | os.O_WRONLY | getattr(os, "O_CLOEXEC", 0), - 0o600, - ) - os.close(descriptor) - - -def _create_fixture(home: Path) -> tuple[Path, str, str, str, bytes, bytes]: - sessions = home / "sessions" - year = sessions / "2026" - month = year / "07" - day = month / "03" - for directory in (home, sessions, year, month, day): - directory.mkdir(mode=0o700) - - target_id = str(UUID(int=TARGET_ORDINAL)) - for ordinal in range(1, FIXTURE_FILE_COUNT + 1): - session_id = str(UUID(int=ordinal)) - _create_private_file(day / _rollout_name(session_id)) - - turn_id = "synthetic-benchmark-turn" - user_text = "synthetic benchmark prompt" - stream_text = "synthetic benchmark incremental output" - tail = _jsonl( - _event("task_started", turn_id), - _message(turn_id, "user", user_text), - ) - append = _jsonl( - _message(turn_id, "assistant", stream_text, phase="commentary") - ) - target = day / _rollout_name(target_id) - with target.open("r+b", buffering=0) as handle: - handle.seek(FIXTURE_SPARSE_BYTES) - handle.write(b"\n") - handle.write(tail) - return target, target_id, turn_id, user_text, stream_text, append - - -def _reset_codex_state() -> None: - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - herdr_turns._CODEX_SESSION_CACHE_LIVE_KEYS = None - herdr_turns._CODEX_SESSION_CACHE_BINDING_GENERATIONS = {} - herdr_turns._CODEX_SESSION_CACHE_BINDING_FINGERPRINTS = {} - - -def _timed(call: Any) -> tuple[Any, int]: - started = perf_counter_ns() - value = call() - return value, perf_counter_ns() - started - - -def _contains_private_value(value: Any, forbidden: tuple[str, ...]) -> bool: - if isinstance(value, Mapping): - return any( - _contains_private_value(key, forbidden) - or _contains_private_value(item, forbidden) - for key, item in value.items() - ) - if isinstance(value, (list, tuple)): - return any(_contains_private_value(item, forbidden) for item in value) - if not isinstance(value, str): - return False - return bool(_UUID_PATTERN.search(value)) or any( - private and private in value for private in forbidden - ) - - -def _command_text() -> str: - return "PYTHONPATH=src python3 scripts/codex_session_reader_benchmark.py --json" - - -def _benchmark() -> dict[str, Any]: - benchmark_started = perf_counter_ns() - load_average = tuple(round(value, 2) for value in os.getloadavg()) - temporary_path: Path | None = None - index_observations: list[int] = [] - read_observations: list[int] = [] - prior_home = os.environ.get("CODEX_HOME") - prior_index_observer = herdr_turns._CODEX_INDEX_BUILD_OBSERVER - prior_read_observer = herdr_turns._CODEX_ISOLATED_READ_OBSERVER - report: dict[str, Any] | None = None - forbidden: tuple[str, ...] = () - - try: - with tempfile.TemporaryDirectory( - prefix="tendwire-codex-reader-benchmark-", - dir="/dev/shm", - ) as raw_root: - temporary_path = Path(raw_root) - root_private = temporary_path.stat().st_mode & 0o777 == 0o700 - home = temporary_path / "codex-home" - ( - target, - target_id, - turn_id, - user_text, - stream_text, - append, - ) = _create_fixture(home) - forbidden = ( - raw_root, - os.fspath(home), - os.fspath(target), - target_id, - turn_id, - user_text, - stream_text, - target.name, - ) - logical_file_bytes = target.stat().st_size - - os.environ["CODEX_HOME"] = os.fspath(home) - _reset_codex_state() - herdr_turns._CODEX_INDEX_BUILD_OBSERVER = index_observations.append - herdr_turns._CODEX_ISOLATED_READ_OBSERVER = read_observations.append - - wildcard_result, wildcard_ns = _timed( - lambda: herdr_turns._find_codex_session_file("*") - ) - builds_after_wildcard = len(index_observations) - - cold_result, cold_lookup_ns = _timed( - lambda: herdr_turns._find_codex_session_file(target_id) - ) - builds_after_cold_lookup = len(index_observations) - warm_result, warm_lookup_ns = _timed( - lambda: herdr_turns._find_codex_session_file(target_id) - ) - builds_after_warm_lookup = len(index_observations) - - with herdr_turns._CODEX_PATH_CACHE_LOCK: - index = herdr_turns._CODEX_INDEX_GENERATION - if index is None: - raise RuntimeError("index_generation_missing") - indexed_sessions = len(index.entries) - retained_index_bytes = index.retained_bytes - index_overflowed = index.overflowed - generation_visited = index.visited - - cold_content, cold_parse_ns = _timed( - lambda: herdr_turns._read_codex_session_turn(target_id) - ) - with target.open("ab", buffering=0) as handle: - handle.write(append) - incremental_content, incremental_poll_ns = _timed( - lambda: herdr_turns._read_codex_session_turn(target_id) - ) - unchanged_content, unchanged_poll_ns = _timed( - lambda: herdr_turns._read_codex_session_turn(target_id) - ) - - if len(read_observations) != 3: - raise RuntimeError("read_observation_count_mismatch") - cold_parse_bytes, incremental_poll_bytes, unchanged_poll_bytes = ( - read_observations - ) - builds_after_all_reads = len(index_observations) - observed_index_visits = sum(index_observations) - - checks = { - "append_sized_second_poll": incremental_poll_bytes == len(append), - "cold_lookup_budget_met": cold_lookup_ns <= COLD_LOOKUP_BUDGET_NS, - "cold_parse_budget_met": cold_parse_ns <= COLD_PARSE_BUDGET_NS, - "cold_parse_bounded": cold_parse_bytes - <= herdr_turns._CODEX_RESYNC_INITIAL_BYTES, - "exact_resolution": cold_result == target.resolve() - and warm_result == target.resolve(), - "fixture_file_count_exact": indexed_sessions == FIXTURE_FILE_COUNT, - "incremental_content_observed": isinstance(incremental_content, Mapping) - and incremental_content.get("assistant_stream_text") == stream_text, - "incremental_poll_budget_met": incremental_poll_ns - <= INCREMENTAL_POLL_BUDGET_NS, - "index_build_bounded": observed_index_visits - <= herdr_turns._CODEX_INDEX_MAX_VISITS, - "index_generation_complete": not index_overflowed, - "one_index_build": builds_after_cold_lookup == 1 - and builds_after_all_reads == 1, - "private_temporary_directory": root_private, - "sparse_large_fixture": logical_file_bytes > FIXTURE_SPARSE_BYTES, - "unchanged_poll_budget_met": unchanged_poll_ns - <= UNCHANGED_POLL_BUDGET_NS, - "unchanged_poll_no_read": unchanged_content is None - and unchanged_poll_bytes == 0, - "warm_lookup_budget_met": warm_lookup_ns <= WARM_LOOKUP_BUDGET_NS, - "warm_lookup_no_walk": builds_after_warm_lookup - == builds_after_cold_lookup, - "wildcard_no_match": wildcard_result is None, - "wildcard_no_walk": builds_after_wildcard == 0, - "cold_content_observed": isinstance(cold_content, Mapping) - and cold_content.get("user_text") == user_text, - } - - report = { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "validating", - "command": _command_text(), - "parameters": { - "fixture_files": FIXTURE_FILE_COUNT, - "sparse_prefix_bytes": FIXTURE_SPARSE_BYTES, - }, - "environment": { - "architecture": platform.machine(), - "fixture_storage": "memory_backed_tmpfs", - "load_average_1m_5m_15m": list(load_average), - "logical_cpus": os.cpu_count(), - "operating_system": platform.system(), - "platform": platform.platform(), - "platform_release": platform.release(), - "python_version": platform.python_version(), - "source_checkout_pythonpath": "src", - "timer": "perf_counter_ns", - }, - "fixture": { - "file_count": FIXTURE_FILE_COUNT, - "logical_session_file_bytes_before_append": logical_file_bytes, - "tail_bytes": logical_file_bytes - FIXTURE_SPARSE_BYTES - 1, - "append_bytes": len(append), - }, - "latency_ns": { - "wildcard_probe": wildcard_ns, - "cold_lookup": { - "elapsed_ns": cold_lookup_ns, - "documented_host_budget_ns": COLD_LOOKUP_BUDGET_NS, - "documented_host_budget_met": checks[ - "cold_lookup_budget_met" - ], - }, - "warm_lookup": { - "elapsed_ns": warm_lookup_ns, - "documented_host_budget_ns": WARM_LOOKUP_BUDGET_NS, - "documented_host_budget_met": checks[ - "warm_lookup_budget_met" - ], - }, - "cold_parse": { - "elapsed_ns": cold_parse_ns, - "documented_host_budget_ns": COLD_PARSE_BUDGET_NS, - "documented_host_budget_met": checks[ - "cold_parse_budget_met" - ], - }, - "incremental_poll": { - "elapsed_ns": incremental_poll_ns, - "documented_host_budget_ns": INCREMENTAL_POLL_BUDGET_NS, - "documented_host_budget_met": checks[ - "incremental_poll_budget_met" - ], - }, - "unchanged_poll": { - "elapsed_ns": unchanged_poll_ns, - "documented_host_budget_ns": UNCHANGED_POLL_BUDGET_NS, - "documented_host_budget_met": checks[ - "unchanged_poll_budget_met" - ], - }, - }, - "bounded_work": { - "index_builds": builds_after_all_reads, - "filesystem_entries_visited": observed_index_visits, - "filesystem_entry_visit_bound": herdr_turns._CODEX_INDEX_MAX_VISITS, - "generation_entries_visited": generation_visited, - "indexed_sessions": indexed_sessions, - "retained_index_bytes": retained_index_bytes, - "retained_index_byte_bound": herdr_turns._CODEX_INDEX_MAX_BYTES, - "wildcard_index_builds": builds_after_wildcard, - "warm_lookup_additional_index_builds": builds_after_warm_lookup - - builds_after_cold_lookup, - "cold_parse_bytes_read": cold_parse_bytes, - "cold_parse_byte_bound": herdr_turns._CODEX_RESYNC_INITIAL_BYTES, - "incremental_poll_bytes_read": incremental_poll_bytes, - "incremental_append_bytes": len(append), - "unchanged_poll_bytes_read": unchanged_poll_bytes, - }, - "checks": checks, - } - finally: - herdr_turns._CODEX_INDEX_BUILD_OBSERVER = prior_index_observer - herdr_turns._CODEX_ISOLATED_READ_OBSERVER = prior_read_observer - _reset_codex_state() - if prior_home is None: - os.environ.pop("CODEX_HOME", None) - else: - os.environ["CODEX_HOME"] = prior_home - - if report is None: - raise RuntimeError("report_not_created") - report["checks"]["temporary_artifacts_removed"] = bool( - temporary_path is not None and not temporary_path.exists() - ) - if _contains_private_value(report, forbidden): - raise RuntimeError("privacy_scan_failed") - report["checks"]["privacy_scan_passed"] = True - failed = sorted( - name for name, passed in report["checks"].items() if passed is not True - ) - if failed: - raise RuntimeError("benchmark_invariants_failed") - report["ok"] = True - report["status"] = "completed" - report["wall_time_ns"] = perf_counter_ns() - benchmark_started - return report - - -def _parser() -> argparse.ArgumentParser: - parser = _Parser( - add_help=False, - description="Run the deterministic synthetic Codex session-reader benchmark.", - ) - parser.add_argument( - "--json", - action="store_true", - help="Emit the aggregate report as one compact JSON object.", - ) - return parser - - -def main() -> int: - try: - args = _parser().parse_args() - except _ArgumentError: - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "invalid_arguments", - } - ) - ) - return 2 - if not args.json: - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "invalid_arguments", - } - ) - ) - return 2 - try: - report = _benchmark() - except Exception as exc: - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "benchmark_failed", - "error_type": type(exc).__name__, - } - ) - ) - return 1 - print(_canonical_json(report)) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/turn_ingestion_benchmark.py b/scripts/turn_ingestion_benchmark.py deleted file mode 100755 index 3568a7d..0000000 --- a/scripts/turn_ingestion_benchmark.py +++ /dev/null @@ -1,1549 +0,0 @@ -#!/usr/bin/env python3 -"""Deterministic synthetic benchmark for background turn ingestion. - -Run from a source checkout with ``PYTHONPATH=src``. The benchmark creates only -private temporary fixtures, exercises a real Unix-domain socket daemon, and -prints one aggregate JSON object. Documented-host latency budgets are evidence -gates for this benchmark run, not generic service-level guarantees. -""" - -from __future__ import annotations - -import argparse -import json -import math -import os -import platform -import sqlite3 -import stat -import sys -import tempfile -import threading -import time -from collections.abc import Callable, Mapping -from pathlib import Path -from time import perf_counter_ns -from typing import Any - -from tendwire.backends.herdr_turns import TurnIngestionScheduler -from tendwire.config import Config -from tendwire.core.commands import ( - DISPOSITION_NO_RECEIPT, - STATUS_NOOP, - CommandEnvelope, - CommandRequest, -) -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.core.turns import recompute_pending_content_fingerprint -from tendwire.daemon import DaemonHooks, TendwireDaemon -from tendwire.daemon_api import ( - MAX_REQUEST_BYTES, - MAX_RESPONSE_BYTES, - DaemonAPIClient, -) -from tendwire.store import sqlite as store - -REPORT_SCHEMA_VERSION = 1 -FIXTURE_HOST = "synthetic-turn-benchmark-host" -FIXTURE_TIMESTAMP = "2026-07-01T00:00:00+00:00" -SCHEDULER_REFRESH_SECONDS = 2.0 -SCHEDULER_WORKERS = 4 -SCHEDULER_QUEUE_CAPACITY = 64 -API_REQUEST_WORKERS = 8 -API_ADMISSION_CAPACITY = 32 -LIST_BUDGET_NS = 350_000_000 -HEALTH_BUDGET_NS = 350_000_000 -COMMAND_BUDGET_NS = 250_000_000 -SHUTDOWN_BOUND_NS = 2_000_000_000 -POLL_SECONDS = 0.005 - - -def _canonical_json(value: Any) -> str: - return json.dumps( - value, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - ) - - -def _nearest_rank(samples: list[int], percentile: float) -> int: - if not samples: - raise ValueError("samples_required") - ordered = sorted(samples) - rank = max(1, math.ceil(percentile * len(ordered))) - return ordered[rank - 1] - - -def _metric( - samples: list[int], - *, - warmups: int, - response_bytes: list[int], - budget_ns: int, -) -> dict[str, Any]: - p95_ns = _nearest_rank(samples, 0.95) - return { - "samples": len(samples), - "warmups": warmups, - "min_ns": min(samples), - "p50_ns": _nearest_rank(samples, 0.50), - "p95_ns": p95_ns, - "max_ns": max(samples), - "response_bytes_max": max(response_bytes), - "documented_host_budget_ns": budget_ns, - "documented_host_budget_met": p95_ns <= budget_ns, - } - - -def _wait_until(predicate: Callable[[], bool], timeout_seconds: float, code: str) -> None: - deadline = time.monotonic() + timeout_seconds - while time.monotonic() < deadline: - if predicate(): - return - time.sleep(POLL_SECONDS) - if not predicate(): - raise RuntimeError(code) - - -def _thread_ids(prefixes: tuple[str, ...]) -> set[int]: - return { - int(thread.ident) - for thread in threading.enumerate() - if thread.ident is not None and thread.name.startswith(prefixes) - } - - -def _process_alive(process_id: int) -> bool: - try: - os.kill(process_id, 0) - except ProcessLookupError: - return False - except PermissionError: - return True - return True - - -def _marker_records(marker_dir: Path) -> list[dict[str, int]]: - records: list[dict[str, int]] = [] - for marker in marker_dir.glob("done-*.json"): - try: - value = json.loads(marker.read_text(encoding="utf-8")) - except (OSError, UnicodeError, json.JSONDecodeError): - continue - if not isinstance(value, Mapping): - continue - try: - ordinal = int(value["ordinal"]) - process_id = int(value["process_id"]) - started_ns = int(value["started_ns"]) - finished_ns = int(value["finished_ns"]) - except (KeyError, TypeError, ValueError): - continue - if ordinal < 0 or process_id <= 0 or finished_ns < started_ns: - continue - records.append( - { - "ordinal": ordinal, - "process_id": process_id, - "started_ns": started_ns, - "finished_ns": finished_ns, - } - ) - return records - - -def _active_markers(marker_dir: Path) -> list[Path]: - return list(marker_dir.glob("active-*.json")) - - -def _source_call_count(marker_dir: Path) -> int: - return len(_active_markers(marker_dir)) + len(_marker_records(marker_dir)) - - -def _interval_maximum(records: list[dict[str, int]]) -> int: - events: list[tuple[int, int]] = [] - for record in records: - events.append((record["started_ns"], 1)) - events.append((record["finished_ns"], -1)) - active = 0 - maximum = 0 - for _at, delta in sorted(events, key=lambda item: (item[0], item[1])): - active += delta - maximum = max(maximum, active) - return maximum - - -def _first_call_overlap_ns( - records: list[dict[str, int]], - blocked_workers: int, -) -> int: - first: list[dict[str, int]] = [] - for ordinal in range(blocked_workers): - matching = sorted( - (record for record in records if record["ordinal"] == ordinal), - key=lambda record: record["started_ns"], - ) - if not matching: - return 0 - first.append(matching[0]) - return max( - 0, - min(record["finished_ns"] for record in first) - - max(record["started_ns"] for record in first), - ) - - -def _write_adapter( - adapter_path: Path, - marker_dir: Path, - release_path: Path, - state_path: Path, -) -> None: - source = f'''#!/usr/bin/env python3 -import json -import os -import pathlib -import sys -import time - -marker_dir = pathlib.Path({str(marker_dir)!r}) -release_path = pathlib.Path({str(release_path)!r}) -state_path = pathlib.Path({str(state_path)!r}) -target = sys.argv[3] if len(sys.argv) > 3 else "" -try: - ordinal = int(target.rsplit("-", 1)[-1]) -except ValueError: - raise SystemExit(2) -process_id = os.getpid() -started_ns = time.monotonic_ns() -active = marker_dir / f"active-{{ordinal}}-{{process_id}}.json" -done = marker_dir / f"done-{{ordinal}}-{{process_id}}.json" -active.write_text(json.dumps({{"ordinal": ordinal, "process_id": process_id, "started_ns": started_ns}}, sort_keys=True), encoding="utf-8") -os.chmod(active, 0o600) -with open(release_path, "rb", buffering=0) as release: - if release.read(1) != b"R": - raise SystemExit(3) -finished_ns = time.monotonic_ns() -done.write_text(json.dumps({{"ordinal": ordinal, "process_id": process_id, "started_ns": started_ns, "finished_ns": finished_ns}}, sort_keys=True), encoding="utf-8") -os.chmod(done, 0o600) -try: - active.unlink() -except FileNotFoundError: - pass -turn = {{"available": True, "user_text": "generated request", "assistant_final_text": "generated response", "complete": True, "has_open_turn": False, "model": "synthetic"}} -try: - pending_state = state_path.read_text(encoding="utf-8").strip() -except OSError: - pending_state = "none" -if pending_state == "open": - turn["pending_decision"] = {{ - "id": "synthetic-private-decision", - "prompt": "generated pending prompt", - "options": [ - {{"id": "approve", "label": "Approve generated choice"}}, - {{"id": "reject", "label": "Reject generated choice"}}, - ], - }} -print(json.dumps({{"result": {{"turn": turn}}}}, sort_keys=True, separators=(",", ":"))) -''' - adapter_path.write_text(source, encoding="utf-8") - adapter_path.chmod(0o700) - - -def _fixture( - blocked_workers: int, -) -> tuple[list[Worker], list[WorkerBinding], dict[str, Any]]: - workers: list[Worker] = [] - bindings: list[WorkerBinding] = [] - for ordinal in range(blocked_workers): - worker = Worker( - id=f"worker-benchmark-{ordinal}", - name=f"Generated Worker {ordinal + 1}", - status="active", - ) - workers.append(worker) - bindings.append( - WorkerBinding( - host_id=FIXTURE_HOST, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value=f"synthetic-agent-{ordinal}", - turn_target_kind="pane_id", - turn_target_value=f"synthetic-pane-{ordinal}", - sendable=True, - reason=None, - observed_at=FIXTURE_TIMESTAMP, - private_fingerprint=f"synthetic-private-binding-{ordinal}", - ) - ) - content = { - "user_text": "generated request", - "assistant_final_text": "generated response", - "complete": True, - "has_open_turn": False, - "model": "synthetic", - } - return workers, bindings, content - - -def _revision_state(db_path: Path) -> dict[str, int]: - with sqlite3.connect(str(db_path)) as conn: - rows = int( - conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - ) - current = int( - conn.execute( - "SELECT COUNT(*) FROM turn_content_revisions WHERE is_current = 1" - ).fetchone()[0] - ) - duplicate_groups = int( - conn.execute( - """ - SELECT COUNT(*) FROM ( - SELECT host_id, turn_id, content_revision - FROM turn_content_revisions - GROUP BY host_id, turn_id, content_revision - HAVING COUNT(*) > 1 - ) - """ - ).fetchone()[0] - ) - return { - "rows": rows, - "current_rows": current, - "duplicate_groups": duplicate_groups, - } - - -def _pending_row_state(db_path: Path) -> dict[str, int]: - with sqlite3.connect(str(db_path)) as conn: - rows = int(conn.execute("SELECT COUNT(*) FROM backend_pending").fetchone()[0]) - open_rows = int( - conn.execute( - "SELECT COUNT(*) FROM backend_pending WHERE observation_state = 'open'" - ).fetchone()[0] - ) - duplicate_groups = int( - conn.execute( - """ - SELECT COUNT(*) FROM ( - SELECT host_id, worker_id - FROM backend_pending - GROUP BY host_id, worker_id - HAVING COUNT(*) > 1 - ) - """ - ).fetchone()[0] - ) - return { - "rows": rows, - "open_rows": open_rows, - "duplicate_groups": duplicate_groups, - } - - -def _outbox_rows(db_path: Path) -> list[tuple[Any, ...]]: - with sqlite3.connect(str(db_path)) as conn: - return conn.execute( - "SELECT * FROM connector_outbox ORDER BY id" - ).fetchall() - - -def _seed_store( - db_path: Path, - workers: list[Worker], - bindings: list[WorkerBinding], - content: Mapping[str, Any], -) -> None: - snapshot = Snapshot( - host_id=FIXTURE_HOST, - updated_at=FIXTURE_TIMESTAMP, - workers=workers, - backend_health=[ - BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - observed_at=FIXTURE_TIMESTAMP, - counts={"workers": len(workers)}, - ) - ], - ) - store.save_snapshot(db_path, snapshot) - if store.upsert_worker_bindings(db_path, bindings) != len(bindings): - raise RuntimeError("binding_seed_failed") - for ordinal, binding in enumerate(bindings): - applied = store.apply_turn_refresh( - db_path, - FIXTURE_HOST, - binding.worker_id, - { - **content, - "source_turn_id": f"synthetic-source-turn-{ordinal}", - }, - expected_binding=binding, - observed_at=FIXTURE_TIMESTAMP, - ) - if applied.updated != 1: - raise RuntimeError("revision_seed_failed") - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - FIXTURE_HOST, - "attention", - "synthetic-delivery-key", - "queued", - '{"generated":true}', - '{"opaque":"generated"}', - FIXTURE_TIMESTAMP, - FIXTURE_TIMESTAMP, - ), - ) - - -class _FixtureEventBackend: - def __init__( - self, - db_path: Path, - workers: list[Worker], - bindings: list[WorkerBinding], - content: Mapping[str, Any], - ) -> None: - self._db_path = db_path - self._workers = workers - self._bindings = bindings - self._content = content - self._callback: Callable[[], None] | None = None - self.started = False - self.stopped = False - self.callback_detached = False - self.flush_calls = 0 - self.committed_events = 0 - self.event_rows_after = 0 - self.callback_notifications = 0 - - def start(self, *, wait_for_reconcile: bool = True) -> None: - if not wait_for_reconcile: - raise RuntimeError("event_reconcile_not_requested") - _seed_store( - self._db_path, - self._workers, - self._bindings, - self._content, - ) - with sqlite3.connect(str(self._db_path)) as conn: - before = int( - conn.execute( - "SELECT COUNT(*) FROM events WHERE host_id = ?", - (FIXTURE_HOST,), - ).fetchone()[0] - ) - for ordinal in range(2): - store.append_event( - self._db_path, - FIXTURE_HOST, - "pane.output_matched", - { - "schema_version": 1, - "generated": True, - "ordinal": ordinal, - }, - observed_at=FIXTURE_TIMESTAMP, - ) - with sqlite3.connect(str(self._db_path)) as conn: - self.event_rows_after = int( - conn.execute( - "SELECT COUNT(*) FROM events WHERE host_id = ?", - (FIXTURE_HOST,), - ).fetchone()[0] - ) - self.committed_events = self.event_rows_after - before - if self.committed_events != 2: - raise RuntimeError("event_commit_count_mismatch") - self.started = True - - def set_turn_refresh_callback( - self, - callback: Callable[[], None] | None, - ) -> None: - self._callback = callback - if callback is None: - self.callback_detached = True - - def emit_committed_burst(self, count: int) -> None: - if not self.started or self.stopped or count <= 0: - raise RuntimeError("event_backend_not_ready") - with sqlite3.connect(str(self._db_path)) as conn: - event_rows = int( - conn.execute( - "SELECT COUNT(*) FROM events WHERE host_id = ?", - (FIXTURE_HOST,), - ).fetchone()[0] - ) - if count != self.committed_events or event_rows != self.event_rows_after: - raise RuntimeError("event_commit_count_mismatch") - callback = self._callback - if callback is None: - raise RuntimeError("event_callback_missing") - self.callback_notifications += 1 - callback() - - def flush(self) -> None: - self.flush_calls += 1 - - def stop(self) -> None: - self.stopped = True - - @property - def operational_status(self) -> Mapping[str, Any]: - return { - "status": "healthy", - "outcome": "healthy_non_empty", - "ready": self.started and not self.stopped, - "running": self.started and not self.stopped, - "reconcile_enabled": False, - "last_event_at": FIXTURE_TIMESTAMP if self.committed_events else None, - "last_snapshot_at": FIXTURE_TIMESTAMP, - "last_reconcile_at": FIXTURE_TIMESTAMP, - } - - -class _APIConcurrency: - def __init__(self, workers: int) -> None: - self.workers = workers - self.lock = threading.Lock() - self.release = threading.Event() - self.active = 0 - self.maximum = 0 - self.probe_entered = 0 - self.dispatches = 0 - self.method_dispatches: dict[str, int] = {} - - def wrap( - self, - dispatcher: Callable[[Any], Mapping[str, Any]], - ) -> Callable[[Any], Mapping[str, Any]]: - def instrumented(request: Any) -> Mapping[str, Any]: - probe = bool( - isinstance(request, Mapping) - and request.get("method") == "ping" - and isinstance(request.get("params"), Mapping) - and request["params"].get("concurrency_probe") is True - ) - with self.lock: - self.active += 1 - self.maximum = max(self.maximum, self.active) - self.dispatches += 1 - method = ( - str(request.get("method")) - if isinstance(request, Mapping) - and isinstance(request.get("method"), str) - else "invalid" - ) - self.method_dispatches[method] = self.method_dispatches.get(method, 0) + 1 - if probe: - self.probe_entered += 1 - if self.probe_entered == self.workers: - self.release.set() - try: - if probe and not self.release.wait(timeout=2.0): - raise RuntimeError("api_probe_barrier_timeout") - return dispatcher(request) - finally: - with self.lock: - self.active -= 1 - - return instrumented - - -def _run_api_probe( - socket_path: Path, - workers: int, -) -> tuple[int, bool]: - ready = threading.Barrier(workers + 1) - results: list[dict[str, Any]] = [] - failures: list[str] = [] - lock = threading.Lock() - - def request() -> None: - try: - ready.wait(timeout=2.0) - response = DaemonAPIClient( - socket_path, - timeout_seconds=3.0, - ).request("ping", {"concurrency_probe": True}) - with lock: - results.append(response) - except BaseException as exc: # noqa: BLE001 - with lock: - failures.append(type(exc).__name__) - - clients = [ - threading.Thread( - target=request, - name=f"tendwire-benchmark-api-probe-{index}", - ) - for index in range(workers) - ] - started = perf_counter_ns() - for client in clients: - client.start() - ready.wait(timeout=2.0) - for client in clients: - client.join(timeout=4.0) - elapsed = perf_counter_ns() - started - clean = all(not client.is_alive() for client in clients) - ok = ( - clean - and not failures - and len(results) == workers - and all(response.get("ok") is True for response in results) - ) - return elapsed, ok - - -def _validate_list(response: Mapping[str, Any], blocked_workers: int) -> None: - result = response.get("result") - turns = result.get("turns") if isinstance(result, Mapping) else None - if ( - response.get("ok") is not True - or not isinstance(result, Mapping) - or result.get("schema_version") != 2 - or not isinstance(turns, list) - or sum( - isinstance(turn, Mapping) - and isinstance(turn.get("content"), Mapping) - and bool(turn["content"].get("content_revision")) - for turn in turns - ) - != blocked_workers - ): - raise RuntimeError("turn_list_contract_failed") - - -def _validate_pending(response: Mapping[str, Any], _blocked_workers: int) -> None: - result = response.get("result") - interactions = ( - result.get("pending_interactions") if isinstance(result, Mapping) else None - ) - health = result.get("pending_health") if isinstance(result, Mapping) else None - counts = health.get("counts") if isinstance(health, Mapping) else None - fingerprint = ( - result.get("content_fingerprint") if isinstance(result, Mapping) else None - ) - if ( - response.get("ok") is not True - or not isinstance(result, Mapping) - or result.get("schema_version") != 1 - or not isinstance(interactions, list) - or not all(isinstance(item, Mapping) for item in interactions) - or not isinstance(health, Mapping) - or health.get("status") not in {"healthy", "degraded"} - or not isinstance(counts, Mapping) - or set(counts) != {"fresh", "stale", "total"} - or any( - isinstance(counts.get(key), bool) - or not isinstance(counts.get(key), int) - or counts[key] < 0 - for key in ("fresh", "stale", "total") - ) - or counts["total"] != counts["fresh"] + counts["stale"] - or not isinstance(fingerprint, str) - or len(fingerprint) != 24 - or any(character not in "0123456789abcdef" for character in fingerprint) - or recompute_pending_content_fingerprint(result) != fingerprint - ): - raise RuntimeError("pending_list_contract_failed") - - -def _wait_for_pending_count( - socket_path: Path, - *, - expected_count: int, - blocked_workers: int, - timeout_seconds: float, - code: str, -) -> tuple[dict[str, Any], int]: - deadline = time.monotonic() + timeout_seconds - polls = 0 - while True: - response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("pending.list") - polls += 1 - _validate_pending(response, blocked_workers) - result = response["result"] - if len(result["pending_interactions"]) == expected_count: - return response, polls - if time.monotonic() >= deadline: - raise RuntimeError(code) - time.sleep(POLL_SECONDS) - - -def _validate_health(response: Mapping[str, Any], blocked_workers: int) -> None: - result = response.get("result") - ingestion = result.get("turn_ingestion") if isinstance(result, Mapping) else None - if ( - response.get("ok") is not True - or not isinstance(ingestion, Mapping) - or int(ingestion.get("active") or 0) < blocked_workers - ): - raise RuntimeError("health_contract_failed") - - -def _validate_command(response: Mapping[str, Any], _blocked_workers: int) -> None: - result = response.get("result") - try: - envelope = ( - CommandEnvelope.from_dict(dict(result)) - if isinstance(result, Mapping) - else None - ) - except (TypeError, ValueError): - envelope = None - if ( - response.get("ok") is not True - or envelope is None - or envelope.ok is not True - or envelope.status != STATUS_NOOP - or envelope.dry_run is not True - or envelope.disposition != DISPOSITION_NO_RECEIPT - ): - raise RuntimeError("command_contract_failed") - - -def _measure_requests( - socket_path: Path, - marker_dir: Path, - scheduler: TurnIngestionScheduler, - *, - method: str, - params: Mapping[str, Any], - validator: Callable[[Mapping[str, Any], int], None], - blocked_workers: int, - warmups: int, - samples: int, - budget_ns: int, -) -> dict[str, Any]: - timings: list[int] = [] - response_bytes: list[int] = [] - for index in range(warmups + samples): - if ( - int(scheduler.operational_status().get("active") or 0) < blocked_workers - or len(_active_markers(marker_dir)) < blocked_workers - ): - raise RuntimeError("adapters_not_blocked_during_request") - started = perf_counter_ns() - response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request(method, params) - elapsed = perf_counter_ns() - started - validator(response, blocked_workers) - if ( - int(scheduler.operational_status().get("active") or 0) < blocked_workers - or len(_active_markers(marker_dir)) < blocked_workers - ): - raise RuntimeError("adapter_block_ended_during_request") - if index >= warmups: - timings.append(elapsed) - response_bytes.append(len(_canonical_json(response).encode("utf-8")) + 1) - return _metric( - timings, - warmups=warmups, - response_bytes=response_bytes, - budget_ns=budget_ns, - ) - - -def _privacy_scan(report: Mapping[str, Any], forbidden_values: list[str]) -> bool: - encoded = _canonical_json(report) - return all(not value or value not in encoded for value in forbidden_values) - - -def _contains_raw_error_field(value: Any) -> bool: - if isinstance(value, Mapping): - return any( - str(key) in {"error", "error_type", "errors"} - or _contains_raw_error_field(item) - for key, item in value.items() - ) - if isinstance(value, list): - return any(_contains_raw_error_field(item) for item in value) - return False - - -def _command_text(args: argparse.Namespace) -> str: - return ( - "PYTHONPATH=src python3 scripts/turn_ingestion_benchmark.py " - f"--workers {args.workers} --blocked-workers {args.blocked_workers} " - f"--blocked-seconds {args.blocked_seconds:g} --warmups {args.warmups} " - f"--samples {args.samples} --json" - ) - - -def _benchmark(args: argparse.Namespace) -> dict[str, Any]: - temporary_path: Path | None = None - forbidden_values: list[str] = [ - FIXTURE_HOST, - "generated request", - "generated response", - "synthetic-delivery-key", - '{"opaque":"generated"}', - "generated pending prompt", - "Approve generated choice", - "Reject generated choice", - "synthetic-private-decision", - ] - report: dict[str, Any] | None = None - with tempfile.TemporaryDirectory( - prefix="tendwire-turn-benchmark-", - dir="/dev/shm", - ) as raw_root: - root = Path(raw_root) - temporary_path = root - db_path = root / "benchmark.db" - socket_path = root / "benchmark.sock" - marker_dir = root / "adapter-markers" - release_path = root / "release-adapters" - adapter_path = root / "generated-herdr" - state_path = root / "pending-state" - marker_dir.mkdir(mode=0o700) - os.mkfifo(release_path, mode=0o600) - _write_adapter(adapter_path, marker_dir, release_path, state_path) - state_path.write_text("none", encoding="utf-8") - state_path.chmod(0o600) - forbidden_values.extend( - [ - str(root), - str(db_path), - str(socket_path), - str(marker_dir), - str(release_path), - str(adapter_path), - str(state_path), - ] - ) - workers, bindings, content = _fixture(args.blocked_workers) - forbidden_values.extend( - [ - worker.id - for worker in workers - ] - ) - forbidden_values.extend( - value - for binding in bindings - for value in ( - binding.target_value, - str(binding.turn_target_value or ""), - binding.private_fingerprint, - ) - ) - - config = Config( - host_id=FIXTURE_HOST, - herdr_bin=str(adapter_path), - data_dir=root, - db_path=db_path, - socket_path=socket_path, - herdr_timeout_seconds=max(60.0, float(args.blocked_seconds) + 30.0), - herdr_backend="socket", - reconcile_interval_seconds=0.0, - turn_refresh_interval_seconds=SCHEDULER_REFRESH_SECONDS, - turn_refresh_workers=SCHEDULER_WORKERS, - ) - event_backend = _FixtureEventBackend( - db_path, - workers, - bindings, - content, - ) - scheduler: TurnIngestionScheduler | None = None - - def scheduler_factory(current: Config) -> TurnIngestionScheduler: - nonlocal scheduler - scheduler = TurnIngestionScheduler(current) - return scheduler - - command_calls = 0 - command_lock = threading.Lock() - - def submit_command(_config: Config, payload: str) -> Mapping[str, Any]: - nonlocal command_calls - parsed = json.loads(payload) - if ( - not isinstance(parsed, dict) - or parsed.get("schema_version") != 1 - or parsed.get("action") != "noop" - or parsed.get("dry_run") is not True - ): - raise RuntimeError("synthetic_command_invalid") - request = CommandRequest.from_dict(parsed) - with command_lock: - command_calls += 1 - return CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_NOOP, - result={}, - ).to_dict() - - baseline_threads = _thread_ids( - ( - "tendwire-turn-", - "tendwire-daemon-api", - "tendwire-benchmark-", - ) - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - event_backend_factory=lambda _config, _stop: event_backend, - turn_scheduler_factory=scheduler_factory, - submit_command=submit_command, - ), - ) - server_thread: threading.Thread | None = None - shutdown_ns = 0 - api_probe_elapsed_ns = 0 - api_probe_ok = False - api_concurrency = _APIConcurrency(args.workers) - initial_revisions: dict[str, int] = {} - initial_outbox: list[tuple[Any, ...]] = [] - during_block_health: dict[str, Any] = {} - final_health: dict[str, Any] = {} - latency: dict[str, Any] = {} - source_calls_before_requests = 0 - source_calls_after_requests = 0 - blocked_observation_started = 0 - release_fd: int | None = None - production_handlers_measured = False - production_pending_handler_measured = False - turn_list_calls_during_pending_measurement = 0 - production_event_callback_bound = False - pending_source_calls_before_measurement = 0 - pending_source_calls_after_measurement = 0 - pending_rows_before_requests: dict[str, int] = {} - pending_rows_after_requests: dict[str, int] = {} - final_pending_rows: dict[str, int] = {} - independent_pending_polls = 0 - independent_turn_list_calls = 0 - independent_prompt_count = 0 - independent_clear_count = -1 - independent_discovery_fingerprint_changed = False - independent_unchanged_fingerprint_stable = False - independent_clear_fingerprint_changed = False - independent_clear_restored_baseline = False - try: - daemon.start() - if scheduler is None or daemon.server is None: - raise RuntimeError("daemon_components_missing") - original_dispatcher = daemon.server.dispatcher - daemon.server.dispatcher = api_concurrency.wrap(original_dispatcher) - api_dispatch = getattr(original_dispatcher, "__self__", None) - production_handlers_measured = bool( - getattr(api_dispatch, "_get_turns", None) == daemon.get_turns - and getattr(api_dispatch, "_get_health", None) == daemon.get_health - ) - production_pending_handler_measured = bool( - getattr(api_dispatch, "_get_pending", None) == daemon.get_pending - ) - production_event_callback_bound = ( - event_backend._callback == scheduler.request_refresh - ) - if ( - not production_handlers_measured - or not production_pending_handler_measured - or not production_event_callback_bound - ): - raise RuntimeError("production_handlers_not_bound") - server_thread = threading.Thread( - target=daemon.serve_forever, - name="tendwire-benchmark-daemon", - ) - server_thread.start() - _wait_until( - lambda: len(_active_markers(marker_dir)) >= args.blocked_workers, - 5.0, - "blocked_adapters_not_entered", - ) - blocked_observation_started = perf_counter_ns() - initial_revisions = _revision_state(db_path) - initial_outbox = _outbox_rows(db_path) - source_calls_before_requests = _source_call_count(marker_dir) - pending_rows_before_requests = _pending_row_state(db_path) - if source_calls_before_requests != args.blocked_workers: - raise RuntimeError("unexpected_initial_source_calls") - - api_probe_elapsed_ns, api_probe_ok = _run_api_probe( - socket_path, - args.workers, - ) - event_backend.emit_committed_burst(2) - _wait_until( - lambda: int(scheduler.operational_status().get("coalesced") or 0) - >= args.blocked_workers, - SCHEDULER_REFRESH_SECONDS + 2.0, - "scheduler_coalescing_not_observed", - ) - during_response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("health.get") - _validate_health(during_response, args.blocked_workers) - during_block_health = dict( - during_response["result"]["turn_ingestion"] - ) - - latency["turn_list"] = _measure_requests( - socket_path, - marker_dir, - scheduler, - method="turn.list", - params={ - "schema_version": 2, - "limit": 100, - "cursor": None, - "since": None, - }, - validator=_validate_list, - blocked_workers=args.blocked_workers, - warmups=args.warmups, - samples=args.samples, - budget_ns=LIST_BUDGET_NS, - ) - pending_source_calls_before_measurement = _source_call_count(marker_dir) - turn_list_calls_before_pending = api_concurrency.method_dispatches.get( - "turn.list", - 0, - ) - latency["pending_list"] = _measure_requests( - socket_path, - marker_dir, - scheduler, - method="pending.list", - params={}, - validator=_validate_pending, - blocked_workers=args.blocked_workers, - warmups=args.warmups, - samples=args.samples, - budget_ns=LIST_BUDGET_NS, - ) - pending_source_calls_after_measurement = _source_call_count(marker_dir) - if ( - pending_source_calls_after_measurement - != pending_source_calls_before_measurement - ): - raise RuntimeError("pending_list_started_source_reads") - turn_list_calls_during_pending_measurement = ( - api_concurrency.method_dispatches.get("turn.list", 0) - - turn_list_calls_before_pending - ) - latency["health_get"] = _measure_requests( - socket_path, - marker_dir, - scheduler, - method="health.get", - params={}, - validator=_validate_health, - blocked_workers=args.blocked_workers, - warmups=args.warmups, - samples=args.samples, - budget_ns=HEALTH_BUDGET_NS, - ) - latency["command_submit"] = _measure_requests( - socket_path, - marker_dir, - scheduler, - method="command.submit", - params={ - "schema_version": 1, - "action": "noop", - "dry_run": True, - }, - validator=_validate_command, - blocked_workers=args.blocked_workers, - warmups=args.warmups, - samples=args.samples, - budget_ns=COMMAND_BUDGET_NS, - ) - source_calls_after_requests = _source_call_count(marker_dir) - if source_calls_after_requests != source_calls_before_requests: - raise RuntimeError("request_path_started_source_reads") - pending_rows_after_requests = _pending_row_state(db_path) - - remaining_ns = int(args.blocked_seconds * 1_000_000_000) - ( - perf_counter_ns() - blocked_observation_started - ) - if remaining_ns > 0: - time.sleep(remaining_ns / 1_000_000_000) - if len(_active_markers(marker_dir)) < args.blocked_workers: - raise RuntimeError("configured_block_not_held") - release_fd = os.open(release_path, os.O_WRONLY | os.O_NONBLOCK) - os.write(release_fd, b"R" * max(64, args.blocked_workers * 4)) - _wait_until( - lambda: ( - _source_call_count(marker_dir) >= args.blocked_workers * 2 - and int(scheduler.operational_status().get("active") or 0) == 0 - and int(scheduler.operational_status().get("queue_depth") or 0) == 0 - ), - 3.0, - "scheduler_did_not_drain", - ) - turn_calls_before_independent = api_concurrency.method_dispatches.get( - "turn.list", - 0, - ) - baseline_pending_response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("pending.list") - independent_pending_polls += 1 - _validate_pending(baseline_pending_response, args.blocked_workers) - baseline_pending = baseline_pending_response["result"] - baseline_fingerprint = str(baseline_pending["content_fingerprint"]) - - state_path.write_text("open", encoding="utf-8") - discovery_source_calls = _source_call_count(marker_dir) - event_backend.emit_committed_burst(2) - _wait_until( - lambda: ( - _source_call_count(marker_dir) - >= discovery_source_calls + args.blocked_workers - and int(scheduler.operational_status().get("active") or 0) == 0 - and int(scheduler.operational_status().get("queue_depth") or 0) == 0 - ), - 3.0, - "independent_pending_discovery_did_not_drain", - ) - open_pending_response, discovery_polls = _wait_for_pending_count( - socket_path, - expected_count=args.blocked_workers, - blocked_workers=args.blocked_workers, - timeout_seconds=3.0, - code="independent_pending_not_discovered", - ) - independent_pending_polls += discovery_polls - open_pending = open_pending_response["result"] - independent_prompt_count = len(open_pending["pending_interactions"]) - open_fingerprint = str(open_pending["content_fingerprint"]) - independent_discovery_fingerprint_changed = ( - open_fingerprint != baseline_fingerprint - ) - - unchanged_pending_response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("pending.list") - independent_pending_polls += 1 - _validate_pending(unchanged_pending_response, args.blocked_workers) - unchanged_fingerprint = str( - unchanged_pending_response["result"]["content_fingerprint"] - ) - independent_unchanged_fingerprint_stable = ( - unchanged_fingerprint == open_fingerprint - ) - - state_path.write_text("none", encoding="utf-8") - clearing_source_calls = _source_call_count(marker_dir) - event_backend.emit_committed_burst(2) - _wait_until( - lambda: ( - _source_call_count(marker_dir) - >= clearing_source_calls + args.blocked_workers - and int(scheduler.operational_status().get("active") or 0) == 0 - and int(scheduler.operational_status().get("queue_depth") or 0) == 0 - ), - 3.0, - "independent_pending_clear_did_not_drain", - ) - cleared_pending_response, clear_polls = _wait_for_pending_count( - socket_path, - expected_count=0, - blocked_workers=args.blocked_workers, - timeout_seconds=3.0, - code="independent_pending_not_cleared", - ) - independent_pending_polls += clear_polls - cleared_pending = cleared_pending_response["result"] - independent_clear_count = len(cleared_pending["pending_interactions"]) - cleared_fingerprint = str(cleared_pending["content_fingerprint"]) - independent_clear_fingerprint_changed = ( - cleared_fingerprint != open_fingerprint - ) - independent_clear_restored_baseline = ( - cleared_fingerprint == baseline_fingerprint - ) - independent_turn_list_calls = ( - api_concurrency.method_dispatches.get("turn.list", 0) - - turn_calls_before_independent - ) - final_pending_rows = _pending_row_state(db_path) - - daemon.server.dispatcher = api_concurrency.wrap(original_dispatcher) - final_response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("health.get") - if final_response.get("ok") is not True: - raise RuntimeError("final_health_failed") - final_health = dict(final_response["result"]["turn_ingestion"]) - finally: - if release_fd is not None: - try: - os.close(release_fd) - except OSError: - pass - shutdown_started = perf_counter_ns() - daemon.stop() - if server_thread is not None: - server_thread.join(timeout=2.0) - shutdown_ns = perf_counter_ns() - shutdown_started - - if scheduler is None or server_thread is None: - raise RuntimeError("daemon_lifecycle_incomplete") - final_revisions = _revision_state(db_path) - final_outbox = _outbox_rows(db_path) - adapter_records = _marker_records(marker_dir) - process_ids = {record["process_id"] for record in adapter_records} - _wait_until( - lambda: all(not _process_alive(process_id) for process_id in process_ids), - 1.0, - "adapter_child_not_reaped", - ) - _wait_until( - lambda: not ( - _thread_ids( - ( - "tendwire-turn-", - "tendwire-daemon-api", - "tendwire-benchmark-", - ) - ) - - baseline_threads - ), - 2.0, - "benchmark_thread_not_reaped", - ) - remaining_threads = _thread_ids( - ( - "tendwire-turn-", - "tendwire-daemon-api", - "tendwire-benchmark-", - ) - ) - baseline_threads - source_calls_final = len(adapter_records) - forbidden_values.extend( - marker - for process_id in process_ids - for marker in ( - f'"process_id":{process_id}', - f'"process_id":"{process_id}"', - f'"pid":{process_id}', - f'"pid":"{process_id}"', - ) - ) - overlap_ns = _first_call_overlap_ns(adapter_records, args.blocked_workers) - response_bytes_max = max( - metric["response_bytes_max"] for metric in latency.values() - ) - scheduler_bounds = { - "refresh_interval_seconds": config.turn_refresh_interval_seconds, - "max_workers": config.turn_refresh_workers, - "queue_capacity": SCHEDULER_QUEUE_CAPACITY, - "adapter_timeout_seconds": config.herdr_timeout_seconds, - } - checks = { - "private_temporary_directory": stat.S_IMODE(root.stat().st_mode) == 0o700, - "private_marker_directory": stat.S_IMODE(marker_dir.stat().st_mode) == 0o700, - "private_adapter_executable": stat.S_IMODE(adapter_path.stat().st_mode) == 0o700, - "private_database_mode": stat.S_IMODE(db_path.stat().st_mode) & 0o077 == 0, - "real_unix_socket_removed": not os.path.lexists(socket_path), - "blocked_adapters_overlapped": overlap_ns >= int(args.blocked_seconds * 1_000_000_000), - "cached_requests_started_no_source_reads": source_calls_after_requests - == source_calls_before_requests, - "api_probe_completed": api_probe_ok, - "api_worker_bound_observed": api_concurrency.maximum == args.workers, - "production_list_health_handlers_measured": production_handlers_measured, - "production_pending_handler_measured": production_pending_handler_measured, - "production_event_callback_bound": production_event_callback_bound, - "pending_list_started_no_turn_reads": turn_list_calls_during_pending_measurement == 0, - "pending_list_started_no_source_reads": ( - pending_source_calls_after_measurement - == pending_source_calls_before_measurement - ), - "pending_list_store_rows_unchanged": ( - pending_rows_after_requests == pending_rows_before_requests - ), - "independent_pending_discovered": ( - independent_prompt_count == args.blocked_workers - ), - "independent_pending_cleared": independent_clear_count == 0, - "independent_pending_zero_turn_calls": independent_turn_list_calls == 0, - "independent_pending_discovery_fingerprint_changed": ( - independent_discovery_fingerprint_changed - ), - "independent_pending_unchanged_fingerprint_stable": ( - independent_unchanged_fingerprint_stable - ), - "independent_pending_clear_fingerprint_changed": ( - independent_clear_fingerprint_changed - ), - "independent_pending_clear_restored_baseline": ( - independent_clear_restored_baseline - ), - "no_duplicate_pending_rows": ( - final_pending_rows.get("duplicate_groups") == 0 - ), - "independent_pending_health_coherent": ( - open_pending["pending_health"] - == { - "status": "healthy", - "counts": { - "fresh": args.blocked_workers, - "stale": 0, - "total": args.blocked_workers, - }, - } - and cleared_pending["pending_health"] - == { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - } - ), - "independent_pending_rows_coherent_after_clear": ( - final_pending_rows.get("rows") == args.blocked_workers - and final_pending_rows.get("open_rows") == 0 - ), - "adapter_worker_bound_observed": _interval_maximum(adapter_records) - == args.blocked_workers, - "event_burst_committed_before_notification": event_backend.committed_events == 2 - and event_backend.callback_notifications == 3, - "scheduler_queue_drained": int(final_health.get("queue") or 0) == 0 - and int(final_health.get("active") or 0) == 0, - "scheduler_coalescing_observed": int(final_health.get("coalesced") or 0) - >= args.blocked_workers, - "scheduler_no_timeouts_or_queue_full": int( - final_health.get("timed_out") or 0 - ) - == 0 - and int(final_health.get("queue_full") or 0) == 0, - "revision_rows_unchanged": final_revisions == initial_revisions, - "no_duplicate_revisions": final_revisions.get("duplicate_groups") == 0, - "outbox_rows_unchanged": final_outbox == initial_outbox, - "expected_outbox_rows_preserved": ( - len(final_outbox) == args.blocked_workers + 1 - ), - "expected_command_calls": command_calls == args.warmups + args.samples, - "list_budget_met": bool(latency["turn_list"]["documented_host_budget_met"]), - "pending_list_budget_met": bool( - latency["pending_list"]["documented_host_budget_met"] - ), - "health_budget_met": bool(latency["health_get"]["documented_host_budget_met"]), - "command_budget_met": bool(latency["command_submit"]["documented_host_budget_met"]), - "shutdown_bounded": shutdown_ns <= SHUTDOWN_BOUND_NS, - "daemon_thread_reaped": not server_thread.is_alive(), - "adapter_children_reaped": all( - not _process_alive(process_id) for process_id in process_ids - ), - "benchmark_threads_reaped": not remaining_threads, - "event_callback_detached": event_backend.callback_detached, - "event_backend_stopped": event_backend.stopped, - } - report = { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "validating", - "command": _command_text(args), - "parameters": { - "api_probe_workers": args.workers, - "blocked_adapter_workers": args.blocked_workers, - "blocked_seconds": args.blocked_seconds, - "warmups_per_operation": args.warmups, - "samples_per_operation": args.samples, - }, - "environment": { - "python_version": platform.python_version(), - "sqlite_version": sqlite3.sqlite_version, - "operating_system": platform.system(), - "platform_release": platform.release(), - "platform": platform.platform(), - "architecture": platform.machine(), - "timer": "perf_counter_ns", - "percentiles": "nearest_rank", - "source_checkout_pythonpath": "src", - "fixture_storage": "memory_backed_tmpfs", - }, - "transport": { - "kind": "unix_stream_socket", - "request_workers": API_REQUEST_WORKERS, - "admission_capacity": API_ADMISSION_CAPACITY, - "request_frame_max_bytes": MAX_REQUEST_BYTES, - "response_frame_max_bytes": MAX_RESPONSE_BYTES, - "observed_max_api_concurrency": api_concurrency.maximum, - "probe_elapsed_ns": api_probe_elapsed_ns, - "dispatches": api_concurrency.dispatches, - "measured_response_bytes_max": response_bytes_max, - "method_dispatches": dict(sorted(api_concurrency.method_dispatches.items())), - "handler_mode": "production_store_backed", - }, - "ingestion": { - "scheduler_bounds": scheduler_bounds, - "source_calls_before_requests": source_calls_before_requests, - "source_calls_after_requests": source_calls_after_requests, - "source_calls_final": source_calls_final, - "turn_list_calls_during_pending_measurement": turn_list_calls_during_pending_measurement, - "pending_source_calls_before_measurement": pending_source_calls_before_measurement, - "pending_source_calls_after_measurement": pending_source_calls_after_measurement, - "independent_pending_polls": independent_pending_polls, - "independent_turn_list_calls": independent_turn_list_calls, - "independent_prompt_count": independent_prompt_count, - "independent_clear_count": independent_clear_count, - "observed_max_adapter_concurrency": _interval_maximum(adapter_records), - "first_call_overlap_ns": overlap_ns, - "event_committed_count": event_backend.committed_events, - "event_callback_notifications": event_backend.callback_notifications, - "during_block": { - "status": during_block_health.get("status"), - "queue": during_block_health.get("queue"), - "active": during_block_health.get("active"), - "refreshed": during_block_health.get("refreshed"), - "failed": during_block_health.get("failed"), - "timed_out": during_block_health.get("timed_out"), - "coalesced": during_block_health.get("coalesced"), - "queue_full": during_block_health.get("queue_full"), - }, - "final": { - "status": final_health.get("status"), - "queue": final_health.get("queue"), - "active": final_health.get("active"), - "refreshed": final_health.get("refreshed"), - "failed": final_health.get("failed"), - "timed_out": final_health.get("timed_out"), - "coalesced": final_health.get("coalesced"), - "queue_full": final_health.get("queue_full"), - }, - }, - "latency_ns": latency, - "store": { - "schema_version": store.STORE_SCHEMA_VERSION, - "event_rows_after": event_backend.event_rows_after, - "generated_event_rows": event_backend.committed_events, - "bindings": len(bindings), - "revision_rows_before": initial_revisions["rows"], - "revision_rows_after": final_revisions["rows"], - "current_revision_rows_after": final_revisions["current_rows"], - "duplicate_revision_groups_after": final_revisions["duplicate_groups"], - "pending_rows_before_requests": pending_rows_before_requests["rows"], - "pending_rows_after_requests": pending_rows_after_requests["rows"], - "pending_rows_after_independent_clear": final_pending_rows["rows"], - "pending_open_rows_after_independent_clear": final_pending_rows["open_rows"], - "duplicate_pending_groups_after": final_pending_rows["duplicate_groups"], - "outbox_rows_before": len(initial_outbox), - "outbox_rows_after": len(final_outbox), - }, - "cleanup": { - "shutdown_ns": shutdown_ns, - "shutdown_bound_ns": SHUTDOWN_BOUND_NS, - "adapter_child_count": len(process_ids), - "adapter_children_alive": sum( - _process_alive(process_id) for process_id in process_ids - ), - "benchmark_threads_alive": len(remaining_threads), - "socket_present_after_shutdown": os.path.lexists(socket_path), - "event_flush_calls": event_backend.flush_calls, - }, - "checks": checks, - } - - if report is None: - raise RuntimeError("report_not_created") - report["checks"]["temporary_artifacts_removed"] = bool( - temporary_path is not None and not temporary_path.exists() - ) - report["checks"]["raw_errors_absent"] = not _contains_raw_error_field(report) - if not _privacy_scan(report, forbidden_values): - raise RuntimeError("privacy_scan_failed") - report["checks"]["privacy_scan_passed"] = True - failed = sorted( - name - for name, passed in report["checks"].items() - if isinstance(passed, bool) and not passed - ) - if failed: - raise RuntimeError("benchmark_invariants_failed") - report["ok"] = True - report["status"] = "completed" - return report - - -def _parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - description="Run the deterministic synthetic turn-ingestion benchmark." - ) - parser.add_argument("--workers", type=int, default=8) - parser.add_argument("--blocked-workers", type=int, default=2) - parser.add_argument("--blocked-seconds", type=float, default=5.0) - parser.add_argument("--warmups", type=int, default=3) - parser.add_argument("--samples", type=int, default=21) - parser.add_argument( - "--json", - action="store_true", - help="Emit the aggregate report as one compact JSON object.", - ) - return parser - - -def main() -> int: - benchmark_started = perf_counter_ns() - args = _parser().parse_args() - if ( - not 1 <= args.workers <= API_REQUEST_WORKERS - or not 2 <= args.blocked_workers <= SCHEDULER_WORKERS - or not math.isfinite(args.blocked_seconds) - or args.blocked_seconds <= 0 - or args.warmups < 0 - or args.samples <= 0 - ): - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "invalid_arguments", - } - ) - ) - return 2 - try: - report = _benchmark(args) - report["wall_time_ns"] = perf_counter_ns() - benchmark_started - except Exception as exc: - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "benchmark_failed", - "error_type": type(exc).__name__, - } - ) - ) - return 1 - print(_canonical_json(report)) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py index 1521343..94a2ac1 100644 --- a/src/tendwire/backends/acp_client.py +++ b/src/tendwire/backends/acp_client.py @@ -24,6 +24,17 @@ from types import MappingProxyType from typing import Any, Callable, TypeVar +from acp.schema import ( + InitializeResponse as UpstreamInitializeResponse, + ListSessionsResponse as UpstreamListSessionsResponse, + LoadSessionResponse as UpstreamLoadSessionResponse, + NewSessionResponse as UpstreamNewSessionResponse, + PromptRequest as UpstreamPromptRequest, + PromptResponse as UpstreamPromptResponse, + ResumeSessionResponse as UpstreamResumeSessionResponse, +) +from pydantic import ValidationError + from tendwire import __version__ from .acp_protocol import ( @@ -134,8 +145,8 @@ class _PendingRequest: SessionEvent = SessionUpdate | PermissionRequest -class AcpClient: - """Thread-safe, blocking ACP v1 client for one agent subprocess.""" +class BoundedAcpConnection: + """Thread-safe ACP connection with bounded subprocess stdio framing.""" def __init__( self, @@ -216,7 +227,7 @@ def __init__( self._exit: ProcessExit | None = None self._initialize_result: InitializeResult | None = None - def __enter__(self) -> "AcpClient": + def __enter__(self) -> "BoundedAcpConnection": self.start() return self @@ -270,7 +281,7 @@ def stderr_tail(self) -> str: data = b"".join(self._stderr_chunks) return data.decode("utf-8", errors="replace") - def start(self) -> "AcpClient": + def start(self) -> "BoundedAcpConnection": with self._state_lock: if self._state in {ClientState.RUNNING, ClientState.INITIALIZED}: return self @@ -361,6 +372,11 @@ def initialize( require_initialized=False, ) raw = _require_mapping(result, "initialize result") + _validate_upstream( + UpstreamInitializeResponse, + raw, + "initialize result", + ) version = raw.get("protocolVersion") if ( not isinstance(version, int) @@ -500,6 +516,7 @@ def new_session( ) result = self.request("session/new", params, timeout=timeout) raw = _require_mapping(result, "session/new result") + _validate_upstream(UpstreamNewSessionResponse, raw, "session/new result") return _parse_session_result(raw, require_session_id=True) def load_session( @@ -530,6 +547,7 @@ def load_session( MappingProxyType({}), ) raw = _require_mapping(result, "session/load result") + _validate_upstream(UpstreamLoadSessionResponse, raw, "session/load result") parsed = _parse_session_result(raw, require_session_id=False) return SessionResult(session_id, parsed.modes, parsed.config_options, parsed.raw) @@ -551,6 +569,7 @@ def resume_session( params["sessionId"] = _nonempty(session_id, "session_id") result = self.request("session/resume", params, timeout=timeout) raw = _require_mapping(result, "session/resume result") + _validate_upstream(UpstreamResumeSessionResponse, raw, "session/resume result") parsed = _parse_session_result(raw, require_session_id=False) return SessionResult(session_id, parsed.modes, parsed.config_options, parsed.raw) @@ -569,6 +588,7 @@ def list_sessions( params["cursor"] = _nonempty(cursor, "cursor") result = self.request("session/list", params, timeout=timeout) raw = _require_mapping(result, "session/list result") + _validate_upstream(UpstreamListSessionsResponse, raw, "session/list result") raw_sessions = raw.get("sessions") if not isinstance(raw_sessions, list): raise AcpEnvelopeError("session/list result.sessions must be an array") @@ -621,6 +641,11 @@ def prompt( self._active_prompts[session_id] = self._active_prompts.get(session_id, 0) + 1 response_received = False try: + _validate_upstream( + UpstreamPromptRequest, + {"sessionId": session_id, "prompt": content}, + "session/prompt params", + ) result = self.request( "session/prompt", {"sessionId": session_id, "prompt": content}, @@ -639,6 +664,7 @@ def prompt( if response_received: self._cancelled_sessions.discard(session_id) raw = _require_mapping(result, "session/prompt result") + _validate_upstream(UpstreamPromptResponse, raw, "session/prompt result") stop_reason = raw.get("stopReason") try: parsed_reason = StopReason(stop_reason) @@ -1601,6 +1627,17 @@ def _require_mapping(value: Any, name: str) -> Mapping[str, Any]: return value +def _validate_upstream(model: Any, value: Mapping[str, Any], name: str) -> None: + """Validate one stable ACP payload with the official generated schema.""" + + try: + model.model_validate(dict(value)) + except ValidationError as exc: + raise AcpEnvelopeError( + f"{name} does not match the upstream ACP schema" + ) from exc + + def _parse_session_result( raw: Mapping[str, Any], *, require_session_id: bool ) -> SessionResult: diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 05d40d0..9168481 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -32,10 +32,10 @@ record_agent_event, upsert_worker_bindings, ) -from .acp_client import AcpClient +from .acp_client import BoundedAcpConnection from .acp_permissions import AcpPermissionBroker from .acp_runtime import ( - AcpRuntime, + AcpWorkerSession, PermissionCallback, RuntimeState, SessionOpenMode, @@ -100,10 +100,10 @@ class HerdrAcpConsoleEndpoint: @dataclass(slots=True) -class _RuntimeSlot: +class _SessionSlot: continuity: WorkerBinding generation: str - runtime: AcpRuntime + runtime: AcpWorkerSession permission_broker: AcpPermissionBroker | None = None console: HerdrAcpConsoleEndpoint | None = None console_input_sequence: int = 0 @@ -121,9 +121,9 @@ class _RuntimeSlot: class _PromptRoute: def __init__( self, - owner: "AcpRuntimeCoordinator", + owner: "AcpSupervisor", worker: Worker, - slot: _RuntimeSlot, + slot: _SessionSlot, ) -> None: self._owner = owner self._worker = worker @@ -196,12 +196,12 @@ def prepare(self): EndpointClientFactory = Callable[[Config], Any] -RuntimeFactory = Callable[..., AcpRuntime] -ClientFactory = Callable[..., AcpClient] +WorkerSessionFactory = Callable[..., AcpWorkerSession] +ConnectionFactory = Callable[..., BoundedAcpConnection] -class AcpRuntimeCoordinator: - """Reconcile Herdr worker authority into per-generation ACP runtimes.""" +class AcpSupervisor: + """Reconcile Herdr endpoint ownership into per-worker ACP sessions.""" def __init__( self, @@ -209,8 +209,8 @@ def __init__( stop_event: threading.Event, *, endpoint_client_factory: EndpointClientFactory | None = None, - runtime_factory: RuntimeFactory = AcpRuntime, - client_factory: ClientFactory = AcpClient, + session_factory: WorkerSessionFactory = AcpWorkerSession, + connection_factory: ConnectionFactory = BoundedAcpConnection, reconcile_interval: float | None = None, permission_callback: PermissionCallback | None = None, require_permission_bridge: bool = False, @@ -227,15 +227,15 @@ def __init__( self._endpoint_client_factory = ( endpoint_client_factory or _default_endpoint_client_factory ) - self._runtime_factory = runtime_factory - self._client_factory = client_factory + self._session_factory = session_factory + self._connection_factory = connection_factory self._permission_callback = permission_callback self._require_permission_bridge = bool(require_permission_bridge) self._durable_permission_bridge = bool(durable_permission_bridge) self._reconcile_interval = max( 1.0, float( - config.turn_refresh_interval_seconds + config.reconcile_interval_seconds if reconcile_interval is None else reconcile_interval ), @@ -247,8 +247,8 @@ def __init__( # after the fact by selecting whichever runtime happened to attach. self._reconcile_lock = threading.RLock() self._stop = threading.Event() - self._slots: dict[str, _RuntimeSlot] = {} - self._retired_slots: list[_RuntimeSlot] = [] + self._slots: dict[str, _SessionSlot] = {} + self._retired_slots: list[_SessionSlot] = [] self._thread: threading.Thread | None = None self._console_thread: threading.Thread | None = None self._state = RuntimeState.NEW @@ -258,20 +258,11 @@ def __init__( self._console_failure_type: str | None = None self._console_failed_workers: set[str] = set() self._console_failed_claims: dict[str, str] = {} - # Exact ACP ownership survives runtime retirement. Preferred mode may - # use legacy PTY I/O only after Herdr positively stops publishing this - # exact worker identity, never merely because reminting failed. + # Exact ACP ownership survives runtime retirement so an outage remains + # distinguishable from a worker that Herdr has actually removed. self._published_acp_claims: dict[str, str] = {} - # Optional ACP policies discover workers from the same Herdr binding - # stream as legacy PTY agents. Remember an exact worker generation - # that positively reported it is not ACP-owned so the periodic pass - # does not issue a mutating endpoint-mint request every interval. - # Cached workers are checked with the non-ticketing status method, so - # a later ACP registration becomes attachable immediately without - # relying on mutable observation fingerprints. - self._optional_endpoint_absences: dict[str, str] = {} - - def start(self) -> "AcpRuntimeCoordinator": + + def start(self) -> "AcpSupervisor": with self._lock: if self._state is RuntimeState.RUNNING: return self @@ -293,7 +284,7 @@ def start(self) -> "AcpRuntimeCoordinator": # Revoke any process-owned rows left by an unclean prior exit before # a fresh Herdr generation is allowed to attach. self._expire_orphaned_bindings() - self._reconcile(strict=self.config.agent_event_source == "acp_required") + self._reconcile(strict=True) except Exception as exc: with self._lock: self._state = RuntimeState.FAILED @@ -481,34 +472,6 @@ def prompt_route(self, worker: Worker) -> _PromptRoute | None: return None return _PromptRoute(self, worker, slot) - def owns_worker(self, worker_id: str, worker_fingerprint: str) -> bool: - """Return whether a healthy ACP slot currently owns this exact worker.""" - with self._lock: - slot = self._slots.get(worker_id) - return bool( - slot is not None - and slot.continuity.worker_fingerprint == worker_fingerprint - and slot.runtime.status().healthy - ) - - def claims_worker(self, worker_id: str, worker_fingerprint: str) -> bool: - """Return whether ACP has published authority for this exact worker. - - Unlike ``owns_worker``, this remains true across a console/runtime - outage so preferred mode cannot fall through to legacy pane I/O. - """ - - with self._lock: - slot = self._slots.get(worker_id) - return bool( - ( - slot is not None - and slot.continuity.worker_fingerprint == worker_fingerprint - ) - or self._console_failed_claims.get(worker_id) == worker_fingerprint - or self._published_acp_claims.get(worker_id) == worker_fingerprint - ) - def owns_permission_decision(self, decision: Any) -> bool: """Return whether one pending decision belongs to an exact live slot.""" worker_id = str(getattr(decision, "worker_id", "") or "") @@ -538,7 +501,7 @@ def answer_permission_decision(self, decision: Any, *, timeout: float) -> None: # writing the complete JSON-RPC response frame. slot.permission_broker.answer(decision, timeout=timeout) - def _current_slot(self, worker: Worker) -> _RuntimeSlot: + def _current_slot(self, worker: Worker) -> _SessionSlot: with self._lock: if self._state is not RuntimeState.RUNNING: raise AcpCoordinatorError("ACP coordinator is not running") @@ -602,7 +565,7 @@ def _bridge_console_slots(self) -> None: slot.console_bridge_thread = thread thread.start() - def _bridge_console_slot_supervised(self, slot: _RuntimeSlot) -> None: + def _bridge_console_slot_supervised(self, slot: _SessionSlot) -> None: worker_id = slot.continuity.worker_id try: self._bridge_console_slot(slot) @@ -660,7 +623,7 @@ def _bridge_console_slot_supervised(self, slot: _RuntimeSlot) -> None: # replacement slot completes a successful console pass. pass - def _bridge_console_slot(self, slot: _RuntimeSlot) -> None: + def _bridge_console_slot(self, slot: _SessionSlot) -> None: with slot.lock: if slot.retired: return @@ -968,7 +931,7 @@ def _bridge_console_slot(self, slot: _RuntimeSlot) -> None: ) def _submit_console_input( - self, slot: _RuntimeSlot, sequence: int, text: str + self, slot: _SessionSlot, sequence: int, text: str ) -> str: with self._reconcile_lock: self._require_reconcile_state(allow_starting=False) @@ -981,7 +944,7 @@ def _submit_console_input( return self._submit_console_input_fenced(slot, sequence, text) def _submit_console_input_fenced( - self, slot: _RuntimeSlot, sequence: int, text: str + self, slot: _SessionSlot, sequence: int, text: str ) -> str: snapshot = latest_snapshot(Path(self.config.db_path), self.config.host_id) worker = next( @@ -1070,9 +1033,6 @@ def _submit_console_input_fenced( self.config, json.dumps(request, sort_keys=True, separators=(",", ":")), acp_prompt_router=self.prompt_route, - acp_worker_owner=self.claims_worker, - acp_required=True, - acp_observation_only=False, acp_permission_router=self, ) except Exception: @@ -1157,10 +1117,6 @@ def _reconcile_locked(self, *, strict: bool) -> None: with self._lock: failed_claims = tuple(self._console_failed_claims.items()) published_claims = tuple(self._published_acp_claims.items()) - for worker_id in tuple(self._optional_endpoint_absences): - binding = current.get(worker_id) - if binding is None: - self._optional_endpoint_absences.pop(worker_id, None) exact_authorities = ( self._herdr_authority_claims() if failed_claims or published_claims @@ -1200,50 +1156,13 @@ def _reconcile_locked(self, *, strict: bool) -> None: ] for worker_id, continuity in current.items(): self._require_reconcile_state(allow_starting=True) - with self._lock: - existing = self._slots.get(worker_id) - optional_absence = worker_id in self._optional_endpoint_absences - if existing is None and optional_absence: - try: - status = self._resolve_status(continuity) - except Exception as exc: # noqa: BLE001 - if _optional_acp_absence(exc): - with self._lock: - self._optional_endpoint_absences[worker_id] = ( - continuity.worker_fingerprint - ) - continue - failures.append(exc) - continue - if status.lifecycle != "acp_owned_ready": - failures.append( - AcpCoordinatorError( - "ACP worker is attached without a local runtime" - ) - ) - continue - with self._lock: - self._optional_endpoint_absences.pop(worker_id, None) try: self._reconcile_binding(continuity) except Exception as exc: # noqa: BLE001 - if ( - existing is None - and self.config.agent_event_source - in {"acp_shadow", "acp_preferred"} - and _optional_acp_absence(exc) - ): - with self._lock: - self._optional_endpoint_absences[worker_id] = ( - continuity.worker_fingerprint - ) - continue failures.append(exc) self._retire_worker(worker_id) with self._lock: - self._required_degraded = bool(failures) and ( - self.config.agent_event_source == "acp_required" - ) + self._required_degraded = bool(failures) if failures: self._failure_type = type(failures[0]).__name__ elif not self._required_degraded: @@ -1272,8 +1191,8 @@ def _reconcile_worker(self, worker_id: str, *, strict: bool) -> None: claimed_fingerprint, ) in self._herdr_authority_claims() except Exception: - # A failed ownership check cannot safely reopen PTY - # fallback; the periodic reconcile can retry it. + # A failed ownership check cannot prove that the + # Herdr endpoint disappeared; retry periodically. authority_remains = True with self._lock: if worker_id not in self._slots and not authority_remains: @@ -1355,7 +1274,7 @@ def _reconcile_binding(self, continuity: WorkerBinding) -> None: runtime_binding.turn_target_value, endpoint.console.generation, ) - slot = _RuntimeSlot( + slot = _SessionSlot( continuity, endpoint.generation, runtime, @@ -1412,7 +1331,7 @@ def _resolve_status(self, continuity: WorkerBinding) -> HerdrAcpStatus: close() return _parse_status(continuity, result) - def _require_attached_generation(self, slot: _RuntimeSlot) -> None: + def _require_attached_generation(self, slot: _SessionSlot) -> None: try: status = self._resolve_status(slot.continuity) except Exception: @@ -1429,7 +1348,7 @@ def _require_attached_generation(self, slot: _RuntimeSlot) -> None: def _submit_prompt( self, worker: Worker, - slot: _RuntimeSlot, + slot: _SessionSlot, text: str, *, producer_turn_id: str, @@ -1459,7 +1378,7 @@ def _submit_prompt( on_send_start=on_send_start, ) - def _supports_steering(self, worker: Worker, slot: _RuntimeSlot) -> bool: + def _supports_steering(self, worker: Worker, slot: _SessionSlot) -> bool: try: return self._current_slot(worker) is slot and slot.runtime.can_steer() except Exception: @@ -1468,7 +1387,7 @@ def _supports_steering(self, worker: Worker, slot: _RuntimeSlot) -> bool: def _submit_steering( self, worker: Worker, - slot: _RuntimeSlot, + slot: _SessionSlot, text: str, *, producer_turn_id: str, @@ -1497,7 +1416,7 @@ def _submit_steering( def _route_binding_fingerprint( self, worker: Worker, - slot: _RuntimeSlot, + slot: _SessionSlot, ) -> str: """Return authority only while this exact route remains current.""" @@ -1516,8 +1435,8 @@ def _build_runtime( self, continuity: WorkerBinding, endpoint: HerdrAcpEndpoint, - ) -> tuple[AcpRuntime, AcpPermissionBroker | None]: - client = self._client_factory( + ) -> tuple[AcpWorkerSession, AcpPermissionBroker | None]: + client = self._connection_factory( endpoint.command, cwd=endpoint.cwd, request_timeout=self.config.acp_request_timeout_seconds, @@ -1545,7 +1464,7 @@ def _build_runtime( else None ) try: - runtime = self._runtime_factory( + runtime = self._session_factory( client, config=self.config, binding=binding, @@ -1554,7 +1473,7 @@ def _build_runtime( session_id=endpoint.session_id, # Herdr's generation authenticates the worker lease and can # remain stable across several freshly minted adapter - # transports. AcpRuntime deliberately creates a new stream + # transports. AcpWorkerSession deliberately creates a new stream # nonce when this argument is omitted; reusing the Herdr # generation would make synthetic notification identities # collide after a Tendwire restart. @@ -1592,7 +1511,7 @@ def _retire_worker( self, worker_id: str, *, - expected: _RuntimeSlot | None = None, + expected: _SessionSlot | None = None, preserve_console_failure: bool = False, ) -> None: with self._reconcile_lock: @@ -1604,7 +1523,7 @@ def _retire_worker( self._retired_slots.append(slot) # A visible-console failure is an exact sticky ownership # claim. Retirement, ambiguity, and failed reminting must not - # reopen legacy PTY fallback while Herdr still publishes that + # erase endpoint ownership while Herdr still publishes that # identity. Only a successful current console pass or the # positive-disappearance path in reconciliation may remove it. if ( @@ -1624,7 +1543,9 @@ def _retire_worker( executor.shutdown(wait=False, cancel_futures=True) self._stop_runtime(slot.runtime) - def _stop_runtime(self, runtime: AcpRuntime, *, timeout: float | None = None) -> None: + def _stop_runtime( + self, runtime: AcpWorkerSession, *, timeout: float | None = None + ) -> None: binding = getattr(runtime, "_binding", None) try: runtime.stop( @@ -1684,7 +1605,7 @@ def _stop_all(self, *, timeout: float | None = None) -> None: ) -def _slot_has_live_work(slot: _RuntimeSlot) -> bool: +def _slot_has_live_work(slot: _SessionSlot) -> bool: with slot.lock: thread = slot.console_bridge_thread futures = tuple((slot.console_submissions or {}).values()) @@ -1761,18 +1682,6 @@ def _same_continuity(left: WorkerBinding, right: WorkerBinding) -> bool: ) -def _optional_acp_absence(exc: BaseException) -> bool: - """Recognize a positive, generation-scoped legacy/non-ACP classification.""" - - if not isinstance(exc, HerdrErrorResponse) or not isinstance(exc.error, Mapping): - return False - return exc.error.get("code") in { - "acp_worker_unauthenticated", - "acp_ownership_required", - "acp_adapter_unsupported", - } - - def _nonempty_text(value: Any, field: str) -> str: if not isinstance(value, str) or not value or value.strip() != value: raise AcpCoordinatorError(f"Herdr ACP endpoint {field} is invalid") @@ -2491,12 +2400,12 @@ def _console_permission_selection( return matches[0] if len(matches) == 1 else None -def production_acp_runtime_factory( +def production_acp_supervisor_factory( config: Config, stop_event: threading.Event, -) -> AcpRuntimeCoordinator: - """Build the stock daemon's Herdr-backed multi-worker ACP coordinator.""" - return AcpRuntimeCoordinator( +) -> AcpSupervisor: + """Build the stock daemon's Herdr-backed ACP session supervisor.""" + return AcpSupervisor( config, stop_event, require_permission_bridge=True, diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py index 7dbd1ab..a8ff07a 100644 --- a/src/tendwire/backends/acp_ingestion.py +++ b/src/tendwire/backends/acp_ingestion.py @@ -14,7 +14,7 @@ from pathlib import Path from typing import Any -from ..config import Config +from ..config import DEFAULT_TURN_MODEL, Config from ..core.agent_events import AgentEvent, agent_event from ..core.models import WorkerBinding, stable_fingerprint from ..store.sqlite import ( @@ -74,8 +74,6 @@ def __init__( or binding.turn_target_value != session_id.strip() ): raise ValueError("ACP session does not match the private worker binding") - if config.agent_event_source == "legacy": - raise ValueError("ACP ingestion is disabled by agent_event_source=legacy") self.config = config self.session_id = session_id.strip() self.stream_generation = stream_generation.strip() @@ -417,13 +415,9 @@ def mark_prompt_complete( self.config.host_id, marker, expected_binding=self.binding, - content=( - None - if self.config.agent_event_source == "acp_shadow" - else content - ), + content=content, observed_at=marker.observed_at, - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ) except BaseException: self._restore_speculation(checkpoint, prior_turn_state) @@ -502,7 +496,6 @@ def _accept( projection: Mapping[str, Any] | None = None if ( kind in {"user_message", "agent_message"} - and self.config.agent_event_source != "acp_shadow" and project_turn ): content = self.projector.project_turn_content(self.session_id) @@ -516,7 +509,7 @@ def _accept( expected_binding=self.binding, content=projection, observed_at=event.observed_at, - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ) except BaseException: self._restore_speculation(checkpoint, prior_turn_state) diff --git a/src/tendwire/backends/acp_probe.py b/src/tendwire/backends/acp_probe.py index 3e63222..6b03f3d 100644 --- a/src/tendwire/backends/acp_probe.py +++ b/src/tendwire/backends/acp_probe.py @@ -25,7 +25,7 @@ from typing import Any from .acp_client import ( - AcpClient, + BoundedAcpConnection, AcpProtocolVersionError, AcpRequestTimeoutError, AcpTransportError, @@ -143,7 +143,7 @@ def probe_adapter( "close_timeout_seconds", maximum=MAX_PROBE_CLOSE_TIMEOUT_SECONDS, ) - client = AcpClient( + client = BoundedAcpConnection( argv, cwd=cwd, env=env, diff --git a/src/tendwire/backends/acp_protocol.py b/src/tendwire/backends/acp_protocol.py index 8d6662b..4e69440 100644 --- a/src/tendwire/backends/acp_protocol.py +++ b/src/tendwire/backends/acp_protocol.py @@ -14,6 +14,12 @@ from types import MappingProxyType from typing import Any, TypeAlias +from acp.schema import ( + RequestPermissionRequest as UpstreamRequestPermissionRequest, + SessionNotification as UpstreamSessionNotification, +) +from pydantic import ValidationError + JSONRPC_VERSION = "2.0" ACP_PROTOCOL_VERSION = 1 DEFAULT_MAX_FRAME_BYTES = 8 * 1024 * 1024 @@ -508,6 +514,13 @@ def parse_session_update(params: Mapping[str, Any]) -> SessionUpdate: except ValueError: # ACP extensions and future stable revisions remain observable. kind = kind_value + else: + try: + UpstreamSessionNotification.model_validate(dict(params)) + except ValidationError as exc: + raise AcpEnvelopeError( + "session/update params do not match the upstream ACP schema" + ) from exc meta = params.get("_meta") if meta is not None and not isinstance(meta, Mapping): meta = None @@ -524,6 +537,25 @@ def parse_permission_request(request: JsonRpcRequest) -> PermissionRequest: if request.method != "session/request_permission": raise AcpEnvelopeError("request is not session/request_permission") params = request.params + try: + UpstreamRequestPermissionRequest.model_validate(dict(params)) + except ValidationError as exc: + errors = exc.errors(include_url=False, include_input=False) + if any( + tuple(error.get("loc", ()))[-1:] == ("kind",) + for error in errors + ): + raise AcpEnvelopeError( + "permission option kind is not valid ACP v1" + ) from exc + locations = ", ".join( + ".".join(_upstream_alias(part) for part in error.get("loc", ())) + for error in errors + ) + suffix = f" ({locations})" if locations else "" + raise AcpEnvelopeError( + f"permission request does not match the upstream ACP schema{suffix}" + ) from exc session_id = _required_string(params, "sessionId") tool_call = params.get("toolCall") if not isinstance(tool_call, Mapping): @@ -566,6 +598,14 @@ def parse_permission_request(request: JsonRpcRequest) -> PermissionRequest: ) +def _upstream_alias(value: object) -> str: + """Render safe validation locations using ACP wire aliases.""" + + text = str(value) + head, *tail = text.split("_") + return head + "".join(part[:1].upper() + part[1:] for part in tail) + + def _required_string(value: Mapping[str, Any], key: str) -> str: result = value.get(key) if not isinstance(result, str) or not result: diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py index e69dd48..36dd196 100644 --- a/src/tendwire/backends/acp_runtime.py +++ b/src/tendwire/backends/acp_runtime.py @@ -69,7 +69,7 @@ class RuntimeState(str, Enum): @dataclass(frozen=True, slots=True) -class AcpRuntimeStatus: +class AcpWorkerSessionStatus: """Public-safe runtime health and counters. This type intentionally has no command, process, session, worker, target, @@ -117,8 +117,8 @@ def __call__( ) -> WorkerBinding: ... -class AcpRuntimeClient(Protocol): - """Adapter-neutral client surface required by :class:`AcpRuntime`.""" +class AcpSessionConnection(Protocol): + """Bounded ACP connection required by :class:`AcpWorkerSession`.""" def initialize( self, @@ -199,7 +199,7 @@ def respond_permission( def close(self) -> None: ... -class AcpRuntime: +class AcpWorkerSession: """Run and durably ingest exactly one ACP session for one worker binding. Permission requests fail closed: the default response is ``cancelled``. @@ -209,7 +209,7 @@ class AcpRuntime: def __init__( self, - client: AcpRuntimeClient, + client: AcpSessionConnection, *, config: Config, binding: WorkerBinding, @@ -323,7 +323,7 @@ def __init__( self._prompts_failed = 0 self._cancellation_requests = 0 - def __enter__(self) -> "AcpRuntime": + def __enter__(self) -> "AcpWorkerSession": return self.start() def __exit__(self, exc_type: object, exc: object, tb: object) -> None: @@ -333,7 +333,7 @@ def __exit__(self, exc_type: object, exc: object, tb: object) -> None: if exc is None: raise - def start(self) -> "AcpRuntime": + def start(self) -> "AcpWorkerSession": """Initialize capabilities, open one session, and start consumers.""" with self._lifecycle_lock: @@ -634,11 +634,11 @@ def cancel(self) -> None: self._record_failure(exc) raise - def status(self) -> AcpRuntimeStatus: + def status(self) -> AcpWorkerSessionStatus: """Return redacted health and counters safe for a public status API.""" with self._state_lock: - return AcpRuntimeStatus( + return AcpWorkerSessionStatus( state=self._state, healthy=self._state is RuntimeState.RUNNING and self._failure is None, updates_ingested=self._updates_ingested, @@ -1161,7 +1161,7 @@ def _permission_source_event_id(request_id: RequestId) -> str: def _prepare_prompt_content( - client: AcpRuntimeClient, + client: AcpSessionConnection, prompt: str | Sequence[Mapping[str, Any]], ) -> tuple[Mapping[str, Any], ...]: """Validate before persistence when the transport exposes its validator.""" @@ -1255,13 +1255,13 @@ def _outcome_has_persisted_event(outcome: object) -> bool: __all__ = [ - "AcpRuntime", + "AcpWorkerSession", "AcpRuntimeBindingError", - "AcpRuntimeClient", + "AcpSessionConnection", "AcpRuntimeError", "AcpRuntimeProtocolError", "AcpRuntimeStateError", - "AcpRuntimeStatus", + "AcpWorkerSessionStatus", "AcpRuntimeStopTimeout", "PermissionCallback", "RuntimeState", diff --git a/src/tendwire/backends/herdr_cli.py b/src/tendwire/backends/herdr_cli.py index bf120eb..a426485 100644 --- a/src/tendwire/backends/herdr_cli.py +++ b/src/tendwire/backends/herdr_cli.py @@ -65,8 +65,6 @@ class HerdrContinuityUnavailableError(RuntimeError): class _WorkerRecord: worker: Worker private_fingerprint: str - turn_target_kind: str | None = None - turn_target_value: str | None = None # Canonical public Herdr identity used exclusively for continuity. Raw # observations remain private and separate so routing compatibility can # never accidentally feed stable-key derivation. @@ -110,9 +108,6 @@ class _WorkerRecord: {"agent_id", "terminal_id", "pane_id", "agent", "name", "label"} ) _AGENT_SCOPED_BACKEND_TARGET_KINDS = frozenset({"agent_id", "agent"}) -_SESSION_SCOPED_TURN_TARGET_KINDS = frozenset( - {"codex_session_id", "omp_session_path"} -) _DEADLINE_EXHAUSTED_OUTCOMES = frozenset({"timeout", "deadline_exhausted"}) _UNAVAILABLE_HEALTH_OUTCOMES = frozenset({"missing_binary", "launch_error", "socket_disconnected"}) _DEGRADED_HEALTH_OUTCOMES = frozenset( @@ -1257,21 +1252,6 @@ def _backend_target_from_item(item: Mapping[str, Any]) -> dict[str, Any] | None: return None -def _turn_target_from_item(item: Mapping[str, Any]) -> tuple[str, str] | None: - """Resolve the private Herdr structured-turn target from backend-observed fields.""" - agent_name = (_first_text(item, ("agent", "name")) or "").strip().lower() - agent_session = _nested_text(item, "agent_session", "value") - if agent_name == "codex" and agent_session: - return "codex_session_id", agent_session - if agent_name == "omp" and agent_session: - # oh-my-pi reports its native session file path (herdr:omp source). - return "omp_session_path", agent_session - pane_id = _first_text(item, ("pane_id",)) - if pane_id: - return "pane_id", pane_id - return None - - def _worker_with_id(worker: Worker, worker_id: str) -> Worker: """Return a worker copy with a disambiguated public id.""" return Worker( @@ -1445,7 +1425,6 @@ def _worker_record_from_item( item = _strip_turn_observation_fields(item) worker = _worker_from_item(item) worker = _worker_with_summary(worker, _bounded_excerpt(worker.summary, _output_excerpt_limit(config))) - turn_target = _turn_target_from_item(item) observed_workspace_id = _first_text(item, ("workspace_id", "workspaceId")) observed_pane_id = _first_text(item, ("pane_id", "paneId")) canonical_identity = canonical_herdr_pane_identity( @@ -1456,8 +1435,6 @@ def _worker_record_from_item( return _WorkerRecord( worker=worker, private_fingerprint=_private_identity_from_item(item, config), - turn_target_kind=turn_target[0] if turn_target is not None else None, - turn_target_value=turn_target[1] if turn_target is not None else None, workspace_id=workspace_id, pane_id=pane_id, observed_workspace_id=observed_workspace_id, @@ -1634,8 +1611,6 @@ def _record_with_worker(record: _WorkerRecord, worker: Worker) -> _WorkerRecord: return _WorkerRecord( worker=worker, private_fingerprint=record.private_fingerprint, - turn_target_kind=record.turn_target_kind, - turn_target_value=record.turn_target_value, workspace_id=record.workspace_id, pane_id=record.pane_id, observed_workspace_id=record.observed_workspace_id, @@ -1793,8 +1768,6 @@ def _binding_from_worker_record( backend=_BACKEND_NAME, target_kind=target_kind, target_value=target_value, - turn_target_kind=record.turn_target_kind, - turn_target_value=record.turn_target_value, sendable=target.get("sendable") is True, reason=str(reason) if reason is not None else None, observed_at=observed_at, @@ -2016,29 +1989,6 @@ def _compatible_backend_target( return pane_target if _backend_target_present(pane_target) else None -def _compatible_turn_target( - agent_record: _WorkerRecord, - pane_record: _WorkerRecord, -) -> tuple[str | None, str | None]: - """Prefer PaneInfo targets unless a compatible session target is more precise.""" - agent_kind = agent_record.turn_target_kind - agent_value = agent_record.turn_target_value - pane_kind = pane_record.turn_target_kind - pane_value = pane_record.turn_target_value - if agent_kind in _SESSION_SCOPED_TURN_TARGET_KINDS and agent_value: - pane_session_id = pane_record.agent_session_id - if pane_session_id and pane_session_id != agent_value: - if pane_kind and pane_value: - return pane_kind, pane_value - return None, None - if pane_kind in _SESSION_SCOPED_TURN_TARGET_KINDS and pane_value: - return pane_kind, pane_value - return agent_kind, agent_value - if pane_kind and pane_value: - return pane_kind, pane_value - return None, None - - def _ambiguous_agent_record( record: _WorkerRecord, *, @@ -2117,18 +2067,11 @@ def _merge_agent_pane_record( backend_target=backend_target, ) - turn_target_kind, turn_target_value = _compatible_turn_target( - agent_record, - pane_record, - ) - workspace_id = pane_record.workspace_id pane_id = pane_record.pane_id terminal_id = pane_record.terminal_id if ( worker == agent_record.worker - and turn_target_kind == agent_record.turn_target_kind - and turn_target_value == agent_record.turn_target_value and workspace_id == agent_record.workspace_id and pane_id == agent_record.pane_id and terminal_id == agent_record.terminal_id @@ -2138,8 +2081,6 @@ def _merge_agent_pane_record( return _WorkerRecord( worker=worker, private_fingerprint=agent_record.private_fingerprint, - turn_target_kind=turn_target_kind, - turn_target_value=turn_target_value, workspace_id=workspace_id, pane_id=pane_id, observed_workspace_id=pane_record.observed_workspace_id, @@ -2198,13 +2139,6 @@ def _record_ownership_keys( str(backend_target.get("value") or ""), ) ) - if record.turn_target_kind and record.turn_target_value: - keys.add( - ( - f"turn:{record.turn_target_kind}", - record.turn_target_value, - ) - ) return keys diff --git a/src/tendwire/backends/herdr_decision.py b/src/tendwire/backends/herdr_decision.py deleted file mode 100644 index c6199bd..0000000 --- a/src/tendwire/backends/herdr_decision.py +++ /dev/null @@ -1,124 +0,0 @@ -"""Translate semantic Claude decisions into private Herdr pane input. - -Calibration assumptions are deliberately confined to this backend module, and -every mapping below was LIVE-VERIFIED against Claude Code 2.1.211 on a real -pane (2026-07-16): - -* Single-choice and plan rows carry 1-based decimal shortcuts, and typing the - ordinal alone SELECTS AND SUBMITS the row instantly — no Enter follows. A - trailing Enter would leak into whatever UI appears next, so none is sent. -* The single-choice write-in row ("Type something", at position N + 1) does - NOT respond to its digit. It is reached by pressing Down exactly N times - from the initial cursor on row 1; the focused row is itself a text input, - so Tendwire then sends the write-in prose and submits with Enter. -* Multi-select digits toggle their row ABSOLUTELY without moving the cursor, - Right switches to the Submit tab (a review screen whose default focus is - "Submit answers"), and Enter there submits the selection set. -* Every driven ordinal must stay a single keystroke, so decisions expose at - most 9 real options (PENDING_DECISION_MAX_OPTIONS in herdr_turns). -* Herdr's private ``pane.send_keys`` accepts decimal character keys plus - ``Down``, ``Up``, ``Right``, and ``Enter``; write-in prose uses - ``pane.send_input`` so Herdr owns terminal text encoding and appends the - final Enter atomically. - -These steps are internal calibration data. They are never accepted from a -connector and there is intentionally no public raw-key command action. -""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Literal - - -MULTI_SELECT_CALIBRATION = { - "submit_tab": "Right", - "submit": "Enter", -} - - -@dataclass(frozen=True) -class HerdrDecisionStep: - """One private, already-calibrated Herdr pane operation.""" - - operation: Literal["keys", "text", "input"] - keys: tuple[str, ...] = () - text: str | None = None - - def __post_init__(self) -> None: - if self.operation == "keys": - if not self.keys or self.text is not None: - raise ValueError("key calibration step requires only keys") - elif self.operation == "text": - raise ValueError("text calibration steps are no longer produced") - elif self.operation == "input": - if not isinstance(self.text, str) or not self.text or self.keys != ("Enter",): - raise ValueError("input calibration step requires text plus Enter") - else: - raise ValueError("unsupported decision calibration operation") - - -def _digit_keys(value: int | str) -> tuple[str, ...]: - text = str(value) - if not text.isdigit() or int(text) < 1: - raise ValueError("decision ordinal must be a positive decimal") - return tuple(text) - - -def calibrate_decision_steps( - *, - kind: Literal["single", "multi", "plan"], - option_count: int, - option_refs: tuple[str, ...] = (), - text: str | None = None, -) -> tuple[HerdrDecisionStep, ...]: - """Return private pane operations for one validated semantic selection.""" - if ( - kind not in {"single", "multi", "plan"} - or not isinstance(option_count, int) - or isinstance(option_count, bool) - or option_count < 1 - ): - raise ValueError("invalid decision calibration context") - if text is not None: - if kind != "single" or option_refs or not isinstance(text, str) or not text: - raise ValueError("invalid decision write-in calibration") - # The write-in row ignores digits; reach it with Down x N from row 1, - # where the focused row is itself the text input. - return ( - HerdrDecisionStep("keys", keys=("Down",) * option_count), - HerdrDecisionStep("input", keys=("Enter",), text=text), - ) - if not option_refs or len(option_refs) != len(set(option_refs)): - raise ValueError("decision option refs must be nonempty and unique") - ordinals: list[int] = [] - for ref in option_refs: - if not isinstance(ref, str) or not ref.isdigit(): - raise ValueError("decision option ref must be a decimal ordinal") - ordinal = int(ref) - if not 1 <= ordinal <= option_count: - raise ValueError("decision option ref is out of range") - ordinals.append(ordinal) - if kind in {"single", "plan"}: - if len(ordinals) != 1: - raise ValueError("single and plan decisions require one option") - # The digit alone selects AND submits; a trailing Enter would leak into - # the next UI (composer, or worse, a modal). - return (HerdrDecisionStep("keys", keys=_digit_keys(ordinals[0])),) - - # Digits toggle rows absolutely (cursor-independent); Right reaches the - # Submit tab whose default focus is "Submit answers"; Enter submits. - steps: list[HerdrDecisionStep] = [ - HerdrDecisionStep("keys", keys=_digit_keys(ordinal)) - for ordinal in sorted(ordinals) - ] - steps.append( - HerdrDecisionStep( - "keys", - keys=( - MULTI_SELECT_CALIBRATION["submit_tab"], - MULTI_SELECT_CALIBRATION["submit"], - ), - ) - ) - return tuple(steps) diff --git a/src/tendwire/backends/herdr_events.py b/src/tendwire/backends/herdr_events.py index 326e508..fa09d08 100644 --- a/src/tendwire/backends/herdr_events.py +++ b/src/tendwire/backends/herdr_events.py @@ -17,7 +17,7 @@ from pathlib import Path from typing import Any -from ..config import Config +from ..config import DEFAULT_TURN_MODEL, Config from ..core.models import ( BackendHealth, Snapshot, @@ -35,22 +35,14 @@ ) from ..core.projector import project_from_observations from ..store.sqlite import ( - HerdrTurnWatermark, SnapshotObservationContext, SnapshotRetentionPolicy, expire_stale_worker_bindings, expire_worker_bindings, - get_herdr_turn_refresh_retry, - get_herdr_turn_watermark, - herdr_turn_refresh_retry_due, latest_snapshot, list_worker_bindings, maybe_run_automatic_store_maintenance, - record_herdr_turn_completeness_break, - record_herdr_turn_completion, - record_herdr_turn_refresh_retry, save_snapshot, - set_herdr_turn_watermark, upsert_worker_bindings, ) from .herdr_cli import ( @@ -67,7 +59,6 @@ from .herdr_protocol import ( HERDR_EVENTS_SUBSCRIBE_METHOD, HERDR_OFFICIAL_EVENT_NAMES, - HERDR_TURN_COMPLETED_EVENT_NAME, HerdrEnvelopeError, HerdrErrorResponse, HerdrMalformedLineError, @@ -92,15 +83,10 @@ _AGENT_PAYLOAD_KEYS = ("agents", "workers", "data", "items", "results", "result") _PANE_PAYLOAD_KEYS = ("panes", "items", "data", "results", "result") -_SUPPORTED_EVENT_NAMES = ( - *HERDR_OFFICIAL_EVENT_NAMES, - HERDR_TURN_COMPLETED_EVENT_NAME, -) +_SUPPORTED_EVENT_NAMES = HERDR_OFFICIAL_EVENT_NAMES _SUPPORTED_EVENT_NAME_SET = frozenset(_SUPPORTED_EVENT_NAMES) _HERDR_074_EVENT_NAMES = tuple( - event_name - for event_name in _SUPPORTED_EVENT_NAMES - if event_name not in {"pane.updated", "pane.turn_completed"} + event_name for event_name in _SUPPORTED_EVENT_NAMES if event_name != "pane.updated" ) _HERDR_074_PANE_SCOPED_REPLAY_EVENT_NAMES = frozenset( { @@ -119,7 +105,6 @@ { "pane.agent_status_changed", "pane.output_matched", - "pane.turn_completed", } ) _GLOBAL_EVENT_NAMES = tuple( @@ -139,39 +124,8 @@ ) _WORKTREE_EVENT_NAMES = frozenset({"worktree.created", "worktree.opened", "worktree.removed"}) # ``pane.updated`` is normalized from Herdr 0.7.5's scalar -# ``PaneOutputChanged`` event. It is a turn-refresh notification, not a -# PaneInfo observation, and therefore must never rebuild worker identity. +# ``PaneOutputChanged`` lifecycle event and must never rebuild worker identity. _PANE_WORKER_EVENT_NAMES = frozenset({"pane.created", "pane.focused"}) -_TURN_REFRESH_EVENT_NAMES = frozenset( - { - "pane.created", - "pane.updated", - "pane.focused", - "pane.moved", - "pane.closed", - "pane.exited", - "pane.agent_detected", - "pane.agent_status_changed", - "pane.output_matched", - } -) -_COMPLETED_TURN_REFRESH_STATUSES = frozenset({"updated", "unchanged", "missing"}) -_RETRYABLE_COMPLETED_TURN_REFRESH_STATUSES = frozenset( - { - "binding_ambiguous", - "binding_missing", - "failed", - "stale_binding", - "store_unavailable", - "timeout", - } -) -_COMPLETED_TURN_REFRESH_MAX_RETRY_AGE_SECONDS = 5 * 60 -_COMPLETED_TURN_REFRESH_MAX_ATTEMPTS = 8 -_COMPLETED_TURN_REFRESH_RETRY_BASE_DELAY_SECONDS = 1 -_COMPLETED_TURN_REFRESH_RETRY_MAX_DELAY_SECONDS = 30 - - class HerdrEventBackendError(Exception): """Base error for the opt-in Herdr socket event backend.""" @@ -227,33 +181,6 @@ class NormalizedHerdrEvent: producer_identity: HerdrProducerIdentity | None -@dataclass(frozen=True) -class HerdrTurnCompletionRecord: - """Validated completion metadata; semantic content still comes from adapters.""" - - pane_id: str - turn: int - turn_epoch: int - outcome: str - completed_unix_ms: int - message: str | None = None - message_truncated: bool = False - agent_session_path: str | None = None - - -@dataclass(frozen=True) -class HerdrPaneTurnsReplay: - pane_id: str - turn_epoch: int - records: tuple[HerdrTurnCompletionRecord, ...] - truncated: bool - oldest_available: int | None - - @property - def newest_turn(self) -> int: - return max((record.turn for record in self.records), default=0) - - def _compact_key(value: object) -> str: return str(value).strip().lower().replace("-", "_").replace(".", "_").replace(":", "_") @@ -304,105 +231,6 @@ def _call_with_optional_keywords( return callback(*args, **dict(kwargs)) -def _nonnegative_protocol_integer(value: Any, field: str) -> int: - if ( - not isinstance(value, int) - or isinstance(value, bool) - or value < 0 - or value > (1 << 63) - 1 - ): - raise HerdrEnvelopeError(f"invalid Herdr {field}") - return int(value) - - -def _turn_completion_record( - value: Any, - *, - pane_id: str | None = None, -) -> HerdrTurnCompletionRecord: - if not isinstance(value, Mapping): - raise HerdrEnvelopeError("invalid Herdr turn completion record") - pane = value.get("pane") - resolved_pane_id = pane_id or ( - _first_text(pane, ("pane_id", "paneId", "id")) - if isinstance(pane, Mapping) - else None - ) or _first_text(value, ("pane_id", "paneId")) - if not resolved_pane_id: - raise HerdrEnvelopeError("Herdr turn completion is missing pane_id") - outcome = value.get("outcome") - if outcome not in {"completed", "aborted"}: - raise HerdrEnvelopeError("invalid Herdr turn completion outcome") - message = value.get("message") - if message is not None and not isinstance(message, str): - raise HerdrEnvelopeError("invalid Herdr turn completion message") - if isinstance(message, str) and len(message.encode("utf-8")) > 8 * 1024: - raise HerdrEnvelopeError("Herdr turn completion message is too large") - message_truncated = value.get("message_truncated", False) - if not isinstance(message_truncated, bool): - raise HerdrEnvelopeError("invalid Herdr message_truncated") - agent_session_path = value.get("agent_session_path") - if agent_session_path is not None and not isinstance(agent_session_path, str): - raise HerdrEnvelopeError("invalid Herdr agent_session_path") - return HerdrTurnCompletionRecord( - pane_id=str(resolved_pane_id), - turn=_nonnegative_protocol_integer(value.get("turn"), "turn"), - turn_epoch=_nonnegative_protocol_integer( - value.get("turn_epoch"), - "turn_epoch", - ), - outcome=str(outcome), - completed_unix_ms=_nonnegative_protocol_integer( - value.get("completed_unix_ms"), - "completed_unix_ms", - ), - message=message, - message_truncated=message_truncated, - agent_session_path=agent_session_path, - ) - - -def _pane_turns_replay(value: Any, pane_id: str) -> HerdrPaneTurnsReplay: - if not isinstance(value, Mapping): - raise HerdrEnvelopeError("invalid pane.turns response") - turns = value.get("turns") - if isinstance(turns, Mapping): - value = turns - response_pane_id = value.get("pane_id") - if not isinstance(response_pane_id, str) or response_pane_id != pane_id: - raise HerdrEnvelopeError("pane.turns returned the wrong pane") - epoch = _nonnegative_protocol_integer(value.get("turn_epoch"), "turn_epoch") - raw_records = value.get("records") - if not isinstance(raw_records, list): - raise HerdrEnvelopeError("pane.turns records must be an array") - records = tuple( - _turn_completion_record(record, pane_id=pane_id) - for record in raw_records - ) - if any(record.turn_epoch != epoch for record in records): - raise HerdrEnvelopeError("pane.turns mixed turn epochs") - if tuple(record.turn for record in records) != tuple( - sorted({record.turn for record in records}) - ): - raise HerdrEnvelopeError("pane.turns records are not strictly ordered") - truncated = value.get("truncated", False) - if not isinstance(truncated, bool): - raise HerdrEnvelopeError("invalid pane.turns truncated marker") - raw_oldest = value.get("oldest_available") - oldest = ( - None - if raw_oldest is None - else _nonnegative_protocol_integer(raw_oldest, "oldest_available") - ) - return HerdrPaneTurnsReplay( - pane_id=pane_id, - turn_epoch=epoch, - records=records, - truncated=truncated, - oldest_available=oldest, - ) - - def _entity_payload_with_source(payload: Mapping[str, Any], *entity_names: str) -> tuple[dict[str, Any], str | None]: """Return an event entity object plus the nested entity name selected.""" merged: dict[str, Any] = {} @@ -769,7 +597,6 @@ def __init__( max_batch_size: int = DEFAULT_MAX_BATCH_SIZE, reconnect_delay_seconds: float = DEFAULT_RECONNECT_DELAY_SECONDS, stop_event: threading.Event | None = None, - turn_completion_processor: Callable[..., Any] | None = None, ) -> None: self.config = config self.client_factory = client_factory or self._default_client_factory @@ -794,13 +621,7 @@ def __init__( self.max_batch_size = max(1, int(max_batch_size)) self.reconnect_delay_seconds = max(0.0, float(reconnect_delay_seconds)) self.stop_event = stop_event or threading.Event() - self.turn_completion_processor = ( - turn_completion_processor - or self._default_turn_completion_processor - ) self._lock = threading.RLock() - self._turn_refresh_callback_lock = threading.Lock() - self._turn_refresh_callback: Callable[[], None] | None = None self._ready = threading.Event() self._thread: threading.Thread | None = None self._producer_dedupe: OrderedDict[HerdrProducerIdentity, None] = OrderedDict() @@ -823,35 +644,13 @@ def __init__( self._last_cap_status_at: str | None = None self._automatic_maintenance_status: dict[str, Any] | None = None self._next_reconcile_monotonic: float | None = None - self._next_turn_replay_monotonic: float | None = None self._subscription_pane_ids: list[str] = [] - self._turn_api_probed = False - self._turn_api_supported = False - self._turn_completion_diagnostic_counts: dict[str, int] = {} self._load_existing_state() @staticmethod def _default_client_factory(config: Config) -> HerdrSocketClient: return HerdrSocketClient(timeout=config.herdr_timeout_seconds) - @staticmethod - def _default_turn_completion_processor( - config: Config, - pane_id: str, - *, - terminal_id: str | None = None, - binding_private_fingerprint: str | None = None, - ) -> Any: - from .herdr_turns import refresh_completed_pane_turn - - return refresh_completed_pane_turn( - config, - pane_id, - terminal_id=terminal_id, - binding_private_fingerprint=binding_private_fingerprint, - adapter_timeout_seconds=config.herdr_timeout_seconds, - ) - @property def db_path(self) -> Path: if self.config.db_path is None: @@ -881,9 +680,6 @@ def operational_status(self) -> dict[str, Any]: if self._automatic_maintenance_status is not None else None ), - "turn_completion_diagnostics": dict( - self._turn_completion_diagnostic_counts - ), } @property @@ -895,24 +691,6 @@ def running(self) -> bool: thread = self._thread return thread is not None and thread.is_alive() - def set_turn_refresh_callback(self, callback: Callable[[], None] | None) -> None: - """Set the post-persistence turn refresh signal.""" - if callback is not None and not callable(callback): - raise TypeError("turn refresh callback must be callable or None") - with self._turn_refresh_callback_lock: - self._turn_refresh_callback = callback - - def _notify_turn_refresh(self) -> None: - with self._turn_refresh_callback_lock: - callback = self._turn_refresh_callback - if callback is None: - return - try: - callback() - except Exception: - # Scheduling is best-effort after the durable backend commit. - pass - def _health_for(self, outcome: str) -> HerdrEventBackendHealth: health = herdr_backend_health(outcome) return HerdrEventBackendHealth( @@ -934,7 +712,7 @@ def _save_snapshot( if not save_snapshot( self.db_path, snapshot, - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, observation=observation, worker_bindings=worker_bindings, binding_backend=BACKEND_NAME if worker_bindings is not None else None, @@ -951,7 +729,7 @@ def _save_snapshot( result = maybe_run_automatic_store_maintenance( self.db_path, policy=policy, - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, acknowledged_final_retention_days=( self.config.acknowledged_final_retention_days ), @@ -1049,33 +827,13 @@ def run_forever(self) -> None: try: client = self.client_factory(self.config) try: - self._turn_api_probed = False - self._turn_api_supported = False self.reconcile_once(client=client) reconciled = True - # Herdr ordinary RPC connections are one-shot. Keep this - # future subscription client out of pane.turns entirely; - # each replay request uses its own short-lived client. - if callable(getattr(client, "pane_turns", None)) or callable( - getattr(client, "request", None) - ): - self._replay_turns_after_reconcile() - else: - self._turn_api_probed = True - self._turn_api_supported = False if self.stop_event.is_set(): break if hasattr(client, "connect"): client.connect() stream = self._subscribe_event_stream(client) - # Herdr subscriptions intentionally start at the server's - # current sequence and deliver no backlog. The first - # replay above capability-gates the new subscription kind; - # this second replay closes the probe-to-subscribe race. - # Events concurrent with it are buffered by the socket - # client and become harmless watermark-deduped duplicates. - if self._turn_api_supported: - self._replay_turns_after_reconcile() self._ready.set() self._read_event_stream(client, stream.subscription_id) finally: @@ -1153,30 +911,12 @@ def _run_periodic_reconcile_if_due(self, client: Any | None = None) -> None: and due_at is not None and current >= due_at ) - turn_replay_due = ( - self._next_turn_replay_monotonic is not None - and current >= self._next_turn_replay_monotonic - ) if self.reconcile_interval_seconds > 0 and due_at is None: self._schedule_next_reconcile() - if not reconcile_due and not turn_replay_due: + if not reconcile_due: return if reconcile_due: self.reconcile_once(client=client) - if self._turn_api_supported: - # Completion refreshes intentionally leave their watermark behind - # while a worker binding is still settling. Periodic snapshot - # reconciliation must therefore also replay the durable Herdr turn - # ledger; otherwise a retryable completion would not be revisited - # until the subscription happened to disconnect. - self._next_turn_replay_monotonic = None - self._replay_turns_after_reconcile() - - def _schedule_turn_replay(self, delay_seconds: float) -> None: - candidate = time.monotonic() + max(0.0, float(delay_seconds)) - due_at = self._next_turn_replay_monotonic - if due_at is None or candidate < due_at: - self._next_turn_replay_monotonic = candidate def _pending_event_count(self) -> int: with self._lock: @@ -1284,7 +1024,6 @@ def reconcile_once(self, *, client: Any | None = None) -> Snapshot: observed_at=health.observed_at or snapshot.updated_at, message=health.message, ) - self._notify_turn_refresh() return snapshot except (HerdrContinuityUnavailableError, InstallationKeyError): snapshot = self._mark_unhealthy("continuity_unavailable") @@ -1331,548 +1070,6 @@ def _herdr_error_message(exc: HerdrErrorResponse) -> str: return message return "" - def _call_pane_turns( - self, - client: Any, - pane_id: str, - *, - since: int, - expected_epoch: int | None, - allow_uncorrelated: bool = False, - ) -> HerdrPaneTurnsReplay: - params: dict[str, Any] = { - "pane_id": str(pane_id), - "since": int(since), - } - if expected_epoch is not None: - params["expected_epoch"] = int(expected_epoch) - optional_keywords = {"timeout": self.config.herdr_timeout_seconds} - if allow_uncorrelated: - optional_keywords["allow_uncorrelated"] = True - method = getattr(client, "pane_turns", None) - if callable(method): - value = _call_with_optional_keywords( - method, - (params,), - optional_keywords, - ) - else: - request = getattr(client, "request", None) - if not callable(request): - raise AttributeError("client does not expose pane.turns") - value = _call_with_optional_keywords( - request, - ("pane.turns", params), - optional_keywords, - ) - return _pane_turns_replay(value, pane_id) - - def _call_pane_turns_isolated( - self, - pane_id: str, - *, - since: int, - expected_epoch: int | None, - allow_uncorrelated: bool = False, - ) -> HerdrPaneTurnsReplay: - """Call one pane.turns RPC on a fresh, always-closed client.""" - client = self.client_factory(self.config) - try: - if hasattr(client, "connect"): - client.connect() - return self._call_pane_turns( - client, - pane_id, - since=since, - expected_epoch=expected_epoch, - allow_uncorrelated=allow_uncorrelated, - ) - finally: - if hasattr(client, "close"): - client.close() - - def _record_turn_diagnostic( - self, - code: str, - pane_id: str, - *, - status: str | None = None, - ) -> None: - count_key = f"{code}:{status}" if status else code - with self._lock: - self._turn_completion_diagnostic_counts[count_key] = ( - self._turn_completion_diagnostic_counts.get(count_key, 0) + 1 - ) - diagnostic = { - "code": code, - "host_id": self.config.host_id, - "pane_id": pane_id, - } - if status: - diagnostic["status"] = status - _LOGGER.warning( - code, - extra={"tendwire_diagnostic": diagnostic}, - ) - - def _record_completeness_break( - self, - replay: HerdrPaneTurnsReplay, - reason: str, - ) -> None: - record_herdr_turn_completeness_break( - self.db_path, - self.config.host_id, - replay.pane_id, - turn_epoch=replay.turn_epoch, - newest_turn=replay.newest_turn, - reason=reason, - ) - _LOGGER.warning( - "herdr_turn_completeness_break", - extra={ - "tendwire_diagnostic": { - "code": "herdr_turn_completeness_break", - "host_id": self.config.host_id, - "pane_id": replay.pane_id, - "reason": reason, - } - }, - ) - - def _completion_processor_result( - self, - record: HerdrTurnCompletionRecord, - ) -> tuple[str, str | None, str | None]: - terminal_id = self._pane_terminals.get(record.pane_id) - owner_ids = self._pane_owners.get(record.pane_id, set()) - owner_bindings = [ - binding - for binding in self._bindings.values() - if binding.worker_id in owner_ids - ] - binding_private_fingerprint = ( - owner_bindings[0].private_fingerprint - if len(owner_bindings) == 1 - else None - ) - result = _call_with_optional_keywords( - self.turn_completion_processor, - (self.config, record.pane_id), - { - "terminal_id": terminal_id, - "binding_private_fingerprint": binding_private_fingerprint, - }, - ) - if isinstance(result, Mapping): - status = str(result.get("status") or "") - worker_id = result.get("worker_id") - refreshed_turn_id = result.get("refreshed_turn_id") - else: - status = str(getattr(result, "status", "")) - worker_id = getattr(result, "worker_id", None) - refreshed_turn_id = getattr(result, "refreshed_turn_id", None) - return ( - status, - str(worker_id) if worker_id else None, - str(refreshed_turn_id) if refreshed_turn_id else None, - ) - - def _process_turn_record( - self, - record: HerdrTurnCompletionRecord, - ) -> None: - watermark = get_herdr_turn_watermark( - self.db_path, - self.config.host_id, - record.pane_id, - ) - if watermark is None: - self._record_completeness_break( - HerdrPaneTurnsReplay( - pane_id=record.pane_id, - turn_epoch=record.turn_epoch, - records=(record,), - truncated=False, - oldest_available=record.turn, - ), - "live_without_baseline", - ) - return - if watermark.turn_epoch != record.turn_epoch: - self._record_completeness_break( - HerdrPaneTurnsReplay( - pane_id=record.pane_id, - turn_epoch=record.turn_epoch, - records=(record,), - truncated=False, - oldest_available=record.turn, - ), - "turn_epoch_mismatch", - ) - return - if record.turn <= watermark.last_turn: - return - if record.turn != watermark.last_turn + 1: - # A later live notification is not evidence of ledger loss while - # the exact next completion is durably waiting for refresh. Leave - # ordering intact; pane.turns replay will revisit the blocker. - pending_predecessor = get_herdr_turn_refresh_retry( - self.db_path, - self.config.host_id, - record.pane_id, - turn_epoch=record.turn_epoch, - turn=watermark.last_turn + 1, - ) - if ( - pending_predecessor is not None - and pending_predecessor.status == "pending" - ): - return - self._record_completeness_break( - HerdrPaneTurnsReplay( - pane_id=record.pane_id, - turn_epoch=record.turn_epoch, - records=(record,), - truncated=False, - oldest_available=record.turn, - ), - "live_gap", - ) - return - existing_retry = get_herdr_turn_refresh_retry( - self.db_path, - self.config.host_id, - record.pane_id, - turn_epoch=record.turn_epoch, - turn=record.turn, - ) - if existing_retry is not None and existing_retry.status == "escalated": - # Escalation and watermark advancement are separate durable writes. - # If the process stopped between them, finalize provenance without - # rerunning the known-poison refresh or losing the terminal marker. - self._record_turn_diagnostic( - "herdr_turn_completion_refresh_escalated_recovered", - record.pane_id, - status=existing_retry.refresh_status, - ) - record_herdr_turn_completion( - self.db_path, - self.config.host_id, - record.pane_id, - turn_epoch=record.turn_epoch, - turn=record.turn, - outcome=record.outcome, - completed_unix_ms=record.completed_unix_ms, - message=record.message, - message_truncated=record.message_truncated, - agent_session_path=record.agent_session_path, - worker_id=None, - refreshed_turn_id=None, - preserve_refresh_retry=True, - ) - return - if not herdr_turn_refresh_retry_due( - self.db_path, - self.config.host_id, - record.pane_id, - turn_epoch=record.turn_epoch, - turn=record.turn, - ): - self._schedule_turn_replay( - _COMPLETED_TURN_REFRESH_RETRY_BASE_DELAY_SECONDS - ) - return - status, worker_id, refreshed_turn_id = self._completion_processor_result( - record - ) - preserve_refresh_retry = False - if status not in _COMPLETED_TURN_REFRESH_STATUSES: - self._record_turn_diagnostic( - "herdr_turn_completion_refresh_skipped", - record.pane_id, - status=status or "unknown", - ) - refreshed_turn_id = None - if status in _RETRYABLE_COMPLETED_TURN_REFRESH_STATUSES: - retry = record_herdr_turn_refresh_retry( - self.db_path, - self.config.host_id, - record.pane_id, - turn_epoch=record.turn_epoch, - turn=record.turn, - refresh_status=status, - base_delay_seconds=( - _COMPLETED_TURN_REFRESH_RETRY_BASE_DELAY_SECONDS - ), - max_delay_seconds=( - _COMPLETED_TURN_REFRESH_RETRY_MAX_DELAY_SECONDS - ), - max_retry_age_seconds=( - _COMPLETED_TURN_REFRESH_MAX_RETRY_AGE_SECONDS - ), - max_attempts=_COMPLETED_TURN_REFRESH_MAX_ATTEMPTS, - ) - if retry.status == "pending": - retry_delay = min( - _COMPLETED_TURN_REFRESH_RETRY_MAX_DELAY_SECONDS, - _COMPLETED_TURN_REFRESH_RETRY_BASE_DELAY_SECONDS - * (2 ** min(retry.attempt_count - 1, 30)), - ) - self._schedule_turn_replay(retry_delay) - return - preserve_refresh_retry = True - self._record_turn_diagnostic( - "herdr_turn_completion_refresh_escalated", - record.pane_id, - status=status, - ) - record_herdr_turn_completion( - self.db_path, - self.config.host_id, - record.pane_id, - turn_epoch=record.turn_epoch, - turn=record.turn, - outcome=record.outcome, - completed_unix_ms=record.completed_unix_ms, - message=record.message, - message_truncated=record.message_truncated, - agent_session_path=record.agent_session_path, - worker_id=worker_id, - refreshed_turn_id=refreshed_turn_id, - preserve_refresh_retry=preserve_refresh_retry, - ) - - def _consume_replay( - self, - replay: HerdrPaneTurnsReplay, - watermark: HerdrTurnWatermark | None, - ) -> None: - if watermark is None: - set_herdr_turn_watermark( - self.db_path, - self.config.host_id, - replay.pane_id, - turn_epoch=replay.turn_epoch, - last_turn=replay.newest_turn, - ) - return - if replay.turn_epoch != watermark.turn_epoch: - self._record_completeness_break(replay, "turn_epoch_mismatch") - return - if replay.truncated: - self._record_completeness_break(replay, "replay_truncated") - return - expected = watermark.last_turn + 1 - actual_turns = tuple(record.turn for record in replay.records) - expected_turns = tuple(range(expected, expected + len(replay.records))) - if actual_turns != expected_turns: - self._record_completeness_break(replay, "replay_gap") - return - for record in replay.records: - self._process_turn_record(record) - current = get_herdr_turn_watermark( - self.db_path, - self.config.host_id, - replay.pane_id, - ) - if current is None or current.last_turn < record.turn: - # Preserve strict same-pane ordering: a pending refresh blocks - # later records in this replay, but never another pane. - return - - @classmethod - def _turn_api_method_unsupported(cls, exc: HerdrErrorResponse) -> bool: - code = cls._herdr_error_code(exc).strip().lower().replace("-", "_") - raw_message = cls._herdr_error_message(exc).lower() - message = raw_message.strip() - if code in { - "method_not_found", - "unknown_method", - "unsupported_method", - "not_implemented", - }: - return True - if code == "invalid_request" and raw_message.startswith( - "invalid request: unknown variant" - ): - return True - return code == "invalid_params" and any( - marker in message - for marker in ( - "unknown method", - "method not found", - "unsupported method", - "pane.turns is not supported", - ) - ) - - def _probe_turn_api( - self, - client: Any | None, - pane_id: str, - watermark: HerdrTurnWatermark | None, - ) -> tuple[ - bool, - HerdrPaneTurnsReplay | None, - HerdrErrorResponse | AttributeError | None, - ]: - """Probe pane.turns once without treating pane-scoped errors as absence.""" - if client is not None and not callable( - getattr(client, "pane_turns", None) - ) and not callable(getattr(client, "request", None)): - return False, None, None - try: - call = self._call_pane_turns_isolated if client is None else ( - lambda current_pane_id, **kwargs: self._call_pane_turns( - client, - current_pane_id, - **kwargs, - ) - ) - replay = call( - pane_id, - since=watermark.last_turn if watermark is not None else 0, - expected_epoch=watermark.turn_epoch if watermark is not None else None, - allow_uncorrelated=True, - ) - except HerdrErrorResponse as exc: - if self._turn_api_method_unsupported(exc): - return False, None, None - return True, None, exc - except AttributeError: - return False, None, None - return True, replay, None - - def _consume_pane_replay( - self, - client: Any | None, - pane_id: str, - watermark: HerdrTurnWatermark | None, - *, - replay: HerdrPaneTurnsReplay | None = None, - error: HerdrErrorResponse | AttributeError | None = None, - ) -> None: - call = self._call_pane_turns_isolated if client is None else ( - lambda current_pane_id, **kwargs: self._call_pane_turns( - client, - current_pane_id, - **kwargs, - ) - ) - try: - if error is not None: - raise error - current_replay = replay or call( - pane_id, - since=watermark.last_turn if watermark is not None else 0, - expected_epoch=watermark.turn_epoch if watermark is not None else None, - ) - except HerdrErrorResponse as exc: - code = self._herdr_error_code(exc) - message = self._herdr_error_message(exc) - if code == "turn_epoch_mismatch": - try: - current_replay = call( - pane_id, - since=0, - expected_epoch=None, - ) - except (HerdrErrorResponse, AttributeError) as retry_exc: - self._record_turn_diagnostic( - "herdr_turn_replay_pane_skipped", - pane_id, - status=( - self._herdr_error_code(retry_exc) - if isinstance(retry_exc, HerdrErrorResponse) - else "method_unavailable" - ), - ) - return - self._record_completeness_break( - current_replay, - "turn_epoch_mismatch", - ) - return - if code == "invalid_params" and "newer than current turn" in message: - try: - current_replay = call( - pane_id, - since=0, - expected_epoch=None, - ) - except (HerdrErrorResponse, AttributeError) as retry_exc: - self._record_turn_diagnostic( - "herdr_turn_replay_pane_skipped", - pane_id, - status=( - self._herdr_error_code(retry_exc) - if isinstance(retry_exc, HerdrErrorResponse) - else "method_unavailable" - ), - ) - return - self._record_completeness_break(current_replay, "watermark_ahead") - return - self._record_turn_diagnostic( - "herdr_turn_replay_pane_skipped", - pane_id, - status=code or "pane_error", - ) - return - except AttributeError: - self._record_turn_diagnostic( - "herdr_turn_replay_pane_skipped", - pane_id, - status="method_unavailable", - ) - return - self._consume_replay(current_replay, watermark) - - def _replay_turns_after_reconcile(self, client: Any | None = None) -> None: - """Probe pane.turns once, then replay each pane independently.""" - if self.stop_event.is_set(): - return - pane_ids = tuple(self._subscription_pane_ids) - if not pane_ids: - self._turn_api_probed = True - self._turn_api_supported = False - return - watermarks = { - pane_id: get_herdr_turn_watermark( - self.db_path, - self.config.host_id, - pane_id, - ) - for pane_id in pane_ids - } - probe_pane_id: str | None = None - probe_replay: HerdrPaneTurnsReplay | None = None - probe_error: HerdrErrorResponse | AttributeError | None = None - if not self._turn_api_probed: - if self.stop_event.is_set(): - return - probe_pane_id = pane_ids[0] - supported, probe_replay, probe_error = self._probe_turn_api( - client, - probe_pane_id, - watermarks[probe_pane_id], - ) - self._turn_api_probed = True - self._turn_api_supported = supported - if not self._turn_api_supported: - return - for pane_id in pane_ids: - if self.stop_event.is_set(): - return - self._consume_pane_replay( - client, - pane_id, - watermarks[pane_id], - replay=probe_replay if pane_id == probe_pane_id else None, - error=probe_error if pane_id == probe_pane_id else None, - ) - def _subscribe_event_stream(self, client: Any) -> Any: # Herdr 0.7.5 strictly validates pane-scoped status subscriptions and # added the general pane.updated event. Use one bounded mixed @@ -1886,11 +1083,6 @@ def _subscribe_event_stream(self, client: Any) -> Any: {"type": "pane.agent_status_changed", "pane_id": pane_id} for pane_id in self._subscription_pane_ids ) - if self._turn_api_supported: - subscriptions.extend( - {"type": "pane.turn_completed", "pane_id": pane_id} - for pane_id in self._subscription_pane_ids - ) params = {"subscriptions": subscriptions} if hasattr(client, "subscribe"): try: @@ -2017,8 +1209,6 @@ def _commit_producer_identities(self, events: Sequence[NormalizedHerdrEvent]) -> def flush(self) -> None: # Draining, application, persistence, and producer-ID commitment share # one lock scope so later batches cannot overtake an earlier flush. - notify_turn_refresh = False - completed_turns: list[HerdrTurnCompletionRecord] = [] with self._lock: if not self._pending_events: return @@ -2026,21 +1216,6 @@ def flush(self) -> None: accepted_at = utc_timestamp() self._pending_events.clear() has_producer_identity = any(event.producer_identity is not None for event in events) - has_turn_refresh_event = any( - event.name in _TURN_REFRESH_EVENT_NAMES for event in events - ) - notify_turn_refresh = has_turn_refresh_event - for event in events: - if event.name != "pane.turn_completed": - continue - try: - completed_turns.append(_turn_completion_record(event.payload)) - except HerdrEnvelopeError: - self._record_turn_diagnostic( - "herdr_turn_completion_record_quarantined", - _first_text(event.payload, ("pane_id", "paneId")) - or "unknown", - ) try: self._event_continuity_revalidated = False changed = False @@ -2056,10 +1231,6 @@ def flush(self) -> None: self._mark_unhealthy("continuity_unavailable") finally: self._event_continuity_revalidated = False - for record in completed_turns: - self._process_turn_record(record) - if notify_turn_refresh: - self._notify_turn_refresh() def _apply_event(self, event: NormalizedHerdrEvent) -> bool: if event.name in _SPACE_EVENT_NAMES: diff --git a/src/tendwire/backends/herdr_protocol.py b/src/tendwire/backends/herdr_protocol.py index a035b58..2cbf4a4 100644 --- a/src/tendwire/backends/herdr_protocol.py +++ b/src/tendwire/backends/herdr_protocol.py @@ -23,7 +23,6 @@ ) HERDR_EVENTS_SUBSCRIBE_METHOD = "events.subscribe" -HERDR_TURN_COMPLETED_EVENT_NAME = "pane.turn_completed" HERDR_OFFICIAL_EVENT_NAMES = ( "workspace.created", "workspace.updated", @@ -282,7 +281,6 @@ def validate_response( envelope: Mapping[str, Any], *, allow_uncorrelated_error: bool = False, - allow_uncorrelated_method_error: bool = False, ) -> dict[str, Any]: """Validate a response envelope while tolerating unknown fields.""" if not is_response(envelope): @@ -301,20 +299,7 @@ def validate_response( and isinstance(error.get("message"), str) and error["message"].startswith("invalid request:") ) - # Stock Herdr 0.7.5 omits the id entirely when an internally tagged RPC - # variant is unknown. Only an explicit capability probe may opt into this - # narrower exception; ordinary requests remain strictly correlated. - uncorrelated_method_error = ( - allow_uncorrelated_method_error - and is_error_response(envelope) - and ("id" not in envelope or envelope.get("id") == "") - and isinstance(error, Mapping) - and error.get("code") == "invalid_request" - and isinstance(error.get("message"), str) - and error["message"].startswith("invalid request: unknown variant") - and "pane.turns" in error["message"] - ) - if not (uncorrelated_subscription_error or uncorrelated_method_error): + if not uncorrelated_subscription_error: _validated_id(envelope) return dict(envelope) @@ -342,14 +327,12 @@ def validate_server_envelope( envelope: Mapping[str, Any], *, allow_uncorrelated_error: bool = False, - allow_uncorrelated_method_error: bool = False, ) -> dict[str, Any]: """Validate a decoded server response or event envelope.""" if is_response(envelope): return validate_response( envelope, allow_uncorrelated_error=allow_uncorrelated_error, - allow_uncorrelated_method_error=allow_uncorrelated_method_error, ) if is_event(envelope): return validate_event(envelope) diff --git a/src/tendwire/backends/herdr_socket.py b/src/tendwire/backends/herdr_socket.py index e3aeca6..d76b143 100644 --- a/src/tendwire/backends/herdr_socket.py +++ b/src/tendwire/backends/herdr_socket.py @@ -152,19 +152,13 @@ def request( params: Mapping[str, Any] | None = None, *, timeout: float | None = None, - allow_uncorrelated: bool = False, ) -> Any: - """Send one request and return its raw result payload. - - ``allow_uncorrelated`` is reserved for an inert method-capability - probe; the default keeps every ordinary request strictly correlated. - """ + """Send one strictly correlated request and return its raw result payload.""" request_id, deadline = self._send_request(method, params, timeout=timeout) response = self._read_response( request_id, deadline=deadline, allow_uncorrelated_error=False, - allow_uncorrelated_method_error=allow_uncorrelated, ) if is_error_response(response): raise HerdrErrorResponse(error_payload(response), request_id) @@ -184,7 +178,6 @@ def subscribe( request_id, deadline=deadline, allow_uncorrelated_error=True, - allow_uncorrelated_method_error=False, ) if is_error_response(response): raise HerdrErrorResponse(error_payload(response), request_id) @@ -278,20 +271,6 @@ def pane_read( ) -> Any: return self.request("pane.read", params, timeout=timeout) - def pane_turns( - self, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - allow_uncorrelated: bool = False, - ) -> Any: - return self.request( - "pane.turns", - params, - timeout=timeout, - allow_uncorrelated=allow_uncorrelated, - ) - def agent_send( self, params: Mapping[str, Any] | None = None, @@ -410,13 +389,11 @@ def _read_response( *, deadline: float, allow_uncorrelated_error: bool, - allow_uncorrelated_method_error: bool, ) -> dict[str, Any]: while True: envelope = self._read_server_envelope( deadline=deadline, allow_uncorrelated_error=allow_uncorrelated_error, - allow_uncorrelated_method_error=allow_uncorrelated_method_error, ) if is_event(envelope): if len(self._pending_events) >= _MAX_PENDING_EVENTS: @@ -426,20 +403,10 @@ def _read_response( self._pending_events.append(envelope) continue if ( - (allow_uncorrelated_error or allow_uncorrelated_method_error) + allow_uncorrelated_error and is_error_response(envelope) and ( - ( - allow_uncorrelated_error - and envelope.get("id") == "" - ) - or ( - allow_uncorrelated_method_error - and ( - "id" not in envelope - or envelope.get("id") == "" - ) - ) + envelope.get("id") == "" ) ): # These narrowly validated Herdr 0.7.5 errors belong to the @@ -463,14 +430,12 @@ def _read_server_envelope( *, deadline: float, allow_uncorrelated_error: bool = False, - allow_uncorrelated_method_error: bool = False, ) -> dict[str, Any]: line = self._read_line(deadline=deadline) envelope = parse_json_line(line) return validate_server_envelope( envelope, allow_uncorrelated_error=allow_uncorrelated_error, - allow_uncorrelated_method_error=allow_uncorrelated_method_error, ) def _read_line(self, *, deadline: float) -> bytes: diff --git a/src/tendwire/backends/herdr_turns.py b/src/tendwire/backends/herdr_turns.py deleted file mode 100644 index 04524e5..0000000 --- a/src/tendwire/backends/herdr_turns.py +++ /dev/null @@ -1,4866 +0,0 @@ -"""Structured turn ingestion through Tendwire-owned private Herdr bindings.""" - -from __future__ import annotations - -import base64 -import hashlib -import json -import multiprocessing -import os -import re -import secrets -import select -import shlex -import socket -import stat -import struct -import subprocess -import threading -import time -from collections import OrderedDict, deque -from collections.abc import Callable, Mapping -from concurrent.futures import FIRST_COMPLETED, Future, ThreadPoolExecutor, wait -from dataclasses import dataclass, field, replace -from datetime import datetime -from pathlib import Path -from typing import Any, Literal -from uuid import UUID - -from ..config import Config -from ..core.models import WorkerBinding, stable_fingerprint, utc_timestamp -from ..core.turns import ( - InteractionChoice, - PendingObservation, - PendingObservedChoice, - is_internal_automation_turn_payload, - redact_private_prompt_text, -) -from ..store.sqlite import ( - apply_turn_refresh, - latest_turn_id_for_worker, - list_worker_bindings, - prune_backend_pending, -) - - -_TURN_CONTENT_KEYS = ( - "user_text", - "assistant_final_text", - "assistant_stream_text", - "model", - "complete", - "has_open_turn", - "awaiting_input", - "pending_decision", -) -_CODEX_SESSION_TURN_KIND = "codex_session_id" -_OMP_SESSION_TURN_KIND = "omp_session_path" -_PANE_TURN_KIND = "pane_id" -_MAX_CODEX_STREAM_MESSAGES = 4 -_CODEX_RECORD_MAX_BYTES = 8 * 1024 * 1024 -_CODEX_TURN_ID_MAX_BYTES = 1024 -_CODEX_READ_CHUNK_BYTES = 64 * 1024 -_CODEX_RESYNC_INITIAL_BYTES = 64 * 1024 -_CODEX_RESYNC_MAX_BYTES = 16 * 1024 * 1024 -_CODEX_RESYNC_MAX_RECORDS = 65_536 -_CODEX_INDEX_MAX_DEPTH = 4 -_CODEX_INDEX_MAX_VISITS = 100_000 -_CODEX_INDEX_MAX_ENTRIES = _CODEX_INDEX_MAX_VISITS -_CODEX_INDEX_MAX_BYTES = 16 * 1024 * 1024 -_CODEX_PATH_CACHE_CAPACITY = 256 -_CODEX_PATH_CACHE_MAX_BYTES = 256 * 1024 -_CODEX_NEGATIVE_TTL_SECONDS = 2.0 -# A found path is inode-validated on every hit. Duplicate discovery is -# intentionally bounded-stale by this complete-index refresh interval so the -# daemon never walks a 20k tree on each two-second poll. -_CODEX_POSITIVE_TTL_SECONDS = 60.0 -_CODEX_SESSION_CACHE_CAPACITY = 64 -_CODEX_SESSION_CACHE_MAX_BYTES = 16 * 1024 * 1024 -_CODEX_STATE_IPC_MAX_BYTES = 12 * 1024 * 1024 -_CODEX_IPC_FRAME_MAX_BYTES = 64 * 1024 * 1024 -_CODEX_POLL_MAX_BYTES = 64 * 1024 * 1024 -_OMP_IPC_RESPONSE_CHUNK_BYTES = 1024 * 1024 -_OMP_TAIL_BYTES = 786432 -_OMP_TOOL_SNIPPET_CHARS = 160 -_OMP_SESSION_CACHE_CAPACITY = 64 -_OMP_SESSION_CACHE_MAX_BYTES = 64 * 1024 -_OMP_REQUEST_MAX_BYTES = 16 * 1024 -_OMP_TARGET_MAX_CHARS = 4096 -_OMP_TEARDOWN_GRACE_SECONDS = 0.25 -_OMP_FRAME_HEADER = struct.Struct("!Q") -_UNCHANGED_TURN = object() -_PROMPTLESS_STATUS_FINAL_RE = re.compile( - r"\b(?:" - r"standing by|" - r"waiting(?: quietly)?|" - r"wait for|" - r"no new state|" - r"repeat tick|" - r"startup line|" - r"initial state|" - r"current state|" - r"monitor(?: reports|ing)?|" - r"review in progress|" - r"gate (?:phase|verdict|held|reached)|" - r"not yet (?:reached|materialized)|" - r"handoff|" - r"phase" - r")\b", - re.IGNORECASE, -) - - -@dataclass(frozen=True) -class _CodexPathResolution: - status: str - root: str - root_file_id: tuple[int, int] - session_id: str - canonical_path: str | None - relative_path: str | None - file_id: tuple[int, int] | None - generation: int - expires_at: float - - -@dataclass(frozen=True) -class _CodexIndexGeneration: - root: str - root_signature: tuple[int, int, int, int] - generation: int - built_at: float - entries: Mapping[str, tuple[str, ...]] - retained_bytes: int - visited: int - overflowed: bool - - -@dataclass(frozen=True) -class _CodexRecordSpan: - start: int - end: int - - -@dataclass(frozen=True) -class _CodexSessionState: - resolver_generation: int - root: str - session_id: str - canonical_path: str - file_id: tuple[int, int] | None - observed_size: int - mtime_ns: int - ctime_ns: int - committed_offset: int - partial_record: bytes - active_turn_id: str - last_content_turn_id: str - turn_open: bool - final_seen: bool - complete: bool - stream_spans: tuple[_CodexRecordSpan, ...] - internal_turn: bool = False - root_file_id: tuple[int, int] | None = None - - -@dataclass(frozen=True) -class _CodexSemanticEvent: - kind: str - turn_id: str - text: str = "" - - -@dataclass -class _CodexWorkState: - resolver_generation: int - root: str - session_id: str - canonical_path: str - file_id: tuple[int, int] - observed_size: int - mtime_ns: int - ctime_ns: int - committed_offset: int = 0 - partial_record: bytes = b"" - active_turn_id: str = "" - last_content_turn_id: str = "" - turn_open: bool = False - final_seen: bool = False - complete: bool = False - internal_turn: bool = False - root_file_id: tuple[int, int] | None = None - stream_items: list[tuple[_CodexRecordSpan, str]] = field(default_factory=list) - user_text: str | None = None - final_text: str | None = None - public_changed: bool = False - - -_CODEX_ROLLOUT_RE = re.compile( - r"^rollout-(\d{4})-(\d{2})-(\d{2})T" - r"(\d{2})-(\d{2})-(\d{2})-" - r"([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})" - r"\.jsonl$", - re.ASCII, -) -_CODEX_PATH_CACHE: OrderedDict[tuple[str, str], _CodexPathResolution] = OrderedDict() -_CODEX_PATH_CACHE_LOCK = threading.RLock() -_CODEX_INDEX_GENERATION: _CodexIndexGeneration | None = None -_CODEX_INDEX_GENERATION_COUNTER = 0 -_CODEX_RESOLUTION_GENERATION_COUNTER = 0 -_CODEX_INDEX_BUILD_OBSERVER: Callable[[int], None] | None = None -_CODEX_SESSION_CACHE: OrderedDict[tuple[str, str], _CodexSessionState] = OrderedDict() -_CODEX_SESSION_CACHE_LOCK = threading.RLock() -_CODEX_SESSION_CACHE_LIVE_KEYS: set[tuple[str, str]] | None = None -_CODEX_SESSION_CACHE_BINDING_GENERATIONS: dict[tuple[str, str], int] = {} -_CODEX_SESSION_CACHE_BINDING_FINGERPRINTS: dict[tuple[str, str], tuple[str, ...]] = {} -_CODEX_SESSION_CACHE_GENERATION_COUNTER = 0 -_CODEX_ISOLATED_READ_OBSERVER: Callable[[int], None] | None = None - - -@dataclass -class _OmpSessionState: - """Constant-size parser coordinates retained between polls.""" - - offset: int = 0 - observed_size: int = 0 - file_id: tuple[int, int] | None = None - mtime_ns: int = 0 - ctime_ns: int = 0 - replay_offset: int = 0 - turn_open: bool = False - project_root: Path | None = None - - -@dataclass -class _OmpTurnState: - """Canonical turn data confined to one parse and one response.""" - - prompt_id: str = "" - user_text: str = "" - stream_parts: list[str] = field(default_factory=list) - final_text: str = "" - tool_count: int = 0 - project_root: Path | None = None - - -@dataclass(frozen=True) -class _PublicOmpToolProgress: - """Progress assembled only from constants and proven-safe relative paths.""" - - action: str - subject: str | None = None - - def render(self, step: int) -> str: - body = f"{self.action}: {self.subject}" if self.subject else self.action - return f"step {step} · {body}" - - - - -_OMP_SESSION_CACHE: dict[str, _OmpSessionState] = {} -_OMP_SESSION_CACHE_LOCK = threading.RLock() -_OMP_SESSION_CACHE_LIVE_KEYS: set[str] | None = None -_OMP_SESSION_CACHE_BINDING_GENERATIONS: dict[str, int] = {} -_OMP_SESSION_CACHE_BINDING_FINGERPRINTS: dict[str, tuple[str, ...]] = {} -_OMP_SESSION_CACHE_GENERATION_COUNTER = 0 -_OMP_ISOLATED_READ_OBSERVER: Callable[[int], None] | None = None - - -def _omp_cache_get_locked(cache_key: str) -> _OmpSessionState | None: - state = _OMP_SESSION_CACHE.pop(cache_key, None) - if state is not None: - _OMP_SESSION_CACHE[cache_key] = state - return state - - -def _omp_cache_store_locked(cache_key: str, state: _OmpSessionState) -> None: - _OMP_SESSION_CACHE.pop(cache_key, None) - _OMP_SESSION_CACHE[cache_key] = state - while _OMP_SESSION_CACHE and ( - len(_OMP_SESSION_CACHE) > _OMP_SESSION_CACHE_CAPACITY - or _omp_cache_weight_locked() > _OMP_SESSION_CACHE_MAX_BYTES - ): - del _OMP_SESSION_CACHE[next(iter(_OMP_SESSION_CACHE))] - - -def _omp_cache_weight_locked() -> int: - return sum( - len(cache_key.encode("utf-8")) - + len( - json.dumps( - _serialize_omp_state(state), - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - ) - for cache_key, state in _OMP_SESSION_CACHE.items() - ) - - -def _omp_cache_binding_generation_locked(cache_key: str) -> int | None: - if _OMP_SESSION_CACHE_LIVE_KEYS is None: - return None - return _OMP_SESSION_CACHE_BINDING_GENERATIONS.get(cache_key) - - -def _serialize_omp_state(state: _OmpSessionState | None) -> dict[str, Any] | None: - if state is None: - return None - return { - "offset": state.offset, - "observed_size": state.observed_size, - "file_id": list(state.file_id) if state.file_id is not None else None, - "mtime_ns": state.mtime_ns, - "ctime_ns": state.ctime_ns, - "replay_offset": state.replay_offset, - "turn_open": state.turn_open, - "project_root": os.fspath(state.project_root) if state.project_root is not None else None, - } - - -def _deserialize_omp_state(value: Any) -> _OmpSessionState | None: - if value is None: - return None - if not isinstance(value, Mapping) or set(value) != { - "offset", - "observed_size", - "file_id", - "mtime_ns", - "ctime_ns", - "replay_offset", - "turn_open", - "project_root", - }: - raise ValueError("invalid OMP parser state") - offset = value["offset"] - observed_size = value["observed_size"] - file_id_value = value["file_id"] - mtime_ns = value["mtime_ns"] - ctime_ns = value["ctime_ns"] - replay_offset = value["replay_offset"] - turn_open = value["turn_open"] - project_root_value = value["project_root"] - if ( - type(offset) is not int - or offset < 0 - or type(observed_size) is not int - or observed_size < offset - or type(replay_offset) is not int - or replay_offset < 0 - or replay_offset > offset - or type(turn_open) is not bool - or type(mtime_ns) is not int - or mtime_ns < 0 - or type(ctime_ns) is not int - or ctime_ns < 0 - ): - raise ValueError("invalid OMP coordinates") - if ( - file_id_value is not None - and ( - not isinstance(file_id_value, (list, tuple)) - or len(file_id_value) != 2 - or any(type(part) is not int or part < 0 for part in file_id_value) - ) - ): - raise ValueError("invalid OMP file identity") - if project_root_value is not None and type(project_root_value) is not str: - raise ValueError("invalid OMP project root") - return _OmpSessionState( - offset=offset, - observed_size=observed_size, - file_id=tuple(file_id_value) if file_id_value is not None else None, - mtime_ns=mtime_ns, - ctime_ns=ctime_ns, - replay_offset=replay_offset, - turn_open=turn_open, - project_root=Path(project_root_value) if project_root_value is not None else None, - ) - - -def _extract_turn_payload(value: Any) -> Mapping[str, Any] | None: - if not isinstance(value, Mapping): - return None - result = value.get("result") - if isinstance(result, Mapping) and isinstance(result.get("turn"), Mapping): - return result["turn"] - if isinstance(value.get("turn"), Mapping): - return value["turn"] - return value - - -# Every driven ordinal must stay a single keystroke (digits select/toggle -# absolutely in the pane, live-verified on Claude Code 2.1.211), so 9 is the -# hard bound; larger decisions fail closed to the read-only interaction. -PENDING_DECISION_MAX_OPTIONS = 9 -_PENDING_TEXT_MAX = 2000 -_SINGLE_WRITE_IN_OPTION_IDS = frozenset( - {"custom", "other", "writein", "write_in", "write-in"} -) - - -def _private_pending_revision(value: Mapping[str, Any]) -> str: - encoded = json.dumps( - value, - sort_keys=True, - separators=(",", ":"), - ensure_ascii=False, - default=str, - ).encode("utf-8") - return hashlib.sha256(encoded).hexdigest()[:24] - - -def _pending_observation_from_turn(turn: Mapping[str, Any]) -> PendingObservation: - """Build one explicit private-neutral pending observation from a pane read.""" - decision = turn.get("pending_decision") - if decision is not None: - if not isinstance(decision, Mapping): - return PendingObservation("read_succeeded_invalid_prompt") - revision = _private_pending_revision(decision) - question = redact_private_prompt_text( - decision.get("prompt") or decision.get("question"), - max_chars=_PENDING_TEXT_MAX, - ) - options = decision.get("options") - if options is None: - options = [] - if not isinstance(options, list): - return PendingObservation("read_succeeded_invalid_prompt") - # A single-choice Claude prompt may have one trailing write-in row in - # addition to the digit-addressable options. Validate the effective - # selectable rows after the decision kind and write-in shape are known. - if len(options) > PENDING_DECISION_MAX_OPTIONS + 1: - return PendingObservation("read_succeeded_unsupported_decision") - choices: list[PendingObservedChoice] = [] - for ordinal, option in enumerate(options, 1): - if not isinstance(option, Mapping): - return PendingObservation("read_succeeded_invalid_prompt") - label = redact_private_prompt_text( - option.get("label"), - max_chars=_PENDING_TEXT_MAX, - ) - if not label: - return PendingObservation("read_succeeded_invalid_prompt") - label = InteractionChoice(label=label).label - choices.append( - PendingObservedChoice( - choice_id=f"choice-{stable_fingerprint({'revision': revision, 'ordinal': ordinal, 'label': label})}", - label=label, - picker_ordinal=ordinal, - ) - ) - if not question: - return PendingObservation("read_succeeded_invalid_prompt") - option_ids = { - str(option.get("id") or "") - for option in options - if isinstance(option, Mapping) - } - raw_kind = str( - decision.get("kind") - or decision.get("tool_name") - or decision.get("name") - or "" - ).strip().lower().replace("-", "_") - compact_kind = raw_kind.replace("_", "") - raw_mode = ( - str(decision.get("mode") or "") - .strip() - .lower() - .replace("-", "_") - ) - compact_mode = raw_mode.replace("_", "") - if compact_kind not in { - "", - "askuserquestion", - "single", - "multi", - "multiselect", - "exitplanmode", - "plan", - } or compact_mode not in { - "", - "buttons", - "single", - "multi", - "multiselect", - "plan", - }: - return PendingObservation("read_succeeded_unsupported_decision") - raw_multi_select = decision.get( - "multi_select", - decision.get("multiSelect", False), - ) - if not isinstance(raw_multi_select, bool): - return PendingObservation("read_succeeded_invalid_prompt") - if ( - compact_mode == "plan" - or compact_kind in {"exitplanmode", "plan"} - or (not compact_kind and "approve" in option_ids) - ): - decision_kind: Literal["single", "multi", "plan"] = "plan" - elif ( - compact_mode in {"multi", "multiselect"} - or raw_multi_select - or compact_kind in {"multi", "multiselect"} - ): - decision_kind = "multi" - else: - decision_kind = "single" - if raw_multi_select is not (decision_kind == "multi"): - return PendingObservation("read_succeeded_unsupported_decision") - raw_question_count = decision.get("question_count") - if raw_question_count is None: - raw_questions = decision.get("questions") - raw_question_count = ( - len(raw_questions) if isinstance(raw_questions, list) else 1 - ) - if ( - not isinstance(raw_question_count, int) - or isinstance(raw_question_count, bool) - or raw_question_count < 1 - ): - return PendingObservation("read_succeeded_invalid_prompt") - decision_option_labels = [choice.label for choice in choices] - if ( - decision_kind == "single" - and options - and isinstance(options[len(decision_option_labels) - 1], Mapping) - and str( - options[len(decision_option_labels) - 1].get("id") or "" - ).strip().lower() - in _SINGLE_WRITE_IN_OPTION_IDS - ): - decision_option_labels.pop() - decision_options = tuple(decision_option_labels) - if not decision_options: - return PendingObservation("read_succeeded_invalid_prompt") - if len(decision_options) > PENDING_DECISION_MAX_OPTIONS: - return PendingObservation("read_succeeded_unsupported_decision") - return PendingObservation( - "open_prompt", - question=question, - pending_kind="approval" if "approve" in option_ids else "question", - choices=tuple(choices), - revision_digest=revision, - decision_kind=decision_kind, - decision_options=decision_options, - decision_multi_select=decision_kind == "multi", - decision_question_count=raw_question_count, - ) - interaction = turn.get("pending_interaction") - if interaction is not None: - if not isinstance(interaction, Mapping): - return PendingObservation("read_succeeded_invalid_prompt") - questions = interaction.get("questions") - if questions is None: - questions = [] - if not isinstance(questions, list): - return PendingObservation("read_succeeded_invalid_prompt") - parts: list[str] = [] - for item in questions[:4]: - if not isinstance(item, Mapping): - return PendingObservation("read_succeeded_invalid_prompt") - part = redact_private_prompt_text(item.get("question")) - if part: - parts.append(part) - if not parts: - return PendingObservation("read_succeeded_invalid_prompt") - return PendingObservation( - "open_prompt", - question=" / ".join(parts)[:_PENDING_TEXT_MAX], - pending_kind="review", - revision_digest=_private_pending_revision(interaction), - ) - return PendingObservation("read_succeeded_no_prompt") - - -def _backend_pending_from_turn(turn: Mapping[str, Any]) -> dict[str, Any] | None: - """Compatibility public projection of the explicit observation.""" - observation = _pending_observation_from_turn(turn) - if observation.kind != "open_prompt": - return None - meta: dict[str, Any] = {"source": "backend"} - if observation.decision_kind is not None: - meta["decision"] = { - "decision_ref": ( - "decision-" - + stable_fingerprint( - {"decision_revision": observation.revision_digest} - ) - ), - "kind": observation.decision_kind, - "prompt": observation.question, - "options": [ - {"ref": str(ordinal), "label": label} - for ordinal, label in enumerate(observation.decision_options, 1) - ], - "multi_select": observation.decision_multi_select, - "question_count": observation.decision_question_count, - } - return { - "question": observation.question, - "kind": observation.pending_kind or "question", - "choices": [ - {"choice_id": choice.choice_id, "label": choice.label} - for choice in observation.choices - ], - "meta": meta, - } - - -def _public_turn_pending_projection( - observation: PendingObservation, -) -> dict[str, Any]: - """Return the private-neutral decision fields persisted with an open turn.""" - if observation.kind != "open_prompt" or observation.decision_kind is None: - return {} - mode = { - "single": "buttons", - "multi": "multi", - "plan": "plan", - }[observation.decision_kind] - return { - "awaiting_input": True, - "pending_decision": { - "prompt": observation.question, - "mode": mode, - "options": [ - {"id": str(ordinal), "label": label} - for ordinal, label in enumerate(observation.decision_options, 1) - ], - "multi_select": observation.decision_multi_select, - "question_count": observation.decision_question_count, - }, - } - - -def _backend_terminal_error_text(turn: Mapping[str, Any]) -> str | None: - """Return a bounded public final for a structured terminal adapter error.""" - - error = turn.get("api_error") - if not isinstance(error, Mapping): - return None - text = redact_private_prompt_text(error.get("text"), max_chars=600) - if text: - return text - code = redact_private_prompt_text(error.get("code"), max_chars=120) - if code: - return f"The agent ended this turn with an API error ({code})." - return "The agent ended this turn with an API error." - - -class _TurnReadTimeout(Exception): - """Fixed internal timeout signal; never serialized with private details.""" - - -class _TurnReadFailed(Exception): - """Fixed internal adapter failure signal; never serialized with raw errors.""" - - -def _read_private_turn( - config: Config, - pane_id: str, - *, - timeout_seconds: float | None = None, - raise_timeout: bool = False, - cancel_event: threading.Event | None = None, -) -> Mapping[str, Any] | None: - argv = [ - config.herdr_bin, - "pane", - "turn", - pane_id, - "--last", - "--format", - "json", - ] - timeout = config.herdr_timeout_seconds if timeout_seconds is None else timeout_seconds - try: - if cancel_event is None: - completed = subprocess.run( - argv, - capture_output=True, - text=True, - check=False, - timeout=timeout, - ) - else: - process = subprocess.Popen( - argv, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - text=True, - ) - deadline = time.monotonic() + float(timeout) - while True: - try: - stdout, stderr = process.communicate( - timeout=max(0.001, min(0.05, deadline - time.monotonic())) - ) - completed = subprocess.CompletedProcess( - argv, - process.returncode, - stdout, - stderr, - ) - break - except subprocess.TimeoutExpired: - if not cancel_event.is_set() and time.monotonic() < deadline: - continue - process.terminate() - try: - process.wait(0.25) - except subprocess.TimeoutExpired: - process.kill() - process.communicate() - raise _TurnReadTimeout from None - except subprocess.TimeoutExpired: - if raise_timeout: - raise _TurnReadTimeout from None - return None - except _TurnReadTimeout: - if raise_timeout: - raise - return None - except (OSError, UnicodeDecodeError, ValueError): - if raise_timeout: - raise _TurnReadFailed from None - return None - if completed.returncode != 0: - if raise_timeout: - raise _TurnReadFailed - return None - try: - payload = json.loads(completed.stdout) - except (json.JSONDecodeError, TypeError, ValueError): - if raise_timeout: - raise _TurnReadFailed from None - return None - turn = _extract_turn_payload(payload) - if not isinstance(turn, Mapping): - if raise_timeout: - raise _TurnReadFailed - return None - pending_observation = _pending_observation_from_turn(turn) - pending_projection = _public_turn_pending_projection(pending_observation) - if turn.get("available") is False: - return ( - {"_backend_pending_observation": pending_observation} - if raise_timeout - else None - ) - open_user_text = turn.get("open_user_text") - open_turn_id = str(turn.get("open_turn_id") or "").strip() - if turn.get("has_open_turn") and (open_user_text or open_turn_id): - # An in-progress turn is reported alongside the last completed one via - # open_* fields. Emit it as its own open turn keyed by the stable prompt - # id so the connector streams a live "working" card that later edits - # into the final once this same id completes. - opened = _open_turn_content( - open_user_text, - turn.get("assistant_stream_text"), - open_turn_id, - ) - terminal_error = _backend_terminal_error_text(turn) - if opened is not None and terminal_error: - opened = { - **dict(opened), - "assistant_stream_text": None, - "assistant_final_text": terminal_error, - "complete": True, - "has_open_turn": False, - } - if raise_timeout: - opened_data = dict(opened or {}) - opened_data.update(pending_projection) - opened_data["_backend_pending_observation"] = pending_observation - return opened_data - - return opened - - content = {key: turn.get(key) for key in _TURN_CONTENT_KEYS if key in turn} - terminal_error = _backend_terminal_error_text(turn) - if terminal_error: - content.update( - { - "assistant_stream_text": None, - "assistant_final_text": terminal_error, - "complete": True, - "has_open_turn": False, - } - ) - content.update(pending_projection) - if raise_timeout: - content["_backend_pending_observation"] = pending_observation - # Prefer the stable prompt-scoped id so a turn keeps one identity from - # open through complete; fall back to turn_id for backends without it. - source_turn_id = str(turn.get("source_turn_id") or turn.get("turn_id") or "").strip() - if source_turn_id: - content["source_turn_id"] = source_turn_id[:160] - if _is_internal_turn_content(content): - return ( - {"_backend_pending_observation": pending_observation} - if raise_timeout - else None - ) - # Never clobber stored text with an empty value: notification-triggered - # turns legitimately carry no prompt, but the previous real prompt and - # stream must survive the merge. - for key in ("user_text", "assistant_final_text", "assistant_stream_text"): - if key in content and not (content.get(key) or "").strip(): - content.pop(key) - if not any(value not in (None, "", False) for value in content.values()): - return None - return content - - -def _pending_public_payload( - observation: PendingObservation, -) -> dict[str, Any] | None: - if observation.kind != "open_prompt": - return None - return { - "question": observation.question, - "kind": observation.pending_kind or "question", - "choices": [ - {"choice_id": choice.choice_id, "label": choice.label} - for choice in observation.choices - ], - "meta": {"source": "backend"}, - } - - -def _pop_backend_pending_observation( - content: Mapping[str, Any] | None, -) -> tuple[dict[str, Any] | None, PendingObservation | None]: - if content is None: - return None, None - data = dict(content) - observation = data.pop("_backend_pending_observation", None) - data.pop("_backend_pending", None) - if not any(value not in (None, "", False) for value in data.values()): - data = None - return ( - data, - observation if isinstance(observation, PendingObservation) else None, - ) - - -def _pop_backend_pending(content: Mapping[str, Any] | None) -> tuple[dict[str, Any] | None, dict[str, Any] | None]: - """Split the reserved _backend_pending key off a turn-content read. Returns (content, pending); - content becomes None when nothing else remains.""" - if content is None: - return None, None - data = dict(content) - pending = data.pop("_backend_pending", None) - if not any(value not in (None, "", False) for value in data.values()): - data = None - return data, pending if isinstance(pending, dict) else None - - -def _open_turn_content( - open_user_text: Any, - stream_text: Any, - open_turn_id: str, -) -> Mapping[str, Any] | None: - if isinstance(open_user_text, str) and _is_internal_user_text(open_user_text): - return None - content: dict[str, Any] = { - "assistant_final_text": None, - "complete": False, - "has_open_turn": True, - } - if isinstance(open_user_text, str) and open_user_text.strip(): - content["user_text"] = open_user_text - if isinstance(stream_text, str) and stream_text.strip(): - content["assistant_stream_text"] = stream_text - if open_turn_id: - content["source_turn_id"] = open_turn_id[:160] - if _is_internal_turn_content(content): - return None - if not (content.get("user_text") or content.get("assistant_stream_text")): - return None - return content - - -def _codex_home() -> Path: - raw = os.environ.get("CODEX_HOME") - if raw: - return Path(raw).expanduser() - return Path.home() / ".codex" - - -def _canonical_codex_session_id(value: Any) -> str | None: - if type(value) is not str or len(value) != 36 or not value.isascii(): - return None - try: - parsed = UUID(value) - except (ValueError, AttributeError): - return None - if parsed.int == 0 or str(parsed) != value: - return None - return value - - -def _codex_rollout_identity( - relative_date: tuple[str, str, str], - basename: str, -) -> str | None: - if len(relative_date) != 3: - return None - match = _CODEX_ROLLOUT_RE.fullmatch(basename) - if match is None: - return None - year, month, day, hour, minute, second, session_id = match.groups() - if relative_date != (year, month, day): - return None - try: - datetime( - int(year), - int(month), - int(day), - int(hour), - int(minute), - int(second), - ) - except ValueError: - return None - return _canonical_codex_session_id(session_id) - - -class _CodexIndexLimit(Exception): - pass - - -def _resolve_codex_sessions_root() -> Path: - lexical_root = _codex_home() / "sessions" - flags = ( - os.O_RDONLY - | getattr(os, "O_DIRECTORY", 0) - | getattr(os, "O_CLOEXEC", 0) - | getattr(os, "O_NOFOLLOW", 0) - ) - try: - descriptor = os.open(lexical_root, flags) - except OSError as exc: - raise OSError("Codex sessions root is unavailable") from exc - try: - opened = os.fstat(descriptor) - root = lexical_root.resolve(strict=True) - current = root.lstat() - if ( - not stat.S_ISDIR(opened.st_mode) - or not stat.S_ISDIR(current.st_mode) - or (int(opened.st_dev), int(opened.st_ino)) - != (int(current.st_dev), int(current.st_ino)) - ): - raise OSError("Codex sessions root changed during resolution") - return root - finally: - os.close(descriptor) - - -def _codex_root_signature(root: Path) -> tuple[int, int, int, int]: - value = root.lstat() - if not stat.S_ISDIR(value.st_mode): - raise OSError("Codex sessions root is not a directory") - return ( - int(value.st_dev), - int(value.st_ino), - int(value.st_mtime_ns), - int(value.st_ctime_ns), - ) - - - - - - -def _build_codex_index(root: Path) -> _CodexIndexGeneration: - global _CODEX_INDEX_GENERATION_COUNTER - entries: dict[str, tuple[str, ...]] = {} - retained_bytes = 0 - visited = 0 - - def visit() -> None: - nonlocal visited - visited += 1 - if visited > _CODEX_INDEX_MAX_VISITS: - raise _CodexIndexLimit - def scan(path: str | os.PathLike[str]) -> list[Any]: - iterator = os.scandir(path) - items: list[Any] = [] - try: - for item in iterator: - visit() - items.append(item) - finally: - close = getattr(iterator, "close", None) - if callable(close): - close() - items.sort(key=lambda item: item.name) - return items - - - try: - for year_entry in scan(root): - if ( - len(year_entry.name) != 4 - or not year_entry.name.isascii() - or not year_entry.name.isdecimal() - or not year_entry.is_dir(follow_symlinks=False) - ): - continue - for month_entry in scan(year_entry.path): - if ( - len(month_entry.name) != 2 - or not month_entry.name.isascii() - or not month_entry.name.isdecimal() - or not month_entry.is_dir(follow_symlinks=False) - ): - continue - for day_entry in scan(month_entry.path): - if ( - len(day_entry.name) != 2 - or not day_entry.name.isascii() - or not day_entry.name.isdecimal() - or not day_entry.is_dir(follow_symlinks=False) - ): - continue - date_parts = ( - year_entry.name, - month_entry.name, - day_entry.name, - ) - for file_entry in scan(day_entry.path): - if not file_entry.is_file(follow_symlinks=False): - continue - session_id = _codex_rollout_identity( - date_parts, - file_entry.name, - ) - if session_id is None: - continue - relative_path = "/".join((*date_parts, file_entry.name)) - previous = entries.get(session_id, ()) - if relative_path in previous: - continue - retained = (*previous, relative_path) - added_weight = len(relative_path.encode("utf-8")) - if not previous: - added_weight += len(session_id) - if ( - len(entries) + (0 if previous else 1) - > _CODEX_INDEX_MAX_ENTRIES - or retained_bytes + added_weight - > _CODEX_INDEX_MAX_BYTES - ): - raise _CodexIndexLimit - entries[session_id] = retained - retained_bytes += added_weight - overflowed = False - except _CodexIndexLimit: - entries = {} - retained_bytes = 0 - overflowed = True - _CODEX_INDEX_GENERATION_COUNTER += 1 - generation = _CodexIndexGeneration( - root=os.fspath(root), - root_signature=_codex_root_signature(root), - generation=_CODEX_INDEX_GENERATION_COUNTER, - built_at=time.monotonic(), - entries=entries, - retained_bytes=retained_bytes, - visited=visited, - overflowed=overflowed, - ) - observer = _CODEX_INDEX_BUILD_OBSERVER - if observer is not None: - observer(visited) - return generation - - -def _codex_path_cache_weight_locked() -> int: - return sum( - len(root.encode("utf-8")) - + len(session_id) - + len((entry.relative_path or "").encode("utf-8")) - + 96 - for (root, session_id), entry in _CODEX_PATH_CACHE.items() - ) - - -def _codex_path_cache_store_locked( - key: tuple[str, str], - resolution: _CodexPathResolution, -) -> None: - _CODEX_PATH_CACHE.pop(key, None) - _CODEX_PATH_CACHE[key] = resolution - while _CODEX_PATH_CACHE and ( - len(_CODEX_PATH_CACHE) > _CODEX_PATH_CACHE_CAPACITY - or _codex_path_cache_weight_locked() > _CODEX_PATH_CACHE_MAX_BYTES - ): - _CODEX_PATH_CACHE.popitem(last=False) - - -def _codex_resolution_from_index_locked( - root: Path, - session_id: str, - index: _CodexIndexGeneration, - previous: _CodexPathResolution | None, -) -> _CodexPathResolution: - global _CODEX_RESOLUTION_GENERATION_COUNTER - relative_paths = index.entries.get(session_id, ()) - status = ( - "index_limit" - if index.overflowed - else "missing" - if not relative_paths - else "ambiguous" - if len(relative_paths) != 1 - else "found" - ) - canonical_path: str | None = None - relative_path: str | None = None - file_id: tuple[int, int] | None = None - if status == "found": - relative_path = relative_paths[0] - candidate = root.joinpath(*relative_path.split("/")) - try: - path_stat = candidate.lstat() - resolved = candidate.resolve(strict=True) - resolved.relative_to(root) - if ( - not stat.S_ISREG(path_stat.st_mode) - or resolved != candidate - or (int(path_stat.st_dev), int(path_stat.st_ino)) - != (int(resolved.stat().st_dev), int(resolved.stat().st_ino)) - ): - status = "unsafe_path" - else: - canonical_path = os.fspath(resolved) - file_id = (int(path_stat.st_dev), int(path_stat.st_ino)) - except (OSError, ValueError): - status = "unsafe_path" - same_resolution = bool( - previous is not None - and previous.status == status - and previous.root_file_id == index.root_signature[:2] - and previous.canonical_path == canonical_path - and previous.relative_path == relative_path - and previous.file_id == file_id - ) - if same_resolution: - generation = previous.generation - else: - _CODEX_RESOLUTION_GENERATION_COUNTER += 1 - generation = _CODEX_RESOLUTION_GENERATION_COUNTER - return _CodexPathResolution( - status=status, - root=os.fspath(root), - root_file_id=(index.root_signature[0], index.root_signature[1]), - session_id=session_id, - canonical_path=canonical_path, - relative_path=relative_path, - file_id=file_id, - generation=generation, - expires_at=time.monotonic() - + ( - _CODEX_POSITIVE_TTL_SECONDS - if status == "found" - else _CODEX_NEGATIVE_TTL_SECONDS - ), - ) - - -def _resolve_codex_session(session_id: Any) -> _CodexPathResolution | None: - canonical_id = _canonical_codex_session_id(session_id) - if canonical_id is None: - return None - try: - root = _resolve_codex_sessions_root() - root_signature = _codex_root_signature(root) - except OSError: - return None - key = (os.fspath(root), canonical_id) - now = time.monotonic() - global _CODEX_INDEX_GENERATION - with _CODEX_PATH_CACHE_LOCK: - cached = _CODEX_PATH_CACHE.get(key) - index = _CODEX_INDEX_GENERATION - invalidated = False - current_root_id = root_signature[:2] - root_changed = bool( - (cached is not None and cached.root_file_id != current_root_id) - or ( - index is not None - and index.root == os.fspath(root) - and index.root_signature[:2] != current_root_id - ) - ) - if root_changed: - for path_key in tuple(_CODEX_PATH_CACHE): - if path_key[0] == os.fspath(root): - del _CODEX_PATH_CACHE[path_key] - _CODEX_INDEX_GENERATION = None - cached = None - index = None - invalidated = True - if cached is not None and cached.expires_at > now: - if cached.status != "found": - _CODEX_PATH_CACHE.move_to_end(key) - return cached - try: - current = Path(cached.canonical_path or "").lstat() - except OSError: - invalidated = True - else: - if ( - stat.S_ISREG(current.st_mode) - and (int(current.st_dev), int(current.st_ino)) == cached.file_id - ): - _CODEX_PATH_CACHE.move_to_end(key) - return cached - invalidated = True - rebuild = bool( - invalidated - or index is None - or index.root != os.fspath(root) - or index.root_signature != root_signature - or now - index.built_at >= _CODEX_POSITIVE_TTL_SECONDS - ) - if rebuild: - index = _build_codex_index(root) - _CODEX_INDEX_GENERATION = index - assert index is not None - resolution = _codex_resolution_from_index_locked( - root, - canonical_id, - index, - cached, - ) - _codex_path_cache_store_locked(key, resolution) - return resolution - - -def _find_codex_session_file(session_id: Any) -> Path | None: - resolution = _resolve_codex_session(session_id) - if resolution is None or resolution.status != "found": - return None - return Path(resolution.canonical_path or "") - - -def _payload_turn_id(payload: Mapping[str, Any]) -> str: - raw = payload.get("turn_id") - if isinstance(raw, str) and raw.strip(): - return raw.strip() - metadata = payload.get("internal_chat_message_metadata_passthrough") - if isinstance(metadata, Mapping): - raw = metadata.get("turn_id") - if isinstance(raw, str) and raw.strip(): - return raw.strip() - return "" - - -def _message_text(payload: Mapping[str, Any]) -> str: - content = payload.get("content") - if isinstance(content, list): - parts: list[str] = [] - for item in content: - if not isinstance(item, Mapping): - continue - text = item.get("text") - if isinstance(text, str) and text.strip(): - parts.append(text) - return "\n".join(parts) - text = payload.get("message") - if isinstance(text, str): - return text - return "" - - -_INTERNAL_USER_TEXT_PREFIXES = ( - "", - "", - "", - "", - "", - "", - "", - "", - "Caveat: The messages below were generated by the user while running local commands.", -) - -def _is_internal_user_text(text: str) -> bool: - clean = text.lstrip().replace("\r\n", "\n") - return clean.startswith(_INTERNAL_USER_TEXT_PREFIXES) or is_internal_automation_turn_payload( - {"user_text": clean} - ) - - -def _is_internal_turn_content(content: Mapping[str, Any]) -> bool: - user_text = content.get("user_text") - if isinstance(user_text, str) and _is_internal_user_text(user_text): - return True - if _is_promptless_status_final(content): - return True - return is_internal_automation_turn_payload(content) - - -def _is_promptless_status_final(content: Mapping[str, Any]) -> bool: - if str(content.get("user_text") or "").strip(): - return False - if content.get("complete") is False or content.get("has_open_turn") is True: - return False - final_text = str(content.get("assistant_final_text") or "").strip() - if not final_text or len(final_text) > 800: - return False - return bool(_PROMPTLESS_STATUS_FINAL_RE.search(final_text)) - - -def _append_unique_recent(items: list[str], text: str) -> None: - clean = text.strip() - if not clean: - return - if clean in items: - items.remove(clean) - items.append(clean) - if len(items) > _MAX_CODEX_STREAM_MESSAGES: - del items[: len(items) - _MAX_CODEX_STREAM_MESSAGES] - - -def _open_verified_codex_file( - resolution: _CodexPathResolution, -) -> tuple[int, os.stat_result]: - if ( - resolution.status != "found" - or resolution.canonical_path is None - or resolution.file_id is None - ): - raise _TurnReadFailed - root = Path(resolution.root) - candidate = Path(resolution.canonical_path) - try: - before = candidate.lstat() - resolved = candidate.resolve(strict=True) - resolved.relative_to(root) - except (OSError, ValueError) as exc: - raise _TurnReadFailed from exc - before_id = (int(before.st_dev), int(before.st_ino)) - if ( - not stat.S_ISREG(before.st_mode) - or resolved != candidate - or before_id != resolution.file_id - ): - raise _TurnReadFailed - if resolution.relative_path is None: - raise _TurnReadFailed - relative_parts = tuple(resolution.relative_path.split("/")) - if len(relative_parts) != _CODEX_INDEX_MAX_DEPTH: - raise _TurnReadFailed - nofollow = getattr(os, "O_NOFOLLOW", 0) - close_on_exec = getattr(os, "O_CLOEXEC", 0) - directory_flags = ( - os.O_RDONLY - | getattr(os, "O_DIRECTORY", 0) - | close_on_exec - | nofollow - ) - file_flags = os.O_RDONLY | close_on_exec | nofollow - directory_fd: int | None = None - try: - directory_fd = os.open(root, directory_flags) - root_stat = os.fstat(directory_fd) - if (int(root_stat.st_dev), int(root_stat.st_ino)) != resolution.root_file_id: - raise _TurnReadFailed - for component in relative_parts[:-1]: - next_fd = os.open( - component, - directory_flags, - dir_fd=directory_fd, - ) - os.close(directory_fd) - directory_fd = next_fd - descriptor = os.open( - relative_parts[-1], - file_flags, - dir_fd=directory_fd, - ) - except OSError as exc: - raise _TurnReadFailed from exc - finally: - if directory_fd is not None: - os.close(directory_fd) - try: - opened = os.fstat(descriptor) - after = candidate.stat() - opened_id = (int(opened.st_dev), int(opened.st_ino)) - if ( - not stat.S_ISREG(opened.st_mode) - or opened_id != before_id - or opened_id != (int(after.st_dev), int(after.st_ino)) - ): - raise _TurnReadFailed - return descriptor, opened - except BaseException: - os.close(descriptor) - raise - - -def _serialize_codex_state(state_value: _CodexSessionState | None) -> dict[str, Any] | None: - if state_value is None: - return None - value = { - "resolver_generation": state_value.resolver_generation, - "root": state_value.root, - "root_file_id": list(state_value.root_file_id) if state_value.root_file_id is not None else None, - "session_id": state_value.session_id, - "canonical_path": state_value.canonical_path, - "file_id": list(state_value.file_id) if state_value.file_id is not None else None, - "observed_size": state_value.observed_size, - "mtime_ns": state_value.mtime_ns, - "ctime_ns": state_value.ctime_ns, - "committed_offset": state_value.committed_offset, - "partial_record_b64": base64.b64encode(state_value.partial_record).decode("ascii"), - "active_turn_id": state_value.active_turn_id, - "last_content_turn_id": state_value.last_content_turn_id, - "turn_open": state_value.turn_open, - "final_seen": state_value.final_seen, - "complete": state_value.complete, - "internal_turn": state_value.internal_turn, - "stream_spans": [ - [span.start, span.end] - for span in state_value.stream_spans - ], - } - encoded = json.dumps( - value, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - if len(encoded) > _CODEX_STATE_IPC_MAX_BYTES: - raise ValueError("Codex parser state exceeds IPC limit") - return value - - -def _deserialize_codex_state(value: Any) -> _CodexSessionState | None: - if value is None: - return None - expected = { - "resolver_generation", - "root", - "root_file_id", - "session_id", - "canonical_path", - "file_id", - "observed_size", - "mtime_ns", - "ctime_ns", - "committed_offset", - "partial_record_b64", - "active_turn_id", - "last_content_turn_id", - "turn_open", - "final_seen", - "complete", - "internal_turn", - "stream_spans", - } - if not isinstance(value, Mapping) or set(value) != expected: - raise ValueError("invalid Codex parser state") - encoded = json.dumps( - value, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - if len(encoded) > _CODEX_STATE_IPC_MAX_BYTES: - raise ValueError("oversized Codex parser state") - resolver_generation = value["resolver_generation"] - root = value["root"] - root_file_id_value = value["root_file_id"] - session_id = value["session_id"] - canonical_path = value["canonical_path"] - file_id_value = value["file_id"] - observed_size = value["observed_size"] - mtime_ns = value["mtime_ns"] - ctime_ns = value["ctime_ns"] - committed_offset = value["committed_offset"] - partial_value = value["partial_record_b64"] - active_turn_id = value["active_turn_id"] - last_content_turn_id = value["last_content_turn_id"] - stream_value = value["stream_spans"] - if ( - type(resolver_generation) is not int - or resolver_generation <= 0 - or type(root) is not str - or not root - or type(session_id) is not str - or _canonical_codex_session_id(session_id) != session_id - or type(canonical_path) is not str - or not canonical_path - or type(observed_size) is not int - or observed_size < 0 - or type(mtime_ns) is not int - or mtime_ns < 0 - or type(ctime_ns) is not int - or ctime_ns < 0 - or type(committed_offset) is not int - or committed_offset < 0 - or type(partial_value) is not str - or type(active_turn_id) is not str - or type(last_content_turn_id) is not str - or type(value["turn_open"]) is not bool - or type(value["final_seen"]) is not bool - or type(value["complete"]) is not bool - or type(value["internal_turn"]) is not bool - or not isinstance(stream_value, list) - or len(stream_value) > _MAX_CODEX_STREAM_MESSAGES - ): - raise ValueError("invalid Codex parser state values") - for turn_id in (active_turn_id, last_content_turn_id): - if len(turn_id.encode("utf-8")) > _CODEX_TURN_ID_MAX_BYTES: - raise ValueError("oversized Codex turn identity") - try: - partial_record = base64.b64decode(partial_value, validate=True) - except (ValueError, base64.binascii.Error) as exc: - raise ValueError("invalid Codex partial record") from exc - if ( - base64.b64encode(partial_record).decode("ascii") != partial_value - or len(partial_record) > _CODEX_RECORD_MAX_BYTES - or committed_offset + len(partial_record) > observed_size - ): - raise ValueError("invalid Codex partial coordinates") - if ( - not isinstance(root_file_id_value, (list, tuple)) - or len(root_file_id_value) != 2 - or any(type(part) is not int or part < 0 for part in root_file_id_value) - ): - raise ValueError("invalid Codex root identity") - if ( - file_id_value is not None - and ( - not isinstance(file_id_value, (list, tuple)) - or len(file_id_value) != 2 - or any(type(part) is not int or part < 0 for part in file_id_value) - ) - ): - raise ValueError("invalid Codex file identity") - if file_id_value is None and ( - observed_size != 0 - or committed_offset != 0 - or partial_record - or mtime_ns != 0 - or ctime_ns != 0 - or active_turn_id - or last_content_turn_id - or value["internal_turn"] - or stream_value - ): - raise ValueError("invalid Codex bootstrap state") - spans: list[_CodexRecordSpan] = [] - prior_end = -1 - for raw_span in stream_value: - if ( - not isinstance(raw_span, (list, tuple)) - or len(raw_span) != 2 - or any(type(part) is not int or part < 0 for part in raw_span) - ): - raise ValueError("invalid Codex record span") - start, end = raw_span - if start >= end or end > committed_offset or end - start > _CODEX_RECORD_MAX_BYTES: - raise ValueError("invalid Codex record coordinates") - if start <= prior_end: - raise ValueError("overlapping Codex record spans") - spans.append(_CodexRecordSpan(start, end)) - prior_end = end - try: - root_path = Path(root) - path = Path(canonical_path) - relative = path.relative_to(root_path) - except ValueError as exc: - raise ValueError("Codex path outside root") from exc - if ( - len(relative.parts) != _CODEX_INDEX_MAX_DEPTH - or _codex_rollout_identity( - (relative.parts[0], relative.parts[1], relative.parts[2]), - relative.parts[3], - ) - != session_id - ): - raise ValueError("invalid Codex rollout path") - return _CodexSessionState( - resolver_generation=resolver_generation, - root=root, - root_file_id=tuple(root_file_id_value), - session_id=session_id, - canonical_path=canonical_path, - file_id=tuple(file_id_value) if file_id_value is not None else None, - observed_size=observed_size, - mtime_ns=mtime_ns, - ctime_ns=ctime_ns, - committed_offset=committed_offset, - partial_record=partial_record, - active_turn_id=active_turn_id, - last_content_turn_id=last_content_turn_id, - turn_open=value["turn_open"], - final_seen=value["final_seen"], - complete=value["complete"], - internal_turn=value["internal_turn"], - stream_spans=tuple(spans), - ) - - -def _codex_cache_weight_locked() -> int: - total = 0 - for (root, session_id), state_value in _CODEX_SESSION_CACHE.items(): - serialized = _serialize_codex_state(state_value) - total += ( - len(root.encode("utf-8")) - + len(session_id) - + len( - json.dumps( - serialized, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - ) - ) - return total - - -def _codex_cache_get_locked( - cache_key: tuple[str, str], -) -> _CodexSessionState | None: - state_value = _CODEX_SESSION_CACHE.pop(cache_key, None) - if state_value is not None: - _CODEX_SESSION_CACHE[cache_key] = state_value - return state_value - - -def _codex_cache_store_locked( - cache_key: tuple[str, str], - state_value: _CodexSessionState, -) -> bool: - encoded = _serialize_codex_state(state_value) - entry_weight = ( - len(cache_key[0].encode("utf-8")) - + len(cache_key[1]) - + len( - json.dumps( - encoded, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - ) - ) - if entry_weight > _CODEX_SESSION_CACHE_MAX_BYTES: - return False - _CODEX_SESSION_CACHE.pop(cache_key, None) - _CODEX_SESSION_CACHE[cache_key] = state_value - while _CODEX_SESSION_CACHE and ( - len(_CODEX_SESSION_CACHE) > _CODEX_SESSION_CACHE_CAPACITY - or _codex_cache_weight_locked() > _CODEX_SESSION_CACHE_MAX_BYTES - ): - _CODEX_SESSION_CACHE.popitem(last=False) - return cache_key in _CODEX_SESSION_CACHE - - -def _codex_binding_cache_key(value: Any) -> tuple[str, str] | None: - session_id = _canonical_codex_session_id(value) - if session_id is None: - return None - try: - root = _resolve_codex_sessions_root() - _codex_root_signature(root) - except OSError: - return None - return (os.fspath(root), session_id) - - -def _codex_cache_binding_generation_locked( - cache_key: tuple[str, str], -) -> int | None: - if _CODEX_SESSION_CACHE_LIVE_KEYS is None: - return None - return _CODEX_SESSION_CACHE_BINDING_GENERATIONS.get(cache_key) - - -def _prune_codex_cache_for_bindings(bindings: list[WorkerBinding]) -> None: - live_fingerprint_sets: dict[tuple[str, str], set[str]] = {} - for binding in bindings: - if ( - binding.turn_target_kind != _CODEX_SESSION_TURN_KIND - or not _eligible_turn_binding(binding) - ): - continue - cache_key = _codex_binding_cache_key(binding.turn_target_value) - if cache_key is not None: - live_fingerprint_sets.setdefault(cache_key, set()).add( - binding.private_fingerprint - ) - live_keys = set(live_fingerprint_sets) - live_fingerprints = { - key: tuple(sorted(values)) - for key, values in live_fingerprint_sets.items() - } - global _CODEX_SESSION_CACHE_GENERATION_COUNTER - global _CODEX_SESSION_CACHE_BINDING_FINGERPRINTS - global _CODEX_SESSION_CACHE_BINDING_GENERATIONS - global _CODEX_SESSION_CACHE_LIVE_KEYS - with _CODEX_SESSION_CACHE_LOCK: - changed = { - key - for key in live_keys - if ( - _CODEX_SESSION_CACHE_LIVE_KEYS is not None - and key in _CODEX_SESSION_CACHE_LIVE_KEYS - and _CODEX_SESSION_CACHE_BINDING_FINGERPRINTS.get(key) - != live_fingerprints[key] - ) - } - generations: dict[tuple[str, str], int] = {} - for key in live_keys: - if ( - _CODEX_SESSION_CACHE_LIVE_KEYS is not None - and key in _CODEX_SESSION_CACHE_LIVE_KEYS - and _CODEX_SESSION_CACHE_BINDING_FINGERPRINTS.get(key) - == live_fingerprints[key] - ): - generations[key] = _CODEX_SESSION_CACHE_BINDING_GENERATIONS[key] - else: - _CODEX_SESSION_CACHE_GENERATION_COUNTER += 1 - generations[key] = _CODEX_SESSION_CACHE_GENERATION_COUNTER - _CODEX_SESSION_CACHE_LIVE_KEYS = live_keys - _CODEX_SESSION_CACHE_BINDING_FINGERPRINTS = live_fingerprints - _CODEX_SESSION_CACHE_BINDING_GENERATIONS = generations - for key in tuple(_CODEX_SESSION_CACHE): - if key not in live_keys or key in changed: - del _CODEX_SESSION_CACHE[key] - - -def _codex_record_event(record: Mapping[str, Any]) -> _CodexSemanticEvent | None: - payload = record.get("payload") - if not isinstance(payload, Mapping): - return None - payload_type = str(payload.get("type") or "") - turn_id = _payload_turn_id(payload) - if turn_id and len(turn_id.encode("utf-8")) > _CODEX_TURN_ID_MAX_BYTES: - raise ValueError("oversized Codex turn identity") - if record.get("type") == "event_msg" and payload_type == "task_started": - return _CodexSemanticEvent("start", turn_id) - if record.get("type") == "event_msg" and payload_type == "task_complete": - text = str(payload.get("last_agent_message") or "") - if not text: - return _CodexSemanticEvent("complete_empty", turn_id) - return _CodexSemanticEvent("final", turn_id, text) - if record.get("type") != "response_item" or payload_type != "message": - return None - text = _message_text(payload) - if not text: - return None - role = str(payload.get("role") or "") - if role == "user": - return _CodexSemanticEvent("user", turn_id, text) - if role != "assistant": - return None - if str(payload.get("phase") or "") == "commentary": - return _CodexSemanticEvent("commentary", turn_id, text) - return _CodexSemanticEvent("final", turn_id, text) - - -def _apply_codex_event( - state_value: _CodexWorkState, - event: _CodexSemanticEvent | None, - span: _CodexRecordSpan, -) -> None: - if event is None: - return - if event.kind == "start": - if not event.turn_id: - return - state_value.active_turn_id = event.turn_id - state_value.last_content_turn_id = "" - state_value.turn_open = True - state_value.final_seen = False - state_value.complete = False - state_value.internal_turn = False - state_value.stream_items.clear() - state_value.user_text = None - state_value.final_text = None - state_value.public_changed = True - return - turn_id = event.turn_id or state_value.active_turn_id - if not turn_id or event.kind == "complete_empty": - return - selected = state_value.active_turn_id or state_value.last_content_turn_id - if state_value.active_turn_id and turn_id != state_value.active_turn_id: - return - if not state_value.active_turn_id and selected and turn_id != selected: - state_value.stream_items.clear() - state_value.final_seen = False - state_value.complete = False - state_value.internal_turn = False - state_value.user_text = None - state_value.final_text = None - if event.kind == "user": - if _is_internal_user_text(event.text): - # Codex may emit environment or command context before the real - # user message under the same turn ID. Suppress that context - # without allowing later internal metadata to erase a user turn - # that has already become public. - if state_value.user_text is None: - state_value.internal_turn = True - state_value.last_content_turn_id = turn_id - state_value.stream_items.clear() - state_value.final_text = None - return - state_value.internal_turn = False - state_value.user_text = event.text - state_value.last_content_turn_id = turn_id - state_value.turn_open = not state_value.final_seen - state_value.public_changed = True - return - if state_value.internal_turn: - return - if event.kind == "commentary": - clean = event.text.strip() - if not clean: - return - state_value.stream_items = [ - item for item in state_value.stream_items if item[1] != clean - ] - state_value.stream_items.append((span, clean)) - if len(state_value.stream_items) > _MAX_CODEX_STREAM_MESSAGES: - del state_value.stream_items[ - : len(state_value.stream_items) - _MAX_CODEX_STREAM_MESSAGES - ] - state_value.last_content_turn_id = turn_id - state_value.turn_open = not state_value.final_seen - state_value.public_changed = True - return - if event.kind == "final": - state_value.final_text = event.text - state_value.last_content_turn_id = turn_id - state_value.final_seen = True - state_value.complete = True - state_value.turn_open = False - state_value.stream_items.clear() - state_value.public_changed = True - - -def _codex_decode_record(raw: bytes) -> Mapping[str, Any]: - if len(raw) > _CODEX_RECORD_MAX_BYTES: - raise ValueError("oversized Codex record") - try: - decoded = raw.decode("utf-8") - value = json.loads(decoded) - except (UnicodeError, json.JSONDecodeError) as exc: - raise ValueError("invalid Codex record") from exc - if not isinstance(value, Mapping): - raise ValueError("invalid Codex record shape") - return value - - -def _codex_materialize_stream_items( - descriptor: int, - spans: tuple[_CodexRecordSpan, ...], -) -> tuple[list[tuple[_CodexRecordSpan, str]], int]: - items: list[tuple[_CodexRecordSpan, str]] = [] - bytes_read = 0 - for span in spans: - length = span.end - span.start - raw = os.pread(descriptor, length, span.start) - bytes_read += len(raw) - if len(raw) != length: - raise _TurnReadFailed - event = _codex_record_event(_codex_decode_record(raw)) - if event is None or event.kind != "commentary": - raise _TurnReadFailed - clean = event.text.strip() - items = [item for item in items if item[1] != clean] - items.append((span, clean)) - return items, bytes_read - - -def _codex_content_from_work( - state_value: _CodexWorkState, -) -> Mapping[str, Any] | None: - if state_value.internal_turn: - return None - if not state_value.public_changed: - return None - turn_id = state_value.active_turn_id or state_value.last_content_turn_id - if not turn_id: - return None - stream_text = "\n\n".join(text for _span, text in state_value.stream_items) or None - content = { - "user_text": state_value.user_text, - "assistant_stream_text": None if state_value.final_seen else stream_text, - "assistant_final_text": state_value.final_text, - "complete": state_value.complete if state_value.final_seen else False, - "has_open_turn": not state_value.final_seen, - "source_turn_id": turn_id[:160], - } - if _is_internal_turn_content(content): - return None - if not any(item not in (None, "", False) for item in content.values()): - return None - return content - - -def _codex_freeze_work(state_value: _CodexWorkState) -> _CodexSessionState: - return _CodexSessionState( - resolver_generation=state_value.resolver_generation, - root=state_value.root, - root_file_id=state_value.root_file_id, - session_id=state_value.session_id, - canonical_path=state_value.canonical_path, - file_id=state_value.file_id, - observed_size=state_value.observed_size, - mtime_ns=state_value.mtime_ns, - ctime_ns=state_value.ctime_ns, - committed_offset=state_value.committed_offset, - partial_record=state_value.partial_record, - active_turn_id=state_value.active_turn_id, - last_content_turn_id=state_value.last_content_turn_id, - turn_open=state_value.turn_open, - final_seen=state_value.final_seen, - complete=state_value.complete, - internal_turn=state_value.internal_turn, - stream_spans=tuple(span for span, _text in state_value.stream_items), - ) - - -def _codex_work_from_prior( - descriptor: int, - prior: _CodexSessionState, - opened: os.stat_result, -) -> tuple[_CodexWorkState, int]: - stream_items, bytes_read = _codex_materialize_stream_items( - descriptor, - prior.stream_spans, - ) - return ( - _CodexWorkState( - resolver_generation=prior.resolver_generation, - root=prior.root, - root_file_id=prior.root_file_id, - session_id=prior.session_id, - canonical_path=prior.canonical_path, - file_id=(int(opened.st_dev), int(opened.st_ino)), - observed_size=int(opened.st_size), - mtime_ns=int(opened.st_mtime_ns), - ctime_ns=int(opened.st_ctime_ns), - committed_offset=prior.committed_offset, - partial_record=prior.partial_record, - active_turn_id=prior.active_turn_id, - last_content_turn_id=prior.last_content_turn_id, - turn_open=prior.turn_open, - final_seen=prior.final_seen, - complete=prior.complete, - internal_turn=prior.internal_turn, - stream_items=stream_items, - ), - bytes_read, - ) - - -def _codex_apply_complete_line( - state_value: _CodexWorkState, - raw: bytes, - start: int, - end: int, -) -> _CodexSemanticEvent | None: - event = None - if raw.strip(): - record = _codex_decode_record(raw) - event = _codex_record_event(record) - _apply_codex_event( - state_value, - event, - _CodexRecordSpan(start, end), - ) - state_value.committed_offset = end + 1 - return event - - -def _read_codex_incremental( - descriptor: int, - prior: _CodexSessionState, - opened: os.stat_result, -) -> tuple[Mapping[str, Any] | None, _CodexSessionState, int]: - state_value, bytes_read = _codex_work_from_prior(descriptor, prior, opened) - buffer = bytearray(prior.partial_record) - line_start = prior.committed_offset - physical_offset = prior.committed_offset + len(prior.partial_record) - target_size = int(opened.st_size) - if bytes_read + (target_size - physical_offset) > _CODEX_POLL_MAX_BYTES: - raise ValueError("Codex poll byte limit exceeded") - while physical_offset < target_size: - amount = min(_CODEX_READ_CHUNK_BYTES, target_size - physical_offset) - chunk = os.pread(descriptor, amount, physical_offset) - if not chunk: - raise _TurnReadFailed - physical_offset += len(chunk) - bytes_read += len(chunk) - buffer.extend(chunk) - while True: - newline = buffer.find(b"\n") - if newline < 0: - if len(buffer) > _CODEX_RECORD_MAX_BYTES: - raise ValueError("oversized Codex record") - break - if newline > _CODEX_RECORD_MAX_BYTES: - raise ValueError("oversized Codex record") - raw = bytes(buffer[:newline]) - was_final = state_value.final_seen - event = _codex_apply_complete_line( - state_value, - raw, - line_start, - line_start + newline, - ) - del buffer[: newline + 1] - line_start = state_value.committed_offset - if ( - event is not None - and event.kind == "final" - and not was_final - and state_value.final_seen - ): - # Publish each newly completed turn before consuming a later - # turn already present in the same append batch. Bytes after - # this record may already have been read into ``buffer``; do - # not checkpoint them. The next refresh rereads from the exact - # committed newline and advances to the following turn. - state_value.partial_record = b"" - state_value.observed_size = state_value.committed_offset - state_value.mtime_ns = int(opened.st_mtime_ns) - state_value.ctime_ns = int(opened.st_ctime_ns) - return ( - _codex_content_from_work(state_value), - _codex_freeze_work(state_value), - bytes_read, - ) - state_value.partial_record = bytes(buffer) - state_value.observed_size = target_size - state_value.mtime_ns = int(opened.st_mtime_ns) - state_value.ctime_ns = int(opened.st_ctime_ns) - return ( - _codex_content_from_work(state_value), - _codex_freeze_work(state_value), - bytes_read, - ) - - -def _codex_tail_candidate( - data: bytes, - absolute_start: int, - file_size: int, -) -> tuple[int, list[tuple[int, int, Mapping[str, Any], _CodexSemanticEvent | None]], bytes] | None: - aligned_start = absolute_start - if absolute_start > 0: - first_lf = data.find(b"\n") - if first_lf < 0: - return None - aligned_start += first_lf + 1 - data = data[first_lf + 1 :] - last_lf = data.rfind(b"\n") - if last_lf < 0: - return None - complete = data[: last_lf + 1] - partial = data[last_lf + 1 :] - if len(partial) > _CODEX_RECORD_MAX_BYTES: - raise ValueError("oversized Codex record") - records: list[ - tuple[int, int, Mapping[str, Any], _CodexSemanticEvent | None] - ] = [] - cursor = 0 - while cursor < len(complete): - newline = complete.find(b"\n", cursor) - if newline < 0: - break - raw = complete[cursor:newline] - start = aligned_start + cursor - end = aligned_start + newline - cursor = newline + 1 - if not raw.strip(): - continue - if len(records) >= _CODEX_RESYNC_MAX_RECORDS: - raise ValueError("too many Codex resync records") - try: - record = _codex_decode_record(raw) - event = _codex_record_event(record) - except ValueError: - records.append((start, end, {}, _CodexSemanticEvent("invalid", ""))) - continue - records.append((start, end, record, event)) - latest_start: int | None = None - for index, (_start, _end, _record, event) in enumerate(records): - if event is not None and event.kind == "start" and event.turn_id: - latest_start = index - boundary = latest_start - if boundary is None: - last_identity = "" - for _start, _end, _record, event in records: - if ( - event is not None - and event.kind in {"user", "commentary", "final"} - and event.turn_id - ): - last_identity = event.turn_id - if last_identity: - boundary = next( - index - for index, (_start, _end, _record, event) in enumerate(records) - if ( - event is not None - and event.turn_id == last_identity - and event.kind in {"user", "commentary", "final"} - ) - ) - if boundary is None: - return None - if any( - event is not None and event.kind == "invalid" - for _start, _end, _record, event in records[boundary:] - ): - raise ValueError("invalid Codex record") - return boundary, records, partial - - -def _resync_codex( - descriptor: int, - bootstrap: _CodexSessionState, - opened: os.stat_result, -) -> tuple[Mapping[str, Any] | None, _CodexSessionState, int] | None: - file_size = int(opened.st_size) - if file_size == 0: - return None - chunks: deque[bytes] = deque() - total = 0 - next_amount = _CODEX_RESYNC_INITIAL_BYTES - candidate = None - absolute_start = file_size - while total < min(file_size, _CODEX_RESYNC_MAX_BYTES): - amount = min( - next_amount, - file_size - total, - _CODEX_RESYNC_MAX_BYTES - total, - ) - absolute_start = file_size - total - amount - chunk = os.pread(descriptor, amount, absolute_start) - if len(chunk) != amount: - raise _TurnReadFailed - chunks.appendleft(chunk) - total += amount - data = b"".join(chunks) - candidate = _codex_tail_candidate(data, absolute_start, file_size) - if candidate is not None: - boundary, records, _partial = candidate - boundary_event = records[boundary][3] - if ( - boundary_event is not None - and boundary_event.kind == "start" - ) or absolute_start == 0 or total >= min(file_size, _CODEX_RESYNC_MAX_BYTES): - break - candidate = None - next_amount = min(next_amount * 2, _CODEX_RESYNC_MAX_BYTES - total) - if next_amount <= 0: - break - if candidate is None: - return None - boundary, records, partial = candidate - first_start = records[boundary][0] - state_value = _CodexWorkState( - resolver_generation=bootstrap.resolver_generation, - root=bootstrap.root, - root_file_id=bootstrap.root_file_id, - session_id=bootstrap.session_id, - canonical_path=bootstrap.canonical_path, - file_id=(int(opened.st_dev), int(opened.st_ino)), - observed_size=file_size, - mtime_ns=int(opened.st_mtime_ns), - ctime_ns=int(opened.st_ctime_ns), - committed_offset=first_start, - ) - for start, end, _record, event in records[boundary:]: - if event is not None and event.kind == "invalid": - raise ValueError("invalid Codex record") - _apply_codex_event( - state_value, - event, - _CodexRecordSpan(start, end), - ) - state_value.committed_offset = end + 1 - state_value.partial_record = partial - if state_value.committed_offset + len(partial) != file_size: - raise _TurnReadFailed - return ( - _codex_content_from_work(state_value), - _codex_freeze_work(state_value), - total, - ) - - -def _codex_resolution_for_state( - state_value: _CodexSessionState, -) -> _CodexPathResolution: - try: - configured_root = _resolve_codex_sessions_root() - configured_signature = _codex_root_signature(configured_root) - state_root = Path(state_value.root) - path = Path(state_value.canonical_path) - relative = path.relative_to(state_root) - except (OSError, ValueError) as exc: - raise _TurnReadFailed from exc - if ( - configured_root != state_root - or state_value.root_file_id != configured_signature[:2] - or len(relative.parts) != _CODEX_INDEX_MAX_DEPTH - ): - raise _TurnReadFailed - parsed_id = _codex_rollout_identity( - (relative.parts[0], relative.parts[1], relative.parts[2]), - relative.parts[3], - ) - if parsed_id != state_value.session_id: - raise _TurnReadFailed - try: - current = path.lstat() - except OSError as exc: - raise _TurnReadFailed from exc - return _CodexPathResolution( - status="found", - root=state_value.root, - root_file_id=(configured_signature[0], configured_signature[1]), - session_id=state_value.session_id, - canonical_path=state_value.canonical_path, - relative_path="/".join(relative.parts), - file_id=(int(current.st_dev), int(current.st_ino)), - generation=state_value.resolver_generation, - expires_at=0.0, - ) - - -def _read_codex_session_turn_with_state( - session_id: str, - supplied_state: _CodexSessionState, -) -> tuple[Mapping[str, Any] | None, _CodexSessionState, int] | None: - if _canonical_codex_session_id(session_id) != supplied_state.session_id: - raise _TurnReadFailed - prior: _CodexSessionState | None = supplied_state - for _attempt in range(2): - resolution = _codex_resolution_for_state(supplied_state) - descriptor, opened = _open_verified_codex_file(resolution) - try: - file_id = (int(opened.st_dev), int(opened.st_ino)) - reusable = bool( - prior is not None - and prior.file_id == file_id - and prior.resolver_generation == supplied_state.resolver_generation - and prior.canonical_path == supplied_state.canonical_path - and int(opened.st_size) - >= prior.committed_offset + len(prior.partial_record) - and ( - int(opened.st_size) > prior.observed_size - or ( - int(opened.st_size) == prior.observed_size - and int(opened.st_mtime_ns) == prior.mtime_ns - and int(opened.st_ctime_ns) == prior.ctime_ns - ) - ) - ) - if reusable: - parsed = _read_codex_incremental(descriptor, prior, opened) - else: - parsed = _resync_codex(descriptor, supplied_state, opened) - after_fd = os.fstat(descriptor) - after_path = Path(supplied_state.canonical_path).stat() - if ( - (int(after_fd.st_dev), int(after_fd.st_ino)) != file_id - or (int(after_path.st_dev), int(after_path.st_ino)) != file_id - or int(after_fd.st_size) != int(opened.st_size) - or int(after_fd.st_mtime_ns) != int(opened.st_mtime_ns) - or int(after_fd.st_ctime_ns) != int(opened.st_ctime_ns) - ): - raise _TurnReadFailed - return parsed - except (_TurnReadFailed, OSError): - prior = None - if _attempt: - raise _TurnReadFailed - finally: - os.close(descriptor) - raise _TurnReadFailed - - -def _codex_checkpoint_matches_stat( - state_value: _CodexSessionState, - resolution: _CodexPathResolution, -) -> bool: - if ( - state_value.file_id is None - or resolution.status != "found" - or resolution.canonical_path != state_value.canonical_path - or resolution.root_file_id != state_value.root_file_id - or resolution.generation != state_value.resolver_generation - ): - return False - try: - current = Path(state_value.canonical_path).stat() - except OSError: - return False - return bool( - stat.S_ISREG(current.st_mode) - and (int(current.st_dev), int(current.st_ino)) == state_value.file_id - and int(current.st_size) == state_value.observed_size - and int(current.st_mtime_ns) == state_value.mtime_ns - and int(current.st_ctime_ns) == state_value.ctime_ns - ) - - -def _publish_codex_cache_state( - cache_key: tuple[str, str], - prior_state_value: Mapping[str, Any] | None, - updated_state: _CodexSessionState, - content: Mapping[str, Any] | None, - binding_generation: int | None, -) -> Mapping[str, Any] | None: - with _CODEX_SESSION_CACHE_LOCK: - current_generation = _codex_cache_binding_generation_locked(cache_key) - if ( - binding_generation != current_generation - or ( - _CODEX_SESSION_CACHE_LIVE_KEYS is not None - and cache_key not in _CODEX_SESSION_CACHE_LIVE_KEYS - ) - ): - return None - current = _CODEX_SESSION_CACHE.get(cache_key) - current_value = _serialize_codex_state(current) - exact_prior = current_value == prior_state_value - monotone = bool( - current is not None - and current.resolver_generation == updated_state.resolver_generation - and current.root_file_id == updated_state.root_file_id - and current.canonical_path == updated_state.canonical_path - and current.file_id == updated_state.file_id - and updated_state.committed_offset > current.committed_offset - ) - if not exact_prior and not monotone: - return None - if prior_state_value is not None and current is None: - return None - if not _codex_cache_store_locked(cache_key, updated_state): - return None - return content - - -def _read_codex_session_turn(session_id: str) -> Mapping[str, Any] | None: - resolution = _resolve_codex_session(session_id) - if resolution is None or resolution.status != "found": - return None - cache_key = (resolution.root, resolution.session_id) - with _CODEX_SESSION_CACHE_LOCK: - prior = _codex_cache_get_locked(cache_key) - if ( - prior is not None - and ( - prior.resolver_generation != resolution.generation - or prior.root_file_id != resolution.root_file_id - or prior.canonical_path != resolution.canonical_path - or prior.file_id != resolution.file_id - ) - ): - _CODEX_SESSION_CACHE.pop(cache_key, None) - prior = None - prior_value = _serialize_codex_state(prior) - binding_generation = _codex_cache_binding_generation_locked(cache_key) - if prior is not None and _codex_checkpoint_matches_stat(prior, resolution): - observer = _CODEX_ISOLATED_READ_OBSERVER - if observer is not None: - observer(0) - return None - supplied = prior - if ( - supplied is None - or supplied.resolver_generation != resolution.generation - or supplied.root_file_id != resolution.root_file_id - or supplied.canonical_path != resolution.canonical_path - ): - supplied = _CodexSessionState( - resolver_generation=resolution.generation, - root=resolution.root, - root_file_id=resolution.root_file_id, - session_id=resolution.session_id, - canonical_path=resolution.canonical_path or "", - file_id=None, - observed_size=0, - mtime_ns=0, - ctime_ns=0, - committed_offset=0, - partial_record=b"", - active_turn_id="", - last_content_turn_id="", - turn_open=False, - final_seen=False, - complete=False, - stream_spans=(), - ) - prior_value = None - parsed = _read_codex_session_turn_with_state(session_id, supplied) - if parsed is None: - return None - content, updated_state, bytes_read = parsed - observer = _CODEX_ISOLATED_READ_OBSERVER - if observer is not None: - observer(bytes_read) - return _publish_codex_cache_state( - cache_key, - prior_value, - updated_state, - content, - binding_generation, - ) - - -def _omp_sessions_root() -> Path: - raw = os.environ.get("OMP_SESSIONS_DIR") - if raw: - return Path(raw).expanduser() - return Path.home() / ".omp" / "agent" / "sessions" - - -def _safe_omp_session_path(value: str) -> Path | None: - candidate = Path(str(value or "")).expanduser() - root = _omp_sessions_root() - try: - candidate.resolve().relative_to(root.resolve()) - except (ValueError, OSError): - return None - if candidate.suffix != ".jsonl" or not candidate.is_file(): - return None - return candidate - - -def _omp_valid_git_directory(path: Path) -> bool: - """Require bounded, parseable Git HEAD evidence inside a git directory.""" - try: - if not path.is_dir(): - return False - with open(path / "HEAD", encoding="ascii") as handle: - head_text = handle.read(257) - except (OSError, UnicodeError): - return False - if len(head_text) > 256: - return False - lines = head_text.splitlines() - if len(lines) != 1: - return False - head = lines[0] - if re.fullmatch(r"[0-9a-fA-F]{40}|[0-9a-fA-F]{64}", head): - return True - if not head.startswith("ref: refs/"): - return False - reference = head[len("ref: ") :] - return bool(reference) and all( - part not in {"", ".", ".."} for part in reference.split("/") - ) and all(character.isalnum() or character in "._-/" for character in reference) - - -def _omp_project_root(session_file: Path) -> Path | None: - """Return the nearest repository root proven by the session cwd.""" - try: - with open(session_file, encoding="utf-8", errors="replace") as handle: - for _index in range(32): - line = handle.readline() - if not line or handle.tell() > 65536: - break - try: - entry = json.loads(line) - except (TypeError, json.JSONDecodeError): - continue - if not isinstance(entry, Mapping) or entry.get("type") != "session": - continue - cwd = entry.get("cwd") - if type(cwd) is not str or not cwd.strip(): - return None - resolved_cwd = Path(cwd).expanduser().resolve(strict=True) - if not resolved_cwd.is_dir(): - return None - for root in (resolved_cwd, *resolved_cwd.parents): - git_marker = root / ".git" - if git_marker.is_dir(): - return root if _omp_valid_git_directory(git_marker) else None - if not git_marker.exists(): - continue - if not git_marker.is_file(): - return None - with open(git_marker, encoding="utf-8") as marker_handle: - marker_text = marker_handle.read(4097) - if len(marker_text) > 4096: - return None - marker_lines = marker_text.splitlines() - if len(marker_lines) != 1 or not marker_lines[0].startswith("gitdir:"): - return None - reference = marker_lines[0][len("gitdir:") :].strip() - if not reference: - return None - git_dir = Path(reference) - if not git_dir.is_absolute(): - git_dir = git_marker.parent / git_dir - resolved_git_dir = git_dir.resolve(strict=True) - return root if _omp_valid_git_directory(resolved_git_dir) else None - return None - except (OSError, RuntimeError, UnicodeError, ValueError): - return None - return None - - -def _omp_file_id(stat_result: os.stat_result) -> tuple[int, int]: - return (int(stat_result.st_dev), int(stat_result.st_ino)) - - -def _omp_checkpoint_matches_stat( - state: _OmpSessionState | None, - stat_result: os.stat_result, -) -> bool: - return bool( - state is not None - and state.file_id == _omp_file_id(stat_result) - and state.observed_size == int(stat_result.st_size) - and state.mtime_ns == int(stat_result.st_mtime_ns) - and state.ctime_ns == int(stat_result.st_ctime_ns) - ) - - -class _OmpFileChanged(Exception): - pass - - -def _omp_binding_cache_key(path_value: str) -> str | None: - candidate = Path(str(path_value or "")).expanduser() - if candidate.suffix != ".jsonl": - return None - try: - resolved = candidate.resolve() - resolved.relative_to(_omp_sessions_root().resolve()) - except (ValueError, OSError): - return None - return os.fspath(resolved) - - -def _omp_cache_key(path_value: str) -> str | None: - if _safe_omp_session_path(path_value) is None: - return None - return _omp_binding_cache_key(path_value) - - -def _prune_omp_cache_for_bindings(bindings: list[WorkerBinding]) -> None: - live_fingerprint_sets: dict[str, set[str]] = {} - live_keys: set[str] = set() - for binding in bindings: - if ( - binding.turn_target_kind != _OMP_SESSION_TURN_KIND - or not _eligible_turn_binding(binding) - ): - continue - cache_key = _omp_binding_cache_key(str(binding.turn_target_value or "")) - if cache_key is not None: - live_keys.add(cache_key) - live_fingerprint_sets.setdefault(cache_key, set()).add( - binding.private_fingerprint - ) - global _OMP_SESSION_CACHE_GENERATION_COUNTER - global _OMP_SESSION_CACHE_BINDING_FINGERPRINTS - global _OMP_SESSION_CACHE_BINDING_GENERATIONS - global _OMP_SESSION_CACHE_LIVE_KEYS - with _OMP_SESSION_CACHE_LOCK: - live_fingerprints = { - cache_key: tuple(sorted(fingerprints)) - for cache_key, fingerprints in live_fingerprint_sets.items() - } - changed_keys = { - cache_key - for cache_key in live_keys - if ( - _OMP_SESSION_CACHE_LIVE_KEYS is not None - and cache_key in _OMP_SESSION_CACHE_LIVE_KEYS - and _OMP_SESSION_CACHE_BINDING_FINGERPRINTS.get(cache_key) - != live_fingerprints.get(cache_key) - ) - } - generations: dict[str, int] = {} - for cache_key in live_keys: - if ( - _OMP_SESSION_CACHE_LIVE_KEYS is not None - and cache_key in _OMP_SESSION_CACHE_LIVE_KEYS - and _OMP_SESSION_CACHE_BINDING_FINGERPRINTS.get(cache_key) - == live_fingerprints.get(cache_key) - ): - generations[cache_key] = _OMP_SESSION_CACHE_BINDING_GENERATIONS[cache_key] - else: - _OMP_SESSION_CACHE_GENERATION_COUNTER += 1 - generations[cache_key] = _OMP_SESSION_CACHE_GENERATION_COUNTER - _OMP_SESSION_CACHE_LIVE_KEYS = live_keys - _OMP_SESSION_CACHE_BINDING_FINGERPRINTS = live_fingerprints - _OMP_SESSION_CACHE_BINDING_GENERATIONS = generations - for cache_key in tuple(_OMP_SESSION_CACHE): - if cache_key not in live_keys or cache_key in changed_keys: - del _OMP_SESSION_CACHE[cache_key] - - -def _read_omp_jsonl_lines( - session_file: Path, - *, - start_offset: int, - drop_first_partial: bool, - expected_file_id: tuple[int, int], -) -> tuple[list[str], int, int]: - try: - with open(session_file, "rb") as handle: - opened_stat = os.fstat(handle.fileno()) - if ( - _omp_file_id(opened_stat) != expected_file_id - or int(opened_stat.st_size) < start_offset - ): - raise _OmpFileChanged - handle.seek(start_offset) - blob = handle.read() - completed_stat = os.fstat(handle.fileno()) - current_stat = session_file.stat() - opened_signature = ( - _omp_file_id(opened_stat), - int(opened_stat.st_size), - int(opened_stat.st_mtime_ns), - int(opened_stat.st_ctime_ns), - ) - completed_signature = ( - _omp_file_id(completed_stat), - int(completed_stat.st_size), - int(completed_stat.st_mtime_ns), - int(completed_stat.st_ctime_ns), - ) - current_signature = ( - _omp_file_id(current_stat), - int(current_stat.st_size), - int(current_stat.st_mtime_ns), - int(current_stat.st_ctime_ns), - ) - if ( - completed_signature != opened_signature - or current_signature != completed_signature - ): - raise _OmpFileChanged - except _OmpFileChanged: - raise - except OSError as exc: - raise _TurnReadFailed from exc - if not blob: - return [], start_offset, 0 - - offset = start_offset - segments = blob.splitlines(keepends=True) - if drop_first_partial and segments: - offset += len(segments[0]) - segments = segments[1:] - - lines: list[str] = [] - for index, segment in enumerate(segments): - line_bytes = segment.rstrip(b"\r\n") - if not line_bytes: - offset += len(segment) - continue - text = line_bytes.decode("utf-8", "replace") - has_line_end = segment.endswith(b"\n") or segment.endswith(b"\r") - if not has_line_end and index == len(segments) - 1: - try: - json.loads(text) - except (TypeError, json.JSONDecodeError): - break - lines.append(text) - offset += len(segment) - return lines, offset, len(blob) - - -def _omp_message_entry_from_line(line: str) -> tuple[Mapping[str, Any], Mapping[str, Any]] | None: - try: - entry = json.loads(line) - except (TypeError, json.JSONDecodeError): - return None - if not isinstance(entry, Mapping) or entry.get("type") != "message": - return None - message = entry.get("message") - if not isinstance(message, Mapping): - return None - return entry, message - - -def _is_omp_user_message(message: Mapping[str, Any]) -> bool: - return str(message.get("role") or "") == "user" and str(message.get("attribution") or "") == "user" - - -def _last_omp_user_line_index(lines: list[str]) -> int | None: - found: int | None = None - for index, line in enumerate(lines): - parsed = _omp_message_entry_from_line(line) - if parsed is None: - continue - _entry, message = parsed - text = _omp_message_text(message) - if _is_omp_user_message(message) and text and not _is_internal_user_text(text): - found = index - return found - - -def _omp_thinking_snippet(message: Mapping[str, Any]) -> str: - """Compact progress line from an omp thinking block: its bold headline, - falling back to a trimmed first line.""" - content = message.get("content") - if not isinstance(content, list): - return "" - for item in content: - if not isinstance(item, Mapping) or item.get("type") != "thinking": - continue - text = str(item.get("thinking") or "").strip() - if not text: - continue - first = text.splitlines()[0].strip() - if first.startswith("**") and first.endswith("**") and len(first) > 4: - return first.strip("*").strip() - return first[:120] - return "" - - -_OMP_FILE_ACTIONS = { - "read": "read", - "read_file": "read", - "write": "write", - "write_file": "write", - "edit": "edit", - "edit_file": "edit", - "apply_patch": "edit", -} -_OMP_ACTIONS = { - "grep": "search", - "search": "search", - "ast_grep": "search", - "glob": "list files", - "list_files": "list files", - "browser": "browse", - "web_search": "search web", - "task": "delegate", - "agent": "delegate", - "lsp": "inspect code", -} -_OMP_PRIVATE_PATH_SEGMENTS = frozenset( - { - ".git", - ".netrc", - ".npmrc", - ".pypirc", - ".ssh", - "auth.json", - "credential", - "credentials", - "credentials.json", - "id_dsa", - "id_ecdsa", - "id_ed25519", - "id_rsa", - "secret", - "secrets", - "secrets.json", - "token", - "tokens", - } -) - - -def _omp_path_segment_is_private(value: str) -> bool: - lowered = value.lower() - return ( - lowered in _OMP_PRIVATE_PATH_SEGMENTS - or lowered.startswith(".env") - or lowered.endswith(".key") - ) -_OMP_SHELL_TOOLS = frozenset({"bash", "shell", "sh", "exec", "execute", "run"}) - - -def _omp_tool_name(item: Mapping[str, Any]) -> str: - raw = item.get("name") - if raw is None: - raw = item.get("toolName") - if raw is None: - raw = item.get("tool") - if type(raw) is not str: - return "" - return re.sub(r"[^a-z0-9]+", "_", raw.strip().lower()).strip("_") - - -def _omp_tool_arguments(item: Mapping[str, Any]) -> Mapping[str, Any]: - for key in ("arguments", "input", "args"): - value = item.get(key) - if isinstance(value, Mapping): - return value - return {} - - -def _omp_repo_relative_path( - arguments: Mapping[str, Any], - project_root: Path | None, -) -> str | None: - if project_root is None: - return None - raw: Any = None - for key in ("path", "file_path", "filepath", "file"): - if key in arguments: - raw = arguments.get(key) - break - if type(raw) is not str: - return None - text = raw.strip() - if not text or text.startswith("~") or any(ord(character) < 32 for character in text): - return None - try: - root = project_root.resolve(strict=True) - candidate = Path(text) - if ".." in candidate.parts: - return None - resolved = candidate.resolve(strict=False) if candidate.is_absolute() else (root / candidate).resolve(strict=False) - relative = resolved.relative_to(root) - except (OSError, RuntimeError, ValueError): - return None - if any(_omp_path_segment_is_private(part) for part in relative.parts): - return None - public_path = relative.as_posix() - if not public_path or public_path == "." or len(public_path) > 120: - return None - allowed = frozenset("._-+@/ ") - if any(not (character.isascii() and (character.isalnum() or character in allowed)) for character in public_path): - return None - return public_path - - -def _omp_shell_progress(arguments: Mapping[str, Any]) -> _PublicOmpToolProgress: - command = arguments.get("command") - if command is None: - command = arguments.get("cmd") - if type(command) is not str or any(character in command for character in "\r\n;&|`$<>"): - return _PublicOmpToolProgress("run command") - try: - tokens = shlex.split(command) - except ValueError: - return _PublicOmpToolProgress("run command") - if not tokens: - return _PublicOmpToolProgress("run command") - if tokens[:2] == ["git", "status"] and all( - token in {"--short", "--porcelain", "--branch", "-s", "-sb"} - for token in tokens[2:] - ): - return _PublicOmpToolProgress("git status") - if tokens[0] == "pytest" or tokens[:3] in (["python", "-m", "pytest"], ["python3", "-m", "pytest"]): - return _PublicOmpToolProgress("test", "pytest") - if tokens[:3] == ["uv", "run", "pytest"]: - return _PublicOmpToolProgress("test", "pytest") - if len(tokens) >= 2 and tokens[0] in {"bun", "cargo", "go", "npm"} and tokens[1] == "test": - return _PublicOmpToolProgress("test", tokens[0]) - if len(tokens) >= 2 and tokens[0] in {"cargo", "go"} and tokens[1] == "build": - return _PublicOmpToolProgress("build", tokens[0]) - if tokens[:3] == ["npm", "run", "build"]: - return _PublicOmpToolProgress("build", "npm") - if tokens[0] == "make": - return _PublicOmpToolProgress("build", "make") - if tokens[:3] in (["python", "-m", "build"], ["python3", "-m", "build"]): - return _PublicOmpToolProgress("build", "python") - return _PublicOmpToolProgress("run command") - - -def _omp_public_tool_progress( - item: Mapping[str, Any], - project_root: Path | None, -) -> _PublicOmpToolProgress: - name = _omp_tool_name(item) - arguments = _omp_tool_arguments(item) - if name in _OMP_FILE_ACTIONS: - action = _OMP_FILE_ACTIONS[name] - subject = _omp_repo_relative_path(arguments, project_root) - return _PublicOmpToolProgress(action, subject) if subject else _PublicOmpToolProgress(f"{action} file") - if name in _OMP_SHELL_TOOLS: - return _omp_shell_progress(arguments) - return _PublicOmpToolProgress(_OMP_ACTIONS.get(name, "tool")) - - -def _omp_tool_snippet( - item: Mapping[str, Any], - step: int, - project_root: Path | None = None, -) -> str: - return _omp_public_tool_progress(item, project_root).render(step) - - -def _apply_omp_progress_message( - state: _OmpTurnState, - message: Mapping[str, Any], -) -> bool: - contributed = False - text = _omp_message_text(message) - if text: - _append_unique_recent(state.stream_parts, text) - contributed = True - - content = message.get("content") - if not isinstance(content, list): - return contributed - for item in content: - if not isinstance(item, Mapping): - continue - kind = str(item.get("type") or "") - if kind == "thinking": - snippet = _omp_thinking_snippet({"content": [item]}) - if snippet: - _append_unique_recent(state.stream_parts, snippet) - contributed = True - continue - if kind == "toolCall": - state.tool_count += 1 - snippet = _omp_tool_snippet(item, state.tool_count, state.project_root) - if snippet: - _append_unique_recent(state.stream_parts, snippet) - contributed = True - return contributed - - -def _omp_message_text(message: Mapping[str, Any]) -> str: - content = message.get("content") - if isinstance(content, str): - return content - if isinstance(content, list): - return "\n".join( - str(item.get("text") or "") - for item in content - if isinstance(item, Mapping) - and item.get("type") == "text" - and str(item.get("text") or "").strip() - ) - return "" - - -def _apply_omp_lines_to_state(state: _OmpTurnState, lines: list[str]) -> None: - for line in lines: - parsed = _omp_message_entry_from_line(line) - if parsed is None: - continue - entry, message = parsed - role = str(message.get("role") or "") - text = _omp_message_text(message) - if role == "user": - if not _is_omp_user_message(message): - continue - if not text or _is_internal_user_text(text): - continue - state.prompt_id = str(entry.get("id") or "") - state.user_text = text - state.stream_parts = [] - state.final_text = "" - state.tool_count = 0 - continue - if role != "assistant" or not state.prompt_id: - continue - if state.final_text: - continue - if str(message.get("stopReason") or "") == "stop" and text: - state.final_text = text - state.stream_parts = [] - continue - _apply_omp_progress_message(state, message) - - -def _read_omp_state_from_recent( - session_file: Path, - size: int, - file_id: tuple[int, int], - project_root: Path | None, -) -> tuple[_OmpTurnState, int, int, int]: - turn = _OmpTurnState(project_root=project_root) - if size <= 0: - return turn, 0, 0, 0 - - window = min(size, max(1, _OMP_TAIL_BYTES)) - selected_lines: list[str] = [] - selected_offset = size - replay_offset = size - bytes_read = 0 - while True: - start = max(0, size - window) - lines, next_offset, consumed = _read_omp_jsonl_lines( - session_file, - start_offset=start, - drop_first_partial=start > 0, - expected_file_id=file_id, - ) - bytes_read += consumed - user_index = _last_omp_user_line_index(lines) - if user_index is not None: - selected_lines = lines[user_index:] - selected_offset = next_offset - replay_offset = start - break - if start == 0: - selected_lines = lines - selected_offset = next_offset - replay_offset = 0 - break - window = min(size, window * 2) - - _apply_omp_lines_to_state(turn, selected_lines) - return turn, selected_offset, replay_offset, bytes_read - - -def _omp_state_to_content(state: _OmpTurnState) -> Mapping[str, Any] | None: - if not state.prompt_id: - return None - has_final = bool(state.final_text) - content: dict[str, Any] = { - "user_text": state.user_text or None, - "assistant_stream_text": None if has_final else ("\n\n".join(state.stream_parts) or None), - "assistant_final_text": state.final_text or None, - "complete": has_final, - "has_open_turn": not has_final, - "source_turn_id": state.prompt_id[:160], - } - if _is_internal_turn_content(content): - return None - if not (content.get("user_text") or content.get("assistant_stream_text") or content.get("assistant_final_text")): - return None - return content - - -def _publish_omp_cache_state( - cache_key: str, - prior_state_value: Mapping[str, Any] | None, - updated_state: _OmpSessionState, - content: Mapping[str, Any] | None, - binding_generation: int | None = None, -) -> Mapping[str, Any] | None: - """Atomically publish a winning parser state and never return a losing view.""" - with _OMP_SESSION_CACHE_LOCK: - current_generation = _omp_cache_binding_generation_locked(cache_key) - if ( - binding_generation != current_generation - or ( - _OMP_SESSION_CACHE_LIVE_KEYS is not None - and cache_key not in _OMP_SESSION_CACHE_LIVE_KEYS - ) - ): - return None - current_state = _OMP_SESSION_CACHE.get(cache_key) - current_value = _serialize_omp_state(current_state) - if ( - current_value == prior_state_value - or current_state is None - or ( - current_state.file_id == updated_state.file_id - and updated_state.offset > current_state.offset - ) - ): - _omp_cache_store_locked(cache_key, updated_state) - return content - return None - - -def _read_omp_session_turn_with_state( - path_value: str, - prior_state: _OmpSessionState | None, -) -> tuple[Mapping[str, Any] | None, _OmpSessionState, int] | None: - """Parse one OMP session while returning only compact checkpoint coordinates.""" - session_file = _safe_omp_session_path(path_value) - if session_file is None: - return None - for _attempt in range(2): - try: - stat_result = session_file.stat() - except OSError as exc: - raise _TurnReadFailed from exc - size = int(stat_result.st_size) - file_id = _omp_file_id(stat_result) - reusable = bool( - prior_state is not None - and prior_state.file_id == file_id - and size >= prior_state.offset - and ( - size > prior_state.observed_size - or ( - size == prior_state.observed_size - and prior_state.mtime_ns == int(stat_result.st_mtime_ns) - and prior_state.ctime_ns == int(stat_result.st_ctime_ns) - ) - ) - ) - try: - if reusable: - start_offset = ( - prior_state.replay_offset if prior_state.turn_open else prior_state.offset - ) - lines, next_offset, bytes_read = _read_omp_jsonl_lines( - session_file, - start_offset=start_offset, - drop_first_partial=False, - expected_file_id=file_id, - ) - turn = _OmpTurnState(project_root=prior_state.project_root) - _apply_omp_lines_to_state(turn, lines) - replay_offset = start_offset - else: - project_root = _omp_project_root(session_file) - turn, next_offset, replay_offset, bytes_read = _read_omp_state_from_recent( - session_file, - size, - file_id, - project_root, - ) - content = _omp_state_to_content(turn) - turn_open = bool(content is not None and content.get("has_open_turn") is True) - checkpoint = _OmpSessionState( - offset=next_offset, - observed_size=max(size, next_offset), - file_id=file_id, - mtime_ns=int(stat_result.st_mtime_ns), - ctime_ns=int(stat_result.st_ctime_ns), - replay_offset=replay_offset if turn_open else next_offset, - turn_open=turn_open, - project_root=turn.project_root, - ) - return content, checkpoint, bytes_read - except _OmpFileChanged: - prior_state = None - raise _TurnReadFailed - - -def _read_omp_session_turn(path_value: str) -> Mapping[str, Any] | None: - """Parse an OMP session while retaining private state in the local process.""" - cache_key = _omp_cache_key(path_value) - if cache_key is None: - return None - with _OMP_SESSION_CACHE_LOCK: - prior_state = _omp_cache_get_locked(cache_key) - prior_state_value = _serialize_omp_state(prior_state) - binding_generation = _omp_cache_binding_generation_locked(cache_key) - try: - unchanged = prior_state is not None and _omp_checkpoint_matches_stat( - prior_state, - Path(cache_key).stat(), - ) - except OSError as exc: - raise _TurnReadFailed from exc - if unchanged: - return None - parsed = _read_omp_session_turn_with_state(path_value, prior_state) - if parsed is None: - return None - content, state, _bytes_read = parsed - return _publish_omp_cache_state( - cache_key, - prior_state_value, - state, - content, - binding_generation, - ) - - -@dataclass(frozen=True) -class TurnRefreshKey: - """Opaque scheduler identity for one durable private binding.""" - - private_fingerprint: str - - -@dataclass(frozen=True) -class _TurnRefreshItem: - key: TurnRefreshKey - worker_id: str - worker_fingerprint: str - turn_target_kind: str - turn_target_value: str - - @classmethod - def from_binding(cls, binding: WorkerBinding) -> "_TurnRefreshItem": - return cls( - key=TurnRefreshKey(binding.private_fingerprint), - worker_id=binding.worker_id, - worker_fingerprint=binding.worker_fingerprint, - turn_target_kind=str(binding.turn_target_kind or ""), - turn_target_value=str(binding.turn_target_value or ""), - ) - - -TurnRefreshStatus = Literal[ - "updated", - "unchanged", - "missing", - "timeout", - "failed", - "stale_binding", -] - - -class _BindingLookupFailed(Exception): - pass - - -@dataclass(frozen=True) -class TurnRefreshResult: - """Public-safe result of one binding refresh.""" - - status: TurnRefreshStatus - updated: int - pending_changed: bool = False - retry_binding_lookup: bool = False - binding_validated: bool = False - - -@dataclass(frozen=True) -class CompletedPaneTurnRefreshResult: - """Result of one completion-authoritative, pane-targeted semantic refresh.""" - - status: str - worker_id: str | None = None - refreshed_turn_id: str | None = None - - -_ELIGIBLE_TURN_TARGET_KINDS = frozenset( - {_CODEX_SESSION_TURN_KIND, _OMP_SESSION_TURN_KIND, _PANE_TURN_KIND} -) -_TURN_INGESTION_QUEUE_CAPACITY = 64 - - -def _eligible_turn_binding(binding: WorkerBinding) -> bool: - return bool( - binding.turn_target_kind in _ELIGIBLE_TURN_TARGET_KINDS - and binding.turn_target_value - and binding.private_fingerprint - ) - - -def _isolated_content_is_valid(value: Any) -> bool: - if not isinstance(value, Mapping): - return False - if not set(value).issubset({*_TURN_CONTENT_KEYS, "source_turn_id"}): - return False - for key in ("user_text", "assistant_final_text", "assistant_stream_text", "model", "source_turn_id"): - if key in value and value[key] is not None and type(value[key]) is not str: - return False - for key in ("complete", "has_open_turn"): - if key in value and type(value[key]) is not bool: - return False - return True - - -@dataclass(frozen=True) -class _OmpCachePublication: - cache_key: str - prior_state_value: Mapping[str, Any] | None - updated_state: _OmpSessionState - binding_generation: int | None -@dataclass(frozen=True) -class _CodexCachePublication: - cache_key: tuple[str, str] - prior_state_value: Mapping[str, Any] | None - updated_state: _CodexSessionState - binding_generation: int | None - resolver_generation: int - - - - -@dataclass(frozen=True) -class _ObservedFileTurn: - content: Mapping[str, Any] | None - publication: _OmpCachePublication | _CodexCachePublication - - -def _blocking_recv_frame(sock: socket.socket, maximum: int) -> bytes: - header = bytearray() - while len(header) < _OMP_FRAME_HEADER.size: - chunk = sock.recv(_OMP_FRAME_HEADER.size - len(header)) - if not chunk: - raise EOFError - header.extend(chunk) - length = _OMP_FRAME_HEADER.unpack(header)[0] - if length > maximum: - raise ValueError("oversized IPC frame") - payload = bytearray() - while len(payload) < length: - chunk = sock.recv(min(65536, length - len(payload))) - if not chunk: - raise EOFError - payload.extend(chunk) - return bytes(payload) - - -def _blocking_send_frame(sock: socket.socket, payload: bytes) -> None: - sock.sendall(_OMP_FRAME_HEADER.pack(len(payload)) + payload) - - -def _check_ipc_deadline( - deadline: float, - cancel_event: threading.Event | None, -) -> float: - if cancel_event is not None and cancel_event.is_set(): - raise _TurnReadTimeout - remaining = deadline - time.monotonic() - if remaining <= 0: - raise _TurnReadTimeout - return remaining - - -def _send_frame_until( - sock: socket.socket, - payload: bytes, - deadline: float, - cancel_event: threading.Event | None, -) -> None: - framed = _OMP_FRAME_HEADER.pack(len(payload)) + payload - view = memoryview(framed) - while view: - remaining = _check_ipc_deadline(deadline, cancel_event) - _readable, writable, _exceptional = select.select( - [], - [sock], - [], - min(0.05, remaining), - ) - if not writable: - continue - sent = sock.send(view) - if sent <= 0: - raise OSError("IPC socket closed") - view = view[sent:] - - -def _recv_frame_until( - sock: socket.socket, - deadline: float, - cancel_event: threading.Event | None, - maximum: int = _CODEX_IPC_FRAME_MAX_BYTES, -) -> bytes: - framed = bytearray() - expected: int | None = None - while expected is None or len(framed) < _OMP_FRAME_HEADER.size + expected: - remaining = _check_ipc_deadline(deadline, cancel_event) - readable, _writable, _exceptional = select.select( - [sock], - [], - [], - min(0.05, remaining), - ) - if not readable: - continue - if expected is None: - remaining_bytes = _OMP_FRAME_HEADER.size - len(framed) - else: - remaining_bytes = _OMP_FRAME_HEADER.size + expected - len(framed) - chunk = sock.recv(min(65536, remaining_bytes)) - if not chunk: - raise EOFError - framed.extend(chunk) - if expected is None and len(framed) >= _OMP_FRAME_HEADER.size: - expected = _OMP_FRAME_HEADER.unpack(framed[: _OMP_FRAME_HEADER.size])[0] - if expected > maximum: - raise ValueError("oversized IPC frame") - assert expected is not None - return bytes(framed[_OMP_FRAME_HEADER.size : _OMP_FRAME_HEADER.size + expected]) -def _blocking_send_streamed_omp_response( - sock: socket.socket, - payload: bytes, - nonce: str, - *, - chunk_bytes: int = _OMP_IPC_RESPONSE_CHUNK_BYTES, -) -> None: - if ( - type(chunk_bytes) is not int - or chunk_bytes <= 0 - or chunk_bytes > _OMP_IPC_RESPONSE_CHUNK_BYTES - ): - raise ValueError("invalid OMP IPC chunk bound") - chunk_count = (len(payload) + chunk_bytes - 1) // chunk_bytes - manifest = { - "protocol": 1, - "nonce": nonce, - "stream": "omp_response", - "chunks": chunk_count, - "total_bytes": len(payload), - "chunk_bytes": chunk_bytes, - } - _blocking_send_frame( - sock, - json.dumps(manifest, separators=(",", ":")).encode("utf-8"), - ) - view = memoryview(payload) - for offset in range(0, len(payload), chunk_bytes): - _blocking_send_frame(sock, bytes(view[offset : offset + chunk_bytes])) - end = { - "protocol": 1, - "nonce": nonce, - "stream": "omp_response_end", - } - _blocking_send_frame( - sock, - json.dumps(end, separators=(",", ":")).encode("utf-8"), - ) - - -def _recv_streamed_omp_response_until( - sock: socket.socket, - first_payload: bytes, - nonce: str, - target_kind: str, - deadline: float, - cancel_event: threading.Event | None, -) -> bytes: - try: - manifest = json.loads(first_payload.decode("utf-8")) - except (UnicodeError, json.JSONDecodeError): - return first_payload - if not isinstance(manifest, Mapping) or "stream" not in manifest: - return first_payload - if ( - target_kind != _OMP_SESSION_TURN_KIND - or set(manifest) - != { - "protocol", - "nonce", - "stream", - "chunks", - "total_bytes", - "chunk_bytes", - } - or manifest["protocol"] != 1 - or manifest["stream"] != "omp_response" - or type(manifest["nonce"]) is not str - or not secrets.compare_digest(manifest["nonce"], nonce) - or type(manifest["chunks"]) is not int - or manifest["chunks"] <= 0 - or type(manifest["total_bytes"]) is not int - or manifest["total_bytes"] <= 0 - or type(manifest["chunk_bytes"]) is not int - or manifest["chunk_bytes"] <= 0 - or manifest["chunk_bytes"] > _OMP_IPC_RESPONSE_CHUNK_BYTES - or manifest["chunks"] - != ( - manifest["total_bytes"] + manifest["chunk_bytes"] - 1 - ) - // manifest["chunk_bytes"] - ): - raise _TurnReadFailed - assembled = bytearray() - for index in range(manifest["chunks"]): - chunk = _recv_frame_until( - sock, - deadline, - cancel_event, - manifest["chunk_bytes"], - ) - expected = ( - manifest["chunk_bytes"] - if index + 1 < manifest["chunks"] - else manifest["total_bytes"] - len(assembled) - ) - if len(chunk) != expected: - raise _TurnReadFailed - assembled.extend(chunk) - if len(assembled) != manifest["total_bytes"]: - raise _TurnReadFailed - end_payload = _recv_frame_until(sock, deadline, cancel_event, 1024) - try: - end = json.loads(end_payload.decode("utf-8")) - except (UnicodeError, json.JSONDecodeError) as exc: - raise _TurnReadFailed from exc - if ( - not isinstance(end, Mapping) - or set(end) != {"protocol", "nonce", "stream"} - or end["protocol"] != 1 - or end["stream"] != "omp_response_end" - or type(end["nonce"]) is not str - or not secrets.compare_digest(end["nonce"], nonce) - ): - raise _TurnReadFailed - while True: - remaining = _check_ipc_deadline(deadline, cancel_event) - readable, _writable, _exceptional = select.select( - [sock], - [], - [], - min(0.05, remaining), - ) - if not readable: - continue - if sock.recv(1): - raise _TurnReadFailed - break - return bytes(assembled) - - - - -def _omp_publication_commit( - publication: _OmpCachePublication, - content: Mapping[str, Any] | None, -) -> Mapping[str, Any] | None: - return _publish_omp_cache_state( - publication.cache_key, - publication.prior_state_value, - publication.updated_state, - content, - publication.binding_generation, - ) -def _codex_publication_commit( - publication: _CodexCachePublication, - content: Mapping[str, Any] | None, -) -> Mapping[str, Any] | None: - resolution = _resolve_codex_session(publication.cache_key[1]) - if ( - resolution is None - or resolution.status != "found" - or resolution.root != publication.cache_key[0] - or resolution.root_file_id != publication.updated_state.root_file_id - or resolution.generation != publication.resolver_generation - or resolution.canonical_path != publication.updated_state.canonical_path - or resolution.file_id != publication.updated_state.file_id - ): - return None - return _publish_codex_cache_state( - publication.cache_key, - publication.prior_state_value, - publication.updated_state, - content, - publication.binding_generation, - ) - - -def _file_publication_commit( - publication: _OmpCachePublication | _CodexCachePublication, - content: Mapping[str, Any] | None, -) -> Mapping[str, Any] | None: - if isinstance(publication, _CodexCachePublication): - return _codex_publication_commit(publication, content) - return _omp_publication_commit(publication, content) - - - - - - -def _file_turn_child(channel: socket.socket) -> None: - """Spawn entry point using one bounded, source-tagged private frame.""" - nonce = "" - try: - payload = _blocking_recv_frame(channel, _CODEX_STATE_IPC_MAX_BYTES) - request = json.loads(payload.decode("utf-8")) - if not isinstance(request, Mapping) or set(request) != { - "protocol", - "nonce", - "target_kind", - "target_value", - "parser_state", - }: - raise ValueError("invalid turn reader request") - nonce = request["nonce"] - target_kind = request["target_kind"] - target_value = request["target_value"] - parser_state = request["parser_state"] - if ( - request["protocol"] != 1 - or type(nonce) is not str - or not nonce - or type(target_kind) is not str - or type(target_value) is not str - or not isinstance(parser_state, Mapping) - or set(parser_state) != {"source", "state"} - ): - raise ValueError("invalid turn reader request values") - source = parser_state["source"] - content: Mapping[str, Any] | None - response_state: dict[str, Any] | None = None - bytes_read = 0 - disposition = "ok" - if target_kind == _CODEX_SESSION_TURN_KIND: - if source != "codex": - raise ValueError("wrong parser state source") - supplied = _deserialize_codex_state(parser_state["state"]) - if supplied is None: - raise ValueError("missing Codex parser state") - parsed = _read_codex_session_turn_with_state(target_value, supplied) - if parsed is None: - content = None - disposition = "missing" - else: - content, updated_state, bytes_read = parsed - response_state = { - "source": "codex", - "state": _serialize_codex_state(updated_state), - } - elif target_kind == _OMP_SESSION_TURN_KIND: - if source != "omp": - raise ValueError("wrong parser state source") - prior_state = _deserialize_omp_state(parser_state["state"]) - parsed = _read_omp_session_turn_with_state(target_value, prior_state) - if parsed is None: - content = None - disposition = "missing" - else: - content, updated_state, bytes_read = parsed - response_state = { - "source": "omp", - "state": _serialize_omp_state(updated_state), - } - else: - raise ValueError("unsupported turn reader target") - response = { - "protocol": 1, - "nonce": nonce, - "disposition": disposition, - "content": dict(content) if content is not None else None, - "parser_state": response_state, - "bytes_read": bytes_read, - } - encoded = json.dumps( - response, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - if target_kind == _CODEX_SESSION_TURN_KIND: - if len(encoded) > _CODEX_IPC_FRAME_MAX_BYTES: - raise ValueError("oversized Codex turn reader response") - _blocking_send_frame(channel, encoded) - elif len(encoded) > _OMP_IPC_RESPONSE_CHUNK_BYTES: - _blocking_send_streamed_omp_response(channel, encoded, nonce) - else: - _blocking_send_frame(channel, encoded) - except BaseException: - try: - response = { - "protocol": 1, - "nonce": nonce, - "disposition": "failed", - "content": None, - "parser_state": None, - "bytes_read": 0, - } - _blocking_send_frame( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - ) - except BaseException: - pass - finally: - channel.close() - - -def _terminate_and_reap(process: multiprocessing.Process) -> None: - """Reap within a fixed grace after the request deadline under normal POSIX scheduling.""" - if process.pid is None: - return - teardown_deadline = time.monotonic() + _OMP_TEARDOWN_GRACE_SECONDS - if not process.is_alive(): - process.join(0) - return - process.terminate() - process.join(min(0.05, max(0.0, teardown_deadline - time.monotonic()))) - if process.is_alive(): - process.kill() - process.join(max(0.0, teardown_deadline - time.monotonic())) - if process.is_alive(): - # Never trade the caller's hard teardown bound for an unbounded join. - # A SIGKILL-resistant kernel task remains an OS-level exceptional case. - process.kill() - - -def _read_file_turn_isolated( - target_kind: str, - target_value: str, - *, - timeout_seconds: float, - cancel_event: threading.Event | None = None, - defer_cache: bool = False, -) -> Mapping[str, Any] | _ObservedFileTurn | object | None: - """Read through one bounded source-tagged IPC request.""" - deadline = time.monotonic() + float(timeout_seconds) - if ( - type(target_kind) is not str - or type(target_value) is not str - or len(target_value) > _OMP_TARGET_MAX_CHARS - ): - raise _TurnReadFailed - codex_resolution: _CodexPathResolution | None = None - codex_cache_key: tuple[str, str] | None = None - codex_prior: _CodexSessionState | None = None - omp_cache_key: str | None = None - omp_prior: _OmpSessionState | None = None - prior_state_value: dict[str, Any] | None = None - binding_generation: int | None = None - parser_state: dict[str, Any] - if target_kind == _CODEX_SESSION_TURN_KIND: - canonical_id = _canonical_codex_session_id(target_value) - if canonical_id is None: - return None - codex_resolution = _resolve_codex_session(canonical_id) - if codex_resolution is None or codex_resolution.status != "found": - return None - codex_cache_key = (codex_resolution.root, canonical_id) - with _CODEX_SESSION_CACHE_LOCK: - codex_prior = _codex_cache_get_locked(codex_cache_key) - if ( - codex_prior is not None - and ( - codex_prior.resolver_generation != codex_resolution.generation - or codex_prior.root_file_id != codex_resolution.root_file_id - or codex_prior.canonical_path != codex_resolution.canonical_path - or codex_prior.file_id != codex_resolution.file_id - ) - ): - _CODEX_SESSION_CACHE.pop(codex_cache_key, None) - codex_prior = None - prior_state_value = _serialize_codex_state(codex_prior) - binding_generation = _codex_cache_binding_generation_locked( - codex_cache_key - ) - if ( - codex_prior is not None - and _codex_checkpoint_matches_stat(codex_prior, codex_resolution) - ): - _check_ipc_deadline(deadline, cancel_event) - observer = _CODEX_ISOLATED_READ_OBSERVER - if observer is not None: - observer(0) - return _UNCHANGED_TURN - supplied = codex_prior or _CodexSessionState( - resolver_generation=codex_resolution.generation, - root=codex_resolution.root, - root_file_id=codex_resolution.root_file_id, - session_id=canonical_id, - canonical_path=codex_resolution.canonical_path or "", - file_id=None, - observed_size=0, - mtime_ns=0, - ctime_ns=0, - committed_offset=0, - partial_record=b"", - active_turn_id="", - last_content_turn_id="", - turn_open=False, - final_seen=False, - complete=False, - stream_spans=(), - ) - parser_state = { - "source": "codex", - "state": _serialize_codex_state(supplied), - } - elif target_kind == _OMP_SESSION_TURN_KIND: - omp_cache_key = _omp_cache_key(target_value) - if omp_cache_key is not None: - with _OMP_SESSION_CACHE_LOCK: - omp_prior = _omp_cache_get_locked(omp_cache_key) - prior_state_value = _serialize_omp_state(omp_prior) - binding_generation = _omp_cache_binding_generation_locked( - omp_cache_key - ) - try: - unchanged = ( - omp_prior is not None - and _omp_checkpoint_matches_stat( - omp_prior, - Path(omp_cache_key).stat(), - ) - ) - except OSError as exc: - raise _TurnReadFailed from exc - if unchanged: - _check_ipc_deadline(deadline, cancel_event) - observer = _OMP_ISOLATED_READ_OBSERVER - if observer is not None: - observer(0) - return _UNCHANGED_TURN - parser_state = {"source": "omp", "state": prior_state_value} - else: - raise _TurnReadFailed - nonce = secrets.token_urlsafe(32) - request = { - "protocol": 1, - "nonce": nonce, - "target_kind": target_kind, - "target_value": target_value, - "parser_state": parser_state, - } - request_payload = json.dumps( - request, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - maximum_request = ( - _OMP_REQUEST_MAX_BYTES - if target_kind == _OMP_SESSION_TURN_KIND - else _CODEX_STATE_IPC_MAX_BYTES - ) - if len(request_payload) > maximum_request: - raise _TurnReadFailed - _check_ipc_deadline(deadline, cancel_event) - context = multiprocessing.get_context("spawn") - parent_channel, child_channel = socket.socketpair() - process: multiprocessing.Process | None = None - try: - parent_channel.setblocking(False) - process = context.Process( - target=_file_turn_child, - args=(child_channel,), - name="tendwire-turn-reader", - ) - process.start() - child_channel.close() - _check_ipc_deadline(deadline, cancel_event) - _send_frame_until(parent_channel, request_payload, deadline, cancel_event) - response_payload = _recv_frame_until( - parent_channel, - deadline, - cancel_event, - ( - _OMP_IPC_RESPONSE_CHUNK_BYTES - if target_kind == _OMP_SESSION_TURN_KIND - else _CODEX_IPC_FRAME_MAX_BYTES - ), - ) - response_payload = _recv_streamed_omp_response_until( - parent_channel, - response_payload, - nonce, - target_kind, - deadline, - cancel_event, - ) - _check_ipc_deadline(deadline, cancel_event) - response = json.loads(response_payload.decode("utf-8")) - _check_ipc_deadline(deadline, cancel_event) - process.join(max(0.0, deadline - time.monotonic())) - if process.is_alive(): - raise _TurnReadTimeout - if not isinstance(response, Mapping) or set(response) != { - "protocol", - "nonce", - "disposition", - "content", - "parser_state", - "bytes_read", - }: - raise _TurnReadFailed - response_nonce = response["nonce"] - disposition = response["disposition"] - content = response["content"] - bytes_read = response["bytes_read"] - response_state = response["parser_state"] - if ( - response["protocol"] != 1 - or type(response_nonce) is not str - or not secrets.compare_digest(response_nonce, nonce) - or disposition not in {"ok", "missing"} - or type(bytes_read) is not int - or bytes_read < 0 - or ( - target_kind == _CODEX_SESSION_TURN_KIND - and bytes_read > _CODEX_POLL_MAX_BYTES - ) - ): - raise _TurnReadFailed - if disposition == "missing": - if content is not None or response_state is not None or bytes_read != 0: - raise _TurnReadFailed - return None - if content is not None and not _isolated_content_is_valid(content): - raise _TurnReadFailed - if ( - not isinstance(response_state, Mapping) - or set(response_state) != {"source", "state"} - ): - raise _TurnReadFailed - if target_kind == _OMP_SESSION_TURN_KIND: - if response_state["source"] != "omp" or omp_cache_key is None: - raise _TurnReadFailed - updated_omp = _deserialize_omp_state(response_state["state"]) - if updated_omp is None: - raise _TurnReadFailed - observer = _OMP_ISOLATED_READ_OBSERVER - if observer is not None: - observer(bytes_read) - publication: _OmpCachePublication | _CodexCachePublication = ( - _OmpCachePublication( - omp_cache_key, - prior_state_value, - updated_omp, - binding_generation, - ) - ) - else: - if ( - response_state["source"] != "codex" - or codex_cache_key is None - or codex_resolution is None - ): - raise _TurnReadFailed - updated_codex = _deserialize_codex_state(response_state["state"]) - if ( - updated_codex is None - or updated_codex.root != codex_resolution.root - or updated_codex.root_file_id != codex_resolution.root_file_id - or updated_codex.session_id != codex_resolution.session_id - or updated_codex.canonical_path - != codex_resolution.canonical_path - or updated_codex.resolver_generation - != codex_resolution.generation - ): - raise _TurnReadFailed - observer = _CODEX_ISOLATED_READ_OBSERVER - if observer is not None: - observer(bytes_read) - publication = _CodexCachePublication( - codex_cache_key, - prior_state_value, - updated_codex, - binding_generation, - codex_resolution.generation, - ) - if defer_cache: - return _ObservedFileTurn(content, publication) - return _file_publication_commit(publication, content) - except _TurnReadTimeout: - raise - except ( - OSError, - EOFError, - BrokenPipeError, - UnicodeError, - ValueError, - json.JSONDecodeError, - ): - raise _TurnReadFailed from None - finally: - parent_channel.close() - child_channel.close() - if process is not None and process.pid is not None: - _terminate_and_reap(process) - - -def _read_turn_for_binding( - config: Config, - binding: WorkerBinding, - *, - timeout_seconds: float | None = None, - cancel_event: threading.Event | None = None, -) -> Mapping[str, Any] | _ObservedFileTurn | object | None: - target_kind = str(binding.turn_target_kind or "") - target_value = str(binding.turn_target_value or "") - if not target_value: - return None - deadline = config.herdr_timeout_seconds if timeout_seconds is None else float(timeout_seconds) - if target_kind in {_CODEX_SESSION_TURN_KIND, _OMP_SESSION_TURN_KIND}: - return _read_file_turn_isolated( - target_kind, - target_value, - timeout_seconds=deadline, - cancel_event=cancel_event, - defer_cache=True, - ) - if target_kind == _PANE_TURN_KIND: - return _read_private_turn( - config, - target_value, - timeout_seconds=deadline, - raise_timeout=True, - cancel_event=cancel_event, - ) - return None - - -def _binding_still_matches( - config: Config, - item: _TurnRefreshItem, -) -> bool: - if config.db_path is None: - return False - try: - current = list_worker_bindings(config.db_path, config.host_id, backend="herdr") - except Exception: - return False - return any( - binding.private_fingerprint == item.key.private_fingerprint - and binding.worker_id == item.worker_id - and binding.worker_fingerprint == item.worker_fingerprint - and str(binding.turn_target_kind or "") == item.turn_target_kind - and str(binding.turn_target_value or "") == item.turn_target_value - for binding in current - ) - - -def _refresh_turn_binding( - config: Config, - binding: WorkerBinding, - *, - pane_target_override: str | None = None, - adapter_timeout_seconds: float | None = None, - cancel_event: threading.Event | None = None, - apply_deadline_monotonic: float | None = None, - observed_at: str | None = None, -) -> TurnRefreshResult: - """Read and atomically apply exactly one immutable private binding. - - Completion-authoritative callers may read through the live pane while the - immutable stored binding remains the ownership and transaction guard. This - prevents a prior agent-session generation from choosing the content source - after Herdr has reported completion for the current pane. - """ - if config.db_path is None or not _eligible_turn_binding(binding): - return TurnRefreshResult("missing", 0) - timeout_seconds = ( - config.herdr_timeout_seconds - if adapter_timeout_seconds is None - else float(adapter_timeout_seconds) - ) - if timeout_seconds <= 0: - return TurnRefreshResult("failed", 0) - item = _TurnRefreshItem.from_binding(binding) - read_binding = binding - if pane_target_override is not None: - live_pane_id = str(pane_target_override).strip() - if not live_pane_id: - return TurnRefreshResult("missing", 0) - read_binding = replace( - binding, - turn_target_kind=_PANE_TURN_KIND, - turn_target_value=live_pane_id, - ) - read_target_kind = str(read_binding.turn_target_kind or "") - current_time = observed_at or utc_timestamp() - grace_seconds = float(config.pending_stale_grace_seconds) - def failed_pending_result(status: Literal["failed", "timeout"]) -> TurnRefreshResult: - if read_target_kind != _PANE_TURN_KIND: - return TurnRefreshResult(status, 0) - if not _binding_still_matches(config, item): - return TurnRefreshResult("stale_binding", 0) - try: - applied_failure = apply_turn_refresh( - config.db_path, - config.host_id, - item.worker_id, - {}, - backend_pending_observation=PendingObservation("read_failed"), - expected_binding=binding, - deadline_monotonic=apply_deadline_monotonic, - cancelled=cancel_event.is_set if cancel_event is not None else None, - observed_at=current_time, - pending_stale_grace_seconds=grace_seconds, - turn_model=config.turn_model, - ) - except Exception: - return TurnRefreshResult(status, 0) - if applied_failure.stale_binding: - return TurnRefreshResult("stale_binding", 0) - return TurnRefreshResult( - status, - 0, - pending_changed=applied_failure.pending_changed, - ) - try: - observed = _read_turn_for_binding( - config, - read_binding, - timeout_seconds=timeout_seconds, - cancel_event=cancel_event, - ) - except _TurnReadTimeout: - return failed_pending_result("timeout") - except Exception: - return failed_pending_result("failed") - if observed is _UNCHANGED_TURN: - return TurnRefreshResult("unchanged", 0) - publication: _OmpCachePublication | _CodexCachePublication | None = None - if isinstance(observed, _ObservedFileTurn): - publication = observed.publication - observed = observed.content - if observed is None: - if not _binding_still_matches(config, item): - return TurnRefreshResult("stale_binding", 0) - _file_publication_commit(publication, None) - return TurnRefreshResult("unchanged", 0) - if observed is None: - return TurnRefreshResult("missing", 0) - if read_target_kind == _PANE_TURN_KIND: - content, pending_observation = _pop_backend_pending_observation(observed) - else: - content, pending_observation = dict(observed), None - if not _binding_still_matches(config, item): - return TurnRefreshResult("stale_binding", 0) - try: - if read_target_kind == _PANE_TURN_KIND: - applied = apply_turn_refresh( - config.db_path, - config.host_id, - item.worker_id, - content or {}, - backend_pending_observation=( - pending_observation - or PendingObservation("read_succeeded_no_prompt") - ), - expected_binding=binding, - deadline_monotonic=apply_deadline_monotonic, - cancelled=cancel_event.is_set if cancel_event is not None else None, - observed_at=current_time, - pending_stale_grace_seconds=grace_seconds, - turn_model=config.turn_model, - ) - elif content is not None: - applied = apply_turn_refresh( - config.db_path, - config.host_id, - item.worker_id, - content, - expected_binding=binding, - deadline_monotonic=apply_deadline_monotonic, - cancelled=cancel_event.is_set if cancel_event is not None else None, - observed_at=current_time, - turn_model=config.turn_model, - ) - else: - return TurnRefreshResult("missing", 0) - except Exception: - return TurnRefreshResult("failed", 0) - if applied.cancelled: - return TurnRefreshResult("timeout", 0) - if applied.stale_binding: - return TurnRefreshResult("stale_binding", 0) - if ( - pending_observation is not None - and pending_observation.kind == "read_failed" - ): - return TurnRefreshResult( - "failed", - int(applied.updated), - bool(applied.pending_changed), - ) - if publication is not None: - _file_publication_commit(publication, content) - updated = int(applied.updated) - return TurnRefreshResult( - "updated" if updated else "unchanged", - updated, - bool(applied.pending_changed), - ) - - -def refresh_turn_binding( - config: Config, - binding: WorkerBinding, - *, - adapter_timeout_seconds: float | None = None, -) -> TurnRefreshResult: - return _refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=adapter_timeout_seconds, - ) - - -def refresh_completed_pane_turn( - config: Config, - pane_id: str, - *, - terminal_id: str | None = None, - binding_private_fingerprint: str | None = None, - adapter_timeout_seconds: float | None = None, -) -> CompletedPaneTurnRefreshResult: - """Capture one completed turn through its current pane adapter target.""" - if config.db_path is None: - return CompletedPaneTurnRefreshResult("store_unavailable") - try: - bindings = list_worker_bindings( - config.db_path, - config.host_id, - backend="herdr", - ) - except Exception: - return CompletedPaneTurnRefreshResult("store_unavailable") - eligible = [binding for binding in bindings if _eligible_turn_binding(binding)] - candidate_tiers: list[list[WorkerBinding]] = [] - if binding_private_fingerprint is not None: - candidate_tiers.append( - [ - binding - for binding in eligible - if binding.private_fingerprint == binding_private_fingerprint - ] - ) - candidate_tiers.append( - [ - binding - for binding in eligible - if binding.turn_target_kind == _PANE_TURN_KIND - and str(binding.turn_target_value or "") == str(pane_id) - ] - ) - if terminal_id is not None: - candidate_tiers.append( - [ - binding - for binding in eligible - if str(binding.target_kind or "") == "terminal_id" - and str(binding.target_value or "") == str(terminal_id) - ] - ) - binding = None - for candidates in candidate_tiers: - if len(candidates) > 1: - return CompletedPaneTurnRefreshResult("binding_ambiguous") - if candidates: - binding = candidates[0] - break - if binding is None: - return CompletedPaneTurnRefreshResult("binding_missing") - refreshed = _refresh_turn_binding( - config, - binding, - pane_target_override=pane_id, - adapter_timeout_seconds=adapter_timeout_seconds, - ) - if refreshed.status not in {"updated", "unchanged", "missing"}: - return CompletedPaneTurnRefreshResult( - refreshed.status, - worker_id=binding.worker_id, - ) - try: - turn_id = latest_turn_id_for_worker( - config.db_path, - config.host_id, - binding.worker_id, - ) - except Exception: - return CompletedPaneTurnRefreshResult( - "store_unavailable", - worker_id=binding.worker_id, - ) - return CompletedPaneTurnRefreshResult( - refreshed.status, - worker_id=binding.worker_id, - refreshed_turn_id=turn_id, - ) - - -def refresh_structured_turn_content( - config: Config, - *, - adapter_timeout_seconds: float | None = None, - max_workers: int | None = None, - total_timeout_seconds: float | None = None, -) -> dict[str, Any]: - """Run one bounded unavailable-only fallback scan with active feeding.""" - if config.db_path is None: - return {"ok": False, "status": "store_unavailable", "updated": 0, "attempted": 0} - try: - bindings = list_worker_bindings(config.db_path, config.host_id, backend="herdr") - except Exception: - return {"ok": False, "status": "store_unavailable", "updated": 0, "attempted": 0} - _prune_omp_cache_for_bindings(bindings) - _prune_codex_cache_for_bindings(bindings) - turn_bindings = [binding for binding in bindings if _eligible_turn_binding(binding)] - worker_limit = max( - 1, - min( - 32, - int( - max_workers - if max_workers is not None - else getattr(config, "turn_refresh_workers", 4) - ), - ), - ) - adapter_deadline = ( - config.herdr_timeout_seconds - if adapter_timeout_seconds is None - else float(adapter_timeout_seconds) - ) - waves = (len(turn_bindings) + worker_limit - 1) // worker_limit - total_deadline = ( - max(1.0, (waves * adapter_deadline) + 1.0) - if total_timeout_seconds is None - else max(0.0, float(total_timeout_seconds)) - ) - deadline = time.monotonic() + total_deadline - shutdown_reserve = min(0.75, total_deadline / 2.0) - work_deadline = deadline - shutdown_reserve - pending_bindings = deque(turn_bindings) - active: dict[Future[TurnRefreshResult], WorkerBinding] = {} - submitted: list[Future[TurnRefreshResult]] = [] - updated = 0 - deadline_reached = False - pool = ThreadPoolExecutor( - max_workers=worker_limit, - thread_name_prefix="tendwire-turn-fallback", - ) - def fallback_cancelled() -> bool: - return time.monotonic() >= deadline - - - def collect(done: set[Future[TurnRefreshResult]]) -> None: - nonlocal updated - for future in done: - active.pop(future, None) - try: - updated += future.result().updated - except Exception: - pass - - while pending_bindings or active: - while ( - pending_bindings - and len(active) < worker_limit - and time.monotonic() < work_deadline - ): - binding = pending_bindings.popleft() - remaining_for_binding = max( - 0.001, - min(adapter_deadline, work_deadline - time.monotonic()), - ) - future = pool.submit( - _refresh_turn_binding, - config, - binding, - adapter_timeout_seconds=remaining_for_binding, - apply_deadline_monotonic=work_deadline, - ) - active[future] = binding - submitted.append(future) - if not active: - deadline_reached = bool(pending_bindings) - break - remaining = max(0.0, work_deadline - time.monotonic()) - done, _ = wait(active, timeout=remaining, return_when=FIRST_COMPLETED) - if not done: - deadline_reached = True - break - collect(done) - if time.monotonic() >= work_deadline and (pending_bindings or active): - deadline_reached = True - break - if deadline_reached: - for future in active: - future.cancel() - remaining = max(0.0, deadline - time.monotonic()) - done, _ = wait(active, timeout=remaining) - collect(done) - # Supported readers terminate and reap at their per-binding deadlines - # inside the reserved shutdown window. Never use an executor context or - # wait=True: the absolute fallback deadline owns return latency. - pool.shutdown(wait=False, cancel_futures=True) - if not deadline_reached: - try: - prune_backend_pending( - config.db_path, - config.host_id, - { - binding.private_fingerprint - for binding in bindings - if binding.turn_target_kind == _PANE_TURN_KIND - }, - deadline_monotonic=deadline, - cancelled=fallback_cancelled, - observed_at=utc_timestamp(), - ) - except Exception: - pass - deadline_reached = fallback_cancelled() - return { - "ok": not deadline_reached, - "status": "deadline_exceeded" if deadline_reached else "ok", - "updated": updated, - "attempted": len(submitted), - } - - -class TurnIngestionScheduler: - """One bounded coordinator and fixed worker pool for turn ingestion.""" - - def __init__( - self, - config: Config, - *, - refresh_interval_seconds: float | None = None, - max_workers: int | None = None, - queue_capacity: int = _TURN_INGESTION_QUEUE_CAPACITY, - adapter_timeout_seconds: float | None = None, - clock: Callable[[], float] = time.monotonic, - utc_clock: Callable[[], str] = utc_timestamp, - reader: Callable[..., TurnRefreshResult] = refresh_turn_binding, - ) -> None: - self.config = config - self.refresh_interval_seconds = float( - refresh_interval_seconds - if refresh_interval_seconds is not None - else getattr(config, "turn_refresh_interval_seconds", 2.0) - ) - self.max_workers = int( - max_workers - if max_workers is not None - else getattr(config, "turn_refresh_workers", 4) - ) - self.queue_capacity = int(queue_capacity) - self.adapter_timeout_seconds = float( - config.herdr_timeout_seconds - if adapter_timeout_seconds is None - else adapter_timeout_seconds - ) - if self.refresh_interval_seconds <= 0: - raise ValueError("refresh_interval_seconds must be positive") - if not 1 <= self.max_workers <= 32: - raise ValueError("max_workers must be between 1 and 32") - if self.queue_capacity <= 0: - raise ValueError("queue_capacity must be positive") - if self.adapter_timeout_seconds <= 0: - raise ValueError("adapter_timeout_seconds must be positive") - self._clock = clock - self._utc_clock = utc_clock - self._reader = reader - self._uses_default_reader = reader is refresh_turn_binding - self._cancel_event = threading.Event() - self._condition = threading.Condition(threading.RLock()) - self._queue: deque[_TurnRefreshItem] = deque() - self._queued: set[TurnRefreshKey] = set() - self._running: dict[TurnRefreshKey, tuple[_TurnRefreshItem, Future[TurnRefreshResult], float]] = {} - self._dirty: set[TurnRefreshKey] = set() - self._latest: dict[TurnRefreshKey, _TurnRefreshItem] = {} - self._deferred_reruns: dict[TurnRefreshKey, _TurnRefreshItem] = {} - self._executor: ThreadPoolExecutor | None = None - self._coordinator: threading.Thread | None = None - self._accepting = False - self._started = False - self._force_exit = False - self._rescan_requested = False - self._next_scan = 0.0 - self._scan_cursor = 0 - self._stopping = False - self._refreshed = 0 - self._failed = 0 - self._timed_out = 0 - self._coalesced = 0 - self._queue_full = 0 - self._last_success: str | None = None - self._last_success_clock: float | None = None - self._last_duration_ms: float | None = None - self._last_completed_outcome: str | None = None - self._consecutive_failures = 0 - self._scan_failed = False - self._scan_retry_remaining = 1 - self._binding_retry_remaining: dict[TurnRefreshKey, int] = {} - self._binding_retry_due: dict[ - TurnRefreshKey, - tuple[_TurnRefreshItem, float], - ] = {} - self._worker_excluded: Callable[[str, str], bool] | None = None - - def set_worker_exclusion( - self, - callback: Callable[[str, str], bool] | None, - ) -> None: - """Exclude workers currently owned by a stronger semantic source.""" - with self._condition: - self._worker_excluded = callback - self._rescan_requested = True - self._condition.notify_all() - - def _is_worker_excluded(self, binding: WorkerBinding) -> bool: - callback = self._worker_excluded - if callback is None: - return False - try: - return callback(binding.worker_id, binding.worker_fingerprint) is True - except Exception: - # Authority uncertainty must not let the legacy reader overwrite - # a potentially ACP-owned worker. - return True - - def start(self) -> None: - with self._condition: - if self._started: - return - if self._stopping: - raise RuntimeError("turn ingestion scheduler cannot restart") - self._executor = ThreadPoolExecutor( - max_workers=self.max_workers, - thread_name_prefix="tendwire-turn-ingestion", - ) - self._accepting = True - self._started = True - self._rescan_requested = True - self._next_scan = self._clock() - self._coordinator = threading.Thread( - target=self._coordinate, - name="tendwire-turn-coordinator", - daemon=False, - ) - self._coordinator.start() - - def request_refresh(self) -> None: - with self._condition: - if not self._accepting: - return - if self._rescan_requested: - self._coalesced += 1 - self._rescan_requested = True - self._condition.notify() - - def _enqueue_locked(self, item: _TurnRefreshItem) -> None: - key = item.key - self._latest[key] = item - if key in self._running: - if key in self._dirty: - self._coalesced += 1 - else: - self._dirty.add(key) - return - if key in self._binding_retry_due: - _previous, due = self._binding_retry_due[key] - self._binding_retry_due[key] = (item, due) - self._coalesced += 1 - return - if key in self._deferred_reruns: - self._deferred_reruns[key] = item - self._coalesced += 1 - return - if key in self._queued: - self._coalesced += 1 - return - if len(self._queue) >= self.queue_capacity: - self._queue_full += 1 - self._latest.pop(key, None) - return - self._queue.append(item) - self._queued.add(key) - - def _scan_bindings(self) -> None: - if self.config.db_path is None: - with self._condition: - self._failed += 1 - self._scan_failed = True - return - try: - bindings = list_worker_bindings( - self.config.db_path, - self.config.host_id, - backend="herdr", - ) - except Exception: - with self._condition: - self._failed += 1 - self._scan_failed = True - if self._accepting and self._scan_retry_remaining > 0: - self._scan_retry_remaining -= 1 - self._next_scan = min( - self._next_scan, - self._clock() + 0.05, - ) - self._condition.notify() - return - _prune_omp_cache_for_bindings(bindings) - _prune_codex_cache_for_bindings(bindings) - items = [ - _TurnRefreshItem.from_binding(binding) - for binding in bindings - if _eligible_turn_binding(binding) and not self._is_worker_excluded(binding) - ] - with self._condition: - self._scan_failed = False - self._scan_retry_remaining = 1 - live_keys = {item.key for item in items} - for key in tuple(self._binding_retry_remaining): - if key not in live_keys: - self._binding_retry_remaining.pop(key, None) - self._binding_retry_due.pop(key, None) - if self._accepting: - if items: - offset = self._scan_cursor % len(items) - items = items[offset:] + items[:offset] - # Rotate each full scan so bindings dropped at the explicit - # queue cap are first on a later cadence instead of starving. - self._scan_cursor = (offset + self.queue_capacity) % len(items) - for item in items: - self._enqueue_locked(item) - self._dispatch_locked() - self._condition.notify() - # Lock order: no scheduler condition is held during store reads/writes, - # adapter/process waits, SQLite transactions, or future waits. - try: - prune_backend_pending( - self.config.db_path, - self.config.host_id, - { - binding.private_fingerprint - for binding in bindings - if binding.turn_target_kind == _PANE_TURN_KIND - }, - cancelled=self._cancel_event.is_set, - observed_at=self._utc_clock(), - ) - except Exception: - pass - - def _binding_for_item(self, item: _TurnRefreshItem) -> WorkerBinding | None: - if self.config.db_path is None: - return None - try: - bindings = list_worker_bindings( - self.config.db_path, - self.config.host_id, - backend="herdr", - ) - except Exception as exc: - raise _BindingLookupFailed from exc - return next( - ( - binding - for binding in bindings - if binding.private_fingerprint == item.key.private_fingerprint - and binding.worker_id == item.worker_id - and binding.worker_fingerprint == item.worker_fingerprint - and str(binding.turn_target_kind or "") == item.turn_target_kind - and str(binding.turn_target_value or "") == item.turn_target_value - ), - None, - ) - - def _run_item(self, item: _TurnRefreshItem) -> TurnRefreshResult: - try: - binding = self._binding_for_item(item) - except _BindingLookupFailed: - return TurnRefreshResult("failed", 0, retry_binding_lookup=True) - if binding is None: - return TurnRefreshResult("stale_binding", 0) - if self._is_worker_excluded(binding): - return TurnRefreshResult("stale_binding", 0) - try: - if self._uses_default_reader: - result = _refresh_turn_binding( - self.config, - binding, - adapter_timeout_seconds=self.adapter_timeout_seconds, - cancel_event=self._cancel_event, - observed_at=self._utc_clock(), - ) - else: - result = self._reader( - self.config, - binding, - adapter_timeout_seconds=self.adapter_timeout_seconds, - ) - except Exception: - result = TurnRefreshResult("failed", 0) - return TurnRefreshResult( - result.status, - result.updated, - result.pending_changed, - binding_validated=True, - ) - - def _future_finished(self, _future: Future[TurnRefreshResult]) -> None: - with self._condition: - self._condition.notify() - - def _collect_finished_locked(self) -> None: - now = self._clock() - for key, (item, future, started_at) in list(self._running.items()): - if not future.done(): - continue - self._running.pop(key, None) - self._last_duration_ms = max(0.0, (now - started_at) * 1000.0) - try: - result = future.result() - except Exception: - result = TurnRefreshResult("failed", 0) - self._last_completed_outcome = result.status - if result.binding_validated or result.status == "stale_binding": - self._binding_retry_remaining.pop(key, None) - self._binding_retry_due.pop(key, None) - if result.status == "timeout": - self._timed_out += 1 - self._consecutive_failures += 1 - elif result.status == "failed": - self._failed += 1 - self._consecutive_failures += 1 - elif result.status == "stale_binding": - self._failed += 1 - else: - self._refreshed += 1 - self._last_success = utc_timestamp() - self._last_success_clock = now - self._consecutive_failures = 0 - if result.retry_binding_lookup: - retry_item = self._latest.get(key, item) - self._dirty.discard(key) - self._latest.pop(key, None) - remaining = self._binding_retry_remaining.get(key, 1) - if self._accepting and remaining > 0: - self._binding_retry_remaining[key] = remaining - 1 - self._binding_retry_due[key] = (retry_item, now + 0.05) - continue - if key in self._dirty: - self._dirty.discard(key) - rerun = self._latest.get(key, item) - if self._accepting and len(self._queue) < self.queue_capacity: - self._queue.append(rerun) - self._queued.add(key) - elif self._accepting: - self._deferred_reruns[key] = rerun - self._queue_full += 1 - else: - self._latest.pop(key, None) - else: - self._latest.pop(key, None) - - def _promote_deferred_locked(self) -> None: - while ( - self._accepting - and self._deferred_reruns - and len(self._queue) < self.queue_capacity - ): - key, item = next(iter(self._deferred_reruns.items())) - del self._deferred_reruns[key] - self._queue.append(item) - self._queued.add(key) - - def _promote_binding_retries_locked(self, now: float) -> None: - for key, (item, due) in tuple(self._binding_retry_due.items()): - if due > now: - continue - self._binding_retry_due.pop(key, None) - self._enqueue_locked(item) - - - def _dispatch_locked(self) -> None: - executor = self._executor - if executor is None: - return - self._promote_deferred_locked() - self._promote_binding_retries_locked(self._clock()) - while self._accepting and self._queue and len(self._running) < self.max_workers: - item = self._queue.popleft() - self._queued.discard(item.key) - self._binding_retry_due.pop(item.key, None) - self._promote_deferred_locked() - future = executor.submit(self._run_item, item) - self._running[item.key] = (item, future, self._clock()) - future.add_done_callback(self._future_finished) - - def _coordinate(self) -> None: - while True: - run_scan = False - with self._condition: - self._collect_finished_locked() - if self._force_exit: - return - if not self._accepting and not self._running: - return - now = self._clock() - self._promote_binding_retries_locked(now) - if self._accepting and ( - self._rescan_requested or now >= self._next_scan - ): - run_scan = True - self._rescan_requested = False - self._next_scan = now + self.refresh_interval_seconds - self._dispatch_locked() - if not run_scan: - timeout: float | None = None - if self._accepting: - timeout = max(0.0, self._next_scan - self._clock()) - if self._binding_retry_due: - retry_timeout = max( - 0.0, - min(due for _item, due in self._binding_retry_due.values()) - - self._clock(), - ) - timeout = min(timeout, retry_timeout) - self._condition.wait(timeout) - continue - self._scan_bindings() - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - timeout = ( - self.adapter_timeout_seconds + 1.0 - if flush_timeout_seconds is None - else max(0.0, float(flush_timeout_seconds)) - ) - with self._condition: - if self._stopping: - return - self._stopping = True - self._accepting = False - self._rescan_requested = False - self._queue.clear() - self._queued.clear() - self._dirty.clear() - self._deferred_reruns.clear() - self._latest.clear() - self._binding_retry_due.clear() - self._binding_retry_remaining.clear() - self._cancel_event.set() - self._condition.notify_all() - coordinator = self._coordinator - executor = self._executor - coordinator_clean = True - if coordinator is not None: - coordinator.join(timeout) - if coordinator.is_alive(): - coordinator_clean = False - with self._condition: - self._force_exit = True - self._condition.notify_all() - coordinator.join(0.25) - coordinator_clean = not coordinator.is_alive() - if executor is not None: - executor.shutdown(wait=coordinator_clean, cancel_futures=True) - - def operational_status(self) -> Mapping[str, Any]: - with self._condition: - now = self._clock() - stale_age = ( - None - if self._last_success_clock is None - else max(0.0, now - self._last_success_clock) - ) - if self._stopping: - status = "stopping" - elif self._scan_failed or self._consecutive_failures > 0: - status = "degraded" - elif self._last_success_clock is None: - status = "stale" - elif stale_age is not None and stale_age > max( - self.refresh_interval_seconds * 3.0, - self.adapter_timeout_seconds * 2.0, - ): - status = "stale" - else: - status = "healthy" - return { - "status": status, - "queue_depth": len(self._queue), - "active": len(self._running), - "refreshed": self._refreshed, - "failed": self._failed, - "timed_out": self._timed_out, - "coalesced": self._coalesced, - "queue_full": self._queue_full, - "last_success": self._last_success, - "last_duration_ms": self._last_duration_ms, - "stale_age_seconds": stale_age, - "max_workers": self.max_workers, - "queue_capacity": self.queue_capacity, - "refresh_interval_seconds": self.refresh_interval_seconds, - "adapter_timeout_seconds": self.adapter_timeout_seconds, - } diff --git a/src/tendwire/cli.py b/src/tendwire/cli.py index 7b26b2a..e15500a 100644 --- a/src/tendwire/cli.py +++ b/src/tendwire/cli.py @@ -23,8 +23,7 @@ herdr_backend_health, rehydrate_workers_from_bindings, ) -from .backends.herdr_turns import refresh_structured_turn_content -from .config import Config, load_config +from .config import DEFAULT_TURN_MODEL, Config, load_config from .core.actions import CommandContext, execute_command from .core.attention import attention_payload_from_snapshot from .core.commands import ( @@ -698,7 +697,7 @@ def observe_public_snapshot( save_snapshot( config.db_path, snapshot, - turn_model=config.turn_model, + turn_model=DEFAULT_TURN_MODEL, observation=SnapshotObservationContext( authority=authority, observed_at=health.observed_at, @@ -1099,13 +1098,6 @@ def cmd_turns( "status": "store_unavailable", } else: - if cursor is None and since is None: - refresh_structured_turn_content( - config, - adapter_timeout_seconds=config.herdr_timeout_seconds, - max_workers=config.turn_refresh_workers, - total_timeout_seconds=config.herdr_timeout_seconds + 1.0, - ) payload = turns_payload_from_store( config.db_path, config.host_id, @@ -1113,8 +1105,7 @@ def cmd_turns( limit=limit, cursor=cursor, since=since, - turn_refresh_interval_seconds=config.turn_refresh_interval_seconds, - turn_model=config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ) elif daemon_attempt.error_kind == "timeout": payload = { @@ -1195,7 +1186,7 @@ def cmd_turn_content_get(config: Config, args: argparse.Namespace) -> int: field=args.field, cursor=args.cursor, schema_version=1, - turn_model=config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ) print(_content_payload_json(payload, indent=2)) return 0 if payload.get("ok") is not False and isinstance(payload.get("text"), str) else 1 @@ -1295,7 +1286,7 @@ def cmd_turn_delta(config: Config, args: argparse.Namespace) -> int: watermark=args.watermark, cursor=args.cursor, limit=args.limit, - turn_model=config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ) elif daemon_attempt.error_kind == "timeout": payload = { @@ -1647,7 +1638,7 @@ def cmd_connector(config: Config, args: argparse.Namespace) -> int: max_lease_seconds=config.connector_max_claim_ttl_seconds, ack_ttl_seconds=config.connector_ack_ttl_seconds, max_attempts=config.max_outbox_attempts, - turn_model=config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ).dispatch(method, params) print(_connector_payload_json(payload, indent=2)) return 0 if payload.get("ok") is not False else 1 diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 30c4e51..9e4366a 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -20,11 +20,8 @@ DISPOSITION_TERMINAL_UNCERTAIN, STATUS_ACCEPTED, STATUS_ANSWER_IN_PROGRESS, - STATUS_AMBIGUOUS_BACKEND_TARGET, STATUS_AMBIGUOUS_TARGET, - STATUS_BACKEND_FAILED, STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, STATUS_DRY_RUN, STATUS_DECISION_NOT_PENDING, STATUS_DUPLICATE_REQUEST, @@ -50,14 +47,12 @@ validate_request, worker_candidate, ) -from .core.models import BackendHealth, Snapshot, Worker, WorkerBinding +from .core.models import BackendHealth, Snapshot, Worker from .core.projector import project_from_observations -from .backends.herdr_decision import calibrate_decision_steps from .store.sqlite import ( abandon_backend_pending_choice_claim, abandon_command_request_reservation, backend_pending_choice_terminal_effect, - claim_backend_pending_choice, claim_backend_pending_decision, command_reservation_is_live, envelope_to_receipt_json, @@ -67,14 +62,11 @@ get_command_request, linked_turn_for_submission, latest_snapshot, - list_worker_bindings, mark_command_send_started, - record_command_send_queued, recover_unresolved_command_send, reserve_command_request, reserve_terminal_command_replay, settle_submission_link_for_request, - start_backend_pending_choice_send, start_backend_pending_decision_send, ) @@ -84,26 +76,8 @@ {"send_instruction", "answer_pending", "answer_decision"} ) _LEGACY_V0_REPLAY_WORKER_ID = "legacy-v0-replay-only" -_PENDING_CHANGED_MESSAGE = "pending interaction changed or is no longer answerable" _DISALLOWED_SEND_STATUSES = frozenset({"closed", "failed", "unknown"}) _AMBIGUOUS_BINDING_REASONS = frozenset({"duplicate_backend_target", "not_unique"}) -_PRIVATE_PANE_CLEAR_KEY_SEQUENCES = ( - ("ctrl+u",), - ("ctrl+a", "ctrl+k"), - ("ctrl+a", "backspace"), -) -_PANE_SUBMIT_TARGET_KINDS = frozenset( - { - "agent_id", - "agent", - "name", - "label", - "terminal_id", - "pane_id", - } -) - -SocketClientFactory = Callable[[Config], Any] class AcpPromptRoute(Protocol): @@ -145,7 +119,6 @@ def steer( AcpPromptRouter = Callable[[Worker], AcpPromptRoute | None] -AcpWorkerOwner = Callable[[str, str], bool] class AcpPermissionDecisionRouter(Protocol): @@ -156,12 +129,6 @@ def owns_permission_decision(self, decision: Any) -> bool: ... def answer_permission_decision(self, decision: Any, *, timeout: float) -> None: ... -@dataclass(frozen=True) -class ResolvedCommandTarget: - worker: Worker - binding: WorkerBinding - - def _raw_payload_from_mapping(params: Mapping[str, Any]) -> str: return json.dumps( dict(params), @@ -170,17 +137,6 @@ def _raw_payload_from_mapping(params: Mapping[str, Any]) -> str: separators=(",", ":"), ) - - - -def _default_socket_client_factory(config: Config) -> Any: - from .backends.herdr_socket import HerdrSocketClient - - return HerdrSocketClient(timeout=config.herdr_timeout_seconds) - - - - def _backend_health(snapshot: Snapshot) -> BackendHealth: for health in snapshot.backend_health: if health.name == HERDR_BACKEND: @@ -275,67 +231,6 @@ def _target_resolution_error( ) -def _binding_error(request: CommandRequest, status: str, message: str) -> CommandEnvelope: - return CommandEnvelope.from_result( - request, - ok=False, - status=status, - error=error_value(status, message), - ) - - -def _binding_for_worker( - request: CommandRequest, - worker: Worker, - bindings: list[WorkerBinding], -) -> ResolvedCommandTarget | CommandEnvelope: - worker_bindings = [ - binding - for binding in bindings - if binding.backend == HERDR_BACKEND and binding.worker_id == worker.id - ] - if not worker_bindings: - return _binding_error( - request, - STATUS_BACKEND_UNSUPPORTED, - "target has no backend-owned sendable private binding", - ) - - exact = [binding for binding in worker_bindings if binding.worker_fingerprint == worker.fingerprint] - if not exact: - return _binding_error( - request, - STATUS_STALE_TARGET, - "target private binding is stale for the current worker", - ) - if len(exact) != 1: - return _binding_error( - request, - STATUS_AMBIGUOUS_BACKEND_TARGET, - "target resolves to an ambiguous backend send target", - ) - - binding = exact[0] - if ( - not binding.sendable - or not binding.target_value - or binding.target_kind not in _PANE_SUBMIT_TARGET_KINDS - ): - if (binding.reason or "") in _AMBIGUOUS_BINDING_REASONS: - return _binding_error( - request, - STATUS_AMBIGUOUS_BACKEND_TARGET, - "target resolves to an ambiguous backend send target", - ) - return _binding_error( - request, - STATUS_BACKEND_UNSUPPORTED, - "target has no backend-owned sendable private binding", - ) - - return ResolvedCommandTarget(worker=worker, binding=binding) - - def _resolve_authoritative_worker( request: CommandRequest, snapshot: Snapshot, @@ -371,125 +266,6 @@ def _worker_status_error( ) -def _socket_request(client: Any, method: str, params: Mapping[str, Any], *, timeout: float) -> Any: - if not hasattr(client, "request"): - raise TypeError("socket client does not expose generic request") - return client.request(method, params, timeout=timeout) - - -def _pane_id_from_agent_info(value: Any) -> str: - if not isinstance(value, Mapping): - return "" - result = value.get("result") - agent = result.get("agent") if isinstance(result, Mapping) else None - if not isinstance(agent, Mapping): - agent = value.get("agent") - if not isinstance(agent, Mapping): - return "" - pane_id = agent.get("pane_id") or agent.get("paneId") - return str(pane_id or "").strip() - - -def _pane_id_from_terminal_listing(value: Any, terminal_id: str) -> str: - if not isinstance(value, Mapping): - raise ValueError("invalid agent.list response") - agents = value.get("agents") - if not isinstance(agents, list): - raise ValueError("invalid agent.list agents") - matches: set[str] = set() - for agent in agents: - if not isinstance(agent, Mapping): - raise ValueError("invalid agent.list entry") - listed_terminal_id = agent.get("terminal_id") - listed_pane_id = agent.get("pane_id") - if ( - not isinstance(listed_terminal_id, str) - or not listed_terminal_id - or not isinstance(listed_pane_id, str) - or not listed_pane_id.strip() - ): - raise ValueError("invalid agent.list entry") - if listed_terminal_id == terminal_id: - matches.add(listed_pane_id.strip()) - if len(matches) > 1: - raise ValueError("ambiguous agent.list terminal_id match") - return next(iter(matches), "") - - -def _private_pane_id_for_binding(client: Any, binding: WorkerBinding, *, timeout: float) -> str: - if binding.target_kind == "pane_id": - return str(binding.target_value or "").strip() - try: - response = _socket_request( - client, - "agent.get", - {"target": binding.target_value}, - timeout=timeout, - ) - except Exception as exc: # noqa: BLE001 - from .backends.herdr_protocol import HerdrErrorResponse - - # Herdr 0.7.5 stopped resolving terminal-id targets through agent.get - # while agent.list still publishes the terminal_id -> pane_id mapping, - # so a definite target-lookup error falls back to the listing before - # the caller terminalizes the request. - if not isinstance(exc, HerdrErrorResponse) or binding.target_kind != "terminal_id": - raise - listing = _socket_request(client, "agent.list", {}, timeout=timeout) - pane_id = _pane_id_from_terminal_listing( - listing, - str(binding.target_value or ""), - ) - if pane_id: - return pane_id - raise - return _pane_id_from_agent_info(response) - - -def _submit_private_pane_input(client: Any, pane_id: str, instruction_text: str, *, timeout: float) -> None: - # A single ctrl+u is not reliable across all foreground TUIs. Clear stale - # input first, then submit text and Enter in one Herdr operation so the - # foreground application cannot observe a staged prompt between requests. - try: - for keys in _PRIVATE_PANE_CLEAR_KEY_SEQUENCES: - _socket_request( - client, - "pane.send_keys", - {"pane_id": pane_id, "keys": list(keys)}, - timeout=timeout, - ) - except Exception as exc: - raise _PaneInputNotStartedError from exc - _socket_request( - client, - "pane.send_input", - {"pane_id": pane_id, "text": instruction_text, "keys": ["Enter"]}, - timeout=timeout, - ) - - -class _PaneInputNotStartedError(RuntimeError): - """The instruction input operation was never attempted.""" - - -def _agent_prompt_delivery(value: Any) -> str: - if not isinstance(value, Mapping): - return "" - result = value.get("result") - candidate = result if isinstance(result, Mapping) else value - delivery = candidate.get("delivery") - return delivery if isinstance(delivery, str) else "" - - -def _herdr_error_code(exc: BaseException) -> str: - from .backends.herdr_protocol import HerdrErrorResponse - - if not isinstance(exc, HerdrErrorResponse) or not isinstance(exc.error, Mapping): - return "" - code = exc.error.get("code") - return code if isinstance(code, str) else "" - - def _target_state_at_send(worker: Worker) -> str: status = str(worker.status or "").strip().lower().replace("-", "_") return status or "unknown" @@ -503,15 +279,6 @@ def _instruction_text(request: CommandRequest) -> str: -def _backend_failure(request: CommandRequest, message: str) -> CommandEnvelope: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_BACKEND_FAILED, - error=error_value(STATUS_BACKEND_FAILED, message), - ) - - def _backend_uncertain(request: CommandRequest, message: str) -> CommandEnvelope: return CommandEnvelope.from_result( request, @@ -566,73 +333,6 @@ def _duplicate_request(request: CommandRequest) -> CommandEnvelope: ) -def _pending_changed_envelope(request: CommandRequest) -> CommandEnvelope: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_STALE_TARGET, - error=error_value(STATUS_STALE_TARGET, _PENDING_CHANGED_MESSAGE), - ) - - -def _pending_public_result( - request: CommandRequest, - claim: Any, - *, - delivery_state: str, -) -> dict[str, Any]: - params = request.params or {} - result: dict[str, Any] = { - "target": {"worker_id": claim.worker_id}, - "pending": { - "id": params.get("pending_id"), - "fingerprint": params.get("pending_fingerprint"), - }, - "choice": {"choice_id": params.get("choice_id")}, - "delivery_state": delivery_state, - } - if delivery_state == "submitted": - result.update( - { - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - } - ) - return result - - -def _pending_claim_has_exact_route(claim: Any) -> bool: - return ( - isinstance(getattr(claim, "worker_id", None), str) - and bool(claim.worker_id) - and isinstance(getattr(claim, "worker_fingerprint", None), str) - and bool(claim.worker_fingerprint) - and isinstance(getattr(claim, "binding_private_fingerprint", None), str) - and bool(claim.binding_private_fingerprint) - and isinstance(getattr(claim, "turn_target_value", None), str) - and bool(claim.turn_target_value.strip()) - and not isinstance(getattr(claim, "picker_ordinal", None), bool) - and isinstance(claim.picker_ordinal, int) - and claim.picker_ordinal >= 1 - ) - - -def _same_pending_route(left: Any, right: Any) -> bool: - return _pending_claim_has_exact_route(left) and _pending_claim_has_exact_route(right) and ( - left.worker_id, - left.worker_fingerprint, - left.binding_private_fingerprint, - left.turn_target_value, - left.picker_ordinal, - ) == ( - right.worker_id, - right.worker_fingerprint, - right.binding_private_fingerprint, - right.turn_target_value, - right.picker_ordinal, - ) - - def _decision_failure_envelope( request: CommandRequest, status: str, @@ -758,15 +458,6 @@ def _safe_transient_pre_send(envelope: CommandEnvelope) -> PreSendFailure: return PreSendFailure(envelope=envelope, certainty=PreSendCertainty.SAFE_TRANSIENT) -def _close_socket_client(client: Any | None) -> None: - if client is None or not hasattr(client, "close"): - return - try: - client.close() - except Exception: - pass - - def _abandon_pending_claim(config: Config, claim_token: str | None) -> bool: if config.db_path is None or not claim_token: return False @@ -799,100 +490,6 @@ def _abandon_request_reservation( return False -def _connect_socket( - config: Config, - request: CommandRequest, - socket_client_factory: SocketClientFactory | None, -) -> Any | CommandEnvelope: - factory = socket_client_factory or _default_socket_client_factory - client: Any | None = None - try: - client = factory(config) - if not hasattr(client, "request"): - raise TypeError("socket client does not expose generic request") - if hasattr(client, "connect"): - client.connect() - return client - except Exception: # noqa: BLE001 - _close_socket_client(client) - return _backend_unavailable(request, "Herdr socket could not be reached") - - -def _resolve_private_pane( - config: Config, - request: CommandRequest, - client: Any, - binding: WorkerBinding, -) -> str | PreSendFailure: - try: - pane_id = _private_pane_id_for_binding( - client, - binding, - timeout=config.herdr_timeout_seconds, - ) - except Exception as exc: # noqa: BLE001 - from .backends.herdr_protocol import HerdrErrorResponse, HerdrProtocolError - from .backends.herdr_socket import ( - HerdrSocketConnectionError, - HerdrSocketDisconnectedError, - HerdrSocketTimeoutError, - ) - - # A definite error response from Herdr is an authoritative answer that - # this target cannot be resolved; a same-ID retry would get the same - # answer, so it terminalizes rather than looping to the retry horizon. - # It is checked before the transport branch because HerdrErrorResponse - # subclasses HerdrProtocolError and must not be mistaken for framing loss. - if isinstance(exc, HerdrErrorResponse): - return _permanent_pre_send( - _backend_failure( - request, - "Herdr socket could not resolve the private send target", - ) - ) - # A transport read that could not complete -- timeout, disconnect, - # connection loss, protocol framing, or an OS-level socket error -- - # proves nothing about the target and never began a send, so it stays - # retryable. - if isinstance( - exc, - HerdrSocketConnectionError - | HerdrSocketTimeoutError - | HerdrSocketDisconnectedError - | HerdrProtocolError, - ) or isinstance(exc, OSError): - return _safe_transient_pre_send( - _backend_unavailable( - request, - "Herdr socket could not resolve the private send target", - ) - ) - # A malformed or unsupported resolution response is a proven target - # property, not a transient operation failure. - if isinstance(exc, (TypeError, ValueError)): - return _permanent_pre_send( - _backend_failure( - request, - "Herdr socket private send target is unsupported", - ) - ) - # An unclassifiable resolution error is not proven safe to retry, so it - # retains the prior terminal behavior rather than looping indefinitely. - return _permanent_pre_send( - _backend_failure( - request, - "Herdr socket private send target resolution failed", - ) - ) - if not pane_id: - # Herdr answered, and the answer is that the target has no resolvable - # pane. That is authoritative target unsuitability, not a read failure. - return _permanent_pre_send( - _backend_failure(request, "Herdr socket private send target has no pane") - ) - return pane_id - - def _transition_payload( request: CommandRequest, *, @@ -1123,74 +720,6 @@ class ReservedCommandMutation: canonical: CanonicalMutation owner_token: str -@dataclass(frozen=True) -class PreparedInstructionMutation: - client: Any - pane_id: str - binding_fingerprint: str - - -def _prepare_instruction( - config: Config, - request: CommandRequest, - worker: Worker, - *, - socket_client_factory: SocketClientFactory | None, -) -> PreparedInstructionMutation | PreSendFailure: - assert config.db_path is not None - try: - bindings = list_worker_bindings( - config.db_path, - config.host_id, - backend=HERDR_BACKEND, - ) - except Exception: - # The binding store raised. This is an operation failure, not a proven - # target property, and no send began: stay retryable under the same ID. - return _safe_transient_pre_send( - _backend_unavailable(request, "private binding store is unavailable") - ) - resolved = _binding_for_worker(request, worker, bindings) - if isinstance(resolved, CommandEnvelope): - # A missing, stale, or ambiguous binding read from current data is a - # proven target property; a same-ID retry would resolve it the same way. - return _permanent_pre_send(resolved) - - binding_fingerprint = str(resolved.binding.private_fingerprint or "").strip() - if not binding_fingerprint: - return _permanent_pre_send( - _binding_error( - request, - STATUS_BACKEND_UNSUPPORTED, - "target private binding has no durable identity", - ) - ) - - # A socket that cannot be reached is an operation failure before any send, - # so it stays retryable rather than burning the request ID. - client_or_error = _connect_socket(config, request, socket_client_factory) - if isinstance(client_or_error, CommandEnvelope): - return _safe_transient_pre_send(client_or_error) - client = client_or_error - # Pane resolution classifies its own outcome: a transport read failure is a - # safe transient, while a definite backend answer (rejection, unsupported, - # or no pane) is a proven-permanent target failure. - pane_or_error = _resolve_private_pane( - config, - request, - client, - resolved.binding, - ) - if isinstance(pane_or_error, PreSendFailure): - _close_socket_client(client) - return pane_or_error - return PreparedInstructionMutation( - client=client, - pane_id=pane_or_error, - binding_fingerprint=binding_fingerprint, - ) - - def _reserve_canonical_request( config: Config, request: CommandRequest, @@ -1527,31 +1056,6 @@ def _accepted_send_envelope( ) -def _queued_send_envelope( - request: CommandRequest, - worker: Worker, -) -> CommandEnvelope: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_PENDING, - disposition=DISPOSITION_IN_PROGRESS, - result={ - "target": {"worker_id": worker.id}, - "delivery_state": "queued", - "transport_state": "queued", - "target_state_at_send": _target_state_at_send(worker), - "turn_id": None, - "observed_turn_state": "pending_observation", - "submission_verdict": "written_to_pty", - }, - error=error_value( - STATUS_PENDING, - "instruction is queued for the next agent turn boundary", - ), - ) - - def _instruction_rejected_envelope( request: CommandRequest, worker: Worker, @@ -1682,419 +1186,61 @@ def _unverified_queued_send_envelope( ) -def _record_queued_send( +def _uncertain_pending_effect( config: Config, - request: CommandRequest, - worker: Worker, - reservation: ReservedCommandMutation, - envelope: CommandEnvelope, -) -> CommandEnvelope: - assert config.db_path is not None + claim_token: str, +) -> Callable[[Any], Any] | None: try: - queued = record_command_send_queued( - config.db_path, + return backend_pending_choice_terminal_effect( host_id=config.host_id, - request_id=request.request_id or "", - canonical_fingerprint=reservation.canonical.fingerprint, - owner_token=reservation.owner_token, - result_json=envelope_to_receipt_json(envelope), - event_payload=_transition_payload( - request, - worker_id=worker.id, - envelope=envelope, - ), + claim_token=claim_token, + accepted=False, ) - except Exception: # noqa: BLE001 - return _recover_request(config, request, reservation.canonical) - if not isinstance(queued, Mapping): - return _recover_request(config, request, reservation.canonical) - return _envelope_from_receipt( - request, - reservation.canonical, - queued.get("receipt"), - ) + except Exception: + return None -def _submit_instruction( +def _validate_pending_decision( config: Config, request: CommandRequest, - worker: Worker, - reservation: ReservedCommandMutation, - prepared: PreparedInstructionMutation, -) -> CommandEnvelope: - assert config.db_path is not None +) -> Any | PreSendFailure: + if config.db_path is None: + return _safe_transient_pre_send( + _backend_unavailable(request, "pending state store is unavailable") + ) + params = request.params or {} + target = request.target or {} try: - send_started = _mark_request_send_started( - config, - request, - reservation, - binding_fingerprint=prepared.binding_fingerprint, - worker=worker, - instruction_text=_instruction_text(request), + validated = claim_backend_pending_decision( + config.db_path, + config.host_id, + str(target.get("worker_id") or ""), + str(params.get("decision_ref") or ""), + params.get("selection") + if isinstance(params.get("selection"), Mapping) + else {}, + claim=False, ) - if isinstance(send_started, CommandEnvelope): - return send_started - if not isinstance(send_started, Mapping): - return _recover_request( - config, + except Exception: + return _safe_transient_pre_send( + _backend_unavailable(request, "pending state store is unavailable") + ) + if validated.status == "validated" and _decision_claim_has_exact_route(validated): + return validated + if validated.status == "acp_authority_unavailable": + return _safe_transient_pre_send( + _backend_unavailable( request, - reservation.canonical, + "ACP permission authority is temporarily unavailable", ) - observed_turn = send_started - - try: - response = _socket_request( - prepared.client, - "agent.prompt", - { - # Herdr 0.7.5 deliberately restricts agent.prompt to a - # current pane id or a unique live agent name. Private - # bindings may instead be keyed by terminal id, so use the - # pane resolved and validated during the pre-send phase. - "target": prepared.pane_id, - "text": _instruction_text(request), - "wait": { - "until": ["working"], - "timeout_ms": max( - 1, - int(config.herdr_timeout_seconds * 1000), - ), - }, - }, - timeout=config.herdr_timeout_seconds + 1.0, - ) - except Exception as exc: # noqa: BLE001 - verdict = _herdr_error_code(exc) - if verdict in { - "agent_not_ready", - "agent_target_ambiguous", - "agent_prompt_not_received", - "agent_prompt_unsubmitted", - "agent_input_pending", - }: - envelope = _instruction_rejected_envelope( - request, - worker, - verdict=verdict, - ) - return _finish_request( - config, - request, - reservation, - envelope, - expected_state="send_started", - terminal_state="rejected", - ) - if verdict == "agent_prompt_stalled": - envelope = _instruction_uncertain_envelope( - request, - worker, - verdict=verdict, - ) - else: - envelope = _instruction_uncertain_envelope( - request, - worker, - verdict="unknown", - ) - return _finish_request( - config, - request, - reservation, - envelope, - expected_state="send_started", - terminal_state="uncertain", - ) - - verdict = _agent_prompt_delivery(response) - if verdict == "written_to_pty": - return _record_queued_send( - config, - request, - worker, - reservation, - _queued_send_envelope(request, worker), - ) - if verdict != "submitted": - return _finish_request( - config, - request, - reservation, - _instruction_uncertain_envelope( - request, - worker, - verdict="unknown", - ), - expected_state="send_started", - terminal_state="uncertain", - ) - - # The observation may arrive while the pane call is in flight. Re-read - # the durable link so the accepted envelope can report it immediately. - try: - refreshed_turn = linked_turn_for_submission( - config.db_path, - host_id=config.host_id, - request_id=request.request_id or "", - ) - except Exception: # noqa: BLE001 - refreshed_turn = None - if isinstance(refreshed_turn, Mapping): - observed_turn = refreshed_turn - finally: - _close_socket_client(prepared.client) - - accepted = _accepted_send_envelope( - request, - worker, - observed_turn, - submission_verdict="submitted", - ) - return _finish_request( - config, - request, - reservation, - accepted, - expected_state="send_started", - terminal_state="accepted", - ) - - -def _validate_pending_choice( - config: Config, - request: CommandRequest, -) -> Any | PreSendFailure: - if config.db_path is None: - return _safe_transient_pre_send( - _backend_unavailable(request, "pending state store is unavailable") - ) - params = request.params or {} - try: - validated = claim_backend_pending_choice( - config.db_path, - config.host_id, - str(params.get("pending_id") or ""), - str(params.get("pending_fingerprint") or ""), - str(params.get("choice_id") or ""), - claim=False, - ) - except Exception: - # The pending store raised; nothing was claimed or sent. - return _safe_transient_pre_send( - _backend_unavailable(request, "pending state store is unavailable") - ) - if validated.status != "validated" or not _pending_claim_has_exact_route(validated): - # The pending interaction provably changed or is no longer answerable. - return _permanent_pre_send(_pending_changed_envelope(request)) - return validated - - -def _claim_pending_choice( - config: Config, - request: CommandRequest, - validated: Any, -) -> Any | CommandEnvelope: - assert config.db_path is not None - params = request.params or {} - try: - claim = claim_backend_pending_choice( - config.db_path, - config.host_id, - str(params.get("pending_id") or ""), - str(params.get("pending_fingerprint") or ""), - str(params.get("choice_id") or ""), - claim=True, - ) - except Exception: - return _backend_uncertain(request, "pending answer claim state is uncertain") - if ( - claim.status != "claimed" - or not isinstance(getattr(claim, "claim_token", None), str) - or not claim.claim_token - or not _same_pending_route(validated, claim) - ): - return _pending_changed_envelope(request) - return claim - - -def _uncertain_pending_effect( - config: Config, - claim_token: str, -) -> Callable[[Any], Any] | None: - try: - return backend_pending_choice_terminal_effect( - host_id=config.host_id, - claim_token=claim_token, - accepted=False, - ) - except Exception: - return None - - -def _answer_pending( - config: Config, - request: CommandRequest, - validated: Any, - reservation: ReservedCommandMutation, - client: Any, -) -> CommandEnvelope: - assert config.db_path is not None - claim = _claim_pending_choice(config, request, validated) - if isinstance(claim, CommandEnvelope): - _close_socket_client(client) - return _finish_before_send(config, request, reservation, claim) - claim_token = claim.claim_token - - send_start_error = _mark_request_send_started( - config, - request, - reservation, - binding_fingerprint=claim.binding_private_fingerprint, - ) - if send_start_error is not None: - _close_socket_client(client) - claim_released = _abandon_pending_claim(config, claim_token) - if send_start_error.status == STATUS_PENDING and not claim_released: - return _finish_before_send( - config, - request, - reservation, - _backend_uncertain( - request, - "pending answer claim could not be safely released", - ), - ) - return send_start_error - - try: - started = start_backend_pending_choice_send( - config.db_path, - config.host_id, - claim_token, - ) - except Exception: - _close_socket_client(client) - _abandon_pending_claim(config, claim_token) - return _finish_request( - config, - request, - reservation, - _backend_uncertain(request, "pending answer start state is uncertain"), - expected_state="send_started", - terminal_state="uncertain", - terminal_effect=_uncertain_pending_effect(config, claim_token), - ) - if getattr(started, "status", None) != "started" or not _same_pending_route(claim, started): - _close_socket_client(client) - _abandon_pending_claim(config, claim_token) - return _finish_request( - config, - request, - reservation, - _backend_uncertain( - request, - "pending answer state is uncertain after send start", - ), - expected_state="send_started", - terminal_state="uncertain", - terminal_effect=_uncertain_pending_effect(config, claim_token), - ) - - try: - _submit_private_pane_input( - client, - started.turn_target_value.strip(), - str(started.picker_ordinal), - timeout=config.herdr_timeout_seconds, - ) - except Exception: # noqa: BLE001 - uncertain = _backend_uncertain( - request, - "Herdr socket pane input state is uncertain after send start", - ) - return _finish_request( - config, - request, - reservation, - uncertain, - expected_state="send_started", - terminal_state="uncertain", - terminal_effect=_uncertain_pending_effect(config, claim_token), - ) - finally: - _close_socket_client(client) - - accepted = CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result=_pending_public_result(request, started, delivery_state="submitted"), - ) - try: - effect = backend_pending_choice_terminal_effect( - host_id=config.host_id, - claim_token=claim_token, - accepted=True, - ) - except Exception: - return _recover_request( - config, - request, - reservation.canonical, - ) - return _finish_request( - config, - request, - reservation, - accepted, - expected_state="send_started", - terminal_state="accepted", - terminal_effect=effect, - ) - - -def _validate_pending_decision( - config: Config, - request: CommandRequest, -) -> Any | PreSendFailure: - if config.db_path is None: - return _safe_transient_pre_send( - _backend_unavailable(request, "pending state store is unavailable") - ) - params = request.params or {} - target = request.target or {} - try: - validated = claim_backend_pending_decision( - config.db_path, - config.host_id, - str(target.get("worker_id") or ""), - str(params.get("decision_ref") or ""), - params.get("selection") - if isinstance(params.get("selection"), Mapping) - else {}, - claim=False, - ) - except Exception: - return _safe_transient_pre_send( - _backend_unavailable(request, "pending state store is unavailable") - ) - if validated.status == "validated" and _decision_claim_has_exact_route(validated): - return validated - if validated.status == "acp_authority_unavailable": - return _safe_transient_pre_send( - _backend_unavailable( - request, - "ACP permission authority is temporarily unavailable", - ) - ) - status = { - "already_claimed": STATUS_ANSWER_IN_PROGRESS, - "unknown_worker": STATUS_UNKNOWN_WORKER, - "invalid_selection": STATUS_INVALID_SELECTION, - "unsupported_decision": STATUS_UNSUPPORTED_DECISION, - }.get(validated.status, STATUS_DECISION_NOT_PENDING) - return _permanent_pre_send(_decision_failure_envelope(request, status)) + ) + status = { + "already_claimed": STATUS_ANSWER_IN_PROGRESS, + "unknown_worker": STATUS_UNKNOWN_WORKER, + "invalid_selection": STATUS_INVALID_SELECTION, + "unsupported_decision": STATUS_UNSUPPORTED_DECISION, + }.get(validated.status, STATUS_DECISION_NOT_PENDING) + return _permanent_pre_send(_decision_failure_envelope(request, status)) def _claim_pending_decision( @@ -2139,43 +1285,6 @@ def _claim_pending_decision( return _decision_failure_envelope(request, status) -def _submit_decision_calibration( - client: Any, - pane_id: str, - decision: Any, - *, - timeout: float, -) -> None: - steps = calibrate_decision_steps( - kind=decision.decision_kind, - option_count=decision.option_count, - option_refs=decision.option_refs, - text=decision.text, - ) - for step in steps: - if step.operation == "keys": - _socket_request( - client, - "pane.send_keys", - {"pane_id": pane_id, "keys": list(step.keys)}, - timeout=timeout, - ) - elif step.operation == "text": - _socket_request( - client, - "pane.send_text", - {"pane_id": pane_id, "text": step.text}, - timeout=timeout, - ) - else: - _socket_request( - client, - "pane.send_input", - {"pane_id": pane_id, "text": step.text, "keys": list(step.keys)}, - timeout=timeout, - ) - - def _decision_public_result( request: CommandRequest, claim: Any, @@ -2194,15 +1303,12 @@ def _answer_decision( request: CommandRequest, validated: Any, reservation: ReservedCommandMutation, - client: Any, *, acp_permission_router: AcpPermissionDecisionRouter | None = None, - acp_handoff: bool = False, ) -> CommandEnvelope: assert config.db_path is not None claim = _claim_pending_decision(config, request, validated) if isinstance(claim, CommandEnvelope): - _close_socket_client(client) if claim.status == STATUS_ANSWER_IN_PROGRESS: _abandon_request_reservation(config, request, reservation) return _answer_in_progress(request, receipt_reserved=True) @@ -2228,7 +1334,6 @@ def _answer_decision( binding_fingerprint=claim.binding_private_fingerprint, ) if send_start_error is not None: - _close_socket_client(client) claim_released = _abandon_pending_claim(config, claim_token) if send_start_error.status == STATUS_PENDING and not claim_released: return _finish_before_send( @@ -2249,7 +1354,6 @@ def _answer_decision( claim_token, ) except Exception: - _close_socket_client(client) _abandon_pending_claim(config, claim_token) return _finish_request( config, @@ -2261,7 +1365,6 @@ def _answer_decision( terminal_effect=_uncertain_pending_effect(config, claim_token), ) if getattr(started, "status", None) != "started" or not _same_decision_route(claim, started): - _close_socket_client(client) _abandon_pending_claim(config, claim_token) return _finish_request( config, @@ -2277,20 +1380,12 @@ def _answer_decision( ) try: - if acp_handoff: - if acp_permission_router is None: - raise RuntimeError("ACP permission bridge is unavailable") - acp_permission_router.answer_permission_decision( - started, - timeout=config.acp_request_timeout_seconds, - ) - else: - _submit_decision_calibration( - client, - started.turn_target_value.strip(), - started, - timeout=config.herdr_timeout_seconds, - ) + if acp_permission_router is None: + raise RuntimeError("ACP permission bridge is unavailable") + acp_permission_router.answer_permission_decision( + started, + timeout=config.acp_request_timeout_seconds, + ) except Exception: # noqa: BLE001 return _finish_request( config, @@ -2304,9 +1399,6 @@ def _answer_decision( terminal_state="uncertain", terminal_effect=_uncertain_pending_effect(config, claim_token), ) - finally: - _close_socket_client(client) - accepted = CommandEnvelope.from_result( request, ok=True, @@ -2708,20 +1800,12 @@ def submit_acp_command( params: Mapping[str, Any] | str, *, prompt_router: AcpPromptRouter, - worker_owner: AcpWorkerOwner | None = None, - required: bool = False, - observation_only: bool = False, ) -> CommandEnvelope | None: - """Submit ``send_instruction`` through a live ACP worker route. + """Submit ``send_instruction`` through the required ACP worker route. - ``None`` means the ACP path made no durable change and an optional policy - may safely use the legacy Herdr sender. Once a receipt reaches - ``send_started``, every failure is terminally uncertain and this function - never permits a second transport attempt. - - In observation-only shadow mode a live ACP route is ownership evidence, - not a transport. Such a target fails closed before receipt reservation; - returning ``None`` would incorrectly fall through to its legacy route. + ``None`` is reserved for input that is not an executable instruction, so + the shared parser/dry-run/non-instruction path can handle it. A valid live + instruction always produces an ACP result and never crosses transports. """ payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) @@ -2733,14 +1817,7 @@ def submit_acp_command( if request.dry_run: return None if request.action != "send_instruction": - return ( - _backend_unavailable( - request, - "command is not supported by the required ACP control path", - ) - if required and request.action in _MUTATING_ACTIONS - else None - ) + return None existing_receipt: Mapping[str, Any] | None = None if config.db_path is not None: @@ -2767,17 +1844,9 @@ def submit_acp_command( except Exception: # noqa: BLE001 if takeover is not None: return _request_in_progress(request) - return ( - _backend_unavailable( - request, - "Current worker authority is temporarily unavailable", - ) - # A configured ownership oracle means preferred mode can route - # both ACP and never-ACP workers. Without the authoritative - # snapshot there is no exact worker identity to ask it about, so - # falling through would treat "unknown" as proof of never-ACP. - if required or worker_owner is not None - else None + return _backend_unavailable( + request, + "Current worker authority is temporarily unavailable", ) health_error = _backend_health_error(config, request, snapshot) @@ -2786,64 +1855,28 @@ def submit_acp_command( if takeover is not None: return _request_in_progress(request) if health_error is not None: - return health_error if required else None + return health_error return worker if takeover is not None and worker.id != takeover.public_worker_id: return _duplicate_request(request) - # Preferred mode may fall back only for workers that ACP has never - # claimed. Once the coordinator publishes an exact worker generation, - # losing its visible console or runtime is an ACP outage, not permission - # to inject keys through the legacy PTY path. - owned_by_acp = False - if worker_owner is not None: - try: - owned_by_acp = bool(worker_owner(worker.id, worker.fingerprint)) - except Exception: # noqa: BLE001 - # An ownership oracle failure cannot prove that legacy pane I/O is - # safe. Prefer a retryable no-send result over crossing transports. - owned_by_acp = True - route_required = required or owned_by_acp - route: AcpPromptRoute | None = None - route_resolved = False - if observation_only: - route_resolved = True - try: - route = prompt_router(worker) - except Exception: # noqa: BLE001 - route = None - if route is not None: - return _backend_unavailable( - request, - "ACP shadow is observation-only for ACP-owned workers; use an " - "isolated ACP preferred or required canary to validate prompt " - "execution", - ) - permanent_error = _worker_status_error(request, worker) or health_error if permanent_error is not None: - if route_required: - canonical = build_canonical_mutation(request, public_worker_id=worker.id) - reservation = _reserve_canonical_request(config, request, canonical) - if isinstance(reservation, CommandEnvelope): - return reservation - return _finish_before_send(config, request, reservation, permanent_error) - return None + canonical = build_canonical_mutation(request, public_worker_id=worker.id) + reservation = _reserve_canonical_request(config, request, canonical) + if isinstance(reservation, CommandEnvelope): + return reservation + return _finish_before_send(config, request, reservation, permanent_error) - if not route_resolved: - try: - route = prompt_router(worker) - except Exception: # noqa: BLE001 - route = None + try: + route = prompt_router(worker) + except Exception: # noqa: BLE001 + route = None if route is None: if takeover is not None: return _request_in_progress(request) - return ( - _backend_unavailable(request, "ACP worker route is unavailable") - if route_required - else None - ) + return _backend_unavailable(request, "ACP worker route is unavailable") def submit_through(active_route: AcpPromptRoute) -> CommandEnvelope | None: try: binding_fingerprint = str( @@ -2854,12 +1887,8 @@ def submit_through(active_route: AcpPromptRoute) -> CommandEnvelope | None: if not binding_fingerprint: if takeover is not None: return _request_in_progress(request) - return ( - _backend_unavailable( - request, "ACP worker route has no durable authority" - ) - if route_required - else None + return _backend_unavailable( + request, "ACP worker route has no durable authority" ) canonical = build_canonical_mutation(request, public_worker_id=worker.id) @@ -3036,12 +2065,8 @@ def retryable_before_transport() -> CommandEnvelope: except Exception: # noqa: BLE001 - no receipt or transport exists yet if takeover is not None: return _request_in_progress(request) - return ( - _backend_unavailable( - request, "ACP worker route could not be prepared" - ) - if route_required - else None + return _backend_unavailable( + request, "ACP worker route could not be prepared" ) if active_route is None: active_route = route @@ -3055,16 +2080,14 @@ def _submit_command_v2( config: Config, params: Mapping[str, Any] | str, *, - socket_client_factory: SocketClientFactory | None = None, acp_permission_router: AcpPermissionDecisionRouter | None = None, ) -> CommandEnvelope: - """Submit one command through the authoritative daemon/socket path.""" + """Handle non-prompt commands and ACP permission decisions.""" + payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) request, parse_error = parse_command_request(payload) if parse_error is not None: - if request is not None: - return CommandEnvelope.from_error(request, parse_error) - return CommandEnvelope.from_error(None, parse_error) + return CommandEnvelope.from_error(request, parse_error) validation_error = validate_request(request) if validation_error is not None: @@ -3074,6 +2097,15 @@ def _submit_command_v2( return _execute_non_mutating(config, request) if request.dry_run: return _mutation_dry_run(request) + if request.action == "answer_pending": + return _backend_unavailable( + request, + "legacy pane choices are unavailable; use an ACP permission decision", + ) + if request.action == "send_instruction": + # Valid live instructions are consumed by submit_acp_command before this + # shared parser path. Never expose a second command transport. + return _backend_unavailable(request, "ACP worker route is unavailable") existing_receipt: Mapping[str, Any] | None = None if config.db_path is not None: @@ -3088,11 +2120,6 @@ def _submit_command_v2( if isinstance(candidate, Mapping): existing_receipt = candidate - # An existing receipt is the authority for its request ID. It decides the - # retry from stored evidence before any mutable worker snapshot is read, so - # a vanished, renamed, or recycled worker can never downgrade a live receipt - # to a no-receipt failure or drive a second backend mutation. Only an - # abandoned reservation returns here, to be re-driven by the normal path. takeover: _ReceiptTakeover | None = None if existing_receipt is not None: decided = _receipt_authority(config, request, existing_receipt) @@ -3100,166 +2127,70 @@ def _submit_command_v2( return decided takeover = decided - try: - snapshot = _current_snapshot(config) - except Exception: # noqa: BLE001 - # No external mutation has begun. Store/open contention while reading - # current authority is safely retryable when no receipt exists. An - # abandoned reservation remains authoritative and stays in progress. - if takeover is not None: - return _request_in_progress(request) - return _backend_unavailable( - request, - "Current worker authority is temporarily unavailable", - ) - health_error = _backend_health_error(config, request, snapshot) - - if request.action == "send_instruction": - worker = _resolve_authoritative_worker(request, snapshot) - if isinstance(worker, CommandEnvelope): - if takeover is not None: - # The receipt says this request is reserved and unsent. Mutable - # authority may not restate that as a no-receipt failure. - return _request_in_progress(request) - # An unhealthy observation cannot authoritatively establish that a - # selector is absent, stale, or ambiguous, so keep the request ID - # retryable until a canonical worker can be proven. - if health_error is not None: - return health_error - return worker - if takeover is not None and worker.id != takeover.public_worker_id: - # The abandoned reservation named a different worker, so this is a - # changed target. Fail before any socket or backend work. - return _duplicate_request(request) - canonical = build_canonical_mutation(request, public_worker_id=worker.id) - # A disallowed worker status or an unavailable backend is an authoritative - # observation of proven target unsuitability: a durable rejection is - # justified, and a same-ID retry replays it. - permanent_error = _worker_status_error(request, worker) or health_error - prepared: PreparedInstructionMutation | PreSendFailure | None = None - if permanent_error is None: - prepared = _prepare_instruction( - config, - request, - worker, - socket_client_factory=socket_client_factory, - ) - # A safe transient preparation failure never began a send and never - # created durable authority. Keep the request ID retryable without - # reserving, so a command that was never sent is never silently dropped. - if isinstance(prepared, PreSendFailure) and prepared.is_transient: - if takeover is not None: - return _request_in_progress(request) - return prepared.envelope - reservation = _reserve_canonical_request(config, request, canonical) - if isinstance(reservation, CommandEnvelope): - if isinstance(prepared, PreparedInstructionMutation): - _close_socket_client(prepared.client) - return reservation - if permanent_error is not None: - return _finish_before_send( - config, - request, - reservation, - permanent_error, - ) - if isinstance(prepared, PreSendFailure): - return _finish_before_send(config, request, reservation, prepared.envelope) - assert isinstance(prepared, PreparedInstructionMutation) - return _submit_instruction( - config, - request, - worker, - reservation, - prepared, - ) - answer_pre_send: PreSendFailure | None = None - validate_answer = ( - _validate_pending_decision - if request.action == "answer_decision" - else _validate_pending_choice - ) if takeover is not None: - # Re-driving an abandoned answer reservation: the receipt already fixed - # which worker this request answers, so a pending interaction that now - # routes elsewhere is a changed target, not a new one. existing_worker_id = takeover.public_worker_id canonical = build_canonical_mutation( request, public_worker_id=existing_worker_id, ) - validated = validate_answer(config, request) + validated = _validate_pending_decision(config, request) if isinstance(validated, PreSendFailure): answer_pre_send = validated elif validated.worker_id != existing_worker_id: answer_pre_send = _permanent_pre_send(_duplicate_request(request)) else: - validated = validate_answer(config, request) + validated = _validate_pending_decision(config, request) if isinstance(validated, PreSendFailure): - # No reservation exists yet, so neither a transient nor a permanent - # validation failure writes a receipt here. Return it directly. - if health_error is not None and request.action != "answer_decision": - return health_error return validated.envelope canonical = build_canonical_mutation( request, public_worker_id=validated.worker_id, ) - # A safe transient pre-send failure (the pending store raised) never began a - # send. Keep it retryable under the same request ID without reserving. if answer_pre_send is not None and answer_pre_send.is_transient: - if takeover is not None: - return _request_in_progress(request) - return answer_pre_send.envelope + return ( + _request_in_progress(request) + if takeover is not None + else answer_pre_send.envelope + ) if ( answer_pre_send is not None and answer_pre_send.envelope.status == STATUS_ANSWER_IN_PROGRESS ): - # Another request owns the still-live decision claim. Keep this - # abandoned reservation nonterminal so it can take over after that - # claim is released or expires. return _answer_in_progress(request, receipt_reserved=True) - acp_handoff = False - if answer_pre_send is None and request.action == "answer_decision": - acp_binding = _decision_uses_acp_binding(config, validated) - if acp_binding: - try: - acp_handoff = bool( - acp_permission_router is not None - and acp_permission_router.owns_permission_decision(validated) - ) - except Exception: - acp_handoff = False - if not acp_handoff: - unavailable = _backend_unavailable( - request, - "ACP permission authority is temporarily unavailable", - ) - if takeover is not None: - return _request_in_progress(request) - return unavailable - - client_or_error: Any | CommandEnvelope | None = None - if answer_pre_send is None and health_error is None: - client_or_error = ( - object() - if acp_handoff - else _connect_socket(config, request, socket_client_factory) - ) - if isinstance(client_or_error, CommandEnvelope): - # The socket could not be reached before any transmission -> safe - # transient. Stay retryable rather than reserving a durable rejection. - if takeover is not None: - return _request_in_progress(request) - return client_or_error + if answer_pre_send is None: + if not _decision_uses_acp_binding(config, validated): + unavailable = _backend_unavailable( + request, + "legacy permission decisions are unavailable; ACP authority is required", + ) + return ( + _request_in_progress(request) + if takeover is not None + else unavailable + ) + try: + owns_decision = bool( + acp_permission_router is not None + and acp_permission_router.owns_permission_decision(validated) + ) + except Exception: + owns_decision = False + if not owns_decision: + unavailable = _backend_unavailable( + request, + "ACP permission authority is temporarily unavailable", + ) + return ( + _request_in_progress(request) + if takeover is not None + else unavailable + ) reservation = _reserve_canonical_request(config, request, canonical) if isinstance(reservation, CommandEnvelope): - if client_or_error is not None and not isinstance(client_or_error, CommandEnvelope): - _close_socket_client(client_or_error) return reservation if answer_pre_send is not None: return _finish_before_send( @@ -3268,30 +2199,12 @@ def _submit_command_v2( reservation, answer_pre_send.envelope, ) - if health_error is not None: - return _finish_before_send( - config, - request, - reservation, - health_error, - ) - assert client_or_error is not None - if request.action == "answer_decision": - return _answer_decision( - config, - request, - validated, - reservation, - client_or_error, - acp_permission_router=acp_permission_router, - acp_handoff=acp_handoff, - ) - return _answer_pending( + return _answer_decision( config, request, validated, reservation, - client_or_error, + acp_permission_router=acp_permission_router, ) @@ -3358,41 +2271,32 @@ def submit_command( config: Config, params: Mapping[str, Any] | str, *, - socket_client_factory: SocketClientFactory | None = None, acp_prompt_router: AcpPromptRouter | None = None, - acp_worker_owner: AcpWorkerOwner | None = None, - acp_required: bool = False, - acp_observation_only: bool = False, acp_permission_router: AcpPermissionDecisionRouter | None = None, ) -> CommandEnvelope: - """Submit one command and apply optional response-envelope negotiation.""" - if acp_prompt_router is not None: - acp_envelope = submit_acp_command( - config, - params, - prompt_router=acp_prompt_router, - worker_owner=acp_worker_owner, - required=acp_required, - observation_only=acp_observation_only, - ) - if acp_envelope is not None: - payload = ( - params - if isinstance(params, str) - else _raw_payload_from_mapping(params) + """Submit one command with ACP as the only instruction transport.""" + acp_envelope = submit_acp_command( + config, + params, + prompt_router=acp_prompt_router or (lambda _worker: None), + ) + if acp_envelope is not None: + payload = ( + params + if isinstance(params, str) + else _raw_payload_from_mapping(params) + ) + request, parse_error = parse_command_request(payload) + if parse_error is None and request is not None: + return _negotiated_submission_envelope( + config, + request, + acp_envelope, ) - request, parse_error = parse_command_request(payload) - if parse_error is None and request is not None: - return _negotiated_submission_envelope( - config, - request, - acp_envelope, - ) - return acp_envelope + return acp_envelope envelope = _submit_command_v2( config, params, - socket_client_factory=socket_client_factory, acp_permission_router=acp_permission_router, ) payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) diff --git a/src/tendwire/config.py b/src/tendwire/config.py index 0ddc887..7866b1b 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -6,7 +6,6 @@ from __future__ import annotations -import logging import math import os import platform @@ -15,14 +14,9 @@ from pathlib import Path HERDR_BACKENDS = frozenset({"cli", "socket"}) -TURN_MODELS = frozenset({"legacy", "dual", "shadow", "observed"}) -AGENT_EVENT_SOURCES = frozenset( - {"legacy", "acp_shadow", "acp_preferred", "acp_required"} -) ACP_THOUGHT_POLICIES = frozenset({"disabled", "private_summary", "private_all"}) ACP_CONSOLE_INPUT_POLICIES = frozenset({"preserve", "live_only"}) DEFAULT_TURN_MODEL = "observed" -DEFAULT_AGENT_EVENT_SOURCE = "legacy" DEFAULT_ACP_THOUGHT_POLICY = "disabled" DEFAULT_ACP_CONSOLE_INPUT_POLICY = "preserve" DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS = 30.0 @@ -34,8 +28,6 @@ DEFAULT_EVENT_RETENTION_DAYS = 7 DEFAULT_OUTPUT_EXCERPT_CHARS = 200 DEFAULT_MAX_WORKERS = 512 -DEFAULT_TURN_REFRESH_INTERVAL_SECONDS = 2.0 -DEFAULT_TURN_REFRESH_WORKERS = 4 DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS = 60 DEFAULT_SUBMISSION_HARD_TTL_SECONDS = 86_400 DEFAULT_PENDING_STALE_GRACE_SECONDS = 30.0 @@ -61,7 +53,6 @@ MAX_RETENTION_DAYS = 365_000 MAX_SQLITE_INTEGER = (1 << 63) - 1 MAX_MAINTENANCE_CADENCE_SECONDS = MAX_RETENTION_DAYS * 24 * 60 * 60 -_LOGGER = logging.getLogger(__name__) @dataclass(frozen=True) @@ -78,8 +69,6 @@ class Config: DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS ) herdr_backend: str = "cli" - turn_model: str = DEFAULT_TURN_MODEL - agent_event_source: str = DEFAULT_AGENT_EVENT_SOURCE acp_thought_policy: str = DEFAULT_ACP_THOUGHT_POLICY acp_console_input_policy: str = DEFAULT_ACP_CONSOLE_INPUT_POLICY acp_request_timeout_seconds: float = DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS @@ -90,8 +79,6 @@ class Config: event_retention_days: int = DEFAULT_EVENT_RETENTION_DAYS output_excerpt_chars: int = DEFAULT_OUTPUT_EXCERPT_CHARS max_workers: int = DEFAULT_MAX_WORKERS - turn_refresh_interval_seconds: float = DEFAULT_TURN_REFRESH_INTERVAL_SECONDS - turn_refresh_workers: int = DEFAULT_TURN_REFRESH_WORKERS submission_link_window_seconds: int = DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS submission_hard_ttl_seconds: int = DEFAULT_SUBMISSION_HARD_TTL_SECONDS pending_stale_grace_seconds: float = DEFAULT_PENDING_STALE_GRACE_SECONDS @@ -151,21 +138,6 @@ def __post_init__(self) -> None: allowed = ", ".join(sorted(HERDR_BACKENDS)) raise ValueError(f"herdr_backend must be one of: {allowed}") object.__setattr__(self, "herdr_backend", backend) - turn_model = str(self.turn_model or "").strip().lower() - if turn_model not in TURN_MODELS: - allowed = ", ".join(sorted(TURN_MODELS)) - raise ValueError(f"turn_model must be one of: {allowed}") - object.__setattr__(self, "turn_model", turn_model) - if turn_model != "observed": - _LOGGER.warning( - "turn_model=%s is a compatibility alias and behaves as observed", - turn_model, - ) - agent_event_source = str(self.agent_event_source or "").strip().lower() - if agent_event_source not in AGENT_EVENT_SOURCES: - allowed = ", ".join(sorted(AGENT_EVENT_SOURCES)) - raise ValueError(f"agent_event_source must be one of: {allowed}") - object.__setattr__(self, "agent_event_source", agent_event_source) acp_thought_policy = str(self.acp_thought_policy or "").strip().lower() if acp_thought_policy not in ACP_THOUGHT_POLICIES: allowed = ", ".join(sorted(ACP_THOUGHT_POLICIES)) @@ -238,25 +210,6 @@ def __post_init__(self) -> None: "max_workers", _positive_int(self.max_workers, "max_workers", minimum=1), ) - object.__setattr__( - self, - "turn_refresh_interval_seconds", - _positive_finite_float( - self.turn_refresh_interval_seconds, - "turn_refresh_interval_seconds", - ), - ) - object.__setattr__( - self, - "turn_refresh_workers", - _bounded_positive_int( - self.turn_refresh_workers, - "turn_refresh_workers", - maximum=32, - ), - ) - if self.turn_refresh_workers > self.max_workers: - raise ValueError("turn_refresh_workers must be <= max_workers") object.__setattr__( self, "submission_link_window_seconds", @@ -526,8 +479,6 @@ def load_config( herdr_timeout_seconds: float | str | None = None, herdr_initial_reconcile_timeout_seconds: float | str | None = None, herdr_backend: str | None = None, - turn_model: str | None = None, - agent_event_source: str | None = None, acp_thought_policy: str | None = None, acp_console_input_policy: str | None = None, acp_request_timeout_seconds: float | str | None = None, @@ -538,8 +489,6 @@ def load_config( event_retention_days: int | str | None = None, output_excerpt_chars: int | str | None = None, max_workers: int | str | None = None, - turn_refresh_interval_seconds: float | str | None = None, - turn_refresh_workers: int | str | None = None, submission_link_window_seconds: int | str | None = None, submission_hard_ttl_seconds: int | str | None = None, pending_stale_grace_seconds: float | str | None = None, @@ -628,16 +577,6 @@ def load_config( DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS, ), herdr_backend=resolved_herdr_backend, - turn_model=_resolve_value( - turn_model, - "TENDWIRE_TURN_MODEL", - DEFAULT_TURN_MODEL, - ), - agent_event_source=_resolve_value( - agent_event_source, - "TENDWIRE_AGENT_EVENT_SOURCE", - DEFAULT_AGENT_EVENT_SOURCE, - ), acp_thought_policy=_resolve_value( acp_thought_policy, "TENDWIRE_ACP_THOUGHT_POLICY", @@ -688,16 +627,6 @@ def load_config( "TENDWIRE_MAX_WORKERS", DEFAULT_MAX_WORKERS, ), - turn_refresh_interval_seconds=_resolve_value( - turn_refresh_interval_seconds, - "TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS", - DEFAULT_TURN_REFRESH_INTERVAL_SECONDS, - ), - turn_refresh_workers=_resolve_value( - turn_refresh_workers, - "TENDWIRE_TURN_REFRESH_WORKERS", - DEFAULT_TURN_REFRESH_WORKERS, - ), submission_link_window_seconds=_resolve_value( submission_link_window_seconds, "TENDWIRE_SUBMISSION_LINK_WINDOW_SECONDS", diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index c3697ae..3c76e31 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -11,7 +11,7 @@ from pathlib import Path from typing import Any -from .config import Config +from .config import DEFAULT_TURN_MODEL, Config from .core.commands import CommandEnvelope from .core.models import Snapshot, sanitize_public_mapping, utc_timestamp from .daemon_api import ( @@ -394,41 +394,6 @@ def _command_requests_health( }, True -def _turn_ingestion_health(config: Config, scheduler: Any | None) -> dict[str, Any]: - raw: Mapping[str, Any] = {} - if scheduler is not None: - try: - status_value = scheduler.operational_status() - except Exception: - status_value = {} - if isinstance(status_value, Mapping): - raw = status_value - status = raw.get("status") - if status not in {"healthy", "stale", "degraded", "stopping"}: - status = "stale" if scheduler is None else "degraded" - return { - "status": status, - "queue": _nonnegative_int(raw.get("queue_depth")), - "active": _nonnegative_int(raw.get("active")), - "refreshed": _nonnegative_int(raw.get("refreshed")), - "failed": _nonnegative_int(raw.get("failed")), - "timed_out": _nonnegative_int(raw.get("timed_out")), - "coalesced": _nonnegative_int(raw.get("coalesced")), - "queue_full": _nonnegative_int(raw.get("queue_full")), - "last_success": _valid_observation_timestamp( - raw.get("last_success") if isinstance(raw.get("last_success"), str) else None - ), - "last_duration_ms": _nonnegative_float(raw.get("last_duration_ms")), - "stale_age": _nonnegative_float(raw.get("stale_age_seconds")), - "bounds": { - "refresh_interval_seconds": config.turn_refresh_interval_seconds, - "max_workers": config.turn_refresh_workers, - "queue_capacity": _nonnegative_int(raw.get("queue_capacity")), - "adapter_timeout_seconds": config.herdr_timeout_seconds, - }, - } - - def _pending_ingestion_health(config: Config) -> dict[str, Any]: """Return the fixed durable pending aggregate without exposing row identity.""" unavailable = { @@ -505,22 +470,10 @@ def _default_observe_initial_snapshot(config: Config) -> Snapshot: return observe_public_snapshot(config, store_snapshot=True) -def _default_submit_command(config: Config, payload: str) -> CommandEnvelope: - from .command_submission import submit_command - - return submit_command(config, payload) - +def _default_acp_supervisor_factory(config: Config, stop_event: threading.Event) -> Any: + from .backends.acp_coordinator import production_acp_supervisor_factory -def _default_turn_scheduler_factory(config: Config) -> Any: - from .backends.herdr_turns import TurnIngestionScheduler - - return TurnIngestionScheduler(config) - - -def _default_acp_runtime_factory(config: Config, stop_event: threading.Event) -> Any: - from .backends.acp_coordinator import production_acp_runtime_factory - - return production_acp_runtime_factory(config, stop_event) + return production_acp_supervisor_factory(config, stop_event) @dataclass(frozen=True) @@ -529,11 +482,9 @@ class DaemonHooks: init_store: Callable[[Path], None] = _default_init_store observe_initial_snapshot: Callable[[Config], Snapshot] = _default_observe_initial_snapshot - submit_command: Callable[[Config, str], CommandEnvelope | Mapping[str, Any]] = _default_submit_command event_backend_factory: Callable[[Config, threading.Event], Any] | None = None - turn_scheduler_factory: Callable[[Config], Any] = _default_turn_scheduler_factory - acp_runtime_factory: Callable[[Config, threading.Event], Any | None] | None = ( - _default_acp_runtime_factory + acp_supervisor_factory: Callable[[Config, threading.Event], Any | None] | None = ( + _default_acp_supervisor_factory ) @@ -557,8 +508,7 @@ def __init__( self._snapshot: Snapshot | None = None self._server: UnixSocketJSONServer | None = None self._event_backend: Any | None = None - self._turn_scheduler: Any | None = None - self._acp_runtime: Any | None = None + self._acp_supervisor: Any | None = None self._acp_startup_failure_type: str | None = None self._stop_lock = threading.Lock() self._automatic_maintenance_status: dict[str, Any] | None = None @@ -610,19 +560,7 @@ def start(self) -> None: self._snapshot = self.hooks.observe_initial_snapshot(self.config) self._after_snapshot_saved() - self._start_acp_runtime() - - scheduler = None - if self.config.agent_event_source != "acp_required": - scheduler = self.hooks.turn_scheduler_factory(self.config) - self._turn_scheduler = scheduler - if self.config.agent_event_source in {"acp_shadow", "acp_preferred"}: - owns_worker = getattr(self._acp_runtime, "claims_worker", None) - if not callable(owns_worker): - owns_worker = getattr(self._acp_runtime, "owns_worker", None) - set_exclusion = getattr(scheduler, "set_worker_exclusion", None) - if callable(owns_worker) and callable(set_exclusion): - set_exclusion(owns_worker) + self._start_acp_supervisor() api = TendwireDaemonAPI( get_snapshot=self.get_snapshot, @@ -650,42 +588,12 @@ def start(self) -> None: # Requests are not served until start() returns successfully. server.start() - backend = self._event_backend - callback_setter = ( - getattr(backend, "set_turn_refresh_callback", None) - if backend is not None - else None - ) - if callable(callback_setter) and scheduler is not None: - callback_setter(scheduler.request_refresh) - if scheduler is not None: - scheduler.start() - scheduler.request_refresh() except Exception: self.stop_event.set() backend = self._event_backend - callback_setter = ( - getattr(backend, "set_turn_refresh_callback", None) - if backend is not None - else None - ) - if callable(callback_setter): - try: - callback_setter(None) - except Exception: - pass - scheduler = self._turn_scheduler - self._turn_scheduler = None - if scheduler is not None: - try: - scheduler.stop( - flush_timeout_seconds=self.config.herdr_timeout_seconds + 1.0 - ) - except Exception: - pass - runtime = self._acp_runtime - self._acp_runtime = None - self._stop_acp_runtime(runtime) + supervisor = self._acp_supervisor + self._acp_supervisor = None + self._stop_acp_supervisor(supervisor) self._event_backend = None if backend is not None: try: @@ -726,12 +634,10 @@ def stop(self) -> None: self.stop_event.set() server = self._server backend = self._event_backend - scheduler = self._turn_scheduler - runtime = self._acp_runtime + supervisor = self._acp_supervisor self._server = None self._event_backend = None - self._turn_scheduler = None - self._acp_runtime = None + self._acp_supervisor = None if server is not None: try: @@ -739,7 +645,7 @@ def stop(self) -> None: except Exception: pass - self._stop_acp_runtime(runtime) + self._stop_acp_supervisor(supervisor) if backend is not None: flush = getattr(backend, "flush", None) @@ -748,86 +654,63 @@ def stop(self) -> None: flush() except Exception: pass - callback_setter = getattr(backend, "set_turn_refresh_callback", None) - if callable(callback_setter): - try: - callback_setter(None) - except Exception: - pass - - if scheduler is not None: - try: - scheduler.stop( - flush_timeout_seconds=self.config.herdr_timeout_seconds + 1.0 - ) - except Exception: - pass - if backend is not None: try: backend.stop() except Exception: pass - def _start_acp_runtime(self) -> None: - """Start an injected ACP runtime according to the configured policy.""" - policy = self.config.agent_event_source + def _start_acp_supervisor(self) -> None: + """Start the required ACP supervisor and fail the daemon closed.""" self._acp_startup_failure_type = None - if policy == "legacy": - return - factory = self.hooks.acp_runtime_factory + factory = self.hooks.acp_supervisor_factory if factory is None: - if policy == "acp_required": - raise RuntimeError("ACP runtime is required but unavailable") - return + raise RuntimeError("ACP supervisor is required but unavailable") - runtime: Any | None = None + supervisor: Any | None = None try: - runtime = factory(self.config, self.stop_event) - if runtime is None: - if policy == "acp_required": - raise RuntimeError("ACP runtime is required but unavailable") - return - self._acp_runtime = runtime - runtime.start() - health = self._acp_runtime_health() + supervisor = factory(self.config, self.stop_event) + if supervisor is None: + raise RuntimeError("ACP supervisor is required but unavailable") + self._acp_supervisor = supervisor + supervisor.start() + health = self._acp_supervisor_health() if health["healthy"] is not True: failure_type = health.get("failure_type") self._acp_startup_failure_type = _public_failure_type(failure_type) - raise RuntimeError("ACP runtime did not become healthy") + raise RuntimeError("ACP supervisor did not become healthy") except Exception as exc: self._acp_startup_failure_type = ( self._acp_startup_failure_type or type(exc).__name__ ) - if runtime is not None: - self._stop_acp_runtime(runtime) - self._acp_runtime = None - if policy == "acp_required": - raise RuntimeError( - "ACP runtime is required but failed to start " - f"({self._acp_startup_failure_type})" - ) from None - - def _stop_acp_runtime(self, runtime: Any | None) -> None: - """Best-effort bounded shutdown for an injected ACP runtime.""" - if runtime is None: + if supervisor is not None: + self._stop_acp_supervisor(supervisor) + self._acp_supervisor = None + raise RuntimeError( + "ACP supervisor is required but failed to start " + f"({self._acp_startup_failure_type})" + ) from None + + def _stop_acp_supervisor(self, supervisor: Any | None) -> None: + """Best-effort bounded shutdown for the ACP supervisor.""" + if supervisor is None: return timeout = self.config.acp_shutdown_timeout_seconds - stop = getattr(runtime, "stop", None) + stop = getattr(supervisor, "stop", None) if callable(stop): try: stop(timeout=timeout) except Exception: pass - join = getattr(runtime, "join", None) + join = getattr(supervisor, "join", None) if callable(join): try: join(timeout=timeout) except Exception: pass - def _acp_runtime_health(self) -> dict[str, Any]: + def _acp_supervisor_health(self) -> dict[str, Any]: """Return a fixed, public-safe ACP lifecycle aggregate.""" counters = { "updates_ingested": 0, @@ -840,21 +723,10 @@ def _acp_runtime_health(self) -> dict[str, Any]: "prompts_failed": 0, "cancellation_requests": 0, } - policy = self.config.agent_event_source - if policy == "legacy": + supervisor = self._acp_supervisor + if supervisor is None: return { - "policy": policy, - "status": "disabled", - "healthy": False, - "state": "disabled", - "failure_type": None, - "counters": counters, - } - - runtime = self._acp_runtime - if runtime is None: - return { - "policy": policy, + "required": True, "status": "unavailable", "healthy": False, "state": "unavailable", @@ -862,12 +734,12 @@ def _acp_runtime_health(self) -> dict[str, Any]: "counters": counters, } - status_method = getattr(runtime, "status", None) + status_method = getattr(supervisor, "status", None) try: raw = status_method() if callable(status_method) else None except Exception as exc: return { - "policy": policy, + "required": True, "status": "degraded", "healthy": False, "state": "failed", @@ -890,7 +762,7 @@ def field(name: str) -> Any: failure_type_value = field("failure_type") failure_type = _public_failure_type(failure_type_value) return { - "policy": policy, + "required": True, "status": "healthy" if healthy else "degraded", "healthy": healthy, "state": state, @@ -918,7 +790,7 @@ def _after_snapshot_saved(self) -> None: policy=policy, agent_event_host_id=self.config.host_id, agent_event_retention_days=self.config.event_retention_days, - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, acknowledged_final_retention_days=( self.config.acknowledged_final_retention_days ), @@ -1018,7 +890,7 @@ def _start_socket_event_backend(self) -> Snapshot: save_snapshot( Path(self.config.db_path), snapshot, - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, observation=SnapshotObservationContext( authority="none", observed_at=_valid_observation_timestamp(backend_health.observed_at), @@ -1187,7 +1059,7 @@ def get_health(self) -> dict[str, Any]: or stored_last_snapshot_at or snapshot.updated_at ) - acp_health = self._acp_runtime_health() + acp_health = self._acp_supervisor_health() payload = { "schema_version": 1, "status": ( @@ -1195,14 +1067,10 @@ def get_health(self) -> dict[str, Any]: if store_ok and not maintenance_degraded and pending_ingestion["status"] == "healthy" - and ( - self.config.agent_event_source != "acp_required" - or acp_health["healthy"] is True - ) + and acp_health["healthy"] is True else "degraded" ), "host_id": self.config.host_id, - "turn_model": self.config.turn_model, "daemon": { "status": "healthy", "started_at": self.started_at, @@ -1242,10 +1110,6 @@ def get_health(self) -> dict[str, Any]: self.config.reconcile_interval_seconds > 0, ), }, - "turn_ingestion": _turn_ingestion_health( - self.config, - self._turn_scheduler, - ), "acp": acp_health, "pending_ingestion": pending_ingestion, "limits": { @@ -1332,8 +1196,7 @@ def get_turns( limit=limit, cursor=cursor, since=since, - turn_refresh_interval_seconds=self.config.turn_refresh_interval_seconds, - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ) def get_turn_content(self, params: Mapping[str, Any]) -> Mapping[str, Any]: @@ -1358,7 +1221,7 @@ def get_turn_content(self, params: Mapping[str, Any]) -> Mapping[str, Any]: field=params.get("field"), cursor=params.get("cursor"), schema_version=params.get("schema_version", 1), - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ) def get_turn_delta( @@ -1385,7 +1248,7 @@ def get_turn_delta( watermark=watermark, cursor=cursor, limit=limit, - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ) def connector_call(self, method: str, params: Mapping[str, Any]) -> Mapping[str, Any]: @@ -1410,7 +1273,7 @@ def connector_call(self, method: str, params: Mapping[str, Any]) -> Mapping[str, max_lease_seconds=self.config.connector_max_claim_ttl_seconds, ack_ttl_seconds=self.config.connector_ack_ttl_seconds, max_attempts=self.config.max_outbox_attempts, - turn_model=self.config.turn_model, + turn_model=DEFAULT_TURN_MODEL, ).dispatch(method, params) def _connector_periodic_tick(self) -> None: @@ -1448,39 +1311,21 @@ def submit_command(self, params: Mapping[str, Any]) -> CommandEnvelope | Mapping sort_keys=True, separators=(",", ":"), ) - policy = self.config.agent_event_source - runtime = self._acp_runtime - route = getattr(runtime, "prompt_route", None) - worker_owner = getattr(runtime, "claims_worker", None) - if not callable(worker_owner): - worker_owner = getattr(runtime, "owns_worker", None) + supervisor = self._acp_supervisor + route = getattr(supervisor, "prompt_route", None) permission_router = ( - runtime - if callable(getattr(runtime, "answer_permission_decision", None)) + supervisor + if callable(getattr(supervisor, "answer_permission_decision", None)) else None ) - if policy == "acp_required" or ( - policy in {"acp_shadow", "acp_preferred"} and callable(route) - ) or permission_router is not None: - from .command_submission import submit_command - - return submit_command( - self.config, - payload, - acp_prompt_router=( - route - if policy in {"acp_shadow", "acp_required", "acp_preferred"} - and callable(route) - else None - ), - acp_worker_owner=( - worker_owner if callable(worker_owner) else None - ), - acp_required=policy == "acp_required", - acp_observation_only=policy == "acp_shadow", - acp_permission_router=permission_router, - ) - return self.hooks.submit_command(self.config, payload) + from .command_submission import submit_command + + return submit_command( + self.config, + payload, + acp_prompt_router=route if callable(route) else None, + acp_permission_router=permission_router, + ) def run_daemon( diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index 51a2b41..76a07e3 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -32,8 +32,6 @@ DEFAULT_SUBMISSION_HARD_TTL_SECONDS, DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS, DEFAULT_TURN_MODEL, - DEFAULT_TURN_REFRESH_INTERVAL_SECONDS, - TURN_MODELS, ) from ..local_state import ( EntryIdentity, @@ -206,6 +204,8 @@ _LOGGER = logging.getLogger(__name__) _SUBMISSION_LINK_SWEEP_LAST_AT: dict[tuple[str, str, str], float] = {} _SUBMISSION_LINK_SWEEP_LOCK = threading.Lock() +TURN_MODELS = frozenset({"legacy", "dual", "shadow", "observed"}) +DEFAULT_SUBMISSION_LINK_SWEEP_INTERVAL_SECONDS = 2.0 _SUBMISSION_LINK_BACKOFF: dict[ tuple[str, str, str, str], datetime | None ] = {} @@ -24462,7 +24462,7 @@ def _turn_delta_payload_from_store( host, purpose="submission_links", current_clock=clock, - refresh_interval_seconds=DEFAULT_TURN_REFRESH_INTERVAL_SECONDS, + refresh_interval_seconds=DEFAULT_SUBMISSION_LINK_SWEEP_INTERVAL_SECONDS, ) if sweep_due: try: diff --git a/tests/fixtures/acp_fake_agent.py b/tests/fixtures/acp_fake_agent.py index fe381eb..5b5dfc6 100644 --- a/tests/fixtures/acp_fake_agent.py +++ b/tests/fixtures/acp_fake_agent.py @@ -168,7 +168,13 @@ def update(session_id: str, kind: str, **values: object) -> None: update("s-new", "agent_message_chunk", content={"type": "text", "text": "hi"}) response( request_id, - {"sessionId": "s-new", "modes": {"currentModeId": "default"}}, + { + "sessionId": "s-new", + "modes": { + "currentModeId": "default", + "availableModes": [{"id": "default", "name": "Default"}], + }, + }, ) elif method == "session/load" or method == "session/resume": if MODE == "load_replay" and method == "session/load": @@ -179,7 +185,20 @@ def update(session_id: str, kind: str, **values: object) -> None: messageId=f"replay-{index}", content={"type": "text", "text": str(index)}, ) - response(request_id, {"configOptions": [{"id": "model", "currentValue": "x"}]}) + response( + request_id, + { + "configOptions": [ + { + "id": "model", + "name": "Model", + "type": "select", + "currentValue": "x", + "options": [{"value": "x", "name": "Model X"}], + } + ] + }, + ) elif method == "session/close" or method == "session/delete": response(request_id, {"_meta": {"vendor.example": {"receipt": method}}}) elif method == "session/list": @@ -280,7 +299,9 @@ def update(session_id: str, kind: str, **values: object) -> None: update( pending_prompt_session, "plan", - entries=[{"content": "done", "status": "completed"}], + entries=[ + {"content": "done", "priority": "medium", "status": "completed"} + ], ) response( pending_prompt_id, diff --git a/tests/test_acp_atomic_ingestion.py b/tests/test_acp_atomic_ingestion.py index bd01ae4..e46f6f4 100644 --- a/tests/test_acp_atomic_ingestion.py +++ b/tests/test_acp_atomic_ingestion.py @@ -28,11 +28,10 @@ def _store( tmp_path: Path, *, - source: str = "acp_preferred", stable_owner: bool = False, ) -> tuple[Config, WorkerBinding]: db_path = tmp_path / "events.db" - config = Config(host_id="host-a", db_path=db_path, agent_event_source=source) + config = Config(host_id="host-a", db_path=db_path) snapshot = project_from_raw( config, workers=[ @@ -519,22 +518,3 @@ def fault(boundary: str) -> None: assert len(list_agent_events(config.db_path, config.host_id)) == 2 turn = turns_payload_from_store(config.db_path, config.host_id)["turns"][0] assert turn["assistant_final_text"] == "answer" - - -def test_shadow_mode_journals_completion_without_projecting_turn( - tmp_path: Path, -) -> None: - config, binding = _store(tmp_path, source="acp_shadow") - ingestor = AcpSessionIngestor( - config, - session_id="session-a", - stream_generation="generation-a", - binding=binding, - ) - ingestor.start_turn(producer_turn_id="producer-a") - ingestor.ingest_update(_update("answer"), source_event_id="message-event-a") - ingestor.mark_prompt_complete() - - events = list_agent_events(config.db_path, config.host_id) - assert [item.event.kind for item in events] == ["agent_message", "extension"] - assert turns_payload_from_store(config.db_path, config.host_id)["turns"] == [] diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py index f0a91a4..1e02b95 100644 --- a/tests/test_acp_client.py +++ b/tests/test_acp_client.py @@ -9,7 +9,7 @@ from tendwire.backends.acp_client import ( AcpCapabilityError, - AcpClient, + BoundedAcpConnection as AcpClient, AcpEventQueueFullError, AcpRequestTimeoutError, AcpTransportError, @@ -109,7 +109,10 @@ def test_initialize_capabilities_and_session_lifecycle() -> None: additional_directories=["/tmp/other"], ) assert created.session_id == "s-new" - assert created.modes == {"currentModeId": "default"} + assert created.modes == { + "currentModeId": "default", + "availableModes": [{"id": "default", "name": "Default"}], + } streamed = acp.next_update(timeout=1) assert streamed.update_kind is SessionUpdateKind.AGENT_MESSAGE_CHUNK diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index f7de34e..39d98f4 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -17,9 +17,9 @@ from tendwire.backends.acp_coordinator import ( AcpConsoleInputGap, AcpCoordinatorError, - AcpRuntimeCoordinator, + AcpSupervisor as AcpRuntimeCoordinator, HerdrAcpConsoleEndpoint, - _RuntimeSlot, + _SessionSlot as _RuntimeSlot, _CONSOLE_BRIDGE_INTERVAL_SECONDS, _derived_binding, _console_event_output, @@ -34,11 +34,10 @@ _parse_status, _prepare_console_event_cursor, _record_console_submission_outcome, - production_acp_runtime_factory, + production_acp_supervisor_factory as production_acp_runtime_factory, ) from tendwire.backends.acp_runtime import RuntimeState, SessionOpenMode from tendwire.backends.herdr_protocol import HerdrErrorResponse -from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult from tendwire.command_submission import submit_acp_command, submit_command from tendwire.config import Config from tendwire.core.models import ( @@ -59,14 +58,13 @@ ) -def _config(tmp_path: Path, *, policy: str = "acp_preferred") -> Config: +def _config(tmp_path: Path) -> Config: return Config( host_id="acp-host", data_dir=tmp_path, db_path=tmp_path / "tendwire.db", herdr_backend="socket", herdr_bin="herdr", - agent_event_source=policy, ) @@ -646,7 +644,7 @@ def bridge(slot: _RuntimeSlot) -> None: def test_console_failure_remains_degraded_after_slot_disappears(tmp_path: Path) -> None: coordinator = AcpRuntimeCoordinator( - _config(tmp_path, policy="acp_required"), + _config(tmp_path), threading.Event(), reconcile_interval=60.0, ) @@ -675,7 +673,7 @@ def test_first_console_failure_immediately_fences_prompt_route_until_success( tmp_path: Path, ) -> None: coordinator = AcpRuntimeCoordinator( - _config(tmp_path, policy="acp_required"), + _config(tmp_path), threading.Event(), reconcile_interval=60.0, ) @@ -762,7 +760,7 @@ def test_superseded_console_success_cannot_clear_replacement_fence( tmp_path: Path, ) -> None: coordinator = AcpRuntimeCoordinator( - _config(tmp_path, policy="acp_required"), + _config(tmp_path), threading.Event(), reconcile_interval=60.0, ) @@ -797,195 +795,6 @@ def test_superseded_console_success_cannot_clear_replacement_fence( assert coordinator.status()["healthy"] is True -def test_failed_remint_retains_exact_acp_claim_and_blocks_preferred_fallback( - tmp_path: Path, -) -> None: - config = _config(tmp_path, policy="acp_preferred") - worker = _seed(config) - coordinator = AcpRuntimeCoordinator( - config, - threading.Event(), - reconcile_interval=60.0, - ) - coordinator._state = RuntimeState.RUNNING - runtime = SimpleNamespace( - status=lambda: SimpleNamespace(healthy=True, failure_type=None), - stop=lambda *, timeout: None, - _binding=_binding(), - ) - slot = _RuntimeSlot( - _binding(), - "42", - runtime, - console=HerdrAcpConsoleEndpoint(42, "console-lease"), - ) - coordinator._slots[worker.id] = slot - - def fail_console(_slot: _RuntimeSlot) -> None: - raise OSError("console unavailable") - - coordinator._bridge_console_slot = fail_console # type: ignore[method-assign] - coordinator._continuity_bindings = ( # type: ignore[method-assign] - lambda: ({worker.id: _binding()}, 0) - ) - - def fail_remint(_continuity: WorkerBinding) -> None: - raise AcpCoordinatorError("replacement attach failed") - - coordinator._reconcile_binding = fail_remint # type: ignore[method-assign] - - for _attempt in range(3): - coordinator._bridge_console_slot_supervised(slot) - - assert worker.id not in coordinator._slots - assert coordinator.claims_worker(worker.id, worker.fingerprint) is True - assert coordinator.prompt_route(worker) is None - - def forbidden_legacy(_config: Config) -> Any: - raise AssertionError("retired ACP claim must not reopen legacy pane I/O") - - envelope = submit_command( - config, - _request("failed-remint-no-fallback"), - socket_client_factory=forbidden_legacy, - acp_prompt_router=coordinator.prompt_route, - acp_worker_owner=coordinator.claims_worker, - ) - assert envelope.status == "backend_unavailable" - assert envelope.disposition == "no_receipt" - assert get_command_request( - config.db_path, - config.host_id, - "failed-remint-no-fallback", - ) is None - - -def test_failed_claim_clears_only_after_exact_herdr_authority_disappears( - tmp_path: Path, -) -> None: - coordinator = AcpRuntimeCoordinator( - _config(tmp_path, policy="acp_preferred"), - threading.Event(), - reconcile_interval=60.0, - ) - coordinator._state = RuntimeState.RUNNING - coordinator._console_failed_workers.add("worker-1") - coordinator._console_failed_claims["worker-1"] = "worker-fingerprint" - coordinator._console_degraded = True - coordinator._continuity_bindings = ( # type: ignore[method-assign] - lambda: ({}, 1) - ) - coordinator._herdr_authority_claims = ( # type: ignore[method-assign] - lambda: {("worker-1", "worker-fingerprint")} - ) - - coordinator._reconcile_locked(strict=False) - assert coordinator.claims_worker("worker-1", "worker-fingerprint") is True - assert coordinator.status()["healthy"] is False - - coordinator._herdr_authority_claims = lambda: set() # type: ignore[method-assign] - coordinator._reconcile_locked(strict=False) - assert coordinator.claims_worker("worker-1", "worker-fingerprint") is False - assert coordinator._console_degraded is False - - -def test_first_console_failure_survives_unique_route_ambiguity_and_retirement( - tmp_path: Path, -) -> None: - coordinator = AcpRuntimeCoordinator( - _config(tmp_path, policy="acp_preferred"), - threading.Event(), - reconcile_interval=60.0, - ) - coordinator._state = RuntimeState.RUNNING - runtime = SimpleNamespace( - status=lambda: SimpleNamespace(healthy=True, failure_type=None), - stop=lambda *, timeout: None, - _binding=_binding(), - ) - slot = _RuntimeSlot( - _binding(), - "42", - runtime, - console=HerdrAcpConsoleEndpoint(42, "console-lease"), - ) - coordinator._slots["worker-1"] = slot - coordinator._bridge_console_slot = ( # type: ignore[method-assign] - lambda _slot: (_ for _ in ()).throw(OSError("console unavailable")) - ) - coordinator._bridge_console_slot_supervised(slot) - assert coordinator.claims_worker("worker-1", "worker-fingerprint") is True - - # Two sendable routes make the worker non-unique. The old slot is stale, - # but exact Herdr authority remains and therefore so must the ACP claim. - coordinator._continuity_bindings = lambda: ({}, 1) # type: ignore[method-assign] - coordinator._herdr_authority_claims = ( # type: ignore[method-assign] - lambda: {("worker-1", "worker-fingerprint")} - ) - coordinator._reconcile_locked(strict=False) - - assert "worker-1" not in coordinator._slots - assert coordinator.claims_worker("worker-1", "worker-fingerprint") is True - assert coordinator.status()["healthy"] is False - - -def test_published_claim_survives_unhealthy_runtime_retire_and_failed_remint( - tmp_path: Path, -) -> None: - config = _config(tmp_path, policy="acp_preferred") - worker = _seed(config) - coordinator = AcpRuntimeCoordinator( - config, - threading.Event(), - reconcile_interval=60.0, - ) - coordinator._state = RuntimeState.RUNNING - runtime = SimpleNamespace( - status=lambda: SimpleNamespace(healthy=False, failure_type="runtime_failed"), - stop=lambda *, timeout: None, - _binding=_binding(), - ) - slot = _RuntimeSlot( - _binding(), - "42", - runtime, - console=HerdrAcpConsoleEndpoint(42, "console-lease"), - ) - coordinator._slots[worker.id] = slot - coordinator._published_acp_claims[worker.id] = worker.fingerprint - coordinator._continuity_bindings = ( # type: ignore[method-assign] - lambda: ({worker.id: _binding()}, 0) - ) - coordinator._resolve_endpoint = ( # type: ignore[method-assign] - lambda _continuity: (_ for _ in ()).throw( - AcpCoordinatorError("replacement attach failed") - ) - ) - - coordinator._reconcile_locked(strict=False) - - assert worker.id not in coordinator._slots - assert coordinator.claims_worker(worker.id, worker.fingerprint) is True - - def forbidden_legacy(_config: Config) -> Any: - raise AssertionError("published ACP ownership must survive failed remint") - - envelope = submit_command( - config, - _request("unhealthy-runtime-failed-remint"), - socket_client_factory=forbidden_legacy, - acp_prompt_router=coordinator.prompt_route, - acp_worker_owner=coordinator.claims_worker, - ) - assert envelope.status == "backend_unavailable" - assert envelope.disposition == "no_receipt" - assert get_command_request( - config.db_path, - config.host_id, - "unhealthy-runtime-failed-remint", - ) is None - - def test_console_submission_rejects_a_retired_generation_before_store_access( tmp_path: Path, ) -> None: @@ -1353,7 +1162,6 @@ def test_live_acp_route_uses_advertised_steering_despite_observer_lag( config, _request("request-active-steer"), acp_prompt_router=lambda routed: route if routed.id == observed.id else None, - acp_required=True, ) assert envelope.status == "accepted" @@ -1397,7 +1205,6 @@ def test_definite_acp_steering_failure_is_rejected_not_uncertain( config, _request("request-active-steer-failed"), acp_prompt_router=lambda _routed: route, - acp_required=True, ) assert envelope.status == "rejected" @@ -1443,7 +1250,6 @@ def test_acp_generation_preflight_failure_is_retryable_before_receipt( config, _request("request-preflight-retry"), acp_prompt_router=lambda routed: route if routed == worker else None, - acp_required=True, ) assert envelope.status == "backend_unavailable" @@ -1486,8 +1292,6 @@ def test_production_route_checks_generation_before_reserving_receipt( config, _request("request-production-preflight"), acp_prompt_router=coordinator.prompt_route, - acp_worker_owner=coordinator.claims_worker, - acp_required=True, ) assert envelope.status == "backend_unavailable" @@ -1510,7 +1314,6 @@ def test_acp_failure_before_transport_boundary_is_immediately_retryable( config, _request("request-prewrite-retry"), acp_prompt_router=lambda routed: failed_route if routed == worker else None, - acp_required=True, ) assert first.status == "backend_unavailable" assert first.disposition == "no_receipt" @@ -1528,7 +1331,6 @@ def test_acp_failure_before_transport_boundary_is_immediately_retryable( config, _request("request-prewrite-retry"), acp_prompt_router=lambda routed: good_route if routed == worker else None, - acp_required=True, ) assert second.status == "accepted" assert second.disposition == "terminal_accepted" @@ -1541,330 +1343,6 @@ def test_acp_failure_before_transport_boundary_is_immediately_retryable( assert receipt is not None and receipt["state"] == "accepted" -def test_preferred_acp_owned_route_loss_fails_closed_without_receipt_or_legacy( - tmp_path: Path, -) -> None: - config = _config(tmp_path, policy="acp_preferred") - worker = _seed(config) - - def forbidden_legacy(_config: Config) -> Any: - raise AssertionError("ACP-owned console loss must not reach legacy pane I/O") - - for _attempt in range(2): - envelope = submit_command( - config, - _request("preferred-owned-console-loss"), - socket_client_factory=forbidden_legacy, - acp_prompt_router=lambda _worker: None, - acp_worker_owner=lambda worker_id, fingerprint: ( - worker_id == worker.id and fingerprint == worker.fingerprint - ), - ) - assert envelope.status == "backend_unavailable" - assert envelope.disposition == "no_receipt" - assert get_command_request( - config.db_path, - config.host_id, - "preferred-owned-console-loss", - ) is None - - -def test_preferred_non_acp_worker_still_uses_legacy_sender(tmp_path: Path) -> None: - config = _config(tmp_path, policy="acp_preferred") - _seed(config) - legacy_calls: list[str] = [] - - class LegacyClient: - def connect(self) -> "LegacyClient": - return self - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - del timeout - legacy_calls.append(method) - if method == "agent.get": - return {"result": {"agent": {"pane_id": "pane-private"}}} - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": "pane-private"}, - "delivery": "submitted", - } - return {"accepted": True, "params": params} - - def close(self) -> None: - return None - - envelope = submit_command( - config, - _request("preferred-non-acp-worker"), - socket_client_factory=lambda _config: LegacyClient(), - acp_prompt_router=lambda _worker: None, - acp_worker_owner=lambda _worker_id, _fingerprint: False, - ) - - assert envelope.status == "accepted" - assert legacy_calls[-1] == "agent.prompt" - - -def test_preferred_snapshot_failure_with_owner_oracle_never_falls_back( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path, policy="acp_preferred") - _seed(config) - - def unavailable_snapshot(_config: Config) -> Snapshot: - raise OSError("authority store temporarily unavailable") - - def forbidden_legacy(_config: Config) -> Any: - raise AssertionError("unknown ACP ownership must not reach legacy pane I/O") - - monkeypatch.setattr( - "tendwire.command_submission._current_snapshot", unavailable_snapshot - ) - envelope = submit_command( - config, - _request("preferred-authority-read-failure"), - socket_client_factory=forbidden_legacy, - acp_prompt_router=lambda _worker: None, - acp_worker_owner=lambda _worker_id, _fingerprint: False, - ) - - assert envelope.status == "backend_unavailable" - assert envelope.disposition == "no_receipt" - assert get_command_request( - config.db_path, - config.host_id, - "preferred-authority-read-failure", - ) is None - - -def test_shadow_owned_command_is_observation_only_before_receipt(tmp_path: Path) -> None: - config = _config(tmp_path, policy="acp_shadow") - worker = replace(_seed(config), status="working") - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-07-31T00:00:01+00:00", - workers=[worker], - backend_health=[ - BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - ) - ], - ), - ) - route = _Route() - - envelope = submit_acp_command( - config, - _request("shadow-observation-only"), - prompt_router=lambda _worker: route, - observation_only=True, - ) - - assert envelope is not None - assert envelope.status == "backend_unavailable" - assert envelope.disposition == "no_receipt" - assert route.calls == [] - assert get_command_request( - config.db_path, - config.host_id, - "shadow-observation-only", - ) is None - - -def test_daemon_shadow_owned_command_never_reaches_legacy_sender(tmp_path: Path) -> None: - config = _config(tmp_path, policy="acp_shadow") - worker = _seed(config) - route = _Route() - legacy_calls: list[str] = [] - - class Runtime: - def prompt_route(self, routed: Worker) -> _Route | None: - return route if routed == worker else None - - def legacy_sender(_config: Config, _payload: str) -> Any: - legacy_calls.append("calibrate-or-write") - raise AssertionError("ACP-owned shadow target must not use legacy PTY I/O") - - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(submit_command=legacy_sender), - ) - daemon._acp_runtime = Runtime() - - envelope = daemon.submit_command(_request("shadow-daemon-fence")) - - assert envelope.status == "backend_unavailable" - assert envelope.disposition == "no_receipt" - assert route.calls == [] - assert legacy_calls == [] - - -def test_daemon_preferred_console_loss_uses_claim_to_block_legacy_sender( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path, policy="acp_preferred") - worker = _seed(config) - - class Runtime: - def prompt_route(self, _worker: Worker) -> None: - return None - - def claims_worker(self, worker_id: str, fingerprint: str) -> bool: - return worker_id == worker.id and fingerprint == worker.fingerprint - - def forbidden_legacy(_config: Config) -> Any: - raise AssertionError("ACP-owned console loss must not use legacy pane I/O") - - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - forbidden_legacy, - ) - daemon = TendwireDaemon(config) - daemon._acp_runtime = Runtime() - - envelope = daemon.submit_command(_request("daemon-preferred-console-loss")) - - assert envelope.status == "backend_unavailable" - assert envelope.disposition == "no_receipt" - assert get_command_request( - config.db_path, - config.host_id, - "daemon-preferred-console-loss", - ) is None - - -def test_daemon_shadow_preserves_ordinary_legacy_worker_submission( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path, policy="acp_shadow") - _seed(config) - legacy_calls: list[str] = [] - - class Runtime: - def prompt_route(self, _worker: Worker) -> None: - return None - - class LegacyClient: - def connect(self) -> "LegacyClient": - return self - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - del timeout - legacy_calls.append(method) - if method == "agent.get": - return {"result": {"agent": {"pane_id": "pane-private"}}} - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": "pane-private"}, - "delivery": "submitted", - } - return {"accepted": True, "params": params} - - def close(self) -> None: - return None - - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - lambda _config: LegacyClient(), - ) - daemon = TendwireDaemon(config) - daemon._acp_runtime = Runtime() - - envelope = daemon.submit_command(_request("shadow-legacy-worker")) - - assert envelope.status == "accepted" - assert legacy_calls[-1] == "agent.prompt" - - -def test_daemon_wires_shadow_ownership_fence_into_legacy_scheduler( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = replace( - _config(tmp_path, policy="acp_shadow"), - herdr_backend="cli", - socket_path=tmp_path / "shadow.sock", - ) - callback: Any | None = None - - class Runtime: - def start(self) -> None: - return None - - def stop(self, *, timeout: float) -> None: - return None - - def status(self) -> dict[str, Any]: - return {"state": "running", "healthy": True} - - def owns_worker(self, worker_id: str, fingerprint: str) -> bool: - return worker_id == "worker-1" and fingerprint == "worker-fingerprint" - - class Scheduler: - def set_worker_exclusion(self, value: Any) -> None: - nonlocal callback - callback = value - - def start(self) -> None: - return None - - def request_refresh(self) -> None: - return None - - def stop(self, *, flush_timeout_seconds: float) -> None: - return None - - def observe(_config: Config) -> Snapshot: - snapshot = Snapshot( - host_id=config.host_id, - updated_at="2026-07-31T00:00:00+00:00", - workers=[], - backend_health=[], - ) - assert config.db_path is not None - save_snapshot(config.db_path, snapshot) - return snapshot - - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - observe_initial_snapshot=observe, - turn_scheduler_factory=lambda _config: Scheduler(), - acp_runtime_factory=lambda _config, _stop: Runtime(), - ), - ) - monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", lambda _self: None) - try: - daemon.start() - assert callable(callback) - assert callback("worker-1", "worker-fingerprint") is True - assert callback("legacy-worker", "legacy-fingerprint") is False - finally: - daemon.stop() - - def test_route_authority_failure_is_safe_before_receipt_reservation(tmp_path: Path) -> None: config = _config(tmp_path) _seed(config) @@ -1877,19 +1355,10 @@ def binding_fingerprint(self) -> str: def prompt(self, *_args: Any, **_kwargs: Any) -> None: raise AssertionError("a route without authority must not send") - assert ( - submit_acp_command( - config, - _request("route-race-preferred"), - prompt_router=lambda _worker: VanishedRoute(), - ) - is None - ) required = submit_acp_command( config, _request("route-race-required"), prompt_router=lambda _worker: VanishedRoute(), - required=True, ) assert required is not None assert required.status == "backend_unavailable" @@ -1945,13 +1414,12 @@ def prompt( def test_required_has_no_legacy_fallback_when_route_is_absent(tmp_path: Path) -> None: - config = _config(tmp_path, policy="acp_required") + config = _config(tmp_path) _seed(config) envelope = submit_command( config, _request("request-required"), acp_prompt_router=lambda _worker: None, - acp_required=True, ) assert envelope.status == "backend_unavailable" assert get_command_request( @@ -2051,8 +1519,8 @@ def submit_prompt(self, *_args: Any, **_kwargs: Any) -> None: config, threading.Event(), endpoint_client_factory=lambda _config: EndpointClient(), - client_factory=lambda *_args, **_kwargs: object(), - runtime_factory=Runtime, + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, reconcile_interval=60.0, ).start() try: @@ -2138,8 +1606,8 @@ def status(self) -> Any: config, threading.Event(), endpoint_client_factory=lambda _config: EndpointClient(), - client_factory=lambda *_args, **_kwargs: object(), - runtime_factory=Runtime, + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, reconcile_interval=60.0, ).start() upsert_worker_bindings(config.db_path, [_binding()]) @@ -2194,8 +1662,8 @@ def status(self) -> Any: config, threading.Event(), endpoint_client_factory=lambda _config: EndpointClient(), - client_factory=lambda *_args, **_kwargs: object(), - runtime_factory=Runtime, + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, reconcile_interval=60.0, ).start() upsert_worker_bindings(config.db_path, [_binding()]) @@ -2276,8 +1744,8 @@ def status(self) -> Any: config, threading.Event(), endpoint_client_factory=lambda _config: EndpointClient(), - client_factory=lambda *_args, **_kwargs: object(), - runtime_factory=Runtime, + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, durable_permission_bridge=True, reconcile_interval=60.0, ).start() @@ -2352,8 +1820,8 @@ def status(self) -> Any: config, threading.Event(), endpoint_client_factory=lambda _config: EndpointClient(), - client_factory=lambda *_args, **_kwargs: object(), - runtime_factory=Runtime, + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, reconcile_interval=60.0, ).start() worker = Worker( @@ -2419,8 +1887,8 @@ def close(self) -> None: config, threading.Event(), endpoint_client_factory=lambda _config: EndpointClient(), - client_factory=lambda *_args, **_kwargs: client, - runtime_factory=lambda *_args, **_kwargs: (_ for _ in ()).throw( + connection_factory=lambda *_args, **_kwargs: client, + session_factory=lambda *_args, **_kwargs: (_ for _ in ()).throw( RuntimeError("constructor failed with --ticket private") ), reconcile_interval=60.0, @@ -2446,7 +1914,7 @@ def test_coordinator_start_revokes_orphaned_process_binding(tmp_path: Path) -> N config, threading.Event(), endpoint_client_factory=lambda _config: object(), - client_factory=lambda *_args, **_kwargs: object(), + connection_factory=lambda *_args, **_kwargs: object(), reconcile_interval=60.0, ).start() try: @@ -2508,8 +1976,8 @@ def permission_bridge(_request: Any) -> str | None: config, threading.Event(), endpoint_client_factory=lambda _config: EndpointClient(), - client_factory=lambda *_args, **_kwargs: object(), - runtime_factory=Runtime, + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, permission_callback=permission_bridge, require_permission_bridge=True, reconcile_interval=60.0, @@ -2520,43 +1988,10 @@ def permission_bridge(_request: Any) -> str | None: coordinator.stop() -@pytest.mark.parametrize("policy", ["acp_shadow", "acp_preferred"]) -def test_acp_owned_worker_is_excluded_from_legacy_scheduler( - tmp_path: Path, - policy: str, -) -> None: - config = _config(tmp_path, policy=policy) - assert config.db_path is not None - init_store(config.db_path) - upsert_worker_bindings(config.db_path, [_binding()]) - read = threading.Event() - - def reader(*_args: Any, **_kwargs: Any) -> TurnRefreshResult: - read.set() - return TurnRefreshResult("updated", 1) - - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=0.05, - max_workers=1, - reader=reader, - ) - scheduler.set_worker_exclusion( - lambda worker_id, fingerprint: ( - worker_id == "worker-1" and fingerprint == "worker-fingerprint" - ) - ) - scheduler.start() - try: - assert not read.wait(0.2) - finally: - scheduler.stop(flush_timeout_seconds=1.0) - - def test_required_zero_workers_is_idle_healthy_then_new_unowned_worker_degrades( tmp_path: Path, ) -> None: - config = _config(tmp_path, policy="acp_required") + config = _config(tmp_path) assert config.db_path is not None init_store(config.db_path) @@ -2584,89 +2019,8 @@ def close(self) -> None: coordinator.stop() -def test_preferred_caches_exact_non_acp_generation_without_endpoint_churn( - tmp_path: Path, -) -> None: - config = _config(tmp_path, policy="acp_preferred") - assert config.db_path is not None - init_store(config.db_path) - upsert_worker_bindings(config.db_path, [_binding()]) - endpoint_calls = 0 - status_calls = 0 - registered = False - - class NonAcpClient: - def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: - nonlocal endpoint_calls - endpoint_calls += 1 - if registered: - return _endpoint() - raise HerdrErrorResponse( - { - "code": "acp_worker_unauthenticated", - "message": "worker is not ACP-owned", - }, - "request-private", - ) - - def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: - nonlocal status_calls - status_calls += 1 - if registered: - return _status(lifecycle="acp_owned_ready") - raise HerdrErrorResponse( - { - "code": "acp_worker_unauthenticated", - "message": "worker is not ACP-owned", - }, - "request-private", - ) - - def close(self) -> None: - return None - - class Runtime: - def __init__(self, _client: Any, **kwargs: Any) -> None: - self._binding = kwargs["binding"] - self.stopped = False - - def start(self) -> None: - return None - - def stop(self, *, timeout: float) -> None: - self.stopped = True - - def status(self) -> Any: - return SimpleNamespace(healthy=not self.stopped, failure_type=None) - - coordinator = AcpRuntimeCoordinator( - config, - threading.Event(), - endpoint_client_factory=lambda _config: NonAcpClient(), - client_factory=lambda *_args, **_kwargs: object(), - runtime_factory=Runtime, - reconcile_interval=60.0, - ).start() - try: - assert endpoint_calls == 1 - assert coordinator.status()["healthy"] is True - coordinator._reconcile(strict=False) - coordinator._reconcile(strict=False) - assert endpoint_calls == 1 - assert status_calls == 2 - assert coordinator.status()["failure_type"] is None - - registered = True - coordinator._reconcile(strict=False) - assert endpoint_calls == 2 - assert status_calls == 3 - assert "worker-1" in coordinator._slots - finally: - coordinator.stop() - - def test_required_does_not_cache_non_acp_endpoint_failure(tmp_path: Path) -> None: - config = _config(tmp_path, policy="acp_required") + config = _config(tmp_path) assert config.db_path is not None init_store(config.db_path) upsert_worker_bindings(config.db_path, [_binding()]) diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py index 3d90cc4..107e72e 100644 --- a/tests/test_acp_ingestion.py +++ b/tests/test_acp_ingestion.py @@ -10,13 +10,14 @@ from tendwire.backends.acp_ingestion import AcpSessionIngestor from tendwire.config import Config from tendwire.core.agent_events import AgentEvent, AppendBoundAgentEventResult -from tendwire.core.models import WorkerBinding +from tendwire.core.models import Snapshot, Worker, WorkerBinding from tendwire.backends.acp_protocol import StopReason from tendwire.store.sqlite import ( AppendProjectedAgentEventResult, TurnRefreshApplyResult, list_agent_events, list_public_agent_events, + save_snapshot, upsert_worker_bindings, ) @@ -81,11 +82,9 @@ def persist( def _config(db_path: Path, **kwargs: object) -> Config: - agent_event_source = str(kwargs.pop("agent_event_source", "acp_preferred")) return Config( host_id="host-a", db_path=db_path, - agent_event_source=agent_event_source, **kwargs, ) @@ -157,44 +156,6 @@ def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): assert turns[-1]["source_turn_id"] == turn_id -def test_shadow_mode_journals_without_turn_projection(tmp_path: Path) -> None: - events: list[AgentEvent] = [] - - def append( - _path: Path | str, - _host: str, - event: AgentEvent, - **_kwargs, - ) -> AppendBoundAgentEventResult: - events.append(event) - return _appended(1, event) - - def unexpected_turn(*_args, **_kwargs): - raise AssertionError("shadow mode must not project turns") - - ingestor = AcpSessionIngestor( - _config(tmp_path / "events.db", agent_event_source="acp_shadow"), - session_id="session-a", - stream_generation="generation-a", - binding=_binding(), - persist_event=_persist(append, unexpected_turn), - ) - result = ingestor.ingest_update( - _update( - "agent_message_chunk", - content={"type": "text", "text": "shadow"}, - ) - ) - - assert result.event is not None - assert result.turn is None - assert len(events) == 1 - assert ingestor.source_turn_id is not None - assert ingestor.projector.project_turn_content("session-a")[ - "assistant_stream_text" - ] == "shadow" - - def test_disabled_thought_policy_discards_before_persistence(tmp_path: Path) -> None: def unexpected_append(*_args, **_kwargs): raise AssertionError("disabled thoughts must not be persisted") @@ -295,6 +256,20 @@ def test_required_mode_fails_closed_when_durable_binding_is_stale( ) -> None: db_path = tmp_path / "events.db" binding = _binding() + save_snapshot( + db_path, + Snapshot( + host_id="host-a", + updated_at="2026-01-01T00:00:00+00:00", + workers=[ + Worker( + id=binding.worker_id, + name="Worker A", + fingerprint=binding.worker_fingerprint, + ) + ], + ), + ) upsert_worker_bindings(db_path, [binding]) replacement = replace( binding, @@ -304,7 +279,7 @@ def test_required_mode_fails_closed_when_durable_binding_is_stale( upsert_worker_bindings(db_path, [replacement]) ingestor = AcpSessionIngestor( - _config(db_path, agent_event_source="acp_required"), + _config(db_path), session_id="session-a", stream_generation="generation-a", binding=binding, @@ -334,7 +309,7 @@ def test_default_authority_check_accepts_the_current_durable_binding( binding = _binding() upsert_worker_bindings(db_path, [binding]) ingestor = AcpSessionIngestor( - _config(db_path, agent_event_source="acp_required"), + _config(db_path), session_id="session-a", stream_generation="generation-a", binding=binding, @@ -347,93 +322,6 @@ def test_default_authority_check_accepts_the_current_durable_binding( assert result.ignored_reason is None -def test_shadow_completion_never_projects_and_finality_is_idempotent( - tmp_path: Path, -) -> None: - events: list[AgentEvent] = [] - - def append( - _path: Path | str, - _host: str, - event: AgentEvent, - **_kwargs, - ) -> AppendBoundAgentEventResult: - events.append(event) - return _appended(len(events), event) - - def unexpected_turn(*_args, **_kwargs): - raise AssertionError("shadow mode must never project, including completion") - - ingestor = AcpSessionIngestor( - _config(tmp_path / "events.db", agent_event_source="acp_shadow"), - session_id="session-a", - stream_generation="generation-a", - binding=_binding(), - persist_event=_persist(append, unexpected_turn), - ) - ingestor.start_turn(producer_turn_id="turn-1") - ingestor.ingest_update( - _update( - "agent_message_chunk", - content={"type": "text", "text": "shadow final"}, - ) - ) - - completed = ingestor.mark_prompt_complete() - repeated = ingestor.mark_prompt_complete() - late = ingestor.ingest_update( - _update( - "agent_message_chunk", - content={"type": "text", "text": "late mutation"}, - ) - ) - - assert completed.turn is None - assert repeated.ignored_reason == "turn_already_complete" - assert late.ignored_reason == "turn_already_complete" - assert len(events) == 2 - assert events[-1].kind == "extension" - - -def test_required_mode_projects_messages_and_final_exactly_once(tmp_path: Path) -> None: - events: list[AgentEvent] = [] - turns: list[dict[str, object]] = [] - - def append( - _path: Path | str, - _host: str, - event: AgentEvent, - **_kwargs, - ) -> AppendBoundAgentEventResult: - events.append(event) - return _appended(len(events), event) - - def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): - turns.append(dict(content)) - return TurnRefreshApplyResult(1, False) - - ingestor = AcpSessionIngestor( - _config(tmp_path / "events.db", agent_event_source="acp_required"), - session_id="session-a", - stream_generation="generation-a", - binding=_binding(), - persist_event=_persist(append, apply), - ) - ingestor.start_turn(producer_turn_id="turn-1") - streamed = ingestor.ingest_update( - _update( - "agent_message_chunk", - content={"type": "text", "text": "answer"}, - ) - ) - completed = ingestor.mark_prompt_complete() - - assert streamed.turn is not None - assert completed.turn is not None - assert [turn["complete"] for turn in turns] == [False, True] - assert turns[-1]["assistant_final_text"] == "answer" - assert turns[-1]["assistant_stream_text"] == "" - @pytest.mark.parametrize( ("stop_reason", "outcome", "notice"), @@ -467,7 +355,7 @@ def apply(_path, _host, _worker, content, **_kwargs): return TurnRefreshApplyResult(len(turns), False) ingestor = AcpSessionIngestor( - _config(tmp_path / "events.db", agent_event_source="acp_required"), + _config(tmp_path / "events.db"), session_id="session-a", stream_generation="generation-a", binding=_binding(), @@ -497,9 +385,23 @@ def test_live_prompt_echo_is_suppressed_but_load_replay_user_message_is_retained ) -> None: db_path = tmp_path / "events.db" binding = _binding() + save_snapshot( + db_path, + Snapshot( + host_id="host-a", + updated_at="2026-01-01T00:00:00+00:00", + workers=[ + Worker( + id=binding.worker_id, + name="Worker A", + fingerprint=binding.worker_fingerprint, + ) + ], + ), + ) upsert_worker_bindings(db_path, [binding]) ingestor = AcpSessionIngestor( - _config(db_path, agent_event_source="acp_shadow"), + _config(db_path), session_id="session-a", stream_generation="generation-a", binding=binding, diff --git a/tests/test_acp_permissions.py b/tests/test_acp_permissions.py index 99cd2a8..f1cb93c 100644 --- a/tests/test_acp_permissions.py +++ b/tests/test_acp_permissions.py @@ -21,23 +21,97 @@ PermissionRequest, SessionResult, ) -from tendwire.backends.acp_runtime import AcpRuntime, SessionOpenMode +from tendwire.backends.acp_runtime import AcpWorkerSession, SessionOpenMode from tendwire.command_submission import submit_command -from tendwire.core.models import Worker +from tendwire.config import Config +from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding from tendwire.daemon import TendwireDaemon from tendwire.store.sqlite import ( expire_worker_bindings, get_command_request, + init_store, list_worker_bindings, pending_payload_from_store, + save_snapshot, upsert_worker_bindings, ) -from tests.test_answer_decision import _answer_request -from tests.test_command_submission import _binding, _config, _seed from tests.test_acp_runtime import FakeClient, FakeIngestor +def _config(tmp_path: Path) -> Config: + return Config( + host_id="cmd-host", + data_dir=tmp_path, + db_path=tmp_path / "commands.db", + herdr_backend="socket", + ) + + +def _binding(worker: Worker) -> WorkerBinding: + return WorkerBinding( + host_id="cmd-host", + worker_id=worker.id, + worker_fingerprint=worker.fingerprint, + backend="herdr", + target_kind="agent_id", + target_value="agent-secret", + turn_target_kind=None, + turn_target_value=None, + sendable=True, + reason=None, + observed_at="2026-01-01T00:00:00+00:00", + private_fingerprint="private-secret", + ) + + +def _seed( + config: Config, + workers: list[Worker], + bindings: list[WorkerBinding], +) -> None: + assert config.db_path is not None + init_store(config.db_path) + save_snapshot( + config.db_path, + Snapshot( + host_id=config.host_id, + updated_at="2026-01-01T00:00:00+00:00", + workers=workers, + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty", + observed_at="2026-01-01T00:00:00+00:00", + counts={"workers": len(workers)}, + ) + ], + ), + ) + upsert_worker_bindings(config.db_path, bindings) + + +def _answer_request( + decision_ref: str, + *, + request_id: str = "decision-request-1", + worker_id: str = "w-1", + selection: dict[str, Any] | None = None, +) -> dict[str, Any]: + return { + "schema_version": 1, + "action": "answer_decision", + "request_id": request_id, + "dry_run": False, + "target": {"worker_id": worker_id}, + "params": { + "decision_ref": decision_ref, + "selection": selection or {"option_refs": ["2"]}, + }, + } + + class _Router: def __init__(self, broker: AcpPermissionBroker) -> None: self.broker = broker @@ -133,7 +207,7 @@ def test_permission_bridge_is_private_durable_and_frame_acknowledged( )[0] client = FakeClient() client.restored_session_result = SessionResult(session_id, None, (), {}) - runtime = AcpRuntime( + runtime = AcpWorkerSession( client, config=config, binding=acp_binding, @@ -202,16 +276,13 @@ def test_acp_permission_never_falls_back_to_legacy_socket(tmp_path: Path) -> Non ], reason="test_retired_before_answer", ) == 1 - socket_calls: list[bool] = [] result = submit_command( config, _answer_request(pending["meta"]["decision"]["decision_ref"]), - socket_client_factory=lambda _config: socket_calls.append(True), acp_permission_router=None, ) assert result.ok is False assert result.disposition == "no_receipt" - assert socket_calls == [] broker.close() thread.join(timeout=2) @@ -379,11 +450,11 @@ def test_v27_provenance_migration_preserves_stale_pending_state( ).fetchone() == ("stale", "legacy") -def test_shadow_daemon_routes_permission_answers_without_enabling_acp_prompts( +def test_daemon_routes_acp_permission_answers_without_a_prompt_route( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - config = replace(_config(tmp_path), agent_event_source="acp_shadow") + config = _config(tmp_path) class Runtime: def answer_permission_decision(self, _decision: Any, *, timeout: float) -> None: @@ -391,7 +462,7 @@ def answer_permission_decision(self, _decision: Any, *, timeout: float) -> None: runtime = Runtime() daemon = TendwireDaemon(config) - daemon._acp_runtime = runtime + daemon._acp_supervisor = runtime captured: dict[str, Any] = {} def submit(_config: Any, _payload: Any, **kwargs: Any) -> str: diff --git a/tests/test_acp_probe.py b/tests/test_acp_probe.py index 11e35ac..5dfd4f4 100644 --- a/tests/test_acp_probe.py +++ b/tests/test_acp_probe.py @@ -189,10 +189,11 @@ def test_extension_count_uses_only_spec_reserved_meta_locations() -> None: assert _extension_capability_count({"forbiddenRootExtension": {}}) == 0 -def test_authentication_count_skips_invalid_stable_schema_items() -> None: +def test_upstream_schema_rejects_invalid_authentication_items() -> None: payload = probe_adapter(adapter_argv("auth_shapes")).to_payload() - assert payload["initialization_compatible"] is True + assert payload["initialization_compatible"] is False assert payload["authentication"] == { - "method_count": 1, + "method_count": 0, "method_count_capped": False, } + assert payload["failure"] == "protocol_error" diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py index 325d57c..ea70c7c 100644 --- a/tests/test_acp_runtime.py +++ b/tests/test_acp_runtime.py @@ -12,7 +12,10 @@ import pytest -from tendwire.backends.acp_client import AcpClient, AcpRequestTimeoutError +from tendwire.backends.acp_client import ( + AcpRequestTimeoutError, + BoundedAcpConnection as AcpClient, +) from tendwire.backends.acp_protocol import ( PermissionOption, PermissionOptionKind, @@ -26,7 +29,7 @@ SteeringResult, ) from tendwire.backends.acp_runtime import ( - AcpRuntime, + AcpWorkerSession as AcpRuntime, AcpRuntimeBindingError, AcpRuntimeProtocolError, AcpRuntimeStateError, @@ -314,7 +317,6 @@ def runtime( config=Config( host_id="host-a", db_path=db_path, - agent_event_source="acp_required", ), binding=continuity, cwd=tmp_path, @@ -340,7 +342,6 @@ def bound_runtime( config=Config( host_id="host-a", db_path=db_path, - agent_event_source="acp_required", ), binding=current_binding, cwd=tmp_path, @@ -504,7 +505,6 @@ def test_load_and_resume_use_requested_session( config=Config( host_id="host-a", db_path=db_path, - agent_event_source="acp_required", ), binding=existing, cwd=tmp_path, @@ -551,7 +551,6 @@ def test_load_and_resume_reject_agent_session_mismatch_and_close( config=Config( host_id="host-a", db_path=db_path, - agent_event_source="acp_required", ), binding=existing, cwd=tmp_path, @@ -592,7 +591,6 @@ def test_load_and_resume_reject_non_acp_binding_before_transport( config=Config( host_id="host-a", db_path=tmp_path / "events.db", - agent_event_source="acp_required", ), binding=legacy, cwd=tmp_path, @@ -1539,7 +1537,6 @@ def test_load_drains_replay_larger_than_client_queue_before_response( config=Config( host_id="host-a", db_path=db_path, - agent_event_source="acp_required", ), binding=current, cwd=tmp_path, diff --git a/tests/test_answer_decision.py b/tests/test_answer_decision.py deleted file mode 100644 index 249513d..0000000 --- a/tests/test_answer_decision.py +++ /dev/null @@ -1,888 +0,0 @@ -"""Semantic connector answers for current backend-owned Claude decisions.""" - -from __future__ import annotations - -from pathlib import Path -from typing import Any - -import pytest -import tendwire.command_submission as command_submission - -from tendwire.backends.herdr_decision import calibrate_decision_steps -from tendwire.backends.herdr_turns import ( - PENDING_DECISION_MAX_OPTIONS, - _pending_observation_from_turn, -) -from tendwire.command_submission import submit_command -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - STATUS_ACCEPTED, - STATUS_ANSWER_IN_PROGRESS, - STATUS_DECISION_NOT_PENDING, - STATUS_INVALID_SELECTION, - STATUS_PENDING, - STATUS_REQUEST_STATE_UNCERTAIN, - STATUS_UNKNOWN_WORKER, - STATUS_UNSUPPORTED_DECISION, - CommandRequest, - build_canonical_mutation, -) -from tendwire.core.models import Worker -from tendwire.store.sqlite import ( - abandon_backend_pending_choice_claim, - apply_backend_pending_observation, - claim_backend_pending_decision, - envelope_to_receipt_json, - get_command_request, - pending_payload_from_store, - reserve_command_request, -) - -from tests.test_command_submission import ( - _FakeSocketClient, - _binding, - _config, - _factory, - _seed, -) - - -def _decision_turn( - *, - prompt: str = "Choose a database", - kind: str = "AskUserQuestion", - multi_select: bool = False, - question_count: int = 1, -) -> dict[str, Any]: - return { - "pending_decision": { - "decision_id": "private-tool-use", - "kind": kind, - "question": prompt, - "options": [ - {"id": "postgres", "label": "Postgres"}, - {"id": "sqlite", "label": "SQLite"}, - {"id": "duckdb", "label": "DuckDB"}, - {"id": "mysql", "label": "MySQL"}, - ], - "multi_select": multi_select, - "question_count": question_count, - } - } - - -def _seed_pending_decision( - tmp_path: Path, - *, - turn: dict[str, Any] | None = None, - turn_model: str = "observed", -) -> tuple[Any, Worker, str]: - config = _config(tmp_path, turn_model=turn_model) - worker = Worker(id="w-1", name="Alpha", status="active") - binding = _binding( - worker, - private_fingerprint="decision-binding-private", - turn_target_value="decision-pane-private", - ) - _seed(config, [worker], [binding]) - assert config.db_path is not None - observation = _pending_observation_from_turn(turn or _decision_turn()) - assert apply_backend_pending_observation( - config.db_path, - config.host_id, - worker.id, - observation, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - payload = pending_payload_from_store(config.db_path, config.host_id) - row = next(item for item in payload["pending_interactions"] if item["worker_id"] == worker.id) - return config, worker, row["meta"]["decision"]["decision_ref"] - - -def _answer_request( - decision_ref: str, - *, - request_id: str = "decision-request-1", - worker_id: str = "w-1", - selection: dict[str, Any] | None = None, -) -> dict[str, Any]: - return { - "schema_version": 1, - "action": "answer_decision", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": worker_id}, - "params": { - "decision_ref": decision_ref, - "selection": selection or {"option_refs": ["2"]}, - }, - } - - -def test_pending_payload_carries_stable_structured_decision_and_rotates_ref( - tmp_path: Path, -) -> None: - config, worker, first_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - first = pending_payload_from_store(config.db_path, config.host_id) - first_row = next(item for item in first["pending_interactions"] if item["worker_id"] == worker.id) - assert first_row["meta"]["decision"] == { - "decision_ref": first_ref, - "kind": "single", - "prompt": "Choose a database", - "options": [ - {"ref": "1", "label": "Postgres"}, - {"ref": "2", "label": "SQLite"}, - {"ref": "3", "label": "DuckDB"}, - {"ref": "4", "label": "MySQL"}, - ], - "multi_select": False, - "question_count": 1, - } - - binding = _binding( - worker, - private_fingerprint="decision-binding-private", - turn_target_value="decision-pane-private", - ) - changed = _pending_observation_from_turn( - _decision_turn(prompt="Choose a durable database") - ) - assert apply_backend_pending_observation( - config.db_path, - config.host_id, - worker.id, - changed, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - second = pending_payload_from_store(config.db_path, config.host_id) - second_row = next(item for item in second["pending_interactions"] if item["worker_id"] == worker.id) - assert second_row["meta"]["decision"]["decision_ref"] != first_ref - - -def test_answer_decision_stale_ref_fails_before_pane_io(tmp_path: Path) -> None: - config, _worker, _decision_ref = _seed_pending_decision(tmp_path) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request("decision-stale"), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_DECISION_NOT_PENDING - assert calls == [] - - -def test_answer_decision_omitted_dry_run_is_preview_only(tmp_path: Path) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - calls: list[dict[str, Any]] = [] - request = _answer_request(decision_ref) - request.pop("dry_run") - - result = submit_command( - config, - request, - socket_client_factory=_factory(calls), - ) - - assert result.ok is True - assert result.status == "dry_run" - assert result.dry_run is True - assert calls == [] - - -def test_answer_decision_unknown_worker_fails_before_pane_io(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request("decision-any", worker_id="worker-missing"), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_UNKNOWN_WORKER - assert calls == [] - - -@pytest.mark.parametrize( - "selection", - [ - {"option_refs": ["9"]}, - {"option_refs": ["1", "2"]}, - {"option_refs": ["2", "2"]}, - {"text": ""}, - {"option_refs": ["1"], "text": "both"}, - ], -) -def test_answer_decision_invalid_selection_fails_before_pane_io( - tmp_path: Path, - selection: dict[str, Any], -) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request(decision_ref, selection=selection), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_INVALID_SELECTION - assert calls == [] - - -def test_answer_decision_refuses_multi_question_before_pane_io(tmp_path: Path) -> None: - config, _worker, decision_ref = _seed_pending_decision( - tmp_path, - turn=_decision_turn(question_count=2), - ) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request(decision_ref), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_UNSUPPORTED_DECISION - assert calls == [] - - -def test_answer_decision_rejects_plan_write_in_before_pane_io(tmp_path: Path) -> None: - config, _worker, decision_ref = _seed_pending_decision( - tmp_path, - turn=_decision_turn(kind="ExitPlanMode"), - ) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request(decision_ref, selection={"text": "Revise this plan"}), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_INVALID_SELECTION - assert calls == [] - - -def test_decision_calibration_steps_cover_single_plan_write_in_and_multi() -> None: - single = calibrate_decision_steps( - kind="single", option_count=4, option_refs=("2",) - ) - assert [(item.operation, item.keys, item.text) for item in single] == [ - ("keys", ("2",), None) # digit alone selects AND submits (live-verified) - ] - - plan = calibrate_decision_steps( - kind="plan", option_count=2, option_refs=("1",) - ) - assert [(item.operation, item.keys, item.text) for item in plan] == [ - ("keys", ("1",), None) - ] - - write_in = calibrate_decision_steps( - kind="single", option_count=4, text="Use another database" - ) - assert [(item.operation, item.keys, item.text) for item in write_in] == [ - ("keys", ("Down", "Down", "Down", "Down"), None), # write-in row ignores digits - ("input", ("Enter",), "Use another database"), - ] - - multi = calibrate_decision_steps( - kind="multi", option_count=4, option_refs=("3", "1") - ) - assert [(item.operation, item.keys, item.text) for item in multi] == [ - ("keys", ("1",), None), # digits toggle rows absolutely - ("keys", ("3",), None), - ("keys", ("Right", "Enter"), None), # Submit tab, then submit - ] - - -def test_production_shaped_multi_decision_end_to_end(tmp_path: Path) -> None: - tool_input = { - "questions": [ - { - "question": "Which databases should we support?", - "header": "Database support", - "options": [ - {"label": "Postgres", "description": "Primary database"}, - {"label": "SQLite", "description": "Local database"}, - {"label": "DuckDB", "description": "Analytics database"}, - {"label": "MySQL", "description": "Compatibility database"}, - ], - "multiSelect": True, - } - ] - } - question = tool_input["questions"][0] - adapter_pending_decision = { - "decision_id": "toolu_multi_123", - "prompt": f'{question["header"]}: {question["question"]}', - "mode": "multi", - "multi_select": question["multiSelect"], - "options": [ - {"id": str(ordinal), "label": option["label"]} - for ordinal, option in enumerate(question["options"], 1) - ], - } - assert adapter_pending_decision == { - "decision_id": "toolu_multi_123", - "prompt": "Database support: Which databases should we support?", - "mode": "multi", - "multi_select": True, - "options": [ - {"id": "1", "label": "Postgres"}, - {"id": "2", "label": "SQLite"}, - {"id": "3", "label": "DuckDB"}, - {"id": "4", "label": "MySQL"}, - ], - } - - config, worker, decision_ref = _seed_pending_decision( - tmp_path, - turn={"pending_decision": adapter_pending_decision}, - ) - assert config.db_path is not None - payload = pending_payload_from_store(config.db_path, config.host_id) - pending = next( - row for row in payload["pending_interactions"] - if row["worker_id"] == worker.id - ) - assert pending["meta"]["decision"] == { - "decision_ref": decision_ref, - "kind": "multi", - "prompt": "Database support: Which databases should we support?", - "options": [ - {"ref": "1", "label": "Postgres"}, - {"ref": "2", "label": "SQLite"}, - {"ref": "3", "label": "DuckDB"}, - {"ref": "4", "label": "MySQL"}, - ], - "multi_select": True, - "question_count": 1, - } - - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request( - decision_ref, - request_id="multi-production-answer", - selection={"option_refs": ["3", "1"]}, - ), - socket_client_factory=_factory(calls), - ) - - assert result.ok is True - assert result.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "pane.send_keys", - "params": {"pane_id": "decision-pane-private", "keys": ["1"]}, - }, - { - "method": "pane.send_keys", - "params": {"pane_id": "decision-pane-private", "keys": ["3"]}, - }, - { - "method": "pane.send_keys", - "params": {"pane_id": "decision-pane-private", "keys": ["Right", "Enter"]}, - }, - ] - - -def _bounded_decision_turn(option_count: int, *, custom_last: bool = False) -> dict[str, Any]: - options = [ - {"id": str(ordinal), "label": f"Option {ordinal}"} - for ordinal in range(1, option_count + 1) - ] - if custom_last: - options.append({"id": "custom", "label": "Type something"}) - return { - "pending_decision": { - "decision_id": "toolu_bound", - "kind": "AskUserQuestion", - "prompt": "Choose one", - "multi_select": False, - "options": options, - } - } - - -def _unknown_decision_turn() -> dict[str, Any]: - turn = _bounded_decision_turn(2) - turn["pending_decision"]["kind"] = "FutureDecisionKind" - return turn - - -def test_decision_option_bound_accepts_exactly_nine(tmp_path: Path) -> None: - config, worker, _decision_ref = _seed_pending_decision( - tmp_path, - turn=_bounded_decision_turn(PENDING_DECISION_MAX_OPTIONS), - ) - assert config.db_path is not None - payload = pending_payload_from_store(config.db_path, config.host_id) - row = next( - item for item in payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert len(row["meta"]["decision"]["options"]) == PENDING_DECISION_MAX_OPTIONS - - -def test_decision_option_bound_accepts_nine_plus_trailing_write_in( - tmp_path: Path, -) -> None: - config, worker, _decision_ref = _seed_pending_decision( - tmp_path, - turn=_bounded_decision_turn( - PENDING_DECISION_MAX_OPTIONS, - custom_last=True, - ), - ) - assert config.db_path is not None - payload = pending_payload_from_store(config.db_path, config.host_id) - row = next( - item for item in payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert len(row["meta"]["decision"]["options"]) == PENDING_DECISION_MAX_OPTIONS - - -@pytest.mark.parametrize( - "turn", - [ - _bounded_decision_turn(PENDING_DECISION_MAX_OPTIONS + 1), - _bounded_decision_turn( - PENDING_DECISION_MAX_OPTIONS + 1, - custom_last=True, - ), - _unknown_decision_turn(), - ], - ids=[ - "ten-real-options", - "custom-row-after-ten-real-options", - "unknown-kind", - ], -) -def test_over_bound_decision_fails_closed_without_pane_io( - tmp_path: Path, - turn: dict[str, Any], -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - binding = _binding( - worker, - private_fingerprint="decision-binding-private", - turn_target_value="decision-pane-private", - ) - _seed(config, [worker], [binding]) - observation = _pending_observation_from_turn(turn) - assert observation.kind == "read_succeeded_unsupported_decision" - assert config.db_path is not None - assert apply_backend_pending_observation( - config.db_path, - config.host_id, - worker.id, - observation, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - payload = pending_payload_from_store(config.db_path, config.host_id) - assert not any( - item["worker_id"] == worker.id - for item in payload["pending_interactions"] - ) - - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request("decision-unsupported-bound"), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_UNSUPPORTED_DECISION - assert calls == [] - - -def test_answer_decision_request_id_replay_does_not_resend_keys(tmp_path: Path) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - calls: list[dict[str, Any]] = [] - request = _answer_request(decision_ref) - - first = submit_command(config, request, socket_client_factory=_factory(calls)) - replay = submit_command(config, request, socket_client_factory=_factory(calls)) - - assert first.ok is True - assert first.status == STATUS_ACCEPTED - assert replay.to_dict() == first.to_dict() - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_answer_decision_observed_mode_completes_without_instruction_turn( - tmp_path: Path, -) -> None: - config, _worker, decision_ref = _seed_pending_decision( - tmp_path, - turn_model="observed", - ) - calls: list[dict[str, Any]] = [] - - result = submit_command( - config, - _answer_request(decision_ref, request_id="observed-answer-decision"), - socket_client_factory=_factory(calls), - ) - - assert result.ok is True - assert result.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_two_requests_race_one_decision_and_only_first_claimant_sends( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config, worker, decision_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - winner = _answer_request(decision_ref, request_id="decision-winner") - loser = _answer_request(decision_ref, request_id="decision-loser") - calls: list[dict[str, Any]] = [] - raced: dict[str, Any] = {} - real_mark = command_submission._mark_request_send_started - - def race_after_claim(*args: Any, **kwargs: Any) -> Any: - pending = pending_payload_from_store(config.db_path, config.host_id) - assert any( - item["worker_id"] == worker.id - and item["meta"]["decision"]["decision_ref"] == decision_ref - for item in pending["pending_interactions"] - ) - raced["loser"] = submit_command( - config, - loser, - socket_client_factory=_factory(calls), - ) - return real_mark(*args, **kwargs) - - monkeypatch.setattr( - command_submission, - "_mark_request_send_started", - race_after_claim, - ) - first = submit_command( - config, - winner, - socket_client_factory=_factory(calls), - ) - - assert first.ok is True - assert first.status == STATUS_ACCEPTED - assert raced["loser"].ok is False - assert raced["loser"].status == STATUS_ANSWER_IN_PROGRESS - assert raced["loser"].disposition == DISPOSITION_NO_RECEIPT - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_winner_safe_presend_failure_releases_claim_for_loser_retry( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - winner = _answer_request(decision_ref, request_id="decision-safe-failure") - loser = _answer_request(decision_ref, request_id="decision-safe-retry") - calls: list[dict[str, Any]] = [] - raced: dict[str, Any] = {} - real_mark = command_submission._mark_request_send_started - first_mark = True - - def fail_winner_before_send(*args: Any, **kwargs: Any) -> Any: - nonlocal first_mark - if first_mark: - first_mark = False - raced["loser"] = submit_command( - config, - loser, - socket_client_factory=_factory(calls), - ) - return command_submission._request_in_progress(args[1]) - return real_mark(*args, **kwargs) - - monkeypatch.setattr( - command_submission, - "_mark_request_send_started", - fail_winner_before_send, - ) - failed_winner = submit_command( - config, - winner, - socket_client_factory=_factory(calls), - ) - retried_loser = submit_command( - config, - loser, - socket_client_factory=_factory(calls), - ) - - assert failed_winner.status == STATUS_PENDING - assert raced["loser"].status == STATUS_ANSWER_IN_PROGRESS - assert raced["loser"].disposition == DISPOSITION_NO_RECEIPT - assert retried_loser.ok is True - assert retried_loser.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_claim_time_loser_releases_unsent_reservation_for_takeover( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config, worker, decision_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - request = _answer_request(decision_ref, request_id="claim-time-loser") - calls: list[dict[str, Any]] = [] - real_claim = command_submission._claim_pending_decision - injected_claim: dict[str, Any] = {} - - def lose_during_claim( - claim_config: Any, - claim_request: CommandRequest, - validated: Any, - ) -> Any: - if "value" not in injected_claim: - injected_claim["value"] = claim_backend_pending_decision( - config.db_path, - config.host_id, - worker.id, - decision_ref, - {"option_refs": ["2"]}, - claim=True, - ) - assert injected_claim["value"].status == "claimed" - return real_claim(claim_config, claim_request, validated) - - monkeypatch.setattr( - command_submission, - "_claim_pending_decision", - lose_during_claim, - ) - first = submit_command( - config, - request, - socket_client_factory=_factory(calls), - ) - receipt = get_command_request( - config.db_path, - config.host_id, - "claim-time-loser", - ) - - assert first.status == STATUS_ANSWER_IN_PROGRESS - assert first.disposition == DISPOSITION_IN_PROGRESS - assert receipt is not None - assert receipt["state"] == "reserved" - assert receipt["status"] == STATUS_PENDING - assert calls == [] - - assert abandon_backend_pending_choice_claim( - config.db_path, - config.host_id, - injected_claim["value"].claim_token, - ) - retry = submit_command( - config, - request, - socket_client_factory=_factory(calls), - ) - assert retry.ok is True - assert retry.status == STATUS_ACCEPTED - assert len(calls) == 1 - - -def test_abandoned_reservation_is_not_terminalized_by_live_claim( - tmp_path: Path, -) -> None: - config, worker, decision_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - payload = _answer_request(decision_ref, request_id="abandoned-loser") - request = CommandRequest.from_dict(payload) - canonical = build_canonical_mutation(request, public_worker_id=worker.id) - initial = reserve_command_request( - config.db_path, - host_id=config.host_id, - request_id=request.request_id or "", - action=canonical.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=envelope_to_receipt_json( - command_submission._request_in_progress(request) - ), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), - owner_lease_seconds=1, - now="2020-01-01T00:00:00+00:00", - ) - assert initial["status"] == "reserved" - competing = claim_backend_pending_decision( - config.db_path, - config.host_id, - worker.id, - decision_ref, - {"option_refs": ["1"]}, - claim=True, - ) - assert competing.status == "claimed" - - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - payload, - socket_client_factory=_factory(calls), - ) - receipt = get_command_request( - config.db_path, - config.host_id, - request.request_id or "", - ) - - assert result.status == STATUS_ANSWER_IN_PROGRESS - assert result.disposition == DISPOSITION_IN_PROGRESS - assert receipt is not None - assert receipt["state"] == "reserved" - assert receipt["status"] == STATUS_PENDING - assert calls == [] - - -def test_process_loss_before_send_started_is_recovered_after_lease_expiry( - tmp_path: Path, -) -> None: - config, worker, decision_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - payload = _answer_request(decision_ref, request_id="crashed-before-send") - request = CommandRequest.from_dict(payload) - canonical = build_canonical_mutation(request, public_worker_id=worker.id) - reservation = reserve_command_request( - config.db_path, - host_id=config.host_id, - request_id=request.request_id or "", - action=canonical.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=envelope_to_receipt_json( - command_submission._request_in_progress(request) - ), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), - owner_lease_seconds=1, - now="2020-01-01T00:00:00+00:00", - ) - assert reservation["status"] == "reserved" - abandoned_claim = claim_backend_pending_decision( - config.db_path, - config.host_id, - worker.id, - decision_ref, - {"option_refs": ["2"]}, - claim=True, - observed_at="2020-01-01T00:00:00+00:00", - claim_lease_seconds=1, - ) - assert abandoned_claim.status == "claimed" - - calls: list[dict[str, Any]] = [] - recovered = submit_command( - config, - payload, - socket_client_factory=_factory(calls), - ) - - assert recovered.ok is True - assert recovered.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_send_uncertainty_is_durable_and_never_resends( - tmp_path: Path, -) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - request = _answer_request(decision_ref, request_id="uncertain-decision") - calls: list[dict[str, Any]] = [] - - class FailingKeyClient(_FakeSocketClient): - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if method == "pane.send_keys": - raise RuntimeError("response lost after key send") - return {"accepted": True} - - first = submit_command( - config, - request, - socket_client_factory=lambda _config: FailingKeyClient(calls), - ) - replay = submit_command( - config, - request, - socket_client_factory=lambda _config: FailingKeyClient(calls), - ) - - assert first.ok is False - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] diff --git a/tests/test_backend.py b/tests/test_backend.py index 918d691..fb9472f 100644 --- a/tests/test_backend.py +++ b/tests/test_backend.py @@ -606,10 +606,10 @@ def test_herdr_backend_target_precedence_and_pane_fallback(monkeypatch) -> None: assert by_id["pane-public"].backend_target["kind"] == "terminal_id" assert by_id["pane-public"].backend_target["value"] == "term-send" assert by_id["pane-public"].backend_target["sendable"] is True - assert bindings_by_id["public-id"].turn_target_kind == "pane_id" - assert bindings_by_id["public-id"].turn_target_value == "pane-fallback" - assert bindings_by_id["pane-public"].turn_target_kind == "pane_id" - assert bindings_by_id["pane-public"].turn_target_value == "pane-send" + assert bindings_by_id["public-id"].turn_target_kind is None + assert bindings_by_id["public-id"].turn_target_value is None + assert bindings_by_id["pane-public"].turn_target_kind is None + assert bindings_by_id["pane-public"].turn_target_value is None assert all( (worker.backend_target or {}).get("value") != "sess-not-sendable" for worker in workers diff --git a/tests/test_backend_pending.py b/tests/test_backend_pending.py deleted file mode 100644 index ffc9afa..0000000 --- a/tests/test_backend_pending.py +++ /dev/null @@ -1,1560 +0,0 @@ -"""Backend-provided pending prompts: a REAL pane prompt (question + choices, captured by the herdres -pending hook through the turn adapter) flows adapter -> herdr_turns -> backend_pending -> pending.list, -superseding the worker's synthetic attention-derived row.""" -from __future__ import annotations - -import json -import sqlite3 -import pytest -from pathlib import Path - -from tendwire.backends.herdr_turns import _backend_pending_from_turn, _pop_backend_pending -from tendwire.backends.herdr_turns import _pending_observation_from_turn -from tendwire.config import Config, DEFAULT_PENDING_STALE_GRACE_SECONDS, load_config -from tendwire.core.projector import project_from_raw -from tendwire.core.models import Snapshot, WorkerBinding -from tendwire.core.turns import PendingObservation, pending_payload_from_snapshot -from tendwire.daemon import TendwireDaemon -from tendwire.store.sqlite import ( - STORE_SCHEMA_VERSION, - abandon_backend_pending_choice_claim, - apply_backend_pending_observation, - backend_pending_health, - claim_backend_pending_choice, - finish_backend_pending_choice_send, - init_store, - list_backend_pending, - merge_backend_pending, - prune_backend_pending, - pending_payload_from_store, - save_snapshot, - start_backend_pending_choice_send, - upsert_worker_bindings, -) - - -def _decision_turn() -> dict: - return { - "available": True, - "complete": False, - "awaiting_input": True, - "user_text": "which db?", - "pending_decision": { - "decision_id": "toolu_123", - "prompt": "Which database should we use?", - "mode": "buttons", - "options": [ - {"id": "1", "label": "Postgres", "send_text": "Postgres"}, - {"id": "2", "label": "SQLite", "send_text": "SQLite"}, - {"id": "custom", "label": "Tell me differently", "send_text": ""}, - ], - }, - } - - -def test_extract_pending_decision(): - pending = _backend_pending_from_turn(_decision_turn()) - assert pending["question"] == "Which database should we use?" - assert pending["kind"] == "question" - assert [c["label"] for c in pending["choices"]] == ["Postgres", "SQLite", "Tell me differently"] - choice_ids = [c["choice_id"] for c in pending["choices"]] - assert all(choice_id.startswith("choice-") for choice_id in choice_ids) - assert choice_ids == [c["choice_id"] for c in _backend_pending_from_turn(_decision_turn())["choices"]] - assert not ({"1", "2", "custom"} & set(choice_ids)) - # The machine-send payload (send_text) is not published; choices carry only id + label. - assert all("value" not in c for c in pending["choices"]) - # decision_id (internal tool_use_id) is not published in public pending. - assert "decision_id" not in pending["meta"] - decision = pending["meta"]["decision"] - assert decision == { - "decision_ref": decision["decision_ref"], - "kind": "single", - "prompt": "Which database should we use?", - "options": [ - {"ref": "1", "label": "Postgres"}, - {"ref": "2", "label": "SQLite"}, - ], - "multi_select": False, - "question_count": 1, - } - assert decision["decision_ref"].startswith("decision-") - - -def test_extract_plan_approval_kind(): - turn = {"pending_decision": {"decision_id": "t", "prompt": "Approve this plan?", - "options": [{"id": "approve", "label": "Approve", "send_text": "1"}, - {"id": "revise", "label": "Revise", "send_text": ""}]}} - assert _backend_pending_from_turn(turn)["kind"] == "approval" - - -def test_extract_none_without_pending(): - assert _backend_pending_from_turn({"complete": True, "assistant_final_text": "done"}) is None - - -def test_pop_backend_pending_splits(): - content, pending = _pop_backend_pending({"user_text": "hi", "_backend_pending": {"question": "q"}}) - assert content == {"user_text": "hi"} and pending == {"question": "q"} - content, pending = _pop_backend_pending({"_backend_pending": {"question": "q"}}) - assert content is None and pending == {"question": "q"} - - -def test_merge_and_prune_backend_pending(tmp_path: Path): - db = tmp_path / "p.db" - init_store(db) - pending = {"question": "Q?", "kind": "question", "choices": [], "meta": {"source": "backend"}} - assert merge_backend_pending(db, "h", "w1", pending) is True - assert merge_backend_pending(db, "h", "w1", pending) is False # unchanged -> no write - assert list_backend_pending(db, "h") == {"w1": pending} - assert merge_backend_pending(db, "h", "w1", None) is True # answered -> pruned - assert list_backend_pending(db, "h") == {} - - -# --- Regression tests for the PR#3 review fixes --------------------------------------------- - -_GOOGLE_API_KEY_SENTINEL = "AI" + "zaSyD-ExampleKey1234567890abcdefghijk" -_OPENAI_KEY_SENTINEL = "sk-" + "live-SENTINELSECRET123ABC" - -_SENTINELS = [ - _OPENAI_KEY_SENTINEL, # sk- secret token - "/run/user/1000/herdr/sock-abcdef123456", # socket path - "w4V:p1", # pseudo pane id - "/home/example/.ssh/id_rsa", # absolute fs path - "toolu_SENTINELDECISION01", # internal tool_use_id (dropped, not published) - _GOOGLE_API_KEY_SENTINEL, # google api key - "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjMifQ.SflKxwRJSMeKKF2QT4fwpMeJf", # JWT - "alice.jones@internal-corp.example", # email / PII - "10.4.2.9:5432", # internal ip:port - "home/alice/.ssh/id_rsa", # home path without leading slash -] - - -def _leaky_decision_turn() -> dict: - return { - "pending_decision": { - "decision_id": "toolu_SENTINELDECISION01", - "prompt": ( - "Approve running against pane w4V:p1 at /home/example/.ssh/id_rsa via " - f"/run/user/1000/herdr/sock-abcdef123456? Also rotate {_GOOGLE_API_KEY_SENTINEL} " - "and eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjMifQ.SflKxwRJSMeKKF2QT4fwpMeJf; " - "notify alice.jones@internal-corp.example on db 10.4.2.9:5432 at home/alice/.ssh/id_rsa" - ), - "options": [ - { - "id": "approve", - "label": f"Use secret {_OPENAI_KEY_SENTINEL}", - "send_text": _OPENAI_KEY_SENTINEL, - }, - {"id": "postgres", "label": "Postgres", "send_text": "Postgres"}, - {"id": "run", "label": "Deploy to 10.4.2.9:5432", "send_text": "tmux send-keys -t w4V:p1 'rm -rf /home/example/.ssh'"}, - {"id": "shell", "label": "bash -lc 'echo untrusted option'", "send_text": "echo untrusted option"}, - ], - } - } - - -def test_ingestion_redacts_private_data_from_pending(): - """Blocker regression: no private path/pane-id/secret/tool-id survives ingestion.""" - pending = _backend_pending_from_turn(_leaky_decision_turn()) - blob = json.dumps(pending) - for sentinel in _SENTINELS: - assert sentinel not in blob, f"private sentinel leaked into ingested pending: {sentinel!r}" - # A benign label/value is preserved verbatim. - labels = [c["label"] for c in pending["choices"]] - assert "Postgres" in labels - assert "[redacted]" in pending["question"] - assert "bash -lc" not in blob - assert "echo untrusted option" not in blob - - -def test_get_pending_public_json_has_no_private_leak(tmp_path: Path): - """End-to-end: the sentinels must not reach the PUBLIC pending.list payload either.""" - db = tmp_path / "leak.db" - config = Config(host_id="h", db_path=db) - snapshot = project_from_raw(config, workers=[{"id": "worker-1", "name": "claude", "status": "blocked", "space_id": "s1"}]) - init_store(db) - save_snapshot(db, snapshot) - pending = _backend_pending_from_turn(_leaky_decision_turn()) - merge_backend_pending(db, "h", "worker-1", pending) - - public = json.dumps(TendwireDaemon(config).get_pending()) - for sentinel in _SENTINELS: - assert sentinel not in public, f"private sentinel leaked into public pending.list: {sentinel!r}" - # The benign choice survives; the raw-command choice value is dropped by _public_choice_value. - assert "Postgres" in public - - -def test_get_pending_recomputes_content_fingerprint_and_shows_choices(tmp_path: Path): - db = tmp_path / "fp.db" - config = Config(host_id="h", db_path=db) - snapshot = project_from_raw(config, workers=[{"id": "worker-1", "name": "claude", "status": "blocked", "space_id": "s1"}]) - init_store(db) - save_snapshot(db, snapshot) - baseline_fp = pending_payload_from_snapshot(snapshot)["content_fingerprint"] - - merge_backend_pending(db, "h", "worker-1", _backend_pending_from_turn(_decision_turn())) - payload = TendwireDaemon(config).get_pending() - - # The overlaid list moves the change-token (was left stale before the fix). - assert payload["content_fingerprint"] != baseline_fp - interactions = [p for p in payload["pending_interactions"] if p["worker_id"] == "worker-1"] - assert interactions and interactions[0]["question"] == "Which database should we use?" - assert [c["label"] for c in interactions[0]["choices"]] == ["Postgres", "SQLite", "Tell me differently"] - - -def test_prune_backend_pending_reaps_orphaned_workers(tmp_path: Path): - db = tmp_path / "orphan.db" - init_store(db) - live = PendingObservation( - "open_prompt", - question="Q?", - pending_kind="question", - revision_digest="live-revision", - ) - gone = PendingObservation( - "open_prompt", - question="Q?", - pending_kind="question", - revision_digest="gone-revision", - ) - apply_backend_pending_observation( - db, "h", "worker-live", live, binding_private_fingerprint="binding-live" - ) - apply_backend_pending_observation( - db, "h", "worker-gone", gone, binding_private_fingerprint="binding-gone" - ) - assert set(list_backend_pending(db, "h")) == {"worker-live", "worker-gone"} - - reaped = prune_backend_pending(db, "h", {"binding-live"}) - assert reaped == 1 - assert set(list_backend_pending(db, "h")) == {"worker-live"} - - -def _pending_fixture(tmp_path: Path) -> tuple[Path, Config, object]: - db = tmp_path / "pending-v10.db" - config = Config(host_id="h", db_path=db) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-1", - "name": "pane worker", - "status": "blocked", - "space_id": "space-1", - } - ], - ) - init_store(db) - save_snapshot(db, snapshot) - return db, config, snapshot - - -def test_pending_stale_grace_config_default_env_and_validation(monkeypatch) -> None: - monkeypatch.delenv("TENDWIRE_PENDING_STALE_GRACE_SECONDS", raising=False) - assert DEFAULT_PENDING_STALE_GRACE_SECONDS == 30.0 - assert load_config().pending_stale_grace_seconds == 30.0 - monkeypatch.setenv("TENDWIRE_PENDING_STALE_GRACE_SECONDS", "12.5") - assert load_config().pending_stale_grace_seconds == 12.5 - assert load_config(pending_stale_grace_seconds="4").pending_stale_grace_seconds == 4 - for invalid in (0, -1, "nan", "inf"): - try: - Config(pending_stale_grace_seconds=invalid) - except ValueError as exc: - assert "pending_stale_grace_seconds must be a finite positive number" in str(exc) - else: - raise AssertionError(f"accepted invalid grace: {invalid!r}") - - -def test_explicit_pending_transition_table_tombstone_stale_and_expiry( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - opened = _pending_observation_from_turn(_decision_turn()) - t0 = "2026-07-13T00:00:00+00:00" - t1 = "2026-07-13T00:00:01+00:00" - t2 = "2026-07-13T00:00:02+00:00" - t20 = "2026-07-13T00:00:20+00:00" - t32 = "2026-07-13T00:00:32+00:00" - - assert apply_backend_pending_observation(db, "h", "worker-1", opened, observed_at=t0) - fresh = pending_payload_from_store(db, "h") - backend_row = next( - row for row in fresh["pending_interactions"] if row["worker_id"] == "worker-1" - ) - assert backend_row["question"] == "Which database should we use?" - assert backend_row["meta"]["freshness"] == "fresh" - fresh_fingerprint = fresh["content_fingerprint"] - - assert not apply_backend_pending_observation( - db, "h", "worker-1", opened, observed_at=t1 - ) - assert pending_payload_from_store(db, "h")["content_fingerprint"] == fresh_fingerprint - with sqlite3.connect(db) as conn: - assert conn.execute( - "SELECT COUNT(*) FROM backend_pending WHERE host_id = 'h'" - ).fetchone()[0] == 1 - assert conn.execute( - "SELECT COUNT(*) FROM connector_outbox" - ).fetchone()[0] == 0 - - assert apply_backend_pending_observation( - db, "h", "worker-1", PendingObservation("read_failed"), observed_at=t2 - ) - stale = pending_payload_from_store(db, "h") - stale_row = next( - row for row in stale["pending_interactions"] if row["worker_id"] == "worker-1" - ) - assert stale_row["meta"]["freshness"] == "stale" - assert stale["pending_health"] == { - "status": "degraded", - "counts": {"fresh": 0, "stale": 1, "total": 1}, - } - assert stale["content_fingerprint"] != fresh_fingerprint - with sqlite3.connect(db) as conn: - deadline = conn.execute( - "SELECT grace_deadline FROM backend_pending WHERE worker_id = 'worker-1'" - ).fetchone()[0] - - assert not apply_backend_pending_observation( - db, "h", "worker-1", PendingObservation("read_failed"), observed_at=t20 - ) - with sqlite3.connect(db) as conn: - assert conn.execute( - "SELECT grace_deadline FROM backend_pending WHERE worker_id = 'worker-1'" - ).fetchone()[0] == deadline - - assert apply_backend_pending_observation( - db, "h", "worker-1", PendingObservation("read_failed"), observed_at=t32 - ) - expired = pending_payload_from_store(db, "h") - assert expired["pending_health"] == { - "status": "degraded", - "counts": {"fresh": 0, "stale": 1, "total": 1}, - } - assert expired["content_fingerprint"] != stale["content_fingerprint"] - assert any( - row["worker_id"] == "worker-1" - for row in expired["pending_interactions"] - ) - - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("read_succeeded_no_prompt"), - observed_at="2026-07-13T00:01:00+00:00", - ) - tombstoned = pending_payload_from_store(db, "h") - assert not any( - row["worker_id"] == "worker-1" - for row in tombstoned["pending_interactions"] - ) - assert tombstoned["pending_health"] == { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - } - assert _pending_observation_from_turn( - {"pending_decision": {"prompt": "Broken?", "options": "not-a-list"}} - ).kind == "read_succeeded_invalid_prompt" - with sqlite3.connect(db) as conn: - assert conn.execute( - "SELECT observation_state, freshness FROM backend_pending" - ).fetchone() == ("none", "fresh") - - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("worker_authoritatively_absent"), - observed_at="2026-07-13T00:01:01+00:00", - ) - assert backend_pending_health(db, "h")["counts"]["total"] == 0 - - -def test_malformed_source_prompt_exposes_snapshot_fallback_and_health( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - opened = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - opened, - observed_at="2026-07-13T00:00:00+00:00", - ) - malformed = _pending_observation_from_turn( - {"pending_decision": {"prompt": "Broken?", "options": "not-a-list"}} - ) - assert malformed.kind == "read_succeeded_invalid_prompt" - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - malformed, - observed_at="2026-07-13T00:00:01+00:00", - ) - fallback = pending_payload_from_store(db, "h") - assert any( - item["worker_id"] == "worker-1" - for item in fallback["pending_interactions"] - ) - assert all( - item["question"] != "Which database should we use?" - for item in fallback["pending_interactions"] - ) - assert fallback["pending_health"] == { - "status": "degraded", - "counts": {"fresh": 0, "stale": 1, "total": 1}, - } - fingerprint = fallback["content_fingerprint"] - assert not apply_backend_pending_observation( - db, - "h", - "worker-1", - malformed, - observed_at="2026-07-13T00:00:02+00:00", - ) - assert ( - pending_payload_from_store(db, "h")["content_fingerprint"] - == fingerprint - ) - with sqlite3.connect(db) as conn: - assert conn.execute( - "SELECT observation_state, freshness FROM backend_pending" - ).fetchone() == ("invalid", "stale") - - -def test_initial_read_failure_remains_degraded_until_success( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - failure = PendingObservation("read_failed") - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - failure, - observed_at="2026-07-13T00:00:00+00:00", - ) - failed = pending_payload_from_store(db, "h") - assert failed["pending_health"] == { - "status": "degraded", - "counts": {"fresh": 0, "stale": 1, "total": 1}, - } - assert any( - item["worker_id"] == "worker-1" - for item in failed["pending_interactions"] - ) - fingerprint = failed["content_fingerprint"] - assert not apply_backend_pending_observation( - db, - "h", - "worker-1", - failure, - observed_at="2026-07-13T00:00:01+00:00", - ) - assert ( - pending_payload_from_store(db, "h")["content_fingerprint"] - == fingerprint - ) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("read_succeeded_no_prompt"), - observed_at="2026-07-13T00:00:02+00:00", - ) - recovered = pending_payload_from_store(db, "h") - assert recovered["pending_health"] == { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - } - assert not any( - item["worker_id"] == "worker-1" - for item in recovered["pending_interactions"] - ) - - -def test_older_pending_observations_cannot_regress_newer_state( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - opened = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - opened, - observed_at="2026-07-13T00:00:10+00:00", - ) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("read_succeeded_no_prompt"), - observed_at="2026-07-13T00:00:20+00:00", - ) - assert not apply_backend_pending_observation( - db, - "h", - "worker-1", - opened, - observed_at="2026-07-13T00:00:15+00:00", - ) - assert not any( - item["worker_id"] == "worker-1" - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - ) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - opened, - observed_at="2026-07-13T00:00:30+00:00", - ) - assert not apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("read_failed"), - observed_at="2026-07-13T00:00:25+00:00", - ) - current = pending_payload_from_store(db, "h") - row = next( - item - for item in current["pending_interactions"] - if item["worker_id"] == "worker-1" - ) - assert row["question"] == "Which database should we use?" - assert row["meta"]["freshness"] == "fresh" - - -def test_revision_bound_opaque_handles_and_two_phase_claim(tmp_path: Path) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="pane_id", - turn_target_value="pane-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private", - ) - upsert_worker_bindings(db, [binding]) - first = _pending_observation_from_turn(_decision_turn()) - changed_turn = _decision_turn() - changed_turn["pending_decision"]["decision_id"] = "toolu_changed_private" - second = _pending_observation_from_turn(changed_turn) - assert [choice.choice_id for choice in first.choices] != [ - choice.choice_id for choice in second.choices - ] - public_blob = json.dumps(_backend_pending_from_turn(changed_turn)) - assert "toolu_changed_private" not in public_blob - assert "send_text" not in public_blob - - assert apply_backend_pending_observation( - db, - "h", - worker.id, - first, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - projected = pending_payload_from_store(db, "h") - row = next(item for item in projected["pending_interactions"] if item["worker_id"] == worker.id) - choice_id = row["choices"][1]["choice_id"] - dry = claim_backend_pending_choice( - db, "h", row["id"], row["fingerprint"], choice_id, - claim=False, observed_at="2026-07-13T00:00:01+00:00", - ) - assert dry.status == "validated" - assert dry.claim_token is None - assert dry.picker_ordinal == 2 - claimed = claim_backend_pending_choice( - db, "h", row["id"], row["fingerprint"], choice_id, - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claimed.status == "claimed" - assert claimed.binding_private_fingerprint == "binding-private" - assert claimed.turn_target_value == "pane-private" - assert claimed.picker_ordinal == 2 - started = start_backend_pending_choice_send( - db, "h", claimed.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ) - assert started.status == "started" - assert started.turn_target_value == "pane-private" - assert started.picker_ordinal == 2 - assert abandon_backend_pending_choice_claim(db, "h", claimed.claim_token) is False - assert finish_backend_pending_choice_send( - db, "h", claimed.claim_token, accepted=False - ) is False - with sqlite3.connect(db) as conn: - assert conn.execute( - """ - SELECT state FROM backend_pending_claims - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ("send_started",) - assert conn.execute( - """ - SELECT observation_state FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ("open",) - assert finish_backend_pending_choice_send( - db, "h", claimed.claim_token, accepted=True - ) is True - assert list_backend_pending(db, "h") == {} - - -def test_accepted_answer_immediately_tombstones_snapshot_fallback( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - fallback = pending_payload_from_store(db, "h") - assert any( - item["worker_id"] == worker.id - for item in fallback["pending_interactions"] - ) - - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private-answer", - turn_target_kind="pane_id", - turn_target_value="pane-private-answer", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private-answer", - ) - upsert_worker_bindings(db, [binding]) - first = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - first, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - overlaid_payload = pending_payload_from_store(db, "h") - overlaid = next( - item - for item in overlaid_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert overlaid["question"] == "Which database should we use?" - assert binding.private_fingerprint not in json.dumps(overlaid_payload) - assert binding.turn_target_value not in json.dumps(overlaid_payload) - - claim = claim_backend_pending_choice( - db, - "h", - overlaid["id"], - overlaid["fingerprint"], - overlaid["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claim.status == "claimed" - assert start_backend_pending_choice_send( - db, - "h", - claim.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "started" - assert finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=True, - observed_at="2026-07-13T00:00:03Z", - ) - - answered = pending_payload_from_store(db, "h") - assert not any( - item["worker_id"] == worker.id - for item in answered["pending_interactions"] - ) - answered_blob = json.dumps(answered) - assert binding.private_fingerprint not in answered_blob - assert binding.turn_target_value not in answered_blob - with sqlite3.connect(db) as conn: - tombstone = conn.execute( - """ - SELECT payload_json, revision_digest, choice_routes_json, - binding_private_fingerprint, observed_turn_target_value, - observation_state, freshness, observed_at, last_success_at, - last_failure_at, grace_deadline, updated_at - FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() - assert tombstone == ( - "{}", - "", - "{}", - binding.private_fingerprint, - binding.turn_target_value, - "none", - "fresh", - "2026-07-13T00:00:03+00:00", - "2026-07-13T00:00:03+00:00", - None, - None, - "2026-07-13T00:00:03+00:00", - ) - assert conn.execute( - """ - SELECT COUNT(*) FROM backend_pending_claims - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == (0,) - - assert not finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=True, - observed_at="2026-07-13T00:00:04+00:00", - ) - with sqlite3.connect(db) as conn: - assert conn.execute( - """ - SELECT observation_state, freshness, observed_at - FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ( - "none", - "fresh", - "2026-07-13T00:00:03+00:00", - ) - - later_turn = _decision_turn() - later_turn["pending_decision"]["decision_id"] = "private-later-answer" - assert apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(later_turn), - observed_at="2026-07-13T00:00:05+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - reopened_payload = pending_payload_from_store(db, "h") - reopened = next( - item - for item in reopened_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert reopened["id"] != overlaid["id"] - assert reopened["fingerprint"] != overlaid["fingerprint"] - assert ( - reopened["choices"][0]["choice_id"] - != overlaid["choices"][0]["choice_id"] - ) - reopened_blob = json.dumps(reopened_payload) - assert binding.private_fingerprint not in reopened_blob - assert binding.turn_target_value not in reopened_blob - - -def test_accepted_finish_tombstones_exact_prompt_after_stale_expiry( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private-expiry", - turn_target_kind="pane_id", - turn_target_value="pane-private-expiry", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private-expiry", - ) - upsert_worker_bindings(db, [binding]) - observation = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - observation, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - pending = next( - item - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - if item["worker_id"] == worker.id - ) - claim = claim_backend_pending_choice( - db, - "h", - pending["id"], - pending["fingerprint"], - pending["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claim.status == "claimed" - assert start_backend_pending_choice_send( - db, - "h", - claim.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "started" - assert apply_backend_pending_observation( - db, - "h", - worker.id, - PendingObservation("read_failed"), - observed_at="2026-07-13T00:00:03+00:00", - stale_grace_seconds=1, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - PendingObservation("read_failed"), - observed_at="2026-07-13T00:00:04+00:00", - stale_grace_seconds=1, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - with sqlite3.connect(db) as conn: - failed_row = conn.execute( - """ - SELECT observation_state, freshness, revision_digest - FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() - assert failed_row[:2] == ("failed", "stale") - assert failed_row[2] - assert conn.execute( - """ - SELECT state FROM backend_pending_claims - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ("send_started",) - - assert finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=True, - observed_at="2026-07-13T00:00:05+00:00", - ) - answered = pending_payload_from_store(db, "h") - assert not any( - item["worker_id"] == worker.id - for item in answered["pending_interactions"] - ) - with sqlite3.connect(db) as conn: - assert conn.execute( - """ - SELECT observation_state, freshness, - binding_private_fingerprint, observed_turn_target_value - FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ( - "none", - "fresh", - binding.private_fingerprint, - binding.turn_target_value, - ) - assert conn.execute( - """ - SELECT COUNT(*) FROM backend_pending_claims - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == (0,) - - -def test_newer_prompt_racing_accepted_finish_is_not_erased( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private-race", - turn_target_kind="pane_id", - turn_target_value="pane-private-race", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private-race", - ) - upsert_worker_bindings(db, [binding]) - first = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - first, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - initial = next( - item - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - if item["worker_id"] == worker.id - ) - claim = claim_backend_pending_choice( - db, - "h", - initial["id"], - initial["fingerprint"], - initial["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claim.status == "claimed" - assert start_backend_pending_choice_send( - db, - "h", - claim.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "started" - - newer_turn = _decision_turn() - newer_turn["pending_decision"]["decision_id"] = "private-racing-revision" - assert apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(newer_turn), - observed_at="2026-07-13T00:00:03+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - newer_before_finish = next( - item - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - if item["worker_id"] == worker.id - ) - assert newer_before_finish["id"] != initial["id"] - assert not finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=True, - observed_at="2026-07-13T00:00:04+00:00", - ) - newer_after_finish_payload = pending_payload_from_store(db, "h") - newer_after_finish = next( - item - for item in newer_after_finish_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert newer_after_finish == newer_before_finish - private_blob = json.dumps(newer_after_finish_payload) - assert binding.private_fingerprint not in private_blob - assert binding.turn_target_value not in private_blob - - -def test_identical_prompt_on_new_source_mints_new_public_handles( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="pane_id", - turn_target_value="pane-a-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="stable-binding-private", - ) - upsert_worker_bindings(db, [binding]) - observation = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - observation, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - before = next( - item - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - if item["worker_id"] == worker.id - ) - with sqlite3.connect(db) as conn: - conn.execute( - """ - UPDATE worker_bindings - SET turn_target_value = 'pane-b-private' - WHERE private_fingerprint = ? - """, - (binding.private_fingerprint,), - ) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - observation, - observed_at="2026-07-13T00:00:01+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value="pane-b-private", - ) - after_payload = pending_payload_from_store(db, "h") - after = next( - item - for item in after_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert after["id"] != before["id"] - assert after["fingerprint"] != before["fingerprint"] - assert after["choices"][0]["choice_id"] != before["choices"][0]["choice_id"] - assert "pane-a-private" not in json.dumps(after_payload) - assert "pane-b-private" not in json.dumps(after_payload) - assert claim_backend_pending_choice( - db, - "h", - before["id"], - before["fingerprint"], - before["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:02+00:00", - ).status == "not_found" - claimed = claim_backend_pending_choice( - db, - "h", - after["id"], - after["fingerprint"], - after["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:02+00:00", - ) - assert claimed.status == "claimed" - assert claimed.turn_target_value == "pane-b-private" - - -def test_claim_and_authoritative_prune_are_bound_to_exact_source_pane( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - source = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-source", - turn_target_kind="pane_id", - turn_target_value="pane-source-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="z-source-binding", - ) - decoy = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-decoy", - turn_target_kind="pane_id", - turn_target_value="pane-decoy-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="a-decoy-binding", - ) - upsert_worker_bindings(db, [decoy, source]) - apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(_decision_turn()), - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=source.private_fingerprint, - observed_turn_target_value=source.turn_target_value, - ) - payload = pending_payload_from_store(db, "h") - row = next( - item for item in payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - with sqlite3.connect(db) as conn: - conn.execute( - """ - UPDATE worker_bindings - SET turn_target_value = 'pane-moved-private' - WHERE private_fingerprint = ? - """, - (source.private_fingerprint,), - ) - assert claim_backend_pending_choice( - db, - "h", - row["id"], - row["fingerprint"], - row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:00.500000+00:00", - ).status == "not_found" - with sqlite3.connect(db) as conn: - conn.execute( - """ - UPDATE worker_bindings - SET turn_target_value = ? - WHERE private_fingerprint = ? - """, - (source.turn_target_value, source.private_fingerprint), - ) - claimed = claim_backend_pending_choice( - db, - "h", - row["id"], - row["fingerprint"], - row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claimed.status == "claimed" - assert claimed.binding_private_fingerprint == source.private_fingerprint - assert claimed.turn_target_value == source.turn_target_value - assert source.turn_target_value not in json.dumps(payload) - with sqlite3.connect(db) as conn: - conn.execute( - """ - UPDATE worker_bindings - SET turn_target_value = 'pane-moved-private' - WHERE private_fingerprint = ? - """, - (source.private_fingerprint,), - ) - assert start_backend_pending_choice_send( - db, - "h", - claimed.claim_token, - observed_at="2026-07-13T00:00:01.500000+00:00", - ).status == "binding_changed" - - assert prune_backend_pending(db, "h", {decoy.private_fingerprint}) == 1 - assert list_backend_pending(db, "h") == {} - assert start_backend_pending_choice_send( - db, - "h", - claimed.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "not_found" - - -@pytest.mark.parametrize("terminal_status", ["closed", "failed"]) -def test_start_send_rejects_latest_terminal_worker_status( - tmp_path: Path, - terminal_status: str, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="pane_id", - turn_target_value="pane-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="terminal-binding", - ) - upsert_worker_bindings(db, [binding]) - apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(_decision_turn()), - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - payload = pending_payload_from_store(db, "h") - row = next( - item for item in payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - claimed = claim_backend_pending_choice( - db, - "h", - row["id"], - row["fingerprint"], - row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claimed.status == "claimed" - terminal = project_from_raw( - config, - workers=[ - { - "id": worker.id, - "name": worker.name, - "status": terminal_status, - "space_id": worker.space_id, - } - ], - ) - terminal_payload = terminal.to_dict() - terminal_payload["workers"][0]["fingerprint"] = worker.fingerprint - terminal = Snapshot.from_dict(terminal_payload) - assert terminal.workers[0].fingerprint == worker.fingerprint - save_snapshot(db, terminal) - assert start_backend_pending_choice_send( - db, - "h", - claimed.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "binding_changed" - - -def test_presend_claim_lease_reclaims_only_unstarted_owner(tmp_path: Path) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - upsert_worker_bindings( - db, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent", - turn_target_kind="pane_id", - turn_target_value="pane", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="lease-binding", - ) - ], - ) - observation = _pending_observation_from_turn(_decision_turn()) - apply_backend_pending_observation( - db, - "h", - worker.id, - observation, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint="lease-binding", - observed_turn_target_value="pane", - ) - projected = pending_payload_from_store(db, "h") - row = next(item for item in projected["pending_interactions"] if item["worker_id"] == worker.id) - args = (db, "h", row["id"], row["fingerprint"], row["choices"][0]["choice_id"]) - old = claim_backend_pending_choice( - *args, - observed_at="2026-07-13T00:00:01+00:00", - claim_lease_seconds=30, - ) - assert old.status == "claimed" - assert claim_backend_pending_choice( - *args, - observed_at="2026-07-13T00:00:30+00:00", - claim_lease_seconds=30, - ).status == "already_claimed" - replacement = claim_backend_pending_choice( - *args, - observed_at="2026-07-13T00:00:31+00:00", - claim_lease_seconds=30, - ) - assert replacement.status == "claimed" - assert replacement.claim_token != old.claim_token - assert start_backend_pending_choice_send( - db, - "h", - old.claim_token, - observed_at="2026-07-13T00:00:31+00:00", - claim_lease_seconds=30, - ).status == "not_found" - assert start_backend_pending_choice_send( - db, - "h", - replacement.claim_token, - observed_at="2026-07-13T00:00:32+00:00", - claim_lease_seconds=30, - ).status == "started" - assert claim_backend_pending_choice( - *args, - observed_at="2026-07-14T00:00:00+00:00", - claim_lease_seconds=1, - ).status == "already_claimed" - - -def test_new_revision_retires_uncertain_claim_and_malformed_overlay_falls_back( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - upsert_worker_bindings( - db, - [ - WorkerBinding( - host_id="h", - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent", - turn_target_kind="pane_id", - turn_target_value="pane", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding", - ) - ], - ) - first = _pending_observation_from_turn(_decision_turn()) - apply_backend_pending_observation( - db, - "h", - worker.id, - first, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint="binding", - observed_turn_target_value="pane", - ) - payload = pending_payload_from_store(db, "h") - row = next(item for item in payload["pending_interactions"] if item["worker_id"] == worker.id) - claim = claim_backend_pending_choice( - db, "h", row["id"], row["fingerprint"], row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claim.status == "claimed" - assert start_backend_pending_choice_send( - db, - "h", - claim.claim_token, - observed_at="2026-07-13T00:00:01.500000+00:00", - ).status == "started" - assert not finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=False, - ) - changed = _decision_turn() - changed["pending_decision"]["decision_id"] = "private-new-revision" - apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(changed), - observed_at="2026-07-13T00:00:02+00:00", - binding_private_fingerprint="binding", - observed_turn_target_value="pane", - ) - assert start_backend_pending_choice_send( - db, "h", claim.claim_token, - observed_at="2026-07-13T00:00:03+00:00", - ).status == "not_found" - changed_payload = pending_payload_from_store(db, "h") - changed_row = next( - item - for item in changed_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - replacement = claim_backend_pending_choice( - db, - "h", - changed_row["id"], - changed_row["fingerprint"], - changed_row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:03.500000+00:00", - ) - assert replacement.status == "claimed" - - with sqlite3.connect(db) as conn: - conn.execute( - "UPDATE backend_pending SET payload_json = ? WHERE worker_id = ?", - ('{"question":"broken","kind":"question","choices":"bad"}', worker.id), - ) - fallback = pending_payload_from_store(db, "h") - assert any(item["worker_id"] == worker.id for item in fallback["pending_interactions"]) - assert all(item["question"] != "broken" for item in fallback["pending_interactions"]) - - -def test_cached_pending_projection_never_migrates_older_store( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - with sqlite3.connect(db) as conn: - conn.execute("PRAGMA user_version = 9") - result = pending_payload_from_store(db, "h") - assert result["status"] == "store_unavailable" - assert result["ok"] is False - with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == 9 - - -def test_current_schema_creation_has_exact_binding_and_claim_state(tmp_path: Path) -> None: - db = tmp_path / "current-schema.db" - init_store(db) - with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 28 - columns = { - str(row[1]) - for row in conn.execute("PRAGMA table_info(backend_pending)").fetchall() - } - assert { - "revision_digest", - "choice_routes_json", - "binding_private_fingerprint", - "observed_turn_target_value", - "observation_state", - "freshness", - "last_success_at", - "last_failure_at", - "grace_deadline", - "updated_at", - } <= columns - claim_columns = { - str(row[1]) - for row in conn.execute( - "PRAGMA table_info(backend_pending_claims)" - ).fetchall() - } - assert { - "binding_private_fingerprint", - "turn_target_value", - "state", - "send_started_at", - } <= claim_columns - assert "backend_pending_claims" in { - str(row[0]) - for row in conn.execute( - "SELECT name FROM sqlite_master WHERE type = 'table'" - ).fetchall() - } - - -def test_v9_pending_migration_preserves_public_row_but_leaves_it_unrouted( - tmp_path: Path, -) -> None: - db = tmp_path / "legacy-v9.db" - init_store(db) - config = Config(host_id="h", db_path=db) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-legacy", - "name": "legacy worker", - "status": "blocked", - } - ], - ) - save_snapshot(db, snapshot) - legacy_payload = json.dumps( - { - "question": "Legacy approval?", - "kind": "approval", - "choices": [{"choice_id": "choice-0123456789abcdef01234567", "label": "Approve"}], - "meta": {"source": "backend"}, - }, - sort_keys=True, - separators=(",", ":"), - ) - with sqlite3.connect(db) as conn: - conn.execute("DROP TABLE backend_pending_claims") - conn.execute("ALTER TABLE backend_pending RENAME TO backend_pending_v10") - conn.execute( - """ - CREATE TABLE backend_pending ( - host_id TEXT NOT NULL, - worker_id TEXT NOT NULL, - payload_json TEXT NOT NULL, - observed_at TEXT NOT NULL, - PRIMARY KEY (host_id, worker_id) - ) - """ - ) - conn.execute( - "INSERT INTO backend_pending VALUES (?, ?, ?, ?)", - ("h", "worker-legacy", legacy_payload, "2026-07-13T00:00:00+00:00"), - ) - conn.execute("DROP TABLE backend_pending_v10") - conn.execute("PRAGMA user_version = 9") - init_store(db) - with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION - migrated = conn.execute( - """ - SELECT payload_json, observation_state, freshness, - choice_routes_json, binding_private_fingerprint, - observed_turn_target_value, last_success_at, updated_at - FROM backend_pending - """ - ).fetchone() - assert migrated[0] == legacy_payload - assert migrated[1:6] == ("open", "fresh", "{}", "", "") - assert migrated[6:] == ( - "2026-07-13T00:00:00+00:00", - "2026-07-13T00:00:00+00:00", - ) - projected = pending_payload_from_store(db, "h") - legacy = next( - row - for row in projected["pending_interactions"] - if row["worker_id"] == "worker-legacy" - ) - assert legacy["question"] == "Legacy approval?" - assert legacy["choices"][0]["label"] == "Approve" diff --git a/tests/test_cli.py b/tests/test_cli.py index 4575c21..d433c23 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -19,7 +19,7 @@ from tendwire.backends import herdr_cli from tendwire.cli import _build_parser, main, observe_public_snapshot -from tendwire.config import Config +from tendwire.config import DEFAULT_TURN_MODEL, Config from tendwire.core.models import AttentionSignal, Snapshot, SuggestedAction, Worker, WorkerBinding from tendwire.core.projector import project_from_raw from tendwire.daemon_api import TendwireDaemonAPI, UnixSocketJSONServer @@ -420,7 +420,7 @@ def test_cli_turns_parser_defaults_bounds_and_exclusive_positions() -> None: ) -def test_cli_turns_definite_unavailable_refreshes_once_then_reads_exact_page( +def test_cli_turns_definite_unavailable_reads_durable_page_without_refresh( tmp_path: Path, capsys, monkeypatch, @@ -460,7 +460,9 @@ def forbidden_snapshot(*_args: Any, **_kwargs: Any) -> Any: db_path = tmp_path / "fallback.db" monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", UnavailableClient) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", refresh) + monkeypatch.setattr( + "tendwire.cli.refresh_structured_turn_content", refresh, raising=False + ) monkeypatch.setattr("tendwire.cli.turns_payload_from_store", read_page) monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden_snapshot) @@ -491,13 +493,7 @@ def forbidden_snapshot(*_args: Any, **_kwargs: Any) -> Any: {"schema_version": 2, "limit": 7, "cursor": None, "since": None}, ) ] - assert refresh_calls == [ - { - "adapter_timeout_seconds": 0.75, - "max_workers": 4, - "total_timeout_seconds": 1.75, - } - ] + assert refresh_calls == [] assert store_calls == [ ( db_path, @@ -507,8 +503,7 @@ def forbidden_snapshot(*_args: Any, **_kwargs: Any) -> Any: "limit": 7, "cursor": None, "since": None, - "turn_refresh_interval_seconds": 2.0, - "turn_model": os.environ.get("TENDWIRE_TURN_MODEL", "observed"), + "turn_model": DEFAULT_TURN_MODEL, }, ) ] @@ -555,7 +550,11 @@ def read_page(_db_path: Path, host_id: str, **kwargs: Any) -> dict[str, Any]: } monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", UnavailableClient) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden_refresh) + monkeypatch.setattr( + "tendwire.cli.refresh_structured_turn_content", + forbidden_refresh, + raising=False, + ) monkeypatch.setattr("tendwire.cli.turns_payload_from_store", read_page) code = main( @@ -584,8 +583,7 @@ def read_page(_db_path: Path, host_id: str, **kwargs: Any) -> dict[str, Any]: "limit": 9, "cursor": position_value if position_flag == "--cursor" else None, "since": position_value if position_flag == "--since" else None, - "turn_refresh_interval_seconds": 2.0, - "turn_model": os.environ.get("TENDWIRE_TURN_MODEL", "observed"), + "turn_model": DEFAULT_TURN_MODEL, } ] @@ -685,7 +683,9 @@ def forbidden_read(*_args: Any, **_kwargs: Any) -> Any: raise AssertionError("ambiguous/reachable failures must not read any source") monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FailingClient) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden_read) + monkeypatch.setattr( + "tendwire.cli.refresh_structured_turn_content", forbidden_read, raising=False + ) monkeypatch.setattr("tendwire.cli.turns_payload_from_store", forbidden_read) monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden_read) monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden_read) @@ -740,7 +740,9 @@ def forbidden_read(*_args: Any, **_kwargs: Any) -> Any: raise AssertionError("reachable page result must be authoritative") monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", AuthoritativeClient) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden_read) + monkeypatch.setattr( + "tendwire.cli.refresh_structured_turn_content", forbidden_read, raising=False + ) monkeypatch.setattr("tendwire.cli.turns_payload_from_store", forbidden_read) monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden_read) @@ -1101,6 +1103,7 @@ def test_cli_long_content_pages_match_direct_store_and_daemon( monkeypatch.setattr( "tendwire.cli.refresh_structured_turn_content", lambda _config, **_kwargs: {"ok": True}, + raising=False, ) v1_code = main( @@ -1289,6 +1292,7 @@ def test_cli_short_v1_compatibility_then_known_incomplete_refusal( monkeypatch.setattr( "tendwire.cli.refresh_structured_turn_content", lambda _config, **_kwargs: {"ok": True}, + raising=False, ) common = [ "--host-id", @@ -1382,7 +1386,9 @@ def forbidden(*_args: Any, **_kwargs: Any) -> Any: monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden) + monkeypatch.setattr( + "tendwire.cli.refresh_structured_turn_content", forbidden, raising=False + ) code = main( [ @@ -1464,7 +1470,9 @@ def forbidden(*_args: Any, **_kwargs: Any) -> Any: monkeypatch.setattr("tendwire.cli.pending_payload_from_store", forbidden) monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden) + monkeypatch.setattr( + "tendwire.cli.refresh_structured_turn_content", forbidden, raising=False + ) code = main( [ @@ -1533,7 +1541,9 @@ def forbidden(*_args: Any, **_kwargs: Any) -> Any: monkeypatch.setattr("tendwire.cli.pending_payload_from_store", forbidden) monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden) + monkeypatch.setattr( + "tendwire.cli.refresh_structured_turn_content", forbidden, raising=False + ) code = main( [ @@ -1900,7 +1910,9 @@ def forbidden(*_args: Any, **_kwargs: Any) -> Any: monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", UnavailableClient) monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden) + monkeypatch.setattr( + "tendwire.cli.refresh_structured_turn_content", forbidden, raising=False + ) code = main( [ @@ -2141,7 +2153,7 @@ def _capture_save( assert captured_atomic == [ ([], "herdr", health.status == "healthy", bool(workers)) ] - assert captured_turn_models == [config.turn_model] + assert captured_turn_models == [DEFAULT_TURN_MODEL] def test_rejected_stale_snapshot_does_not_persist_stale_worker_bindings( @@ -2320,7 +2332,7 @@ def _capture_save( assert len(captured) == 1 assert captured[0].authority == "none" assert captured_atomic == [([], "herdr")] - assert captured_turn_models == [config.turn_model] + assert captured_turn_models == [DEFAULT_TURN_MODEL] def test_cli_attention_json_reads_store_backed_lifecycle( @@ -2691,13 +2703,7 @@ def test_cli_module_invocation() -> None: env=env, ) assert result.returncode == 0, result.stderr - turn_model = env.get("TENDWIRE_TURN_MODEL", "observed").strip().lower() - expected_stderr = ( - "" - if turn_model == "observed" - else f"turn_model={turn_model} is a compatibility alias and behaves as observed\n" - ) - assert result.stderr == expected_stderr + assert result.stderr == "" payload = json.loads(result.stdout) assert payload["schema_version"] == 2 assert payload["host_id"] == "module-host" diff --git a/tests/test_cli_command.py b/tests/test_cli_command.py index 86bff4e..4e9535a 100644 --- a/tests/test_cli_command.py +++ b/tests/test_cli_command.py @@ -388,7 +388,6 @@ def forbidden(*args: Any, **kwargs: Any) -> Any: monkeypatch.setattr("tendwire.cli._try_daemon_attempt", forbidden) monkeypatch.setattr("tendwire.command_submission.get_command_request", forbidden) monkeypatch.setattr("tendwire.command_submission._current_snapshot", forbidden) - monkeypatch.setattr("tendwire.command_submission._validate_pending_choice", forbidden) monkeypatch.setattr("tendwire.command_submission.reserve_command_request", forbidden) db_path = tmp_path / f"{request_payload['action']}.db" @@ -1260,7 +1259,6 @@ def guarded(*args: Any, **kwargs: Any) -> Any: raise AssertionError("invalid request_id must stop before backend or store mutation") monkeypatch.setattr("tendwire.command_submission.reserve_command_request", guarded) - monkeypatch.setattr("tendwire.command_submission._default_socket_client_factory", guarded) payload: dict[str, Any] = { "schema_version": 1, "action": "send_instruction", @@ -1719,7 +1717,6 @@ def guarded(*args: Any, **kwargs: Any) -> Any: raise AssertionError("invalid schema_version must stop before pipeline work") monkeypatch.setattr("tendwire.command_submission.reserve_command_request", guarded) - monkeypatch.setattr("tendwire.command_submission._default_socket_client_factory", guarded) monkeypatch.setattr( "sys.stdin", io.StringIO( @@ -1983,10 +1980,6 @@ def guarded_backend(*args: Any, **kwargs: Any) -> Any: calls.append("backend") raise AssertionError("changed duplicate receipt must not reach the backend") - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - guarded_backend, - ) monkeypatch.setattr( "sys.stdin", io.StringIO( @@ -2421,10 +2414,6 @@ def guarded_send(*args: Any, **kwargs: Any) -> Any: monkeypatch.setattr("tendwire.cli.fetch_herdr_state", guarded_fetch) monkeypatch.setattr("tendwire.cli.project_from_observations", guarded_project) monkeypatch.setattr("tendwire.cli.execute_command", guarded_execute) - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - guarded_send, - ) monkeypatch.setattr( "sys.stdin", io.StringIO( diff --git a/tests/test_codex_session_reader.py b/tests/test_codex_session_reader.py deleted file mode 100644 index b5e8141..0000000 --- a/tests/test_codex_session_reader.py +++ /dev/null @@ -1,1226 +0,0 @@ -"""Deterministic contracts for the private Codex session resolver and reader.""" - -from __future__ import annotations -from dataclasses import replace - -import json -import os -from pathlib import Path -from uuid import UUID -from types import SimpleNamespace - -import pytest - -from tendwire.backends import herdr_turns - - -SESSION_A = "019f2307-092b-7810-8323-418d7c55bd26" -SESSION_B = "019f2307-092b-7810-8323-418d7c55bd27" -SESSION_C = "11111111-1111-4111-8111-111111111111" - - -def _event(kind: str, turn_id: str, **extra): - return { - "type": "event_msg", - "payload": {"type": kind, "turn_id": turn_id, **extra}, - } - - -def _message(turn_id: str, role: str, text: str, *, phase: str | None = None): - payload = { - "type": "message", - "role": role, - "content": [{"type": "output_text", "text": text}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - } - if phase is not None: - payload["phase"] = phase - return {"type": "response_item", "payload": payload} - - -def _jsonl(*records, terminate: bool = True) -> bytes: - body = b"\n".join( - json.dumps(record, ensure_ascii=False, separators=(",", ":")).encode("utf-8") - for record in records - ) - return body + (b"\n" if terminate and records else b"") - - -def _session_path(home: Path, session_id: str = SESSION_A, date: str = "2026-07-03") -> Path: - year, month, day = date.split("-") - return ( - home - / "sessions" - / year - / month - / day - / f"rollout-{date}T00-00-00-{session_id}.jsonl" - ) - - -def _write_session( - home: Path, - records: tuple[dict, ...] | list[dict], - *, - session_id: str = SESSION_A, - date: str = "2026-07-03", - terminate: bool = True, -) -> Path: - path = _session_path(home, session_id, date) - path.parent.mkdir(parents=True, exist_ok=True) - path.write_bytes(_jsonl(*records, terminate=terminate)) - return path - - -def _reset_codex() -> None: - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - herdr_turns._CODEX_SESSION_CACHE_LIVE_KEYS = None - herdr_turns._CODEX_SESSION_CACHE_BINDING_GENERATIONS = {} - herdr_turns._CODEX_SESSION_CACHE_BINDING_FINGERPRINTS = {} - - -@pytest.fixture(autouse=True) -def _isolated_codex_state(monkeypatch): - _reset_codex() - monkeypatch.setattr(herdr_turns, "_CODEX_INDEX_BUILD_OBSERVER", None) - monkeypatch.setattr(herdr_turns, "_CODEX_ISOLATED_READ_OBSERVER", None) - yield - _reset_codex() - - -def test_canonical_uuid_rejects_adversarial_spellings_before_filesystem_work( - tmp_path: Path, - monkeypatch, -) -> None: - invalid = [ - "", - "*", - "?", - "[abc]", - ".", - "..", - "/", - "\\", - f"../{SESSION_A}", - f"{SESSION_A}/x", - f"prefix-{SESSION_A}", - f"{SESSION_A}-suffix", - f" {SESSION_A}", - f"{SESSION_A} ", - SESSION_A.upper(), - SESSION_A.replace("-", ""), - "{" + SESSION_A + "}", - "urn:uuid:" + SESSION_A, - SESSION_A.replace("-", "‐"), - SESSION_A.replace("d", "ԁ"), - "00000000-0000-0000-0000-000000000000", - "x" * 10_000, - ] - monkeypatch.setenv("CODEX_HOME", str(tmp_path / "must-not-touch")) - monkeypatch.setattr( - herdr_turns, - "_build_codex_index", - lambda *_args: (_ for _ in ()).throw(AssertionError("invalid identity walked")), - ) - for value in invalid: - assert herdr_turns._canonical_codex_session_id(value) is None - assert herdr_turns._find_codex_session_file(value) is None - assert herdr_turns._canonical_codex_session_id(SESSION_A) == SESSION_A - assert herdr_turns._canonical_codex_session_id(SESSION_C) == SESSION_C - - -def test_invalid_identity_is_rejected_before_socket_or_process(monkeypatch) -> None: - monkeypatch.setattr( - herdr_turns.socket, - "socketpair", - lambda: (_ for _ in ()).throw(AssertionError("socket created")), - ) - assert ( - herdr_turns._read_file_turn_isolated( - "codex_session_id", - "*", - timeout_seconds=1, - ) - is None - ) - - -@pytest.mark.parametrize( - ("parts", "name", "expected"), - [ - (("2026", "07", "03"), f"rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", SESSION_A), - (("2024", "02", "29"), f"rollout-2024-02-29T23-59-59-{SESSION_C}.jsonl", SESSION_C), - (("2026", "07", "04"), f"rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", None), - (("2026", "02", "30"), f"rollout-2026-02-30T00-00-00-{SESSION_A}.jsonl", None), - (("2026", "07", "03"), f"rollout-2026-07-03T24-00-00-{SESSION_A}.jsonl", None), - (("2026", "07", "03"), f"rollout-2026-07-03T00-00-00-{SESSION_A}.JSONL", None), - (("2026", "07", "03"), f"rollout-2026-07-03T00-00-00.1-{SESSION_A}.jsonl", None), - (("2026", "07", "03"), f"copy-rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", None), - (("2026", "07", "03"), f"rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl.zst", None), - ], -) -def test_exact_rollout_filename_and_date_grammar(parts, name, expected) -> None: - assert herdr_turns._codex_rollout_identity(parts, name) == expected - - -def test_resolver_selects_only_exact_regular_and_rejects_decoys_and_symlinks( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - target = _write_session(home, [_event("task_started", "turn-a")]) - decoy = _write_session( - home, - [_event("task_started", "turn-b")], - session_id=SESSION_B, - ) - os.utime(decoy, (2_000_000_000, 2_000_000_000)) - target.parent.joinpath(f"prefix-{SESSION_A}.jsonl").write_bytes(b"decoy\n") - target.parent.joinpath( - f"rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl.zst" - ).write_bytes(b"compressed") - outside = tmp_path / "outside.jsonl" - outside.write_bytes(b"outside") - target.parent.joinpath( - f"rollout-2026-07-03T01-00-00-{SESSION_C}.jsonl" - ).symlink_to(outside) - monkeypatch.setenv("CODEX_HOME", str(home)) - - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert herdr_turns._find_codex_session_file(SESSION_B) == decoy.resolve() - assert herdr_turns._find_codex_session_file(SESSION_C) is None - - -def test_duplicate_exact_identity_is_ambiguous_independent_of_mtime( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - first = _write_session(home, [], date="2026-07-03") - second = _write_session(home, [], date="2026-07-04") - os.utime(first, (2_000_000_000, 2_000_000_000)) - os.utime(second, (1, 1)) - monkeypatch.setenv("CODEX_HOME", str(home)) - - resolution = herdr_turns._resolve_codex_session(SESSION_A) - assert resolution is not None - assert resolution.status == "ambiguous" - assert herdr_turns._find_codex_session_file(SESSION_A) is None - - -def test_complete_34k_index_build_is_bounded_and_deterministic(monkeypatch) -> None: - root = Path("/virtual/sessions") - - class Entry: - def __init__(self, path: str, name: str, kind: str): - self.path = path - self.name = name - self.kind = kind - - def is_dir(self, *, follow_symlinks: bool) -> bool: - assert follow_symlinks is False - return self.kind == "dir" - - def is_file(self, *, follow_symlinks: bool) -> bool: - assert follow_symlinks is False - return self.kind == "file" - - year = Entry(f"{root}/2026", "2026", "dir") - month = Entry(f"{year.path}/07", "07", "dir") - day = Entry(f"{month.path}/03", "03", "dir") - files = [] - for ordinal in range(1, 34_001): - session_id = str(UUID(int=ordinal)) - name = f"rollout-2026-07-03T00-00-00-{session_id}.jsonl" - files.append(Entry(f"{day.path}/{name}", name, "file")) - tree = { - str(root): [year], - year.path: [month], - month.path: [day], - day.path: files, - } - monkeypatch.setattr(herdr_turns.os, "scandir", lambda path: tree[os.fspath(path)]) - monkeypatch.setattr(herdr_turns, "_codex_root_signature", lambda _root: (1, 2, 3, 4)) - - generation = herdr_turns._build_codex_index(root) - assert generation.overflowed is False - assert len(generation.entries) == 34_000 - assert generation.entries[str(UUID(int=1))][0].endswith("000000000001.jsonl") - assert generation.entries[str(UUID(int=34_000))][0].endswith("0000000084d0.jsonl") - assert generation.visited == 34_003 - assert generation.retained_bytes <= herdr_turns._CODEX_INDEX_MAX_BYTES - - -def test_index_iterator_stops_at_first_overflow_sentinel(monkeypatch) -> None: - produced = 0 - closed = 0 - - class Entry: - def __init__(self, ordinal: int): - self.name = f"entry-{ordinal}" - self.path = f"/virtual/{self.name}" - - class UnboundedDirectory: - def __iter__(self): - return self - - def __next__(self): - nonlocal produced - produced += 1 - return Entry(produced) - - def close(self): - nonlocal closed - closed += 1 - - observed = [] - monkeypatch.setattr(herdr_turns, "_CODEX_INDEX_MAX_VISITS", 5) - monkeypatch.setattr( - herdr_turns, - "_codex_root_signature", - lambda _root: (1, 2, 3, 4), - ) - monkeypatch.setattr( - herdr_turns.os, - "scandir", - lambda _path: UnboundedDirectory(), - ) - monkeypatch.setattr( - herdr_turns, - "_CODEX_INDEX_BUILD_OBSERVER", - observed.append, - ) - - generation = herdr_turns._build_codex_index(Path("/virtual")) - - assert generation.overflowed is True - assert generation.entries == {} - assert generation.visited == 6 - assert produced == 6 - assert closed == 1 - assert observed == [6] - - -def test_warm_positive_and_negative_lookup_use_one_index_build( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - target = _write_session(home, []) - monkeypatch.setenv("CODEX_HOME", str(home)) - visits = [] - monkeypatch.setattr(herdr_turns, "_CODEX_INDEX_BUILD_OBSERVER", visits.append) - clock = [100.0] - monkeypatch.setattr(herdr_turns.time, "monotonic", lambda: clock[0]) - - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert herdr_turns._find_codex_session_file(SESSION_B) is None - assert herdr_turns._find_codex_session_file(SESSION_B) is None - assert len(visits) == 1 - clock[0] += herdr_turns._CODEX_NEGATIVE_TTL_SECONDS + 0.5 - assert herdr_turns._find_codex_session_file(SESSION_B) is None - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert len(visits) == 1 - clock[0] += herdr_turns._CODEX_POSITIVE_TTL_SECONDS - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert len(visits) == 2 - - -def test_pure_interpreter_preserves_turn_id_precedence_and_stream_window() -> None: - work = herdr_turns._CodexWorkState( - resolver_generation=1, - root="/root", - root_file_id=(9, 9), - session_id=SESSION_A, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", - file_id=(1, 2), - observed_size=0, - mtime_ns=0, - ctime_ns=0, - ) - start = herdr_turns._codex_record_event(_event("task_started", "active")) - herdr_turns._apply_codex_event(work, start, herdr_turns._CodexRecordSpan(0, 1)) - for index, text in enumerate(("one", "two", "three", "four", "two", "five"), 1): - event = herdr_turns._codex_record_event( - _message("active", "assistant", text, phase="commentary") - ) - herdr_turns._apply_codex_event( - work, - event, - herdr_turns._CodexRecordSpan(index * 10, index * 10 + 5), - ) - assert [text for _span, text in work.stream_items] == ["three", "four", "two", "five"] - direct = { - "type": "response_item", - "payload": { - "type": "message", - "role": "user", - "turn_id": "direct", - "content": [{"text": "x"}], - "internal_chat_message_metadata_passthrough": {"turn_id": "metadata"}, - }, - } - assert herdr_turns._codex_record_event(direct).turn_id == "direct" - - -def test_partial_record_is_invisible_until_lf_and_commits_once( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "turn-partial" - base = [_event("task_started", turn_id), _message(turn_id, "user", "prompt")] - path = _write_session(home, base) - completion = _jsonl( - _event("task_complete", turn_id, last_agent_message="final once"), - terminate=False, - ) - split = len(completion) // 2 - with path.open("ab") as handle: - handle.write(completion[:split]) - monkeypatch.setenv("CODEX_HOME", str(home)) - observed = [] - monkeypatch.setattr(herdr_turns, "_CODEX_ISOLATED_READ_OBSERVER", observed.append) - - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["user_text"] == "prompt" - assert first["assistant_final_text"] is None - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - first_state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - committed = first_state.committed_offset - assert first_state.partial_record == completion[:split] - - with path.open("ab") as handle: - handle.write(completion[split:]) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - second_state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - assert second_state.committed_offset == committed - assert second_state.partial_record == completion - - with path.open("ab") as handle: - handle.write(b"\n") - final = herdr_turns._read_codex_session_turn(SESSION_A) - assert final["assistant_final_text"] == "final once" - assert final["complete"] is True - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - assert observed[-1] == 0 - - -def test_sparse_cold_read_and_warm_append_are_bounded( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - path = _session_path(home) - path.parent.mkdir(parents=True) - turn_id = "turn-sparse" - tail = _jsonl( - _event("task_started", turn_id), - _message(turn_id, "user", "sparse prompt"), - ) - with path.open("wb") as handle: - handle.seek(20 * 1024 * 1024) - handle.write(b"\n") - handle.write(tail) - monkeypatch.setenv("CODEX_HOME", str(home)) - observed = [] - monkeypatch.setattr(herdr_turns, "_CODEX_ISOLATED_READ_OBSERVER", observed.append) - - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["user_text"] == "sparse prompt" - assert observed[-1] <= herdr_turns._CODEX_RESYNC_INITIAL_BYTES - append = _jsonl(_message(turn_id, "assistant", "working", phase="commentary")) - with path.open("ab") as handle: - handle.write(append) - second = herdr_turns._read_codex_session_turn(SESSION_A) - assert second["assistant_stream_text"] == "working" - assert observed[-1] == len(append) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - assert observed[-1] == 0 - - -def test_delayed_incremental_poll_publishes_completed_turn_before_next_turn( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - first_turn = "turn-before-gap" - next_turn = "turn-after-gap" - path = _write_session( - home, - [ - _event("task_started", first_turn), - _message(first_turn, "user", "first prompt"), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - opened = herdr_turns._read_codex_session_turn(SESSION_A) - assert opened["source_turn_id"] == first_turn - assert opened["complete"] is False - - final_record = _message( - first_turn, - "assistant", - "first final", - phase="final_answer", - ) - first_batch = _jsonl( - _message(first_turn, "assistant", "first progress", phase="commentary"), - final_record, - ) - later_batch = _jsonl( - _event("task_started", next_turn), - _message(next_turn, "user", "next prompt"), - _message(next_turn, "assistant", "next progress", phase="commentary"), - ) - with path.open("ab") as handle: - handle.write(first_batch) - handle.write(later_batch) - - completed = herdr_turns._read_codex_session_turn(SESSION_A) - assert completed == { - # The coordinate-only checkpoint deliberately retains no canonical - # prompt body. Store merge preserves the previously persisted prompt. - "user_text": None, - "assistant_stream_text": None, - "assistant_final_text": "first final", - "complete": True, - "has_open_turn": False, - "source_turn_id": first_turn, - } - cache_key = (str((home / "sessions").resolve()), SESSION_A) - expected_offset = len( - _jsonl( - _event("task_started", first_turn), - _message(first_turn, "user", "first prompt"), - ) - ) + len(first_batch) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - completed_state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - assert completed_state.committed_offset == expected_offset - assert completed_state.observed_size == expected_offset - assert completed_state.partial_record == b"" - - following = herdr_turns._read_codex_session_turn(SESSION_A) - assert following["source_turn_id"] == next_turn - assert following["user_text"] == "next prompt" - assert following["assistant_stream_text"] == "next progress" - assert following["assistant_final_text"] is None - assert following["complete"] is False - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - - -def test_malformed_and_oversized_records_block_without_checkpoint_advance( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - path = _write_session( - home, - [_event("task_started", "blocked"), _message("blocked", "user", "safe")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["user_text"] == "safe" - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - before = herdr_turns._serialize_codex_state( - herdr_turns._CODEX_SESSION_CACHE[cache_key] - ) - with path.open("ab") as handle: - handle.write(b"{not-json}\n") - with pytest.raises(ValueError, match="invalid Codex record"): - herdr_turns._read_codex_session_turn(SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert herdr_turns._serialize_codex_state( - herdr_turns._CODEX_SESSION_CACHE[cache_key] - ) == before - - path.write_bytes(_jsonl(_event("task_started", "oversize")) + b"x" * 65) - monkeypatch.setattr(herdr_turns, "_CODEX_RECORD_MAX_BYTES", 64) - _reset_codex() - with pytest.raises(ValueError, match="oversized Codex record"): - herdr_turns._read_codex_session_turn(SESSION_A) - - -def test_truncate_and_inode_replacement_recover_latest_exact_turn( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - old_prompt = "old prompt " + ("x" * 1024) - old = [_event("task_started", "old"), _message("old", "user", old_prompt)] - path = _write_session(home, old) - monkeypatch.setenv("CODEX_HOME", str(home)) - assert herdr_turns._read_codex_session_turn(SESSION_A)["source_turn_id"] == "old" - - new_prompt = "new prompt" - new = [_event("task_started", "new"), _message("new", "user", new_prompt)] - path.write_bytes(_jsonl(*new)) - os.utime(path, ns=(1_800_000_000_000_000_000, 1_800_000_000_000_000_000)) - truncated = herdr_turns._read_codex_session_turn(SESSION_A) - assert truncated["source_turn_id"] == "new" - assert truncated["user_text"] == new_prompt - - replacement = path.with_name("replacement.tmp") - newest = [ - _event("task_started", "replacement"), - _message("replacement", "user", "replacement prompt"), - ] - replacement.write_bytes(_jsonl(*newest)) - replacement.replace(path) - replaced = herdr_turns._read_codex_session_turn(SESSION_A) - assert replaced["source_turn_id"] == "replacement" - assert replaced["user_text"] == "replacement prompt" - - -def test_huge_admitted_prompt_and_final_are_exact_and_not_cached( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - prompt = "π" * 300_000 - final = "終" * 300_000 - turn_id = "huge" - _write_session( - home, - [ - _event("task_started", turn_id), - _message(turn_id, "user", prompt), - _event("task_complete", turn_id, last_agent_message=final), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - content = herdr_turns._read_codex_session_turn(SESSION_A) - assert content["user_text"] == prompt - assert content["assistant_final_text"] == final - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - serialized = herdr_turns._serialize_codex_state( - herdr_turns._CODEX_SESSION_CACHE[cache_key] - ) - private_json = json.dumps(serialized, separators=(",", ":")) - assert prompt[:100] not in private_json - assert final[:100] not in private_json - assert serialized["stream_spans"] == [] - - -def test_parser_lru_enforces_count_and_weight_with_mru_touch(monkeypatch) -> None: - monkeypatch.setattr(herdr_turns, "_CODEX_SESSION_CACHE_CAPACITY", 3) - monkeypatch.setattr(herdr_turns, "_CODEX_SESSION_CACHE_MAX_BYTES", 64 * 1024) - - def state(session_id: str, ordinal: int): - return herdr_turns._CodexSessionState( - resolver_generation=1, - root="/root", - root_file_id=(9, 9), - session_id=session_id, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{session_id}.jsonl", - file_id=(1, ordinal), - observed_size=ordinal, - mtime_ns=ordinal, - ctime_ns=ordinal, - committed_offset=ordinal, - partial_record=b"", - active_turn_id="", - last_content_turn_id="", - turn_open=False, - final_seen=False, - complete=False, - stream_spans=(), - ) - - ids = [str(UUID(int=index)) for index in range(1, 5)] - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - for index, session_id in enumerate(ids[:3], 1): - key = ("/root", session_id) - assert herdr_turns._codex_cache_store_locked(key, state(session_id, index)) - herdr_turns._codex_cache_get_locked(("/root", ids[0])) - herdr_turns._codex_cache_store_locked(("/root", ids[3]), state(ids[3], 4)) - assert [key[1] for key in herdr_turns._CODEX_SESSION_CACHE] == [ - ids[2], - ids[0], - ids[3], - ] - assert herdr_turns._codex_cache_weight_locked() <= 64 * 1024 - - -def test_state_deserializer_rejects_bodies_overlap_and_wrong_rollout_path() -> None: - state = herdr_turns._CodexSessionState( - resolver_generation=1, - root="/root", - root_file_id=(9, 9), - session_id=SESSION_A, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", - file_id=(1, 2), - observed_size=100, - mtime_ns=1, - ctime_ns=1, - committed_offset=100, - partial_record=b"", - active_turn_id="turn", - last_content_turn_id="turn", - turn_open=True, - final_seen=False, - complete=False, - stream_spans=(herdr_turns._CodexRecordSpan(10, 20),), - ) - serialized = herdr_turns._serialize_codex_state(state) - assert herdr_turns._deserialize_codex_state(serialized) == state - body = dict(serialized) - body["user_text"] = "forbidden" - with pytest.raises(ValueError, match="invalid Codex parser state"): - herdr_turns._deserialize_codex_state(body) - overlap = dict(serialized) - overlap["stream_spans"] = [[10, 20], [20, 30]] - with pytest.raises(ValueError, match="overlapping"): - herdr_turns._deserialize_codex_state(overlap) - wrong = dict(serialized) - wrong["canonical_path"] = f"/root/2026/07/04/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl" - with pytest.raises(ValueError, match="invalid Codex rollout path"): - herdr_turns._deserialize_codex_state(wrong) - - -def test_partial_completion_is_transactional_at_every_byte_split( - tmp_path: Path, - monkeypatch, -) -> None: - turn_id = "all-splits" - completion = _jsonl( - _event("task_complete", turn_id, last_agent_message="split final"), - terminate=False, - ) - for split in range(len(completion) + 1): - _reset_codex() - home = tmp_path / f"split-{split}" - path = _write_session( - home, - [_event("task_started", turn_id), _message(turn_id, "user", "split prompt")], - ) - base_size = path.stat().st_size - with path.open("ab") as handle: - handle.write(completion[:split]) - monkeypatch.setenv("CODEX_HOME", str(home)) - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["assistant_final_text"] is None - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - assert state.committed_offset == base_size - assert state.partial_record == completion[:split] - if split < len(completion): - with path.open("ab") as handle: - handle.write(completion[split:]) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - with path.open("ab") as handle: - handle.write(b"\n") - completed = herdr_turns._read_codex_session_turn(SESSION_A) - assert completed["assistant_final_text"] == "split final" - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - - -def test_path_result_lru_has_deterministic_mru_and_capacity( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - session_ids = [str(UUID(int=index)) for index in range(101, 105)] - for session_id in session_ids: - _write_session(home, [], session_id=session_id) - monkeypatch.setenv("CODEX_HOME", str(home)) - monkeypatch.setattr(herdr_turns, "_CODEX_PATH_CACHE_CAPACITY", 3) - for session_id in session_ids[:3]: - assert herdr_turns._resolve_codex_session(session_id).status == "found" - assert herdr_turns._resolve_codex_session(session_ids[0]).status == "found" - assert herdr_turns._resolve_codex_session(session_ids[3]).status == "found" - with herdr_turns._CODEX_PATH_CACHE_LOCK: - assert [key[1] for key in herdr_turns._CODEX_PATH_CACHE] == [ - session_ids[2], - session_ids[0], - session_ids[3], - ] - assert ( - herdr_turns._codex_path_cache_weight_locked() - <= herdr_turns._CODEX_PATH_CACHE_MAX_BYTES - ) - - -def test_rotation_duplicate_and_parser_cold_start_are_fail_closed( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - path = _write_session( - home, - [_event("task_started", "rotation"), _message("rotation", "user", "rotating")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - assert herdr_turns._read_codex_session_turn(SESSION_A)["user_text"] == "rotating" - - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - cold = herdr_turns._read_codex_session_turn(SESSION_A) - assert cold["source_turn_id"] == "rotation" - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - fully_cold = herdr_turns._read_codex_session_turn(SESSION_A) - assert fully_cold["user_text"] == "rotating" - - duplicate = _session_path(home, SESSION_A, "2026-07-04") - duplicate.parent.mkdir(parents=True) - duplicate.write_bytes(path.read_bytes()) - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - assert herdr_turns._find_codex_session_file(SESSION_A) is None - assert herdr_turns._resolve_codex_session(SESSION_A).status == "ambiguous" - - -def test_long_warm_turn_advances_beyond_cold_resync_horizon( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "long-warm" - path = _write_session( - home, - [_event("task_started", turn_id), _message(turn_id, "user", "long prompt")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - monkeypatch.setattr(herdr_turns, "_CODEX_RESYNC_MAX_BYTES", 1024) - observed = [] - monkeypatch.setattr(herdr_turns, "_CODEX_ISOLATED_READ_OBSERVER", observed.append) - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["user_text"] == "long prompt" - - newest = [] - for index in range(12): - text = f"progress-{index}-" + ("x" * 220) - newest.append(text) - append = _jsonl(_message(turn_id, "assistant", text, phase="commentary")) - with path.open("ab") as handle: - handle.write(append) - current = herdr_turns._read_codex_session_turn(SESSION_A) - assert current["source_turn_id"] == turn_id - assert current["complete"] is False - assert current["assistant_stream_text"].split("\n\n") == newest[-4:] - assert observed[-1] <= len(append) + 4 * herdr_turns._CODEX_RECORD_MAX_BYTES - assert path.stat().st_size > herdr_turns._CODEX_RESYNC_MAX_BYTES - - final_record = _jsonl( - _event("task_complete", turn_id, last_agent_message="long exact final") - ) - with path.open("ab") as handle: - handle.write(final_record) - final = herdr_turns._read_codex_session_turn(SESSION_A) - assert final["assistant_final_text"] == "long exact final" - assert final["assistant_stream_text"] is None - assert final["complete"] is True - - -def test_codex_cache_cas_is_monotone_and_does_not_resurrect_eviction() -> None: - cache_key = ("/root", SESSION_A) - prior = herdr_turns._CodexSessionState( - resolver_generation=1, - root="/root", - root_file_id=(9, 9), - session_id=SESSION_A, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", - file_id=(1, 2), - observed_size=10, - mtime_ns=1, - ctime_ns=1, - committed_offset=10, - partial_record=b"", - active_turn_id="turn", - last_content_turn_id="turn", - turn_open=True, - final_seen=False, - complete=False, - stream_spans=(), - ) - newer = replace( - prior, - observed_size=30, - mtime_ns=3, - ctime_ns=3, - committed_offset=30, - ) - older = replace( - prior, - observed_size=20, - mtime_ns=2, - ctime_ns=2, - committed_offset=20, - ) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE[cache_key] = prior - prior_value = herdr_turns._serialize_codex_state(prior) - assert herdr_turns._publish_codex_cache_state( - cache_key, - prior_value, - newer, - {"source_turn_id": "turn"}, - None, - ) == {"source_turn_id": "turn"} - assert herdr_turns._publish_codex_cache_state( - cache_key, - prior_value, - older, - {"source_turn_id": "turn"}, - None, - ) is None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert herdr_turns._CODEX_SESSION_CACHE[cache_key].committed_offset == 30 - herdr_turns._CODEX_SESSION_CACHE.clear() - assert herdr_turns._publish_codex_cache_state( - cache_key, - prior_value, - newer, - {"source_turn_id": "turn"}, - None, - ) is None - - -def test_symlinked_sessions_root_is_rejected( - tmp_path: Path, - monkeypatch, -) -> None: - outside_home = tmp_path / "outside" - target = _write_session(outside_home, [_event("task_started", "outside")]) - configured = tmp_path / "configured" - configured.mkdir() - (configured / "sessions").symlink_to(outside_home / "sessions", target_is_directory=True) - monkeypatch.setenv("CODEX_HOME", str(configured)) - - assert target.is_file() - assert herdr_turns._find_codex_session_file(SESSION_A) is None - - -def test_internal_turn_remains_suppressed_when_later_commentary_arrives( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "internal-turn" - path = _write_session( - home, - [ - _event("task_started", turn_id), - _message( - turn_id, - "user", - "Acme job\n\nTemplate: security-review\nTemplate instructions:", - ), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert herdr_turns._CODEX_SESSION_CACHE[cache_key].internal_turn is True - - with path.open("ab") as handle: - handle.write( - _jsonl( - _message( - turn_id, - "assistant", - "ordinary-looking internal progress", - phase="commentary", - ) - ) - ) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - assert state.internal_turn is True - assert state.stream_spans == () - - -def test_real_user_after_environment_context_makes_turn_public( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "context-then-user" - _write_session( - home, - [ - _event("task_started", turn_id), - _message( - turn_id, - "user", - "\n /private/path\n", - ), - _message(turn_id, "user", "fix the Telegram delivery"), - _message( - turn_id, - "assistant", - "Tracing the delivery path.", - phase="commentary", - ), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - content = herdr_turns._read_codex_session_turn(SESSION_A) - - assert content == { - "user_text": "fix the Telegram delivery", - "assistant_stream_text": "Tracing the delivery path.", - "assistant_final_text": None, - "complete": False, - "has_open_turn": True, - "source_turn_id": turn_id, - } - assert "/private/path" not in json.dumps(content) - - -def test_internal_context_after_real_user_does_not_hide_turn( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "user-then-context" - _write_session( - home, - [ - _event("task_started", turn_id), - _message(turn_id, "user", "keep this prompt"), - _message( - turn_id, - "user", - "private runtime metadata", - ), - _message( - turn_id, - "assistant", - "Visible progress.", - phase="commentary", - ), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - content = herdr_turns._read_codex_session_turn(SESSION_A) - - assert content["user_text"] == "keep this prompt" - assert content["assistant_stream_text"] == "Visible progress." - assert "private runtime metadata" not in json.dumps(content) - - -def test_empty_task_complete_preserves_existing_open_turn_semantics( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "empty-complete" - _write_session( - home, - [ - _event("task_started", turn_id), - _message(turn_id, "user", "still open by existing contract"), - _event("task_complete", turn_id), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - content = herdr_turns._read_codex_session_turn(SESSION_A) - assert content["user_text"] == "still open by existing contract" - assert content["assistant_final_text"] is None - assert content["complete"] is False - assert content["has_open_turn"] is True - - -def test_descriptor_relative_open_rejects_ancestor_symlink_swap( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - path = _write_session( - home, - [_event("task_started", "inside"), _message("inside", "user", "inside")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - resolution = herdr_turns._resolve_codex_session(SESSION_A) - assert resolution is not None and resolution.status == "found" - - outside_day = tmp_path / "outside" / "2026" / "07" / "03" - outside_day.mkdir(parents=True) - outside_file = outside_day / path.name - outside_file.write_bytes( - _jsonl( - _event("task_started", "outside"), - _message("outside", "user", "outside sentinel"), - ) - ) - original_open = herdr_turns.os.open - swapped = False - - def racing_open(path_value, flags, mode=0o777, *, dir_fd=None): - nonlocal swapped - if path_value == path.name and dir_fd is not None and not swapped: - swapped = True - saved = path.parent.with_name("03-saved") - path.parent.rename(saved) - path.parent.symlink_to(outside_day, target_is_directory=True) - return original_open(path_value, flags, mode, dir_fd=dir_fd) - - monkeypatch.setattr(herdr_turns.os, "open", racing_open) - with pytest.raises(herdr_turns._TurnReadFailed): - herdr_turns._open_verified_codex_file(resolution) - assert swapped is True - assert outside_file.read_bytes().endswith(b"\n") - - -def test_sessions_root_swap_during_resolution_is_rejected( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "configured" - _write_session(home, [_event("task_started", "inside")]) - outside = tmp_path / "outside" - _write_session(outside, [_event("task_started", "outside")]) - monkeypatch.setenv("CODEX_HOME", str(home)) - lexical_root = home / "sessions" - original_resolve = Path.resolve - swapped = False - - def racing_resolve(path_value, *args, **kwargs): - nonlocal swapped - if path_value == lexical_root and not swapped: - swapped = True - lexical_root.rename(home / "sessions-saved") - lexical_root.symlink_to(outside / "sessions", target_is_directory=True) - return original_resolve(path_value, *args, **kwargs) - - monkeypatch.setattr(Path, "resolve", racing_resolve) - assert herdr_turns._find_codex_session_file(SESSION_A) is None - assert swapped is True - - -def test_incremental_poll_limit_rejects_before_reading() -> None: - prior = herdr_turns._CodexSessionState( - resolver_generation=1, - root="/root", - root_file_id=(1, 1), - session_id=SESSION_A, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", - file_id=(2, 2), - observed_size=10, - mtime_ns=1, - ctime_ns=1, - committed_offset=10, - partial_record=b"", - active_turn_id="turn", - last_content_turn_id="turn", - turn_open=True, - final_seen=False, - complete=False, - stream_spans=(), - ) - opened = SimpleNamespace( - st_dev=2, - st_ino=2, - st_size=10 + herdr_turns._CODEX_POLL_MAX_BYTES + 1, - st_mtime_ns=2, - st_ctime_ns=2, - ) - with pytest.raises(ValueError, match="poll byte limit"): - herdr_turns._read_codex_incremental(-1, prior, opened) - - -def test_ipc_frame_bound_covers_maximum_compact_state_and_visible_turn() -> None: - assert herdr_turns._CODEX_IPC_FRAME_MAX_BYTES >= ( - herdr_turns._CODEX_STATE_IPC_MAX_BYTES - + (1 + herdr_turns._MAX_CODEX_STREAM_MESSAGES) - * herdr_turns._CODEX_RECORD_MAX_BYTES - ) - - -def test_positive_snapshot_discovers_duplicate_by_named_refresh_bound( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - original = _write_session(home, [], date="2026-07-03") - monkeypatch.setenv("CODEX_HOME", str(home)) - clock = [500.0] - monkeypatch.setattr(herdr_turns.time, "monotonic", lambda: clock[0]) - visits = [] - monkeypatch.setattr(herdr_turns, "_CODEX_INDEX_BUILD_OBSERVER", visits.append) - - assert herdr_turns._find_codex_session_file(SESSION_A) == original.resolve() - duplicate = _write_session(home, [], date="2026-07-04") - assert duplicate.is_file() - assert herdr_turns._find_codex_session_file(SESSION_A) == original.resolve() - assert len(visits) == 1 - - clock[0] += herdr_turns._CODEX_POSITIVE_TTL_SECONDS + 0.001 - assert herdr_turns._find_codex_session_file(SESSION_A) is None - assert herdr_turns._resolve_codex_session(SESSION_A).status == "ambiguous" - assert len(visits) == 2 - - -def test_root_replacement_invalidates_found_and_nonfound_even_with_same_rollout_inode( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - original_path = _write_session( - home, - [_event("task_started", "root-one")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - builds = [] - monkeypatch.setattr( - herdr_turns, - "_CODEX_INDEX_BUILD_OBSERVER", - builds.append, - ) - - first = herdr_turns._resolve_codex_session(SESSION_A) - assert first is not None and first.status == "found" - assert herdr_turns._resolve_codex_session(SESSION_B).status == "missing" - assert len(builds) == 1 - original_file_id = first.file_id - original_root_id = first.root_file_id - - old_root = home / "sessions-old" - (home / "sessions").rename(old_root) - replacement_path = _session_path(home) - replacement_path.parent.mkdir(parents=True) - os.link( - old_root / replacement_path.relative_to(home / "sessions"), - replacement_path, - ) - assert ( - replacement_path.stat().st_dev, - replacement_path.stat().st_ino, - ) == original_file_id - - refreshed_found = herdr_turns._resolve_codex_session(SESSION_A) - assert refreshed_found is not None - assert refreshed_found.status == "found" - assert refreshed_found.file_id == original_file_id - assert refreshed_found.root_file_id != original_root_id - assert refreshed_found.generation != first.generation - assert len(builds) == 2 - refreshed_missing = herdr_turns._resolve_codex_session(SESSION_B) - assert refreshed_missing is not None - assert refreshed_missing.status == "missing" - assert refreshed_missing.root_file_id == refreshed_found.root_file_id - assert len(builds) == 2 diff --git a/tests/test_command_presend_retryability.py b/tests/test_command_presend_retryability.py deleted file mode 100644 index 9a01d97..0000000 --- a/tests/test_command_presend_retryability.py +++ /dev/null @@ -1,1076 +0,0 @@ -"""Transient pre-send failures stay retryable; permanent ones stay terminal. - -Goal 11B made an existing receipt authoritative for its retry. Goal 11C closes -the gap that stress testing exposed: a transient *local* failure before any -backend send -- the binding store or receipt store raising, a socket that will -not connect, a pane read that times out -- was reserved and written as a durable -``terminal_rejected`` receipt, permanently dropping a command that was never -sent. - -The corrected rule classifies a pre-send failure by the last irreversible stage -it reached. A failed local or backend *operation* proves nothing durable and -stays ``no_receipt`` / retryable under the same request ID. Only an authoritative -observation of proven target unsuitability -- a disallowed worker status, an -unavailable backend, or a missing/stale/ambiguous private binding -- may -terminalize. Anything after a send may have started stays terminal uncertainty. -""" - -from __future__ import annotations - -import sqlite3 -import threading -from pathlib import Path -from typing import Any - -import pytest - -import tendwire.command_submission as command_submission - -from tendwire.backends.herdr_socket import HerdrSocketTimeoutError -from tendwire.command_submission import submit_command -from tendwire.config import Config -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - DISPOSITION_TERMINAL_ACCEPTED, - DISPOSITION_TERMINAL_REJECTED, - STATUS_ACCEPTED, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_PENDING, - STATUS_REJECTED, - STATUS_STALE_TARGET, -) -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.daemon_api import TendwireDaemonAPI -from tendwire.local_state import LocalStateError, LocalStateErrorCode -from tendwire.store.sqlite import ( - get_command_request, - init_store, - save_snapshot, - upsert_worker_bindings, -) - - -HOST_ID = "cmd-host" - - -def _config(tmp_path: Path) -> Config: - return Config( - host_id=HOST_ID, - data_dir=tmp_path, - db_path=tmp_path / "commands.db", - herdr_backend="socket", - herdr_timeout_seconds=5.0, - ) - - -def _worker(*, worker_id: str = "w-1", status: str = "active") -> Worker: - return Worker(id=worker_id, name="Alpha", status=status, space_id="space-1") - - -def _binding( - worker: Worker, - *, - sendable: bool = True, - fingerprint: str | None = None, -) -> WorkerBinding: - return WorkerBinding( - host_id=HOST_ID, - worker_id=worker.id, - worker_fingerprint=fingerprint or worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value=f"agent-{worker.id}", - turn_target_kind="pane_id", - turn_target_value=f"pane-{worker.id}", - sendable=sendable, - reason=None if sendable else "not_sendable", - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint=f"private-{worker.id}", - ) - - -def _health(status: str = "healthy") -> BackendHealth: - return BackendHealth( - name="herdr", - status=status, - outcome="healthy_non_empty" if status == "healthy" else "timeout", - observed_at="2026-01-01T00:00:00+00:00", - counts={"workers": 1}, - ) - - -def _seed( - config: Config, - workers: list[Worker], - bindings: list[WorkerBinding], - *, - health: str = "healthy", -) -> None: - assert config.db_path is not None - init_store(config.db_path) - save_snapshot( - config.db_path, - Snapshot( - host_id=HOST_ID, - updated_at="2026-01-01T00:00:00+00:00", - workers=workers, - backend_health=[_health(health)], - ), - ) - if bindings: - upsert_worker_bindings(config.db_path, bindings) - - -def _request( - *, - request_id: str, - worker_id: str = "w-1", - text: str = "hello", -) -> dict[str, Any]: - return { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": worker_id}, - "instruction": {"text": text}, - } - - -class _FakeSocketClient: - def __init__( - self, - calls: list[dict[str, Any]], - *, - agent_get_raises: BaseException | None = None, - agent_get_response: dict[str, Any] | None = None, - ) -> None: - self.calls = calls - self.agent_get_raises = agent_get_raises - self.agent_get_response = agent_get_response - - def connect(self) -> "_FakeSocketClient": - return self - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if method == "agent.get": - if self.agent_get_raises is not None: - raise self.agent_get_raises - if self.agent_get_response is not None: - return self.agent_get_response - return {"result": {"agent": {"pane_id": "pane-w-1"}}} - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": "Completed previous turn.\n── status: idle ──"}, - } - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": "pane-w-1"}, - "delivery": "submitted", - } - return {"accepted": True} - - def close(self) -> None: - return None - - -def _factory(calls: list[dict[str, Any]], **kwargs: Any): - def make_client(config: Config) -> _FakeSocketClient: - return _FakeSocketClient(calls, **kwargs) - - return make_client - - -def _forbidden_factory(config: Config) -> Any: - pytest.fail("a receipt replay must not create a socket client") - - -def _sent_texts(calls: list[dict[str, Any]]) -> list[str]: - return [ - str(call["params"].get("text")) - for call in calls - if call["method"] == "agent.prompt" - ] - - -def _receipt(config: Config, request_id: str) -> dict[str, Any] | None: - assert config.db_path is not None - return get_command_request(config.db_path, config.host_id, request_id) - - -def _receipt_count(config: Config) -> int: - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - return conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] - - -# --------------------------------------------------------------------------- -# Deterministic injected transients: each must stay retryable -# --------------------------------------------------------------------------- - - -class _TransientInjection: - """One armed pre-send transient that clears after the first attempt.""" - - def __init__(self, kind: str) -> None: - self.kind = kind - self.armed = True - - def install( - self, - monkeypatch: pytest.MonkeyPatch, - calls: list[dict[str, Any]], - ) -> Any: - real_bindings = command_submission.list_worker_bindings - real_latest_snapshot = command_submission.latest_snapshot - real_reserve = command_submission.reserve_command_request - - if self.kind == "snapshot_store_local_state": - def latest_snapshot(*a: Any, **k: Any) -> Any: - if self.armed: - raise LocalStateError( - LocalStateErrorCode.ENTRY_CHANGED, - LocalStateErrorCode.ENTRY_CHANGED, - "local-state entry changed during validation", - ) - return real_latest_snapshot(*a, **k) - - monkeypatch.setattr(command_submission, "latest_snapshot", latest_snapshot) - return _factory(calls) - - if self.kind == "binding_store_sqlite": - def bindings(*a: Any, **k: Any) -> Any: - if self.armed: - raise sqlite3.OperationalError("database is locked") - return real_bindings(*a, **k) - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings) - return _factory(calls) - - if self.kind == "binding_store_local_state": - def bindings(*a: Any, **k: Any) -> Any: - if self.armed: - raise LocalStateError( - LocalStateErrorCode.ENTRY_CHANGED, - "local-state entry changed during validation", - ) - return real_bindings(*a, **k) - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings) - return _factory(calls) - - if self.kind == "receipt_store_open": - def reserve(*a: Any, **k: Any) -> Any: - if self.armed: - raise sqlite3.OperationalError("database is locked") - return real_reserve(*a, **k) - - monkeypatch.setattr(command_submission, "reserve_command_request", reserve) - return _factory(calls) - - if self.kind == "socket_connect": - def make_client(config: Config) -> Any: - if self.armed: - raise ConnectionRefusedError("socket refused") - return _FakeSocketClient(calls) - - return make_client - - if self.kind == "pane_resolution": - def make_client(config: Config) -> _FakeSocketClient: - if self.armed: - return _FakeSocketClient( - calls, - agent_get_raises=HerdrSocketTimeoutError("pane read timeout"), - ) - return _FakeSocketClient(calls) - - return make_client - - raise AssertionError(f"unknown injection {self.kind!r}") - - -TRANSIENT_KINDS = [ - "snapshot_store_local_state", - "binding_store_sqlite", - "binding_store_local_state", - "receipt_store_open", - "socket_connect", - "pane_resolution", -] - - -@pytest.mark.parametrize("kind", TRANSIENT_KINDS) -def test_pre_send_transient_stays_retryable_then_succeeds_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - kind: str, -) -> None: - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - injection = _TransientInjection(kind) - factory = injection.install(monkeypatch, calls) - request_id = f"transient-{kind}" - - first = submit_command(config, _request(request_id=request_id), socket_client_factory=factory) - - # No external mutation began and no durable authority was written. - assert first.ok is False - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_NO_RECEIPT - assert _sent_texts(calls) == [] - assert _receipt(config, request_id) is None - assert _receipt_count(config) == 0 - - # The transient clears; the same request ID succeeds exactly once. - injection.armed = False - recovered = submit_command( - config, _request(request_id=request_id), socket_client_factory=factory - ) - assert recovered.status == STATUS_ACCEPTED - assert recovered.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert _sent_texts(calls) == ["hello"] - receipt = _receipt(config, request_id) - assert receipt is not None - assert receipt["state"] == "accepted" - - # A later replay returns the stored accepted result without another send. - replay = submit_command( - config, _request(request_id=request_id), socket_client_factory=_forbidden_factory - ) - assert replay.to_dict() == recovered.to_dict() - assert _sent_texts(calls) == ["hello"] - - -@pytest.mark.parametrize("kind", TRANSIENT_KINDS) -def test_pre_send_transient_never_requires_a_new_request_id( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - kind: str, -) -> None: - """Retrying is legal under the SAME id; a different id would double-send.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - injection = _TransientInjection(kind) - factory = injection.install(monkeypatch, calls) - - first = submit_command( - config, _request(request_id="same-id"), socket_client_factory=factory - ) - assert first.disposition == DISPOSITION_NO_RECEIPT - - # Two more attempts under the same id while still transient: still no receipt, - # still no send. Nothing accumulates. - injection.armed = True - again = submit_command( - config, _request(request_id="same-id"), socket_client_factory=factory - ) - assert again.disposition == DISPOSITION_NO_RECEIPT - assert _receipt_count(config) == 0 - assert _sent_texts(calls) == [] - - injection.armed = False - done = submit_command( - config, _request(request_id="same-id"), socket_client_factory=factory - ) - assert done.status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - # Exactly one receipt for the one request id. - assert _receipt_count(config) == 1 - - -def test_receipt_store_transient_closes_the_prepared_client( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A receipt-store transient after a successful prepare must not leak the socket.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - closed: list[bool] = [] - - class _TrackingClient(_FakeSocketClient): - def close(self) -> None: - closed.append(True) - - def factory(_config: Config) -> _TrackingClient: - return _TrackingClient([]) - - def reserve_raises(*a: Any, **k: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "reserve_command_request", reserve_raises) - - envelope = submit_command( - config, _request(request_id="store-transient"), socket_client_factory=factory - ) - - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert _receipt(config, "store-transient") is None - # The prepared socket was opened (pane resolved) but never sent, and closed. - assert closed == [True] - - -# --------------------------------------------------------------------------- -# Permanent pre-send failures: terminal and non-sending -# --------------------------------------------------------------------------- - - -def test_disallowed_worker_status_is_terminal( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = _worker(status="closed") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, _request(request_id="disallowed"), socket_client_factory=_factory(calls) - ) - replay = submit_command( - config, _request(request_id="disallowed"), socket_client_factory=_forbidden_factory - ) - - assert first.status == STATUS_REJECTED - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, "disallowed") - assert receipt is not None - assert receipt["state"] == "rejected" - - -def test_missing_private_binding_is_terminal( - tmp_path: Path, -) -> None: - """A worker with no sendable backend binding is an unsupported target.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], []) # worker resolves, but no herdr binding exists - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, _request(request_id="no-binding"), socket_client_factory=_factory(calls) - ) - replay = submit_command( - config, _request(request_id="no-binding"), socket_client_factory=_forbidden_factory - ) - - assert first.status == STATUS_BACKEND_UNSUPPORTED - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, "no-binding") - assert receipt is not None - assert receipt["state"] == "rejected" - - -def test_stale_private_binding_is_terminal( - tmp_path: Path, -) -> None: - """A binding whose fingerprint no longer matches the worker is a proven stale target.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker, fingerprint="stale-observation")]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, _request(request_id="stale-binding"), socket_client_factory=_factory(calls) - ) - replay = submit_command( - config, _request(request_id="stale-binding"), socket_client_factory=_forbidden_factory - ) - - assert first.status == STATUS_STALE_TARGET - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, "stale-binding") - assert receipt is not None - assert receipt["state"] == "rejected" - - -def test_authoritative_degraded_backend_is_terminal( - tmp_path: Path, -) -> None: - """A resolved worker plus a degraded backend is an authoritative rejection.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)], health="degraded") - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, _request(request_id="degraded"), socket_client_factory=_factory(calls) - ) - replay = submit_command( - config, _request(request_id="degraded"), socket_client_factory=_forbidden_factory - ) - - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, "degraded") - assert receipt is not None - assert receipt["state"] == "rejected" - - -def _pane_resolution_factory(kind: str, calls: list[dict[str, Any]]): - """A socket factory whose pane resolution fails in a specific way.""" - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - def make_client(config: Config) -> _FakeSocketClient: - if kind == "no_pane": - # Herdr answered, but the agent has no resolvable pane. - return _FakeSocketClient(calls, agent_get_response={"result": {"agent": {}}}) - if kind == "herdr_error_response": - # Herdr returned an authoritative error response. - return _FakeSocketClient( - calls, - agent_get_raises=HerdrErrorResponse({"message": "no such agent"}, "rid"), - ) - if kind == "unsupported": - # The resolution response was malformed / unsupported. - return _FakeSocketClient(calls, agent_get_raises=ValueError("bad agent info")) - raise AssertionError(f"unknown pane failure {kind!r}") - - return make_client - - -@pytest.mark.parametrize("kind", ["no_pane", "herdr_error_response", "unsupported"]) -def test_authoritative_pane_resolution_failure_is_terminal( - tmp_path: Path, - kind: str, -) -> None: - """A definite backend answer during pane resolution is a proven target failure. - - These are not transport read failures: Herdr answered (or the response was - unusable), and a same-ID retry would get the same answer. They must stay - terminal so the connector advances instead of retrying to its horizon. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - factory = _pane_resolution_factory(kind, calls) - request_id = f"pane-{kind}" - - first = submit_command(config, _request(request_id=request_id), socket_client_factory=factory) - replay = submit_command( - config, _request(request_id=request_id), socket_client_factory=_forbidden_factory - ) - - # A durable terminal rejection, replayed on retry, with no send. - assert first.ok is False - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert first.status in {"backend_failed", "backend_unavailable"} - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, request_id) - assert receipt is not None - assert receipt["state"] == "rejected" - # The pane read was attempted exactly once and never repeated by the replay. - assert [call["method"] for call in calls].count("agent.get") == 1 - - -def test_transient_pane_read_failure_stays_retryable( - tmp_path: Path, -) -> None: - """A pane-read timeout is a transport failure, not an authoritative answer.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - connect_ok = {"value": False} - - def flaky_factory(config: Config) -> _FakeSocketClient: - if not connect_ok["value"]: - return _FakeSocketClient( - calls, - agent_get_raises=HerdrSocketTimeoutError("pane read timeout"), - ) - return _FakeSocketClient(calls) - - first = submit_command( - config, _request(request_id="pane-timeout"), socket_client_factory=flaky_factory - ) - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_NO_RECEIPT - assert _receipt(config, "pane-timeout") is None - assert _sent_texts(calls) == [] - - connect_ok["value"] = True - recovered = submit_command( - config, _request(request_id="pane-timeout"), socket_client_factory=flaky_factory - ) - assert recovered.status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - - -def test_transient_binding_store_never_masks_a_permanent_rejection( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Once the store recovers and the target is unsuitable, the truth is terminal.""" - config = _config(tmp_path) - worker = _worker() - # No binding: the target is permanently unsupported, but the binding store - # is transiently unavailable on the first attempt. - _seed(config, [worker], []) - calls: list[dict[str, Any]] = [] - injection = _TransientInjection("binding_store_sqlite") - factory = injection.install(monkeypatch, calls) - - transient = submit_command( - config, _request(request_id="mixed"), socket_client_factory=factory - ) - assert transient.disposition == DISPOSITION_NO_RECEIPT - assert _receipt(config, "mixed") is None - - injection.armed = False - terminal = submit_command( - config, _request(request_id="mixed"), socket_client_factory=factory - ) - assert terminal.status == STATUS_BACKEND_UNSUPPORTED - assert terminal.disposition == DISPOSITION_TERMINAL_REJECTED - assert _sent_texts(calls) == [] - - -# --------------------------------------------------------------------------- -# Concurrency (barriers, not sleeps) -# --------------------------------------------------------------------------- - - -def test_concurrent_callers_with_one_transient_send_at_most_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Three same-id callers, a subset transient: one send, one terminal, no rejection.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - calls_lock = threading.Lock() - real_bindings = command_submission.list_worker_bindings - fail_budget = {"n": 2} # two of three callers hit a transient binding-store error - budget_lock = threading.Lock() - - def flaky_bindings(*a: Any, **k: Any) -> Any: - with budget_lock: - if fail_budget["n"] > 0: - fail_budget["n"] -= 1 - raise sqlite3.OperationalError("database is locked") - return real_bindings(*a, **k) - - monkeypatch.setattr(command_submission, "list_worker_bindings", flaky_bindings) - - class _SerializedClient(_FakeSocketClient): - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None): - with calls_lock: - return super().request(method, params, timeout=timeout) - - def factory(_config: Config) -> _SerializedClient: - return _SerializedClient(calls) - - barrier = threading.Barrier(3) - results: list[Any] = [] - results_lock = threading.Lock() - - def attempt() -> None: - barrier.wait(timeout=30) - envelope = submit_command( - config, _request(request_id="race"), socket_client_factory=factory - ) - with results_lock: - results.append(envelope) - - threads = [threading.Thread(target=attempt) for _ in range(3)] - for thread in threads: - thread.start() - for thread in threads: - thread.join(timeout=30) - - assert not any(thread.is_alive() for thread in threads) - assert len(results) == 3 - # At most one send, at most one terminal receipt, and no caller was durably - # rejected because of the transient binding-store failure. - assert _sent_texts(calls).count("hello") <= 1 - assert _receipt_count(config) <= 1 - for envelope in results: - assert envelope.disposition != DISPOSITION_TERMINAL_REJECTED - assert envelope.status in { - STATUS_ACCEPTED, - STATUS_PENDING, - STATUS_BACKEND_UNAVAILABLE, - } - # The transient callers can retry the same id and converge on the result. - settled = submit_command( - config, _request(request_id="race"), socket_client_factory=factory - ) - assert settled.status == STATUS_ACCEPTED - assert _sent_texts(calls).count("hello") == 1 - - -def test_transient_racing_reservation_creation_sends_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A transient caller cannot corrupt a concurrent caller's reservation+send.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - calls_lock = threading.Lock() - real_reserve = command_submission.reserve_command_request - at_reservation = threading.Event() - let_reservation_proceed = threading.Event() - - def gated_reserve(*a: Any, **k: Any) -> Any: - at_reservation.set() - assert let_reservation_proceed.wait(timeout=30) - return real_reserve(*a, **k) - - class _SerializedClient(_FakeSocketClient): - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None): - with calls_lock: - return super().request(method, params, timeout=timeout) - - def factory(_config: Config) -> _SerializedClient: - return _SerializedClient(calls) - - monkeypatch.setattr(command_submission, "reserve_command_request", gated_reserve) - - sender_result: list[Any] = [] - - def sender() -> None: - sender_result.append( - submit_command(config, _request(request_id="rr"), socket_client_factory=factory) - ) - - sender_thread = threading.Thread(target=sender, name="sender") - sender_thread.start() - assert at_reservation.wait(timeout=30), "sender never reached reservation" - - # While the sender is parked at reservation, a second caller hits a transient - # binding-store failure. It must not reserve, send, or reject durably. - real_bindings = command_submission.list_worker_bindings - - def bindings_raise(*a: Any, **k: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings_raise) - transient = submit_command( - config, _request(request_id="rr"), socket_client_factory=factory - ) - monkeypatch.setattr(command_submission, "list_worker_bindings", real_bindings) - - let_reservation_proceed.set() - sender_thread.join(timeout=30) - - assert not sender_thread.is_alive() - assert transient.disposition == DISPOSITION_NO_RECEIPT - assert transient.status == STATUS_BACKEND_UNAVAILABLE - assert sender_result[0].status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert _receipt_count(config) == 1 - - -def test_transient_racing_send_start_cas_sends_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A transient retry arriving during the sender's send-start replays in-progress.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - calls_lock = threading.Lock() - real_mark = command_submission.mark_command_send_started - at_send_start = threading.Event() - let_send_start_proceed = threading.Event() - - def gated_mark(*a: Any, **k: Any) -> Any: - at_send_start.set() - assert let_send_start_proceed.wait(timeout=30) - return real_mark(*a, **k) - - monkeypatch.setattr(command_submission, "mark_command_send_started", gated_mark) - - class _SerializedClient(_FakeSocketClient): - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None): - with calls_lock: - return super().request(method, params, timeout=timeout) - - def factory(_config: Config) -> _SerializedClient: - return _SerializedClient(calls) - - sender_result: list[Any] = [] - - def sender() -> None: - sender_result.append( - submit_command(config, _request(request_id="ss"), socket_client_factory=factory) - ) - - sender_thread = threading.Thread(target=sender, name="sender") - sender_thread.start() - assert at_send_start.wait(timeout=30), "sender never reached send-start" - - # The reservation now exists (state reserved). A retry that then hits a - # transient binding-store failure must read in-progress from that receipt, - # never a no-receipt failure and never a second send. - real_bindings = command_submission.list_worker_bindings - - def bindings_raise(*a: Any, **k: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings_raise) - retry = submit_command(config, _request(request_id="ss"), socket_client_factory=factory) - monkeypatch.setattr(command_submission, "list_worker_bindings", real_bindings) - - let_send_start_proceed.set() - sender_thread.join(timeout=30) - - assert not sender_thread.is_alive() - assert retry.status == STATUS_PENDING - assert retry.disposition == DISPOSITION_IN_PROGRESS - assert sender_result[0].status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert _receipt_count(config) == 1 - - -def test_retry_after_abandoned_reservation_with_transient_reports_in_progress( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """An abandoned reservation plus a transient retry stays in-progress, not rejected.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - # Reserve then crash before send by losing the send-start response. - real_mark = command_submission.mark_command_send_started - - def lose_send_start(*a: Any, **k: Any) -> Any: - raise HerdrSocketTimeoutError("send-start response lost") - - monkeypatch.setattr(command_submission, "mark_command_send_started", lose_send_start) - reserved = submit_command( - config, _request(request_id="abandoned"), socket_client_factory=_factory(calls) - ) - assert reserved.status == STATUS_PENDING - monkeypatch.setattr(command_submission, "mark_command_send_started", real_mark) - - # Expire the crashed owner's lease, then retry while a transient binding-store - # failure is active. The abandoned reservation must not be terminalized. - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET owner_expires_at = ? WHERE request_id = ?", - ("2020-01-01T00:00:00+00:00", "abandoned"), - ) - - def bindings_raise(*a: Any, **k: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings_raise) - retry = submit_command( - config, _request(request_id="abandoned"), socket_client_factory=_factory(calls) - ) - - assert retry.status == STATUS_PENDING - assert retry.disposition == DISPOSITION_IN_PROGRESS - assert _sent_texts(calls) == [] - receipt = _receipt(config, "abandoned") - assert receipt is not None - assert receipt["state"] == "reserved" - - -def test_process_response_loss_after_send_replays_without_second_send( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Once a send committed, a lost finish response is recovered, never re-sent.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - real_finish = command_submission.finish_command_request - finished_state: dict[str, Any] = {} - - def finish_then_lose(*a: Any, **k: Any) -> Any: - result = real_finish(*a, **k) - finished_state["result"] = result - raise HerdrSocketTimeoutError("finish response lost") - - monkeypatch.setattr(command_submission, "finish_command_request", finish_then_lose) - first = submit_command( - config, _request(request_id="resp-loss"), socket_client_factory=_factory(calls) - ) - monkeypatch.setattr(command_submission, "finish_command_request", real_finish) - - # The send happened and the receipt committed accepted; the caller only lost - # the response. A same-id retry replays it without a second send. - assert _sent_texts(calls) == ["hello"] - replay = submit_command( - config, _request(request_id="resp-loss"), socket_client_factory=_forbidden_factory - ) - assert replay.status == STATUS_ACCEPTED - assert replay.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert _receipt_count(config) == 1 - - -# --------------------------------------------------------------------------- -# Paired connector boundary: the exact tuple Herdres retries -# --------------------------------------------------------------------------- - - -@pytest.mark.parametrize("kind", TRANSIENT_KINDS) -def test_daemon_emits_the_retryable_tuple_herdres_expects( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - kind: str, -) -> None: - """The daemon JSON for a transient is the ``backend_unavailable/no_receipt`` - tuple that the Herdres client validates and the reduce loop retries under the - same request ID (see the Herdres client's ``_RETRY_DISPOSITIONS``). A durable - ``terminal_rejected`` would instead advance the offset and drop the command. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - injection = _TransientInjection(kind) - factory = injection.install(monkeypatch, calls) - payload = _request(request_id=f"paired-{kind}") - - api = TendwireDaemonAPI( - get_snapshot=lambda: Snapshot( - host_id=HOST_ID, updated_at="2026-01-01T00:00:00+00:00", workers=[worker] - ), - get_health=lambda: {"ok": True}, - submit_command=lambda params: submit_command( - config, params, socket_client_factory=factory - ), - ) - response = api.dispatch({"method": "command.submit", "params": payload, "id": "1"}) - result = response["result"] - - # The exact tuple asserted by the paired Herdres test - # ``test_backend_unavailable_authority_comes_only_from_disposition``. - assert result["ok"] is False - assert result["status"] == STATUS_BACKEND_UNAVAILABLE - assert result["disposition"] == DISPOSITION_NO_RECEIPT - assert result["request_id"] == payload["request_id"] - assert result["dry_run"] is False - assert _receipt(config, payload["request_id"]) is None - assert _sent_texts(calls) == [] - - # The same request ID, once the transient clears, produces the accepted tuple - # Herdres marks terminal -- exactly once. - injection.armed = False - accepted = api.dispatch({"method": "command.submit", "params": payload, "id": "2"}) - assert accepted["result"]["status"] == STATUS_ACCEPTED - assert accepted["result"]["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - assert _sent_texts(calls) == ["hello"] - - -# --------------------------------------------------------------------------- -# Bounded real SQLite/local-state stress fixture -# --------------------------------------------------------------------------- - - -def test_real_store_contention_never_durably_rejects_an_unsent_command( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Reproduce the original contention shape against a real store, bounded. - - Many concurrent same-id submits run against a real SQLite store while a real - SQLite ``OperationalError`` is injected into the binding-store read on a - bounded fraction of attempts -- the exact failure the stress run surfaced. - The invariant: at most one send, at most one terminal receipt, and no receipt - is ever a rejection produced solely by a pre-send transient. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - calls_lock = threading.Lock() - real_bindings = command_submission.list_worker_bindings - attempt_counter = {"n": 0} - counter_lock = threading.Lock() - - def contended_bindings(*a: Any, **k: Any) -> Any: - with counter_lock: - attempt_counter["n"] += 1 - # Fail the binding-store read on odd attempts with a real SQLite error. - fail = attempt_counter["n"] % 2 == 1 - if fail: - raise sqlite3.OperationalError("database is locked") - return real_bindings(*a, **k) - - monkeypatch.setattr(command_submission, "list_worker_bindings", contended_bindings) - - class _SerializedClient(_FakeSocketClient): - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None): - with calls_lock: - return super().request(method, params, timeout=timeout) - - def factory(_config: Config) -> _SerializedClient: - return _SerializedClient(calls) - - barrier = threading.Barrier(8) - results: list[Any] = [] - results_lock = threading.Lock() - - def attempt() -> None: - barrier.wait(timeout=30) - envelope = submit_command( - config, _request(request_id="contended"), socket_client_factory=factory - ) - with results_lock: - results.append(envelope) - - threads = [threading.Thread(target=attempt) for _ in range(8)] - for thread in threads: - thread.start() - for thread in threads: - thread.join(timeout=30) - - assert not any(thread.is_alive() for thread in threads) - assert len(results) == 8 - assert _sent_texts(calls).count("hello") <= 1 - assert _receipt_count(config) <= 1 - # No durable rejection was ever manufactured from the transient store errors. - receipt = _receipt(config, "contended") - if receipt is not None: - assert receipt["state"] in {"reserved", "send_started", "accepted"} - for envelope in results: - if envelope.disposition == DISPOSITION_TERMINAL_REJECTED: - pytest.fail("a pre-send transient produced a durable rejection") - - # After contention clears, exactly one accepted result stands. - monkeypatch.setattr(command_submission, "list_worker_bindings", real_bindings) - settled = submit_command( - config, _request(request_id="contended"), socket_client_factory=factory - ) - assert settled.status == STATUS_ACCEPTED - assert _sent_texts(calls).count("hello") == 1 diff --git a/tests/test_command_replay_authority.py b/tests/test_command_replay_authority.py deleted file mode 100644 index d76022d..0000000 --- a/tests/test_command_replay_authority.py +++ /dev/null @@ -1,1442 +0,0 @@ -"""Receipt authority over mutable worker resolution for retried commands. - -Goal 11 made ``request_id`` the sole idempotency key. This module proves the -follow-up property: an existing receipt decides its own retry from stored -evidence, before any mutable worker snapshot is consulted. A worker that -vanishes, is renamed, or is recycled must never turn a live receipt into a -no-receipt failure, and must never let one request mutate the backend twice. -""" - -from __future__ import annotations - -import json -import sqlite3 -import threading -from pathlib import Path -from typing import Any - -import pytest - -import tendwire.command_submission as command_submission -import tendwire.store.sqlite as store_sqlite - -from tendwire.backends.herdr_socket import HerdrSocketTimeoutError -from tendwire.command_submission import replay_command_receipt, submit_command -from tendwire.config import Config -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - DISPOSITION_TERMINAL_ACCEPTED, - DISPOSITION_TERMINAL_REJECTED, - DISPOSITION_TERMINAL_UNCERTAIN, - STATUS_ACCEPTED, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DUPLICATE_REQUEST, - STATUS_INVALID_REQUEST, - STATUS_PENDING, - STATUS_REQUEST_STATE_UNCERTAIN, - CommandRequest, - build_selector_proof, - is_selector_proof, -) -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.daemon_api import TendwireDaemonAPI -from tendwire.store.sqlite import ( - get_command_request, - init_store, - run_store_maintenance, - save_snapshot, - store_status, - upsert_worker_bindings, -) - - -HOST_ID = "cmd-host" - -RECEIPT_STATES = ("reserved", "send_started", "accepted", "rejected", "uncertain") -SELECTOR_KINDS = ( - "worker_id", - "worker_id_fingerprint", - "name", - "space_id", - "name_and_space", -) - -_EXPECTED_REPLAY = { - "reserved": (STATUS_PENDING, DISPOSITION_IN_PROGRESS), - "send_started": (STATUS_PENDING, DISPOSITION_IN_PROGRESS), - "accepted": (STATUS_ACCEPTED, DISPOSITION_TERMINAL_ACCEPTED), - "rejected": (STATUS_BACKEND_UNSUPPORTED, DISPOSITION_TERMINAL_REJECTED), - "uncertain": (STATUS_REQUEST_STATE_UNCERTAIN, DISPOSITION_TERMINAL_UNCERTAIN), -} - -# What reaching each state costs at the backend. Only a send that was actually -# attempted puts text on a pane; the rest fail earlier. -_SENDS_TO_REACH = { - "reserved": [], - "send_started": [], - "accepted": ["hello"], - "rejected": [], - "uncertain": ["hello"], -} - - -def _config(tmp_path: Path) -> Config: - return Config( - host_id=HOST_ID, - data_dir=tmp_path, - db_path=tmp_path / "commands.db", - herdr_backend="socket", - herdr_timeout_seconds=5.0, - ) - - -def _worker( - *, - worker_id: str = "w-1", - name: str = "Alpha", - space_id: str | None = "space-1", - status: str = "active", -) -> Worker: - return Worker(id=worker_id, name=name, status=status, space_id=space_id) - - -def _binding(worker: Worker, *, sendable: bool = True) -> WorkerBinding: - # Each worker owns a distinct private route. Sharing one would make every - # worker an ambiguous backend target and mask what these tests measure. - return WorkerBinding( - host_id=HOST_ID, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value=f"agent-{worker.id}", - turn_target_kind="pane_id", - turn_target_value=f"pane-{worker.id}", - sendable=sendable, - reason=None if sendable else "not_sendable", - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint=f"private-{worker.id}", - ) - - -def _health(status: str = "healthy") -> BackendHealth: - return BackendHealth( - name="herdr", - status=status, - outcome="healthy_non_empty" if status == "healthy" else "timeout", - observed_at="2026-01-01T00:00:00+00:00", - counts={"workers": 1}, - ) - - -def _snapshot( - workers: list[Worker], - *, - health: str = "healthy", - updated_at: str = "2026-01-01T00:00:00+00:00", -) -> Snapshot: - return Snapshot( - host_id=HOST_ID, - updated_at=updated_at, - workers=workers, - backend_health=[_health(health)], - ) - - -def _seed( - config: Config, - workers: list[Worker], - bindings: list[WorkerBinding], -) -> None: - assert config.db_path is not None - init_store(config.db_path) - save_snapshot(config.db_path, _snapshot(workers)) - if bindings: - upsert_worker_bindings(config.db_path, bindings) - - -def _remove_workers(config: Config, *, health: str = "healthy") -> None: - """Publish a newer authoritative snapshot in which the worker is gone.""" - assert config.db_path is not None - save_snapshot( - config.db_path, - _snapshot([], health=health, updated_at="2026-01-01T00:05:00+00:00"), - ) - - -def _selector(kind: str, worker: Worker) -> dict[str, Any]: - return { - "worker_id": {"worker_id": worker.id}, - "worker_id_fingerprint": { - "worker_id": worker.id, - "worker_fingerprint": worker.fingerprint, - }, - "name": {"name": worker.name}, - "space_id": {"space_id": worker.space_id}, - "name_and_space": {"name": worker.name, "space_id": worker.space_id}, - }[kind] - - -def _request( - *, - request_id: str, - target: dict[str, Any], - text: str = "hello", -) -> dict[str, Any]: - return { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": dict(target), - "instruction": {"text": text}, - } - - -class _FakeSocketClient: - def __init__( - self, - calls: list[dict[str, Any]], - *, - raises: BaseException | None = None, - ) -> None: - self.calls = calls - self.raises = raises - - def connect(self) -> "_FakeSocketClient": - return self - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if self.raises is not None and method == "agent.prompt": - raise self.raises - if method == "agent.get": - return {"result": {"agent": {"pane_id": "pane-secret"}}} - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": "Completed previous turn.\n── status: idle ──"}, - } - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": "pane-secret"}, - "delivery": "submitted", - } - return {"accepted": True} - - def close(self) -> None: - return None - - -def _factory(calls: list[dict[str, Any]], *, raises: BaseException | None = None): - def make_client(config: Config) -> _FakeSocketClient: - return _FakeSocketClient(calls, raises=raises) - - return make_client - - -def _forbidden_factory(config: Config) -> Any: - pytest.fail("a receipt replay must not create a socket client") - - -def _receipt(config: Config, request_id: str) -> dict[str, Any]: - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, request_id) - assert receipt is not None - return receipt - - -def _receipt_rows(config: Config) -> list[tuple[Any, ...]]: - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - return conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - - -def _sent_texts(calls: list[dict[str, Any]]) -> list[str]: - return [ - str(call["params"].get("text")) - for call in calls - if call["method"] == "agent.prompt" - ] - - -def _drive_to_state( - config: Config, - payload: dict[str, Any], - state: str, - calls: list[dict[str, Any]], - monkeypatch: pytest.MonkeyPatch, -) -> Any: - """Submit one request and leave its receipt in the requested state. - - ``rejected`` is reached through an unsendable private binding, which the - submission path terminalizes before any send; the caller seeds that binding. - """ - if state == "accepted": - return submit_command(config, payload, socket_client_factory=_factory(calls)) - if state == "rejected": - return submit_command(config, payload, socket_client_factory=_factory(calls)) - if state == "uncertain": - return submit_command( - config, - payload, - socket_client_factory=_factory( - calls, - raises=HerdrSocketTimeoutError("send response lost"), - ), - ) - - real_mark = command_submission.mark_command_send_started - - def lose_send_start(*args: Any, **kwargs: Any) -> Any: - if state == "send_started": - result = real_mark(*args, **kwargs) - assert result["status"] == "send_started" - raise HerdrSocketTimeoutError("send-start response lost") - - monkeypatch.setattr(command_submission, "mark_command_send_started", lose_send_start) - envelope = submit_command(config, payload, socket_client_factory=_factory(calls)) - monkeypatch.setattr(command_submission, "mark_command_send_started", real_mark) - return envelope - - -@pytest.mark.parametrize("selector_kind", SELECTOR_KINDS) -@pytest.mark.parametrize("receipt_state", RECEIPT_STATES) -def test_exact_retry_replays_receipt_after_healthy_worker_removal( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - receipt_state: str, - selector_kind: str, -) -> None: - """Every receipt state replays for every selector shape once the worker is gone.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker, sendable=receipt_state != "rejected")]) - request_id = f"replay-{receipt_state}-{selector_kind}" - payload = _request(request_id=request_id, target=_selector(selector_kind, worker)) - calls: list[dict[str, Any]] = [] - - first = _drive_to_state(config, payload, receipt_state, calls, monkeypatch) - stored = _receipt(config, request_id) - assert stored["state"] == receipt_state - - _remove_workers(config) - rows_before = _receipt_rows(config) - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert (retry.status, retry.disposition) == _EXPECTED_REPLAY[receipt_state] - if receipt_state in {"accepted", "rejected"}: - assert retry.to_dict() == first.to_dict() - # The retry itself sent nothing and left the authoritative receipt intact. - assert _sent_texts(calls) == _SENDS_TO_REACH[receipt_state] - assert _receipt_rows(config) == rows_before - - -@pytest.mark.parametrize("selector_kind", SELECTOR_KINDS) -def test_exact_retry_replays_accepted_receipt_while_backend_is_degraded( - tmp_path: Path, - selector_kind: str, -) -> None: - """A degraded observation cannot override truth the receipt already holds.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - request_id = f"degraded-{selector_kind}" - payload = _request(request_id=request_id, target=_selector(selector_kind, worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - _remove_workers(config, health="degraded") - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.to_dict() == accepted.to_dict() - assert _sent_texts(calls) == ["hello"] - - -def test_fingerprint_only_targets_cannot_claim_another_workers_receipt( - tmp_path: Path, -) -> None: - """The collision that blocked 2edc6cc: two fingerprints, one request ID. - - A fingerprint is a mutable precondition, not identity, so the selector proof - excludes it. If a fingerprint could stand alone as the whole target, every - such target would share one proof -- and reusing the request ID with a - fingerprint naming a different worker would replay the first worker's - accepted result. Rejecting the shape outright is what closes that hole. - """ - config = _config(tmp_path) - first_worker = _worker() - second_worker = _worker(worker_id="w-2", name="Beta", space_id="space-2") - _seed( - config, - [first_worker, second_worker], - [_binding(first_worker), _binding(second_worker)], - ) - assert first_worker.fingerprint != second_worker.fingerprint - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request( - request_id="fingerprint-collision", - target={"worker_fingerprint": first_worker.fingerprint}, - ), - socket_client_factory=_factory(calls), - ) - changed = submit_command( - config, - _request( - request_id="fingerprint-collision", - target={"worker_fingerprint": second_worker.fingerprint}, - ), - socket_client_factory=_factory(calls), - ) - - # Under 2edc6cc the first was accepted and the second replayed its stored - # terminal_accepted body: one worker's result claimed for another. - for envelope in (first, changed): - assert envelope.ok is False - assert envelope.status == STATUS_INVALID_REQUEST - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert _sent_texts(calls) == [] - assert _receipt_rows(config) == [] - - -def test_fingerprint_only_target_does_no_store_source_or_backend_work( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Rejection happens in the parser, before anything durable or observable.""" - config = _config(tmp_path) - touched: list[str] = [] - - def forbidden(*args: Any, **kwargs: Any) -> Any: - touched.append("io") - raise AssertionError("an invalid target must not reach store or source work") - - monkeypatch.setattr(command_submission, "get_command_request", forbidden) - monkeypatch.setattr(command_submission, "_current_snapshot", forbidden) - monkeypatch.setattr(command_submission, "latest_snapshot", forbidden) - monkeypatch.setattr(command_submission, "reserve_command_request", forbidden) - monkeypatch.setattr( - "tendwire.command_submission.project_from_observations", - forbidden, - ) - - envelope = submit_command( - config, - _request( - request_id="fingerprint-only", - target={"worker_fingerprint": "fingerprint-A"}, - ), - socket_client_factory=forbidden, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_INVALID_REQUEST - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert touched == [] - assert config.db_path is not None - assert not config.db_path.exists() - - -def test_fingerprint_only_target_is_rejected_by_daemon_and_read_only_replay( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request( - request_id="fingerprint-only", - target={"worker_fingerprint": worker.fingerprint}, - ) - api = TendwireDaemonAPI( - get_snapshot=lambda: _snapshot([worker]), - get_health=lambda: {"ok": True}, - submit_command=lambda params: submit_command( - config, - params, - socket_client_factory=_forbidden_factory, - ), - ) - - response = api.dispatch({"method": "command.submit", "params": payload, "id": "1"}) - - assert response["result"]["ok"] is False - assert response["result"]["status"] == STATUS_INVALID_REQUEST - assert response["result"]["disposition"] == DISPOSITION_NO_RECEIPT - # The rejection names the rule it broke without echoing the observation the - # caller sent, and it leaves no durable trace of the request. - rendered = json.dumps(response) - assert worker.fingerprint not in rendered - assert "worker_fingerprint" in response["result"]["error"]["message"] - assert response["result"]["error"]["details"]["allowed"] == [ - "name", - "space_id", - "stable_key", - "worker_id", - ] - # The read-only response-loss path cannot resolve it either, and must never - # invent a receipt for a request that could not have created one. - assert replay_command_receipt(config, payload) is None - assert _receipt_rows(config) == [] - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - command_events = conn.execute( - "SELECT COUNT(*) FROM events WHERE aggregate_type = 'command_request'" - ).fetchone()[0] - assert command_events == 0 - - -@pytest.mark.parametrize("selector_kind", ["name", "space_id", "name_and_space"]) -def test_refreshed_fingerprint_beside_a_stable_selector_replays_stored_result( - tmp_path: Path, - selector_kind: str, -) -> None: - """A refreshed fingerprint is a precondition, not a different command. - - This is the alias path, so equivalence is proven by the stored selector - proof -- which is exactly the evidence that must ignore the fingerprint. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - stable = _selector(selector_kind, worker) - payload = _request( - request_id=f"fingerprint-beside-{selector_kind}", - target={**stable, "worker_fingerprint": worker.fingerprint}, - ) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - - # The worker is re-observed with fresh data, so its fingerprint moves. - refreshed_worker = _worker(status="waiting") - assert refreshed_worker.fingerprint != worker.fingerprint - assert config.db_path is not None - save_snapshot( - config.db_path, - _snapshot([refreshed_worker], updated_at="2026-01-01T00:06:00+00:00"), - ) - refreshed = _request( - request_id=f"fingerprint-beside-{selector_kind}", - target={**stable, "worker_fingerprint": refreshed_worker.fingerprint}, - ) - - retry = submit_command(config, refreshed, socket_client_factory=_forbidden_factory) - - assert retry.to_dict() == accepted.to_dict() - assert _sent_texts(calls) == ["hello"] - - -def test_refreshed_worker_fingerprint_stays_noncanonical_on_retry( - tmp_path: Path, -) -> None: - """A worker ID that survives with new observation data is the same target.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request( - request_id="fingerprint-churn", - target={"worker_id": worker.id, "worker_fingerprint": worker.fingerprint}, - ) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - - # The worker keeps its public ID but every observed attribute changes. - recycled = _worker(name="Renamed", space_id="space-9", status="waiting") - assert config.db_path is not None - save_snapshot( - config.db_path, - _snapshot([recycled], updated_at="2026-01-01T00:06:00+00:00"), - ) - refreshed = _request( - request_id="fingerprint-churn", - target={"worker_id": worker.id, "worker_fingerprint": recycled.fingerprint}, - ) - - retry = submit_command(config, refreshed, socket_client_factory=_forbidden_factory) - - assert retry.to_dict() == accepted.to_dict() - assert _sent_texts(calls) == ["hello"] - - -@pytest.mark.parametrize("receipt_state", RECEIPT_STATES) -@pytest.mark.parametrize( - ("collision", "changed"), - [ - ("instruction", {"instruction": {"text": "different"}}), - ("worker_id", {"target": {"worker_id": "w-2"}}), - ("name", {"target": {"name": "Beta"}}), - ("space_id", {"target": {"space_id": "space-2"}}), - ("action", {"action": "answer_pending"}), - ], -) -def test_changed_request_cannot_claim_a_stored_receipt( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - receipt_state: str, - collision: str, - changed: dict[str, Any], -) -> None: - """Reusing a request ID with any changed canonical field mutates nothing.""" - config = _config(tmp_path) - worker = _worker() - other = _worker(worker_id="w-2", name="Beta", space_id="space-2") - _seed( - config, - [worker, other], - [ - _binding(worker, sendable=receipt_state != "rejected"), - _binding(other, sendable=receipt_state != "rejected"), - ], - ) - request_id = f"collision-{receipt_state}-{collision}" - payload = _request(request_id=request_id, target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, receipt_state, calls, monkeypatch) - assert _receipt(config, request_id)["state"] == receipt_state - sent_before = list(_sent_texts(calls)) - rows_before = _receipt_rows(config) - - reused = _request(request_id=request_id, target=_selector("name", worker)) - reused.update(changed) - if collision == "action": - reused.pop("target", None) - reused.pop("instruction", None) - reused["params"] = { - "pending_id": "pending-public", - "pending_fingerprint": "revision-public", - "choice_id": "choice-public", - } - - conflict = submit_command( - config, - reused, - socket_client_factory=lambda _config: pytest.fail( - "a changed request must not create a socket client" - ), - ) - - assert conflict.status == STATUS_DUPLICATE_REQUEST - assert conflict.disposition == DISPOSITION_TERMINAL_REJECTED - assert _sent_texts(calls) == sent_before - assert _receipt_rows(config) == rows_before - - -@pytest.mark.parametrize("receipt_state", RECEIPT_STATES) -def test_unsupported_stored_selector_proof_fails_closed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - receipt_state: str, -) -> None: - """Evidence this version cannot read decides nothing rather than deciding wrong.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker, sendable=receipt_state != "rejected")]) - request_id = f"proof-{receipt_state}" - payload = _request(request_id=request_id, target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, receipt_state, calls, monkeypatch) - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET selector_proof = ? WHERE request_id = ?", - ("v9:not-a-supported-proof", request_id), - ) - _remove_workers(config) - sent_before = list(_sent_texts(calls)) - rows_before = _receipt_rows(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert _sent_texts(calls) == sent_before - assert _receipt_rows(config) == rows_before - - -def test_legacy_receipt_without_selector_proof_fails_closed_on_alias_retry( - tmp_path: Path, -) -> None: - """A migrated v12 alias receipt is never guessed into a replay.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="legacy-alias", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - # Reproduce what the v12 migration leaves behind: a canonical receipt whose - # original selector spelling was never recorded. - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute("UPDATE command_receipts SET selector_proof = ''") - _remove_workers(config) - rows_before = _receipt_rows(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert _sent_texts(calls) == ["hello"] - assert _receipt_rows(config) == rows_before - - -def test_legacy_alias_receipt_fails_closed_when_current_authority_store_raises( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A transient authority read cannot erase a receipt or permit a resend.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="legacy-store-race", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute("UPDATE command_receipts SET selector_proof = ''") - - def contended_snapshot(*args: Any, **kwargs: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "latest_snapshot", contended_snapshot) - rows_before = _receipt_rows(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert _sent_texts(calls) == ["hello"] - assert _receipt_rows(config) == rows_before - - -def test_legacy_receipt_without_selector_proof_replays_explicit_worker_id( - tmp_path: Path, -) -> None: - """Explicit worker IDs still replay from a legacy receipt: the ID is the proof.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="legacy-explicit", target={"worker_id": worker.id}) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute("UPDATE command_receipts SET selector_proof = ''") - _remove_workers(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.to_dict() == accepted.to_dict() - assert _sent_texts(calls) == ["hello"] - - -def test_abandoned_reservation_is_redriven_not_replayed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """An expired reservation lease is the existing crash-recovery path, not a replay.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="abandoned", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, "reserved", calls, monkeypatch) - assert _receipt(config, "abandoned")["state"] == "reserved" - assert _sent_texts(calls) == [] - - # Expire the owner lease the crashed sender left behind. - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET owner_expires_at = ? WHERE request_id = ?", - ("2020-01-01T00:00:00+00:00", "abandoned"), - ) - - recovered = submit_command(config, payload, socket_client_factory=_factory(calls)) - - assert recovered.status == STATUS_ACCEPTED - assert recovered.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert _receipt(config, "abandoned")["state"] == "accepted" - - -def test_abandoned_reservation_reports_in_progress_when_worker_is_gone( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A vanished worker never restates a stored reservation as a no-receipt failure.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="abandoned-gone", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, "reserved", calls, monkeypatch) - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET owner_expires_at = ? WHERE request_id = ?", - ("2020-01-01T00:00:00+00:00", "abandoned-gone"), - ) - _remove_workers(config) - rows_before = _receipt_rows(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.status == STATUS_PENDING - assert retry.disposition == DISPOSITION_IN_PROGRESS - assert _sent_texts(calls) == [] - assert _receipt_rows(config) == rows_before - - -def test_abandoned_reservation_conflicts_when_selector_now_names_another_worker( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Re-driving an abandoned send never redirects it to a different worker.""" - config = _config(tmp_path) - worker = _worker() - other = _worker(worker_id="w-2", name="Beta", space_id="space-2") - _seed(config, [worker, other], [_binding(worker), _binding(other)]) - payload = _request(request_id="abandoned-drift", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, "reserved", calls, monkeypatch) - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET owner_expires_at = ?, selector_proof = ''", - ("2020-01-01T00:00:00+00:00",), - ) - # "Alpha" now names the other worker, so the abandoned reservation's target - # can no longer be reached by the spelling that created it. - save_snapshot( - config.db_path, - _snapshot( - [_worker(worker_id="w-2", name="Alpha", space_id="space-2")], - updated_at="2026-01-01T00:07:00+00:00", - ), - ) - rows_before = _receipt_rows(config) - - conflict = submit_command( - config, - payload, - socket_client_factory=lambda _config: pytest.fail( - "a redirected takeover must not create a socket client" - ), - ) - - assert conflict.status == STATUS_DUPLICATE_REQUEST - assert _sent_texts(calls) == [] - assert _receipt_rows(config) == rows_before - - -@pytest.mark.parametrize("selector_kind", SELECTOR_KINDS) -@pytest.mark.parametrize("stage", ["reservation", "send_start", "completion"]) -def test_exact_retry_racing_a_live_mutation_never_sends_twice( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - stage: str, - selector_kind: str, -) -> None: - """A retry landing mid-mutation replays the receipt, even as the worker vanishes. - - The sender is held at one stage of the state machine while an exact retry - runs against a snapshot that no longer contains the worker. The retry must - read in-progress from the receipt, open no socket, and leave the in-flight - mutation to finish exactly once. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - request_id = f"race-{stage}-{selector_kind}" - payload = _request(request_id=request_id, target=_selector(selector_kind, worker)) - calls: list[dict[str, Any]] = [] - - reached = threading.Event() - release = threading.Event() - - def pause() -> None: - reached.set() - assert release.wait(timeout=30), "the racing retry never released the sender" - - real_reserve = command_submission.reserve_command_request - real_mark = command_submission.mark_command_send_started - real_finish = command_submission.finish_command_request - - def reserve(*args: Any, **kwargs: Any) -> Any: - reserved = real_reserve(*args, **kwargs) - if stage == "reservation": - pause() - return reserved - - def mark(*args: Any, **kwargs: Any) -> Any: - started = real_mark(*args, **kwargs) - if stage == "send_start": - pause() - return started - - def finish(*args: Any, **kwargs: Any) -> Any: - if stage == "completion": - pause() - return real_finish(*args, **kwargs) - - monkeypatch.setattr(command_submission, "reserve_command_request", reserve) - monkeypatch.setattr(command_submission, "mark_command_send_started", mark) - monkeypatch.setattr(command_submission, "finish_command_request", finish) - - sent: list[Any] = [] - - def send() -> None: - sent.append( - submit_command(config, payload, socket_client_factory=_factory(calls)) - ) - - sender = threading.Thread(target=send, name=f"sender-{stage}") - sender.start() - assert reached.wait(timeout=30), "the sender never reached the raced stage" - - # Healthy authority loses the worker while the mutation is still in flight. - _remove_workers(config) - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - release.set() - sender.join(timeout=30) - - assert not sender.is_alive() - assert len(sent) == 1 - assert retry.status == STATUS_PENDING - assert retry.disposition == DISPOSITION_IN_PROGRESS - assert sent[0].status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert len(_receipt_rows(config)) == 1 - assert _receipt(config, request_id)["state"] == "accepted" - - # Once the race settles, the retry converges on the one stored result. - settled = submit_command(config, payload, socket_client_factory=_forbidden_factory) - assert settled.to_dict() == sent[0].to_dict() - assert _sent_texts(calls) == ["hello"] - - -def test_retry_racing_receipt_retention_never_sends_twice( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Retention deleting a receipt mid-replay must not reopen the mutation.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="retention-race", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - - real_reserve = command_submission.reserve_terminal_command_replay - - def delete_then_reserve(*args: Any, **kwargs: Any) -> Any: - # Retention wins the race between reading the receipt and replaying it. - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute("DELETE FROM command_receipts") - return real_reserve(*args, **kwargs) - - monkeypatch.setattr( - command_submission, - "reserve_terminal_command_replay", - delete_then_reserve, - ) - _remove_workers(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - # The accepted body is gone, so the only honest answer is terminal - # uncertainty. It must never become a fresh send. - assert retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert _sent_texts(calls) == ["hello"] - restored = _receipt(config, "retention-race") - assert restored["state"] == "uncertain" - # The rebuilt row keeps the original spelling's evidence. - assert is_selector_proof(restored["selector_proof"]) - assert restored["selector_proof"] == build_selector_proof( - CommandRequest.from_dict(payload) - ) - - -def test_read_only_replay_honors_selector_proof_without_observing( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """The CLI's response-loss path replays an exact alias without any observation.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="read-only", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - monkeypatch.setattr( - command_submission, - "_current_snapshot", - lambda _config: pytest.fail("read-only replay must not consult authority"), - ) - rows_before = _receipt_rows(config) - - exact = replay_command_receipt(config, payload) - changed = _request(request_id="read-only", target={"name": "Beta"}) - unprovable = replay_command_receipt(config, changed) - - assert exact is not None - assert exact.to_dict() == accepted.to_dict() - # Proving a different spelling would need an observation this path may not - # make, so it stays unresolved rather than guessing. - assert unprovable is None - assert _receipt_rows(config) == rows_before - - -# --------------------------------------------------------------------------- -# Migration -# --------------------------------------------------------------------------- - -# A literal, frozen copy of the schema-v12 receipt table. It must not be rebuilt -# from the live DDL: the point is to migrate what v12 actually wrote. -_V12_COMMAND_RECEIPTS_TABLE = """ -CREATE TABLE command_receipts ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - request_id TEXT NOT NULL, - action TEXT NOT NULL, - canonical_version INTEGER NOT NULL CHECK (canonical_version >= 0), - canonical_fingerprint TEXT NOT NULL, - canonical_request_json TEXT NOT NULL, - public_worker_id TEXT NOT NULL, - state TEXT NOT NULL CHECK ( - state IN ('reserved', 'send_started', 'accepted', 'rejected', 'uncertain') - ), - status TEXT NOT NULL, - result_json TEXT NOT NULL, - owner_token_hash TEXT NOT NULL DEFAULT '', - owner_expires_at TEXT, - binding_fingerprint TEXT, - created_at TEXT NOT NULL, - reserved_at TEXT NOT NULL, - send_started_at TEXT, - terminal_at TEXT, - updated_at TEXT NOT NULL, - legacy_collision INTEGER NOT NULL DEFAULT 0 CHECK (legacy_collision IN (0, 1)), - legacy_collision_count INTEGER NOT NULL DEFAULT 0 CHECK ( - legacy_collision_count >= 0 - ), - CHECK ( - ( - state IN ('reserved', 'send_started') - AND terminal_at IS NULL - AND owner_token_hash <> '' - ) - OR ( - state IN ('accepted', 'rejected', 'uncertain') - AND terminal_at IS NOT NULL - AND owner_token_hash = '' - AND owner_expires_at IS NULL - ) - ), - CHECK (state NOT IN ('reserved', 'send_started') OR status = 'pending'), - CHECK ( - state != 'accepted' - OR (status = 'accepted' AND send_started_at IS NOT NULL) - ), - CHECK (state != 'uncertain' OR status = 'request_state_uncertain'), - CHECK ( - state != 'rejected' - OR status NOT IN ('pending', 'accepted', 'request_state_uncertain') - ), - CHECK ( - legacy_collision = 0 - OR (state = 'uncertain' AND legacy_collision_count >= 2) - ) -); -""" - -_V12_ACTIVE = ( - "host-a", - "v12-active", - "send_instruction", - 1, - "fingerprint-active", - '{"action":"send_instruction"}', - "w-1", - "reserved", - "pending", - '{"ok":false,"status":"pending"}', - "owner-hash", - "2026-01-01T00:00:30+00:00", - None, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - None, - None, - "2026-01-01T00:00:00+00:00", -) -_V12_TERMINAL = ( - "host-a", - "v12-terminal", - "send_instruction", - 1, - "fingerprint-terminal", - '{"action":"send_instruction"}', - "w-2", - "accepted", - "accepted", - '{"ok":true,"status":"accepted"}', - "", - None, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - "2026-01-01T00:00:02+00:00", - "2026-01-01T00:00:02+00:00", -) - - -def _write_v12_store(db_path: Path) -> None: - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - _V12_COMMAND_RECEIPTS_TABLE + store_sqlite.CREATE_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - _V12_ACTIVE, - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - created_at, reserved_at, send_started_at, terminal_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - _V12_TERMINAL, - ) - conn.execute("PRAGMA user_version = 12") - - -def test_v12_receipts_migrate_to_empty_selector_proof(tmp_path: Path) -> None: - """Active and terminal v12 receipts survive with no invented selector evidence.""" - db_path = tmp_path / "v12.db" - _write_v12_store(db_path) - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert ( - int(conn.execute("PRAGMA user_version").fetchone()[0]) - == store_sqlite.STORE_SCHEMA_VERSION - ) - proofs = conn.execute( - "SELECT request_id, selector_proof FROM command_receipts ORDER BY request_id" - ).fetchall() - # A v12 row records the worker a request resolved to, never how it was - # spelled. Deriving a proof from that would let a changed target replay it. - assert proofs == [("v12-active", ""), ("v12-terminal", "")] - - active = get_command_request(db_path, "host-a", "v12-active") - terminal = get_command_request(db_path, "host-a", "v12-terminal") - assert active is not None and terminal is not None - assert (active["state"], active["status"]) == ("reserved", "pending") - assert (terminal["state"], terminal["status"]) == ("accepted", "accepted") - assert terminal["result_json"] == '{"ok":true,"status":"accepted"}' - - -def test_v12_migration_is_idempotent_across_reruns(tmp_path: Path) -> None: - db_path = tmp_path / "v12-idempotent.db" - _write_v12_store(db_path) - - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - first = conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - init_store(db_path) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - second = conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - version = int(conn.execute("PRAGMA user_version").fetchone()[0]) - - assert first == second - assert version == store_sqlite.STORE_SCHEMA_VERSION - - -def test_v12_migration_rolls_back_without_partial_schema( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A failing v13 transition leaves the v12 store exactly as it was.""" - db_path = tmp_path / "v12-rollback.db" - _write_v12_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - before = conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - - real_migrate = store_sqlite._migrate_v12_to_v13_conn - - def failing_migrate(conn: sqlite3.Connection) -> None: - real_migrate(conn) - raise RuntimeError("v13 migration failed after adding the column") - - monkeypatch.setattr( - store_sqlite, - "MIGRATIONS", - tuple( - store_sqlite.Migration(item.from_version, item.to_version, failing_migrate) - if item.to_version == 13 - else item - for item in store_sqlite.MIGRATIONS - ), - ) - - with pytest.raises(RuntimeError): - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert int(conn.execute("PRAGMA user_version").fetchone()[0]) == 12 - columns = { - str(row[1]) - for row in conn.execute("PRAGMA table_info(command_receipts)").fetchall() - } - assert "selector_proof" not in columns - assert ( - conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - == before - ) - - -def _downgrade_store_to_v12(db_path: Path) -> None: - """Rebuild a live store as schema v12, dropping every selector proof.""" - with sqlite3.connect(str(db_path)) as conn: - conn.execute("ALTER TABLE command_receipts RENAME TO command_receipts_v13") - conn.executescript(_V12_COMMAND_RECEIPTS_TABLE) - conn.execute( - """ - INSERT INTO command_receipts ( - id, host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at, legacy_collision, legacy_collision_count - ) - SELECT - id, host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at, legacy_collision, legacy_collision_count - FROM command_receipts_v13 - """ - ) - conn.execute("DROP TABLE command_receipts_v13") - for statement in store_sqlite.CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - conn.execute("PRAGMA user_version = 12") - - -def test_migrated_v12_store_proves_new_requests_but_never_old_ones( - tmp_path: Path, -) -> None: - """Migration is conservative: it earns proofs forward, it does not backfill them.""" - config = _config(tmp_path) - assert config.db_path is not None - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - legacy_payload = _request( - request_id="pre-migration", - target=_selector("name", worker), - ) - legacy_accepted = submit_command( - config, - legacy_payload, - socket_client_factory=_factory(calls), - ) - assert legacy_accepted.status == STATUS_ACCEPTED - - _downgrade_store_to_v12(config.db_path) - init_store(config.db_path) - - with sqlite3.connect(str(config.db_path)) as conn: - assert ( - int(conn.execute("PRAGMA user_version").fetchone()[0]) - == store_sqlite.STORE_SCHEMA_VERSION - ) - assert _receipt(config, "pre-migration")["selector_proof"] == "" - - fresh_payload = _request( - request_id="post-migration", - target=_selector("name", worker), - ) - fresh_accepted = submit_command( - config, - fresh_payload, - socket_client_factory=_factory(calls), - ) - assert fresh_accepted.status == STATUS_ACCEPTED - assert is_selector_proof(_receipt(config, "post-migration")["selector_proof"]) - - _remove_workers(config) - legacy_retry = submit_command( - config, - legacy_payload, - socket_client_factory=_forbidden_factory, - ) - fresh_retry = submit_command( - config, - fresh_payload, - socket_client_factory=_forbidden_factory, - ) - - # The pre-migration alias has no evidence of how it was spelled, so it fails - # closed. The post-migration one carries its own proof and replays. - assert legacy_retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert legacy_retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert fresh_retry.to_dict() == fresh_accepted.to_dict() - assert _sent_texts(calls) == ["hello", "hello"] - - -# --------------------------------------------------------------------------- -# Public boundary -# --------------------------------------------------------------------------- - - -def _assert_no_selector_proof(value: Any, proof: str, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - assert "selector" not in str(key).lower().replace("-", "_"), ( - f"selector evidence leaked at {path}.{key}" - ) - _assert_no_selector_proof(item, proof, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_selector_proof(item, proof, f"{path}[{index}]") - elif isinstance(value, str): - assert proof not in value, f"selector proof leaked at {path}" - - -def test_selector_proof_never_reaches_a_public_surface(tmp_path: Path) -> None: - """The proof stays private evidence: not in envelopes, events, or the audit row.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="boundary", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - _remove_workers(config) - replay = submit_command(config, payload, socket_client_factory=_forbidden_factory) - proof = _receipt(config, "boundary")["selector_proof"] - assert is_selector_proof(proof) - - for envelope in (accepted, replay): - _assert_no_selector_proof(envelope.to_dict(), proof) - _assert_no_selector_proof(json.loads(envelope.to_json()), proof) - - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - events = conn.execute("SELECT payload_json FROM events ORDER BY id").fetchall() - audit = conn.execute("SELECT * FROM commands ORDER BY id").fetchall() - audit_columns = { - str(row[1]) for row in conn.execute("PRAGMA table_info(commands)").fetchall() - } - assert events - for (payload_json,) in events: - _assert_no_selector_proof(json.loads(payload_json), proof) - assert proof not in payload_json - # The non-authoritative audit projection must not carry the proof at all. - assert "selector_proof" not in audit_columns - for row in audit: - assert proof not in json.dumps(row, default=str) - - # Daemon JSON, and the health and maintenance surfaces that summarize - # command receipts, are built from the same receipts and must stay clean. - health = store_status(config.db_path, config.host_id) - api = TendwireDaemonAPI( - get_snapshot=lambda: _snapshot([]), - get_health=lambda: health, - submit_command=lambda params: submit_command( - config, - params, - socket_client_factory=_forbidden_factory, - ), - ) - daemon_response = api.dispatch( - {"method": "command.submit", "params": payload, "id": "1"} - ) - assert daemon_response["result"]["status"] == STATUS_ACCEPTED - _assert_no_selector_proof(daemon_response, proof) - _assert_no_selector_proof(api.dispatch({"method": "health.get", "id": "2"}), proof) - _assert_no_selector_proof(health, proof) - _assert_no_selector_proof( - run_store_maintenance( - config.db_path, - config.host_id, - retention_days=14, - max_outbox_attempts=5, - ), - proof, - ) - - -def test_selector_proof_field_is_rejected_from_a_request(tmp_path: Path) -> None: - """No caller can inject or forge selector evidence through the public request.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - - for injected in ( - {"target": {"worker_id": "w-1", "selector_proof": "v1:" + "a" * 64}}, - {"instruction": {"text": "hello", "selector_proof": "v1:" + "a" * 64}}, - {"selector_proof": "v1:" + "a" * 64}, - ): - payload = _request(request_id="injected", target={"worker_id": worker.id}) - payload.update(injected) - envelope = submit_command( - config, - payload, - socket_client_factory=lambda _config: pytest.fail( - "an invalid request must not create a socket client" - ), - ) - assert envelope.ok is False - assert envelope.status == "invalid_request" - - assert _receipt_rows(config) == [] diff --git a/tests/test_command_submission.py b/tests/test_command_submission.py deleted file mode 100644 index 75fcd54..0000000 --- a/tests/test_command_submission.py +++ /dev/null @@ -1,4542 +0,0 @@ -"""Tests for the authoritative daemon command submission path.""" - -from __future__ import annotations - -from concurrent.futures import ThreadPoolExecutor -from datetime import datetime, timedelta, timezone -from threading import Event - -import json -import sqlite3 -from pathlib import Path -from typing import Any - -import pytest -import tendwire.command_submission as command_submission -import tendwire.store.sqlite as store_sqlite - -from tendwire.backends.herdr_protocol import HerdrErrorResponse, HerdrProtocolError -from tendwire.backends.herdr_socket import ( - HerdrSocketDisconnectedError, - HerdrSocketTimeoutError, -) -from tendwire.command_submission import replay_command_receipt, submit_command -from tendwire.config import TURN_MODELS, Config -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - DISPOSITION_TERMINAL_ACCEPTED, - DISPOSITION_TERMINAL_REJECTED, - DISPOSITION_TERMINAL_UNCERTAIN, - STATUS_ACCEPTED, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DUPLICATE_REQUEST, - STATUS_INVALID_REQUEST, - STATUS_NOT_FOUND, - STATUS_PENDING, - STATUS_REJECTED, - STATUS_REQUEST_STATE_UNCERTAIN, - STATUS_STALE_TARGET, - CommandEnvelope, - CommandRequest, - build_canonical_mutation, - instruction_fingerprint, - turn_submission_id, -) -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.core.turns import PendingObservation, PendingObservedChoice, Turn -from tendwire.store.sqlite import ( - apply_backend_pending_observation, - cleanup_command_request_retention, - get_command_request, - init_store, - merge_turn_content, - pending_payload_from_store, - save_snapshot, - turns_payload_from_store, - upsert_worker_bindings, - reserve_command_request, -) - - -def _receipt_for_action( - db_path: Path, - host_id: str, - request_id: str, - action: str, -) -> dict[str, Any] | None: - receipt = get_command_request(db_path, host_id, request_id) - if receipt is None or receipt["action"] != action: - return None - return {**receipt, "uncertain": receipt["state"] in {"send_started", "uncertain"}} - - -_FORBIDDEN_PUBLIC_KEYS = { - "pane_id", - "terminal_id", - "backend_target", - "agent_session", - "argv", - "shell", - "target_kind", - "target_value", - "private_fingerprint", -} - - -def _assert_no_private_json(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - assert key not in _FORBIDDEN_PUBLIC_KEYS, f"forbidden field {path}.{key}" - _assert_no_private_json(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_private_json(item, f"{path}[{index}]") - - -def _config( - tmp_path: Path, - *, - backend: str = "socket", - timeout: float = 5.0, - turn_model: str = "observed", - submission_link_window_seconds: int = 60, - submission_hard_ttl_seconds: int = 86_400, -) -> Config: - return Config( - host_id="cmd-host", - data_dir=tmp_path, - db_path=tmp_path / "commands.db", - herdr_backend=backend, - herdr_timeout_seconds=timeout, - turn_model=turn_model, - submission_link_window_seconds=submission_link_window_seconds, - submission_hard_ttl_seconds=submission_hard_ttl_seconds, - ) - - -def _request( - *, - request_id: str = "req-1", - worker_id: str = "w-1", - text: str = "hello", - worker_fingerprint: str | None = None, - response_schema_version: int | None = None, -) -> dict[str, Any]: - target: dict[str, Any] = {"worker_id": worker_id} - if worker_fingerprint is not None: - target["worker_fingerprint"] = worker_fingerprint - payload = { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": target, - "instruction": {"text": text}, - } - if response_schema_version is not None: - payload["response_schema_version"] = response_schema_version - return payload - - -def _healthy_backend() -> BackendHealth: - return BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - counts={"workers": 1}, - ) - - -def _degraded_backend() -> BackendHealth: - return BackendHealth( - name="herdr", - status="degraded", - outcome="timeout", - observed_at="2026-01-01T00:01:00+00:00", - ) - - -def _seed( - config: Config, - workers: list[Worker], - bindings: list[WorkerBinding] | None = None, - *, - health: BackendHealth | None = None, -) -> None: - assert config.db_path is not None - init_store(config.db_path) - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:00:00+00:00", - workers=workers, - backend_health=[health or _healthy_backend()], - ), - turn_model=config.turn_model, - ) - if bindings: - upsert_worker_bindings(config.db_path, bindings) - - -def _binding( - worker: Worker, - *, - target_kind: str = "agent_id", - value: str = "agent-secret", - sendable: bool = True, - reason: str | None = None, - fingerprint: str | None = None, - private_fingerprint: str = "private-secret", - turn_target_kind: str | None = "pane_id", - turn_target_value: str | None = "pane-secret", -) -> WorkerBinding: - return WorkerBinding( - host_id="cmd-host", - worker_id=worker.id, - worker_fingerprint=fingerprint or worker.fingerprint, - backend="herdr", - target_kind=target_kind, - target_value=value, - turn_target_kind=turn_target_kind, - turn_target_value=turn_target_value, - sendable=sendable, - reason=reason, - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint=private_fingerprint, - ) - - -_REALISTIC_VISIBLE_PANE = """\ -╭─ Claude Code ─────────────────────────────────────────────────────────────╮ -│ Completed the previous task. │ -╰───────────────────────────────────────────────────────────────────────────╯ -───────────────────────────────────────────────────────────────────────────── - ⏵⏵ accept edits on · esc to interrupt -""" - - -class _FakeSocketClient: - def __init__( - self, - calls: list[dict[str, Any]], - *, - raises: BaseException | None = None, - pane_id: str = "pane-secret", - ) -> None: - self.calls = calls - self.raises = raises - self.pane_id = pane_id - self.close_count = 0 - - def connect(self) -> "_FakeSocketClient": - return self - - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if self.raises is not None and method in {"pane.send_input", "agent.prompt"}: - raise self.raises - if method == "agent.get": - return {"result": {"agent": {"pane_id": self.pane_id}}} - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": _REALISTIC_VISIBLE_PANE}, - } - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": self.pane_id}, - "delivery": "submitted", - } - return {"accepted": True} - - def close(self) -> None: - self.close_count += 1 - - -class _PromptVerdictClient(_FakeSocketClient): - def __init__( - self, - calls: list[dict[str, Any]], - *, - delivery: str = "submitted", - error_code: str | None = None, - pane_reads: list[str] | None = None, - ) -> None: - super().__init__(calls) - self.delivery = delivery - self.error_code = error_code - self.pane_reads = list(pane_reads or [_REALISTIC_VISIBLE_PANE]) - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - if method == "pane.read": - self.calls.append({"method": method, "params": dict(params)}) - text = self.pane_reads.pop(0) if self.pane_reads else "" - return {"type": "pane_read", "read": {"text": text}} - if method == "agent.prompt": - self.calls.append({"method": method, "params": dict(params)}) - if self.error_code is not None: - raise HerdrErrorResponse( - { - "code": self.error_code, - "message": self.error_code, - }, - "test-request", - ) - return { - "type": "agent_prompted", - "agent": {"pane_id": self.pane_id}, - "delivery": self.delivery, - } - return super().request(method, params, timeout=timeout) - - -def _factory(calls: list[dict[str, Any]], *, raises: BaseException | None = None, pane_id: str = "pane-secret"): - def make_client(config: Config) -> _FakeSocketClient: - return _FakeSocketClient(calls, raises=raises, pane_id=pane_id) - - return make_client - - -def _expected_submit_calls( - target: str = "agent-secret", - *, - resolved_target: str = "pane-secret", - text: str = "hello", - timeout_ms: int = 5000, -) -> list[dict[str, Any]]: - return [ - {"method": "agent.get", "params": {"target": target}}, - { - "method": "agent.prompt", - "params": { - "target": resolved_target, - "text": text, - "wait": {"until": ["working"], "timeout_ms": timeout_ms}, - }, - }, - ] - - -def _expected_private_clear_calls(pane_id: str = "pane-secret") -> list[dict[str, Any]]: - return [ - {"method": "pane.send_keys", "params": {"pane_id": pane_id, "keys": ["ctrl+u"]}}, - {"method": "pane.send_keys", "params": {"pane_id": pane_id, "keys": ["ctrl+a", "ctrl+k"]}}, - {"method": "pane.send_keys", "params": {"pane_id": pane_id, "keys": ["ctrl+a", "backspace"]}}, - ] - -@pytest.mark.parametrize( - "action", ["send_instruction", "answer_pending", "answer_decision"] -) -@pytest.mark.parametrize( - ("request_id", "include_request_id"), - [ - pytest.param(None, False, id="missing"), - pytest.param(None, True, id="null"), - pytest.param(123, True, id="non-string"), - pytest.param("", True, id="empty"), - pytest.param("x" * 129, True, id="max-plus-one"), - pytest.param("x" * ((1024 * 1024) - 256), True, id="near-frame-size"), - pytest.param(" leading", True, id="leading-space"), - pytest.param("trailing ", True, id="trailing-space"), - pytest.param("interior space", True, id="interior-space"), - pytest.param("\t", True, id="tab"), - pytest.param("\n", True, id="newline"), - pytest.param("\0", True, id="nul"), - pytest.param("é", True, id="unicode-nfc"), - pytest.param("e\u0301", True, id="unicode-nfd"), - pytest.param("Ａ", True, id="unicode-fullwidth"), - ], -) -def test_submit_command_rejects_invalid_request_id_before_mutation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - action: str, - request_id: Any, - include_request_id: bool, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - init_store(config.db_path) - calls: list[str] = [] - - def guarded_store(*args: Any, **kwargs: Any) -> Any: - calls.append("store") - raise AssertionError("invalid request_id must not access the store") - - def guarded_observation(config: Config) -> Snapshot: - calls.append("observe") - raise AssertionError("invalid request_id must not observe") - - def guarded_socket_factory(config: Config) -> _FakeSocketClient: - calls.append("socket") - raise AssertionError("invalid request_id must not construct a socket client") - - monkeypatch.setattr("tendwire.command_submission.project_from_observations", guarded_observation) - monkeypatch.setattr(command_submission, "get_command_request", guarded_store) - monkeypatch.setattr(command_submission, "latest_snapshot", guarded_store) - payload = _request() - if action == "answer_pending": - payload = { - "schema_version": 1, - "action": "answer_pending", - "request_id": request_id, - "dry_run": False, - "params": { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - }, - } - elif action == "answer_decision": - payload = { - "schema_version": 1, - "action": "answer_decision", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": "w-1"}, - "params": { - "decision_ref": "decision-public", - "selection": {"option_refs": ["1"]}, - }, - } - if include_request_id: - payload["request_id"] = request_id - else: - del payload["request_id"] - - envelope = submit_command(config, payload, socket_client_factory=guarded_socket_factory) - - assert envelope.status == STATUS_INVALID_REQUEST - assert envelope.error is not None - assert envelope.error["code"] == STATUS_INVALID_REQUEST - assert calls == [] - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 0 - assert conn.execute("SELECT COUNT(*) FROM events").fetchone()[0] == 0 - - -@pytest.mark.parametrize( - ("label", "factory_exception", "connect_exception"), - [ - ("factory", RuntimeError("raw setup failure with private detail"), None), - ("path", ValueError("bad socket path /private/herdr.sock"), None), - ("missing", None, FileNotFoundError("missing /private/herdr.sock")), - ("refused", None, ConnectionRefusedError("refused /private/herdr.sock")), - ("permission", None, PermissionError("denied /private/herdr.sock")), - ], -) -def test_submit_command_socket_setup_failures_are_backend_unavailable( - tmp_path: Path, - label: str, - factory_exception: BaseException | None, - connect_exception: BaseException | None, -) -> None: - config = _config(tmp_path / label) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - class SetupFailsClient: - def connect(self) -> "SetupFailsClient": - assert connect_exception is not None - raise connect_exception - - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None) -> dict[str, Any]: - calls.append({"method": method, "params": dict(params)}) - raise AssertionError("private send request must not run before setup succeeds") - - def close(self) -> None: - return None - - def make_client(config: Config) -> SetupFailsClient: - if factory_exception is not None: - raise factory_exception - return SetupFailsClient() - - envelope = submit_command( - config, - _request(request_id=f"setup-{label}"), - socket_client_factory=make_client, - ) - - # Socket setup failed before any transmission. That is a safe pre-send - # transient: no send began, so the request ID stays retryable and no durable - # rejection receipt is written. - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert envelope.error is not None - assert envelope.error["code"] == STATUS_BACKEND_UNAVAILABLE - assert "private" not in json.dumps(envelope.to_dict()) - assert calls == [] - assert envelope.status != STATUS_NOT_FOUND - assert envelope.status != STATUS_REQUEST_STATE_UNCERTAIN - assert config.db_path is not None - assert get_command_request(config.db_path, "cmd-host", f"setup-{label}") is None - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 0 - command_events = conn.execute( - "SELECT COUNT(*) FROM events WHERE aggregate_type = 'command_request'" - ).fetchone()[0] - assert command_events == 0 - - # Once the transient clears, the same request ID succeeds exactly once. - recovery_calls: list[dict[str, Any]] = [] - recovered = submit_command( - config, - _request(request_id=f"setup-{label}"), - socket_client_factory=_factory(recovery_calls), - ) - assert recovered.status == STATUS_ACCEPTED - assert recovered.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert [call["method"] for call in recovery_calls].count("agent.prompt") == 1 - receipt = get_command_request(config.db_path, "cmd-host", f"setup-{label}") - assert receipt is not None - assert receipt["state"] == "accepted" - - -@pytest.mark.parametrize( - "exc", - [ - HerdrSocketDisconnectedError("disconnected"), - HerdrProtocolError("malformed response"), - OSError("transport failed after write"), - ], -) -def test_submit_command_post_send_transport_failures_are_uncertain( - tmp_path: Path, - exc: BaseException, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _request(request_id=f"uncertain-{type(exc).__name__}"), - socket_client_factory=_factory(calls, raises=exc), - ) - - assert envelope.status == STATUS_REQUEST_STATE_UNCERTAIN - assert envelope.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert envelope.status != STATUS_BACKEND_UNAVAILABLE - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, "cmd-host", f"uncertain-{type(exc).__name__}", "send_instruction") - assert receipt is not None - assert receipt["uncertain"] is True - with sqlite3.connect(str(config.db_path)) as conn: - events = [row[0] for row in conn.execute("SELECT event_type FROM events ORDER BY id").fetchall()] - ledger = conn.execute( - """ - SELECT state, terminal_at, submitted_at - FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, f"uncertain-{type(exc).__name__}"), - ).fetchone() - assert ledger is not None - assert ledger[0] == "uncertain" - assert ledger[1] is not None - assert ledger[2] is None - assert "command.request.send_started" in events - assert "command.request.uncertain" in events - - -def test_submit_command_uses_socket_pane_input_once_and_caches_result(tmp_path: Path) -> None: - config = _config(tmp_path) - stable_key = "wsk1_" + ("9" * 64) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={"stable_key": stable_key, "stable_key_version": 1}, - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command(config, _request(), socket_client_factory=_factory(calls)) - second = submit_command(config, _request(), socket_client_factory=_factory(calls)) - duplicate = submit_command( - config, - _request(text="changed"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_ACCEPTED - assert first.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert first.result["turn_id"] is None - assert first.result == { - "target": {"worker_id": "w-1"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "target_state_at_send": "active", - "observed_turn_state": "pending_observation", - "turn_id": first.result["turn_id"], - "submission_verdict": "submitted", - } - assert second.to_dict() == first.to_dict() - assert duplicate.status == STATUS_DUPLICATE_REQUEST - assert duplicate.disposition == DISPOSITION_TERMINAL_REJECTED - assert calls == _expected_submit_calls() - - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, "cmd-host", "req-1", "send_instruction") - assert receipt is not None - assert receipt["status"] == STATUS_ACCEPTED - assert receipt["uncertain"] is False - stored_result = json.loads(receipt["result_json"]) - assert stored_result["schema_version"] == 2 - assert stored_result["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - assert set(stored_result) == { - "schema_version", - "action", - "request_id", - "ok", - "dry_run", - "status", - "disposition", - "result", - "error", - "warnings", - } - turns_payload = turns_payload_from_store(config.db_path, "cmd-host") - assert turns_payload["turns"] == [] - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[worker], - backend_health=[_healthy_backend()], - ), - ) - turns_after_snapshot = turns_payload_from_store(config.db_path, "cmd-host") - assert turns_after_snapshot["turns"] == [] - - with sqlite3.connect(str(config.db_path)) as conn: - event_rows = conn.execute("SELECT event_type, payload_json FROM events ORDER BY id").fetchall() - command_row = conn.execute( - "SELECT request_json, result_json FROM commands WHERE request_id = 'req-1'" - ).fetchone() - submission_rows = conn.execute( - """ - SELECT submission_id, request_id, owner_key, owner_key_version, - instruction_fingerprint, state, linked_turn_id, - link_not_before, link_expires_at, hard_expires_at, - terminal_at, submitted_at, send_started_at, updated_at - FROM turn_submissions - """ - ).fetchall() - supersession_count = conn.execute( - "SELECT COUNT(*) FROM turn_supersessions" - ).fetchone()[0] - assert len(submission_rows) == 1 - submission = submission_rows[0] - assert submission[:7] == ( - turn_submission_id("cmd-host", "req-1"), - "req-1", - stable_key, - 1, - instruction_fingerprint("hello"), - "submitted", - None, - ) - link_not_before = datetime.fromisoformat(submission[7]) - link_expires_at = datetime.fromisoformat(submission[8]) - hard_expires_at = datetime.fromisoformat(submission[9]) - send_started_at = datetime.fromisoformat(submission[12]) - assert (send_started_at - link_not_before).total_seconds() == 60 - assert (link_expires_at - send_started_at).total_seconds() == 60 - assert (hard_expires_at - send_started_at).total_seconds() == 86_400 - assert submission[10] == submission[11] == submission[13] - assert datetime.fromisoformat(submission[11]) >= send_started_at - assert supersession_count == 0 - assert [row[0] for row in event_rows] == [ - "snapshot.saved", - "command.request.reserved", - "command.request.send_started", - "command.request.accepted", - ] - command_events = [json.loads(row[1]) for row in event_rows[1:]] - assert [ - (event["state"], event["status"]) - for event in command_events - ] == [ - ("reserved", STATUS_PENDING), - ("send_started", STATUS_PENDING), - ("accepted", STATUS_ACCEPTED), - ] - assert "detail" not in command_events[0] - assert command_events[1]["detail"]["target"] == {"worker_id": "w-1"} - assert command_events[2]["detail"]["envelope"] == first.to_dict() - assert command_row is not None - assert json.loads(command_row[0])["target"] == {"worker_id": "w-1"} - assert "request_id" not in json.loads(command_row[0]) - - public_surfaces = [ - first.to_dict(), - duplicate.to_dict(), - turns_payload, - turns_after_snapshot, - json.loads(receipt["result_json"]), - json.loads(command_row[1]), - *[json.loads(row[1]) for row in event_rows], - ] - encoded = json.dumps(public_surfaces) - assert "agent-secret" not in encoded - assert "private-secret" not in encoded - for surface in public_surfaces: - _assert_no_private_json(surface) - - - -def test_request_id_can_be_resubmitted_after_receipt_retention_purge( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="purged-request"), - socket_client_factory=_factory(calls), - ) - survivor = submit_command( - config, - _request(request_id="retention-survivor"), - socket_client_factory=_factory(calls), - ) - assert first.status == survivor.status == STATUS_ACCEPTED - assert config.db_path is not None - - cleanup = cleanup_command_request_retention( - config.db_path, - retry_horizon_seconds=604_800, - retention_seconds=691_200, - retention_count=1, - host_id=config.host_id, - now="2099-01-01T00:00:00+00:00", - ) - assert cleanup["deleted"] == 1 - assert get_command_request( - config.db_path, - config.host_id, - "purged-request", - ) is None - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - """ - SELECT COUNT(*) FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, "purged-request"), - ).fetchone() == (0,) - - resubmitted = submit_command( - config, - _request(request_id="purged-request"), - socket_client_factory=_factory(calls), - ) - - assert resubmitted.status == STATUS_ACCEPTED - assert resubmitted.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert [call["method"] for call in calls].count("agent.prompt") == 3 - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - """ - SELECT state FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, "purged-request"), - ).fetchone() == ("submitted",) - - -def test_submission_envelope_v3_requires_explicit_negotiation(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - default = submit_command( - config, - _request(request_id="default-v2"), - socket_client_factory=_factory(calls), - ) - opted_in = submit_command( - config, - _request(request_id="opted-v3", response_schema_version=3), - socket_client_factory=_factory(calls), - ) - replayed = submit_command( - config, - _request(request_id="opted-v3", response_schema_version=3), - socket_client_factory=_factory(calls), - ) - - assert default.schema_version == 2 - assert "submission_id" not in default.result - assert set(default.to_dict()) == { - "schema_version", "action", "request_id", "ok", "dry_run", - "status", "disposition", "result", "error", "warnings", - } - assert opted_in.schema_version == 3 - assert opted_in.result["submission_id"] == turn_submission_id( - config.host_id, - "opted-v3", - ) - assert opted_in.result["turn_id"] is None - assert replayed.to_dict() == opted_in.to_dict() - assert [call["method"] for call in calls].count("agent.prompt") == 2 - - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, "opted-v3") - assert receipt is not None - stored = json.loads(receipt["result_json"]) - assert stored["schema_version"] == 2 - assert "submission_id" not in stored["result"] - - -def test_submission_fingerprint_is_owner_isolated(tmp_path: Path) -> None: - config = _config(tmp_path) - first_key = "wsk1_" + ("1" * 64) - second_key = "wsk1_" + ("2" * 64) - first_worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={"stable_key": first_key, "stable_key_version": 1}, - ) - second_worker = Worker( - id="w-2", - name="Beta", - status="active", - meta={"stable_key": second_key, "stable_key_version": 1}, - ) - _seed( - config, - [first_worker, second_worker], - [ - _binding( - first_worker, - value="agent-1", - private_fingerprint="private-1", - turn_target_value="pane-1", - ), - _binding( - second_worker, - value="agent-2", - private_fingerprint="private-2", - turn_target_value="pane-2", - ), - ], - ) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="owner-1", worker_id="w-1", text="hello world"), - socket_client_factory=_factory(calls, pane_id="pane-1"), - ) - second = submit_command( - config, - _request(request_id="owner-2", worker_id="w-2", text=" hello world "), - socket_client_factory=_factory(calls, pane_id="pane-2"), - ) - - assert first.status == second.status == STATUS_ACCEPTED - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - rows = conn.execute( - """ - SELECT owner_key, owner_key_version, instruction_fingerprint - FROM turn_submissions ORDER BY request_id - """ - ).fetchall() - assert rows == [ - (first_key, 1, instruction_fingerprint("hello world")), - (second_key, 1, instruction_fingerprint("hello world")), - ] - - - - - -def test_submission_first_keeps_observation_authoritative_during_send( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={ - "stable_key": "wsk1_" + ("c" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - class ObservingClient(_FakeSocketClient): - def request(self, method, params, *, timeout=None): - result = super().request(method, params, timeout=timeout) - if method == "agent.prompt": - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "submission-first-source", - "user_text": "hello", - "assistant_final_text": "observed during send", - "complete": True, - "has_open_turn": False, - }, - observed_at=datetime.now(timezone.utc).isoformat(), - ) == 1 - return result - - accepted = submit_command( - config, - _request(request_id="submission-first-request"), - socket_client_factory=lambda _config: ObservingClient(calls), - ) - - assert accepted.status == STATUS_ACCEPTED - assert accepted.result["observed_turn_state"] == "complete" - assert isinstance(accepted.result["turn_id"], str) - turns = turns_payload_from_store( - config.db_path, - config.host_id, - schema_version=2, - )["turns"] - matching = [turn for turn in turns if turn.get("user_text") == "hello"] - assert len(matching) == 1 - assert accepted.result["turn_id"] == matching[0]["id"] - assert matching[0]["assistant_final_text"] == "observed during send" - - - -def test_submit_command_sends_identical_100_character_instructions_without_turn_rows( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={ - "stable_key": "wsk1_" + ("d" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - text = "x" * 100 - - first = submit_command( - config, - _request(request_id="long-1", text=text), - socket_client_factory=_factory(calls), - ) - second = submit_command( - config, - _request(request_id="long-2", text=text), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_ACCEPTED - assert second.status == STATUS_ACCEPTED - assert first.result["turn_id"] is None - assert second.result["turn_id"] is None - expected_send = _expected_submit_calls(text=text) - assert calls == [*expected_send, *expected_send] - - assert config.db_path is not None - first_receipt = get_command_request(config.db_path, "cmd-host", "long-1") - second_receipt = get_command_request(config.db_path, "cmd-host", "long-2") - assert first_receipt is not None - assert first_receipt["state"] == "accepted" - assert second_receipt is not None - assert second_receipt["state"] == "accepted" - with sqlite3.connect(str(config.db_path)) as conn: - turn_count = conn.execute("SELECT COUNT(*) FROM turns").fetchone()[0] - events = [ - row[0] - for row in conn.execute( - "SELECT event_type FROM events WHERE aggregate_type = 'command_request' ORDER BY id" - ).fetchall() - ] - assert turn_count == 0 - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "ambiguous-identical-source", - "user_text": text, - "assistant_final_text": "independent observation", - "complete": True, - "has_open_turn": False, - }, - observed_at="2099-07-19T10:00:00+00:00", - ) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - stored = [ - (str(turn_id), json.loads(payload_json)) - for turn_id, payload_json in conn.execute( - "SELECT turn_id, payload_json FROM turns WHERE host_id = ?", - (config.host_id,), - ).fetchall() - ] - relevant = [ - (turn_id, payload) - for turn_id, payload in stored - if payload.get("source_turn_id") - ] - assert len(relevant) == 1 - observed = [ - (turn_id, payload) - for turn_id, payload in relevant - if payload.get("source_turn_id") - ] - assert len(observed) == 1 - assert events == [ - "command.request.reserved", - "command.request.send_started", - "command.request.accepted", - "command.request.reserved", - "command.request.send_started", - "command.request.accepted", - ] - public_json = json.dumps( - [ - first.to_dict(), - second.to_dict(), - json.loads(first_receipt["result_json"]), - json.loads(second_receipt["result_json"]), - ] - ) - assert text not in public_json - assert "agent-secret" not in public_json - assert "private-secret" not in public_json - - -def test_submit_command_allows_same_instruction_after_worker_fingerprint_changes( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - old_worker = Worker(id="w-1", name="Alpha", status="active", fingerprint="old-fp") - new_worker = Worker(id="w-1", name="Alpha", status="active", fingerprint="new-fp") - _seed( - config, - [old_worker], - [_binding(old_worker, value="old-agent-secret", private_fingerprint="old-private-secret")], - ) - calls: list[dict[str, Any]] = [] - text = "When this exact long Telegram instruction appears for a new binding, it should send." - - first = submit_command( - config, - _request(request_id="fingerprint-1", text=text, worker_fingerprint="old-fp"), - socket_client_factory=_factory(calls), - ) - assert first.status == STATUS_ACCEPTED - - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[new_worker], - backend_health=[_healthy_backend()], - ), - ) - upsert_worker_bindings( - config.db_path, - [_binding(new_worker, value="new-agent-secret", private_fingerprint="new-private-secret")], - ) - - second = submit_command( - config, - _request(request_id="fingerprint-2", text=text, worker_fingerprint="new-fp"), - socket_client_factory=_factory(calls), - ) - - assert second.status == STATUS_ACCEPTED - assert calls == [ - *_expected_submit_calls("old-agent-secret", text=text), - *_expected_submit_calls("new-agent-secret", text=text), - ] - - -def test_submit_command_terminal_worker_id_and_fingerprint_replays_after_healthy_worker_churn( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - fingerprint="worker-fingerprint-1", - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request = _request( - request_id="terminal-worker-churn", - worker_fingerprint=worker.fingerprint, - ) - - accepted = submit_command( - config, - request, - socket_client_factory=_factory(calls), - ) - assert accepted.status == STATUS_ACCEPTED - assert accepted.disposition == DISPOSITION_TERMINAL_ACCEPTED - - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[], - backend_health=[_healthy_backend()], - ), - ) - no_backend = lambda _config: pytest.fail( - "terminal replay after worker churn must not create a socket client" - ) - - exact_replay = submit_command( - config, - request, - socket_client_factory=no_backend, - ) - refreshed_fingerprint = submit_command( - config, - _request( - request_id="terminal-worker-churn", - worker_fingerprint="worker-fingerprint-2", - ), - socket_client_factory=no_backend, - ) - changed_worker = submit_command( - config, - _request( - request_id="terminal-worker-churn", - worker_id="w-2", - worker_fingerprint=worker.fingerprint, - ), - socket_client_factory=no_backend, - ) - changed_instruction = submit_command( - config, - _request( - request_id="terminal-worker-churn", - text="changed", - worker_fingerprint=worker.fingerprint, - ), - socket_client_factory=no_backend, - ) - - assert exact_replay.to_dict() == accepted.to_dict() - assert refreshed_fingerprint.to_dict() == accepted.to_dict() - assert changed_worker.status == STATUS_DUPLICATE_REQUEST - assert changed_worker.disposition == DISPOSITION_TERMINAL_REJECTED - assert changed_instruction.status == STATUS_DUPLICATE_REQUEST - assert changed_instruction.disposition == DISPOSITION_TERMINAL_REJECTED - assert calls == _expected_submit_calls() - - -def test_submit_command_uses_verified_agent_prompt(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls)) - - assert envelope.status == STATUS_ACCEPTED - assert calls == _expected_submit_calls() - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - assert not any(call["method"] == "pane.read" for call in calls) - assert not any(call["method"] == "pane.send_text" for call in calls) - assert not any(call["method"] == "pane.send_keys" for call in calls) - - -def test_written_to_pty_prior_running_turn_observed_later_stays_queued( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker( - id="w-1", - name="Alpha", - status="working", - meta={ - "stable_key": "wsk1_" + ("7" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "queued-written-to-pty" - - queued = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - delivery="written_to_pty", - ), - ) - replay = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "queued replay must not issue a second prompt" - ), - ) - - assert queued.status == replay.status == STATUS_PENDING - assert queued.disposition == replay.disposition == DISPOSITION_IN_PROGRESS - assert queued.result["submission_verdict"] == "written_to_pty" - assert queued.result["delivery_state"] == "queued" - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - UPDATE turn_submissions - SET submitted_at = ?, send_started_at = ?, link_not_before = ? - WHERE host_id = ? AND request_id = ? - """, - ( - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - config.host_id, - request_id, - ), - ) - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "queued-observed-source", - "user_text": "hello", - "assistant_final_text": None, - "complete": False, - "has_open_turn": True, - }, - observed_at="2026-01-01T00:00:01+00:00", - ) == 1 - - still_queued = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "queued replay with an observed prior turn must not issue a second prompt" - ), - ) - - assert still_queued.status == STATUS_PENDING - assert still_queued.disposition == DISPOSITION_IN_PROGRESS - assert still_queued.result["submission_verdict"] == "written_to_pty" - assert still_queued.result["transport_state"] == "queued" - assert still_queued.result["turn_id"] is None - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - """ - SELECT linked_turn_id FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, request_id), - ).fetchone() == (None,) - - -def test_written_to_pty_expired_verification_becomes_terminal_uncertain( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker(id="w-1", name="Alpha", status="working") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "queued-verification-expired" - request = _request(request_id=request_id) - - queued = submit_command( - config, - request, - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - delivery="written_to_pty", - ), - ) - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - UPDATE turn_submissions - SET link_expires_at = ?, hard_expires_at = ? - WHERE host_id = ? AND request_id = ? - """, - ( - "2000-01-01T00:00:00+00:00", - "2000-01-01T00:00:00+00:00", - config.host_id, - request_id, - ), - ) - - uncertain = submit_command( - config, - request, - socket_client_factory=lambda _config: pytest.fail( - "expired queued replay must not issue another prompt" - ), - ) - replay = submit_command( - config, - request, - socket_client_factory=lambda _config: pytest.fail( - "terminal-uncertain replay must not issue another prompt" - ), - ) - - assert queued.status == STATUS_PENDING - assert uncertain.to_dict() == replay.to_dict() - assert uncertain.status == STATUS_REQUEST_STATE_UNCERTAIN - assert uncertain.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert uncertain.result["submission_verdict"] == "written_to_pty" - assert uncertain.result["delivery_state"] == "unknown" - assert uncertain.error is not None - assert "verification expired" in uncertain.error["message"] - assert "queued" not in uncertain.error["message"].lower() - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - """ - SELECT state, status - FROM command_receipts - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, request_id), - ).fetchone() == ("uncertain", STATUS_REQUEST_STATE_UNCERTAIN) - assert conn.execute( - """ - SELECT state - FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, request_id), - ).fetchone() == ("expired",) - - -def test_written_to_pty_replay_settles_only_its_component( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker(id="w-1", name="Alpha", status="working") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "queued-one-of-forty" - request = _request(request_id=request_id) - - queued = submit_command( - config, - request, - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - delivery="written_to_pty", - ), - ) - assert queued.status == STATUS_PENDING - with sqlite3.connect(str(config.db_path)) as conn: - for index in range(1, 40): - conn.execute( - """ - INSERT INTO turn_submissions ( - host_id, submission_id, request_id, owner_key, - owner_key_version, instruction_fingerprint, state, - linked_turn_id, link_not_before, link_expires_at, - hard_expires_at, linked_at, terminal_at, submitted_at, - send_started_at, updated_at - ) - SELECT host_id, ?, ?, ?, owner_key_version, ?, state, - linked_turn_id, link_not_before, link_expires_at, - hard_expires_at, linked_at, terminal_at, submitted_at, - send_started_at, updated_at - FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - ( - f"submission-extra-{index}", - f"request-extra-{index}", - f"owner-extra-{index}", - f"fingerprint-extra-{index}", - config.host_id, - request_id, - ), - ) - assert conn.execute( - """ - SELECT COUNT(DISTINCT owner_key || ':' || instruction_fingerprint) - FROM turn_submissions - WHERE host_id = ? - """, - (config.host_id,), - ).fetchone() == (40,) - own_component = conn.execute( - """ - SELECT owner_key, instruction_fingerprint - FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, request_id), - ).fetchone() - assert own_component is not None - - original_settle = store_sqlite.settle_submission_links_conn - settled_components: list[tuple[str, str]] = [] - - def counted_settle( - conn: sqlite3.Connection, - host_id: str, - owner_key: str, - instruction_fingerprint_value: str, - **kwargs: Any, - ) -> int: - settled_components.append((owner_key, instruction_fingerprint_value)) - return original_settle( - conn, - host_id, - owner_key, - instruction_fingerprint_value, - **kwargs, - ) - - monkeypatch.setattr( - store_sqlite, - "settle_submission_links_conn", - counted_settle, - ) - replay = submit_command( - config, - request, - socket_client_factory=lambda _config: pytest.fail( - "queued replay must not issue another prompt" - ), - ) - - assert replay.status == STATUS_PENDING - assert settled_components == [own_component] - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -@pytest.mark.parametrize( - "verdict", - [ - "agent_not_ready", - "agent_target_ambiguous", - "agent_prompt_not_received", - "agent_prompt_unsubmitted", - "agent_input_pending", - ], -) -def test_positive_non_delivery_verdict_is_terminal_rejected_without_retry( - tmp_path: Path, - verdict: str, -) -> None: - config = _config(tmp_path / verdict) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = f"not-delivered-{verdict}" - - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - error_code=verdict, - ), - ) - second = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "terminal non-delivery replay must not issue another prompt" - ), - ) - - assert first.to_dict() == second.to_dict() - assert first.status == STATUS_REJECTED - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert first.result["submission_verdict"] == verdict - assert first.result["delivery_state"] == "not_delivered" - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_agent_prompt_stalled_never_infers_composer_from_stale_scrollback( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "stalled-stale-scrollback" - - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - error_code="agent_prompt_stalled", - pane_reads=[ - f"{_REALISTIC_VISIBLE_PANE}\n" - "User: hello\n" - "Assistant: an answer from an old completed turn" - ], - ), - ) - second = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "stalled replay must not issue another prompt" - ), - ) - - assert first.to_dict() == second.to_dict() - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert first.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert first.result["submission_verdict"] == "agent_prompt_stalled" - assert "composer_state" not in first.result - assert not any(call["method"] == "pane.read" for call in calls) - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_agent_not_found_remains_unknown_without_retry(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "agent-not-found-unknown" - - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - error_code="agent_not_found", - ), - ) - second = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "unknown agent_not_found replay must not issue another prompt" - ), - ) - - assert first.to_dict() == second.to_dict() - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert first.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert first.result["submission_verdict"] == "unknown" - assert first.result["delivery_state"] == "unknown" - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_dirty_composer_non_delivery_is_surfaced_without_clear_or_resend( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "dirty-composer" - - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - error_code="agent_prompt_unsubmitted", - ), - ) - second = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "dirty-composer non-delivery replay must not resend" - ), - ) - - assert first.to_dict() == second.to_dict() - assert first.status == STATUS_REJECTED - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert first.result["submission_verdict"] == "agent_prompt_unsubmitted" - assert first.result["delivery_state"] == "not_delivered" - assert calls == _expected_submit_calls() - assert not any(call["method"] == "pane.send_keys" for call in calls) - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_crash_after_prompt_write_recovers_unknown_without_resend( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "write-before-verdict-record" - real_finish = command_submission.finish_command_request - - def crash_before_record(*args: Any, **kwargs: Any) -> Any: - raise RuntimeError("simulated process loss after prompt write") - - monkeypatch.setattr( - command_submission, - "finish_command_request", - crash_before_record, - ) - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient(calls), - ) - assert first.status == STATUS_PENDING - assert first.disposition == DISPOSITION_IN_PROGRESS - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - monkeypatch.setattr( - command_submission, - "finish_command_request", - real_finish, - ) - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - UPDATE command_receipts - SET owner_expires_at = ? - WHERE host_id = ? AND request_id = ? - """, - ("2000-01-01T00:00:00+00:00", config.host_id, request_id), - ) - - recovered = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "unknown recovery must never resend" - ), - ) - - assert recovered.status == STATUS_REQUEST_STATE_UNCERTAIN - assert recovered.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert recovered.result["submission_verdict"] == "unknown" - assert recovered.result["delivery_state"] == "unknown" - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_submit_command_reports_submitted_transport_and_worker_state(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="working") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls)) - - assert envelope.status == STATUS_ACCEPTED - assert envelope.result["turn_id"] is None - assert envelope.result == { - "target": {"worker_id": "w-1"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "target_state_at_send": "active", - "observed_turn_state": "pending_observation", - "turn_id": envelope.result["turn_id"], - "submission_verdict": "submitted", - } - - -def test_submit_command_marks_idle_worker_delivery_as_submitted(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="idle") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls)) - - assert envelope.status == STATUS_ACCEPTED - assert envelope.result["turn_id"] is None - assert envelope.result == { - "target": {"worker_id": "w-1"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "target_state_at_send": "idle", - "observed_turn_state": "pending_observation", - "turn_id": envelope.result["turn_id"], - "submission_verdict": "submitted", - } - - -def test_submit_command_terminal_binding_resolves_pane_and_submits_input(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker, target_kind="terminal_id", value="term-secret")]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls, pane_id="pane-private")) - - assert envelope.status == STATUS_ACCEPTED - assert calls == _expected_submit_calls( - "term-secret", - resolved_target="pane-private", - ) - public_json = json.dumps(envelope.to_dict()) - assert "term-secret" not in public_json - assert "pane-private" not in public_json - _assert_no_private_json(envelope.to_dict()) - - -def test_submit_command_pane_binding_submits_without_public_pane_leak(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker, target_kind="pane_id", value="pane-private")]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls)) - - assert envelope.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "agent.prompt", - "params": { - "target": "pane-private", - "text": "hello", - "wait": {"until": ["working"], "timeout_ms": 5000}, - }, - }, - ] - public_json = json.dumps(envelope.to_dict()) - assert "pane-private" not in public_json - _assert_no_private_json(envelope.to_dict()) - - -def test_submit_command_backend_unavailable_prevents_not_found_and_send(tmp_path: Path) -> None: - config = _config(tmp_path) - health = BackendHealth( - name="herdr", - status="degraded", - outcome="protocol_error", - observed_at="2026-01-01T00:00:00+00:00", - ) - _seed(config, [], [], health=health) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _request(worker_id="missing", request_id="degraded-1"), - socket_client_factory=_factory(calls), - ) - - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert calls == [] - assert envelope.status != STATUS_NOT_FOUND - assert config.db_path is not None - assert get_command_request(config.db_path, config.host_id, "degraded-1") is None - - -def test_submit_command_missing_worker_can_return_not_found_when_backend_healthy(tmp_path: Path) -> None: - config = _config(tmp_path) - _seed(config, [], []) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _request(worker_id="missing", request_id="missing-1"), - socket_client_factory=_factory(calls), - ) - - assert envelope.status == STATUS_NOT_FOUND - assert calls == [] -def test_submit_command_resolved_health_failure_is_terminal_and_replayed( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)], health=_degraded_backend()) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="resolved-degraded"), - socket_client_factory=_factory(calls), - ) - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:02:00+00:00", - workers=[worker], - backend_health=[_healthy_backend()], - ), - ) - replay = submit_command( - config, - _request(request_id="resolved-degraded"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert calls == [] - receipt = get_command_request( - config.db_path, - config.host_id, - "resolved-degraded", - ) - assert receipt is not None - assert receipt["state"] == "rejected" - assert receipt["status"] == STATUS_BACKEND_UNAVAILABLE - stored_result = json.loads(receipt["result_json"]) - assert stored_result["schema_version"] == 2 - assert stored_result["disposition"] == DISPOSITION_TERMINAL_REJECTED - - -def test_backend_unavailable_disposition_depends_on_receipt_authority( - tmp_path: Path, -) -> None: - no_authority_config = _config(tmp_path / "no-authority") - _seed(no_authority_config, [], [], health=_degraded_backend()) - no_authority = submit_command( - no_authority_config, - _request(request_id="unavailable-no-authority", worker_id="missing"), - socket_client_factory=lambda _config: pytest.fail( - "unresolved authority must not create a socket client" - ), - ) - - rejected_config = _config(tmp_path / "terminal-rejection") - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - rejected_config, - [worker], - [_binding(worker)], - health=_degraded_backend(), - ) - terminal_rejection = submit_command( - rejected_config, - _request(request_id="unavailable-terminal-rejection"), - socket_client_factory=lambda _config: pytest.fail( - "failed health must not create a socket client" - ), - ) - - assert no_authority.status == terminal_rejection.status == STATUS_BACKEND_UNAVAILABLE - assert no_authority.disposition == DISPOSITION_NO_RECEIPT - assert terminal_rejection.disposition == DISPOSITION_TERMINAL_REJECTED - assert no_authority_config.db_path is not None - assert rejected_config.db_path is not None - assert get_command_request( - no_authority_config.db_path, - no_authority_config.host_id, - "unavailable-no-authority", - ) is None - receipt = get_command_request( - rejected_config.db_path, - rejected_config.host_id, - "unavailable-terminal-rejection", - ) - assert receipt is not None - assert receipt["state"] == "rejected" - - -def test_submit_command_disallowed_worker_rejection_is_terminal_and_replayed( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - closed = Worker(id="w-1", name="Alpha", status="closed") - _seed(config, [closed], [_binding(closed)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="closed-worker"), - socket_client_factory=_factory(calls), - ) - active = Worker(id="w-1", name="Alpha", status="active") - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:02:00+00:00", - workers=[active], - backend_health=[_healthy_backend()], - ), - ) - replay = submit_command( - config, - _request(request_id="closed-worker"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_REJECTED - assert replay.to_dict() == first.to_dict() - assert calls == [] - receipt = get_command_request(config.db_path, config.host_id, "closed-worker") - assert receipt is not None - assert receipt["state"] == "rejected" - assert receipt["status"] == STATUS_REJECTED - - -def test_submit_command_rejects_stale_worker_fingerprint_and_binding(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker, fingerprint="old-fingerprint")]) - calls: list[dict[str, Any]] = [] - - stale_request = submit_command( - config, - _request(request_id="stale-request", worker_fingerprint="old-fingerprint"), - socket_client_factory=_factory(calls), - ) - stale_binding = submit_command( - config, - _request(request_id="stale-binding"), - socket_client_factory=_factory(calls), - ) - - assert stale_request.status == STATUS_STALE_TARGET - assert stale_binding.status == STATUS_STALE_TARGET - assert calls == [] - - -def test_submit_command_rejects_duplicate_missing_and_unsendable_bindings(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - - _seed(config, [worker], []) - calls: list[dict[str, Any]] = [] - missing = submit_command( - config, - _request(request_id="missing-binding"), - socket_client_factory=_factory(calls), - ) - assert missing.status == STATUS_BACKEND_UNSUPPORTED - - config = _config(tmp_path / "unsendable") - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker, sendable=False, reason="disabled")]) - unsendable = submit_command( - config, - _request(request_id="unsendable-binding"), - socket_client_factory=_factory(calls), - ) - assert unsendable.status == STATUS_BACKEND_UNSUPPORTED - - config = _config(tmp_path / "duplicate") - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [ - _binding(worker, value="agent-a", private_fingerprint="private-a"), - _binding(worker, value="agent-b", private_fingerprint="private-b"), - ], - ) - duplicate = submit_command( - config, - _request(request_id="duplicate-binding"), - socket_client_factory=_factory(calls), - ) - assert duplicate.status == STATUS_AMBIGUOUS_BACKEND_TARGET - assert calls == [] - - -def test_submit_command_timeout_after_send_start_is_uncertain_and_not_retried(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="timeout-1"), - socket_client_factory=_factory(calls, raises=HerdrSocketTimeoutError("timeout")), - ) - second = submit_command( - config, - _request(request_id="timeout-1"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert first.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert second.status == STATUS_REQUEST_STATE_UNCERTAIN - assert second.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == [ - *_expected_submit_calls(), - ] - - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, "cmd-host", "timeout-1", "send_instruction") - assert receipt is not None - assert receipt["uncertain"] is True - with sqlite3.connect(str(config.db_path)) as conn: - event_rows = conn.execute( - "SELECT event_type, payload_json FROM events " - "WHERE aggregate_id = ? ORDER BY id", - ("timeout-1",), - ).fetchall() - assert [row[0] for row in event_rows] == [ - "command.request.reserved", - "command.request.send_started", - "command.request.uncertain", - ] - payloads = [json.loads(row[1]) for row in event_rows] - assert [(item["state"], item["status"]) for item in payloads] == [ - ("reserved", STATUS_PENDING), - ("send_started", STATUS_PENDING), - ("uncertain", STATUS_REQUEST_STATE_UNCERTAIN), - ] - - -def test_submit_command_unprovable_selector_spelling_fails_closed( - tmp_path: Path, -) -> None: - """A spelling the receipt cannot vouch for is never resolved by a degraded snapshot.""" - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="selector-unavailable"), - socket_client_factory=_factory(calls), - ) - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[], - backend_health=[_degraded_backend()], - ), - ) - # The receipt was issued for an explicit worker ID, so it holds no proof of - # either alias. Only a healthy observation could show they mean the same - # worker, and a degraded one may not stand in for it. - by_name = _request(request_id="selector-unavailable") - by_name["target"] = {"name": "Alpha"} - by_space = _request(request_id="selector-unavailable") - by_space["target"] = {"space_id": "space-1"} - - name_replay = submit_command(config, by_name, socket_client_factory=_factory(calls)) - space_replay = submit_command(config, by_space, socket_client_factory=_factory(calls)) - - assert first.status == STATUS_ACCEPTED - assert name_replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert name_replay.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert space_replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert space_replay.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == _expected_submit_calls() - receipt = get_command_request(config.db_path, config.host_id, "selector-unavailable") - assert receipt is not None - assert (receipt["state"], receipt["status"]) == ("accepted", STATUS_ACCEPTED) - - -def test_submit_command_equivalent_resolved_selectors_replay_once( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - by_name = _request(request_id="selector-equivalent") - by_name["target"] = {"name": "Alpha"} - by_space_with_origin = _request(request_id="selector-equivalent") - by_space_with_origin["target"] = {"space_id": "space-1"} - by_space_with_origin["params"] = {"origin": "connector-observation"} - - first = submit_command( - config, - _request(request_id="selector-equivalent"), - socket_client_factory=_factory(calls), - ) - second = submit_command(config, by_name, socket_client_factory=_factory(calls)) - third = submit_command( - config, - by_space_with_origin, - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_ACCEPTED - assert second.to_dict() == first.to_dict() - assert third.to_dict() == first.to_dict() - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, "selector-equivalent") - assert receipt is not None - assert receipt["public_worker_id"] == worker.id - canonical = json.loads(receipt["canonical_request_json"]) - assert canonical["target"] == {"worker_id": worker.id} - assert canonical["options"] == {} - assert "origin" not in receipt["canonical_request_json"] - - -@pytest.mark.parametrize("selector_kind", ["name", "space_id"]) -def test_submit_command_exact_selector_retry_survives_selector_reuse( - tmp_path: Path, - selector_kind: str, -) -> None: - """A reused name or space is worker churn, not a changed request.""" - config = _config(tmp_path / selector_kind) - first_worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - second_worker = Worker( - id="w-2", - name="Beta", - status="active", - space_id="space-2", - ) - _seed(config, [first_worker, second_worker], [_binding(first_worker)]) - selector = {"name": "Alpha"} if selector_kind == "name" else {"space_id": "space-1"} - request = _request(request_id=f"selector-reused-{selector_kind}") - request["target"] = selector - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, request, socket_client_factory=_factory(calls)) - # The selector the caller spelled now names a different public worker. - if selector_kind == "name": - changed_workers = [ - Worker(id="w-1", name="Former", status="active", space_id="space-1"), - Worker(id="w-2", name="Alpha", status="active", space_id="space-2"), - ] - else: - changed_workers = [ - Worker(id="w-1", name="Alpha", status="active", space_id="former-space"), - Worker(id="w-2", name="Beta", status="active", space_id="space-1"), - ] - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:02:00+00:00", - workers=changed_workers, - backend_health=[_healthy_backend()], - ), - ) - replay = submit_command( - config, - request, - socket_client_factory=lambda _config: pytest.fail( - "an exact retry must not create a socket client" - ), - ) - - assert accepted.status == STATUS_ACCEPTED - # Re-resolving the selector would deliver the instruction a second time, to - # a worker the caller never addressed. The receipt outranks the snapshot. - assert replay.to_dict() == accepted.to_dict() - assert replay.result is not None - assert replay.result["target"] == {"worker_id": "w-1"} - assert calls == _expected_submit_calls() - - -@pytest.mark.parametrize( - ("selector_kind", "changed_selector"), - [ - ("name", {"name": "Beta"}), - ("space_id", {"space_id": "space-2"}), - ("name_and_space", {"name": "Beta", "space_id": "space-2"}), - ], -) -def test_submit_command_changed_selector_conflicts_without_backend( - tmp_path: Path, - selector_kind: str, - changed_selector: dict[str, Any], -) -> None: - """Reusing a request ID with a different selector cannot claim its result.""" - config = _config(tmp_path / selector_kind) - first_worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - second_worker = Worker( - id="w-2", - name="Beta", - status="active", - space_id="space-2", - ) - _seed(config, [first_worker, second_worker], [_binding(first_worker)]) - request = _request(request_id=f"selector-changed-{selector_kind}") - request["target"] = {"name": "Alpha"} - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, request, socket_client_factory=_factory(calls)) - changed = _request(request_id=f"selector-changed-{selector_kind}") - changed["target"] = changed_selector - conflict = submit_command( - config, - changed, - socket_client_factory=lambda _config: pytest.fail( - "a changed selector must not create a socket client" - ), - ) - - assert accepted.status == STATUS_ACCEPTED - assert conflict.status == STATUS_DUPLICATE_REQUEST - assert conflict.disposition == DISPOSITION_TERMINAL_REJECTED - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - f"selector-changed-{selector_kind}", - ) - assert receipt is not None - assert (receipt["state"], receipt["status"]) == ("accepted", STATUS_ACCEPTED) - assert json.loads(receipt["result_json"]) == accepted.to_dict() - - -def test_submit_command_migrated_v11_exact_raw_request_replays( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - request_payload = _request(request_id="legacy-exact") - request = CommandRequest.from_dict(request_payload) - accepted = CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result={ - "target": {"worker_id": "w-1"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "target_state_at_send": "active", - "observed_turn_state": "pending_observation", - }, - ) - legacy_result = accepted.to_dict() - legacy_result.pop("disposition") - legacy_result["schema_version"] = 1 - legacy_fingerprint = request.payload_fingerprint() - result_json = json.dumps( - legacy_result, - sort_keys=True, - separators=(",", ":"), - ) - request_json = json.dumps( - request.to_dict(), - sort_keys=True, - separators=(",", ":"), - ) - with sqlite3.connect(str(config.db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0) - """, - ( - config.host_id, - request.request_id, - request.action, - legacy_fingerprint, - STATUS_ACCEPTED, - result_json, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES (?, ?, ?, ?, ?, 0, 0, ?, ?, ?, ?, ?, ?) - """, - ( - config.host_id, - request.request_id, - request.action, - legacy_fingerprint, - STATUS_ACCEPTED, - request_json, - result_json, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute("PRAGMA user_version = 11") - - init_store(config.db_path) - replay = submit_command( - config, - request_payload, - socket_client_factory=lambda _config: pytest.fail( - "legacy terminal replay must not create a socket client" - ), - ) - changed = submit_command( - config, - _request(request_id="legacy-exact", text="changed"), - socket_client_factory=lambda _config: pytest.fail( - "legacy request collision must not create a socket client" - ), - ) - - assert replay.to_dict() == accepted.to_dict() - assert replay.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert changed.status == STATUS_DUPLICATE_REQUEST - assert changed.disposition == DISPOSITION_TERMINAL_REJECTED - - -def test_submit_command_same_id_text_target_and_action_collide_without_backend( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - first_worker = Worker(id="w-1", name="Alpha", status="active") - second_worker = Worker(id="w-2", name="Beta", status="active") - _seed( - config, - [first_worker, second_worker], - [ - _binding(first_worker), - _binding( - second_worker, - value="other-agent-secret", - private_fingerprint="other-private-secret", - ), - ], - ) - calls: list[dict[str, Any]] = [] - request_id = "collision-1" - - accepted = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=_factory(calls), - ) - changed_text = submit_command( - config, - _request(request_id=request_id, text="changed"), - socket_client_factory=lambda _config: pytest.fail( - "text collision must not create a socket client" - ), - ) - changed_target = submit_command( - config, - _request(request_id=request_id, worker_id=second_worker.id), - socket_client_factory=lambda _config: pytest.fail( - "target collision must not create a socket client" - ), - ) - changed_action = submit_command( - config, - _answer_request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "action collision must not create a socket client" - ), - ) - unknown_options = _request(request_id=request_id) - unknown_options["options"] = {"mode": "invented"} - invalid_options = submit_command( - config, - unknown_options, - socket_client_factory=lambda _config: pytest.fail( - "invalid options must not create a socket client" - ), - ) - fresh_unknown_options = _request(request_id="options-invalid-fresh") - fresh_unknown_options["options"] = {"mode": "invented"} - fresh_invalid_options = submit_command( - config, - fresh_unknown_options, - socket_client_factory=lambda _config: pytest.fail( - "invalid options must not create a socket client" - ), - ) - - assert accepted.status == STATUS_ACCEPTED - assert accepted.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert changed_text.status == STATUS_DUPLICATE_REQUEST - assert changed_text.disposition == DISPOSITION_TERMINAL_REJECTED - assert changed_target.status == STATUS_DUPLICATE_REQUEST - assert changed_target.disposition == DISPOSITION_TERMINAL_REJECTED - assert changed_action.status == STATUS_DUPLICATE_REQUEST - assert changed_action.disposition == DISPOSITION_TERMINAL_REJECTED - assert invalid_options.status == STATUS_INVALID_REQUEST - assert fresh_invalid_options.status == STATUS_INVALID_REQUEST - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, request_id) - assert receipt is not None - assert receipt["state"] == "accepted" - assert receipt["public_worker_id"] == first_worker.id - assert ( - get_command_request(config.db_path, config.host_id, "options-invalid-fresh") - is None - ) - - -def test_submit_command_private_preparation_over_30_second_budget_precedes_reservation_and_sends_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path, timeout=31) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - pane_lookup_started = Event() - release_pane_lookup = Event() - first_calls: list[dict[str, Any]] = [] - second_calls: list[dict[str, Any]] = [] - clients: list[_FakeSocketClient] = [] - - class BlockingClient(_FakeSocketClient): - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - if method == "agent.get": - assert timeout == 31 - pane_lookup_started.set() - assert release_pane_lookup.wait(timeout=5) - return super().request(method, params, timeout=timeout) - - def first_factory(config: Config) -> BlockingClient: - client = BlockingClient(first_calls) - clients.append(client) - return client - - def second_factory(config: Config) -> _FakeSocketClient: - client = _FakeSocketClient(second_calls) - clients.append(client) - return client - - with ThreadPoolExecutor(max_workers=2) as executor: - first_future = executor.submit( - submit_command, - config, - _request(request_id="concurrent-1"), - socket_client_factory=first_factory, - ) - assert pane_lookup_started.wait(timeout=5) - assert config.db_path is not None - assert get_command_request(config.db_path, config.host_id, "concurrent-1") is None - second = submit_command( - config, - _request(request_id="concurrent-1"), - socket_client_factory=second_factory, - ) - release_pane_lookup.set() - first = first_future.result(timeout=5) - - assert first.status == STATUS_ACCEPTED - assert second.status == STATUS_ACCEPTED - assert first_calls == [{"method": "agent.get", "params": {"target": "agent-secret"}}] - assert second_calls == _expected_submit_calls(timeout_ms=31_000) - assert [client.close_count for client in clients] == [1, 1] - receipt = get_command_request(config.db_path, config.host_id, "concurrent-1") - assert receipt is not None - assert receipt["state"] == "accepted" - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - "SELECT COUNT(*) FROM command_receipts " - "WHERE host_id = ? AND request_id = ? AND state = 'accepted'", - (config.host_id, "concurrent-1"), - ).fetchone()[0] == 1 - assert sum(call["method"] == "agent.prompt" for call in first_calls + second_calls) == 1 - - -@pytest.mark.parametrize( - ("after_commit", "expected_status", "expected_state"), - [ - (False, STATUS_PENDING, "reserved"), - (True, STATUS_PENDING, "send_started"), - ], -) -def test_submit_command_send_start_exception_recovers_durable_state_and_closes_prepared_clients( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - after_commit: bool, - expected_status: str, - expected_state: str, -) -> None: - config = _config(tmp_path / expected_state) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - real_mark = command_submission.mark_command_send_started - calls: list[dict[str, Any]] = [] - clients: list[_FakeSocketClient] = [] - - def lose_send_start_response(*args: Any, **kwargs: Any) -> Any: - if after_commit: - result = real_mark(*args, **kwargs) - assert result["status"] == "send_started" - raise HerdrSocketTimeoutError("send-start response lost") - - def factory(config: Config) -> _FakeSocketClient: - client = _FakeSocketClient(calls) - clients.append(client) - return client - - monkeypatch.setattr( - command_submission, - "mark_command_send_started", - lose_send_start_response, - ) - first = submit_command( - config, - _request(request_id="send-start-loss"), - socket_client_factory=factory, - ) - - assert first.status == expected_status - assert first.disposition == DISPOSITION_IN_PROGRESS - assert calls == _expected_submit_calls()[:-1] - assert clients[0].close_count == 1 - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, "send-start-loss") - assert receipt is not None - assert receipt["state"] == expected_state - - if after_commit: - replay = submit_command( - config, - _request(request_id="send-start-loss"), - socket_client_factory=lambda _config: pytest.fail( - "send-started replay must not prepare another client" - ), - ) - assert replay.status == STATUS_PENDING - assert replay.disposition == DISPOSITION_IN_PROGRESS - assert len(clients) == 1 - else: - # The reservation is still owned, so both retries are decided by the - # receipt alone: no second client, no second private observation. - replay = submit_command( - config, - _request(request_id="send-start-loss"), - socket_client_factory=factory, - ) - assert replay.status == STATUS_PENDING - assert replay.disposition == DISPOSITION_IN_PROGRESS - conflict = submit_command( - config, - _request(request_id="send-start-loss", text="different"), - socket_client_factory=factory, - ) - assert conflict.status == STATUS_DUPLICATE_REQUEST - assert [client.close_count for client in clients] == [1] - assert calls == _expected_submit_calls()[:-1] - assert not any(call["method"] == "pane.send_input" for call in calls) - - -def test_submit_command_accepted_terminal_replay_delete_fences_prepared_takeover( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - initial_calls: list[dict[str, Any]] = [] - first = submit_command( - config, - _request(request_id="terminal-delete-race"), - socket_client_factory=_factory(initial_calls), - ) - assert first.status == STATUS_ACCEPTED - assert first.result["submission_verdict"] == "submitted" - assert sum(call["method"] == "agent.prompt" for call in initial_calls) == 1 - assert not any(call["method"] == "pane.send_input" for call in initial_calls) - assert config.db_path is not None - - active_worker = Worker(id="w-1", name="Alpha", status="active") - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:02:00+00:00", - workers=[active_worker], - backend_health=[_healthy_backend()], - ), - ) - upsert_worker_bindings(config.db_path, [_binding(active_worker)]) - - real_atomic_replay = command_submission.reserve_terminal_command_replay - deleted = Event() - contender_prepared = Event() - release_contender = Event() - atomic_calls = 0 - - def delete_then_insert_terminal(*args: Any, **kwargs: Any) -> Any: - nonlocal atomic_calls - atomic_calls += 1 - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "DELETE FROM command_receipts WHERE host_id = ? AND request_id = ?", - (config.host_id, "terminal-delete-race"), - ) - conn.commit() - deleted.set() - assert contender_prepared.wait(timeout=5) - try: - return real_atomic_replay(*args, **kwargs) - finally: - release_contender.set() - - monkeypatch.setattr( - command_submission, - "reserve_terminal_command_replay", - delete_then_insert_terminal, - ) - contender_calls: list[dict[str, Any]] = [] - - class PreparedContenderClient(_FakeSocketClient): - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - if method == "agent.get": - contender_prepared.set() - assert release_contender.wait(timeout=5) - return super().request(method, params, timeout=timeout) - - contender_client = PreparedContenderClient(contender_calls) - with ThreadPoolExecutor(max_workers=2) as executor: - replay_future = executor.submit( - submit_command, - config, - _request(request_id="terminal-delete-race"), - socket_client_factory=lambda _config: pytest.fail( - "terminal replay must not prepare another client" - ), - ) - assert deleted.wait(timeout=5) - contender_future = executor.submit( - submit_command, - config, - _request(request_id="terminal-delete-race"), - socket_client_factory=lambda _config: contender_client, - ) - replay = replay_future.result(timeout=5) - contender = contender_future.result(timeout=5) - - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert contender.to_dict() == replay.to_dict() - assert atomic_calls == 1 - assert contender_client.close_count == 1 - assert contender_calls == [ - {"method": "agent.get", "params": {"target": "agent-secret"}} - ] - receipt = get_command_request(config.db_path, config.host_id, "terminal-delete-race") - assert receipt is not None - assert receipt["state"] == "uncertain" - assert receipt["state"] != "reserved" - assert sum(call["method"] == "agent.prompt" for call in initial_calls) == 1 - - -def test_submit_command_timeout_before_send_start_stays_retryable( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - - class ConnectTimeoutClient: - def connect(self) -> None: - raise HerdrSocketTimeoutError("connect timeout") - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - raise AssertionError("pane operations must not run") - - def close(self) -> None: - return None - - first = submit_command( - config, - _request(request_id="before-timeout"), - socket_client_factory=lambda _config: ConnectTimeoutClient(), - ) - - # A connect timeout occurs before any request transmission, so no send began. - # It must stay retryable rather than durably reject the unsent command. - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_NO_RECEIPT - assert config.db_path is not None - assert get_command_request(config.db_path, config.host_id, "before-timeout") is None - - # A retry under the same request ID re-attempts and, once the socket - # responds, sends exactly once. - recovery_calls: list[dict[str, Any]] = [] - recovered = submit_command( - config, - _request(request_id="before-timeout"), - socket_client_factory=_factory(recovery_calls), - ) - assert recovered.status == STATUS_ACCEPTED - assert [call["method"] for call in recovery_calls].count("agent.prompt") == 1 - receipt = get_command_request(config.db_path, config.host_id, "before-timeout") - assert receipt is not None - assert receipt["state"] == "accepted" - assert receipt["send_started_at"] is not None - - -def test_submit_command_finalization_timeout_after_send_is_uncertain_and_not_retried( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - def timeout_finish(*args: Any, **kwargs: Any) -> Any: - raise HerdrSocketTimeoutError("receipt finalization timeout") - - monkeypatch.setattr(command_submission, "finish_command_request", timeout_finish) - first = submit_command( - config, - _request(request_id="after-timeout"), - socket_client_factory=_factory(calls), - ) - replay = submit_command( - config, - _request(request_id="after-timeout"), - socket_client_factory=lambda _config: pytest.fail( - "send-started replay must not create a socket client" - ), - ) - - assert first.status == STATUS_PENDING - assert replay.status == STATUS_PENDING - assert first.disposition == DISPOSITION_IN_PROGRESS - assert replay.disposition == DISPOSITION_IN_PROGRESS - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, "after-timeout") - assert receipt is not None - assert receipt["state"] == "send_started" - maintenance = cleanup_command_request_retention( - config.db_path, - retry_horizon_seconds=60, - retention_seconds=691_200, - retention_count=4096, - host_id=config.host_id, - now="2099-01-01T00:00:00+00:00", - ) - terminal = replay_command_receipt( - config, - _request(request_id="after-timeout"), - ) - assert maintenance["stale_active"] == 1 - assert terminal is not None - assert terminal.status == STATUS_REQUEST_STATE_UNCERTAIN - assert terminal.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == _expected_submit_calls() - - -def test_submit_command_accepted_finalization_response_loss_replays_accepted( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - real_finish = command_submission.finish_command_request - - def finish_then_lose_response(*args: Any, **kwargs: Any) -> Any: - result = real_finish(*args, **kwargs) - assert result["status"] == "accepted" - raise HerdrSocketTimeoutError("accepted response lost") - - monkeypatch.setattr( - command_submission, - "finish_command_request", - finish_then_lose_response, - ) - first = submit_command( - config, - _request(request_id="accepted-response-loss"), - socket_client_factory=_factory(calls), - ) - replay = submit_command( - config, - _request(request_id="accepted-response-loss"), - socket_client_factory=lambda _config: pytest.fail( - "accepted replay must not create a socket client" - ), - ) - - assert first.status == STATUS_ACCEPTED - assert first.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert replay.to_dict() == first.to_dict() - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - "accepted-response-loss", - ) - assert receipt is not None - assert receipt["state"] == "accepted" - turns = turns_payload_from_store(config.db_path, config.host_id)["turns"] - assert turns == [] - - -@pytest.mark.parametrize( - ("schema_version", "legacy_v1"), - [ - pytest.param(True, True, id="v1-bool-alias"), - pytest.param(1.0, True, id="v1-float-alias"), - pytest.param(2.0, False, id="v2-float-alias"), - ], -) -def test_replay_command_receipt_rejects_non_exact_stored_schema_versions( - tmp_path: Path, - schema_version: Any, - legacy_v1: bool, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_payload = _request(request_id=f"schema-alias-{type(schema_version).__name__}") - accepted = submit_command( - config, - request_payload, - socket_client_factory=_factory(calls), - ) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - request_payload["request_id"], - ) - assert receipt is not None - stored = json.loads(receipt["result_json"]) - if legacy_v1: - stored.pop("disposition") - stored["schema_version"] = schema_version - malformed_json = json.dumps(stored, sort_keys=True, separators=(",", ":")) - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET result_json = ? " - "WHERE host_id = ? AND request_id = ?", - (malformed_json, config.host_id, request_payload["request_id"]), - ) - conn.commit() - rows_before = conn.execute( - "SELECT * FROM command_receipts ORDER BY id" - ).fetchall() - - replay = replay_command_receipt(config, request_payload) - - assert replay is not None - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert replay.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == _expected_submit_calls() - with sqlite3.connect(str(config.db_path)) as conn: - assert ( - conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - == rows_before - ) - - -def test_response_loss_replay_uses_only_current_exact_worker_selector( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_payload = _request(request_id="response-loss-target") - accepted = submit_command( - config, - request_payload, - socket_client_factory=_factory(calls), - ) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - - def stored_rows() -> tuple[list[tuple[Any, ...]], list[tuple[Any, ...]]]: - with sqlite3.connect(str(config.db_path)) as conn: - return ( - conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall(), - conn.execute("SELECT * FROM events ORDER BY id").fetchall(), - ) - - rows_before = stored_rows() - exact = replay_command_receipt(config, request_payload) - changed_request = _request(request_id="response-loss-target", worker_id="w-2") - changed = replay_command_receipt(config, changed_request) - - monkeypatch.setattr( - command_submission, - "_current_snapshot", - lambda _config: pytest.fail( - "read-only response-loss reconciliation must not consult current authority" - ), - ) - mutable_request = _request(request_id="response-loss-target") - mutable_request["target"] = {"name": "Alpha"} - mutable = replay_command_receipt(config, mutable_request) - - assert exact is not None - assert exact.to_dict() == accepted.to_dict() - assert changed is not None - assert changed.status == STATUS_DUPLICATE_REQUEST - assert changed.disposition == DISPOSITION_TERMINAL_REJECTED - assert mutable is None - assert calls == _expected_submit_calls() - assert stored_rows() == rows_before - - -@pytest.mark.parametrize("damage", ["malformed_result", "illegal_state"]) -def test_submit_command_illegal_or_malformed_terminal_receipt_fails_closed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - damage: str, -) -> None: - config = _config(tmp_path / damage) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = f"damaged-{damage}" - accepted = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=_factory(calls), - ) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - - if damage == "malformed_result": - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET result_json = '{' " - "WHERE host_id = ? AND request_id = ?", - (config.host_id, request_id), - ) - conn.commit() - else: - real_reserve = command_submission.reserve_terminal_command_replay - - def illegal_reserve(*args: Any, **kwargs: Any) -> dict[str, Any]: - result = real_reserve(*args, **kwargs) - receipt = dict(result["receipt"]) - receipt["state"] = "illegal" - return {**result, "receipt": receipt} - - monkeypatch.setattr( - command_submission, - "reserve_terminal_command_replay", - illegal_reserve, - ) - - replay = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "damaged receipt replay must not create a socket client" - ), - ) - - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert replay.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == _expected_submit_calls() - - -def _answer_request( - *, - request_id: str = "answer-1", - dry_run: bool = False, - pending_id: str = "pending-public", - pending_fingerprint: str = "revision-public", - choice_id: str = "choice-public", -) -> dict[str, Any]: - return { - "schema_version": 1, - "action": "answer_pending", - "request_id": request_id, - "dry_run": dry_run, - "params": { - "pending_id": pending_id, - "pending_fingerprint": pending_fingerprint, - "choice_id": choice_id, - }, - } - - -@pytest.mark.parametrize("terminal_status", [STATUS_ACCEPTED, STATUS_REJECTED]) -def test_answer_pending_migrated_v11_terminal_replays_without_current_authority( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - terminal_status: str, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - request_payload = _answer_request(request_id=f"legacy-answer-{terminal_status}") - request = CommandRequest.from_dict(request_payload) - if terminal_status == STATUS_ACCEPTED: - terminal = CommandEnvelope.from_result( - request, - ok=True, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - status=terminal_status, - result={ - "target": {"worker_id": "legacy-worker"}, - "pending": { - "id": "pending-public", - "fingerprint": "revision-public", - }, - "choice": {"choice_id": "choice-public"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - }, - ) - else: - terminal = CommandEnvelope.from_result( - request, - ok=False, - disposition=DISPOSITION_TERMINAL_REJECTED, - status=terminal_status, - error={ - "code": terminal_status, - "message": "legacy pending answer was rejected before send", - }, - ) - raw_fingerprint = request.payload_fingerprint() - expected_terminal = terminal.to_dict() - legacy_terminal = dict(expected_terminal) - legacy_terminal.pop("disposition") - legacy_terminal["schema_version"] = 1 - result_json = json.dumps( - legacy_terminal, - sort_keys=True, - separators=(",", ":"), - ) - request_json = json.dumps( - request.to_dict(), - sort_keys=True, - separators=(",", ":"), - ) - with sqlite3.connect(str(config.db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0) - """, - ( - config.host_id, - request.request_id, - request.action, - raw_fingerprint, - terminal_status, - result_json, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES (?, ?, ?, ?, ?, 0, 0, ?, ?, ?, ?, ?, ?) - """, - ( - config.host_id, - request.request_id, - request.action, - raw_fingerprint, - terminal_status, - request_json, - result_json, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute("PRAGMA user_version = 11") - - init_store(config.db_path) - migrated = get_command_request( - config.db_path, - config.host_id, - request.request_id or "", - ) - assert migrated is not None - assert migrated["canonical_version"] == 0 - assert migrated["canonical_fingerprint"] == raw_fingerprint - assert migrated["public_worker_id"] == "" - assert migrated["state"] == ( - "accepted" if terminal_status == STATUS_ACCEPTED else "rejected" - ) - assert migrated["legacy_collision"] is False - - def command_rows() -> tuple[list[tuple[Any, ...]], list[tuple[Any, ...]]]: - with sqlite3.connect(str(config.db_path)) as conn: - return ( - conn.execute( - "SELECT * FROM command_receipts ORDER BY id" - ).fetchall(), - conn.execute("SELECT * FROM commands ORDER BY id").fetchall(), - ) - - rows_before = command_rows() - - def forbidden(*args: Any, **kwargs: Any) -> Any: - pytest.fail("legacy terminal replay must not consult or mutate current authority") - - monkeypatch.setattr(command_submission, "_validate_pending_choice", forbidden) - monkeypatch.setattr(command_submission, "_answer_pending", forbidden) - monkeypatch.setattr(command_submission, "reserve_command_request", forbidden) - - read_only_replay = replay_command_receipt(config, request_payload) - replay = submit_command( - config, - request_payload, - socket_client_factory=forbidden, - ) - changed_choice = submit_command( - config, - _answer_request( - request_id=request.request_id or "", - choice_id="changed-choice", - ), - socket_client_factory=forbidden, - ) - - assert read_only_replay is not None - assert read_only_replay.to_dict() == expected_terminal - assert replay.to_dict() == expected_terminal - assert replay.disposition == ( - DISPOSITION_TERMINAL_ACCEPTED - if terminal_status == STATUS_ACCEPTED - else DISPOSITION_TERMINAL_REJECTED - ) - assert changed_choice.status == STATUS_DUPLICATE_REQUEST - assert changed_choice.disposition == DISPOSITION_TERMINAL_REJECTED - assert command_rows() == rows_before - - -class _PendingClaim: - def __init__( - self, - status: str, - worker: Worker, - *, - claim_token: str | None, - private_fingerprint: str = "binding-private", - turn_target_value: str = "pane-secret", - picker_ordinal: int = 2, - ) -> None: - self.status = status - self.claim_token = claim_token - self.worker_id = worker.id if status in {"claimed", "validated"} else None - self.worker_fingerprint = worker.fingerprint if status in {"claimed", "validated"} else None - self.binding_private_fingerprint = ( - private_fingerprint if status in {"claimed", "validated"} else None - ) - self.turn_target_value = ( - turn_target_value if status in {"claimed", "validated"} else None - ) - self.picker_ordinal = picker_ordinal if status in {"claimed", "validated"} else None - - -class _PendingSend: - def __init__( - self, - status: str, - worker: Worker, - *, - private_fingerprint: str = "binding-private", - turn_target_value: str = "pane-secret", - picker_ordinal: int = 2, - ) -> None: - self.status = status - self.worker_id = worker.id if status == "started" else None - self.worker_fingerprint = worker.fingerprint if status == "started" else None - self.binding_private_fingerprint = private_fingerprint if status == "started" else None - self.turn_target_value = turn_target_value if status == "started" else None - self.picker_ordinal = picker_ordinal if status == "started" else None - - -def _patch_pending_store_flow( - monkeypatch: pytest.MonkeyPatch, - worker: Worker, - *, - claim_status: str = "claimed", - private_fingerprint: str = "binding-private", - picker_ordinal: int = 2, - turn_target_value: str = "pane-secret", - finish_result: bool = True, -) -> list[tuple[Any, ...]]: - transitions: list[tuple[Any, ...]] = [] - - def claim( - db_path: Path, - host_id: str, - pending_id: str, - pending_fingerprint: str, - choice_id: str, - *, - claim: bool = True, - observed_at: str | None = None, - ) -> _PendingClaim: - transitions.append( - ("claim", claim, pending_id, pending_fingerprint, choice_id, observed_at) - ) - status = claim_status - token: str | None = "claim-private" if status == "claimed" else None - if not claim and status == "claimed": - status = "validated" - return _PendingClaim( - status, - worker, - claim_token=token, - private_fingerprint=private_fingerprint, - turn_target_value=turn_target_value, - picker_ordinal=picker_ordinal, - ) - - def start( - db_path: Path, - host_id: str, - claim_token: str, - *, - observed_at: str | None = None, - ) -> _PendingSend: - transitions.append(("start", claim_token, observed_at)) - return _PendingSend( - "started", - worker, - private_fingerprint=private_fingerprint, - turn_target_value=turn_target_value, - picker_ordinal=picker_ordinal, - ) - - def finish_effect( - *, - host_id: str, - claim_token: str, - accepted: bool, - ) -> Any: - def apply(conn: sqlite3.Connection) -> None: - transitions.append(("finish", claim_token, accepted)) - if not finish_result: - raise RuntimeError("pending finish failed") - - return apply - - def abandon(db_path: Path, host_id: str, claim_token: str) -> bool: - transitions.append(("abandon", claim_token)) - return True - - monkeypatch.setattr(command_submission, "claim_backend_pending_choice", claim) - monkeypatch.setattr(command_submission, "start_backend_pending_choice_send", start) - monkeypatch.setattr( - command_submission, - "backend_pending_choice_terminal_effect", - finish_effect, - ) - monkeypatch.setattr(command_submission, "abandon_backend_pending_choice_claim", abandon) - return transitions - - -def _expected_answer_calls( - *, - pane_id: str = "pane-secret", - ordinal: int = 2, -) -> list[dict[str, Any]]: - return [ - *_expected_private_clear_calls(pane_id), - {"method": "pane.send_input", "params": {"pane_id": pane_id, "text": str(ordinal), "keys": ["Enter"]}}, - ] - - -@pytest.mark.parametrize( - ("payload", "expected_result"), - [ - ( - { - "schema_version": 1, - "action": "send_instruction", - "dry_run": True, - "target": {"name": "Alpha", "space_id": "space-1"}, - "instruction": {"text": "hello"}, - }, - { - "target": {"name": "Alpha", "space_id": "space-1"}, - "instruction": {"text": "hello"}, - }, - ), - ( - _answer_request(request_id="", dry_run=True), - { - "pending": { - "id": "pending-public", - "fingerprint": "revision-public", - }, - "choice": {"choice_id": "choice-public"}, - "delivery_state": "not_submitted", - }, - ), - ], -) -def test_mutation_dry_run_is_pure_without_store_snapshot_or_socket( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - payload: dict[str, Any], - expected_result: dict[str, Any], -) -> None: - config = _config(tmp_path, backend="cli") - calls: list[str] = [] - - def forbidden(*args: Any, **kwargs: Any) -> Any: - calls.append("io") - raise AssertionError("dry-run must not consult mutable command authority") - - monkeypatch.setattr(command_submission, "get_command_request", forbidden) - monkeypatch.setattr(command_submission, "_current_snapshot", forbidden) - monkeypatch.setattr(command_submission, "_validate_pending_choice", forbidden) - monkeypatch.setattr(command_submission, "reserve_command_request", forbidden) - - envelope = submit_command( - config, - payload, - socket_client_factory=forbidden, - ) - - assert envelope.ok is True - assert envelope.status == "dry_run" - assert envelope.result == expected_result - assert calls == [] - assert config.db_path is not None - assert not config.db_path.exists() -def test_answer_pending_reacquired_reservation_worker_drift_finishes_rejected( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - current_worker = Worker(id="w-2", name="Beta", status="active") - _seed(config, [current_worker], [_binding(current_worker)]) - payload = _answer_request(request_id="answer-reacquired") - request = CommandRequest.from_dict(payload) - canonical = build_canonical_mutation(request, public_worker_id="w-1") - pending = command_submission._request_in_progress(request) - assert config.db_path is not None - initial = reserve_command_request( - config.db_path, - host_id=config.host_id, - request_id=request.request_id or "", - action=canonical.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=store_sqlite.envelope_to_receipt_json(pending), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), - owner_lease_seconds=1, - now="2020-01-01T00:00:00+00:00", - ) - assert initial["status"] == "reserved" - transitions = _patch_pending_store_flow(monkeypatch, current_worker) - calls: list[dict[str, Any]] = [] - - drift = submit_command( - config, - payload, - socket_client_factory=_factory(calls), - ) - replay = submit_command( - config, - payload, - socket_client_factory=_factory(calls), - ) - - assert drift.status == STATUS_DUPLICATE_REQUEST - assert replay.to_dict() == drift.to_dict() - assert calls == [] - assert transitions == [ - ( - "claim", - False, - "pending-public", - "revision-public", - "choice-public", - None, - ) - ] - receipt = get_command_request( - config.db_path, - config.host_id, - request.request_id or "", - ) - assert receipt is not None - assert receipt["state"] == "rejected" - assert receipt["status"] == STATUS_DUPLICATE_REQUEST - - -def test_answer_pending_claims_sends_only_ordinal_and_replays_receipt( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - monkeypatch.setattr( - command_submission, - "list_worker_bindings", - lambda *args, **kwargs: pytest.fail( - "answer_pending must use the binding authenticated by the claim API" - ), - ) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _answer_request(), - socket_client_factory=_factory(calls), - ) - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[worker], - backend_health=[_degraded_backend()], - ), - ) - replay = submit_command( - config, - _answer_request(), - socket_client_factory=_factory(calls), - ) - changed_choice = submit_command( - config, - _answer_request(choice_id="different-choice"), - socket_client_factory=lambda _config: pytest.fail( - "choice collision must not create a socket client" - ), - ) - changed_pending_revision = submit_command( - config, - _answer_request(pending_fingerprint="different-revision"), - socket_client_factory=lambda _config: pytest.fail( - "pending collision must not create a socket client" - ), - ) - - assert first.ok is True - assert first.status == STATUS_ACCEPTED - assert first.result == { - "target": {"worker_id": "w-1"}, - "pending": {"id": "pending-public", "fingerprint": "revision-public"}, - "choice": {"choice_id": "choice-public"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - } - assert replay.to_dict() == first.to_dict() - assert changed_choice.status == STATUS_DUPLICATE_REQUEST - assert changed_pending_revision.status == STATUS_DUPLICATE_REQUEST - assert calls == _expected_answer_calls() - assert transitions == [ - ( - "claim", - False, - "pending-public", - "revision-public", - "choice-public", - None, - ), - ( - "claim", - True, - "pending-public", - "revision-public", - "choice-public", - None, - ), - ("start", "claim-private", None), - ("finish", "claim-private", True), - ] - - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, "cmd-host", "answer-1", "answer_pending") - assert receipt is not None - assert receipt["status"] == STATUS_ACCEPTED - turns = turns_payload_from_store(config.db_path, config.host_id)["turns"] - assert not any(turn.get("origin_command_id") == "answer-1" for turn in turns) - - -def test_answer_pending_observed_mode_completes_without_instruction_turn( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path, turn_model="observed") - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - calls: list[dict[str, Any]] = [] - - result = submit_command( - config, - _answer_request(request_id="observed-answer-pending"), - socket_client_factory=_factory(calls), - ) - - assert result.ok is True - assert result.status == STATUS_ACCEPTED - assert calls == _expected_answer_calls() - assert transitions[-2:] == [ - ("start", "claim-private", None), - ("finish", "claim-private", True), - ] - - -@pytest.mark.parametrize( - "claim_status", - ["not_found", "stale", "changed", "unknown_choice", "already_claimed"], -) -def test_answer_pending_changed_disappeared_stale_or_unknown_fails_before_socket( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - claim_status: str, -) -> None: - config = _config(tmp_path / claim_status) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow( - monkeypatch, - worker, - claim_status=claim_status, - ) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _answer_request(request_id=f"answer-{claim_status}"), - socket_client_factory=_factory(calls), - ) - - assert envelope.ok is False - assert envelope.status == STATUS_STALE_TARGET - assert envelope.error == { - "code": STATUS_STALE_TARGET, - "message": "pending interaction changed or is no longer answerable", - "details": {}, - } - assert calls == [] - assert len(transitions) == 1 - assert transitions[0][0] == "claim" - - - - -def test_answer_pending_claim_race_closes_prepared_loser_without_second_send( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - original_claim = command_submission.claim_backend_pending_choice - mutating_claim_count = 0 - - def racing_claim(*args: Any, **kwargs: Any) -> _PendingClaim: - nonlocal mutating_claim_count - if kwargs.get("claim", True): - mutating_claim_count += 1 - if mutating_claim_count == 2: - transitions.append(("claim_race_lost",)) - return _PendingClaim("already_claimed", worker, claim_token=None) - return original_claim(*args, **kwargs) - - monkeypatch.setattr(command_submission, "claim_backend_pending_choice", racing_claim) - calls: list[dict[str, Any]] = [] - nested_result: list[Any] = [] - clients: list[_FakeSocketClient] = [] - - def nested_factory(config: Config) -> _FakeSocketClient: - client = _FakeSocketClient(calls) - clients.append(client) - return client - - class _RacingClient(_FakeSocketClient): - def connect(self) -> "_FakeSocketClient": - nested_result.append( - submit_command( - config, - _answer_request(request_id="answer-race-nested"), - socket_client_factory=nested_factory, - ) - ) - return self - - outer_client = _RacingClient(calls) - clients.append(outer_client) - outer = submit_command( - config, - _answer_request(request_id="answer-race-outer"), - socket_client_factory=lambda _config: outer_client, - ) - - assert outer.status == STATUS_STALE_TARGET - assert len(nested_result) == 1 - assert nested_result[0].status == STATUS_ACCEPTED - assert calls == _expected_answer_calls() - assert [client.close_count for client in clients] == [1, 1] - - -def test_answer_pending_post_send_failure_is_uncertain_and_not_retried( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _answer_request(request_id="answer-uncertain"), - socket_client_factory=_factory( - calls, - raises=HerdrSocketDisconnectedError("disconnected"), - ), - ) - replay = submit_command( - config, - _answer_request(request_id="answer-uncertain"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert transitions[-1] == ("finish", "claim-private", False) - assert calls == [ - *_expected_private_clear_calls(), - {"method": "pane.send_input", "params": {"pane_id": "pane-secret", "text": "2", "keys": ["Enter"]}}, - ] - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, - "cmd-host", - "answer-uncertain", - "answer_pending",) - assert receipt is not None - assert receipt["uncertain"] is True - - -def test_answer_pending_public_surfaces_recursively_exclude_private_route_values( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - private_binding = "raw-binding-sentinel" - private_target = "raw-target-sentinel" - private_pane = "raw-pane-sentinel" - _seed( - config, - [worker], - [ - _binding( - worker, - value=private_target, - private_fingerprint=private_binding, - turn_target_kind="pane_id", - turn_target_value=private_pane, - ) - ], - ) - _patch_pending_store_flow( - monkeypatch, - worker, - private_fingerprint=private_binding, - turn_target_value=private_pane, - picker_ordinal=3, - ) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _answer_request(request_id="answer-private"), - socket_client_factory=_factory(calls, pane_id=private_pane), - ) - - assert envelope.status == STATUS_ACCEPTED - assert calls == _expected_answer_calls( - pane_id=private_pane, - ordinal=3, - ) - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, - "cmd-host", - "answer-private", - "answer_pending",) - assert receipt is not None - with sqlite3.connect(str(config.db_path)) as conn: - event_payloads = [ - json.loads(row[0]) - for row in conn.execute( - "SELECT payload_json FROM events WHERE aggregate_id = ? ORDER BY id", - ("answer-private",), - ).fetchall() - ] - public_surfaces = [ - envelope.to_dict(), - json.loads(receipt["result_json"]), - *event_payloads, - ] - encoded = json.dumps(public_surfaces) - assert private_binding not in encoded - assert private_target not in encoded - assert private_pane not in encoded - for surface in public_surfaces: - _assert_no_private_json(surface) - - -def test_answer_pending_integrates_with_durable_two_phase_claim( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - binding = _binding( - worker, - private_fingerprint="binding-private", - turn_target_kind="pane_id", - turn_target_value="pane-secret", - ) - _seed(config, [worker], [binding]) - assert config.db_path is not None - assert apply_backend_pending_observation( - config.db_path, - config.host_id, - worker.id, - PendingObservation( - kind="open_prompt", - question="Choose a safe option", - pending_kind="choice", - choices=( - PendingObservedChoice( - choice_id="choice-aaaaaaaaaaaaaaaaaaaaaaaa", - label="First", - picker_ordinal=1, - ), - PendingObservedChoice( - choice_id="choice-bbbbbbbbbbbbbbbbbbbbbbbb", - label="Second", - picker_ordinal=2, - ), - ), - revision_digest="private-revision-digest", - ), - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - pending_before = pending_payload_from_store(config.db_path, config.host_id) - assert len(pending_before["pending_interactions"]) == 1 - interaction = pending_before["pending_interactions"][0] - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _answer_request( - request_id="answer-real-claim", - pending_id=interaction["id"], - pending_fingerprint=interaction["fingerprint"], - choice_id=interaction["choices"][1]["choice_id"], - ), - socket_client_factory=_factory(calls), - ) - - assert envelope.status == STATUS_ACCEPTED - assert envelope.result == { - "target": {"worker_id": worker.id}, - "pending": { - "id": interaction["id"], - "fingerprint": interaction["fingerprint"], - }, - "choice": {"choice_id": interaction["choices"][1]["choice_id"]}, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - } - assert calls == _expected_answer_calls(ordinal=2) - pending_after = pending_payload_from_store(config.db_path, config.host_id) - assert pending_after["pending_interactions"] == [] - - -@pytest.mark.parametrize("start_status", ["changed", "stale", "binding_changed"]) -def test_answer_pending_post_receipt_send_start_cas_change_is_uncertain_without_pane_mutation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - start_status: str, -) -> None: - config = _config(tmp_path / start_status) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - - def changed_start( - db_path: Path, - host_id: str, - claim_token: str, - *, - observed_at: str | None = None, - ) -> _PendingSend: - transitions.append(("start", claim_token, start_status)) - return _PendingSend(start_status, worker) - - monkeypatch.setattr( - command_submission, - "start_backend_pending_choice_send", - changed_start, - ) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _answer_request(request_id=f"answer-start-{start_status}"), - socket_client_factory=_factory(calls), - ) - - assert envelope.status == STATUS_REQUEST_STATE_UNCERTAIN - assert envelope.error == { - "code": STATUS_REQUEST_STATE_UNCERTAIN, - "message": "previous request state is uncertain; not retrying mutation", - "details": {}, - } - assert calls == [] - assert transitions[-2:] == [ - ("abandon", "claim-private"), - ("finish", "claim-private", False), - ] - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - f"answer-start-{start_status}", - ) - assert receipt is not None - assert receipt["state"] == "uncertain" - - - -@pytest.mark.parametrize( - ("claim_released", "expected_status", "expected_state"), - [ - (True, STATUS_PENDING, "reserved"), - (False, STATUS_REQUEST_STATE_UNCERTAIN, "uncertain"), - ], -) -def test_answer_pending_send_start_exception_is_retryable_only_after_claim_release( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - claim_released: bool, - expected_status: str, - expected_state: str, -) -> None: - config = _config(tmp_path / expected_state) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - - if not claim_released: - def fail_abandon(db_path: Path, host_id: str, claim_token: str) -> bool: - transitions.append(("abandon_failed", claim_token)) - return False - - monkeypatch.setattr( - command_submission, - "abandon_backend_pending_choice_claim", - fail_abandon, - ) - - def fail_before_send_start(*args: Any, **kwargs: Any) -> Any: - raise HerdrSocketTimeoutError("send-start unavailable before commit") - - monkeypatch.setattr( - command_submission, - "mark_command_send_started", - fail_before_send_start, - ) - client = _FakeSocketClient([]) - envelope = submit_command( - config, - _answer_request(request_id="answer-mark-failure"), - socket_client_factory=lambda _config: client, - ) - - assert envelope.status == expected_status - assert client.close_count == 1 - assert client.calls == [] - assert transitions[-1] == ( - ("abandon", "claim-private") - if claim_released - else ("abandon_failed", "claim-private") - ) - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - "answer-mark-failure", - ) - assert receipt is not None - assert receipt["state"] == expected_state - -def test_answer_pending_socket_setup_failure_precedes_claim_and_stays_retryable( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="replacement-private")], - ) - transitions = _patch_pending_store_flow( - monkeypatch, - worker, - private_fingerprint="claimed-private", - ) - - connect_ok = {"value": False} - - def flaky_factory(config: Config) -> Any: - if not connect_ok["value"]: - raise OSError("socket unavailable") - return _FakeSocketClient([]) - - envelope = submit_command( - config, - _answer_request(request_id="answer-setup-failed"), - socket_client_factory=flaky_factory, - ) - - # The socket could not be reached, before any pending choice was claimed and - # before any transmission. That is a safe pre-send transient. - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert envelope.error == { - "code": STATUS_BACKEND_UNAVAILABLE, - "message": "Herdr socket could not be reached", - "details": {}, - } - # Only the read-only validation ran; nothing was claimed or sent. - assert transitions == [ - ( - "claim", - False, - "pending-public", - "revision-public", - "choice-public", - None, - ) - ] - assert config.db_path is not None - assert _receipt_for_action( - config.db_path, - config.host_id, - "answer-setup-failed", - "answer_pending", - ) is None - - # The same request ID answers exactly once after the socket recovers. - connect_ok["value"] = True - recovered = submit_command( - config, - _answer_request(request_id="answer-setup-failed"), - socket_client_factory=flaky_factory, - ) - assert recovered.status == STATUS_ACCEPTED - assert recovered.disposition == DISPOSITION_TERMINAL_ACCEPTED - receipt = _receipt_for_action( - config.db_path, - config.host_id, - "answer-setup-failed", - "answer_pending", - ) - assert receipt is not None - assert receipt["state"] == "accepted" - assert ("finish", "claim-private", True) in transitions - - -@pytest.mark.parametrize("turn_model", sorted(TURN_MODELS)) -def test_submit_under_any_turn_model_never_creates_a_turn_row( - tmp_path: Path, - turn_model: str, -) -> None: - config = _config(tmp_path, turn_model=turn_model) - assert config.db_path is not None - stable_key = "wsk1_" + ("e" * 64) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={"stable_key": stable_key, "stable_key_version": 1}, - ) - _seed(config, [worker], [_binding(worker)]) - - accepted = submit_command( - config, - _request(request_id=f"{turn_model}-submit", response_schema_version=3), - socket_client_factory=_factory([]), - ) - - assert accepted.status == STATUS_ACCEPTED - assert accepted.schema_version == 3 - assert accepted.result["turn_id"] is None - assert accepted.result["submission_id"] == turn_submission_id( - config.host_id, - f"{turn_model}-submit", - ) - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM turns").fetchone() == (0,) - assert conn.execute( - """ - SELECT state, linked_turn_id FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, f"{turn_model}-submit"), - ).fetchone() == ("submitted", None) - - -def test_observed_turn_identity_and_link_are_order_independent( - tmp_path: Path, -) -> None: - def exercise(order: str) -> tuple[str, dict[str, Any]]: - case_path = tmp_path / order - config = _config( - case_path, - turn_model="observed", - submission_link_window_seconds=30, - ) - assert config.db_path is not None - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={ - "stable_key": "wsk1_" + ("f" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - request = _request( - request_id=f"observed-{order}", - response_schema_version=3, - ) - - if order == "observation-first": - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "shared-source-turn", - "user_text": "hello", - "assistant_final_text": "done", - "complete": True, - "has_open_turn": False, - }, - turn_model="observed", - ) == 1 - accepted = submit_command( - config, - request, - socket_client_factory=_factory([]), - ) - if order == "submission-first": - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "shared-source-turn", - "user_text": "hello", - "assistant_final_text": "done", - "complete": True, - "has_open_turn": False, - }, - turn_model="observed", - ) == 1 - assert accepted.result["turn_id"] is None - with sqlite3.connect(str(config.db_path)) as conn: - expires_at = str( - conn.execute( - """ - SELECT link_expires_at FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, f"observed-{order}"), - ).fetchone()[0] - ) - store_sqlite.sweep_submission_links( - config.db_path, - host_id=config.host_id, - now=(datetime.fromisoformat(expires_at) + timedelta(seconds=1)).isoformat(), - ) - replayed = submit_command( - config, - request, - socket_client_factory=_factory([]), - ) - assert replayed.schema_version == 3 - assert isinstance(replayed.result["turn_id"], str) - with sqlite3.connect(str(config.db_path)) as conn: - rows = conn.execute( - "SELECT turn_id, payload_json FROM turns ORDER BY turn_id" - ).fetchall() - link = conn.execute( - """ - SELECT state, linked_turn_id FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, f"observed-{order}"), - ).fetchone() - assert len(rows) == 1 - turn_id, payload_json = rows[0] - payload = json.loads(payload_json) - assert payload.get("origin_command_id") is None - assert Turn.from_dict(payload).id == turn_id - assert link == ("linked", turn_id) - return str(turn_id), replayed.result - - submission_first = exercise("submission-first") - observation_first = exercise("observation-first") - assert submission_first[0] == observation_first[0] - assert submission_first[1]["turn_id"] == observation_first[1]["turn_id"] - - -def test_observed_identical_submissions_fail_closed_and_unobserved_expires( - tmp_path: Path, -) -> None: - config = _config( - tmp_path, - turn_model="observed", - submission_link_window_seconds=5, - submission_hard_ttl_seconds=30, - ) - assert config.db_path is not None - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={ - "stable_key": "wsk1_" + ("1" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - for request_id in ("identical-a", "identical-b"): - assert submit_command( - config, - _request(request_id=request_id, response_schema_version=3), - socket_client_factory=_factory([]), - ).status == STATUS_ACCEPTED - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "identical-source", - "user_text": "hello", - "assistant_final_text": "one observation", - "complete": True, - "has_open_turn": False, - }, - turn_model="observed", - ) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - latest_expiry = max( - datetime.fromisoformat(str(row[0])) - for row in conn.execute( - "SELECT link_expires_at FROM turn_submissions" - ).fetchall() - ) - store_sqlite.sweep_submission_links( - config.db_path, - host_id=config.host_id, - now=(latest_expiry + timedelta(seconds=1)).isoformat(), - ) - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - "SELECT state, linked_turn_id FROM turn_submissions ORDER BY request_id" - ).fetchall() == [("ambiguous", None), ("ambiguous", None)] - - expiry_config = _config( - tmp_path / "expiry", - turn_model="observed", - submission_link_window_seconds=1, - submission_hard_ttl_seconds=1, - ) - assert expiry_config.db_path is not None - _seed(expiry_config, [worker], [_binding(worker)]) - assert submit_command( - expiry_config, - _request(request_id="never-observed", response_schema_version=3), - socket_client_factory=_factory([]), - ).status == STATUS_ACCEPTED - with sqlite3.connect(str(expiry_config.db_path)) as conn: - hard_expiry = datetime.fromisoformat( - str( - conn.execute( - "SELECT hard_expires_at FROM turn_submissions" - ).fetchone()[0] - ) - ) - store_sqlite.sweep_submission_links( - expiry_config.db_path, - host_id=expiry_config.host_id, - now=(hard_expiry + timedelta(seconds=1)).isoformat(), - ) - with sqlite3.connect(str(expiry_config.db_path)) as conn: - assert conn.execute("SELECT state FROM turn_submissions").fetchone() == ( - "expired", - ) - assert conn.execute("SELECT COUNT(*) FROM turns").fetchone() == (0,) - assert conn.execute( - "SELECT COUNT(*) FROM turn_change_journal" - ).fetchone() == (0,) - - -def test_terminal_id_binding_falls_back_to_agent_list_when_agent_get_refuses(monkeypatch) -> None: - # Herdr 0.7.5 regression: agent.get no longer resolves terminal-id targets; - # agent.list still publishes terminal_id -> pane_id. A definite error from - # agent.get must fall back to the listing instead of terminalizing. - from tendwire import command_submission as cs - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - calls = [] - - def fake_socket_request(client, method, params, *, timeout): - calls.append(method) - if method == "agent.get": - raise HerdrErrorResponse({"code": "agent_not_found", "message": "agent target term_new not found"}, "req-1") - if method == "agent.list": - return { - "agents": [ - {"terminal_id": "term_other", "pane_id": "w1:p1"}, - {"terminal_id": "term_new", "pane_id": "w1:p9"}, - ] - } - raise AssertionError(f"unexpected method {method}") - - monkeypatch.setattr(cs, "_socket_request", fake_socket_request) - - class _Binding: - target_kind = "terminal_id" - target_value = "term_new" - - pane_id = cs._private_pane_id_for_binding(object(), _Binding(), timeout=5.0) - assert pane_id == "w1:p9" - assert calls == ["agent.get", "agent.list"] - - # A non-terminal binding kind must NOT consult the listing: the original - # error propagates. - class _SessionBinding: - target_kind = "agent_session" - target_value = "sess-1" - - calls.clear() - try: - cs._private_pane_id_for_binding(object(), _SessionBinding(), timeout=5.0) - raise AssertionError("expected HerdrErrorResponse to propagate") - except HerdrErrorResponse: - pass - assert calls == ["agent.get"] - - -def test_terminal_id_agent_list_fallback_rejects_distinct_conflicting_matches( - monkeypatch, -) -> None: - from tendwire import command_submission as cs - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - original_error = HerdrErrorResponse( - {"code": "agent_not_found", "message": "agent target term_dup not found"}, - "req-1", - ) - - def fake_socket_request(client, method, params, *, timeout): - if method == "agent.get": - raise original_error - if method == "agent.list": - return { - "agents": [ - {"terminal_id": "term_dup", "pane_id": "w1:p1"}, - {"terminal_id": "term_dup", "pane_id": "w1:p2"}, - ] - } - raise AssertionError(f"unexpected method {method}") - - monkeypatch.setattr(cs, "_socket_request", fake_socket_request) - - class _Binding: - target_kind = "terminal_id" - target_value = "term_dup" - - with pytest.raises(ValueError, match="ambiguous agent.list terminal_id match"): - cs._private_pane_id_for_binding(object(), _Binding(), timeout=5.0) - - -def test_terminal_id_agent_list_zero_matches_reraise_original_error( - monkeypatch, -) -> None: - from tendwire import command_submission as cs - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - original_error = HerdrErrorResponse( - {"code": "agent_not_found", "message": "agent target term_gone not found"}, - "req-1", - ) - - def fake_socket_request(client, method, params, *, timeout): - if method == "agent.get": - raise original_error - if method == "agent.list": - return { - "agents": [ - {"terminal_id": "term_other", "pane_id": "w1:p9"}, - ] - } - raise AssertionError(f"unexpected method {method}") - - monkeypatch.setattr(cs, "_socket_request", fake_socket_request) - - class _Binding: - target_kind = "terminal_id" - target_value = "term_gone" - - with pytest.raises(HerdrErrorResponse) as excinfo: - cs._private_pane_id_for_binding(object(), _Binding(), timeout=5.0) - assert excinfo.value is original_error - - -def test_terminal_id_agent_list_identical_duplicates_converge_and_send_succeeds( - tmp_path: Path, -) -> None: - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [ - _binding( - worker, - target_kind="terminal_id", - value="term-dup", - ) - ], - ) - calls: list[dict[str, Any]] = [] - - class DuplicateListingClient(_FakeSocketClient): - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if method == "agent.get": - raise HerdrErrorResponse( - {"code": "agent_not_found", "message": "target not found"}, - "req-1", - ) - if method == "agent.list": - return { - "agents": [ - {"terminal_id": "term-dup", "pane_id": "w1:p1"}, - {"terminal_id": "term-dup", "pane_id": "w1:p1"}, - ] - } - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": _REALISTIC_VISIBLE_PANE}, - } - if method == "agent.prompt": - if params.get("target") != "w1:p1": - raise HerdrErrorResponse( - {"code": "agent_not_found", "message": "target not found"}, - "req-2", - ) - return { - "type": "agent_prompted", - "agent": {"pane_id": "w1:p1"}, - "delivery": "submitted", - } - return {"accepted": True} - - result = submit_command( - config, - _request(request_id="identical-terminal-rows"), - socket_client_factory=lambda _config: DuplicateListingClient(calls), - ) - - assert result.status == STATUS_ACCEPTED - assert calls == [ - {"method": "agent.get", "params": {"target": "term-dup"}}, - {"method": "agent.list", "params": {}}, - { - "method": "agent.prompt", - "params": { - "target": "w1:p1", - "text": "hello", - "wait": {"until": ["working"], "timeout_ms": 5000}, - }, - }, - ] - - -@pytest.mark.parametrize( - "listing", - [ - None, - {}, - {"agents": {}}, - {"agents": ["not-an-agent"]}, - {"agents": [{"terminal_id": "term-shared"}]}, - {"agents": [{"terminal_id": "", "pane_id": "w1:p1"}]}, - {"agents": [{"terminal_id": "term-shared", "pane_id": ""}]}, - {"agents": [{"terminal_id": "term-shared", "pane_id": 7}]}, - ], -) -def test_terminal_id_agent_list_fallback_rejects_malformed_listing(listing) -> None: - from tendwire import command_submission as cs - - with pytest.raises(ValueError, match="invalid agent.list"): - cs._pane_id_from_terminal_listing(listing, "term-shared") diff --git a/tests/test_config.py b/tests/test_config.py index 8563b58..a005764 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -14,7 +14,6 @@ DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS, DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS, DEFAULT_ACP_THOUGHT_POLICY, - DEFAULT_AGENT_EVENT_SOURCE, DEFAULT_COMMAND_RECEIPT_RETENTION_COUNT, DEFAULT_COMMAND_RECEIPT_RETENTION_SECONDS, DEFAULT_COMMAND_RETRY_HORIZON_SECONDS, @@ -24,8 +23,6 @@ DEFAULT_TURN_MODEL, MAX_COMMAND_RETRY_HORIZON_SECONDS, MIN_COMMAND_RECEIPT_RETENTION_SECONDS, - DEFAULT_TURN_REFRESH_INTERVAL_SECONDS, - DEFAULT_TURN_REFRESH_WORKERS, MAX_MAINTENANCE_CADENCE_SECONDS, MAX_RETENTION_DAYS, MAX_SQLITE_INTEGER, @@ -74,7 +71,7 @@ def test_initial_reconcile_timeout_rejects_invalid_environment( load_config() -def test_acp_event_source_defaults_to_legacy_with_thoughts_disabled( +def test_acp_defaults_are_required_runtime_settings_with_thoughts_disabled( monkeypatch, ) -> None: for name in ( @@ -89,7 +86,7 @@ def test_acp_event_source_defaults_to_legacy_with_thoughts_disabled( config = load_config() - assert config.agent_event_source == DEFAULT_AGENT_EVENT_SOURCE == "legacy" + assert not hasattr(config, "agent_event_source") assert config.acp_thought_policy == DEFAULT_ACP_THOUGHT_POLICY == "disabled" assert config.acp_request_timeout_seconds == DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS == 30.0 assert config.acp_shutdown_timeout_seconds == DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS == 5.0 @@ -107,7 +104,6 @@ def test_acp_configuration_uses_explicit_before_environment(monkeypatch) -> None environment = load_config() explicit = load_config( - agent_event_source="acp_required", acp_thought_policy="disabled", acp_request_timeout_seconds="7.5", acp_shutdown_timeout_seconds="2.5", @@ -115,13 +111,12 @@ def test_acp_configuration_uses_explicit_before_environment(monkeypatch) -> None acp_console_input_policy="preserve", ) - assert environment.agent_event_source == "acp_shadow" + assert not hasattr(environment, "agent_event_source") assert environment.acp_thought_policy == "private_all" assert environment.acp_request_timeout_seconds == 11.0 assert environment.acp_shutdown_timeout_seconds == 3.0 assert environment.acp_max_frame_bytes == 4096 assert environment.acp_console_input_policy == "live_only" - assert explicit.agent_event_source == "acp_required" assert explicit.acp_thought_policy == "disabled" assert explicit.acp_request_timeout_seconds == 7.5 assert explicit.acp_shutdown_timeout_seconds == 2.5 @@ -135,12 +130,6 @@ def test_acp_console_input_policy_rejects_unknown_values(value: str) -> None: Config(acp_console_input_policy=value) -@pytest.mark.parametrize("value", ["", "acp", "preferred", "future"]) -def test_acp_event_source_rejects_unknown_values(value: str) -> None: - with pytest.raises(ValueError, match="agent_event_source must be one of"): - Config(agent_event_source=value) - - @pytest.mark.parametrize("value", ["", "public", "summary", "future"]) def test_acp_thought_policy_rejects_unknown_values(value: str) -> None: with pytest.raises(ValueError, match="acp_thought_policy must be one of"): @@ -164,24 +153,13 @@ def test_acp_bounds_reject_invalid_values(field: str, value: object) -> None: Config(**{field: value}) -def test_turn_model_defaults_to_observed_and_accepts_compatibility_aliases( - monkeypatch, - caplog, -) -> None: +def test_runtime_turn_model_modes_are_removed(monkeypatch) -> None: monkeypatch.delenv("TENDWIRE_TURN_MODEL", raising=False) assert DEFAULT_TURN_MODEL == "observed" - assert load_config().turn_model == "observed" + assert not hasattr(load_config(), "turn_model") monkeypatch.setenv("TENDWIRE_TURN_MODEL", "shadow") - assert load_config().turn_model == "shadow" - assert load_config(turn_model="dual").turn_model == "dual" - assert "behaves as observed" in caplog.text - - -@pytest.mark.parametrize("value", ["", "future", "legacy,dual"]) -def test_turn_model_rejects_unknown_values(value: str) -> None: - with pytest.raises(ValueError, match="turn_model must be one of"): - Config(turn_model=value) + assert not hasattr(load_config(), "turn_model") def test_submission_windows_have_defaults_and_explicit_precedence(monkeypatch) -> None: @@ -536,83 +514,6 @@ def test_command_receipt_retention_must_strictly_exceed_retry_horizon( ) -TURN_REFRESH_ENV_NAMES = ( - "TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS", - "TENDWIRE_TURN_REFRESH_WORKERS", -) - - -def test_turn_refresh_knobs_have_documented_defaults(monkeypatch) -> None: - for name in TURN_REFRESH_ENV_NAMES: - monkeypatch.delenv(name, raising=False) - - config = load_config() - - assert DEFAULT_TURN_REFRESH_INTERVAL_SECONDS == 2.0 - assert DEFAULT_TURN_REFRESH_WORKERS == 4 - assert config.turn_refresh_interval_seconds == 2.0 - assert config.turn_refresh_workers == 4 - - -def test_turn_refresh_knobs_use_explicit_before_environment(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS", "3.5") - monkeypatch.setenv("TENDWIRE_TURN_REFRESH_WORKERS", "8") - - env_config = load_config(max_workers=16) - explicit = load_config( - max_workers=16, - turn_refresh_interval_seconds="0.25", - turn_refresh_workers="6", - ) - - assert env_config.turn_refresh_interval_seconds == 3.5 - assert env_config.turn_refresh_workers == 8 - assert explicit.turn_refresh_interval_seconds == 0.25 - assert explicit.turn_refresh_workers == 6 - - -@pytest.mark.parametrize("value", [0, -0.01, "nan", "inf", "-inf"]) -def test_turn_refresh_interval_rejects_nonpositive_or_nonfinite(value: object) -> None: - with pytest.raises( - ValueError, - match="turn_refresh_interval_seconds must be a finite positive number", - ): - Config(turn_refresh_interval_seconds=value) - - -@pytest.mark.parametrize( - ("value", "message"), - [ - (True, "turn_refresh_workers must be an integer >= 1"), - (0, "turn_refresh_workers must be >= 1"), - (-1, "turn_refresh_workers must be >= 1"), - (33, "turn_refresh_workers must be <= 32"), - (1.5, "turn_refresh_workers must be an integer >= 1"), - ], -) -def test_turn_refresh_workers_reject_invalid_bounds( - value: object, - message: str, -) -> None: - with pytest.raises(ValueError, match=message): - Config(turn_refresh_workers=value) - - -def test_turn_refresh_workers_cannot_exceed_observed_worker_max(monkeypatch) -> None: - with pytest.raises( - ValueError, - match="turn_refresh_workers must be <= max_workers", - ): - Config(max_workers=3, turn_refresh_workers=4) - - monkeypatch.setenv("TENDWIRE_TURN_REFRESH_WORKERS", "5") - with pytest.raises( - ValueError, - match="turn_refresh_workers must be <= max_workers", - ): - load_config(max_workers=4) - - SNAPSHOT_MAINTENANCE_ENV_NAMES = ( "TENDWIRE_SNAPSHOT_RETENTION_DAYS", "TENDWIRE_SNAPSHOT_RETENTION_COUNT", diff --git a/tests/test_daemon.py b/tests/test_daemon.py index 92ae7b0..2f3cb2e 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -23,7 +23,6 @@ from tendwire import __version__ from tendwire.backends.herdr_socket import HerdrSocketTimeoutError -from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult from tendwire.cli import main from tendwire.config import Config from tendwire.core.commands import ( @@ -106,6 +105,49 @@ _PUBLIC_JSON_FORBIDDEN_COMPACT = {key.replace("_", "") for key in _PUBLIC_JSON_FORBIDDEN_KEYS} +@pytest.fixture(autouse=True) +def _required_acp_supervisor_for_daemon_unit_tests( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Keep non-ACP daemon tests focused on their own boundary.""" + + class Supervisor: + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + del timeout + + def join(self, *, timeout: float) -> bool: + del timeout + return True + + def status(self) -> dict[str, Any]: + return {"state": "running", "healthy": True} + + def prompt_route(self, _worker: Worker) -> None: + return None + + original_init = TendwireDaemon.__init__ + + def init_with_required_acp(self: TendwireDaemon, *args: Any, **kwargs: Any) -> None: + original_init(self, *args, **kwargs) + hooks = self.hooks + if ( + hooks.acp_supervisor_factory is not None + and getattr(hooks.acp_supervisor_factory, "__name__", "") + == "_default_acp_supervisor_factory" + ): + object.__setattr__( + hooks, + "acp_supervisor_factory", + lambda _config, _stop: Supervisor(), + ) + self._acp_supervisor = Supervisor() + + monkeypatch.setattr(TendwireDaemon, "__init__", init_with_required_acp) + + def _assert_no_public_json_forbidden(value: Any, path: str = "$") -> None: if isinstance(value, dict): for key, item in value.items(): @@ -899,14 +941,8 @@ def test_daemon_turn_list_is_store_projection_only( updated_at="2026-01-01T00:00:00+00:00", ) save_snapshot(db_path, snapshot) - source_calls = 0 projection_calls: list[dict[str, Any]] = [] - def forbidden_source_refresh(*_args: Any, **_kwargs: Any) -> None: - nonlocal source_calls - source_calls += 1 - raise AssertionError("turn source read reached a cached daemon handler") - def project( path: Path, host_id: str, @@ -921,10 +957,6 @@ def project( "turns": [], } - monkeypatch.setattr( - "tendwire.backends.herdr_turns.refresh_structured_turn_content", - forbidden_source_refresh, - ) monkeypatch.setattr("tendwire.store.sqlite.turns_payload_from_store", project) daemon = TendwireDaemon(config) @@ -937,7 +969,6 @@ def project( ) assert result["status"] == "ok" - assert source_calls == 0 assert len(projection_calls) == 3 assert all( call == { @@ -948,8 +979,7 @@ def project( "limit": 17, "cursor": "twlist1.public", "since": None, - "turn_refresh_interval_seconds": 2.0, - "turn_model": config.turn_model, + "turn_model": "observed", } for call in projection_calls ) @@ -1383,7 +1413,7 @@ def _capture_save( turn_model: str, observation: SnapshotObservationContext, ) -> None: - assert turn_model == config.turn_model + assert turn_model == "observed" captured.append(observation) backend = _Backend() @@ -1425,7 +1455,6 @@ def test_daemon_health_exposes_public_operational_status_without_private_values( command_retry_horizon_seconds=120, command_receipt_retention_seconds=691_200, command_receipt_retention_count=77, - turn_model="shadow", ) snapshot = project_from_raw( config, @@ -1467,35 +1496,13 @@ def test_daemon_health_exposes_public_operational_status_without_private_values( ), ) - class PrivateSchedulerStatus: - def operational_status(self) -> dict[str, Any]: - return { - "status": "healthy", - "queue_depth": 2, - "active": 1, - "refreshed": 7, - "failed": 3, - "timed_out": 2, - "coalesced": 11, - "queue_full": 5, - "last_success": "2026-01-02T00:00:00+00:00", - "last_duration_ms": 12.5, - "stale_age_seconds": 0.25, - "max_workers": 999, - "queue_capacity": 64, - "refresh_interval_seconds": 999, - "adapter_timeout_seconds": 999, - "private_fingerprint": "sentinel-private-fingerprint", - "error": f"sentinel-private failure at {tmp_path}", - } - daemon = TendwireDaemon(config) - daemon._turn_scheduler = PrivateSchedulerStatus() health = daemon.get_health() encoded = json.dumps(health) assert health["status"] == "ok" - assert health["turn_model"] == "shadow" + assert health["acp"]["required"] is True + assert health["acp"]["healthy"] is True assert health["daemon"]["started_at"] assert health["store"]["counts"]["snapshots"] == 1 assert health["store"]["outbox"]["pending"] == 1 @@ -1559,25 +1566,6 @@ def operational_status(self) -> dict[str, Any]: "snapshot_maintenance_batch_size": 6, "store_maintenance_cadence_seconds": 44, } - assert health["turn_ingestion"] == { - "status": "healthy", - "queue": 2, - "active": 1, - "refreshed": 7, - "failed": 3, - "timed_out": 2, - "coalesced": 11, - "queue_full": 5, - "last_success": "2026-01-02T00:00:00+00:00", - "last_duration_ms": 12.5, - "stale_age": 0.25, - "bounds": { - "refresh_interval_seconds": 2.0, - "max_workers": 4, - "queue_capacity": 64, - "adapter_timeout_seconds": 5.0, - }, - } assert health["pending_ingestion"] == { "status": "healthy", "counts": {"fresh": 0, "stale": 0, "total": 0}, @@ -2320,7 +2308,7 @@ def maintenance( now: str | None = None, ) -> dict[str, Any]: assert now is None - assert turn_model == config.turn_model + assert turn_model == "observed" calls.append( ( path, @@ -3666,194 +3654,6 @@ def test_unix_socket_server_close_preserves_substituted_socket(tmp_path: Path) - socket_path.unlink(missing_ok=True) -@_UNIX_SOCKET_TEST -def test_daemon_binds_socket_after_store_observation_before_scheduler_io( - tmp_path: Path, -) -> None: - socket_path = tmp_path / "ordered-cli.sock" - calls: list[str] = [] - - def record_unpublished(stage: str) -> None: - assert not os.path.lexists(socket_path) - calls.append(stage) - - def initialize_store(path: Path) -> None: - record_unpublished("init_store") - init_store(path) - - def observe(_config: Config) -> Snapshot: - record_unpublished("observe") - snapshot = _public_snapshot() - save_snapshot(tmp_path / "ordered-cli.db", snapshot) - return snapshot - - class RecordingScheduler: - parent_fd: int | None = None - - def start(self) -> None: - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - import fcntl - - self.parent_fd = os.open( - tmp_path, - os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC, - ) - fcntl.flock(self.parent_fd, fcntl.LOCK_SH) - calls.append("scheduler_start") - - def request_refresh(self) -> None: - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - calls.append("scheduler_request") - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - if self.parent_fd is not None: - os.close(self.parent_fd) - self.parent_fd = None - calls.append(f"scheduler_stop:{flush_timeout_seconds}") - - def scheduler_factory(_config: Config) -> RecordingScheduler: - record_unpublished("scheduler_factory") - return RecordingScheduler() - - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=tmp_path / "ordered-cli.db", - socket_path=socket_path, - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - init_store=initialize_store, - observe_initial_snapshot=observe, - turn_scheduler_factory=scheduler_factory, - ), - ) - - try: - daemon.start() - assert calls == [ - "init_store", - "observe", - "scheduler_factory", - "scheduler_start", - "scheduler_request", - ] - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - _assert_unix_socket_connects(socket_path) - finally: - daemon.stop() - - assert calls[-1] == "scheduler_stop:6.0" - assert not os.path.lexists(socket_path) - - -@_UNIX_SOCKET_TEST -def test_daemon_event_callback_is_attached_after_reconcile_before_ingestion( - tmp_path: Path, -) -> None: - socket_path = tmp_path / "ordered-events.sock" - db_path = tmp_path / "ordered-events.db" - calls: list[str] = [] - - def record_unpublished(stage: str) -> None: - assert not os.path.lexists(socket_path) - calls.append(stage) - - def initialize_store(path: Path) -> None: - record_unpublished("init_store") - init_store(path) - - class RecordingEventBackend: - def __init__(self) -> None: - self.callback: Any | None = None - self.stopped = False - - def start(self, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - record_unpublished("backend_start") - save_snapshot(db_path, _public_snapshot()) - - def set_turn_refresh_callback(self, callback: Any | None) -> None: - if callback is not None: - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - self.callback = callback - calls.append("callback_attached" if callback is not None else "callback_detached") - - def flush(self) -> None: - calls.append("backend_flush") - if self.callback is not None: - self.callback() - - def stop(self) -> None: - calls.append("backend_stop") - self.stopped = True - - class RecordingScheduler: - def start(self) -> None: - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - calls.append("scheduler_start") - - def request_refresh(self) -> None: - calls.append("scheduler_request") - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - calls.append(f"scheduler_stop:{flush_timeout_seconds}") - - backend = RecordingEventBackend() - - def event_backend_factory(_config: Config, _stop_event: threading.Event) -> Any: - record_unpublished("backend_factory") - return backend - - def scheduler_factory(_config: Config) -> RecordingScheduler: - record_unpublished("scheduler_factory") - return RecordingScheduler() - - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - init_store=initialize_store, - event_backend_factory=event_backend_factory, - turn_scheduler_factory=scheduler_factory, - ), - ) - - daemon.start() - assert calls == [ - "init_store", - "backend_factory", - "backend_start", - "scheduler_factory", - "callback_attached", - "scheduler_start", - "scheduler_request", - ] - assert backend.callback is not None - backend.callback() - assert calls[-1] == "scheduler_request" - daemon.stop() - daemon.stop() - - assert calls[-5:] == [ - "backend_flush", - "scheduler_request", - "callback_detached", - "scheduler_stop:6.0", - "backend_stop", - ] - assert backend.callback is None - assert backend.stopped is True - assert not os.path.lexists(socket_path) - - @_UNIX_SOCKET_TEST @pytest.mark.parametrize("failure_stage", ["init_store", "observe"]) def test_daemon_startup_failure_never_publishes_socket( @@ -3988,13 +3788,12 @@ def forbidden_acp_factory(_config: Config, _stop_event: threading.Event) -> Any: db_path=tmp_path / "backend-timeout.db", socket_path=tmp_path / "backend-timeout.sock", herdr_backend="socket", - agent_event_source="acp_preferred", ) daemon = TendwireDaemon( config, hooks=DaemonHooks( event_backend_factory=lambda _config, _stop_event: TimedOutEventBackend(), - acp_runtime_factory=forbidden_acp_factory, + acp_supervisor_factory=forbidden_acp_factory, ), ) @@ -4053,72 +3852,6 @@ def time_out_start(self: Any, *, wait_for_reconcile: bool) -> None: @_UNIX_SOCKET_TEST -def test_daemon_scheduler_start_failure_detaches_callback_and_cleans_components( - tmp_path: Path, -) -> None: - socket_path = tmp_path / "scheduler-failure.sock" - db_path = tmp_path / "scheduler-failure.db" - calls: list[str] = [] - - class Backend: - callback: Any | None = None - - def start(self, *, wait_for_reconcile: bool) -> None: - calls.append("backend_start") - save_snapshot(db_path, _public_snapshot()) - - def set_turn_refresh_callback(self, callback: Any | None) -> None: - self.callback = callback - calls.append("callback_set" if callback is not None else "callback_clear") - - def stop(self) -> None: - calls.append("backend_stop") - - class FailingScheduler: - def request_refresh(self) -> None: - calls.append("scheduler_request") - - def start(self) -> None: - calls.append("scheduler_start") - raise RuntimeError("sentinel scheduler startup failure") - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - calls.append(f"scheduler_stop:{flush_timeout_seconds}") - - backend = Backend() - scheduler = FailingScheduler() - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - event_backend_factory=lambda _config, _stop_event: backend, - turn_scheduler_factory=lambda _config: scheduler, - ), - ) - - with pytest.raises(RuntimeError, match="sentinel scheduler startup failure"): - daemon.start() - - assert calls == [ - "backend_start", - "callback_set", - "scheduler_start", - "callback_clear", - "scheduler_stop:6.0", - "backend_stop", - ] - assert backend.callback is None - assert daemon.server is None - assert daemon._turn_scheduler is None - assert not os.path.lexists(socket_path) - - def test_daemon_starts_observes_persists_serves_and_removes_socket(tmp_path: Path) -> None: db_path = tmp_path / "daemon.db" socket_path = tmp_path / "daemon.sock" @@ -4172,271 +3905,6 @@ def observe(config: Config) -> Snapshot: @_UNIX_SOCKET_TEST -def test_blocked_turn_ingestion_does_not_delay_cached_real_socket_handlers( - tmp_path: Path, -) -> None: - db_path = tmp_path / "blocked-ingestion.db" - socket_path = tmp_path / "blocked-ingestion.sock" - entered = threading.Event() - release = threading.Event() - source_calls: list[str] = [] - worker = Worker(id="worker-1", name="Worker One", status="active") - binding = WorkerBinding( - host_id="daemon-host", - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="sentinel-private-agent", - turn_target_kind="pane_id", - turn_target_value="sentinel-private-pane", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="sentinel-private-binding", - ) - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_timeout_seconds=1, - turn_refresh_interval_seconds=3600, - turn_refresh_workers=1, - ) - - def observe(_config: Config) -> Snapshot: - snapshot = Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:00:00+00:00", - workers=[worker], - backend_health=[ - BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - ) - ], - ) - save_snapshot(db_path, snapshot) - upsert_worker_bindings(db_path, [binding]) - return snapshot - - def blocked_reader( - _config: Config, - current: WorkerBinding, - *, - adapter_timeout_seconds: float, - ) -> TurnRefreshResult: - source_calls.append(current.private_fingerprint) - entered.set() - assert release.wait(timeout=10) - return TurnRefreshResult("unchanged", 0) - - scheduler: TurnIngestionScheduler | None = None - - def scheduler_factory(current: Config) -> TurnIngestionScheduler: - nonlocal scheduler - scheduler = TurnIngestionScheduler( - current, - refresh_interval_seconds=3600, - max_workers=1, - reader=blocked_reader, - ) - return scheduler - - command_calls: list[str] = [] - - def submit_command(_config: Config, payload: str) -> CommandEnvelope: - command_calls.append(payload) - return CommandEnvelope( - ok=True, - status="accepted", - action="noop", - result={"accepted": True}, - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - observe_initial_snapshot=observe, - submit_command=submit_command, - turn_scheduler_factory=scheduler_factory, - ), - ) - server_thread: threading.Thread | None = None - try: - daemon.start() - server_thread = threading.Thread(target=daemon.serve_forever) - server_thread.start() - assert entered.wait(timeout=10) - client = DaemonAPIClient(socket_path, timeout_seconds=1) - - for _ in range(3): - listed = client.request( - "turn.list", - {"schema_version": 2, "limit": 10, "cursor": None, "since": None}, - ) - health = client.request("health.get") - snapshot = client.request("snapshot.get") - pending = client.request("pending.list") - assert listed["ok"] is True - assert health["result"]["turn_ingestion"]["active"] == 1 - assert snapshot["result"]["host_id"] == config.host_id - assert pending["ok"] is True - - command = client.request( - "command.submit", - {"schema_version": 1, "action": "noop", "dry_run": True}, - ) - assert command["ok"] is True - assert command["result"]["status"] == "accepted" - assert source_calls == ["sentinel-private-binding"] - assert len(command_calls) == 1 - finally: - release.set() - daemon.stop() - if server_thread is not None: - server_thread.join(timeout=2) - - assert scheduler is not None - assert server_thread is not None and not server_thread.is_alive() - assert not os.path.lexists(socket_path) - - -@_UNIX_SOCKET_TEST -def test_daemon_restart_scans_durable_bindings_without_touching_final_or_outbox( - tmp_path: Path, -) -> None: - db_path = tmp_path / "restart.db" - socket_path = tmp_path / "restart.sock" - config = Config( - host_id="restart-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - ) - init_store(db_path) - worker = Worker(id="worker-1", name="Worker One", status="idle") - snapshot = Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:00:00+00:00", - workers=[worker], - ) - save_snapshot(db_path, snapshot) - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="sentinel-private-agent", - turn_target_kind="pane_id", - turn_target_value="sentinel-private-pane", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="sentinel-private-binding", - ) - ], - ) - assert merge_turn_content( - db_path, - config.host_id, - worker.id, - { - "source_turn_id": "source-turn-1", - "assistant_final_text": "durable final", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:01:00+00:00", - ) == 1 - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - config.host_id, - "attention", - "durable-delivery", - "queued", - '{"safe":"kept"}', - '{"opaque":"kept"}', - "2026-01-01T00:02:00+00:00", - "2026-01-01T00:02:00+00:00", - ), - ) - before_turns = conn.execute( - "SELECT * FROM turns WHERE host_id = ? ORDER BY turn_id", - (config.host_id,), - ).fetchall() - before_outbox = conn.execute( - "SELECT * FROM connector_outbox WHERE host_id = ? ORDER BY id", - (config.host_id,), - ).fetchall() - - scheduler_calls: list[tuple[str, int]] = [] - - class DurableScanScheduler: - def start(self) -> None: - scheduler_calls.append(("start", 0)) - - def request_refresh(self) -> None: - with sqlite3.connect(str(db_path)) as conn: - count = conn.execute( - "SELECT COUNT(*) FROM worker_bindings WHERE host_id = ?", - (config.host_id,), - ).fetchone()[0] - scheduler_calls.append(("request", int(count))) - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - scheduler_calls.append(("stop", int(flush_timeout_seconds or 0))) - - for _ in range(2): - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - observe_initial_snapshot=lambda _config: latest_snapshot( - db_path, - config.host_id, - ), - turn_scheduler_factory=lambda _config: DurableScanScheduler(), - ), - ) - daemon.start() - daemon.stop() - - with sqlite3.connect(str(db_path)) as conn: - after_turns = conn.execute( - "SELECT * FROM turns WHERE host_id = ? ORDER BY turn_id", - (config.host_id,), - ).fetchall() - after_outbox = conn.execute( - "SELECT * FROM connector_outbox WHERE host_id = ? ORDER BY id", - (config.host_id,), - ).fetchall() - - assert scheduler_calls == [ - ("start", 0), - ("request", 1), - ("stop", 6), - ("start", 0), - ("request", 1), - ("stop", 6), - ] - assert after_turns == before_turns - assert after_outbox == before_outbox - assert not os.path.lexists(socket_path) - - def test_daemon_server_survives_client_disconnect_during_response(tmp_path: Path) -> None: socket_path = tmp_path / "daemon.sock" request_seen = threading.Event() @@ -4859,226 +4327,6 @@ def request_into(target: list[BaseException | dict[str, Any]]) -> None: assert remaining == set() -def test_daemon_concurrent_same_request_id_sends_once_and_replays_accepted( - tmp_path: Path, - monkeypatch, - capsys, -) -> None: - db_path = tmp_path / "commands.db" - socket_path = tmp_path / "commands.sock" - config = Config( - host_id="cmd-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - init_store(db_path) - calls: list[dict[str, Any]] = [] - worker = Worker(id="w-1", name="Alpha", status="active") - binding = WorkerBinding( - host_id="cmd-host", - worker_id="w-1", - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="private-binding", - ) - - class FakeHealth: - def to_backend_health(self) -> BackendHealth: - return BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - counts={"workers": 1}, - ) - - class FakeEventBackend: - health = FakeHealth() - - def __init__(self, config: Config, stop_event: threading.Event) -> None: - self.config = config - - def start(self, *, wait_for_reconcile: bool = True) -> None: - snapshot = Snapshot( - host_id="cmd-host", - updated_at="2026-01-01T00:00:00+00:00", - workers=[worker], - backend_health=[self.health.to_backend_health()], - ) - save_snapshot(db_path, snapshot) - upsert_worker_bindings(db_path, [binding]) - - def stop(self) -> None: - return None - - class FakeHerdrSocketClient: - def connect(self) -> "FakeHerdrSocketClient": - return self - - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None) -> dict[str, Any]: - calls.append({"method": method, "params": dict(params)}) - if method == "agent.get": - return {"result": {"agent": {"pane_id": "pane-private"}}} - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": "Completed previous turn.\n── status: idle ──"}, - } - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": "pane-private"}, - "delivery": "submitted", - } - return {"accepted": True} - - def close(self) -> None: - return None - - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - lambda config: FakeHerdrSocketClient(), - ) - from tendwire import command_submission - - real_reserve = command_submission.reserve_command_request - reservation_barrier = threading.Barrier(2, timeout=5) - - def synchronized_reserve(*args: Any, **kwargs: Any) -> dict[str, Any]: - result = real_reserve(*args, **kwargs) - reservation_barrier.wait() - return result - - monkeypatch.setattr( - command_submission, - "reserve_command_request", - synchronized_reserve, - ) - - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(event_backend_factory=lambda config, stop_event: FakeEventBackend(config, stop_event)), - ) - daemon.start() - thread = threading.Thread(target=daemon.serve_forever) - thread.start() - try: - request = { - "schema_version": 1, - "action": "send_instruction", - "request_id": "req-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - start_barrier = threading.Barrier(3, timeout=5) - results: list[dict[str, Any] | None] = [None, None] - errors: list[BaseException] = [] - - def submit(index: int) -> None: - try: - start_barrier.wait() - results[index] = DaemonAPIClient( - socket_path, - timeout_seconds=5, - ).request("command.submit", request) - except BaseException as exc: # noqa: BLE001 - errors.append(exc) - - clients = [ - threading.Thread(target=submit, args=(index,)) - for index in range(2) - ] - for client_thread in clients: - client_thread.start() - start_barrier.wait() - for client_thread in clients: - client_thread.join(timeout=5) - - assert errors == [] - assert all(not client_thread.is_alive() for client_thread in clients) - responses = [result for result in results if result is not None] - assert len(responses) == 2 - assert all(response["ok"] is True for response in responses) - assert sorted(response["result"]["status"] for response in responses) == [ - STATUS_ACCEPTED, - STATUS_PENDING, - ] - assert sorted( - response["result"]["disposition"] for response in responses - ) == [ - DISPOSITION_IN_PROGRESS, - DISPOSITION_TERMINAL_ACCEPTED, - ] - assert all(response["schema_version"] == 1 for response in responses) - assert all(response["result"]["schema_version"] == 2 for response in responses) - assert calls == [ - {"method": "agent.get", "params": {"target": "agent-private"}}, - {"method": "agent.get", "params": {"target": "agent-private"}}, - { - "method": "agent.prompt", - "params": { - "target": "pane-private", - "text": "hello", - "wait": {"until": ["working"], "timeout_ms": 5000}, - }, - }, - ] - assert not any(call["method"] == "pane.send_keys" for call in calls) - receipt = get_command_request(db_path, "cmd-host", "req-1") - assert receipt is not None - assert receipt["state"] == "accepted" - assert receipt["status"] == STATUS_ACCEPTED - assert receipt["terminal_at"] is not None - - monkeypatch.setattr( - command_submission, - "reserve_command_request", - real_reserve, - ) - replay = DaemonAPIClient(socket_path).request("command.submit", request) - assert replay["ok"] is True - assert replay["result"]["status"] == STATUS_ACCEPTED - assert replay["result"]["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "cli-state")) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request))) - cli_code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(socket_path), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - cli_result = json.loads(captured.out) - assert cli_code == 0 - assert captured.err == "" - assert cli_result == replay["result"] - assert cli_result["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - _assert_no_public_json_forbidden(cli_result) - assert len([call for call in calls if call["method"] == "agent.prompt"]) == 1 - for response in [*responses, replay]: - encoded = json.dumps(response) - assert "agent-private" not in encoded - assert "pane-private" not in encoded - _assert_no_public_json_forbidden(response) - finally: - daemon.stop() - thread.join(timeout=2) - - def test_daemon_command_submit_rejects_blank_request_id_before_mutation( tmp_path: Path, monkeypatch, @@ -5093,14 +4341,6 @@ def test_daemon_command_submit_rejects_blank_request_id_before_mutation( init_store(db_path) calls: list[str] = [] - def guarded_socket_factory(config: Config) -> Any: - calls.append("socket") - raise AssertionError("invalid request_id must not construct Herdr socket client") - - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - guarded_socket_factory, - ) daemon = TendwireDaemon(config) request = { "schema_version": 1, @@ -5716,7 +4956,6 @@ def test_isolated_daemon_survives_deterministic_real_wal_retirement_without_reso data_dir=tmp_path, db_path=db_path, socket_path=socket_path, - turn_refresh_interval_seconds=3600, acknowledged_final_retention_days=36500, ) worker = Worker(id="worker-race", name="Worker Race", status="active") @@ -5845,27 +5084,6 @@ def churn_wal() -> None: except threading.BrokenBarrierError: pass - scheduler_calls: list[str] = [] - - class NoopScheduler: - def start(self) -> None: - scheduler_calls.append("start") - - def request_refresh(self) -> None: - scheduler_calls.append("request") - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - del flush_timeout_seconds - scheduler_calls.append("stop") - - def operational_status(self) -> dict[str, Any]: - return { - "status": "healthy", - "queue_depth": 0, - "active": 0, - "queue_capacity": 1, - } - def direct_child_processes() -> set[int]: children: set[int] = set() for task in (Path("/proc/self/task")).iterdir(): @@ -5907,7 +5125,6 @@ def fd_targets() -> dict[str, tuple[str, int, int, int]]: db_path, config.host_id, ), - turn_scheduler_factory=lambda _config: NoopScheduler(), ), ) server_thread: threading.Thread | None = None @@ -5987,7 +5204,6 @@ def fd_targets() -> dict[str, tuple[str, int, int, int]]: assert phase_calls == [ ("captured", LocalStateKind.DATABASE_WAL) ] * cycle_count - assert scheduler_calls == ["start", "request", "stop"] assert len(responses) == cycle_count * 3 assert (db_path.stat().st_dev, db_path.stat().st_ino) == main_identity assert not os.path.lexists(socket_path) diff --git a/tests/test_daemon_acp.py b/tests/test_daemon_acp.py index a2262ac..f672006 100644 --- a/tests/test_daemon_acp.py +++ b/tests/test_daemon_acp.py @@ -1,10 +1,8 @@ -"""ACP lifecycle policy tests for the Tendwire daemon.""" +"""Required ACP lifecycle contract tests for the Tendwire daemon.""" from __future__ import annotations import json -import os -import stat import threading from pathlib import Path from typing import Any @@ -20,36 +18,18 @@ def _snapshot() -> Snapshot: return Snapshot( host_id="daemon-host", - updated_at="2026-01-01T00:00:00+00:00", + updated_at="2026-08-04T00:00:00+00:00", backend_health=[ BackendHealth( name="herdr", status="healthy", outcome="empty_healthy", - observed_at="2026-01-01T00:00:00+00:00", ) ], ) -class _Scheduler: - def __init__(self, calls: list[str]) -> None: - self.calls = calls - - def start(self) -> None: - self.calls.append("scheduler_start") - - def request_refresh(self) -> None: - self.calls.append("scheduler_request") - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - self.calls.append(f"scheduler_stop:{flush_timeout_seconds}") - - def operational_status(self) -> dict[str, Any]: - return {"status": "healthy"} - - -class _Runtime: +class _Supervisor: def __init__( self, calls: list[str], @@ -72,18 +52,10 @@ def status(self) -> dict[str, Any]: "healthy": self.healthy, "updates_ingested": 7, "permissions_ingested": 2, - "permissions_selected": 1, - "permissions_cancelled": 1, - "invalid_permission_selections": 0, - "prompts_started": 3, - "prompts_completed": 2, - "prompts_failed": 1, - "cancellation_requests": 1, + "prompts_completed": 3, "failure_type": None if self.healthy else "AcpTransportError", - # Deliberately private transport material must never be projected. "argv": ["sentinel-private-command"], "session_id": "sentinel-private-session", - "binding_id": "sentinel-private-binding", } def stop(self, *, timeout: float) -> None: @@ -94,14 +66,24 @@ def join(self, *, timeout: float) -> bool: return True +def _config(tmp_path: Path) -> Config: + tmp_path.chmod(0o700) + return Config( + host_id="daemon-host", + data_dir=tmp_path, + db_path=tmp_path / "daemon.db", + socket_path=tmp_path / "daemon.sock", + herdr_backend="cli", + acp_shutdown_timeout_seconds=1.25, + ) + + def _hooks( - tmp_path: Path, + config: Config, + supervisor_factory: Any, calls: list[str], - *, - acp_runtime_factory: Any = None, - scheduler_factory: Any = None, ) -> DaemonHooks: - db_path = tmp_path / "daemon.db" + assert config.db_path is not None def initialize(path: Path) -> None: calls.append("init_store") @@ -110,285 +92,102 @@ def initialize(path: Path) -> None: def observe(_config: Config) -> Snapshot: calls.append("observe") snapshot = _snapshot() - save_snapshot(db_path, snapshot) + save_snapshot(config.db_path, snapshot) return snapshot - def make_scheduler(_config: Config) -> _Scheduler: - calls.append("scheduler_factory") - return _Scheduler(calls) - return DaemonHooks( init_store=initialize, observe_initial_snapshot=observe, - turn_scheduler_factory=scheduler_factory or make_scheduler, - acp_runtime_factory=acp_runtime_factory, - ) - - -def _config(tmp_path: Path, policy: str) -> Config: - return Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=tmp_path / "daemon.db", - socket_path=tmp_path / "daemon.sock", - agent_event_source=policy, - acp_shutdown_timeout_seconds=1.25, - ) - - -def test_legacy_policy_never_calls_acp_factory(tmp_path: Path) -> None: - calls: list[str] = [] - - def forbidden_factory(_config: Config, _stop_event: threading.Event) -> Any: - raise AssertionError("legacy mode must never discover or start ACP") - - daemon = TendwireDaemon( - _config(tmp_path, "legacy"), - hooks=_hooks(tmp_path, calls, acp_runtime_factory=forbidden_factory), + acp_supervisor_factory=supervisor_factory, ) - daemon.start() - try: - assert daemon.get_health()["acp"] == { - "policy": "legacy", - "status": "disabled", - "healthy": False, - "state": "disabled", - "failure_type": None, - "counters": { - "updates_ingested": 0, - "permissions_ingested": 0, - "permissions_selected": 0, - "permissions_cancelled": 0, - "invalid_permission_selections": 0, - "prompts_started": 0, - "prompts_completed": 0, - "prompts_failed": 0, - "cancellation_requests": 0, - }, - } - finally: - daemon.stop() - - assert calls[-1] == "scheduler_stop:6.0" -@pytest.mark.parametrize("policy", ["acp_shadow", "acp_preferred"]) -def test_optional_acp_policy_tolerates_unavailable_runtime( - tmp_path: Path, - policy: str, -) -> None: +def test_daemon_requires_an_acp_supervisor_before_binding_socket(tmp_path: Path) -> None: + config = _config(tmp_path) calls: list[str] = [] + daemon = TendwireDaemon(config, hooks=_hooks(config, None, calls)) - def unavailable(_config: Config, _stop_event: threading.Event) -> None: - calls.append("acp_factory") - return None - - daemon = TendwireDaemon( - _config(tmp_path, policy), - hooks=_hooks(tmp_path, calls, acp_runtime_factory=unavailable), - ) - daemon.start() - try: - assert calls[-2:] == ["scheduler_start", "scheduler_request"] - assert daemon.get_health()["acp"]["status"] == "unavailable" - finally: - daemon.stop() - - -def test_required_acp_without_factory_fails_before_socket_or_scheduler( - tmp_path: Path, -) -> None: - calls: list[str] = [] - socket_path = tmp_path / "daemon.sock" - daemon = TendwireDaemon( - _config(tmp_path, "acp_required"), - hooks=_hooks(tmp_path, calls), - ) - - with pytest.raises(RuntimeError, match="ACP runtime is required"): + with pytest.raises(RuntimeError, match="ACP supervisor is required"): daemon.start() + assert not config.socket_path.exists() assert calls == ["init_store", "observe"] - assert not os.path.lexists(socket_path) - assert daemon.server is None -def test_required_acp_starts_before_socket_and_exposes_only_redacted_health( +def test_daemon_starts_required_acp_and_exposes_only_public_health( tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, ) -> None: + monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", lambda _self: None) + config = _config(tmp_path) calls: list[str] = [] - socket_path = tmp_path / "daemon.sock" - runtime = _Runtime(calls) - - def runtime_factory(config: Config, stop_event: threading.Event) -> _Runtime: - assert config.agent_event_source == "acp_required" - assert stop_event.is_set() is False - assert not os.path.lexists(socket_path) - calls.append("acp_factory") - return runtime - + supervisor = _Supervisor(calls) daemon = TendwireDaemon( - _config(tmp_path, "acp_required"), - hooks=_hooks(tmp_path, calls, acp_runtime_factory=runtime_factory), + config, + hooks=_hooks(config, lambda _config, _stop: supervisor, calls), ) + daemon.start() try: - assert calls == [ - "init_store", - "observe", - "acp_factory", - "acp_start", - ] - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) health = daemon.get_health() assert health["status"] == "ok" - acp = health["acp"] - assert acp == { - "policy": "acp_required", - "status": "healthy", - "healthy": True, - "state": "running", - "failure_type": None, - "counters": { - "updates_ingested": 7, - "permissions_ingested": 2, - "permissions_selected": 1, - "permissions_cancelled": 1, - "invalid_permission_selections": 0, - "prompts_started": 3, - "prompts_completed": 2, - "prompts_failed": 1, - "cancellation_requests": 1, - }, - } - encoded = json.dumps(acp) - assert "sentinel-private" not in encoded - assert "argv" not in encoded - assert "session" not in encoded - assert "binding" not in encoded - runtime.healthy = False - degraded = daemon.get_health() - assert degraded["status"] == "degraded" - assert degraded["acp"]["status"] == "degraded" + assert health["acp"]["required"] is True + assert health["acp"]["healthy"] is True + assert health["acp"]["state"] == "running" + assert health["acp"]["counters"]["updates_ingested"] == 7 + assert "sentinel-private" not in json.dumps(health) + assert calls[:3] == ["init_store", "observe", "acp_start"] finally: daemon.stop() - assert calls[-2:] == [ - "acp_stop:1.25", - "acp_join:1.25", - ] + assert calls[-2:] == ["acp_stop:1.25", "acp_join:1.25"] -@pytest.mark.parametrize("policy", ["acp_shadow", "acp_preferred"]) -def test_optional_unhealthy_acp_is_stopped_and_legacy_scheduler_continues( +@pytest.mark.parametrize( + "supervisor", + [ + pytest.param(_Supervisor([], healthy=False), id="unhealthy"), + pytest.param( + _Supervisor([], start_failure=OSError("private transport detail")), + id="start-failure", + ), + ], +) +def test_daemon_fails_closed_when_required_acp_cannot_start( tmp_path: Path, - policy: str, + supervisor: _Supervisor, ) -> None: - calls: list[str] = [] - runtime = _Runtime(calls, healthy=False) + config = _config(tmp_path) + calls = supervisor.calls daemon = TendwireDaemon( - _config(tmp_path, policy), - hooks=_hooks( - tmp_path, - calls, - acp_runtime_factory=lambda _config, _stop_event: runtime, - ), + config, + hooks=_hooks(config, lambda _config, _stop: supervisor, calls), ) - daemon.start() - try: - assert calls == [ - "init_store", - "observe", - "acp_start", - "acp_stop:1.25", - "acp_join:1.25", - "scheduler_factory", - "scheduler_start", - "scheduler_request", - ] - acp = daemon.get_health()["acp"] - assert acp["status"] == "unavailable" - assert acp["failure_type"] == "AcpTransportError" - finally: - daemon.stop() - - -def test_required_unhealthy_acp_stops_and_fails_closed(tmp_path: Path) -> None: - calls: list[str] = [] - runtime = _Runtime(calls, healthy=False) - daemon = TendwireDaemon( - _config(tmp_path, "acp_required"), - hooks=_hooks( - tmp_path, - calls, - acp_runtime_factory=lambda _config, _stop_event: runtime, - ), - ) - - with pytest.raises(RuntimeError, match=r"failed to start \(AcpTransportError\)"): + with pytest.raises(RuntimeError, match="ACP supervisor is required"): daemon.start() - assert calls == [ - "init_store", - "observe", - "acp_start", - "acp_stop:1.25", - "acp_join:1.25", - ] - assert not os.path.lexists(tmp_path / "daemon.sock") + assert not config.socket_path.exists() + assert any(call.startswith("acp_stop:") for call in calls) + assert "private transport detail" not in repr(daemon._acp_startup_failure_type) -def test_optional_acp_start_failure_is_cleaned_up_before_legacy_fallback( +def test_supervisor_receives_daemon_stop_event( tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, ) -> None: + monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", lambda _self: None) + config = _config(tmp_path) calls: list[str] = [] - runtime = _Runtime( - calls, - start_failure=RuntimeError("sentinel-private-command --session secret"), - ) - daemon = TendwireDaemon( - _config(tmp_path, "acp_preferred"), - hooks=_hooks( - tmp_path, - calls, - acp_runtime_factory=lambda _config, _stop_event: runtime, - ), - ) + seen: list[threading.Event] = [] - daemon.start() - try: - assert calls[2:5] == ["acp_start", "acp_stop:1.25", "acp_join:1.25"] - acp = daemon.get_health()["acp"] - assert acp["status"] == "unavailable" - assert acp["failure_type"] == "RuntimeError" - assert "sentinel-private" not in json.dumps(acp) - assert calls[-2:] == ["scheduler_start", "scheduler_request"] - finally: - daemon.stop() - - -def test_required_acp_never_constructs_legacy_scheduler(tmp_path: Path) -> None: - calls: list[str] = [] - runtime = _Runtime(calls) - - def forbidden_scheduler(_config: Config) -> _Scheduler: - raise AssertionError("acp_required must never construct legacy ingestion") - - daemon = TendwireDaemon( - _config(tmp_path, "acp_required"), - hooks=_hooks( - tmp_path, - calls, - acp_runtime_factory=lambda _config, _stop_event: runtime, - scheduler_factory=forbidden_scheduler, - ), - ) + def factory(_config: Config, stop_event: threading.Event) -> _Supervisor: + seen.append(stop_event) + return _Supervisor(calls) + daemon = TendwireDaemon(config, hooks=_hooks(config, factory, calls)) daemon.start() try: - assert calls == ["init_store", "observe", "acp_start"] + assert seen == [daemon.stop_event] finally: daemon.stop() - - assert calls[-2:] == ["acp_stop:1.25", "acp_join:1.25"] diff --git a/tests/test_herdr_events.py b/tests/test_herdr_events.py index 95a8fe2..945eb72 100644 --- a/tests/test_herdr_events.py +++ b/tests/test_herdr_events.py @@ -20,7 +20,7 @@ import pytest -from tendwire.backends import herdr_cli, herdr_events, herdr_turns +from tendwire.backends import herdr_cli, herdr_events from tendwire.backends.herdr_events import ( DEFAULT_SUBSCRIBE_METHOD, HerdrEventBackend, @@ -52,9 +52,6 @@ SnapshotObservationContext, SnapshotRetentionPolicy, apply_backend_pending_observation, - get_herdr_turn_refresh_retry, - get_herdr_turn_watermark, - herdr_turn_refresh_retry_due, init_store, latest_snapshot, list_attention_items, @@ -63,9 +60,7 @@ maybe_run_automatic_store_maintenance, merge_turn_content, pending_payload_from_store, - record_herdr_turn_refresh_retry, save_snapshot, - set_herdr_turn_watermark, turns_payload_from_store, ) @@ -98,22 +93,6 @@ _PUBLIC_JSON_FORBIDDEN_COMPACT = {key.replace("_", "") for key in _PUBLIC_JSON_FORBIDDEN_KEYS} -def _force_herdr_turn_refresh_retry_due( - backend: HerdrEventBackend, - pane_id: str, - turn: int, -) -> None: - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - conn.execute( - """ - UPDATE herdr_turn_refresh_retries - SET next_attempt_at = '1970-01-01T00:00:00+00:00' - WHERE host_id = ? AND pane_id = ? AND turn = ? - """, - (backend.config.host_id, pane_id, turn), - ) - - def _assert_no_public_json_forbidden(value: Any, path: str = "$") -> None: if isinstance(value, dict): for key, item in value.items(): @@ -942,635 +921,14 @@ def connect(self) -> None: assert client.connected == 0 -def test_herdr_075_observation_paths_preserve_474_identity_inputs_byte_for_byte( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Feed one pane through every event path without changing its identity bytes.""" - fixed_installation_key = b"tendwire-identity-regression-key" - assert len(fixed_installation_key) == 32 - derivation_inputs: list[bytes] = [] - real_stable_worker_key = herdr_cli.stable_worker_key - - def capture_stable_worker_key( - installation_key: bytes, - *, - backend: str, - host_id: str, - workspace_id: str, - pane_id: str, - ) -> str: - derivation_inputs.append( - installation_key - + b"\0" - + json.dumps( - { - "backend": backend, - "host_id": host_id, - "pane_id": pane_id, - "workspace_id": workspace_id, - }, - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") - ) - return real_stable_worker_key( - installation_key, - backend=backend, - host_id=host_id, - workspace_id=workspace_id, - pane_id=pane_id, - ) - - monkeypatch.setattr( - herdr_cli, - "load_or_create_installation_key", - lambda _data_dir: fixed_installation_key, - ) - monkeypatch.setattr(herdr_cli, "stable_worker_key", capture_stable_worker_key) - - pane = { - "pane_id": "w123456789abcde:pA", - "terminal_id": "terminal-identity", - "agent": "claude", - "workspace_id": "w123456789abcde", - "agent_status": "working", - "label": "identity-pane", - } - workspaces = [{"id": "w123456789abcde", "name": "Build"}] - - def reconciled_backend(name: str) -> HerdrEventBackend: - data_dir = tmp_path / name - config = Config( - host_id="identity-host", - data_dir=data_dir, - db_path=data_dir / "tendwire.db", - herdr_backend="socket", - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - backend.reconcile_once( - client=_StaticClient( - workspaces=workspaces, - panes=[dict(pane)], - agents=[], - ) - ) - return backend - - def stable_key_bytes(backend: HerdrEventBackend) -> bytes: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - return snapshot.workers[0].meta["stable_key"].encode("ascii") - - def worker_meta_bytes(backend: HerdrEventBackend) -> bytes: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - return json.dumps( - snapshot.workers[0].meta, - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") - - reference_backend = reconciled_backend("474-reference-reconcile") - assert len(derivation_inputs) == 1 - reference_input = derivation_inputs[0] - reference_key = stable_key_bytes(reference_backend) - reference_meta = worker_meta_bytes(reference_backend) - - legacy_backend = reconciled_backend("herdr-074-status-path") - legacy_path_start = len(derivation_inputs) - 1 - assert legacy_backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": {**pane, "agent_status": "blocked"}, - } - ) - # Scalar status payloads are not PaneInfo and must not re-derive identity. - assert derivation_inputs[legacy_path_start:] == [reference_input] - assert stable_key_bytes(legacy_backend) == reference_key - assert worker_meta_bytes(legacy_backend) == reference_meta - - new_backend = reconciled_backend("herdr-075-pane-updated-path") - new_path_start = len(derivation_inputs) - 1 - before_new_meta = worker_meta_bytes(new_backend) - refreshes: list[None] = [] - new_backend.set_turn_refresh_callback(lambda: refreshes.append(None)) - - # Herdr 0.7.5's pane.updated is the scalar PaneOutputChanged event. It may - # trigger a turn refresh but must not rebuild worker identity. - assert new_backend.queue_event_envelope( - { - "event": "pane.updated", - "data": { - "type": "pane_updated", - "workspace_id": pane["workspace_id"], - "pane_id": pane["pane_id"], - "revision": 2, - }, - } - ) - assert refreshes == [None] - assert derivation_inputs[new_path_start:] == [reference_input] - assert stable_key_bytes(new_backend) == reference_key - assert before_new_meta == worker_meta_bytes(new_backend) == reference_meta - - class Herdr074FallbackClient: - def __init__(self) -> None: - self.params: list[dict[str, Any]] = [] - self.closed = 0 - self.connected = 0 - - def subscribe(self, _method: str, params: Mapping[str, Any], **_kwargs: Any) -> Any: - copied = json.loads(json.dumps(params)) - self.params.append(copied) - if any( - item.get("type") == "pane.updated" - for item in copied["subscriptions"] - ): - raise HerdrErrorResponse( - { - "code": "invalid_request", - "message": "invalid request: unknown variant pane.updated", - }, - "subscribe-1", - uncorrelated=True, - ) - return SimpleNamespace(subscription_id="herdr-074-compatible") - - def close(self) -> None: - self.closed += 1 - - def connect(self) -> None: - self.connected += 1 - - fallback = Herdr074FallbackClient() - before_fallback_inputs = list(derivation_inputs) - before_fallback = latest_snapshot(new_backend.db_path, new_backend.config.host_id) - stream = new_backend._subscribe_event_stream(fallback) - after_fallback = latest_snapshot(new_backend.db_path, new_backend.config.host_id) - assert stream.subscription_id == "herdr-074-compatible" - assert fallback.closed == fallback.connected == 1 - assert len(fallback.params) == 2 - assert all( - item["type"] != "pane.updated" - for item in fallback.params[1]["subscriptions"] - ) - assert {item["pane_id"] for item in fallback.params[1]["subscriptions"]} == { - pane["pane_id"] - } - assert derivation_inputs == before_fallback_inputs - assert before_fallback == after_fallback - - fallback_event_start = len(derivation_inputs) - assert new_backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": {**pane, "agent_status": "blocked"}, - } - ) - assert derivation_inputs[fallback_event_start:] == [] - assert stable_key_bytes(new_backend) == reference_key - assert worker_meta_bytes(new_backend) == reference_meta - - # This models the observation-layer failure: a scalar refresh arrives with - # a second identity-looking tuple. It must neither call the derivation - # function nor replace any persisted worker metadata. - before_dangerous_update_inputs = list(derivation_inputs) - before_dangerous_update_meta = worker_meta_bytes(new_backend) - assert new_backend.queue_event_envelope( - { - "event": "pane.updated", - "data": { - "type": "pane_updated", - "workspace_id": "wD2", - "pane_id": "wD2:p7", - "revision": 3, - }, - } - ) - assert refreshes == [None, None, None] - assert derivation_inputs == before_dangerous_update_inputs - assert stable_key_bytes(new_backend) == reference_key - assert worker_meta_bytes(new_backend) == before_dangerous_update_meta - - -def test_cross_path_and_cross_representation_observations_keep_one_stable_key( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - adapter = tmp_path / "identity-turn-adapter" - adapter.write_text( - "#!/usr/bin/env python3\n" - "import json\n" - "print(json.dumps({'result': {'turn': {" - "'available': True, 'complete': True, 'user_text': 'identity read', " - "'assistant_final_text': 'stable', 'source_turn_id': 'identity-turn', " - "'workspace_id': 7, 'pane_id': 41}}}))\n", - encoding="utf-8", - ) - adapter.chmod(0o700) - config = Config( - host_id="cross-path-identity", - data_dir=tmp_path, - db_path=tmp_path / "cross-path.db", - herdr_backend="socket", - herdr_bin=str(adapter), - herdr_timeout_seconds=1, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - record_observations: list[ - tuple[str, bool, str | None, str | None, str | None, str | None] - ] = [] - real_worker_record_from_item = herdr_cli._worker_record_from_item - - def capture_worker_record( - item: Mapping[str, Any], - record_config: Config | None = None, - *, - pane_info_observed: bool = False, - identity_source: str = "unknown", - ) -> Any: - record = real_worker_record_from_item( - item, - record_config, - pane_info_observed=pane_info_observed, - identity_source=identity_source, - ) - record_observations.append( - ( - record.identity_source, - record.pane_info_observed, - record.observed_workspace_id, - record.observed_pane_id, - record.workspace_id, - record.pane_id, - ) - ) - return record - - monkeypatch.setattr(herdr_cli, "_worker_record_from_item", capture_worker_record) - monkeypatch.setattr(herdr_events, "_worker_record_from_item", capture_worker_record) - pane = { - "workspace_id": "w65383a2e877513", - "pane_id": "w65383a2e877513:pA", - "terminal_id": "terminal-cross-path", - "agent": "claude", - "agent_status": "working", - "label": "cross-path", - } - client = _StaticClient( - workspaces=[{"id": pane["workspace_id"], "name": "Build"}], - panes=[pane], - ) - - stable_keys: list[str] = [] - - def capture_key() -> None: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert len(snapshot.workers) == 1 - stable_keys.append(str(snapshot.workers[0].meta["stable_key"])) - - backend.reconcile_once(client=client) - capture_key() - assert ( - "pane.list", - True, - pane["workspace_id"], - pane["pane_id"], - pane["workspace_id"], - pane["pane_id"], - ) in record_observations - records = backend._records_from_reconcile_payloads( - {"agents": []}, - {"panes": [pane]}, - ) - assert len(records) == 1 - assert records[0].identity_source == "pane.list" - assert (records[0].workspace_id, records[0].pane_id) == ( - pane["workspace_id"], - pane["pane_id"], - ) - - # Full PaneInfo events may use aliases, but record construction still - # stores the exact canonical public pair used by pane.list. - assert backend.queue_event_envelope( - { - "event": "pane.created", - "data": { - "type": "pane_created", - "pane": { - "workspaceId": pane["workspace_id"], - "paneId": pane["pane_id"], - "terminalId": pane["terminal_id"], - "agent": "claude", - "agentStatus": "idle", - "label": "cross-path", - }, - }, - } - ) - capture_key() - assert ( - "event:pane.created", - True, - pane["workspace_id"], - pane["pane_id"], - pane["workspace_id"], - pane["pane_id"], - ) in record_observations - - # pane.updated is a scalar refresh notification in Herdr 0.7.5. Even a - # different identity-looking representation cannot enter worker records. - assert backend.queue_event_envelope( - { - "event": "pane.updated", - "data": { - "type": "pane_output_changed", - "workspace_id": 7, - "pane_id": 41, - "revision": 2, - }, - } - ) - capture_key() - assert all(source != "event:pane.updated" for source, *_rest in record_observations) - - # Scalar events may carry raw runtime representations. They can update a - # matched worker but cannot become PaneInfo or feed stable-key derivation. - assert backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": { - "workspace_id": 7, - "pane_id": 41, - "terminal_id": pane["terminal_id"], - "agent": "claude", - "agent_status": "working", - }, - } - ) - capture_key() - assert ( - "event:pane.agent_status_changed", - False, - "7", - "41", - None, - None, - ) in record_observations - - binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - assert binding.turn_target_kind == "pane_id" - assert herdr_turns.refresh_turn_binding(config, binding).status in { - "updated", - "unchanged", - } - capture_key() - - backend.reconcile_once(client=client) - capture_key() - assert len(set(stable_keys)) == 1 - - -def test_one_hundred_interleaved_identity_observations_never_drift( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - real_stable_worker_key = herdr_cli.stable_worker_key - derived_keys: set[str] = set() - - def capture_stable_worker_key(*args: Any, **kwargs: Any) -> str: - stable_key = real_stable_worker_key(*args, **kwargs) - derived_keys.add(stable_key) - return stable_key - - monkeypatch.setattr(herdr_cli, "stable_worker_key", capture_stable_worker_key) - backend = _backend(tmp_path, "interleaved-identity") - pane = { - "workspace_id": "w65383a2e877513", - "pane_id": "w65383a2e877513:pA", - "terminal_id": "terminal-interleaved", - "agent": "claude", - "agent_status": "working", - } - client = _StaticClient( - workspaces=[{"id": pane["workspace_id"], "name": "Build"}], - panes=[pane], - ) - backend.reconcile_once(client=client) - binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - monkeypatch.setattr( - herdr_turns, - "_read_turn_for_binding", - lambda *_args, **_kwargs: { - "complete": True, - "user_text": "identity read", - "assistant_final_text": "stable", - "source_turn_id": "identity-turn", - "workspace_id": 7, - "pane_id": 41, - }, - ) - - observed_keys: set[str] = set() - - def remember_key() -> None: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - observed_keys.add(str(snapshot.workers[0].meta["stable_key"])) - - remember_key() - for index in range(100): - path = index % 5 - if path == 0: - assert backend.queue_event_envelope( - { - "event": "pane.created", - "data": {"pane": {**pane, "agent_status": "idle"}}, - } - ) - elif path == 1: - assert backend.queue_event_envelope( - { - "event": "pane.updated", - "data": { - "workspace_id": 7, - "pane_id": 41, - "revision": index, - }, - } - ) - elif path == 2: - assert backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": { - "workspace_id": 7, - "pane_id": 41, - "terminal_id": pane["terminal_id"], - "agent": "claude", - "agent_status": "working", - }, - } - ) - elif path == 3: - backend.reconcile_once(client=client) - binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - else: - assert herdr_turns.refresh_turn_binding( - backend.config, - binding, - ).status in {"updated", "unchanged"} - remember_key() - - assert len(observed_keys) == 1 - assert derived_keys == observed_keys - - -@pytest.mark.parametrize("turn_model", ["legacy", "observed"]) -@pytest.mark.parametrize( - "event_envelope", - [ - pytest.param( - { - "event": "pane.agent_status_changed", - "data": { - "pane_id": "w123456789abcde:pA", - "workspace_id": "w123456789abcde", - "agent": "claude", - "agent_status": "blocked", - }, - }, - id="herdr-074-status-event", - ), - pytest.param( - { - "event": "pane_updated", - "data": { - "type": "pane_updated", - "pane": { - "pane_id": "w123456789abcde:pA", - "workspace_id": "w123456789abcde", - "terminal_id": "terminal-decision", - "agent": "claude", - "agent_status": "blocked", - }, - }, - }, - id="herdr-075-pane-updated-event", - ), - pytest.param( - { - "event": "pane_agent_status_changed", - "data": { - "type": "pane_agent_status_changed", - "pane_id": "w123456789abcde:pA", - "workspace_id": "w123456789abcde", - "agent": "claude", - "display_agent": "Claude", - "agent_status": "blocked", - "state_labels": {}, - "title": None, - }, - }, - id="herdr-075-agent-status-event", - ), - ], -) -def test_idless_blocked_event_persists_and_lists_decision( - tmp_path: Path, - turn_model: str, - event_envelope: dict[str, Any], -) -> None: - backend, _binding = _decision_backend(tmp_path, turn_model) - refreshes: list[Any] = [] - - def refresh_current() -> None: - binding = next(iter(backend._bindings.values())) - refreshes.append( - herdr_turns.refresh_turn_binding( - backend.config, binding, adapter_timeout_seconds=2 - ) - ) - - backend.set_turn_refresh_callback(refresh_current) - def handler(conn: _SocketConnection) -> None: - request = conn.read_request() - subscriptions = request["params"]["subscriptions"] - assert {"type": "pane.updated"} in subscriptions - assert { - "type": "pane.agent_status_changed", - "pane_id": "w123456789abcde:pA", - } in subscriptions - conn.send_json( - {"id": request["id"], "result": {"type": "subscription_started"}} - ) - conn.send_json(event_envelope) - with _FakeHerdrSocketServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = backend._subscribe_event_stream(client) - envelope = client.read_event(stream.subscription_id, timeout=1) - assert envelope.get("id") is None - assert backend.queue_event_envelope(envelope) is True - client.close() - assert refreshes == [herdr_turns.TurnRefreshResult("updated", 1, True)] - _assert_decision_persisted(backend) -@pytest.mark.parametrize("turn_model", ["legacy", "observed"]) -def test_reconcile_fallback_refreshes_and_lists_decision_without_status_event( - tmp_path: Path, - turn_model: str, -) -> None: - backend, _binding = _decision_backend(tmp_path, turn_model) - refreshes: list[Any] = [] - - def refresh_current() -> None: - binding = next(iter(backend._bindings.values())) - refreshes.append( - herdr_turns.refresh_turn_binding( - backend.config, binding, adapter_timeout_seconds=2 - ) - ) - backend.set_turn_refresh_callback(refresh_current) - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[ - { - "pane_id": "w123456789abcde:pA", - "terminal_id": "terminal-decision", - "agent": "claude", - "workspace_id": "w123456789abcde", - "agent_status": "blocked", - } - ], - agents=[], - ) - ) - assert refreshes == [herdr_turns.TurnRefreshResult("updated", 1, True)] - _assert_decision_persisted(backend) def test_backend_rejects_non_official_subscribe_method(tmp_path: Path) -> None: @@ -1658,97 +1016,29 @@ def test_supported_nested_agent_or_worker_canonical_fields_cannot_mint_continuit _assert_no_public_json_forbidden(json.loads(snapshot.to_json())) -def test_nested_compatibility_event_cannot_duplicate_authenticated_turn_owner( + + +@pytest.mark.parametrize("entity_source", ["top_level", "pane"]) +def test_official_pane_tuple_provenance_mints_continuity( tmp_path: Path, + entity_source: str, ) -> None: - backend = _backend(tmp_path, "nested-turn-owner-conflict") - session = { - "source": "old-source-secret", + backend = _backend(tmp_path, f"official-pane-{entity_source}") + backend.reconcile_once( + client=_StaticClient(workspaces=[{"id": "wR9", "name": "Build"}]) + ) + pane = { "agent": "codex", - "kind": "id", - "value": "shared-session-secret", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": session, - "status": "running", - } - ], - agents=[ - { - "worker_id": "public-old-owner", - "agent_id": "old-agent-target-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": session, - "status": "running", - } - ], - ) - ) - - assert backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "worker_id": "public-compatibility-claimant", - "agent_id": "new-agent-target-secret", - "agent": "codex", - "agent_session": { - "source": "new-source-secret", - "agent": "codex", - "kind": "id", - "value": "shared-session-secret", - }, - "status": "running", - } - }} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert len(snapshot.workers) == len(bindings) == 1 - assert "stable_key" not in snapshot.workers[0].meta - assert bindings[0].sendable is False - assert bindings[0].reason == "ambiguous_pane_match" - assert bindings[0].turn_target_kind is None - assert bindings[0].turn_target_value is None - - -@pytest.mark.parametrize("entity_source", ["top_level", "pane"]) -def test_official_pane_tuple_provenance_mints_continuity( - tmp_path: Path, - entity_source: str, -) -> None: - backend = _backend(tmp_path, f"official-pane-{entity_source}") - backend.reconcile_once( - client=_StaticClient(workspaces=[{"id": "wR9", "name": "Build"}]) - ) - pane = { - "agent": "codex", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "official-terminal-secret", - "status": "running", - "agent_session": { - "source": "official-source-secret", - "agent": "codex", - "kind": "id", - "value": "official-session-secret", - }, + "workspace_id": "wR9", + "pane_id": "wR9:pA", + "terminal_id": "official-terminal-secret", + "status": "running", + "agent_session": { + "source": "official-source-secret", + "agent": "codex", + "kind": "id", + "value": "official-session-secret", + }, } payload = pane if entity_source == "top_level" else {"pane": pane} @@ -1764,263 +1054,10 @@ def test_official_pane_tuple_provenance_mints_continuity( assert backend.config.installation_key_path.exists() -@pytest.mark.parametrize( - "event_name", - ["pane.agent_detected", "pane.agent_status_changed"], -) -@pytest.mark.parametrize("key_failure", [False, True]) -def test_official_idless_event_reuses_single_authenticated_pane_owner( - tmp_path: Path, - key_failure: bool, - event_name: str, -) -> None: - backend = _backend( - tmp_path, - f"event-pane-owner-id-churn-{key_failure}-{event_name}", - ) - old_session = { - "source": "old-source-secret", - "agent": "codex", - "kind": "id", - "value": "old-session-secret", - } - pane = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": old_session, - "status": "running", - } - agent = { - "worker_id": "public-old-owner", - "agent_id": "old-agent-target-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": old_session, - "status": "running", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane], - agents=[agent], - ) - ) - before = latest_snapshot(backend.db_path, backend.config.host_id) - before_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert before is not None - assert len(before.workers) == len(before_bindings) == 1 - worker_id = before.workers[0].id - stable_key = before.workers[0].meta["stable_key"] - if key_failure: - backend.config.installation_key_marker_path.unlink() - - event_payload = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": { - "worker_id": "public-new-owner", - "agent_id": "new-agent-target-secret", - "terminal_id": "new-terminal-secret", - "agent": "codex", - "agent_session": { - "source": "new-source-secret", - "agent": "codex", - "kind": "id", - "value": "new-session-secret", - }, - "status": "working", - }, - } - assert backend.queue_event_envelope( - {"event": event_name, "data": event_payload} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert after is not None - assert len(after.workers) == len(bindings) == 1 - assert after.workers[0].id == worker_id - assert after.workers[0].meta["stable_key"] == stable_key - assert after.backend_health[0].status == "healthy" - if key_failure: - assert not backend.config.installation_key_marker_path.exists() - assert bindings[0].worker_id == worker_id - assert bindings[0].private_fingerprint == before_bindings[0].private_fingerprint - assert (bindings[0].target_kind, bindings[0].target_value) == ( - before_bindings[0].target_kind, - before_bindings[0].target_value, - ) - assert (bindings[0].turn_target_kind, bindings[0].turn_target_value) == ( - before_bindings[0].turn_target_kind, - before_bindings[0].turn_target_value, - ) - _assert_no_public_json_forbidden(json.loads(after.to_json())) - - -@pytest.mark.parametrize("shared_owner", ["terminal_id", "agent_session"]) -@pytest.mark.parametrize("key_failure", [False, True]) -def test_incremental_shared_private_owner_fails_closed_across_canonical_panes( - tmp_path: Path, - shared_owner: str, - key_failure: bool, -) -> None: - backend = _backend( - tmp_path, - f"event-shared-{shared_owner}-owner-{key_failure}", - ) - backend.reconcile_once( - client=_StaticClient(workspaces=[{"id": "wR9", "name": "Build"}]) - ) - - for suffix, pane_id in (("a", "wR9:pA"), ("b", "wR9:pB")): - assert backend.queue_event_envelope( - {"event": "pane.created", "data": { - "workspace_id": "wR9", - "pane_id": pane_id, - "terminal_id": ( - "shared-terminal-secret" - if shared_owner == "terminal_id" - else f"terminal-{suffix}-secret" - ), - "agent_id": f"agent-{suffix}-secret", - "agent": "codex", - "agent_session": { - "source": f"source-{suffix}-secret", - "agent": "codex", - "kind": "id", - "value": ( - "shared-session-secret" - if shared_owner == "agent_session" - else f"session-{suffix}-secret" - ), - }, - "status": "running", - }} - ) - if suffix == "a" and key_failure: - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.created", "data": { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": ( - "shared-terminal-secret" - if shared_owner == "terminal_id" - else "terminal-b-secret" - ), - "agent_id": "agent-b-secret", - "agent": "codex", - "agent_session": { - "source": "source-b-secret", - "agent": "codex", - "kind": "id", - "value": ( - "shared-session-secret" - if shared_owner == "agent_session" - else "session-b-secret" - ), - }, - "status": "running", - }} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert len(snapshot.workers) == len(bindings) == 1 - if key_failure: - assert snapshot.workers[0].meta["stable_key"].startswith("wsk1_") - assert snapshot.workers[0].meta["stable_key_version"] == 1 - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "continuity_unavailable" - assert bindings[0].sendable is True - assert bindings[0].reason is None - assert bindings[0].turn_target_kind is not None - assert bindings[0].turn_target_value is not None - else: - assert "stable_key" not in snapshot.workers[0].meta - assert "stable_key_version" not in snapshot.workers[0].meta - assert bindings[0].sendable is False - assert bindings[0].reason == "ambiguous_pane_match" - assert bindings[0].turn_target_kind is None - assert bindings[0].turn_target_value is None -@pytest.mark.parametrize("complete_move", [False, True]) -def test_move_into_owned_pane_fails_closed_for_both_owners( - tmp_path: Path, - complete_move: bool, -) -> None: - backend = _backend( - tmp_path, - f"event-move-owned-destination-{complete_move}", - ) - panes = [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent A", - "status": "running", - }, - { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "agent": "Agent B", - "status": "running", - }, - ] - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=panes, - ) - ) - payload: dict[str, Any] = { - "previous_pane_id": "wR9:pB", - "new_pane_id": "wR9:pA", - } - if complete_move: - payload["pane"] = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent B", - "status": "running", - } - assert backend.queue_event_envelope( - {"event": "pane.moved", "data": payload} - ) - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert len(snapshot.workers) == len(bindings) == 2 - assert all("stable_key" not in worker.meta for worker in snapshot.workers) - assert all("stable_key_version" not in worker.meta for worker in snapshot.workers) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert all(binding.turn_target_kind is None for binding in bindings) - assert all(binding.turn_target_value is None for binding in bindings) def test_key_failure_precedes_move_conflict_mutation(tmp_path: Path) -> None: @@ -2080,108 +1117,6 @@ def test_key_failure_precedes_move_conflict_mutation(tmp_path: Path) -> None: ) -@pytest.mark.parametrize("key_failure", [False, True]) -def test_authoritative_move_resolves_agent_targeted_source_by_previous_pane( - tmp_path: Path, - key_failure: bool, -) -> None: - backend = _backend( - tmp_path, - f"event-move-agent-targeted-source-{key_failure}", - ) - old_session = { - "source": "old-source-secret", - "agent": "codex", - "kind": "id", - "value": "old-session-secret", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[ - {"id": "wR9", "name": "Source"}, - {"id": "wD2", "name": "Destination"}, - ], - panes=[ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": old_session, - "status": "running", - } - ], - agents=[ - { - "worker_id": "public-source-owner", - "agent_id": "old-agent-target-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": old_session, - "status": "running", - } - ], - ) - ) - before = latest_snapshot(backend.db_path, backend.config.host_id) - before_binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - assert before is not None - worker_id = before.workers[0].id - original_key = before.workers[0].meta["stable_key"] - if key_failure: - key_bytes = backend.config.installation_key_path.read_bytes() - backend.config.installation_key_path.write_bytes( - bytes(byte ^ 0xFF for byte in key_bytes) - ) - - assert backend.queue_event_envelope( - {"event": "pane.moved", "data": { - "previous_pane_id": "wR9:pA", - "pane": { - "workspace_id": "wD2", - "pane_id": "wD2:p7", - "terminal_id": "new-terminal-secret", - "agent": "codex", - "agent_session": { - "source": "new-source-secret", - "agent": "codex", - "kind": "id", - "value": "new-session-secret", - }, - "status": "running", - }, - }} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert after is not None - assert len(after.workers) == len(bindings) == 1 - assert after.workers[0].id == worker_id - if key_failure: - assert after.workers == before.workers - assert after.workers[0].meta["stable_key"] == original_key - assert bindings == [before_binding] - assert after.backend_health[0].status == "degraded" - assert after.backend_health[0].outcome == "continuity_unavailable" - else: - assert after.workers[0].meta["stable_key"] != original_key - assert after.workers[0].space_id == "wD2" - assert bindings[0].private_fingerprint == before_binding.private_fingerprint - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == "new-terminal-secret" - assert bindings[0].turn_target_kind == "codex_session_id" - assert bindings[0].turn_target_value == "new-session-secret" def test_reconcile_retains_authenticated_snapshot_until_installation_key_recovers( @@ -2872,89 +1807,13 @@ def wait(self, timeout: float | None = None) -> bool: backend.stop() -def test_start_timeout_cancels_remaining_per_pane_replay_and_joins_worker( + + +@pytest.mark.parametrize("batched", [False, True], ids=["one-flush-per-event", "one-batch"]) +def test_real_idless_working_idle_working_preserves_every_transition( tmp_path: Path, monkeypatch: Any, -) -> None: - config = Config( - host_id="initial-reconcile-cancel", - data_dir=tmp_path, - db_path=tmp_path / "initial-reconcile-cancel.db", - herdr_backend="socket", - herdr_timeout_seconds=0.1, - herdr_initial_reconcile_timeout_seconds=0.01, - ) - init_store(Path(config.db_path)) - - class Client: - def close(self) -> None: - return None - - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - raise AssertionError("the patched replay hooks own this test") - - backend = HerdrEventBackend( - config, - client_factory=lambda _config: Client(), - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - consumed: list[str] = [] - replay_started = threading.Event() - - class ReadinessGate: - def clear(self) -> None: - return None - - def is_set(self) -> bool: - return False - - def set(self) -> None: - return None - - def wait(self, timeout: float | None = None) -> bool: - assert replay_started.wait(1.0) - return False - - backend._ready = ReadinessGate() # type: ignore[assignment] - - def reconcile_once(*, client: Any) -> None: - backend._subscription_pane_ids = ["pane-1", "pane-2"] - - def probe_turn_api( - client: Any, - pane_id: str, - watermark: Any, - ) -> tuple[bool, None, None]: - return True, None, None - - def consume_pane_replay( - client: Any, - pane_id: str, - watermark: Any, - **kwargs: Any, - ) -> None: - consumed.append(pane_id) - replay_started.set() - time.sleep(0.05) - - monkeypatch.setattr(backend, "reconcile_once", reconcile_once) - monkeypatch.setattr(backend, "_probe_turn_api", probe_turn_api) - monkeypatch.setattr(backend, "_consume_pane_replay", consume_pane_replay) - - with pytest.raises(HerdrSocketTimeoutError): - backend.start(wait_for_reconcile=True) - - assert consumed == ["pane-1"] - assert backend.running is False - assert backend._thread is None - - -@pytest.mark.parametrize("batched", [False, True], ids=["one-flush-per-event", "one-batch"]) -def test_real_idless_working_idle_working_preserves_every_transition( - tmp_path: Path, - monkeypatch: Any, - batched: bool, + batched: bool, ) -> None: backend = _backend( tmp_path, @@ -3795,289 +2654,20 @@ def queue_idle_event() -> None: assert len(snapshot.workers) == 1 assert len(list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")) == 1 -def test_reconcile_turn_refresh_observes_durable_state_and_replaced_ownership_maps( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "reconcile-turn-refresh") - client = _StaticClient( - workspaces=[{"id": "space-1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "pane-1", - "terminal_id": "terminal-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - } - ], - agents=[], - ) - observed: list[tuple[str, str]] = [] - - def callback() -> None: - assert not backend._lock._is_owned() - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert bindings - assert backend._pane_terminals == {"pane-1": "terminal-1"} - observed.append((snapshot.to_json(), bindings[0].private_fingerprint)) - # The callback-state lock must also be released before invocation. - backend.set_turn_refresh_callback(None) - - backend.set_turn_refresh_callback(callback) - reconciled = backend.reconcile_once(client=client) - - assert observed == [ - ( - reconciled.to_json(), - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0].private_fingerprint, - ) - ] - - -@pytest.mark.parametrize( - ("event_name", "data"), - [ - ( - "pane.created", - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "working", - }, - ), - ( - "pane.focused", - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "working", - }, - ), - ( - "pane.updated", - { - "pane": { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "working", - }, - }, - ), - ( - "pane.moved", - { - "old_pane_id": "pane-1", - "pane_id": "pane-2", - "agent": "Agent One", - "workspace_id": "space-1", - }, - ), - ("pane.closed", {"pane_id": "pane-1"}), - ("pane.exited", {"pane_id": "pane-1"}), - ( - "pane.agent_detected", - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "working", - }, - ), - ( - "pane.agent_status_changed", - { - "pane_id": "pane-1", - "agent": "Agent One", - "status": "blocked", - }, - ), - ("pane.output_matched", {"pane_id": "pane-1"}), - ], -) -def test_each_turn_relevant_event_notifies_once( - tmp_path: Path, - event_name: str, - data: dict[str, Any], -) -> None: - backend = _backend(tmp_path, f"turn-refresh-{event_name.replace('.', '-')}") - backend.reconcile_once(client=_initial_pane_client()) - calls: list[None] = [] - backend.set_turn_refresh_callback(lambda: calls.append(None)) - - assert backend.queue_event_envelope({"event": event_name, "data": data}) is True - - assert calls == [None] - -def test_relevant_event_burst_notifies_once_after_batch_commit(tmp_path: Path) -> None: - backend = _backend(tmp_path, "turn-refresh-burst", debounce_seconds=60) - backend.reconcile_once(client=_initial_pane_client()) - calls: list[str] = [] - identity = HerdrEventId("turn-refresh-burst") - - def callback() -> None: - assert not backend._lock._is_owned() - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "blocked" - assert identity in backend._producer_dedupe - calls.append(snapshot.updated_at) - - backend.set_turn_refresh_callback(callback) - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane_id": "pane-1", "agent": "Agent One"}}, - flush=False, - ) - assert backend.queue_event_envelope( - {**_status_event("blocked"), "event_id": identity.value}, - flush=False, - ) - assert backend.queue_event_envelope( - {"event": "pane.output_matched", "data": {"pane_id": "pane-1"}}, - flush=False, - ) - - backend.flush() - - assert len(calls) == 1 - - -def test_projection_neutral_output_match_notifies_without_rewriting_snapshot(tmp_path: Path) -> None: - backend = _backend(tmp_path, "turn-refresh-output") - backend.reconcile_once(client=_initial_pane_client()) - before = latest_snapshot(backend.db_path, backend.config.host_id) - assert before is not None - calls: list[None] = [] - backend.set_turn_refresh_callback(lambda: calls.append(None)) - - assert backend.queue_event_envelope( - {"event": "pane.output_matched", "data": {"pane_id": "pane-1"}} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.to_json() == before.to_json() - assert calls == [None] - - -def test_irrelevant_duplicate_and_invalid_events_do_not_notify(tmp_path: Path) -> None: - backend = _backend(tmp_path, "turn-refresh-noop") - backend.reconcile_once(client=_initial_pane_client()) - calls: list[None] = [] - backend.set_turn_refresh_callback(lambda: calls.append(None)) - - assert backend.queue_event_envelope( - { - "event": "workspace.updated", - "data": {"workspace_id": "space-1", "name": "Renamed"}, - } - ) - assert backend.queue_event_envelope( - { - "event": "worktree.created", - "data": {"workspace_id": "missing-space", "name": "Ignored"}, - } - ) - assert backend.queue_event_envelope({"event": "not.official", "data": {}}) is False - assert backend.queue_event_envelope({"event": 7, "data": {}}) is False - assert calls == [] - - backend.set_turn_refresh_callback(None) - duplicate = { - "event": "pane.output_matched", - "event_id": "already-applied", - "data": {"pane_id": "pane-1"}, - } - assert backend.queue_event_envelope(duplicate) is True - backend.set_turn_refresh_callback(lambda: calls.append(None)) - assert backend.queue_event_envelope(duplicate) is False - assert calls == [] - - -def test_persistence_failure_does_not_notify_turn_refresh( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "turn-refresh-persist-failure") - backend.reconcile_once(client=_initial_pane_client()) - calls: list[None] = [] - backend.set_turn_refresh_callback(lambda: calls.append(None)) - def fail_persistence(*, observed_at: str | None = None) -> Any: - raise RuntimeError(f"snapshot unavailable at {observed_at}") - monkeypatch.setattr(backend, "_persist_current_state", fail_persistence) - with pytest.raises(RuntimeError, match="snapshot unavailable"): - backend.queue_event_envelope(_status_event("blocked")) - assert calls == [] -def test_callback_failure_cannot_roll_back_durable_event_state(tmp_path: Path) -> None: - backend = _backend(tmp_path, "turn-refresh-callback-failure") - backend.reconcile_once(client=_initial_pane_client()) - identity = HerdrEventId("callback-failure") - def fail_callback() -> None: - raise RuntimeError("scheduler unavailable") - backend.set_turn_refresh_callback(fail_callback) - assert backend.queue_event_envelope( - {**_status_event("blocked"), "event_id": identity.value} - ) - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "blocked" - assert identity in backend._producer_dedupe - assert backend.queue_event_envelope( - {**_status_event("blocked"), "event_id": identity.value} - ) is False -@pytest.mark.parametrize("event_name", ["pane.closed", "pane.exited"]) -def test_close_refresh_callback_observes_expired_binding( - tmp_path: Path, - event_name: str, -) -> None: - backend = _backend(tmp_path, f"turn-refresh-{event_name.replace('.', '-')}-binding") - backend.reconcile_once(client=_initial_pane_client()) - observed_reasons: list[str | None] = [] - def callback() -> None: - assert list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) == [] - expired = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - include_expired=True, - ) - assert len(expired) == 1 - observed_reasons.append(expired[0].reason) - backend.set_turn_refresh_callback(callback) - assert backend.queue_event_envelope( - {"event": event_name, "data": {"pane_id": "pane-1"}} - ) - assert observed_reasons == [event_name.replace(".", "_")] @@ -4176,38 +2766,6 @@ def test_healthy_empty_reconnect_closes_missing_workers_and_expires_bindings(tmp assert list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") == [] -def test_worker_cap_exceeded_preserves_previous_authoritative_snapshot(tmp_path: Path) -> None: - config = Config( - host_id="worker-cap", - data_dir=tmp_path, - db_path=tmp_path / "worker-cap.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - max_workers=1, - turn_refresh_workers=1, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - backend.reconcile_once(client=_initial_pane_client()) - - capped = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "space-1", "name": "Build"}], - panes=[ - {"pane_id": "pane-1", "agent": "Agent One", "workspace_id": "space-1"}, - {"pane_id": "pane-2", "agent": "Agent Two", "workspace_id": "space-1"}, - ], - ) - ) - latest = latest_snapshot(backend.db_path, backend.config.host_id) - - assert latest is not None - assert capped.content_fingerprint == latest.content_fingerprint - assert [worker.name for worker in capped.workers] == ["Agent One"] - assert capped.backend_health[0].status == "degraded" - assert capped.backend_health[0].outcome == "worker_cap_exceeded" - assert list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") - _assert_no_public_json_forbidden(json.loads(capped.to_json())) def test_output_excerpt_limit_bounds_public_worker_summary(tmp_path: Path) -> None: @@ -4245,188 +2803,32 @@ def test_output_excerpt_limit_bounds_public_worker_summary(tmp_path: Path) -> No assert long_summary not in snapshot.to_json() -def test_incremental_event_over_worker_cap_is_ignored_with_degraded_health(tmp_path: Path) -> None: - config = Config( - host_id="event-worker-cap", - data_dir=tmp_path, - db_path=tmp_path / "event-worker-cap.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - max_workers=1, - turn_refresh_workers=1, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - backend.reconcile_once(client=_initial_pane_client()) - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent_id": "agent-2", - "name": "Agent Two", - "workspace_id": "space-1", - "pane_id": "pane-2", - } - }} - ) - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert [worker.name for worker in snapshot.workers] == ["Agent One"] - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "worker_cap_exceeded" -def test_closed_worker_reactivation_over_worker_cap_is_ignored(tmp_path: Path) -> None: - config = Config( - host_id="event-worker-cap-reactivate", + + +def test_periodic_reconcile_uses_config_and_zero_disables_it(tmp_path: Path) -> None: + disabled_config = Config( + host_id="periodic-disabled", data_dir=tmp_path, - db_path=tmp_path / "event-worker-cap-reactivate.db", + db_path=tmp_path / "periodic-disabled.db", herdr_backend="socket", - herdr_timeout_seconds=0.5, - max_workers=1, - turn_refresh_workers=1, + reconcile_interval_seconds=0, ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - backend.reconcile_once(client=_initial_pane_client()) + init_store(Path(disabled_config.db_path)) + disabled = HerdrEventBackend(disabled_config, debounce_seconds=0) + disabled_client = _initial_pane_client() + disabled._next_reconcile_monotonic = time.monotonic() - 1 + disabled._run_periodic_reconcile_if_due(disabled_client) - backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane": {"pane_id": "pane-1", "agent": "Agent One", "workspace_id": "space-1"}}} - ) - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent_id": "agent-2", - "name": "Agent Two", - "workspace_id": "space-1", - "pane_id": "pane-2", - "status": "running", - } - }} - ) - - before_reactivation = latest_snapshot(backend.db_path, backend.config.host_id) - assert before_reactivation is not None - assert {worker.name: worker.status for worker in before_reactivation.workers} == { - "Agent One": "closed", - "Agent Two": "active", - } - - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent": "Agent One", - "workspace_id": "space-1", - "pane_id": "pane-3", - "status": "running", - } - }} - ) - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - - assert snapshot is not None - assert {worker.name: worker.status for worker in snapshot.workers} == { - "Agent One": "closed", - "Agent Two": "active", - } - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "worker_cap_exceeded" - - -def test_pane_moved_reactivation_over_worker_cap_is_ignored(tmp_path: Path) -> None: - config = Config( - host_id="event-worker-cap-moved-reactivate", - data_dir=tmp_path, - db_path=tmp_path / "event-worker-cap-moved-reactivate.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - max_workers=1, - turn_refresh_workers=1, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - first = backend.reconcile_once(client=_initial_pane_client()) - first_worker = first.workers[0] - closed_worker = Worker( - id=first_worker.id, - name=first_worker.name, - status="closed", - space_id=first_worker.space_id, - meta=first_worker.meta, - last_seen_at=first_worker.last_seen_at, - summary=first_worker.summary, - backend_target=first_worker.backend_target, - ) - backend._workers[closed_worker.id] = closed_worker - save_snapshot( - backend.db_path, - project_from_observations( - backend.config, - spaces=first.spaces, - workers=[closed_worker], - backend_health=first.backend_health, - ), - ) - - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent_id": "agent-2", - "name": "Agent Two", - "workspace_id": "space-1", - "pane_id": "pane-2", - "status": "running", - } - }} - ) - before_move = latest_snapshot(backend.db_path, backend.config.host_id) - assert before_move is not None - assert {worker.name: worker.status for worker in before_move.workers} == { - "Agent One": "closed", - "Agent Two": "active", - } - - backend.queue_event_envelope( - {"event": "pane.moved", "data": { - "old_pane_id": "pane-1", - "pane_id": "pane-3", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - }} - ) - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - - assert snapshot is not None - assert {worker.name: worker.status for worker in snapshot.workers} == { - "Agent One": "closed", - "Agent Two": "active", - } - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "worker_cap_exceeded" - - -def test_periodic_reconcile_uses_config_and_zero_disables_it(tmp_path: Path) -> None: - disabled_config = Config( - host_id="periodic-disabled", - data_dir=tmp_path, - db_path=tmp_path / "periodic-disabled.db", - herdr_backend="socket", - reconcile_interval_seconds=0, - ) - init_store(Path(disabled_config.db_path)) - disabled = HerdrEventBackend(disabled_config, debounce_seconds=0) - disabled_client = _initial_pane_client() - disabled._next_reconcile_monotonic = time.monotonic() - 1 - disabled._run_periodic_reconcile_if_due(disabled_client) - - enabled_config = Config( - host_id="periodic-enabled", - data_dir=tmp_path, - db_path=tmp_path / "periodic-enabled.db", - herdr_backend="socket", - reconcile_interval_seconds=0.001, + enabled_config = Config( + host_id="periodic-enabled", + data_dir=tmp_path, + db_path=tmp_path / "periodic-enabled.db", + herdr_backend="socket", + reconcile_interval_seconds=0.001, ) init_store(Path(enabled_config.db_path)) enabled = HerdrEventBackend(enabled_config, debounce_seconds=0) @@ -4439,161 +2841,10 @@ def test_periodic_reconcile_uses_config_and_zero_disables_it(tmp_path: Path) -> assert enabled.operational_status["last_reconcile_at"] is not None -def test_periodic_reconcile_replays_supported_turn_ledger( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = Config( - host_id="periodic-turn-replay", - data_dir=tmp_path, - db_path=tmp_path / "periodic-turn-replay.db", - herdr_backend="socket", - reconcile_interval_seconds=0.001, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - backend._turn_api_supported = True - replays: list[bool] = [] - monkeypatch.setattr( - backend, - "_replay_turns_after_reconcile", - lambda: replays.append(True), - ) - backend._next_reconcile_monotonic = time.monotonic() - 1 - - backend._run_periodic_reconcile_if_due(_initial_pane_client()) - - assert replays == [True] - - -def test_periodic_reconcile_recovers_retry_without_stream_disconnect( - tmp_path: Path, -) -> None: - calls = 0 - - def process(_config: Config, _pane_id: str, **_kwargs: Any) -> Any: - nonlocal calls - calls += 1 - return SimpleNamespace( - status="binding_missing" if calls == 1 else "updated", - worker_id="claude", - refreshed_turn_id=None if calls == 1 else "periodic-public-turn", - ) - - config = Config( - host_id="periodic-live-turn-retry", - data_dir=tmp_path, - db_path=tmp_path / "periodic-live-turn-retry.db", - herdr_backend="socket", - reconcile_interval_seconds=300, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend( - config, - debounce_seconds=0, - reconnect_delay_seconds=0, - turn_completion_processor=process, - ) - pane = _turn_api_pane() - pane_id = pane["pane_id"] - reconcile_client = _StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[pane], - ) - backend.reconcile_once(client=reconcile_client) - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - record = herdr_events._turn_completion_record( - {"pane": pane, **_turn_record(1)} - ) - backend._process_turn_record(record) - assert calls == 1 - assert backend._next_turn_replay_monotonic is not None - _force_herdr_turn_refresh_retry_due(backend, pane_id, 1) - backend._turn_api_probed = True - backend._turn_api_supported = True - - class ReplayClient: - def connect(self) -> None: - return None - - def close(self) -> None: - return None - - def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: - assert params == { - "pane_id": pane_id, - "since": 0, - "expected_epoch": 7, - } - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": [_turn_record(1)], - "truncated": False, - "oldest_available": 1, - } - } - - backend.client_factory = lambda _config: ReplayClient() - backend._next_reconcile_monotonic = time.monotonic() + 300 - backend._next_turn_replay_monotonic = time.monotonic() - 1 - backend._run_periodic_reconcile_if_due(reconcile_client) - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert calls == 2 - assert watermark is not None and watermark.last_turn == 1 - assert reconcile_client.calls == [ - "workspace.list", - "tab.list", - "pane.list", - "agent.list", - ] -def test_refresh_retry_ignores_producer_time_and_survives_clock_rollback( - tmp_path: Path, -) -> None: - db_path = tmp_path / "local-retry-clock.db" - init_store(db_path) - retry = record_herdr_turn_refresh_retry( - db_path, - "local-clock-host", - "pane-clock", - turn_epoch=1, - turn=1, - refresh_status="binding_missing", - now="2026-08-02T00:00:10+00:00", - ) - assert retry.first_seen_at == "2026-08-02T00:00:10+00:00" - assert retry.attempt_count == 1 - assert not herdr_turn_refresh_retry_due( - db_path, - "local-clock-host", - "pane-clock", - turn_epoch=1, - turn=1, - now="2026-08-02T00:00:10.500000+00:00", - ) - assert herdr_turn_refresh_retry_due( - db_path, - "local-clock-host", - "pane-clock", - turn_epoch=1, - turn=1, - now="2026-08-01T23:59:00+00:00", - ) def test_debounce_batches_until_flush_and_shutdown_flushes(tmp_path: Path) -> None: @@ -4840,44 +3091,6 @@ def test_status_event_with_pane_id_only_updates_bound_worker_not_a_phantom(tmp_p assert workers[idle_worker_id].status in {"idle", "done"} -def test_pane_id_only_status_event_resolves_codex_binding_via_pane_terminal_map(tmp_path: Path) -> None: - """Regression: codex bindings' turn target is a session id, so pane-id-only - status events must resolve through the pane->terminal map remembered from - reconcile instead of inserting a phantom bare 'codex' worker.""" - backend = _backend(tmp_path, "codex-phantom-host") - client = _StaticClient( - workspaces=[{"id": "wX8", "name": "projectx", "status": "active"}], - panes=[ - { - "pane_id": "wX8:p1", - "terminal_id": "term-ctx", - "agent": "codex", - "agent_session": {"agent": "codex", "kind": "id", "value": "019f-session"}, - "workspace_id": "wX8", - "agent_status": "working", - } - ], - agents=[], - ) - backend.reconcile_once(client=client) - assert sorted(backend._workers) == ["codex"] - binding = next(iter(backend._bindings.values())) - assert binding.turn_target_kind == "codex_session_id" - assert backend._pane_terminals == {"wX8:p1": "term-ctx"} - - event = normalize_event( - {"event": "pane.agent_status_changed", "data": {"pane_id": "wX8:p1", "workspace_id": "wX8", "agent_status": "idle"}} - ) - assert event is not None - assert backend._apply_event(event) is True - assert sorted(backend._workers) == ["codex"], f"phantom inserted: {sorted(backend._workers)}" - assert backend._workers["codex"].status in {"idle", "done"} - updated_binding = next(iter(backend._bindings.values())) - assert updated_binding.private_fingerprint == binding.private_fingerprint - assert updated_binding.target_kind == binding.target_kind == "terminal_id" - assert updated_binding.target_value == binding.target_value == "term-ctx" - assert updated_binding.turn_target_kind == binding.turn_target_kind == "codex_session_id" - assert updated_binding.turn_target_value == binding.turn_target_value == "019f-session" def test_reconcile_drops_unbound_missing_workers_but_keeps_bound_closed(tmp_path: Path) -> None: @@ -5044,2399 +3257,3 @@ def test_accepted_move_removes_source_pane_terminal_alias_before_stale_close( assert backend._workers[worker.id].status == worker.status assert backend._workers[worker.id].status != "closed" assert backend._pane_terminals == {"P2": "T1"} - - -def test_pane_only_status_preserves_unobserved_owner_aliases_for_later_conflict( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "status-owner-alias-provenance") - session = { - "source": "old-source-secret", - "agent": "codex", - "kind": "id", - "value": "old-session-secret", - } - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "wR9:pA", - "terminal_id": "shared-terminal-secret", - "workspace_id": "wR9", - "agent": "codex", - "agent_session": session, - "agent_status": "working", - } - ], - agents=[ - { - "worker_id": "public-owner", - "agent_id": "old-agent-target-secret", - "pane_id": "wR9:pA", - "terminal_id": "shared-terminal-secret", - "workspace_id": "wR9", - "agent": "codex", - "agent_session": session, - "agent_status": "working", - } - ], - ) - ) - worker = initial.workers[0] - binding = next(iter(backend._bindings.values())) - assert binding.target_kind == "agent_id" - assert binding.turn_target_kind == "codex_session_id" - - assert backend.queue_event_envelope( - {"event": "pane.agent_status_changed", "data": { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "status": "idle", - }} - ) - status_binding = next(iter(backend._bindings.values())) - assert status_binding.private_fingerprint == binding.private_fingerprint - assert status_binding.target_kind == binding.target_kind - assert status_binding.target_value == binding.target_value - assert status_binding.turn_target_kind == binding.turn_target_kind - assert status_binding.turn_target_value == binding.turn_target_value - assert backend._terminal_owners == { - "shared-terminal-secret": {worker.id}, - } - assert backend._session_owners == { - "old-session-secret": {worker.id}, - } - - assert backend.queue_event_envelope( - {"event": "pane.created", "data": { - "pane": { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "shared-terminal-secret", - "agent_id": "new-agent-target-secret", - "agent": "codex", - "agent_session": { - "source": "new-source-secret", - "agent": "codex", - "kind": "id", - "value": "new-session-secret", - }, - "status": "working", - } - }} - ) - - assert set(backend._workers) == {worker.id} - conflicted = next(iter(backend._bindings.values())) - assert conflicted.worker_id == worker.id - assert conflicted.sendable is False - assert conflicted.reason == "ambiguous_pane_match" - assert "wR9:pB" not in backend._pane_terminals - - -def test_nested_compatibility_replay_cannot_rehabilitate_ambiguous_binding( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "compatibility-ambiguity-provenance") - session = { - "source": "shared-source-secret", - "agent": "codex", - "kind": "id", - "value": "shared-session-secret", - } - original_agent = { - "worker_id": "public-owner-a", - "agent_id": "agent-target-a-secret", - "pane_id": "wR9:pA", - "terminal_id": "terminal-a-secret", - "workspace_id": "wR9", - "agent": "codex", - "agent_session": session, - "status": "working", - } - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "wR9:pA", - "terminal_id": "terminal-a-secret", - "workspace_id": "wR9", - "agent": "codex", - "agent_session": session, - "status": "working", - } - ], - agents=[original_agent], - ) - ) - worker = initial.workers[0] - - assert backend.queue_event_envelope( - {"event": "pane.created", "data": { - "pane": { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "terminal-b-secret", - "agent_id": "agent-target-b-secret", - "agent": "codex", - "agent_session": session, - "status": "working", - } - }} - ) - ambiguous = next(iter(backend._bindings.values())) - assert ambiguous.worker_id == worker.id - assert ambiguous.target_value == "agent-target-a-secret" - assert ambiguous.sendable is False - assert ambiguous.reason == "ambiguous_pane_match" - assert ambiguous.turn_target_kind is None - assert ambiguous.turn_target_value is None - assert backend._workers[worker.id].backend_target == ambiguous.backend_target() - - assert backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": {"agent": original_agent}} - ) - - replayed = next(iter(backend._bindings.values())) - assert replayed.private_fingerprint == ambiguous.private_fingerprint - assert replayed.target_kind == ambiguous.target_kind - assert replayed.target_value == ambiguous.target_value - assert replayed.sendable is False - assert replayed.reason == "ambiguous_pane_match" - assert replayed.turn_target_kind is None - assert replayed.turn_target_value is None - assert backend._workers[worker.id].backend_target == replayed.backend_target() - assert backend._pane_terminals == { - "wR9:pA": "terminal-a-secret", - } - - -def _run_authoritative_recovery_trace( - tmp_path: Path, - monkeypatch: Any, -) -> dict[str, Any]: - backend = _backend(tmp_path, "authoritative-turn-recovery") - session = { - "source": "codex", - "agent": "codex", - "kind": "id", - "value": "session-private-recovery", - } - agent = { - "worker_id": "public-recovery-worker", - "agent_id": "agent-private-recovery", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-private-a", - "agent": "codex", - "agent_session": session, - "status": "working", - } - pane_a = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-private-a", - "agent": "codex", - "agent_session": session, - "status": "working", - } - workspace = [{"id": "wR9", "name": "Build", "status": "active"}] - - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=workspace, - panes=[pane_a], - agents=[agent], - ) - ) - worker = initial.workers[0] - initial_binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - assert initial_binding.worker_id == worker.id - assert initial_binding.turn_target_kind == "codex_session_id" - assert initial_binding.turn_target_value == "session-private-recovery" - assert merge_turn_content( - backend.db_path, - backend.config.host_id, - worker.id, - { - "user_text": "Recover the deterministic producer final.", - "assistant_stream_text": "Still working.", - "assistant_final_text": None, - "complete": False, - "has_open_turn": True, - "source_turn_id": "producer-turn-42", - }, - ) == 1 - seeded_payload = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=initial, - ) - seeded_turns = [ - turn - for turn in seeded_payload["turns"] - if turn.get("user_text") == "Recover the deterministic producer final." - ] - assert len(seeded_turns) == 1 - public_turn_id = seeded_turns[0]["id"] - public_source_turn_id = seeded_turns[0]["source_turn_id"] - assert public_source_turn_id != "producer-turn-42" - assert seeded_turns[0]["worker_id"] == worker.id - assert seeded_turns[0]["complete"] is False - - adapter_calls: list[tuple[str, str | None]] = [] - - def refresh_existing_final( - config: Config, - binding: Any, - *, - adapter_timeout_seconds: float | None = None, - ) -> Any: - del adapter_timeout_seconds - adapter_calls.append((binding.worker_id, binding.turn_target_value)) - applied = herdr_turns.apply_turn_refresh( - config.db_path, - config.host_id, - binding.worker_id, - { - "user_text": "Recover the deterministic producer final.", - "assistant_stream_text": None, - "assistant_final_text": "The durable producer final.", - "complete": True, - "has_open_turn": False, - "source_turn_id": "producer-turn-42", - }, - expected_binding=binding, - ) - assert applied.updated == 1 - return herdr_turns.TurnRefreshResult("updated", 1, applied.pending_changed) - - monkeypatch.setattr(herdr_turns, "refresh_turn_binding", refresh_existing_final) - pane_b = { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "terminal-private-b", - "agent": "codex", - "agent_session": session, - "status": "working", - } - quarantined = backend.reconcile_once( - client=_StaticClient( - workspaces=workspace, - panes=[pane_a, pane_b], - agents=[agent], - ) - ) - quarantined_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert len(quarantined_bindings) == 1 - quarantined_binding = quarantined_bindings[0] - assert quarantined_binding.worker_id == worker.id - assert quarantined_binding.sendable is False - assert quarantined_binding.reason == "ambiguous_pane_match" - assert quarantined_binding.turn_target_kind is None - assert quarantined_binding.turn_target_value is None - assert herdr_turns.refresh_structured_turn_content(backend.config) == { - "ok": True, - "status": "ok", - "updated": 0, - "attempted": 0, - } - assert adapter_calls == [] - quarantined_payload = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=quarantined, - ) - quarantined_turn = next( - turn for turn in quarantined_payload["turns"] if turn["id"] == public_turn_id - ) - assert quarantined_turn["source_turn_id"] == public_source_turn_id - assert quarantined_turn["assistant_final_text"] is None - assert quarantined_turn["assistant_stream_text"] == "Still working." - assert quarantined_turn["complete"] is False - - restarted = HerdrEventBackend( - backend.config, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - assert len(restarted._bindings) == 1 - restarted_binding = next(iter(restarted._bindings.values())) - assert restarted_binding.worker_id == worker.id - assert restarted_binding.reason == "ambiguous_pane_match" - assert restarted_binding.turn_target_value is None - - recovered = restarted.reconcile_once( - client=_StaticClient( - workspaces=workspace, - panes=[pane_a], - agents=[agent], - ) - ) - recovered_bindings = list_worker_bindings( - restarted.db_path, - restarted.config.host_id, - backend="herdr", - ) - assert len(recovered_bindings) == 1 - recovered_binding = recovered_bindings[0] - assert recovered.workers[0].id == worker.id - assert recovered_binding.worker_id == initial_binding.worker_id - assert recovered_binding.private_fingerprint == initial_binding.private_fingerprint - assert recovered_binding.sendable is True - assert recovered_binding.reason is None - assert recovered_binding.turn_target_kind == "codex_session_id" - assert recovered_binding.turn_target_value == "session-private-recovery" - - assert herdr_turns.refresh_turn_binding( - restarted.config, - recovered_binding, - ) == herdr_turns.TurnRefreshResult("updated", 1) - assert adapter_calls == [(worker.id, "session-private-recovery")] - - final_payload = turns_payload_from_store( - restarted.db_path, - restarted.config.host_id, - snapshot=recovered, - ) - final_turns = [ - turn - for turn in final_payload["turns"] - if turn.get("assistant_final_text") == "The durable producer final." - ] - assert len(final_turns) == 1 - assert final_turns[0]["id"] == public_turn_id - assert final_turns[0]["source_turn_id"] == public_source_turn_id - assert final_turns[0]["worker_id"] == worker.id - assert final_turns[0]["assistant_stream_text"] is None - assert final_turns[0]["complete"] is True - assert final_turns[0]["has_open_turn"] is False - public_json = json.dumps(final_payload, sort_keys=True) - for private_value in ( - "producer-turn-42", - "agent-private-recovery", - "session-private-recovery", - "terminal-private-a", - "terminal-private-b", - "wR9:pA", - "wR9:pB", - ): - assert private_value not in public_json - _assert_no_public_json_forbidden(json.loads(public_json)) - snapshot_payload = json.loads(recovered.to_json()) - snapshot_payload["ok"] = True - return { - "turns_payload": final_payload, - "snapshot_payload": snapshot_payload, - "public_turn": final_turns[0], - } - - -def test_authoritative_reconcile_quarantines_then_recovers_existing_final_once( - tmp_path: Path, - monkeypatch: Any, -) -> None: - _run_authoritative_recovery_trace(tmp_path, monkeypatch) - - -@pytest.mark.skipif( - not os.environ.get("HERDRES_SOURCE_DIR"), - reason="HERDRES_SOURCE_DIR is required for the cross-repository recovery contract", -) -def test_authoritative_recovery_payload_promotes_herdres_working_once( - tmp_path: Path, - monkeypatch: Any, -) -> None: - trace = _run_authoritative_recovery_trace(tmp_path, monkeypatch) - turns_payload = trace["turns_payload"] - snapshot_payload = trace["snapshot_payload"] - public_turn = trace["public_turn"] - assert turns_payload["schema_version"] == 1 - assert public_turn["id"] - - source_dir = Path(os.environ["HERDRES_SOURCE_DIR"]).expanduser().resolve() - package_roots = [ - candidate - for candidate in (source_dir, source_dir / "src") - if (candidate / "herdres_connector" / "source_sync.py").is_file() - ] - assert len(package_roots) == 1, ( - "HERDRES_SOURCE_DIR must contain herdres_connector/source_sync.py " - "either directly or under src/" - ) - package_root = package_roots[0] - monkeypatch.syspath_prepend(str(package_root)) - preloaded_herdres_modules = sorted( - name - for name in sys.modules - if name == "herdres_connector" or name.startswith("herdres_connector.") - ) - assert preloaded_herdres_modules == [] - direct_boundary_attempts: list[str] = [] - - def reject_direct_boundary(*_args: Any, **_kwargs: Any) -> Any: - direct_boundary_attempts.append("process_or_socket") - raise AssertionError("Herdres source sync must not access Herdr outside Tendwire") - - class RejectDirectSocket(socket.socket): - def __new__(cls, *_args: Any, **_kwargs: Any) -> Any: - direct_boundary_attempts.append("socket") - raise AssertionError("Herdres source sync must not open a direct socket") - - - real_import = builtins.__import__ - - def guarded_import( - name: str, - globals: Any = None, - locals: Any = None, - fromlist: Any = (), - level: int = 0, - ) -> Any: - private_roots = { - "herdr", - "herdr_turn_adapter", - "herdr_socket", - "herdr_cli", - "herdr_events", - } - if name.split(".", 1)[0] in private_roots or name.startswith( - "tendwire.backends" - ): - direct_boundary_attempts.append(f"import:{name[:80]}") - raise AssertionError("Herdres source sync must not import a direct Herdr client") - return real_import(name, globals, locals, fromlist, level) - - monkeypatch.setattr(builtins, "__import__", guarded_import) - monkeypatch.setattr(subprocess, "run", reject_direct_boundary) - monkeypatch.setattr(subprocess, "Popen", reject_direct_boundary) - monkeypatch.setattr(socket, "socket", RejectDirectSocket) - monkeypatch.setattr(socket, "create_connection", reject_direct_boundary) - monkeypatch.setenv("HERDRES_TENDWIRE_MODE", "source") - monkeypatch.setenv("HERDRES_PINNED_STATUS", "0") - source_sync = importlib.import_module("herdres_connector.source_sync") - herdres_state = importlib.import_module("herdres_connector.state") - source_sync_path = Path(source_sync.__file__).resolve() - loaded_herdres_files = { - name: Path(module.__file__).resolve() - for name, module in sys.modules.items() - if (name == "herdres_connector" or name.startswith("herdres_connector.")) - and getattr(module, "__file__", None) - } - assert source_sync_path == package_root / "herdres_connector" / "source_sync.py" - assert loaded_herdres_files - assert all(path.is_relative_to(package_root) for path in loaded_herdres_files.values()) - - worker_observation = next( - worker - for worker in snapshot_payload["workers"] - if worker["id"] == public_turn["worker_id"] - ) - space_observation = next( - space - for space in snapshot_payload["spaces"] - if space["id"] == public_turn["space_id"] - ) - store = { - "enabled": True, - "telegram": {"chat_id": "-100", "general_thread_id": "1"}, - "panes": {}, - "spaces": {}, - "tendwired_bootstrap_complete": True, - } - _worker_key, worker_entry, _created = herdres_state.upsert_worker_entry( - store, - worker_observation, - topic_id="77", - ) - herdres_state.upsert_space_entry( - store, - space_observation, - topic_id="77", - ) - worker_entry.update( - { - "last_stream_turn_id": public_turn["id"], - "last_stream_hash": "persisted-working-hash", - "last_stream_message_id": "555", - "last_stream_bot_kind": "manager", - } - ) - herdres_state.bind_message_to_worker( - store, - "555", - worker_entry, - topic_id="77", - kind="working", - turn_id=public_turn["id"], - bot_kind="manager", - ) - - class RecoveryTendwire: - def __init__(self) -> None: - self.calls: list[str] = [] - self.turn_payload_objects: list[dict[str, Any]] = [] - - def snapshot(self) -> dict[str, Any]: - self.calls.append("snapshot") - return snapshot_payload - - def turns(self) -> dict[str, Any]: - self.calls.append("turns") - self.turn_payload_objects.append(turns_payload) - return turns_payload - - def pending(self) -> dict[str, Any]: - self.calls.append("pending") - return {"ok": True, "pending_interactions": []} - - class RecoveryTelegram: - dry_run = False - - def __init__( - self, - token: str = "fake", - shared: dict[str, list[Any]] | None = None, - ) -> None: - self.token = token - self.shared = shared or { - "sent": [], - "edited": [], - "topics": [], - "deleted_topics": [], - "renamed_topics": [], - "pins": [], - "api_calls": [], - "icon_edits": [], - } - self.sent = self.shared["sent"] - self.edited = self.shared["edited"] - self.topics = self.shared["topics"] - self.deleted_topics = self.shared["deleted_topics"] - self.renamed_topics = self.shared["renamed_topics"] - self.pins = self.shared["pins"] - self.api_calls = self.shared["api_calls"] - self.icon_edits = self.shared["icon_edits"] - - def with_token(self, token: str) -> Any: - return RecoveryTelegram(token=token, shared=self.shared) - - def api(self, method: str, payload: dict[str, Any]) -> dict[str, Any]: - self.api_calls.append((method, dict(payload), self.token)) - if method == "editMessageText": - rich_payload = payload.get("rich_message") - rich = json.loads(rich_payload) if rich_payload else {} - html = str(rich.get("html") or payload.get("text") or "") - message_id = str(payload.get("message_id") or "") - self.edited.append((str(payload.get("chat_id") or ""), message_id, html)) - return {"ok": True, "result": {"message_id": message_id}} - if method == "sendRichMessage": - message_id = str(100 + len(self.sent)) - rich = json.loads(payload.get("rich_message") or "{}") - self.sent.append( - ( - str(payload.get("chat_id") or ""), - str(rich.get("html") or ""), - {"thread_id": str(payload.get("message_thread_id") or "")}, - message_id, - ) - ) - return {"ok": True, "result": {"message_id": message_id}} - return {"ok": True, "result": {"message_id": "0"}} - - def create_topic( - self, - _chat_id: str, - name: str, - icon_color: int | None = None, - ) -> dict[str, Any]: - self.topics.append((name, icon_color)) - return {"ok": True, "topic_id": str(76 + len(self.topics))} - - def rename_topic( - self, - chat_id: str, - thread_id: str, - name: str, - ) -> dict[str, Any]: - self.renamed_topics.append((chat_id, thread_id, name)) - return {"ok": True} - - def edit_topic_icon( - self, - chat_id: str, - thread_id: str, - emoji_id: str, - ) -> dict[str, Any]: - self.icon_edits.append((chat_id, thread_id, emoji_id)) - return {"ok": True} - - def delete_topic(self, _chat_id: str, thread_id: str) -> dict[str, Any]: - self.deleted_topics.append(thread_id) - return {"ok": True} - - def send_message( - self, - chat_id: str, - html: str, - **kwargs: Any, - ) -> dict[str, Any]: - message_id = str(100 + len(self.sent)) - self.sent.append((chat_id, html, dict(kwargs), message_id)) - return {"ok": True, "message_id": message_id} - - def edit_message( - self, - chat_id: str, - message_id: str, - html: str, - ) -> dict[str, Any]: - self.edited.append((chat_id, str(message_id), html)) - return {"ok": True, "message_id": str(message_id)} - - def pin_message(self, chat_id: str, message_id: str) -> dict[str, Any]: - self.pins.append((chat_id, str(message_id))) - return {"ok": True} - - tendwire = RecoveryTendwire() - telegram = RecoveryTelegram() - runtime = source_sync.SyncRuntime(tendwire, telegram, with_outbox=False) - - first = source_sync.sync_once(store, runtime) - ledger_after_first = json.loads( - json.dumps(herdres_state.delivered_turns(store), sort_keys=True) - ) - edits_after_first = list(telegram.edited) - sends_after_first = list(telegram.sent) - second = source_sync.sync_once(store, runtime) - ledger_after_second = json.loads( - json.dumps(herdres_state.delivered_turns(store), sort_keys=True) - ) - third = source_sync.sync_once(store, runtime) - - binding = herdres_state.find_message_binding(store, "555", topic_id="77") - ledger = herdres_state.delivered_turns(store) - assert tendwire.calls == ["snapshot", "turns", "pending"] * 3 - assert tendwire.turn_payload_objects == [turns_payload] * 3 - assert all(payload is turns_payload for payload in tendwire.turn_payload_objects) - assert first["feed_sent"] == first["sent"] == 1 - assert len(telegram.edited) == 1 - assert telegram.edited[0][1] == "555" - assert public_turn["assistant_final_text"] in telegram.edited[0][2] - assert telegram.sent == [] - assert second["feed_sent"] == second["sent"] == second["turn_updates"] == 0 - assert third["feed_sent"] == third["sent"] == third["turn_updates"] == 0 - assert telegram.edited == edits_after_first - assert telegram.sent == sends_after_first - assert ledger_after_second == ledger_after_first - assert ledger == ledger_after_first - assert len(ledger) == 1 - assert list(ledger.values())[0]["turn_id"] == public_turn["id"] - assert binding is not None - assert binding["kind"] == "final" - assert binding["turn_id"] == public_turn["id"] - assert binding["topic_id"] == "77" - assert worker_entry["topic_id"] == "77" - assert worker_entry["last_turn_id"] == public_turn["id"] - assert worker_entry["last_clean_message_id"] == "555" - assert worker_entry["last_clean_message_ids"] == ["555"] - assert "last_stream_turn_id" not in worker_entry - assert "last_stream_hash" not in worker_entry - assert "last_stream_message_id" not in worker_entry - assert "last_stream_bot_kind" not in worker_entry - assert direct_boundary_attempts == [] - - -def _turn_api_pane() -> dict[str, Any]: - return { - "pane_id": "w123456789abcde:pA", - "terminal_id": "terminal-turn-api", - "agent": "claude", - "workspace_id": "w123456789abcde", - "agent_status": "idle", - "last_completed_turn": { - "turn": 0, - "turn_epoch": 7, - "completed_unix_ms": 0, - }, - "turn": 0, - "turn_epoch": 7, - "outcome": "completed", - } - - -def _turn_record( - turn: int, - *, - epoch: int = 7, - outcome: str = "completed", -) -> dict[str, Any]: - return { - "turn": turn, - "turn_epoch": epoch, - "outcome": outcome, - "completed_unix_ms": 1_700_000_000_000 + turn, - "message": f"turn {turn}", - "message_truncated": False, - "agent_session_path": None, - } - - -def _turn_api_backend( - tmp_path: Path, - host_id: str, - processor: Callable[..., Any], -) -> HerdrEventBackend: - backend = _backend(tmp_path, host_id) - backend.turn_completion_processor = processor - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[_turn_api_pane()], - ) - ) - return backend - - -def test_turn_api_absent_preserves_old_subscription_negotiation(tmp_path: Path) -> None: - backend = _turn_api_backend( - tmp_path, - "turn-api-absent", - lambda *_args, **_kwargs: SimpleNamespace(status="updated"), - ) - - class OldClient: - subscriptions: list[dict[str, Any]] = [] - - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - raise HerdrErrorResponse( - {"code": "invalid_params", "message": "unknown method"}, - "probe", - ) - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - self.subscriptions.append(dict(params)) - return SimpleNamespace(subscription_id="old") - - client = OldClient() - backend._replay_turns_after_reconcile(client) - backend._subscribe_event_stream(client) - - assert backend._turn_api_supported is False - assert all( - item["type"] != "pane.turn_completed" - for item in client.subscriptions[0]["subscriptions"] - ) - assert get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - _turn_api_pane()["pane_id"], - ) is None - - -def test_turn_api_idless_unknown_variant_probe_is_inert( - tmp_path: Path, -) -> None: - backend = _turn_api_backend( - tmp_path, - "turn-api-idless-unknown-variant", - lambda *_args, **_kwargs: SimpleNamespace(status="updated"), - ) - # Exact envelope emitted by live stock Herdr 0.7.5 (reported upstream): - # the id is present but EMPTY, and the method name is backticked. - raw_probe_error = { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: unknown variant `pane.turns`, expected one of `pane.read`, `pane.list`, `agent.list`", - }, - } - - def handler(conn: _SocketConnection) -> None: - probe = conn.read_request() - assert probe["method"] == "pane.turns" - conn.send_json(raw_probe_error) - - ordinary = conn.read_request() - assert ordinary["method"] == "workspace.list" - conn.send_json({"id": ordinary["id"], "result": {"workspaces": []}}) - - subscription = conn.read_request() - assert subscription["method"] == HERDR_EVENTS_SUBSCRIBE_METHOD - assert all( - item["type"] != "pane.turn_completed" - for item in subscription["params"]["subscriptions"] - ) - conn.send_json( - { - "id": subscription["id"], - "result": {"type": "subscription_started"}, - } - ) - - with _FakeHerdrSocketServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - backend._replay_turns_after_reconcile(client) - assert client.workspace_list() == {"workspaces": []} - backend._subscribe_event_stream(client) - client.close() - - assert [request["method"] for request in server.requests] == [ - "pane.turns", - "workspace.list", - HERDR_EVENTS_SUBSCRIBE_METHOD, - ] - assert backend._turn_api_supported is False - assert ( - _table_count( - backend.db_path, - backend.config.host_id, - "herdr_turn_watermarks", - ) - == 0 - ) - assert get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - _turn_api_pane()["pane_id"], - ) is None - - -def test_turn_api_unknown_variant_classifier_requires_exact_code_and_prefix() -> None: - assert HerdrEventBackend._turn_api_method_unsupported( - HerdrErrorResponse( - { - "code": "invalid_request", - "message": "invalid request: unknown variant pane.turns", - }, - "probe", - ) - ) - assert not HerdrEventBackend._turn_api_method_unsupported( - HerdrErrorResponse( - { - "code": "invalid_params", - "message": "invalid request: unknown variant pane.turns", - }, - "probe", - ) - ) - assert not HerdrEventBackend._turn_api_method_unsupported( - HerdrErrorResponse( - { - "code": "invalid_request", - "message": "pane.turns failed: unknown variant", - }, - "probe", - ) - ) - - -def test_turn_api_first_connect_baselines_full_ring_without_processing( - tmp_path: Path, -) -> None: - processed: list[str] = [] - binding_hints: list[str | None] = [] - - def process(_config: Config, pane_id: str, **kwargs: Any) -> Any: - processed.append(pane_id) - binding_hints.append(kwargs.get("binding_private_fingerprint")) - return SimpleNamespace( - status="updated", - worker_id="claude", - refreshed_turn_id=f"public-turn-{len(processed)}", - ) - - backend = _turn_api_backend(tmp_path, "turn-api-happy", process) - - class NewClient: - subscriptions: list[dict[str, Any]] = [] - - def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: - assert params == { - "pane_id": _turn_api_pane()["pane_id"], - "since": 0, - } - return { - "type": "pane_turns", - "turns": { - "pane_id": _turn_api_pane()["pane_id"], - "turn_epoch": 7, - "records": [_turn_record(turn) for turn in range(1, 65)], - "truncated": False, - "oldest_available": 1, - }, - } - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - self.subscriptions.append(dict(params)) - return SimpleNamespace(subscription_id="new") - - client = NewClient() - backend._replay_turns_after_reconcile(client) - backend._subscribe_event_stream(client) - - assert processed == [] - assert binding_hints == [] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - _turn_api_pane()["pane_id"], - ) - assert watermark is not None - assert (watermark.turn_epoch, watermark.last_turn) == (7, 64) - assert watermark.completeness_break_count == 0 - assert { - "type": "pane.turn_completed", - "pane_id": _turn_api_pane()["pane_id"], - } in client.subscriptions[0]["subscriptions"] - with sqlite3.connect(str(backend.db_path)) as conn: - assert conn.execute( - """ - SELECT turn, outcome, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? - ORDER BY turn - """, - (backend.config.host_id,), - ).fetchall() == [] - - -def test_turn_completion_late_attaches_final_after_idle_event_wins_race( - tmp_path: Path, - monkeypatch: Any, -) -> None: - config = Config( - host_id="turn-api-content-race", - data_dir=tmp_path, - db_path=tmp_path / "turn-api-content-race.db", - herdr_backend="socket", - herdr_bin="turn-adapter", - herdr_timeout_seconds=0.5, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend( - config, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - pane_id = _turn_api_pane()["pane_id"] - working_pane = { - **_turn_api_pane(), - "agent_status": "working", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[working_pane], - ) - ) - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - - # Reproduce the production ordering: the status stream closes the public - # projection first, before any semantic adapter read has run. - assert backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": { - **working_pane, - "agent_status": "idle", - }, - } - ) - idle_snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert idle_snapshot is not None - idle_payload = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=idle_snapshot, - ) - assert idle_snapshot.workers[0].status == "idle" - assert not any( - turn.get("assistant_final_text") for turn in idle_payload["turns"] - ) - - adapter_calls: list[list[str]] = [] - final_payload = { - "result": { - "turn": { - "available": True, - "user_text": "reply with purple elephant 42", - "assistant_final_text": "purple elephant 42", - "complete": True, - "has_open_turn": False, - "source_turn_id": "event-close-race-turn", - } - } - } - - def fake_run(args: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]: - adapter_calls.append(args) - return subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps(final_payload), - stderr="", - ) - - monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) - backend._turn_api_supported = True - assert backend.queue_event_envelope( - { - "event": "pane.turn_completed", - "data": { - "pane": { - **working_pane, - "agent_status": "idle", - }, - **_turn_record(1), - }, - } - ) - - completed_snapshot = latest_snapshot( - backend.db_path, - backend.config.host_id, - ) - assert completed_snapshot is not None - captured_payload = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=completed_snapshot, - ) - captured = next( - turn - for turn in captured_payload["turns"] - if turn.get("assistant_final_text") == "purple elephant 42" - ) - assert captured["status"] == "idle" - assert captured["complete"] is True - assert adapter_calls == [ - [ - "turn-adapter", - "pane", - "turn", - pane_id, - "--last", - "--format", - "json", - ] - ] - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - completion = conn.execute( - """ - SELECT turn, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - """, - (backend.config.host_id, pane_id), - ).fetchone() - assert completion == (1, captured["id"]) - - -@pytest.mark.parametrize( - "refresh_status", - [ - "timeout", - "failed", - "stale_binding", - "binding_missing", - "binding_ambiguous", - "store_unavailable", - ], -) -def test_default_completion_processor_retryable_status_preserves_watermark( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - refresh_status: str, -) -> None: - monkeypatch.setattr(herdr_events.time, "time", lambda: 1_700_000_000.1) - backend = _backend(tmp_path, f"default-composition-{refresh_status}") - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[_turn_api_pane()], - ) - ) - pane_id = _turn_api_pane()["pane_id"] - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert len(bindings) == 1 - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - - if refresh_status in {"timeout", "failed", "stale_binding"}: - monkeypatch.setattr( - herdr_turns, - "_refresh_turn_binding", - lambda *_args, **_kwargs: herdr_turns.TurnRefreshResult( - refresh_status, - 0, - ), - ) - elif refresh_status == "binding_missing": - monkeypatch.setattr( - herdr_turns, - "list_worker_bindings", - lambda *_args, **_kwargs: [], - ) - elif refresh_status == "binding_ambiguous": - monkeypatch.setattr( - herdr_turns, - "list_worker_bindings", - lambda *_args, **_kwargs: [bindings[0], bindings[0]], - ) - else: - def unavailable(*_args: Any, **_kwargs: Any) -> Any: - raise RuntimeError("store unavailable") - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", unavailable) - - assert backend.queue_event_envelope( - { - "event": "pane.turn_completed", - "data": {"pane": _turn_api_pane(), **_turn_record(1)}, - } - ) - - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 0 - assert backend.operational_status["turn_completion_diagnostics"] == { - f"herdr_turn_completion_refresh_skipped:{refresh_status}": 1 - } - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - """, - (backend.config.host_id, pane_id), - ).fetchall() == [] - - -def test_completion_binding_race_replays_once_after_binding_appears( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - monkeypatch.setattr(herdr_events.time, "time", lambda: 1_700_000_000.1) - config = Config( - host_id="turn-api-binding-race", - data_dir=tmp_path, - db_path=tmp_path / "turn-api-binding-race.db", - herdr_backend="socket", - herdr_bin="turn-adapter", - herdr_timeout_seconds=0.5, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend( - config, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - pane = _turn_api_pane() - pane_id = pane["pane_id"] - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[pane], - ) - ) - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - - original_list_worker_bindings = herdr_turns.list_worker_bindings - monkeypatch.setattr( - herdr_turns, - "list_worker_bindings", - lambda *_args, **_kwargs: [], - ) - event_data = {"pane": pane, **_turn_record(1)} - assert backend.queue_event_envelope( - {"event": "pane.turn_completed", "data": event_data} - ) - - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 0 - - final_payload = { - "result": { - "turn": { - "available": True, - "user_text": "binding race prompt", - "assistant_final_text": "binding race response", - "complete": True, - "has_open_turn": False, - "source_turn_id": "binding-race-turn", - } - } - } - adapter_calls: list[list[str]] = [] - - def fake_run(args: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]: - adapter_calls.append(args) - return subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps(final_payload), - stderr="", - ) - - monkeypatch.setattr( - herdr_turns, - "list_worker_bindings", - original_list_worker_bindings, - ) - monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) - record = herdr_events._turn_completion_record(event_data) - _force_herdr_turn_refresh_retry_due(backend, pane_id, 1) - backend._process_turn_record(record) - backend._process_turn_record(record) - - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 1 - assert len(adapter_calls) == 1 - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - completion_rows = conn.execute( - """ - SELECT turn, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - """, - (backend.config.host_id, pane_id), - ).fetchall() - final_ready_count = conn.execute( - """ - SELECT COUNT(*) - FROM connector_outbox - WHERE host_id = ? - AND connector = 'turn-final' - AND delivery_kind = 'final_ready' - """, - (backend.config.host_id,), - ).fetchone()[0] - assert len(completion_rows) == 1 - assert completion_rows[0][0] == 1 - assert completion_rows[0][1] - assert final_ready_count == 1 - - -def test_completed_turn_missing_content_remains_terminal( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - backend = _backend(tmp_path, "default-composition-missing") - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[_turn_api_pane()], - ) - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - monkeypatch.setattr( - herdr_turns, - "_refresh_turn_binding", - lambda *_args, **_kwargs: herdr_turns.TurnRefreshResult("missing", 0), - ) - - assert backend.queue_event_envelope( - { - "event": "pane.turn_completed", - "data": {"pane": _turn_api_pane(), **_turn_record(1)}, - } - ) - - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 1 - assert backend.operational_status["turn_completion_diagnostics"] == {} - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - """, - (backend.config.host_id, pane_id), - ).fetchall() == [(1, None)] - - -def test_poison_completion_escalates_then_later_same_pane_turn_advances( - tmp_path: Path, -) -> None: - calls = 0 - - def process(_config: Config, _pane_id: str, **_kwargs: Any) -> Any: - nonlocal calls - calls += 1 - if calls <= herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS: - return SimpleNamespace( - status="failed", - worker_id="claude", - refreshed_turn_id=None, - ) - return SimpleNamespace( - status="updated", - worker_id="claude", - refreshed_turn_id=f"public-turn-{calls}", - ) - - backend = _turn_api_backend(tmp_path, "turn-api-poison", process) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - - poison_record = herdr_events._turn_completion_record( - {"pane": _turn_api_pane(), **_turn_record(1)} - ) - for attempt in range(herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS): - if attempt: - _force_herdr_turn_refresh_retry_due(backend, pane_id, 1) - backend._process_turn_record(poison_record) - backend._process_turn_record( - herdr_events._turn_completion_record( - {"pane": _turn_api_pane(), **_turn_record(2)} - ) - ) - - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 2 - assert backend.operational_status["turn_completion_diagnostics"] == { - "herdr_turn_completion_refresh_skipped:failed": ( - herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS - ), - "herdr_turn_completion_refresh_escalated:failed": 1, - } - retry = get_herdr_turn_refresh_retry( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - turn=1, - ) - assert retry is not None - assert retry.status == "escalated" - assert retry.attempt_count == herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS - assert retry.escalated_at is not None - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - ORDER BY turn - """, - (backend.config.host_id, pane_id), - ).fetchall() == [ - ( - 1, - None, - ), - ( - 2, - f"public-turn-{herdr_events._COMPLETED_TURN_REFRESH_MAX_ATTEMPTS + 1}", - ), - ] - - -def test_retryable_completion_watermark_survives_backend_restart( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - monkeypatch.setattr(herdr_events.time, "time", lambda: 1_700_000_000.1) - first = _turn_api_backend( - tmp_path, - "turn-api-retry-restart", - lambda *_args, **_kwargs: SimpleNamespace( - status="binding_missing", - worker_id=None, - refreshed_turn_id=None, - ), - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - first.db_path, - first.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - record = herdr_events._turn_completion_record( - {"pane": _turn_api_pane(), **_turn_record(1)} - ) - first._process_turn_record(record) - before_restart = get_herdr_turn_watermark( - first.db_path, - first.config.host_id, - pane_id, - ) - assert before_restart is not None and before_restart.last_turn == 0 - - calls = 0 - - def process_after_restart( - _config: Config, - _pane_id: str, - **_kwargs: Any, - ) -> Any: - nonlocal calls - calls += 1 - return SimpleNamespace( - status="updated", - worker_id="claude", - refreshed_turn_id="public-turn-after-restart", - ) - - restarted = HerdrEventBackend( - first.config, - debounce_seconds=0, - reconnect_delay_seconds=0, - turn_completion_processor=process_after_restart, - ) - _force_herdr_turn_refresh_retry_due(restarted, pane_id, 1) - restarted._process_turn_record(record) - restarted._process_turn_record(record) - - after_restart = get_herdr_turn_watermark( - restarted.db_path, - restarted.config.host_id, - pane_id, - ) - assert after_restart is not None and after_restart.last_turn == 1 - assert calls == 1 - with closing(sqlite3.connect(str(restarted.db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - """, - (restarted.config.host_id, pane_id), - ).fetchall() == [(1, "public-turn-after-restart")] - - -def test_restart_finishes_escalation_written_before_completion_watermark( - tmp_path: Path, -) -> None: - processor_calls = 0 - - def process(_config: Config, _pane_id: str, **_kwargs: Any) -> Any: - nonlocal processor_calls - processor_calls += 1 - return SimpleNamespace(status="failed") - - backend = _turn_api_backend( - tmp_path, - "turn-api-escalation-crash-boundary", - process, - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - retry = record_herdr_turn_refresh_retry( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - turn=1, - refresh_status="failed", - max_attempts=1, - ) - assert retry.status == "escalated" - - backend._process_turn_record( - herdr_events._turn_completion_record( - {"pane": _turn_api_pane(), **_turn_record(1)} - ) - ) - - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert processor_calls == 0 - assert watermark is not None and watermark.last_turn == 1 - assert backend.operational_status["turn_completion_diagnostics"] == { - "herdr_turn_completion_refresh_escalated_recovered:failed": 1 - } - - -def test_replay_timeout_on_one_pane_does_not_block_other_panes_or_subscribe( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - monkeypatch.setattr(herdr_events.time, "time", lambda: 1_700_000_000.1) - pane_a = _turn_api_pane() - pane_b = { - **_turn_api_pane(), - "pane_id": "w123456789abcde:pB", - "terminal_id": "terminal-turn-api-b", - "agent": "codex", - } - processed: list[str] = [] - - def process(_config: Config, pane_id: str, **_kwargs: Any) -> Any: - processed.append(pane_id) - return SimpleNamespace( - status="timeout" if pane_id == pane_a["pane_id"] else "updated", - worker_id=pane_id, - refreshed_turn_id=f"public-{pane_id}", - ) - - backend = _backend(tmp_path, "turn-api-pane-timeout") - backend.turn_completion_processor = process - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[pane_a, pane_b], - ) - ) - for pane in (pane_a, pane_b): - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane["pane_id"], - turn_epoch=7, - last_turn=0, - ) - - class Client: - subscriptions: list[dict[str, Any]] = [] - - def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: - pane_id = str(params["pane_id"]) - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": [{**_turn_record(1), "pane_id": pane_id}], - "truncated": False, - "oldest_available": 1, - } - } - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - self.subscriptions.append(dict(params)) - return SimpleNamespace(subscription_id="healthy") - - client = Client() - backend._replay_turns_after_reconcile(client) - backend._subscribe_event_stream(client) - - assert set(processed) == {pane_a["pane_id"], pane_b["pane_id"]} - assert len(client.subscriptions) == 1 - timed_out = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_a["pane_id"], - ) - completed = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_b["pane_id"], - ) - assert timed_out is not None and timed_out.last_turn == 0 - assert completed is not None and completed.last_turn == 1 - - -def test_first_probe_pane_not_found_skips_only_that_pane(tmp_path: Path) -> None: - pane_a = _turn_api_pane() - pane_b = { - **_turn_api_pane(), - "pane_id": "w123456789abcde:pB", - "terminal_id": "terminal-turn-api-b", - "agent": "codex", - } - processed: list[str] = [] - backend = _backend(tmp_path, "turn-api-probe-pane-missing") - backend.turn_completion_processor = ( - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) or SimpleNamespace(status="unchanged") - ) - ) - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[pane_a, pane_b], - ) - ) - for pane in (pane_a, pane_b): - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane["pane_id"], - turn_epoch=7, - last_turn=0, - ) - - class Client: - subscriptions: list[dict[str, Any]] = [] - - def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: - pane_id = str(params["pane_id"]) - if pane_id == pane_a["pane_id"]: - raise HerdrErrorResponse( - {"code": "pane_not_found", "message": "pane disappeared"}, - "probe", - ) - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": [{**_turn_record(1), "pane_id": pane_id}], - "truncated": False, - "oldest_available": 1, - } - } - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - self.subscriptions.append(dict(params)) - return SimpleNamespace(subscription_id="new") - - client = Client() - backend._replay_turns_after_reconcile(client) - backend._subscribe_event_stream(client) - - assert backend._turn_api_supported is True - assert processed == [pane_b["pane_id"]] - assert len(client.subscriptions) == 1 - assert { - "type": "pane.turn_completed", - "pane_id": pane_b["pane_id"], - } in client.subscriptions[0]["subscriptions"] - - -def test_malformed_completion_is_quarantined_without_poisoning_batch( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "turn-api-malformed-batch", debounce_seconds=1) - backend.reconcile_once(client=_initial_pane_client()) - assert backend.queue_event_envelope( - { - "event": "pane.turn_completed", - "data": { - "pane_id": "pane-1", - "turn": "not-an-integer", - "turn_epoch": 7, - "outcome": "completed", - "completed_unix_ms": 1, - }, - }, - flush=False, - ) - assert backend.queue_event_envelope(_status_event("idle"), flush=False) - - backend.flush() - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "idle" - assert backend.health.outcome != "protocol_error" - assert backend.operational_status["turn_completion_diagnostics"] == { - "herdr_turn_completion_record_quarantined": 1 - } - - -def test_replay_gap_inside_records_records_break_without_processing( - tmp_path: Path, -) -> None: - processed: list[str] = [] - backend = _turn_api_backend( - tmp_path, - "turn-api-interior-gap", - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) or SimpleNamespace(status="updated") - ), - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=2, - ) - - class GapClient: - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": [_turn_record(3), _turn_record(5)], - "truncated": False, - "oldest_available": 3, - } - } - - backend._replay_turns_after_reconcile(GapClient()) - - assert processed == [] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None - assert watermark.last_turn == 5 - assert watermark.last_completeness_break_reason == "replay_gap" - - -def test_optional_signature_fallback_never_double_invokes_typeerror( - tmp_path: Path, -) -> None: - calls = 0 - - def broken_processor(*_args: Any, **_kwargs: Any) -> Any: - nonlocal calls - calls += 1 - raise TypeError("processor body failed") - - backend = _turn_api_backend( - tmp_path, - "turn-api-no-double-invoke", - broken_processor, - ) - with pytest.raises(TypeError, match="processor body failed"): - backend._completion_processor_result( - herdr_events._turn_completion_record( - {"pane_id": _turn_api_pane()["pane_id"], **_turn_record(1)} - ) - ) - assert calls == 1 - - pane_calls = 0 - - class BrokenClient: - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - nonlocal pane_calls - pane_calls += 1 - raise TypeError("pane.turns body failed") - - with pytest.raises(TypeError, match="pane.turns body failed"): - backend._call_pane_turns( - BrokenClient(), - _turn_api_pane()["pane_id"], - since=0, - expected_epoch=None, - ) - assert pane_calls == 1 - - -def test_turn_api_run_loop_closes_probe_to_subscribe_completion_race( - tmp_path: Path, -) -> None: - config = _config(tmp_path, "turn-api-subscribe-race") - init_store(Path(config.db_path)) - pane_id = _turn_api_pane()["pane_id"] - processed: list[str] = [] - operations: list[str] = [] - - class RaceClient(_StaticClient): - def __init__(self) -> None: - super().__init__( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[_turn_api_pane()], - ) - self.subscribed = False - - def connect(self) -> None: - return None - - def close(self) -> None: - return None - - def pane_turns( - self, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - operations.append("replay" if self.subscribed else "probe") - records = [_turn_record(1)] if self.subscribed else [] - if self.subscribed: - backend.stop_event.set() - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": records, - "truncated": False, - "oldest_available": 1 if records else None, - } - } - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - operations.append("subscribe") - assert { - "type": "pane.turn_completed", - "pane_id": pane_id, - } in params["subscriptions"] - self.subscribed = True - return SimpleNamespace(subscription_id="race") - - client = RaceClient() - backend = HerdrEventBackend( - config, - client_factory=lambda _config: client, - debounce_seconds=0, - reconnect_delay_seconds=0, - turn_completion_processor=lambda _config, current_pane, **_kwargs: ( - processed.append(current_pane) - or SimpleNamespace(status="unchanged") - ), - ) - - backend.run_forever() - - assert operations == ["probe", "subscribe", "replay"] - assert processed == [pane_id] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None - assert (watermark.turn_epoch, watermark.last_turn) == (7, 1) - - -def test_turn_api_run_loop_keeps_stream_client_exclusive_and_replay_calls_one_shot( - tmp_path: Path, -) -> None: - config = _config(tmp_path, "turn-api-one-shot-connections") - init_store(Path(config.db_path)) - first_pane = _turn_api_pane() - second_pane = { - **_turn_api_pane(), - "pane_id": "w123456789abcde:pB", - "terminal_id": "terminal-turn-api-b", - } - operations: list[str] = [] - subscribed = threading.Event() - second_post_replay_started = threading.Event() - release_second_post_replay = threading.Event() - per_pane_calls = {first_pane["pane_id"]: 0, second_pane["pane_id"]: 0} - replay_clients: list[Any] = [] - - class SubscriptionClient(_StaticClient): - def __init__(self) -> None: - super().__init__( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[first_pane, second_pane], - ) - self.pane_turn_calls = 0 - - def connect(self) -> None: - operations.append("stream.connect") - - def close(self) -> None: - operations.append("stream.close") - - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - self.pane_turn_calls += 1 - raise AssertionError("the subscription client must never carry pane.turns") - - def subscribe( - self, - _method: str, - _params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - operations.append("subscribe") - subscribed.set() - return SimpleNamespace(subscription_id="one-shot-stream") - - def read_event( - self, - _subscription_id: str, - *, - timeout: float | None = None, - ) -> dict[str, Any]: - assert backend.ready is True - backend.stop_event.set() - raise HerdrSocketTimeoutError("idle") - - class OneShotReplayClient: - def __init__(self) -> None: - self.connected = False - self.closed = False - self.calls = 0 - - def connect(self) -> None: - assert self.connected is False - self.connected = True - - def close(self) -> None: - self.closed = True - - def pane_turns( - self, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - assert self.connected is True - assert self.closed is False - self.calls += 1 - assert self.calls == 1, "ordinary Herdr RPC connections are one-shot" - pane_id = str(params["pane_id"]) - per_pane_calls[pane_id] += 1 - phase = "pre" if per_pane_calls[pane_id] == 1 else "post" - operations.append(f"{phase}:{pane_id}") - if phase == "post" and pane_id == second_pane["pane_id"]: - second_post_replay_started.set() - assert release_second_post_replay.wait(1) - records = [_turn_record(1)] if phase == "post" else [] - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": records, - "truncated": False, - "oldest_available": 1 if records else None, - } - } - - stream_client = SubscriptionClient() - - def client_factory(_config: Config) -> Any: - if not replay_clients and not operations: - return stream_client - client = OneShotReplayClient() - replay_clients.append(client) - return client - - backend = HerdrEventBackend( - config, - client_factory=client_factory, - debounce_seconds=0, - reconnect_delay_seconds=0, - turn_completion_processor=lambda *_args, **_kwargs: SimpleNamespace( - status="unchanged" - ), - ) - thread = threading.Thread(target=backend.run_forever, daemon=True) - thread.start() - - assert subscribed.wait(1) - assert second_post_replay_started.wait(1) - assert backend.ready is False - assert stream_client.pane_turn_calls == 0 - - release_second_post_replay.set() - thread.join(timeout=2) - - assert thread.is_alive() is False - assert backend.ready is True - assert len(replay_clients) == 4 - assert all(client.calls == 1 for client in replay_clients) - assert all(client.connected is True and client.closed is True for client in replay_clients) - replay_order = [ - operation - for operation in operations - if operation == "subscribe" or operation.startswith(("pre:", "post:")) - ] - assert replay_order == [ - f"pre:{first_pane['pane_id']}", - f"pre:{second_pane['pane_id']}", - "subscribe", - f"post:{first_pane['pane_id']}", - f"post:{second_pane['pane_id']}", - ] - - -def test_turn_api_production_retry_uses_another_short_lived_client( - tmp_path: Path, -) -> None: - backend = _turn_api_backend( - tmp_path, - "turn-api-isolated-retry", - lambda *_args, **_kwargs: SimpleNamespace(status="unchanged"), - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=2, - ) - backend._turn_api_probed = True - backend._turn_api_supported = True - clients: list[Any] = [] - - class RetryClient: - def __init__(self, ordinal: int) -> None: - self.ordinal = ordinal - self.connected = False - self.closed = False - self.calls = 0 - - def connect(self) -> None: - self.connected = True - - def close(self) -> None: - self.closed = True - - def pane_turns( - self, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - self.calls += 1 - assert self.calls == 1 - if self.ordinal == 1: - assert params["expected_epoch"] == 7 - raise HerdrErrorResponse( - {"code": "turn_epoch_mismatch", "message": "epoch changed"}, - "isolated-retry", - ) - assert params == {"pane_id": pane_id, "since": 0} - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 8, - "records": [_turn_record(1, epoch=8)], - "truncated": False, - "oldest_available": 1, - } - } - - def client_factory(_config: Config) -> RetryClient: - client = RetryClient(len(clients) + 1) - clients.append(client) - return client - - backend.client_factory = client_factory - backend._replay_turns_after_reconcile() - - assert len(clients) == 2 - assert all(client.calls == 1 for client in clients) - assert all(client.connected is True and client.closed is True for client in clients) - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None - assert (watermark.turn_epoch, watermark.last_turn) == (8, 1) - assert watermark.last_completeness_break_reason == "turn_epoch_mismatch" - - -def test_turn_api_restart_replays_exactly_the_missed_turn(tmp_path: Path) -> None: - pane_id = _turn_api_pane()["pane_id"] - first = _turn_api_backend( - tmp_path, - "turn-api-restart", - lambda *_args, **_kwargs: SimpleNamespace(status="unchanged"), - ) - set_herdr_turn_watermark( - first.db_path, - first.config.host_id, - pane_id, - turn_epoch=7, - last_turn=2, - ) - processed: list[str] = [] - restarted = HerdrEventBackend( - first.config, - debounce_seconds=0, - reconnect_delay_seconds=0, - turn_completion_processor=lambda _config, current_pane, **_kwargs: ( - processed.append(current_pane) - or SimpleNamespace(status="unchanged") - ), - ) - restarted.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[_turn_api_pane()], - ) - ) - - class ReplayClient: - params: dict[str, Any] | None = None - - def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: - self.params = dict(params) - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": [_turn_record(3)], - "truncated": False, - "oldest_available": 1, - } - } - - client = ReplayClient() - restarted._replay_turns_after_reconcile(client) - - assert client.params == { - "pane_id": pane_id, - "since": 2, - "expected_epoch": 7, - } - assert processed == [pane_id] - watermark = get_herdr_turn_watermark( - restarted.db_path, - restarted.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 3 - - -@pytest.mark.parametrize( - ("mode", "expected_epoch", "expected_turn", "expected_reason"), - [ - ("truncated", 7, 12, "replay_truncated"), - ("epoch", 8, 1, "turn_epoch_mismatch"), - ], -) -def test_turn_api_breaks_rebaseline_without_inferring_completion( - tmp_path: Path, - mode: str, - expected_epoch: int, - expected_turn: int, - expected_reason: str, -) -> None: - processed: list[str] = [] - backend = _turn_api_backend( - tmp_path, - f"turn-api-break-{mode}", - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) - or SimpleNamespace(status="updated") - ), - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=2, - ) - - class BreakClient: - calls = 0 - - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - self.calls += 1 - if mode == "epoch" and self.calls == 1: - raise HerdrErrorResponse( - { - "code": "turn_epoch_mismatch", - "message": "epoch changed", - }, - "epoch", - ) - records = ( - [_turn_record(1, epoch=8)] - if mode == "epoch" - else [_turn_record(11), _turn_record(12)] - ) - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": expected_epoch, - "records": records, - "truncated": mode == "truncated", - "oldest_available": records[0]["turn"], - } - } - - backend._replay_turns_after_reconcile(BreakClient()) - - assert processed == [] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None - assert (watermark.turn_epoch, watermark.last_turn) == ( - expected_epoch, - expected_turn, - ) - assert watermark.completeness_break_count == 1 - assert watermark.last_completeness_break_reason == expected_reason - - -def test_live_aborted_completion_refreshes_then_advances_with_provenance( - tmp_path: Path, -) -> None: - processed: list[str] = [] - backend = _turn_api_backend( - tmp_path, - "turn-api-live-abort", - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) - or SimpleNamespace( - status="updated", - worker_id="claude", - refreshed_turn_id="public-aborted-turn", - ) - ), - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=1, - ) - backend._turn_api_supported = True - - assert backend.queue_event_envelope( - { - "event": "pane.turn_completed", - "data": { - "pane": _turn_api_pane(), - **_turn_record(2, outcome="aborted"), - }, - } - ) - - assert processed == [pane_id] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 2 - with sqlite3.connect(str(backend.db_path)) as conn: - assert conn.execute( - """ - SELECT outcome, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? AND turn = 2 - """, - (backend.config.host_id, pane_id), - ).fetchone() == ("aborted", "public-aborted-turn") - - -def test_status_turn_hints_never_advance_completion_watermarks( - tmp_path: Path, -) -> None: - processed: list[str] = [] - backend = _turn_api_backend( - tmp_path, - "turn-api-hints-only", - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) - or SimpleNamespace(status="updated") - ), - ) - pane_id = _turn_api_pane()["pane_id"] - - assert backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": { - "pane_id": pane_id, - "workspace_id": "w123456789abcde", - "agent": "claude", - "agent_status": "idle", - "turn": 19, - "turn_epoch": 7, - }, - } - ) - - assert processed == [] - assert get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) is None diff --git a/tests/test_herdr_socket.py b/tests/test_herdr_socket.py index 038c866..b9011f1 100644 --- a/tests/test_herdr_socket.py +++ b/tests/test_herdr_socket.py @@ -180,31 +180,6 @@ def test_client_successful_request_matches_id_and_returns_raw_result(tmp_path: P assert server.requests[0]["params"] == {"scope": "all"} -def test_client_pane_turns_wrapper_uses_additive_method(tmp_path: Path) -> None: - result = { - "type": "pane_turns", - "turns": { - "pane_id": "w1:p1", - "turn_epoch": 3, - "records": [], - "truncated": False, - }, - } - with _FakeHerdrServer(tmp_path, _responding_handler(result)) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert client.pane_turns( - {"pane_id": "w1:p1", "since": 4, "expected_epoch": 3} - ) == result - client.close() - - assert server.requests[0]["method"] == "pane.turns" - assert server.requests[0]["params"] == { - "pane_id": "w1:p1", - "since": 4, - "expected_epoch": 3, - } - assert isinstance(server.requests[0]["id"], str) def test_client_reconnects_after_one_shot_response_connection_closes(tmp_path: Path) -> None: @@ -283,26 +258,6 @@ def handler(conn: _Connection) -> None: client.close() -def test_ordinary_request_rejects_idless_unknown_variant_error( - tmp_path: Path, -) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json( - { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: unknown variant `pane.turns`, expected one of `pane.read`, `pane.list`, `agent.list`", - }, - } - ) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.pane_turns({"pane_id": "w1:p1", "since": 0}) - client.close() def test_client_non_utf8_response_raises_protocol_error(tmp_path: Path) -> None: diff --git a/tests/test_herdr_turns.py b/tests/test_herdr_turns.py deleted file mode 100644 index 6a32aa6..0000000 --- a/tests/test_herdr_turns.py +++ /dev/null @@ -1,2380 +0,0 @@ -"""Tests for private Herdr turn ingestion into public Tendwire turns.""" - -from __future__ import annotations - -import json -import multiprocessing -import sqlite3 -import subprocess -import threading -import pytest -from pathlib import Path -from typing import Any - -from tendwire.backends import herdr_turns -from tendwire.backends.herdr_turns import refresh_structured_turn_content -from tendwire.config import Config -from tendwire.core.models import WorkerBinding, sanitize_canonical_turn_text -from tendwire.core.projector import project_from_raw -from tendwire.core.turns import ( - TURN_CONTENT_PAGE_MAX_UTF8_BYTES, - TURN_STREAM_TEXT_MAX_CHARS, - is_internal_automation_turn_payload, -) -from tendwire.store import sqlite as store_sqlite -from tendwire.store.sqlite import ( - init_store, - save_snapshot, - turns_payload_from_store, - upsert_worker_bindings, -) - - -def _read_test_ipc_request(channel): - return json.loads( - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._CODEX_STATE_IPC_MAX_BYTES, - ).decode("utf-8") - ) - - -def _send_test_ipc_response(channel, response) -> None: - herdr_turns._blocking_send_frame( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - ) - - -def _failed_isolated_turn_child(channel) -> None: - try: - request = _read_test_ipc_request(channel) - _send_test_ipc_response( - channel, - { - "protocol": 1, - "nonce": request["nonce"], - "disposition": "failed", - "content": None, - "parser_state": None, - "bytes_read": 0, - }, - ) - finally: - channel.close() - - -def _invalid_isolated_turn_child(channel) -> None: - try: - _read_test_ipc_request(channel) - _send_test_ipc_response( - channel, - { - "protocol": 1, - "nonce": "not-the-request-nonce", - "disposition": "ok", - "content": None, - "parser_state": None, - "bytes_read": 0, - }, - ) - finally: - channel.close() - - -def _blocked_isolated_turn_child(channel) -> None: - _read_test_ipc_request(channel) - threading.Event().wait(30) - - -def _growing_isolated_turn_child(channel) -> None: - try: - request = _read_test_ipc_request(channel) - large_final = "grew-during-read-" + ("g" * (2 * 1024 * 1024)) - with open(request["target_value"], "a", encoding="utf-8") as handle: - handle.write( - "\n" - + json.dumps( - { - "type": "message", - "id": "grown-final", - "message": { - "role": "assistant", - "stopReason": "stop", - "content": [{"type": "text", "text": large_final}], - }, - }, - separators=(",", ":"), - ) - ) - parser_state = request["parser_state"] - assert parser_state["source"] == "omp" - parsed = herdr_turns._read_omp_session_turn_with_state( - request["target_value"], - herdr_turns._deserialize_omp_state(parser_state["state"]), - ) - content, checkpoint, bytes_read = parsed - response = { - "protocol": 1, - "nonce": request["nonce"], - "disposition": "ok", - "content": content, - "parser_state": { - "source": "omp", - "state": herdr_turns._serialize_omp_state(checkpoint), - }, - "bytes_read": bytes_read, - } - herdr_turns._blocking_send_streamed_omp_response( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - request["nonce"], - ) - finally: - channel.close() - - -def test_refresh_structured_turn_content_uses_private_binding_without_public_leak( - tmp_path: Path, - monkeypatch, -) -> None: - db_path = tmp_path / "turns.db" - config = Config( - host_id="turn-host", - db_path=db_path, - herdr_bin="herdr_turn_adapter.py", - herdr_timeout_seconds=2, - ) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "codex", "status": "active", "space_id": "space-1"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="pane_id", - turn_target_value="pane-private", - sendable=True, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-binding", - ) - ], - ) - calls: list[tuple[list[str], dict[str, Any]]] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append((args, kwargs)) - return subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps( - { - "result": { - "turn": { - "available": True, - "source_turn_id": "private-binding-source", - "user_text": "Why is Telegram showing lifecycle status?", - "assistant_final_text": "Use Tendwire turn text, not pane_id pane-private.", - "assistant_stream_text": "Checking source mode...", - "complete": True, - "has_open_turn": False, - } - } - } - ), - stderr="", - ) - - monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) - - result = refresh_structured_turn_content(config) - payload = turns_payload_from_store(db_path, config.host_id, snapshot=snapshot) - - assert result["attempted"] == 1 - assert result["updated"] == 1 - assert calls[0][0] == [ - "herdr_turn_adapter.py", - "pane", - "turn", - "pane-private", - "--last", - "--format", - "json", - ] - assert calls[0][1]["timeout"] == 2 - turn = payload["turns"][0] - assert turn["user_text"] == "Why is Telegram showing lifecycle status?" - assert "Use Tendwire turn text" in turn["assistant_final_text"] - public_json = json.dumps(payload) - assert "pane-private" not in public_json - assert "agent-private" not in public_json - assert turn["complete"] is True - assert turn["has_open_turn"] is False - - -def test_refresh_structured_turn_content_reads_codex_session_jsonl( - tmp_path: Path, - monkeypatch, -) -> None: - db_path = tmp_path / "turns.db" - codex_home = tmp_path / "codex-home" - session_id = "019f2307-092b-7810-8323-418d7c55bd26" - session_file = ( - codex_home - / "sessions" - / "2026" - / "07" - / "03" - / f"rollout-2026-07-03T00-00-00-{session_id}.jsonl" - ) - session_file.parent.mkdir(parents=True) - turn_id = "turn-live" - lines = [ - { - "timestamp": "2026-07-03T08:00:00Z", - "type": "event_msg", - "payload": {"type": "task_started", "turn_id": turn_id}, - }, - { - "timestamp": "2026-07-03T08:00:01Z", - "type": "response_item", - "payload": { - "type": "message", - "role": "user", - "content": [{"type": "input_text", "text": "Please fix the source feed"}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - }, - }, - { - "timestamp": "2026-07-03T08:00:02Z", - "type": "response_item", - "payload": { - "type": "message", - "role": "assistant", - "phase": "commentary", - "content": [{"type": "output_text", "text": "Checking source state."}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - }, - }, - { - "timestamp": "2026-07-03T08:00:03Z", - "type": "event_msg", - "payload": { - "type": "task_complete", - "turn_id": turn_id, - "last_agent_message": "Fixed the source feed.", - }, - }, - ] - session_file.write_text( - "\n".join(json.dumps(item) for item in lines) + "\n", - encoding="utf-8", - ) - monkeypatch.setenv("CODEX_HOME", str(codex_home)) - monkeypatch.setattr( - herdr_turns.subprocess, - "run", - lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("pane turn fallback should not run")), - ) - - config = Config(host_id="turn-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "codex", "status": "active", "space_id": "space-1"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value="term-private", - turn_target_kind="codex_session_id", - turn_target_value=session_id, - sendable=True, - observed_at="2026-07-03T08:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-binding", - ) - ], - ) - - result = refresh_structured_turn_content(config) - payload = turns_payload_from_store(db_path, config.host_id, snapshot=snapshot) - - assert result == {"ok": True, "status": "ok", "updated": 1, "attempted": 1} - turn = payload["turns"][0] - assert turn["user_text"] == "Please fix the source feed" - assert turn["assistant_final_text"] == "Fixed the source feed." - assert turn["assistant_stream_text"] is None - assert turn["complete"] is True - assert turn["has_open_turn"] is False - public_json = json.dumps(payload) - assert session_id not in public_json - assert "term-private" not in public_json - - -def test_refresh_structured_turn_content_skips_codex_automation_protocol_turn( - tmp_path: Path, - monkeypatch, -) -> None: - db_path = tmp_path / "turns.db" - codex_home = tmp_path / "codex-home" - session_id = "019f31a8-57cf-7353-b4f0-c25e523267af" - session_file = ( - codex_home - / "sessions" - / "2026" - / "07" - / "05" - / f"rollout-2026-07-05T00-00-00-{session_id}.jsonl" - ) - session_file.parent.mkdir(parents=True) - turn_id = "automation-turn" - lines = [ - { - "timestamp": "2026-07-05T08:00:00Z", - "type": "event_msg", - "payload": {"type": "task_started", "turn_id": turn_id}, - }, - { - "timestamp": "2026-07-05T08:00:01Z", - "type": "response_item", - "payload": { - "type": "message", - "role": "user", - "content": [ - { - "type": "input_text", - "text": "Acme job\n\nTemplate: review-lead\nTemplate instructions:", - } - ], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - }, - }, - { - "timestamp": "2026-07-05T08:00:02Z", - "type": "event_msg", - "payload": { - "type": "task_complete", - "turn_id": turn_id, - "last_agent_message": '{"acme_result":{"decision":"approved","summary":"internal job result"}}', - }, - }, - ] - session_file.write_text( - "\n".join(json.dumps(item) for item in lines) + "\n", - encoding="utf-8", - ) - monkeypatch.setenv("CODEX_HOME", str(codex_home)) - - config = Config(host_id="turn-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "codex", "status": "active", "space_id": "space-1"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value="term-private", - turn_target_kind="codex_session_id", - turn_target_value=session_id, - sendable=True, - observed_at="2026-07-05T08:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-binding", - ) - ], - ) - - result = refresh_structured_turn_content(config) - payload = turns_payload_from_store(db_path, config.host_id, snapshot=snapshot) - public_json = json.dumps(payload) - - assert result == {"ok": True, "status": "ok", "updated": 0, "attempted": 1} - assert "Acme job" not in public_json - assert "acme_result" not in public_json - - -def test_turns_payload_from_store_quarantines_existing_automation_protocol_rows( - tmp_path: Path, -) -> None: - db_path = tmp_path / "turns.db" - host_id = "turn-host" - init_store(db_path) - with sqlite3.connect(db_path) as conn: - rows = [ - ( - host_id, - "bad-turn", - "worker-1", - "active", - "task", - "2026-07-05T08:00:00+00:00", - "bad-fp", - "snap-fp", - "2026-07-05T08:00:00+00:00", - json.dumps( - { - "host_id": host_id, - "worker_id": "worker-1", - "status": "active", - "kind": "task", - "user_text": "Acme job\n\nTemplate: review-lead\nTemplate instructions:", - "assistant_final_text": '{"acme_result":{"decision":"approved","summary":"internal"}}', - } - ), - 1, - ), - ( - host_id, - "good-turn", - "worker-1", - "active", - "task", - "2026-07-05T08:01:00+00:00", - "good-fp", - "snap-fp", - "2026-07-05T08:01:00+00:00", - json.dumps( - { - "host_id": host_id, - "worker_id": "worker-1", - "status": "active", - "kind": "task", - "user_text": "Please review the issue", - "assistant_final_text": "Normal answer.", - } - ), - 2, - ), - ] - conn.executemany( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, list_sequence - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - rows, - ) - - payload = turns_payload_from_store(db_path, host_id) - public_json = json.dumps(payload) - - assert len(payload["turns"]) == 1 - assert payload["turns"][0]["assistant_final_text"] == "Normal answer." - assert "Acme job" not in public_json - assert "acme_result" not in public_json - - -def test_internal_user_text_detects_local_command_artifacts() -> None: - assert herdr_turns._is_internal_user_text("Caveat: ...") - assert herdr_turns._is_internal_user_text(" /model") - assert herdr_turns._is_internal_user_text("Set model") - assert herdr_turns._is_internal_user_text("context") - assert herdr_turns._is_internal_user_text("done") - assert not herdr_turns._is_internal_user_text("another test") - - -def test_internal_turn_filter_detects_automation_protocol_without_blocking_discussion() -> None: - assert herdr_turns._is_internal_user_text( - "Acme job\n\nTemplate: review-lead\nTemplate instructions:" - ) - assert herdr_turns._is_internal_user_text( - "Your previous response did not contain a valid acme_result JSON object.\n" - "Validation errors (fix every line):" - ) - assert is_internal_automation_turn_payload( - {"assistant_final_text": '{"acme_result":{"decision":"approved","summary":"internal job result"}}'} - ) - assert is_internal_automation_turn_payload( - {"assistant_final_text": '```json\n{"acme_result":{"decision":"blocked"}}\n```'} - ) - assert not herdr_turns._is_internal_user_text("Can you investigate why automation job responses leaked?") - assert not is_internal_automation_turn_payload( - {"assistant_final_text": "I found a leaked automation_result row in the Tendwire DB."} - ) - assert not is_internal_automation_turn_payload( - { - "user_text": "Please return a JSON status object.", - "assistant_final_text": '{"acme_result":{"status":"ok"}}', - } - ) - - -def test_read_private_turn_skips_local_command_turns(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "user_text": "Caveat: The messages below were generated by the user while running local commands.", - "has_open_turn": True, - "complete": False, - } - } - } - - def fake_run(args, **kwargs): - return subprocess.CompletedProcess(args=args, returncode=0, stdout=json.dumps(payload), stderr="") - - monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) - assert herdr_turns._read_private_turn(config, "pane-1") is None - - -def test_read_private_turn_skips_automation_protocol_turns(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "user_text": "Acme job\n\nTemplate: review-lead\nTemplate instructions:", - "assistant_final_text": '{"acme_result":{"decision":"approved"}}', - "has_open_turn": False, - "complete": True, - "source_turn_id": "automation-turn", - } - } - } - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - assert herdr_turns._read_private_turn(config, "pane-1") is None - - -def test_read_private_turn_skips_promptless_status_finals(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "assistant_final_text": "Initial state (review in progress; gate phase not yet reached). Waiting quietly for the gate verdict or merge. Standing by.", - "complete": True, - "has_open_turn": False, - "source_turn_id": "status-only-turn", - "model": "claude-opus-4-8", - } - } - } - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - assert herdr_turns._read_private_turn(config, "pane-1") is None - - -def test_read_private_turn_keeps_prompted_status_like_final(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "user_text": "What is the current state?", - "assistant_final_text": "Current state: the review is complete.", - "complete": True, - "has_open_turn": False, - "source_turn_id": "prompted-turn", - } - } - } - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - content = herdr_turns._read_private_turn(config, "pane-1") - assert content is not None - assert content["user_text"] == "What is the current state?" - assert content["assistant_final_text"] == "Current state: the review is complete." - - -def _run_returning(payload): - def fake_run(args, **kwargs): - return subprocess.CompletedProcess(args=args, returncode=0, stdout=json.dumps(payload), stderr="") - return fake_run - - -def test_read_private_turn_emits_open_turn_from_open_fields(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - # top level is the PREVIOUS completed turn - "complete": True, - "has_open_turn": True, - "user_text": "previous prompt", - "assistant_final_text": "previous answer", - "source_turn_id": "prompt-prev", - # the in-progress turn is carried in open_* fields - "open_turn_id": "prompt-open", - "open_user_text": "current prompt", - "assistant_stream_text": "thinking live...", - } - } - } - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - content = herdr_turns._read_private_turn(config, "pane-1") - assert content is not None - assert content["user_text"] == "current prompt" - assert content["assistant_stream_text"] == "thinking live..." - assert content["assistant_final_text"] is None - assert content["complete"] is False - assert content["has_open_turn"] is True - # keyed by the OPEN turn's stable prompt id, not the completed one - assert content["source_turn_id"] == "prompt-open" - - -def test_read_private_turn_terminalizes_structured_api_error(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "turn_id": "prompt-error", - "user_text": "Please answer this.", - "assistant_final_text": "", - "complete": False, - "api_error": { - "code": "rate_limit_error", - "text": "You've hit your weekly limit. Try again after the reset.", - }, - } - } - } - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - content = herdr_turns._read_private_turn(config, "pane-1") - - assert content is not None - assert content["source_turn_id"] == "prompt-error" - assert content["assistant_final_text"] == ( - "You've hit your weekly limit. Try again after the reset." - ) - assert content.get("assistant_stream_text") is None - assert content["complete"] is True - assert content["has_open_turn"] is False - - -def test_read_private_turn_terminalizes_open_fields_api_error(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "complete": True, - "has_open_turn": True, - "turn_id": "older", - "user_text": "older prompt", - "assistant_final_text": "older answer", - "open_turn_id": "prompt-error", - "open_user_text": "current prompt", - "api_error": {"code": "overloaded_error", "text": "Provider overloaded."}, - } - } - } - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - content = herdr_turns._read_private_turn(config, "pane-1") - - assert content is not None - assert content["source_turn_id"] == "prompt-error" - assert content["user_text"] == "current prompt" - assert content["assistant_final_text"] == "Provider overloaded." - assert content["complete"] is True - assert content["has_open_turn"] is False - - -def test_open_turn_and_its_completion_share_source_turn_id(monkeypatch) -> None: - """The open turn (prompt-open) and its later completion must share the id so - a working card edits into the final instead of duplicating.""" - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - open_payload = { - "result": { - "turn": { - "available": True, - "complete": True, - "has_open_turn": True, - "user_text": "older", - "assistant_final_text": "older answer", - "source_turn_id": "prompt-older", - "open_turn_id": "prompt-X", - "open_user_text": "the question", - "assistant_stream_text": "working...", - } - } - } - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(open_payload)) - open_content = herdr_turns._read_private_turn(config, "pane-1") - assert open_content["source_turn_id"] == "prompt-X" - assert open_content["complete"] is False - - # Now the same turn completes (no open fields; it is the last completed one). - done_payload = { - "result": { - "turn": { - "available": True, - "complete": True, - "has_open_turn": False, - "user_text": "the question", - "assistant_final_text": "the answer", - "source_turn_id": "prompt-X", - "turn_id": "assistant-uuid-differs", - } - } - } - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(done_payload)) - done_content = herdr_turns._read_private_turn(config, "pane-1") - assert done_content["source_turn_id"] == "prompt-X" # same id, not the assistant uuid - assert done_content["complete"] is True - assert done_content["assistant_final_text"] == "the answer" - - -def test_read_private_turn_prefers_source_turn_id_over_turn_id(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "complete": True, - "has_open_turn": False, - "user_text": "q", - "assistant_final_text": "a", - "source_turn_id": "stable-prompt", - "turn_id": "assistant-uuid", - } - } - } - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - content = herdr_turns._read_private_turn(config, "pane-1") - assert content["source_turn_id"] == "stable-prompt" - - -def test_omp_agent_session_id_also_maps_to_omp_turn_target() -> None: - from tendwire.backends.herdr_cli import _turn_target_from_item - - item = { - "agent": "omp", - "pane_id": "wX:p1", - "agent_session": {"agent": "omp", "kind": "id", "value": "019f-omp-session"}, - } - assert _turn_target_from_item(item) == ("omp_session_path", "019f-omp-session") - - -def _write_omp_session(tmp_path, lines): - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - herdr_turns._OMP_SESSION_CACHE_BINDING_GENERATIONS.clear() - root = tmp_path / "omp-sessions" - session_dir = root / "-demoapp" - session_dir.mkdir(parents=True) - path = session_dir / "2026-07-05T00-00-00-000Z_session.jsonl" - path.write_text("\n".join(json.dumps(line) for line in lines), encoding="utf-8") - return root, path - - -def _write_valid_git_head(git_dir: Path) -> None: - git_dir.mkdir(parents=True) - (git_dir / "HEAD").write_text("ref: refs/heads/main\n", encoding="ascii") - - -def _mark_git_repository(path: Path) -> None: - _write_valid_git_head(path / ".git") - - -def _omp_msg(entry_id, role, text, stop=None, attribution=None): - message = {"role": role, "content": [{"type": "text", "text": text}]} - if stop: - message["stopReason"] = stop - if attribution: - message["attribution"] = attribution - return {"type": "message", "id": entry_id, "message": message} - - -def _omp_read_msg(entry_id, path_value): - return { - "type": "message", - "id": entry_id, - "message": { - "role": "assistant", - "stopReason": "toolUse", - "content": [ - { - "type": "toolCall", - "id": f"{entry_id}-tool", - "name": "read", - "arguments": {"path": path_value}, - } - ], - }, - } - - -def test_read_omp_session_open_then_complete_turn(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session( - tmp_path, - [ - {"type": "session", "id": "s1"}, - _omp_msg("u1", "user", "please fix the bug", attribution="user"), - _omp_msg("a1", "assistant", "looking at the code", stop="toolUse"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - content = herdr_turns._read_omp_session_turn(str(path)) - assert content["user_text"] == "please fix the bug" - assert content["assistant_stream_text"] == "looking at the code" - assert content["complete"] is False - assert content["has_open_turn"] is True - assert content["source_turn_id"] == "u1" - - # Same turn completes: same source id, final text, stream cleared. - path.write_text( - path.read_text(encoding="utf-8") - + "\n" - + json.dumps(_omp_msg("a2", "assistant", "fixed and pushed", stop="stop")), - encoding="utf-8", - ) - done = herdr_turns._read_omp_session_turn(str(path)) - assert done["source_turn_id"] == "u1" - assert done["assistant_final_text"] == "fixed and pushed" - assert done["complete"] is True - assert done["assistant_stream_text"] is None - - -def test_read_omp_session_rejects_paths_outside_root(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session(tmp_path, [_omp_msg("u1", "user", "hi", attribution="user")]) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(tmp_path / "elsewhere")) - assert herdr_turns._read_omp_session_turn(str(path)) is None - - -def test_omp_agent_session_path_maps_to_omp_turn_target() -> None: - from tendwire.backends.herdr_cli import _turn_target_from_item - - item = { - "agent": "omp", - "pane_id": "wX:p1", - "agent_session": {"agent": "omp", "kind": "path", "value": "/home/user/.omp/agent/sessions/-x/a.jsonl"}, - } - assert _turn_target_from_item(item) == ("omp_session_path", "/home/user/.omp/agent/sessions/-x/a.jsonl") - - -def test_omp_open_turn_streams_thinking_headlines(tmp_path, monkeypatch) -> None: - def thinking(entry_id, text): - return {"type": "message", "id": entry_id, "message": {"role": "assistant", "stopReason": "toolUse", "content": [{"type": "thinking", "thinking": text}, {"type": "toolCall", "id": "c1", "name": "bash"}]}} - - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("u1", "user", "add the feature", attribution="user"), - thinking("a1", "**Reading the goal doc**\n\nlong reasoning body..."), - thinking("a2", "checking the branch state first\nmore detail"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - content = herdr_turns._read_omp_session_turn(str(path)) - assert content["has_open_turn"] is True - assert content["assistant_stream_text"] == ( - "Reading the goal doc\n\n" - "step 1 · run command\n\n" - "checking the branch state first\n\n" - "step 2 · run command" - ) - - -def test_omp_cold_start_scans_back_until_current_user_prompt(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("u1", "user", "large turn please", attribution="user"), - _omp_msg("a1", "assistant", "x" * 512, stop="toolUse"), - _omp_msg("a2", "assistant", "finished large turn", stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr(herdr_turns, "_OMP_TAIL_BYTES", 80) - - content = herdr_turns._read_omp_session_turn(str(path)) - - assert content["source_turn_id"] == "u1" - assert content["user_text"] == "large turn please" - assert content["assistant_final_text"] == "finished large turn" - assert content["complete"] is True - - -def test_omp_cold_start_ignores_internal_user_lines_when_finding_prompt(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("u1", "user", "real prompt", attribution="user"), - _omp_msg("a1", "assistant", "x" * 512, stop="toolUse"), - _omp_msg("internal", "user", "\nignore me", attribution="user"), - _omp_msg("a2", "assistant", "still answering real prompt", stop="toolUse"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr(herdr_turns, "_OMP_TAIL_BYTES", 80) - - content = herdr_turns._read_omp_session_turn(str(path)) - - assert content["source_turn_id"] == "u1" - assert content["user_text"] == "real prompt" - assert "still answering real prompt" in content["assistant_stream_text"] - assert "" not in content["assistant_stream_text"] - - -def test_omp_incremental_cache_keeps_turn_state_when_prompt_leaves_tail(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("u1", "user", "keep streaming this", attribution="user"), - _omp_msg("a1", "assistant", "started", stop="toolUse"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr(herdr_turns, "_OMP_TAIL_BYTES", 80) - - first = herdr_turns._read_omp_session_turn(str(path)) - assert first["source_turn_id"] == "u1" - assert first["assistant_stream_text"] == "started" - - def fail_cold_start(*_args): - raise AssertionError("incremental read should not cold-scan a cached growing file") - - monkeypatch.setattr(herdr_turns, "_read_omp_state_from_recent", fail_cold_start) - path.write_text( - path.read_text(encoding="utf-8") - + "\n" - + json.dumps( - { - "type": "message", - "id": "a2", - "message": { - "role": "assistant", - "stopReason": "toolUse", - "content": [ - {"type": "thinking", "thinking": "**Still working**\n\n" + ("x" * 512)}, - {"type": "toolCall", "id": "c1", "name": "bash", "arguments": {"command": "git status"}}, - ], - }, - } - ), - encoding="utf-8", - ) - - second = herdr_turns._read_omp_session_turn(str(path)) - - assert second["source_turn_id"] == "u1" - assert second["user_text"] == "keep streaming this" - assert second["complete"] is False - assert "Still working" in second["assistant_stream_text"] - assert "step 1 · git status" in second["assistant_stream_text"] - - -def test_isolated_omp_response_bound_tracks_valid_growth_during_child_read( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [_omp_msg("growing-user", "user", "wait for large final", attribution="user")], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr(herdr_turns, "_file_turn_child", _growing_isolated_turn_child) - - content = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=5, - ) - - assert content["source_turn_id"] == "growing-user" - assert content["assistant_final_text"].startswith("grew-during-read-") - assert len(content["assistant_final_text"]) > 2 * 1024 * 1024 - assert content["complete"] is True - - -def test_isolated_omp_authoritative_final_over_64mib_streams_losslessly( - tmp_path: Path, - monkeypatch, -) -> None: - large_final = "lossless-omp-" + ( - "z" * (herdr_turns._CODEX_POLL_MAX_BYTES + 257) - ) - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("over-limit-user", "user", "retain exact OMP final", attribution="user"), - _omp_msg("over-limit-final", "assistant", large_final, stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - before_children = {child.pid for child in multiprocessing.active_children()} - before_threads = { - thread.ident - for thread in threading.enumerate() - if thread.name.startswith("tendwire-turn") - } - - content = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=60, - ) - - assert content["assistant_final_text"] == large_final - assert content["complete"] is True - assert {child.pid for child in multiprocessing.active_children()} == before_children - assert { - thread.ident - for thread in threading.enumerate() - if thread.name.startswith("tendwire-turn") - } == before_threads - - -def test_isolated_omp_large_final_unchanged_fast_path_and_appended_turn( - tmp_path: Path, - monkeypatch, -) -> None: - large_final = "canonical-" + ("x" * (6 * 1024 * 1024)) - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("six-mib-user", "user", "first private prompt", attribution="user"), - _omp_msg("six-mib-final", "assistant", large_final, stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - byte_reads: list[int] = [] - monkeypatch.setattr(herdr_turns, "_OMP_ISOLATED_READ_OBSERVER", byte_reads.append) - - first = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=10, - ) - assert first["assistant_final_text"] == large_final - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._serialize_omp_state( - herdr_turns._OMP_SESSION_CACHE[cache_key] - ) - checkpoint_json = json.dumps(checkpoint, separators=(",", ":")) - assert len(checkpoint_json.encode("utf-8")) < 1024 - assert "canonical-" not in checkpoint_json - assert set(checkpoint) == { - "offset", - "observed_size", - "file_id", - "mtime_ns", - "ctime_ns", - "replay_offset", - "turn_open", - "project_root", - } - assert checkpoint["turn_open"] is False - - original_get_context = herdr_turns.multiprocessing.get_context - - def fail_if_spawned(*_args, **_kwargs): - raise AssertionError("unchanged OMP poll spawned a child") - - monkeypatch.setattr(herdr_turns.multiprocessing, "get_context", fail_if_spawned) - second = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=1, - ) - monkeypatch.setattr(herdr_turns.multiprocessing, "get_context", original_get_context) - assert second is herdr_turns._UNCHANGED_TURN - assert byte_reads[-1] == 0 - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert ( - herdr_turns._serialize_omp_state(herdr_turns._OMP_SESSION_CACHE[cache_key]) - == checkpoint - ) - - appended = "\n".join( - [ - "", - json.dumps( - _omp_msg("next-user", "user", "new appended prompt", attribution="user"), - separators=(",", ":"), - ), - json.dumps( - _omp_msg("next-final", "assistant", "new answer", stop="stop"), - separators=(",", ":"), - ), - ] - ) - with open(path, "a", encoding="utf-8") as handle: - handle.write(appended) - third = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=5, - ) - assert third["source_turn_id"] == "next-user" - assert third["user_text"] == "new appended prompt" - assert third["assistant_final_text"] == "new answer" - assert large_final not in json.dumps(third) - assert byte_reads[-1] == len(appended.encode("utf-8")) - - -def test_isolated_omp_cache_validates_identity_and_retains_good_state_on_failures( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [_omp_msg("original-user", "user", "original prompt", attribution="user")], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - original_child = herdr_turns._file_turn_child - - first = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - - def cached_state(): - with herdr_turns._OMP_SESSION_CACHE_LOCK: - return herdr_turns._serialize_omp_state(herdr_turns._OMP_SESSION_CACHE[cache_key]) - - first_state = cached_state() - assert first["source_turn_id"] == "original-user" - - for child, expected_error, timeout_seconds in ( - (_blocked_isolated_turn_child, herdr_turns._TurnReadTimeout, 0.1), - (_failed_isolated_turn_child, herdr_turns._TurnReadFailed, 5), - (_invalid_isolated_turn_child, herdr_turns._TurnReadFailed, 5), - ): - with open(path, "a", encoding="utf-8") as handle: - handle.write( - "\n" - + json.dumps( - {"type": "ignored", "failure_attempt": child.__name__}, - separators=(",", ":"), - ) - ) - monkeypatch.setattr(herdr_turns, "_file_turn_child", child) - with pytest.raises(expected_error): - herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=timeout_seconds, - ) - assert cached_state() == first_state - monkeypatch.setattr(herdr_turns, "_file_turn_child", original_child) - - replacement = path.with_name("replacement.jsonl") - replacement.write_text( - json.dumps(_omp_msg("replacement-user", "user", "replacement prompt", attribution="user")), - encoding="utf-8", - ) - replacement.replace(path) - replaced = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - replaced_state = cached_state() - assert replaced["source_turn_id"] == "replacement-user" - assert replaced_state["file_id"] != first_state["file_id"] - - replaced_inode = path.stat().st_ino - path.write_text( - json.dumps(_omp_msg("truncated-user", "user", "short", attribution="user")), - encoding="utf-8", - ) - assert path.stat().st_ino == replaced_inode - truncated = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - assert truncated["source_turn_id"] == "truncated-user" - - final_line = json.dumps(_omp_msg("final-answer", "assistant", "published", stop="stop")) - with open(path, "a", encoding="utf-8") as handle: - handle.write("\n" + final_line) - final = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - final_state = cached_state() - assert final["source_turn_id"] == "truncated-user" - assert final["assistant_final_text"] == "published" - assert final_state["offset"] == path.stat().st_size - - -def test_omp_same_inode_same_size_rewrite_forces_cold_rescan( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("user-old", "user", "prompt-old", attribution="user"), - _omp_msg("final-old", "assistant", "answer-old", stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - first = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - original_stat = path.stat() - original_size = original_stat.st_size - assert first["assistant_final_text"] == "answer-old" - - rewritten = "\n".join( - json.dumps(line) - for line in ( - _omp_msg("user-new", "user", "prompt-new", attribution="user"), - _omp_msg("final-new", "assistant", "answer-new", stop="stop"), - ) - ) - assert len(rewritten.encode("utf-8")) == original_size - path.write_text(rewritten, encoding="utf-8") - rewritten_stat = path.stat() - assert rewritten_stat.st_ino == original_stat.st_ino - assert rewritten_stat.st_size == original_size - assert ( - rewritten_stat.st_mtime_ns != original_stat.st_mtime_ns - or rewritten_stat.st_ctime_ns != original_stat.st_ctime_ns - ) - - second = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - - assert second["source_turn_id"] == "user-new" - assert second["user_text"] == "prompt-new" - assert second["assistant_final_text"] == "answer-new" - - -def test_omp_every_assistant_after_final_is_ignored_until_next_user( - tmp_path: Path, - monkeypatch, -) -> None: - irrelevant = { - "type": "message", - "id": "metadata-only", - "message": { - "role": "assistant", - "stopReason": "toolUse", - "content": [{"type": "metadata", "value": "private bookkeeping"}], - }, - } - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("stable-user", "user", "stable prompt", attribution="user"), - _omp_msg("stable-final", "assistant", "stable answer", stop="stop"), - irrelevant, - _omp_msg( - "renderable-progress", - "assistant", - "must not reopen the completed turn", - stop="toolUse", - ), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - - assert content["assistant_final_text"] == "stable answer" - assert content["complete"] is True - assert content["assistant_stream_text"] is None - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._OMP_SESSION_CACHE[cache_key] - assert checkpoint.turn_open is False - assert checkpoint.replay_offset == checkpoint.offset - - -def test_omp_appended_assistant_after_committed_final_advances_idle_checkpoint( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("done-user", "user", "finish once", attribution="user"), - _omp_msg("done-final", "assistant", "finished once", stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - first = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - assert first["assistant_final_text"] == "finished once" - - appended = "\n" + json.dumps( - _omp_msg( - "late-progress", - "assistant", - "late renderable progress", - stop="toolUse", - ), - separators=(",", ":"), - ) - with open(path, "a", encoding="utf-8") as handle: - handle.write(appended) - second = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - - assert second is None - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._OMP_SESSION_CACHE[cache_key] - assert checkpoint.turn_open is False - assert checkpoint.offset == path.stat().st_size - assert checkpoint.replay_offset == checkpoint.offset - assert ( - herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=1, - ) - is herdr_turns._UNCHANGED_TURN - ) - - -def test_omp_retries_atomic_replacement_that_occurs_during_read( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("old-user", "user", "old prompt", attribution="user"), - _omp_msg("old-final", "assistant", "stale answer", stop="stop"), - ], - ) - replacement = path.with_name("during-read-replacement.jsonl") - replacement.write_text( - "\n".join( - json.dumps(line, separators=(",", ":")) - for line in ( - _omp_msg("new-user", "user", "new prompt", attribution="user"), - _omp_msg("new-final", "assistant", "current answer", stop="stop"), - ) - ), - encoding="utf-8", - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - original_open = open - replaced = False - - class ReplacingReader: - def __init__(self, handle): - self.handle = handle - - def __enter__(self): - self.handle.__enter__() - return self - - def __exit__(self, *args): - return self.handle.__exit__(*args) - - def fileno(self): - return self.handle.fileno() - - def seek(self, *args): - return self.handle.seek(*args) - - def read(self, *args): - nonlocal replaced - if not replaced: - replaced = True - replacement.replace(path) - return self.handle.read(*args) - - def racing_open(file, mode="r", *args, **kwargs): - handle = original_open(file, mode, *args, **kwargs) - if Path(file) == path and mode == "rb" and not replaced: - return ReplacingReader(handle) - return handle - - monkeypatch.setattr("builtins.open", racing_open) - content = herdr_turns._read_omp_session_turn(str(path)) - - assert replaced is True - assert content["source_turn_id"] == "new-user" - assert content["assistant_final_text"] == "current answer" - assert "stale answer" not in json.dumps(content) - - -def test_omp_concurrent_cache_loser_cannot_overwrite_accepted_checkpoint() -> None: - cache_key = "concurrent-publication" - prior = herdr_turns._OmpSessionState( - offset=100, - observed_size=100, - file_id=(7, 11), - ) - accepted = herdr_turns._OmpSessionState( - offset=300, - observed_size=300, - file_id=(7, 11), - ) - losing = herdr_turns._OmpSessionState( - offset=300, - observed_size=300, - file_id=(7, 11), - turn_open=True, - ) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - herdr_turns._OMP_SESSION_CACHE[cache_key] = accepted - - returned = herdr_turns._publish_omp_cache_state( - cache_key, - herdr_turns._serialize_omp_state(prior), - losing, - {"assistant_final_text": "losing duplicate"}, - ) - - assert returned is None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert herdr_turns._OMP_SESSION_CACHE[cache_key] is accepted - - -def test_omp_cache_lru_capacity_moves_hits_and_evicts_oldest(monkeypatch) -> None: - monkeypatch.setattr(herdr_turns, "_OMP_SESSION_CACHE_CAPACITY", 3) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - for index in range(3): - state = herdr_turns._OmpSessionState( - offset=index + 1, - observed_size=index + 1, - file_id=(1, index + 1), - ) - herdr_turns._publish_omp_cache_state( - f"key-{index}", - None, - state, - None, - ) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert herdr_turns._omp_cache_get_locked("key-0") is not None - - newest = herdr_turns._OmpSessionState( - offset=4, - observed_size=4, - file_id=(1, 4), - ) - herdr_turns._publish_omp_cache_state( - "key-3", - None, - newest, - None, - ) - - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert list(herdr_turns._OMP_SESSION_CACHE) == ["key-2", "key-0", "key-3"] - assert len(herdr_turns._OMP_SESSION_CACHE) == 3 - - -def test_omp_sixty_four_large_completed_sessions_keep_only_bounded_coordinates( - tmp_path: Path, - monkeypatch, -) -> None: - root = tmp_path / "omp-sessions" - session_dir = root / "-many" - session_dir.mkdir(parents=True) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - - for index in range(64): - large_final = f"large-final-{index}-" + ("z" * (128 * 1024)) - path = session_dir / f"{index:02d}.jsonl" - path.write_text( - "\n".join( - json.dumps(line, separators=(",", ":")) - for line in ( - _omp_msg(f"user-{index}", "user", f"prompt-{index}", attribution="user"), - _omp_msg(f"final-{index}", "assistant", large_final, stop="stop"), - ) - ), - encoding="utf-8", - ) - content = herdr_turns._read_omp_session_turn(str(path)) - assert content["assistant_final_text"] == large_final - - with herdr_turns._OMP_SESSION_CACHE_LOCK: - serialized = json.dumps( - { - key: herdr_turns._serialize_omp_state(state) - for key, state in herdr_turns._OMP_SESSION_CACHE.items() - }, - separators=(",", ":"), - ) - assert len(herdr_turns._OMP_SESSION_CACHE) == 64 - assert herdr_turns._omp_cache_weight_locked() <= herdr_turns._OMP_SESSION_CACHE_MAX_BYTES - assert all(not state.turn_open for state in herdr_turns._OMP_SESSION_CACHE.values()) - assert "large-final-" not in serialized - assert "z" * 1024 not in serialized - - -def test_omp_cache_enforces_serialized_byte_bound(monkeypatch) -> None: - monkeypatch.setattr(herdr_turns, "_OMP_SESSION_CACHE_CAPACITY", 64) - monkeypatch.setattr(herdr_turns, "_OMP_SESSION_CACHE_MAX_BYTES", 600) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - for index in range(20): - herdr_turns._omp_cache_store_locked( - f"long-cache-key-{index}-" + ("k" * 40), - herdr_turns._OmpSessionState( - offset=index, - observed_size=index, - file_id=(1, index), - ), - ) - assert len(herdr_turns._OMP_SESSION_CACHE) < 20 - assert herdr_turns._omp_cache_weight_locked() <= 600 - - -def test_omp_cache_prunes_disappeared_bindings_and_keeps_live_binding( - tmp_path: Path, - monkeypatch, -) -> None: - root, live_path = _write_omp_session( - tmp_path, - [_omp_msg("live", "user", "live", attribution="user")], - ) - stale_path = live_path.with_name("stale.jsonl") - stale_path.write_text( - json.dumps(_omp_msg("stale", "user", "stale", attribution="user")), - encoding="utf-8", - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - live_key = herdr_turns._omp_cache_key(str(live_path)) - stale_key = herdr_turns._omp_cache_key(str(stale_path)) - assert live_key is not None - assert stale_key is not None - live_state = herdr_turns._OmpSessionState( - offset=1, - observed_size=1, - file_id=(1, 1), - ) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._omp_cache_store_locked(live_key, live_state) - herdr_turns._omp_cache_store_locked( - stale_key, - herdr_turns._OmpSessionState(offset=1, file_id=(1, 2)), - ) - live_binding = WorkerBinding( - host_id="host", - worker_id="worker", - worker_fingerprint="worker-fingerprint", - backend="herdr", - target_kind="agent_id", - target_value="agent", - turn_target_kind="omp_session_path", - turn_target_value=str(live_path), - sendable=True, - observed_at="2026-07-12T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private", - ) - - herdr_turns._prune_omp_cache_for_bindings([live_binding]) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert list(herdr_turns._OMP_SESSION_CACHE) == [live_key] - retained_prior = herdr_turns._serialize_omp_state( - herdr_turns._OMP_SESSION_CACHE[live_key] - ) - retained_generation = herdr_turns._omp_cache_binding_generation_locked(live_key) - - live_path.unlink() - herdr_turns._prune_omp_cache_for_bindings([live_binding]) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert list(herdr_turns._OMP_SESSION_CACHE) == [live_key] - assert herdr_turns._omp_cache_binding_generation_locked(live_key) == retained_generation - - herdr_turns._prune_omp_cache_for_bindings([]) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert not herdr_turns._OMP_SESSION_CACHE - - herdr_turns._prune_omp_cache_for_bindings([live_binding]) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert ( - herdr_turns._omp_cache_binding_generation_locked(live_key) - != retained_generation - ) - - resurrected = herdr_turns._OmpSessionState( - offset=2, - observed_size=2, - file_id=(1, 1), - ) - returned = herdr_turns._publish_omp_cache_state( - live_key, - retained_prior, - resurrected, - None, - retained_generation, - ) - assert returned is None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert live_key not in herdr_turns._OMP_SESSION_CACHE - - -def test_omp_tool_progress_uses_only_allowlisted_structured_summaries(tmp_path, monkeypatch) -> None: - project = tmp_path / "project" - project.mkdir() - _mark_git_repository(project) - (project / "README.md").write_text("public", encoding="utf-8") - private_key_path = "/home/alice/.ssh/id_ed25519" - herdr_socket_path = "/run/user/1000/herdr/private.sock" - credential_url = "https://alice:password@internal.example/private" - provider_key = "sk-" + "proj-" + "PUBLICSAFETY1234567890" - raw_tool_id = "toolu_PUBLICSAFETYTOOL01" - tool_message = { - "type": "message", - "id": "a1", - "message": { - "role": "assistant", - "stopReason": "toolUse", - "content": [ - { - "type": "toolCall", - "id": raw_tool_id, - "name": "bash", - "arguments": { - "command": f"cat {private_key_path}; connect {herdr_socket_path} {credential_url}", - "env": {"TOKEN": provider_key}, - }, - }, - { - "type": "toolCall", - "id": "c2", - "toolName": "bash", - "input": {"command": "python -m pytest -q tests/test_turns.py"}, - }, - { - "type": "toolCall", - "id": "c3", - "tool": "read", - "args": {"path": "README.md", "stdout": private_key_path}, - }, - { - "type": "toolCall", - "id": "c4", - "name": raw_tool_id, - "arguments": { - "nested": [private_key_path, herdr_socket_path, provider_key], - "url": credential_url, - }, - }, - ], - }, - } - root, path = _write_omp_session( - tmp_path, - [ - {"type": "session", "id": "private-session", "cwd": str(project)}, - _omp_msg("u1", "user", "show safe tool progress", attribution="user"), - tool_message, - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(path)) - stream = content["assistant_stream_text"] - - assert stream.split("\n\n") == [ - "step 1 · run command", - "step 2 · test: pytest", - "step 3 · read: README.md", - "step 4 · tool", - ] - for private_value in ( - private_key_path, - herdr_socket_path, - credential_url, - provider_key, - raw_tool_id, - ): - assert private_value not in stream - - -def test_omp_shell_progress_uses_a_small_constant_allowlist() -> None: - cases = [ - ("git status --short", "step 1 · git status"), - ("pytest -q tests/test_turns.py", "step 1 · test: pytest"), - ("uv run pytest tests/test_turns.py", "step 1 · test: pytest"), - ("cargo test --workspace", "step 1 · test: cargo"), - ("npm run build", "step 1 · build: npm"), - ("make all", "step 1 · build: make"), - ("git checkout -b private-branch", "step 1 · run command"), - ("echo arbitrary private text", "step 1 · run command"), - ] - - for command, expected in cases: - item = {"name": "bash", "arguments": {"command": command}} - assert herdr_turns._omp_tool_snippet(item, 1) == expected - - -def test_omp_file_progress_requires_repository_root_proof(tmp_path: Path) -> None: - project = tmp_path / "repo" - docs = project / "docs" - docs.mkdir(parents=True) - _mark_git_repository(project) - readme = project / "README.md" - guide = docs / "guide.md" - outside = tmp_path / "outside.txt" - readme.write_text("readme", encoding="utf-8") - guide.write_text("guide", encoding="utf-8") - outside.write_text("private", encoding="utf-8") - escape = project / "escape" - escape.symlink_to(outside) - - def snippet(path_value: str, root: Path | None = project) -> str: - return herdr_turns._omp_tool_snippet( - {"name": "read", "arguments": {"path": path_value}}, - 1, - root, - ) - - assert snippet("README.md") == "step 1 · read: README.md" - assert snippet(str(guide)) == "step 1 · read: docs/guide.md" - assert snippet("README.md", None) == "step 1 · read file" - assert snippet(str(outside)) == "step 1 · read file" - assert snippet("../outside.txt") == "step 1 · read file" - assert snippet(str(escape)) == "step 1 · read file" - assert snippet("~/.ssh/id_ed25519") == "step 1 · read file" - assert snippet("docs/../README.md") == "step 1 · read file" - assert snippet(".env") == "step 1 · read file" - assert snippet(".git/config") == "step 1 · read file" - assert snippet("secrets/key.txt") == "step 1 · read file" - assert snippet("credentials.json") == "step 1 · read file" - - -def test_omp_file_progress_rejects_unproven_session_cwds(tmp_path: Path, monkeypatch) -> None: - home = tmp_path / "home" / "alice" - home.mkdir(parents=True) - (home / ".bashrc").write_text("operator shell settings", encoding="utf-8") - notes = tmp_path / "operator-work" - notes.mkdir() - (notes / "operator-notes.txt").write_text("private operator notes", encoding="utf-8") - - cases = ( - ("filesystem-root", Path("/"), "/etc/passwd"), - ("home", home, ".bashrc"), - ("notes", notes, "operator-notes.txt"), - ) - for label, cwd, path_value in cases: - root, session_path = _write_omp_session( - tmp_path / label, - [ - {"type": "session", "id": label, "cwd": str(cwd)}, - _omp_msg(f"{label}-user", "user", "inspect a file", attribution="user"), - _omp_read_msg(f"{label}-assistant", path_value), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(session_path)) - - assert content is not None - assert content["assistant_stream_text"] == "step 1 · read file" - - -def test_omp_file_progress_finds_repository_above_session_cwd( - tmp_path: Path, - monkeypatch, -) -> None: - project = tmp_path / "repo-with-subdir" - docs = project / "docs" - docs.mkdir(parents=True) - _mark_git_repository(project) - guide = docs / "guide.md" - guide.write_text("public", encoding="utf-8") - root, session_path = _write_omp_session( - tmp_path / "subdir-session", - [ - {"type": "session", "id": "subdir", "cwd": str(docs)}, - _omp_msg("subdir-user", "user", "inspect the guide", attribution="user"), - _omp_read_msg("subdir-assistant", str(guide)), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(session_path)) - - assert content is not None - assert content["assistant_stream_text"] == "step 1 · read: docs/guide.md" - - -def test_omp_file_progress_accepts_worktree_gitdir_file(tmp_path: Path, monkeypatch) -> None: - checkout = tmp_path / "checkout" - checkout.mkdir() - (checkout / "README.md").write_text("public", encoding="utf-8") - worktree_git_dir = tmp_path / "main" / ".git" / "worktrees" / "checkout" - _write_valid_git_head(worktree_git_dir) - (checkout / ".git").write_text( - f"gitdir: {worktree_git_dir}\n", - encoding="utf-8", - ) - root, session_path = _write_omp_session( - tmp_path / "worktree-session", - [ - {"type": "session", "id": "worktree", "cwd": str(checkout)}, - _omp_msg("worktree-user", "user", "inspect the readme", attribution="user"), - _omp_read_msg("worktree-assistant", "README.md"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(session_path)) - - assert content is not None - assert content["assistant_stream_text"] == "step 1 · read: README.md" - - -def test_omp_file_progress_rejects_invalid_worktree_gitdir_file( - tmp_path: Path, - monkeypatch, -) -> None: - checkout = tmp_path / "invalid-checkout" - checkout.mkdir() - (checkout / "README.md").write_text("public", encoding="utf-8") - (checkout / ".git").write_text("gitdir: ../missing-git-dir\n", encoding="utf-8") - root, session_path = _write_omp_session( - tmp_path / "invalid-worktree-session", - [ - {"type": "session", "id": "invalid-worktree", "cwd": str(checkout)}, - _omp_msg("invalid-user", "user", "inspect the readme", attribution="user"), - _omp_read_msg("invalid-assistant", "README.md"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(session_path)) - - assert content is not None - assert content["assistant_stream_text"] == "step 1 · read file" - - -def _prepare_pane_turn_store( - tmp_path: Path, - *, - herdr_bin: str = "herdr", -) -> tuple[Config, Any]: - db_path = tmp_path / "turns.db" - config = Config( - host_id="turn-host", - db_path=db_path, - herdr_bin=herdr_bin, - herdr_timeout_seconds=10, - ) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "codex", "status": "active", "space_id": "space-1"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="private-agent", - turn_target_kind="pane_id", - turn_target_value="private-pane", - sendable=True, - observed_at="2026-07-11T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-binding", - ) - ], - ) - return config, snapshot - - -def _pane_turn_payload( - *, - user_text: str | None, - final_text: str | None, - stream_text: str | None = None, - complete: bool = True, - source_turn_id: str = "source-turn-long", -) -> dict[str, Any]: - return { - "result": { - "turn": { - "available": True, - "user_text": user_text, - "assistant_final_text": final_text, - "assistant_stream_text": stream_text, - "complete": complete, - "has_open_turn": not complete, - "source_turn_id": source_turn_id, - } - } - } - - -def _current_content_turn( - config: Config, - snapshot: Any, -) -> tuple[dict[str, Any], str]: - payload = turns_payload_from_store( - config.db_path, - config.host_id, - snapshot=snapshot, - schema_version=2, - ) - turn = next( - item - for item in payload["turns"] - if (item.get("content") or {}).get("content_revision") - ) - return turn, str(turn["content"]["content_revision"]) - - -def _reconstruct_turn_field( - config: Config, - *, - turn_id: str, - revision: str, - field: str, -) -> str: - cursor: str | None = None - pages: list[dict[str, Any]] = [] - while True: - page = store_sqlite.get_turn_content( - config.db_path, - config.host_id, - turn_id=turn_id, - content_revision=revision, - field=field, - cursor=cursor, - schema_version=1, - ) - assert page["availability"] == "complete" - assert page["index"] == len(pages) - pages.append(page) - cursor = page["next_cursor"] - if cursor is None: - break - assert all(page["count"] == len(pages) for page in pages) - assert all( - len(str(page["text"]).encode("utf-8")) <= TURN_CONTENT_PAGE_MAX_UTF8_BYTES - for page in pages - ) - return "".join(str(page["text"]) for page in pages) - - -@pytest.mark.parametrize("content_size", [20_000, 1024 * 1024 + 257]) -def test_wrapper_adapter_round_trips_unbounded_authoritative_content( - tmp_path: Path, - monkeypatch, - content_size: int, -) -> None: - adapter = tmp_path / "long_turn_adapter.py" - adapter.write_text( - r"""#!/usr/bin/env python3 -import json -import os - -size = int(os.environ["TENDWIRE_TEST_TURN_SIZE"]) -prompt = " Prompt fi\n" + ("p" * size) + "\u200b\x00\n" -final = "\n# Final\n" + ("f" * size) + "\u200b\x00 " -print(json.dumps({"result": {"turn": { - "available": True, - "user_text": prompt, - "assistant_final_text": final, - "assistant_stream_text": None, - "complete": True, - "has_open_turn": False, - "source_turn_id": "source-turn-long", -}}})) -""", - encoding="utf-8", - ) - adapter.chmod(0o700) - monkeypatch.setenv("TENDWIRE_TEST_TURN_SIZE", str(content_size)) - config, snapshot = _prepare_pane_turn_store(tmp_path, herdr_bin=str(adapter)) - - result = refresh_structured_turn_content(config) - turn, revision = _current_content_turn(config, snapshot) - expected_prompt = sanitize_canonical_turn_text( - " Prompt fi\n" + ("p" * content_size) + "\u200b\x00\n" - ) - expected_final = sanitize_canonical_turn_text( - "\n# Final\n" + ("f" * content_size) + "\u200b\x00 " - ) - serialized_source = json.dumps( - _pane_turn_payload( - user_text=" Prompt fi\n" + ("p" * content_size) + "\u200b\x00\n", - final_text="\n# Final\n" + ("f" * content_size) + "\u200b\x00 ", - ) - ).encode("utf-8") - assert len(serialized_source) > content_size * 2 - - assert result == {"ok": True, "status": "ok", "updated": 1, "attempted": 1} - assert expected_prompt is not None - assert expected_final is not None - assert _reconstruct_turn_field( - config, - turn_id=turn["id"], - revision=revision, - field="user_text", - ) == expected_prompt - assert _reconstruct_turn_field( - config, - turn_id=turn["id"], - revision=revision, - field="assistant_final_text", - ) == expected_final - - -def test_refresh_keeps_only_a_rolling_bounded_stream( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot = _prepare_pane_turn_store(tmp_path) - stream = "".join(str(index % 10) for index in range(TURN_STREAM_TEXT_MAX_CHARS + 137)) - monkeypatch.setattr( - herdr_turns.subprocess, - "run", - _run_returning( - _pane_turn_payload( - user_text="stream this turn", - final_text=None, - stream_text=stream, - complete=False, - ) - ), - ) - - assert refresh_structured_turn_content(config)["updated"] == 1 - turn, _revision = _current_content_turn(config, snapshot) - - assert turn["assistant_stream_text"] == stream[-TURN_STREAM_TEXT_MAX_CHARS:] - assert len(turn["assistant_stream_text"]) == TURN_STREAM_TEXT_MAX_CHARS - - -def test_empty_later_observation_does_not_erase_authoritative_final( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot = _prepare_pane_turn_store(tmp_path) - final_text = "authoritative final" - observations = iter( - [ - _pane_turn_payload(user_text="prompt", final_text=final_text), - _pane_turn_payload(user_text="", final_text=""), - ] - ) - monkeypatch.setattr( - herdr_turns.subprocess, - "run", - lambda args, **kwargs: subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps(next(observations)), - stderr="", - ), - ) - - assert refresh_structured_turn_content(config)["updated"] == 1 - first_turn, first_revision = _current_content_turn(config, snapshot) - second_result = refresh_structured_turn_content(config) - second_turn, second_revision = _current_content_turn(config, snapshot) - - assert second_result["updated"] == 0 - assert second_turn["id"] == first_turn["id"] - assert second_revision == first_revision - assert _reconstruct_turn_field( - config, - turn_id=second_turn["id"], - revision=second_revision, - field="assistant_final_text", - ) == final_text - - -def test_identical_source_turn_observation_is_a_revision_noop( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot = _prepare_pane_turn_store(tmp_path) - payload = _pane_turn_payload(user_text="same prompt", final_text="same final") - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - - assert refresh_structured_turn_content(config)["updated"] == 1 - first_turn, first_revision = _current_content_turn(config, snapshot) - with sqlite3.connect(config.db_path) as conn: - first_count = conn.execute( - "SELECT COUNT(*) FROM turn_content_revisions WHERE host_id = ? AND turn_id = ?", - (config.host_id, first_turn["id"]), - ).fetchone()[0] - - assert refresh_structured_turn_content(config)["updated"] == 0 - second_turn, second_revision = _current_content_turn(config, snapshot) - with sqlite3.connect(config.db_path) as conn: - second_count = conn.execute( - "SELECT COUNT(*) FROM turn_content_revisions WHERE host_id = ? AND turn_id = ?", - (config.host_id, second_turn["id"]), - ).fetchone()[0] - - assert second_turn["id"] == first_turn["id"] - assert second_revision == first_revision - assert second_count == first_count - - -def test_complete_reobservation_recovers_known_incomplete_source_turn( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot = _prepare_pane_turn_store(tmp_path) - fragment = ("legacy fragment " * 900)[:11_988] + "\n[truncated]" - complete_final = fragment.removesuffix("\n[truncated]") + " recovered authoritative suffix" - first_payload = _pane_turn_payload(user_text="recover this", final_text=fragment) - complete_payload = _pane_turn_payload(user_text="recover this", final_text=complete_final) - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(first_payload)) - - assert refresh_structured_turn_content(config)["updated"] == 1 - turn, incomplete_revision = _current_content_turn(config, snapshot) - with sqlite3.connect(config.db_path) as conn: - conn.execute( - """ - UPDATE turn_content_revisions - SET final_state = 'known_incomplete' - WHERE host_id = ? AND turn_id = ? AND content_revision = ? - """, - (config.host_id, turn["id"], incomplete_revision), - ) - conn.commit() - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(complete_payload)) - assert refresh_structured_turn_content(config)["updated"] == 1 - recovered_turn, recovered_revision = _current_content_turn(config, snapshot) - with sqlite3.connect(config.db_path) as conn: - revisions = conn.execute( - """ - SELECT content_revision, final_state, is_current - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? - ORDER BY created_at, content_revision - """, - (config.host_id, turn["id"]), - ).fetchall() - - assert recovered_turn["id"] == turn["id"] - assert recovered_revision != incomplete_revision - assert (incomplete_revision, "known_incomplete", 0) in revisions - assert (recovered_revision, "complete", 1) in revisions - assert _reconstruct_turn_field( - config, - turn_id=recovered_turn["id"], - revision=recovered_revision, - field="assistant_final_text", - ) == complete_final - - -def test_completed_pane_refresh_uses_authoritative_binding_hint( - tmp_path: Path, - monkeypatch: Any, -) -> None: - config = Config(host_id="completion-route", db_path=tmp_path / "route.db") - binding = WorkerBinding( - host_id=config.host_id, - worker_id="worker-1", - worker_fingerprint="worker-fingerprint", - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="codex_session_id", - turn_target_value="session-private", - sendable=True, - observed_at="2026-07-23T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private", - ) - monkeypatch.setattr( - herdr_turns, - "list_worker_bindings", - lambda *_args, **_kwargs: [binding], - ) - refreshed: list[WorkerBinding] = [] - monkeypatch.setattr( - herdr_turns, - "_refresh_turn_binding", - lambda _config, current, **_kwargs: ( - refreshed.append((current, _kwargs.get("pane_target_override"))) - or herdr_turns.TurnRefreshResult("updated", 1) - ), - ) - monkeypatch.setattr( - herdr_turns, - "latest_turn_id_for_worker", - lambda *_args, **_kwargs: "public-turn-1", - ) - - result = herdr_turns.refresh_completed_pane_turn( - config, - "w123456789abcde:pA", - terminal_id="different-terminal", - binding_private_fingerprint="binding-private", - ) - - assert refreshed == [(binding, "w123456789abcde:pA")] - assert result == herdr_turns.CompletedPaneTurnRefreshResult( - "updated", - worker_id="worker-1", - refreshed_turn_id="public-turn-1", - ) - - -def test_completed_pane_refresh_ignores_stale_session_generation_for_content_read( - tmp_path: Path, - monkeypatch: Any, -) -> None: - db_path = tmp_path / "generation-mismatch.db" - config = Config( - host_id="generation-mismatch", - db_path=db_path, - herdr_bin="turn-adapter", - herdr_timeout_seconds=2, - ) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-generation-a", - "name": "claude", - "status": "idle", - "space_id": "space-1", - } - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - stale_binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value="terminal-stable", - turn_target_kind="codex_session_id", - turn_target_value="session-generation-a", - sendable=True, - observed_at="2026-07-23T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-generation-a", - ) - upsert_worker_bindings(db_path, [stale_binding]) - adapter_calls: list[list[str]] = [] - payload = _pane_turn_payload( - user_text="reply with purple elephant 42", - final_text="purple elephant 42", - source_turn_id="generation-b-turn", - ) - - def fake_run(args: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]: - adapter_calls.append(args) - return subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps(payload), - stderr="", - ) - - monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) - monkeypatch.setattr( - herdr_turns, - "_read_file_turn_isolated", - lambda *_args, **_kwargs: pytest.fail( - "completion must not read the stale session generation" - ), - ) - - result = herdr_turns.refresh_completed_pane_turn( - config, - "pane-generation-b", - terminal_id="terminal-stable", - binding_private_fingerprint="binding-generation-a", - ) - payload_from_store = turns_payload_from_store( - db_path, - config.host_id, - snapshot=snapshot, - ) - - assert result.status == "updated" - assert result.worker_id == worker.id - assert adapter_calls == [ - [ - "turn-adapter", - "pane", - "turn", - "pane-generation-b", - "--last", - "--format", - "json", - ] - ] - captured = next( - turn - for turn in payload_from_store["turns"] - if turn.get("assistant_final_text") - ) - assert captured["status"] == "idle" - assert captured["assistant_final_text"] == "purple elephant 42" diff --git a/tests/test_turn_ingestion.py b/tests/test_turn_ingestion.py deleted file mode 100644 index 04213c1..0000000 --- a/tests/test_turn_ingestion.py +++ /dev/null @@ -1,2371 +0,0 @@ -"""Deterministic concurrency tests for daemon-owned turn ingestion.""" - -from __future__ import annotations - -import json -import multiprocessing -import os -import sqlite3 -import threading -import time -from pathlib import Path -from typing import Any - -from tendwire.backends import herdr_turns -from tendwire.backends.herdr_turns import ( - TurnIngestionScheduler, - TurnRefreshResult, -) -from tendwire.config import Config -from tendwire.core.models import Snapshot, Worker, WorkerBinding -from tendwire.core.turns import PendingObservation, PendingObservedChoice -from tendwire.core.projector import project_from_raw -from tendwire.store.sqlite import ( - apply_backend_pending_observation, - init_store, - merge_turn_content, - pending_payload_from_store, - save_snapshot, - turns_payload_from_store, - upsert_worker_bindings, -) - - -def _blocked_codex_child(channel) -> None: - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._CODEX_STATE_IPC_MAX_BYTES, - ) - threading.Event().wait(30) - - -def _wrong_source_codex_child(channel) -> None: - try: - request = json.loads( - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._CODEX_STATE_IPC_MAX_BYTES, - ).decode("utf-8") - ) - response = { - "protocol": 1, - "nonce": request["nonce"], - "disposition": "ok", - "content": { - "user_text": "must not publish", - "assistant_final_text": "must not publish", - "complete": True, - "has_open_turn": False, - "source_turn_id": "wrong-source", - }, - "parser_state": {"source": "omp", "state": None}, - "bytes_read": 0, - } - herdr_turns._blocking_send_frame( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - ) - finally: - channel.close() - - -def _oversized_direct_omp_child(channel) -> None: - try: - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._OMP_REQUEST_MAX_BYTES, - ) - herdr_turns._blocking_send_frame( - channel, - b"x" * (herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES + 1), - ) - finally: - channel.close() - - -def _large_direct_codex_child(channel) -> None: - try: - request = json.loads( - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._CODEX_STATE_IPC_MAX_BYTES, - ).decode("utf-8") - ) - final = "codex-frame-" + ( - "c" * (herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES + 1024) - ) - response = { - "protocol": 1, - "nonce": request["nonce"], - "disposition": "ok", - "content": { - "assistant_final_text": final, - "complete": True, - "has_open_turn": False, - "source_turn_id": "codex-large-frame", - }, - "parser_state": request["parser_state"], - "bytes_read": 0, - } - herdr_turns._blocking_send_frame( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - ) - finally: - channel.close() - - -def _wait_until(predicate, timeout: float = 2.0) -> None: - deadline = time.monotonic() + timeout - while not predicate(): - remaining = deadline - time.monotonic() - if remaining <= 0: - raise AssertionError("condition did not become true") - threading.Event().wait(min(0.01, remaining)) - - -def _binding(config: Config, worker: Any, ordinal: int, *, target: str | None = None) -> WorkerBinding: - return WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value=f"agent-{ordinal}", - turn_target_kind="pane_id", - turn_target_value=target or f"pane-{ordinal}", - sendable=True, - observed_at="2026-07-12T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint=f"private-{ordinal}", - ) - - -def _scheduler_store(tmp_path: Path, count: int) -> tuple[Config, Any, list[WorkerBinding]]: - config = Config( - host_id="ingestion-host", - db_path=tmp_path / "ingestion.db", - herdr_timeout_seconds=0.5, - turn_refresh_interval_seconds=100.0, - turn_refresh_workers=4, - ) - snapshot = project_from_raw( - config, - workers=[ - {"id": f"worker-{index}", "name": f"worker {index}", "status": "active"} - for index in range(count) - ], - ) - init_store(config.db_path) - save_snapshot(config.db_path, snapshot) - bindings = [_binding(config, worker, index) for index, worker in enumerate(snapshot.workers)] - upsert_worker_bindings(config.db_path, bindings) - return config, snapshot, bindings - - -def test_background_scheduler_discovers_and_clears_pending_without_turn_list( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 1) - current = { - "observation": PendingObservation( - "open_prompt", - question="Background choice?", - pending_kind="question", - choices=( - PendingObservedChoice( - "choice-0123456789abcdef01234567", - "Continue", - 1, - ), - ), - revision_digest="revision-background", - ) - } - utc_now = ["2026-07-13T00:00:00+00:00"] - - def read_pending(*_args, **_kwargs): - return {"_backend_pending_observation": current["observation"]} - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_pending) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - utc_clock=lambda: utc_now[0], - ) - scheduler.start() - try: - _wait_until( - lambda: any( - row["question"] == "Background choice?" - for row in pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - ) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute( - """ - SELECT observation_state, binding_private_fingerprint - FROM backend_pending - WHERE host_id = ? AND worker_id = ? - """, - (config.host_id, bindings[0].worker_id), - ).fetchone() == ("open", bindings[0].private_fingerprint) - current["observation"] = PendingObservation("read_succeeded_no_prompt") - utc_now[0] = "2026-07-13T00:00:01+00:00" - scheduler.request_refresh() - _wait_until( - lambda: not pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute( - """ - SELECT observation_state, freshness, binding_private_fingerprint - FROM backend_pending - WHERE host_id = ? AND worker_id = ? - """, - (config.host_id, bindings[0].worker_id), - ).fetchone() == ("none", "fresh", bindings[0].private_fingerprint) - finally: - scheduler.stop() - - -def test_background_authoritative_scan_reaps_removed_pane_pending( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot, bindings = _scheduler_store(tmp_path, 1) - observation = PendingObservation( - "open_prompt", - question="Will be removed?", - pending_kind="question", - revision_digest="revision-removal", - ) - def read_pending(_config, binding, **_kwargs): - selected = ( - observation - if binding.private_fingerprint == bindings[0].private_fingerprint - else PendingObservation("read_succeeded_no_prompt") - ) - return {"_backend_pending_observation": selected} - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_pending) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - utc_clock=lambda: "2026-07-13T00:00:00+00:00", - ) - scheduler.start() - try: - _wait_until( - lambda: bool( - pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - ) - decoy = _binding(config, snapshot.workers[0], 99) - upsert_worker_bindings(config.db_path, [decoy]) - with sqlite3.connect(config.db_path) as conn: - conn.execute( - """ - DELETE FROM worker_bindings - WHERE host_id = ? AND private_fingerprint = ? - """, - (config.host_id, bindings[0].private_fingerprint), - ) - scheduler.request_refresh() - _wait_until( - lambda: not pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute( - """ - SELECT private_fingerprint FROM worker_bindings - WHERE host_id = ? - """, - (config.host_id,), - ).fetchall() == [(decoy.private_fingerprint,)] - finally: - scheduler.stop() - - -def test_same_key_never_overlaps_and_burst_causes_one_rerun(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - first_entered = threading.Event() - release_first = threading.Event() - second_entered = threading.Event() - calls = 0 - active = 0 - maximum_active = 0 - lock = threading.Lock() - - def reader(_config, _binding, *, adapter_timeout_seconds): - nonlocal calls, active, maximum_active - with lock: - calls += 1 - call = calls - active += 1 - maximum_active = max(maximum_active, active) - if call == 1: - first_entered.set() - assert release_first.wait(2) - else: - second_entered.set() - with lock: - active -= 1 - return TurnRefreshResult("unchanged", 0) - - original_list = herdr_turns.list_worker_bindings - scan_after_burst = threading.Event() - list_calls = 0 - - def observed_list(*args, **kwargs): - nonlocal list_calls - result = original_list(*args, **kwargs) - list_calls += 1 - if first_entered.is_set() and list_calls >= 3: - scan_after_burst.set() - return result - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", observed_list) - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, reader=reader) - scheduler.start() - assert first_entered.wait(2) - for _ in range(20): - scheduler.request_refresh() - assert scan_after_burst.wait(2) - release_first.set() - assert second_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - scheduler.stop() - - assert calls == 2 - assert maximum_active == 1 - assert scheduler.operational_status()["coalesced"] >= 19 - - -def test_scan_dispatches_reader_before_blocked_orphan_prune_completes( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - prune_entered = threading.Event() - reader_entered = threading.Event() - original_prune = herdr_turns.prune_backend_pending - - def observed_prune(*args, **kwargs): - prune_entered.set() - return original_prune(*args, **kwargs) - - def reader(_config, _binding, *, adapter_timeout_seconds): - reader_entered.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "prune_backend_pending", observed_prune) - writer = sqlite3.connect(config.db_path, isolation_level=None, timeout=1) - writer.execute("BEGIN IMMEDIATE") - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - try: - assert prune_entered.wait(2) - assert reader_entered.wait(2) - finally: - writer.rollback() - writer.close() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - scheduler.stop() - - - -def test_transient_initial_binding_scan_retries_once_and_dispatches( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - original_list = herdr_turns.list_worker_bindings - reader_entered = threading.Event() - calls = 0 - lock = threading.Lock() - - def transient_list(*args, **kwargs): - nonlocal calls - with lock: - calls += 1 - call = calls - if call == 1: - raise sqlite3.OperationalError("transient initial read") - return original_list(*args, **kwargs) - - def reader(_config, _binding, *, adapter_timeout_seconds): - reader_entered.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", transient_list) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - assert reader_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - status = scheduler.operational_status() - assert status["failed"] == 1 - assert status["status"] == "healthy" - scheduler.stop() - - -def test_persistent_initial_binding_scan_failure_retries_once_without_spin( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 1) - second_call = threading.Event() - calls = 0 - lock = threading.Lock() - - prune_calls = 0 - - apply_backend_pending_observation( - config.db_path, - config.host_id, - bindings[0].worker_id, - PendingObservation( - "open_prompt", - question="Must survive failed scan?", - pending_kind="question", - revision_digest="failed-scan-revision", - ), - binding_private_fingerprint=bindings[0].private_fingerprint, - observed_turn_target_value=bindings[0].turn_target_value, - ) - def failed_list(*_args, **_kwargs): - nonlocal calls - with lock: - calls += 1 - if calls == 2: - second_call.set() - raise sqlite3.OperationalError("persistent read failure") - - def observed_prune(*_args, **_kwargs): - nonlocal prune_calls - prune_calls += 1 - return 0 - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", failed_list) - monkeypatch.setattr(herdr_turns, "prune_backend_pending", observed_prune) - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100) - scheduler.start() - assert second_call.wait(2) - _wait_until(lambda: scheduler.operational_status()["failed"] == 2) - with scheduler._condition: - assert calls == 2 - assert scheduler._scan_retry_remaining == 0 - assert scheduler._rescan_requested is False - assert scheduler.operational_status()["status"] == "degraded" - assert prune_calls == 0 - assert any( - row["question"] == "Must survive failed scan?" - for row in pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - scheduler.stop() - - -def test_transient_binding_revalidation_failure_retries_after_prune( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - original_list = herdr_turns.list_worker_bindings - original_prune = herdr_turns.prune_backend_pending - validation_failed = threading.Event() - prune_entered = threading.Event() - release_prune = threading.Event() - reader_entered = threading.Event() - validation_calls = 0 - lock = threading.Lock() - - def transient_validation(*args, **kwargs): - nonlocal validation_calls - if threading.current_thread().name.startswith("tendwire-turn-ingestion"): - with lock: - validation_calls += 1 - call = validation_calls - if call == 1: - validation_failed.set() - raise sqlite3.OperationalError("transient validation read") - return original_list(*args, **kwargs) - - def blocked_prune(*args, **kwargs): - prune_entered.set() - assert release_prune.wait(5) - return original_prune(*args, **kwargs) - - def reader(_config, _binding, *, adapter_timeout_seconds): - reader_entered.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", transient_validation) - monkeypatch.setattr(herdr_turns, "prune_backend_pending", blocked_prune) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - try: - assert validation_failed.wait(2) - assert prune_entered.wait(2) - assert not reader_entered.is_set() - release_prune.set() - assert reader_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - finally: - release_prune.set() - scheduler.stop() - status = scheduler.operational_status() - assert validation_calls == 2 - assert status["failed"] == 1 - - -def test_persistent_binding_revalidation_failure_is_bounded_and_skips_reader( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - original_list = herdr_turns.list_worker_bindings - second_failure = threading.Event() - reader_entered = threading.Event() - validation_calls = 0 - lock = threading.Lock() - - def failed_validation(*args, **kwargs): - nonlocal validation_calls - if threading.current_thread().name.startswith("tendwire-turn-ingestion"): - with lock: - validation_calls += 1 - if validation_calls == 2: - second_failure.set() - raise sqlite3.OperationalError("persistent validation read") - return original_list(*args, **kwargs) - - def reader(_config, _binding, *, adapter_timeout_seconds): - reader_entered.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", failed_validation) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - assert second_failure.wait(2) - _wait_until(lambda: scheduler.operational_status()["failed"] == 2) - with scheduler._condition: - key = next(iter(scheduler._binding_retry_remaining)) - assert scheduler._binding_retry_remaining[key] == 0 - assert key not in scheduler._binding_retry_due - assert validation_calls == 2 - assert not reader_entered.is_set() - assert scheduler.operational_status()["status"] == "degraded" - scheduler.stop() - - -def test_dirty_rerun_survives_full_queue_at_completion(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 3) - a_entered = threading.Event() - release_a = threading.Event() - a_finished = threading.Event() - b_entered = threading.Event() - release_b = threading.Event() - c_entered = threading.Event() - release_c = threading.Event() - third_prune_entered = threading.Event() - release_third_prune = threading.Event() - rerun_a_entered = threading.Event() - calls: list[str] = [] - first_target: str | None = None - b_target: str | None = None - target_calls: dict[str, int] = {} - lock = threading.Lock() - - def reader(_config, binding, *, adapter_timeout_seconds): - nonlocal first_target, b_target - target = str(binding.turn_target_value) - with lock: - calls.append(target) - if first_target is None: - first_target = target - target_calls[target] = target_calls.get(target, 0) + 1 - current_target_call = target_calls[target] - if target != first_target and b_target is None: - b_target = target - role = "a" if target == first_target else ("b" if target == b_target else "c") - if role == "a" and current_target_call == 1: - a_entered.set() - assert release_a.wait(5) - a_finished.set() - elif role == "a": - rerun_a_entered.set() - elif role == "b": - b_entered.set() - assert release_b.wait(5) - else: - c_entered.set() - assert release_c.wait(5) - return TurnRefreshResult("unchanged", 0) - - prune_calls = 0 - prune_lock = threading.Lock() - - def block_third_prune(*_args, **_kwargs): - nonlocal prune_calls - with prune_lock: - prune_calls += 1 - call = prune_calls - if call == 3: - third_prune_entered.set() - assert release_third_prune.wait(5) - return 0 - - monkeypatch.setattr(herdr_turns, "prune_backend_pending", block_third_prune) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=2, - queue_capacity=1, - reader=reader, - ) - scheduler.start() - try: - assert a_entered.wait(2) - scheduler.request_refresh() - assert b_entered.wait(2) - assert b_target is not None - with sqlite3.connect(config.db_path) as conn: - conn.execute( - "DELETE FROM worker_bindings WHERE host_id = ? AND turn_target_value = ?", - (config.host_id, b_target), - ) - scheduler.request_refresh() - assert third_prune_entered.wait(2) - assert not c_entered.is_set() - - release_a.set() - assert a_finished.wait(2) - _wait_until( - lambda: any( - item.turn_target_value == first_target and future.done() - for item, future, _started_at in scheduler._running.values() - ) - ) - release_third_prune.set() - assert c_entered.wait(2) - release_b.set() - release_c.set() - assert rerun_a_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - finally: - release_a.set() - release_b.set() - release_c.set() - release_third_prune.set() - scheduler.stop() - - assert first_target is not None - assert calls.count(first_target) == 2 - assert len(calls) == 4 - assert scheduler.operational_status()["queue_full"] >= 2 - - -def test_distinct_keys_use_four_workers_and_fifth_waits(tmp_path: Path) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 5) - release = threading.Event() - four_entered = threading.Event() - fifth_entered = threading.Event() - lock = threading.Lock() - active = 0 - maximum_active = 0 - calls = 0 - - def reader(_config, _binding, *, adapter_timeout_seconds): - nonlocal active, maximum_active, calls - with lock: - calls += 1 - active += 1 - maximum_active = max(maximum_active, active) - if calls == 4: - four_entered.set() - elif calls == 5: - fifth_entered.set() - assert release.wait(2) - with lock: - active -= 1 - return TurnRefreshResult("unchanged", 0) - - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=4, - reader=reader, - ) - scheduler.start() - assert four_entered.wait(2) - assert not fifth_entered.is_set() - status = scheduler.operational_status() - assert status["active"] == 4 - assert status["queue_depth"] == 1 - release.set() - assert fifth_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - scheduler.stop() - assert maximum_active == 4 - - -def test_queue_saturation_is_recovered_by_next_cadence_scan(tmp_path: Path) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 4) - - class Clock: - def __init__(self) -> None: - self.value = 0.0 - self.lock = threading.Lock() - - def __call__(self) -> float: - with self.lock: - return self.value - - def advance(self, seconds: float) -> None: - with self.lock: - self.value += seconds - - clock = Clock() - first_entered = threading.Event() - release = threading.Event() - all_seen = threading.Event() - seen: set[str] = set() - lock = threading.Lock() - - def reader(_config, binding, *, adapter_timeout_seconds): - with lock: - seen.add(binding.private_fingerprint) - if len(seen) == len(bindings): - all_seen.set() - if not first_entered.is_set(): - first_entered.set() - assert release.wait(2) - return TurnRefreshResult("unchanged", 0) - - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=2, - max_workers=1, - queue_capacity=2, - clock=clock, - reader=reader, - ) - scheduler.start() - assert first_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["queue_full"] >= 1) - release.set() - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - assert len(seen) == 2 - clock.advance(2.1) - with scheduler._condition: - scheduler._condition.notify_all() - assert all_seen.wait(2) - scheduler.stop() - assert scheduler.operational_status()["queue_full"] >= 1 - - -def test_target_change_discards_old_result_and_runs_latest_binding(tmp_path: Path, monkeypatch) -> None: - config, snapshot, bindings = _scheduler_store(tmp_path, 1) - old_binding = bindings[0] - old_entered = threading.Event() - release_old = threading.Event() - new_entered = threading.Event() - reads: list[str] = [] - - def read_binding(_config, binding, *, timeout_seconds, cancel_event=None): - target = str(binding.turn_target_value) - reads.append(target) - if target == "pane-0": - old_entered.set() - assert release_old.wait(2) - else: - new_entered.set() - return { - "source_turn_id": "source-stable", - "user_text": "question", - "assistant_final_text": target, - "complete": True, - "has_open_turn": False, - } - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_binding) - original_list = herdr_turns.list_worker_bindings - latest_scan = threading.Event() - - def observed_list(*args, **kwargs): - result = original_list(*args, **kwargs) - if old_entered.is_set() and any(item.turn_target_value == "pane-new" for item in result): - latest_scan.set() - return result - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", observed_list) - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, max_workers=1) - scheduler.start() - assert old_entered.wait(2) - replacement = WorkerBinding( - **{ - **old_binding.__dict__, - "turn_target_value": "pane-new", - "observed_at": "2026-07-12T00:00:01+00:00", - } - ) - upsert_worker_bindings(config.db_path, [replacement]) - scheduler.request_refresh() - assert latest_scan.wait(2) - release_old.set() - assert new_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - scheduler.stop() - - - turns = turns_payload_from_store(config.db_path, config.host_id, snapshot=snapshot)["turns"] - assert any(turn.get("assistant_final_text") == "pane-new" for turn in turns) - assert reads == ["pane-0", "pane-new"] - assert scheduler.operational_status()["failed"] == 1 - - -def test_atomic_binding_guard_closes_revalidation_commit_race(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 1) - original = bindings[0] - with sqlite3.connect(config.db_path) as conn: - baseline_revisions = conn.execute( - "SELECT COUNT(*) FROM turn_content_revisions" - ).fetchone()[0] - replacement = WorkerBinding( - **{ - **original.__dict__, - "turn_target_value": "pane-replaced-after-check", - "observed_at": "2026-07-12T00:00:02+00:00", - } - ) - monkeypatch.setattr( - herdr_turns, - "_read_turn_for_binding", - lambda _config, _binding, *, timeout_seconds, cancel_event=None: { - "source_turn_id": "must-not-commit", - "user_text": "stale question", - "assistant_final_text": "stale final", - "complete": True, - "has_open_turn": False, - }, - ) - - def race_after_revalidation(_config, _item): - upsert_worker_bindings(config.db_path, [replacement]) - return True - - monkeypatch.setattr(herdr_turns, "_binding_still_matches", race_after_revalidation) - result = herdr_turns.refresh_turn_binding(config, original) - - assert result == TurnRefreshResult("stale_binding", 0) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] == baseline_revisions - assert conn.execute( - "SELECT COUNT(*) FROM turns WHERE payload_json LIKE '%must-not-commit%'" - ).fetchone()[0] == 0 - - -def test_omp_compact_checkpoint_publishes_only_after_validated_durable_apply( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 1) - root = tmp_path / "omp-sessions" - session_dir = root / "-retry" - session_dir.mkdir(parents=True) - path = session_dir / "retry.jsonl" - path.write_text( - "\n".join( - json.dumps(line, separators=(",", ":")) - for line in ( - { - "type": "message", - "id": "retry-user", - "message": { - "role": "user", - "attribution": "user", - "content": [{"type": "text", "text": "retry prompt"}], - }, - }, - { - "type": "message", - "id": "retry-final", - "message": { - "role": "assistant", - "stopReason": "stop", - "content": [{"type": "text", "text": "durable final"}], - }, - }, - ) - ), - encoding="utf-8", - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - original = bindings[0] - omp_binding = WorkerBinding( - **{ - **original.__dict__, - "turn_target_kind": "omp_session_path", - "turn_target_value": str(path), - "private_fingerprint": "omp-retry-private", - } - ) - upsert_worker_bindings(config.db_path, [omp_binding]) - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - - original_matches = herdr_turns._binding_still_matches - monkeypatch.setattr(herdr_turns, "_binding_still_matches", lambda *_args: False) - stale = herdr_turns._refresh_turn_binding( - config, - omp_binding, - adapter_timeout_seconds=10, - ) - assert stale == TurnRefreshResult("stale_binding", 0) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._OMP_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "_binding_still_matches", original_matches) - original_apply = herdr_turns.apply_turn_refresh - - def fail_apply(*_args, **_kwargs): - raise sqlite3.OperationalError("injected apply failure") - - monkeypatch.setattr(herdr_turns, "apply_turn_refresh", fail_apply) - failed = herdr_turns._refresh_turn_binding( - config, - omp_binding, - adapter_timeout_seconds=10, - ) - assert failed == TurnRefreshResult("failed", 0) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._OMP_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "apply_turn_refresh", original_apply) - applied = herdr_turns._refresh_turn_binding( - config, - omp_binding, - adapter_timeout_seconds=10, - ) - assert applied.status == "updated" - with herdr_turns._OMP_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._serialize_omp_state( - herdr_turns._OMP_SESSION_CACHE[cache_key] - ) - assert checkpoint["turn_open"] is False - assert "retry prompt" not in json.dumps(checkpoint) - assert "durable final" not in json.dumps(checkpoint) - - unchanged = herdr_turns._refresh_turn_binding( - config, - omp_binding, - adapter_timeout_seconds=10, - ) - assert unchanged == TurnRefreshResult("unchanged", 0) - - -def test_codex_checkpoint_publication_requires_validated_durable_apply_and_cas( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot, bindings = _scheduler_store(tmp_path, 1) - session_id = "019f5590-3333-7333-8333-333333333333" - home = tmp_path / "codex-publication" - path = ( - home - / "sessions" - / "2026" - / "07" - / "12" - / f"rollout-2026-07-12T00-00-00-{session_id}.jsonl" - ) - path.parent.mkdir(parents=True) - turn_id = "codex-durable-turn" - records = ( - { - "type": "event_msg", - "payload": {"type": "task_started", "turn_id": turn_id}, - }, - { - "type": "response_item", - "payload": { - "type": "message", - "role": "user", - "content": [{"type": "input_text", "text": "durable prompt"}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - }, - }, - { - "type": "event_msg", - "payload": { - "type": "task_complete", - "turn_id": turn_id, - "last_agent_message": "durable Codex final", - }, - }, - ) - path.write_text( - "\n".join(json.dumps(record, separators=(",", ":")) for record in records) - + "\n", - encoding="utf-8", - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - original = bindings[0] - binding = WorkerBinding( - **{ - **original.__dict__, - "turn_target_kind": "codex_session_id", - "turn_target_value": session_id, - "private_fingerprint": "codex-publication-private", - } - ) - upsert_worker_bindings(config.db_path, [binding]) - cache_key = (str((home / "sessions").resolve()), session_id) - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - herdr_turns._CODEX_SESSION_CACHE_LIVE_KEYS = None - - original_matches = herdr_turns._binding_still_matches - monkeypatch.setattr(herdr_turns, "_binding_still_matches", lambda *_args: False) - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) == TurnRefreshResult("stale_binding", 0) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "_binding_still_matches", original_matches) - original_apply = herdr_turns.apply_turn_refresh - monkeypatch.setattr( - herdr_turns, - "apply_turn_refresh", - lambda *_args, **_kwargs: (_ for _ in ()).throw( - sqlite3.OperationalError("injected Codex apply failure") - ), - ) - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) == TurnRefreshResult("failed", 0) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "apply_turn_refresh", original_apply) - real_child = herdr_turns._file_turn_child - monkeypatch.setattr(herdr_turns, "_file_turn_child", _wrong_source_codex_child) - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) == TurnRefreshResult("failed", 0) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "_file_turn_child", _blocked_codex_child) - before_children = {child.pid for child in multiprocessing.active_children()} - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=0.05, - ) == TurnRefreshResult("timeout", 0) - assert {child.pid for child in multiprocessing.active_children()} == before_children - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "_file_turn_child", real_child) - real_commit = herdr_turns._file_publication_commit - monkeypatch.setattr( - herdr_turns, - "_file_publication_commit", - lambda _publication, content: content, - ) - applied_without_checkpoint = herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) - assert applied_without_checkpoint.status == "updated" - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - payload = turns_payload_from_store( - config.db_path, - config.host_id, - snapshot=snapshot, - ) - assert payload["turns"][0]["assistant_final_text"] == "durable Codex final" - - monkeypatch.setattr(herdr_turns, "_file_publication_commit", real_commit) - retry = herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) - assert retry.status == "unchanged" - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._serialize_codex_state( - herdr_turns._CODEX_SESSION_CACHE[cache_key] - ) - assert "durable prompt" not in json.dumps(checkpoint) - assert "durable Codex final" not in json.dumps(checkpoint) - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) == TurnRefreshResult("unchanged", 0) - - -def _codex_lifecycle_file(tmp_path: Path, monkeypatch, session_id: str) -> None: - home = tmp_path / "codex-lifecycle" - path = ( - home - / "sessions" - / "2026" - / "07" - / "12" - / f"rollout-2026-07-12T00-00-00-{session_id}.jsonl" - ) - path.parent.mkdir(parents=True, exist_ok=True) - path.write_bytes(b"") - monkeypatch.setenv("CODEX_HOME", str(home)) - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None -def test_direct_omp_first_frame_over_chunk_bound_is_rejected_and_reaped( - tmp_path: Path, - monkeypatch, -) -> None: - root = tmp_path / "omp-direct-frame" - path = root / "-session" / "session.jsonl" - path.parent.mkdir(parents=True) - path.write_bytes(b"") - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr( - herdr_turns, - "_file_turn_child", - _oversized_direct_omp_child, - ) - before_children = {child.pid for child in multiprocessing.active_children()} - - try: - herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=5, - ) - except herdr_turns._TurnReadFailed: - pass - else: - raise AssertionError("oversized direct OMP frame was accepted") - assert {child.pid for child in multiprocessing.active_children()} == before_children - - -def test_codex_direct_frame_can_exceed_omp_chunk_bound( - tmp_path: Path, - monkeypatch, -) -> None: - session_id = "019f5590-4444-7444-8444-444444444444" - _codex_lifecycle_file(tmp_path, monkeypatch, session_id) - monkeypatch.setattr( - herdr_turns, - "_file_turn_child", - _large_direct_codex_child, - ) - - observed = herdr_turns._read_file_turn_isolated( - "codex_session_id", - session_id, - timeout_seconds=5, - defer_cache=True, - ) - - assert isinstance(observed, herdr_turns._ObservedFileTurn) - final = observed.content["assistant_final_text"] - assert len(final) > herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES - assert len(final) < herdr_turns._CODEX_IPC_FRAME_MAX_BYTES - - - - -def test_oversized_isolated_request_is_rejected_without_process_or_helper( - monkeypatch, -) -> None: - before_children = {child.pid for child in multiprocessing.active_children()} - before_threads = {thread.ident for thread in threading.enumerate()} - started = time.monotonic() - try: - herdr_turns._read_file_turn_isolated( - "codex_session_id", - "x" * (8 * 1024 * 1024), - timeout_seconds=0.05, - ) - except herdr_turns._TurnReadFailed: - pass - else: - raise AssertionError("oversized private request was accepted") - elapsed = time.monotonic() - started - - assert elapsed < 0.2 - assert {child.pid for child in multiprocessing.active_children()} == before_children - assert {thread.ident for thread in threading.enumerate()} == before_threads - - -def test_isolated_process_construction_failure_closes_both_socket_fds( - tmp_path: Path, - monkeypatch, -) -> None: - session_id = "019f5590-1111-7111-8111-111111111111" - _codex_lifecycle_file(tmp_path, monkeypatch, session_id) - real_socketpair = herdr_turns.socket.socketpair - opened = [] - - def tracked_socketpair(): - pair = real_socketpair() - opened.extend(pair) - return pair - - real_context = multiprocessing.get_context("spawn") - - class FailingContext: - def Process(self, **_kwargs): - raise RuntimeError("injected process construction failure") - - monkeypatch.setattr(herdr_turns.socket, "socketpair", tracked_socketpair) - monkeypatch.setattr( - herdr_turns.multiprocessing, - "get_context", - lambda _method: FailingContext(), - ) - try: - herdr_turns._read_file_turn_isolated( - "codex_session_id", - session_id, - timeout_seconds=0.1, - ) - except RuntimeError: - pass - else: - raise AssertionError("process construction failure was hidden") - - assert len(opened) == 2 - assert all(channel.fileno() == -1 for channel in opened) - assert not real_context.active_children() - - -def test_isolated_start_delay_times_out_then_reaps_with_bounded_grace( - tmp_path: Path, - monkeypatch, -) -> None: - session_id = "019f5590-2222-7222-8222-222222222222" - _codex_lifecycle_file(tmp_path, monkeypatch, session_id) - context = multiprocessing.get_context("spawn") - process_type = type(context.Process()) - original_start = process_type.start - original_deadline_check = herdr_turns._check_ipc_deadline - start_entered = False - - def delayed_start(process): - nonlocal start_entered - start_entered = True - time.sleep(0.08) - return original_start(process) - - def deadline_after_start(deadline, cancel_event): - if start_entered: - original_deadline_check(deadline, cancel_event) - - monkeypatch.setattr(process_type, "start", delayed_start) - monkeypatch.setattr( - herdr_turns, - "_check_ipc_deadline", - deadline_after_start, - ) - before_children = {child.pid for child in multiprocessing.active_children()} - started = time.monotonic() - try: - herdr_turns._read_file_turn_isolated( - "codex_session_id", - session_id, - timeout_seconds=0.02, - ) - except herdr_turns._TurnReadTimeout: - pass - else: - raise AssertionError("delayed process start ignored the request deadline") - elapsed = time.monotonic() - started - - assert elapsed >= 0.08 - assert elapsed < 0.08 + herdr_turns._OMP_TEARDOWN_GRACE_SECONDS + 0.25 - assert {child.pid for child in multiprocessing.active_children()} == before_children - assert not any( - thread.name == "tendwire-turn-ipc" - for thread in threading.enumerate() - ) - - -def test_pathological_reap_never_uses_unbounded_join() -> None: - class NeverReaped: - pid = 123 - - def __init__(self): - self.joins = [] - self.kills = 0 - self.terminates = 0 - - def is_alive(self): - return True - - def join(self, timeout=None): - self.joins.append(timeout) - - def terminate(self): - self.terminates += 1 - - def kill(self): - self.kills += 1 - - process = NeverReaped() - started = time.monotonic() - herdr_turns._terminate_and_reap(process) - elapsed = time.monotonic() - started - - assert process.terminates == 1 - assert process.kills == 2 - assert process.joins - assert all(timeout is not None and timeout >= 0 for timeout in process.joins) - assert elapsed < herdr_turns._OMP_TEARDOWN_GRACE_SECONDS - - - - -def test_back_to_back_frames_are_received_without_trailing_byte_loss() -> None: - sender, receiver = herdr_turns.socket.socketpair() - try: - payloads = (b"first-frame", b"second-frame") - sender.sendall( - b"".join( - herdr_turns._OMP_FRAME_HEADER.pack(len(payload)) + payload - for payload in payloads - ) - ) - receiver.setblocking(False) - deadline = time.monotonic() + 1 - assert herdr_turns._recv_frame_until(receiver, deadline, None, 1024) == payloads[0] - assert herdr_turns._recv_frame_until(receiver, deadline, None, 1024) == payloads[1] - finally: - sender.close() - receiver.close() - - -def test_streamed_omp_response_reassembles_small_coalesced_chunks_without_leaks() -> None: - sender, receiver = herdr_turns.socket.socketpair() - receiver.setblocking(False) - payload = (b"chunked-private-response-" * 257) + b"end" - nonce = "stream-nonce" - failures = [] - - def send() -> None: - try: - herdr_turns._blocking_send_streamed_omp_response( - sender, - payload, - nonce, - chunk_bytes=7, - ) - except BaseException as exc: - failures.append(exc) - finally: - sender.close() - - thread = threading.Thread(target=send, name="test-omp-stream-sender") - thread.start() - try: - deadline = time.monotonic() + 5 - first = herdr_turns._recv_frame_until(receiver, deadline, None, 1024) - assembled = herdr_turns._recv_streamed_omp_response_until( - receiver, - first, - nonce, - "omp_session_path", - deadline, - None, - ) - finally: - receiver.close() - thread.join(5) - assert assembled == payload - assert failures == [] - assert thread.is_alive() is False - -def test_stream_manifest_rejects_chunk_bound_above_one_mib() -> None: - sender, receiver = herdr_turns.socket.socketpair() - receiver.setblocking(False) - nonce = "oversized-chunk-manifest" - chunk_bytes = herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES + 1 - try: - herdr_turns._blocking_send_streamed_omp_response( - sender, - b"x", - nonce, - chunk_bytes=chunk_bytes, - ) - except ValueError: - pass - else: - raise AssertionError("oversized OMP sender chunk bound was accepted") - manifest = json.dumps( - { - "protocol": 1, - "nonce": nonce, - "stream": "omp_response", - "chunks": 1, - "total_bytes": 1, - "chunk_bytes": chunk_bytes, - }, - separators=(",", ":"), - ).encode("utf-8") - herdr_turns._blocking_send_frame(sender, manifest) - sender.close() - try: - deadline = time.monotonic() + 1 - first = herdr_turns._recv_frame_until( - receiver, - deadline, - None, - herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES, - ) - try: - herdr_turns._recv_streamed_omp_response_until( - receiver, - first, - nonce, - "omp_session_path", - deadline, - None, - ) - except herdr_turns._TurnReadFailed: - pass - else: - raise AssertionError("oversized OMP chunk manifest was accepted") - finally: - receiver.close() - - -def test_streamed_omp_response_rejects_extra_frame_after_terminator() -> None: - sender, receiver = herdr_turns.socket.socketpair() - receiver.setblocking(False) - nonce = "extra-frame-nonce" - manifest = json.dumps( - { - "protocol": 1, - "nonce": nonce, - "stream": "omp_response", - "chunks": 1, - "total_bytes": 3, - "chunk_bytes": 3, - }, - separators=(",", ":"), - ).encode("utf-8") - end = json.dumps( - {"protocol": 1, "nonce": nonce, "stream": "omp_response_end"}, - separators=(",", ":"), - ).encode("utf-8") - frames = (manifest, b"abc", end, b"unexpected") - sender.sendall( - b"".join( - herdr_turns._OMP_FRAME_HEADER.pack(len(payload)) + payload - for payload in frames - ) - ) - sender.close() - try: - deadline = time.monotonic() + 1 - first = herdr_turns._recv_frame_until(receiver, deadline, None, 1024) - try: - herdr_turns._recv_streamed_omp_response_until( - receiver, - first, - nonce, - "omp_session_path", - deadline, - None, - ) - except herdr_turns._TurnReadFailed: - pass - else: - raise AssertionError("extra streamed IPC frame was accepted") - finally: - receiver.close() - - - -def test_file_adapter_timeout_kills_reaps_and_leaves_no_ipc_threads(tmp_path: Path, monkeypatch) -> None: - root = tmp_path / "sessions" - root.mkdir() - fifo = root / "blocked.jsonl" - os.mkfifo(fifo) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - before_children = {child.pid for child in multiprocessing.active_children()} - before_threads = {thread.ident for thread in threading.enumerate() if thread.name == "tendwire-turn-ipc"} - started = time.monotonic() - - for _ in range(3): - try: - herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(fifo), - timeout_seconds=0.1, - ) - except herdr_turns._TurnReadTimeout: - pass - else: - raise AssertionError("blocked file reader did not time out") - elapsed = time.monotonic() - started - - assert {child.pid for child in multiprocessing.active_children()} == before_children - assert {thread.ident for thread in threading.enumerate() if thread.name == "tendwire-turn-ipc"} == before_threads - assert elapsed < 2.0 - - -def test_stop_terminates_and_reaps_active_pane_adapter(tmp_path: Path, monkeypatch) -> None: - base_config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - pid_file = tmp_path / "adapter.pid" - adapter = tmp_path / "blocked_adapter.py" - adapter.write_text( - "#!/usr/bin/env python3\n" - "import os, time\n" - "with open(os.environ['TENDWIRE_TEST_ADAPTER_PID'], 'w') as handle:\n" - " handle.write(str(os.getpid()))\n" - "time.sleep(30)\n", - encoding="utf-8", - ) - adapter.chmod(0o700) - monkeypatch.setenv("TENDWIRE_TEST_ADAPTER_PID", str(pid_file)) - config = Config( - host_id=base_config.host_id, - db_path=base_config.db_path, - herdr_bin=str(adapter), - herdr_timeout_seconds=10, - turn_refresh_interval_seconds=100, - turn_refresh_workers=1, - ) - scheduler = TurnIngestionScheduler(config) - scheduler.start() - _wait_until( - lambda: pid_file.exists() - and bool(pid_file.read_text(encoding="utf-8").strip()) - ) - pid = int(pid_file.read_text(encoding="utf-8")) - started = time.monotonic() - scheduler.stop(flush_timeout_seconds=1) - elapsed = time.monotonic() - started - - assert elapsed < 1 - assert scheduler.operational_status()["active"] == 0 - assert not any( - thread.name.startswith("tendwire-turn-ingestion") - for thread in threading.enumerate() - ) - try: - os.kill(pid, 0) - except ProcessLookupError: - pass - else: - raise AssertionError("pane adapter child was not reaped") - - -def test_stop_rejects_new_refresh_and_boundedly_drains_started_work(tmp_path: Path) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - entered = threading.Event() - release = threading.Event() - finished = threading.Event() - calls = 0 - - def reader(_config, _binding, *, adapter_timeout_seconds): - nonlocal calls - calls += 1 - entered.set() - assert release.wait(2) - finished.set() - return TurnRefreshResult("unchanged", 0) - - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, reader=reader) - scheduler.start() - assert entered.wait(2) - stopper = threading.Thread( - target=lambda: scheduler.stop(flush_timeout_seconds=0.5), - name="test-scheduler-stopper", - ) - started = time.monotonic() - stopper.start() - _wait_until(lambda: scheduler.operational_status()["status"] == "stopping") - scheduler.request_refresh() - release.set() - stopper.join(1) - assert not stopper.is_alive() - assert time.monotonic() - started < 1 - assert finished.wait(1) - assert calls == 1 - assert scheduler.operational_status()["queue_depth"] == 0 - - -def test_direct_fallback_actively_feeds_only_worker_bound(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 5) - release = threading.Event() - two_entered = threading.Event() - lock = threading.Lock() - calls: list[str] = [] - active = 0 - maximum_active = 0 - - def reader( - _config, - binding, - *, - adapter_timeout_seconds, - cancel_event=None, - apply_deadline_monotonic=None, - ): - nonlocal active, maximum_active - with lock: - calls.append(binding.private_fingerprint) - active += 1 - maximum_active = max(maximum_active, active) - if len(calls) == 2: - two_entered.set() - assert release.wait(2) - with lock: - active -= 1 - return TurnRefreshResult("updated", 1) - - monkeypatch.setattr(herdr_turns, "_refresh_turn_binding", reader) - returned: list[dict[str, Any]] = [] - fallback = threading.Thread( - target=lambda: returned.append( - herdr_turns.refresh_structured_turn_content( - config, - max_workers=2, - total_timeout_seconds=2, - ) - ), - name="test-turn-fallback", - ) - fallback.start() - assert two_entered.wait(2) - assert len(calls) == 2 - release.set() - fallback.join(2) - assert not fallback.is_alive() - assert returned == [{"ok": True, "status": "ok", "updated": 5, "attempted": 5}] - assert len(calls) == len(bindings) - assert len(set(calls)) == len(bindings) - assert maximum_active == 2 - - -def test_direct_fallback_total_deadline_stops_feeding_new_work(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 5) - calls = 0 - lock = threading.Lock() - - def reader( - _config, - _binding, - *, - adapter_timeout_seconds, - cancel_event=None, - apply_deadline_monotonic=None, - ): - nonlocal calls - with lock: - calls += 1 - threading.Event().wait(adapter_timeout_seconds) - return TurnRefreshResult("timeout", 0) - - monkeypatch.setattr(herdr_turns, "_refresh_turn_binding", reader) - started = time.monotonic() - result = herdr_turns.refresh_structured_turn_content( - config, - max_workers=2, - total_timeout_seconds=1, - ) - assert result == { - "ok": False, - "status": "deadline_exceeded", - "updated": 0, - "attempted": 2, - } - assert calls == 2 - assert time.monotonic() - started < 1 - _wait_until( - lambda: not any( - thread.name.startswith("tendwire-turn-fallback") - for thread in threading.enumerate() - ) - ) - - -def test_direct_fallback_deadline_reaps_real_pane_child(tmp_path: Path, monkeypatch) -> None: - base_config, _snapshot, _bindings = _scheduler_store(tmp_path, 3) - pid_file = tmp_path / "fallback-adapter.pid" - adapter = tmp_path / "fallback_blocked_adapter.py" - adapter.write_text( - "#!/usr/bin/env python3\n" - "import os, time\n" - "with open(os.environ['TENDWIRE_TEST_FALLBACK_PID'], 'w') as handle:\n" - " handle.write(str(os.getpid()))\n" - "time.sleep(30)\n", - encoding="utf-8", - ) - adapter.chmod(0o700) - monkeypatch.setenv("TENDWIRE_TEST_FALLBACK_PID", str(pid_file)) - config = Config( - host_id=base_config.host_id, - db_path=base_config.db_path, - herdr_bin=str(adapter), - herdr_timeout_seconds=10, - turn_refresh_interval_seconds=100, - turn_refresh_workers=1, - ) - - started = time.monotonic() - result = herdr_turns.refresh_structured_turn_content( - config, - max_workers=1, - total_timeout_seconds=1, - ) - elapsed = time.monotonic() - started - - assert result == { - "ok": False, - "status": "deadline_exceeded", - "updated": 0, - "attempted": 1, - } - assert elapsed < 1 - assert pid_file.exists() - pid = int(pid_file.read_text(encoding="utf-8")) - _wait_until( - lambda: not any( - thread.name.startswith("tendwire-turn-fallback") - for thread in threading.enumerate() - ) - ) - try: - os.kill(pid, 0) - except ProcessLookupError: - pass - else: - raise AssertionError("fallback pane child was not reaped") - - -def test_fallback_deadline_cancels_blocked_store_apply_without_late_commit( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 3) - content = { - "source_turn_id": "blocked-fallback-source", - "user_text": "must not commit", - "assistant_final_text": "must not commit", - "complete": True, - "has_open_turn": False, - } - read_done = threading.Event() - - def read_now(_config, _binding, *, timeout_seconds, cancel_event=None): - read_done.set() - return content - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_now) - with sqlite3.connect(config.db_path) as conn: - baseline = conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - blocker = sqlite3.connect(config.db_path, isolation_level=None) - blocker.execute("BEGIN IMMEDIATE") - returned: list[dict[str, Any]] = [] - fallback = threading.Thread( - target=lambda: returned.append( - herdr_turns.refresh_structured_turn_content( - config, - max_workers=1, - total_timeout_seconds=1, - ) - ), - name="test-locked-fallback", - ) - started = time.monotonic() - fallback.start() - assert read_done.wait(1) - fallback.join(1.1) - elapsed = time.monotonic() - started - assert not fallback.is_alive() - assert elapsed < 1.1 - assert returned[0]["status"] == "deadline_exceeded" - blocker.rollback() - blocker.close() - _wait_until( - lambda: not any( - thread.name.startswith("tendwire-turn-fallback") - for thread in threading.enumerate() - ) - ) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] == baseline - assert conn.execute( - "SELECT COUNT(*) FROM turns WHERE payload_json LIKE '%blocked-fallback-source%'" - ).fetchone()[0] == 0 - - - - -def test_fallback_prune_obeys_total_deadline_without_late_delete( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - with sqlite3.connect(config.db_path) as conn: - conn.execute( - """ - INSERT INTO backend_pending (host_id, worker_id, payload_json, observed_at) - VALUES (?, ?, ?, ?) - """, - ( - config.host_id, - "orphan-worker", - '{"prompt":"must survive"}', - "2026-07-12T00:00:00+00:00", - ), - ) - job_finished = threading.Event() - - def reader( - _config, - _binding, - *, - adapter_timeout_seconds, - cancel_event=None, - apply_deadline_monotonic=None, - ): - job_finished.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "_refresh_turn_binding", reader) - blocker = sqlite3.connect(config.db_path, isolation_level=None) - blocker.execute("BEGIN IMMEDIATE") - started = time.monotonic() - result = herdr_turns.refresh_structured_turn_content( - config, - max_workers=1, - total_timeout_seconds=0.4, - ) - elapsed = time.monotonic() - started - - assert job_finished.is_set() - assert result == { - "ok": False, - "status": "deadline_exceeded", - "updated": 0, - "attempted": 1, - } - assert elapsed < 0.7 - blocker.rollback() - blocker.close() - threading.Event().wait(0.2) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute( - """ - SELECT COUNT(*) FROM backend_pending - WHERE host_id = ? AND worker_id = ? - """, - (config.host_id, "orphan-worker"), - ).fetchone()[0] == 1 - _wait_until( - lambda: not any( - thread.name.startswith("tendwire-turn-fallback") - for thread in threading.enumerate() - ) - ) - - -def test_scheduler_stop_cancels_blocked_store_apply_without_late_commit( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - read_done = threading.Event() - - def read_now(_config, _binding, *, timeout_seconds, cancel_event=None): - read_done.set() - return { - "source_turn_id": "blocked-scheduler-source", - "user_text": "must not commit", - "assistant_final_text": "must not commit", - "complete": True, - "has_open_turn": False, - } - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_now) - monkeypatch.setattr(herdr_turns, "prune_backend_pending", lambda *args, **kwargs: 0) - with sqlite3.connect(config.db_path) as conn: - baseline = conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - blocker = sqlite3.connect(config.db_path, isolation_level=None) - blocker.execute("BEGIN IMMEDIATE") - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, max_workers=1) - scheduler.start() - assert read_done.wait(1) - started = time.monotonic() - scheduler.stop(flush_timeout_seconds=0.5) - elapsed = time.monotonic() - started - assert elapsed < 0.5 - assert scheduler.operational_status()["active"] == 0 - blocker.rollback() - blocker.close() - with sqlite3.connect(config.db_path) as conn: - assert conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] == baseline - assert conn.execute( - "SELECT COUNT(*) FROM turns WHERE payload_json LIKE '%blocked-scheduler-source%'" - ).fetchone()[0] == 0 - - -def test_scheduler_restart_with_identical_final_is_revision_noop(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - content = { - "source_turn_id": "restart-source", - "user_text": "same prompt", - "assistant_final_text": "same final", - "complete": True, - "has_open_turn": False, - } - monkeypatch.setattr( - herdr_turns, - "_read_turn_for_binding", - lambda _config, _binding, *, timeout_seconds, cancel_event=None: content, - ) - - def run_once() -> None: - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, max_workers=1) - scheduler.start() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - scheduler.stop() - - run_once() - with sqlite3.connect(config.db_path) as conn: - first = conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - run_once() - with sqlite3.connect(config.db_path) as conn: - second = conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - assert second == first - - -def test_operational_status_recovers_after_successful_empty_binding_scan( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 0) - stale_cache_key = "disappeared-omp-binding" - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._omp_cache_store_locked( - stale_cache_key, - herdr_turns._OmpSessionState(offset=1, file_id=(1, 1)), - ) - original_list = herdr_turns.list_worker_bindings - allow_success = threading.Event() - successful_scan = threading.Event() - - def flaky_list(*args, **kwargs): - if not allow_success.is_set(): - raise sqlite3.OperationalError("deterministic scan failure") - bindings = original_list(*args, **kwargs) - successful_scan.set() - return bindings - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", flaky_list) - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100) - scheduler.start() - _wait_until(lambda: scheduler.operational_status()["failed"] == 1) - assert scheduler.operational_status()["status"] == "degraded" - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert stale_cache_key in herdr_turns._OMP_SESSION_CACHE - - allow_success.set() - scheduler.request_refresh() - assert successful_scan.wait(2) - _wait_until(lambda: scheduler.operational_status()["status"] == "stale") - recovered = scheduler.operational_status() - assert recovered["failed"] == 1 - assert recovered["refreshed"] == 0 - assert recovered["timed_out"] == 0 - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert stale_cache_key not in herdr_turns._OMP_SESSION_CACHE - scheduler.stop() - - -def test_fallback_successful_empty_scan_prunes_disappeared_omp_cache( - tmp_path: Path, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 0) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._omp_cache_store_locked( - "fallback-disappeared-omp", - herdr_turns._OmpSessionState(offset=1, file_id=(1, 1)), - ) - - result = herdr_turns.refresh_structured_turn_content(config) - - assert result == {"ok": True, "status": "ok", "updated": 0, "attempted": 0} - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert not herdr_turns._OMP_SESSION_CACHE - - -def test_operational_status_recovers_from_completed_failures_and_stale_churn( - tmp_path: Path, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - outcomes = [ - "failed", - "unchanged", - "stale_binding", - "updated", - "timeout", - "unchanged", - "blocked", - ] - blocked_entered = threading.Event() - release_blocked = threading.Event() - calls = 0 - lock = threading.Lock() - - def reader(_config, _binding, *, adapter_timeout_seconds): - nonlocal calls - with lock: - outcome = outcomes[calls] - calls += 1 - if outcome == "blocked": - blocked_entered.set() - assert release_blocked.wait(2) - outcome = "unchanged" - return TurnRefreshResult(outcome, 1 if outcome == "updated" else 0) - - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - _wait_until(lambda: scheduler.operational_status()["failed"] == 1) - failed_now = scheduler.operational_status() - assert failed_now["status"] == "degraded" - assert failed_now["refreshed"] == 0 - assert failed_now["timed_out"] == 0 - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - recovered = scheduler.operational_status() - assert recovered["status"] == "healthy" - assert recovered["failed"] == 1 - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["failed"] == 2) - stale_churn = scheduler.operational_status() - assert stale_churn["status"] == "healthy" - assert stale_churn["refreshed"] == 1 - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 2) - assert scheduler.operational_status()["status"] == "healthy" - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["timed_out"] == 1) - current_timeout = scheduler.operational_status() - assert current_timeout["status"] == "degraded" - assert current_timeout["failed"] == 2 - assert current_timeout["refreshed"] == 2 - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 3) - assert scheduler.operational_status()["status"] == "healthy" - - scheduler.request_refresh() - assert blocked_entered.wait(2) - active_fresh = scheduler.operational_status() - assert active_fresh["active"] == 1 - assert active_fresh["status"] == "healthy" - release_blocked.set() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 4) - final = scheduler.operational_status() - assert final["status"] == "healthy" - assert final["failed"] == 2 - assert final["timed_out"] == 1 - assert final["refreshed"] == 4 - scheduler.stop() - - -def test_operational_status_has_only_fixed_aggregate_fields(tmp_path: Path) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 0) - scheduler = TurnIngestionScheduler(config) - assert set(scheduler.operational_status()) == { - "status", - "queue_depth", - "active", - "refreshed", - "failed", - "timed_out", - "coalesced", - "queue_full", - "last_success", - "last_duration_ms", - "stale_age_seconds", - "max_workers", - "queue_capacity", - "refresh_interval_seconds", - "adapter_timeout_seconds", - } - - -def test_structured_refresh_rebinds_same_owner_after_stale_a_rejection_and_current_b_retry( - tmp_path: Path, - monkeypatch, -) -> None: - config = Config( - host_id="ingestion-owner-host", - db_path=tmp_path / "ingestion-owner.db", - herdr_timeout_seconds=0.5, - turn_refresh_interval_seconds=100.0, - turn_refresh_workers=1, - ) - assert config.db_path is not None - stable_key = "wsk1_" + ("7" * 64) - worker_a = Worker( - id="structured-worker-a", - name="Structured Worker A", - status="active", - space_id="structured-space-a", - fingerprint="structured-fingerprint-a", - meta={"stable_key": stable_key, "stable_key_version": 1}, - ) - worker_b = Worker( - id="structured-worker-b", - name="Structured Worker B", - status="waiting", - space_id="structured-space-b", - fingerprint="structured-fingerprint-b", - meta={"stable_key": stable_key, "stable_key_version": 1}, - ) - snapshot_a = Snapshot( - host_id=config.host_id, - updated_at="2026-07-13T05:00:00+00:00", - workers=[worker_a], - ) - snapshot_b = Snapshot( - host_id=config.host_id, - updated_at="2026-07-13T05:01:00+00:00", - workers=[worker_b], - ) - binding_a = _binding(config, worker_a, 70, target="structured-pane-a-private") - binding_b = _binding(config, worker_b, 71, target="structured-pane-b-private") - raw_source = "019f5590-4444-7444-8444-444444444444" - - init_store(config.db_path) - save_snapshot(config.db_path, snapshot_a) - upsert_worker_bindings(config.db_path, [binding_a]) - assert merge_turn_content( - config.db_path, - config.host_id, - worker_a.id, - { - "source_turn_id": raw_source, - "user_text": "stable owner prompt", - "assistant_final_text": "initial A final", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-07-13T05:00:01+00:00", - ) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - source_before = conn.execute( - """ - SELECT turn_id, list_sequence, - json_extract(payload_json, '$.source_turn_id') - FROM turns - WHERE host_id = ? - AND json_extract(payload_json, '$.source_turn_id') IS NOT NULL - """, - (config.host_id,), - ).fetchone() - list_state_before = conn.execute( - """ - SELECT next_sequence, traversal_generation - FROM turn_list_hosts - WHERE host_id = ? - """, - (config.host_id,), - ).fetchone() - assert source_before is not None - - old_entered = threading.Event() - release_old = threading.Event() - current_entered = threading.Event() - reads: list[tuple[str, str]] = [] - - def read_binding(_config, binding, *, timeout_seconds, cancel_event=None): - reads.append((str(binding.worker_id), str(binding.turn_target_value))) - if binding.worker_id == worker_a.id: - old_entered.set() - assert release_old.wait(2) - final_text = "stale A final must not commit" - else: - current_entered.set() - final_text = "current B final" - return { - "source_turn_id": raw_source, - "user_text": "stable owner prompt", - "assistant_final_text": final_text, - "complete": True, - "has_open_turn": False, - } - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_binding) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - ) - scheduler.start() - try: - assert old_entered.wait(2) - save_snapshot(config.db_path, snapshot_b) - upsert_worker_bindings(config.db_path, [binding_b]) - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - DELETE FROM worker_bindings - WHERE host_id = ? AND private_fingerprint = ? - """, - (config.host_id, binding_a.private_fingerprint), - ) - scheduler.request_refresh() - release_old.set() - assert current_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - finally: - release_old.set() - scheduler.stop() - - public_payload = turns_payload_from_store( - config.db_path, - config.host_id, - snapshot=snapshot_b, - schema_version=2, - ) - source_turns = [ - turn for turn in public_payload["turns"] if turn.get("source_turn_id") - ] - assert len(source_turns) == 1 - source_turn = source_turns[0] - assert source_turn["id"] == source_before[0] - assert source_turn["source_turn_id"] == source_before[2] - assert source_turn["source_turn_id"] != raw_source - assert source_turn["worker_id"] == worker_b.id - assert source_turn["worker_fingerprint"] == worker_b.fingerprint - assert source_turn["space_id"] == worker_b.space_id - assert source_turn["assistant_final_text"] == "current B final" - assert source_turn["complete"] is True - assert source_turn["has_open_turn"] is False - assert reads == [ - (worker_a.id, "structured-pane-a-private"), - (worker_b.id, "structured-pane-b-private"), - ] - assert scheduler.operational_status()["failed"] == 1 - with sqlite3.connect(str(config.db_path)) as conn: - persisted_source = conn.execute( - """ - SELECT turn_id, list_sequence, - json_extract(payload_json, '$.source_turn_id') - FROM turns - WHERE host_id = ? - AND json_extract(payload_json, '$.source_turn_id') IS NOT NULL - """, - (config.host_id,), - ).fetchone() - list_state_after = conn.execute( - """ - SELECT next_sequence, traversal_generation - FROM turn_list_hosts - WHERE host_id = ? - """, - (config.host_id,), - ).fetchone() - current_revisions = conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - """, - (config.host_id, source_turn["id"]), - ).fetchone()[0] - foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert persisted_source == source_before - assert persisted_source[1] == source_before[1] - assert list_state_after == list_state_before - assert current_revisions == 1 - assert foreign_keys == [] - encoded = json.dumps( - [public_payload, scheduler.operational_status()], - sort_keys=True, - ) - for private_value in ( - raw_source, - "structured-pane-a-private", - "structured-pane-b-private", - binding_a.private_fingerprint, - binding_b.private_fingerprint, - "stale A final must not commit", - ): - assert private_value not in encoded diff --git a/tests/test_turn_ingestion_benchmark.py b/tests/test_turn_ingestion_benchmark.py deleted file mode 100644 index e143e4f..0000000 --- a/tests/test_turn_ingestion_benchmark.py +++ /dev/null @@ -1,168 +0,0 @@ -from __future__ import annotations - -import importlib.util -import json -import os -import subprocess -import sys -from pathlib import Path -from types import ModuleType - -import pytest - - -ROOT = Path(__file__).resolve().parents[1] -SCRIPT = ROOT / "scripts" / "turn_ingestion_benchmark.py" - - -def _load_driver() -> ModuleType: - spec = importlib.util.spec_from_file_location("turn_ingestion_benchmark", SCRIPT) - assert spec is not None and spec.loader is not None - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -def _invoke(*arguments: str) -> subprocess.CompletedProcess[str]: - environment = os.environ.copy() - environment["PYTHONPATH"] = str(ROOT / "src") - return subprocess.run( - [sys.executable, str(SCRIPT), *arguments], - cwd=ROOT, - stdin=subprocess.DEVNULL, - capture_output=True, - text=True, - env=environment, - check=False, - timeout=30, - ) - - -def _single_object(completed: subprocess.CompletedProcess[str]) -> dict[str, object]: - assert completed.stderr == "" - lines = completed.stdout.splitlines() - assert len(lines) == 1 - payload = json.loads(lines[0]) - assert isinstance(payload, dict) - assert lines[0] == json.dumps( - payload, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - ) - return payload - - -def test_pending_validator_requires_fixed_durable_health_shape() -> None: - driver = _load_driver() - result = { - "schema_version": 1, - "host_id": "benchmark-validator-host", - "pending_interactions": [], - "backend_health": [], - "pending_health": { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - }, - } - result["content_fingerprint"] = ( - driver.recompute_pending_content_fingerprint(result) - ) - valid = {"ok": True, "result": result} - - driver._validate_pending(valid, 2) - for invalid in ( - {**valid, "ok": False}, - { - "ok": True, - "result": { - **valid["result"], - "pending_health": { - "status": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - }, - }, - }, - { - "ok": True, - "result": { - **valid["result"], - "pending_health": { - "status": "healthy", - "counts": {"fresh": 1, "stale": 0, "total": 0}, - }, - }, - }, - { - "ok": True, - "result": { - **valid["result"], - "content_fingerprint": "0" * 24, - }, - }, - ): - with pytest.raises(RuntimeError, match="pending_list_contract_failed"): - driver._validate_pending(invalid, 2) - - -@pytest.mark.skipif( - not sys.platform.startswith("linux") or not Path("/dev/shm").is_dir(), - reason="benchmark contract requires Linux tmpfs and Unix sockets", -) -def test_tiny_run_measures_production_pending_without_source_or_turn_reads() -> None: - completed = _invoke( - "--workers", - "2", - "--blocked-workers", - "2", - "--blocked-seconds", - "0.1", - "--warmups", - "0", - "--samples", - "1", - "--json", - ) - - assert completed.returncode == 0 - report = _single_object(completed) - assert report["ok"] is True - assert report["status"] == "completed" - assert report["latency_ns"]["pending_list"]["samples"] == 1 - assert ( - report["latency_ns"]["pending_list"]["documented_host_budget_ns"] - == 350_000_000 - ) - assert report["latency_ns"]["pending_list"]["documented_host_budget_met"] is True - assert report["checks"]["production_pending_handler_measured"] is True - assert report["checks"]["production_event_callback_bound"] is True - assert report["checks"]["pending_list_started_no_turn_reads"] is True - assert report["checks"]["pending_list_started_no_source_reads"] is True - assert report["checks"]["pending_list_store_rows_unchanged"] is True - assert report["checks"]["cached_requests_started_no_source_reads"] is True - assert report["checks"]["independent_pending_discovered"] is True - assert report["checks"]["independent_pending_cleared"] is True - assert report["checks"]["independent_pending_zero_turn_calls"] is True - assert ( - report["checks"]["independent_pending_discovery_fingerprint_changed"] - is True - ) - assert ( - report["checks"]["independent_pending_unchanged_fingerprint_stable"] - is True - ) - assert report["checks"]["independent_pending_clear_fingerprint_changed"] is True - assert report["checks"]["independent_pending_clear_restored_baseline"] is True - assert report["checks"]["no_duplicate_pending_rows"] is True - assert report["checks"]["independent_pending_health_coherent"] is True - assert ( - report["checks"]["independent_pending_rows_coherent_after_clear"] is True - ) - assert report["ingestion"]["turn_list_calls_during_pending_measurement"] == 0 - assert report["ingestion"]["independent_turn_list_calls"] == 0 - assert report["ingestion"]["independent_prompt_count"] == 2 - assert report["ingestion"]["independent_clear_count"] == 0 - assert ( - report["transport"]["method_dispatches"]["pending.list"] - == 1 + report["ingestion"]["independent_pending_polls"] - ) diff --git a/tests/test_worker_label_and_model.py b/tests/test_worker_label_and_model.py index 1a8cc76..9c162b0 100644 --- a/tests/test_worker_label_and_model.py +++ b/tests/test_worker_label_and_model.py @@ -6,7 +6,6 @@ from tendwire.backends.herdr_cli import _worker_from_item, _workers_and_bindings_from_records from tendwire.backends.herdr_events import HerdrEventBackend -from tendwire.backends.herdr_turns import _TURN_CONTENT_KEYS from tendwire.config import Config from tendwire.core.turns import Turn from tendwire.core.projector import project_from_raw @@ -92,35 +91,8 @@ def test_reconcile_drops_agent_and_pane_cwd_from_public_worker(tmp_path: Path) - assert "/root/pane-cwd" not in str(records[0].worker.to_dict()) -def test_reconcile_keeps_agent_turn_target_when_present(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - agent = {**_agent_item(), "agent": "codex", "name": "codex"} - records = backend._records_from_reconcile_payloads({"agents": [agent]}, {"panes": [_pane_item()]}) - - assert len(records) == 1 - assert records[0].turn_target_kind == "codex_session_id" - assert records[0].turn_target_value == "sess-1" - assert records[0].worker.meta.get("label") == "review-pane" - -def test_reconcile_uses_matched_pane_turn_target_when_agent_lacks_one(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - agent = _agent_item() - agent.pop("pane_id") - records = backend._records_from_reconcile_payloads({"agents": [agent]}, {"panes": [_pane_item()]}) - workers, bindings = _workers_and_bindings_from_records(config, records) - assert len(records) == 1 - assert records[0].turn_target_kind == "pane_id" - assert records[0].turn_target_value == "ws-1:p2Q" - assert len(bindings) == 1 - assert bindings[0].turn_target_kind == "pane_id" - assert bindings[0].turn_target_value == "ws-1:p2Q" - assert workers[0].meta.get("label") == "review-pane" def test_reconcile_only_fills_missing_agent_backend_target_from_pane(tmp_path: Path) -> None: @@ -167,8 +139,6 @@ def test_turn_model_round_trip_and_id_stability() -> None: assert plain.fingerprint != with_model.fingerprint # but the content fingerprint reflects it -def test_turn_content_keys_include_model() -> None: - assert "model" in _TURN_CONTENT_KEYS def test_merge_turn_content_persists_model(tmp_path: Path) -> None: diff --git a/tests/test_worker_stable_key.py b/tests/test_worker_stable_key.py index 3819fd4..89edb24 100644 --- a/tests/test_worker_stable_key.py +++ b/tests/test_worker_stable_key.py @@ -534,7 +534,6 @@ def test_session_targeted_agent_adopts_matched_pane_identity_privately(tmp_path: assert len(records) == len(workers) == 1 assert records[0].workspace_id == "wR9" assert records[0].pane_id == "wR9:pA" - assert records[0].turn_target_kind == "codex_session_id" assert _STABLE_KEY.fullmatch(_stable(workers[0])) public = json.dumps(workers[0].to_dict(), sort_keys=True) assert "wR9:pA" not in public @@ -579,8 +578,6 @@ def test_matched_pane_overrides_conflicting_agent_continuity_and_workspace( assert merged_records[0].pane_id != agent_records[0].pane_id assert merged_workers[0].space_id == pane_workers[0].space_id == "wR9" assert merged_workers[0].space_id != agent_workers[0].space_id - assert merged_records[0].turn_target_kind == "codex_session_id" - assert merged_records[0].turn_target_value == pane["agent_session"]["value"] assert merged_workers[0].backend_target == { "kind": "agent_id", "value": "agent-send-secret", @@ -590,8 +587,8 @@ def test_matched_pane_overrides_conflicting_agent_continuity_and_workspace( assert len(merged_bindings) == 1 assert merged_bindings[0].target_kind == "agent_id" assert merged_bindings[0].target_value == "agent-send-secret" - assert merged_bindings[0].turn_target_kind == "codex_session_id" - assert merged_bindings[0].turn_target_value == pane["agent_session"]["value"] + assert merged_bindings[0].turn_target_kind is None + assert merged_bindings[0].turn_target_value is None public = json.dumps(merged_workers[0].to_dict(), sort_keys=True) for private_value in ( @@ -654,8 +651,6 @@ def test_matched_incomplete_or_invalid_pane_suppresses_agent_identity_derivation assert merged_records[0].workspace_id == pane_records[0].workspace_id assert merged_records[0].pane_id == pane_records[0].pane_id assert merged_workers[0].space_id == pane_workers[0].space_id - assert merged_records[0].turn_target_kind == "codex_session_id" - assert merged_records[0].turn_target_value == pane["agent_session"]["value"] assert merged_workers[0].backend_target is not None assert merged_workers[0].backend_target["kind"] == "agent_id" assert merged_workers[0].backend_target["value"] == "agent-send-secret" @@ -683,8 +678,8 @@ def test_unmatched_agent_list_identity_never_authorizes_continuity( "sendable": True, "reason": None, } - assert bindings[0].turn_target_kind == "codex_session_id" - assert bindings[0].turn_target_value == agent["agent_session"]["value"] + assert bindings[0].turn_target_kind is None + assert bindings[0].turn_target_value is None public = json.dumps(workers[0].to_dict(), sort_keys=True) for private_value in ( @@ -729,8 +724,6 @@ def test_conflicting_match_keys_across_two_panes_fail_closed( assert len(records) == len(workers) == len(bindings) == 1 assert records[0].pane_info_observed is False - assert records[0].turn_target_kind is None - assert records[0].turn_target_value is None assert "stable_key" not in workers[0].meta assert "stable_key_version" not in workers[0].meta assert workers[0].backend_target == { @@ -787,8 +780,6 @@ def test_two_agents_claiming_one_pane_fail_closed_independent_of_order( assert len(records) == len(workers) == len(bindings) == 2 assert all(record.pane_info_observed is False for record in records) - assert all(record.turn_target_kind is None for record in records) - assert all(record.turn_target_value is None for record in records) assert all("stable_key" not in worker.meta for worker in workers) assert all("stable_key_version" not in worker.meta for worker in workers) assert all( @@ -888,8 +879,6 @@ def test_distinct_panes_with_shared_agent_owner_fail_closed_in_any_order( assert len(records) == len(workers) == len(bindings) == 2 assert all(record.pane_info_observed is False for record in records) - assert all(record.turn_target_kind is None for record in records) - assert all(record.turn_target_value is None for record in records) assert all("stable_key" not in worker.meta for worker in workers) assert all( worker.backend_target is not None @@ -957,8 +946,6 @@ def test_conflicting_pane_owner_key_fails_closed_independent_of_row_order( assert len(records) == len(workers) == len(bindings) == 2 assert all(record.pane_info_observed is False for record in records) - assert all(record.turn_target_kind is None for record in records) - assert all(record.turn_target_value is None for record in records) assert all("stable_key" not in worker.meta for worker in workers) assert all("stable_key_version" not in worker.meta for worker in workers) assert all( @@ -1010,8 +997,6 @@ def test_unmatched_agent_send_token_colliding_with_pane_fails_closed( assert len(records) == len(workers) == len(bindings) == 2 assert all(record.pane_info_observed is False for record in records) - assert all(record.turn_target_kind is None for record in records) - assert all(record.turn_target_value is None for record in records) assert all( worker.backend_target is not None and worker.backend_target["sendable"] is False @@ -1068,12 +1053,9 @@ def test_matched_pane_replaces_conflicting_pane_scoped_targets( } assert bindings[0].target_kind == "terminal_id" assert bindings[0].target_value == pane["terminal_id"] - assert bindings[0].turn_target_kind == "codex_session_id" - assert bindings[0].turn_target_value == pane["agent_session"]["value"] - assert agent["pane_id"] not in { - bindings[0].target_value, - bindings[0].turn_target_value, - } + assert bindings[0].turn_target_kind is None + assert bindings[0].turn_target_value is None + assert agent["pane_id"] != bindings[0].target_value public = json.dumps(workers[0].to_dict(), sort_keys=True) for private_value in ( From b88023f4b990bec0c6907b3020a85d32aaf42c72 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 20:37:41 +0800 Subject: [PATCH 72/83] docs: record Tendwire reduction baseline --- docs/reduction-baseline.md | 97 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 docs/reduction-baseline.md diff --git a/docs/reduction-baseline.md b/docs/reduction-baseline.md new file mode 100644 index 0000000..6ff2976 --- /dev/null +++ b/docs/reduction-baseline.md @@ -0,0 +1,97 @@ +# Tendwire reduction baseline + +This baseline was measured at commit `91891bf` on branch `wave-0/baseline` on +2026-08-04. The production root is `src/`. + +## SLOC calibration + +Canonical production SLOC was measured with: + +```console +python3 /home/smith/acp-reduction-goal/tools/sloc_count.py \ + /home/smith/tendwire/.worktrees/wave-0-baseline/src +``` + +The counter reports **57,007 SLOC**, exactly matching the stated baseline of +57,007: a difference of **0 lines (0.00%)**. The canonical checkout at +`/home/smith/tendwire/src` independently reports the same total. + +Physical lines are newline-terminated lines reported by `wc -l` for the same +production Python files. Canonical SLOC excludes blank lines, full-line +comments, and module/class/function docstrings according to +`tools/sloc_count.py`. Tests, docs, tooling, generated code, and vendored code +are outside both tables. + +### Directory summary + +| Directory | Physical lines | Canonical SLOC | +|---|---:|---:| +| `tendwire/` (direct modules) | 12,337 | 10,978 | +| `tendwire/backends/` | 14,761 | 12,724 | +| `tendwire/connectors/` | 686 | 624 | +| `tendwire/core/` | 6,884 | 5,925 | +| `tendwire/store/` | 28,239 | 26,756 | +| **Total** | **62,907** | **57,007** | + +### Per-module measurements + +| Module | Physical lines | Canonical SLOC | +|---|---:|---:| +| `tendwire/__init__.py` | 5 | 2 | +| `tendwire/_version.py` | 3 | 1 | +| `tendwire/backends/__init__.py` | 1 | 0 | +| `tendwire/backends/acp_client.py` | 1,688 | 1,483 | +| `tendwire/backends/acp_coordinator.py` | 2,413 | 2,148 | +| `tendwire/backends/acp_ingestion.py` | 714 | 612 | +| `tendwire/backends/acp_permissions.py` | 295 | 263 | +| `tendwire/backends/acp_probe.py` | 360 | 295 | +| `tendwire/backends/acp_projection.py` | 1,299 | 1,117 | +| `tendwire/backends/acp_protocol.py` | 613 | 480 | +| `tendwire/backends/acp_runtime.py` | 1,270 | 1,036 | +| `tendwire/backends/herdr_cli.py` | 2,671 | 2,304 | +| `tendwire/backends/herdr_command.py` | 156 | 125 | +| `tendwire/backends/herdr_events.py` | 2,454 | 2,209 | +| `tendwire/backends/herdr_protocol.py` | 363 | 260 | +| `tendwire/backends/herdr_socket.py` | 464 | 392 | +| `tendwire/cli.py` | 1,856 | 1,669 | +| `tendwire/command_submission.py` | 2,306 | 2,016 | +| `tendwire/config.py` | 726 | 686 | +| `tendwire/connectors/__init__.py` | 5 | 2 | +| `tendwire/connectors/outbox.py` | 681 | 622 | +| `tendwire/core/__init__.py` | 1 | 0 | +| `tendwire/core/actions.py` | 219 | 179 | +| `tendwire/core/agent_events.py` | 393 | 328 | +| `tendwire/core/attention.py` | 196 | 155 | +| `tendwire/core/commands.py` | 1,385 | 1,141 | +| `tendwire/core/models.py` | 1,961 | 1,706 | +| `tendwire/core/projector.py` | 68 | 51 | +| `tendwire/core/turns.py` | 2,661 | 2,365 | +| `tendwire/daemon.py` | 1,361 | 1,226 | +| `tendwire/daemon_api.py` | 1,833 | 1,706 | +| `tendwire/local_state.py` | 3,868 | 3,342 | +| `tendwire/store/__init__.py` | 1 | 0 | +| `tendwire/store/sqlite.py` | 28,238 | 26,756 | +| `tendwire/worker_identity.py` | 379 | 330 | +| **Total** | **62,907** | **57,007** | + +## Test baseline + +The existing repository virtual environment was used without installing or +upgrading anything. This worktree's source tree was selected explicitly: + +```console +env PYTHONPATH=/home/smith/tendwire/.worktrees/wave-0-baseline/src \ + /home/smith/tendwire/.venv/bin/python -m pytest tests/ -x -q +``` + +Result: + +```text +2879 passed, 2 skipped in 496.88s (0:08:16) +``` + +The authoritative run was performed outside the managed filesystem sandbox. +Inside that sandbox, a hardened Unix-socket test cannot bind through its +`/proc/self/fd/...` path and stopped the first run after 428 passes. The same +isolated test passed outside the sandbox (`1 passed in 3.06s`), after which the +unchanged full-suite command above completed green. From bf84e5487579835fee7770cb9dd780c714a1e788 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 21:16:24 +0800 Subject: [PATCH 73/83] delete dead command paths and v0 receipt replay --- src/tendwire/backends/herdr_command.py | 156 -------------- src/tendwire/command_submission.py | 45 +---- src/tendwire/core/actions.py | 85 -------- tests/test_actions.py | 270 ------------------------- tests/test_backend.py | 193 +----------------- tests/test_herdr_socket.py | 1 - 6 files changed, 4 insertions(+), 746 deletions(-) delete mode 100644 src/tendwire/backends/herdr_command.py diff --git a/src/tendwire/backends/herdr_command.py b/src/tendwire/backends/herdr_command.py deleted file mode 100644 index 3fb6d46..0000000 --- a/src/tendwire/backends/herdr_command.py +++ /dev/null @@ -1,156 +0,0 @@ -"""Narrow mutating command adapter for Herdr. - -Only the high-level ``herdr agent send `` API is used here. -This module must not fall back to pane control, key sending, shell commands, -PTY control, signals, paste buffers, raw argv, or client-provided backend -parameters. -""" - -from __future__ import annotations - -from dataclasses import dataclass - -import shutil -import subprocess -from typing import Any - -from ..config import Config -from ..core.commands import ( - STATUS_ACCEPTED, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_FAILED, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_REQUEST_STATE_UNCERTAIN, - error_value, - sanitize_command_result, -) -from ..core.models import _string_value - -_BACKEND_TARGET_KINDS = frozenset( - {"agent_id", "terminal_id", "pane_id", "agent", "name", "label"} -) - - -@dataclass(frozen=True) -class _HerdrCommandResult: - """Internal backend outcome, never a public command receipt envelope.""" - - ok: bool - status: str - result: dict[str, Any] | None = None - error: dict[str, Any] | None = None - - def to_dict(self) -> dict[str, Any]: - return { - "ok": self.ok, - "status": self.status, - "result": sanitize_command_result(self.result), - "error": sanitize_command_result(self.error), - } - - -def _run_agent_send( - config: Config, - target_value: str, - instruction_text: str, -) -> subprocess.CompletedProcess[str]: - """Run the single allowed Herdr send surface with an argv list.""" - return subprocess.run( - [config.herdr_bin, "agent", "send", target_value, instruction_text], - capture_output=True, - text=True, - check=False, - timeout=config.herdr_timeout_seconds, - ) - - -def _backend_error( - status: str, - message: str, - details: dict[str, Any] | None = None, -) -> _HerdrCommandResult: - return _HerdrCommandResult( - ok=False, - status=status, - error=error_value(status, message, details=details), - ) - - -def send_instruction( - config: Config, - target: dict[str, Any], - instruction: dict[str, Any], -) -> _HerdrCommandResult: - """Send instruction text to the backend-resolved private Herdr target.""" - backend_target = target.get("backend_target") - target_value = "" - target_kind = "" - target_reason = "" - if isinstance(backend_target, dict): - target_value = _string_value(backend_target.get("value")) - target_kind = _string_value(backend_target.get("kind")) - target_reason = _string_value(backend_target.get("reason")) - public_worker_id = _string_value(target.get("worker_id")) - instruction_text = instruction.get("text") - - if not isinstance(instruction_text, str) or not instruction_text: - return _backend_error( - STATUS_BACKEND_FAILED, - "instruction text is missing after validation", - ) - - try: - if shutil.which(config.herdr_bin) is None: - return _backend_error( - STATUS_BACKEND_UNAVAILABLE, - "Herdr binary is unavailable", - ) - except (OSError, TypeError, ValueError): - return _backend_error( - STATUS_BACKEND_UNAVAILABLE, - "Herdr binary is unavailable", - ) - - if ( - not isinstance(backend_target, dict) - or backend_target.get("sendable") is not True - or target_kind not in _BACKEND_TARGET_KINDS - or not target_value - ): - if target_reason in {"duplicate_backend_target", "not_unique"}: - return _backend_error( - STATUS_AMBIGUOUS_BACKEND_TARGET, - "resolved target is ambiguous for backend send", - ) - return _backend_error( - STATUS_BACKEND_UNSUPPORTED, - "resolved target has no backend-owned sendable target", - ) - - try: - completed = _run_agent_send(config, target_value, instruction_text) - except subprocess.TimeoutExpired: - return _backend_error( - STATUS_REQUEST_STATE_UNCERTAIN, - "Herdr agent send timed out after starting", - details={"timeout_seconds": config.herdr_timeout_seconds}, - ) - except (OSError, UnicodeDecodeError, ValueError, TypeError): - return _backend_error( - STATUS_BACKEND_UNAVAILABLE, - "Herdr agent send could not be launched", - ) - - if completed.returncode == 0: - return _HerdrCommandResult( - ok=True, - status=STATUS_ACCEPTED, - result={"target": {"worker_id": public_worker_id}}, - ) - - return _backend_error( - STATUS_BACKEND_FAILED, - "Herdr agent send exited non-zero", - details={"exit_code": int(completed.returncode)}, - ) diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 9e4366a..3f08ff9 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -75,7 +75,6 @@ _MUTATING_ACTIONS = frozenset( {"send_instruction", "answer_pending", "answer_decision"} ) -_LEGACY_V0_REPLAY_WORKER_ID = "legacy-v0-replay-only" _DISALLOWED_SEND_STATUSES = frozenset({"closed", "failed", "unknown"}) _AMBIGUOUS_BINDING_REASONS = frozenset({"duplicate_backend_target", "not_unique"}) @@ -521,11 +520,6 @@ def _receipt_is_canonical( ) if not common_identity: return False - if version == 0: - return ( - receipt.get("legacy_collision") is False - and receipt.get("canonical_fingerprint") == request.payload_fingerprint() - ) return ( version == canonical.canonical_version and receipt.get("canonical_fingerprint") == canonical.fingerprint @@ -566,33 +560,9 @@ def _stored_terminal_envelope( try: if type(schema_version) is not int: raise ValueError("stored envelope schema_version must be an exact integer") - if schema_version == COMMAND_ENVELOPE_SCHEMA_VERSION: - envelope = CommandEnvelope.from_dict(data) - elif schema_version == 1: - legacy_fields = { - "schema_version", - "action", - "request_id", - "ok", - "dry_run", - "status", - "result", - "error", - "warnings", - } - if set(data) != legacy_fields: - raise ValueError("legacy envelope has an invalid field set") - upgraded = dict(data) - upgraded["schema_version"] = COMMAND_ENVELOPE_SCHEMA_VERSION - upgraded["disposition"] = expected_disposition - envelope = CommandEnvelope.from_dict(upgraded) - roundtrip = envelope.to_dict() - roundtrip.pop("disposition") - roundtrip["schema_version"] = 1 - if roundtrip != data: - raise ValueError("legacy envelope is not an exact public roundtrip") - else: + if schema_version != COMMAND_ENVELOPE_SCHEMA_VERSION: raise ValueError("unsupported stored envelope schema") + envelope = CommandEnvelope.from_dict(data) except (TypeError, ValueError): return _backend_uncertain(request, malformed) @@ -1546,11 +1516,6 @@ def _proven_replay_worker_id( stored = receipt.get("public_worker_id") stored_worker_id = stored if isinstance(stored, str) and stored else "" - # A v0 receipt is validated against the exact raw request payload, which - # already pins the original selector spelling. It needs no resolution, and - # a changed payload fails its canonical check rather than replaying here. - if version == 0: - return stored_worker_id or _LEGACY_V0_REPLAY_WORKER_ID if not stored_worker_id: return _receipt_malformed(request) if request.action == "answer_decision": @@ -1577,7 +1542,7 @@ def _proven_replay_worker_id( # 3. Only a current, healthy observation can prove that a different spelling # names the same canonical worker. A degraded one proves nothing, and a - # legacy receipt carries no proof to fall back on. + # receipt without a proof has no fallback evidence. if not allow_current_authority: return None try: @@ -1631,10 +1596,6 @@ def _receipt_authority( if replay.status == STATUS_DUPLICATE_REQUEST: # A changed canonical mutation never rewrites the original receipt. return replay - if receipt.get("canonical_version") == 0: - # Legacy evidence cannot be re-expressed as a canonical v1 row, so - # replay it as read instead of inventing one. - return replay return _reserve_terminal_replay(config, request, canonical, receipt, replay) if replay.status != STATUS_PENDING: return replay diff --git a/src/tendwire/core/actions.py b/src/tendwire/core/actions.py index 373633b..06b7713 100644 --- a/src/tendwire/core/actions.py +++ b/src/tendwire/core/actions.py @@ -13,9 +13,6 @@ from ..config import Config from .commands import ( STATUS_AMBIGUOUS_TARGET, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DRY_RUN, STATUS_NOOP, STATUS_NOT_FOUND, STATUS_REJECTED, @@ -30,7 +27,6 @@ resolve_target, snapshot_result, validate_request, - worker_candidate, ) from .projector import project_from_observations @@ -110,84 +106,6 @@ def _resolve_target_result(request: CommandRequest, workers: list[Worker]) -> Co ) -def _send_instruction_result(request: CommandRequest, context: CommandContext) -> CommandEnvelope: - resolved, candidates, status = resolve_target( - request.target, - context.workers, - allow_disallowed_status=True, - include_backend_target=True, - ) - if status != STATUS_RESOLVED: - return _resolve_target_result(request, context.workers) - - # Even though resolve_target succeeded, send_instruction must reject workers - # whose current status is closed, failed, or unknown. - resolved_worker = next( - (w for w in context.workers if w.id == (resolved or {}).get("worker_id")), - None, - ) - disallowed = {"closed", "failed", "unknown"} - if resolved_worker is not None and resolved_worker.status in disallowed: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_REJECTED, - result={"candidates": [worker_candidate(resolved_worker)]}, - error=error_value( - STATUS_REJECTED, - f"target worker status does not allow instructions: {resolved_worker.status!r}", - ), - ) - - instruction = request.instruction or {} - target = resolved or {} - text = instruction.get("text", "") - - if request.dry_run: - public_target = worker_candidate(resolved_worker) if resolved_worker is not None else target - return CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_DRY_RUN, - result={"target": public_target, "instruction": {"text": text}}, - ) - - backend_target = target.get("backend_target") - backend_reason = "" - if isinstance(backend_target, dict): - backend_reason = str(backend_target.get("reason") or "") - if not isinstance(backend_target, dict) or backend_target.get("sendable") is not True: - if backend_reason in {"duplicate_backend_target", "not_unique"}: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_AMBIGUOUS_BACKEND_TARGET, - error=error_value( - STATUS_AMBIGUOUS_BACKEND_TARGET, - "target resolves to an ambiguous backend send target", - ), - ) - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_BACKEND_UNSUPPORTED, - error=error_value( - STATUS_BACKEND_UNSUPPORTED, - "target has no backend-owned sendable private binding", - ), - ) - - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_BACKEND_UNSUPPORTED, - error=error_value( - STATUS_BACKEND_UNSUPPORTED, - "live mutations require the authoritative command submission path", - ), - ) - - def execute_command(request: CommandRequest, context: CommandContext) -> CommandEnvelope: """Execute a validated command request and return a neutral envelope.""" validation_error = validate_request(request) @@ -210,9 +128,6 @@ def execute_command(request: CommandRequest, context: CommandContext) -> Command if request.action == "resolve_target": return _resolve_target_result(request, context.workers) - if request.action == "send_instruction": - return _send_instruction_result(request, context) - return CommandEnvelope.from_error( request, error_value(STATUS_REJECTED, f"unknown action {request.action!r}"), diff --git a/tests/test_actions.py b/tests/test_actions.py index c571acb..f95f4a9 100644 --- a/tests/test_actions.py +++ b/tests/test_actions.py @@ -5,15 +5,9 @@ import json from typing import Any -import pytest - from tendwire.config import Config from tendwire.core.actions import CommandContext, execute_command from tendwire.core.commands import ( - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DRY_RUN, - STATUS_INVALID_REQUEST, STATUS_NOT_FOUND, STATUS_REJECTED, STATUS_RESOLVED, @@ -44,52 +38,6 @@ def _workers(snapshot: Snapshot) -> list[Worker]: return list(snapshot.workers) -def _sendable_worker( - worker_id: str, - name: str, - *, - status: str = "active", - space_id: str | None = "s-1", - target_value: str | None = None, - sendable: bool = True, - reason: str | None = None, -) -> Worker: - return Worker( - id=worker_id, - name=name, - status=status, - space_id=space_id, - backend_target={ - "kind": "agent_id", - "value": target_value or f"agent-{worker_id}", - "sendable": sendable, - "reason": reason, - }, - ) - - -def _workers_with_backend_targets(snapshot: Snapshot) -> list[Worker]: - return [ - Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - fingerprint=worker.fingerprint, - backend_target={ - "kind": "agent_id", - "value": f"agent-{worker.id}", - "sendable": True, - "reason": None, - }, - ) - for worker in snapshot.workers - ] - - def test_noop_action_succeeds() -> None: request = CommandRequest(action="noop") context = CommandContext(host_id="host", workers=[]) @@ -170,224 +118,6 @@ def test_resolve_target_disallowed_status() -> None: assert envelope.status == STATUS_REJECTED -def test_send_instruction_dry_run_is_pure() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - dry_run=True, - ) - context = CommandContext( - host_id=snapshot.host_id, - workers=_workers_with_backend_targets(snapshot), - ) - envelope = execute_command(request, context) - assert envelope.ok is True - assert envelope.status == STATUS_DRY_RUN - assert envelope.result == { - "target": { - "worker_id": "w-1", - "name": "Alpha", - "space_id": "s-1", - "status": "active", - "worker_fingerprint": snapshot.workers[0].fingerprint, - }, - "instruction": {"text": "hello"}, - } - - -def test_send_instruction_non_dry_run_requires_request_id() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - envelope = execute_command(request, context) - assert envelope.ok is False - assert envelope.status == STATUS_INVALID_REQUEST - - -def test_send_instruction_non_dry_run_returns_backend_unsupported() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-1", - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - envelope = execute_command(request, context) - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - assert envelope.request_id == "req-1" - assert envelope.dry_run is False - - -def test_send_instruction_without_sendable_backend_target_is_unsupported() -> None: - request = CommandRequest( - action="send_instruction", - request_id="req-no-binding", - dry_run=False, - target={"worker_id": "w-no-binding"}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id="host", - workers=[ - _sendable_worker( - "w-no-binding", - "No Binding", - sendable=False, - reason="backend_unsupported", - ) - ], - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - - -def test_send_instruction_ambiguous_backend_target_is_rejected() -> None: - request = CommandRequest( - action="send_instruction", - request_id="req-ambiguous-binding", - dry_run=False, - target={"worker_id": "w-ambiguous"}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id="host", - workers=[ - _sendable_worker( - "w-ambiguous", - "Ambiguous", - sendable=False, - reason="duplicate_backend_target", - ) - ], - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_AMBIGUOUS_BACKEND_TARGET - - -def test_send_instruction_rejects_empty_target_before_resolution() -> None: - request = CommandRequest( - action="send_instruction", - request_id="req-empty", - dry_run=False, - target={}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id="host", - workers=[Worker(id="only-worker", name="Only", status="active")], - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_INVALID_REQUEST - - -def test_send_instruction_done_worker_still_requires_authoritative_submission() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-done", - dry_run=False, - target={"worker_id": "w-5"}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id=snapshot.host_id, - workers=_workers_with_backend_targets(snapshot), - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - assert envelope.error is not None - assert "authoritative command submission" in envelope.error["message"] - - -@pytest.mark.parametrize("worker_id", ["w-4", "w-6", "w-7", "w-8"]) -def test_send_instruction_rejects_closed_failed_unknown_statuses(worker_id: str) -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id=f"req-{worker_id}", - dry_run=False, - target={"worker_id": worker_id}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_REJECTED - - -def test_send_instruction_resolves_selector_but_does_not_mutate() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-1", - dry_run=False, - target={"name": "Beta"}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id=snapshot.host_id, - workers=_workers_with_backend_targets(snapshot), - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - -def test_send_instruction_respects_ambiguous_target_before_backend() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-1", - dry_run=False, - target={"name": "Alpha"}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - envelope = execute_command(request, context) - assert envelope.ok is False - assert envelope.status == "ambiguous_target" - - -def test_send_instruction_respects_rejected_status_before_backend() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-1", - dry_run=False, - target={"worker_id": "w-4"}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - envelope = execute_command(request, context) - assert envelope.ok is False - assert envelope.status == STATUS_REJECTED - - def test_public_result_contains_no_connector_fields() -> None: snapshot = _snapshot() request = CommandRequest(action="read_snapshot") diff --git a/tests/test_backend.py b/tests/test_backend.py index fb9472f..fa82967 100644 --- a/tests/test_backend.py +++ b/tests/test_backend.py @@ -11,18 +11,9 @@ import pytest from tendwire import cli as tendwire_cli -from tendwire.backends import herdr_cli, herdr_command +from tendwire.backends import herdr_cli from tendwire.backends.herdr_cli import fetch_herdr_state from tendwire.config import Config -from tendwire.core.commands import ( - STATUS_ACCEPTED, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_FAILED, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_REQUEST_STATE_UNCERTAIN, - CommandEnvelope, -) from tendwire.core.models import Worker, WorkerBinding, worker_binding_private_fingerprint from tendwire.core.projector import project_from_observations from tendwire.store.sqlite import init_store, list_worker_bindings @@ -103,188 +94,6 @@ def _assert_no_forbidden_fields(value: Any, path: str = "$") -> None: _assert_no_forbidden_fields(item, f"{path}[{index}]") -def _send_completed(returncode: int = 0) -> subprocess.CompletedProcess[str]: - return subprocess.CompletedProcess( - args=["herdr", "agent", "send", "worker-1", "hello"], - returncode=returncode, - stdout="", - stderr="", - ) - - -def test_send_instruction_uses_agent_send_argv(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[list[str], dict[str, Any]]] = [] - instruction_text = "line one\nline two\tindented" - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append((args, kwargs)) - return _send_completed() - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command.subprocess, "run", fake_run) - - envelope = herdr_command.send_instruction( - config, - { - "worker_id": "public-worker-1", - "backend_target": {"kind": "agent_id", "value": "agent-send-1", "sendable": True, "reason": None}, - "terminal_id": "ignored", - }, - {"text": instruction_text}, - ) - - assert envelope.ok is True - assert envelope.status == STATUS_ACCEPTED - assert envelope.result == {"target": {"worker_id": "public-worker-1"}} - assert not isinstance(envelope, CommandEnvelope) - assert set(envelope.to_dict()) == {"ok", "status", "result", "error"} - assert calls == [ - ( - ["herdr", "agent", "send", "agent-send-1", instruction_text], - { - "capture_output": True, - "text": True, - "check": False, - "timeout": config.herdr_timeout_seconds, - }, - ) - ] - assert "shell" not in calls[0][1] - - -def test_send_instruction_requires_private_backend_target(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[Any] = [] - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", lambda *args: calls.append(args)) - - envelope = herdr_command.send_instruction( - config, - {"worker_id": "public-worker-1", "agent_session": {"value": "sess-ignored"}}, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - assert calls == [] - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_maps_missing_binary_to_backend_unavailable(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="missing-herdr") - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: None) - monkeypatch.setattr( - herdr_command, - "_run_agent_send", - lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("should not run")), - ) - - envelope = herdr_command.send_instruction( - config, - {"worker_id": "worker-1", "backend_target": {"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}}, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_maps_nonzero_exit_to_backend_failed(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", lambda *args: _send_completed(returncode=2)) - - envelope = herdr_command.send_instruction( - config, - {"worker_id": "worker-1", "backend_target": {"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}}, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_FAILED - assert envelope.error["details"] == {"exit_code": 2} - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_maps_timeout_to_uncertain(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - - def raise_timeout(*args: Any) -> subprocess.CompletedProcess[str]: - raise subprocess.TimeoutExpired(cmd=["herdr", "agent", "send"], timeout=5.0) - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", raise_timeout) - - envelope = herdr_command.send_instruction( - config, - {"worker_id": "worker-1", "backend_target": {"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}}, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_REQUEST_STATE_UNCERTAIN - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_rejects_ambiguous_private_backend_target(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[Any] = [] - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", lambda *args: calls.append(args)) - - envelope = herdr_command.send_instruction( - config, - { - "worker_id": "public-worker-1", - "backend_target": { - "kind": "agent_id", - "value": "agent-1", - "sendable": False, - "reason": "duplicate_backend_target", - }, - }, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_AMBIGUOUS_BACKEND_TARGET - assert calls == [] - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_rejects_unsupported_private_backend_target_kind(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[Any] = [] - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", lambda *args: calls.append(args)) - - envelope = herdr_command.send_instruction( - config, - { - "worker_id": "public-worker-1", - "backend_target": { - "kind": "session_id", - "value": "session-must-not-send", - "sendable": True, - "reason": None, - }, - }, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - assert calls == [] - serialized = json.dumps(envelope.to_dict()) - assert "session-must-not-send" not in serialized - _assert_no_forbidden_fields(envelope.to_dict()) - - def test_fetch_herdr_state_returns_empty_when_binary_missing() -> None: config = Config(host_id="testhost", herdr_bin="definitely-not-a-real-herdr-binary") spaces, workers = fetch_herdr_state(config) diff --git a/tests/test_herdr_socket.py b/tests/test_herdr_socket.py index b9011f1..0a7ff3e 100644 --- a/tests/test_herdr_socket.py +++ b/tests/test_herdr_socket.py @@ -586,7 +586,6 @@ def test_existing_production_backend_files_do_not_import_socket_client() -> None for relative in ( "src/tendwire/cli.py", "src/tendwire/backends/herdr_cli.py", - "src/tendwire/backends/herdr_command.py", ): text = (root / relative).read_text(encoding="utf-8") assert "herdr_socket" not in text From 67c4606fc3dfe5b1d265c25f159172ed878ac9ae Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 21:19:57 +0800 Subject: [PATCH 74/83] refactor: make Herdr socket lifecycle-only --- src/tendwire/backends/acp_coordinator.py | 367 ++- src/tendwire/backends/herdr_cli.py | 2671 ------------------ src/tendwire/backends/herdr_events.py | 2454 ---------------- src/tendwire/backends/herdr_protocol.py | 132 +- src/tendwire/backends/herdr_socket.py | 212 +- src/tendwire/cli.py | 643 +---- src/tendwire/command_submission.py | 5 - src/tendwire/config.py | 62 +- src/tendwire/daemon.py | 103 +- tests/test_acp_coordinator.py | 19 +- tests/test_acp_permissions.py | 1 - tests/test_backend.py | 1598 ----------- tests/test_cli.py | 3223 +-------------------- tests/test_cli_command.py | 2638 +---------------- tests/test_config.py | 96 +- tests/test_connector_daemon_cli.py | 116 - tests/test_daemon.py | 429 +-- tests/test_daemon_acp.py | 10 +- tests/test_diagnostics.py | 1394 --------- tests/test_herdr_events.py | 3259 ---------------------- tests/test_herdr_protocol.py | 278 +- tests/test_herdr_smoke.py | 678 ----- tests/test_herdr_socket.py | 660 +---- tests/test_local_state_permissions.py | 2 +- tests/test_release_readiness.py | 39 +- tests/test_turn_delta.py | 16 +- tests/test_worker_label_and_model.py | 178 +- tests/test_worker_stable_key.py | 2479 ++-------------- 28 files changed, 952 insertions(+), 22810 deletions(-) delete mode 100644 src/tendwire/backends/herdr_cli.py delete mode 100644 src/tendwire/backends/herdr_events.py delete mode 100644 tests/test_backend.py delete mode 100644 tests/test_diagnostics.py delete mode 100644 tests/test_herdr_events.py delete mode 100644 tests/test_herdr_smoke.py diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index 9168481..a9a678d 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -12,7 +12,8 @@ import json import threading import time -from collections.abc import Callable, Mapping +from collections import Counter +from collections.abc import Callable, Mapping, Sequence from contextlib import contextmanager from concurrent.futures import Future, ThreadPoolExecutor from dataclasses import dataclass, field, replace @@ -20,7 +21,16 @@ from typing import Any from ..config import Config -from ..core.models import Worker, WorkerBinding, utc_timestamp +from ..core.models import ( + BackendHealth, + Space, + Worker, + WorkerBinding, + normalize_status, + separate_duplicate_worker_bindings, + utc_timestamp, + worker_binding_private_fingerprint, +) from ..core.commands import turn_submission_id from ..core.models import stable_fingerprint from ..store.sqlite import ( @@ -30,8 +40,16 @@ list_worker_bindings, pending_payload_from_store, record_agent_event, + save_snapshot, upsert_worker_bindings, ) +from ..core.projector import project_from_observations +from ..worker_identity import ( + STABLE_KEY_VERSION, + canonical_herdr_pane_identity, + load_or_create_installation_key, + stable_worker_key, +) from .acp_client import BoundedAcpConnection from .acp_permissions import AcpPermissionBroker from .acp_runtime import ( @@ -209,6 +227,7 @@ def __init__( stop_event: threading.Event, *, endpoint_client_factory: EndpointClientFactory | None = None, + discovery_client_factory: EndpointClientFactory | None = None, session_factory: WorkerSessionFactory = AcpWorkerSession, connection_factory: ConnectionFactory = BoundedAcpConnection, reconcile_interval: float | None = None, @@ -227,6 +246,9 @@ def __init__( self._endpoint_client_factory = ( endpoint_client_factory or _default_endpoint_client_factory ) + self._discovery_client_factory = discovery_client_factory + if self._discovery_client_factory is None and endpoint_client_factory is None: + self._discovery_client_factory = _default_endpoint_client_factory self._session_factory = session_factory self._connection_factory = connection_factory self._permission_callback = permission_callback @@ -261,6 +283,8 @@ def __init__( # Exact ACP ownership survives runtime retirement so an outage remains # distinguishable from a worker that Herdr has actually removed. self._published_acp_claims: dict[str, str] = {} + self._last_discovery_at: str | None = None + self._worker_count = 0 def start(self) -> "AcpSupervisor": with self._lock: @@ -451,6 +475,8 @@ def status(self) -> dict[str, Any]: "state": state.value, "healthy": healthy, "failure_type": failure_type, + "last_reconcile_at": self._last_discovery_at, + "worker_count": self._worker_count, **counters, } @@ -1113,6 +1139,16 @@ def _reconcile(self, *, strict: bool) -> None: self._stop_all() def _reconcile_locked(self, *, strict: bool) -> None: + if self._discovery_client_factory is not None: + try: + self._discover_continuity() + except Exception as exc: + with self._lock: + self._required_degraded = True + self._failure_type = type(exc).__name__ + if strict: + raise + return current, ambiguities = self._continuity_bindings() with self._lock: failed_claims = tuple(self._console_failed_claims.items()) @@ -1170,6 +1206,67 @@ def _reconcile_locked(self, *, strict: bool) -> None: if strict and failures: raise AcpCoordinatorError("one or more ACP workers failed to attach") + def _discover_continuity(self) -> None: + """Refresh the one Herdr lifecycle projection consumed by ACP routing.""" + + client = self._discovery_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + workspaces = client.workspace_list(timeout=self.config.herdr_timeout_seconds) + panes = client.pane_list(timeout=self.config.herdr_timeout_seconds) + agents = client.agent_list(timeout=self.config.herdr_timeout_seconds) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + + observed_at = utc_timestamp() + spaces = _discovered_spaces(workspaces) + prior_bindings = list_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="herdr", + ) + workers, bindings = _discovered_workers( + self.config, + panes, + agents, + observed_at, + prior_bindings=prior_bindings, + ) + health = BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty" if spaces or workers else "empty_healthy", + observed_at=observed_at, + counts={"spaces": len(spaces), "workers": len(workers)}, + ) + snapshot = project_from_observations( + self.config, + spaces=spaces, + workers=workers, + backend_health=[health], + ) + from ..store.sqlite import SnapshotObservationContext + + save_snapshot( + Path(self.config.db_path), + snapshot, + observation=SnapshotObservationContext( + authority="complete", + observed_at=observed_at, + ), + worker_bindings=bindings, + binding_backend="herdr", + binding_observation_authoritative=True, + binding_workers_present=bool(workers), + ) + with self._lock: + self._last_discovery_at = observed_at + self._worker_count = len(workers) + def _reconcile_worker(self, worker_id: str, *, strict: bool) -> None: with self._reconcile_lock: try: @@ -1682,6 +1779,272 @@ def _same_continuity(left: WorkerBinding, right: WorkerBinding) -> bool: ) +def _field(item: Mapping[str, Any], name: str) -> Any: + expected = name.replace("_", "").lower() + for key, value in item.items(): + if str(key).replace("_", "").replace("-", "").lower() == expected: + return value + return None + + +def _text(item: Mapping[str, Any], *names: str) -> str | None: + for name in names: + value = _field(item, name) + if isinstance(value, (str, int)) and not isinstance(value, bool): + text = str(value).strip() + if text: + return text + return None + + +def _items(payload: Any, *names: str) -> list[dict[str, Any]]: + if isinstance(payload, list): + values = payload + elif isinstance(payload, Mapping): + values = [] + for name in (*names, "data", "payload"): + candidate = _field(payload, name) + if isinstance(candidate, list): + values = candidate + break + if isinstance(candidate, Mapping): + nested = _items(candidate, *names) + if nested: + return nested + else: + return [] + return [dict(value) for value in values if isinstance(value, Mapping)] + + +def _discovered_spaces(payload: Any) -> list[Space]: + spaces: list[Space] = [] + for item in _items(payload, "workspaces", "spaces", "items", "result"): + space_id = _text(item, "workspace_id", "space_id", "id", "name") + if space_id is None: + continue + spaces.append( + Space( + id=space_id, + name=_text(item, "label", "name", "title") or space_id, + status=normalize_status(_text(item, "status", "state")), + updated_at=_text(item, "updated_at", "observed_at"), + ) + ) + return spaces + + +def _agent_match( + pane: Mapping[str, Any], + agents: Sequence[Mapping[str, Any]], +) -> tuple[Mapping[str, Any] | None, bool]: + pane_id = _text(pane, "pane_id") + terminal_id = _text(pane, "terminal_id") + matches = [ + agent + for agent in agents + if (pane_id and _text(agent, "pane_id") == pane_id) + or (terminal_id and _text(agent, "terminal_id") == terminal_id) + ] + return (matches[0], False) if len(matches) == 1 else (None, len(matches) > 1) + + +def _assign_prior_worker_ids( + rows: list[dict[str, Any]], + prior_bindings: Sequence[WorkerBinding], +) -> None: + private_counts = Counter(str(row["private_fingerprint"]) for row in rows) + target_key_counts = Counter( + (str(row["target_kind"]), str(row["target_value"])) for row in rows + ) + stored_by_private: dict[str, list[WorkerBinding]] = {} + stored_by_target: dict[tuple[str, str], list[WorkerBinding]] = {} + for binding in prior_bindings: + if ( + binding.backend != "herdr" + or not binding.worker_id + or binding.reason in {"duplicate_backend_target", "ambiguous_pane_match"} + ): + continue + stored_by_private.setdefault(binding.private_fingerprint, []).append(binding) + stored_by_target.setdefault( + (binding.target_kind, binding.target_value), [] + ).append(binding) + for row in rows: + prior: WorkerBinding | None = None + private_fingerprint = str(row["private_fingerprint"]) + private_candidates = stored_by_private.get(private_fingerprint, []) + if private_counts[private_fingerprint] == 1 and len(private_candidates) == 1: + prior = private_candidates[0] + target_key = (str(row["target_kind"]), str(row["target_value"])) + target_candidates = stored_by_target.get(target_key, []) + if prior is None and target_key_counts[target_key] == 1 and len(target_candidates) == 1: + prior = target_candidates[0] + row["desired_id"] = prior.worker_id if prior is not None else row["base_id"] + + +def _materialize_discovered_workers( + config: Config, + rows: list[dict[str, Any]], + observed_at: str, +) -> tuple[list[Worker], list[WorkerBinding]]: + target_counts = Counter(str(row["target_value"]) for row in rows) + id_counts = Counter(str(row["desired_id"]) for row in rows) + workers: list[Worker] = [] + bindings: list[WorkerBinding] = [] + duplicate_indexes: dict[str, int] = {} + for row in sorted(rows, key=lambda value: str(value["private_fingerprint"])): + base_id = str(row["desired_id"]) + worker_id = base_id + if id_counts[base_id] > 1: + duplicate_indexes[base_id] = duplicate_indexes.get(base_id, 0) + 1 + worker_id = f"{base_id}-{duplicate_indexes[base_id]}" + pane = row["pane"] + agent = row["agent"] + meta: dict[str, Any] = {} + label = _text(pane, "label") + if label: + meta["label"] = label + stable_key = row["stable_key"] + if stable_key is not None: + meta["stable_key"] = stable_key + meta["stable_key_version"] = STABLE_KEY_VERSION + reason = row["reason"] + sendable = reason is None and target_counts[str(row["target_value"])] == 1 + if not sendable and reason is None: + reason = "duplicate_backend_target" + worker = Worker( + id=worker_id, + name=( + _text(agent, "agent", "name") + or _text(pane, "agent", "label", "name") + or worker_id + ), + status=normalize_status( + _text(agent, "status", "agent_status") + or _text(pane, "agent_status", "status") + ), + space_id=_text(pane, "workspace_id"), + meta=meta, + last_seen_at=observed_at, + backend_target={ + "kind": row["target_kind"], + "value": row["target_value"], + "sendable": sendable, + "reason": reason, + }, + ) + workers.append(worker) + bindings.append( + WorkerBinding( + host_id=config.host_id, + worker_id=worker.id, + worker_fingerprint=worker.fingerprint, + backend="herdr", + target_kind=str(row["target_kind"]), + target_value=str(row["target_value"]), + sendable=sendable, + reason=reason, + observed_at=observed_at, + expires_at=None, + private_fingerprint=str(row["private_fingerprint"]), + ) + ) + separated = separate_duplicate_worker_bindings(bindings) + final_workers: list[Worker] = [] + final_bindings: list[WorkerBinding] = [] + for worker, binding in zip(workers, separated, strict=True): + if worker.backend_target != binding.backend_target(): + worker = replace(worker, backend_target=binding.backend_target()) + binding = replace(binding, worker_fingerprint=worker.fingerprint) + final_workers.append(worker) + final_bindings.append(binding) + return final_workers, final_bindings + + +def _discovered_workers( + config: Config, + pane_payload: Any, + agent_payload: Any, + observed_at: str, + *, + prior_bindings: Sequence[WorkerBinding] = (), +) -> tuple[list[Worker], list[WorkerBinding]]: + panes = _items(pane_payload, "panes", "items", "result") + agents = _items(agent_payload, "agents", "workers", "items", "result") + rows: list[dict[str, Any]] = [] + needs_stable_key = False + for pane in panes: + agent, ambiguous_agent = _agent_match(pane, agents) + if agent is None and not ambiguous_agent and not _text(pane, "agent", "name", "label"): + continue + workspace_id = _text(pane, "workspace_id") + pane_id = _text(pane, "pane_id") + identity = canonical_herdr_pane_identity(workspace_id, pane_id) + needs_stable_key = needs_stable_key or identity is not None + target_kind = "" + target_value = "" + for kind, value in ( + ("agent_id", _text(agent or {}, "agent_id")), + ("terminal_id", _text(pane, "terminal_id")), + ("pane_id", pane_id), + ): + if value: + target_kind, target_value = kind, value + break + if not target_value: + continue + base_id = ( + _text(agent or {}, "agent", "name") + or _text(pane, "agent", "name", "label") + or "worker" + ) + private_fingerprint = worker_binding_private_fingerprint( + host_id=config.host_id, + backend="herdr", + identity_material={ + "workspace_id": workspace_id, + "pane_id": pane_id, + "terminal_id": _text(pane, "terminal_id"), + "agent_id": _text(agent or {}, "agent_id"), + }, + ) + rows.append( + { + "agent": agent or {}, + "base_id": base_id, + "identity": identity, + "pane": pane, + "private_fingerprint": private_fingerprint, + "target_kind": target_kind, + "target_value": target_value, + "reason": ( + "ambiguous_pane_match" + if ambiguous_agent + else ("invalid_pane_identity" if identity is None else None) + ), + } + ) + + installation_key = ( + load_or_create_installation_key(config.data_dir) if needs_stable_key else None + ) + for row in rows: + identity = row["identity"] + if installation_key is not None and identity is not None: + workspace_id, pane_id = identity + row["stable_key"] = stable_worker_key( + installation_key, + backend="herdr", + host_id=config.host_id, + workspace_id=workspace_id, + pane_id=pane_id, + ) + else: + row["stable_key"] = None + _assign_prior_worker_ids(rows, prior_bindings) + return _materialize_discovered_workers(config, rows, observed_at) + + def _nonempty_text(value: Any, field: str) -> str: if not isinstance(value, str) or not value or value.strip() != value: raise AcpCoordinatorError(f"Herdr ACP endpoint {field} is invalid") diff --git a/src/tendwire/backends/herdr_cli.py b/src/tendwire/backends/herdr_cli.py deleted file mode 100644 index a426485..0000000 --- a/src/tendwire/backends/herdr_cli.py +++ /dev/null @@ -1,2671 +0,0 @@ -"""Thin adapter boundary around the Herdr CLI. - -This module shells out read-only to a `herdr` binary when available and parses -output on a best-effort basis. If the binary is missing or fails, it returns -empty neutral data rather than blocking the snapshot contract. - -This module must not import Herdres code or leak delivery/routing state into -core models. -""" - -from __future__ import annotations - -import hashlib -import json -import shutil -import subprocess -import time -from collections import Counter -from collections.abc import Iterable, Mapping, Sequence -from dataclasses import dataclass, field, replace -from datetime import datetime, timedelta, timezone -from typing import Any - -from ..config import Config -from ..core.models import ( - BackendHealth, - Space, - Worker, - WorkerBinding, - normalize_status, - separate_duplicate_worker_bindings, - sanitize_public_text, - stable_fingerprint, - utc_timestamp, - worker_binding_private_fingerprint, -) -from ..local_state import ( - ConfigStateReport, - LocalStateErrorCode, - LocalStateKind, - PermissionState, - inspect_config_state, -) -from ..worker_identity import ( - InstallationKeyError, - STABLE_KEY_VERSION, - canonical_herdr_pane_identity, - load_or_create_installation_key, - stable_worker_key, -) - - -_HERDR_TIMEOUT_SECONDS = 5.0 -_BACKEND_NAME = "herdr" -_AMBIGUOUS_BINDING_REASONS = frozenset( - {"ambiguous_pane_match", "duplicate_backend_target", "not_unique"} -) - - -class HerdrContinuityUnavailableError(RuntimeError): - """A healthy-looking Herdr observation cannot authenticate worker continuity.""" - - -@dataclass(frozen=True) -class _WorkerRecord: - worker: Worker - private_fingerprint: str - # Canonical public Herdr identity used exclusively for continuity. Raw - # observations remain private and separate so routing compatibility can - # never accidentally feed stable-key derivation. - workspace_id: str | None = None - pane_id: str | None = None - observed_workspace_id: str | None = None - observed_pane_id: str | None = None - identity_source: str = "unknown" - terminal_id: str | None = None - agent_session_id: str | None = None - # Continuity is authorized only by a PaneInfo observation, never by - # workspace/pane-shaped fields reported by agent.list. - pane_info_observed: bool = False - unmatched_agent_observation: bool = False - -_FORBIDDEN_CONNECTOR_FIELDS = { - "telegram", - "chat_id", - "topic_id", - "message_id", - "thread_id", - "token", - "bot_token", - "delivery", - "route", - "herdres_delivery", - "backend_target", -} - -_STATUS_KEYS = ( - "agent_status", - "status", - "state", - "phase", - "lifecycle", - "lifecycle_state", - "raw_status", -) - -_BACKEND_TARGET_KINDS = frozenset( - {"agent_id", "terminal_id", "pane_id", "agent", "name", "label"} -) -_AGENT_SCOPED_BACKEND_TARGET_KINDS = frozenset({"agent_id", "agent"}) -_DEADLINE_EXHAUSTED_OUTCOMES = frozenset({"timeout", "deadline_exhausted"}) -_UNAVAILABLE_HEALTH_OUTCOMES = frozenset({"missing_binary", "launch_error", "socket_disconnected"}) -_DEGRADED_HEALTH_OUTCOMES = frozenset( - { - "timeout", - "deadline_exhausted", - "nonzero", - "malformed_json", - "protocol_error", - "worker_cap_exceeded", - "continuity_unavailable", - } -) - -_HEALTH_MESSAGES = { - "healthy_non_empty": "Herdr observation is healthy", - "empty_healthy": "Herdr observation is healthy but empty", - "missing_binary": "Herdr binary is unavailable", - "launch_error": "Herdr launch failed", - "timeout": "Herdr observation timed out", - "deadline_exhausted": "Herdr observation deadline was exhausted", - "nonzero": "Herdr command returned nonzero status", - "malformed_json": "Herdr command returned malformed JSON", - "protocol_error": "Herdr protocol returned an invalid envelope", - "socket_disconnected": "Herdr socket disconnected", - "worker_cap_exceeded": "Herdr observation exceeded the configured worker cap", - "continuity_unavailable": "Herdr continuity identity is unavailable", - "unknown": "Herdr observation state is unknown", -} - - -@dataclass(frozen=True) -class _ProbeBudget: - """Aggregate deadline for a read-only Herdr observation chain.""" - - started_at: float - per_probe_timeout_seconds: float - aggregate_deadline_seconds: float - - @classmethod - def from_config(cls, config: Config, *, planned_probes: int) -> "_ProbeBudget": - per_probe = float(config.herdr_timeout_seconds) - planned = max(1, int(planned_probes)) - return cls( - started_at=time.monotonic(), - per_probe_timeout_seconds=per_probe, - aggregate_deadline_seconds=per_probe * planned, - ) - - def remaining_seconds(self) -> float: - return self.aggregate_deadline_seconds - (time.monotonic() - self.started_at) - - def subprocess_timeout_seconds(self) -> float | None: - remaining = self.remaining_seconds() - if remaining <= 0: - return None - if remaining >= self.per_probe_timeout_seconds: - return self.per_probe_timeout_seconds - return max(0.001, remaining) - - -def herdr_health_status_for_outcome(outcome: str) -> str: - """Map a Herdr adapter outcome into the public backend health status.""" - normalized = str(outcome or "unknown").strip().lower().replace("-", "_") - if normalized in {"healthy_non_empty", "empty_healthy"}: - return "healthy" - if normalized in _UNAVAILABLE_HEALTH_OUTCOMES: - return "unavailable" - if normalized in _DEGRADED_HEALTH_OUTCOMES: - return "degraded" - return "unknown" - - -def herdr_backend_health( - outcome: str, - *, - observed_at: str | None = None, - message: str | None = None, - spaces: Sequence[Space] | None = None, - workers: Sequence[Worker] | None = None, -) -> BackendHealth: - """Return the fixed public-safe health object for a Herdr observation.""" - normalized_outcome = str(outcome or "unknown").strip().lower().replace("-", "_") - if normalized_outcome == "ok": - normalized_outcome = ( - "healthy_non_empty" - if (spaces and len(spaces) > 0) or (workers and len(workers) > 0) - else "empty_healthy" - ) - if normalized_outcome not in { - "healthy_non_empty", - "empty_healthy", - "missing_binary", - "launch_error", - "timeout", - "deadline_exhausted", - "nonzero", - "malformed_json", - "protocol_error", - "socket_disconnected", - "worker_cap_exceeded", - "continuity_unavailable", - "unknown", - }: - normalized_outcome = "unknown" - counts = { - "spaces": len(spaces or []), - "workers": len(workers or []), - } - return BackendHealth( - name=_BACKEND_NAME, - status=herdr_health_status_for_outcome(normalized_outcome), - outcome=normalized_outcome, - observed_at=observed_at or utc_timestamp(), - message=message if message is not None else _HEALTH_MESSAGES[normalized_outcome], - counts=counts, - ) - - -@dataclass(frozen=True) -class HerdrSnapshotObservation: - """Snapshot observation plus public backend health and private bindings.""" - - spaces: list[Space] - workers: list[Worker] - bindings: list[WorkerBinding] = field(default_factory=list) - backend_health: list[BackendHealth] = field(default_factory=list) - - def __post_init__(self) -> None: - spaces = list(self.spaces) - workers = list(self.workers) - bindings = list(self.bindings) - backend_health = list(self.backend_health) - if not backend_health: - outcome = "healthy_non_empty" if spaces or workers else "empty_healthy" - backend_health = [herdr_backend_health(outcome, spaces=spaces, workers=workers)] - object.__setattr__(self, "spaces", spaces) - object.__setattr__(self, "workers", workers) - object.__setattr__(self, "bindings", bindings) - object.__setattr__(self, "backend_health", backend_health) - - @property - def health(self) -> BackendHealth: - for item in self.backend_health: - if item.name == _BACKEND_NAME: - return item - return herdr_backend_health("unknown", spaces=self.spaces, workers=self.workers) - - @property - def authoritative(self) -> bool: - return self.health.status == "healthy" - - -@dataclass(frozen=True) -class HerdrCommandObservation: - """Command execution observation with health metadata.""" - - spaces: list[Space] - workers: list[Worker] - status: str - outcome: str - message: str = "" - bindings: list[WorkerBinding] = field(default_factory=list) - backend_health: list[BackendHealth] = field(default_factory=list) - - def __post_init__(self) -> None: - spaces = list(self.spaces) - workers = list(self.workers) - bindings = list(self.bindings) - backend_health = list(self.backend_health) - if not backend_health: - backend_health = [ - herdr_backend_health( - self.outcome, - message=self.message or None, - spaces=spaces, - workers=workers, - ) - ] - object.__setattr__(self, "spaces", spaces) - object.__setattr__(self, "workers", workers) - object.__setattr__(self, "bindings", bindings) - object.__setattr__(self, "backend_health", backend_health) - - @property - def healthy(self) -> bool: - return self.status == "healthy" and self.health.status == "healthy" - - @property - def health(self) -> BackendHealth: - for item in self.backend_health: - if item.name == _BACKEND_NAME: - return item - return herdr_backend_health(self.outcome, message=self.message or None, spaces=self.spaces, workers=self.workers) - - -_FORBIDDEN_CONNECTOR_FIELDS_COMPACT = {field.replace("_", "") for field in _FORBIDDEN_CONNECTOR_FIELDS} - - -def _compact_field_name(key: object) -> str: - """Normalize field names for conservative connector/status-key matching.""" - return str(key).lower().replace("-", "_").replace(".", "_").replace("_", "") - - -def _field_matches(key: object, expected: str) -> bool: - """Return True when a payload key matches snake_case or camelCase spelling.""" - return _compact_field_name(key) == _compact_field_name(expected) - -def _is_reserved_stable_key_field(key: object) -> bool: - """Match the entire current and future normalized stable-key family.""" - compact = str(key).lower().replace("_", "").replace("-", "").replace(".", "") - return compact.startswith("stablekey") - - -def _strip_stable_key_fields(value: Any) -> Any: - """Recursively remove every source-controlled stable-key family field.""" - if isinstance(value, Mapping): - return { - str(key): _strip_stable_key_fields(child) - for key, child in value.items() - if not _is_reserved_stable_key_field(key) - } - if isinstance(value, list): - return [_strip_stable_key_fields(item) for item in value] - if isinstance(value, tuple): - return [_strip_stable_key_fields(item) for item in value] - return value - - -_TURN_OBSERVATION_FIELD_NAMES = frozenset( - { - "turn", - "turnepoch", - "lastcompletedturn", - "outcome", - } -) - - -def _strip_turn_observation_fields(value: Any) -> Any: - """Remove turn counters and outcomes from every Worker identity surface.""" - if isinstance(value, Mapping): - return { - str(key): _strip_turn_observation_fields(child) - for key, child in value.items() - if _compact_field_name(key) not in _TURN_OBSERVATION_FIELD_NAMES - } - if isinstance(value, list): - return [_strip_turn_observation_fields(item) for item in value] - if isinstance(value, tuple): - return tuple(_strip_turn_observation_fields(item) for item in value) - return value - - -def _private_fingerprint(value: Any) -> str: - """Return a private adapter-only fingerprint without public sanitization.""" - encoded = json.dumps( - value, - sort_keys=True, - separators=(",", ":"), - default=str, - ).encode("utf-8") - return hashlib.sha256(encoded).hexdigest()[:24] - - -def _is_forbidden_connector_field(key: object) -> bool: - """Return True for forbidden connector fields, including common case variants.""" - normalized = str(key).lower().replace("-", "_").replace(".", "_") - compact = _compact_field_name(key) - segments = {part for part in normalized.split("_") if part} - return ( - normalized in _FORBIDDEN_CONNECTOR_FIELDS - or compact in _FORBIDDEN_CONNECTOR_FIELDS_COMPACT - or bool(segments & _FORBIDDEN_CONNECTOR_FIELDS) - ) - - -def _contains_forbidden_connector_text(value: object) -> bool: - """Return True when diagnostic text names connector/private delivery fields.""" - normalized = str(value).lower().replace("-", "_").replace(".", "_") - compact = normalized.replace("_", "") - return any(field in normalized for field in _FORBIDDEN_CONNECTOR_FIELDS) or any( - field in compact for field in _FORBIDDEN_CONNECTOR_FIELDS_COMPACT - ) - - -def _run_herdr( - args: Sequence[str], - config: Config, - *, - timeout_seconds: float | None = None, -) -> subprocess.CompletedProcess[str] | None: - """Run the Herdr CLI with read-only arguments; return None on launch failure.""" - timeout = config.herdr_timeout_seconds if timeout_seconds is None else timeout_seconds - try: - return subprocess.run( - [config.herdr_bin, *args], - capture_output=True, - text=True, - check=False, - timeout=timeout, - ) - except (OSError, UnicodeDecodeError, ValueError, TypeError): - return None - - -def _run_herdr_probe( - args: Sequence[str], - config: Config, - timeout_seconds: float | None, -) -> subprocess.CompletedProcess[str] | None: - """Call _run_herdr with timeout support while preserving simple test fakes.""" - if timeout_seconds is None: - return _run_herdr(args, config) - try: - return _run_herdr(args, config, timeout_seconds=timeout_seconds) - except TypeError: - return _run_herdr(args, config) - - -def _probe_herdr( - args: Sequence[str], - config: Config, - budget: _ProbeBudget | None = None, -) -> tuple[str, Any]: - """Run a read-only Herdr command and retain failure class for mutations.""" - timeout_seconds: float | None = None - if budget is not None: - timeout_seconds = budget.subprocess_timeout_seconds() - if timeout_seconds is None: - return "deadline_exhausted", None - try: - completed = _run_herdr_probe(args, config, timeout_seconds) - except subprocess.TimeoutExpired: - return "timeout", None - if completed is None: - return "launch_error", None - - if completed.returncode != 0: - return "nonzero", None - payload = _parse_json_output(completed.stdout) - if payload is None: - return "malformed_json", None - return "ok", payload - - -def _parse_json_output(stdout: str | None) -> Any: - """Best-effort parse of herdr JSON output; None on failure.""" - if not stdout or not stdout.strip(): - return None - try: - return json.loads(stdout) - except (json.JSONDecodeError, TypeError, ValueError): - return None - - -def _command_payload(args: Sequence[str], config: Config) -> Any: - """Return parsed JSON for a single herdr command, or None on any bad output.""" - outcome, payload = _probe_herdr(args, config) - if outcome != "ok": - return None - return payload - - -def _command_payload_variants(variants: Sequence[Sequence[str]], config: Config) -> Any: - """Try a sequence of herdr arg lists in order; return first successful payload.""" - outcome, payload = _probe_payload_variants(variants, config) - return payload if outcome == "ok" else None - - -def _safe_text_sample(value: str | None) -> str | None: - """Return a short diagnostic sample through the shared public sanitizer.""" - if not value or _contains_forbidden_connector_text(value): - return None - sanitized = sanitize_public_text( - value, - max_chars=200, - collapse_whitespace=True, - ) - return sanitized or None - - -_LOCAL_STATE_COMPLIANT_REMEDIATION = "No action required." -_LOCAL_STATE_UNINITIALIZED_REMEDIATION = ( - "No action required while local state is uninitialized." -) -_LOCAL_STATE_STOPPED_REMEDIATION = "No action required while the daemon is stopped." -_LOCAL_STATE_REPAIR_REMEDIATION = ( - "Restart Tendwire to repair local state permissions." -) -_LOCAL_STATE_UNSAFE_REMEDIATION = ( - "Move unsafe local state aside and restore from a trusted backup." -) -_LOCAL_STATE_CHECK_GROUPS = ( - ( - "state_directory_permissions", - frozenset({LocalStateKind.STATE_DIRECTORY}), - "not_initialized", - _LOCAL_STATE_UNINITIALIZED_REMEDIATION, - ), - ( - "database_permissions", - frozenset( - { - LocalStateKind.DATABASE, - LocalStateKind.DATABASE_WAL, - LocalStateKind.DATABASE_SHM, - LocalStateKind.DATABASE_JOURNAL, - } - ), - "not_initialized", - _LOCAL_STATE_UNINITIALIZED_REMEDIATION, - ), - ( - "identity_permissions", - frozenset({LocalStateKind.PRIVATE_FILE}), - "not_initialized", - _LOCAL_STATE_UNINITIALIZED_REMEDIATION, - ), - ( - "daemon_socket_permissions", - frozenset({LocalStateKind.SOCKET, LocalStateKind.SOCKET_GROUP}), - "not_running", - _LOCAL_STATE_STOPPED_REMEDIATION, - ), -) - - -def _local_state_check( - name: str, - kinds: frozenset[LocalStateKind], - neutral_outcome: str, - neutral_remediation: str, - *, - entries: Sequence[Any], - issues: Sequence[Any], -) -> dict[str, Any]: - """Fold one fixed local-state category into a path-free doctor record.""" - category_entries = [entry for entry in entries if entry.kind in kinds] - category_issues = [issue for issue in issues if issue.kind in kinds] - issue_codes = {issue.code for issue in category_issues} - if issue_codes - {LocalStateErrorCode.INSECURE_MODE}: - return { - "name": name, - "ok": False, - "outcome": "unsafe", - "remediation": _LOCAL_STATE_UNSAFE_REMEDIATION, - } - if ( - LocalStateErrorCode.INSECURE_MODE in issue_codes - or any( - entry.state is PermissionState.REPAIR_REQUIRED - for entry in category_entries - ) - ): - return { - "name": name, - "ok": False, - "outcome": "repair_required", - "remediation": _LOCAL_STATE_REPAIR_REMEDIATION, - } - if not category_entries or all( - entry.state is PermissionState.ABSENT for entry in category_entries - ): - return { - "name": name, - "ok": True, - "outcome": neutral_outcome, - "remediation": neutral_remediation, - } - return { - "name": name, - "ok": True, - "outcome": "compliant", - "remediation": _LOCAL_STATE_COMPLIANT_REMEDIATION, - } - - -def _inspect_local_state(config: Config) -> ConfigStateReport | None: - """Inspect configured local state once without creating or repairing it.""" - try: - if config.db_path is None: - raise ValueError - socket_path = config.socket_path or config.data_dir / "tendwire.sock" - return inspect_config_state( - config.data_dir, - config.db_path, - socket_path=socket_path, - private_files=( - config.installation_key_path, - config.installation_key_marker_path, - config.installation_key_sentinel_path, - ), - socket_group=config.socket_group, - ) - except Exception: - return None - - -def _local_state_checks( - report: ConfigStateReport | None, -) -> tuple[list[dict[str, Any]], bool]: - """Fold a path-free inspection report into the fixed local-state checks.""" - inspected = report - if inspected is None: - checks = [ - { - "name": name, - "ok": False, - "outcome": "unsafe", - "remediation": _LOCAL_STATE_UNSAFE_REMEDIATION, - } - for name, _kinds, _neutral, _remediation in _LOCAL_STATE_CHECK_GROUPS - ] - return checks, False - - checks = [ - _local_state_check( - name, - kinds, - neutral_outcome, - neutral_remediation, - entries=inspected.entries, - issues=inspected.issues, - ) - for name, kinds, neutral_outcome, neutral_remediation in _LOCAL_STATE_CHECK_GROUPS - ] - return checks, inspected.ok - - -def _public_herdr_bin(value: str) -> str: - """Retain the doctor key while redacting configured path-shaped values.""" - sanitized = sanitize_public_text( - value, - max_chars=200, - collapse_whitespace=True, - ) - return sanitized or "[redacted]" - - -_STORE_MAINTENANCE_REMEDIATION = { - "ok": "No action required.", - "overdue": "Keep Tendwire running to resume automatic store maintenance.", - "backlog": "Keep Tendwire running to drain the maintenance backlog.", - "not_initialized": "No action required while the store is uninitialized.", - "unsafe": "Move unsafe local state aside and restore from a trusted backup.", - "unavailable": "Check store availability before retrying diagnostics.", -} -_STORE_KINDS = frozenset( - { - LocalStateKind.STATE_DIRECTORY, - LocalStateKind.DATABASE, - LocalStateKind.DATABASE_WAL, - LocalStateKind.DATABASE_SHM, - LocalStateKind.DATABASE_JOURNAL, - } -) - - -def _store_maintenance_record( - config: Config, - outcome: str, - *, - snapshot_count: int = 0, - last_completed_at: str | None = None, -) -> dict[str, Any]: - """Build the fixed public-safe maintenance doctor record.""" - return { - "name": "store_maintenance", - "ok": outcome in {"ok", "not_initialized"}, - "outcome": outcome, - "remediation": _STORE_MAINTENANCE_REMEDIATION[outcome], - "snapshot_retention_days": config.snapshot_retention_days, - "snapshot_retention_count": config.snapshot_retention_count, - "maintenance_batch_size": config.snapshot_maintenance_batch_size, - "maintenance_cadence_seconds": config.store_maintenance_cadence_seconds, - "snapshot_count": snapshot_count, - "last_completed_at": last_completed_at, - } - - -def _public_utc_timestamp(value: Any) -> tuple[str | None, datetime | None]: - """Return a normalized public UTC timestamp, rejecting malformed/private text.""" - if not isinstance(value, str) or not value.strip(): - return None, None - raw = value.strip() - if raw.endswith("Z"): - raw = raw[:-1] + "+00:00" - try: - parsed = datetime.fromisoformat(raw) - except (TypeError, ValueError): - return None, None - if parsed.tzinfo is None: - return None, None - normalized = parsed.astimezone(timezone.utc) - return normalized.isoformat(), normalized - - - - -def _store_maintenance_check( - config: Config, - report: ConfigStateReport | None, -) -> dict[str, Any]: - """Return one fixed, path-free, non-mutating store-maintenance check.""" - if report is None: - return _store_maintenance_record(config, "unsafe") - relevant_issues = [issue for issue in report.issues if issue.kind in _STORE_KINDS] - relevant_entries = [entry for entry in report.entries if entry.kind in _STORE_KINDS] - if relevant_issues or any( - entry.state is PermissionState.REPAIR_REQUIRED for entry in relevant_entries - ): - return _store_maintenance_record(config, "unsafe") - database = next( - (entry for entry in relevant_entries if entry.kind is LocalStateKind.DATABASE), - None, - ) - if database is None or database.state is PermissionState.ABSENT: - return _store_maintenance_record(config, "not_initialized") - - try: - from ..store.sqlite import store_status - - status = store_status( - config.db_path, - config.host_id, - snapshot_retention_days=config.snapshot_retention_days, - snapshot_retention_count=config.snapshot_retention_count, - maintenance_batch_size=config.snapshot_maintenance_batch_size, - maintenance_cadence_seconds=config.store_maintenance_cadence_seconds, - require_current_schema=True, - ) - maintenance = status.get("maintenance") - if ( - status.get("ok") is not True - or not isinstance(maintenance, Mapping) - or isinstance(maintenance.get("snapshot_count"), bool) - or not isinstance(maintenance.get("snapshot_count"), int) - or int(maintenance["snapshot_count"]) < 0 - or not isinstance(maintenance.get("backlog"), bool) - or maintenance.get("status") not in {"never", "ok", "failed"} - ): - return _store_maintenance_record(config, "unavailable") - snapshot_count = int(maintenance["snapshot_count"]) - maintenance_status = str(maintenance["status"]) - last_value = maintenance.get("last_completed_at") - if maintenance_status == "failed" or ( - maintenance_status == "never" and last_value is not None - ): - return _store_maintenance_record(config, "unavailable") - if last_value is None: - last_completed_at = None - completed = None - else: - last_completed_at, completed = _public_utc_timestamp(last_value) - if completed is None: - return _store_maintenance_record(config, "unavailable") - if maintenance["backlog"]: - outcome = "backlog" - elif completed is None: - outcome = "overdue" - else: - _now_value, now = _public_utc_timestamp(utc_timestamp()) - if now is None: - return _store_maintenance_record(config, "unavailable") - due_at = completed + timedelta( - seconds=config.store_maintenance_cadence_seconds - ) - outcome = "overdue" if now >= due_at else "ok" - return _store_maintenance_record( - config, - outcome, - snapshot_count=snapshot_count, - last_completed_at=last_completed_at, - ) - except Exception: - return _store_maintenance_record(config, "unavailable") - - -def _pending_ingestion_check(config: Config) -> dict[str, Any]: - """Return one fixed non-mutating check from durable pending state only.""" - unavailable = { - "status": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - } - try: - from ..store.sqlite import backend_pending_health - - value = backend_pending_health(config.db_path, config.host_id) - except Exception: - value = unavailable - raw = value if isinstance(value, Mapping) else unavailable - raw_counts = raw.get("counts") - status = raw.get("status") - count_values = ( - tuple(raw_counts.get(key) for key in ("fresh", "stale", "total")) - if isinstance(raw_counts, Mapping) - else () - ) - valid_counts = len(count_values) == 3 and all( - isinstance(item, int) and not isinstance(item, bool) and item >= 0 - for item in count_values - ) - if ( - status not in {"healthy", "degraded", "store_unavailable"} - or not valid_counts - or count_values[2] != count_values[0] + count_values[1] - or (status == "healthy" and count_values[1] != 0) - or (status == "degraded" and count_values[1] == 0) - or (status == "store_unavailable" and count_values != (0, 0, 0)) - ): - status = "store_unavailable" - counts = dict(unavailable["counts"]) - else: - counts = dict(zip(("fresh", "stale", "total"), count_values, strict=True)) - return { - "name": "pending_ingestion", - "ok": status == "healthy", - "outcome": status, - "counts": counts, - "stale_grace_seconds": config.pending_stale_grace_seconds, - } - - -def _finish_diagnostics(result: dict[str, Any], config: Config) -> dict[str, Any]: - report = _inspect_local_state(config) - local_checks, local_ok = _local_state_checks(report) - maintenance = _store_maintenance_check(config, report) - pending_ingestion = _pending_ingestion_check(config) - result["checks"].extend(local_checks) - result["checks"].append(maintenance) - result["checks"].append(pending_ingestion) - if ( - (not local_ok or not maintenance["ok"] or not pending_ingestion["ok"]) - and result["status"] == "ok" - ): - result["status"] = "degraded" - return result - - -def _diagnostic_item_count(payload: Any, keys: Sequence[str]) -> int: - return len(_payload_items(payload, keys)) - - -def _diagnostic_check( - name: str, - args: Sequence[str], - config: Config, - keys: Sequence[str], - budget: _ProbeBudget, -) -> dict[str, Any]: - """Run one read-only Herdr command and return a sanitized diagnostic record.""" - check: dict[str, Any] = { - "name": name, - "ok": False, - "outcome": "unknown", - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": budget.aggregate_deadline_seconds, - } - timeout_seconds = budget.subprocess_timeout_seconds() - if timeout_seconds is None: - check["outcome"] = "deadline_exhausted" - return check - try: - completed = subprocess.run( - [config.herdr_bin, *args], - capture_output=True, - text=True, - check=False, - timeout=timeout_seconds, - ) - except subprocess.TimeoutExpired: - check["outcome"] = "timeout" - return check - except (OSError, UnicodeDecodeError, ValueError, TypeError): - check["outcome"] = "launch_error" - return check - - check["exit_code"] = int(completed.returncode) - if completed.returncode != 0: - check["outcome"] = "nonzero" - stdout_sample = _safe_text_sample(completed.stdout) - stderr_sample = _safe_text_sample(completed.stderr) - if stdout_sample is not None: - check["stdout_sample"] = stdout_sample - if stderr_sample is not None: - check["stderr_sample"] = stderr_sample - return check - - payload = _parse_json_output(completed.stdout) - if payload is None: - check["outcome"] = "malformed_json" - stdout_sample = _safe_text_sample(completed.stdout) - if stdout_sample is not None: - check["stdout_sample"] = stdout_sample - return check - - item_count = _diagnostic_item_count(payload, keys) - check["ok"] = True - check["item_count"] = item_count - check["outcome"] = "healthy_non_empty" if item_count else "empty_healthy" - return check - - -def diagnose_herdr(config: Config) -> dict[str, Any]: - """Return JSON-serializable read-only Herdr CLI diagnostics.""" - groups = [ - [ - ("workspace_list", ["workspace", "list"], ("workspaces", "spaces", "data", "items", "results", "result")), - ("workspace_list_json", ["workspace", "list", "--json"], ("workspaces", "spaces", "data", "items", "results", "result")), - ], - [ - ("agent_list", ["agent", "list"], ("agents", "workers", "data", "items", "results", "result")), - ("agent_list_json", ["agent", "list", "--json"], ("agents", "workers", "data", "items", "results", "result")), - ], - [ - ("pane_list", ["pane", "list"], ("panes", "items", "data", "results", "result")), - ("pane_list_json", ["pane", "list", "--json"], ("panes", "items", "data", "results", "result")), - ], - ] - planned = [check for group in groups for check in group] - result: dict[str, Any] = { - "schema_version": 1, - "command": "doctor", - "herdr_bin": _public_herdr_bin(config.herdr_bin), - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": config.herdr_timeout_seconds * len(planned), - "status": "ok", - "checks": [], - } - try: - binary_path = shutil.which(config.herdr_bin) - except (TypeError, ValueError, OSError): - binary_path = None - - if binary_path is None: - result["status"] = "unavailable" - result["checks"] = [ - { - "name": name, - "ok": False, - "outcome": "missing_binary", - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": result["aggregate_deadline_seconds"], - } - for name, args, _keys in planned - ] - return _finish_diagnostics(result, config) - - budget = _ProbeBudget.from_config(config, planned_probes=len(planned)) - checks: list[dict[str, Any]] = [] - stop_after_timeout = False - stop_after_outcome = "" - for group in groups: - if stop_after_timeout: - break - for index, (name, args, keys) in enumerate(group): - if index > 0 and checks[-1]["ok"]: - break - check = _diagnostic_check(name, args, config, keys, budget) - checks.append(check) - if check["outcome"] in _DEADLINE_EXHAUSTED_OUTCOMES: - stop_after_timeout = True - stop_after_outcome = str(check["outcome"]) - break - - names_seen = {str(check["name"]) for check in checks} - remaining_planned = [ - (name, args, keys) - for name, args, keys in planned - if name not in names_seen - ] - if stop_after_timeout: - skipped_outcome = ( - "skipped_after_deadline" - if stop_after_outcome == "deadline_exhausted" - else "skipped_after_timeout" - ) - for name, args, _keys in remaining_planned: - checks.append( - { - "name": name, - "ok": False, - "outcome": skipped_outcome, - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": budget.aggregate_deadline_seconds, - } - ) - else: - for name, args, _keys in remaining_planned: - checks.append( - { - "name": name, - "ok": True, - "outcome": "skipped_not_needed", - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": budget.aggregate_deadline_seconds, - } - ) - - result["checks"] = checks - if any(check["outcome"] in _DEADLINE_EXHAUSTED_OUTCOMES for check in checks): - result["status"] = "timeout" - elif any(not check["ok"] for check in checks): - result["status"] = "degraded" - return _finish_diagnostics(result, config) - - -def _strip_connector_fields(value: Any) -> Any: - """Recursively drop connector/delivery fields from arbitrary JSON-like values.""" - if isinstance(value, Mapping): - clean: dict[str, Any] = {} - for key, child in value.items(): - key_text = str(key) - if _is_forbidden_connector_field(key): - continue - clean[key_text] = _strip_connector_fields(child) - return clean - if isinstance(value, list): - return [_strip_connector_fields(item) for item in value] - if isinstance(value, tuple): - return [_strip_connector_fields(item) for item in value] - return value - - -def _strip_status_fields(value: Any) -> Any: - """Recursively drop raw status fields from metadata values.""" - if isinstance(value, Mapping): - clean: dict[str, Any] = {} - for key, child in value.items(): - if any(_field_matches(key, status_key) for status_key in _STATUS_KEYS): - continue - clean[str(key)] = _strip_status_fields(child) - return clean - if isinstance(value, list): - return [_strip_status_fields(item) for item in value] - if isinstance(value, tuple): - return [_strip_status_fields(item) for item in value] - return value - - -def _payload_items(payload: Any, keys: Sequence[str]) -> list[dict[str, Any]]: - """Extract object records from conservative herdr list payload shapes.""" - if isinstance(payload, list): - candidates: Iterable[Any] = payload - elif isinstance(payload, Mapping): - candidates = () - for key in keys: - value = _value_for_key(payload, key) - if isinstance(value, list): - candidates = value - break - if isinstance(value, Mapping): - nested = _payload_items(value, keys) - if nested: - return nested - else: - return [] - - items: list[dict[str, Any]] = [] - for item in candidates: - if isinstance(item, Mapping): - stripped = _strip_connector_fields(item) - if isinstance(stripped, dict): - items.append(stripped) - return items - - -def _first_text(item: Mapping[str, Any], keys: Sequence[str]) -> str | None: - """Return the first scalar string value for any key.""" - for key in keys: - value = _value_for_key(item, key) - if value is None: - continue - if isinstance(value, (str, int, float, bool)): - return str(value) - return None - - -def _nested_text(item: Mapping[str, Any], *path: str) -> str | None: - """Return the first scalar string value reachable via a dotted key path.""" - current: Any = item - for key in path: - if not isinstance(current, Mapping): - return None - current = _value_for_key(current, key) - if current is None: - return None - if isinstance(current, (str, int, float, bool)): - return str(current) - if isinstance(current, Mapping): - return _first_text(current, ("id", "value", "name", "label")) - return None - - -def _related_id(value: Any) -> str | None: - """Return a neutral related-object id from a scalar or mapping.""" - if value is None: - return None - if isinstance(value, Mapping): - return _first_text(value, ("id", "workspace_id", "space_id", "slug", "name")) - if isinstance(value, (str, int, float, bool)): - return str(value) - return None - - -def _normalize_status(raw_status: Any) -> tuple[str, str | None]: - """Return canonical status plus original raw string when normalization changed it.""" - if raw_status is None: - return "unknown", None - - raw_text = str(raw_status) - canonical = normalize_status(raw_text) - raw_key = raw_text.strip().lower().replace("_", "-") - raw_meta = raw_text if raw_text and raw_key != canonical else None - return canonical, raw_meta - - -def _value_for_key(item: Mapping[str, Any], expected_key: str) -> Any: - """Return a value by exact key or snake/camel-case equivalent.""" - if expected_key in item: - return item[expected_key] - for key, value in item.items(): - if _field_matches(key, expected_key): - return value - return None - - -def _status_from_item(item: Mapping[str, Any]) -> tuple[str, str | None]: - """Extract and normalize status-like fields from a herdr record.""" - for key in _STATUS_KEYS: - value = _value_for_key(item, key) - if value is not None: - return _normalize_status(value) - return "unknown", None - - -def _meta_from_item(item: Mapping[str, Any], excluded_keys: set[str], raw_status: str | None) -> dict[str, Any]: - """Build sanitized neutral metadata for a projected model.""" - item = _strip_stable_key_fields(item) - explicit_meta = _value_for_key(item, "meta") - meta = { - str(key): _strip_status_fields(value) - for key, value in item.items() - if not _field_matches(key, "meta") - and not any(_field_matches(key, excluded_key) for excluded_key in excluded_keys) - and not any(_field_matches(key, status_key) for status_key in _STATUS_KEYS) - } - if isinstance(explicit_meta, Mapping): - for key, value in explicit_meta.items(): - if _is_forbidden_connector_field(key): - continue - if any(_field_matches(key, status_key) for status_key in _STATUS_KEYS): - continue - meta[str(key)] = _strip_status_fields(value) - if raw_status is not None: - meta["raw_status"] = raw_status - return meta - - -def _space_id_from_item(item: Mapping[str, Any]) -> str: - """Resolve a stable space id, preferring explicit Herdr workspace_id.""" - return _first_text(item, ("workspace_id", "space_id", "id", "slug", "name")) or "unknown" - - -def _space_name_from_item(item: Mapping[str, Any], space_id: str) -> str: - """Resolve a space name, preferring label then workspace_id.""" - return _first_text(item, ("label", "name", "title", "workspace_id", "space_id")) or space_id - - -def _public_worker_id_base_from_item(item: Mapping[str, Any]) -> str: - """Resolve a neutral public worker id without terminal/session handles.""" - return ( - _first_text(item, ("worker_id", "id", "slug", "agent_id")) - or _first_text(item, ("agent", "name", "label", "title")) - or "unknown" - ) - - -def _worker_id_from_item(item: Mapping[str, Any]) -> str: - """Resolve a stable public worker id.""" - return _public_worker_id_base_from_item(item) - - -def _private_identity_material_from_item(item: Mapping[str, Any]) -> dict[str, Any]: - """Return private Herdr identity material that is never serialized publicly.""" - agent_id = _first_text(item, ("agent_id",)) - agent_session = _nested_text(item, "agent_session", "value") - session_id = _first_text(item, ("session_id",)) - if agent_id or agent_session or session_id: - return { - "agent_id": agent_id, - "agent_session": agent_session, - "session_id": session_id, - "space_id": _worker_space_id_from_item(item), - } - base = { - "public_id": _public_worker_id_base_from_item(item), - "name": _first_text(item, ("agent", "name", "label", "title")), - "space_id": _worker_space_id_from_item(item), - } - base["terminal_id"] = _first_text(item, ("terminal_id",)) - base["pane_id"] = _first_text(item, ("pane_id",)) - return base - - -def _private_identity_from_item(item: Mapping[str, Any], config: Config | None = None) -> str: - """Return a private identity used only to avoid collapsing distinct workers.""" - material = _private_identity_material_from_item(item) - if config is None: - return _private_fingerprint(material) - return worker_binding_private_fingerprint( - host_id=config.host_id, - backend=_BACKEND_NAME, - identity_material=material, - ) - - -def _private_backend_target(kind: str, value: str, *, sendable: bool = True, reason: str | None = None) -> dict[str, Any]: - """Return the internal backend target shape.""" - return { - "kind": kind, - "value": value, - "sendable": bool(sendable), - "reason": reason, - } - - -def _backend_target_from_item(item: Mapping[str, Any]) -> dict[str, Any] | None: - """Resolve the private Herdr send target from backend-observed fields.""" - candidates = ( - ("agent_id", _first_text(item, ("agent_id",))), - ("terminal_id", _first_text(item, ("terminal_id",))), - ("pane_id", _first_text(item, ("pane_id",))), - ("agent", _first_text(item, ("agent",))), - ("name", _first_text(item, ("name",))), - ("label", _first_text(item, ("label",))), - ) - for kind, value in candidates: - if value: - return _private_backend_target(kind, value) - return None - - -def _worker_with_id(worker: Worker, worker_id: str) -> Worker: - """Return a worker copy with a disambiguated public id.""" - return Worker( - id=worker_id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - backend_target=worker.backend_target, - ) - - -def _worker_name_from_item(item: Mapping[str, Any], worker_id: str) -> str: - """Resolve a worker display name, preferring agent then label.""" - return _first_text(item, ("agent", "label", "name", "title")) or worker_id - - -def _worker_space_id_from_item(item: Mapping[str, Any]) -> str | None: - """Resolve a worker's parent space id, preferring workspace_id.""" - return ( - _first_text(item, ("workspace_id", "space_id", "spaceId", "workspaceId")) - or _related_id(_value_for_key(item, "space")) - or _related_id(_value_for_key(item, "workspace")) - ) - - -def _spaces_from_payload(payload: Any) -> list[Space]: - """Extract neutral Space objects from a herdr workspace-list payload.""" - spaces: list[Space] = [] - for item in _payload_items(payload, ("workspaces", "spaces", "data", "items", "results", "result")): - space_id = _space_id_from_item(item) - name = _space_name_from_item(item, space_id) - status, raw_status = _status_from_item(item) - updated_at = _first_text(item, ("updated_at", "last_seen_at", "observed_at", "timestamp")) - status_line = _first_text(item, ("status_line", "summary", "description")) - meta = _meta_from_item( - item, - { - "id", - "workspace_id", - "space_id", - "slug", - "name", - "title", - "label", - "meta", - "updated_at", - "last_seen_at", - "observed_at", - "timestamp", - "status_line", - "summary", - "description", - "fingerprint", - "agent_status", - }, - raw_status, - ) - spaces.append( - Space( - id=space_id, - name=name, - status=status, - meta=meta, - updated_at=updated_at, - status_line=status_line, - ) - ) - return spaces - - -def _worker_from_item(item: Mapping[str, Any]) -> Worker: - """Build a neutral Worker from a single herdr agent/pane record.""" - worker_id = _worker_id_from_item(item) - name = _worker_name_from_item(item, worker_id) - status, raw_status = _status_from_item(item) - last_seen_at = _first_text(item, ("last_seen_at", "updated_at", "observed_at", "timestamp")) - summary = _first_text(item, ("summary", "status_line", "description")) - space_id = _worker_space_id_from_item(item) - meta = _meta_from_item( - item, - { - "id", - "agent_id", - "worker_id", - "slug", - "name", - "title", - "agent", - "meta", - "space_id", - "workspace_id", - "spaceId", - "workspaceId", - "space", - "workspace", - "last_seen_at", - "updated_at", - "observed_at", - "timestamp", - "summary", - "status_line", - "description", - "fingerprint", - "agent_status", - "backend_target", - "terminal_id", - "pane_id", - "agent_session", - "session_id", - }, - raw_status, - ) - return Worker( - id=worker_id, - name=name, - status=status, - space_id=space_id, - meta=meta, - last_seen_at=last_seen_at, - summary=summary, - backend_target=_backend_target_from_item(item), - ) - - -def _output_excerpt_limit(config: Config | None) -> int | None: - if config is None: - return None - try: - limit = int(getattr(config, "output_excerpt_chars")) - except (TypeError, ValueError): - return None - return max(1, limit) - - -def _bounded_excerpt(value: str | None, limit: int | None) -> str | None: - if value is None or limit is None: - return value - text = str(value) - if len(text) <= limit: - return text - if limit <= 3: - return text[:limit] - return text[: limit - 3] + "..." - - -def _worker_with_summary(worker: Worker, summary: str | None) -> Worker: - if summary == worker.summary: - return worker - return Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=summary, - backend_target=worker.backend_target, - ) - - -def _worker_record_from_item( - item: Mapping[str, Any], - config: Config | None = None, - *, - pane_info_observed: bool = False, - identity_source: str = "unknown", -) -> _WorkerRecord: - item = _strip_turn_observation_fields(item) - worker = _worker_from_item(item) - worker = _worker_with_summary(worker, _bounded_excerpt(worker.summary, _output_excerpt_limit(config))) - observed_workspace_id = _first_text(item, ("workspace_id", "workspaceId")) - observed_pane_id = _first_text(item, ("pane_id", "paneId")) - canonical_identity = canonical_herdr_pane_identity( - observed_workspace_id, - observed_pane_id, - ) - workspace_id, pane_id = canonical_identity or (None, None) - return _WorkerRecord( - worker=worker, - private_fingerprint=_private_identity_from_item(item, config), - workspace_id=workspace_id, - pane_id=pane_id, - observed_workspace_id=observed_workspace_id, - observed_pane_id=observed_pane_id, - identity_source=identity_source, - terminal_id=_first_text(item, ("terminal_id", "terminalId")), - agent_session_id=( - _nested_text(item, "agent_session", "value") - or _first_text(item, ("session_id", "sessionId")) - ), - pane_info_observed=pane_info_observed, - ) - - -def _worker_with_backend_target(worker: Worker, backend_target: dict[str, Any] | None) -> Worker: - return Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - fingerprint=worker.fingerprint, - backend_target=backend_target, - ) - - -def _agent_observation_record( - item: Mapping[str, Any], - config: Config | None = None, -) -> _WorkerRecord: - """Record agent.list provenance without authorizing pane continuity.""" - return replace( - _worker_record_from_item( - item, - config, - pane_info_observed=False, - identity_source="agent.list", - ), - unmatched_agent_observation=True, - ) - - -def _stable_pane_identity(record: _WorkerRecord) -> tuple[str, str] | None: - """Return a PaneInfo-verified workspace/public-pane pair, never a runtime id.""" - if not record.pane_info_observed: - return None - return canonical_herdr_pane_identity(record.workspace_id, record.pane_id) - - -def _worker_with_stable_key( - config: Config, - record: _WorkerRecord, - installation_key: bytes | None, -) -> Worker: - """Replace source continuity metadata with a locally authenticated key.""" - worker = record.worker - meta = _strip_stable_key_fields(worker.meta) - identity = _stable_pane_identity(record) - if installation_key is not None and identity is not None: - workspace_id, pane_id = identity - meta["stable_key"] = stable_worker_key( - installation_key, - backend=_BACKEND_NAME, - host_id=config.host_id, - workspace_id=workspace_id, - pane_id=pane_id, - ) - meta["stable_key_version"] = STABLE_KEY_VERSION - return Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - backend_target=worker.backend_target, - ) - - -def _backend_target_send_token(target: Mapping[str, Any] | None) -> str: - """Return the exact value passed as herdr agent send's target argv token.""" - if not isinstance(target, Mapping): - return "" - return str(target.get("value") or "") - - -def _mark_backend_sendability(workers: list[Worker]) -> list[Worker]: - """Mark unsupported or duplicate final Herdr send tokens as not sendable.""" - marked: list[Worker] = [] - for worker in workers: - target = worker.backend_target - if not isinstance(target, dict): - marked.append(worker) - continue - kind = str(target.get("kind") or "") - value = _backend_target_send_token(target) - if target.get("sendable") is False: - marked.append(worker) - continue - if kind not in _BACKEND_TARGET_KINDS or not value: - marked.append( - _worker_with_backend_target( - worker, - _private_backend_target(kind or "agent", value, sendable=False, reason="backend_unsupported"), - ) - ) - continue - marked.append(_worker_with_backend_target(worker, _private_backend_target(kind, value))) - - send_token_counts = Counter( - _backend_target_send_token(worker.backend_target) - for worker in marked - if isinstance(worker.backend_target, dict) and worker.backend_target.get("sendable") is True - ) - final: list[Worker] = [] - for worker in marked: - target = worker.backend_target - if not isinstance(target, dict) or target.get("sendable") is not True: - final.append(worker) - continue - kind = str(target.get("kind") or "") - value = _backend_target_send_token(target) - if send_token_counts[value] > 1: - final.append( - _worker_with_backend_target( - worker, - _private_backend_target(kind, value, sendable=False, reason="duplicate_backend_target"), - ) - ) - continue - final.append(worker) - return final - - -def assert_unique_sendable_backend_targets(workers: Iterable[Worker]) -> bool: - """Prove no sendable workers share the same final Herdr argv target token.""" - seen: set[str] = set() - for worker in workers: - target = worker.backend_target - if not isinstance(target, Mapping) or target.get("sendable") is not True: - continue - token = _backend_target_send_token(target) - if not token: - raise AssertionError("sendable backend target is missing a send token") - if token in seen: - raise AssertionError("duplicate sendable backend target token") - seen.add(token) - return True - - -def _binding_target_key(binding: WorkerBinding) -> tuple[str, str]: - return (binding.target_kind, binding.target_value) - - -def _safe_stored_binding_for_reuse(binding: WorkerBinding) -> bool: - return bool(binding.worker_id) and (binding.reason or "") not in _AMBIGUOUS_BINDING_REASONS - - -def _worker_target_key(worker: Worker) -> tuple[str, str] | None: - target = worker.backend_target - if not isinstance(target, Mapping): - return None - kind = str(target.get("kind") or "") - value = str(target.get("value") or "") - if not kind or not value: - return None - return kind, value - - -def _record_with_worker(record: _WorkerRecord, worker: Worker) -> _WorkerRecord: - return _WorkerRecord( - worker=worker, - private_fingerprint=record.private_fingerprint, - workspace_id=record.workspace_id, - pane_id=record.pane_id, - observed_workspace_id=record.observed_workspace_id, - observed_pane_id=record.observed_pane_id, - identity_source=record.identity_source, - terminal_id=record.terminal_id, - agent_session_id=record.agent_session_id, - pane_info_observed=record.pane_info_observed, - unmatched_agent_observation=record.unmatched_agent_observation, - ) - - -def _reuse_worker_ids_from_bindings( - records: list[_WorkerRecord], - stored_bindings: Sequence[WorkerBinding] | None, -) -> list[_WorkerRecord]: - """Reuse stable public ids from private binding matches when safe.""" - if not stored_bindings: - return records - - by_private: dict[str, list[WorkerBinding]] = {} - by_target: dict[tuple[str, str], list[WorkerBinding]] = {} - for binding in stored_bindings: - if binding.backend != _BACKEND_NAME: - continue - by_private.setdefault(binding.private_fingerprint, []).append(binding) - if _safe_stored_binding_for_reuse(binding): - key = _binding_target_key(binding) - if key[0] and key[1]: - by_target.setdefault(key, []).append(binding) - - current_private_counts = Counter(record.private_fingerprint for record in records) - current_target_counts = Counter( - key - for key in (_worker_target_key(record.worker) for record in records) - if key is not None - ) - - reused: list[_WorkerRecord] = [] - for record in records: - worker = record.worker - private_fingerprint = record.private_fingerprint - matched = None - private_candidates = [ - binding - for binding in by_private.get(private_fingerprint, []) - if _safe_stored_binding_for_reuse(binding) - ] - if current_private_counts[private_fingerprint] == 1 and len(private_candidates) == 1: - matched = private_candidates[0] - if matched is None: - key = _worker_target_key(worker) - candidates = by_target.get(key or ("", ""), []) - if key is not None and current_target_counts[key] == 1 and len(candidates) == 1: - matched = candidates[0] - if matched is not None and matched.worker_id: - reused.append(_record_with_worker(record, _worker_with_id(worker, matched.worker_id))) - else: - reused.append(record) - return reused - - -def _deduplicated_worker_records( - records: list[_WorkerRecord], - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[_WorkerRecord]: - """Drop exact duplicates, then disambiguate duplicate public ids.""" - records = _reuse_worker_ids_from_bindings(records, stored_bindings) - seen: set[tuple[str, str, str | None, str, str, str]] = set() - unique: list[_WorkerRecord] = [] - for record in records: - worker = record.worker - backend_kind = "" - backend_value = "" - if worker.backend_target: - backend_kind = str(worker.backend_target.get("kind", "")) - backend_value = str(worker.backend_target.get("value", "")) - key = (worker.id, worker.name, worker.space_id, backend_kind, backend_value, record.private_fingerprint) - if key in seen: - continue - seen.add(key) - unique.append(record) - - groups: dict[str, list[_WorkerRecord]] = {} - for record in unique: - groups.setdefault(record.worker.id, []).append(record) - - disambiguated: list[_WorkerRecord] = [] - for worker_id, group in groups.items(): - if len(group) == 1: - disambiguated.append(group[0]) - continue - ordered = sorted( - group, - key=lambda record: ( - record.worker.name, - record.worker.space_id or "", - str((record.worker.backend_target or {}).get("kind", "")), - str((record.worker.backend_target or {}).get("value", "")), - record.private_fingerprint, - stable_fingerprint( - { - "id": record.worker.id, - "name": record.worker.name, - "space_id": record.worker.space_id, - "status": record.worker.status, - "summary": record.worker.summary, - } - ), - ), - ) - for index, record in enumerate(ordered, start=1): - disambiguated.append(_record_with_worker(record, _worker_with_id(record.worker, f"{worker_id}-{index}"))) - - disambiguated = sorted(disambiguated, key=lambda record: record.worker.id) - workers = _mark_backend_sendability([record.worker for record in disambiguated]) - assert_unique_sendable_backend_targets(workers) - return [ - _record_with_worker(record, worker) - for record, worker in zip(disambiguated, workers, strict=True) - ] - - -def _deduplicate_worker_records( - records: list[_WorkerRecord], - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[Worker]: - return [record.worker for record in _deduplicated_worker_records(records, stored_bindings)] - - -def _deduplicate_workers(workers: list[Worker]) -> list[Worker]: - return _deduplicate_worker_records( - [_WorkerRecord(worker=worker, private_fingerprint=worker.fingerprint) for worker in workers] - ) - - -def _binding_from_worker_record( - config: Config, - record: _WorkerRecord, - observed_at: str, -) -> WorkerBinding | None: - worker = record.worker - target = worker.backend_target - if not isinstance(target, Mapping): - return None - target_kind = str(target.get("kind") or "") - target_value = str(target.get("value") or "") - if not target_kind or not target_value: - return None - reason = target.get("reason") - return WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend=_BACKEND_NAME, - target_kind=target_kind, - target_value=target_value, - sendable=target.get("sendable") is True, - reason=str(reason) if reason is not None else None, - observed_at=observed_at, - expires_at=None, - private_fingerprint=record.private_fingerprint, - ) - - -def _workers_and_bindings_from_records( - config: Config, - records: list[_WorkerRecord], - *, - stored_bindings: Sequence[WorkerBinding] | None = None, - require_authenticated_continuity: bool = False, -) -> tuple[list[Worker], list[WorkerBinding]]: - observed_at = utc_timestamp() - deduplicated = _deduplicated_worker_records(records, stored_bindings) - if require_authenticated_continuity and any( - record.unmatched_agent_observation for record in deduplicated - ): - raise HerdrContinuityUnavailableError( - "Herdr agent observation has no authoritative pane owner" - ) - installation_key = None - if any( - _stable_pane_identity(record) is not None - or (require_authenticated_continuity and record.pane_info_observed) - for record in deduplicated - ): - try: - installation_key = load_or_create_installation_key(config.data_dir) - except InstallationKeyError: - if require_authenticated_continuity: - raise - installation_key = None - finalized = [ - _record_with_worker( - record, - _worker_with_stable_key(config, record, installation_key), - ) - for record in deduplicated - ] - workers = [record.worker for record in finalized] - bindings = [ - binding - for record in finalized - if (binding := _binding_from_worker_record(config, record, observed_at)) is not None - ] - return workers, separate_duplicate_worker_bindings(bindings) - - -def bindings_from_workers( - config: Config, - workers: Sequence[Worker], - *, - observed_at: str | None = None, -) -> list[WorkerBinding]: - """Build private Herdr bindings from in-memory workers when raw records are absent.""" - timestamp = observed_at or utc_timestamp() - workers = _mark_backend_sendability(list(workers)) - bindings: list[WorkerBinding] = [] - for worker in workers: - target = worker.backend_target - if not isinstance(target, Mapping): - continue - target_kind = str(target.get("kind") or "") - target_value = str(target.get("value") or "") - if not target_kind or not target_value: - continue - private_fingerprint = worker_binding_private_fingerprint( - host_id=config.host_id, - backend=_BACKEND_NAME, - identity_material={ - "worker_id": worker.id, - "worker_fingerprint": worker.fingerprint, - "target_kind": target_kind, - "target_value": target_value, - }, - ) - record = _WorkerRecord(worker=worker, private_fingerprint=private_fingerprint) - binding = _binding_from_worker_record(config, record, timestamp) - if binding is not None: - bindings.append(binding) - return separate_duplicate_worker_bindings(bindings) - - -def _binding_for_worker(worker: Worker, bindings: Sequence[WorkerBinding]) -> WorkerBinding | None: - candidates = [binding for binding in bindings if binding.worker_id == worker.id] - if not candidates: - return None - exact = [binding for binding in candidates if binding.worker_fingerprint == worker.fingerprint] - if len(exact) == 1: - return exact[0] - if exact: - return None - if len(candidates) == 1: - return candidates[0] - return None - - -def rehydrate_workers_from_bindings( - workers: Sequence[Worker], - current_bindings: Sequence[WorkerBinding] | None = None, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[Worker]: - """Attach private backend targets to workers from current, then stored bindings.""" - current = list(current_bindings or []) - stored = list(stored_bindings or []) - rehydrated: list[Worker] = [] - for worker in workers: - binding = _binding_for_worker(worker, current) - if binding is not None: - rehydrated.append(_worker_with_backend_target(worker, binding.backend_target())) - continue - if isinstance(worker.backend_target, Mapping): - rehydrated.append(worker) - continue - binding = _binding_for_worker(worker, stored) - if binding is None: - rehydrated.append(worker) - else: - rehydrated.append(_worker_with_backend_target(worker, binding.backend_target())) - return rehydrated - - -def _workers_from_payload( - payload: Any, - config: Config | None = None, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[Worker]: - """Extract neutral Worker objects from a herdr agent-list payload.""" - records: list[_WorkerRecord] = [] - for item in _payload_items(payload, ("agents", "workers", "data", "items", "results", "result")): - records.append(_agent_observation_record(item, config)) - return _deduplicate_worker_records(records, stored_bindings) - - -def _workers_and_bindings_from_payload( - payload: Any, - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> tuple[list[Worker], list[WorkerBinding]]: - records: list[_WorkerRecord] = [] - for item in _payload_items(payload, ("agents", "workers", "data", "items", "results", "result")): - records.append(_agent_observation_record(item, config)) - return _workers_and_bindings_from_records(config, records, stored_bindings=stored_bindings) - - -def _pane_has_agent(item: Mapping[str, Any]) -> bool: - """Return True when a pane record carries an agent or explicit agent marker.""" - if _value_for_key(item, "agent") is not None: - return True - if _value_for_key(item, "agent_session") is not None: - return True - if _nested_text(item, "agent_session", "value"): - return True - markers = _value_for_key(item, "state_labels") or _value_for_key(item, "labels") or [] - if isinstance(markers, list): - for marker in markers: - if isinstance(marker, str) and "agent" in marker.lower(): - return True - return False - - -def _record_match_keys( - item: Mapping[str, Any], - record: _WorkerRecord, - *, - pane_shaped: bool = False, -) -> list[tuple[str, str]]: - keys: list[tuple[str, str]] = [] - pane_id_keys = ("pane_id", "paneId", "id") if pane_shaped else ("pane_id", "paneId") - agent_session = ( - _nested_text(item, "agent_session", "value") - or _first_text(item, ("session_id", "sessionId")) - ) - for kind, value in ( - ("pane_id", _first_text(item, pane_id_keys)), - ("terminal_id", _first_text(item, ("terminal_id", "terminalId"))), - ("agent_session", agent_session), - ("private_fingerprint", str(record.private_fingerprint)), - ): - if value and (kind, value) not in keys: - keys.append((kind, value)) - return keys - - -def _backend_target_present(target: Any) -> bool: - return ( - isinstance(target, Mapping) - and bool(str(target.get("kind") or "")) - and bool(str(target.get("value") or "")) - ) - - -def _compatible_backend_target( - agent_record: _WorkerRecord, - pane_record: _WorkerRecord, -) -> dict[str, Any] | None: - """Retain only verified agent-scoped targets; PaneInfo owns pane targets.""" - agent_target = agent_record.worker.backend_target - pane_target = pane_record.worker.backend_target - if not _backend_target_present(agent_target): - return pane_target if _backend_target_present(pane_target) else None - - kind = str(agent_target.get("kind") or "") - value = str(agent_target.get("value") or "") - if ( - kind == "agent_id" - and _backend_target_present(pane_target) - and str(pane_target.get("kind") or "") == "agent_id" - and str(pane_target.get("value") or "") != value - ): - return pane_target - if kind in _AGENT_SCOPED_BACKEND_TARGET_KINDS and ( - kind != "agent" or value == pane_record.worker.name - ): - return agent_target - return pane_target if _backend_target_present(pane_target) else None - - -def _ambiguous_agent_record( - record: _WorkerRecord, - *, - config: Config | None = None, - observation: Mapping[str, Any] | None = None, -) -> _WorkerRecord: - """Fail closed and retain a unique auditable row for ambiguous ownership.""" - worker = record.worker - target = worker.backend_target - if _backend_target_present(target): - worker = _worker_with_backend_target( - worker, - _private_backend_target( - str(target.get("kind") or ""), - str(target.get("value") or ""), - sendable=False, - reason="ambiguous_pane_match", - ), - ) - private_fingerprint = record.private_fingerprint - if config is not None and observation is not None: - private_fingerprint = worker_binding_private_fingerprint( - host_id=config.host_id, - backend=_BACKEND_NAME, - identity_material={ - "ambiguous_pane_ownership": dict(observation), - "source_private_fingerprint": record.private_fingerprint, - }, - ) - return _WorkerRecord( - worker=worker, - private_fingerprint=private_fingerprint, - workspace_id=record.workspace_id, - pane_id=record.pane_id, - observed_workspace_id=record.observed_workspace_id, - observed_pane_id=record.observed_pane_id, - identity_source=record.identity_source, - terminal_id=record.terminal_id, - agent_session_id=record.agent_session_id, - pane_info_observed=False, - ) - - -def _merge_agent_pane_record( - agent_record: _WorkerRecord, - pane_record: _WorkerRecord | None, -) -> _WorkerRecord: - if pane_record is None: - return agent_record - pane_meta = pane_record.worker.meta - merged_meta = dict(agent_record.worker.meta) - label = pane_meta.get("label") - if isinstance(label, str) and label.strip(): - merged_meta["label"] = label - for key in ("foreground_cwd", "cwd"): - value = pane_meta.get(key) - if isinstance(value, str) and value.strip() and not merged_meta.get(key): - merged_meta[key] = value - - backend_target = _compatible_backend_target(agent_record, pane_record) - worker = agent_record.worker - pane_space_id = pane_record.worker.space_id - if ( - merged_meta != worker.meta - or backend_target != worker.backend_target - or pane_space_id != worker.space_id - ): - worker = Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=pane_space_id, - meta=merged_meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - backend_target=backend_target, - ) - - workspace_id = pane_record.workspace_id - pane_id = pane_record.pane_id - terminal_id = pane_record.terminal_id - if ( - worker == agent_record.worker - and workspace_id == agent_record.workspace_id - and pane_id == agent_record.pane_id - and terminal_id == agent_record.terminal_id - and agent_record.pane_info_observed - ): - return agent_record - return _WorkerRecord( - worker=worker, - private_fingerprint=agent_record.private_fingerprint, - workspace_id=workspace_id, - pane_id=pane_id, - observed_workspace_id=pane_record.observed_workspace_id, - observed_pane_id=pane_record.observed_pane_id, - identity_source=pane_record.identity_source, - terminal_id=terminal_id, - agent_session_id=pane_record.agent_session_id, - pane_info_observed=True, - ) - - -def _collapse_exact_observation_items( - items: Sequence[Mapping[str, Any]], -) -> list[dict[str, Any]]: - """Collapse byte-equivalent JSON observations before ownership cardinality.""" - seen: set[str] = set() - unique: list[dict[str, Any]] = [] - for item in items: - signature = json.dumps( - item, - sort_keys=True, - separators=(",", ":"), - default=str, - ) - if signature in seen: - continue - seen.add(signature) - unique.append(dict(item)) - return unique - - -def _record_ownership_keys( - item: Mapping[str, Any], - record: _WorkerRecord, - *, - pane_shaped: bool = False, -) -> set[tuple[str, str]]: - keys = set( - _record_match_keys( - item, - record, - pane_shaped=pane_shaped, - ) - ) - backend_target = record.worker.backend_target - if _backend_target_present(backend_target): - keys.add( - ( - f"backend:{backend_target.get('kind')}", - str(backend_target.get("value") or ""), - ) - ) - keys.add( - ( - "backend_send_token", - str(backend_target.get("value") or ""), - ) - ) - return keys - - -def _pane_ownership_graph( - agent_items: Sequence[Mapping[str, Any]], - agent_records: Sequence[_WorkerRecord], - pane_items: Sequence[Mapping[str, Any]], - pane_records: Sequence[_WorkerRecord], -) -> tuple[list[set[int]], set[int], set[int]]: - """Return agent-to-pane edges and every ambiguous ownership component.""" - agent_nodes = [("agent", index) for index in range(len(agent_records))] - pane_nodes = [("pane", index) for index in range(len(pane_records))] - adjacency: dict[tuple[str, int], set[tuple[str, int]]] = { - node: set() for node in [*agent_nodes, *pane_nodes] - } - ambiguous_seeds: set[tuple[str, int]] = set() - - pane_indices_by_match: dict[tuple[str, str], set[int]] = {} - pane_indices_by_owner: dict[tuple[str, str], set[int]] = {} - for pane_index, (item, record) in enumerate( - zip(pane_items, pane_records, strict=True) - ): - match_keys = _record_match_keys(item, record, pane_shaped=True) - for key in match_keys: - pane_indices_by_match.setdefault(key, set()).add(pane_index) - for key in _record_ownership_keys(item, record, pane_shaped=True): - pane_indices_by_owner.setdefault(key, set()).add(pane_index) - - # Each private pane, terminal, session, or agent identity may own only one - # PaneInfo row. Exact duplicate rows were already collapsed above. - for pane_indices in pane_indices_by_owner.values(): - if len(pane_indices) < 2: - continue - ordered = sorted(pane_indices) - first_node = ("pane", ordered[0]) - ambiguous_seeds.update(("pane", index) for index in ordered) - for pane_index in ordered[1:]: - pane_node = ("pane", pane_index) - adjacency[first_node].add(pane_node) - adjacency[pane_node].add(first_node) - - agent_matches: list[set[int]] = [] - pane_claimants: dict[int, set[int]] = {} - agent_indices_by_owner: dict[tuple[str, str], set[int]] = {} - for agent_index, (item, record) in enumerate( - zip(agent_items, agent_records, strict=True) - ): - matches: set[int] = set() - for key in _record_match_keys(item, record): - matches.update(pane_indices_by_match.get(key, ())) - agent_matches.append(matches) - agent_node = ("agent", agent_index) - owner_keys = _record_ownership_keys(item, record) - for key in owner_keys: - agent_indices_by_owner.setdefault(key, set()).add(agent_index) - if key[0] != "backend_send_token": - continue - for pane_index in pane_indices_by_owner.get(key, ()): - if pane_index in matches: - continue - pane_node = ("pane", pane_index) - adjacency[agent_node].add(pane_node) - adjacency[pane_node].add(agent_node) - ambiguous_seeds.add(agent_node) - ambiguous_seeds.add(pane_node) - for pane_index in matches: - pane_node = ("pane", pane_index) - adjacency[agent_node].add(pane_node) - adjacency[pane_node].add(agent_node) - pane_claimants.setdefault(pane_index, set()).add(agent_index) - if len(matches) > 1: - ambiguous_seeds.add(agent_node) - ambiguous_seeds.update(("pane", index) for index in matches) - - for agent_indices in agent_indices_by_owner.values(): - if len(agent_indices) < 2: - continue - if not any(agent_matches[index] for index in agent_indices): - continue - ordered = sorted(agent_indices) - first_node = ("agent", ordered[0]) - ambiguous_seeds.update(("agent", index) for index in ordered) - for agent_index in ordered[1:]: - agent_node = ("agent", agent_index) - adjacency[first_node].add(agent_node) - adjacency[agent_node].add(first_node) - - for pane_index, claimants in pane_claimants.items(): - if len(claimants) < 2: - continue - ambiguous_seeds.add(("pane", pane_index)) - ambiguous_seeds.update(("agent", index) for index in claimants) - - ambiguous_nodes = set(ambiguous_seeds) - pending = list(ambiguous_seeds) - while pending: - node = pending.pop() - for adjacent in adjacency[node]: - if adjacent in ambiguous_nodes: - continue - ambiguous_nodes.add(adjacent) - pending.append(adjacent) - - return ( - agent_matches, - {index for kind, index in ambiguous_nodes if kind == "agent"}, - {index for kind, index in ambiguous_nodes if kind == "pane"}, - ) - - -def _records_from_agent_and_pane_payloads( - config: Config | None, - agent_payload: Any, - pane_payload: Any, - *, - include_unmatched_panes: bool = True, -) -> list[_WorkerRecord]: - """Merge PaneInfo identity only across a one-to-one ownership graph.""" - pane_items = _collapse_exact_observation_items( - [ - item - for item in _payload_items( - pane_payload, - ("panes", "items", "data", "results", "result"), - ) - if _pane_has_agent(item) - ] - ) - pane_records = [ - _worker_record_from_item( - item, - config, - pane_info_observed=True, - identity_source="pane.list", - ) - for item in pane_items - ] - agent_items = _collapse_exact_observation_items( - _payload_items( - agent_payload, - ("agents", "workers", "data", "items", "results", "result"), - ) - ) - agent_records = [ - _agent_observation_record(item, config) - for item in agent_items - ] - agent_matches, ambiguous_agents, ambiguous_panes = _pane_ownership_graph( - agent_items, - agent_records, - pane_items, - pane_records, - ) - - records: list[_WorkerRecord] = [] - consumed_pane_indices: set[int] = set() - for agent_index, record in enumerate(agent_records): - matched_pane_indices = agent_matches[agent_index] - consumed_pane_indices.update(matched_pane_indices) - if agent_index in ambiguous_agents: - records.append( - _ambiguous_agent_record( - record, - config=config, - observation=agent_items[agent_index], - ) - ) - elif len(matched_pane_indices) == 1: - matched_pane_index = next(iter(matched_pane_indices)) - records.append( - _merge_agent_pane_record(record, pane_records[matched_pane_index]) - ) - else: - records.append(record) - if include_unmatched_panes: - records.extend( - _ambiguous_agent_record( - record, - config=config, - observation=pane_items[index], - ) - if index in ambiguous_panes - else record - for index, record in enumerate(pane_records) - if index not in consumed_pane_indices - ) - return records - - -def _workers_from_pane_payload( - payload: Any, - config: Config | None = None, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[Worker]: - """Extract pane workers through the ownership graph.""" - records = _records_from_agent_and_pane_payloads( - config, - None, - payload, - ) - return _deduplicate_worker_records(records, stored_bindings) - - -def _workers_and_bindings_from_pane_payload( - payload: Any, - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> tuple[list[Worker], list[WorkerBinding]]: - records = _records_from_agent_and_pane_payloads( - config, - None, - payload, - ) - return _workers_and_bindings_from_records( - config, - records, - stored_bindings=stored_bindings, - ) - - -def _probe_payload_variants( - variants: Sequence[Sequence[str]], - config: Config, - budget: _ProbeBudget | None = None, -) -> tuple[str, Any]: - outcomes: list[str] = [] - for args in variants: - if budget is None: - outcome, payload = _probe_herdr(args, config) - else: - try: - outcome, payload = _probe_herdr(args, config, budget) - except TypeError: - outcome, payload = _probe_herdr(args, config) - if outcome == "ok": - return outcome, payload - if outcome in _DEADLINE_EXHAUSTED_OUTCOMES: - return outcome, None - outcomes.append(outcome) - if outcomes and all(outcome == "launch_error" for outcome in outcomes): - return "launch_error", None - if "malformed_json" in outcomes: - return "malformed_json", None - if "nonzero" in outcomes: - return "nonzero", None - return outcomes[-1] if outcomes else "nonzero", None - - -def _degraded_observation( - outcome: str, - message: str, - *, - spaces: Sequence[Space] | None = None, - workers: Sequence[Worker] | None = None, -) -> HerdrCommandObservation: - observed_spaces = list(spaces or []) - observed_workers = list(workers or []) - health = herdr_backend_health( - outcome, - message=message, - spaces=observed_spaces, - workers=observed_workers, - ) - return HerdrCommandObservation( - spaces=observed_spaces, - workers=observed_workers, - status=health.status, - outcome=health.outcome, - message=health.message, - backend_health=[health], - ) - - -def _snapshot_observation( - spaces: list[Space], - workers: list[Worker], - bindings: list[WorkerBinding], - outcome: str, - *, - message: str | None = None, -) -> HerdrSnapshotObservation: - health = herdr_backend_health( - outcome, - message=message, - spaces=spaces, - workers=workers, - ) - return HerdrSnapshotObservation( - spaces=spaces, - workers=workers, - bindings=bindings, - backend_health=[health], - ) - - -def fetch_herdr_command_observation( - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> HerdrCommandObservation: - """Return Herdr observations plus health metadata for mutation safety.""" - try: - if shutil.which(config.herdr_bin) is None: - return _degraded_observation("missing_binary", "Herdr binary is unavailable") - except (TypeError, ValueError, OSError): - return _degraded_observation("launch_error", "Herdr binary could not be inspected") - - budget = _ProbeBudget.from_config(config, planned_probes=5) - workspace_outcome, workspace_payload = _probe_payload_variants( - [ - ["workspace", "list"], - ["workspace", "list", "--json"], - ], - config, - budget, - ) - if workspace_outcome != "ok": - return _degraded_observation( - workspace_outcome, - "Herdr workspace observation is not healthy", - ) - - agent_outcome, agent_payload = _probe_payload_variants( - [ - ["agent", "list"], - ["agent", "list", "--json"], - ], - config, - budget, - ) - if agent_outcome != "ok": - return _degraded_observation( - agent_outcome, - "Herdr agent observation is not healthy", - ) - - spaces = _spaces_from_payload(workspace_payload) - try: - pane_outcome, pane_payload = _probe_herdr(["pane", "list"], config, budget) - except TypeError: - pane_outcome, pane_payload = _probe_herdr(["pane", "list"], config) - records = _records_from_agent_and_pane_payloads( - config, - agent_payload, - pane_payload if pane_outcome == "ok" else None, - include_unmatched_panes=not bool( - _payload_items( - agent_payload, - ("agents", "workers", "data", "items", "results", "result"), - ) - ), - ) - try: - workers, bindings = _workers_and_bindings_from_records( - config, - records, - stored_bindings=stored_bindings, - require_authenticated_continuity=pane_outcome == "ok", - ) - except (HerdrContinuityUnavailableError, InstallationKeyError): - return _degraded_observation( - "continuity_unavailable", - _HEALTH_MESSAGES["continuity_unavailable"], - spaces=spaces, - ) - if pane_outcome != "ok": - return _degraded_observation( - pane_outcome, - "Herdr pane continuity observation is not healthy", - spaces=spaces, - workers=workers, - ) - - return HerdrCommandObservation( - spaces=spaces, - workers=workers, - status="healthy", - outcome="healthy_non_empty" if spaces or workers else "empty_healthy", - bindings=bindings, - backend_health=[ - herdr_backend_health( - "healthy_non_empty" if spaces or workers else "empty_healthy", - spaces=spaces, - workers=workers, - ) - ], - ) - - -def _state_result( - spaces: list[Space], - workers: list[Worker], - bindings: list[WorkerBinding], - include_bindings: bool, -) -> tuple[list[Space], list[Worker]] | tuple[list[Space], list[Worker], list[WorkerBinding]]: - if include_bindings: - return spaces, workers, bindings - return spaces, workers - - -def fetch_herdr_snapshot_observation( - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, - *, - require_authenticated_continuity: bool = True, -) -> HerdrSnapshotObservation: - """Return Herdr snapshot observations plus public backend health.""" - try: - if shutil.which(config.herdr_bin) is None: - return _snapshot_observation( - [], - [], - [], - "missing_binary", - message=_HEALTH_MESSAGES["missing_binary"], - ) - except (TypeError, ValueError, OSError): - return _snapshot_observation( - [], - [], - [], - "launch_error", - message="Herdr binary could not be inspected", - ) - - budget = _ProbeBudget.from_config(config, planned_probes=5) - workspace_outcome, workspace_payload = _probe_payload_variants( - [ - ["workspace", "list"], - ["workspace", "list", "--json"], - ], - config, - budget, - ) - if workspace_outcome in _DEADLINE_EXHAUSTED_OUTCOMES: - return _snapshot_observation( - [], - [], - [], - workspace_outcome, - message=_HEALTH_MESSAGES[workspace_outcome], - ) - - agent_outcome, agent_payload = _probe_payload_variants( - [ - ["agent", "list"], - ["agent", "list", "--json"], - ], - config, - budget, - ) - - spaces = _spaces_from_payload(workspace_payload) - if agent_outcome in _DEADLINE_EXHAUSTED_OUTCOMES: - return _snapshot_observation( - spaces, - [], - [], - agent_outcome, - message=_HEALTH_MESSAGES[agent_outcome], - ) - - try: - pane_outcome, pane_payload = _probe_herdr(["pane", "list"], config, budget) - except TypeError: - pane_outcome, pane_payload = _probe_herdr(["pane", "list"], config) - records = _records_from_agent_and_pane_payloads( - config, - agent_payload, - pane_payload if pane_outcome == "ok" else None, - include_unmatched_panes=not bool( - _payload_items( - agent_payload, - ("agents", "workers", "data", "items", "results", "result"), - ) - ), - ) - try: - workers, bindings = _workers_and_bindings_from_records( - config, - records, - stored_bindings=stored_bindings, - require_authenticated_continuity=( - require_authenticated_continuity and pane_outcome == "ok" - ), - ) - except (HerdrContinuityUnavailableError, InstallationKeyError): - return _snapshot_observation( - spaces, - [], - [], - "continuity_unavailable", - message=_HEALTH_MESSAGES["continuity_unavailable"], - ) - - failed_outcomes = [ - outcome - for outcome in (workspace_outcome, agent_outcome, pane_outcome) - if outcome not in {"ok"} - ] - if failed_outcomes: - outcome = failed_outcomes[0] - return _snapshot_observation( - spaces, - workers, - bindings, - outcome, - message=_HEALTH_MESSAGES.get(outcome, _HEALTH_MESSAGES["unknown"]), - ) - - return _snapshot_observation( - spaces, - workers, - bindings, - "healthy_non_empty" if spaces or workers else "empty_healthy", - ) - - -def fetch_herdr_state( - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, - *, - include_bindings: bool = False, -) -> tuple[list[Space], list[Worker]] | tuple[list[Space], list[Worker], list[WorkerBinding]]: - """Return neutral spaces and workers from the Herdr CLI, or empty lists.""" - observation = fetch_herdr_snapshot_observation( - config, - stored_bindings=stored_bindings, - require_authenticated_continuity=False, - ) - return _state_result(observation.spaces, observation.workers, observation.bindings, include_bindings) diff --git a/src/tendwire/backends/herdr_events.py b/src/tendwire/backends/herdr_events.py deleted file mode 100644 index fa09d08..0000000 --- a/src/tendwire/backends/herdr_events.py +++ /dev/null @@ -1,2454 +0,0 @@ -"""Opt-in Herdr socket event backend and reconciliation layer. - -This module is intentionally imported only from the explicit socket backend -path. It reuses the PR8 socket client for transport and the Herdr CLI adapter's -projection helpers for Tendwire model normalization. -""" - -from __future__ import annotations - -import inspect -import logging -import threading -import time -from collections import OrderedDict -from collections.abc import Callable, Iterable, Mapping, Sequence -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -from ..config import DEFAULT_TURN_MODEL, Config -from ..core.models import ( - BackendHealth, - Snapshot, - Space, - Worker, - WorkerBinding, - normalize_status, - utc_timestamp, -) -from ..worker_identity import ( - InstallationKeyError, - STABLE_KEY_VERSION, - canonical_herdr_pane_identity, - is_stable_worker_key, -) -from ..core.projector import project_from_observations -from ..store.sqlite import ( - SnapshotObservationContext, - SnapshotRetentionPolicy, - expire_stale_worker_bindings, - expire_worker_bindings, - latest_snapshot, - list_worker_bindings, - maybe_run_automatic_store_maintenance, - save_snapshot, - upsert_worker_bindings, -) -from .herdr_cli import ( - HerdrContinuityUnavailableError, - _pane_has_agent, - _payload_items, - _spaces_from_payload, - _records_from_agent_and_pane_payloads, - _worker_record_from_item, - _workers_and_bindings_from_records, - _strip_stable_key_fields, - herdr_backend_health, -) -from .herdr_protocol import ( - HERDR_EVENTS_SUBSCRIBE_METHOD, - HERDR_OFFICIAL_EVENT_NAMES, - HerdrEnvelopeError, - HerdrErrorResponse, - HerdrMalformedLineError, - HerdrProtocolError, - build_events_subscribe_params, -) -from .herdr_socket import ( - HerdrSocketClient, - HerdrSocketConnectionError, - HerdrSocketDisconnectedError, - HerdrSocketTimeoutError, -) - - -BACKEND_NAME = "herdr" -DEFAULT_SUBSCRIBE_METHOD = HERDR_EVENTS_SUBSCRIBE_METHOD -DEFAULT_DEBOUNCE_SECONDS = 0.05 -DEFAULT_DEDUPE_SIZE = 512 -DEFAULT_MAX_BATCH_SIZE = 64 -DEFAULT_RECONNECT_DELAY_SECONDS = 0.25 -_LOGGER = logging.getLogger(__name__) - -_AGENT_PAYLOAD_KEYS = ("agents", "workers", "data", "items", "results", "result") -_PANE_PAYLOAD_KEYS = ("panes", "items", "data", "results", "result") -_SUPPORTED_EVENT_NAMES = HERDR_OFFICIAL_EVENT_NAMES -_SUPPORTED_EVENT_NAME_SET = frozenset(_SUPPORTED_EVENT_NAMES) -_HERDR_074_EVENT_NAMES = tuple( - event_name for event_name in _SUPPORTED_EVENT_NAMES if event_name != "pane.updated" -) -_HERDR_074_PANE_SCOPED_REPLAY_EVENT_NAMES = frozenset( - { - "workspace.focused", - "pane.focused", - "pane.agent_detected", - "pane.output_matched", - } -) -_HERDR_074_PANE_SCOPED_FALLBACK_EVENT_NAMES = tuple( - event_name - for event_name in _HERDR_074_EVENT_NAMES - if event_name not in _HERDR_074_PANE_SCOPED_REPLAY_EVENT_NAMES -) -_PARAMETERIZED_EVENT_NAMES = frozenset( - { - "pane.agent_status_changed", - "pane.output_matched", - } -) -_GLOBAL_EVENT_NAMES = tuple( - event_name - for event_name in _SUPPORTED_EVENT_NAMES - if event_name not in _PARAMETERIZED_EVENT_NAMES -) -_CLOSED_EVENT_NAMES = frozenset({"pane.closed", "pane.exited"}) -_SPACE_EVENT_NAMES = frozenset( - { - "workspace.created", - "workspace.updated", - "workspace.renamed", - "workspace.closed", - "workspace.focused", - } -) -_WORKTREE_EVENT_NAMES = frozenset({"worktree.created", "worktree.opened", "worktree.removed"}) -# ``pane.updated`` is normalized from Herdr 0.7.5's scalar -# ``PaneOutputChanged`` lifecycle event and must never rebuild worker identity. -_PANE_WORKER_EVENT_NAMES = frozenset({"pane.created", "pane.focused"}) -class HerdrEventBackendError(Exception): - """Base error for the opt-in Herdr socket event backend.""" - - -@dataclass(frozen=True) -class HerdrEventBackendHealth: - """Small in-memory health state for the Herdr socket backend.""" - - status: str - outcome: str - observed_at: str - message: str - - def to_backend_health( - self, - *, - spaces: Sequence[Space] | None = None, - workers: Sequence[Worker] | None = None, - ) -> BackendHealth: - return herdr_backend_health( - self.outcome, - observed_at=self.observed_at, - message=self.message, - spaces=spaces or [], - workers=workers or [], - ) - - -@dataclass(frozen=True) -class HerdrEventId: - """Forward-compatible authoritative producer event identifier.""" - - value: str - - -@dataclass(frozen=True) -class HerdrProducerSequence: - """Forward-compatible producer-scoped sequence identifier.""" - - producer_id: str - sequence: str - - -HerdrProducerIdentity = HerdrEventId | HerdrProducerSequence - - -@dataclass(frozen=True) -class NormalizedHerdrEvent: - """A validated Herdr event with optional durable producer identity.""" - - name: str - payload: Mapping[str, Any] - producer_identity: HerdrProducerIdentity | None - - -def _compact_key(value: object) -> str: - return str(value).strip().lower().replace("-", "_").replace(".", "_").replace(":", "_") - - -def _field_value(item: Mapping[str, Any], expected_key: str) -> Any: - expected = _compact_key(expected_key) - for key, value in item.items(): - if _compact_key(key) == expected: - return value - return None - - -def _first_text(item: Mapping[str, Any], keys: Iterable[str]) -> str | None: - for key in keys: - value = _field_value(item, key) - if value is None: - continue - if isinstance(value, Mapping): - nested = _first_text(value, ("id", "value", "name", "label")) - if nested: - return nested - continue - if isinstance(value, (str, int, float, bool)): - text = str(value) - if text: - return text - return None - - -def _safe_mapping(value: Any) -> dict[str, Any]: - return dict(value) if isinstance(value, Mapping) else {} - - -def _call_with_optional_keywords( - callback: Callable[..., Any], - args: tuple[Any, ...], - kwargs: Mapping[str, Any], -) -> Any: - """Invoke once, omitting optional keywords only when the signature requires it.""" - try: - signature = inspect.signature(callback) - except (TypeError, ValueError): - return callback(*args, **dict(kwargs)) - try: - signature.bind(*args, **dict(kwargs)) - except TypeError: - return callback(*args) - return callback(*args, **dict(kwargs)) - - -def _entity_payload_with_source(payload: Mapping[str, Any], *entity_names: str) -> tuple[dict[str, Any], str | None]: - """Return an event entity object plus the nested entity name selected.""" - merged: dict[str, Any] = {} - nested_entity_keys: set[str] = set() - selected_entity: str | None = None - for entity_name in entity_names: - nested = _field_value(payload, entity_name) - if isinstance(nested, Mapping): - merged.update(dict(nested)) - compact_name = _compact_key(entity_name) - nested_entity_keys.add(compact_name) - selected_entity = compact_name - break - for key, value in payload.items(): - if _compact_key(key) in nested_entity_keys: - continue - merged.setdefault(str(key), value) - return merged, selected_entity - - -def _entity_payload(payload: Mapping[str, Any], *entity_names: str) -> dict[str, Any]: - """Return a single object for an event entity while preserving scalar hints.""" - item, _selected_entity = _entity_payload_with_source(payload, *entity_names) - return item - - -def _privatize_pane_event_id(item: dict[str, Any]) -> dict[str, Any]: - """Treat generic ``id`` on pane events as a private pane identifier.""" - raw_id = _first_text(item, ("id",)) - if raw_id and _first_text(item, ("pane_id", "paneId")) is None: - item["pane_id"] = raw_id - for key in list(item): - if _compact_key(key) == "id": - item.pop(key, None) - return item - - -def _pane_event_payload_with_provenance( - payload: Mapping[str, Any], - *entity_names: str, - allow_top_level_pane_info: bool = False, -) -> tuple[dict[str, Any], bool]: - """Return a pane-event item and whether a full PaneInfo authorizes it.""" - item, selected_entity = _entity_payload_with_source(payload, *entity_names) - # EventData's internally tagged discriminator is envelope metadata, not a - # PaneInfo field. Keeping it would make event and pane.list projections - # differ even after their identity pair was canonicalized. - for key in list(item): - if _compact_key(key) == "type": - item.pop(key, None) - # Scalar agent/status events and nested agent/worker objects are not - # PaneInfo, even when they repeat workspace_id/pane_id. Treating them as - # PaneInfo was the observation-layer path that admitted alternate identity - # representations. Pane lifecycle events historically also carry a full - # top-level PaneInfo, so their caller opts into that established shape. - top_level_identity = canonical_herdr_pane_identity( - _first_text(item, ("workspace_id", "workspaceId")), - _first_text(item, ("pane_id", "paneId")), - ) - pane_info_observed = selected_entity == "pane" or ( - selected_entity is None - and allow_top_level_pane_info - and top_level_identity is not None - ) - return _privatize_pane_event_id(item), pane_info_observed - - -def _pane_event_payload(payload: Mapping[str, Any], *entity_names: str) -> dict[str, Any]: - """Return a pane-event item without exposing pane ``id`` as public worker id.""" - item, _pane_info_observed = _pane_event_payload_with_provenance( - payload, - *entity_names, - ) - return item - - -def _event_alias_key(name: str) -> str: - return "_".join(part for part in _compact_key(name).split("_") if part) - - -def _canonical_event_name(raw_name: Any) -> str | None: - if not isinstance(raw_name, str) or not raw_name.strip(): - return None - event_name = raw_name.strip() - if event_name in _SUPPORTED_EVENT_NAME_SET: - return event_name - aliases = {_event_alias_key(name): name for name in _SUPPORTED_EVENT_NAMES} - aliases.update( - { - "agent_detected": "pane.agent_detected", - "agent_observed": "pane.agent_detected", - "agent_status_changed": "pane.agent_status_changed", - "agent_status_updated": "pane.agent_status_changed", - "pane_output_changed": "pane.updated", - "pane_observed": "pane.created", - "pane_detected": "pane.created", - "workspace_observed": "workspace.updated", - "workspace_detected": "workspace.created", - "worktree_observed": "worktree.opened", - "worktree_detected": "worktree.created", - "worktree_updated": "worktree.opened", - "worktree_changed": "worktree.opened", - "worktree_closed": "worktree.removed", - "worktree_deleted": "worktree.removed", - } - ) - return aliases.get(_event_alias_key(event_name)) - - -def _strict_producer_id(value: object) -> str | None: - if type(value) is not str or not value or any(character.isspace() for character in value): - return None - return value - - -def _strict_producer_sequence(value: object) -> str | None: - if type(value) is not int or value < 0: - return None - return str(value) - - -def _producer_identity(envelope: Mapping[str, Any]) -> HerdrProducerIdentity | None: - """Return only valid explicit top-level producer identity when present. - - Herdr's confirmed EventEnvelope has only ``event`` and ``data``. The - identity fields handled here are forward-compatible optional metadata; - malformed metadata leaves the event idless, and entity fields inside - ``data`` are never durable event identity. - """ - if "event_id" in envelope: - event_id = _strict_producer_id(envelope.get("event_id")) - return HerdrEventId(event_id) if event_id is not None else None - server_present = "server_id" in envelope - sequence_present = "sequence" in envelope - if not server_present and not sequence_present: - return None - producer_id = _strict_producer_id(envelope.get("server_id")) - sequence = _strict_producer_sequence(envelope.get("sequence")) - if not server_present or not sequence_present or producer_id is None or sequence is None: - return None - return HerdrProducerSequence(producer_id, sequence) - - -def normalize_event(envelope: Mapping[str, Any]) -> NormalizedHerdrEvent | None: - """Normalize a Herdr event envelope; unsupported events return ``None``. - - Confirmed Herdr envelopes use ``event`` and ``data`` and are intentionally - idless. ``payload`` remains receive-only compatibility for older clients. - """ - name = _canonical_event_name(envelope.get("event")) - if name is None: - return None - payload = envelope.get("data") if "data" in envelope else envelope.get("payload", {}) - if payload is None: - payload = {} - if not isinstance(payload, Mapping): - return None - return NormalizedHerdrEvent( - name=name, - payload=dict(payload), - producer_identity=_producer_identity(envelope), - ) - - -def _worker_copy( - worker: Worker, - *, - worker_id: str | None = None, - name: str | None = None, - status: str | None = None, - space_id: str | None = None, - meta: Mapping[str, Any] | None = None, - last_seen_at: str | None = None, - summary: str | None = None, - backend_target: Mapping[str, Any] | None = None, -) -> Worker: - return Worker( - id=worker_id if worker_id is not None else worker.id, - name=name if name is not None else worker.name, - status=status if status is not None else worker.status, - space_id=space_id if space_id is not None else worker.space_id, - meta=dict(meta) if meta is not None else dict(worker.meta), - last_seen_at=last_seen_at if last_seen_at is not None else worker.last_seen_at, - summary=summary if summary is not None else worker.summary, - backend_target=dict(backend_target) if isinstance(backend_target, Mapping) else worker.backend_target, - ) - - -def _merge_worker_update( - existing: Worker | None, - observed: Worker, - *, - status: str | None = None, - preserve_existing_continuity: bool = False, -) -> Worker: - if existing is None: - if status is not None: - return _worker_copy(observed, status=status) - return observed - merged_meta = _strip_stable_key_fields(existing.meta) - if ( - preserve_existing_continuity - and is_stable_worker_key(existing.meta.get("stable_key")) - and type(existing.meta.get("stable_key_version")) is int - and existing.meta.get("stable_key_version") == STABLE_KEY_VERSION - ): - merged_meta["stable_key"] = existing.meta["stable_key"] - merged_meta["stable_key_version"] = STABLE_KEY_VERSION - merged_meta.update(observed.meta) - observed_name_is_identity = observed.name in {observed.id, "unknown"} - resolved_status = status if status is not None else observed.status - if resolved_status == "unknown" and existing.status != "unknown": - resolved_status = existing.status - return Worker( - id=existing.id, - name=existing.name if observed_name_is_identity else observed.name, - status=resolved_status, - space_id=observed.space_id or existing.space_id, - meta=merged_meta, - last_seen_at=observed.last_seen_at or utc_timestamp(), - summary=observed.summary or existing.summary, - backend_target=observed.backend_target or existing.backend_target, - ) - - -def _closed_worker(worker: Worker) -> Worker: - return _worker_copy(worker, status="closed", last_seen_at=worker.last_seen_at or utc_timestamp()) - - -def _observed_worker_count(workers: Sequence[Worker]) -> int: - return len([worker for worker in workers if worker.status != "closed"]) - - -def _binding_target(binding: WorkerBinding) -> tuple[str, str]: - return (binding.target_kind, binding.target_value) - -def _worker_state_equal(left: Worker, right: Worker) -> bool: - """Compare effective worker state while ignoring observation timestamps.""" - return left.fingerprint == right.fingerprint and left.backend_target == right.backend_target - - -def _binding_state_equal(left: WorkerBinding, right: WorkerBinding) -> bool: - """Compare effective private routing while ignoring observation timestamps.""" - return ( - left.host_id, - left.worker_id, - left.worker_fingerprint, - left.backend, - left.target_kind, - left.target_value, - left.turn_target_kind, - left.turn_target_value, - left.sendable, - left.reason, - left.expires_at, - left.private_fingerprint, - ) == ( - right.host_id, - right.worker_id, - right.worker_fingerprint, - right.backend, - right.target_kind, - right.target_value, - right.turn_target_kind, - right.turn_target_value, - right.sendable, - right.reason, - right.expires_at, - right.private_fingerprint, - ) - - - - -def _target_pairs_from_item(item: Mapping[str, Any], *, old_first: bool = False) -> list[tuple[str, str]]: - old_pairs = [ - ("pane_id", _first_text(item, ("old_pane_id", "previous_pane_id", "from_pane_id", "source_pane_id"))), - ( - "terminal_id", - _first_text(item, ("old_terminal_id", "previous_terminal_id", "from_terminal_id", "source_terminal_id")), - ), - ] - current_pairs = [ - ("agent_id", _first_text(item, ("agent_id", "agentId"))), - ("terminal_id", _first_text(item, ("terminal_id", "terminalId"))), - ("pane_id", _first_text(item, ("pane_id", "paneId", "id"))), - ("agent", _first_text(item, ("agent",))), - ("name", _first_text(item, ("name", "label"))), - ] - pairs = [*old_pairs, *current_pairs] if old_first else [*current_pairs, *old_pairs] - seen: set[tuple[str, str]] = set() - result: list[tuple[str, str]] = [] - for kind, value in pairs: - if not value: - continue - pair = (kind, value) - if pair in seen: - continue - seen.add(pair) - result.append(pair) - return result - - -def _new_move_target(item: Mapping[str, Any]) -> tuple[str, str] | None: - candidates = ( - ("pane_id", ("new_pane_id", "to_pane_id", "target_pane_id", "pane_id", "paneId", "id")), - ("terminal_id", ("new_terminal_id", "to_terminal_id", "target_terminal_id", "terminal_id", "terminalId")), - ("agent_id", ("agent_id", "agentId")), - ) - for kind, keys in candidates: - value = _first_text(item, keys) - if value: - return kind, value - return None - - -def _has_public_worker_identity(item: Mapping[str, Any]) -> bool: - return ( - _first_text(item, ("worker_id", "id", "slug", "agent_id", "agent", "name", "label", "title")) - is not None - ) - -def _has_authoritative_identity_tuple(item: Mapping[str, Any]) -> bool: - return ( - _field_value(item, "workspace_id") is not None - and _field_value(item, "pane_id") is not None - ) - - -def _has_authoritative_binding_target(item: Mapping[str, Any]) -> bool: - agent_session = _safe_mapping(_field_value(item, "agent_session")) - return ( - _first_text(item, ("agent_id", "terminal_id")) is not None - or _first_text(agent_session, ("value", "id")) is not None - ) - - -def _authenticated_local_stable_key(worker: Worker) -> str | None: - value = worker.meta.get("stable_key") - version = worker.meta.get("stable_key_version") - if ( - is_stable_worker_key(value) - and type(version) is int - and version == STABLE_KEY_VERSION - ): - return str(value) - return None - - -class HerdrEventBackend: - """Maintain Tendwire projections from Herdr socket reconcile and events.""" - - def __init__( - self, - config: Config, - *, - client_factory: Callable[[Config], HerdrSocketClient] | None = None, - subscribe_method: str = DEFAULT_SUBSCRIBE_METHOD, - debounce_seconds: float | None = None, - reconcile_interval_seconds: float | None = None, - max_workers: int | None = None, - output_excerpt_chars: int | None = None, - dedupe_size: int = DEFAULT_DEDUPE_SIZE, - max_batch_size: int = DEFAULT_MAX_BATCH_SIZE, - reconnect_delay_seconds: float = DEFAULT_RECONNECT_DELAY_SECONDS, - stop_event: threading.Event | None = None, - ) -> None: - self.config = config - self.client_factory = client_factory or self._default_client_factory - requested_subscribe_method = str(subscribe_method or DEFAULT_SUBSCRIBE_METHOD) - if requested_subscribe_method != HERDR_EVENTS_SUBSCRIBE_METHOD: - raise HerdrEventBackendError("Herdr event backend requires events.subscribe") - self.subscribe_method = HERDR_EVENTS_SUBSCRIBE_METHOD - configured_debounce = config.event_debounce_seconds if debounce_seconds is None else debounce_seconds - configured_reconcile = ( - config.reconcile_interval_seconds - if reconcile_interval_seconds is None - else reconcile_interval_seconds - ) - self.debounce_seconds = max(0.0, float(configured_debounce)) - self.reconcile_interval_seconds = max(0.0, float(configured_reconcile)) - self.max_workers = max(1, int(config.max_workers if max_workers is None else max_workers)) - self.output_excerpt_chars = max( - 1, - int(config.output_excerpt_chars if output_excerpt_chars is None else output_excerpt_chars), - ) - self.dedupe_size = max(1, int(dedupe_size)) - self.max_batch_size = max(1, int(max_batch_size)) - self.reconnect_delay_seconds = max(0.0, float(reconnect_delay_seconds)) - self.stop_event = stop_event or threading.Event() - self._lock = threading.RLock() - self._ready = threading.Event() - self._thread: threading.Thread | None = None - self._producer_dedupe: OrderedDict[HerdrProducerIdentity, None] = OrderedDict() - self._pending_events: list[NormalizedHerdrEvent] = [] - self._spaces: dict[str, Space] = {} - self._workers: dict[str, Worker] = {} - self._bindings: dict[str, WorkerBinding] = {} - # pane_id -> terminal_id, remembered from reconcile pane lists so that - # pane-id-only events (pane.agent_status_changed carries no terminal id) - # can still resolve to the terminal-targeted stored binding. - self._pane_terminals: dict[str, str] = {} - self._pane_owners: dict[str, set[str]] = {} - self._terminal_owners: dict[str, set[str]] = {} - self._session_owners: dict[str, set[str]] = {} - self._event_continuity_revalidated = False - self._health = self._health_for("unknown") - self._last_event_at: str | None = None - self._last_reconcile_at: str | None = None - self._last_snapshot_at: str | None = None - self._last_cap_status_at: str | None = None - self._automatic_maintenance_status: dict[str, Any] | None = None - self._next_reconcile_monotonic: float | None = None - self._subscription_pane_ids: list[str] = [] - self._load_existing_state() - - @staticmethod - def _default_client_factory(config: Config) -> HerdrSocketClient: - return HerdrSocketClient(timeout=config.herdr_timeout_seconds) - - @property - def db_path(self) -> Path: - if self.config.db_path is None: - raise HerdrEventBackendError("socket event backend requires a sqlite db path") - return Path(self.config.db_path) - - @property - def health(self) -> HerdrEventBackendHealth: - with self._lock: - return self._health - - @property - def operational_status(self) -> dict[str, Any]: - with self._lock: - return { - "status": self._health.status, - "outcome": self._health.outcome, - "ready": self.ready, - "running": self.running, - "last_event_at": self._last_event_at, - "last_reconcile_at": self._last_reconcile_at, - "last_snapshot_at": self._last_snapshot_at, - "last_cap_status_at": self._last_cap_status_at, - "reconcile_enabled": self.reconcile_interval_seconds > 0, - "automatic_maintenance": ( - dict(self._automatic_maintenance_status) - if self._automatic_maintenance_status is not None - else None - ), - } - - @property - def ready(self) -> bool: - return self._ready.is_set() - - @property - def running(self) -> bool: - thread = self._thread - return thread is not None and thread.is_alive() - - def _health_for(self, outcome: str) -> HerdrEventBackendHealth: - health = herdr_backend_health(outcome) - return HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or utc_timestamp(), - message=health.message, - ) - - def _save_snapshot( - self, - snapshot: Snapshot, - *, - observation: SnapshotObservationContext, - worker_bindings: Iterable[WorkerBinding] | None = None, - binding_observation_authoritative: bool = False, - binding_workers_present: bool = True, - ) -> None: - if not save_snapshot( - self.db_path, - snapshot, - turn_model=DEFAULT_TURN_MODEL, - observation=observation, - worker_bindings=worker_bindings, - binding_backend=BACKEND_NAME if worker_bindings is not None else None, - binding_observation_authoritative=binding_observation_authoritative, - binding_workers_present=binding_workers_present, - ): - raise RuntimeError("snapshot rejected by store ordering") - policy = SnapshotRetentionPolicy( - retention_days=self.config.snapshot_retention_days, - retention_count=self.config.snapshot_retention_count, - batch_size=self.config.snapshot_maintenance_batch_size, - ) - try: - result = maybe_run_automatic_store_maintenance( - self.db_path, - policy=policy, - turn_model=DEFAULT_TURN_MODEL, - acknowledged_final_retention_days=( - self.config.acknowledged_final_retention_days - ), - acknowledged_final_retention_count=( - self.config.acknowledged_final_retention_count - ), - command_retry_horizon_seconds=( - self.config.command_retry_horizon_seconds - ), - command_receipt_retention_seconds=( - self.config.command_receipt_retention_seconds - ), - command_receipt_retention_count=( - self.config.command_receipt_retention_count - ), - cadence_seconds=self.config.store_maintenance_cadence_seconds, - ) - snapshot_result = result.get("snapshot") - snapshot_counts = snapshot_result if isinstance(snapshot_result, Mapping) else {} - maintenance_status = { - "ok": bool(result.get("ok")), - "status": str(result.get("status") or "unknown"), - "due": bool(result.get("due")), - "examined": int(snapshot_counts.get("examined") or 0), - "deleted": int(snapshot_counts.get("deleted") or 0), - "remaining_candidates": bool(snapshot_counts.get("remaining_candidates")), - } - except Exception: - self._automatic_maintenance_status = { - "ok": False, - "status": "failed", - "due": False, - "examined": 0, - "deleted": 0, - "remaining_candidates": False, - } - else: - self._automatic_maintenance_status = maintenance_status - - def _load_existing_state(self) -> None: - try: - snapshot = latest_snapshot(self.db_path, self.config.host_id) - except Exception: - snapshot = None - if snapshot is not None: - self._spaces = {space.id: space for space in snapshot.spaces} - self._workers = {worker.id: worker for worker in snapshot.workers} - self._last_snapshot_at = snapshot.updated_at - for health in snapshot.backend_health: - if health.name == BACKEND_NAME: - self._health = HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or utc_timestamp(), - message=health.message, - ) - break - try: - bindings = list_worker_bindings(self.db_path, self.config.host_id, backend=BACKEND_NAME) - except Exception: - bindings = [] - self._bindings = {binding.private_fingerprint: binding for binding in bindings} - self._replace_ownership_maps([], bindings) - - def start(self, *, wait_for_reconcile: bool = True, timeout_seconds: float | None = None) -> None: - if self._thread is not None: - return - self.stop_event.clear() - self._ready.clear() - thread = threading.Thread(target=self.run_forever, name="tendwire-herdr-events", daemon=True) - self._thread = thread - thread.start() - if wait_for_reconcile: - timeout = ( - self.config.herdr_initial_reconcile_timeout_seconds - if timeout_seconds is None - else timeout_seconds - ) - if not self._ready.wait(max(0.001, float(timeout))): - self.stop() - raise HerdrSocketTimeoutError("initial Herdr reconciliation timed out") - - def stop(self) -> None: - self.stop_event.set() - self.flush() - thread = self._thread - if thread is not None and thread is not threading.current_thread(): - thread.join(timeout=max(1.0, self.config.herdr_timeout_seconds + 1.0)) - if thread is None or not thread.is_alive(): - self._thread = None - - def run_forever(self) -> None: - while not self.stop_event.is_set(): - reconciled = False - try: - client = self.client_factory(self.config) - try: - self.reconcile_once(client=client) - reconciled = True - if self.stop_event.is_set(): - break - if hasattr(client, "connect"): - client.connect() - stream = self._subscribe_event_stream(client) - self._ready.set() - self._read_event_stream(client, stream.subscription_id) - finally: - if hasattr(client, "close"): - client.close() - except HerdrSocketTimeoutError: - self._mark_unhealthy_safe("timeout") - except (HerdrSocketDisconnectedError, HerdrSocketConnectionError): - # A complete list reconciliation is authoritative. The event - # stream only accelerates later observations, so its closure - # must not replace that healthy snapshot with unavailable. - if not reconciled: - self._mark_unhealthy_safe("socket_disconnected") - except (HerdrMalformedLineError, HerdrEnvelopeError, HerdrProtocolError, ValueError, TypeError): - self._mark_unhealthy_safe("protocol_error") - except Exception: - self._mark_unhealthy_safe("unknown") - if self.stop_event.is_set(): - break - delay = self._reconnect_delay_seconds() - if delay: - self.stop_event.wait(delay) - - def _reconnect_delay_seconds(self) -> float: - delay = self.reconnect_delay_seconds - with self._lock: - outcome = self._health.outcome - if outcome == "protocol_error" and self.ready and self.reconcile_interval_seconds > 0: - return max(delay, min(self.reconcile_interval_seconds, 60.0)) - return delay - - def _read_event_stream(self, client: Any, subscription_id: str) -> None: - while not self.stop_event.is_set(): - self._run_periodic_reconcile_if_due() - try: - envelope = client.read_event(subscription_id, timeout=self.config.herdr_timeout_seconds) - except HerdrSocketTimeoutError: - self._run_periodic_reconcile_if_due() - continue - self.queue_event_envelope(envelope) - disconnected = False - deadline = time.monotonic() + self.debounce_seconds - while ( - self.debounce_seconds > 0 - and self._pending_event_count() < self.max_batch_size - and not self.stop_event.is_set() - ): - remaining = deadline - time.monotonic() - if remaining <= 0: - break - try: - extra = client.read_event(subscription_id, timeout=max(0.001, remaining)) - except HerdrSocketTimeoutError: - break - except HerdrSocketDisconnectedError: - disconnected = True - break - self.queue_event_envelope(extra, flush=False) - self.flush() - self._run_periodic_reconcile_if_due() - if disconnected: - raise HerdrSocketDisconnectedError("Herdr socket disconnected during event drain") - - def _schedule_next_reconcile(self) -> None: - if self.reconcile_interval_seconds <= 0: - self._next_reconcile_monotonic = None - return - self._next_reconcile_monotonic = time.monotonic() + self.reconcile_interval_seconds - - def _run_periodic_reconcile_if_due(self, client: Any | None = None) -> None: - current = time.monotonic() - due_at = self._next_reconcile_monotonic - reconcile_due = ( - self.reconcile_interval_seconds > 0 - and due_at is not None - and current >= due_at - ) - if self.reconcile_interval_seconds > 0 and due_at is None: - self._schedule_next_reconcile() - if not reconcile_due: - return - if reconcile_due: - self.reconcile_once(client=client) - - def _pending_event_count(self) -> int: - with self._lock: - return len(self._pending_events) - - def _current_bindings(self) -> list[WorkerBinding]: - return list(self._bindings.values()) - - def _records_from_reconcile_payloads(self, agent_payload: Any, pane_payload: Any) -> list[Any]: - return _records_from_agent_and_pane_payloads( - self.config, - agent_payload, - pane_payload, - ) - - def _pane_subscription_ids(self, pane_payload: Any) -> list[str]: - pane_ids: list[str] = [] - seen: set[str] = set() - for item in _payload_items(pane_payload, _PANE_PAYLOAD_KEYS): - if not _pane_has_agent(item): - continue - pane_id = _first_text(item, ("pane_id", "paneId", "id")) - if not pane_id or pane_id in seen: - continue - seen.add(pane_id) - pane_ids.append(pane_id) - if len(pane_ids) >= self.max_workers: - break - return pane_ids - - def reconcile_once(self, *, client: Any | None = None) -> Snapshot: - """Perform a full Herdr list reconcile and persist Tendwire projections.""" - owns_client = client is None - if client is None: - client = self.client_factory(self.config) - try: - if owns_client and hasattr(client, "connect"): - client.connect() - payloads = { - "workspace.list": self._call_list_method(client, "workspace_list"), - "tab.list": self._call_list_method(client, "tab_list"), - "pane.list": self._call_list_method(client, "pane_list"), - "agent.list": self._call_list_method(client, "agent_list"), - } - # tab.list is intentionally part of the authoritative reconcile barrier; - # Tendwire has no current public tab model to project into. - _ = payloads["tab.list"] - with self._lock: - stored_bindings = list_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - ) - spaces = _spaces_from_payload(payloads["workspace.list"]) - records = self._records_from_reconcile_payloads( - payloads["agent.list"], - payloads["pane.list"], - ) - subscription_pane_ids = self._pane_subscription_ids(payloads["pane.list"]) - workers, bindings = _workers_and_bindings_from_records( - self.config, - records, - stored_bindings=stored_bindings, - require_authenticated_continuity=True, - ) - if _observed_worker_count(workers) > self.max_workers: - return self._mark_worker_cap_exceeded_locked( - _observed_worker_count(workers) - ) - outcome = "healthy_non_empty" if spaces or workers else "empty_healthy" - health = herdr_backend_health(outcome, spaces=spaces, workers=workers) - previous = latest_snapshot(self.db_path, self.config.host_id) - snapshot_workers = self._workers_with_closed_missing( - previous.workers if previous is not None else [], - workers, - bound_worker_ids={binding.worker_id for binding in stored_bindings}, - ) - snapshot = project_from_observations( - self.config, - spaces=spaces, - workers=snapshot_workers, - backend_health=[health], - ) - self._save_snapshot( - snapshot, - observation=SnapshotObservationContext( - authority="complete", - observed_at=health.observed_at or snapshot.updated_at, - ), - worker_bindings=bindings, - binding_observation_authoritative=True, - binding_workers_present=bool(workers), - ) - self._last_reconcile_at = snapshot.updated_at - self._last_snapshot_at = snapshot.updated_at - self._schedule_next_reconcile() - self._spaces = {space.id: space for space in snapshot.spaces} - self._workers = {worker.id: worker for worker in snapshot.workers} - self._bindings = {binding.private_fingerprint: binding for binding in bindings} - self._replace_ownership_maps(records, bindings) - self._subscription_pane_ids = subscription_pane_ids - self._health = HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or snapshot.updated_at, - message=health.message, - ) - return snapshot - except (HerdrContinuityUnavailableError, InstallationKeyError): - snapshot = self._mark_unhealthy("continuity_unavailable") - with self._lock: - self._last_reconcile_at = snapshot.updated_at - self._schedule_next_reconcile() - return snapshot - except Exception: - self._mark_unhealthy("unknown") - raise - finally: - if owns_client and hasattr(client, "close"): - client.close() - - def _call_list_method(self, client: Any, method_name: str) -> Any: - method = getattr(client, method_name) - try: - try: - return method(timeout=self.config.herdr_timeout_seconds) - except TypeError: - return method() - finally: - # Herdr 0.7.x may close a list connection immediately after its - # response. Reconnect between authoritative read-only probes so a - # late close cannot race the next request after its write. - if hasattr(client, "close"): - client.close() - - @staticmethod - def _herdr_error_code(exc: HerdrErrorResponse) -> str: - error = exc.error - if isinstance(error, Mapping): - code = error.get("code") - if isinstance(code, str): - return code - return "" - - @staticmethod - def _herdr_error_message(exc: HerdrErrorResponse) -> str: - error = exc.error - if isinstance(error, Mapping): - message = error.get("message") - if isinstance(message, str): - return message - return "" - - def _subscribe_event_stream(self, client: Any) -> Any: - # Herdr 0.7.5 strictly validates pane-scoped status subscriptions and - # added the general pane.updated event. Use one bounded mixed - # subscription: one global entry per lifecycle/update type and one - # status entry per pane. - # pane.output_matched is intentionally absent because 0.7.5 requires a - # caller-provided match expression; pane.updated is the generic turn - # and stream refresh signal. - subscriptions = [{"type": event_name} for event_name in _GLOBAL_EVENT_NAMES] - subscriptions.extend( - {"type": "pane.agent_status_changed", "pane_id": pane_id} - for pane_id in self._subscription_pane_ids - ) - params = {"subscriptions": subscriptions} - if hasattr(client, "subscribe"): - try: - return client.subscribe( - self.subscribe_method, - params, - timeout=self.config.herdr_timeout_seconds, - event_timeout=self.config.herdr_timeout_seconds, - ) - except TypeError: - return client.subscribe(self.subscribe_method, params) - except HerdrErrorResponse as exc: - if not exc.uncorrelated: - raise - if hasattr(client, "close"): - client.close() - if hasattr(client, "connect"): - client.connect() - return self._subscribe_legacy_event_stream(client) - if hasattr(client, "events_subscribe"): - try: - return client.events_subscribe( - _HERDR_074_EVENT_NAMES, - timeout=self.config.herdr_timeout_seconds, - event_timeout=self.config.herdr_timeout_seconds, - ) - except TypeError: - return client.events_subscribe(_HERDR_074_EVENT_NAMES) - try: - return client.subscribe( - self.subscribe_method, - params, - timeout=self.config.herdr_timeout_seconds, - event_timeout=self.config.herdr_timeout_seconds, - ) - except TypeError: - return client.subscribe(self.subscribe_method, params) - - def _subscribe_legacy_event_stream(self, client: Any) -> Any: - # Preserve 0.7.4's proven compatibility request: its status event is - # parameterized by pane, while unrelated global event variants tolerate - # the same pane_id field. Replay-only events remain excluded exactly as - # before so reconnecting does not synthesize focus/detection activity. - if self._subscription_pane_ids: - subscriptions = [ - {"pane_id": pane_id, "type": event_name} - for pane_id in self._subscription_pane_ids - for event_name in _HERDR_074_PANE_SCOPED_FALLBACK_EVENT_NAMES - ] - else: - # An empty installation has no pane id with which to build the - # pane-scoped compatibility request. Fall back to the pre-0.7.5 - # global shape instead of retrying the rejected mixed shape on - # every reconnect. - subscriptions = [ - {"type": event_name} for event_name in _HERDR_074_EVENT_NAMES - ] - params = {"subscriptions": subscriptions} - try: - return client.subscribe( - self.subscribe_method, - params, - timeout=self.config.herdr_timeout_seconds, - event_timeout=self.config.herdr_timeout_seconds, - ) - except TypeError: - return client.subscribe(self.subscribe_method, params) - - def _workers_with_closed_missing( - self, - previous_workers: Sequence[Worker], - current_workers: Sequence[Worker], - *, - bound_worker_ids: set[str] | None = None, - ) -> list[Worker]: - current_by_id = {worker.id: worker for worker in current_workers} - merged = list(current_workers) - for worker in previous_workers: - if worker.id in current_by_id: - continue - if bound_worker_ids is not None and worker.id not in bound_worker_ids: - # A missing worker with no live binding is a phantom (event - # projections that never matched a binding); dropping it here - # keeps it from riding along as "closed" forever. - continue - merged.append(_closed_worker(worker)) - return merged - - def queue_event_envelope(self, envelope: Mapping[str, Any], *, flush: bool | None = None) -> bool: - event = normalize_event(envelope) - if event is None: - return False - with self._lock: - if ( - event.producer_identity is not None - and self._is_duplicate_producer_identity(event.producer_identity) - ): - return False - # Confirmed Herdr envelopes are idless. Every such event is queued; - # current-state idempotence, not historical content, prevents side effects. - self._pending_events.append(event) - self._last_event_at = utc_timestamp() - should_flush = self.debounce_seconds <= 0 if flush is None else flush - if should_flush: - self.flush() - return True - - def _is_duplicate_producer_identity(self, identity: HerdrProducerIdentity) -> bool: - if identity in self._producer_dedupe: - self._producer_dedupe.move_to_end(identity) - return True - return any(event.producer_identity == identity for event in self._pending_events) - - def _commit_producer_identities(self, events: Sequence[NormalizedHerdrEvent]) -> None: - for event in events: - identity = event.producer_identity - if identity is None: - continue - self._producer_dedupe[identity] = None - self._producer_dedupe.move_to_end(identity) - while len(self._producer_dedupe) > self.dedupe_size: - self._producer_dedupe.popitem(last=False) - - def flush(self) -> None: - # Draining, application, persistence, and producer-ID commitment share - # one lock scope so later batches cannot overtake an earlier flush. - with self._lock: - if not self._pending_events: - return - events = list(self._pending_events) - accepted_at = utc_timestamp() - self._pending_events.clear() - has_producer_identity = any(event.producer_identity is not None for event in events) - try: - self._event_continuity_revalidated = False - changed = False - for event in events: - changed = self._apply_event(event) or changed - # Producer identities become durable only after this barrier. - # It also persists dirty memory when a failed first attempt made - # a retry appear idempotent before any snapshot reached storage. - if changed or has_producer_identity: - self._persist_current_state(observed_at=accepted_at) - self._commit_producer_identities(events) - except (HerdrContinuityUnavailableError, InstallationKeyError): - self._mark_unhealthy("continuity_unavailable") - finally: - self._event_continuity_revalidated = False - - def _apply_event(self, event: NormalizedHerdrEvent) -> bool: - if event.name in _SPACE_EVENT_NAMES: - status = "closed" if event.name == "workspace.closed" else None - return self._apply_space_event(event.payload, status=status) - if event.name in _WORKTREE_EVENT_NAMES: - status = "closed" if event.name == "worktree.removed" else None - return self._apply_worktree_event(event.payload, status=status) - if event.name in _PANE_WORKER_EVENT_NAMES: - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "pane", - "agent", - "worker", - allow_top_level_pane_info=True, - ) - if not _pane_has_agent(item) and self._match_binding(item) is None: - return False - return self._upsert_worker_from_item( - item, - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name == "pane.agent_detected": - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "agent", - "worker", - "pane", - ) - return self._upsert_worker_from_item( - item, - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name == "pane.agent_status_changed": - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "agent", - "worker", - "pane", - ) - raw_status = _first_text(item, ("status", "agent_status", "state", "phase")) - return self._upsert_worker_from_item( - item, - status=normalize_status(raw_status), - update_binding=( - pane_info_observed - and _has_authoritative_identity_tuple(item) - and _has_authoritative_binding_target(item) - ), - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name == "pane.moved": - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "pane", - allow_top_level_pane_info=True, - ) - return self._apply_pane_moved( - item, - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name in _CLOSED_EVENT_NAMES: - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "pane", - allow_top_level_pane_info=True, - ) - return self._apply_pane_closed( - item, - reason=event.name.replace(".", "_"), - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name == "pane.output_matched": - return False - return False - - def _apply_space_event(self, payload: Mapping[str, Any], *, status: str | None = None) -> bool: - item = _entity_payload(payload, "workspace", "space") - direct_name_hint = _first_text(item, ("label", "name", "title")) - rename_hint = _first_text(item, ("new_name", "newName")) - if _first_text(item, ("workspace_id", "space_id", "id", "slug", "name", "label", "title")) is None: - return False - name_hint = direct_name_hint or rename_hint - if rename_hint and direct_name_hint is None: - item["name"] = rename_hint - if status is not None: - item["status"] = status - spaces = _spaces_from_payload([item] if item else []) - if not spaces: - return False - space = spaces[0] - existing = self._spaces.get(space.id) - if existing is not None and status is None: - meta = dict(existing.meta) - meta.update(space.meta) - space = Space( - id=existing.id, - name=space.name if name_hint else existing.name, - status=space.status, - meta=meta, - updated_at=space.updated_at or utc_timestamp(), - status_line=space.status_line or existing.status_line, - ) - if existing is not None and space.fingerprint == existing.fingerprint: - return False - self._spaces[space.id] = space - return True - - def _apply_worktree_event(self, payload: Mapping[str, Any], *, status: str | None = None) -> bool: - item = _entity_payload(payload, "workspace", "space", "worktree") - workspace_id = _first_text(item, ("workspace_id", "space_id")) - if workspace_id is None or workspace_id not in self._spaces: - return False - item.setdefault("id", workspace_id) - if status is not None: - item["status"] = status - return self._apply_space_event({"workspace": item}, status=status) - - def _event_worker_and_binding( - self, - item: Mapping[str, Any], - *, - status: str | None = None, - pane_info_observed: bool = False, - identity_source: str = "event", - ) -> tuple[Worker | None, WorkerBinding | None, WorkerBinding | None]: - try: - record = _worker_record_from_item( - item, - self.config, - pane_info_observed=pane_info_observed, - identity_source=identity_source, - ) - if pane_info_observed and canonical_herdr_pane_identity( - record.workspace_id, - record.pane_id, - ) is None: - matched_owner = self._match_binding(item) - existing_owner = ( - self._workers.get(matched_owner.worker_id) - if matched_owner is not None - else None - ) - if ( - existing_owner is not None - and _authenticated_local_stable_key(existing_owner) is not None - ): - raise HerdrContinuityUnavailableError( - "Herdr event PaneInfo has no canonical public pane identity" - ) - workers, bindings = _workers_and_bindings_from_records( - self.config, - [record], - stored_bindings=self._current_bindings(), - require_authenticated_continuity=True, - ) - except (HerdrContinuityUnavailableError, InstallationKeyError): - raise - except Exception: - return None, None, self._match_binding(item) - worker = workers[0] if workers else None - binding = bindings[0] if bindings else None - matched_binding = self._match_binding(item) - if matched_binding is not None and worker is not None: - worker = _worker_copy(worker, worker_id=matched_binding.worker_id) - if worker is not None and status is not None: - worker = _worker_copy(worker, status=status) - return worker, binding, matched_binding - - @staticmethod - def _add_owner( - owners: dict[str, set[str]], - value: str | None, - worker_id: str, - ) -> None: - if value: - owners.setdefault(value, set()).add(worker_id) - - def _remove_owner(self, worker_id: str) -> None: - for owners in ( - self._pane_owners, - self._terminal_owners, - self._session_owners, - ): - for value in list(owners): - owner_ids = owners[value] - owner_ids.discard(worker_id) - if not owner_ids: - owners.pop(value, None) - - def _remember_item_owner( - self, - item: Mapping[str, Any], - worker_id: str, - *, - replace: bool, - ) -> None: - if replace: - self._remove_owner(worker_id) - agent_session = _safe_mapping(_field_value(item, "agent_session")) - session_id = ( - _first_text(agent_session, ("value", "id")) - or _first_text(item, ("session_id", "sessionId")) - ) - self._add_owner( - self._pane_owners, - _first_text(item, ("pane_id", "paneId")), - worker_id, - ) - self._add_owner( - self._terminal_owners, - _first_text(item, ("terminal_id", "terminalId")), - worker_id, - ) - self._add_owner(self._session_owners, session_id, worker_id) - - def _replace_ownership_maps( - self, - records: Sequence[Any], - bindings: Sequence[WorkerBinding], - ) -> None: - self._pane_terminals = {} - self._pane_owners = {} - self._terminal_owners = {} - self._session_owners = {} - worker_ids_by_private: dict[str, set[str]] = {} - for binding in bindings: - worker_ids_by_private.setdefault( - binding.private_fingerprint, - set(), - ).add(binding.worker_id) - if binding.target_kind == "pane_id": - self._add_owner( - self._pane_owners, - binding.target_value, - binding.worker_id, - ) - if binding.target_kind == "terminal_id": - self._add_owner( - self._terminal_owners, - binding.target_value, - binding.worker_id, - ) - if binding.turn_target_value: - self._add_owner( - self._session_owners, - binding.turn_target_value, - binding.worker_id, - ) - for record in records: - if not record.pane_info_observed: - continue - owner_ids = worker_ids_by_private.get( - record.private_fingerprint, - set(), - ) - if len(owner_ids) != 1: - continue - worker_id = next(iter(owner_ids)) - observed_pane_id = record.observed_pane_id or record.pane_id - if observed_pane_id and record.terminal_id: - self._pane_terminals[observed_pane_id] = record.terminal_id - self._add_owner( - self._pane_owners, - observed_pane_id, - worker_id, - ) - self._add_owner( - self._terminal_owners, - record.terminal_id, - worker_id, - ) - self._add_owner( - self._session_owners, - record.agent_session_id, - worker_id, - ) - - def _ownership_worker_ids( - self, - item: Mapping[str, Any], - observed_binding: WorkerBinding | None = None, - ) -> set[str]: - owner_ids = { - binding.worker_id - for binding in self._matching_bindings(item) - if binding.worker_id - } - if observed_binding is not None: - owner_ids.update( - binding.worker_id - for binding in self._bindings.values() - if binding.worker_id - and binding.target_value - == observed_binding.target_value - ) - pane_id = _first_text(item, ("pane_id", "paneId")) - terminal_id = _first_text(item, ("terminal_id", "terminalId")) - agent_session = _safe_mapping(_field_value(item, "agent_session")) - session_id = ( - _first_text(agent_session, ("value", "id")) - or _first_text(item, ("session_id", "sessionId")) - ) - owner_ids.update(self._pane_owners.get(pane_id or "", ())) - owner_ids.update(self._terminal_owners.get(terminal_id or "", ())) - owner_ids.update(self._session_owners.get(session_id or "", ())) - return owner_ids - - def _target_worker_ids( - self, - target_kind: str, - target_value: str, - ) -> set[str]: - owner_ids = { - binding.worker_id - for binding in self._bindings.values() - if binding.worker_id - and ( - binding.target_value == target_value - or ( - binding.turn_target_kind == target_kind - and binding.turn_target_value == target_value - ) - ) - } - if target_kind == "pane_id": - owner_ids.update(self._pane_owners.get(target_value, ())) - if target_kind == "terminal_id": - owner_ids.update(self._terminal_owners.get(target_value, ())) - return owner_ids - - - def _matching_bindings( - self, - item: Mapping[str, Any], - *, - old_first: bool = False, - ) -> list[WorkerBinding]: - pairs = _target_pairs_from_item(item, old_first=old_first) - mapped_pairs = [ - ("terminal_id", self._pane_terminals[value]) - for kind, value in pairs - if kind == "pane_id" and value in self._pane_terminals - ] - agent_session = _safe_mapping(_field_value(item, "agent_session")) - session_id = ( - _first_text(agent_session, ("value", "id")) - or _first_text(item, ("session_id", "sessionId")) - ) - keys = set([*pairs, *mapped_pairs]) - if session_id: - keys.add(("agent_session", session_id)) - if not keys: - return [] - return [ - binding - for binding in self._bindings.values() - if _binding_target(binding) in keys - or ( - str(binding.turn_target_kind or ""), - str(binding.turn_target_value or ""), - ) - in keys - or ( - "agent_session", - str(binding.turn_target_value or ""), - ) - in keys - ] - - def _fail_closed_ownership(self, worker_ids: set[str]) -> bool: - """Remove continuity and routing from every current ambiguous owner.""" - if not worker_ids: - return False - if ( - self._health.outcome == "continuity_unavailable" - and not self._event_continuity_revalidated - ): - return False - changed = False - binding_updates: list[WorkerBinding] = [] - bindings_by_worker: dict[str, list[WorkerBinding]] = {} - for private_fingerprint, binding in list(self._bindings.items()): - if binding.worker_id not in worker_ids: - continue - bindings_by_worker.setdefault(binding.worker_id, []).append(binding) - ambiguous = WorkerBinding( - host_id=binding.host_id, - worker_id=binding.worker_id, - worker_fingerprint=binding.worker_fingerprint, - backend=binding.backend, - target_kind=binding.target_kind, - target_value=binding.target_value, - turn_target_kind=None, - turn_target_value=None, - sendable=False, - reason="ambiguous_pane_match", - observed_at=utc_timestamp(), - expires_at=binding.expires_at, - private_fingerprint=binding.private_fingerprint, - ) - if _binding_state_equal(binding, ambiguous): - continue - self._bindings[private_fingerprint] = ambiguous - binding_updates.append(ambiguous) - changed = True - - for worker_id in worker_ids: - worker = self._workers.get(worker_id) - if worker is None: - continue - target = worker.backend_target - if not isinstance(target, Mapping): - candidates = bindings_by_worker.get(worker_id, []) - target = candidates[0].backend_target() if candidates else None - ambiguous_target = None - if isinstance(target, Mapping): - kind = str(target.get("kind") or "") - value = str(target.get("value") or "") - if kind and value: - ambiguous_target = { - "kind": kind, - "value": value, - "sendable": False, - "reason": "ambiguous_pane_match", - } - ambiguous_worker = _worker_copy( - worker, - meta=_strip_stable_key_fields(worker.meta), - backend_target=ambiguous_target, - ) - if _worker_state_equal(worker, ambiguous_worker): - continue - self._workers[worker_id] = ambiguous_worker - changed = True - if binding_updates: - upsert_worker_bindings(self.db_path, binding_updates) - return changed - - - def _upsert_worker_from_item( - self, - item: Mapping[str, Any], - *, - status: str | None = None, - update_binding: bool = True, - pane_info_observed: bool = False, - identity_source: str = "event", - ) -> bool: - if not item: - return False - if not _has_public_worker_identity(item) and self._match_binding(item) is None: - return False - worker, binding, matched_binding = self._event_worker_and_binding( - item, - status=status, - pane_info_observed=pane_info_observed, - identity_source=identity_source, - ) - if worker is None: - return False - - authoritative_identity = ( - pane_info_observed and _has_authoritative_identity_tuple(item) - ) - stable_owner_reused = False - if authoritative_identity: - observed_stable_key = _authenticated_local_stable_key(worker) - stable_owner_ids = ( - { - current.id - for current in self._workers.values() - if _authenticated_local_stable_key(current) - == observed_stable_key - } - if observed_stable_key is not None - else set() - ) - target_owner_ids = self._ownership_worker_ids( - item, - binding, - ) - conflicting_owner_ids: set[str] = set() - if len(stable_owner_ids) > 1 or len(target_owner_ids) > 1: - conflicting_owner_ids.update(stable_owner_ids) - conflicting_owner_ids.update(target_owner_ids) - elif len(stable_owner_ids) == 1: - stable_owner_id = next(iter(stable_owner_ids)) - other_target_owners = target_owner_ids - {stable_owner_id} - if other_target_owners: - conflicting_owner_ids.add(stable_owner_id) - conflicting_owner_ids.update(other_target_owners) - else: - worker = _worker_copy( - worker, - worker_id=stable_owner_id, - ) - stable_owner_reused = True - elif len(target_owner_ids) == 1: - target_owner_id = next(iter(target_owner_ids)) - target_owner = self._workers.get(target_owner_id) - target_stable_key = ( - _authenticated_local_stable_key(target_owner) - if target_owner is not None - else None - ) - observed_pane_id = _first_text( - item, - ("pane_id", "paneId"), - ) - observed_canonical_identity = canonical_herdr_pane_identity( - _first_text(item, ("workspace_id", "workspaceId")), - observed_pane_id, - ) - same_pane_owner_ids = self._pane_owners.get( - observed_pane_id or "", - set(), - ) - target_is_ambiguous = any( - current_binding.worker_id == target_owner_id - and current_binding.reason == "ambiguous_pane_match" - for current_binding in self._bindings.values() - ) - if target_is_ambiguous or ( - observed_stable_key is None - and ( - observed_canonical_identity is None - or target_owner_id not in same_pane_owner_ids - ) - ) or ( - observed_stable_key is not None - and target_stable_key is not None - and target_stable_key != observed_stable_key - ): - conflicting_owner_ids.add(target_owner_id) - else: - worker = _worker_copy( - worker, - worker_id=target_owner_id, - ) - stable_owner_reused = True - if conflicting_owner_ids: - return self._fail_closed_ownership( - conflicting_owner_ids - ) - else: - compatibility_owner_ids = self._ownership_worker_ids( - item, - binding, - ) - compatibility_identity = canonical_herdr_pane_identity( - _first_text(item, ("workspace_id", "workspaceId")), - _first_text(item, ("pane_id", "paneId")), - ) - pane_owner_ids = ( - self._pane_owners.get(compatibility_identity[1], set()) - if compatibility_identity is not None - else set() - ) - if ( - matched_binding is None - and len(compatibility_owner_ids) == 1 - and compatibility_owner_ids <= pane_owner_ids - ): - compatibility_owner_id = next(iter(compatibility_owner_ids)) - owner_bindings = [ - current_binding - for current_binding in self._bindings.values() - if current_binding.worker_id == compatibility_owner_id - ] - if len(owner_bindings) == 1: - # Scalar events can locate an already-authenticated owner - # through the private pane map, but cannot replace that - # owner's identity or binding with event-only fields. - matched_binding = owner_bindings[0] - matched_owner_ids = ( - {matched_binding.worker_id} - if matched_binding is not None - else set() - ) - if compatibility_owner_ids - matched_owner_ids: - return self._fail_closed_ownership( - compatibility_owner_ids - ) - - existing = self._workers.get(worker.id) - if existing is None and matched_binding is not None: - existing = self._workers.get(matched_binding.worker_id) - if matched_binding is not None and not stable_owner_reused: - worker = _worker_copy(worker, worker_id=matched_binding.worker_id) - worker = _merge_worker_update( - existing, - worker, - status=status, - preserve_existing_continuity=not authoritative_identity, - ) - if not authoritative_identity and matched_binding is not None: - worker = _worker_copy( - worker, - backend_target=matched_binding.backend_target(), - ) - if self._would_exceed_worker_cap(worker, existing=existing): - self._mark_worker_cap_exceeded_locked(_observed_worker_count(list(self._workers.values())) + 1) - return False - changed = existing is None or not _worker_state_equal(existing, worker) - if changed: - self._workers[worker.id] = worker - else: - assert existing is not None - worker = existing - if update_binding and binding is not None: - if stable_owner_reused: - stale_private_fingerprints = [ - private_fingerprint - for private_fingerprint, current_binding in self._bindings.items() - if current_binding.worker_id == worker.id - and private_fingerprint != binding.private_fingerprint - ] - if stale_private_fingerprints: - expire_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - private_fingerprints=stale_private_fingerprints, - reason="identity_replaced", - ) - for private_fingerprint in stale_private_fingerprints: - self._bindings.pop(private_fingerprint, None) - changed = True - binding = self._binding_with_worker(binding, worker) - elif matched_binding is not None and ( - not authoritative_identity - or binding.private_fingerprint - != matched_binding.private_fingerprint - ): - binding = self._binding_with_worker(matched_binding, worker) - else: - binding = self._binding_with_worker(binding, worker) - current_binding = self._bindings.get(binding.private_fingerprint) - if current_binding is None or not _binding_state_equal(current_binding, binding): - self._bindings[binding.private_fingerprint] = binding - upsert_worker_bindings(self.db_path, [binding]) - changed = True - if authoritative_identity: - self._remember_item_owner( - item, - worker.id, - replace=update_binding, - ) - self._note_pane_terminal(item) - if _authenticated_local_stable_key(worker) is not None: - if self._health.outcome == "continuity_unavailable": - changed = True - self._event_continuity_revalidated = True - return changed - - def _binding_with_worker(self, binding: WorkerBinding, worker: Worker) -> WorkerBinding: - return WorkerBinding( - host_id=binding.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend=binding.backend, - target_kind=binding.target_kind, - target_value=binding.target_value, - turn_target_kind=binding.turn_target_kind, - turn_target_value=binding.turn_target_value, - sendable=binding.sendable, - reason=binding.reason, - observed_at=utc_timestamp(), - expires_at=None, - private_fingerprint=binding.private_fingerprint, - ) - - def _match_binding(self, item: Mapping[str, Any], *, old_first: bool = False) -> WorkerBinding | None: - pairs = _target_pairs_from_item(item, old_first=old_first) - if not pairs: - return None - binding_by_target = {_binding_target(binding): binding for binding in self._bindings.values()} - # Event payloads often carry only a pane id while stored bindings target - # a terminal id; fall back to the turn target so a known pane never - # spawns a duplicate re-lettered worker. - for binding in self._bindings.values(): - turn_key = (str(binding.turn_target_kind or ""), str(binding.turn_target_value or "")) - if turn_key[0] and turn_key[1]: - binding_by_target.setdefault(turn_key, binding) - # Translate pane ids to the terminal ids remembered from the last - # reconcile: agent kinds whose turn target is not a pane id (codex - # session ids) would otherwise never match a pane-id-only event. - mapped_pairs = [ - ("terminal_id", self._pane_terminals[value]) - for kind, value in pairs - if kind == "pane_id" and value in self._pane_terminals - ] - for pair in [*pairs, *mapped_pairs]: - binding = binding_by_target.get(pair) - if binding is not None: - return binding - return None - - def _previous_ownership_worker_ids( - self, - item: Mapping[str, Any], - ) -> set[str]: - previous_pane_id = _first_text( - item, - ( - "old_pane_id", - "previous_pane_id", - "from_pane_id", - "source_pane_id", - ), - ) - previous_terminal_id = _first_text( - item, - ( - "old_terminal_id", - "previous_terminal_id", - "from_terminal_id", - "source_terminal_id", - ), - ) - owner_ids = set(self._pane_owners.get(previous_pane_id or "", ())) - owner_ids.update( - self._terminal_owners.get(previous_terminal_id or "", ()) - ) - if previous_pane_id and previous_pane_id in self._pane_terminals: - previous_terminal_id = self._pane_terminals[previous_pane_id] - owner_ids.update( - self._terminal_owners.get(previous_terminal_id, ()) - ) - previous_keys = { - ("pane_id", previous_pane_id or ""), - ("terminal_id", previous_terminal_id or ""), - } - owner_ids.update( - binding.worker_id - for binding in self._bindings.values() - if binding.worker_id - and ( - _binding_target(binding) in previous_keys - or ( - str(binding.turn_target_kind or ""), - str(binding.turn_target_value or ""), - ) - in previous_keys - ) - ) - return owner_ids - - - def _note_pane_terminal(self, item: Mapping[str, Any]) -> None: - pane_id = _first_text(item, ("pane_id", "paneId")) - terminal_id = _first_text(item, ("terminal_id", "terminalId")) - if pane_id and terminal_id: - self._pane_terminals[pane_id] = terminal_id - - def _apply_pane_moved( - self, - item: Mapping[str, Any], - *, - pane_info_observed: bool = False, - identity_source: str = "event:pane.moved", - ) -> bool: - authoritative_identity = ( - pane_info_observed and _has_authoritative_identity_tuple(item) - ) - observed_worker, observed_binding, _matched = self._event_worker_and_binding( - item, - pane_info_observed=pane_info_observed, - identity_source=identity_source, - ) - source_owner_ids = self._previous_ownership_worker_ids(item) - if len(source_owner_ids) > 1: - return self._fail_closed_ownership(source_owner_ids) - if not source_owner_ids: - return False - source_owner_id = next(iter(source_owner_ids)) - existing = self._workers.get(source_owner_id) - if existing is None: - return False - source_bindings = [ - binding - for binding in self._bindings.values() - if binding.worker_id == source_owner_id - ] - new_target = _new_move_target(item) - if new_target is not None: - destination_owner_ids = self._target_worker_ids(*new_target) - conflicting_destination_ids = destination_owner_ids - { - source_owner_id - } - if conflicting_destination_ids: - return self._fail_closed_ownership( - {source_owner_id, *conflicting_destination_ids} - ) - - if authoritative_identity: - if observed_worker is None: - return False - observed_stable_key = _authenticated_local_stable_key( - observed_worker - ) - destination_owner_ids = self._ownership_worker_ids( - item, - observed_binding, - ) - if observed_stable_key is not None: - destination_owner_ids.update( - current.id - for current in self._workers.values() - if _authenticated_local_stable_key(current) - == observed_stable_key - ) - conflicting_owner_ids = destination_owner_ids - { - source_owner_id - } - if conflicting_owner_ids: - return self._fail_closed_ownership( - {source_owner_id, *conflicting_owner_ids} - ) - worker = _merge_worker_update( - existing, - _worker_copy( - observed_worker, - worker_id=source_owner_id, - ), - preserve_existing_continuity=False, - ) - else: - worker = existing - if observed_worker is not None: - worker = _merge_worker_update( - existing, - _worker_copy( - observed_worker, - worker_id=source_owner_id, - ), - preserve_existing_continuity=True, - ) - - if self._would_exceed_worker_cap(worker, existing=existing): - self._mark_worker_cap_exceeded_locked( - _observed_worker_count(list(self._workers.values())) + 1 - ) - return False - - if authoritative_identity and observed_binding is not None: - if len(source_bindings) != 1: - return self._fail_closed_ownership({source_owner_id}) - old_binding = source_bindings[0] - moved_binding = WorkerBinding( - host_id=observed_binding.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend=observed_binding.backend, - target_kind=observed_binding.target_kind, - target_value=observed_binding.target_value, - turn_target_kind=observed_binding.turn_target_kind, - turn_target_value=observed_binding.turn_target_value, - sendable=observed_binding.sendable, - reason=observed_binding.reason, - observed_at=utc_timestamp(), - expires_at=None, - private_fingerprint=old_binding.private_fingerprint, - ) - else: - if new_target is None or len(source_bindings) != 1: - return False - old_binding = source_bindings[0] - target_kind, target_value = new_target - moved_binding = WorkerBinding( - host_id=old_binding.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend=old_binding.backend, - target_kind=target_kind, - target_value=target_value, - turn_target_kind=old_binding.turn_target_kind, - turn_target_value=( - target_value - if old_binding.turn_target_kind == "pane_id" - and target_kind == "pane_id" - else old_binding.turn_target_value - ), - sendable=old_binding.sendable, - reason=old_binding.reason, - observed_at=utc_timestamp(), - expires_at=None, - private_fingerprint=old_binding.private_fingerprint, - ) - - stale_private_fingerprints = [ - binding.private_fingerprint - for binding in source_bindings - if binding.private_fingerprint - != moved_binding.private_fingerprint - ] - if stale_private_fingerprints: - expire_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - private_fingerprints=stale_private_fingerprints, - reason="identity_replaced", - ) - for private_fingerprint in stale_private_fingerprints: - self._bindings.pop(private_fingerprint, None) - - worker = _worker_copy( - worker, - backend_target=moved_binding.backend_target(), - ) - moved_binding = self._binding_with_worker(moved_binding, worker) - self._workers[worker.id] = worker - self._bindings[moved_binding.private_fingerprint] = moved_binding - upsert_worker_bindings(self.db_path, [moved_binding]) - - previous_pane_ids: set[str] = set() - previous_pane_id = _first_text( - item, - ( - "old_pane_id", - "previous_pane_id", - "from_pane_id", - "source_pane_id", - ), - ) - if previous_pane_id: - previous_pane_ids.add(previous_pane_id) - previous_terminal_id = _first_text( - item, - ( - "old_terminal_id", - "previous_terminal_id", - "from_terminal_id", - "source_terminal_id", - ), - ) - if previous_terminal_id: - previous_pane_ids.update( - pane_id - for pane_id, terminal_id in self._pane_terminals.items() - if terminal_id == previous_terminal_id - and source_owner_id in self._pane_owners.get(pane_id, ()) - ) - current_pane_id = _first_text(item, ("pane_id", "paneId")) - for source_pane_id in previous_pane_ids: - if source_pane_id != current_pane_id: - self._pane_terminals.pop(source_pane_id, None) - - self._remove_owner(worker.id) - if authoritative_identity: - self._remember_item_owner( - item, - worker.id, - replace=False, - ) - self._note_pane_terminal(item) - if _authenticated_local_stable_key(worker) is not None: - self._event_continuity_revalidated = True - else: - if moved_binding.target_kind == "pane_id": - self._add_owner( - self._pane_owners, - moved_binding.target_value, - worker.id, - ) - if moved_binding.target_kind == "terminal_id": - self._add_owner( - self._terminal_owners, - moved_binding.target_value, - worker.id, - ) - if moved_binding.turn_target_value: - self._add_owner( - self._session_owners, - moved_binding.turn_target_value, - worker.id, - ) - return True - - def _apply_pane_closed( - self, - item: Mapping[str, Any], - *, - reason: str, - pane_info_observed: bool = False, - identity_source: str = "event", - ) -> bool: - observed_worker: Worker | None = None - matched_binding: WorkerBinding | None = None - observed_binding: WorkerBinding | None = None - if pane_info_observed: - observed_worker, observed_binding, matched_binding = self._event_worker_and_binding( - item, - status="closed", - pane_info_observed=True, - identity_source=identity_source, - ) - observed_stable_key = ( - _authenticated_local_stable_key(observed_worker) - if observed_worker is not None - else None - ) - if observed_stable_key is not None: - stable_owner_ids = { - current.id - for current in self._workers.values() - if _authenticated_local_stable_key(current) == observed_stable_key - } - target_owner_ids = self._ownership_worker_ids(item, observed_binding) - if len(stable_owner_ids | target_owner_ids) > 1: - return False - binding = matched_binding or self._match_binding(item) - worker: Worker | None = None - if binding is not None: - worker = self._workers.get(binding.worker_id) - if worker is None: - if binding is None and not _has_public_worker_identity(item): - return False - if observed_worker is None: - observed_worker, _event_binding, matched_binding = self._event_worker_and_binding( - item, - status="closed", - pane_info_observed=False, - identity_source=identity_source, - ) - if matched_binding is not None: - binding = matched_binding - worker = observed_worker - if worker is None: - return False - if binding is None and self._would_exceed_worker_cap(worker, existing=self._workers.get(worker.id)): - self._mark_worker_cap_exceeded_locked(_observed_worker_count(list(self._workers.values())) + 1) - return False - closed = _closed_worker(worker) - current = self._workers.get(closed.id) - changed = current is None or not _worker_state_equal(current, closed) - if changed: - self._workers[closed.id] = closed - if binding is not None: - expire_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - private_fingerprints=[binding.private_fingerprint], - now=utc_timestamp(), - reason=reason, - ) - self._bindings.pop(binding.private_fingerprint, None) - changed = True - elif changed: - expire_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - worker_id=closed.id, - now=utc_timestamp(), - reason=reason, - ) - if ( - pane_info_observed - and observed_worker is not None - and _authenticated_local_stable_key(observed_worker) is not None - ): - if self._health.outcome == "continuity_unavailable": - changed = True - self._event_continuity_revalidated = True - return changed - - def _persist_current_state(self, *, observed_at: str | None = None) -> Snapshot: - accepted_at = observed_at or utc_timestamp() - spaces = list(self._spaces.values()) - workers = list(self._workers.values()) - if ( - self._health.outcome == "continuity_unavailable" - and not self._event_continuity_revalidated - ): - health = self._health.to_backend_health(spaces=spaces, workers=workers) - else: - outcome = ( - "healthy_non_empty" - if spaces or _observed_worker_count(workers) - else "empty_healthy" - ) - health = herdr_backend_health( - outcome, - observed_at=accepted_at, - spaces=spaces, - workers=workers, - ) - snapshot = project_from_observations( - self.config, - spaces=spaces, - workers=workers, - backend_health=[health], - ) - self._save_snapshot( - snapshot, - observation=SnapshotObservationContext( - authority="positive" if health.status == "healthy" else "none", - observed_at=health.observed_at or accepted_at, - ), - ) - self._last_snapshot_at = snapshot.updated_at - self._health = HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or snapshot.updated_at, - message=health.message, - ) - return snapshot - - def _would_exceed_worker_cap(self, worker: Worker, *, existing: Worker | None = None) -> bool: - if worker.status == "closed": - return False - previous = existing if existing is not None else self._workers.get(worker.id) - if previous is not None and previous.status != "closed": - return False - return _observed_worker_count(list(self._workers.values())) + 1 > self.max_workers - - def _mark_worker_cap_exceeded_locked(self, observed_workers: int) -> Snapshot: - now = utc_timestamp() - previous = latest_snapshot(self.db_path, self.config.host_id) - spaces = list(previous.spaces) if previous is not None else list(self._spaces.values()) - workers = list(previous.workers) if previous is not None else list(self._workers.values()) - if ( - self._health.outcome == "continuity_unavailable" - and not self._event_continuity_revalidated - ): - health = self._health.to_backend_health(spaces=spaces, workers=workers) - else: - health = herdr_backend_health( - "worker_cap_exceeded", - observed_at=now, - message="Herdr observation exceeded the configured worker cap", - spaces=spaces, - workers=workers, - ) - snapshot = project_from_observations( - self.config, - spaces=spaces, - workers=workers, - backend_health=[health], - ) - self._save_snapshot( - snapshot, - observation=SnapshotObservationContext( - authority="none", - observed_at=health.observed_at or now, - ), - ) - self._spaces = {space.id: space for space in snapshot.spaces} - self._workers = {worker.id: worker for worker in snapshot.workers} - self._health = HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or now, - message=health.message, - ) - self._last_cap_status_at = now - self._last_reconcile_at = now - self._last_snapshot_at = snapshot.updated_at - self._schedule_next_reconcile() - return snapshot - - def _mark_unhealthy(self, outcome: str) -> Snapshot: - with self._lock: - if ( - self._health.outcome == "continuity_unavailable" - and not self._event_continuity_revalidated - ): - health_state = self._health - else: - health_state = self._health_for(outcome) - self._health = health_state - self._event_continuity_revalidated = False - spaces = list(self._spaces.values()) - workers = list(self._workers.values()) - if not spaces and not workers: - snapshot = latest_snapshot(self.db_path, self.config.host_id) - if snapshot is not None: - spaces = list(snapshot.spaces) - workers = list(snapshot.workers) - health = health_state.to_backend_health(spaces=spaces, workers=workers) - snapshot = project_from_observations( - self.config, - spaces=spaces, - workers=workers, - backend_health=[health], - ) - self._save_snapshot( - snapshot, - observation=SnapshotObservationContext( - authority="none", - observed_at=health.observed_at or snapshot.updated_at, - ), - ) - self._last_snapshot_at = snapshot.updated_at - self._spaces = {space.id: space for space in snapshot.spaces} - self._workers = {worker.id: worker for worker in snapshot.workers} - return snapshot - - def _mark_unhealthy_safe(self, outcome: str) -> Snapshot | None: - try: - return self._mark_unhealthy(outcome) - except Exception: - # Health persistence is secondary to keeping the long-running - # observation loop alive. In particular, another store operation - # can briefly hold the secure SQLite parent lock and make this - # best-effort write fail closed. The next loop iteration performs - # a complete reconciliation, so retain the in-memory unhealthy - # state and let that authoritative retry recover the backend. - return None - finally: - self._ready.set() diff --git a/src/tendwire/backends/herdr_protocol.py b/src/tendwire/backends/herdr_protocol.py index 2cbf4a4..ff68069 100644 --- a/src/tendwire/backends/herdr_protocol.py +++ b/src/tendwire/backends/herdr_protocol.py @@ -9,7 +9,7 @@ import json import os import uuid -from collections.abc import Iterable, Mapping +from collections.abc import Mapping from pathlib import Path from typing import Any @@ -22,29 +22,6 @@ "HERDR_SESSION", ) -HERDR_EVENTS_SUBSCRIBE_METHOD = "events.subscribe" -HERDR_OFFICIAL_EVENT_NAMES = ( - "workspace.created", - "workspace.updated", - "workspace.renamed", - "workspace.closed", - "workspace.focused", - "pane.created", - "pane.closed", - "pane.updated", - "pane.focused", - "pane.moved", - "pane.exited", - "pane.agent_detected", - "pane.output_matched", - "pane.agent_status_changed", - "worktree.created", - "worktree.opened", - "worktree.removed", -) -HERDR_OFFICIAL_EVENT_NAME_SET = frozenset(HERDR_OFFICIAL_EVENT_NAMES) - - class HerdrProtocolError(Exception): """Base error for Herdr socket protocol failures.""" @@ -61,6 +38,10 @@ class HerdrEnvelopeError(HerdrProtocolError, ValueError): """Raised when a decoded JSON object is not a valid protocol envelope.""" +class HerdrFrameTooLargeError(HerdrProtocolError, ValueError): + """Raised when one JSON-line frame exceeds the fixed transport bound.""" + + class HerdrRequestIdMismatchError(HerdrEnvelopeError): """Raised when a server envelope is not correlated to the expected id.""" @@ -72,12 +53,9 @@ def __init__( self, error: Any, request_id: str, - *, - uncorrelated: bool = False, ) -> None: self.error = error self.request_id = request_id - self.uncorrelated = uncorrelated message = "Herdr returned an error response" if isinstance(error, Mapping): raw_message = error.get("message") @@ -169,43 +147,6 @@ def build_request( return {"id": request_id, "method": method, "params": dict(params)} -def _validate_event_subscription_name(name: Any) -> str: - if not isinstance(name, str): - raise HerdrEnvelopeError("Herdr event subscription names must be strings") - if not name: - raise HerdrEnvelopeError("Herdr event subscription names must not be empty") - if name.strip() != name or name not in HERDR_OFFICIAL_EVENT_NAME_SET: - raise HerdrEnvelopeError(f"unsupported Herdr event subscription {name!r}") - return name - - -def build_events_subscribe_params(event_names: Iterable[str] | str | None = None) -> dict[str, Any]: - """Return official events.subscribe params for validated Herdr event names.""" - if event_names is None: - names = HERDR_OFFICIAL_EVENT_NAMES - elif isinstance(event_names, str): - names = (event_names,) - else: - try: - names = tuple(event_names) - except TypeError as exc: - raise HerdrEnvelopeError("Herdr event subscriptions must be iterable") from exc - return {"subscriptions": [{"type": _validate_event_subscription_name(name)} for name in names]} - - -def build_events_subscribe_request( - event_names: Iterable[str] | str | None = None, - *, - request_id: str | None = None, -) -> dict[str, Any]: - """Build an official Herdr event subscription request envelope.""" - return build_request( - HERDR_EVENTS_SUBSCRIBE_METHOD, - build_events_subscribe_params(event_names), - request_id=request_id, - ) - - def frame_request(request: Mapping[str, Any]) -> bytes: """Encode one request object as UTF-8 JSON Lines.""" try: @@ -273,71 +214,20 @@ def is_response(envelope: Mapping[str, Any]) -> bool: return is_result_response(envelope) or is_error_response(envelope) -def is_event(envelope: Mapping[str, Any]) -> bool: - return "event" in envelope and "result" not in envelope and "error" not in envelope - - -def validate_response( - envelope: Mapping[str, Any], - *, - allow_uncorrelated_error: bool = False, -) -> dict[str, Any]: +def validate_response(envelope: Mapping[str, Any]) -> dict[str, Any]: """Validate a response envelope while tolerating unknown fields.""" if not is_response(envelope): raise HerdrEnvelopeError("Herdr response must contain exactly one of result or error") - # Herdr 0.7.5 emits ``{"id":"", "error":...}`` when subscription - # parameters fail schema validation. Only the subscription negotiation - # path may opt into that compatibility exception; ordinary requests remain - # strictly correlated. - error = envelope.get("error") - uncorrelated_subscription_error = ( - allow_uncorrelated_error - and is_error_response(envelope) - and envelope.get("id") == "" - and isinstance(error, Mapping) - and error.get("code") == "invalid_request" - and isinstance(error.get("message"), str) - and error["message"].startswith("invalid request:") - ) - if not uncorrelated_subscription_error: - _validated_id(envelope) - return dict(envelope) - - -def validate_event(envelope: Mapping[str, Any]) -> dict[str, Any]: - """Validate an event envelope while preserving its raw data. - - The confirmed Herdr ``EventEnvelope`` consists of ``event`` and ``data``. - A generic ``id`` is tolerated as subscription correlation only; it is not - authoritative producer event identity. Unknown top-level fields remain - available for forward-compatible consumers. - """ - request_id = envelope.get("id") - if request_id is not None and (not isinstance(request_id, str) or not request_id): - raise HerdrEnvelopeError("Herdr event id must be a non-empty string when present") - event_name = envelope.get("event") - if not isinstance(event_name, str) or not event_name: - raise HerdrEnvelopeError("Herdr event name must be a non-empty string") - if not is_event(envelope): - raise HerdrEnvelopeError("Herdr event must not contain result or error fields") + _validated_id(envelope) return dict(envelope) -def validate_server_envelope( - envelope: Mapping[str, Any], - *, - allow_uncorrelated_error: bool = False, -) -> dict[str, Any]: - """Validate a decoded server response or event envelope.""" +def validate_server_envelope(envelope: Mapping[str, Any]) -> dict[str, Any]: + """Validate a decoded server response envelope.""" if is_response(envelope): - return validate_response( - envelope, - allow_uncorrelated_error=allow_uncorrelated_error, - ) - if is_event(envelope): - return validate_event(envelope) + return validate_response(envelope) _validated_id(envelope) - raise HerdrEnvelopeError("Herdr envelope is neither a response nor an event") + raise HerdrEnvelopeError("Herdr envelope is not a response") def ensure_response_id(envelope: Mapping[str, Any], expected_id: str) -> None: diff --git a/src/tendwire/backends/herdr_socket.py b/src/tendwire/backends/herdr_socket.py index d76b143..6a2710b 100644 --- a/src/tendwire/backends/herdr_socket.py +++ b/src/tendwire/backends/herdr_socket.py @@ -1,31 +1,22 @@ -"""Inactive stdlib Herdr Unix socket client. - -This module is additive and is not imported by Tendwire's production -observation or CLI paths. It exposes a low-level JSON-line client plus thin -wrappers for the PR8-allowed Herdr methods only. -""" +"""Synchronous client for Herdr lifecycle discovery and ACP ownership.""" from __future__ import annotations import socket import time -from collections import deque -from collections.abc import Iterable, Iterator, Mapping +from collections.abc import Iterable, Mapping from typing import Any from .herdr_protocol import ( HerdrEnvelopeError, HerdrErrorResponse, + HerdrFrameTooLargeError, HerdrProtocolError, - HerdrRequestIdMismatchError, - HERDR_EVENTS_SUBSCRIBE_METHOD, - build_events_subscribe_params, build_request, ensure_response_id, error_payload, frame_request, is_error_response, - is_event, is_result_response, parse_json_line, resolve_socket_path, @@ -35,7 +26,7 @@ _DEFAULT_TIMEOUT_SECONDS = 5.0 _RECV_SIZE = 4096 -_MAX_PENDING_EVENTS = 1024 +_MAX_FRAME_BYTES = 8 * 1024 * 1024 class HerdrSocketError(HerdrProtocolError): @@ -54,39 +45,6 @@ class HerdrSocketConnectionError(HerdrSocketError, ConnectionError): """Raised when the Unix socket cannot be opened.""" -class HerdrEventStream(Iterator[dict[str, Any]]): - """Iterator over events correlated to a subscription request id.""" - - def __init__( - self, - client: "HerdrSocketClient", - subscription_id: str, - ack: Any, - *, - timeout: float | None = None, - ) -> None: - self.client = client - self.subscription_id = subscription_id - self.ack = ack - self.timeout = timeout - self._closed = False - - def __iter__(self) -> "HerdrEventStream": - return self - - def __next__(self) -> dict[str, Any]: - if self._closed: - raise StopIteration - try: - return self.client.read_event(self.subscription_id, timeout=self.timeout) - except HerdrSocketDisconnectedError: - self._closed = True - raise StopIteration from None - - def close(self) -> None: - self._closed = True - - class HerdrSocketClient: """Synchronous Herdr JSON-line client over a Unix domain socket.""" @@ -100,7 +58,6 @@ def __init__( self.timeout = self._validate_timeout(timeout) self._socket: socket.socket | None = None self._buffer = bytearray() - self._pending_events: deque[dict[str, Any]] = deque() def __enter__(self) -> "HerdrSocketClient": self.connect() @@ -137,7 +94,6 @@ def close(self) -> None: sock = self._socket self._socket = None self._buffer.clear() - self._pending_events.clear() if sock is None: return try: @@ -155,66 +111,11 @@ def request( ) -> Any: """Send one strictly correlated request and return its raw result payload.""" request_id, deadline = self._send_request(method, params, timeout=timeout) - response = self._read_response( - request_id, - deadline=deadline, - allow_uncorrelated_error=False, - ) + response = self._read_response(request_id, deadline=deadline) if is_error_response(response): raise HerdrErrorResponse(error_payload(response), request_id) return result_payload(response) - def subscribe( - self, - method: str, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> HerdrEventStream: - """Send a subscription request and return an iterator over its events.""" - request_id, deadline = self._send_request(method, params, timeout=timeout) - response = self._read_response( - request_id, - deadline=deadline, - allow_uncorrelated_error=True, - ) - if is_error_response(response): - raise HerdrErrorResponse(error_payload(response), request_id) - return HerdrEventStream( - self, - request_id, - result_payload(response), - timeout=self.timeout if event_timeout is None else event_timeout, - ) - - def events_subscribe( - self, - event_names: Iterable[str] | str | None = None, - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> HerdrEventStream: - """Subscribe to the official Herdr event stream.""" - return self.subscribe( - HERDR_EVENTS_SUBSCRIBE_METHOD, - build_events_subscribe_params(event_names), - timeout=timeout, - event_timeout=event_timeout, - ) - - def read_event(self, subscription_id: str, *, timeout: float | None = None) -> dict[str, Any]: - envelope = ( - self._pending_events.popleft() - if self._pending_events - else self._read_server_envelope(deadline=self._deadline(timeout)) - ) - if not is_event(envelope): - raise HerdrEnvelopeError("expected Herdr event envelope") - if envelope.get("id") is not None: - ensure_response_id(envelope, subscription_id) - return envelope - def workspace_list( self, params: Mapping[str, Any] | None = None, @@ -223,14 +124,6 @@ def workspace_list( ) -> Any: return self.request("workspace.list", params, timeout=timeout) - def tab_list( - self, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - ) -> Any: - return self.request("tab.list", params, timeout=timeout) - def pane_list( self, params: Mapping[str, Any] | None = None, @@ -247,38 +140,6 @@ def agent_list( ) -> Any: return self.request("agent.list", params, timeout=timeout) - def pane_get( - self, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - ) -> Any: - return self.request("pane.get", params, timeout=timeout) - - def agent_get( - self, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - ) -> Any: - return self.request("agent.get", params, timeout=timeout) - - def pane_read( - self, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - ) -> Any: - return self.request("pane.read", params, timeout=timeout) - - def agent_send( - self, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - ) -> Any: - return self.request("agent.send", params, timeout=timeout) - def agent_acp_endpoint( self, target: str, @@ -341,7 +202,10 @@ def _send_request( request = build_request(method, params) request_id = str(request["id"]) deadline = self._deadline(timeout) - self._write(frame_request(request), deadline=deadline) + frame = frame_request(request) + if len(frame) > _MAX_FRAME_BYTES: + raise HerdrFrameTooLargeError("Herdr request frame is too large") + self._write(frame, deadline=deadline) return request_id, deadline def _deadline(self, timeout: float | None) -> float: @@ -388,60 +252,25 @@ def _read_response( request_id: str, *, deadline: float, - allow_uncorrelated_error: bool, ) -> dict[str, Any]: - while True: - envelope = self._read_server_envelope( - deadline=deadline, - allow_uncorrelated_error=allow_uncorrelated_error, - ) - if is_event(envelope): - if len(self._pending_events) >= _MAX_PENDING_EVENTS: - raise HerdrEnvelopeError( - "too many Herdr events arrived before the response" - ) - self._pending_events.append(envelope) - continue - if ( - allow_uncorrelated_error - and is_error_response(envelope) - and ( - envelope.get("id") == "" - ) - ): - # These narrowly validated Herdr 0.7.5 errors belong to the - # only in-flight synchronous request and must reach the caller - # as server errors rather than poison the connection. - payload = envelope.get("error") - if not isinstance(payload, Mapping): - raise HerdrEnvelopeError("Herdr error payload must be an object") - raise HerdrErrorResponse( - dict(payload), - request_id, - uncorrelated=True, - ) - ensure_response_id(envelope, request_id) - if not (is_result_response(envelope) or is_error_response(envelope)): - raise HerdrEnvelopeError("expected Herdr response envelope") - return envelope + envelope = self._read_server_envelope(deadline=deadline) + ensure_response_id(envelope, request_id) + if not (is_result_response(envelope) or is_error_response(envelope)): + raise HerdrEnvelopeError("expected Herdr response envelope") + return envelope - def _read_server_envelope( - self, - *, - deadline: float, - allow_uncorrelated_error: bool = False, - ) -> dict[str, Any]: + def _read_server_envelope(self, *, deadline: float) -> dict[str, Any]: line = self._read_line(deadline=deadline) envelope = parse_json_line(line) - return validate_server_envelope( - envelope, - allow_uncorrelated_error=allow_uncorrelated_error, - ) + return validate_server_envelope(envelope) def _read_line(self, *, deadline: float) -> bytes: while True: newline_index = self._buffer.find(b"\n") if newline_index >= 0: + if newline_index + 1 > _MAX_FRAME_BYTES: + self.close() + raise HerdrFrameTooLargeError("Herdr response frame is too large") line = bytes(self._buffer[: newline_index + 1]) del self._buffer[: newline_index + 1] return line @@ -462,3 +291,6 @@ def _read_line(self, *, deadline: float) -> bytes: "Herdr socket disconnected before a complete line was received" ) self._buffer.extend(chunk) + if len(self._buffer) > _MAX_FRAME_BYTES: + self.close() + raise HerdrFrameTooLargeError("Herdr response frame is too large") diff --git a/src/tendwire/cli.py b/src/tendwire/cli.py index e15500a..80b2940 100644 --- a/src/tendwire/cli.py +++ b/src/tendwire/cli.py @@ -15,17 +15,8 @@ from pathlib import Path from typing import Any, Mapping -from .backends.herdr_cli import ( - bindings_from_workers, - diagnose_herdr, - fetch_herdr_snapshot_observation, - fetch_herdr_state, - herdr_backend_health, - rehydrate_workers_from_bindings, -) -from .config import DEFAULT_TURN_MODEL, Config, load_config +from .config import Config, load_config from .core.actions import CommandContext, execute_command -from .core.attention import attention_payload_from_snapshot from .core.commands import ( STATUS_BACKEND_UNAVAILABLE, CommandEnvelope, @@ -33,42 +24,26 @@ parse_command_request, validate_request, ) -from .core.projector import project_from_observations from .core.models import ( - BackendHealth, - WorkerBinding, public_json_dumps, sanitize_public_mapping, - separate_duplicate_worker_bindings, ) from .core.turns import ( TURN_DELTA_DEFAULT_LIMIT, TURN_DELTA_MAX_LIMIT, TURN_LIST_DEFAULT_LIMIT, TURN_LIST_MAX_LIMIT, - turns_payload_from_snapshot, ) from .local_state import repair_config_state from .store.sqlite import ( CompactionOptions, - attention_payload_from_store, compact_store, - expire_stale_worker_bindings, - latest_healthy_backend_snapshot, - latest_snapshot, - list_worker_bindings, - pending_payload_from_store, run_store_maintenance, store_status, tail_event_metadata, - turns_payload_from_store, - turn_delta_payload_from_store, - upsert_worker_bindings, ) -_HERDR_BACKEND = "herdr" -_DEFAULT_FETCH_HERDR_STATE = fetch_herdr_state _DAEMON_FAST_CLIENT_TIMEOUT_SECONDS = 2.0 _DAEMON_CONTENT_CLIENT_TIMEOUT_SECONDS = 10.0 _DAEMON_CONNECTOR_CLIENT_TIMEOUT_SECONDS = 30.0 @@ -152,13 +127,13 @@ def _build_parser() -> argparse.ArgumentParser: "--herdr-timeout", dest="herdr_timeout_seconds", default=None, - help="Seconds to wait for each Herdr CLI probe (default: 5.0).", + help="Seconds to wait for each Herdr socket request (default: 5.0).", ) parser.add_argument( "--socket-path", dest="socket_path", default=None, - help="Unix socket path for daemon-backed requests when explicitly enabled.", + help="Unix socket path for daemon requests (default: data-dir/tendwire.sock).", ) subparsers = parser.add_subparsers(dest="command") @@ -174,19 +149,6 @@ def _build_parser() -> argparse.ArgumentParser: default=True, help="Print snapshot as JSON (default).", ) - snapshot_parser.add_argument( - "--store", - dest="store_snapshot", - action="store_true", - default=False, - help="Persist the snapshot to the sqlite store without changing stdout.", - ) - snapshot_parser.add_argument( - "--db-path", - dest="db_path", - default=None, - help="SQLite database path to use with --store (default: config path).", - ) attention_parser = subparsers.add_parser( "attention", @@ -199,19 +161,6 @@ def _build_parser() -> argparse.ArgumentParser: default=True, help="Print attention as JSON (default).", ) - attention_parser.add_argument( - "--store", - dest="store_snapshot", - action="store_true", - default=False, - help="Persist a fresh snapshot before listing store-backed attention.", - ) - attention_parser.add_argument( - "--db-path", - dest="db_path", - default=None, - help="SQLite database path for store-backed attention (default: config path).", - ) turns_parser = subparsers.add_parser( "turns", @@ -537,184 +486,6 @@ def add_common(action_parser: argparse.ArgumentParser) -> None: action_parser.add_argument("--lease-seconds", dest="lease_seconds", type=int, default=None) -def _load_worker_bindings(config: Config) -> list[WorkerBinding]: - if config.db_path is None: - return [] - return list_worker_bindings( - config.db_path, - config.host_id, - backend=_HERDR_BACKEND, - ) - - -def _fetch_state_with_bindings( - config: Config, - stored_bindings: list[WorkerBinding], -) -> tuple[list[Any], list[Any], list[WorkerBinding]]: - try: - result = fetch_herdr_state( - config, - stored_bindings=stored_bindings, - include_bindings=True, - ) - except TypeError: - spaces, workers = fetch_herdr_state(config) - return spaces, workers, bindings_from_workers(config, workers) - - if len(result) == 3: - spaces, workers, bindings = result - return spaces, workers, bindings - spaces, workers = result - return spaces, workers, bindings_from_workers(config, workers) - - -def _legacy_backend_health(spaces: list[Any], workers: list[Any]) -> list[BackendHealth]: - return [ - herdr_backend_health( - "healthy_non_empty" if spaces or workers else "unknown", - spaces=spaces, - workers=workers, - ) - ] - - -def _fetch_snapshot_observation_with_bindings( - config: Config, - stored_bindings: list[WorkerBinding], -) -> tuple[list[Any], list[Any], list[WorkerBinding], list[BackendHealth], bool]: - complete_barrier = False - if fetch_herdr_state is not _DEFAULT_FETCH_HERDR_STATE: - spaces, workers, bindings = _fetch_state_with_bindings(config, stored_bindings) - backend_health = _legacy_backend_health(spaces, workers) - else: - try: - observation = fetch_herdr_snapshot_observation( - config, - stored_bindings=stored_bindings, - ) - except TypeError: - spaces, workers, bindings = _fetch_state_with_bindings(config, stored_bindings) - backend_health = _legacy_backend_health(spaces, workers) - else: - spaces = list(getattr(observation, "spaces", []) or []) - workers = list(getattr(observation, "workers", []) or []) - bindings = list(getattr(observation, "bindings", []) or []) - backend_health = list(getattr(observation, "backend_health", []) or []) - complete_barrier = bool(backend_health) - if not backend_health: - backend_health = _legacy_backend_health(spaces, workers) - - health = _herdr_health_from_items(backend_health) - if health.status == "healthy": - return spaces, workers, bindings, backend_health, complete_barrier - - # Failed observations are not an authority for routing or continuity. - # Never persist their bindings, and retain the last authenticated public - # state when one has already been stored. - bindings = [] - if config.db_path is None: - return spaces, workers, bindings, backend_health, complete_barrier - - db_path = Path(config.db_path) - latest = latest_snapshot(db_path, config.host_id) - if latest is not None: - latest_health = _herdr_health_from_items(list(latest.backend_health)) - if latest_health.outcome == "continuity_unavailable": - health = latest_health - - previous = latest_healthy_backend_snapshot( - db_path, - config.host_id, - backend=_HERDR_BACKEND, - ) - if previous is not None: - spaces = list(previous.spaces) - workers = list(previous.workers) - - retained_health = herdr_backend_health( - health.outcome, - observed_at=health.observed_at, - message=health.message, - spaces=spaces, - workers=workers, - ) - backend_health = [ - retained_health if item.name == _HERDR_BACKEND else item - for item in backend_health - ] - if not any(item.name == _HERDR_BACKEND for item in backend_health): - backend_health.append(retained_health) - return spaces, workers, bindings, backend_health, complete_barrier - - - - -def _herdr_health_from_items(items: list[BackendHealth]) -> BackendHealth: - for item in items: - if getattr(item, "name", "") == _HERDR_BACKEND: - return item - return herdr_backend_health("unknown") - - - - -def observe_public_snapshot( - config: Config, - *, - store_snapshot: bool = False, -) -> Any: - """Build the public snapshot through the existing one-shot observation path.""" - # Always seed observation with stored bindings: they are what keeps public - # worker ids stable across snapshots. Skipping them re-letters duplicate - # worker names (claude, claude-1, ...) from scratch on every observation. - stored_bindings = _load_worker_bindings(config) - spaces, workers, bindings, backend_health, complete_barrier = ( - _fetch_snapshot_observation_with_bindings( - config, - stored_bindings, - ) - ) - snapshot = project_from_observations( - config, - spaces=spaces, - workers=workers, - backend_health=backend_health, - ) - - if store_snapshot: - from .store.sqlite import SnapshotObservationContext, save_snapshot - - if config.db_path is None: - raise RuntimeError("snapshot persistence requires a db path") - health = _herdr_health_from_items(backend_health) - authority = ( - "complete" - if complete_barrier - and health.status == "healthy" - and health.outcome in {"healthy_non_empty", "empty_healthy"} - else "none" - ) - save_snapshot( - config.db_path, - snapshot, - turn_model=DEFAULT_TURN_MODEL, - observation=SnapshotObservationContext( - authority=authority, - observed_at=health.observed_at, - ), - worker_bindings=bindings, - binding_backend=_HERDR_BACKEND, - binding_observation_authoritative=health.status == "healthy", - binding_workers_present=bool(workers), - ) - - return snapshot - - -def _current_public_snapshot(config: Config) -> Any: - return observe_public_snapshot(config) - - def _try_daemon_attempt( config: Config, method: str, @@ -722,10 +493,10 @@ def _try_daemon_attempt( *, preserve_content_text: bool = False, ) -> _DaemonAttempt: - """Return a daemon result only when a daemon socket was explicitly selected.""" - if config.socket_path is None: - return _DaemonAttempt(error_kind="unavailable", request_started=False) - socket_path = config.socket_path + """Return one result from the daemon's authoritative socket API.""" + from .daemon import default_socket_path + + socket_path = default_socket_path(config) try: from .daemon_api import ( @@ -800,61 +571,20 @@ def _try_daemon_attempt( return _DaemonAttempt(error_kind="protocol", request_started=True) -def _try_daemon_result( - config: Config, - method: str, - params: dict[str, Any] | None = None, -) -> dict[str, Any] | None: - """Return only a daemon result, preserving read-only fallback behavior.""" - return _try_daemon_attempt(config, method, params).result - - -def _persist_binding_observation( - config: Config, - bindings: list[WorkerBinding], - *, - observed_at: str, - workers_present: bool, - authoritative: bool = True, -) -> list[WorkerBinding]: - bindings = separate_duplicate_worker_bindings(bindings) - if config.db_path is None: - return bindings - if bindings: - upsert_worker_bindings(config.db_path, bindings) - if authoritative and (bindings or not workers_present): - expire_stale_worker_bindings( - config.db_path, - config.host_id, - backend=_HERDR_BACKEND, - current_private_fingerprints=[binding.private_fingerprint for binding in bindings], - now=observed_at, - ) - return bindings - - def cmd_snapshot( config: Config, *, json_output: bool = True, - store_snapshot: bool = False, ) -> int: - """Build and print a neutral snapshot.""" + """Read the daemon's current neutral snapshot.""" if json_output: - daemon_attempt = ( - _DaemonAttempt(error_kind="unavailable", request_started=False) - if store_snapshot - else _try_daemon_attempt(config, "snapshot.get") - ) + daemon_attempt = _try_daemon_attempt(config, "snapshot.get") if daemon_attempt.result is not None: payload = daemon_attempt.result code = 0 elif daemon_attempt.response_error is not None: payload = daemon_attempt.response_error code = 1 - elif daemon_attempt.request_started is False: - payload = observe_public_snapshot(config, store_snapshot=store_snapshot).to_dict() - code = 0 elif daemon_attempt.error_kind == "timeout": payload = { "schema_version": 2, @@ -870,10 +600,22 @@ def cmd_snapshot( payload = { "schema_version": 2, "ok": False, - "status": "daemon_protocol_error", + "status": ( + "daemon_unavailable" + if daemon_attempt.request_started is False + else "daemon_protocol_error" + ), "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", + "code": ( + "daemon_unavailable" + if daemon_attempt.request_started is False + else "daemon_protocol_error" + ), + "message": ( + "Tendwire daemon is unavailable" + if daemon_attempt.request_started is False + else "Tendwire daemon returned an invalid response" + ), }, } code = 1 @@ -1062,6 +804,41 @@ def _content_payload_json(payload: dict[str, Any], *, indent: int | None = None) ) +def _daemon_read_payload( + attempt: _DaemonAttempt, + *, + schema_version: int = 1, + extra: Mapping[str, Any] | None = None, +) -> dict[str, Any]: + if attempt.result is not None: + return attempt.result + if attempt.response_error is not None: + return attempt.response_error + status = ( + "daemon_timeout" + if attempt.error_kind == "timeout" + else "daemon_unavailable" + if attempt.request_started is False + else "daemon_protocol_error" + ) + payload: dict[str, Any] = { + "schema_version": schema_version, + "ok": False, + "status": status, + "error": { + "code": status, + "message": { + "daemon_timeout": "Tendwire daemon request timed out", + "daemon_unavailable": "Tendwire daemon is unavailable", + "daemon_protocol_error": "Tendwire daemon returned an invalid response", + }[status], + }, + } + if extra: + payload.update(extra) + return payload + + def cmd_turns( config: Config, *, @@ -1081,58 +858,17 @@ def cmd_turns( "cursor": cursor, "since": since, } - daemon_attempt = _try_daemon_attempt(config, "turn.list", params) - if daemon_attempt.result is not None: - payload = daemon_attempt.result - elif daemon_attempt.response_error is not None: - payload = daemon_attempt.response_error - elif ( - daemon_attempt.error_kind in {"unavailable", "timeout"} - and daemon_attempt.request_started is False - ): - if config.db_path is None: - payload = { - "schema_version": schema_version, - "host_id": config.host_id, - "ok": False, - "status": "store_unavailable", - } - else: - payload = turns_payload_from_store( - config.db_path, - config.host_id, - schema_version=schema_version, - limit=limit, - cursor=cursor, - since=since, - turn_model=DEFAULT_TURN_MODEL, - ) - elif daemon_attempt.error_kind == "timeout": - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_timeout", - "error": { - "code": "daemon_timeout", - "message": "Tendwire daemon request timed out", - }, - } - else: - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", - }, - } + payload = _daemon_read_payload( + _try_daemon_attempt(config, "turn.list", params), + schema_version=schema_version, + extra={"host_id": config.host_id}, + ) print(_turn_list_payload_json(payload, indent=2)) return 0 if payload.get("ok") is not False else 1 def cmd_turn_content_get(config: Config, args: argparse.Namespace) -> int: - """Fetch one bounded canonical content page with daemon/store parity.""" + """Fetch one bounded canonical content page from the daemon.""" params: dict[str, Any] = { "schema_version": 1, "turn_id": args.turn_id, @@ -1141,53 +877,14 @@ def cmd_turn_content_get(config: Config, args: argparse.Namespace) -> int: } if args.cursor is not None: params["cursor"] = args.cursor - daemon_attempt = _try_daemon_attempt( - config, - "turn.content.get", - params, - preserve_content_text=True, - ) - if daemon_attempt.result is not None: - payload = daemon_attempt.result - elif daemon_attempt.response_error is not None: - payload = daemon_attempt.response_error - elif daemon_attempt.error_kind not in {"unavailable", "timeout"}: - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "daemon returned an invalid response", - }, - } - elif config.db_path is None: - payload = { - "schema_version": 1, - "ok": False, - "status": "store_unavailable", - "error": { - "code": "store_unavailable", - "message": "command requires --db-path or a reachable daemon", - }, - } - else: - from .store.sqlite import get_turn_content, init_store - - init_store( - config.db_path, - connector_ack_ttl_seconds=config.connector_ack_ttl_seconds, - ) - payload = get_turn_content( - config.db_path, - config.host_id, - turn_id=args.turn_id, - content_revision=args.content_revision, - field=args.field, - cursor=args.cursor, - schema_version=1, - turn_model=DEFAULT_TURN_MODEL, + payload = _daemon_read_payload( + _try_daemon_attempt( + config, + "turn.content.get", + params, + preserve_content_text=True, ) + ) print(_content_payload_json(payload, indent=2)) return 0 if payload.get("ok") is not False and isinstance(payload.get("text"), str) else 1 @@ -1196,27 +893,14 @@ def cmd_attention( config: Config, *, json_output: bool = True, - store_snapshot: bool = False, ) -> int: """Print neutral public attention items.""" if not json_output: print("error: only --json output is supported", file=sys.stderr) return 2 - if not store_snapshot: - daemon_result = _try_daemon_result(config, "attention.list") - if daemon_result is not None: - print(public_json_dumps(daemon_result, indent=2)) - return 0 - if store_snapshot: - observe_public_snapshot(config, store_snapshot=True) - if config.db_path is not None: - payload = attention_payload_from_store(config.db_path, config.host_id) - if payload is not None: - print(public_json_dumps(payload, indent=2)) - return 0 - snapshot = _current_public_snapshot(config) - print(public_json_dumps(attention_payload_from_snapshot(snapshot), indent=2)) - return 0 + payload = _daemon_read_payload(_try_daemon_attempt(config, "attention.list")) + print(public_json_dumps(payload, indent=2)) + return 0 if payload.get("ok") is not False else 1 def cmd_pending( @@ -1224,84 +908,27 @@ def cmd_pending( *, json_output: bool = True, ) -> int: - """Print pending interactions from one daemon attempt or durable fallback.""" + """Print pending interactions from the daemon.""" if not json_output: print("error: only --json output is supported", file=sys.stderr) return 2 - daemon_attempt = _try_daemon_attempt(config, "pending.list") - if daemon_attempt.result is not None: - payload = daemon_attempt.result - elif daemon_attempt.response_error is not None: - payload = daemon_attempt.response_error - elif ( - daemon_attempt.error_kind == "unavailable" - and daemon_attempt.request_started is False - ): - payload = pending_payload_from_store(config.db_path, config.host_id) - elif daemon_attempt.error_kind == "timeout": - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_timeout", - "error": { - "code": "daemon_timeout", - "message": "Tendwire daemon request timed out", - }, - } - else: - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", - }, - } + payload = _daemon_read_payload(_try_daemon_attempt(config, "pending.list")) print(public_json_dumps(payload, indent=2)) return 0 if payload.get("ok") is not False else 1 def cmd_turn_delta(config: Config, args: argparse.Namespace) -> int: - """Read one delta page via daemon, with read-only store fallback.""" + """Read one delta page from the daemon.""" params = { "limit": args.limit, "watermark": args.watermark, "cursor": args.cursor, } - daemon_attempt = _try_daemon_attempt(config, "turn.delta", params) - if daemon_attempt.result is not None: - payload = daemon_attempt.result - elif daemon_attempt.response_error is not None: - payload = daemon_attempt.response_error - elif ( - daemon_attempt.error_kind in {"unavailable", "timeout"} - and daemon_attempt.request_started is False - and config.db_path is not None - ): - payload = turn_delta_payload_from_store( - config.db_path, - config.host_id, - watermark=args.watermark, - cursor=args.cursor, - limit=args.limit, - turn_model=DEFAULT_TURN_MODEL, - ) - elif daemon_attempt.error_kind == "timeout": - payload = { - "schema_version": 1, - "projection_schema_version": 2, - "ok": False, - "status": "daemon_timeout", - } - else: - payload = { - "schema_version": 1, - "projection_schema_version": 2, - "ok": False, - "status": "store_unavailable" if config.db_path is None else "daemon_protocol_error", - } + payload = _daemon_read_payload( + _try_daemon_attempt(config, "turn.delta", params), + extra={"projection_schema_version": 2}, + ) print(_turn_delta_payload_json(payload, indent=2)) return 0 if payload.get("ok") is not False else 1 @@ -1311,13 +938,13 @@ def cmd_doctor( *, json_output: bool = True, ) -> int: - """Run read-only backend diagnostics and print a JSON result.""" + """Read daemon and lifecycle diagnostics.""" if not json_output: print("error: only --json output is supported", file=sys.stderr) return 2 - payload = diagnose_herdr(config) + payload = _daemon_read_payload(_try_daemon_attempt(config, "health.get")) print(json.dumps(payload, indent=2, sort_keys=True)) - return 0 if payload.get("status") == "ok" else 1 + return 0 if payload.get("status") == "ok" and payload.get("ok") is not False else 1 def command_envelope_from_payload(config: Config, payload: str) -> CommandEnvelope: @@ -1346,31 +973,9 @@ def command_envelope_from_payload(config: Config, payload: str) -> CommandEnvelo CommandContext(host_id=config.host_id, workers=[]), ) - stored_bindings = _load_worker_bindings(config) - spaces, workers, current_bindings, backend_health, _complete_barrier = ( - _fetch_snapshot_observation_with_bindings( - config, - stored_bindings, - ) - ) - workers = rehydrate_workers_from_bindings( - workers, - current_bindings, - stored_bindings, - ) - snapshot = project_from_observations( - config, - spaces=spaces, - workers=workers, - backend_health=backend_health, - ) - return execute_command( + return CommandEnvelope.from_error( request, - CommandContext( - host_id=config.host_id, - workers=workers, - snapshot=snapshot, - ), + error_value(STATUS_BACKEND_UNAVAILABLE, "Tendwire daemon backend is unavailable"), ) @@ -1378,24 +983,6 @@ def _command_exit_code(envelope: CommandEnvelope) -> int: return 0 if envelope.ok else 1 -def _requires_daemon_for_mutating_command(config: Config, payload: str) -> Any | None: - """Return a live mutating request that must not fall back from the daemon.""" - if config.socket_path is None and config.herdr_backend != "socket": - return None - request, parse_error = parse_command_request(payload) - if parse_error is not None or request is None: - return None - validation_error = validate_request(request) - if validation_error is not None: - return None - if ( - request.action in {"send_instruction", "answer_pending", "answer_decision"} - and not request.dry_run - ): - return request - return None - - def _daemon_backend_failure_envelope( request: Any, attempt: _DaemonAttempt, @@ -1428,15 +1015,6 @@ def _strict_daemon_command_envelope( return envelope -def _replay_daemon_command_receipt( - config: Config, - payload: str, -) -> CommandEnvelope | None: - from .command_submission import replay_command_receipt - - return replay_command_receipt(config, payload) - - def cmd_command( config: Config, *, @@ -1463,7 +1041,6 @@ def cmd_command( in {"send_instruction", "answer_pending", "answer_decision"} and parsed_request.dry_run ) - daemon_required_request = _requires_daemon_for_mutating_command(config, payload) daemon_eligible = ( isinstance(request_payload, dict) and parse_error is None @@ -1485,18 +1062,14 @@ def cmd_command( error_kind="protocol", request_started=True, ) - if daemon_required_request is not None: + if parsed_request is not None and not parsed_request.dry_run: if daemon_attempt.request_started is False: envelope = _daemon_backend_failure_envelope( - daemon_required_request, + parsed_request, daemon_attempt, ) print(envelope.to_json(indent=2)) return _command_exit_code(envelope) - envelope = _replay_daemon_command_receipt(config, payload) - if envelope is not None: - print(envelope.to_json(indent=2)) - return _command_exit_code(envelope) print( "error: Tendwire daemon command result is unresolved", file=sys.stderr, @@ -1610,38 +1183,12 @@ def cmd_connector(config: Config, args: argparse.Namespace) -> int: } print(_connector_payload_json(payload, indent=2)) return 1 - if config.db_path is None: - payload = { - "schema_version": 1, - "ok": False, - "status": "store_unavailable", - "host_id": config.host_id, - "name": params.get("name", ""), - "error": { - "code": "store_unavailable", - "message": "command requires --db-path or a reachable daemon", - }, - } - print(public_json_dumps(payload, indent=2)) - return 1 - from .connectors import ConnectorOutboxAPI - from .store.sqlite import init_store - - init_store( - config.db_path, - connector_ack_ttl_seconds=config.connector_ack_ttl_seconds, + payload = _daemon_read_payload( + daemon_attempt, + extra={"host_id": config.host_id, "name": params.get("name", "")}, ) - payload = ConnectorOutboxAPI( - config.db_path, - config.host_id, - default_lease_seconds=config.connector_claim_ttl_seconds, - max_lease_seconds=config.connector_max_claim_ttl_seconds, - ack_ttl_seconds=config.connector_ack_ttl_seconds, - max_attempts=config.max_outbox_attempts, - turn_model=DEFAULT_TURN_MODEL, - ).dispatch(method, params) print(_connector_payload_json(payload, indent=2)) - return 0 if payload.get("ok") is not False else 1 + return 1 def cmd_store(config: Config, args: argparse.Namespace) -> int: @@ -1788,7 +1335,7 @@ def main(argv: list[str] | None = None) -> int: socket_group=getattr(args, "socket_group", None), herdr_timeout_seconds=args.herdr_timeout_seconds, ) - if args.command not in {"daemon", "doctor"} and not ( + if args.command in {"daemon", "store"} and not ( args.command == "store" and args.store_action == "compact" ): repair_config_state( @@ -1805,14 +1352,12 @@ def main(argv: list[str] | None = None) -> int: return cmd_snapshot( config, json_output=args.json_output, - store_snapshot=args.store_snapshot, ) if args.command == "attention": return cmd_attention( config, json_output=args.json_output, - store_snapshot=args.store_snapshot, ) if args.command == "turns": diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 3f08ff9..36a17c3 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -171,11 +171,6 @@ def _backend_unavailable( def _backend_health_error(config: Config, request: CommandRequest, snapshot: Snapshot) -> CommandEnvelope | None: - if config.herdr_backend != "socket": - return _backend_unavailable( - request, - "Herdr socket backend is not enabled", - ) health = _backend_health(snapshot) if health.status != "healthy": return _backend_unavailable( diff --git a/src/tendwire/config.py b/src/tendwire/config.py index 7866b1b..788bb7a 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -13,7 +13,6 @@ from dataclasses import dataclass, field from pathlib import Path -HERDR_BACKENDS = frozenset({"cli", "socket"}) ACP_THOUGHT_POLICIES = frozenset({"disabled", "private_summary", "private_all"}) ACP_CONSOLE_INPUT_POLICIES = frozenset({"preserve", "live_only"}) DEFAULT_TURN_MODEL = "observed" @@ -22,11 +21,8 @@ DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS = 30.0 DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS = 5.0 DEFAULT_ACP_MAX_FRAME_BYTES = 8 * 1024 * 1024 -DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS = 120.0 -DEFAULT_EVENT_DEBOUNCE_SECONDS = 0.05 -DEFAULT_RECONCILE_INTERVAL_SECONDS = 300.0 +DEFAULT_RECONCILE_INTERVAL_SECONDS = 15.0 DEFAULT_EVENT_RETENTION_DAYS = 7 -DEFAULT_OUTPUT_EXCERPT_CHARS = 200 DEFAULT_MAX_WORKERS = 512 DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS = 60 DEFAULT_SUBMISSION_HARD_TTL_SECONDS = 86_400 @@ -65,19 +61,13 @@ class Config: db_path: Path | None = None socket_path: Path | None = None herdr_timeout_seconds: float = 5.0 - herdr_initial_reconcile_timeout_seconds: float = ( - DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS - ) - herdr_backend: str = "cli" acp_thought_policy: str = DEFAULT_ACP_THOUGHT_POLICY acp_console_input_policy: str = DEFAULT_ACP_CONSOLE_INPUT_POLICY acp_request_timeout_seconds: float = DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS acp_shutdown_timeout_seconds: float = DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS acp_max_frame_bytes: int = DEFAULT_ACP_MAX_FRAME_BYTES - event_debounce_seconds: float = DEFAULT_EVENT_DEBOUNCE_SECONDS reconcile_interval_seconds: float = DEFAULT_RECONCILE_INTERVAL_SECONDS event_retention_days: int = DEFAULT_EVENT_RETENTION_DAYS - output_excerpt_chars: int = DEFAULT_OUTPUT_EXCERPT_CHARS max_workers: int = DEFAULT_MAX_WORKERS submission_link_window_seconds: int = DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS submission_hard_ttl_seconds: int = DEFAULT_SUBMISSION_HARD_TTL_SECONDS @@ -125,19 +115,6 @@ def __post_init__(self) -> None: "herdr_timeout_seconds", ), ) - object.__setattr__( - self, - "herdr_initial_reconcile_timeout_seconds", - _positive_finite_float( - self.herdr_initial_reconcile_timeout_seconds, - "herdr_initial_reconcile_timeout_seconds", - ), - ) - backend = str(self.herdr_backend or "").strip().lower() - if backend not in HERDR_BACKENDS: - allowed = ", ".join(sorted(HERDR_BACKENDS)) - raise ValueError(f"herdr_backend must be one of: {allowed}") - object.__setattr__(self, "herdr_backend", backend) acp_thought_policy = str(self.acp_thought_policy or "").strip().lower() if acp_thought_policy not in ACP_THOUGHT_POLICIES: allowed = ", ".join(sorted(ACP_THOUGHT_POLICIES)) @@ -181,11 +158,6 @@ def __post_init__(self) -> None: maximum=64 * 1024 * 1024, ), ) - object.__setattr__( - self, - "event_debounce_seconds", - _non_negative_float(self.event_debounce_seconds, "event_debounce_seconds"), - ) object.__setattr__( self, "reconcile_interval_seconds", @@ -200,11 +172,6 @@ def __post_init__(self) -> None: maximum=MAX_RETENTION_DAYS, ), ) - object.__setattr__( - self, - "output_excerpt_chars", - _positive_int(self.output_excerpt_chars, "output_excerpt_chars", minimum=1), - ) object.__setattr__( self, "max_workers", @@ -477,17 +444,13 @@ def load_config( socket_path: str | Path | None = None, socket_group: str | None = None, herdr_timeout_seconds: float | str | None = None, - herdr_initial_reconcile_timeout_seconds: float | str | None = None, - herdr_backend: str | None = None, acp_thought_policy: str | None = None, acp_console_input_policy: str | None = None, acp_request_timeout_seconds: float | str | None = None, acp_shutdown_timeout_seconds: float | str | None = None, acp_max_frame_bytes: int | str | None = None, - event_debounce_seconds: float | str | None = None, reconcile_interval_seconds: float | str | None = None, event_retention_days: int | str | None = None, - output_excerpt_chars: int | str | None = None, max_workers: int | str | None = None, submission_link_window_seconds: int | str | None = None, submission_hard_ttl_seconds: int | str | None = None, @@ -517,7 +480,6 @@ def load_config( env_socket_path = os.environ.get("TENDWIRE_SOCKET_PATH") env_socket_group = os.environ.get("TENDWIRE_SOCKET_GROUP") env_herdr_timeout_seconds = os.environ.get("TENDWIRE_HERDR_TIMEOUT_SECONDS") - env_herdr_backend = os.environ.get("TENDWIRE_HERDR_BACKEND") resolved_host_id = host_id or env_host_id or (platform.node() or "unknown") resolved_herdr_bin = herdr_bin or env_herdr_bin or "herdr" @@ -558,12 +520,6 @@ def load_config( except (TypeError, ValueError) as exc: raise ValueError("herdr timeout must be a positive number") from exc - resolved_herdr_backend = herdr_backend - if resolved_herdr_backend is None: - resolved_herdr_backend = env_herdr_backend - if resolved_herdr_backend is None: - resolved_herdr_backend = "cli" - return Config( host_id=resolved_host_id, herdr_bin=resolved_herdr_bin, @@ -571,12 +527,6 @@ def load_config( db_path=resolved_db_path, socket_path=resolved_socket_path, herdr_timeout_seconds=resolved_herdr_timeout_seconds, - herdr_initial_reconcile_timeout_seconds=_resolve_value( - herdr_initial_reconcile_timeout_seconds, - "TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS", - DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS, - ), - herdr_backend=resolved_herdr_backend, acp_thought_policy=_resolve_value( acp_thought_policy, "TENDWIRE_ACP_THOUGHT_POLICY", @@ -602,11 +552,6 @@ def load_config( "TENDWIRE_ACP_MAX_FRAME_BYTES", DEFAULT_ACP_MAX_FRAME_BYTES, ), - event_debounce_seconds=_resolve_value( - event_debounce_seconds, - "TENDWIRE_EVENT_DEBOUNCE_SECONDS", - DEFAULT_EVENT_DEBOUNCE_SECONDS, - ), reconcile_interval_seconds=_resolve_value( reconcile_interval_seconds, "TENDWIRE_RECONCILE_INTERVAL_SECONDS", @@ -617,11 +562,6 @@ def load_config( "TENDWIRE_EVENT_RETENTION_DAYS", DEFAULT_EVENT_RETENTION_DAYS, ), - output_excerpt_chars=_resolve_value( - output_excerpt_chars, - "TENDWIRE_OUTPUT_EXCERPT_CHARS", - DEFAULT_OUTPUT_EXCERPT_CHARS, - ), max_workers=_resolve_value( max_workers, "TENDWIRE_MAX_WORKERS", diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index 3c76e31..2a7690f 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -464,12 +464,6 @@ def _default_init_store( init_store(db_path, **kwargs) -def _default_observe_initial_snapshot(config: Config) -> Snapshot: - from .cli import observe_public_snapshot - - return observe_public_snapshot(config, store_snapshot=True) - - def _default_acp_supervisor_factory(config: Config, stop_event: threading.Event) -> Any: from .backends.acp_coordinator import production_acp_supervisor_factory @@ -481,8 +475,6 @@ class DaemonHooks: """Dependency injection points for deterministic daemon tests.""" init_store: Callable[[Path], None] = _default_init_store - observe_initial_snapshot: Callable[[Config], Snapshot] = _default_observe_initial_snapshot - event_backend_factory: Callable[[Config, threading.Event], Any] | None = None acp_supervisor_factory: Callable[[Config, threading.Event], Any | None] | None = ( _default_acp_supervisor_factory ) @@ -507,7 +499,6 @@ def __init__( self.started_at = utc_timestamp() self._snapshot: Snapshot | None = None self._server: UnixSocketJSONServer | None = None - self._event_backend: Any | None = None self._acp_supervisor: Any | None = None self._acp_startup_failure_type: str | None = None self._stop_lock = threading.Lock() @@ -554,13 +545,15 @@ def start(self) -> None: else: self.hooks.init_store(Path(self.config.db_path)) self._connector_periodic_tick() - if self.config.herdr_backend == "socket": - self._snapshot = self._start_socket_event_backend() - else: - self._snapshot = self.hooks.observe_initial_snapshot(self.config) - self._after_snapshot_saved() - self._start_acp_supervisor() + from .store.sqlite import latest_snapshot + + self._snapshot = latest_snapshot( + Path(self.config.db_path), self.config.host_id + ) + if self._snapshot is None: + raise RuntimeError("ACP supervisor did not publish a lifecycle snapshot") + self._after_snapshot_saved() api = TendwireDaemonAPI( get_snapshot=self.get_snapshot, @@ -590,16 +583,9 @@ def start(self) -> None: except Exception: self.stop_event.set() - backend = self._event_backend supervisor = self._acp_supervisor self._acp_supervisor = None self._stop_acp_supervisor(supervisor) - self._event_backend = None - if backend is not None: - try: - backend.stop() - except Exception: - pass self._server = None if server is not None: try: @@ -633,10 +619,8 @@ def stop(self) -> None: with self._stop_lock: self.stop_event.set() server = self._server - backend = self._event_backend supervisor = self._acp_supervisor self._server = None - self._event_backend = None self._acp_supervisor = None if server is not None: @@ -647,19 +631,6 @@ def stop(self) -> None: self._stop_acp_supervisor(supervisor) - if backend is not None: - flush = getattr(backend, "flush", None) - if callable(flush): - try: - flush() - except Exception: - pass - if backend is not None: - try: - backend.stop() - except Exception: - pass - def _start_acp_supervisor(self) -> None: """Start the required ACP supervisor and fail the daemon closed.""" self._acp_startup_failure_type = None @@ -767,6 +738,10 @@ def field(name: str) -> Any: "healthy": healthy, "state": state, "failure_type": failure_type, + "last_reconcile_at": _valid_observation_timestamp( + field("last_reconcile_at") + ), + "worker_count": _nonnegative_int(field("worker_count")), "counters": counters, } @@ -861,44 +836,6 @@ def _after_snapshot_saved(self) -> None: else: self._automatic_maintenance_status = maintenance_status - def _start_socket_event_backend(self) -> Snapshot: - if self.hooks.event_backend_factory is None: - from .backends.herdr_events import HerdrEventBackend - - backend = HerdrEventBackend(self.config, stop_event=self.stop_event) - else: - backend = self.hooks.event_backend_factory(self.config, self.stop_event) - self._event_backend = backend - backend.start(wait_for_reconcile=True) - from .store.sqlite import SnapshotObservationContext, latest_snapshot, save_snapshot - - snapshot = latest_snapshot(Path(self.config.db_path), self.config.host_id) - if snapshot is not None: - return snapshot - from .backends.herdr_cli import herdr_backend_health - from .core.projector import project_from_observations - - backend_health = ( - backend.health.to_backend_health() - if hasattr(backend, "health") - else herdr_backend_health("unknown") - ) - snapshot = project_from_observations( - self.config, - backend_health=[backend_health], - ) - save_snapshot( - Path(self.config.db_path), - snapshot, - turn_model=DEFAULT_TURN_MODEL, - observation=SnapshotObservationContext( - authority="none", - observed_at=_valid_observation_timestamp(backend_health.observed_at), - ), - ) - self._after_snapshot_saved() - return snapshot - def get_snapshot(self) -> Snapshot: if self.config.db_path is not None: from .store.sqlite import latest_snapshot @@ -1015,11 +952,14 @@ def get_health(self) -> dict[str, Any]: and command_requests_valid and maintenance_valid ) - backend_runtime: dict[str, Any] = {} - if self._event_backend is not None and hasattr(self._event_backend, "operational_status"): - status_value = getattr(self._event_backend, "operational_status") - if isinstance(status_value, Mapping): - backend_runtime = dict(status_value) + acp_health = self._acp_supervisor_health() + backend_runtime: dict[str, Any] = { + "status": "healthy" if acp_health["healthy"] else "degraded", + "outcome": "healthy_non_empty" if acp_health.get("worker_count", 0) else "empty_healthy", + "ready": acp_health["healthy"], + "running": acp_health.get("state") == "running", + "last_reconcile_at": acp_health.get("last_reconcile_at"), + } backend_maintenance = backend_runtime.get("automatic_maintenance") runtime_maintenance = ( backend_maintenance @@ -1059,7 +999,6 @@ def get_health(self) -> dict[str, Any]: or stored_last_snapshot_at or snapshot.updated_at ) - acp_health = self._acp_supervisor_health() payload = { "schema_version": 1, "status": ( @@ -1113,10 +1052,8 @@ def get_health(self) -> dict[str, Any]: "acp": acp_health, "pending_ingestion": pending_ingestion, "limits": { - "event_debounce_seconds": self.config.event_debounce_seconds, "reconcile_interval_seconds": self.config.reconcile_interval_seconds, "event_retention_days": self.config.event_retention_days, - "output_excerpt_chars": self.config.output_excerpt_chars, "max_workers": self.config.max_workers, "max_outbox_attempts": self.config.max_outbox_attempts, "outbox_claim_ttl_seconds": self.config.connector_claim_ttl_seconds, diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 39d98f4..7a94147 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -63,7 +63,6 @@ def _config(tmp_path: Path) -> Config: host_id="acp-host", data_dir=tmp_path, db_path=tmp_path / "tendwire.db", - herdr_backend="socket", herdr_bin="herdr", ) @@ -1927,7 +1926,25 @@ def test_coordinator_start_revokes_orphaned_process_binding(tmp_path: Path) -> N def test_production_coordinator_installs_durable_permission_bridge( tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, ) -> None: + class EmptyLifecycleClient: + def workspace_list(self, *, timeout: float) -> list[Any]: + return [] + + def pane_list(self, *, timeout: float) -> list[Any]: + return [] + + def agent_list(self, *, timeout: float) -> list[Any]: + return [] + + def close(self) -> None: + return None + + monkeypatch.setattr( + "tendwire.backends.acp_coordinator._default_endpoint_client_factory", + lambda _config: EmptyLifecycleClient(), + ) config = _config(tmp_path) assert config.db_path is not None init_store(config.db_path) diff --git a/tests/test_acp_permissions.py b/tests/test_acp_permissions.py index f1cb93c..6acb602 100644 --- a/tests/test_acp_permissions.py +++ b/tests/test_acp_permissions.py @@ -44,7 +44,6 @@ def _config(tmp_path: Path) -> Config: host_id="cmd-host", data_dir=tmp_path, db_path=tmp_path / "commands.db", - herdr_backend="socket", ) diff --git a/tests/test_backend.py b/tests/test_backend.py deleted file mode 100644 index fa82967..0000000 --- a/tests/test_backend.py +++ /dev/null @@ -1,1598 +0,0 @@ -"""Tests for the Herdr CLI backend adapter contract.""" - -from __future__ import annotations - -import json -import os -import subprocess -from collections.abc import Sequence -from typing import Any - -import pytest - -from tendwire import cli as tendwire_cli -from tendwire.backends import herdr_cli -from tendwire.backends.herdr_cli import fetch_herdr_state -from tendwire.config import Config -from tendwire.core.models import Worker, WorkerBinding, worker_binding_private_fingerprint -from tendwire.core.projector import project_from_observations -from tendwire.store.sqlite import init_store, list_worker_bindings - - -_FORBIDDEN_FIELDS = { - "telegram", - "chat_id", - "topic_id", - "message_id", - "thread_id", - "token", - "bot_token", - "delivery", - "route", - "herdres_delivery", - "pane_id", - "terminal_id", - "backend_target", - "agent_session", - "session_id", - "herdr_state", - "herdres_state", - "target_kind", - "target_value", - "turn_target_kind", - "turn_target_value", - "private_fingerprint", - "argv", - "command", - "env", - "stderr", - "stdout", - "secret", - "secrets", - "shell", - "connector", - "connectors", -} -_FORBIDDEN_FIELDS_COMPACT = {field.replace("_", "") for field in _FORBIDDEN_FIELDS} - - -def _completed(stdout: str, returncode: int = 0) -> subprocess.CompletedProcess[str]: - return subprocess.CompletedProcess( - args=["herdr", "workspace", "list", "--json"], - returncode=returncode, - stdout=stdout, - stderr="", - ) - - -def _respond(args: Sequence[str], responses: dict[tuple[str, ...], Any]) -> subprocess.CompletedProcess[str] | None: - """Return a canned response for a herdr command tuple.""" - key = tuple(args) - if key not in responses: - return _completed("", returncode=1) - response = responses[key] - if isinstance(response, subprocess.CompletedProcess): - return response - if isinstance(response, str): - return _completed(response) - return _completed(json.dumps(response)) - - -def _assert_no_forbidden_fields(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - normalized = str(key).lower().replace("-", "_").replace(".", "_") - compact = normalized.replace("_", "") - segments = {part for part in normalized.split("_") if part} - assert ( - normalized not in _FORBIDDEN_FIELDS and compact not in _FORBIDDEN_FIELDS_COMPACT - and not (segments & _FORBIDDEN_FIELDS) - ), f"forbidden field {path}.{key}" - _assert_no_forbidden_fields(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_forbidden_fields(item, f"{path}[{index}]") - - -def test_fetch_herdr_state_returns_empty_when_binary_missing() -> None: - config = Config(host_id="testhost", herdr_bin="definitely-not-a-real-herdr-binary") - spaces, workers = fetch_herdr_state(config) - assert spaces == [] - assert workers == [] - - -def test_fetch_herdr_state_returns_empty_on_cli_failure(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli, - "_run_herdr", - lambda args, cfg: _completed('{"workers":[{"id":"leaked"}]}', returncode=2), - ) - - spaces, workers = fetch_herdr_state(config) - - assert spaces == [] - assert workers == [] - - -def test_fetch_herdr_state_returns_empty_on_malformed_json(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _completed("not json")) - - spaces, workers = fetch_herdr_state(config) - - assert spaces == [] - assert workers == [] - - -def test_sample_herdr_projection_is_neutral_and_fingerprinted(monkeypatch) -> None: - config = Config(host_id="herdr-host", herdr_bin="herdr") - sample_payload = { - "spaces": [ - { - "id": "space-1", - "name": "Build", - "status": "running", - "status_line": "building package", - "telegram": "forbidden", - "chat_id": 111, - "safe": "space-meta", - } - ], - "workers": [ - { - "id": "worker-1", - "name": "Agent One", - "status": "panic", - "space_id": "space-1", - "summary": "crashed", - "topic_id": 222, - "message_id": 333, - "route": "telegram", - "delivery": {"chat_id": 444}, - "safe": "worker-meta", - } - ], - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli, - "_run_herdr", - lambda args, cfg: _completed(json.dumps(sample_payload)), - ) - - spaces, workers = fetch_herdr_state(config) - snapshot = project_from_observations(config, spaces=spaces, workers=workers) - payload = json.loads(snapshot.to_json()) - - assert payload["schema_version"] == 2 - assert len(payload["content_fingerprint"]) == 24 - assert payload["spaces"][0]["status"] == "active" - assert payload["spaces"][0]["fingerprint"] - assert payload["spaces"][0]["meta"]["safe"] == "space-meta" - assert payload["workers"][0]["status"] == "failed" - assert payload["workers"][0]["fingerprint"] - assert payload["workers"][0]["meta"]["raw_status"] == "panic" - assert payload["workers"][0]["meta"]["safe"] == "worker-meta" - assert payload["attention"][0]["source"] == "worker:worker-1" - assert payload["attention"][0]["fingerprint"] - _assert_no_forbidden_fields(payload) - - -def test_herdr_cli_strip_connector_fields_drops_dot_separated_aliases() -> None: - raw = { - "safe": "kept", - "backend.target": "sentinel-private-backend", - "message.id": "sentinel-private-message", - "bot.token": "sentinel-private-token", - "herdres.delivery": {"message.id": "sentinel-private-delivery"}, - "delivery.route": "sentinel-private-route", - "telegram.message.id": "sentinel-private-telegram", - "children": [ - { - "safe_child": "kept", - "topic.id": "sentinel-private-topic", - } - ], - } - - stripped = herdr_cli._strip_connector_fields(raw) - - assert stripped == {"safe": "kept", "children": [{"safe_child": "kept"}]} - assert "sentinel-private" not in json.dumps(stripped, sort_keys=True) - _assert_no_forbidden_fields(stripped) - assert herdr_cli._safe_text_sample("failed backend.target=sentinel-private-backend") is None - assert all( - herdr_cli._safe_text_sample(sample) is None - for sample in ( - "failed bot.token=sentinel-private-token", - "failed bot_token=sentinel-private-token", - "failed bot-token=sentinel-private-token", - "failed botToken=sentinel-private-token", - ) - ) - assert herdr_cli._safe_text_sample("plain diagnostic text") == "plain diagnostic text" - - -def test_herdr_agent_public_id_and_private_backend_target_are_separate(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-worker", - "agent_id": "send-agent", - "agent": "Coder", - "agent_session": {"value": "sess-must-not-leak"}, - "terminal_id": "term-must-not-leak", - "pane_id": "pane-must-not-leak", - "session_id": "session-must-not-leak", - "workspace_id": "ws-1", - } - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - snapshot = project_from_observations(config, spaces=spaces, workers=workers) - payload = json.loads(snapshot.to_json()) - - assert len(workers) == 1 - assert workers[0].id == "public-worker" - assert workers[0].backend_target is not None - assert workers[0].backend_target["kind"] == "agent_id" - assert workers[0].backend_target["value"] == "send-agent" - assert workers[0].backend_target["sendable"] is True - assert workers[0].backend_target["reason"] is None - assert payload["workers"][0]["id"] == "public-worker" - assert payload["workers"][0]["name"] == "Coder" - assert "sess-must-not-leak" not in json.dumps(payload) - assert "term-must-not-leak" not in json.dumps(payload) - assert "pane-must-not-leak" not in json.dumps(payload) - assert "session-must-not-leak" not in json.dumps(payload) - _assert_no_forbidden_fields(payload) - - -def test_herdr_bindings_reuse_worker_id_by_private_fingerprint_and_update_moved_target(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - first_responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-before", - "agent": "Coder", - "agent_session": {"value": "sess-stable"}, - "terminal_id": "term-before", - "pane_id": "pane-before", - "workspace_id": "ws-1", - } - ] - } - }, - } - second_responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-after", - "agent": "Coder", - "agent_session": {"value": "sess-stable"}, - "terminal_id": "term-after", - "pane_id": "pane-after", - "workspace_id": "ws-1", - } - ] - } - }, - } - responses = [first_responses, second_responses] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str] | None: - return _respond(args, responses[0]) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - _spaces, workers, bindings = fetch_herdr_state(config, include_bindings=True) - assert workers[0].id == "public-before" - assert bindings[0].worker_id == "public-before" - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == "term-before" - - responses.pop(0) - _spaces2, workers2, bindings2 = fetch_herdr_state( - config, - stored_bindings=bindings, - include_bindings=True, - ) - - assert workers2[0].id == "public-before" - assert bindings2[0].worker_id == "public-before" - assert bindings2[0].private_fingerprint == bindings[0].private_fingerprint - assert bindings2[0].target_kind == "terminal_id" - assert bindings2[0].target_value == "term-after" - payload = json.loads(project_from_observations(config, workers=workers2).to_json()) - assert "term-after" not in json.dumps(payload) - assert "pane-after" not in json.dumps(payload) - _assert_no_forbidden_fields(payload) - - -def test_herdr_reuses_worker_id_by_unique_backend_target_fallback(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - stored = [ - WorkerBinding( - host_id="testhost", - worker_id="stored-public", - worker_fingerprint="stored-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-send", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="2026-01-02T00:00:00+00:00", - private_fingerprint="old-private", - ) - ] - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "new-public", - "agent_id": "agent-send", - "agent": "Coder", - } - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _spaces, workers, bindings = fetch_herdr_state( - config, - stored_bindings=stored, - include_bindings=True, - ) - - assert workers[0].id == "stored-public" - assert bindings[0].worker_id == "stored-public" - assert bindings[0].target_value == "agent-send" - assert bindings[0].private_fingerprint != "old-private" - - -def test_herdr_backend_target_precedence_and_pane_fallback(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "id": "public-id", - "agent_id": "agent-send", - "agent": "agent-name", - "label": "agent-label", - "terminal_id": "term-fallback", - "pane_id": "pane-fallback", - }, - { - "slug": "pane-public", - "agent_session": {"value": "sess-not-sendable"}, - "terminal_id": "term-send", - "pane_id": "pane-send", - "state_labels": ["agent"], - }, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _, workers, bindings = fetch_herdr_state(config, include_bindings=True) - by_id = {worker.id: worker for worker in workers} - bindings_by_id = {binding.worker_id: binding for binding in bindings} - - assert by_id["public-id"].backend_target is not None - assert by_id["public-id"].backend_target["kind"] == "agent_id" - assert by_id["public-id"].backend_target["value"] == "agent-send" - assert by_id["public-id"].backend_target["sendable"] is True - assert by_id["pane-public"].backend_target is not None - assert by_id["pane-public"].backend_target["kind"] == "terminal_id" - assert by_id["pane-public"].backend_target["value"] == "term-send" - assert by_id["pane-public"].backend_target["sendable"] is True - assert bindings_by_id["public-id"].turn_target_kind is None - assert bindings_by_id["public-id"].turn_target_value is None - assert bindings_by_id["pane-public"].turn_target_kind is None - assert bindings_by_id["pane-public"].turn_target_value is None - assert all( - (worker.backend_target or {}).get("value") != "sess-not-sendable" - for worker in workers - ) - - -def test_duplicate_sendable_backend_targets_are_marked_not_sendable(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "public-a", "agent_id": "same-send", "agent": "A"}, - {"worker_id": "public-b", "agent_id": "same-send", "agent": "B"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _, workers = fetch_herdr_state(config) - - assert {worker.id for worker in workers} == {"public-a", "public-b"} - assert all((worker.backend_target or {}).get("kind") == "agent_id" for worker in workers) - assert all((worker.backend_target or {}).get("value") == "same-send" for worker in workers) - assert all((worker.backend_target or {}).get("sendable") is False for worker in workers) - assert all( - (worker.backend_target or {}).get("reason") == "duplicate_backend_target" - for worker in workers - ) - assert herdr_cli.assert_unique_sendable_backend_targets(workers) is True - - -def test_duplicate_backend_targets_mark_bindings_unsendable(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "public-a", "agent_id": "same-send", "agent": "A"}, - {"worker_id": "public-b", "agent_id": "same-send", "agent": "B"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _spaces, workers, bindings = fetch_herdr_state(config, include_bindings=True) - - assert {worker.id for worker in workers} == {"public-a", "public-b"} - assert len(bindings) == 2 - assert all(binding.target_kind == "agent_id" for binding in bindings) - assert all(binding.target_value == "same-send" for binding in bindings) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "duplicate_backend_target" for binding in bindings) - assert len({binding.private_fingerprint for binding in bindings}) == 2 - - -def test_bindings_from_workers_marks_duplicate_targets_unsendable() -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - workers = [ - Worker( - id="public-a", - name="A", - status="active", - backend_target={"kind": "agent_id", "value": "same-agent", "sendable": True, "reason": None}, - ), - Worker( - id="public-b", - name="B", - status="active", - backend_target={"kind": "agent_id", "value": "same-agent", "sendable": True, "reason": None}, - ), - ] - - bindings = herdr_cli.bindings_from_workers(config, workers, observed_at="2026-01-01T00:00:00+00:00") - - assert len(bindings) == 2 - assert {binding.worker_id for binding in bindings} == {"public-a", "public-b"} - assert {binding.sendable for binding in bindings} == {False} - assert {binding.reason for binding in bindings} == {"duplicate_backend_target"} - - -def test_duplicate_private_identity_bindings_stay_separate_across_reobserve(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-a", - "agent_id": "same-agent", - "agent": "A", - "workspace_id": "ws-1", - }, - { - "worker_id": "public-b", - "agent_id": "same-agent", - "agent": "B", - "workspace_id": "ws-1", - }, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _spaces, workers, bindings = fetch_herdr_state(config, include_bindings=True) - _spaces2, workers2, bindings2 = fetch_herdr_state( - config, - stored_bindings=bindings, - include_bindings=True, - ) - - assert {worker.id for worker in workers} == {"public-a", "public-b"} - assert {worker.id for worker in workers2} == {"public-a", "public-b"} - assert len(bindings) == 2 - assert len(bindings2) == 2 - assert {binding.sendable for binding in bindings} == {False} - assert {binding.reason for binding in bindings} == {"duplicate_backend_target"} - assert len({binding.private_fingerprint for binding in bindings}) == 2 - assert {binding.private_fingerprint for binding in bindings2} == { - binding.private_fingerprint for binding in bindings - } - - -def test_legacy_collapsed_private_identity_does_not_rewrite_duplicate_public_ids(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - original_private = worker_binding_private_fingerprint( - host_id="testhost", - backend="herdr", - identity_material={ - "agent_id": "same-agent", - "agent_session": None, - "session_id": None, - "space_id": "ws-1", - }, - ) - stored = [ - WorkerBinding( - host_id="testhost", - worker_id="collapsed-public", - worker_fingerprint="collapsed-fp", - backend="herdr", - target_kind="agent_id", - target_value="same-agent", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="2026-01-02T00:00:00+00:00", - private_fingerprint=original_private, - ) - ] - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-a", - "agent_id": "same-agent", - "agent": "A", - "workspace_id": "ws-1", - }, - { - "worker_id": "public-b", - "agent_id": "same-agent", - "agent": "B", - "workspace_id": "ws-1", - }, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _spaces, workers, bindings = fetch_herdr_state( - config, - stored_bindings=stored, - include_bindings=True, - ) - - assert {worker.id for worker in workers} == {"public-a", "public-b"} - assert "collapsed-public" not in {binding.worker_id for binding in bindings} - assert len({binding.private_fingerprint for binding in bindings}) == 2 - - -def test_duplicate_final_send_tokens_across_backend_kinds_are_not_sendable(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "agent-id-worker", "agent_id": "same-send", "agent": "A"}, - {"worker_id": "name-worker", "name": "same-send"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _, workers = fetch_herdr_state(config) - by_id = {worker.id: worker for worker in workers} - - assert by_id["agent-id-worker"].backend_target["kind"] == "agent_id" - assert by_id["name-worker"].backend_target["kind"] == "name" - assert all((worker.backend_target or {}).get("value") == "same-send" for worker in workers) - assert all((worker.backend_target or {}).get("sendable") is False for worker in workers) - assert all( - (worker.backend_target or {}).get("reason") == "duplicate_backend_target" - for worker in workers - ) - assert herdr_cli.assert_unique_sendable_backend_targets(workers) is True - - -def test_name_and_label_backend_fallbacks_are_sendable_only_when_unique(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "name-unique", "name": "NameUnique"}, - {"worker_id": "label-unique", "label": "LabelUnique"}, - {"worker_id": "name-dupe-a", "name": "DupText"}, - {"worker_id": "label-dupe-b", "label": "DupText"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _, workers = fetch_herdr_state(config) - by_id = {worker.id: worker for worker in workers} - - assert by_id["name-unique"].backend_target == { - "kind": "name", - "value": "NameUnique", - "sendable": True, - "reason": None, - } - assert by_id["label-unique"].backend_target == { - "kind": "label", - "value": "LabelUnique", - "sendable": True, - "reason": None, - } - assert by_id["name-dupe-a"].backend_target["sendable"] is False - assert by_id["name-dupe-a"].backend_target["reason"] == "duplicate_backend_target" - assert by_id["label-dupe-b"].backend_target["sendable"] is False - assert by_id["label-dupe-b"].backend_target["reason"] == "duplicate_backend_target" - - -def test_fetch_herdr_command_observation_reports_healthy_empty(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): {"result": {"panes": []}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", lambda args, cfg: ("ok", _respond(args, responses).stdout and json.loads(_respond(args, responses).stdout))) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is True - assert observation.outcome == "empty_healthy" - assert observation.workers == [] - assert observation.backend_health[0].name == "herdr" - assert observation.backend_health[0].status == "healthy" - assert observation.backend_health[0].outcome == "empty_healthy" - assert observation.backend_health[0].counts == {"spaces": 0, "workers": 0} - - -def test_fetch_herdr_command_observation_degrades_unmatched_agent(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): { - "result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]} - }, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "w-1", "agent_id": "agent-1", "agent": "Coder"} - ] - } - }, - ("pane", "list"): {"result": {"panes": []}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli, - "_run_herdr", - lambda args, cfg: _respond(args, responses), - ) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is False - assert observation.status == "degraded" - assert observation.outcome == "continuity_unavailable" - assert observation.workers == [] - assert observation.bindings == [] - - -def test_fetch_herdr_snapshot_observation_reports_healthy_non_empty(monkeypatch, tmp_path) -> None: - config = Config(host_id="testhost", herdr_bin="herdr", data_dir=tmp_path / "state") - responses = { - ("workspace", "list"): {"result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "w-1", - "agent_id": "agent-1", - "terminal_id": "terminal-1", - "agent": "Coder", - } - ] - } - }, - ("pane", "list"): { - "result": { - "panes": [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-1", - "agent": "Coder", - } - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - health = observation.backend_health[0] - - assert [space.id for space in observation.spaces] == ["wR9"] - assert [worker.id for worker in observation.workers] == ["w-1"] - assert observation.workers[0].meta["stable_key"].startswith("wsk1_") - assert health.to_dict() == { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": health.observed_at, - "message": "Herdr observation is healthy", - "counts": {"spaces": 1, "workers": 1}, - } - - -def test_fetch_herdr_snapshot_observation_degrades_unmatched_agent(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): { - "result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]} - }, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "w-1", "agent_id": "agent-1", "agent": "Coder"} - ] - } - }, - ("pane", "list"): {"result": {"panes": []}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli, - "_run_herdr", - lambda args, cfg: _respond(args, responses), - ) - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - - assert [space.id for space in observation.spaces] == ["wR9"] - assert observation.workers == [] - assert observation.bindings == [] - assert observation.backend_health[0].status == "degraded" - assert observation.backend_health[0].outcome == "continuity_unavailable" - - -def test_fetch_herdr_snapshot_observation_reports_healthy_empty(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): {"result": {"panes": []}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - health = observation.backend_health[0] - - assert observation.spaces == [] - assert observation.workers == [] - assert health.status == "healthy" - assert health.outcome == "empty_healthy" - assert health.counts == {"spaces": 0, "workers": 0} - - -def test_fetch_herdr_snapshot_observation_reports_missing_binary() -> None: - config = Config(host_id="testhost", herdr_bin="definitely-not-a-real-herdr-binary") - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - health = observation.backend_health[0] - - assert observation.spaces == [] - assert observation.workers == [] - assert health.status == "unavailable" - assert health.outcome == "missing_binary" - - -@pytest.mark.parametrize( - ("probe_outcome", "expected_status", "expected_outcome"), - [ - ("launch_error", "unavailable", "launch_error"), - ("timeout", "degraded", "timeout"), - ("deadline_exhausted", "degraded", "deadline_exhausted"), - ("malformed_json", "degraded", "malformed_json"), - ("nonzero", "degraded", "nonzero"), - ("unknown", "unknown", "unknown"), - ], -) -def test_fetch_herdr_snapshot_observation_maps_failure_outcomes( - monkeypatch, - probe_outcome: str, - expected_status: str, - expected_outcome: str, -) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - - def fake_probe(args: Sequence[str], cfg: Config) -> tuple[str, Any]: - return probe_outcome, None - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - health = observation.backend_health[0] - - assert observation.spaces == [] - assert observation.workers == [] - assert health.status == expected_status - assert health.outcome == expected_outcome - assert health.message - - -def test_herdr_health_mapping_includes_socket_disconnect() -> None: - health = herdr_cli.herdr_backend_health("socket_disconnected") - - assert health.status == "unavailable" - assert health.outcome == "socket_disconnected" - - -def test_cli_snapshot_retains_authenticated_worker_while_pane_probe_recovers( - monkeypatch, - tmp_path, -) -> None: - config = Config( - host_id="pane-recovery", - herdr_bin="herdr", - data_dir=tmp_path / "state", - db_path=tmp_path / "pane-recovery.db", - ) - init_store(config.db_path) - responses = { - ("workspace", "list"): { - "result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]} - }, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-worker", - "agent_id": "private-agent", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "private-terminal", - "agent": "Coder", - } - ] - } - }, - ("pane", "list"): { - "result": { - "panes": [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "private-terminal", - "agent": "Coder", - } - ] - } - }, - } - pane_available = True - - def fake_probe(args: Sequence[str], cfg: Config, *unused: Any) -> tuple[str, Any]: - if tuple(args) == ("pane", "list") and not pane_available: - return "timeout", None - payload = responses.get(tuple(args)) - return ("ok", payload) if payload is not None else ("nonzero", None) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - first = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - first_worker = first.workers[0] - first_binding = list_worker_bindings(config.db_path, config.host_id, backend="herdr") - assert first.backend_health[0].status == "healthy" - assert first_worker.meta["stable_key"].startswith("wsk1_") - - pane_available = False - degraded = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert degraded.workers == first.workers - assert degraded.spaces == first.spaces - assert degraded.backend_health[0].status == "degraded" - assert degraded.backend_health[0].outcome == "timeout" - assert degraded.backend_health[0].counts == {"spaces": 1, "workers": 1} - assert list_worker_bindings(config.db_path, config.host_id, backend="herdr") == first_binding - - pane_available = True - recovered = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert recovered.backend_health[0].status == "healthy" - assert recovered.workers[0].meta["stable_key"] == first_worker.meta["stable_key"] - recovered_binding = list_worker_bindings( - config.db_path, - config.host_id, - backend="herdr", - ) - - healthy_panes = responses[("pane", "list")] - responses[("pane", "list")] = {"result": {"panes": []}} - unmatched = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert unmatched.workers == first.workers - assert unmatched.spaces == first.spaces - assert unmatched.backend_health[0].status == "degraded" - assert unmatched.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings(config.db_path, config.host_id, backend="herdr") - == recovered_binding - ) - - responses[("pane", "list")] = healthy_panes - recovered_again = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert recovered_again.backend_health[0].status == "healthy" - assert recovered_again.workers[0].meta["stable_key"] == first_worker.meta["stable_key"] - - -def test_cli_snapshot_retains_authenticated_worker_while_installation_key_recovers( - monkeypatch, - tmp_path, -) -> None: - config = Config( - host_id="key-recovery", - herdr_bin="herdr", - data_dir=tmp_path / "state", - db_path=tmp_path / "key-recovery.db", - ) - init_store(config.db_path) - responses = { - ("workspace", "list"): { - "result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]} - }, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-worker", - "agent_id": "private-agent", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "private-terminal", - "agent": "Coder", - } - ] - } - }, - ("pane", "list"): { - "result": { - "panes": [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "private-terminal", - "agent": "Coder", - } - ] - } - }, - } - - def fake_probe(args: Sequence[str], cfg: Config, *unused: Any) -> tuple[str, Any]: - payload = responses.get(tuple(args)) - return ("ok", payload) if payload is not None else ("nonzero", None) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - first = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - first_worker = first.workers[0] - first_binding = list_worker_bindings(config.db_path, config.host_id, backend="herdr") - marker = config.installation_key_marker_path.read_bytes() - config.installation_key_marker_path.unlink() - - degraded = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert degraded.workers == first.workers - assert degraded.backend_health[0].status == "degraded" - assert degraded.backend_health[0].outcome == "continuity_unavailable" - assert degraded.backend_health[0].message == "Herdr continuity identity is unavailable" - assert degraded.backend_health[0].counts == {"spaces": 1, "workers": 1} - assert json.loads(degraded.to_json())["backend_health"][0]["outcome"] == "continuity_unavailable" - assert list_worker_bindings(config.db_path, config.host_id, backend="herdr") == first_binding - - degraded_again = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - assert degraded_again.workers == first.workers - assert degraded_again.spaces == first.spaces - assert degraded_again.backend_health[0].outcome == "continuity_unavailable" - assert degraded_again.backend_health[0].counts == {"spaces": 1, "workers": 1} - - def timeout_pane_probe( - args: Sequence[str], - cfg: Config, - *unused: Any, - ) -> tuple[str, Any]: - if tuple(args) == ("pane", "list"): - return "timeout", None - return fake_probe(args, cfg, *unused) - - monkeypatch.setattr(herdr_cli, "_probe_herdr", timeout_pane_probe) - alternate_failure = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - assert alternate_failure.workers == first.workers - assert alternate_failure.backend_health[0].outcome == "continuity_unavailable" - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - config.installation_key_marker_path.write_bytes(marker) - os.chmod(config.installation_key_marker_path, 0o600) - recovered = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert recovered.backend_health[0].status == "healthy" - assert recovered.workers[0].meta["stable_key"] == first_worker.meta["stable_key"] - - -def test_probe_payload_variants_stops_after_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[str, ...]] = [] - - def fake_probe(args: Sequence[str], cfg: Config) -> tuple[str, Any]: - calls.append(tuple(args)) - return "timeout", None - - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - outcome, payload = herdr_cli._probe_payload_variants( - [["agent", "list"], ["agent", "list", "--json"]], - config, - ) - - assert outcome == "timeout" - assert payload is None - assert calls == [("agent", "list")] - - -def test_fetch_herdr_command_observation_stops_fallbacks_after_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[str, ...]] = [] - - def fake_probe(args: Sequence[str], cfg: Config) -> tuple[str, Any]: - calls.append(tuple(args)) - if tuple(args) == ("workspace", "list"): - return "ok", {"result": {"workspaces": []}} - return "timeout", None - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is False - assert observation.outcome == "timeout" - assert observation.workers == [] - assert calls == [("workspace", "list"), ("agent", "list")] - - -def test_fetch_herdr_state_short_circuits_after_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args)) - raise subprocess.TimeoutExpired(cmd=["herdr", *args], timeout=config.herdr_timeout_seconds) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert spaces == [] - assert workers == [] - assert calls == [("workspace", "list")] - - -def test_fetch_herdr_state_returns_spaces_and_skips_worker_fallback_after_agent_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[str, ...]] = [] - - responses = { - ("workspace", "list"): {"result": {"workspaces": [{"workspace_id": "ws-1", "label": "Build"}]}}, - } - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args)) - if tuple(args) == ("workspace", "list"): - return _respond(args, responses) - raise subprocess.TimeoutExpired(cmd=["herdr", *args], timeout=config.herdr_timeout_seconds) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert [space.id for space in spaces] == ["ws-1"] - assert workers == [] - assert calls == [("workspace", "list"), ("agent", "list")] - - -def test_fetch_herdr_state_uses_aggregate_deadline_for_remaining_probe_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr", herdr_timeout_seconds=1.0) - current_time = [0.0] - calls: list[tuple[tuple[str, ...], float]] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append((tuple(args[1:]), float(kwargs["timeout"]))) - if len(calls) == 1: - current_time[0] += 4.75 - else: - current_time[0] += 0.30 - return subprocess.CompletedProcess(args=args, returncode=2, stdout="", stderr="") - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.time, "monotonic", lambda: current_time[0]) - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert spaces == [] - assert workers == [] - assert calls[0] == (("workspace", "list"), 1.0) - assert calls[1][0] == ("workspace", "list", "--json") - assert 0 < calls[1][1] <= 0.25 - assert len(calls) == 2 - - -def test_fetch_herdr_command_observation_stops_after_aggregate_deadline(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr", herdr_timeout_seconds=1.0) - current_time = [0.0] - calls: list[tuple[str, ...]] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args[1:])) - current_time[0] += 5.1 - return subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps({"result": {"workspaces": []}}), - stderr="", - ) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.time, "monotonic", lambda: current_time[0]) - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is False - assert observation.status == "degraded" - assert observation.outcome == "deadline_exhausted" - assert calls == [("workspace", "list")] - - -@pytest.mark.parametrize("outcome", ["timeout", "malformed_json", "nonzero"]) -def test_fetch_herdr_command_observation_degraded_agent_probe(monkeypatch, outcome: str) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - - def fake_probe(args: Sequence[str], cfg: Config) -> tuple[str, Any]: - if tuple(args) == ("workspace", "list"): - return "ok", {"result": {"workspaces": []}} - if tuple(args) == ("workspace", "list", "--json"): - return "ok", {"result": {"workspaces": []}} - return outcome, None - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is False - assert observation.status == "degraded" - assert observation.outcome == outcome - assert observation.workers == [] - - -def test_no_flag_workspace_and_agent_lists_preferred_without_json_calls(monkeypatch) -> None: - """Herdr 0.7.0 no-flag envelopes are used before compatibility --json fallbacks.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): { - "result": { - "workspaces": [ - { - "workspace_id": "ws-1", - "label": "Build", - "agent_status": "working", - "active_tab_id": "tab-1", - } - ] - } - }, - ("agent", "list"): { - "result": { - "agents": [ - { - "agent_session": {"value": "sess-1"}, - "agent": "Coder", - "workspace_id": "ws-1", - "agent_status": "done", - "cwd": "/home/dev", - } - ] - } - }, - ("pane", "list"): {"result": {"panes": []}}, - } - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str] | None: - calls.append(tuple(args)) - if "--json" in args: - raise AssertionError("--json fallback should not be called after valid no-flag output") - return _respond(args, responses) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert calls == [("workspace", "list"), ("agent", "list"), ("pane", "list")] - assert len(spaces) == 1 - assert spaces[0].id == "ws-1" - assert spaces[0].name == "Build" - assert spaces[0].status == "active" - assert spaces[0].meta.get("active_tab_id") == "tab-1" - assert spaces[0].meta.get("raw_status") == "working" - assert len(workers) == 1 - assert workers[0].id == "Coder" - assert workers[0].name == "Coder" - assert workers[0].status == "done" - assert workers[0].space_id == "ws-1" - assert workers[0].backend_target is not None - assert workers[0].backend_target["kind"] == "agent" - assert workers[0].backend_target["value"] == "Coder" - assert workers[0].backend_target["sendable"] is True - assert "cwd" not in workers[0].meta - assert "/home/dev" not in json.dumps(workers[0].to_dict()) - assert "raw_status" not in workers[0].meta - - -def test_json_workspace_list_fallback_when_no_flag_fails(monkeypatch) -> None: - """Compatibility --json workspace list is tried when no-flag output fails.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): _completed("usage", returncode=2), - ("workspace", "list", "--json"): { - "result": { - "workspaces": [{"workspace_id": "ws-json", "label": "Compat", "agent_status": "idle"}] - } - }, - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): {"result": {"panes": []}}, - } - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str] | None: - calls.append(tuple(args)) - return _respond(args, responses) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert calls[:2] == [("workspace", "list"), ("workspace", "list", "--json")] - assert len(spaces) == 1 - assert spaces[0].id == "ws-json" - assert spaces[0].name == "Compat" - assert workers == [] - - -def test_json_agent_list_fallback_when_no_flag_is_malformed(monkeypatch) -> None: - """Compatibility --json agent list is tried when no-flag output is malformed.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): _completed("not json"), - ("agent", "list", "--json"): { - "result": { - "agents": [ - { - "agent_session": {"value": "sess-json"}, - "agent": "CompatAgent", - "workspace_id": "ws-1", - } - ] - } - }, - ("pane", "list"): {"result": {"panes": []}}, - } - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str] | None: - calls.append(tuple(args)) - return _respond(args, responses) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("agent", "list", "--json"), - ("pane", "list"), - ] - assert spaces == [] - assert len(workers) == 1 - assert workers[0].id == "CompatAgent" - assert workers[0].name == "CompatAgent" - assert workers[0].space_id == "ws-1" - - -def test_result_envelopes_parse(monkeypatch) -> None: - """result.workspaces, result.agents, and result.panes envelopes parse.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): { - "result": { - "workspaces": [{"workspace_id": "ws-result", "label": "ResultSpace", "agent_status": "idle"}] - } - }, - ("agent", "list", "--json"): { - "result": {"agents": [{"agent_session": {"value": "sess-result"}, "agent": "Agent"}]} - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(spaces) == 1 - assert spaces[0].id == "ws-result" - assert len(workers) == 1 - assert workers[0].id == "Agent" - - -def test_pane_fallback_only_when_agent_list_yields_none(monkeypatch) -> None: - """Pane list fallback runs only when agents are empty and only for agent-bearing panes.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): _completed("", returncode=1), - ("workspace", "list"): _completed("", returncode=1), - ("agent", "list", "--json"): _completed("", returncode=1), - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): { - "result": { - "panes": [ - {"pane_id": "pane-agent", "agent": "Runner", "workspace_id": "ws-1"}, - {"pane_id": "pane-plain", "workspace_id": "ws-1"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(workers) == 1 - assert workers[0].id == "Runner" - assert workers[0].name == "Runner" - assert workers[0].backend_target is not None - assert workers[0].backend_target["kind"] == "pane_id" - assert workers[0].backend_target["value"] == "pane-agent" - assert workers[0].backend_target["sendable"] is True - - -def test_pane_list_enriches_without_adding_unmatched_panes_when_agents_present(monkeypatch) -> None: - """Pane list enriches matching agents without projecting unmatched fallback panes.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): _completed("", returncode=1), - ("workspace", "list"): _completed("", returncode=1), - ("agent", "list", "--json"): { - "result": {"agents": [{"agent_session": {"value": "sess-1"}, "agent": "Agent"}]} - }, - ("pane", "list"): {"result": {"panes": [{"pane_id": "sess-1", "agent": "Agent"}]}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(workers) == 1 - assert workers[0].id == "Agent" - - -def test_agent_and_pane_duplicates_emit_one_worker(monkeypatch) -> None: - """A worker described by both agent and pane payloads is deduplicated.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): _completed("", returncode=1), - ("workspace", "list"): _completed("", returncode=1), - ("agent", "list", "--json"): _completed("", returncode=1), - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): { - "result": { - "panes": [ - {"pane_id": "pane-1", "agent": "Runner", "workspace_id": "ws-1"}, - {"pane_id": "pane-1", "agent": "Runner", "workspace_id": "ws-1"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(workers) == 1 - assert workers[0].id == "Runner" - - -def test_repeated_agent_names_with_distinct_ids_remain_distinct(monkeypatch) -> None: - """Workers with the same display name but different session ids are kept.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): _completed("", returncode=1), - ("workspace", "list"): _completed("", returncode=1), - ("agent", "list", "--json"): { - "result": { - "agents": [ - {"agent_session": {"value": "sess-a"}, "agent": "Coder", "workspace_id": "ws-1"}, - {"agent_session": {"value": "sess-b"}, "agent": "Coder", "workspace_id": "ws-1"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(workers) == 2 - assert {w.id for w in workers} == {"Coder-1", "Coder-2"} - assert workers[0].id < workers[1].id - assert all((w.backend_target or {}).get("kind") == "agent" for w in workers) - assert all((w.backend_target or {}).get("value") == "Coder" for w in workers) - assert all((w.backend_target or {}).get("sendable") is False for w in workers) - assert all((w.backend_target or {}).get("reason") == "duplicate_backend_target" for w in workers) - - -def test_status_aliases_for_live_herdr(monkeypatch) -> None: - """Working maps to active, done maps to done, and raw_status is preserved.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): { - "result": { - "workspaces": [ - {"workspace_id": "ws-1", "label": "Space", "agent_status": "working"}, - {"workspace_id": "ws-2", "label": "Space2", "agent_status": "responding"}, - ] - } - }, - ("agent", "list", "--json"): { - "result": { - "agents": [ - {"agent_session": {"value": "sess-1"}, "agent": "Agent", "workspace_id": "ws-1", "agent_status": "done"}, - {"agent_session": {"value": "sess-2"}, "agent": "Agent", "workspace_id": "ws-2", "agent_status": "awaiting-input"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert spaces[0].status == "active" - assert spaces[0].meta.get("raw_status") == "working" - assert spaces[1].status == "waiting" - assert spaces[1].meta.get("raw_status") == "responding" - by_space = {w.space_id: w for w in workers} - assert by_space["ws-1"].status == "done" - assert "raw_status" not in by_space["ws-1"].meta - assert by_space["ws-2"].status == "waiting" - assert by_space["ws-2"].meta.get("raw_status") == "awaiting-input" - - -def test_forbidden_connector_fields_stripped_in_herdr_070(monkeypatch) -> None: - """Connector/delivery fields are stripped from live Herdr 0.7.0 payloads.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): { - "result": { - "workspaces": [ - { - "workspace_id": "ws-1", - "label": "Space", - "agent_status": "idle", - "telegram": "leaked", - "chat_id": 123, - "bot.token": "leaked", - "message.id": "leaked", - "backend.target": "leaked", - } - ] - } - }, - ("agent", "list", "--json"): { - "result": { - "agents": [ - { - "agent_session": {"value": "sess-1"}, - "agent": "Agent", - "workspace_id": "ws-1", - "agent_status": "active", - "route": "telegram", - "delivery": {"topic_id": 456}, - "herdres_delivery": {"message_id": 789}, - "herdres.delivery": {"message.id": 789}, - "delivery.route": "telegram", - "telegram.message.id": "leaked", - } - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - snapshot = project_from_observations(config, spaces=spaces, workers=workers) - payload = json.loads(snapshot.to_json()) - - _assert_no_forbidden_fields(payload) - assert "telegram" not in payload["spaces"][0]["meta"] - assert "route" not in payload["workers"][0]["meta"] diff --git a/tests/test_cli.py b/tests/test_cli.py index d433c23..06c1199 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,3224 +1,23 @@ -"""Tests for tendwire CLI snapshot JSON output and optional storage.""" - from __future__ import annotations -import io import json -import os -import sqlite3 -import subprocess -import sys -import threading -import time -from datetime import datetime -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest - -from tendwire.backends import herdr_cli -from tendwire.cli import _build_parser, main, observe_public_snapshot -from tendwire.config import DEFAULT_TURN_MODEL, Config -from tendwire.core.models import AttentionSignal, Snapshot, SuggestedAction, Worker, WorkerBinding -from tendwire.core.projector import project_from_raw -from tendwire.daemon_api import TendwireDaemonAPI, UnixSocketJSONServer -from tendwire.store.sqlite import ( - SnapshotObservationContext, - append_event, - get_turn_content, - init_store, - latest_snapshot, - list_worker_bindings, - merge_backend_pending, - pending_payload_from_store, - merge_turn_content, - save_snapshot, - turns_payload_from_store, -) - - -@pytest.fixture(autouse=True) -def _isolate_cli_paths(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - private_home = tmp_path / "home" - private_home.mkdir(mode=0o700) - monkeypatch.setenv("HOME", str(private_home)) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "tendwire-data")) - monkeypatch.delenv("TENDWIRE_DB_PATH", raising=False) - - -_PUBLIC_JSON_FORBIDDEN_KEYS = { - "tty", - "pty", - "pid", - "pids", - "process_id", - "process_ids", - "tmux", - "tmux_session", - "tmux_sessions", - "screen_session", - "screen_sessions", - "window_id", - "window_ids", - "tab_id", - "tab_ids", - "pane_id", - "pane_ids", - "terminal_id", - "terminal_ids", - "backend_target", - "backend_targets", - "session_id", - "private", - "private_binding", - "private_bindings", - "private_fingerprint", - "private_fingerprints", - "route", - "routes", - "delivery", - "deliveries", - "connector", - "connectors", - "command", - "command_args", - "command_argv", - "command_line", - "command_payload", - "command_text", - "raw_args", - "raw_argv", - "raw_command", - "raw_command_line", - "shell_command", - "chat_id", - "chat_ids", - "topic_id", - "topic_ids", - "message_id", - "message_ids", - "token", - "tokens", - "secret", - "secrets", - "password", - "passwords", - "credentials", - "cookie", - "auth_token", - "auth_tokens", -} -_PUBLIC_JSON_FORBIDDEN_COMPACT = { - key.replace("_", "") for key in _PUBLIC_JSON_FORBIDDEN_KEYS -} - - -def _assert_no_public_json_forbidden(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - normalized = str(key).lower().replace("-", "_") - assert ( - normalized not in _PUBLIC_JSON_FORBIDDEN_KEYS - and normalized.replace("_", "") not in _PUBLIC_JSON_FORBIDDEN_COMPACT - ), f"forbidden field {path}.{key}" - _assert_no_public_json_forbidden(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_public_json_forbidden(item, f"{path}[{index}]") - -def test_cli_snapshot_json_prints_contract_json_only(capsys) -> None: - code = main( - [ - "--host-id", - "cli-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--json", - ] - ) - captured = capsys.readouterr() +from tendwire.cli import main - assert code == 0 - assert captured.err == "" - payload = json.loads(captured.out) - assert payload["schema_version"] == 2 - assert payload["host_id"] == "cli-host" - assert len(payload["content_fingerprint"]) == 24 - assert {"updated_at", "spaces", "workers", "attention", "backend_health"} <= set(payload) - assert payload["backend_health"][0]["name"] == "herdr" - assert payload["backend_health"][0]["status"] == "unavailable" - assert payload["backend_health"][0]["outcome"] == "missing_binary" - -def test_cli_snapshot_no_herdr_works() -> None: - """Empty snapshot works even when herdr is not installed.""" - code = main(["--herdr-bin", "definitely-not-a-real-herdr-binary", "snapshot", "--json"]) - assert code == 0 - - -def test_cli_snapshot_post_send_timeout_never_observes_source( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - class TimeoutClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, _method: str, _params: dict[str, Any] | None = None) -> dict[str, Any]: - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("post-send timeout must not observe Herdr or mutate the store") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", TimeoutClient) - monkeypatch.setattr("tendwire.cli.observe_public_snapshot", forbidden) - - code = main( - [ - "--socket-path", - str(tmp_path / "daemon.sock"), - "snapshot", - "--json", - ] - ) +def test_snapshot_uses_daemon_and_never_reads_store(capsys, monkeypatch, tmp_path) -> None: + monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path)) + code = main(["snapshot", "--json"]) payload = json.loads(capsys.readouterr().out) - - assert code == 1 - assert payload == { - "schema_version": 2, - "ok": False, - "status": "daemon_timeout", - "error": { - "code": "daemon_timeout", - "message": "Tendwire daemon request timed out", - }, - } - - -def test_cli_socket_group_option_is_daemon_only_and_normalized(monkeypatch) -> None: - captured: list[Config] = [] - - def capture_daemon_config(config: Config) -> int: - captured.append(config) - return 0 - - monkeypatch.delenv("TENDWIRE_SOCKET_GROUP", raising=False) - monkeypatch.setattr("tendwire.cli.cmd_daemon", capture_daemon_config) - - snapshot_args = _build_parser().parse_args(["snapshot"]) - assert not hasattr(snapshot_args, "socket_group") - assert main(["daemon", "--socket-group", " daemon-clients "]) == 0 - assert captured[0].socket_group == "daemon-clients" - - -def test_cli_daemon_startup_conflict_is_clear_and_nonzero( - tmp_path: Path, - capsys, - monkeypatch: pytest.MonkeyPatch, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - def active_socket(_config: Config) -> int: - raise DaemonUnavailable( - "daemon socket is already active: holder is tendwire 0.1.0rc4 " - "(PID 4242); refusing to start tendwire 0.1.0rc5" - ) - - monkeypatch.setattr("tendwire.daemon.run_daemon", active_socket) - - code = main(["daemon", "--db-path", str(tmp_path / "daemon.db")]) - captured = capsys.readouterr() - - assert code == 1 - assert captured.out == "" - assert captured.err == ( - "tendwire daemon 0.1.0rc5: startup failed: " - "daemon socket is already active: holder is tendwire 0.1.0rc4 " - "(PID 4242); refusing to start tendwire 0.1.0rc5\n" - ) - - -def test_cli_turns_json_without_cached_store_is_publicly_unavailable(capsys) -> None: - code = main( - [ - "--host-id", - "turns-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload == { - "schema_version": 1, - "host_id": "turns-host", - "ok": False, - "status": "store_unavailable", - } - - -def test_cli_turns_schema_v2_daemon_request_requires_no_content_fetch( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - calls: list[tuple[str, dict[str, Any]]] = [] - - class FakeDaemonAPIClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - calls.append((method, dict(params or {}))) - return { - "ok": True, - "result": { - "schema_version": 2, - "host_id": "turns-host", - "turns": [ - { - "id": "turn-public", - "assistant_final_text": "short final", - "content": { - "schema_version": 1, - "content_revision": "twrev1.public", - "known_incomplete": False, - "fields": { - "assistant_final_text": { - "availability": "complete", - "inline": True, - "char_length": 11, - "byte_length": 11, - "page_count": 1, - "first_cursor": None, - } - }, - }, - } - ], - }, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - code = main( - [ - "--host-id", - "turns-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--schema-version", - "2", - "--json", - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 0 - assert payload["schema_version"] == 2 - assert payload["turns"][0]["assistant_final_text"] == "short final" - assert payload["turns"][0]["content"]["fields"]["assistant_final_text"]["inline"] is True - assert calls == [ - ( - "turn.list", - { - "schema_version": 2, - "limit": 100, - "cursor": None, - "since": None, - }, - ) - ] - - -def test_cli_turns_v1_upgrade_required_is_json_and_nonzero( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - class FakeDaemonAPIClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - assert method == "turn.list" - assert params == { - "schema_version": 1, - "limit": 100, - "cursor": None, - "since": None, - } - return { - "ok": True, - "result": { - "schema_version": 1, - "ok": False, - "status": "upgrade_required", - "required_turn_schema_version": 2, - "error": { - "code": "upgrade_required", - "message": "turn content requires schema version 2", - }, - }, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - code = main( - [ - "--host-id", - "turns-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["status"] == "upgrade_required" - assert payload["required_turn_schema_version"] == 2 - assert payload["error"]["code"] == "upgrade_required" - - -def test_cli_turns_parser_defaults_bounds_and_exclusive_positions() -> None: - parser = _build_parser() - - defaults = parser.parse_args(["turns"]) - assert defaults.limit == 100 - assert defaults.cursor is None - assert defaults.since is None - - bounded = parser.parse_args(["turns", "--limit", "250", "--cursor", "twlist1.page"]) - assert bounded.limit == 250 - assert bounded.cursor == "twlist1.page" - assert bounded.since is None - - for invalid_limit in ("0", "251", "1.5"): - with pytest.raises(SystemExit): - parser.parse_args(["turns", "--limit", invalid_limit]) - with pytest.raises(SystemExit): - parser.parse_args( - ["turns", "--cursor", "twlist1.page", "--since", "twsince1.new"] - ) - - -def test_cli_turns_definite_unavailable_reads_durable_page_without_refresh( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - daemon_calls: list[tuple[str, dict[str, Any]]] = [] - refresh_calls: list[dict[str, Any]] = [] - store_calls: list[tuple[Any, ...]] = [] - - class UnavailableClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - daemon_calls.append((method, dict(params or {}))) - raise DaemonUnavailable("not listening", request_started=False) - - def refresh(_config: Config, **kwargs: Any) -> dict[str, Any]: - refresh_calls.append(kwargs) - return {"ok": True, "status": "ok", "updated": 1, "attempted": 1} - - def read_page(db_path: Path, host_id: str, **kwargs: Any) -> dict[str, Any]: - store_calls.append((db_path, host_id, kwargs)) - return { - "schema_version": 2, - "host_id": host_id, - "ok": True, - "status": "ok", - "turns": [{"id": "cached-turn"}], - "next_cursor": "twlist1.next", - "since": "twsince1.done", - } - - def forbidden_snapshot(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("turn fallback must not observe a snapshot") - - db_path = tmp_path / "fallback.db" - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", UnavailableClient) - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", refresh, raising=False - ) - monkeypatch.setattr("tendwire.cli.turns_payload_from_store", read_page) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden_snapshot) - - code = main( - [ - "--host-id", - "fallback-host", - "--herdr-timeout", - "0.75", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--schema-version", - "2", - "--limit", - "7", - "--db-path", - str(db_path), - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 0 - assert payload["turns"] == [{"id": "cached-turn"}] - assert daemon_calls == [ - ( - "turn.list", - {"schema_version": 2, "limit": 7, "cursor": None, "since": None}, - ) - ] - assert refresh_calls == [] - assert store_calls == [ - ( - db_path, - "fallback-host", - { - "schema_version": 2, - "limit": 7, - "cursor": None, - "since": None, - "turn_model": DEFAULT_TURN_MODEL, - }, - ) - ] - - -@pytest.mark.parametrize( - ("position_flag", "position_value"), - [ - ("--cursor", "twlist1.page"), - ("--since", "twsince1.done"), - ], -) -def test_cli_turns_continuation_unavailable_reads_cache_without_refresh( - tmp_path: Path, - capsys, - monkeypatch, - position_flag: str, - position_value: str, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - store_calls: list[dict[str, Any]] = [] - - class UnavailableClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, _method: str, _params: dict[str, Any] | None = None) -> dict[str, Any]: - raise DaemonUnavailable("not listening", request_started=False) - - def forbidden_refresh(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("continuation must never refresh") - - def read_page(_db_path: Path, host_id: str, **kwargs: Any) -> dict[str, Any]: - store_calls.append(kwargs) - return { - "schema_version": 2, - "host_id": host_id, - "ok": True, - "status": "ok", - "turns": [], - "next_cursor": None, - "since": "twsince1.next", - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", UnavailableClient) - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", - forbidden_refresh, - raising=False, - ) - monkeypatch.setattr("tendwire.cli.turns_payload_from_store", read_page) - - code = main( - [ - "--host-id", - "continuation-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--schema-version", - "2", - "--limit", - "9", - position_flag, - position_value, - "--db-path", - str(tmp_path / "cache.db"), - ] - ) - json.loads(capsys.readouterr().out) - - assert code == 0 - assert store_calls == [ - { - "schema_version": 2, - "limit": 9, - "cursor": position_value if position_flag == "--cursor" else None, - "since": position_value if position_flag == "--since" else None, - "turn_model": DEFAULT_TURN_MODEL, - } - ] - - -@pytest.mark.parametrize( - ("mode", "expected"), - [ - ( - "timeout", - { - "schema_version": 1, - "ok": False, - "status": "daemon_timeout", - "error": { - "code": "daemon_timeout", - "message": "Tendwire daemon request timed out", - }, - }, - ), - ( - "protocol", - { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", - }, - }, - ), - ( - "malformed", - { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", - }, - }, - ), - ( - "daemon_error", - { - "schema_version": 1, - "ok": False, - "status": "error", - "result": None, - "error": { - "code": "invalid_params", - "message": "invalid parameters", - }, - }, - ), - ], -) -def test_cli_turns_reachable_or_ambiguous_failure_never_reads_sources( - tmp_path: Path, - capsys, - monkeypatch, - mode: str, - expected: dict[str, Any], -) -> None: - from tendwire.daemon_api import DaemonProtocolError, DaemonUnavailable - - calls = 0 - - class FailingClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "turn.list" - assert params == { - "schema_version": 1, - "limit": 100, - "cursor": None, - "since": None, - } - if mode == "timeout": - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) - if mode == "protocol": - raise DaemonProtocolError("invalid frame", request_started=True) - if mode == "malformed": - return {"ok": True, "result": ["not", "a", "mapping"]} - return expected - - def forbidden_read(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("ambiguous/reachable failures must not read any source") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FailingClient) - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", forbidden_read, raising=False - ) - monkeypatch.setattr("tendwire.cli.turns_payload_from_store", forbidden_read) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden_read) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden_read) - - code = main( - [ - "--host-id", - "failure-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--db-path", - str(tmp_path / "cache.db"), - ] - ) - captured = capsys.readouterr() - - assert code == 1 - assert captured.err == "" - assert json.loads(captured.out) == expected - assert calls == 1 - - -@pytest.mark.parametrize("status", ["invalid_cursor", "cursor_expired", "since_expired"]) -def test_cli_turns_reachable_invalid_or_expired_page_is_authoritative( - tmp_path: Path, - capsys, - monkeypatch, - status: str, -) -> None: - result = { - "schema_version": 2, - "ok": False, - "status": status, - } - - class AuthoritativeClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - assert method == "turn.list" - assert params == { - "schema_version": 2, - "limit": 11, - "cursor": "twlist1.requested", - "since": None, - } - return {"ok": True, "result": result} - - def forbidden_read(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("reachable page result must be authoritative") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", AuthoritativeClient) - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", forbidden_read, raising=False - ) - monkeypatch.setattr("tendwire.cli.turns_payload_from_store", forbidden_read) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden_read) - - code = main( - [ - "--host-id", - "authoritative-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--schema-version", - "2", - "--limit", - "11", - "--cursor", - "twlist1.requested", - "--db-path", - str(tmp_path / "cache.db"), - ] - ) - assert code == 1 - assert json.loads(capsys.readouterr().out) == result - - -def test_cli_turns_traverses_over_one_mib_across_bounded_daemon_pages( - tmp_path: Path, - capsys, -) -> None: - socket_path = tmp_path / "paged-turns.sock" - snapshot = Snapshot(host_id="paged-host", updated_at="2026-01-01T00:00:00+00:00") - canonical_text = { - f"turn-{index:03d}": ( - f"\n# Turn {index:03d}\n" - + (f"exact-{index:03d}-αβγ\n" * 650) - + "終\n" - ) - for index in range(110) - } - ordered_ids = list(canonical_text) - seen_requests: list[dict[str, Any]] = [] - - def get_turns( - *, - schema_version: int, - limit: int, - cursor: str | None, - since: str | None, - ) -> dict[str, Any]: - seen_requests.append( - { - "schema_version": schema_version, - "limit": limit, - "cursor": cursor, - "since": since, - } - ) - start = 0 if cursor is None else 55 - page_ids = ordered_ids[start : start + 55] - return { - "schema_version": 2, - "host_id": "paged-host", - "ok": True, - "status": "ok", - "turns": [ - { - "id": turn_id, - "assistant_final_text": canonical_text[turn_id], - "content": { - "schema_version": 1, - "content_revision": f"twrev1.{turn_id}", - "known_incomplete": False, - "fields": { - "assistant_final_text": { - "availability": "complete", - "inline": True, - "char_length": len(canonical_text[turn_id]), - "byte_length": len(canonical_text[turn_id].encode("utf-8")), - "page_count": 1, - "first_cursor": None, - } - }, - }, - } - for turn_id in page_ids - ], - "next_cursor": "twlist1.second" if start == 0 else None, - "since": "twsince1.complete" if start else None, - } - - api = TendwireDaemonAPI( - get_snapshot=lambda: snapshot, - get_health=lambda: {"schema_version": 1, "status": "ok"}, - submit_command=lambda _params: {}, - get_turns=get_turns, - ) - server = UnixSocketJSONServer( - socket_path, - api.dispatch, - accept_timeout_seconds=0.05, - ) - thread = threading.Thread(target=server.serve_forever) - thread.start() - pages: list[dict[str, Any]] = [] - encoded_sizes: list[int] = [] - cursor: str | None = None - try: - deadline = time.monotonic() + 2 - while not server.listening and time.monotonic() < deadline: - time.sleep(0.01) - while True: - argv = [ - "--host-id", - "paged-host", - "--socket-path", - str(socket_path), - "turns", - "--schema-version", - "2", - "--limit", - "55", - ] - if cursor is not None: - argv += ["--cursor", cursor] - assert main(argv) == 0 - captured = capsys.readouterr() - assert captured.err == "" - encoded = captured.out.encode("utf-8") - assert len(encoded) < 1024 * 1024 - encoded_sizes.append(len(encoded)) - page = json.loads(captured.out) - pages.append(page) - cursor = page["next_cursor"] - if cursor is None: - break - finally: - server.close() - thread.join(timeout=2) - - listed = [turn for page in pages for turn in page["turns"]] - assert sum(encoded_sizes) > 1024 * 1024 - assert [turn["id"] for turn in listed] == ordered_ids - assert { - turn["id"]: turn["assistant_final_text"] for turn in listed - } == canonical_text - assert seen_requests == [ - { - "schema_version": 2, - "limit": 55, - "cursor": None, - "since": None, - }, - { - "schema_version": 2, - "limit": 55, - "cursor": "twlist1.second", - "since": None, - }, - ] - assert not thread.is_alive() - - -def test_cli_turn_content_get_preserves_exact_page_and_params( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - page_text = "\n " + ("界" * 20_000) + "\r\n " - revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" - calls: list[tuple[str, dict[str, Any]]] = [] - - class FakeDaemonAPIClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - calls.append((method, dict(params or {}))) - return { - "ok": True, - "result": { - "schema_version": 1, - "ok": True, - "status": "ok", - "turn_id": "turn-public", - "content_revision": revision, - "field": "assistant_final_text", - "availability": "complete", - "segment_id": "twseg1.public", - "index": 1, - "count": 2, - "text": page_text, - "segment_char_length": len(page_text), - "segment_byte_length": len(page_text.encode("utf-8")), - "total_char_length": 40_000, - "total_byte_length": 120_000, - "next_cursor": None, - }, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - code = main( - [ - "--host-id", - "turns-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turn", - "content", - "get", - "--json", - "--turn-id", - "turn-public", - "--revision", - revision, - "--field", - "assistant_final_text", - "--cursor", - "twcur1.public", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert captured.err == "" - assert payload["turn_id"] == "turn-public" - assert payload["content_revision"] == revision - assert payload["segment_id"] == "twseg1.public" - assert payload["text"] == page_text - assert calls == [ - ( - "turn.content.get", - { - "schema_version": 1, - "turn_id": "turn-public", - "content_revision": revision, - "field": "assistant_final_text", - "cursor": "twcur1.public", - }, - ) - ] - - -@pytest.mark.parametrize("with_db_path", [False, True]) -@pytest.mark.parametrize( - ("error_code", "details"), - [ - ("internal_error", {"type": "RuntimeError"}), - ("response_too_large", {"max_response_bytes": 1024 * 1024}), - ], -) -def test_cli_turn_content_preserves_reachable_daemon_errors_without_store_fallback( - tmp_path: Path, - capsys, - monkeypatch, - with_db_path: bool, - error_code: str, - details: dict[str, Any], -) -> None: - direct_calls: list[str] = [] - original_error = { - "code": error_code, - "message": f"daemon {error_code}", - "details": details, - } - - class FakeDaemonAPIClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - assert method == "turn.content.get" - return { - "schema_version": 1, - "ok": False, - "status": "error", - "result": None, - "error": original_error, - } - - def forbidden_store_call(*_args: Any, **_kwargs: Any) -> dict[str, Any]: - direct_calls.append("store") - raise AssertionError("reachable daemon errors must not fall back to the store") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - monkeypatch.setattr("tendwire.store.sqlite.init_store", forbidden_store_call) - monkeypatch.setattr("tendwire.store.sqlite.get_turn_content", forbidden_store_call) - argv = [ - "--host-id", - "turns-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turn", - "content", - "get", - "--json", - "--turn-id", - "turn-public", - "--revision", - "twrev1.public", - "--field", - "assistant_final_text", - ] - if with_db_path: - argv += ["--db-path", str(tmp_path / "direct.db")] - - code = main(argv) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["status"] == "error" - assert payload["error"] == original_error - assert direct_calls == [] - - -def test_cli_long_content_pages_match_direct_store_and_daemon( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "long-content.db" - socket_path = tmp_path / "long-content.sock" - config = Config(host_id="long-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "Worker", "status": "active"}], - ) - canonical = ( - "# Exact heading\n\n" - + ("safe-value αβγ\n- nested-looking item\n```text\ncode\n```\n" * 30_000) - )[:1_100_000] + "終" - init_store(db_path) - save_snapshot(db_path, snapshot) - assert merge_turn_content( - db_path, - "long-host", - "worker-1", - { - "source_turn_id": "cli-long-content-source", - "user_text": "short prompt", - "assistant_final_text": canonical, - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:00+00:00", - ) == 1 - listed = turns_payload_from_store( - db_path, - "long-host", - snapshot=snapshot, - schema_version=2, - ) - turn = listed["turns"][0] - revision = turn["content"]["content_revision"] - descriptor = turn["content"]["fields"]["assistant_final_text"] - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", - lambda _config, **_kwargs: {"ok": True}, - raising=False, - ) + assert payload["status"] == "daemon_unavailable" + assert not (tmp_path / "tendwire.db").exists() - v1_code = main( - [ - "--host-id", - "long-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--db-path", - str(db_path), - "--json", - ] - ) - v1_payload = json.loads(capsys.readouterr().out) - v2_code = main( - [ - "--host-id", - "long-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--db-path", - str(db_path), - "--schema-version", - "2", - "--json", - ] - ) - v2_payload = json.loads(capsys.readouterr().out) - assert v1_code == 1 - assert v1_payload["status"] == "upgrade_required" - assert v1_payload["required_turn_schema_version"] == 2 - assert v2_code == 0 - assert v2_payload["schema_version"] == 2 - assert descriptor["inline"] is False - assert descriptor["char_length"] == len(canonical) - assert descriptor["byte_length"] == len(canonical.encode("utf-8")) - assert descriptor["page_count"] > 1 - - def fetch_pages(*, socket: Path | None, direct_db: Path | None) -> list[dict[str, Any]]: - pages: list[dict[str, Any]] = [] - cursor: str | None = None - while True: - argv = ["--host-id", "long-host"] - if socket is not None: - argv += ["--socket-path", str(socket)] - argv += [ - "turn", - "content", - "get", - "--json", - "--turn-id", - turn["id"], - "--revision", - revision, - "--field", - "assistant_final_text", - ] - if direct_db is not None: - argv += ["--db-path", str(direct_db)] - if cursor is not None: - argv += ["--cursor", cursor] - assert main(argv) == 0 - captured = capsys.readouterr() - assert captured.err == "" - page = json.loads(captured.out) - assert len(json.dumps(page, ensure_ascii=False).encode("utf-8")) < 1024 * 1024 - pages.append(page) - next_cursor = page["next_cursor"] - if next_cursor is None: - return pages - assert next_cursor not in {item.get("next_cursor") for item in pages[:-1]} - cursor = next_cursor - - direct_pages = fetch_pages(socket=None, direct_db=db_path) - bad_cursor_code = main( - [ - "--host-id", - "long-host", - "turn", - "content", - "get", - "--json", - "--turn-id", - turn["id"], - "--revision", - revision, - "--field", - "assistant_final_text", - "--cursor", - "twcur1.tampered", - "--db-path", - str(db_path), - ] - ) - bad_cursor_payload = json.loads(capsys.readouterr().out) - assert bad_cursor_code == 1 - assert bad_cursor_payload["status"] == "invalid_cursor" - - api = TendwireDaemonAPI( - get_snapshot=lambda: snapshot, - get_health=lambda: {"schema_version": 1, "status": "ok"}, - submit_command=lambda _params: {}, - get_turn_content=lambda params: get_turn_content( - db_path, - "long-host", - turn_id=params["turn_id"], - content_revision=params["content_revision"], - field=params["field"], - cursor=params.get("cursor"), - schema_version=params.get("schema_version", 1), - ), - ) - server = UnixSocketJSONServer( - socket_path, - api.dispatch, - accept_timeout_seconds=0.05, - ) - thread = threading.Thread(target=server.serve_forever) - thread.start() - try: - deadline = time.monotonic() + 2 - while not server.listening and time.monotonic() < deadline: - time.sleep(0.01) - daemon_pages = fetch_pages(socket=socket_path, direct_db=None) - daemon_bad_code = main( - [ - "--host-id", - "long-host", - "--socket-path", - str(socket_path), - "turn", - "content", - "get", - "--json", - "--turn-id", - turn["id"], - "--revision", - revision, - "--field", - "assistant_final_text", - "--cursor", - "twcur1.tampered", - ] - ) - daemon_bad_payload = json.loads(capsys.readouterr().out) - assert daemon_bad_code == 1 - assert daemon_bad_payload == bad_cursor_payload - finally: - server.close() - thread.join(timeout=2) - - assert daemon_pages == direct_pages - assert "".join(page["text"] for page in direct_pages) == canonical - assert [page["index"] for page in direct_pages] == list(range(len(direct_pages))) - assert all(page["count"] == len(direct_pages) for page in direct_pages) - assert not thread.is_alive() - - -def test_cli_short_v1_compatibility_then_known_incomplete_refusal( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "content-compatibility.db" - config = Config(host_id="compat-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "Worker", "status": "active"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - assert merge_turn_content( - db_path, - "compat-host", - "worker-1", - { - "source_turn_id": "cli-short-content-source", - "user_text": " short prompt\n", - "assistant_final_text": "\n short final ", - "complete": True, - }, - ) == 1 - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", - lambda _config, **_kwargs: {"ok": True}, - raising=False, - ) - common = [ - "--host-id", - "compat-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--db-path", - str(db_path), - "--json", - ] - - short_v1_code = main(common) - short_v1 = json.loads(capsys.readouterr().out) - short_v2_code = main([*common, "--schema-version", "2"]) - short_v2 = json.loads(capsys.readouterr().out) - short_turn = short_v2["turns"][0] - - assert short_v1_code == 0 - assert short_v1["schema_version"] == 1 - assert short_v1["turns"][0]["assistant_final_text"] == "\n short final " - assert short_v1["turns"][0]["user_text"] == " short prompt\n" - assert "content" not in short_v1["turns"][0] - assert short_v2_code == 0 - assert short_turn["assistant_final_text"] == "\n short final " - assert short_turn["content"]["fields"]["assistant_final_text"]["inline"] is True - - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - UPDATE turn_content_revisions - SET final_state = 'known_incomplete' - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - """, - ("compat-host", short_turn["id"]), - ) - - incomplete_v1_code = main(common) - incomplete_v1 = json.loads(capsys.readouterr().out) - incomplete_v2_code = main([*common, "--schema-version", "2"]) - incomplete_v2 = json.loads(capsys.readouterr().out) - revision = incomplete_v2["turns"][0]["content"]["content_revision"] - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - UPDATE turn_content_revisions - SET content_revision = ? - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - """, - (revision, "compat-host", short_turn["id"]), - ) - content_code = main( - [ - "--host-id", - "compat-host", - "turn", - "content", - "get", - "--json", - "--turn-id", - short_turn["id"], - "--revision", - revision, - "--field", - "assistant_final_text", - "--db-path", - str(db_path), - ] - ) - content_error = json.loads(capsys.readouterr().out) - - assert incomplete_v1_code == 1 - assert incomplete_v1["status"] == "upgrade_required" - assert incomplete_v1["required_turn_schema_version"] == 2 - assert incomplete_v2_code == 0 - incomplete_field = incomplete_v2["turns"][0]["content"]["fields"]["assistant_final_text"] - assert incomplete_field["availability"] == "known_incomplete" - assert incomplete_field["inline"] is False - assert "assistant_final_text" not in incomplete_v2["turns"][0] - assert content_code == 1 - assert content_error["status"] == "content_known_incomplete" - - -def test_cli_pending_missing_store_is_fixed_and_never_observes_sources( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("pending fallback must not observe Herdr or source state") - - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", forbidden, raising=False - ) - - code = main( - [ - "--host-id", - "pending-host", - "pending", - "--json", - "--db-path", - str(tmp_path / "missing.db"), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload == { - "schema_version": 1, - "host_id": "pending-host", - "ok": False, - "status": "store_unavailable", - "pending_interactions": [], - "backend_health": [], - "pending_health": { - "status": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - }, - } - _assert_no_public_json_forbidden(payload) - - -@pytest.mark.parametrize("mode", ["success", "error"]) -def test_cli_pending_structured_daemon_result_or_error_is_authoritative( - tmp_path: Path, - capsys, - monkeypatch, - mode: str, -) -> None: - success = { - "schema_version": 1, - "host_id": "authoritative-pending", - "pending_interactions": [], - "backend_health": [], - "pending_health": { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - }, - "content_fingerprint": "a" * 24, - } - error = { - "schema_version": 1, - "ok": False, - "status": "error", - "result": None, - "error": { - "code": "invalid_params", - "message": "invalid pending request", - }, - } - calls = 0 - - class AuthoritativeClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "pending.list" - assert params == {} - if mode == "success": - return {"ok": True, "result": success} - return error - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("authoritative daemon response must not read fallback state") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", AuthoritativeClient) - monkeypatch.setattr("tendwire.cli.pending_payload_from_store", forbidden) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", forbidden, raising=False - ) - - code = main( - [ - "--host-id", - "authoritative-pending", - "--socket-path", - str(tmp_path / "daemon.sock"), - "pending", - "--json", - "--db-path", - str(tmp_path / "cache.db"), - ] - ) +def test_attention_uses_daemon_and_never_reads_store(capsys, monkeypatch, tmp_path) -> None: + monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path)) + code = main(["attention", "--json"]) payload = json.loads(capsys.readouterr().out) - - assert calls == 1 - assert payload == (success if mode == "success" else error) - assert code == (0 if mode == "success" else 1) - - -@pytest.mark.parametrize( - ("mode", "expected_status"), - [ - ("timeout", "daemon_timeout"), - ("protocol", "daemon_protocol_error"), - ("malformed", "daemon_protocol_error"), - ("post_send_unavailable", "daemon_protocol_error"), - ], -) -def test_cli_pending_post_send_failure_never_reads_or_retries( - tmp_path: Path, - capsys, - monkeypatch, - mode: str, - expected_status: str, -) -> None: - from tendwire.daemon_api import DaemonProtocolError, DaemonUnavailable - - calls = 0 - - class FailingClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "pending.list" - assert params == {} - if mode == "timeout": - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) - if mode == "protocol": - raise DaemonProtocolError("invalid frame", request_started=True) - if mode == "post_send_unavailable": - raise DaemonUnavailable("connection lost", request_started=True) - return {"ok": True, "result": ["not", "a", "mapping"]} - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("post-send failure must not read fallback state") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FailingClient) - monkeypatch.setattr("tendwire.cli.pending_payload_from_store", forbidden) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", forbidden, raising=False - ) - - code = main( - [ - "--host-id", - "failed-pending", - "--socket-path", - str(tmp_path / "daemon.sock"), - "pending", - "--json", - "--db-path", - str(tmp_path / "cache.db"), - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 1 - assert calls == 1 - assert payload == { - "schema_version": 1, - "ok": False, - "status": expected_status, - "error": { - "code": expected_status, - "message": ( - "Tendwire daemon request timed out" - if expected_status == "daemon_timeout" - else "Tendwire daemon returned an invalid response" - ), - }, - } - - -@pytest.mark.parametrize( - ("mode", "expected_status"), - [ - ("timeout", "daemon_timeout"), - ("protocol", "daemon_protocol_error"), - ("post_send_unavailable", "daemon_protocol_error"), - ], -) -def test_cli_connector_post_send_failure_never_mutates_store_fallback( - tmp_path: Path, - capsys, - monkeypatch, - mode: str, - expected_status: str, -) -> None: - from tendwire.daemon_api import DaemonProtocolError, DaemonUnavailable - - calls = 0 - - class FailingClient: - def __init__( - self, - _socket_path: Any, - *, - timeout_seconds: float, - **_kwargs: Any, - ) -> None: - assert timeout_seconds == 30.0 - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "connector.poll" - assert params == { - "name": "turn-final", - "limit": 1, - "lease_seconds": 60, - } - if mode == "timeout": - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) - if mode == "protocol": - raise DaemonProtocolError( - "invalid frame", - request_started=True, - ) - raise DaemonUnavailable( - "connection lost", - request_started=True, - ) - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError( - "post-send connector failure must not execute store fallback" - ) - - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - FailingClient, - ) - monkeypatch.setattr("tendwire.store.sqlite.init_store", forbidden) - - code = main( - [ - "--host-id", - "connector-timeout-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "connector", - "poll", - "--db-path", - str(tmp_path / "cache.db"), - "--name", - "turn-final", - "--limit", - "1", - "--lease-seconds", - "60", - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 1 - assert calls == 1 - assert payload == { - "schema_version": 1, - "ok": False, - "status": expected_status, - "host_id": "connector-timeout-host", - "name": "turn-final", - "error": { - "code": expected_status, - "message": ( - "Tendwire daemon request timed out" - if expected_status == "daemon_timeout" - else "Tendwire daemon returned an invalid response" - ), - }, - } - - -def test_cli_connector_pre_send_unavailable_uses_store_fallback( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - db_path = tmp_path / "connector-fallback.db" - - class UnavailableClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - assert method == "connector.poll" - raise DaemonUnavailable( - "not listening", - request_started=False, - ) - - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - UnavailableClient, - ) - - code = main( - [ - "--host-id", - "connector-fallback-host", - "--socket-path", - str(tmp_path / "missing.sock"), - "connector", - "poll", - "--db-path", - str(db_path), - "--name", - "turn-final", - "--limit", - "1", - "--lease-seconds", - "60", - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 0 - assert payload["schema_version"] == 1 - assert payload["ok"] is True - assert payload["host_id"] == "connector-fallback-host" - assert payload["name"] == "turn-final" - assert payload["items"] == [] - - -def test_cli_store_hooks_print_json_only_and_support_dry_run(tmp_path: Path, capsys) -> None: - db_path = tmp_path / "store-cli.db" - init_store(db_path) - append_event( - db_path, - "store-cli", - "private.event", - {"pane_id": "sentinel-private-pane", "raw_payload": "sentinel-private-raw"}, - observed_at="2026-01-01T00:00:00+00:00", - ) - append_event( - db_path, - "store-cli", - "public.event", - {"safe": "kept"}, - observed_at="9999-01-09T00:00:00+00:00", - ) - - status_code = main(["--host-id", "store-cli", "store", "status", "--db-path", str(db_path)]) - status_captured = capsys.readouterr() - status_payload = json.loads(status_captured.out) - - tail_code = main( - [ - "--host-id", - "store-cli", - "store", - "events-tail", - "--db-path", - str(db_path), - "--limit", - "5", - ] - ) - tail_captured = capsys.readouterr() - tail_payload = json.loads(tail_captured.out) - - cleanup_code = main( - [ - "--host-id", - "store-cli", - "store", - "cleanup", - "--db-path", - str(db_path), - "--retention-days", - "7", - "--dry-run", - ] - ) - cleanup_captured = capsys.readouterr() - cleanup_payload = json.loads(cleanup_captured.out) - - missing_code = main( - [ - "--host-id", - "store-cli", - "store", - "status", - "--db-path", - str(tmp_path / "missing.db"), - ] - ) - missing_captured = capsys.readouterr() - missing_payload = json.loads(missing_captured.out) - - with sqlite3.connect(str(db_path)) as conn: - event_count = conn.execute("SELECT COUNT(*) FROM events WHERE host_id = ?", ("store-cli",)).fetchone()[0] - - assert status_code == 0 - assert tail_code == 0 - assert cleanup_code == 0 - assert missing_code == 1 - assert status_captured.err == tail_captured.err == cleanup_captured.err == missing_captured.err == "" - assert status_payload["counts"]["events"] == 2 - assert tail_payload["events"] - assert "sentinel-private" not in json.dumps(tail_payload) - assert "payload_json" not in json.dumps(tail_payload) - assert cleanup_payload["dry_run"] is True - assert cleanup_payload["retention"]["deleted"] == 1 - assert "last_examined_id" not in json.dumps(cleanup_payload) - assert event_count == 2 - assert missing_payload["status"] == "store_unavailable" - - -@pytest.mark.parametrize("timed_out", [False, True]) -def test_cli_pending_pre_send_unavailable_uses_durable_overlay_only( - tmp_path: Path, - capsys, - monkeypatch, - timed_out: bool, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - db_path = tmp_path / "pending-fallback.db" - snapshot = Snapshot( - host_id="projection-cli", - updated_at="2026-01-01T00:00:00+00:00", - workers=[ - Worker( - id="worker-1", - name="Worker One", - status="pending", - space_id="space-1", - summary="human approval required before continuing", - meta={ - "needs_human": True, - "pane_id": "sentinel-cli-pane", - }, - backend_target={ - "kind": "agent_id", - "value": "sentinel-cli-target", - "sendable": True, - }, - ) - ], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-01T00:00:00+00:00", - } - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - merge_backend_pending( - db_path, - snapshot.host_id, - "worker-1", - { - "question": "Which durable option?", - "kind": "choice", - "choices": [ - {"choice_id": "safe", "label": "Safe"}, - { - "choice_id": "private", - "label": "sentinel-cli-private", - "value": "sentinel-cli-command", - }, - ], - "meta": {"source": "backend", "pane_id": "sentinel-cli-pane"}, - }, - ) - expected = pending_payload_from_store(db_path, snapshot.host_id) - calls = 0 - - class UnavailableClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "pending.list" - assert params == {} - raise DaemonUnavailable( - "not listening", - timed_out=timed_out, - request_started=False, - ) - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("durable pending fallback must not observe Herdr") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", UnavailableClient) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", forbidden, raising=False - ) - - code = main( - [ - "--host-id", - snapshot.host_id, - "--socket-path", - str(tmp_path / "missing.sock"), - "pending", - "--json", - "--db-path", - str(db_path), - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 0 - assert calls == 1 - assert payload == expected - assert payload["pending_interactions"][0]["question"] == "Which durable option?" - assert "sentinel-cli" not in json.dumps(payload, sort_keys=True) - _assert_no_public_json_forbidden(payload) - - -def test_cli_pending_durable_snapshot_strips_raw_command_action_material( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "raw-command-pending.db" - snapshot = Snapshot( - host_id="raw-command-cli", - updated_at="2026-01-01T00:00:00+00:00", - workers=[ - Worker( - id="worker-1", - name="Worker One", - status="waiting", - space_id="space-1", - summary="waiting for action", - ) - ], - attention=[ - AttentionSignal( - kind="worker_status", - severity="warning", - status="waiting", - reason="Choose next action", - source="worker:worker-1", - updated_at="2026-01-01T00:00:00+00:00", - suggested_actions=[ - SuggestedAction( - command="sentinel-cli-safe-looking-command-alias", - params={ - "safe_choice": "kept", - "commandLine": "sentinel-cli-command-line", - "terminal_id": "sentinel-cli-terminal", - "backendTarget": "sentinel-cli-backend", - "session-id": "sentinel-cli-session", - "token": "sentinel-cli-token", - "secret": "sentinel-cli-secret", - }, - ) - ], - meta={ - "worker_id": "worker-1", - "space_id": "space-1", - "needs_human": True, - }, - host_id="raw-command-cli", - ) - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("pending projection must not observe current state") - - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - - code = main( - [ - "--host-id", - snapshot.host_id, - "pending", - "--json", - "--db-path", - str(db_path), - ] - ) - payload = json.loads(capsys.readouterr().out) - encoded = json.dumps(payload, sort_keys=True) - - assert code == 0 - assert payload["pending_interactions"][0]["choices"] == [ - { - "choice_id": payload["pending_interactions"][0]["choices"][0]["choice_id"], - "label": "Action", - } - ] - assert "sentinel-cli-" not in encoded - _assert_no_public_json_forbidden(payload) - - -def test_cli_snapshot_json_reports_healthy_empty_herdr(capsys, monkeypatch) -> None: - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): {"result": {"panes": []}}, - } - - def _fake_run_herdr(args, cfg): - if tuple(args) in responses: - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps(responses[tuple(args)]), - stderr="", - ) - return subprocess.CompletedProcess(args=list(args), returncode=1, stdout="", stderr="") - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", _fake_run_herdr) - - code = main(["--host-id", "cli-empty", "--herdr-bin", "herdr", "snapshot", "--json"]) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert payload["spaces"] == [] - assert payload["workers"] == [] - assert payload["backend_health"][0]["status"] == "healthy" - assert payload["backend_health"][0]["outcome"] == "empty_healthy" - assert payload["backend_health"][0]["counts"] == {"spaces": 0, "workers": 0} - - -def test_cli_snapshot_store_persists_printed_snapshot(tmp_path: Path, capsys) -> None: - db_path = tmp_path / "cli.db" - code = main( - [ - "--host-id", - "cli-store", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--db-path", - str(db_path), - "--json", - "--store", - ] - ) - captured = capsys.readouterr() - - assert code == 0 - payload = json.loads(captured.out) - assert captured.err == "" - restored = latest_snapshot(db_path) - assert restored is not None - assert restored.host_id == "cli-store" - assert restored.content_fingerprint == payload["content_fingerprint"] - - -@pytest.mark.parametrize( - ("outcome", "has_workers", "expected_authority"), - [ - ("healthy_non_empty", True, "complete"), - ("empty_healthy", False, "complete"), - ("missing_binary", False, "none"), - ("timeout", False, "none"), - ("malformed_json", False, "none"), - ("continuity_unavailable", True, "none"), - ("unknown", False, "none"), - ], -) -def test_cli_snapshot_persistence_passes_explicit_observation_authority( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - outcome: str, - has_workers: bool, - expected_authority: str, -) -> None: - db_path = tmp_path / f"{outcome}.db" - init_store(db_path) - config = Config(host_id=f"cli-{outcome}", db_path=db_path) - observed_at = "2026-01-01T00:00:00+00:00" - workers = [Worker(id="worker-1", name="Worker One", status="active")] if has_workers else [] - health = herdr_cli.herdr_backend_health( - outcome, - observed_at=observed_at, - workers=workers, - ) - observation = SimpleNamespace( - spaces=[], - workers=workers, - bindings=[], - backend_health=[health], - ) - captured: list[SnapshotObservationContext] = [] - captured_atomic: list[tuple[list[WorkerBinding], str | None, bool, bool]] = [] - captured_turn_models: list[str] = [] - - monkeypatch.setattr( - "tendwire.cli.fetch_herdr_snapshot_observation", - lambda _config, *, stored_bindings: observation, - ) - - def _capture_save( - _db_path: Path, - _snapshot: Snapshot, - *, - turn_model: str, - observation: SnapshotObservationContext, - worker_bindings: list[WorkerBinding], - binding_backend: str | None, - binding_observation_authoritative: bool, - binding_workers_present: bool, - ) -> bool: - captured_turn_models.append(turn_model) - captured.append(observation) - captured_atomic.append( - ( - worker_bindings, - binding_backend, - binding_observation_authoritative, - binding_workers_present, - ) - ) - return True - - monkeypatch.setattr("tendwire.store.sqlite.save_snapshot", _capture_save) - - observe_public_snapshot(config, store_snapshot=True) - - assert len(captured) == 1 - assert captured[0].authority == expected_authority - assert captured[0].observed_at == observed_at - assert captured_atomic == [ - ([], "herdr", health.status == "healthy", bool(workers)) - ] - assert captured_turn_models == [DEFAULT_TURN_MODEL] - - -def test_rejected_stale_snapshot_does_not_persist_stale_worker_bindings( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "rejected-stale-binding.db" - init_store(db_path) - config = Config(host_id="cli-stale-binding", db_path=db_path) - worker = Worker(id="worker-stale", name="Worker Stale", status="active") - health = herdr_cli.herdr_backend_health( - "healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - workers=[worker], - ) - observation = SimpleNamespace( - spaces=[], - workers=[worker], - bindings=[{"worker_id": worker.id, "turn_target_value": "stale-target"}], - backend_health=[health], - ) - monkeypatch.setattr( - "tendwire.cli.fetch_herdr_snapshot_observation", - lambda _config, *, stored_bindings: observation, - ) - monkeypatch.setattr( - "tendwire.store.sqlite.save_snapshot", - lambda *_args, **_kwargs: False, - ) - - def _forbidden_binding_persistence(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("rejected snapshot must not update private bindings") - - monkeypatch.setattr( - "tendwire.cli._persist_binding_observation", - _forbidden_binding_persistence, - ) - - observed = observe_public_snapshot(config, store_snapshot=True) - assert observed.host_id == config.host_id - - -def test_cli_atomic_snapshot_binding_write_serializes_delayed_older_observer( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - from tendwire.store import sqlite as store_sqlite - - db_path = tmp_path / "cli-atomic-binding-race.db" - config = Config(host_id="cli-atomic-binding", db_path=db_path) - init_store(db_path) - worker = Worker(id="worker-atomic", name="Worker Atomic", status="active") - - def binding(target: str, observed_at: str) -> WorkerBinding: - return WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value=target, - sendable=True, - observed_at=observed_at, - private_fingerprint="same-private-owner", - ) - - observations = { - "older-observer": SimpleNamespace( - spaces=[], - workers=[worker], - bindings=[binding("older-private-target", "2026-01-01T00:00:00+00:00")], - backend_health=[ - herdr_cli.herdr_backend_health( - "healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - workers=[worker], - ) - ], - ), - "newer-observer": SimpleNamespace( - spaces=[], - workers=[worker], - bindings=[binding("newer-private-target", "2026-01-01T00:00:01+00:00")], - backend_health=[ - herdr_cli.herdr_backend_health( - "healthy_non_empty", - observed_at="2026-01-01T00:00:01+00:00", - workers=[worker], - ) - ], - ), - } - monkeypatch.setattr( - "tendwire.cli.fetch_herdr_snapshot_observation", - lambda _config, *, stored_bindings: observations[threading.current_thread().name], - ) - original_upsert = store_sqlite._upsert_worker_bindings_conn - older_inside_transaction = threading.Event() - release_older = threading.Event() - - def delayed_upsert(conn: sqlite3.Connection, bindings: Any) -> int: - binding_list = list(bindings) - if binding_list and binding_list[0].target_value == "older-private-target": - older_inside_transaction.set() - assert release_older.wait(timeout=10) - return original_upsert(conn, binding_list) - - monkeypatch.setattr(store_sqlite, "_upsert_worker_bindings_conn", delayed_upsert) - errors: list[BaseException] = [] - - def observe() -> None: - try: - observe_public_snapshot(config, store_snapshot=True) - except BaseException as exc: - errors.append(exc) - - older = threading.Thread(target=observe, name="older-observer") - newer = threading.Thread(target=observe, name="newer-observer") - older.start() - assert older_inside_transaction.wait(timeout=10) - newer.start() - time.sleep(0.05) - release_older.set() - older.join(timeout=10) - newer.join(timeout=10) - - assert not errors - assert not older.is_alive() and not newer.is_alive() - stored = list_worker_bindings( - db_path, - config.host_id, - backend="herdr", - include_expired=True, - ) - assert len(stored) == 1 - assert stored[0].target_value == "newer-private-target" - - -def test_cli_legacy_observation_cannot_claim_complete_authority( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "legacy-observation.db" - init_store(db_path) - config = Config(host_id="cli-legacy", db_path=db_path) - worker = Worker(id="worker-1", name="Worker One", status="blocked") - captured: list[SnapshotObservationContext] = [] - captured_atomic: list[tuple[list[WorkerBinding], str | None]] = [] - captured_turn_models: list[str] = [] - - monkeypatch.setattr( - "tendwire.cli.fetch_herdr_state", - lambda _config, **_kwargs: ([], [worker]), - ) - - def _capture_save( - _db_path: Path, - _snapshot: Snapshot, - *, - turn_model: str, - observation: SnapshotObservationContext, - worker_bindings: list[WorkerBinding], - binding_backend: str | None, - binding_observation_authoritative: bool, - binding_workers_present: bool, - ) -> bool: - captured_turn_models.append(turn_model) - captured.append(observation) - captured_atomic.append((worker_bindings, binding_backend)) - return True - - monkeypatch.setattr("tendwire.store.sqlite.save_snapshot", _capture_save) - - observe_public_snapshot(config, store_snapshot=True) - - assert len(captured) == 1 - assert captured[0].authority == "none" - assert captured_atomic == [([], "herdr")] - assert captured_turn_models == [DEFAULT_TURN_MODEL] - - -def test_cli_attention_json_reads_store_backed_lifecycle( - tmp_path: Path, - capsys, -) -> None: - db_path = tmp_path / "attention.db" - socket_path = tmp_path / "absent.sock" - config = Config(host_id="cli-attention", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-1", - "name": "Worker One", - "status": "blocked", - "meta": { - "safe": "kept", - "pane_id": "sentinel-private-pane", - "terminalId": "sentinel-private-terminal", - "backendTarget": "sentinel-private-backend", - "authToken": "sentinel-private-token", - }, - } - ], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-01T00:00:00+00:00", - "counts": {"workers": 1}, - } - ], - timestamp=datetime.fromisoformat("2026-01-01T00:00:00+00:00"), - ) - save_snapshot( - db_path, - snapshot, - observation=SnapshotObservationContext( - authority="complete", - observed_at="2026-01-01T00:00:00+00:00", - ), - ) - - code = main( - [ - "--host-id", - "cli-attention", - "--socket-path", - str(socket_path), - "attention", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert captured.err == "" - assert payload["host_id"] == "cli-attention" - assert payload["attention"][0]["lifecycle_status"] == "open" - assert payload["attention"][0]["first_seen_at"] == "2026-01-01T00:00:00+00:00" - assert payload["attention"][0]["signal_count"] == 1 - assert len(payload["attention"]) == 1 - assert not { - "family_key", - "generation", - "first_missing_at", - "missing_observation_count", - "last_accepted_at", - "last_observation_key", - "max_notified_severity_rank", - }.intersection(payload["attention"][0]) - assert "sentinel-private" not in json.dumps(payload, sort_keys=True) - _assert_no_public_json_forbidden(payload) - - -def test_cli_attention_json_falls_back_to_snapshot_when_store_is_unavailable( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "missing.db" - socket_path = tmp_path / "absent.sock" - - def _fake_herdr_state(config): - return [], [ - Worker( - id="worker-1", - name="Worker One", - status="blocked", - meta={"pane_id": "sentinel-private-pane"}, - ) - ] - - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - - code = main( - [ - "--host-id", - "cli-attention-fallback", - "--socket-path", - str(socket_path), - "attention", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert captured.err == "" - assert len(payload["attention"]) == 1 - assert payload["attention"][0]["status"] == "blocked" - assert "first_seen_at" not in payload["attention"][0] - assert "sentinel-private" not in json.dumps(payload, sort_keys=True) - _assert_no_public_json_forbidden(payload) - - -def test_cli_public_json_does_not_emit_connector_private_store_rows( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "connector-private.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "public-host", - "sentinel-connector-private", - "sentinel-delivery-key", - "queued", - '{"safe":"kept"}', - '{"chat_id":"sentinel-chat","route":"sentinel-route"}', - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - outbox_id = int(conn.execute("SELECT last_insert_rowid()").fetchone()[0]) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - outbox_id, - "public-host", - "sentinel-connector-private", - "sentinel-delivery-key", - 1, - "delivered", - '{"ok":true}', - '{"message_id":"sentinel-message","token":"sentinel-token"}', - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - - payloads: list[dict[str, Any]] = [] - - snapshot_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--json", - "--store", - "--db-path", - str(db_path), - ] - ) - snapshot_captured = capsys.readouterr() - payloads.append(json.loads(snapshot_captured.out)) - - turns_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--json", - ] - ) - turns_captured = capsys.readouterr() - payloads.append(json.loads(turns_captured.out)) - - pending_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "pending", - "--json", - "--db-path", - str(db_path), - ] - ) - pending_captured = capsys.readouterr() - payloads.append(json.loads(pending_captured.out)) - - doctor_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "doctor", - "--json", - ] - ) - doctor_captured = capsys.readouterr() - payloads.append(json.loads(doctor_captured.out)) - - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "read_snapshot"})), - ) - command_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - command_captured = capsys.readouterr() - payloads.append(json.loads(command_captured.out)) - - with sqlite3.connect(str(db_path)) as conn: - private_counts = ( - conn.execute("SELECT COUNT(*) FROM connector_outbox").fetchone()[0], - conn.execute("SELECT COUNT(*) FROM connector_deliveries").fetchone()[0], - ) - - encoded = json.dumps(payloads, sort_keys=True) - assert snapshot_code == 0 - assert turns_code == 1 - assert pending_code == 0 - assert doctor_code == 1 - assert command_code == 0 - assert private_counts == (1, 1) - assert "sentinel-" not in encoded - - -def test_cli_snapshot_store_persists_private_bindings_outside_snapshot_payload( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "bindings.db" - responses = { - ("workspace", "list"): { - "result": { - "workspaces": [ - {"workspace_id": "wA", "label": "Bindings"} - ] - } - }, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-worker", - "agent_id": "agent-secret", - "agent": "Worker", - "workspace_id": "wA", - "pane_id": "wA:p1", - } - ] - } - }, - ("pane", "list"): { - "result": { - "panes": [ - { - "workspace_id": "wA", - "pane_id": "wA:p1", - "terminal_id": "terminal-secret", - "agent": "Worker", - } - ] - } - }, - } - - def _fake_run_herdr(args, cfg): - if tuple(args) in responses: - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps(responses[tuple(args)]), - stderr="", - ) - return subprocess.CompletedProcess(args=list(args), returncode=1, stdout="", stderr="") - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", _fake_run_herdr) - - code = main( - [ - "--host-id", - "cli-bindings", - "--herdr-bin", - "herdr", - "snapshot", - "--db-path", - str(db_path), - "--json", - "--store", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - bindings = list_worker_bindings(db_path, "cli-bindings", backend="herdr") - - assert code == 0 - assert len(bindings) == 1 - assert bindings[0].worker_id == "public-worker" - assert bindings[0].target_kind == "agent_id" - assert bindings[0].target_value == "agent-secret" - encoded = json.dumps(payload) - assert "agent-secret" not in encoded - assert "wA:p1" not in encoded - assert "target_kind" not in encoded - - -def test_cli_module_invocation() -> None: - """python -m tendwire.cli snapshot --json works.""" - env = os.environ.copy() - env["PYTHONPATH"] = os.path.join(os.path.dirname(__file__), "..", "src") - result = subprocess.run( - [ - sys.executable, - "-m", - "tendwire.cli", - "--host-id", - "module-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--json", - ], - capture_output=True, - text=True, - check=False, - env=env, - ) - assert result.returncode == 0, result.stderr - assert result.stderr == "" - payload = json.loads(result.stdout) - assert payload["schema_version"] == 2 - assert payload["host_id"] == "module-host" - assert len(payload["content_fingerprint"]) == 24 - - -def test_cli_snapshot_with_live_shaped_herdr_fixtures(capsys, monkeypatch) -> None: - """CLI emits schema v2 JSON with non-empty spaces and workers from Herdr fixtures.""" - - def _fake_run_herdr(args, cfg): - if tuple(args) == ("workspace", "list", "--json"): - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps({ - "result": { - "workspaces": [ - { - "workspace_id": "wA", - "label": "CLI Space", - "agent_status": "working", - "focused": True, - } - ] - } - }), - stderr="", - ) - if tuple(args) == ("agent", "list", "--json"): - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps({ - "result": { - "agents": [ - { - "agent_session": {"value": "sess-cli"}, - "agent": "CLI Agent", - "workspace_id": "wA", - "pane_id": "wA:p1", - "agent_status": "executing", - "cwd": "/tmp", - } - ] - } - }), - stderr="", - ) - if tuple(args) == ("pane", "list"): - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps({ - "result": { - "panes": [ - { - "workspace_id": "wA", - "pane_id": "wA:p1", - "terminal_id": "terminal-cli", - "agent": "CLI Agent", - "agent_session": {"value": "sess-cli"}, - "agent_status": "executing", - } - ] - } - }), - stderr="", - ) - return subprocess.CompletedProcess(args=list(args), returncode=1, stdout="", stderr="") - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", _fake_run_herdr) - - code = main(["--host-id", "cli-live", "--herdr-bin", "herdr", "snapshot", "--json"]) - captured = capsys.readouterr() - - assert code == 0 - assert captured.err == "" - payload = json.loads(captured.out) - assert payload["schema_version"] == 2 - assert payload["host_id"] == "cli-live" - assert len(payload["spaces"]) == 1 - assert payload["spaces"][0]["id"] == "wA" - assert payload["spaces"][0]["status"] == "active" - assert len(payload["workers"]) == 1 - assert payload["workers"][0]["id"] == "CLI Agent" - assert payload["workers"][0]["status"] == "active" - assert payload["backend_health"][0]["name"] == "herdr" - assert payload["backend_health"][0]["status"] == "healthy" - assert payload["backend_health"][0]["outcome"] == "healthy_non_empty" - assert payload["backend_health"][0]["counts"] == {"spaces": 1, "workers": 1} - assert "agent_session" not in json.dumps(payload) - assert "sess-cli" not in json.dumps(payload) - - -def test_cli_store_compact_parser_requires_exactly_one_mode() -> None: - parser = _build_parser() - parsed = parser.parse_args( - [ - "store", - "compact", - "--db-path", - "private.db", - "--dry-run", - "--snapshot-retention-days", - "9", - "--snapshot-retention-count", - "17", - "--batch-size", - "3", - ] - ) - - assert parsed.store_action == "compact" - assert parsed.compact_dry_run is True - assert parsed.compact_execute is False - assert parsed.snapshot_retention_days == 9 - assert parsed.snapshot_retention_count == 17 - assert parsed.snapshot_batch_size == 3 - with pytest.raises(SystemExit): - parser.parse_args(["store", "compact"]) - with pytest.raises(SystemExit): - parser.parse_args( - ["store", "compact", "--dry-run", "--execute"] - ) - - -@pytest.mark.parametrize( - "extra", - [ - ["--execute"], - ["--execute", "--acknowledge-offline"], - ["--execute", "--backup-path", "private-backup.db"], - ["--dry-run", "--acknowledge-offline"], - ["--dry-run", "--backup-path", "private-backup.db"], - ["--dry-run", "--batch-size", "0"], - ], -) -def test_cli_store_compact_rejects_invalid_authority_as_one_json_object( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - extra: list[str], -) -> None: - db_path = tmp_path / "must-not-be-opened.db" - - code = main( - [ - "store", - "compact", - "--db-path", - str(db_path), - *extra, - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["status"] == "invalid_request" - assert payload["command"] == "store.compact" - assert not db_path.exists() - - -def test_cli_store_compact_dry_run_is_read_only_json_and_skips_generic_repair( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "compact-dry-run.db" - init_store(db_path) - before = tuple( - sorted( - ( - path.name, - path.stat().st_size, - path.stat().st_mtime_ns, - ) - for path in tmp_path.iterdir() - ) - ) - - def forbidden_repair(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("compact must not run generic permission repair") - - monkeypatch.setattr("tendwire.cli.repair_config_state", forbidden_repair) - code = main( - [ - "store", - "compact", - "--db-path", - str(db_path), - "--dry-run", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - after = tuple( - sorted( - ( - path.name, - path.stat().st_size, - path.stat().st_mtime_ns, - ) - for path in tmp_path.iterdir() - ) - ) - - assert code == 0 - assert captured.err == "" - assert payload["status"] == "dry_run" - assert payload["ok"] is True - assert payload["backup"]["created"] is False - assert after == before - - -def test_cli_store_compact_execute_success_is_public_safe_and_retains_backup( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], -) -> None: - db_path = tmp_path / "sentinel-private-store-name.db" - backup_path = tmp_path / "sentinel-private-backup-name.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - conn.execute("CREATE TABLE compact_private (value TEXT NOT NULL)") - conn.execute( - "INSERT INTO compact_private (value) VALUES (?)", - ("sentinel-private-payload",), - ) - - code = main( - [ - "store", - "compact", - "--db-path", - str(db_path), - "--execute", - "--acknowledge-offline", - "--backup-path", - str(backup_path), - "--snapshot-retention-days", - "7", - "--snapshot-retention-count", - "2", - "--batch-size", - "1", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - encoded = json.dumps(payload, sort_keys=True) - - assert code == 0 - assert captured.err == "" - assert payload["status"] == "completed" - assert payload["command"] == "store.compact" - assert payload["backup"] == { - "required": True, - "created": True, - "verified": True, - } - assert backup_path.is_file() - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - "SELECT value FROM compact_private" - ).fetchone()[0] == "sentinel-private-payload" - assert conn.execute("PRAGMA quick_check").fetchone()[0] == "ok" - for private_value in ( - str(db_path), - db_path.name, - str(backup_path), - backup_path.name, - "sentinel-private-payload", - ): - assert private_value not in encoded - - -def test_cli_store_cleanup_passes_snapshot_policy_overrides( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "cleanup-policy.db" - init_store(db_path) - captured_options: dict[str, Any] = {} - - def capture_maintenance( - _db_path: Path, - _host_id: str, - **kwargs: Any, - ) -> dict[str, Any]: - captured_options.update(kwargs) - return {"schema_version": 1, "ok": True, "status": "ok"} - - monkeypatch.setattr( - "tendwire.cli.run_store_maintenance", - capture_maintenance, - ) - code = main( - [ - "store", - "cleanup", - "--db-path", - str(db_path), - "--dry-run", - "--acknowledged-final-retention-days", - "17", - "--acknowledged-final-retention-count", - "29", - "--snapshot-retention-days", - "11", - "--snapshot-retention-count", - "23", - "--snapshot-batch-size", - "5", - ] - ) - captured = capsys.readouterr() - - assert code == 0 - assert captured.err == "" - assert json.loads(captured.out)["ok"] is True - assert captured_options["dry_run"] is True - assert captured_options["acknowledged_final_retention_days"] == 17 - assert captured_options["acknowledged_final_retention_count"] == 29 - assert captured_options["snapshot_retention_days"] == 11 - assert captured_options["snapshot_retention_count"] == 23 - assert captured_options["snapshot_batch_size"] == 5 - - -def test_cli_store_status_passes_configured_maintenance_policy( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "status-policy.db" - init_store(db_path) - monkeypatch.setenv("TENDWIRE_SNAPSHOT_RETENTION_DAYS", "19") - monkeypatch.setenv("TENDWIRE_SNAPSHOT_RETENTION_COUNT", "211") - monkeypatch.setenv("TENDWIRE_SNAPSHOT_MAINTENANCE_BATCH_SIZE", "37") - monkeypatch.setenv("TENDWIRE_STORE_MAINTENANCE_CADENCE_SECONDS", "7200") - monkeypatch.setenv("TENDWIRE_ACKNOWLEDGED_FINAL_RETENTION_DAYS", "31") - monkeypatch.setenv("TENDWIRE_ACKNOWLEDGED_FINAL_RETENTION_COUNT", "422") - received: dict[str, Any] = {} - - def capture_status( - _db_path: Path, - _host_id: str, - **kwargs: Any, - ) -> dict[str, Any]: - received.update(kwargs) - return {"schema_version": 1, "ok": True, "status": "ok"} - - monkeypatch.setattr("tendwire.cli.store_status", capture_status) - code = main( - [ - "store", - "status", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert code == 0 - assert captured.err == "" - assert json.loads(captured.out)["ok"] is True - assert received == { - "acknowledged_final_retention_days": 31, - "acknowledged_final_retention_count": 422, - "snapshot_retention_days": 19, - "snapshot_retention_count": 211, - "maintenance_batch_size": 37, - "maintenance_cadence_seconds": 7200, - "command_retry_horizon_seconds": 604800, - "command_receipt_retention_seconds": 2592000, - "command_receipt_retention_count": 4096, - } - - -def test_cli_doctor_with_absent_sqlite_sidecars_is_noncreating( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, -) -> None: - state_dir = tmp_path / "private-cli-doctor-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-cli-doctor-database" - init_store(db_path) - sidecars = tuple( - Path(f"{db_path}{suffix}") for suffix in ("-wal", "-shm", "-journal") - ) - assert all(not os.path.lexists(path) for path in sidecars) - before = ( - tuple(sorted(path.name for path in state_dir.iterdir())), - db_path.stat().st_ino, - db_path.stat().st_size, - db_path.stat().st_mtime_ns, - ) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(state_dir)) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(db_path)) - - def forbidden_repair(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("doctor must remain validation-only") - - monkeypatch.setattr("tendwire.cli.repair_config_state", forbidden_repair) - code = main( - [ - "--host-id", - "doctor-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "doctor", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - database_check = next( - check for check in payload["checks"] if check["name"] == "database_permissions" - ) - after = ( - tuple(sorted(path.name for path in state_dir.iterdir())), - db_path.stat().st_ino, - db_path.stat().st_size, - db_path.stat().st_mtime_ns, - ) - - assert code == 1 - assert captured.err == "" - assert database_check == { - "name": "database_permissions", - "ok": True, - "outcome": "compliant", - "remediation": "No action required.", - } - assert after == before - assert all(not os.path.lexists(path) for path in sidecars) - - -@pytest.mark.parametrize("hostile_member", ["main", "wal"]) -def test_cli_doctor_sqlite_failures_are_one_fixed_path_free_record( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, - hostile_member: str, -) -> None: - state_dir = tmp_path / "private-cli-hostile-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-cli-hostile-database" - target = state_dir / "private-cli-hostile-target" - private_contents = b"raw-OSError-private-cli-target" - target.write_bytes(private_contents) - os.chmod(target, 0o600) - if hostile_member == "main": - hostile_path = db_path - else: - init_store(db_path) - hostile_path = Path(f"{db_path}-wal") - hostile_path.symlink_to(target) - hostile_inode = str(os.lstat(hostile_path).st_ino) - target_before = target.read_bytes() - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(state_dir)) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(db_path)) - - def forbidden_repair(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("doctor must not repair hostile SQLite entries") - - monkeypatch.setattr("tendwire.cli.repair_config_state", forbidden_repair) - code = main( - [ - "--host-id", - "doctor-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "doctor", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - database_check = [ - check for check in payload["checks"] if check["name"] == "database_permissions" - ] - assert code == 1 - assert captured.err == "" - assert database_check == [ - { - "name": "database_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - ] - assert hostile_path.is_symlink() - assert target.read_bytes() == target_before - serialized = json.dumps(payload, sort_keys=True) - for forbidden in ( - str(state_dir), - str(db_path), - db_path.name, - str(hostile_path), - hostile_path.name, - str(target), - target.name, - private_contents.decode(), - hostile_inode, - "-wal", - "-shm", - "-journal", - "OSError", - "[Errno", - '"uid"', - '"gid"', - '"inode"', - ): - assert forbidden not in serialized + assert payload["status"] == "daemon_unavailable" + assert not (tmp_path / "tendwire.db").exists() diff --git a/tests/test_cli_command.py b/tests/test_cli_command.py index 4e9535a..1db0265 100644 --- a/tests/test_cli_command.py +++ b/tests/test_cli_command.py @@ -1,2645 +1,43 @@ -"""Tests for the `tendwire command --json` CLI orchestration.""" - from __future__ import annotations import io import json -import socket -import sqlite3 -from pathlib import Path -from typing import Any - -import pytest -from tendwire.backends.herdr_cli import HerdrCommandObservation from tendwire.cli import main -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - DISPOSITION_TERMINAL_ACCEPTED, - DISPOSITION_TERMINAL_REJECTED, - DISPOSITION_TERMINAL_UNCERTAIN, - STATUS_ACCEPTED, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_FAILED, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DRY_RUN, - STATUS_DUPLICATE_REQUEST, - STATUS_INVALID_REQUEST, - STATUS_NOT_FOUND, - STATUS_REQUEST_STATE_UNCERTAIN, - CommandEnvelope, - CommandRequest, - build_canonical_mutation, -) -from tendwire.core.models import Space, Worker, WorkerBinding -from tendwire.store.sqlite import ( - finish_command_request, - get_command_request, - init_store, - list_worker_bindings, - mark_command_send_started, - reserve_command_request, - upsert_worker_bindings, -) - - -@pytest.fixture(autouse=True) -def _isolate_cli_paths(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - private_home = tmp_path / "home" - private_home.mkdir(mode=0o700) - monkeypatch.setenv("HOME", str(private_home)) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "tendwire-data")) - monkeypatch.delenv("TENDWIRE_DB_PATH", raising=False) - - -def _fake_herdr_state(config: Any) -> tuple[list[Space], list[Worker]]: - workers = [ - Worker( - id="w-1", - name="Alpha", - status="active", - space_id="s-1", - backend_target={"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}, - ), - Worker( - id="w-2", - name="Beta", - status="idle", - space_id="s-1", - backend_target={"kind": "agent_id", "value": "agent-2", "sendable": True, "reason": None}, - ), - ] - return [], workers - - -def _fake_herdr_command_observation(config: Any) -> HerdrCommandObservation: - spaces, workers = _fake_herdr_state(config) - return HerdrCommandObservation( - spaces=spaces, - workers=workers, - status="healthy", - outcome="healthy_non_empty", - ) - - - - -def _seed_uncertain_request(db_path: Path, request: CommandRequest) -> None: - canonical = build_canonical_mutation(request, public_worker_id="w-1") - pending = CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_REQUEST_STATE_UNCERTAIN, - disposition=DISPOSITION_TERMINAL_UNCERTAIN, - error={ - "code": STATUS_REQUEST_STATE_UNCERTAIN, - "message": "pending", - "details": {}, - }, - ) - reservation = reserve_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - action=request.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=pending.to_json(), - ) - owner_token = reservation["owner_token"] - assert isinstance(owner_token, str) - started = mark_command_send_started( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - binding_fingerprint="seed-binding", - ) - assert started["status"] == "send_started" - finished = finish_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - expected_state="send_started", - terminal_state="uncertain", - status=STATUS_REQUEST_STATE_UNCERTAIN, - result_json=pending.to_json(), - ) - assert finished["status"] == "uncertain" -def _seed_accepted_request( - db_path: Path, - request: CommandRequest, - *, - worker_id: str = "w-1", -) -> None: - init_store(db_path) - canonical = build_canonical_mutation(request, public_worker_id=worker_id) - pending = CommandEnvelope.from_result( - request, - ok=False, - status="pending", - disposition=DISPOSITION_IN_PROGRESS, - error={"code": "pending", "message": "pending"}, - ) - reservation = reserve_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - action=request.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=pending.to_json(), - ) - owner_token = reservation["owner_token"] - assert isinstance(owner_token, str) - started = mark_command_send_started( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - binding_fingerprint="seed-binding", - ) - assert started["status"] == "send_started" - accepted = CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result={"target": {"worker_id": worker_id}}, - ) - finished = finish_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - expected_state="send_started", - terminal_state="accepted", - status=STATUS_ACCEPTED, - result_json=accepted.to_json(), - ) - assert finished["status"] == "accepted" +def _run(monkeypatch, capsys, payload): + monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(payload))) + code = main(["command", "--json"]) + return code, json.loads(capsys.readouterr().out) -def test_cli_command_invalid_json(capsys, monkeypatch) -> None: - monkeypatch.setattr("sys.stdin", io.StringIO("not json")) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() +def test_invalid_command_is_rejected_locally(monkeypatch, capsys) -> None: + code, payload = _run(monkeypatch, capsys, {"schema_version": 1, "action": "unknown"}) assert code == 1 - payload = json.loads(captured.out) assert payload["ok"] is False - assert payload["status"] == STATUS_INVALID_REQUEST - - -def test_cli_command_noop_success(capsys, monkeypatch) -> None: - calls: list[str] = [] - def guarded_fetch(config: Any) -> tuple[list[Space], list[Worker]]: - calls.append("fetch") - raise AssertionError("noop must not fetch Herdr state") - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", guarded_fetch) - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "noop"})), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() +def test_noop_remains_pure(monkeypatch, capsys) -> None: + code, payload = _run(monkeypatch, capsys, {"schema_version": 1, "action": "noop"}) assert code == 0 - payload = json.loads(captured.out) - assert payload["ok"] is True assert payload["status"] == "noop" - assert payload["schema_version"] == 2 - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert captured.err == "" - assert calls == [] - - -def test_cli_command_unknown_action_rejected(capsys, monkeypatch) -> None: - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "explode"})), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - assert payload["ok"] is False - assert captured.err == "" - - -def test_cli_command_fingerprint_only_target_rejected(capsys, monkeypatch, tmp_path) -> None: - """A fingerprint-only send is invalid at the CLI, before any store or backend work.""" - db_path = tmp_path / "fingerprint-only.db" - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "fingerprint-only", - "dry_run": False, - "target": {"worker_fingerprint": "fingerprint-A"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "unavailable.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - - captured = capsys.readouterr() - payload = json.loads(captured.out) - assert code == 1 - assert payload["ok"] is False - assert payload["status"] == "invalid_request" - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert captured.err == "" - # A request this malformed never reaches durable state. - assert not db_path.exists() - - -def test_cli_command_read_snapshot_neutral_result(capsys, monkeypatch) -> None: - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "read_snapshot"})), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 0 - payload = json.loads(captured.out) - assert payload["ok"] is True - assert payload["status"] == "snapshot" - assert payload["result"]["snapshot"]["schema_version"] == 2 - assert payload["result"]["snapshot"]["backend_health"][0]["status"] == "unavailable" - assert payload["result"]["snapshot"]["backend_health"][0]["outcome"] == "missing_binary" - assert captured.err == "" - - -@pytest.mark.parametrize( - ("request_payload", "expected_result"), - [ - ( - { - "schema_version": 1, - "action": "send_instruction", - "target": {"name": "Alpha"}, - "instruction": {"text": "hello"}, - }, - { - "target": {"name": "Alpha"}, - "instruction": {"text": "hello"}, - }, - ), - ( - { - "schema_version": 1, - "action": "answer_pending", - "params": { - "pending_id": "pending-public", - "pending_fingerprint": "revision-public", - "choice_id": "choice-public", - }, - }, - { - "pending": { - "id": "pending-public", - "fingerprint": "revision-public", - }, - "choice": {"choice_id": "choice-public"}, - "delivery_state": "not_submitted", - }, - ), - ], -) -def test_cli_command_mutation_dry_run_is_pure_and_creates_no_receipt( - request_payload: dict[str, Any], - expected_result: dict[str, Any], - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[str] = [] - - def forbidden(*args: Any, **kwargs: Any) -> Any: - calls.append("io") - raise AssertionError("dry-run must not call daemon, backend, or command store") - - monkeypatch.setattr("tendwire.cli._try_daemon_attempt", forbidden) - monkeypatch.setattr("tendwire.command_submission.get_command_request", forbidden) - monkeypatch.setattr("tendwire.command_submission._current_snapshot", forbidden) - monkeypatch.setattr("tendwire.command_submission.reserve_command_request", forbidden) - - db_path = tmp_path / f"{request_payload['action']}.db" - init_store(db_path) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request_payload))) - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "unavailable.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert payload["ok"] is True - assert payload["status"] == "dry_run" - assert payload["dry_run"] is True - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert payload["result"] == expected_result - assert calls == [] - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 0 - assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 0 - - -@pytest.mark.parametrize("mode", ["cli_socket_path", "env_socket_path", "env_backend_socket"]) -def test_cli_command_socket_mode_mutation_unavailable_does_not_fallback( - mode: str, - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[str] = [] - - def guarded_fetch(*args: Any, **kwargs: Any) -> HerdrCommandObservation: - calls.append("fetch") - raise AssertionError("explicit daemon/socket mode must not fall back to Herdr observation") - - def guarded_send(*args: Any, **kwargs: Any) -> CommandEnvelope: - calls.append("send") - raise AssertionError("explicit daemon/socket mode must not send through Herdr CLI") - - monkeypatch.delenv("TENDWIRE_SOCKET_PATH", raising=False) - monkeypatch.delenv("TENDWIRE_HERDR_BACKEND", raising=False) - monkeypatch.delenv("TENDWIRE_DATA_DIR", raising=False) - - - - socket_path = tmp_path / f"{mode}.sock" - data_dir = tmp_path / "data" - args = [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - ] - forbidden_fragments = [str(socket_path)] - if mode == "cli_socket_path": - args.extend(["--socket-path", str(socket_path)]) - elif mode == "env_socket_path": - monkeypatch.setenv("TENDWIRE_SOCKET_PATH", str(socket_path)) - elif mode == "env_backend_socket": - monkeypatch.setenv("TENDWIRE_HERDR_BACKEND", "socket") - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(data_dir)) - forbidden_fragments.extend([str(data_dir), "tendwire.sock"]) - else: - raise AssertionError(f"unexpected mode {mode}") - - db_path = tmp_path / f"{mode}.db" - request_id = f"daemon-unavailable-{mode}" - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - *args, - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - serialized = json.dumps(payload) - - assert code == 1 - assert captured.err == "" - assert payload["ok"] is False - assert payload["status"] == STATUS_BACKEND_UNAVAILABLE - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert payload["request_id"] == request_id - assert calls == [] - for fragment in forbidden_fragments: - assert fragment not in serialized - _assert_no_command_public_forbidden_fields(payload) - - # A definite pre-start daemon failure never creates a local receipt. - assert get_command_request(db_path, "cmd-host", request_id) is None - - -def test_cli_daemon_client_uses_method_specific_timeouts( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[tuple[str, float]] = [] - - class FakeDaemonAPIClient: - def __init__(self, socket_path: Any, *, timeout_seconds: float, max_response_bytes: int = 1024 * 1024): - self.timeout_seconds = timeout_seconds - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - calls.append((method, self.timeout_seconds)) - if method == "snapshot.get": - return {"ok": True, "result": {"schema_version": 2, "spaces": [], "workers": []}} - return { - "ok": True, - "result": { - "schema_version": 2, - "action": "send_instruction", - "request_id": "daemon-timeout-method", - "ok": True, - "dry_run": False, - "status": STATUS_ACCEPTED, - "disposition": DISPOSITION_TERMINAL_ACCEPTED, - "result": {}, - "error": None, - "warnings": [], - }, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - socket_path = tmp_path / "daemon.sock" - - assert ( - main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(socket_path), - "snapshot", - "--json", - ] - ) - == 0 - ) - capsys.readouterr() - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "daemon-timeout-method", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - assert ( - main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(socket_path), - "command", - "--json", - "--db-path", - str(tmp_path / "cmd.db"), - ] - ) - == 0 - ) - - assert calls[0] == ("snapshot.get", 2.0) - assert calls[1][0] == "command.submit" - assert calls[1][1] > calls[0][1] - assert calls[1][1] >= 5.0 - - -def test_cli_command_socket_mode_daemon_timeout_is_uncertain( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[str] = [] - - def guarded_fetch(*args: Any, **kwargs: Any) -> HerdrCommandObservation: - calls.append("fetch") - raise AssertionError("explicit daemon/socket mode must not fall back to Herdr observation") - - def guarded_send(*args: Any, **kwargs: Any) -> CommandEnvelope: - calls.append("send") - raise AssertionError("explicit daemon/socket mode must not send through Herdr CLI") - - class TimeoutDaemonAPIClient: - def __init__(self, socket_path: Any, *, timeout_seconds: float, max_response_bytes: int = 1024 * 1024): - self.timeout_seconds = timeout_seconds - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - from tendwire.daemon_api import DaemonUnavailable - - try: - raise socket.timeout("timed out") - except socket.timeout as exc: - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) from exc - - - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", TimeoutDaemonAPIClient) - - db_path = tmp_path / "timeout.db" - request_id = "daemon-timeout-uncertain" - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - assert calls == [] - - # A lost daemon response with no authoritative receipt is not a command - # envelope and must never be labeled terminal uncertainty. - assert get_command_request(db_path, "cmd-host", request_id) is None - - -def test_cli_rejects_malformed_daemon_inner_envelope_without_fabricating_json( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - request = { - "schema_version": 1, - "action": "send_instruction", - "request_id": "malformed-daemon-result", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - - class MalformedResultClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - assert method == "command.submit" - assert params == request - return { - "schema_version": 1, - "ok": True, - "status": "ok", - "result": { - "schema_version": 2, - "action": request["action"], - "request_id": request["request_id"], - "ok": True, - "dry_run": False, - "status": STATUS_ACCEPTED, - "result": {}, - "error": None, - "warnings": [], - }, - "error": None, - } - - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - MalformedResultClient, - ) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request))) - db_path = tmp_path / "malformed-result.db" - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - assert get_command_request( - db_path, - "cmd-host", - "malformed-daemon-result", - ) is None - - -@pytest.mark.parametrize( - ("case", "response_request_id", "ok", "status", "disposition"), - [ - ( - "receipt-null-id", - None, - True, - STATUS_ACCEPTED, - DISPOSITION_TERMINAL_ACCEPTED, - ), - ( - "receipt-invalid-id", - "invalid id", - True, - STATUS_ACCEPTED, - DISPOSITION_TERMINAL_ACCEPTED, - ), - ( - "no-receipt-success", - "illegal-daemon-tuple", - True, - STATUS_BACKEND_UNAVAILABLE, - DISPOSITION_NO_RECEIPT, - ), - ( - "no-receipt-accepted", - "illegal-daemon-tuple", - False, - STATUS_ACCEPTED, - DISPOSITION_NO_RECEIPT, - ), - ( - "no-receipt-pending", - "illegal-daemon-tuple", - False, - "pending", - DISPOSITION_NO_RECEIPT, - ), - ( - "no-receipt-uncertain", - "illegal-daemon-tuple", - False, - STATUS_REQUEST_STATE_UNCERTAIN, - DISPOSITION_NO_RECEIPT, - ), - ], -) -def test_cli_strictly_rejects_illegal_daemon_disposition_tuples( - case: str, - response_request_id: Any, - ok: bool, - status: str, - disposition: str, - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - request = { - "schema_version": 1, - "action": "send_instruction", - "request_id": "illegal-daemon-tuple", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - inner = { - "schema_version": 2, - "action": request["action"], - "request_id": response_request_id, - "ok": ok, - "dry_run": False, - "status": status, - "disposition": disposition, - "result": {}, - "error": None if ok else {"code": status, "message": "invalid tuple"}, - "warnings": [], - } - - class IllegalTupleClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - assert method == "command.submit" - assert params == request - return { - "schema_version": 1, - "ok": True, - "status": "ok", - "result": inner, - "error": None, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", IllegalTupleClient) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request))) - db_path = tmp_path / f"{case}.db" - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - assert get_command_request( - db_path, - "cmd-host", - request["request_id"], - ) is None - - -def test_cli_daemon_response_loss_recovers_accepted_receipt_exactly_once( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - from tendwire.daemon_api import DaemonProtocolError - - db_path = tmp_path / "accepted-loss.db" - init_store(db_path) - request_payload = { - "schema_version": 1, - "action": "send_instruction", - "request_id": "accepted-loss", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - request = CommandRequest.from_dict(request_payload) - canonical = build_canonical_mutation(request, public_worker_id="w-1") - effects: list[str] = [] - - class LostAcceptedResponseClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - assert method == "command.submit" - assert params == request_payload - pending = CommandEnvelope.from_result( - request, - ok=False, - status="pending", - disposition=DISPOSITION_IN_PROGRESS, - error={"code": "pending", "message": "pending"}, - ) - reservation = reserve_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - action=request.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=pending.to_json(), - ) - owner_token = reservation["owner_token"] - mark_command_send_started( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - binding_fingerprint="private-binding", - ) - effects.append("sent") - accepted = CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result={"target": {"worker_id": "w-1"}}, - ) - finish_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - expected_state="send_started", - terminal_state="accepted", - status=STATUS_ACCEPTED, - result_json=accepted.to_json(), - ) - raise DaemonProtocolError( - "accepted response lost", - request_started=True, - ) - - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - LostAcceptedResponseClient, - ) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request_payload))) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert captured.err == "" - assert payload["status"] == STATUS_ACCEPTED - assert payload["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - assert effects == ["sent"] - receipt = get_command_request(db_path, "cmd-host", "accepted-loss") - assert receipt is not None - assert receipt["state"] == "accepted" - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize( - ("case", "current_target", "expected_status"), - [ - ( - "changed-worker-id", - {"worker_id": "w-2"}, - STATUS_DUPLICATE_REQUEST, - ), - ("mutable-name", {"name": "Alpha"}, None), - ("mutable-space", {"space_id": "space-1"}, None), - ( - "worker-precondition", - {"worker_id": "w-1", "worker_fingerprint": "current-fingerprint"}, - STATUS_ACCEPTED, - ), - ( - "worker-plus-null-alias", - {"worker_id": "w-1", "name": None}, - None, - ), - ], -) -def test_cli_response_loss_reconciliation_only_uses_stored_explicit_worker_identity( - case: str, - current_target: dict[str, Any], - expected_status: str | None, - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - request_id = f"response-loss-{case}" - db_path = tmp_path / f"{case}.db" - stored_request = CommandRequest( - action="send_instruction", - request_id=request_id, - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - _seed_accepted_request(db_path, stored_request) - current_request = { - **stored_request.to_dict(), - "target": current_target, - } - daemon_calls: list[tuple[str, dict[str, Any]]] = [] - - def stored_rows() -> tuple[list[tuple[Any, ...]], list[tuple[Any, ...]]]: - with sqlite3.connect(str(db_path)) as conn: - return ( - conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall(), - conn.execute("SELECT * FROM events ORDER BY id").fetchall(), - ) - - rows_before = stored_rows() - - class LostResponseClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - from tendwire.daemon_api import DaemonProtocolError - - daemon_calls.append((method, dict(params or {}))) - raise DaemonProtocolError("response lost", request_started=True) - - def forbidden(*args: Any, **kwargs: Any) -> Any: - del args, kwargs - raise AssertionError( - "response-loss reconciliation must not submit or resolve mutable authority" - ) - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", LostResponseClient) - monkeypatch.setattr("tendwire.command_submission._current_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.command_envelope_from_payload", forbidden) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(current_request))) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert daemon_calls == [("command.submit", current_request)] - assert stored_rows() == rows_before - if expected_status is None: - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - elif expected_status == STATUS_ACCEPTED: - assert code == 0 - assert captured.err == "" - payload = json.loads(captured.out) - assert payload["status"] == STATUS_ACCEPTED - assert payload["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - _assert_no_command_public_forbidden_fields(payload) - else: - assert code == 1 - assert captured.err == "" - payload = json.loads(captured.out) - assert payload["status"] == expected_status - assert payload["status"] != STATUS_ACCEPTED - assert payload["disposition"] == DISPOSITION_TERMINAL_REJECTED - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize( - ("action", "action_fields"), - [ - ( - "send_instruction", - { - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - }, - ), - ( - "answer_pending", - { - "params": { - "pending_id": "pending-" + ("a" * 24), - "pending_fingerprint": "b" * 24, - "choice_id": "choice-" + ("c" * 24), - }, - }, - ), - ], - ids=["send", "answer"], -) -@pytest.mark.parametrize( - "request_started", - [False, True], - ids=["pre-start", "may-have-started"], -) -def test_cli_mutations_never_fallback_or_write_receipt_on_daemon_edge_failure( - action: str, - action_fields: dict[str, Any], - request_started: bool, - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[tuple[str, dict[str, Any]]] = [] - - class FailingDaemonAPIClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - from tendwire.daemon_api import DaemonProtocolError - - calls.append((method, dict(params or {}))) - raise DaemonProtocolError( - "deterministic protocol failure", - request_started=request_started, - ) - - request_id = f"{action}-{request_started}" - request = { - "schema_version": 1, - "action": action, - "request_id": request_id, - "dry_run": False, - **action_fields, - } - db_path = tmp_path / f"{action}-{request_started}.db" - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - FailingDaemonAPIClient, - ) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request))) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert calls == [("command.submit", request)] - assert get_command_request(db_path, "cmd-host", request_id) is None - if request_started: - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - else: - payload = json.loads(captured.out) - assert code == 1 - assert captured.err == "" - assert payload["status"] == STATUS_BACKEND_UNAVAILABLE - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert payload["request_id"] == request_id - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize( - ("request_id", "include_request_id"), - [ - (None, False), - (None, True), - ("", True), - (" \t", True), - (" leading", True), - ("trailing ", True), - ("\twrapped\t", True), - ], -) -def test_cli_command_send_instruction_non_dry_run_requires_request_id( - capsys, - monkeypatch, - tmp_path: Path, - request_id: Any, - include_request_id: bool, -) -> None: - calls: list[str] = [] - - def guarded(*args: Any, **kwargs: Any) -> Any: - calls.append("called") - raise AssertionError("invalid request_id must stop before backend or store mutation") - - monkeypatch.setattr("tendwire.command_submission.reserve_command_request", guarded) - payload: dict[str, Any] = { - "schema_version": 1, - "action": "send_instruction", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - if include_request_id: - payload["request_id"] = request_id - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(payload))) - db_path = tmp_path / "invalid-request-id.db" - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - - captured = capsys.readouterr() - assert code == 1 - payload_out = json.loads(captured.out) - assert payload_out["ok"] is False - assert payload_out["status"] == STATUS_INVALID_REQUEST - assert calls == [] - assert not db_path.exists() - -def test_cli_command_send_instruction_requires_socket_backend_for_mutation( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "literal-false.db" - calls: list[tuple[Any, Any]] = [] - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "literal-false", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="literal-false") - assert payload["dry_run"] is False - assert calls == [] - - assert get_command_request(db_path, "cmd-host", "literal-false") is None - - -def test_cli_command_default_backend_does_not_rehydrate_private_stored_binding( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - db_path = tmp_path / "stored-binding.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="w-1", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-stored", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="stored-private", - ) - ], - ) - calls: list[tuple[Any, Any]] = [] - - def targetless_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[Worker(id="w-1", name="Alpha", status="active", space_id="s-1")], - status="healthy", - outcome="healthy_non_empty", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "stored-binding", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="stored-binding") - assert calls == [] - serialized = json.dumps(payload) - assert "agent-stored" not in serialized - assert "stored-private" not in serialized - _assert_no_command_public_forbidden_fields(payload) - - -def test_cli_command_does_not_send_through_expired_stored_binding( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - db_path = tmp_path / "expired-binding.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="w-1", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-expired", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="2026-01-02T00:00:00+00:00", - private_fingerprint="expired-private", - ) - ], - ) - calls: list[tuple[Any, Any]] = [] - - def targetless_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[Worker(id="w-1", name="Alpha", status="active", space_id="s-1")], - status="healthy", - outcome="healthy_non_empty", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "expired-binding", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="expired-binding") - assert calls == [] - assert "agent-expired" not in json.dumps(payload) - _assert_no_command_public_forbidden_fields(payload) -def test_cli_command_duplicate_current_binding_is_ambiguous_and_not_expired( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - db_path = tmp_path / "current-duplicates.db" - calls: list[tuple[Any, Any]] = [] - - def duplicate_observation(config: Any) -> HerdrCommandObservation: - worker_a = Worker( - id="dup-a", - name="Duplicate A", - status="active", - backend_target={"kind": "agent_id", "value": "same-agent", "sendable": True, "reason": None}, - ) - worker_b = Worker( - id="dup-b", - name="Duplicate B", - status="active", - backend_target={"kind": "agent_id", "value": "same-agent", "sendable": True, "reason": None}, - ) - return HerdrCommandObservation( - spaces=[], - workers=[worker_a, worker_b], - status="healthy", - outcome="healthy_non_empty", - bindings=[ - WorkerBinding( - host_id="cmd-host", - worker_id=worker_a.id, - worker_fingerprint=worker_a.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="same-agent", - sendable=False, - reason="duplicate_backend_target", - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="colliding-private", - ), - WorkerBinding( - host_id="cmd-host", - worker_id=worker_b.id, - worker_fingerprint=worker_b.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="same-agent", - sendable=False, - reason="duplicate_backend_target", - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="colliding-private", - ), - ], - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "current-duplicate", - "dry_run": False, - "target": {"worker_id": "dup-a"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - current = list_worker_bindings(db_path, "cmd-host", backend="herdr") - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="current-duplicate") - assert calls == [] - assert current == [] - assert "colliding-private" not in json.dumps(payload) - _assert_no_command_public_forbidden_fields(payload) - - -def test_cli_command_stored_duplicate_binding_is_ambiguous_and_skips_backend( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - db_path = tmp_path / "stored-duplicate.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="dup-stored", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="same-agent", - sendable=False, - reason="duplicate_backend_target", - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-duplicate-fingerprint", - ) - ], - ) - calls: list[tuple[Any, Any]] = [] - - def targetless_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[Worker(id="dup-stored", name="Duplicate", status="active", space_id="s-1")], - status="healthy", - outcome="healthy_non_empty", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "stored-duplicate", - "dry_run": False, - "target": {"worker_id": "dup-stored"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="stored-duplicate") - assert calls == [] - serialized = json.dumps(payload) - assert "same-agent" not in serialized - assert "private-duplicate-fingerprint" not in serialized - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize("value", ["false", "true", 0, 1, None, [], {}]) -def test_cli_command_rejects_non_boolean_dry_run_before_backend( - value: Any, capsys, monkeypatch -) -> None: - calls: list[str] = [] - - def guarded_observation(config: Any) -> HerdrCommandObservation: - calls.append("fetch") - raise AssertionError("invalid dry_run must not fetch") - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "bad-dry-run", - "dry_run": value, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert payload["status"] == STATUS_INVALID_REQUEST - assert calls == [] - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize("value", ["1", 1.0, True, False, None, [], {}, 2]) -def test_cli_command_rejects_malformed_schema_version_before_pipeline( - value: Any, capsys, monkeypatch -) -> None: - calls: list[str] = [] - - def guarded(*args: Any, **kwargs: Any) -> Any: - calls.append("called") - raise AssertionError("invalid schema_version must stop before pipeline work") - - monkeypatch.setattr("tendwire.command_submission.reserve_command_request", guarded) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": value, - "action": "send_instruction", - "request_id": "bad-schema", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert payload["status"] == STATUS_INVALID_REQUEST - assert calls == [] - _assert_no_command_public_forbidden_fields(payload) - - -def test_cli_command_send_instruction_empty_target_rejects_before_fetch(capsys, monkeypatch) -> None: - calls: list[str] = [] - - def guarded_fetch(config: Any) -> tuple[list[Space], list[Worker]]: - calls.append("fetch") - raise AssertionError("empty target must reject before Herdr fetch") - - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", guarded_fetch) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "empty-target", - "dry_run": False, - "target": {}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - assert code == 1 - assert payload["status"] == STATUS_INVALID_REQUEST - assert calls == [] - - -def test_cli_command_duplicate_request_id_same_payload_returns_cached(capsys, monkeypatch, tmp_path: Path) -> None: - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - - calls: list[tuple[Any, Any]] = [] - - db_path = tmp_path / "cmd.db" - payload = json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "dup-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - - monkeypatch.setattr("sys.stdin", io.StringIO(payload)) - code1 = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured1 = capsys.readouterr() - assert code1 == 1 - result1 = json.loads(captured1.out) - _assert_socket_backend_required_payload(result1, request_id="dup-1") - - assert get_command_request(db_path, "cmd-host", "dup-1") is None - - monkeypatch.setattr("sys.stdin", io.StringIO(payload)) - code2 = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured2 = capsys.readouterr() - assert code2 == 1 - result2 = json.loads(captured2.out) - assert result2["status"] == STATUS_BACKEND_UNAVAILABLE - assert result2 == result1 - assert calls == [] - - -def test_cli_command_duplicate_request_id_different_payload_rejects(capsys, monkeypatch, tmp_path: Path) -> None: - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - - calls: list[tuple[Any, Any]] = [] - - db_path = tmp_path / "cmd.db" - payload1 = json.dumps( +def test_live_mutation_never_falls_back_from_daemon(monkeypatch, capsys, tmp_path) -> None: + monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path)) + code, payload = _run( + monkeypatch, + capsys, { "schema_version": 1, "action": "send_instruction", - "request_id": "dup-2", + "request_id": "r1", "dry_run": False, - "target": {"worker_id": "w-1"}, + "target": {"worker_id": "worker"}, "instruction": {"text": "hello"}, - } - ) - payload2 = json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "dup-2", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "world"}, - } - ) - - monkeypatch.setattr("sys.stdin", io.StringIO(payload1)) - code1 = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - capsys.readouterr() - assert code1 == 1 - - monkeypatch.setattr("sys.stdin", io.StringIO(payload2)) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - assert code == 1 - result = json.loads(captured.out) - assert result["status"] == STATUS_BACKEND_UNAVAILABLE - assert calls == [] - - -def test_cli_command_pending_receipt_rejects_without_retry(capsys, monkeypatch, tmp_path: Path) -> None: - db_path = tmp_path / "cmd.db" - pending_request = CommandRequest( - action="send_instruction", - request_id="uncertain-1", - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - # Seed terminal uncertainty through the v12 CAS lifecycle. - init_store(db_path) - _seed_uncertain_request(db_path, pending_request) - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "uncertain-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - assert code == 1 - result = json.loads(captured.out) - assert result["status"] == STATUS_REQUEST_STATE_UNCERTAIN - assert result["disposition"] == DISPOSITION_TERMINAL_UNCERTAIN - - -def test_cli_command_uncertain_receipt_changed_payload_is_duplicate_without_retry( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "cmd.db" - - original_request = CommandRequest( - action="send_instruction", - request_id="uncertain-changed", - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - init_store(db_path) - _seed_uncertain_request(db_path, original_request) - calls: list[str] = [] - - def guarded_backend(*args: Any, **kwargs: Any) -> Any: - calls.append("backend") - raise AssertionError("changed duplicate receipt must not reach the backend") - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "uncertain-changed", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "world"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - result = json.loads(captured.out) - - assert code == 1 - assert result["status"] == STATUS_DUPLICATE_REQUEST - assert calls == [] - - -def test_cli_command_forbidden_field_rejected(capsys, monkeypatch) -> None: - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "noop", - "params": {"pane_id": "leaked"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - assert payload["status"] == STATUS_INVALID_REQUEST - - -def test_cli_command_rejects_control_sequence_instruction_as_json_only(capsys, monkeypatch) -> None: - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello\x1b[31mworld"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["ok"] is False - assert payload["status"] == STATUS_INVALID_REQUEST - assert payload["error"]["details"] == {"field": "instruction.text"} - _assert_no_command_public_forbidden_fields(payload) - - -def test_cli_command_legacy_backend_guard_and_receipt_are_sanitized( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "cmd.db" - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - - calls: list[tuple[Any, Any]] = [] - - def leaky_backend(config: Any, target: Any, instruction: Any) -> CommandEnvelope: - calls.append((target, instruction)) - return CommandEnvelope( - ok=False, - status=STATUS_BACKEND_FAILED, - action="send_instruction", - result={ - "target": target, - "pane_id": "p-1", - "nested": {"argv": ["herdr"], "safe": "kept"}, - }, - error={ - "code": STATUS_BACKEND_FAILED, - "message": "failed", - "details": {"terminal_id": "t-1", "safe": "kept"}, - }, - ) - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "sanitize-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - _assert_socket_backend_required_payload(payload, request_id="sanitize-1") - assert calls == [] - _assert_no_command_public_forbidden_fields(payload) - - assert get_command_request(db_path, "cmd-host", "sanitize-1") is None - assert captured.err == "" - - -_COMMAND_PUBLIC_FORBIDDEN_KEYS = { - "pane_id", - "terminal_id", - "pid", - "tty", - "pty", - "tmux", - "screen_session", - "window_id", - "tab_id", - "argv", - "shell", - "command", - "route", - "routes", - "delivery", - "deliveries", - "token", - "tokens", - "connector", - "connectors", - "backend_target", - "agent_session", - "session_id", - "herdr_state", - "herdres_state", - "target_kind", - "target_value", - "turn_target_kind", - "turn_target_value", - "private_fingerprint", -} - - -def _assert_no_command_public_forbidden_fields(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - assert key not in _COMMAND_PUBLIC_FORBIDDEN_KEYS, f"forbidden field {path}.{key}" - _assert_no_command_public_forbidden_fields(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_command_public_forbidden_fields(item, f"{path}[{index}]") - - -def _assert_socket_backend_required_payload(value: dict[str, Any], *, request_id: str) -> None: - assert value["ok"] is False - assert value["status"] == STATUS_BACKEND_UNAVAILABLE - assert value["request_id"] == request_id - assert value["error"]["code"] == STATUS_BACKEND_UNAVAILABLE - assert value["error"]["message"] == "Herdr socket backend is not enabled" - _assert_no_command_public_forbidden_fields(value) - - -def _fake_herdr_state_with_terminal(config: Any) -> tuple[list[Space], list[Worker]]: - return [], [ - Worker( - id="w-terminal", - name="Terminal", - status="active", - space_id="s-1", - meta={ - "pane_id": "p-1", - "terminal_id": "t-1", - "pid": 123, - "tty": "/dev/pts/0", - "pty": "pts", - "tmux": "sess", - "screen_session": "scr", - "window_id": "win-1", - "tab_id": "tab-1", - "argv": ["bash"], - "shell": "bash", - "command": "python app.py", - "route": "telegram", - "routes": ["r1"], - "delivery": {"id": 1}, - "deliveries": [{"id": 2}], - "token": "secret", - "tokens": ["t1"], - "connector": {"x": 1}, - "connectors": [{"y": 2}], - "backend_target": {"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}, - "agent_session": {"value": "sess-1"}, - "session_id": "session-1", - "safe": "kept", - }, - backend_target={"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}, - ) - ] - - -def test_cli_command_read_snapshot_strips_command_public_terminal_fields( - capsys, monkeypatch -) -> None: - """Command-public read_snapshot strips terminal/connector identifiers while - leaving the ordinary snapshot --json output unchanged. - """ - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state_with_terminal) - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "read_snapshot"})), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 0 - payload = json.loads(captured.out) - assert payload["ok"] is True - assert payload["status"] == "snapshot" - assert payload["action"] == "read_snapshot" - assert "request_id" in payload - meta = payload["result"]["snapshot"]["workers"][0]["meta"] - for key in _COMMAND_PUBLIC_FORBIDDEN_KEYS: - assert key not in meta, key - assert meta["safe"] == "kept" - - # The standalone snapshot path is public too, so backend identifiers are absent there as well. - code2 = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--json", - ] - ) - captured2 = capsys.readouterr() - assert code2 == 0 - snapshot = json.loads(captured2.out) - assert snapshot["schema_version"] == 2 - snap_meta = snapshot["workers"][0]["meta"] - for key in ("pane_id", "terminal_id", "backend_target", "agent_session", "session_id"): - assert key not in snap_meta - assert snap_meta["safe"] == "kept" - - -def test_cli_command_does_not_auto_discover_default_daemon_socket( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - """A default socket file must not opt ordinary CLI commands into daemon mode.""" - data_dir = tmp_path / "data" - data_dir.mkdir() - (data_dir / "tendwire.sock").touch() - calls: list[str] = [] - - class GuardedDaemonAPIClient: - def __init__(self, *args: Any, **kwargs: Any) -> None: - calls.append("daemon") - raise AssertionError("implicit default socket should not be contacted") - - monkeypatch.delenv("TENDWIRE_SOCKET_PATH", raising=False) - monkeypatch.delenv("TENDWIRE_HERDR_BACKEND", raising=False) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(data_dir)) - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", GuardedDaemonAPIClient) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "read_snapshot"})), - ) - - code = main(["--host-id", "cmd-host", "command", "--json"]) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert calls == [] - assert payload["ok"] is True - assert payload["status"] == "snapshot" - assert payload["result"]["snapshot"]["workers"][0]["id"] == "w-1" - - -def test_cli_command_forbidden_field_rejects_before_backend_and_store( - capsys, monkeypatch -) -> None: - """A contract-invalid request must be rejected before any backend or store call.""" - calls: list[str] = [] - - def guarded_fetch(config: Any) -> tuple[list[Space], list[Worker]]: - calls.append("fetch") - raise AssertionError("fetch_herdr_state called before validation") - - def guarded_reserve_request(*args: Any, **kwargs: Any) -> Any: - calls.append("reserve_request") - raise AssertionError("reserve_command_request called before validation") - - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", guarded_fetch) - monkeypatch.setattr( - "tendwire.command_submission.reserve_command_request", - guarded_reserve_request, - ) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "rej-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - "params": {"pane_id": "leaked"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - assert payload["status"] == STATUS_INVALID_REQUEST - assert payload["request_id"] == "rej-1" - assert calls == [] - - -def test_cli_command_raw_top_level_forbidden_rejects_before_pipeline( - capsys, monkeypatch -) -> None: - """Raw top-level forbidden fields reject before store, projection, or backend work.""" - calls: list[str] = [] - - def guarded_reserve_request(*args: Any, **kwargs: Any) -> Any: - calls.append("reserve_request") - raise AssertionError("reserve_command_request called before raw validation") - - def guarded_fetch(config: Any) -> tuple[list[Space], list[Worker]]: - calls.append("fetch") - raise AssertionError("fetch_herdr_state called before raw validation") - - def guarded_project(*args: Any, **kwargs: Any) -> Any: - calls.append("project") - raise AssertionError("project_from_observations called before raw validation") - - def guarded_execute(*args: Any, **kwargs: Any) -> Any: - calls.append("execute") - raise AssertionError("execute_command called before raw validation") - - def guarded_send(*args: Any, **kwargs: Any) -> Any: - calls.append("backend") - raise AssertionError("backend sender called before raw validation") - - monkeypatch.setattr( - "tendwire.command_submission.reserve_command_request", - guarded_reserve_request, - ) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", guarded_fetch) - monkeypatch.setattr("tendwire.cli.project_from_observations", guarded_project) - monkeypatch.setattr("tendwire.cli.execute_command", guarded_execute) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "raw-rej", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - "pane_id": "leaked", - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - assert code == 1 - assert payload["status"] == STATUS_INVALID_REQUEST - assert payload["request_id"] is None - assert "pane_id" in str(payload["error"]["details"]) - assert calls == [] - - -def test_cli_command_backend_unavailable_preserves_request_id( - capsys, monkeypatch, tmp_path: Path -) -> None: - """A pre-start backend failure preserves request_id without store authority.""" - db_path = tmp_path / "req.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="still-live", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-still-live", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="still-live-private", - ) - ], - ) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "req-visible", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - assert payload["status"] == STATUS_BACKEND_UNAVAILABLE - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert payload["request_id"] == "req-visible" - - assert get_command_request(db_path, "cmd-host", "req-visible") is None - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT COUNT(*) - FROM commands - WHERE host_id = ? - AND request_id = ? - """, - ("cmd-host", "req-visible"), - ).fetchone()[0] == 0 - current = list_worker_bindings(db_path, "cmd-host", backend="herdr") - assert [binding.private_fingerprint for binding in current] == ["still-live-private"] - - -def test_cli_command_default_backend_blocks_degraded_observation_send( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "degraded.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="still-live", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-still-live", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="still-live-private", - ) - ], - ) - - def degraded_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[], - status="degraded", - outcome="malformed_json", - message="Herdr agent observation is not healthy", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "degraded-1", - "dry_run": False, - "target": {"worker_id": "missing"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="degraded-1") - assert get_command_request(db_path, "cmd-host", "degraded-1") is None - current = list_worker_bindings(db_path, "cmd-host", backend="herdr") - assert [binding.private_fingerprint for binding in current] == ["still-live-private"] - - -def test_cli_command_default_backend_blocks_healthy_empty_observation_send( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "empty.db" - - def empty_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[], - status="healthy", - outcome="empty_healthy", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "empty-1", - "dry_run": False, - "target": {"worker_id": "missing"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] + }, ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="empty-1") - assert get_command_request(db_path, "cmd-host", "empty-1") is None + assert payload["status"] == "backend_unavailable" + assert not (tmp_path / "tendwire.db").exists() diff --git a/tests/test_config.py b/tests/test_config.py index a005764..58673b8 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -17,7 +17,6 @@ DEFAULT_COMMAND_RECEIPT_RETENTION_COUNT, DEFAULT_COMMAND_RECEIPT_RETENTION_SECONDS, DEFAULT_COMMAND_RETRY_HORIZON_SECONDS, - DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS, DEFAULT_SUBMISSION_HARD_TTL_SECONDS, DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS, DEFAULT_TURN_MODEL, @@ -31,46 +30,6 @@ ) -def test_initial_reconcile_timeout_is_distinct_and_configurable(monkeypatch) -> None: - monkeypatch.delenv( - "TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS", - raising=False, - ) - defaults = load_config(herdr_timeout_seconds="2.5") - assert defaults.herdr_timeout_seconds == 2.5 - assert ( - defaults.herdr_initial_reconcile_timeout_seconds - == DEFAULT_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS - == 120.0 - ) - - monkeypatch.setenv("TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS", "45") - environment = load_config(herdr_timeout_seconds="1.5") - explicit = load_config( - herdr_timeout_seconds="0.75", - herdr_initial_reconcile_timeout_seconds="90", - ) - - assert environment.herdr_timeout_seconds == 1.5 - assert environment.herdr_initial_reconcile_timeout_seconds == 45.0 - assert explicit.herdr_timeout_seconds == 0.75 - assert explicit.herdr_initial_reconcile_timeout_seconds == 90.0 - - -@pytest.mark.parametrize("value", ["", "invalid", "0", "-1", "inf", "nan"]) -def test_initial_reconcile_timeout_rejects_invalid_environment( - monkeypatch, - value: str, -) -> None: - monkeypatch.setenv("TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS", value) - - with pytest.raises( - ValueError, - match="herdr_initial_reconcile_timeout_seconds must be a finite positive number", - ): - load_config() - - def test_acp_defaults_are_required_runtime_settings_with_thoughts_disabled( monkeypatch, ) -> None: @@ -216,10 +175,8 @@ def test_submission_hard_ttl_must_cover_link_window() -> None: def test_pr16_runtime_knobs_have_documented_defaults(monkeypatch) -> None: for name in ( - "TENDWIRE_EVENT_DEBOUNCE_SECONDS", "TENDWIRE_RECONCILE_INTERVAL_SECONDS", "TENDWIRE_EVENT_RETENTION_DAYS", - "TENDWIRE_OUTPUT_EXCERPT_CHARS", "TENDWIRE_MAX_WORKERS", "TENDWIRE_MAX_OUTBOX_ATTEMPTS", "TENDWIRE_CONNECTOR_CLAIM_TTL_SECONDS", @@ -233,10 +190,8 @@ def test_pr16_runtime_knobs_have_documented_defaults(monkeypatch) -> None: config = load_config() - assert config.event_debounce_seconds == 0.05 - assert config.reconcile_interval_seconds == 300.0 + assert config.reconcile_interval_seconds == 15.0 assert config.event_retention_days == 7 - assert config.output_excerpt_chars == 200 assert config.max_workers == 512 assert config.max_outbox_attempts == 10 assert config.connector_claim_ttl_seconds == 60 @@ -248,10 +203,8 @@ def test_pr16_runtime_knobs_have_documented_defaults(monkeypatch) -> None: def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_EVENT_DEBOUNCE_SECONDS", "0.25") monkeypatch.setenv("TENDWIRE_RECONCILE_INTERVAL_SECONDS", "0") monkeypatch.setenv("TENDWIRE_EVENT_RETENTION_DAYS", "14") - monkeypatch.setenv("TENDWIRE_OUTPUT_EXCERPT_CHARS", "123") monkeypatch.setenv("TENDWIRE_MAX_WORKERS", "64") monkeypatch.setenv("TENDWIRE_MAX_OUTBOX_ATTEMPTS", "3") monkeypatch.setenv("TENDWIRE_CONNECTOR_CLAIM_TTL_SECONDS", "45") @@ -263,10 +216,8 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: env_config = load_config() explicit = load_config( - event_debounce_seconds="0.1", reconcile_interval_seconds="5", event_retention_days="2", - output_excerpt_chars="50", max_workers="9", max_outbox_attempts="4", connector_claim_ttl_seconds="15", @@ -276,10 +227,8 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: command_receipt_retention_seconds="691200", command_receipt_retention_count="12", ) - assert env_config.event_debounce_seconds == 0.25 assert env_config.reconcile_interval_seconds == 0 assert env_config.event_retention_days == 14 - assert env_config.output_excerpt_chars == 123 assert env_config.max_workers == 64 assert env_config.max_outbox_attempts == 3 assert env_config.connector_claim_ttl_seconds == 45 @@ -288,10 +237,8 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: assert env_config.command_retry_horizon_seconds == 120 assert env_config.command_receipt_retention_seconds == 691_200 assert env_config.command_receipt_retention_count == 99 - assert explicit.event_debounce_seconds == 0.1 assert explicit.reconcile_interval_seconds == 5 assert explicit.event_retention_days == 2 - assert explicit.output_excerpt_chars == 50 assert explicit.max_workers == 9 assert explicit.max_outbox_attempts == 4 assert explicit.connector_claim_ttl_seconds == 15 @@ -305,10 +252,8 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: @pytest.mark.parametrize( ("field", "value", "message"), [ - ("event_debounce_seconds", -0.1, "event_debounce_seconds must be non-negative"), ("reconcile_interval_seconds", -1, "reconcile_interval_seconds must be non-negative"), ("event_retention_days", 0, "event_retention_days must be >= 1"), - ("output_excerpt_chars", 0, "output_excerpt_chars must be >= 1"), ("max_workers", 0, "max_workers must be >= 1"), ("max_outbox_attempts", 0, "max_outbox_attempts must be >= 1"), ("connector_claim_ttl_seconds", 0, "connector_claim_ttl_seconds must be >= 1"), @@ -615,43 +560,7 @@ def test_socket_group_defaults_private_and_normalizes_without_lookup(monkeypatch assert load_config(socket_group=" ").socket_group is None -def test_herdr_backend_defaults_to_cli(monkeypatch) -> None: - monkeypatch.delenv("TENDWIRE_HERDR_BACKEND", raising=False) - - assert Config().herdr_backend == "cli" - assert load_config().herdr_backend == "cli" - - -@pytest.mark.parametrize( - ("raw", "expected"), - [ - ("cli", "cli"), - ("socket", "socket"), - (" CLI ", "cli"), - ("SOCKET", "socket"), - ], -) -def test_herdr_backend_accepts_explicit_values(monkeypatch, raw: str, expected: str) -> None: - monkeypatch.delenv("TENDWIRE_HERDR_BACKEND", raising=False) - - assert Config(herdr_backend=raw).herdr_backend == expected - assert load_config(herdr_backend=raw).herdr_backend == expected - - -def test_herdr_backend_reads_environment(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_HERDR_BACKEND", "socket") - - assert load_config().herdr_backend == "socket" - - -def test_herdr_backend_invalid_value_fails_clearly(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_HERDR_BACKEND", "events") - - with pytest.raises(ValueError, match="herdr_backend must be one of: cli, socket"): - load_config() - - -def test_cli_default_import_does_not_load_socket_event_backend() -> None: +def test_cli_default_import_does_not_load_socket_transport() -> None: code = """ import sys before = set(sys.modules) @@ -659,7 +568,6 @@ def test_cli_default_import_does_not_load_socket_event_backend() -> None: loaded = set(sys.modules) - before for name in sorted(loaded): if name in { - "tendwire.backends.herdr_events", "tendwire.backends.herdr_socket", "tendwire.backends.herdr_protocol", }: diff --git a/tests/test_connector_daemon_cli.py b/tests/test_connector_daemon_cli.py index 09ef2e2..196fd24 100644 --- a/tests/test_connector_daemon_cli.py +++ b/tests/test_connector_daemon_cli.py @@ -215,122 +215,6 @@ def reject_write(*_args: Any, **_kwargs: Any) -> None: daemon._connector_periodic_tick() -def test_cli_connector_poll_and_ack_print_json_only(tmp_path: Path, capsys) -> None: - db_path = tmp_path / "cli-connector.db" - _enqueue(db_path) - poll_code = main( - [ - "--host-id", - "host-a", - "connector", - "poll", - "--name", - "attention", - "--db-path", - str(db_path), - "--lease-seconds", - "60", - ] - ) - poll_captured = capsys.readouterr() - poll_payload = json.loads(poll_captured.out) - ref = poll_payload["items"][0]["ref"] - - ack_code = main( - [ - "--host-id", - "host-a", - "connector", - "ack", - "--name", - "attention", - "--ref", - ref, - "--response-json", - json.dumps({"safe": "kept", "chat_id": "must-strip", "provider": "telegram"}), - "--db-path", - str(db_path), - ] - ) - ack_captured = capsys.readouterr() - ack_payload = json.loads(ack_captured.out) - - assert poll_code == 0 - assert ack_code == 0 - assert poll_captured.err == "" - assert ack_captured.err == "" - assert poll_payload["items"][0]["payload"]["safe"] == "kept" - assert ack_payload["status"] == "acknowledged" - _assert_json_only_and_safe(poll_payload) - _assert_json_only_and_safe(ack_payload) - - -def test_cli_connector_renew_and_release_forward_live_ref_options( - tmp_path: Path, - capsys, -) -> None: - db_path = tmp_path / "cli-renew-release.db" - _enqueue(db_path, key="cli-renew-release") - poll_code = main( - [ - "--host-id", - "host-a", - "connector", - "poll", - "--name", - "attention", - "--db-path", - str(db_path), - ] - ) - poll_payload = json.loads(capsys.readouterr().out) - ref = poll_payload["items"][0]["ref"] - - renew_code = main( - [ - "--host-id", - "host-a", - "connector", - "renew", - "--name", - "attention", - "--ref", - ref, - "--lease-seconds", - "120", - "--db-path", - str(db_path), - ] - ) - renew_capture = capsys.readouterr() - renew_payload = json.loads(renew_capture.out) - release_code = main( - [ - "--host-id", - "host-a", - "connector", - "release", - "--name", - "attention", - "--ref", - ref, - "--db-path", - str(db_path), - ] - ) - release_capture = capsys.readouterr() - release_payload = json.loads(release_capture.out) - - assert poll_code == renew_code == release_code == 0 - assert renew_capture.err == release_capture.err == "" - assert renew_payload["status"] == "renewed" - assert renew_payload["leased_until"] - assert release_payload["status"] == "released" - assert release_payload["available_at"] - _assert_json_only_and_safe(renew_payload) - _assert_json_only_and_safe(release_payload) - - def test_cli_connector_prepare_reads_bounded_action_from_stdin( tmp_path: Path, capsys, diff --git a/tests/test_daemon.py b/tests/test_daemon.py index 2f3cb2e..77b9578 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -22,7 +22,6 @@ import pytest from tendwire import __version__ -from tendwire.backends.herdr_socket import HerdrSocketTimeoutError from tendwire.cli import main from tendwire.config import Config from tendwire.core.commands import ( @@ -112,7 +111,27 @@ def _required_acp_supervisor_for_daemon_unit_tests( """Keep non-ACP daemon tests focused on their own boundary.""" class Supervisor: + def __init__(self, config: Config) -> None: + self.config = config + def start(self) -> None: + assert self.config.db_path is not None + init_store(self.config.db_path) + if latest_snapshot(self.config.db_path, self.config.host_id) is None: + save_snapshot( + self.config.db_path, + Snapshot( + host_id=self.config.host_id, + updated_at="2026-08-04T00:00:00+00:00", + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="empty_healthy", + ) + ], + ), + ) return None def stop(self, *, timeout: float) -> None: @@ -141,9 +160,9 @@ def init_with_required_acp(self: TendwireDaemon, *args: Any, **kwargs: Any) -> N object.__setattr__( hooks, "acp_supervisor_factory", - lambda _config, _stop: Supervisor(), + lambda config, _stop: Supervisor(config), ) - self._acp_supervisor = Supervisor() + self._acp_supervisor = Supervisor(self.config) monkeypatch.setattr(TendwireDaemon, "__init__", init_with_required_acp) @@ -661,7 +680,9 @@ def test_malformed_durable_snapshot_is_fixed_unavailable_for_daemon_and_cli( } assert cli_code == 1 - assert daemon_payload == cli_payload == expected + assert daemon_payload == expected + assert cli_payload["ok"] is False + assert cli_payload["status"] == "daemon_unavailable" assert "sentinel-private" not in json.dumps( {"daemon": daemon_payload, "cli": cli_payload}, sort_keys=True, @@ -1319,127 +1340,13 @@ def test_attention_recurrence_after_two_complete_misses_re_notifies(tmp_path: Pa assert generation == 2 -def test_socket_daemon_synthesized_fallback_has_no_lifecycle_authority( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "socket-fallback.db" - config = Config(host_id="socket-fallback-host", db_path=db_path, herdr_backend="socket") - base = datetime(2026, 1, 1, tzinfo=timezone.utc) - save_snapshot( - db_path, - project_from_raw( - config, - workers=_blocked_worker("blocked"), - backend_health=_HEALTHY_BACKEND, - timestamp=base, - ), - observation=_complete_observation(base), - ) - - class _HealthyState: - def to_backend_health(self) -> BackendHealth: - return BackendHealth( - name="herdr", - status="healthy", - outcome="empty_healthy", - observed_at=(base + timedelta(seconds=300)).isoformat(), - ) - - class _Backend: - health = _HealthyState() - - def start(self, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - - def stop(self) -> None: - pass - - backend = _Backend() - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(event_backend_factory=lambda _config, _stop_event: backend), - ) - monkeypatch.setattr("tendwire.store.sqlite.latest_snapshot", lambda _path, _host_id: None) - - fallback = daemon._start_socket_event_backend() - - assert fallback.attention == [] - assert len(attention_payload_from_store(db_path, config.host_id)["attention"]) == 1 - assert _attention_outbox_count(db_path, config.host_id) == 1 - with sqlite3.connect(str(db_path)) as conn: - missing_count = conn.execute( - "SELECT missing_observation_count FROM attention_lifecycles WHERE host_id = ?", - (config.host_id,), - ).fetchone()[0] - assert missing_count == 0 - - -@pytest.mark.parametrize( - "observed_at", - ["not-a-timestamp", "2026-01-01T00:00:00"], -) -def test_socket_daemon_fallback_drops_unordered_health_timestamp( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - observed_at: str, -) -> None: - config = Config( - host_id="socket-fallback-invalid-time", - db_path=tmp_path / "socket-fallback-invalid-time.db", - herdr_backend="socket", - ) - captured: list[SnapshotObservationContext] = [] - - class _HealthState: - def to_backend_health(self) -> BackendHealth: - return BackendHealth( - name="herdr", - status="healthy", - outcome="empty_healthy", - observed_at=observed_at, - ) - - class _Backend: - health = _HealthState() - - def start(self, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - - def _capture_save( - _db_path: Path, - _snapshot: Snapshot, - *, - turn_model: str, - observation: SnapshotObservationContext, - ) -> None: - assert turn_model == "observed" - captured.append(observation) - - backend = _Backend() - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(event_backend_factory=lambda _config, _stop_event: backend), - ) - monkeypatch.setattr("tendwire.store.sqlite.latest_snapshot", lambda _path, _host_id: None) - monkeypatch.setattr("tendwire.store.sqlite.save_snapshot", _capture_save) - - daemon._start_socket_event_backend() - - assert len(captured) == 1 - assert captured[0].authority == "none" - assert captured[0].observed_at is None - - def test_daemon_health_exposes_public_operational_status_without_private_values(tmp_path: Path) -> None: db_path = tmp_path / "health.db" config = Config( host_id="health-host", db_path=db_path, - event_debounce_seconds=0.2, reconcile_interval_seconds=0, event_retention_days=3, - output_excerpt_chars=80, max_workers=8, max_outbox_attempts=4, connector_claim_ttl_seconds=33, @@ -1547,10 +1454,8 @@ def test_daemon_health_exposes_public_operational_status_without_private_values( "backlog": False, } assert health["limits"] == { - "event_debounce_seconds": 0.2, "reconcile_interval_seconds": 0, "event_retention_days": 3, - "output_excerpt_chars": 80, "max_workers": 8, "max_outbox_attempts": 4, "outbox_claim_ttl_seconds": 33, @@ -2150,22 +2055,6 @@ def capture_signal(signum: int, handler: Any) -> Any: ] -@_UNIX_SOCKET_TEST -def test_service_group_sigterm_exits_closes_socket_and_reaps_child() -> None: - root = Path(__file__).resolve().parents[1] - result = subprocess.run( - ["bash", str(root / "scripts/tendwired_lifecycle_smoke.sh")], - check=False, - capture_output=True, - text=True, - timeout=15, - env={**os.environ, "PYTHON": sys.executable}, - ) - - assert result.returncode == 0, result.stderr - assert result.stdout == "tendwired lifecycle smoke: ok\n" - - def _socket_mode(path: Path) -> int: return stat.S_IMODE(os.lstat(path).st_mode) @@ -2287,10 +2176,10 @@ def test_cli_snapshot_barrier_checks_maintenance_once_and_reads_do_not( tuple[Path, Any, str | None, int | None, int, int, int, int, int, int] ] = [] - def observe(_config: Config) -> Snapshot: + def initialize(path: Path) -> None: + init_store(path) snapshot = _public_snapshot() save_snapshot(db_path, snapshot) - return snapshot def maintenance( path: Path, @@ -2349,7 +2238,7 @@ def maintenance( ) daemon = TendwireDaemon( config, - hooks=DaemonHooks(init_store=init_store, observe_initial_snapshot=observe), + hooks=DaemonHooks(init_store=initialize), ) try: daemon.start() @@ -2423,10 +2312,10 @@ def test_cli_snapshot_persists_when_automatic_maintenance_fails( config = Config(host_id="daemon-host", data_dir=data_dir, db_path=db_path) calls = 0 - def observe(_config: Config) -> Snapshot: + def initialize(path: Path) -> None: + init_store(path) snapshot = _public_snapshot() save_snapshot(db_path, snapshot) - return snapshot def maintenance_failure(*_args: Any, **_kwargs: Any) -> dict[str, Any]: nonlocal calls @@ -2439,7 +2328,7 @@ def maintenance_failure(*_args: Any, **_kwargs: Any) -> dict[str, Any]: ) daemon = TendwireDaemon( config, - hooks=DaemonHooks(init_store=init_store, observe_initial_snapshot=observe), + hooks=DaemonHooks(init_store=initialize), ) try: daemon.start() @@ -2493,10 +2382,7 @@ def test_daemon_default_socket_parent_and_endpoint_are_private_under_umask_zero( ) daemon = TendwireDaemon( config, - hooks=DaemonHooks( - init_store=lambda _path: None, - observe_initial_snapshot=lambda _config: _public_snapshot(), - ), + hooks=DaemonHooks(init_store=lambda _path: None), ) try: @@ -2524,7 +2410,7 @@ def test_daemon_startup_repairs_all_existing_state_before_empty_observation( data_dir.mkdir() os.chmod(data_dir, 0o755) db_path = data_dir / "daemon.db" - db_path.write_bytes(b"existing-database") + init_store(db_path) os.chmod(db_path, 0o644) config = Config(host_id="daemon-host", data_dir=data_dir, db_path=db_path) identity_paths = ( @@ -2535,29 +2421,16 @@ def test_daemon_startup_repairs_all_existing_state_before_empty_observation( for path in identity_paths: path.write_bytes(b"existing-identity") os.chmod(path, 0o644) - observations: list[Snapshot] = [] - def initialize_store(path: Path) -> None: assert path == db_path assert _socket_mode(data_dir) == 0o700 assert _socket_mode(db_path) == 0o600 assert all(_socket_mode(identity_path) == 0o600 for identity_path in identity_paths) - def observe(_config: Config) -> Snapshot: - snapshot = Snapshot( - host_id="daemon-host", - updated_at="2026-01-01T00:00:00+00:00", - ) - observations.append(snapshot) - return snapshot - for _attempt in range(2): daemon = TendwireDaemon( config, - hooks=DaemonHooks( - init_store=initialize_store, - observe_initial_snapshot=observe, - ), + hooks=DaemonHooks(init_store=initialize_store), ) try: daemon.start() @@ -2572,7 +2445,6 @@ def observe(_config: Config) -> Snapshot: finally: daemon.stop() - assert len(observations) == 2 assert not os.path.lexists(data_dir / "tendwire.sock") @@ -2598,16 +2470,9 @@ def initialize_store(_path: Path) -> None: hook_calls.append("init_store") raise AssertionError("store hook must not run") - def observe(_config: Config) -> Snapshot: - hook_calls.append("observe") - raise AssertionError("observation hook must not run") - daemon = TendwireDaemon( Config(host_id="daemon-host", data_dir=data_dir, db_path=db_path), - hooks=DaemonHooks( - init_store=initialize_store, - observe_initial_snapshot=observe, - ), + hooks=DaemonHooks(init_store=initialize_store), ) with pytest.raises(LocalStateError) as caught: @@ -2698,10 +2563,7 @@ def test_daemon_group_socket_and_client_use_exact_shared_mode( ) daemon = TendwireDaemon( config, - hooks=DaemonHooks( - init_store=lambda _path: None, - observe_initial_snapshot=lambda _config: _public_snapshot(), - ), + hooks=DaemonHooks(init_store=lambda _path: None), ) thread: threading.Thread | None = None @@ -2976,10 +2838,7 @@ def test_daemon_rejects_group_sharing_on_implicit_private_parent_before_mutation db_path=tmp_path / "daemon.db", socket_group=group_name, ), - hooks=DaemonHooks( - init_store=lambda _path: None, - observe_initial_snapshot=lambda _config: _public_snapshot(), - ), + hooks=DaemonHooks(init_store=lambda _path: None), ) with pytest.raises(DaemonUnavailable) as caught: @@ -3382,10 +3241,6 @@ def test_daemon_active_socket_fails_before_store_or_backend_work(tmp_path: Path) def forbidden_store(_path: Path) -> None: calls.append("init_store") - def forbidden_observe(_config: Config) -> Snapshot: - calls.append("observe") - raise AssertionError("live-socket guard must precede backend work") - daemon = TendwireDaemon( Config( host_id="fail-fast-host", @@ -3393,10 +3248,7 @@ def forbidden_observe(_config: Config) -> Snapshot: db_path=tmp_path / "fail-fast.db", socket_path=socket_path, ), - hooks=DaemonHooks( - init_store=forbidden_store, - observe_initial_snapshot=forbidden_observe, - ), + hooks=DaemonHooks(init_store=forbidden_store), ) try: with pytest.raises(DaemonUnavailable) as caught: @@ -3655,40 +3507,26 @@ def test_unix_socket_server_close_preserves_substituted_socket(tmp_path: Path) - @_UNIX_SOCKET_TEST -@pytest.mark.parametrize("failure_stage", ["init_store", "observe"]) -def test_daemon_startup_failure_never_publishes_socket( - tmp_path: Path, - failure_stage: str, -) -> None: - socket_path = tmp_path / f"{failure_stage}.sock" +def test_daemon_store_startup_failure_never_publishes_socket(tmp_path: Path) -> None: + socket_path = tmp_path / "init-store.sock" def assert_unpublished() -> None: assert not os.path.lexists(socket_path) def initialize_store(path: Path) -> None: assert_unpublished() - if failure_stage == "init_store": - raise RuntimeError("sentinel startup failure") - init_store(path) - - def observe(_config: Config) -> Snapshot: - assert_unpublished() - if failure_stage == "observe": - raise RuntimeError("sentinel startup failure") - return _public_snapshot() + del path + raise RuntimeError("sentinel startup failure") config = Config( host_id="daemon-host", data_dir=tmp_path, - db_path=tmp_path / f"{failure_stage}.db", + db_path=tmp_path / "init-store.db", socket_path=socket_path, ) daemon = TendwireDaemon( config, - hooks=DaemonHooks( - init_store=initialize_store, - observe_initial_snapshot=observe, - ), + hooks=DaemonHooks(init_store=initialize_store), ) try: @@ -3705,159 +3543,13 @@ def observe(_config: Config) -> Snapshot: @_UNIX_SOCKET_TEST -def test_daemon_backend_start_failure_stops_backend_without_publishing_socket( - tmp_path: Path, -) -> None: - socket_path = tmp_path / "backend-failure.sock" - - def assert_unpublished() -> None: - assert not os.path.lexists(socket_path) - - class FailingEventBackend: - def __init__(self) -> None: - self.started = False - self.stopped = False - - def start(self, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - assert_unpublished() - self.started = True - raise RuntimeError("sentinel backend startup failure") - - def stop(self) -> None: - self.stopped = True - - backend = FailingEventBackend() - - def initialize_store(path: Path) -> None: - assert_unpublished() - init_store(path) - - def event_backend_factory(_config: Config, _stop_event: threading.Event) -> Any: - assert_unpublished() - return backend - - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=tmp_path / "backend-failure.db", - socket_path=socket_path, - herdr_backend="socket", - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - init_store=initialize_store, - event_backend_factory=event_backend_factory, - ), - ) - - try: - with pytest.raises(RuntimeError, match="sentinel backend startup failure") as caught: - daemon.start() - - assert backend.started is True - assert backend.stopped is True - _assert_private_daemon_failure(caught.value, socket_path) - assert daemon.server is None - assert not os.path.lexists(socket_path) - finally: - daemon.stop() - - -@_UNIX_SOCKET_TEST -def test_daemon_backend_timeout_never_reaches_acp_startup(tmp_path: Path) -> None: - calls: list[str] = [] - - class TimedOutEventBackend: - def start(self, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - calls.append("backend_start") - raise HerdrSocketTimeoutError("initial Herdr reconciliation timed out") - - def stop(self) -> None: - calls.append("backend_stop") - - def forbidden_acp_factory(_config: Config, _stop_event: threading.Event) -> Any: - calls.append("acp_factory") - raise AssertionError("ACP startup must not follow a Herdr readiness timeout") - - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=tmp_path / "backend-timeout.db", - socket_path=tmp_path / "backend-timeout.sock", - herdr_backend="socket", - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - event_backend_factory=lambda _config, _stop_event: TimedOutEventBackend(), - acp_supervisor_factory=forbidden_acp_factory, - ), - ) - - try: - with pytest.raises( - HerdrSocketTimeoutError, - match="initial Herdr reconciliation timed out", - ): - daemon.start() - - assert calls == ["backend_start", "backend_stop"] - assert daemon.server is None - assert not os.path.lexists(config.socket_path) - finally: - daemon.stop() - - -@_UNIX_SOCKET_TEST -def test_daemon_default_backend_keeps_startup_and_rpc_timeouts_distinct( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - captured: list[tuple[float, float]] = [] - - def time_out_start(self: Any, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - captured.append( - ( - self.config.herdr_timeout_seconds, - self.config.herdr_initial_reconcile_timeout_seconds, - ) - ) - raise HerdrSocketTimeoutError("initial Herdr reconciliation timed out") - - monkeypatch.setattr( - "tendwire.backends.herdr_events.HerdrEventBackend.start", - time_out_start, - ) - config = Config( - host_id="daemon-distinct-timeouts", - data_dir=tmp_path, - db_path=tmp_path / "daemon-distinct-timeouts.db", - socket_path=tmp_path / "daemon-distinct-timeouts.sock", - herdr_backend="socket", - herdr_timeout_seconds=0.25, - herdr_initial_reconcile_timeout_seconds=17, - ) - daemon = TendwireDaemon(config) - - try: - with pytest.raises(HerdrSocketTimeoutError): - daemon.start() - assert captured == [(0.25, 17.0)] - finally: - daemon.stop() - - -@_UNIX_SOCKET_TEST -def test_daemon_starts_observes_persists_serves_and_removes_socket(tmp_path: Path) -> None: +def test_daemon_starts_persists_serves_and_removes_socket(tmp_path: Path) -> None: db_path = tmp_path / "daemon.db" socket_path = tmp_path / "daemon.sock" config = Config(host_id="daemon-host", data_dir=tmp_path, db_path=db_path, socket_path=socket_path) - def observe(config: Config) -> Snapshot: + def initialize(path: Path) -> None: + init_store(path) snapshot = project_from_raw( config, workers=[{"id": "worker-1", "name": "Worker One", "status": "active"}], @@ -3872,11 +3564,10 @@ def observe(config: Config) -> Snapshot: ], ) save_snapshot(db_path, snapshot) - return snapshot daemon = TendwireDaemon( config, - hooks=DaemonHooks(observe_initial_snapshot=observe), + hooks=DaemonHooks(init_store=initialize), ) daemon.start() thread = threading.Thread(target=daemon.serve_forever) @@ -4336,7 +4027,6 @@ def test_daemon_command_submit_rejects_blank_request_id_before_mutation( host_id="cmd-host", data_dir=tmp_path, db_path=db_path, - herdr_backend="socket", ) init_store(db_path) calls: list[str] = [] @@ -4368,7 +4058,7 @@ def test_daemon_command_submit_rejects_blank_request_id_before_mutation( assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 0 assert conn.execute("SELECT COUNT(*) FROM events").fetchone()[0] == 0 -def test_cli_snapshot_falls_back_when_configured_socket_is_absent( +def test_cli_snapshot_requires_daemon_when_configured_socket_is_absent( tmp_path: Path, capsys, monkeypatch, @@ -4379,11 +4069,6 @@ def test_cli_snapshot_falls_back_when_configured_socket_is_absent( monkeypatch.setenv("TENDWIRE_DATA_DIR", os.fspath(data_dir)) monkeypatch.delenv("TENDWIRE_DB_PATH", raising=False) - def fake_state(config: Config) -> tuple[list[Any], list[Worker]]: - return [], [Worker(id="fallback-worker", name="Fallback", status="active")] - - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", fake_state) - code = main( [ "--host-id", @@ -4397,9 +4082,11 @@ def fake_state(config: Config) -> tuple[list[Any], list[Worker]]: captured = capsys.readouterr() payload = json.loads(captured.out) - assert code == 0 + assert code == 1 assert captured.err == "" - assert payload["workers"][0]["id"] == "fallback-worker" + assert payload["ok"] is False + assert payload["status"] == "daemon_unavailable" + assert not (data_dir / "tendwire.db").exists() def test_cli_command_falls_back_when_configured_socket_is_stale( @@ -5118,15 +4805,7 @@ def fd_targets() -> dict[str, tuple[str, int, int, int]]: baseline_threads = {id(thread) for thread in threading.enumerate()} baseline_children = direct_child_processes() main_identity = (db_path.stat().st_dev, db_path.stat().st_ino) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - observe_initial_snapshot=lambda _config: latest_snapshot( - db_path, - config.host_id, - ), - ), - ) + daemon = TendwireDaemon(config) server_thread: threading.Thread | None = None writer_thread = threading.Thread(target=churn_wal) writer_started = False diff --git a/tests/test_daemon_acp.py b/tests/test_daemon_acp.py index f672006..97aef58 100644 --- a/tests/test_daemon_acp.py +++ b/tests/test_daemon_acp.py @@ -73,7 +73,6 @@ def _config(tmp_path: Path) -> Config: data_dir=tmp_path, db_path=tmp_path / "daemon.db", socket_path=tmp_path / "daemon.sock", - herdr_backend="cli", acp_shutdown_timeout_seconds=1.25, ) @@ -88,16 +87,11 @@ def _hooks( def initialize(path: Path) -> None: calls.append("init_store") init_store(path) - - def observe(_config: Config) -> Snapshot: - calls.append("observe") snapshot = _snapshot() save_snapshot(config.db_path, snapshot) - return snapshot return DaemonHooks( init_store=initialize, - observe_initial_snapshot=observe, acp_supervisor_factory=supervisor_factory, ) @@ -111,7 +105,7 @@ def test_daemon_requires_an_acp_supervisor_before_binding_socket(tmp_path: Path) daemon.start() assert not config.socket_path.exists() - assert calls == ["init_store", "observe"] + assert calls == ["init_store"] def test_daemon_starts_required_acp_and_exposes_only_public_health( @@ -136,7 +130,7 @@ def test_daemon_starts_required_acp_and_exposes_only_public_health( assert health["acp"]["state"] == "running" assert health["acp"]["counters"]["updates_ingested"] == 7 assert "sentinel-private" not in json.dumps(health) - assert calls[:3] == ["init_store", "observe", "acp_start"] + assert calls[:2] == ["init_store", "acp_start"] finally: daemon.stop() diff --git a/tests/test_diagnostics.py b/tests/test_diagnostics.py deleted file mode 100644 index dc3e034..0000000 --- a/tests/test_diagnostics.py +++ /dev/null @@ -1,1394 +0,0 @@ -"""Tests for the read-only Herdr doctor diagnostics.""" - -from __future__ import annotations - -import json -import os -import subprocess -import socket -import sqlite3 -import stat -from collections.abc import Sequence -from pathlib import Path -from typing import Any - -import pytest - -from tendwire.backends import herdr_cli -from tendwire.backends.herdr_cli import diagnose_herdr, fetch_herdr_state -from tendwire.cli import main -from tendwire.config import Config, load_config -from tendwire.local_state import ( - ConfigStateReport, - LocalStateErrorCode, - LocalStateIssue, - LocalStateKind, - PermissionResult, - PermissionState, -) -from tendwire.store.sqlite import init_store - - -FIXTURES = Path(__file__).parent / "fixtures" / "herdr" - -_FORBIDDEN_TEXT = ( - "telegram", - "chat_id", - "topic_id", - "message_id", - "thread_id", - "argv", - "token", - "bot_token", - "delivery", - "route", - "backend_target", -) - - -def _fixture(name: str) -> str: - return (FIXTURES / name).read_text(encoding="utf-8") - - -def _completed(args: Sequence[str], stdout: str = "", stderr: str = "", returncode: int = 0) -> subprocess.CompletedProcess[str]: - return subprocess.CompletedProcess( - args=["herdr", *args], - returncode=returncode, - stdout=stdout, - stderr=stderr, - ) - - -def _doctor_outcomes(payload: dict[str, Any]) -> dict[str, str]: - return {str(check["name"]): str(check["outcome"]) for check in payload["checks"]} - -_HERDR_CHECK_NAMES = frozenset( - { - "workspace_list", - "workspace_list_json", - "agent_list", - "agent_list_json", - "pane_list", - "pane_list_json", - } -) -_LOCAL_STATE_CHECK_NAMES = ( - "state_directory_permissions", - "database_permissions", - "identity_permissions", - "daemon_socket_permissions", -) - - -def _herdr_checks(payload: dict[str, Any]) -> list[dict[str, Any]]: - return [ - check - for check in payload["checks"] - if str(check["name"]) in _HERDR_CHECK_NAMES - ] - - -def _local_state_checks(payload: dict[str, Any]) -> dict[str, dict[str, Any]]: - return { - str(check["name"]): check - for check in payload["checks"] - if str(check["name"]) in _LOCAL_STATE_CHECK_NAMES - } - - -def _maintenance_check(payload: dict[str, Any]) -> dict[str, Any]: - matches = [ - check - for check in payload["checks"] - if check.get("name") == "store_maintenance" - ] - assert len(matches) == 1 - return matches[0] -def _pending_check(payload: dict[str, Any]) -> dict[str, Any]: - matches = [ - check - for check in payload["checks"] - if check.get("name") == "pending_ingestion" - ] - assert len(matches) == 1 - return matches[0] - - - - -def _patch_healthy_herdr(monkeypatch, calls: list[tuple[str, ...]] | None = None) -> None: - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - if calls is not None: - calls.append(tuple(args[1:])) - return _completed(args[1:], stdout='{"items": []}') - - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - -def _write_mode(path: Path, mode: int) -> None: - path.write_bytes(b"local-state") - os.chmod(path, mode) - - -def test_doctor_reports_missing_herdr_binary(monkeypatch, tmp_path: Path) -> None: - config = Config( - host_id="testhost", - herdr_bin="missing-herdr", - data_dir=tmp_path / "state", - ) - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: None) - - payload = diagnose_herdr(config) - - assert payload["status"] == "unavailable" - assert _doctor_outcomes(payload)["workspace_list"] == "missing_binary" - assert all(check["ok"] is False for check in _herdr_checks(payload)) - - -def test_doctor_reports_timeout_and_skips_remaining_checks(monkeypatch, tmp_path: Path) -> None: - config = Config( - host_id="testhost", - herdr_bin="herdr", - herdr_timeout_seconds=0.25, - data_dir=tmp_path / "state", - ) - calls: list[tuple[str, ...]] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args[1:])) - assert kwargs["timeout"] == 0.25 - raise subprocess.TimeoutExpired(cmd=args, timeout=kwargs["timeout"]) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - payload = diagnose_herdr(config) - outcomes = _doctor_outcomes(payload) - - assert payload["status"] == "timeout" - assert payload["aggregate_deadline_seconds"] == 1.5 - assert calls == [("workspace", "list")] - assert outcomes["workspace_list"] == "timeout" - assert outcomes["agent_list"] == "skipped_after_timeout" - - -def test_doctor_distinguishes_nonzero_malformed_empty_and_nonempty(monkeypatch, tmp_path: Path) -> None: - config = Config( - host_id="testhost", - herdr_bin="herdr", - data_dir=tmp_path / "state", - ) - responses = { - ("workspace", "list"): _completed( - ["workspace", "list"], - stderr=_fixture("nonzero_stderr.txt"), - returncode=2, - ), - ("workspace", "list", "--json"): _completed( - ["workspace", "list", "--json"], - stdout=_fixture("workspace_list_json_empty.json"), - ), - ("agent", "list"): _completed( - ["agent", "list"], - stdout=_fixture("malformed.txt"), - ), - ("agent", "list", "--json"): _completed( - ["agent", "list", "--json"], - stdout=_fixture("agent_list_no_flag_nonempty.json"), - ), - ("pane", "list"): _completed( - ["pane", "list"], - stdout=_fixture("pane_list_empty.json"), - ), - } - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - return responses[tuple(args[1:])] - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - payload = diagnose_herdr(config) - outcomes = _doctor_outcomes(payload) - - assert payload["status"] == "degraded" - assert outcomes["workspace_list"] == "nonzero" - assert outcomes["workspace_list_json"] == "empty_healthy" - assert outcomes["agent_list"] == "malformed_json" - assert outcomes["agent_list_json"] == "healthy_non_empty" - assert outcomes["pane_list"] == "empty_healthy" - assert "herdr fixture error" in json.dumps(payload) - - -def test_doctor_cli_outputs_json_only_and_sanitizes_samples(capsys, monkeypatch, tmp_path: Path) -> None: - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - return _completed(args[1:], stdout="not json token=must-not-leak", returncode=0) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "state")) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(tmp_path / "state" / "tendwire.db")) - - code = main(["--host-id", "doctor-host", "--herdr-bin", "herdr", "doctor", "--json"]) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["schema_version"] == 1 - assert payload["command"] == "doctor" - serialized = json.dumps(payload).lower() - assert not any(forbidden in serialized for forbidden in _FORBIDDEN_TEXT) - - -def test_cli_herdr_timeout_knob_is_used_by_doctor(capsys, monkeypatch, tmp_path: Path) -> None: - timeouts: list[float] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - timeouts.append(float(kwargs["timeout"])) - return _completed(args[1:], stdout=_fixture("pane_list_empty.json")) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "state")) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(tmp_path / "state" / "tendwire.db")) - - code = main(["--herdr-timeout", "0.75", "--herdr-bin", "herdr", "doctor", "--json"]) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert timeouts == [0.75, 0.75, 0.75] - assert payload["timeout_seconds"] == 0.75 - assert payload["aggregate_deadline_seconds"] == 4.5 - assert all("aggregate_deadline_seconds" in check for check in _herdr_checks(payload)) - assert _pending_check(payload)["outcome"] == "store_unavailable" - - -def test_doctor_appends_fixed_compliant_local_state_checks( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "state" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "tendwire.db", - socket_path=state_dir / "tendwire.sock", - ) - assert config.db_path is not None - init_store(config.db_path) - for identity_path in ( - config.installation_key_path, - config.installation_key_marker_path, - config.installation_key_sentinel_path, - ): - _write_mode(identity_path, 0o600) - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(config.socket_path)) - assert config.socket_path is not None - os.chmod(config.socket_path, 0o600) - calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, calls) - try: - payload = diagnose_herdr(config) - finally: - listener.close() - config.socket_path.unlink(missing_ok=True) - - assert set(payload) == { - "schema_version", - "command", - "herdr_bin", - "timeout_seconds", - "aggregate_deadline_seconds", - "status", - "checks", - } - assert payload["schema_version"] == 1 - assert payload["command"] == "doctor" - assert payload["status"] == "degraded" - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - assert len(_herdr_checks(payload)) == 6 - local_checks = _local_state_checks(payload) - assert tuple(local_checks) == _LOCAL_STATE_CHECK_NAMES - assert all( - check == { - "name": name, - "ok": True, - "outcome": "compliant", - "remediation": "No action required.", - } - for name, check in local_checks.items() - ) - assert _maintenance_check(payload)["outcome"] == "overdue" - - -def test_doctor_treats_uninitialized_state_and_stopped_socket_as_neutral( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "never-created-state" - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "never-created.db", - ) - assert config.socket_path is None - calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, calls) - - payload = diagnose_herdr(config) - - assert payload["status"] == "degraded" - assert not state_dir.exists() - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - local_checks = _local_state_checks(payload) - assert { - name: (check["ok"], check["outcome"], check["remediation"]) - for name, check in local_checks.items() - } == { - "state_directory_permissions": ( - True, - "not_initialized", - "No action required while local state is uninitialized.", - ), - "database_permissions": ( - True, - "not_initialized", - "No action required while local state is uninitialized.", - ), - "identity_permissions": ( - True, - "not_initialized", - "No action required while local state is uninitialized.", - ), - "daemon_socket_permissions": ( - True, - "not_running", - "No action required while the daemon is stopped.", - ), - } - assert _maintenance_check(payload) == { - "name": "store_maintenance", - "ok": True, - "outcome": "not_initialized", - "remediation": "No action required while the store is uninitialized.", - "snapshot_retention_days": 14, - "snapshot_retention_count": 4096, - "maintenance_batch_size": 100, - "maintenance_cadence_seconds": 3600, - "snapshot_count": 0, - "last_completed_at": None, - } - assert _pending_check(payload) == { - "name": "pending_ingestion", - "ok": False, - "outcome": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - "stale_grace_seconds": 30.0, - } - - -def test_doctor_inspects_broad_default_socket_without_mutating_or_disclosing_path( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "s" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - default_socket = state_dir / "tendwire.sock" - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(default_socket)) - os.chmod(default_socket, 0o666) - before = ( - os.lstat(default_socket).st_ino, - stat.S_IMODE(os.lstat(default_socket).st_mode), - ) - config = Config(host_id="doctor-host", herdr_bin="herdr", data_dir=state_dir) - assert config.socket_path is None - calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, calls) - try: - payload = diagnose_herdr(config) - after = ( - os.lstat(default_socket).st_ino, - stat.S_IMODE(os.lstat(default_socket).st_mode), - ) - finally: - listener.close() - default_socket.unlink(missing_ok=True) - - assert payload["status"] == "degraded" - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - assert after == before - assert _local_state_checks(payload)["daemon_socket_permissions"] == { - "name": "daemon_socket_permissions", - "ok": False, - "outcome": "repair_required", - "remediation": "Restart Tendwire to repair local state permissions.", - } - serialized = json.dumps(payload) - assert str(state_dir) not in serialized - assert str(default_socket) not in serialized - assert "tendwire.sock" not in serialized - - -def test_doctor_rejects_wrong_type_default_socket_without_mutating_or_disclosing_path( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "wrong-type-default-socket-state-value" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - default_socket = state_dir / "tendwire.sock" - private_contents = b"wrong-default-socket-content-value" - default_socket.write_bytes(private_contents) - os.chmod(default_socket, 0o600) - before = ( - os.lstat(default_socket).st_ino, - stat.S_IMODE(os.lstat(default_socket).st_mode), - ) - config = Config(host_id="doctor-host", herdr_bin="herdr", data_dir=state_dir) - assert config.socket_path is None - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - after = ( - os.lstat(default_socket).st_ino, - stat.S_IMODE(os.lstat(default_socket).st_mode), - ) - assert payload["status"] == "degraded" - assert after == before - assert default_socket.read_bytes() == private_contents - assert _local_state_checks(payload)["daemon_socket_permissions"] == { - "name": "daemon_socket_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - serialized = json.dumps(payload) - assert str(state_dir) not in serialized - assert str(default_socket) not in serialized - assert private_contents.decode() not in serialized - - -def test_doctor_rejects_symlink_default_socket_without_following_or_disclosing_path( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "symlink-default-socket-state-value" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - target = state_dir / "default-socket-target-value" - private_contents = b"default-socket-target-content-value" - target.write_bytes(private_contents) - os.chmod(target, 0o600) - default_socket = state_dir / "tendwire.sock" - default_socket.symlink_to(target) - before_target = ( - os.lstat(target).st_ino, - stat.S_IMODE(os.lstat(target).st_mode), - ) - config = Config(host_id="doctor-host", herdr_bin="herdr", data_dir=state_dir) - assert config.socket_path is None - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - after_target = ( - os.lstat(target).st_ino, - stat.S_IMODE(os.lstat(target).st_mode), - ) - assert payload["status"] == "degraded" - assert default_socket.is_symlink() - assert after_target == before_target - assert target.read_bytes() == private_contents - assert _local_state_checks(payload)["daemon_socket_permissions"] == { - "name": "daemon_socket_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - serialized = json.dumps(payload) - for forbidden in ( - str(state_dir), - str(default_socket), - str(target), - private_contents.decode(), - ): - assert forbidden not in serialized - - -def test_doctor_reports_broad_modes_without_mutating_and_cli_exits_degraded( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "state" - state_dir.mkdir() - os.chmod(state_dir, 0o755) - db_path = state_dir / "tendwire.db" - identity_paths = ( - state_dir / "installation.key", - state_dir / "installation.key.sha256", - state_dir / "installation.key.initialized", - ) - for path in (db_path, *identity_paths): - _write_mode(path, 0o644) - socket_path = state_dir / "tendwire.sock" - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(socket_path)) - os.chmod(socket_path, 0o666) - observed_paths = (state_dir, db_path, *identity_paths, socket_path) - before = { - str(path): (os.lstat(path).st_ino, stat.S_IMODE(os.lstat(path).st_mode)) - for path in observed_paths - } - calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, calls) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(state_dir)) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(db_path)) - try: - code = main( - [ - "--herdr-bin", - "herdr", - "--socket-path", - str(socket_path), - "doctor", - "--json", - ] - ) - captured = capsys.readouterr() - finally: - listener.close() - socket_path.unlink(missing_ok=True) - - payload = json.loads(captured.out) - after = { - str(path): (os.lstat(path).st_ino, stat.S_IMODE(os.lstat(path).st_mode)) - for path in observed_paths - if path != socket_path - } - assert code == 1 - assert captured.err == "" - assert payload["status"] == "degraded" - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - assert after == {key: value for key, value in before.items() if key != str(socket_path)} - assert all( - check == { - "name": name, - "ok": False, - "outcome": "repair_required", - "remediation": "Restart Tendwire to repair local state permissions.", - } - for name, check in _local_state_checks(payload).items() - ) - assert _maintenance_check(payload)["outcome"] == "unsafe" - assert _maintenance_check(payload)["ok"] is False - - -def test_doctor_degrades_for_symlinks_and_wrong_entry_types_without_following( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "state-value-that-must-not-leak" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - target = state_dir / "symlink-target-that-must-not-leak" - _write_mode(target, 0o600) - db_path = state_dir / "database-value-that-must-not-leak" - db_path.symlink_to(target) - identity_path = state_dir / "installation.key" - identity_path.mkdir() - socket_path = state_dir / "socket-value-that-must-not-leak" - socket_path.write_bytes(b"wrong socket type") - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - socket_path=socket_path, - socket_group="group-value-that-must-not-leak", - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - local_checks = _local_state_checks(payload) - assert payload["status"] == "degraded" - assert local_checks["state_directory_permissions"]["outcome"] == "compliant" - assert local_checks["database_permissions"]["outcome"] == "unsafe" - assert local_checks["identity_permissions"]["outcome"] == "unsafe" - assert local_checks["daemon_socket_permissions"]["outcome"] == "unsafe" - assert db_path.is_symlink() - assert target.read_bytes() == b"local-state" - assert _maintenance_check(payload)["outcome"] == "unsafe" - assert _maintenance_check(payload)["ok"] is False - serialized = json.dumps(payload) - for forbidden in ( - str(state_dir), - str(db_path), - str(socket_path), - str(target), - "group-value-that-must-not-leak", - "wrong socket type", - ): - assert forbidden not in serialized - - -def test_store_maintenance_rejects_wrong_database_type_without_mutation( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "wrong-type-private-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "wrong-type-private-database" - db_path.mkdir(mode=0o700) - sentinel = db_path / "private-sentinel" - sentinel.write_text("wrong-type-private-content", encoding="utf-8") - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - before = (stat.S_IMODE(db_path.stat().st_mode), sentinel.read_bytes()) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - assert payload["status"] == "degraded" - assert _maintenance_check(payload)["outcome"] == "unsafe" - assert (stat.S_IMODE(db_path.stat().st_mode), sentinel.read_bytes()) == before - serialized = json.dumps(payload) - for private in ( - str(state_dir), - str(db_path), - str(sentinel), - "wrong-type-private-content", - ): - assert private not in serialized - - -def test_store_maintenance_refuses_outdated_schema_without_migration( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "outdated-private-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "outdated-private-store.db" - private_table = "private_schema_sentinel" - with sqlite3.connect(str(db_path)) as conn: - conn.execute(f"CREATE TABLE {private_table} (private_value TEXT)") - conn.execute( - f"INSERT INTO {private_table} (private_value) VALUES (?)", - ("outdated-private-content",), - ) - conn.execute("PRAGMA user_version = 1") - os.chmod(db_path, 0o600) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - before = db_path.read_bytes() - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - assert payload["status"] == "degraded" - assert _maintenance_check(payload)["outcome"] == "unavailable" - assert db_path.read_bytes() == before - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == 1 - assert conn.execute( - f"SELECT private_value FROM {private_table}" - ).fetchone()[0] == "outdated-private-content" - serialized = json.dumps(payload) - for private in ( - str(state_dir), - str(db_path), - private_table, - "outdated-private-content", - ): - assert private not in serialized - - -def _seed_maintenance_store( - config: Config, - *, - last_completed_at: str, - snapshot_count: int, -) -> tuple[bytes, str]: - assert config.db_path is not None - init_store(config.db_path) - private = "maintenance-private-payload-sentinel" - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - UPDATE store_maintenance_state - SET last_started_at = ?, - last_completed_at = ?, - last_status = 'ok' - WHERE scope = 'automatic' - """, - (last_completed_at, last_completed_at), - ) - for index in range(snapshot_count): - conn.execute( - """ - INSERT INTO snapshots ( - host_id, created_at, content_fingerprint, payload - ) VALUES (?, ?, ?, ?) - """, - ( - config.host_id, - f"2026-01-01T00:00:0{index}+00:00", - f"private-fingerprint-{index}", - json.dumps({"private": private, "index": index}), - ), - ) - return config.db_path.read_bytes(), private - - -def test_store_maintenance_reports_current_fixed_aggregate_without_mutation( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "current-private-state" - state_dir.mkdir(mode=0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "current-private-store.db", - snapshot_retention_days=21, - snapshot_retention_count=10, - snapshot_maintenance_batch_size=7, - store_maintenance_cadence_seconds=3600, - ) - before, private = _seed_maintenance_store( - config, - last_completed_at="2026-01-01T00:00:00+00:00", - snapshot_count=1, - ) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-01T00:30:00+00:00", - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - assert payload["status"] == "ok" - assert _maintenance_check(payload) == { - "name": "store_maintenance", - "ok": True, - "outcome": "ok", - "remediation": "No action required.", - "snapshot_retention_days": 21, - "snapshot_retention_count": 10, - "maintenance_batch_size": 7, - "maintenance_cadence_seconds": 3600, - "snapshot_count": 1, - "last_completed_at": "2026-01-01T00:00:00+00:00", - } - assert _pending_check(payload) == { - "name": "pending_ingestion", - "ok": True, - "outcome": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - "stale_grace_seconds": 30.0, - } - assert config.db_path is not None - assert config.db_path.read_bytes() == before - assert private not in json.dumps(payload) - - -def test_doctor_pending_ingestion_is_fixed_nonmutating_and_public_safe( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, -) -> None: - from tendwire.store import sqlite as store_sqlite - - state_dir = tmp_path / "pending-health-private-state" - state_dir.mkdir(mode=0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "pending-health-private.db", - pending_stale_grace_seconds=17, - ) - before, private = _seed_maintenance_store( - config, - last_completed_at="2026-01-01T00:00:00+00:00", - snapshot_count=1, - ) - herdr_calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, herdr_calls) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-01T00:30:00+00:00", - ) - durable_calls: list[tuple[Path, str]] = [] - - def durable_health(db_path: Path, host_id: str) -> dict[str, Any]: - durable_calls.append((db_path, host_id)) - return { - "status": "degraded", - "counts": {"fresh": 2, "stale": 1, "total": 3}, - "pane_id": "sentinel-private-pane", - "source_path": str(tmp_path / "sentinel-private-source"), - "tool_id": "sentinel-private-tool", - "error": "sentinel-private-error", - } - - monkeypatch.setattr(store_sqlite, "backend_pending_health", durable_health) - - payload = diagnose_herdr(config) - - assert payload["status"] == "degraded" - assert durable_calls == [(config.db_path, config.host_id)] - assert herdr_calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - assert _pending_check(payload) == { - "name": "pending_ingestion", - "ok": False, - "outcome": "degraded", - "counts": {"fresh": 2, "stale": 1, "total": 3}, - "stale_grace_seconds": 17.0, - } - assert config.db_path.read_bytes() == before - serialized = json.dumps(payload, sort_keys=True) - assert private not in serialized - assert "sentinel-private" not in serialized - monkeypatch.setattr( - store_sqlite, - "backend_pending_health", - lambda *_args: { - "status": "healthy", - "counts": {"fresh": 1, "stale": 1, "total": 2}, - }, - ) - fail_closed = diagnose_herdr(config) - assert _pending_check(fail_closed) == { - "name": "pending_ingestion", - "ok": False, - "outcome": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - "stale_grace_seconds": 17.0, - } - monkeypatch.setattr( - store_sqlite, - "backend_pending_health", - lambda *_args: { - "status": "healthy", - "counts": {"fresh": 3, "stale": 0, "total": 3}, - }, - ) - recovered = diagnose_herdr(config) - assert recovered["status"] == "ok" - assert _pending_check(recovered) == { - "name": "pending_ingestion", - "ok": True, - "outcome": "healthy", - "counts": {"fresh": 3, "stale": 0, "total": 3}, - "stale_grace_seconds": 17.0, - } - - -def test_store_maintenance_reports_overdue_without_mutation( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "overdue-private-state" - state_dir.mkdir(mode=0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "overdue-private-store.db", - snapshot_retention_count=10, - store_maintenance_cadence_seconds=3600, - ) - before, private = _seed_maintenance_store( - config, - last_completed_at="2026-01-01T00:00:00+00:00", - snapshot_count=1, - ) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-01T02:00:00+00:00", - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - check = _maintenance_check(payload) - assert payload["status"] == "degraded" - assert check["outcome"] == "overdue" - assert check["ok"] is False - assert check["snapshot_count"] == 1 - assert check["last_completed_at"] == "2026-01-01T00:00:00+00:00" - assert config.db_path is not None - assert config.db_path.read_bytes() == before - assert private not in json.dumps(payload) - - -def test_store_maintenance_reports_backlog_before_cadence_without_mutation( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "backlog-private-state" - state_dir.mkdir(mode=0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "backlog-private-store.db", - snapshot_retention_days=36500, - snapshot_retention_count=1, - snapshot_maintenance_batch_size=1, - store_maintenance_cadence_seconds=3600, - ) - before, private = _seed_maintenance_store( - config, - last_completed_at="2026-01-01T00:00:00+00:00", - snapshot_count=2, - ) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-01T00:30:00+00:00", - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - check = _maintenance_check(payload) - assert payload["status"] == "degraded" - assert check["outcome"] == "backlog" - assert check["ok"] is False - assert check["snapshot_count"] == 2 - assert check["last_completed_at"] == "2026-01-01T00:00:00+00:00" - assert config.db_path is not None - assert config.db_path.read_bytes() == before - assert private not in json.dumps(payload) - - -def test_doctor_maps_owner_and_group_failures_to_fixed_unsafe_records( - monkeypatch, - tmp_path: Path, -) -> None: - private_remediation = "private-remediation-value-that-must-not-leak" - report = ConfigStateReport( - ok=False, - entries=( - PermissionResult( - LocalStateKind.STATE_DIRECTORY, - PermissionState.PRIVATE, - 0o700, - ), - ), - issues=( - LocalStateIssue( - LocalStateKind.PRIVATE_FILE, - LocalStateErrorCode.WRONG_OWNER, - private_remediation, - ), - LocalStateIssue( - LocalStateKind.SOCKET_GROUP, - LocalStateErrorCode.WRONG_GROUP, - private_remediation, - ), - ), - ) - monkeypatch.setattr(herdr_cli, "inspect_config_state", lambda *args, **kwargs: report) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr( - Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=tmp_path / "private-state", - ) - ) - - local_checks = _local_state_checks(payload) - assert payload["status"] == "degraded" - assert local_checks["identity_permissions"] == { - "name": "identity_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - assert local_checks["daemon_socket_permissions"] == { - "name": "daemon_socket_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - assert private_remediation not in json.dumps(payload) - - -def test_doctor_recursively_redacts_configured_and_subprocess_private_values( - monkeypatch, - tmp_path: Path, -) -> None: - private_bin = str(tmp_path / "configured-bin-value" / "herdr") - private_state = tmp_path / "configured-state-value" - private_db = private_state / "configured-database-value" - private_socket = private_state / "configured-socket-value" - private_group = "configured-group-value-that-must-not-leak" - private_sample = "subprocess-sample-value-that-must-not-leak" - which_values: list[str] = [] - subprocess_values: list[str] = [] - - def fake_which(value: str) -> str: - which_values.append(value) - return "/usr/bin/herdr" - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - subprocess_values.append(args[0]) - return _completed( - args[1:], - stdout=f"stdout: {private_sample} /home/alice/private-output", - stderr=f"stderr: {private_sample} /run/user/1000/private.sock", - returncode=2, - ) - - monkeypatch.setattr(herdr_cli.shutil, "which", fake_which) - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - payload = diagnose_herdr( - Config( - host_id="doctor-host", - herdr_bin=private_bin, - data_dir=private_state, - db_path=private_db, - socket_path=private_socket, - socket_group=private_group, - ) - ) - - assert which_values == [private_bin] - assert subprocess_values == [private_bin] * 6 - assert isinstance(payload["herdr_bin"], str) - assert payload["herdr_bin"] != private_bin - assert all( - "stdout_sample" in check and "stderr_sample" in check - for check in _herdr_checks(payload) - ) - serialized = json.dumps(payload) - for forbidden in ( - private_bin, - str(private_state), - str(private_db), - str(private_socket), - private_group, - private_sample, - "/home/alice/private-output", - "/run/user/1000/private.sock", - ): - assert forbidden not in serialized - - -def test_doctor_never_serializes_raw_inspection_or_launch_exceptions( - monkeypatch, - tmp_path: Path, -) -> None: - raw_error = "raw-exception-value-that-must-not-leak" - - def fail(*args: Any, **kwargs: Any) -> Any: - raise OSError(raw_error) - - monkeypatch.setattr(herdr_cli.shutil, "which", fail) - monkeypatch.setattr(herdr_cli, "inspect_config_state", fail) - - payload = diagnose_herdr( - Config( - host_id="doctor-host", - herdr_bin=str(tmp_path / "private-bin"), - data_dir=tmp_path / "private-state", - ) - ) - - assert payload["status"] == "unavailable" - assert raw_error not in json.dumps(payload) - assert all( - check["outcome"] == "unsafe" and check["ok"] is False - for check in _local_state_checks(payload).values() - ) - - -def test_tilde_path_expansion_for_configured_paths(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_DATA_DIR", "~/tendwire-data") - monkeypatch.setenv("TENDWIRE_DB_PATH", "~/tendwire-data/tendwire.db") - monkeypatch.setenv("TENDWIRE_HERDR_BIN", "~/bin/herdr") - monkeypatch.setenv("TENDWIRE_HERDR_TIMEOUT_SECONDS", "0.5") - - config = load_config() - - assert str(config.data_dir).startswith(str(Path.home())) - assert str(config.db_path).startswith(str(Path.home())) - assert config.herdr_bin.startswith(str(Path.home())) - assert config.herdr_timeout_seconds == 0.5 - - -def test_fixture_outputs_parse_through_snapshot_fail_soft_path(monkeypatch) -> None: - config = Config(host_id="fixture-host", herdr_bin="herdr") - responses = { - ("workspace", "list"): _completed( - ["workspace", "list"], - stdout=_fixture("workspace_list_no_flag_nonempty.json"), - ), - ("agent", "list"): _completed( - ["agent", "list"], - stdout=_fixture("agent_list_no_flag_nonempty.json"), - ), - } - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args)) - return responses.get(tuple(args), _completed(args, stdout="", returncode=1)) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert calls == [("workspace", "list"), ("agent", "list"), ("pane", "list")] - assert len(spaces) == 1 - assert spaces[0].id == "ws-fixture" - assert len(workers) == 1 - assert workers[0].id == "Fixture Agent" - - -def test_doctor_initialized_store_with_absent_sqlite_sidecars_is_validation_only( - monkeypatch: Any, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "private-doctor-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-doctor-database" - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - init_store(db_path) - sidecars = tuple( - Path(f"{db_path}{suffix}") for suffix in ("-wal", "-shm", "-journal") - ) - assert all(not os.path.lexists(path) for path in sidecars) - before = ( - tuple(sorted(path.name for path in state_dir.iterdir())), - db_path.stat().st_ino, - db_path.stat().st_size, - db_path.stat().st_mtime_ns, - stat.S_IMODE(db_path.stat().st_mode), - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - after = ( - tuple(sorted(path.name for path in state_dir.iterdir())), - db_path.stat().st_ino, - db_path.stat().st_size, - db_path.stat().st_mtime_ns, - stat.S_IMODE(db_path.stat().st_mode), - ) - assert _local_state_checks(payload)["database_permissions"] == { - "name": "database_permissions", - "ok": True, - "outcome": "compliant", - "remediation": "No action required.", - } - assert after == before - assert all(not os.path.lexists(path) for path in sidecars) - - -@pytest.mark.parametrize("hostile_member", ["main", "wal"]) -def test_doctor_sqlite_failures_are_fixed_typed_and_path_free( - monkeypatch: Any, - tmp_path: Path, - hostile_member: str, -) -> None: - state_dir = tmp_path / "private-doctor-hostile-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-doctor-hostile-database" - target = state_dir / "private-doctor-hostile-target" - private_contents = b"raw-OSError-private-sidecar-target" - target.write_bytes(private_contents) - os.chmod(target, 0o600) - if hostile_member == "main": - hostile_path = db_path - else: - init_store(db_path) - hostile_path = Path(f"{db_path}-wal") - hostile_path.symlink_to(target) - target_before = ( - target.read_bytes(), - target.stat().st_ino, - stat.S_IMODE(target.stat().st_mode), - ) - hostile_inode = str(os.lstat(hostile_path).st_ino) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - database_check = _local_state_checks(payload)["database_permissions"] - assert database_check == { - "name": "database_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - assert _maintenance_check(payload)["outcome"] == "unsafe" - assert hostile_path.is_symlink() - assert ( - target.read_bytes(), - target.stat().st_ino, - stat.S_IMODE(target.stat().st_mode), - ) == target_before - serialized = json.dumps(payload, sort_keys=True) - for forbidden in ( - str(state_dir), - str(db_path), - db_path.name, - str(hostile_path), - hostile_path.name, - str(target), - target.name, - private_contents.decode(), - hostile_inode, - "-wal", - "-shm", - "-journal", - "OSError", - "[Errno", - '"uid"', - '"gid"', - '"inode"', - ): - assert forbidden not in serialized - - -def test_doctor_selected_main_disappearance_is_typed_and_publicly_fixed( - monkeypatch: Any, - tmp_path: Path, -) -> None: - from tendwire import local_state as local_state_module - - state_dir = tmp_path / "private-doctor-main-race-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-doctor-main-race-database" - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - init_store(db_path) - selected_inode = str(db_path.stat().st_ino) - removed = False - observed_codes: list[LocalStateErrorCode] = [] - original_inspect = herdr_cli.inspect_config_state - - def remove_selected_main(phase: str, kind: LocalStateKind) -> None: - nonlocal removed - if phase == "captured" and kind is LocalStateKind.DATABASE and not removed: - removed = True - db_path.unlink() - - def capture_typed_failure(*args: Any, **kwargs: Any) -> ConfigStateReport: - report = original_inspect(*args, **kwargs) - observed_codes.extend( - issue.code - for issue in report.issues - if issue.kind is LocalStateKind.DATABASE - ) - return report - - monkeypatch.setattr( - local_state_module, - "_sqlite_family_test_phase", - remove_selected_main, - ) - monkeypatch.setattr( - herdr_cli, - "inspect_config_state", - capture_typed_failure, - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - assert removed - assert observed_codes == [LocalStateErrorCode.ENTRY_CHANGED] - assert not os.path.lexists(db_path) - assert _local_state_checks(payload)["database_permissions"] == { - "name": "database_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - assert _maintenance_check(payload)["outcome"] == "unsafe" - serialized = json.dumps(payload, sort_keys=True) - for forbidden in ( - str(state_dir), - str(db_path), - db_path.name, - selected_inode, - "-wal", - "-shm", - "-journal", - "OSError", - "[Errno", - '"uid"', - '"gid"', - '"inode"', - ): - assert forbidden not in serialized diff --git a/tests/test_herdr_events.py b/tests/test_herdr_events.py deleted file mode 100644 index 945eb72..0000000 --- a/tests/test_herdr_events.py +++ /dev/null @@ -1,3259 +0,0 @@ -"""Tests for the opt-in Herdr socket event backend.""" - -from __future__ import annotations - -import builtins -import importlib -import json -import os -import sqlite3 -import socket -import subprocess -import sys -import threading -import time -from collections.abc import Callable, Mapping -from contextlib import closing -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest - -from tendwire.backends import herdr_cli, herdr_events -from tendwire.backends.herdr_events import ( - DEFAULT_SUBSCRIBE_METHOD, - HerdrEventBackend, - HerdrEventBackendError, - HerdrEventId, - HerdrProducerSequence, - normalize_event, -) -from tendwire.backends.herdr_cli import HerdrContinuityUnavailableError -from tendwire.backends.herdr_socket import ( - HerdrSocketClient, - HerdrSocketDisconnectedError, - HerdrSocketTimeoutError, -) -from tendwire.backends.herdr_protocol import ( - HERDR_EVENTS_SUBSCRIBE_METHOD, - HERDR_OFFICIAL_EVENT_NAMES, - HerdrEnvelopeError, - HerdrErrorResponse, - build_events_subscribe_params, -) -from tendwire.config import Config -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.core.projector import project_from_observations -from tendwire.core.turns import PendingObservation -from tendwire.daemon import DaemonHooks, TendwireDaemon -from tendwire.local_state import LocalStateErrorCode, local_state_error -from tendwire.store.sqlite import ( - SnapshotObservationContext, - SnapshotRetentionPolicy, - apply_backend_pending_observation, - init_store, - latest_snapshot, - list_attention_items, - list_backend_pending, - list_worker_bindings, - maybe_run_automatic_store_maintenance, - merge_turn_content, - pending_payload_from_store, - save_snapshot, - turns_payload_from_store, -) - - -_PUBLIC_JSON_FORBIDDEN_KEYS = { - "pane_id", - "terminal_id", - "backend_target", - "chat_id", - "topic_id", - "message_id", - "connector", - "argv", - "args", - "env", - "environment", - "stdin", - "stdout", - "stderr", - "token", - "tokens", - "secret", - "secrets", - "raw_payload", - "socket_path", - "target_kind", - "target_value", - "private_fingerprint", -} -_PUBLIC_JSON_FORBIDDEN_COMPACT = {key.replace("_", "") for key in _PUBLIC_JSON_FORBIDDEN_KEYS} - - -def _assert_no_public_json_forbidden(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - normalized = str(key).lower().replace("-", "_") - assert ( - normalized not in _PUBLIC_JSON_FORBIDDEN_KEYS - and normalized.replace("_", "") not in _PUBLIC_JSON_FORBIDDEN_COMPACT - ), f"forbidden field {path}.{key}" - _assert_no_public_json_forbidden(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_public_json_forbidden(item, f"{path}[{index}]") - - - -_NO_OP_TABLES = ( - "commands", - "command_receipts", - "turns", - "attention_items", - "connector_outbox", - "connector_deliveries", -) - -_PERSISTED_EVENT_EFFECT_TABLES = ( - "snapshots", - "events", - "workers", - "worker_bindings", - "attention_items", - "connector_outbox", -) - - -def _table_count(db_path: Path, host_id: str, table: str) -> int: - with closing(sqlite3.connect(str(db_path))) as conn, conn: - return int(conn.execute(f"SELECT COUNT(*) FROM {table} WHERE host_id = ?", (host_id,)).fetchone()[0]) - -def _persisted_event_effect_counts(backend: HerdrEventBackend) -> dict[str, int]: - return { - table: _table_count(backend.db_path, backend.config.host_id, table) - for table in _PERSISTED_EVENT_EFFECT_TABLES - } - - -def _attention_lifecycle_rows(backend: HerdrEventBackend) -> tuple[tuple[Any, ...], ...]: - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - return tuple( - conn.execute( - """ - SELECT - generation, - lifecycle_status, - current_attention_id, - first_seen_at, - last_positive_at, - first_missing_at, - missing_observation_count, - last_accepted_at, - last_observation_key, - max_notified_severity_rank - FROM attention_lifecycles - WHERE host_id = ? - ORDER BY family_key - """, - (backend.config.host_id,), - ).fetchall() - ) - - -def _attention_event_types(backend: HerdrEventBackend) -> list[str]: - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - rows = conn.execute( - """ - SELECT payload_json - FROM connector_outbox - WHERE host_id = ? AND connector = 'attention' - ORDER BY id - """, - (backend.config.host_id,), - ).fetchall() - return [str(json.loads(row[0])["event_type"]) for row in rows] - - -def _set_observation_time(monkeypatch: Any, value: str) -> None: - monkeypatch.setattr("tendwire.backends.herdr_cli.utc_timestamp", lambda: value) - monkeypatch.setattr("tendwire.backends.herdr_events.utc_timestamp", lambda: value) - -def _no_op_state(backend: HerdrEventBackend) -> tuple[str, tuple[tuple[Any, ...], ...], dict[str, int]]: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - bindings = tuple( - sorted( - ( - binding.worker_id, - binding.private_fingerprint, - binding.target_kind, - binding.target_value, - binding.sendable, - binding.reason, - binding.expires_at, - ) - for binding in list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - include_expired=True, - ) - ) - ) - counts = {table: _table_count(backend.db_path, backend.config.host_id, table) for table in _NO_OP_TABLES} - return snapshot.to_json(), bindings, counts - -class _SocketConnection: - def __init__(self, conn: socket.socket, requests: list[dict[str, Any]]) -> None: - self.conn = conn - self.requests = requests - self._buffer = bytearray() - - def read_request(self) -> dict[str, Any]: - while b"\n" not in self._buffer: - chunk = self.conn.recv(4096) - if not chunk: - raise ConnectionError("client disconnected before request") - self._buffer.extend(chunk) - index = self._buffer.index(b"\n") - line = bytes(self._buffer[: index + 1]) - del self._buffer[: index + 1] - request = json.loads(line.decode("utf-8")) - self.requests.append(request) - return request - - def send_json(self, payload: Mapping[str, Any]) -> None: - self.conn.sendall(json.dumps(dict(payload), separators=(",", ":")).encode("utf-8") + b"\n") - - -class _FakeHerdrSocketServer: - def __init__( - self, - tmp_path: Path, - handler: Callable[[_SocketConnection], None], - *, - connections: int = 1, - ) -> None: - self.path = tmp_path / f"herdr-events-{time.monotonic_ns()}.sock" - self.handler = handler - self.connections = connections - self.requests: list[dict[str, Any]] = [] - self.errors: list[BaseException] = [] - self._ready = threading.Event() - self._listener: socket.socket | None = None - self._thread: threading.Thread | None = None - - def __enter__(self) -> "_FakeHerdrSocketServer": - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(self.path)) - listener.listen(self.connections) - listener.settimeout(0.2) - self._listener = listener - self._thread = threading.Thread(target=self._run, daemon=True) - self._thread.start() - assert self._ready.wait(1) - return self - - def __exit__(self, exc_type: object, exc: object, tb: object) -> None: - if self._listener is not None: - self._listener.close() - if self._thread is not None: - self._thread.join(timeout=1) - try: - self.path.unlink() - except FileNotFoundError: - pass - if exc_type is None and self.errors: - raise AssertionError(f"fake Herdr socket failed: {self.errors!r}") - - def _run(self) -> None: - self._ready.set() - try: - assert self._listener is not None - for _index in range(self.connections): - conn, _addr = self._listener.accept() - with conn: - self.handler(_SocketConnection(conn, self.requests)) - except OSError: - pass - except BaseException as exc: - self.errors.append(exc) - - -class _StaticClient: - def __init__( - self, - *, - workspaces: Any | None = None, - tabs: Any | None = None, - panes: Any | None = None, - agents: Any | None = None, - ) -> None: - self.workspaces = {"workspaces": list(workspaces or [])} - self.tabs = {"tabs": list(tabs or [])} - self.panes = {"panes": list(panes or [])} - self.agents = {"agents": list(agents or [])} - self.calls: list[str] = [] - - def workspace_list(self) -> Any: - self.calls.append("workspace.list") - return self.workspaces - - def tab_list(self) -> Any: - self.calls.append("tab.list") - return self.tabs - - def pane_list(self) -> Any: - self.calls.append("pane.list") - return self.panes - - def agent_list(self) -> Any: - self.calls.append("agent.list") - return self.agents - - -def _config(tmp_path: Path, host_id: str = "events-host") -> Config: - return Config( - host_id=host_id, - data_dir=tmp_path, - db_path=tmp_path / f"{host_id}.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - ) - - -def _backend(tmp_path: Path, host_id: str = "events-host", *, debounce_seconds: float = 0) -> HerdrEventBackend: - config = _config(tmp_path, host_id) - init_store(Path(config.db_path)) - return HerdrEventBackend( - config, - debounce_seconds=debounce_seconds, - reconnect_delay_seconds=0, - ) - - -def _initial_pane_client() -> _StaticClient: - return _StaticClient( - workspaces=[{"id": "space-1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - } - ], - agents=[], - ) - -def _status_event(status: str) -> dict[str, Any]: - """Return the confirmed idless Herdr EventEnvelope shape.""" - return { - "event": "pane_agent_status_changed", - "data": { - "pane_id": "pane-1", - "agent": "Agent One", - "status": status, - }, - } - - -def _write_decision_adapter(tmp_path: Path) -> Path: - adapter = tmp_path / "fake-herdr-turn-adapter" - adapter.write_text( - "#!/usr/bin/env python3\n" - "import json\n" - "print(json.dumps({'result': {'turn': {" - "'available': True, 'complete': False, 'awaiting_input': True, " - "'user_text': 'Choose a rollout.', 'source_turn_id': 'producer-decision-turn', " - "'pending_decision': {'prompt': 'Choose a rollout.', 'mode': 'buttons', " - "'options': [{'id': 'alpha', 'label': 'Alpha', 'send_text': 'Alpha'}, " - "{'id': 'beta', 'label': 'Beta', 'send_text': 'Beta'}]}}}}))\n", - encoding="utf-8", - ) - adapter.chmod(0o700) - return adapter - - -def _decision_backend( - tmp_path: Path, - turn_model: str, -) -> tuple[HerdrEventBackend, WorkerBinding]: - adapter = _write_decision_adapter(tmp_path) - config = Config( - host_id=f"decision-{turn_model}", - data_dir=tmp_path, - db_path=tmp_path / f"decision-{turn_model}.db", - herdr_backend="socket", - herdr_bin=str(adapter), - herdr_timeout_seconds=1, - turn_model=turn_model, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) - snapshot = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[ - { - "pane_id": "w123456789abcde:pA", - "terminal_id": "terminal-decision", - "agent": "claude", - "workspace_id": "w123456789abcde", - "agent_status": "working", - } - ], - agents=[], - ) - ) - binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - # Reproduce the production wedge: the stable worker still has a durable - # no-prompt row owned by an expired pane binding. - assert apply_backend_pending_observation( - backend.db_path, - backend.config.host_id, - snapshot.workers[0].id, - PendingObservation("read_succeeded_no_prompt"), - binding_private_fingerprint="expired-pane-binding", - observed_turn_target_value="old-pane", - ) - return backend, binding - - -def _assert_decision_persisted(backend: HerdrEventBackend) -> None: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - worker = snapshot.workers[0] - backend_pending = list_backend_pending(backend.db_path, backend.config.host_id) - assert list(backend_pending) == [worker.id] - assert [choice["label"] for choice in backend_pending[worker.id]["choices"]] == [ - "Alpha", - "Beta", - ] - pending = pending_payload_from_store(backend.db_path, backend.config.host_id) - interaction = next( - item for item in pending["pending_interactions"] if item["worker_id"] == worker.id - ) - assert [choice["label"] for choice in interaction["choices"]] == ["Alpha", "Beta"] - turns = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=snapshot, - )["turns"] - turn = next(item for item in turns if item.get("awaiting_input") is True) - assert turn["complete"] is False - assert turn["pending_decision"] == { - "prompt": "Choose a rollout.", - "mode": "buttons", - "options": [ - {"id": "1", "label": "Alpha"}, - {"id": "2", "label": "Beta"}, - ], - "multi_select": False, - "question_count": 1, - } - - -def test_startup_reconcile_uses_socket_client_persists_projection_and_private_bindings(tmp_path: Path) -> None: - def handler(conn: _SocketConnection) -> None: - results = { - "workspace.list": { - "workspaces": [ - { - "id": "space-1", - "name": "Build", - "status": "active", - "pane_id": "private-pane", - } - ] - }, - "tab.list": {"tabs": [{"id": "tab-private", "workspace_id": "space-1"}]}, - "pane.list": { - "panes": [ - { - "pane_id": "pane-1", - "terminal_id": "terminal-private", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - } - ] - }, - "agent.list": { - "agents": [ - { - "agent_id": "agent-private", - "name": "Agent One", - "workspace_id": "space-1", - "status": "waiting", - "pane_id": "pane-1", - } - ] - }, - } - request = conn.read_request() - conn.send_json({"id": request["id"], "result": results[request["method"]]}) - - config = _config(tmp_path, "socket-reconcile") - init_store(Path(config.db_path)) - with _FakeHerdrSocketServer(tmp_path, handler, connections=4) as server: - backend = HerdrEventBackend(config, debounce_seconds=0) - client = HerdrSocketClient(str(server.path), timeout=1) - snapshot = backend.reconcile_once(client=client) - client.close() - - assert [request["method"] for request in server.requests] == [ - "workspace.list", - "tab.list", - "pane.list", - "agent.list", - ] - assert snapshot.backend_health[0].status == "healthy" - assert "agent-private" not in {worker.id for worker in snapshot.workers} - assert all("private" not in worker.id.lower() for worker in snapshot.workers) - bindings = list_worker_bindings(Path(config.db_path), config.host_id, backend="herdr") - assert bindings - assert bindings[0].target_value == "agent-private" - encoded = snapshot.to_json() - assert "agent-private" not in encoded - assert "private-pane" not in encoded - assert "terminal-private" not in encoded - _assert_no_public_json_forbidden(json.loads(encoded)) - - -@pytest.mark.parametrize("event_name", HERDR_OFFICIAL_EVENT_NAMES) -def test_normalize_event_accepts_each_official_event_name(event_name: str) -> None: - event = normalize_event({"event": event_name, "data": {}}) - - assert event is not None - assert event.name == event_name - assert event.producer_identity is None - - -@pytest.mark.parametrize( - ("raw_name", "canonical_name"), - [ - ("agent.status_changed", "pane.agent_status_changed"), - ("agent_status_changed", "pane.agent_status_changed"), - ("agent.detected", "pane.agent_detected"), - ("pane_output_changed", "pane.updated"), - ("pane.observed", "pane.created"), - ("workspace.observed", "workspace.updated"), - ("worktree.updated", "worktree.opened"), - ("worktree.closed", "worktree.removed"), - ], -) -def test_normalize_event_tolerates_legacy_inbound_aliases_only_after_receive( - raw_name: str, - canonical_name: str, -) -> None: - event = normalize_event({"event": raw_name, "data": {}}) - - assert event is not None - assert event.name == canonical_name - - -def test_normalize_event_accepts_confirmed_live_idless_event_data_shape() -> None: - event = normalize_event( - { - "event": "pane_agent_status_changed", - "data": {"agent": "Agent One", "status": "blocked"}, - } - ) - - assert event is not None - assert event.name == "pane.agent_status_changed" - assert event.payload == {"agent": "Agent One", "status": "blocked"} - assert event.producer_identity is None - - -def test_normalize_event_prefers_confirmed_data_over_legacy_payload() -> None: - event = normalize_event( - { - "event": "pane_agent_status_changed", - "data": {"status": "working"}, - "payload": {"status": "idle"}, - } - ) - - assert event is not None - assert event.payload == {"status": "working"} - assert event.producer_identity is None - - -def test_normalize_event_keeps_receive_only_legacy_payload_compatibility() -> None: - event = normalize_event( - { - "event": "pane_agent_status_changed", - "payload": {"status": "idle"}, - } - ) - - assert event is not None - assert event.payload == {"status": "idle"} - assert event.producer_identity is None - - -def test_normalize_event_exposes_forward_compatible_producer_identity_types() -> None: - by_id = normalize_event( - {"event": "pane_agent_status_changed", "data": {}, "event_id": "event-1"} - ) - by_sequence = normalize_event( - { - "event": "pane_agent_status_changed", - "data": {}, - "server_id": "server-1", - "sequence": 7, - } - ) - - assert by_id is not None - assert by_id.producer_identity == HerdrEventId("event-1") - assert by_sequence is not None - assert by_sequence.producer_identity == HerdrProducerSequence("server-1", "7") - - -def test_normalize_event_never_uses_entity_data_as_producer_identity() -> None: - event = normalize_event( - { - "event": "pane_agent_status_changed", - "data": { - "event_id": "entity-event", - "server_id": "entity-server", - "sequence": 9, - "revision": 10, - }, - } - ) - - assert event is not None - assert event.producer_identity is None - - -@pytest.mark.parametrize( - "metadata", - [ - {"event_id": True}, - {"event_id": 1}, - {"event_id": 1.5}, - {"event_id": "event id"}, - {"event_id": {"id": "nested-event"}}, - {"event_id": ["nested-event"]}, - {"server_id": True, "sequence": 1}, - {"server_id": "server id", "sequence": 1}, - {"server_id": {"id": "nested-server"}, "sequence": 1}, - {"server_id": "server-1", "sequence": True}, - {"server_id": "server-1", "sequence": 1.5}, - {"server_id": "server-1", "sequence": "1"}, - {"server_id": "server-1", "sequence": {"value": 1}}, - {"server_id": "server-1"}, - {"sequence": 1}, - {"event_id": True, "server_id": "server-1", "sequence": 1}, - ], -) -def test_malformed_producer_metadata_is_idless_and_preserves_transitions( - tmp_path: Path, - metadata: dict[str, Any], -) -> None: - backend = _backend(tmp_path, f"malformed-producer-{len(str(metadata))}") - backend.reconcile_once(client=_initial_pane_client()) - accepted: list[bool] = [] - - for status in ("working", "idle", "working"): - envelope = {**_status_event(status), **metadata} - normalized = normalize_event(envelope) - assert normalized is not None - assert normalized.producer_identity is None - accepted.append(backend.queue_event_envelope(envelope)) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert accepted == [True, True, True] - assert snapshot.workers[0].status == "active" - - -def test_backend_default_subscription_uses_official_shape_without_legacy_defaults(tmp_path: Path) -> None: - config = _config(tmp_path, "default-subscribe") - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) - - class SubscribeClient(_StaticClient): - def __init__(self) -> None: - super().__init__() - self.subscriptions: list[tuple[str, dict[str, Any]]] = [] - - def connect(self) -> None: - return None - - def close(self) -> None: - return None - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - self.subscriptions.append((method, dict(params))) - backend.stop_event.set() - return SimpleNamespace(subscription_id="sub-default") - - client = SubscribeClient() - backend.client_factory = lambda _config: client - - backend.run_forever() - - expected_params = { - "subscriptions": [ - {"type": name} - for name in HERDR_OFFICIAL_EVENT_NAMES - if name not in {"pane.agent_status_changed", "pane.output_matched"} - ] - } - assert DEFAULT_SUBSCRIBE_METHOD == HERDR_EVENTS_SUBSCRIBE_METHOD - assert client.subscriptions == [(HERDR_EVENTS_SUBSCRIBE_METHOD, expected_params)] - subscribed_names = {subscription["type"] for subscription in expected_params["subscriptions"]} - assert { - "pane.observed", - "workspace.observed", - "agent.status_changed", - "worktree.updated", - }.isdisjoint(subscribed_names) - - -def test_backend_falls_back_to_herdr_074_pane_scoped_event_subscriptions(tmp_path: Path) -> None: - config = _config(tmp_path, "pane-scoped-subscribe") - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) - - class PaneScopedClient(_StaticClient): - def __init__(self) -> None: - super().__init__( - workspaces=[{"id": "space-1", "name": "Build"}], - panes=[ - { - "pane_id": "pane-private", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - } - ], - ) - self.mixed_attempts = 0 - self.subscriptions: list[tuple[str, dict[str, Any]]] = [] - self.closed = 0 - self.connected = 0 - - def connect(self) -> None: - self.connected += 1 - - def close(self) -> None: - self.closed += 1 - - def events_subscribe( - self, - event_names: Any, - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - raise AssertionError("0.7.4 fallback must retain the pane-scoped request") - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - self.subscriptions.append((method, dict(params))) - if any( - item.get("type") == "pane.updated" - for item in params.get("subscriptions", []) - ): - self.mixed_attempts += 1 - raise HerdrErrorResponse( - { - "code": "invalid_request", - "message": "invalid request: unknown variant pane.updated", - }, - "subscribe-1", - uncorrelated=True, - ) - backend.stop_event.set() - return SimpleNamespace(subscription_id="pane-scoped-sub") - - client = PaneScopedClient() - backend.client_factory = lambda _config: client - - backend.run_forever() - - assert client.mixed_attempts == 1 - assert client.closed >= 1 - assert client.connected >= 1 - assert len(client.subscriptions) == 2 - method, params = client.subscriptions[1] - assert method == HERDR_EVENTS_SUBSCRIBE_METHOD - subscriptions = params["subscriptions"] - fallback_names = set(HERDR_OFFICIAL_EVENT_NAMES) - { - "workspace.focused", - "pane.updated", - "pane.focused", - "pane.agent_detected", - "pane.output_matched", - } - assert len(subscriptions) == len(fallback_names) - assert {item["type"] for item in subscriptions} == fallback_names - assert {item["pane_id"] for item in subscriptions} == {"pane-private"} - - -def test_backend_empty_installation_falls_back_to_herdr_074_global_subscription( - tmp_path: Path, -) -> None: - config = _config(tmp_path, "empty-global-fallback") - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) - - class EmptyInstallationClient(_StaticClient): - def __init__(self) -> None: - super().__init__() - self.subscriptions: list[tuple[str, dict[str, Any]]] = [] - self.closed = 0 - self.connected = 0 - - def connect(self) -> None: - self.connected += 1 - - def close(self) -> None: - self.closed += 1 - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - copied = (method, {"subscriptions": [dict(item) for item in params["subscriptions"]]}) - self.subscriptions.append(copied) - if any(item.get("type") == "pane.updated" for item in params["subscriptions"]): - raise HerdrErrorResponse( - { - "code": "invalid_request", - "message": "invalid request: unknown variant pane.updated", - }, - "subscribe-1", - uncorrelated=True, - ) - backend.stop_event.set() - return SimpleNamespace(subscription_id="empty-global-sub") - - client = EmptyInstallationClient() - backend.client_factory = lambda _config: client - - backend.run_forever() - - assert client.closed >= 1 - assert client.connected >= 1 - assert len(client.subscriptions) == 2 - method, params = client.subscriptions[1] - assert method == HERDR_EVENTS_SUBSCRIBE_METHOD - subscriptions = params["subscriptions"] - assert subscriptions - assert all(set(item) == {"type"} for item in subscriptions) - assert {item["type"] for item in subscriptions} == set(HERDR_OFFICIAL_EVENT_NAMES) - { - "pane.updated" - } - - -@pytest.mark.parametrize( - "failure", - [ - HerdrErrorResponse( - {"code": "permission_denied", "message": "subscription denied"}, - "subscribe-1", - ), - HerdrErrorResponse( - {"code": "invalid_request", "message": "invalid request: correlated"}, - "subscribe-1", - ), - HerdrEnvelopeError("malformed subscription response"), - ], -) -def test_backend_reconnects_instead_of_downgrading_unrelated_subscription_failures( - tmp_path: Path, - failure: Exception, -) -> None: - backend = _backend(tmp_path, "no-unrelated-subscription-downgrade") - - class RejectingClient: - def __init__(self) -> None: - self.calls = 0 - self.closed = 0 - self.connected = 0 - - def subscribe(self, method, params, **kwargs): - self.calls += 1 - raise failure - - def close(self) -> None: - self.closed += 1 - - def connect(self) -> None: - self.connected += 1 - - client = RejectingClient() - with pytest.raises(type(failure)): - backend._subscribe_event_stream(client) - assert client.calls == 1 - assert client.closed == 0 - assert client.connected == 0 - - - - - - - - - - - - -def test_backend_rejects_non_official_subscribe_method(tmp_path: Path) -> None: - config = _config(tmp_path, "custom-subscribe") - init_store(Path(config.db_path)) - - with pytest.raises(HerdrEventBackendError): - HerdrEventBackend(config, subscribe_method="custom.subscribe") - - -def test_pane_agent_detected_official_event_updates_worker_and_private_binding(tmp_path: Path) -> None: - backend = _backend(tmp_path, "agent-detected") - backend.reconcile_once(client=_StaticClient(workspaces=[{"id": "space-1", "name": "Build"}])) - - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent_id": "agent-2", - "name": "Agent Two", - "workspace_id": "space-1", - "pane_id": "pane-2", - "status": "running", - } - }} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") - assert snapshot is not None - assert {worker.id for worker in snapshot.workers} == {"agent-2"} - assert bindings[0].target_kind == "agent_id" - assert bindings[0].target_value == "agent-2" - _assert_no_public_json_forbidden(json.loads(snapshot.to_json())) - - -@pytest.mark.parametrize( - "event_name", - [ - "pane.created", - "pane.focused", - "pane.agent_detected", - "pane.agent_status_changed", - ], -) -@pytest.mark.parametrize("entity_name", ["agent", "worker"]) -def test_supported_nested_agent_or_worker_canonical_fields_cannot_mint_continuity( - tmp_path: Path, - event_name: str, - entity_name: str, -) -> None: - backend = _backend( - tmp_path, - f"nested-no-mint-{event_name}-{entity_name}", - ) - backend.reconcile_once( - client=_StaticClient(workspaces=[{"id": "wR9", "name": "Build"}]) - ) - entity = { - "worker_id": f"public-{entity_name}", - "agent_id": f"{entity_name}-target-secret", - "name": "codex", - "agent": "codex", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "status": "running", - "agent_session": { - "source": "compatibility-secret", - "agent": "codex", - "kind": "id", - "value": "compatibility-session-secret", - }, - } - - assert backend.queue_event_envelope( - {"event": event_name, "data": {entity_name: entity}} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert len(snapshot.workers) == 1 - worker = snapshot.workers[0] - assert "stable_key" not in worker.meta - assert "stable_key_version" not in worker.meta - assert not backend.config.installation_key_path.exists() - _assert_no_public_json_forbidden(json.loads(snapshot.to_json())) - - - - -@pytest.mark.parametrize("entity_source", ["top_level", "pane"]) -def test_official_pane_tuple_provenance_mints_continuity( - tmp_path: Path, - entity_source: str, -) -> None: - backend = _backend(tmp_path, f"official-pane-{entity_source}") - backend.reconcile_once( - client=_StaticClient(workspaces=[{"id": "wR9", "name": "Build"}]) - ) - pane = { - "agent": "codex", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "official-terminal-secret", - "status": "running", - "agent_session": { - "source": "official-source-secret", - "agent": "codex", - "kind": "id", - "value": "official-session-secret", - }, - } - payload = pane if entity_source == "top_level" else {"pane": pane} - - assert backend.queue_event_envelope( - {"event": "pane.created", "data": payload} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert len(snapshot.workers) == 1 - assert snapshot.workers[0].meta["stable_key"].startswith("wsk1_") - assert snapshot.workers[0].meta["stable_key_version"] == 1 - assert backend.config.installation_key_path.exists() - - - - - - - - -def test_key_failure_precedes_move_conflict_mutation(tmp_path: Path) -> None: - backend = _backend(tmp_path, "event-move-conflict-key-failure") - panes = [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent A", - "status": "running", - }, - { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "agent": "Agent B", - "status": "running", - }, - ] - before = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=panes, - ) - ) - before_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - backend.config.installation_key_marker_path.unlink() - - assert backend.queue_event_envelope( - {"event": "pane.moved", "data": { - "previous_pane_id": "wR9:pB", - "pane": { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent B", - "status": "running", - }, - }} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.workers == before.workers - assert after.spaces == before.spaces - assert after.backend_health[0].status == "degraded" - assert after.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == before_bindings - ) - - - - -def test_reconcile_retains_authenticated_snapshot_until_installation_key_recovers( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "reconcile-key-recovery") - client = _StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-secret", - "agent": "codex", - "status": "running", - } - ], - agents=[ - { - "worker_id": "public-worker", - "agent_id": "agent-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-secret", - "agent": "codex", - "status": "running", - } - ], - ) - first = backend.reconcile_once(client=client) - first_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - stable_key = first.workers[0].meta["stable_key"] - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - - degraded = backend.reconcile_once(client=client) - - assert degraded.workers == first.workers - assert degraded.spaces == first.spaces - assert degraded.backend_health[0].status == "degraded" - assert degraded.backend_health[0].outcome == "continuity_unavailable" - assert degraded.backend_health[0].counts == {"spaces": 1, "workers": 1} - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == first_bindings - ) - - previous_max_workers = backend.max_workers - backend.max_workers = 1 - capped = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - agents=[ - {"worker_id": "agent-a", "agent": "Agent A"}, - {"worker_id": "agent-b", "agent": "Agent B"}, - ], - ) - ) - assert capped.backend_health[0].outcome == "continuity_unavailable" - backend.max_workers = previous_max_workers - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - recovered = backend.reconcile_once(client=client) - - assert recovered.backend_health[0].status == "healthy" - assert recovered.workers[0].meta["stable_key"] == stable_key - recovered_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - - unmatched = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[], - agents=[ - { - "worker_id": "public-worker", - "agent_id": "agent-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-secret", - "agent": "codex", - "status": "running", - } - ], - ) - ) - - assert unmatched.workers == recovered.workers - assert unmatched.spaces == recovered.spaces - assert unmatched.backend_health[0].status == "degraded" - assert unmatched.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == recovered_bindings - ) - - recovered_again = backend.reconcile_once(client=client) - assert recovered_again.backend_health[0].status == "healthy" - assert recovered_again.workers[0].meta["stable_key"] == stable_key - - -def test_incomplete_pane_event_does_not_clear_continuity_failure( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "incomplete-pane-key-recovery") - pane = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-secret", - "agent": "codex", - "status": "running", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane], - ) - ) - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane": pane}} - ) - failed = latest_snapshot(backend.db_path, backend.config.host_id) - assert failed is not None - assert failed.backend_health[0].outcome == "continuity_unavailable" - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane": {"pane_id": "wR9:pA"}}} - ) - - incomplete = latest_snapshot(backend.db_path, backend.config.host_id) - assert incomplete is not None - assert incomplete.backend_health[0].status == "degraded" - assert incomplete.backend_health[0].outcome == "continuity_unavailable" - - -def test_over_cap_authenticated_retry_does_not_clear_continuity_failure( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "over-cap-key-recovery") - pane_a = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent A", - "status": "running", - } - pane_b = { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "agent": "Agent B", - "status": "running", - } - first = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane_a], - ) - ) - first_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - backend.max_workers = 1 - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.created", "data": {"pane": pane_b}} - ) - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - assert backend.queue_event_envelope( - {"event": "pane.created", "data": {"pane": pane_b}} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.workers == first.workers - assert after.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == first_bindings - ) - - -def test_conflicting_close_does_not_revalidate_continuity( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "conflicting-close-key-recovery") - pane_a = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-a-secret", - "agent": "Agent A", - "status": "running", - } - pane_b = { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "terminal-b-secret", - "agent": "Agent B", - "status": "running", - } - first = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane_a, pane_b], - ) - ) - first_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane": pane_a}} - ) - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - conflicting_close = { - **pane_a, - "terminal_id": pane_b["terminal_id"], - } - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane": conflicting_close}} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.workers == first.workers - assert after.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == first_bindings - ) - - -def test_conflicting_upsert_preserves_latched_authenticated_state( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "conflicting-upsert-key-recovery") - pane_a = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-a-secret", - "agent": "Agent A", - "status": "running", - } - pane_b = { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "terminal-b-secret", - "agent": "Agent B", - "status": "running", - } - first = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane_a, pane_b], - ) - ) - first_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane": pane_a}} - ) - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - conflicting_pane = { - **pane_a, - "terminal_id": pane_b["terminal_id"], - "status": "blocked", - } - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane": conflicting_pane}} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.workers == first.workers - assert after.backend_health[0].status == "degraded" - assert after.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == first_bindings - ) - - -def test_official_pane_event_generic_id_remains_private_binding_only(tmp_path: Path) -> None: - backend = _backend(tmp_path, "pane-id-private") - backend.reconcile_once(client=_StaticClient(workspaces=[{"id": "space-1", "name": "Build"}])) - - assert ( - backend.queue_event_envelope( - {"event": "pane.created", "data": { - "id": "pane-secret", - "agent": "Agent Two", - "workspace_id": "space-1", - "status": "running", - }} - ) - is True - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") - assert snapshot is not None - public_json = snapshot.to_json() - assert "pane-secret" not in public_json - assert {worker.id for worker in snapshot.workers} == {"Agent Two"} - assert bindings[0].target_kind == "pane_id" - assert bindings[0].target_value == "pane-secret" - _assert_no_public_json_forbidden(json.loads(public_json)) - - -def test_unknown_and_malformed_known_events_do_not_mutate_any_public_or_private_state( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "event-noop") - backend.reconcile_once(client=_initial_pane_client()) - before = _no_op_state(backend) - - for envelope in ( - {"event": "unknown.future", "data": {"pane_id": "pane-secret", "stdout": "secret"}}, - {"event": "workspace.created", "data": {"pane_id": "pane-secret", "stdout": "secret"}}, - {"event": "workspace.renamed", "data": {"new_name": "secret"}}, - {"event": "worktree.created", "data": {"worktree_id": "worktree-secret", "stderr": "secret"}}, - {"event": "pane.created", "data": {"labels": ["agent"], "argv": ["secret"]}}, - {"event": "pane.agent_detected", "data": []}, - {"event": "pane.agent_status_changed", "data": {"status": "failed", "stderr": "secret"}}, - {"event": "pane.output_matched", "data": { - "pane_id": "pane-secret", - "terminal_id": "terminal-secret", - "stdout": "secret", - "stderr": "secret", - "token": "secret", - }}, - ): - backend.queue_event_envelope(envelope) - - after = _no_op_state(backend) - assert after == before - _assert_no_public_json_forbidden(json.loads(after[0])) - assert "pane-secret" not in after[0] - assert "terminal-secret" not in after[0] - assert "secret" not in after[0] - - -def test_worktree_events_only_update_existing_workspace_observations(tmp_path: Path) -> None: - backend = _backend(tmp_path, "worktree-adjacent") - backend.reconcile_once(client=_StaticClient(workspaces=[{"id": "space-1", "name": "Build"}])) - before = latest_snapshot(backend.db_path, backend.config.host_id) - assert before is not None - - assert ( - backend.queue_event_envelope( - {"event": "worktree.created", "data": {"workspace_id": "new-space", "name": "Should Not Appear"}} - ) - is True - ) - unchanged = latest_snapshot(backend.db_path, backend.config.host_id) - assert unchanged is not None - assert [space.id for space in unchanged.spaces] == ["space-1"] - assert unchanged.spaces[0].name == "Build" - - assert ( - backend.queue_event_envelope( - {"event": "worktree.opened", "data": { - "workspace_id": "space-1", - "name": "Build Worktree", - "status": "active", - }} - ) - is True - ) - updated = latest_snapshot(backend.db_path, backend.config.host_id) - assert updated is not None - assert [space.id for space in updated.spaces] == ["space-1"] - assert updated.spaces[0].name == "Build Worktree" - _assert_no_public_json_forbidden(json.loads(updated.to_json())) - -def test_run_forever_reconnect_accepts_identical_idless_event_again(tmp_path: Path) -> None: - config = _config(tmp_path, "reconnect-resubscribe") - init_store(Path(config.db_path)) - - class SequenceClient(_StaticClient): - def __init__(self, label: str, events: list[Any], *, pane_status: str) -> None: - super().__init__( - workspaces=[{"id": "space-1", "name": "Build"}], - panes=[ - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": pane_status, - } - ], - ) - self.label = label - self.events = list(events) - self.subscriptions: list[tuple[str, dict[str, Any]]] = [] - self.read_calls = 0 - self.closed = False - - def connect(self) -> None: - return None - - def close(self) -> None: - self.closed = True - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - self.subscriptions.append((method, dict(params))) - return SimpleNamespace(subscription_id=f"{self.label}-sub") - - def read_event(self, subscription_id: str, *, timeout: float | None = None) -> dict[str, Any]: - self.read_calls += 1 - if not self.events: - backend.stop_event.set() - raise HerdrSocketTimeoutError("idle") - event = self.events.pop(0) - if event == "disconnect": - raise HerdrSocketDisconnectedError("disconnect") - return dict(event) - - working = _status_event("working") - first = SequenceClient("first", [working, "disconnect"], pane_status="idle") - second = SequenceClient("second", [working], pane_status="idle") - clients = [first, second] - backend = HerdrEventBackend( - config, - client_factory=lambda _config: clients.pop(0), - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - - backend.run_forever() - - expected = { - "subscriptions": [ - {"type": name} - for name in HERDR_OFFICIAL_EVENT_NAMES - if name not in {"pane.agent_status_changed", "pane.output_matched"} - ] - + [{"type": "pane.agent_status_changed", "pane_id": "pane-1"}] - } - assert first.subscriptions == [(HERDR_EVENTS_SUBSCRIBE_METHOD, expected)] - assert second.subscriptions == [(HERDR_EVENTS_SUBSCRIBE_METHOD, expected)] - assert first.read_calls == 2 - assert second.read_calls == 2 - assert first.closed is True - assert second.closed is True - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "active" - - -def test_run_forever_retains_complete_reconcile_when_event_stream_disconnects( - tmp_path: Path, -) -> None: - config = _config(tmp_path, "reconcile-before-stream-disconnect") - init_store(Path(config.db_path)) - - class DisconnectingClient(_StaticClient): - def __init__(self) -> None: - super().__init__(workspaces=[{"id": "space-1", "name": "Build"}]) - - def connect(self) -> None: - return None - - def close(self) -> None: - return None - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - return SimpleNamespace(subscription_id="disconnect-sub") - - def read_event( - self, - subscription_id: str, - *, - timeout: float | None = None, - ) -> dict[str, Any]: - backend.stop_event.set() - raise HerdrSocketDisconnectedError("event stream closed") - - backend = HerdrEventBackend( - config, - client_factory=lambda _config: DisconnectingClient(), - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - - backend.run_forever() - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.backend_health[0].status == "healthy" - assert snapshot.backend_health[0].outcome == "healthy_non_empty" - - -def test_start_stop_are_idempotent_and_bounded_for_idle_subscription(tmp_path: Path) -> None: - config = Config( - host_id="bounded-stop", - data_dir=tmp_path, - db_path=tmp_path / "bounded-stop.db", - herdr_backend="socket", - herdr_timeout_seconds=0.05, - ) - init_store(Path(config.db_path)) - - class IdleClient(_StaticClient): - def __init__(self) -> None: - super().__init__() - self.subscriptions = 0 - - def connect(self) -> None: - return None - - def close(self) -> None: - return None - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - self.subscriptions += 1 - return SimpleNamespace(subscription_id="idle-sub") - - def read_event(self, subscription_id: str, *, timeout: float | None = None) -> dict[str, Any]: - raise HerdrSocketTimeoutError("idle") - - client = IdleClient() - backend = HerdrEventBackend( - config, - client_factory=lambda _config: client, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - - started = time.monotonic() - backend.start(wait_for_reconcile=True, timeout_seconds=0.2) - deadline = time.monotonic() + 0.5 - while client.subscriptions < 1 and time.monotonic() < deadline: - time.sleep(0.01) - assert backend.ready is True - assert client.subscriptions >= 1 - - backend.stop() - backend.stop() - - assert backend.running is False - assert time.monotonic() - started < 2.0 - - -def test_start_raises_instead_of_proceeding_with_stale_state_when_not_ready( - tmp_path: Path, - monkeypatch: Any, -) -> None: - config = _config(tmp_path, "initial-reconcile-timeout") - init_store(Path(config.db_path)) - save_snapshot( - Path(config.db_path), - project_from_observations( - config, - workers=[Worker(id="stale-worker", name="Stale Worker", status="waiting")], - ), - ) - backend = HerdrEventBackend( - config, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - assert latest_snapshot(backend.db_path, backend.config.host_id) is not None - - def wait_until_stopped() -> None: - backend.stop_event.wait() - - monkeypatch.setattr(backend, "run_forever", wait_until_stopped) - - try: - with pytest.raises( - HerdrSocketTimeoutError, - match="initial Herdr reconciliation timed out", - ): - backend.start(wait_for_reconcile=True, timeout_seconds=0.01) - - assert backend.ready is False - assert backend.running is False - assert backend._thread is None - finally: - backend.stop() - - -def test_start_defaults_to_dedicated_initial_reconcile_timeout( - tmp_path: Path, - monkeypatch: Any, -) -> None: - config = Config( - host_id="initial-reconcile-budget", - data_dir=tmp_path, - db_path=tmp_path / "initial-reconcile-budget.db", - herdr_backend="socket", - herdr_timeout_seconds=0.25, - herdr_initial_reconcile_timeout_seconds=42, - ) - backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) - waited: list[float] = [] - - class NeverReady: - def clear(self) -> None: - return None - - def is_set(self) -> bool: - return False - - def wait(self, timeout: float | None = None) -> bool: - assert timeout is not None - waited.append(timeout) - return False - - backend._ready = NeverReady() # type: ignore[assignment] - monkeypatch.setattr(backend, "run_forever", backend.stop_event.wait) - - try: - with pytest.raises(HerdrSocketTimeoutError): - backend.start(wait_for_reconcile=True) - assert waited == [42.0] - assert config.herdr_timeout_seconds == 0.25 - finally: - backend.stop() - - - - -@pytest.mark.parametrize("batched", [False, True], ids=["one-flush-per-event", "one-batch"]) -def test_real_idless_working_idle_working_preserves_every_transition( - tmp_path: Path, - monkeypatch: Any, - batched: bool, -) -> None: - backend = _backend( - tmp_path, - f"real-idless-transitions-{batched}", - debounce_seconds=60 if batched else 0, - ) - backend.reconcile_once(client=_initial_pane_client()) - bindings_before = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - applied: list[str] = [] - original_apply = backend._apply_event - - def recording_apply(event: Any) -> bool: - applied.append(str(event.payload["status"])) - return original_apply(event) - - monkeypatch.setattr(backend, "_apply_event", recording_apply) - observed_statuses: list[str] = [] - accepted = [] - for status in ("working", "idle", "working"): - accepted.append(backend.queue_event_envelope(_status_event(status), flush=not batched)) - if not batched: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - observed_statuses.append(snapshot.workers[0].status) - if batched: - backend.flush() - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings_after = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert accepted == [True, True, True] - assert applied == ["working", "idle", "working"] - if not batched: - assert observed_statuses == ["active", "idle", "active"] - assert snapshot.workers[0].status == "active" - assert len(snapshot.workers) == 1 - assert bindings_after == bindings_before - - -@pytest.mark.parametrize("batched", [False, True], ids=["separate-flushes", "one-batch"]) -def test_adjacent_idless_duplicates_do_not_duplicate_persisted_effects( - tmp_path: Path, - batched: bool, -) -> None: - backend = _backend( - tmp_path, - f"idless-repeat-effects-{batched}", - debounce_seconds=60 if batched else 0, - ) - backend.reconcile_once(client=_initial_pane_client()) - before = _persisted_event_effect_counts(backend) - blocked = _status_event("blocked") - - assert backend.queue_event_envelope(blocked, flush=not batched) is True - if batched: - assert backend.queue_event_envelope(blocked, flush=False) is True - backend.flush() - after = _persisted_event_effect_counts(backend) - else: - after_first = _persisted_event_effect_counts(backend) - assert backend.queue_event_envelope(blocked, flush=True) is True - after = _persisted_event_effect_counts(backend) - assert after == after_first - - assert after["snapshots"] == before["snapshots"] + 1 - assert after["events"] == before["events"] + 1 - assert after["workers"] == before["workers"] == 1 - assert after["worker_bindings"] == before["worker_bindings"] == 1 - assert after["attention_items"] == before["attention_items"] + 1 - assert after["connector_outbox"] == before["connector_outbox"] + 1 - - -def test_snapshot_observation_context_matches_each_herdr_persistence_barrier( - tmp_path: Path, - monkeypatch: Any, -) -> None: - calls: list[SnapshotObservationContext] = [] - binding_calls: list[tuple[Any, str | None, bool, bool]] = [] - original_save_snapshot = save_snapshot - - def recording_save_snapshot( - db_path: Path, - snapshot: Any, - *, - turn_model: str, - observation: SnapshotObservationContext | None = None, - worker_bindings: Any = None, - binding_backend: str | None = None, - binding_observation_authoritative: bool = False, - binding_workers_present: bool = True, - ) -> bool: - assert observation is not None - calls.append(observation) - binding_calls.append( - ( - worker_bindings, - binding_backend, - binding_observation_authoritative, - binding_workers_present, - ) - ) - return original_save_snapshot( - db_path, - snapshot, - turn_model=turn_model, - observation=observation, - worker_bindings=worker_bindings, - binding_backend=binding_backend, - binding_observation_authoritative=binding_observation_authoritative, - binding_workers_present=binding_workers_present, - ) - - monkeypatch.setattr( - "tendwire.backends.herdr_events.save_snapshot", - recording_save_snapshot, - ) - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, "herdr-observation-context") - - backend.reconcile_once(client=_initial_pane_client()) - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - backend.queue_event_envelope(_status_event("blocked")) - _set_observation_time(monkeypatch, "2026-01-01T00:00:20Z") - backend._mark_worker_cap_exceeded_locked(999) - _set_observation_time(monkeypatch, "2026-01-01T00:00:30Z") - backend._mark_unhealthy("socket_disconnected") - - assert [(call.authority, call.observed_at) for call in calls] == [ - ("complete", "2026-01-01T00:00:00Z"), - ("positive", "2026-01-01T00:00:10Z"), - ("none", "2026-01-01T00:00:20Z"), - ("none", "2026-01-01T00:00:30Z"), - ] - assert binding_calls[0][0] - assert binding_calls[0][1:] == ("herdr", True, True) - assert all(call[0] is None for call in binding_calls[1:]) - -def test_event_snapshot_expiry_serializes_delayed_older_observer( - tmp_path: Path, - monkeypatch: Any, -) -> None: - from tendwire.store import sqlite as store_sqlite - - backend = _backend(tmp_path, "herdr-atomic-expiry") - worker = Worker(id="worker-expiry", name="Worker Expiry", status="active") - - def binding(target: str, observed_at: str, fingerprint: str) -> WorkerBinding: - return WorkerBinding( - host_id=backend.config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value=target, - sendable=True, - observed_at=observed_at, - private_fingerprint=fingerprint, - ) - - initial = Snapshot( - host_id=backend.config.host_id, - updated_at="2026-01-01T00:00:00+00:00", - workers=[worker], - ) - init_store(backend.db_path) - assert save_snapshot( - backend.db_path, - initial, - worker_bindings=[ - binding( - "initial-private-target", - "2026-01-01T00:00:00+00:00", - "initial-private-owner", - ) - ], - binding_backend="herdr", - binding_observation_authoritative=True, - ) is True - older = Snapshot( - host_id=backend.config.host_id, - updated_at="2026-01-01T00:00:01+00:00", - workers=[], - ) - newer = Snapshot( - host_id=backend.config.host_id, - updated_at="2026-01-01T00:00:02+00:00", - workers=[worker], - ) - newer_binding = binding( - "newer-private-target", - "2026-01-01T00:00:02+00:00", - "newer-private-owner", - ) - original_expire = store_sqlite._expire_stale_worker_bindings_conn - older_inside_transaction = threading.Event() - release_older = threading.Event() - - def delayed_expire(conn: sqlite3.Connection, host_id: str, **kwargs: Any) -> int: - if kwargs.get("now") == "2026-01-01T00:00:01+00:00": - older_inside_transaction.set() - assert release_older.wait(timeout=10) - return original_expire(conn, host_id, **kwargs) - - monkeypatch.setattr( - store_sqlite, - "_expire_stale_worker_bindings_conn", - delayed_expire, - ) - errors: list[BaseException] = [] - - def save_older() -> None: - try: - backend._save_snapshot( - older, - observation=SnapshotObservationContext( - authority="complete", - observed_at=older.updated_at, - ), - worker_bindings=[], - binding_observation_authoritative=True, - binding_workers_present=False, - ) - except BaseException as exc: - errors.append(exc) - - def save_newer() -> None: - try: - backend._save_snapshot( - newer, - observation=SnapshotObservationContext( - authority="complete", - observed_at=newer.updated_at, - ), - worker_bindings=[newer_binding], - binding_observation_authoritative=True, - binding_workers_present=True, - ) - except BaseException as exc: - errors.append(exc) - - older_thread = threading.Thread(target=save_older) - newer_thread = threading.Thread(target=save_newer) - older_thread.start() - assert older_inside_transaction.wait(timeout=10) - newer_thread.start() - time.sleep(0.05) - release_older.set() - older_thread.join(timeout=10) - newer_thread.join(timeout=10) - - assert not errors - assert not older_thread.is_alive() and not newer_thread.is_alive() - active = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - now="2026-01-01T00:00:03+00:00", - ) - assert len(active) == 1 - assert active[0].target_value == "newer-private-target" - - -def test_same_fingerprint_observations_refresh_attention_and_run_bounded_cadence( - tmp_path: Path, - monkeypatch: Any, -) -> None: - config = Config( - host_id="same-fingerprint-cadence", - data_dir=tmp_path, - db_path=tmp_path / "same-fingerprint-cadence.db", - herdr_backend="socket", - snapshot_retention_days=14, - snapshot_retention_count=1, - snapshot_maintenance_batch_size=100, - store_maintenance_cadence_seconds=3600, - acknowledged_final_retention_days=33, - acknowledged_final_retention_count=456, - command_retry_horizon_seconds=120, - command_receipt_retention_seconds=691_200, - command_receipt_retention_count=77, - ) - init_store(Path(config.db_path)) - maintenance_times = iter( - ( - "2026-01-01T00:00:00Z", - "2026-01-01T00:00:10Z", - "2026-01-01T00:00:20Z", - "2026-01-01T01:00:10Z", - ) - ) - maintenance_calls: list[ - tuple[SnapshotRetentionPolicy, int, int, int, int, int, int, dict[str, Any]] - ] = [] - - def fixed_clock_maintenance( - db_path: Path, - *, - policy: SnapshotRetentionPolicy, - turn_model: str = "legacy", - acknowledged_final_retention_days: int = 30, - acknowledged_final_retention_count: int = 4096, - command_retry_horizon_seconds: int = 604_800, - command_receipt_retention_seconds: int = 2_592_000, - command_receipt_retention_count: int = 4096, - cadence_seconds: int = 3600, - now: str | None = None, - ) -> dict[str, Any]: - assert now is None - result = maybe_run_automatic_store_maintenance( - db_path, - policy=policy, - turn_model=turn_model, - cadence_seconds=cadence_seconds, - acknowledged_final_retention_days=acknowledged_final_retention_days, - acknowledged_final_retention_count=acknowledged_final_retention_count, - command_retry_horizon_seconds=command_retry_horizon_seconds, - command_receipt_retention_seconds=command_receipt_retention_seconds, - command_receipt_retention_count=command_receipt_retention_count, - now=next(maintenance_times), - ) - maintenance_calls.append( - ( - policy, - acknowledged_final_retention_days, - acknowledged_final_retention_count, - command_retry_horizon_seconds, - command_receipt_retention_seconds, - command_receipt_retention_count, - cadence_seconds, - result, - ) - ) - return result - - monkeypatch.setattr( - "tendwire.backends.herdr_events.maybe_run_automatic_store_maintenance", - fixed_clock_maintenance, - ) - saved_fingerprints: list[str] = [] - - def recording_save( - db_path: Path, - snapshot: Any, - *, - turn_model: str, - observation: SnapshotObservationContext | None = None, - worker_bindings: Any = None, - binding_backend: str | None = None, - binding_observation_authoritative: bool = False, - binding_workers_present: bool = True, - ) -> bool: - saved_fingerprints.append(snapshot.content_fingerprint) - return save_snapshot( - db_path, - snapshot, - turn_model=turn_model, - observation=observation, - worker_bindings=worker_bindings, - binding_backend=binding_backend, - binding_observation_authoritative=binding_observation_authoritative, - binding_workers_present=binding_workers_present, - ) - - monkeypatch.setattr("tendwire.backends.herdr_events.save_snapshot", recording_save) - backend = HerdrEventBackend(config, debounce_seconds=0) - - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend.reconcile_once(client=_initial_pane_client()) - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - assert backend.queue_event_envelope(_status_event("blocked")) is True - assert _attention_lifecycle_rows(backend)[0][4] == "2026-01-01T00:00:10+00:00" - - _set_observation_time(monkeypatch, "2026-01-01T00:00:20Z") - backend._persist_current_state(observed_at="2026-01-01T00:00:20Z") - assert _attention_lifecycle_rows(backend)[0][4] == "2026-01-01T00:00:20+00:00" - - _set_observation_time(monkeypatch, "2026-01-01T01:00:10Z") - backend._persist_current_state(observed_at="2026-01-01T01:00:10Z") - - assert len(maintenance_calls) == 4 - assert len(saved_fingerprints) == 4 - assert len(set(saved_fingerprints[1:])) == 1 - assert [result["status"] for *_, result in maintenance_calls] == [ - "ok", - "not_due", - "not_due", - "ok", - ] - assert sum(bool(result["due"]) for *_, result in maintenance_calls) == 2 - assert { - ( - policy.retention_days, - policy.retention_count, - policy.batch_size, - final_days, - final_count, - retry_horizon, - retention_seconds, - retention_count, - cadence, - ) - for ( - policy, - final_days, - final_count, - retry_horizon, - retention_seconds, - retention_count, - cadence, - _, - ) in maintenance_calls - } == {(14, 1, 100, 33, 456, 120, 691_200, 77, 3600)} - assert _table_count(backend.db_path, config.host_id, "snapshots") == 1 - assert _attention_lifecycle_rows(backend)[0][4] == "2026-01-01T01:00:10+00:00" - assert backend.operational_status["automatic_maintenance"] == { - "ok": True, - "status": "ok", - "due": True, - "examined": 1, - "deleted": 1, - "remaining_candidates": False, - } - - -def test_maintenance_failure_cannot_undo_committed_backend_snapshot( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "maintenance-failure-contained") - - def fail_maintenance(*_args: Any, **_kwargs: Any) -> dict[str, Any]: - raise RuntimeError(f"private maintenance failure at {tmp_path}/secret.db") - - monkeypatch.setattr( - "tendwire.backends.herdr_events.maybe_run_automatic_store_maintenance", - fail_maintenance, - ) - - snapshot = backend.reconcile_once(client=_initial_pane_client()) - persisted = latest_snapshot(backend.db_path, backend.config.host_id) - status = backend.operational_status - encoded = json.dumps(status, sort_keys=True) - - assert persisted is not None - assert persisted.content_fingerprint == snapshot.content_fingerprint - assert status["automatic_maintenance"] == { - "ok": False, - "status": "failed", - "due": False, - "examined": 0, - "deleted": 0, - "remaining_candidates": False, - } - assert str(tmp_path) not in encoded - assert "secret.db" not in encoded - _assert_no_public_json_forbidden(status) - - -def test_incremental_positive_lifecycle_and_complete_absence_are_separate( - tmp_path: Path, - monkeypatch: Any, -) -> None: - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, "incremental-lifecycle") - backend.reconcile_once(client=_initial_pane_client()) - assert _attention_lifecycle_rows(backend) == () - - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - backend.queue_event_envelope(_status_event("blocked")) - opened = _attention_lifecycle_rows(backend) - assert len(opened) == 1 - assert opened[0][0:2] == (1, "open") - assert opened[0][5:7] == (None, 0) - initial_rank = int(opened[0][9]) - assert _attention_event_types(backend) == ["attention_created"] - - _set_observation_time(monkeypatch, "2026-01-01T00:00:20Z") - backend.queue_event_envelope(_status_event("failed")) - escalated = _attention_lifecycle_rows(backend) - assert len(escalated) == 1 - assert escalated[0][0:2] == (1, "open") - assert int(escalated[0][9]) > initial_rank - assert _attention_event_types(backend) == [ - "attention_created", - "attention_escalated", - ] - current = list_attention_items(backend.db_path, backend.config.host_id) - assert len(current) == 1 - assert current[0]["severity"] == "critical" - - _set_observation_time(monkeypatch, "2026-01-01T00:00:30Z") - backend.reconcile_once(client=_initial_pane_client()) - first_missing = _attention_lifecycle_rows(backend) - assert first_missing[0][0:2] == (1, "open") - assert first_missing[0][5] is not None - assert first_missing[0][6] == 1 - - _set_observation_time(monkeypatch, "2026-01-01T00:00:40Z") - backend.queue_event_envelope(_status_event("blocked")) - cleared = _attention_lifecycle_rows(backend) - assert cleared[0][0:2] == (1, "open") - assert cleared[0][5:7] == (None, 0) - assert _attention_event_types(backend) == [ - "attention_created", - "attention_escalated", - ] - - _set_observation_time(monkeypatch, "2026-01-01T00:00:50Z") - backend.queue_event_envelope(_status_event("idle")) - assert _attention_lifecycle_rows(backend) == cleared - - _set_observation_time(monkeypatch, "2026-01-01T00:01:40Z") - backend.reconcile_once(client=_initial_pane_client()) - pending = _attention_lifecycle_rows(backend) - assert pending[0][0:2] == (1, "open") - assert pending[0][5] is not None - assert pending[0][6] == 1 - pending_since = pending[0][5] - - _set_observation_time(monkeypatch, "2026-01-01T00:03:40Z") - backend.reconcile_once(client=_initial_pane_client()) - resolved = _attention_lifecycle_rows(backend) - assert resolved[0][0:3] == (1, "resolved", None) - assert resolved[0][5:7] == (pending_since, 2) - assert list_attention_items(backend.db_path, backend.config.host_id) == [] - assert _attention_event_types(backend) == [ - "attention_created", - "attention_escalated", - ] - - -def test_non_authoritative_herdr_saves_do_not_advance_pending_absence( - tmp_path: Path, - monkeypatch: Any, -) -> None: - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, "non-authoritative-lifecycle") - backend.reconcile_once(client=_initial_pane_client()) - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - backend.queue_event_envelope(_status_event("blocked")) - _set_observation_time(monkeypatch, "2026-01-01T00:01:40Z") - backend.reconcile_once(client=_initial_pane_client()) - - pending = _attention_lifecycle_rows(backend) - outbox = _attention_event_types(backend) - assert pending[0][5] is not None - assert pending[0][6] == 1 - - for timestamp, outcome in ( - ("2026-01-01T00:03:40Z", "socket_disconnected"), - ("2026-01-01T00:04:40Z", "protocol_error"), - ("2026-01-01T00:05:40Z", "continuity_unavailable"), - ): - _set_observation_time(monkeypatch, timestamp) - backend._mark_unhealthy(outcome) - assert _attention_lifecycle_rows(backend) == pending - assert _attention_event_types(backend) == outbox - - failed_worker = backend._workers[next(iter(backend._workers))] - backend._workers[failed_worker.id] = Worker( - id=failed_worker.id, - name=failed_worker.name, - status="failed", - space_id=failed_worker.space_id, - meta=failed_worker.meta, - last_seen_at=failed_worker.last_seen_at, - summary=failed_worker.summary, - backend_target=failed_worker.backend_target, - ) - _set_observation_time(monkeypatch, "2026-01-01T00:06:40Z") - backend._persist_current_state(observed_at="2026-01-01T00:06:40Z") - assert _attention_lifecycle_rows(backend) == pending - assert _attention_event_types(backend) == outbox - - _set_observation_time(monkeypatch, "2026-01-01T00:07:40Z") - backend._mark_worker_cap_exceeded_locked(999) - assert _attention_lifecycle_rows(backend) == pending - assert _attention_event_types(backend) == outbox - - -def test_event_flush_and_direct_persistence_use_the_same_lifecycle_executor( - tmp_path: Path, - monkeypatch: Any, -) -> None: - def exercise(host_id: str, *, direct: bool) -> tuple[Any, ...]: - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, host_id) - backend.reconcile_once(client=_initial_pane_client()) - for timestamp, status in ( - ("2026-01-01T00:00:10Z", "blocked"), - ("2026-01-01T00:00:20Z", "failed"), - ): - _set_observation_time(monkeypatch, timestamp) - envelope = _status_event(status) - if direct: - event = normalize_event(envelope) - assert event is not None - assert backend._apply_event(event) is True - backend._persist_current_state(observed_at=timestamp) - else: - assert backend.queue_event_envelope(envelope) is True - lifecycle = _attention_lifecycle_rows(backend) - public = list_attention_items(backend.db_path, backend.config.host_id) - return ( - lifecycle[0][0], - lifecycle[0][1], - lifecycle[0][5], - lifecycle[0][6], - lifecycle[0][9], - _attention_event_types(backend), - public[0]["severity"], - public[0]["status"], - public[0]["lifecycle_status"], - ) - - assert exercise("event-lifecycle-path", direct=False) == exercise( - "direct-lifecycle-path", - direct=True, - ) - - -def test_restart_preserves_pending_absence_until_complete_confirmation( - tmp_path: Path, - monkeypatch: Any, -) -> None: - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, "restart-pending-lifecycle") - backend.reconcile_once(client=_initial_pane_client()) - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - backend.queue_event_envelope(_status_event("blocked")) - _set_observation_time(monkeypatch, "2026-01-01T00:01:40Z") - backend.reconcile_once(client=_initial_pane_client()) - pending = _attention_lifecycle_rows(backend) - assert pending[0][0:2] == (1, "open") - assert pending[0][5] is not None - assert pending[0][6] == 1 - pending_since = pending[0][5] - - restarted = HerdrEventBackend( - backend.config, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - assert _attention_lifecycle_rows(restarted) == pending - - _set_observation_time(monkeypatch, "2026-01-01T00:03:40Z") - restarted.reconcile_once(client=_initial_pane_client()) - resolved = _attention_lifecycle_rows(restarted) - assert resolved[0][0:3] == (1, "resolved", None) - assert resolved[0][5:7] == (pending_since, 2) - assert _attention_event_types(restarted) == ["attention_created"] - - - -@pytest.mark.parametrize("identity_kind", ["event_id", "producer_sequence"]) -def test_forward_compatible_producer_identity_dedupes_retries_with_bounded_lru( - tmp_path: Path, - identity_kind: str, -) -> None: - config = _config(tmp_path, f"producer-dedupe-{identity_kind}") - init_store(Path(config.db_path)) - backend = HerdrEventBackend( - config, - debounce_seconds=0, - reconnect_delay_seconds=0, - dedupe_size=2, - ) - backend.reconcile_once(client=_initial_pane_client()) - - def identity(value: int) -> dict[str, Any]: - if identity_kind == "event_id": - return {"event_id": f"event-{value}"} - return {"server_id": "server-1", "sequence": value} - - first = {**_status_event("blocked"), **identity(1)} - retry = {**_status_event("failed"), **identity(1)} - assert backend.queue_event_envelope(first) is True - assert backend.queue_event_envelope(retry) is False - assert backend.queue_event_envelope({**_status_event("idle"), **identity(2)}) is True - assert backend.queue_event_envelope({**_status_event("working"), **identity(3)}) is True - assert len(backend._producer_dedupe) == 2 - assert backend.queue_event_envelope({**_status_event("waiting"), **identity(1)}) is True - assert len(backend._producer_dedupe) == 2 - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "waiting" - - -def test_pending_producer_identity_is_not_committed_before_successful_flush( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "pending-producer-dedupe", debounce_seconds=60) - backend.reconcile_once(client=_initial_pane_client()) - identity = HerdrEventId("pending-event") - first = {**_status_event("blocked"), "event_id": identity.value} - duplicate = {**_status_event("failed"), "event_id": identity.value} - - assert backend.queue_event_envelope(first, flush=False) is True - assert backend._producer_dedupe == {} - assert len(backend._pending_events) == 1 - assert backend.queue_event_envelope(duplicate, flush=False) is False - assert backend._producer_dedupe == {} - assert len(backend._pending_events) == 1 - - backend.flush() - - assert list(backend._producer_dedupe) == [identity] - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "blocked" - - -def test_continuity_failure_leaves_producer_identity_retryable( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "continuity-producer-retry") - backend.reconcile_once(client=_initial_pane_client()) - identity = HerdrEventId("continuity-retry") - envelope = {**_status_event("blocked"), "event_id": identity.value} - original_apply = backend._apply_event - fail_next = True - - def fail_once(event: Any) -> bool: - nonlocal fail_next - if fail_next: - fail_next = False - raise HerdrContinuityUnavailableError("continuity unavailable") - return original_apply(event) - - monkeypatch.setattr(backend, "_apply_event", fail_once) - - assert backend.queue_event_envelope(envelope) is True - assert identity not in backend._producer_dedupe - failed = latest_snapshot(backend.db_path, backend.config.host_id) - assert failed is not None - assert failed.workers[0].status == "active" - assert failed.backend_health[0].outcome == "continuity_unavailable" - - assert backend.queue_event_envelope(envelope) is True - recovered = latest_snapshot(backend.db_path, backend.config.host_id) - assert recovered is not None - assert recovered.workers[0].status == "blocked" - assert identity in backend._producer_dedupe - assert backend.queue_event_envelope(envelope) is False - - -def test_snapshot_failure_leaves_identity_retryable_and_retry_persists_dirty_state( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "snapshot-producer-retry") - backend.reconcile_once(client=_initial_pane_client()) - identity = HerdrEventId("snapshot-retry") - envelope = {**_status_event("blocked"), "event_id": identity.value} - original_persist = backend._persist_current_state - fail_next = True - - def fail_once(*, observed_at: str | None = None) -> Any: - nonlocal fail_next - if fail_next: - fail_next = False - raise RuntimeError("snapshot unavailable") - return original_persist(observed_at=observed_at) - - monkeypatch.setattr(backend, "_persist_current_state", fail_once) - - with pytest.raises(RuntimeError, match="snapshot unavailable"): - backend.queue_event_envelope(envelope) - assert identity not in backend._producer_dedupe - stale = latest_snapshot(backend.db_path, backend.config.host_id) - assert stale is not None - assert stale.workers[0].status == "active" - assert backend._workers[stale.workers[0].id].status == "blocked" - - assert backend.queue_event_envelope(envelope) is True - persisted = latest_snapshot(backend.db_path, backend.config.host_id) - assert persisted is not None - assert persisted.workers[0].status == "blocked" - assert identity in backend._producer_dedupe - assert backend.queue_event_envelope(envelope) is False - - -def test_concurrent_queueing_applies_events_in_backend_lock_order( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "concurrent-event-order", debounce_seconds=60) - backend.reconcile_once(client=_initial_pane_client()) - first_queued = threading.Event() - idle_queued = threading.Event() - accepted: dict[str, bool] = {} - applied: list[str] = [] - original_apply = backend._apply_event - - def recording_apply(event: Any) -> bool: - applied.append(str(event.payload["status"])) - return original_apply(event) - - monkeypatch.setattr(backend, "_apply_event", recording_apply) - - def queue_working_events() -> None: - accepted["first"] = backend.queue_event_envelope(_status_event("working"), flush=False) - first_queued.set() - if idle_queued.wait(1): - accepted["last"] = backend.queue_event_envelope(_status_event("working"), flush=False) - - def queue_idle_event() -> None: - if first_queued.wait(1): - accepted["middle"] = backend.queue_event_envelope(_status_event("idle"), flush=False) - idle_queued.set() - - working_thread = threading.Thread(target=queue_working_events) - idle_thread = threading.Thread(target=queue_idle_event) - working_thread.start() - idle_thread.start() - working_thread.join(timeout=1) - idle_thread.join(timeout=1) - - assert working_thread.is_alive() is False - assert idle_thread.is_alive() is False - assert accepted == {"first": True, "middle": True, "last": True} - backend.flush() - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert applied == ["working", "idle", "working"] - assert snapshot.workers[0].status == "active" - assert len(snapshot.workers) == 1 - assert len(list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")) == 1 - - - - - - - - - - - - - - - - - - -def test_pane_moved_preserves_public_worker_and_updates_private_binding(tmp_path: Path) -> None: - backend = _backend(tmp_path, "pane-moved") - backend.reconcile_once(client=_initial_pane_client()) - before = latest_snapshot(backend.db_path, backend.config.host_id) - assert before is not None - worker_id = before.workers[0].id - binding = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - - backend.queue_event_envelope( - {"event": "pane.moved", "data": { - "old_pane_id": "pane-1", - "pane_id": "pane-2", - "agent": "Agent One", - "workspace_id": "space-1", - }} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - moved_binding = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - assert after is not None - assert after.workers[0].id == worker_id - assert moved_binding.private_fingerprint == binding.private_fingerprint - assert moved_binding.target_kind == "pane_id" - assert moved_binding.target_value == "pane-2" - - -def test_pane_closed_closes_worker_and_expires_matching_binding(tmp_path: Path) -> None: - backend = _backend(tmp_path, "pane-closed") - backend.reconcile_once(client=_initial_pane_client()) - - backend.queue_event_envelope({"event": "pane.closed", "data": {"pane_id": "pane-1"}}) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "closed" - assert list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") == [] - expired = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - include_expired=True, - ) - assert expired[0].sendable is False - assert expired[0].reason == "pane_closed" - -def test_pane_exited_closes_worker_and_expires_matching_binding(tmp_path: Path) -> None: - backend = _backend(tmp_path, "pane-exited") - backend.reconcile_once(client=_initial_pane_client()) - - backend.queue_event_envelope({"event": "pane.exited", "data": {"pane_id": "pane-1"}}) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "closed" - assert list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") == [] - expired = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - include_expired=True, - ) - assert expired[0].sendable is False - assert expired[0].reason == "pane_exited" - - -def test_disconnect_degraded_state_preserves_workers_and_bindings(tmp_path: Path) -> None: - backend = _backend(tmp_path, "degraded") - backend.reconcile_once(client=_initial_pane_client()) - binding_before = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - - backend._mark_unhealthy("socket_disconnected") - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - binding_after = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - assert snapshot is not None - assert snapshot.workers[0].status == "active" - assert snapshot.backend_health[0].status == "unavailable" - assert binding_after.private_fingerprint == binding_before.private_fingerprint - assert binding_after.sendable is True - - -def test_healthy_empty_reconnect_closes_missing_workers_and_expires_bindings(tmp_path: Path) -> None: - backend = _backend(tmp_path, "healthy-empty") - backend.reconcile_once(client=_initial_pane_client()) - - backend.reconcile_once(client=_StaticClient(workspaces=[], tabs=[], panes=[], agents=[])) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.backend_health[0].status == "healthy" - assert snapshot.backend_health[0].outcome == "empty_healthy" - assert snapshot.workers[0].status == "closed" - assert list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") == [] - - - - -def test_output_excerpt_limit_bounds_public_worker_summary(tmp_path: Path) -> None: - config = Config( - host_id="output-excerpt", - data_dir=tmp_path, - db_path=tmp_path / "output-excerpt.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - output_excerpt_chars=12, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - long_summary = "x" * 40 - - snapshot = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "space-1", "name": "Build"}], - panes=[ - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "description": long_summary, - } - ], - ) - ) - binding = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - - assert snapshot.workers[0].summary == "xxxxxxxxx..." - assert len(snapshot.workers[0].summary or "") == 12 - assert latest_snapshot(backend.db_path, backend.config.host_id).workers[0].summary == "xxxxxxxxx..." - assert binding.worker_fingerprint == snapshot.workers[0].fingerprint - assert long_summary not in snapshot.to_json() - - - - - - - - -def test_periodic_reconcile_uses_config_and_zero_disables_it(tmp_path: Path) -> None: - disabled_config = Config( - host_id="periodic-disabled", - data_dir=tmp_path, - db_path=tmp_path / "periodic-disabled.db", - herdr_backend="socket", - reconcile_interval_seconds=0, - ) - init_store(Path(disabled_config.db_path)) - disabled = HerdrEventBackend(disabled_config, debounce_seconds=0) - disabled_client = _initial_pane_client() - disabled._next_reconcile_monotonic = time.monotonic() - 1 - disabled._run_periodic_reconcile_if_due(disabled_client) - - enabled_config = Config( - host_id="periodic-enabled", - data_dir=tmp_path, - db_path=tmp_path / "periodic-enabled.db", - herdr_backend="socket", - reconcile_interval_seconds=0.001, - ) - init_store(Path(enabled_config.db_path)) - enabled = HerdrEventBackend(enabled_config, debounce_seconds=0) - enabled_client = _initial_pane_client() - enabled._next_reconcile_monotonic = time.monotonic() - 1 - enabled._run_periodic_reconcile_if_due(enabled_client) - - assert disabled_client.calls == [] - assert enabled_client.calls == ["workspace.list", "tab.list", "pane.list", "agent.list"] - assert enabled.operational_status["last_reconcile_at"] is not None - - - - - - - - -def test_debounce_batches_until_flush_and_shutdown_flushes(tmp_path: Path) -> None: - backend = _backend(tmp_path, "debounce", debounce_seconds=60) - backend.reconcile_once(client=_initial_pane_client()) - - backend.queue_event_envelope( - {"event": "pane.agent_status_changed", "data": {"agent": "Agent One", "status": "blocked"}} - ) - pending = latest_snapshot(backend.db_path, backend.config.host_id) - assert pending is not None - assert pending.workers[0].status == "active" - - backend.stop() - - flushed = latest_snapshot(backend.db_path, backend.config.host_id) - assert flushed is not None - assert flushed.workers[0].status == "blocked" - - -def test_idle_event_timeout_keeps_polling_without_marking_backend_unhealthy(tmp_path: Path) -> None: - backend = _backend(tmp_path, "idle-timeout") - backend.reconcile_once(client=_initial_pane_client()) - - class IdleThenEventClient: - def __init__(self) -> None: - self.calls = 0 - - def read_event(self, subscription_id: str, *, timeout: float | None = None) -> dict[str, Any]: - self.calls += 1 - if self.calls == 1: - raise HerdrSocketTimeoutError("idle") - backend.stop_event.set() - return { - "id": subscription_id, - "event": "pane.agent_status_changed", - "data": {"agent": "Agent One", "status": "blocked"}, - } - - client = IdleThenEventClient() - backend._read_event_stream(client, "sub-1") - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "blocked" - assert snapshot.backend_health[0].status == "healthy" - assert client.calls == 2 - - -def test_mark_unhealthy_safe_sets_ready_even_when_persist_fails(tmp_path: Path, monkeypatch: Any) -> None: - backend = _backend(tmp_path, "ready-on-error") - - def boom(*_args: Any, **_kwargs: Any) -> None: - raise RuntimeError("store unavailable") - - monkeypatch.setattr("tendwire.backends.herdr_events.save_snapshot", boom) - - assert backend._mark_unhealthy_safe("protocol_error") is None - assert backend.ready is True - - -def test_run_forever_retries_when_unhealthy_persistence_fails( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "retry-after-health-persist-error") - reconcile_calls = 0 - clients_created = 0 - - def client_factory(_config: Config) -> object: - nonlocal clients_created - clients_created += 1 - return object() - - def reconcile_once(*, client: object) -> None: - nonlocal reconcile_calls - reconcile_calls += 1 - if reconcile_calls == 1: - raise RuntimeError("observation failed") - backend.stop_event.set() - - def persist_failure(*_args: Any, **_kwargs: Any) -> None: - raise local_state_error(LocalStateErrorCode.OPERATION_FAILED) - - backend.client_factory = client_factory - monkeypatch.setattr(backend, "reconcile_once", reconcile_once) - monkeypatch.setattr("tendwire.backends.herdr_events.save_snapshot", persist_failure) - - backend.run_forever() - - assert reconcile_calls == 2 - assert clients_created == 2 - assert backend.ready is True - assert backend.health.outcome == "unknown" - - -def test_protocol_error_health_is_degraded_and_specific(tmp_path: Path) -> None: - backend = _backend(tmp_path, "protocol-health") - - backend._mark_unhealthy("protocol_error") - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "protocol_error" - - -def test_daemon_starts_socket_backend_only_when_configured(tmp_path: Path) -> None: - db_path = tmp_path / "daemon-socket.db" - socket_path = tmp_path / "daemon.sock" - config = Config( - host_id="daemon-socket", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - calls: list[str] = [] - - class FakeBackend: - def __init__(self, config: Config, stop_event: threading.Event) -> None: - self.config = config - self.stop_event = stop_event - - def start(self, *, wait_for_reconcile: bool = True) -> None: - calls.append(f"start:{wait_for_reconcile}") - snapshot = project_from_observations( - self.config, - workers=[Worker(id="worker-1", name="Worker", status="active")], - backend_health=[ - BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - counts={"workers": 1}, - ) - ], - ) - save_snapshot(Path(self.config.db_path), snapshot) - - def stop(self) -> None: - calls.append("stop") - - def observe_cli(_config: Config) -> Any: - raise AssertionError("CLI observation must not run in socket mode") - - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - observe_initial_snapshot=observe_cli, - event_backend_factory=lambda cfg, stop_event: FakeBackend(cfg, stop_event), - ), - ) - daemon.start() - try: - assert calls == ["start:True"] - assert daemon.get_snapshot().workers[0].id == "worker-1" - _assert_no_public_json_forbidden(daemon.get_health()) - finally: - daemon.stop() - - assert "stop" in calls - - -def test_daemon_socket_fallback_uses_backend_health_when_snapshot_missing(tmp_path: Path) -> None: - db_path = tmp_path / "daemon-fallback.db" - socket_path = tmp_path / "daemon-fallback.sock" - config = Config( - host_id="daemon-fallback", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - - class FakeBackend: - def __init__(self, config: Config, stop_event: threading.Event) -> None: - self.config = config - self.stop_event = stop_event - self.health = HerdrEventBackend(config)._health_for("protocol_error") - - def start(self, *, wait_for_reconcile: bool = True) -> None: - return None - - def stop(self) -> None: - return None - - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(event_backend_factory=lambda cfg, stop_event: FakeBackend(cfg, stop_event)), - ) - daemon.start() - try: - snapshot = daemon.get_snapshot() - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "protocol_error" - finally: - daemon.stop() - - -def test_status_event_with_pane_id_only_updates_bound_worker_not_a_phantom(tmp_path: Path) -> None: - """Regression: status events that only carry a pane id must resolve through - the binding turn target instead of inserting a duplicate re-lettered worker - that freezes the real worker's status (the 'stuck working icon' bug).""" - backend = _backend(tmp_path, "phantom-host") - client = _StaticClient( - workspaces=[{"id": "space-1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "w1:p1", - "terminal_id": "term-1", - "agent": "claude", - "workspace_id": "space-1", - "agent_status": "working", - }, - { - "pane_id": "w1:p2", - "terminal_id": "term-2", - "agent": "claude", - "workspace_id": "space-1", - "agent_status": "working", - }, - ], - agents=[], - ) - backend.reconcile_once(client=client) - snapshot = latest_snapshot(Path(backend.db_path), backend.config.host_id) - assert snapshot is not None - ids_before = sorted(worker.id for worker in snapshot.workers) - assert ids_before == ["claude-1", "claude-2"] - - event = normalize_event( - {"event": "pane.agent_status_changed", "data": {"pane": {"pane_id": "w1:p2", "agent": "claude", "status": "idle"}}} - ) - assert event is not None - assert backend._apply_event(event) is True - backend._persist_projection_locked() if hasattr(backend, "_persist_projection_locked") else None - workers = backend._workers - assert sorted(workers) == ["claude-1", "claude-2"], f"phantom worker inserted: {sorted(workers)}" - by_target = {} - for binding in backend._bindings.values(): - by_target[binding.target_value] = binding.worker_id - idle_worker_id = by_target["term-2"] - assert workers[idle_worker_id].status in {"idle", "done"} - - - - -def test_reconcile_drops_unbound_missing_workers_but_keeps_bound_closed(tmp_path: Path) -> None: - backend = _backend(tmp_path, "phantom-aging-host") - worker_bound = Worker(id="codex-1", name="codex", status="active", space_id="wX8") - worker_phantom = Worker(id="codex", name="codex", status="working", space_id="wX8") - merged = backend._workers_with_closed_missing( - [worker_bound, worker_phantom], - [], - bound_worker_ids={"codex-1"}, - ) - ids = {worker.id: worker.status for worker in merged} - assert "codex" not in ids, "unbound phantom must be dropped, not carried as closed" - assert ids.get("codex-1") == "closed" - - -def test_nested_agent_pane_claim_cannot_poison_terminal_close_matching( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "pane-terminal-provenance") - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "W1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "P1", - "terminal_id": "T1", - "workspace_id": "W1", - "agent": "codex", - "agent_status": "working", - } - ], - ) - ) - worker = initial.workers[0] - binding = next(iter(backend._bindings.values())) - assert backend._pane_terminals == {"P1": "T1"} - - assert backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "worker_id": "nested-agent-claimant", - "pane_id": "Pfake", - "terminal_id": "T1", - "agent": "codex", - "status": "working", - } - }} - ) - assert backend._pane_terminals == {"P1": "T1"} - assert backend._pane_owners == {"P1": {worker.id}} - - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane_id": "Pfake"}} - ) - - current = backend._workers[worker.id] - assert current.status == worker.status - assert current.status != "closed" - assert "Pfake" not in backend._pane_terminals - assert backend._pane_owners == {"P1": {worker.id}} - stored = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert len(stored) == 1 - assert stored[0].private_fingerprint == binding.private_fingerprint - assert stored[0].sendable is True - - -def test_reconcile_maps_only_accepted_pane_info_rows( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "reconcile-pane-map-provenance") - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "W1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "P1", - "terminal_id": "T1", - "workspace_id": "W1", - "agent": "codex", - "agent_status": "working", - }, - { - "pane_id": "Pfake", - "terminal_id": "T1", - "workspace_id": "W1", - "agent_status": "working", - }, - ], - ) - ) - worker = initial.workers[0] - binding = next(iter(backend._bindings.values())) - assert backend._pane_terminals == {"P1": "T1"} - assert backend._pane_owners == {"P1": {worker.id}} - - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane_id": "Pfake"}} - ) - - assert backend._workers[worker.id].status == worker.status - assert backend._workers[worker.id].status != "closed" - stored = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert len(stored) == 1 - assert stored[0].private_fingerprint == binding.private_fingerprint - - -@pytest.mark.parametrize( - ("source_field", "source_value"), - [ - ("previous_pane_id", "P1"), - ("previous_terminal_id", "T1"), - ], -) -def test_accepted_move_removes_source_pane_terminal_alias_before_stale_close( - tmp_path: Path, - source_field: str, - source_value: str, -) -> None: - backend = _backend(tmp_path, "moved-pane-map-provenance") - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "W1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "P1", - "terminal_id": "T1", - "workspace_id": "W1", - "agent": "codex", - "agent_status": "working", - } - ], - ) - ) - worker = initial.workers[0] - - assert backend.queue_event_envelope( - {"event": "pane.moved", "data": { - source_field: source_value, - "pane": { - "pane_id": "P2", - "terminal_id": "T1", - "workspace_id": "W1", - "agent": "codex", - "agent_status": "working", - }, - }} - ) - assert backend._pane_terminals == {"P2": "T1"} - assert backend._pane_owners == {"P2": {worker.id}} - - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane_id": "P1"}} - ) - - assert backend._workers[worker.id].status == worker.status - assert backend._workers[worker.id].status != "closed" - assert backend._pane_terminals == {"P2": "T1"} diff --git a/tests/test_herdr_protocol.py b/tests/test_herdr_protocol.py index d425611..0b786cf 100644 --- a/tests/test_herdr_protocol.py +++ b/tests/test_herdr_protocol.py @@ -1,288 +1,38 @@ -"""Tests for the inactive Herdr socket protocol helpers.""" - from __future__ import annotations -import json -from pathlib import Path - import pytest from tendwire.backends.herdr_protocol import ( HerdrEnvelopeError, HerdrMalformedLineError, HerdrRequestIdMismatchError, - HerdrSocketPathError, - HERDR_EVENTS_SUBSCRIBE_METHOD, - HERDR_OFFICIAL_EVENT_NAME_SET, - HERDR_OFFICIAL_EVENT_NAMES, - build_events_subscribe_params, - build_events_subscribe_request, build_request, ensure_response_id, - error_payload, frame_request, - is_error_response, - is_event, - is_result_response, parse_json_line, resolve_socket_path, - result_payload, - validate_event, - validate_response, validate_server_envelope, ) -def test_resolve_socket_path_prefers_explicit_absolute_path(tmp_path: Path) -> None: - socket_path = tmp_path / "herdr.sock" - env = {"TENDWIRE_HERDR_SOCKET": str(tmp_path / "ignored.sock")} - - assert resolve_socket_path(socket_path, env=env) == str(socket_path) - - -def test_resolve_socket_path_expands_home_for_explicit_path(tmp_path: Path) -> None: - home = tmp_path / "home" - - assert resolve_socket_path("~/custom.sock", home=home) == str(home / "custom.sock") - - -def test_resolve_socket_path_rejects_relative_explicit_path() -> None: - with pytest.raises(HerdrSocketPathError): - resolve_socket_path("relative.sock") - - -def test_resolve_socket_path_env_order_and_home_expansion(tmp_path: Path) -> None: - home = tmp_path / "home" - env = { - "TENDWIRE_HERDR_SOCKET": "~/primary.sock", - "HERDR_SOCKET_PATH": str(tmp_path / "secondary.sock"), - "TENDWIRE_HERDR_SESSION": "session-a", - "HERDR_SESSION": "session-b", - } - - assert resolve_socket_path(env=env, home=home) == str(home / "primary.sock") - - -def test_resolve_socket_path_uses_herdr_socket_path_when_primary_empty(tmp_path: Path) -> None: - env = { - "TENDWIRE_HERDR_SOCKET": "", - "HERDR_SOCKET_PATH": str(tmp_path / "secondary.sock"), - "TENDWIRE_HERDR_SESSION": "ignored", - } - - assert resolve_socket_path(env=env, home=tmp_path) == str(tmp_path / "secondary.sock") - - -def test_resolve_socket_path_uses_tendwire_session_before_herdr_session(tmp_path: Path) -> None: - env = { - "TENDWIRE_HERDR_SESSION": "alpha", - "HERDR_SESSION": "beta", - } - - assert resolve_socket_path(env=env, home=tmp_path) == str( - tmp_path / ".config" / "herdr" / "sessions" / "alpha" / "herdr.sock" - ) - - -def test_resolve_socket_path_uses_herdr_session_when_tendwire_session_empty(tmp_path: Path) -> None: - env = { - "TENDWIRE_HERDR_SOCKET": "", - "HERDR_SOCKET_PATH": " ", - "TENDWIRE_HERDR_SESSION": "", - "HERDR_SESSION": "beta", - } - - assert resolve_socket_path(env=env, home=tmp_path) == str( - tmp_path / ".config" / "herdr" / "sessions" / "beta" / "herdr.sock" - ) - - -def test_resolve_socket_path_defaults_to_config_socket(tmp_path: Path) -> None: - assert resolve_socket_path(env={}, home=tmp_path) == str( - tmp_path / ".config" / "herdr" / "herdr.sock" - ) - - -def test_resolve_socket_path_rejects_relative_env_socket_path() -> None: - with pytest.raises(HerdrSocketPathError): - resolve_socket_path(env={"TENDWIRE_HERDR_SOCKET": "relative.sock"}) - - -def test_build_request_uses_unique_string_ids_and_newline_framing() -> None: - first = build_request("pane.read", {"pane_id": "p-1"}) - second = build_request("pane.read", {"pane_id": "p-1"}) - - assert isinstance(first["id"], str) - assert isinstance(second["id"], str) - assert first["id"] != second["id"] - assert first["method"] == "pane.read" - assert first["params"] == {"pane_id": "p-1"} - - line = frame_request(first) - assert line.endswith(b"\n") - assert json.loads(line.decode("utf-8")) == first - - - -def test_build_events_subscribe_request_uses_official_method_params_and_default_order() -> None: - params = build_events_subscribe_params() - - assert params == {"subscriptions": [{"type": name} for name in HERDR_OFFICIAL_EVENT_NAMES]} - assert build_events_subscribe_request(request_id="sub-1") == { - "id": "sub-1", - "method": HERDR_EVENTS_SUBSCRIBE_METHOD, - "params": params, - } - assert HERDR_OFFICIAL_EVENT_NAME_SET == frozenset(HERDR_OFFICIAL_EVENT_NAMES) - - -@pytest.mark.parametrize("event_name", HERDR_OFFICIAL_EVENT_NAMES) -def test_build_events_subscribe_params_accepts_each_official_event_name(event_name: str) -> None: - assert build_events_subscribe_params([event_name]) == {"subscriptions": [{"type": event_name}]} - - -def test_official_event_subscription_names_exclude_legacy_aliases() -> None: - legacy = { - "pane.observed", - "workspace.observed", - "agent.status_changed", - "agent.detected", - "worktree.updated", - } - - assert legacy.isdisjoint(HERDR_OFFICIAL_EVENT_NAME_SET) +def test_request_round_trip_is_one_correlated_json_line() -> None: + request = build_request("agent.acp_status", {"target": "worker"}, request_id="r1") + assert parse_json_line(frame_request(request)) == request + ensure_response_id({"id": "r1", "result": {}}, "r1") + with pytest.raises(HerdrRequestIdMismatchError): + ensure_response_id({"id": "other", "result": {}}, "r1") -@pytest.mark.parametrize( - "event_names", - [ - ["pane.observed"], - ["workspace.observed"], - ["agent.status_changed"], - ["worktree.updated"], - [""], - [" "], - [" pane.created "], - [123], - [None], - object(), - ], -) -def test_build_events_subscribe_params_rejects_unknown_non_string_and_empty_names( - event_names: object, -) -> None: +def test_protocol_rejects_events_and_malformed_lines() -> None: with pytest.raises(HerdrEnvelopeError): - build_events_subscribe_params(event_names) # type: ignore[arg-type] - -def test_parse_valid_result_error_and_event_envelopes() -> None: - result = validate_response( - parse_json_line( - b'{"id":"req-1","result":{"items":[{"id":"a"}]},"future":"ignored"}\n' - ) - ) - error = validate_response(parse_json_line(b'{"id":"req-2","error":{"message":"no"}}\n')) - event = validate_event( - parse_json_line( - b'{"id":"sub-1","event":"pane.output_matched","data":{"text":"hello"}}\n' - ) - ) - idless_event = validate_event( - parse_json_line( - b'{"event":"pane.agent_status_changed","data":{"status":"blocked"}}\n' - ) - ) - - assert is_result_response(result) is True - assert result_payload(result) == {"items": [{"id": "a"}]} - assert is_error_response(error) is True - assert error_payload(error) == {"message": "no"} - assert is_event(event) is True - assert event == { - "id": "sub-1", - "event": "pane.output_matched", - "data": {"text": "hello"}, - } - assert is_event(idless_event) is True - assert idless_event == { - "event": "pane.agent_status_changed", - "data": {"status": "blocked"}, - } - - -def test_parse_json_line_rejects_malformed_json() -> None: + validate_server_envelope({"event": "pane.created", "data": {}}) with pytest.raises(HerdrMalformedLineError): - parse_json_line(b"{not json}\n") - + parse_json_line(b"not-json\n") -def test_parse_json_line_rejects_malformed_utf8() -> None: - with pytest.raises(HerdrMalformedLineError): - parse_json_line(b"\xff\n") - -def test_validate_server_envelope_rejects_missing_id() -> None: - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope({"result": {"ok": True}}) - - -def test_validate_server_envelope_scopes_herdr_075_uncorrelated_error() -> None: - envelope = { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: missing field pane_id", - }, - } - - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope(envelope) - - assert validate_server_envelope( - envelope, - allow_uncorrelated_error=True, - ) == envelope - - for error in ( - {"code": "permission_denied", "message": "invalid request: denied"}, - {"code": "invalid_request", "message": "subscription unavailable"}, - ): - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope( - {"id": "", "error": error}, - allow_uncorrelated_error=True, - ) - - -def test_validate_server_envelope_rejects_missing_id_error() -> None: - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope({"error": {"message": "uncorrelated"}}) - - -@pytest.mark.parametrize( - "envelope", - [ - {"id": "req-1"}, - {"id": "req-1", "result": {}, "error": {}}, - {"id": "req-1", "event": "pane.output", "result": {}}, - ], -) -def test_validate_server_envelope_rejects_wrong_envelope_shape(envelope: dict[str, object]) -> None: - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope(envelope) - - -def test_parse_json_line_rejects_non_object_envelope() -> None: - with pytest.raises(HerdrEnvelopeError): - parse_json_line(b'["not","an","object"]\n') - - -def test_unknown_fields_are_tolerated_without_changing_result_payload() -> None: - response = validate_response( - {"id": "req-1", "result": {"raw": {"unknown": [1, 2]}}, "extra": {"ignored": True}} +def test_socket_resolution_keeps_frozen_precedence(tmp_path) -> None: + explicit = tmp_path / "explicit.sock" + assert resolve_socket_path(explicit, env={"HERDR_SESSION": "ignored"}) == str(explicit) + assert resolve_socket_path(env={"TENDWIRE_HERDR_SESSION": "work"}, home=tmp_path) == str( + tmp_path / ".config" / "herdr" / "sessions" / "work" / "herdr.sock" ) - - assert result_payload(response) == {"raw": {"unknown": [1, 2]}} - - -def test_ensure_response_id_rejects_mismatch() -> None: - with pytest.raises(HerdrRequestIdMismatchError): - ensure_response_id({"id": "actual", "result": {}}, "expected") diff --git a/tests/test_herdr_smoke.py b/tests/test_herdr_smoke.py deleted file mode 100644 index f333523..0000000 --- a/tests/test_herdr_smoke.py +++ /dev/null @@ -1,678 +0,0 @@ -import importlib.util -import json -import types -from pathlib import Path - - -import pytest - - -PROJECT_ROOT = Path(__file__).resolve().parents[1] -SMOKE_SCRIPT = PROJECT_ROOT / "scripts" / "herdr_smoke.py" -FIXTURE_ROOT = PROJECT_ROOT / "tests" / "fixtures" / "herdr" / "live_smoke" -OK_FIXTURES = FIXTURE_ROOT / "ok" -NEGATIVE_FIXTURES = FIXTURE_ROOT / "negative_private" -RAW_STATUS_FIXTURE = PROJECT_ROOT / "tests" / "fixtures" / "herdr" / "event_replay" / "status_transitions.json" - -REQUIRED_CHECKS = { - "create_attach", - "observe", - "send_addressing", - "target_validation", - "event_subscription", - "status_agent_status_changed", - "pane_moved_binding_update", - "close_exited", - "degraded_backend_preserves_workers", - "public_safety", -} - -# The public schema deliberately contains scenario names such as -# target_validation and pane_moved_binding_update. The banned list therefore -# names concrete private surfaces rather than the neutral words "target" or -# "binding" by themselves. -FORBIDDEN_PUBLIC_TERMS = ( - "telegram", - "herdres", - "raw pane", - "pane_id", - "pane-id", - "terminal_id", - "terminal", - "socket", - "backend_target", - "target_kind", - "target_value", - "private_binding", - "private_fingerprint", - "connector", - "outbox", - "delivery", - "argv", - "env", - "stdout", - "stderr", - "token", - "secret", - "fingerprint", -) - -PRIVATE_MARKERS = ( - "explicit-smoke", - "caller-smoke", - "tendwire-smoke", - "do-not-leak-token", - "do-not-leak-secret", - "actual-private-session", - "socket:///tmp/forbidden.sock", - "private fingerprint abc123", - "pane-secret", - "agent-secret", - "pane-1", - "status_transitions.json", -) - - -def _load_smoke_module(): - spec = importlib.util.spec_from_file_location("tendwire_herdr_smoke_under_test", SMOKE_SCRIPT) - module = importlib.util.module_from_spec(spec) - assert spec.loader is not None - spec.loader.exec_module(module) - return module - - -@pytest.fixture() -def smoke_module(): - return _load_smoke_module() - - -def _patch_which(monkeypatch, module, result): - if not hasattr(module, "shutil"): - monkeypatch.setattr(module, "shutil", types.SimpleNamespace(), raising=False) - monkeypatch.setattr(module.shutil, "which", lambda _name: result, raising=False) - - -def _run_main(module, argv, capsys, *, env=None, runner=None): - try: - return_code = module.main(argv, env={} if env is None else env, runner=runner) - except SystemExit as exc: - return_code = exc.code - - captured = capsys.readouterr() - assert captured.err == "" - public_text = captured.out.strip() - assert public_text, "smoke harness must print one JSON summary" - data = json.loads(public_text) - assert isinstance(data, dict) - module.validate_public_summary(data) - return return_code, public_text, data - - -def _all_strings(value): - if isinstance(value, dict): - for key, child in value.items(): - yield str(key) - yield from _all_strings(child) - elif isinstance(value, list): - for child in value: - yield from _all_strings(child) - elif isinstance(value, (str, int, float, bool)) or value is None: - yield str(value) - - -def _summary_text(data): - return " ".join(_all_strings(data)).lower() - - -def _assert_public_json_safe(public_text, *extra_absent): - lowered = public_text.lower() - for term in FORBIDDEN_PUBLIC_TERMS: - assert term not in lowered - for marker in extra_absent: - assert marker.lower() not in lowered - - -def _is_skip(data): - text = _summary_text(data) - return "skip" in text or "skipped" in text - - -def _is_failure_or_skip(data): - text = _summary_text(data) - return "fail" in text or "failed" in text or "error" in text or _is_skip(data) - - -def _check_records(data): - checks = data.get("checks", []) - if isinstance(checks, dict): - records = [] - for name, record in checks.items(): - if isinstance(record, dict): - records.append({"name": name, **record}) - else: - records.append({"name": name, "status": record}) - return records - assert isinstance(checks, list), "checks must be a list or object" - return [record for record in checks if isinstance(record, dict)] - - -def _check_names(data): - return {record.get("name") for record in _check_records(data)} - - -def _check_by_name(data, name): - for record in _check_records(data): - if record.get("name") == name: - return record - raise AssertionError(f"missing check {name}") - - -class ExplodingRunner: - def __init__(self): - self.calls = [] - - def __call__(self, *args, **kwargs): - self.calls.append((args, kwargs)) - raise AssertionError("runner must not be called") - - -class RecordingRunner: - def __init__(self): - self.calls = [] - - def __call__(self, *args, **kwargs): - argv = args[0] if args else kwargs.get("args") or kwargs.get("argv") - assert isinstance(argv, list), "Herdr commands must be argv lists, not shell strings" - assert all(isinstance(part, str) for part in argv) - assert kwargs.get("shell") is not True - - child_env = kwargs.get("env") - if child_env is None: - for value in args[1:]: - if isinstance(value, dict): - child_env = value - break - assert isinstance(child_env, dict), "runner must receive an explicit child environment" - - self.calls.append({"argv": list(argv), "env": dict(child_env), "kwargs": dict(kwargs)}) - return types.SimpleNamespace(returncode=0, stdout=self._stdout_for(argv), stderr="") - - def _stdout_for(self, argv): - joined = " ".join(part.lower() for part in argv) - if "status" in joined and "server" in joined: - return "status: running\n" - if argv[3:5] == ["agent", "start"]: - return json.dumps( - { - "id": "cli:agent:start", - "result": { - "type": "agent_started", - "agent": { - "name": "tendwire-smoke-address-probe", - "pane_id": "private-pane-id", - }, - }, - } - ) - if argv[3:5] == ["pane", "move"]: - return json.dumps({"id": "cli:pane:move", "result": {"type": "ok"}}) - if argv[3:5] == ["pane", "close"]: - return json.dumps({"id": "cli:pane:close", "result": {"type": "ok"}}) - if "workspace" in joined and "list" in joined: - return json.dumps({"status": "ok", "items": [{"label": "smoke-space"}], "count": 1}) - if "agent" in joined and "list" in joined: - return json.dumps({"status": "ok", "items": [{"name": "smoke-worker"}], "count": 1}) - if "status" in joined or "event" in joined: - return (OK_FIXTURES / "status_agent_status_changed.json").read_text() - if "send" in joined or "address" in joined: - return (OK_FIXTURES / "send_addressing.json").read_text() - return json.dumps({"status": "ok", "items": [{"name": "generic-worker"}], "count": 1}) - - -class StoppedSessionRunner(RecordingRunner): - def _stdout_for(self, argv): - joined = " ".join(part.lower() for part in argv) - if "status" in joined and "server" in joined: - return "status: not running\nsocket: /private/path\n" - raise AssertionError("stopped smoke scope must fail before observe/send commands") - - -class StoppedUnderscoreSessionRunner(RecordingRunner): - def _stdout_for(self, argv): - joined = " ".join(part.lower() for part in argv) - if "status" in joined and "server" in joined: - return json.dumps({"status": "not_running"}) - raise AssertionError("stopped smoke scope must fail before observe/send commands") - - -class NonzeroSendRunner(RecordingRunner): - def __call__(self, *args, **kwargs): - result = super().__call__(*args, **kwargs) - argv = self.calls[-1]["argv"] - if argv[3:5] == ["agent", "send"]: - result.returncode = 1 - result.stdout = "" - result.stderr = "private send failure" - return result - - -class ZeroAcceptedSendRunner(RecordingRunner): - def __call__(self, *args, **kwargs): - result = super().__call__(*args, **kwargs) - argv = self.calls[-1]["argv"] - if argv[3:5] == ["agent", "send"]: - result.returncode = 0 - result.stdout = json.dumps({"status": "ok", "accepted_count": 0}) - result.stderr = "" - return result - - -def test_no_live_opt_in_skips_without_subprocess_calls(smoke_module, monkeypatch, capsys): - runner = ExplodingRunner() - _patch_which(monkeypatch, smoke_module, "/unused/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, [], capsys, env={}, runner=runner) - - assert return_code in (0, None) - assert runner.calls == [] - assert _is_skip(data) - assert data.get("mode") != "live" - assert REQUIRED_CHECKS <= _check_names(data) - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -@pytest.mark.parametrize( - ("argv", "env", "expected_session", "expected_default", "expected_explicit", "expect_send"), - [ - (["--live"], {}, "tendwire-smoke", True, False, True), - (["--live"], {"HERDR_SESSION": "tendwire-smoke"}, "tendwire-smoke", False, True, True), - (["--live", "--session", "explicit-smoke"], {}, "explicit-smoke", False, True, False), - (["--live"], {"HERDR_SESSION": "caller-smoke"}, "caller-smoke", False, True, False), - ], -) -def test_live_session_selection_and_argv_construction( - smoke_module, - monkeypatch, - capsys, - argv, - env, - expected_session, - expected_default, - expected_explicit, - expect_send, -): - runner = RecordingRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, argv, capsys, env=env, runner=runner) - - assert return_code in (0, None) - assert runner.calls, "live opt-in must execute Herdr checks through the injected runner" - for call in runner.calls: - assert "HERDR_SESSION" not in call["env"] - assert isinstance(call["argv"], list) - assert call["kwargs"].get("shell") is not True - assert call["argv"][1:3] == ["--session", expected_session] - send_calls = [call for call in runner.calls if call["argv"][3:5] == ["agent", "send"]] - assert bool(send_calls) is expect_send - start_calls = [call for call in runner.calls if call["argv"][3:5] == ["agent", "start"]] - move_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "move"]] - close_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "close"]] - assert bool(start_calls) is expect_send - assert bool(move_calls) is expect_send - assert bool(close_calls) is expect_send - assert data.get("default_isolated_session") is expected_default - assert data.get("explicit_session") is expected_explicit - assert REQUIRED_CHECKS <= _check_names(data) - assert _check_by_name(data, "observe")["observed"] is True - create = _check_by_name(data, "create_attach") - if expect_send: - assert create["status"] == "ok" - assert create["detail"] == "live_created" - assert "limitation" not in create - assert _check_by_name(data, "pane_moved_binding_update")["detail"] == "live_moved" - assert _check_by_name(data, "close_exited")["detail"] == "live_closed" - else: - assert create["limitation"] == "caller_override" - assert _check_by_name(data, "pane_moved_binding_update")["limitation"] == "live_skipped_unreliable" - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_environment_variable_opts_into_live_mode(smoke_module, monkeypatch, capsys): - runner = RecordingRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main( - smoke_module, - [], - capsys, - env={"TENDWIRE_HERDR_LIVE_SMOKE": "1"}, - runner=runner, - ) - - assert return_code in (0, None) - assert runner.calls - for call in runner.calls: - assert "HERDR_SESSION" not in call["env"] - assert call["argv"][1:3] == ["--session", "tendwire-smoke"] - assert data.get("mode") == "live" - assert _check_by_name(data, "create_attach")["detail"] == "live_created" - assert _check_by_name(data, "send_addressing")["send_attempts"] == 1 - assert _check_by_name(data, "pane_moved_binding_update")["detail"] == "live_moved" - assert _check_by_name(data, "close_exited")["detail"] == "live_closed" - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_live_stopped_selected_scope_fails_before_observe_or_send(smoke_module, monkeypatch, capsys): - runner = StoppedSessionRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}, runner=runner) - - assert return_code not in (0, None) - assert data.get("ok") is False - assert data.get("status") == "unavailable" - assert len(runner.calls) == 1 - assert runner.calls[0]["argv"][1:5] == ["--session", "tendwire-smoke", "status", "server"] - observe = _check_by_name(data, "observe") - assert observe["ok"] is False - assert observe["workspace_count"] == 0 - assert observe["worker_count"] == 0 - send = _check_by_name(data, "send_addressing") - assert send["status"] == "skipped" - assert send["send_attempts"] == 0 - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_live_not_running_machine_status_fails_before_observe_or_send(smoke_module, monkeypatch, capsys): - runner = StoppedUnderscoreSessionRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}, runner=runner) - - assert return_code not in (0, None) - assert data.get("ok") is False - assert data.get("status") == "unavailable" - assert len(runner.calls) == 1 - assert runner.calls[0]["argv"][1:5] == ["--session", "tendwire-smoke", "status", "server"] - observe = _check_by_name(data, "observe") - assert observe["ok"] is False - assert observe["status"] == "unavailable" - send = _check_by_name(data, "send_addressing") - assert send["status"] == "skipped" - assert send["send_attempts"] == 0 - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_live_nonzero_send_is_not_ok(smoke_module, monkeypatch, capsys): - runner = NonzeroSendRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}, runner=runner) - - assert return_code not in (0, None) - assert data.get("ok") is False - assert data.get("status") == "failed" - send = _check_by_name(data, "send_addressing") - assert send["ok"] is False - assert send["status"] == "nonzero" - assert send["exit_code"] == 1 - assert send["accepted_count"] == 0 - send_calls = [call for call in runner.calls if call["argv"][3:5] == ["agent", "send"]] - move_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "move"]] - close_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "close"]] - assert len(send_calls) == 1 - assert len(move_calls) == 1 - assert len(close_calls) == 1 - assert send_calls[0]["argv"][1:3] == ["--session", "tendwire-smoke"] - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_live_zero_accepted_send_is_not_ok(smoke_module, monkeypatch, capsys): - runner = ZeroAcceptedSendRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}, runner=runner) - - assert return_code not in (0, None) - assert data.get("ok") is False - assert data.get("status") == "failed" - send = _check_by_name(data, "send_addressing") - assert send["ok"] is False - assert send["status"] == "zero_accepted" - assert send["exit_code"] == 0 - assert send["json_status"] == "valid" - assert send["accepted_count"] == 0 - send_calls = [call for call in runner.calls if call["argv"][3:5] == ["agent", "send"]] - move_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "move"]] - close_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "close"]] - assert len(send_calls) == 1 - assert len(move_calls) == 1 - assert len(close_calls) == 1 - assert send_calls[0]["argv"][1:3] == ["--session", "tendwire-smoke"] - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_send_ok_envelope_counts_as_one_accepted(smoke_module): - accepted_count, json_status = smoke_module._send_accepted_count( - json.dumps({"id": "cli:agent:send", "result": {"type": "ok"}}) - ) - - assert accepted_count == 1 - assert json_status == "valid" - - -def test_fixture_replay_is_deterministic_and_public_safe(smoke_module, monkeypatch, capsys): - runner = ExplodingRunner() - _patch_which(monkeypatch, smoke_module, None) - - return_code, public_text, data = _run_main( - smoke_module, - ["--fixture-dir", str(OK_FIXTURES)], - capsys, - env={}, - runner=runner, - ) - - assert return_code in (0, None) - assert runner.calls == [] - assert data.get("ok") is True - assert data.get("mode") == "fixture" - assert REQUIRED_CHECKS <= _check_names(data) - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - assert _check_by_name(data, "create_attach")["created_count"] == 1 - assert _check_by_name(data, "observe")["worker_count"] == 2 - assert _check_by_name(data, "send_addressing")["send_attempts"] == 1 - assert _check_by_name(data, "target_validation")["rejected_send_attempts"] == 0 - status_check = _check_by_name(data, "status_agent_status_changed") - assert status_check["changed_count"] == 1 - assert status_check["event_count"] == 4 - assert status_check["accepted_count"] == 4 - assert status_check["exact_shape"] is True - assert status_check["idless"] is True - assert status_check["order_preserved"] is True - assert status_check["final_source_status"] == "working" - assert status_check["final_status"] == "active" - assert status_check["status_buckets"] == ["active"] - assert status_check["persistence_unchanged"] is True - assert status_check["repeat_effect_count"] == 0 - assert status_check["repeat_snapshot_delta"] == 0 - assert status_check["repeat_event_delta"] == 0 - assert status_check["repeat_attention_delta"] == 0 - assert status_check["repeat_queue_delta"] == 0 - assert _check_by_name(data, "pane_moved_binding_update")["preserved"] is True - assert _check_by_name(data, "close_exited")["exited_count"] == 1 - assert _check_by_name(data, "degraded_backend_preserves_workers")["preserved"] is True - event_check = _check_by_name(data, "event_subscription") - assert event_check.get("method") == "events.subscribe" - assert event_check.get("official_event_count") == len(smoke_module.OFFICIAL_EVENT_TYPES) - assert event_check.get("params_shape_ok") is True - assert event_check.get("legacy_event_count") == 0 - assert "subscriptions" not in event_check - assert "subscriptions" not in public_text - for raw_name in ( - "workspace.created", - "pane.created", - "pane.observed", - "workspace.observed", - "agent.status_changed", - "worktree.updated", - ): - assert raw_name not in public_text - - -def test_raw_status_fixture_replays_exact_idless_envelopes_without_duplicate_effects(smoke_module): - aggregate = json.loads((OK_FIXTURES / "status_agent_status_changed.json").read_text()) - replay_path = (OK_FIXTURES / aggregate["replay_fixture"]).resolve() - assert replay_path == RAW_STATUS_FIXTURE.resolve() - - envelopes = json.loads(replay_path.read_text()) - assert len(envelopes) == 4 - assert all(list(envelope) == ["event", "data"] for envelope in envelopes) - assert all(envelope["event"] == "pane.agent_status_changed" for envelope in envelopes) - assert [envelope["data"]["status"] for envelope in envelopes] == [ - "working", - "idle", - "working", - "working", - ] - assert envelopes[-1] == envelopes[-2] - assert smoke_module._contains_synthetic_event_identity(envelopes) is False - - replay = smoke_module._replay_fixture_status_events(replay_path) - assert replay["exact_shape"] is True - assert replay["idless"] is True - assert replay["source_statuses"] == ("working", "idle", "working", "working") - assert replay["accepted"] == (True, True, True, True) - assert replay["canonical_statuses"] == ("active", "idle", "active", "active") - assert replay["order_preserved"] is True - assert replay["final_source_status"] == "working" - assert replay["final_status"] == "active" - assert replay["changed_count"] == 1 - assert replay["repeat_row_deltas"] == { - "snapshots": 0, - "events": 0, - "attention_items": 0, - "connector_outbox": 0, - } - - -def test_deterministic_target_validation_sends_only_valid_case(smoke_module): - calls = [] - - check = smoke_module._deterministic_target_validation_check(calls.append) - - assert check["ok"] is True - assert check["valid_cases"] == 1 - assert check["invalid_cases"] == 2 - assert check["ambiguous_cases"] == 1 - assert check["send_attempts"] == 1 - assert check["rejected_send_attempts"] == 0 - assert calls == ["valid"] - - -def test_deterministic_event_backend_covers_move_close_exited_and_degraded(smoke_module, monkeypatch): - smoke_module._ensure_src_on_path() - from tendwire.backends.herdr_events import HerdrEventBackend - - recorded_envelopes = [] - original_queue = HerdrEventBackend.queue_event_envelope - - def recording_queue(self, envelope, *, flush=None): - recorded_envelopes.append(dict(envelope)) - return original_queue(self, envelope, flush=flush) - - monkeypatch.setattr(HerdrEventBackend, "queue_event_envelope", recording_queue) - checks = smoke_module._deterministic_event_backend_checks() - - assert checks["status_agent_status_changed"]["changed_count"] == 1 - assert checks["pane_moved_binding_update"]["preserved"] is True - assert checks["pane_moved_binding_update"]["worker_count_before"] == checks["pane_moved_binding_update"]["worker_count_after"] - assert checks["close_exited"]["closed_count"] == 1 - assert checks["close_exited"]["exited_count"] == 1 - assert checks["degraded_backend_preserves_workers"]["preserved"] is True - assert checks["degraded_backend_preserves_workers"]["worker_count_before"] == checks["degraded_backend_preserves_workers"]["worker_count_after"] - - assert [envelope["event"] for envelope in recorded_envelopes] == [ - "pane.agent_status_changed", - "pane.moved", - "pane.exited", - ] - assert all(set(envelope) == {"event", "data"} for envelope in recorded_envelopes) - assert smoke_module._contains_synthetic_event_identity(recorded_envelopes) is False - - -def test_event_subscription_builder_rejects_unknown_and_legacy_names(smoke_module): - params = smoke_module._event_subscription_params() - assert list(params) == ["subscriptions"] - assert len(params["subscriptions"]) == len(smoke_module.OFFICIAL_EVENT_TYPES) - assert smoke_module._event_subscription_params_shape_ok(params) is True - - for bad_name in ( - "", - " workspace.created ", - "pane.observed", - "workspace.observed", - "agent.status_changed", - "worktree.updated", - 123, - ): - names = list(smoke_module.OFFICIAL_EVENT_TYPES) - names[0] = bad_name - with pytest.raises(ValueError): - smoke_module._event_subscription_params(names) - - -def test_negative_fixture_rejects_recursive_forbidden_data(smoke_module, monkeypatch, capsys): - runner = ExplodingRunner() - _patch_which(monkeypatch, smoke_module, None) - - return_code, public_text, data = _run_main( - smoke_module, - ["--fixture-dir", str(NEGATIVE_FIXTURES)], - capsys, - env={}, - runner=runner, - ) - - assert return_code not in (0, None) or data.get("ok") is False - assert runner.calls == [] - assert _is_failure_or_skip(data) - assert any(word in _summary_text(data) for word in ("forbidden", "unsafe", "rejected")) - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_public_safety_rejects_forbidden_keys_values_and_allows_neutral_record_names(smoke_module): - for value in ( - {"pane_id": "hidden"}, - {"nested": [{"backend_target": "hidden"}]}, - {"detail": "socket:///tmp/forbidden.sock"}, - {"detail": "actual target value"}, - {"detail": "private fingerprint abc123"}, - {"stdout": "hidden"}, - ): - with pytest.raises(smoke_module.PublicSafetyError): - smoke_module.validate_public_summary(value) - - smoke_module.validate_public_summary( - { - "checks": [ - {"name": "target_validation", "status": "ok", "required": True, "ok": True}, - {"name": "pane_moved_binding_update", "status": "ok", "required": True, "ok": True}, - ] - } - ) - - -def test_missing_herdr_binary_reports_clear_skip_without_runner(smoke_module, monkeypatch, capsys): - _patch_which(monkeypatch, smoke_module, None) - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}) - - assert return_code not in (0, None) or data.get("ok") is False - assert _is_failure_or_skip(data) - assert any(word in _summary_text(data) for word in ("missing", "not found", "unavailable", "requires")) - assert REQUIRED_CHECKS <= _check_names(data) - assert _check_by_name(data, "observe")["ok"] is False - assert _check_by_name(data, "target_validation")["ok"] is True - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) diff --git a/tests/test_herdr_socket.py b/tests/test_herdr_socket.py index 0a7ff3e..ab9c88d 100644 --- a/tests/test_herdr_socket.py +++ b/tests/test_herdr_socket.py @@ -1,591 +1,109 @@ -"""Tests for the inactive Herdr Unix socket client.""" - from __future__ import annotations import json -import os import socket -import subprocess -import sys import threading import time -from collections.abc import Callable -from pathlib import Path -from typing import Any import pytest -from tendwire.backends.herdr_protocol import ( - HerdrEnvelopeError, - HerdrErrorResponse, - HerdrMalformedLineError, - HerdrRequestIdMismatchError, - HERDR_EVENTS_SUBSCRIBE_METHOD, - HERDR_OFFICIAL_EVENT_NAMES, - build_events_subscribe_params, -) +from tendwire.backends.herdr_protocol import HerdrFrameTooLargeError, HerdrRequestIdMismatchError from tendwire.backends.herdr_socket import ( HerdrSocketClient, - HerdrSocketDisconnectedError, + HerdrSocketConnectionError, HerdrSocketTimeoutError, + _MAX_FRAME_BYTES, ) -class _Connection: - def __init__(self, conn: socket.socket, requests: list[dict[str, Any]]) -> None: - self.conn = conn - self.requests = requests - self._buffer = bytearray() - - def read_request(self) -> dict[str, Any]: - while b"\n" not in self._buffer: - chunk = self.conn.recv(4096) - if not chunk: - raise ConnectionError("client disconnected before request") - self._buffer.extend(chunk) - index = self._buffer.index(b"\n") - line = bytes(self._buffer[: index + 1]) - del self._buffer[: index + 1] - request = json.loads(line.decode("utf-8")) - self.requests.append(request) - return request - - def send_json(self, payload: dict[str, Any]) -> None: - self.conn.sendall(json.dumps(payload, separators=(",", ":")).encode("utf-8") + b"\n") - - def send_bytes(self, payload: bytes) -> None: - self.conn.sendall(payload) - - -class _FakeHerdrServer: - def __init__(self, tmp_path: Path, handler: Callable[[_Connection], None]) -> None: - self.path = tmp_path / f"herdr-{time.monotonic_ns()}.sock" - self.handler = handler - self.requests: list[dict[str, Any]] = [] - self.errors: list[BaseException] = [] - self._ready = threading.Event() - self._done = threading.Event() - self._listener: socket.socket | None = None - self._thread: threading.Thread | None = None - - def __enter__(self) -> "_FakeHerdrServer": - try: - self.path.unlink() - except FileNotFoundError: - pass - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(self.path)) - listener.listen(1) - listener.settimeout(0.2) - self._listener = listener - self._thread = threading.Thread(target=self._run, daemon=True) - self._thread.start() - if not self._ready.wait(1): - raise AssertionError("fake Herdr server did not start") - return self - - def __exit__(self, exc_type: object, exc: object, tb: object) -> None: - if self._listener is not None: - self._listener.close() - if self._thread is not None: - self._thread.join(timeout=1) - try: - self.path.unlink() - except FileNotFoundError: - pass - if exc_type is None and self.errors: - raise AssertionError(f"fake Herdr server failed: {self.errors!r}") - - def _run(self) -> None: - self._ready.set() - try: - assert self._listener is not None - conn, _addr = self._listener.accept() - with conn: - self.handler(_Connection(conn, self.requests)) - except OSError: - pass - except BaseException as exc: - self.errors.append(exc) - finally: - self._done.set() - - -class _FakeOneShotHerdrServer: - def __init__(self, tmp_path: Path, handler: Callable[[_Connection], None], *, connections: int) -> None: - self.path = tmp_path / f"herdr-oneshot-{time.monotonic_ns()}.sock" - self.handler = handler - self.connections = connections - self.requests: list[dict[str, Any]] = [] - self.errors: list[BaseException] = [] - self._ready = threading.Event() - self._listener: socket.socket | None = None - self._thread: threading.Thread | None = None - - def __enter__(self) -> "_FakeOneShotHerdrServer": - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(self.path)) - listener.listen(self.connections) - listener.settimeout(0.5) - self._listener = listener - self._thread = threading.Thread(target=self._run, daemon=True) - self._thread.start() - if not self._ready.wait(1): - raise AssertionError("fake one-shot Herdr server did not start") - return self - - def __exit__(self, exc_type: object, exc: object, tb: object) -> None: - if self._listener is not None: - self._listener.close() - if self._thread is not None: - self._thread.join(timeout=1) - try: - self.path.unlink() - except FileNotFoundError: - pass - if exc_type is None and self.errors: - raise AssertionError(f"fake one-shot Herdr server failed: {self.errors!r}") +def _serve(path, responses, requests) -> threading.Thread: + ready = threading.Event() - def _run(self) -> None: - self._ready.set() - try: - assert self._listener is not None - for _index in range(self.connections): - conn, _addr = self._listener.accept() + def run() -> None: + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as server: + server.bind(str(path)) + server.listen() + ready.set() + for result in responses: + conn, _ = server.accept() with conn: - self.handler(_Connection(conn, self.requests)) - except OSError: - pass - except BaseException as exc: - self.errors.append(exc) - - -def _responding_handler(result: Any) -> Callable[[_Connection], None]: - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": result}) - - return handler - - -def test_client_successful_request_matches_id_and_returns_raw_result(tmp_path: Path) -> None: - result = {"items": [{"id": "w-1", "future": {"kept": True}}]} - with _FakeHerdrServer(tmp_path, _responding_handler(result)) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert client.request("workspace.list", {"scope": "all"}) == result - client.close() - - assert server.requests[0]["method"] == "workspace.list" - assert server.requests[0]["params"] == {"scope": "all"} - - - - -def test_client_reconnects_after_one_shot_response_connection_closes(tmp_path: Path) -> None: - results = { - "workspace.list": {"workspaces": []}, - "agent.list": {"agents": []}, - } - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": results[request["method"]]}) - - with _FakeOneShotHerdrServer(tmp_path, handler, connections=2) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert client.request("workspace.list") == {"workspaces": []} - assert client.request("agent.list") == {"agents": []} - client.close() - - assert [request["method"] for request in server.requests] == ["workspace.list", "agent.list"] - - -def test_client_timeout_waiting_for_response(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - time.sleep(0.15) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=0.05) - with pytest.raises(HerdrSocketTimeoutError): - client.request("workspace.list") - client.close() - - -def test_client_malformed_response_raises_protocol_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_bytes(b"{not json}\n") - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrMalformedLineError): - client.request("workspace.list") - client.close() - - -def test_client_malformed_envelope_shape_raises_protocol_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "not_result": True}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.request("workspace.list") - client.close() - - -def test_ordinary_request_rejects_uncorrelated_empty_id_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json( - { - "id": "", - "error": { - "code": "invalid_request", - "message": "uncorrelated ordinary request error", - }, - } - ) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.request("workspace.list") - client.close() - - - - -def test_client_non_utf8_response_raises_protocol_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_bytes(b"\xff\n") - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrMalformedLineError): - client.request("workspace.list") - client.close() - - -def test_client_disconnect_before_response_raises(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrSocketDisconnectedError): - client.request("workspace.list") - client.close() - - -def test_client_handles_partial_reads_split_across_recv_boundaries(tmp_path: Path) -> None: - result = {"text": "hello"} - - def handler(conn: _Connection) -> None: - request = conn.read_request() - response = json.dumps({"id": request["id"], "result": result}).encode("utf-8") + b"\n" - conn.send_bytes(response[:7]) - time.sleep(0.01) - conn.send_bytes(response[7:]) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - assert client.request("pane.read", {"pane_id": "p-1"}) == result - client.close() - - -def test_client_error_response_raises_with_raw_error_payload(tmp_path: Path) -> None: - error = {"code": "not_found", "message": "missing", "extra": {"kept": True}} - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "error": error, "future": "ignored"}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrErrorResponse) as excinfo: - client.request("pane.get", {"pane_id": "missing"}) - client.close() - - assert excinfo.value.error == error - - -def test_client_response_id_mismatch_raises(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json({"id": "wrong-id", "result": {"ok": True}}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrRequestIdMismatchError): - client.request("workspace.list") - client.close() - - -def test_subscription_ack_events_and_stream_termination(tmp_path: Path) -> None: - events = [ - {"event": "pane.output", "payload": {"text": "one"}, "future": {"kept": 1}}, - {"event": "pane.output", "payload": {"text": "two"}, "future": {"kept": 2}}, - ] - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": {"subscribed": True, "raw": [1]}}) - for event in events: - conn.send_json({"id": request["id"], **event}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.subscribe("pane.watch", {"pane_id": "p-1"}) - - assert stream.ack == {"subscribed": True, "raw": [1]} - assert list(stream) == [ - {"id": server.requests[0]["id"], **events[0]}, - {"id": server.requests[0]["id"], **events[1]}, - ] - client.close() - - -def test_subscription_accepts_uncorrelated_idless_event_data_frames(tmp_path: Path) -> None: - events = [ - {"event": "pane.agent_status_changed", "data": {"status": "blocked"}}, - {"event": "pane.closed", "data": {"pane_id": "p-1"}}, - ] - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": {"subscribed": True}}) - for event in events: - conn.send_json(event) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.subscribe("events.subscribe", {"subscriptions": []}) - - assert stream.ack == {"subscribed": True} - assert list(stream) == events - client.close() - - -def test_subscription_buffers_idless_event_before_correlated_ack(tmp_path: Path) -> None: - event = { - "event": "pane.output_matched", - "data": {"pane_id": "p-1"}, - } - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json(event) - conn.send_json({"id": request["id"], "result": {"subscribed": True}}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.events_subscribe(["pane.output_matched"]) - - assert stream.ack == {"subscribed": True} - assert list(stream) == [event] - client.close() - - -def test_subscription_surfaces_herdr_075_empty_id_schema_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json( - { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: missing field pane_id", - }, - } - ) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrErrorResponse, match="missing field pane_id") as raised: - client.events_subscribe(["pane.agent_status_changed"]) - assert raised.value.uncorrelated is True - client.close() - - -@pytest.mark.parametrize( - "error", - [ - {"code": "permission_denied", "message": "invalid request: denied"}, - {"code": "invalid_request", "message": "subscription unavailable"}, - ], -) -def test_subscription_rejects_other_empty_id_errors( - tmp_path: Path, - error: dict[str, str], -) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json({"id": "", "error": error}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.events_subscribe(["pane.agent_status_changed"]) - client.close() - - -def test_ordinary_request_rejects_herdr_075_empty_id_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json( - { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: ordinary request error", - }, - } - ) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.pane_list() - client.close() - - -def test_events_subscribe_wrapper_sends_official_method_and_params(tmp_path: Path) -> None: - event_names = ("workspace.created", "pane.agent_status_changed") - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": {"subscribed": True}}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.events_subscribe(event_names) - client.close() - - assert stream.ack == {"subscribed": True} - assert server.requests[0]["method"] == HERDR_EVENTS_SUBSCRIBE_METHOD - assert server.requests[0]["params"] == build_events_subscribe_params(event_names) - - -@pytest.mark.parametrize("event_name", HERDR_OFFICIAL_EVENT_NAMES) -def test_events_subscribe_wrapper_accepts_each_official_event_name( - tmp_path: Path, - event_name: str, -) -> None: - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": {"subscribed": event_name}}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.events_subscribe([event_name]) - client.close() - - assert stream.ack == {"subscribed": event_name} - assert server.requests[0]["method"] == HERDR_EVENTS_SUBSCRIBE_METHOD - assert server.requests[0]["params"] == {"subscriptions": [{"type": event_name}]} - - -def test_context_manager_and_close_are_idempotent(tmp_path: Path) -> None: - with _FakeHerdrServer(tmp_path, _responding_handler({"ok": True})) as server: - with HerdrSocketClient(str(server.path), timeout=1) as client: - assert client.request("agent.get", {"agent_id": "a-1"}) == {"ok": True} - client.close() - client.close() - - -@pytest.mark.parametrize( - ("wrapper_name", "method", "params", "result"), - [ - ("workspace_list", "workspace.list", {"all": True}, {"workspaces": [{"id": "w"}]}), - ("tab_list", "tab.list", {"workspace_id": "w"}, {"tabs": [{"id": "t"}]}), - ("pane_list", "pane.list", {"tab_id": "t"}, {"panes": [{"id": "p"}]}), - ("agent_list", "agent.list", {"workspace_id": "w"}, {"agents": [{"id": "a"}]}), - ("pane_get", "pane.get", {"pane_id": "p"}, {"id": "p", "raw": {"kept": True}}), - ("agent_get", "agent.get", {"agent_id": "a"}, {"id": "a", "raw": {"kept": True}}), - ("pane_read", "pane.read", {"pane_id": "p", "limit": 20}, {"text": "raw"}), - ], -) -def test_allowed_read_wrappers_send_exact_method_and_params( - tmp_path: Path, - wrapper_name: str, - method: str, - params: dict[str, Any], - result: dict[str, Any], -) -> None: - with _FakeHerdrServer(tmp_path, _responding_handler(result)) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert getattr(client, wrapper_name)(params) == result - client.close() - - assert server.requests[0]["method"] == method - assert server.requests[0]["params"] == params - - -def test_agent_send_is_the_only_exposed_mutate_wrapper_and_shape_is_exact(tmp_path: Path) -> None: - result = {"accepted": True, "opaque": {"server": "kept"}} - params = {"agent_id": "a-1", "text": "hello"} - with _FakeHerdrServer(tmp_path, _responding_handler(result)) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert client.agent_send(params) == result - client.close() - - assert server.requests[0]["method"] == "agent.send" - assert server.requests[0]["params"] == params - - excluded_public_api = { - "pane_send_text", - "pane_send_keys", - "pane_run", - "send_text", - "send_keys", - "run", - "shell", - "raw_terminal_control", - "source_mode", - "connector_polling", - "poll_connectors", - "event_backend_replacement", - } - for name in excluded_public_api: - assert not hasattr(client, name), name - - -def test_cli_import_does_not_load_socket_client_by_default() -> None: - code = """ -import sys -before = set(sys.modules) -import tendwire.cli -loaded = set(sys.modules) - before -for name in sorted(loaded): - if name in {"tendwire.backends.herdr_socket", "tendwire.backends.herdr_protocol"}: - print(name) -""" - env = os.environ.copy() - env["PYTHONPATH"] = os.path.join(os.path.dirname(__file__), "..", "src") - result = subprocess.run( - [sys.executable, "-c", code], - capture_output=True, - text=True, - check=False, - env=env, + request = json.loads(conn.makefile("rb").readline()) + requests.append(request) + response_id = result.pop("response_id", request["id"]) + conn.sendall(json.dumps({"id": response_id, **result}).encode() + b"\n") + + thread = threading.Thread(target=run, daemon=True) + thread.start() + assert ready.wait(2) + return thread + + +def test_lifecycle_and_acp_methods_use_frozen_socket_shapes(tmp_path) -> None: + path = tmp_path / "herdr.sock" + requests: list[dict] = [] + thread = _serve( + path, + [ + {"result": {"panes": []}}, + {"result": {"type": "agent_acp_status"}}, + {"result": {"type": "agent_acp_endpoint"}}, + ], + requests, ) - assert result.returncode == 0 - assert result.stdout == "" + client = HerdrSocketClient(str(path), timeout=1) + assert client.pane_list() == {"panes": []} + client.close() + assert client.agent_acp_status("term") == {"type": "agent_acp_status"} + client.close() + assert client.agent_acp_endpoint("term") == {"type": "agent_acp_endpoint"} + client.close() + thread.join(2) + assert [(item["method"], item["params"]) for item in requests] == [ + ("pane.list", {}), + ("agent.acp_status", {"target": "term"}), + ("agent.acp_endpoint", {"target": "term"}), + ] -def test_existing_production_backend_files_do_not_import_socket_client() -> None: - root = Path(__file__).resolve().parents[1] - for relative in ( - "src/tendwire/cli.py", - "src/tendwire/backends/herdr_cli.py", - ): - text = (root / relative).read_text(encoding="utf-8") - assert "herdr_socket" not in text +def test_response_id_mismatch_fails_closed(tmp_path) -> None: + path = tmp_path / "herdr.sock" + thread = _serve(path, [{"response_id": "wrong", "result": {}}], []) + with pytest.raises(HerdrRequestIdMismatchError): + HerdrSocketClient(str(path), timeout=1).agent_acp_status("term") + thread.join(2) + + +def test_connection_and_timeout_fail_closed(tmp_path) -> None: + with pytest.raises(HerdrSocketConnectionError): + HerdrSocketClient(str(tmp_path / "missing.sock"), timeout=0.01).pane_list() + + class TimedOutSocket: + def settimeout(self, _value): return None + def recv(self, _size): raise socket.timeout + def shutdown(self, _how): return None + def close(self): return None + + client = HerdrSocketClient(str(tmp_path / "unused.sock"), timeout=1) + client._socket = TimedOutSocket() + with pytest.raises(HerdrSocketTimeoutError): + client._read_line(deadline=time.monotonic() + 1) + + +def test_request_and_response_frames_are_bounded(tmp_path) -> None: + class UnusedSocket: + def settimeout(self, _value): return None + def sendall(self, _payload): raise AssertionError("oversize request must not be sent") + def recv(self, _size): return b"x" + def shutdown(self, _how): return None + def close(self): return None + + outbound = HerdrSocketClient(str(tmp_path / "unused.sock"), timeout=1) + outbound._socket = UnusedSocket() + with pytest.raises(HerdrFrameTooLargeError): + outbound.request("pane.list", {"padding": "x" * _MAX_FRAME_BYTES}) + + inbound = HerdrSocketClient(str(tmp_path / "unused.sock"), timeout=1) + inbound._socket = UnusedSocket() + inbound._buffer.extend(b"x" * _MAX_FRAME_BYTES) + with pytest.raises(HerdrFrameTooLargeError): + inbound._read_line(deadline=time.monotonic() + 1) diff --git a/tests/test_local_state_permissions.py b/tests/test_local_state_permissions.py index 7480eea..661be4f 100644 --- a/tests/test_local_state_permissions.py +++ b/tests/test_local_state_permissions.py @@ -550,7 +550,7 @@ def remove_at_preflight(phase: str, selected_kind: LocalStateKind) -> None: assert results[result_index].mode is None assert not sidecar.exists() assert _mode(db_path) == 0o600 - assert set(os.listdir("/proc/self/fd")) == before_fds + assert set(os.listdir("/proc/self/fd")) <= before_fds finally: os.close(parent_fd) diff --git a/tests/test_release_readiness.py b/tests/test_release_readiness.py index 962f126..afd1686 100644 --- a/tests/test_release_readiness.py +++ b/tests/test_release_readiness.py @@ -17,8 +17,6 @@ from pathlib import Path from typing import Any -from tendwire.backends import herdr_cli -from tendwire.backends.herdr_cli import diagnose_herdr from tendwire.daemon import TendwireDaemon from tendwire.config import Config @@ -244,24 +242,6 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( ) health = TendwireDaemon(config).get_health() - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _value: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli.subprocess, - "run", - lambda args, **_kwargs: subprocess.CompletedProcess( - args=args, - returncode=0, - stdout='{"items":[]}', - stderr="", - ), - ) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-10T00:30:00+00:00", - ) - doctor = diagnose_herdr(config) - before_compaction = db_path.read_bytes() compaction = compact_store( db_path, @@ -449,23 +429,6 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( assert health["limits"]["acknowledged_final_retention_days"] == 36500 assert health["limits"]["acknowledged_final_retention_count"] == 100 assert health["store"]["final_retention"] == status["final_retention"] - maintenance_checks = [ - check for check in doctor["checks"] if check["name"] == "store_maintenance" - ] - assert maintenance_checks == [ - { - "name": "store_maintenance", - "ok": True, - "outcome": "ok", - "remediation": "No action required.", - "snapshot_retention_days": 36500, - "snapshot_retention_count": 100, - "maintenance_batch_size": 5, - "maintenance_cadence_seconds": 3600, - "snapshot_count": 1, - "last_completed_at": "2026-01-10T00:00:00+00:00", - } - ] assert compaction["command"] == "store.compact" assert compaction["scope"] == "database" assert compaction["dry_run"] is True @@ -474,7 +437,7 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( assert compaction["snapshots"]["deleted"] == 0 assert db_path.read_bytes() == before_compaction - public_surfaces = [status, automatic, cleanup, health, doctor, compaction] + public_surfaces = [status, automatic, cleanup, health, compaction] for surface in public_surfaces: _assert_public_clean(surface) serialized = json.dumps(public_surfaces, sort_keys=True) diff --git a/tests/test_turn_delta.py b/tests/test_turn_delta.py index 540a08d..f93a84e 100644 --- a/tests/test_turn_delta.py +++ b/tests/test_turn_delta.py @@ -976,7 +976,7 @@ def test_turn_delta_rpc_advertises_feature_and_cannot_invoke_delivery(tmp_path: assert delivery_calls == [] -def test_turn_delta_cli_bootstrap_and_incremental_read(tmp_path: Path, capsys) -> None: +def test_turn_delta_cli_requires_daemon_and_does_not_read_store(tmp_path: Path, capsys) -> None: db_path = tmp_path / "cli.db" socket_path = tmp_path / "missing.sock" init_store(db_path) @@ -994,15 +994,11 @@ def test_turn_delta_cli_bootstrap_and_incremental_read(tmp_path: Path, capsys) - "--db-path", str(db_path), ] - assert main(base_args) == 0 - bootstrap = json.loads(capsys.readouterr().out) - assert bootstrap["changes"][0]["turn"]["summary"] == "first CLI projection" - checkpoint = str(bootstrap["checkpoint"]) - - _mutate_turn(db_path, "cli-turn", summary="second CLI projection") - assert main([*base_args, "--watermark", checkpoint]) == 0 - changed = json.loads(capsys.readouterr().out) - assert changed["changes"][0]["turn"]["summary"] == "second CLI projection" + assert main(base_args) == 1 + payload = json.loads(capsys.readouterr().out) + assert payload["ok"] is False + assert payload["status"] == "daemon_unavailable" + assert "changes" not in payload def test_turn_change_retention_config_defaults_env_and_bounds( diff --git a/tests/test_worker_label_and_model.py b/tests/test_worker_label_and_model.py index 9c162b0..fee7405 100644 --- a/tests/test_worker_label_and_model.py +++ b/tests/test_worker_label_and_model.py @@ -1,166 +1,32 @@ -"""Pane label in public worker meta + model on turns (consumed by herdres for topic names and the -pinned status board).""" from __future__ import annotations -from pathlib import Path - -from tendwire.backends.herdr_cli import _worker_from_item, _workers_and_bindings_from_records -from tendwire.backends.herdr_events import HerdrEventBackend +from tendwire.backends.acp_coordinator import _discovered_workers from tendwire.config import Config from tendwire.core.turns import Turn -from tendwire.core.projector import project_from_raw -from tendwire.store.sqlite import init_store, merge_turn_content, save_snapshot, turns_payload_from_store -def _config(tmp_path: Path) -> Config: - return Config( - host_id="label-host", - data_dir=tmp_path, - db_path=tmp_path / "label-host.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, +def test_pane_label_is_public_but_cwd_and_target_are_private(tmp_path) -> None: + workers, bindings = _discovered_workers( + Config(host_id="h", data_dir=tmp_path, db_path=tmp_path / "db"), + {"panes": [{ + "workspace_id": "wR9", + "pane_id": "wR9:pA", + "terminal_id": "term-private", + "agent": "claude", + "label": "Review pane", + "cwd": "/private/path", + }]}, + {"agents": []}, + "2026-01-01T00:00:00+00:00", ) + assert workers[0].meta["label"] == "Review pane" + assert "private" not in str(workers[0].to_dict()) + assert bindings[0].target_value == "term-private" -def _pane_item(label: str = "review-pane") -> dict: - return { - "pane_id": "ws-1:p2Q", - "terminal_id": "term-1", - "workspace_id": "ws-1", - "agent": "claude", - "agent_session": {"kind": "id", "value": "sess-1"}, - "label": label, - "cwd": "/root/temp", - "agent_status": "idle", - } - - -def _agent_item() -> dict: - return { - "agent_id": "agent-private", - "name": "claude", - "agent": "claude", - "workspace_id": "ws-1", - "status": "waiting", - "pane_id": "ws-1:p2Q", - "terminal_id": "term-1", - "agent_session": {"kind": "id", "value": "sess-1"}, - } - - -def test_pane_label_lands_in_public_worker_meta() -> None: - worker = _worker_from_item(_pane_item()) - assert worker is not None - assert worker.meta.get("label") == "review-pane" - # name resolution unchanged: agent-first - assert worker.name == "claude" - - -def test_reconcile_merges_pane_label_into_agent_record_without_replacing_it(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - records = backend._records_from_reconcile_payloads( - {"agents": [_agent_item()]}, - {"panes": [_pane_item()]}, - ) - workers, bindings = _workers_and_bindings_from_records(config, records) - - assert len(records) == 1 - assert len(workers) == 1 - assert len(bindings) == 1 - assert records[0].worker.meta.get("label") == "review-pane" - assert "cwd" not in records[0].worker.meta - assert records[0].worker.status == "waiting" - assert bindings[0].target_kind == "agent_id" - assert bindings[0].target_value == "agent-private" - - -def test_reconcile_drops_agent_and_pane_cwd_from_public_worker(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - agent = {**_agent_item(), "cwd": "/root/agent-cwd"} - pane = {**_pane_item(), "cwd": "/root/pane-cwd"} - records = backend._records_from_reconcile_payloads({"agents": [agent]}, {"panes": [pane]}) - - assert len(records) == 1 - assert records[0].worker.meta.get("label") == "review-pane" - assert "cwd" not in records[0].worker.meta - assert "/root/agent-cwd" not in str(records[0].worker.to_dict()) - assert "/root/pane-cwd" not in str(records[0].worker.to_dict()) - - - - - - -def test_reconcile_only_fills_missing_agent_backend_target_from_pane(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - agent = { - "workspace_id": "ws-1", - "status": "waiting", - "agent_session": {"kind": "id", "value": "sess-1"}, - } - records = backend._records_from_reconcile_payloads({"agents": [agent]}, {"panes": [_pane_item()]}) - workers, bindings = _workers_and_bindings_from_records(config, records) - - assert len(records) == 1 - assert workers[0].backend_target is not None - assert workers[0].backend_target["kind"] == "terminal_id" - assert workers[0].backend_target["value"] == "term-1" - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == "term-1" - - -def test_reconcile_preserves_pane_only_worker_when_no_agent_record(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - records = backend._records_from_reconcile_payloads({"agents": []}, {"panes": [_pane_item()]}) - workers, bindings = _workers_and_bindings_from_records(config, records) - - assert len(workers) == 1 - assert workers[0].meta.get("label") == "review-pane" - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == "term-1" - - -def test_turn_model_round_trip_and_id_stability() -> None: - base = {"host_id": "h", "worker_id": "w1", "kind": "turn", "source": "herdr", - "user_text": "hi", "assistant_final_text": "done", "complete": True} +def test_turn_model_remains_content_not_identity() -> None: + base = {"host_id": "h", "worker_id": "w", "kind": "turn", "source": "acp", "complete": True} plain = Turn.from_dict(base) - with_model = Turn.from_dict({**base, "model": "claude-fable-5[1m]"}) - assert with_model.model == "claude-fable-5[1m]" - assert with_model.to_dict()["model"] == "claude-fable-5[1m]" - assert plain.id == with_model.id # model is content, NOT identity (no id re-mint) - assert plain.fingerprint != with_model.fingerprint # but the content fingerprint reflects it - - - - -def test_merge_turn_content_persists_model(tmp_path: Path) -> None: - db = tmp_path / "turns.db" - config = Config(host_id="turn-host", db_path=db) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "claude", "status": "active", "space_id": "space-1"}], - ) - init_store(db) - save_snapshot(db, snapshot) - updated = merge_turn_content( - db, "turn-host", "worker-1", - { - "source_turn_id": "model-source", - "user_text": "hi", - "assistant_final_text": "done", - "complete": True, - "model": "claude-fable-5", - }, - observed_at="2026-01-01T00:00:00+00:00", - ) - payload = turns_payload_from_store(db, "turn-host", snapshot=snapshot) - assert updated == 1 - assert payload["turns"][0].get("model") == "claude-fable-5" + modeled = Turn.from_dict({**base, "model": "claude"}) + assert plain.id == modeled.id + assert plain.fingerprint != modeled.fingerprint diff --git a/tests/test_worker_stable_key.py b/tests/test_worker_stable_key.py index 89edb24..743b81d 100644 --- a/tests/test_worker_stable_key.py +++ b/tests/test_worker_stable_key.py @@ -1,2334 +1,257 @@ -"""Stable worker continuity from Herdr's persisted workspace/public-pane identity.""" - from __future__ import annotations -from concurrent.futures import ThreadPoolExecutor -import hashlib -import hmac import json -import os import re -import stat -import subprocess -from copy import deepcopy -from pathlib import Path -from typing import Any +import threading +from dataclasses import replace import pytest -from tendwire import worker_identity -from tendwire.backends import herdr_cli -from tendwire.backends.herdr_cli import ( - _private_identity_material_from_item, - _worker_record_from_item, - _workers_and_bindings_from_records, +from tendwire.backends.acp_coordinator import ( + AcpSupervisor, + _discovered_spaces, + _discovered_workers, ) -from tendwire.backends.herdr_events import HerdrEventBackend from tendwire.config import Config -from tendwire.core.models import Worker, worker_binding_private_fingerprint -from tendwire.store.sqlite import init_store, latest_snapshot, list_worker_bindings -from tendwire.worker_identity import ( - InstallationKeyError, - load_or_create_installation_key, - reset_installation_key, +from tendwire.core.models import Snapshot, WorkerBinding +from tendwire.store.sqlite import ( + init_store, + latest_snapshot, + list_worker_bindings, + save_snapshot, + upsert_worker_bindings, ) -_STABLE_KEY = re.compile(r"^wsk1_[0-9a-f]{64}$") -_FIXTURE_PATH = Path(__file__).parent / "fixtures" / "herdr" / "worker_identity_restore.json" - -@pytest.fixture(autouse=True) -def _isolated_home(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - home = tmp_path / "unused-home" - home.mkdir() - monkeypatch.setenv("HOME", str(home)) +OBSERVED_AT = "2026-01-01T00:00:00+00:00" -def _fixture() -> dict[str, Any]: - return json.loads(_FIXTURE_PATH.read_text(encoding="utf-8")) +def _config(tmp_path) -> Config: + return Config(host_id="host", data_dir=tmp_path, db_path=tmp_path / "db.sqlite") -def _config(data_dir: Path, *, host_id: str = "stable-host") -> Config: - return Config( - host_id=host_id, - data_dir=data_dir, - db_path=data_dir / "stable-host.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, +def _discover(tmp_path, panes, agents=(), *, prior_bindings=()): + return _discovered_workers( + _config(tmp_path), + {"panes": panes}, + {"agents": list(agents)}, + OBSERVED_AT, + prior_bindings=prior_bindings, ) -def _project( - config: Config, - agents: list[dict[str, Any]], - panes: list[dict[str, Any]] | None = None, -) -> tuple[HerdrEventBackend, list[Any], list[Any], list[Any]]: - config.data_dir.mkdir(parents=True, mode=0o700, exist_ok=True) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - records = backend._records_from_reconcile_payloads( - {"agents": deepcopy(agents)}, - {"panes": deepcopy(panes or [])}, - ) - workers, bindings = _workers_and_bindings_from_records(config, records) - return backend, workers, bindings, records - - -def _single_worker(config: Config, item: dict[str, Any]) -> Any: - _backend, workers, _bindings, _records = _project(config, [], [item]) - assert len(workers) == 1 - return workers[0] - - -def _stable(worker: Any) -> str: - value = worker.meta.get("stable_key") - assert isinstance(value, str) - return value - - -def _mode(path: Path) -> int: - return stat.S_IMODE(os.lstat(path).st_mode) - - -def _tree_snapshot(root: Path) -> dict[str, tuple[int, int, bytes | None]]: - snapshot: dict[str, tuple[int, int, bytes | None]] = {} - for entry in [root, *sorted(root.rglob("*"))]: - current = os.lstat(entry) - content = entry.read_bytes() if stat.S_ISREG(current.st_mode) else None - snapshot[str(entry.relative_to(root))] = ( - current.st_ino, - stat.S_IMODE(current.st_mode), - content, - ) - return snapshot - - -def _reserved_meta_keys(value: Any, *, include_root: bool = True) -> list[str]: - found: list[str] = [] - if isinstance(value, dict): - for key, child in value.items(): - compact = str(key).lower().replace("_", "").replace("-", "").replace(".", "") - if include_root and compact.startswith("stablekey"): - found.append(str(key)) - found.extend(_reserved_meta_keys(child)) - elif isinstance(value, list): - for child in value: - found.extend(_reserved_meta_keys(child)) - return found - - -def test_restore_fixture_matches_verified_herdr_contract() -> None: - """Fixture follows authoritative Herdr commit 46174563489273199a17c982356c6e4674ef00d4.""" - fixture = _fixture() - session = fixture["session_snapshot"] - workspace = session["workspaces"][0] - tab = workspace["tabs"][0] - before = fixture["pre_restore"] - after = fixture["post_restore"] - - assert session["version"] == 3 - assert workspace["id"] == "wR9" - assert workspace["public_pane_numbers"] == {"41": 10, "42": 11} - assert tab["layout"]["Split"]["first"] == {"Pane": 41} - assert tab["layout"]["Split"]["second"] == {"Pane": 42} - assert set(tab["panes"]) == {"41", "42"} - assert before["pane_info"]["pane_id"] == after["pane_info"]["pane_id"] == "wR9:pA" - assert before["sibling_pane_info"]["pane_id"] == after["sibling_pane_info"]["pane_id"] == "wR9:pB" - for field in ("raw_pane_id", "runtime_id", "worker_id", "agent_id"): - assert before[field] != after[field] - for field in ("terminal_id", "agent"): - assert before["pane_info"][field] != after["pane_info"][field] - assert before["pane_info"]["agent_session"] != after["pane_info"]["agent_session"] - split = fixture["split_creation"] - assert split["event"] == split["data"]["type"] == "pane_created" - assert split["data"]["pane"]["pane_id"] == before["sibling_pane_info"]["pane_id"] - assert split["data"]["pane"]["workspace_id"] == workspace["id"] - for move in (fixture["same_workspace_move"], fixture["cross_workspace_move"]): - assert move["event"] == "pane_moved" - assert move["data"]["type"] == "pane_moved" - assert "pane" in move["data"] - - -def test_turn_observation_fields_are_byte_identical_identity_exclusions( - tmp_path: Path, -) -> None: - config = _config(tmp_path / "identity-turn-exclusion") - pane = deepcopy(_fixture()["pre_restore"]["pane_info"]) - pane["meta"] = {"provider": {"label": "stable"}} - _backend, baseline_workers, baseline_bindings, _records = _project( - config, - [], - [pane], - ) - observed = deepcopy(pane) - observed.update( - { - "turn": 41, - "turn_epoch": 99, - "last_completed_turn": { - "turn": 41, - "turn_epoch": 99, - "completed_unix_ms": 1_700_000_000_000, - }, - "outcome": "aborted", - "state_change_seq": 100, - } - ) - observed["meta"]["provider"].update( - { - "turn": 41, - "turn_epoch": 99, - "last_completed_turn": {"turn": 41}, - "outcome": "aborted", - "state_change_seq": 100, - } - ) - _backend, turn_workers, turn_bindings, _records = _project( - config, - [], - [observed], - ) - - assert len(baseline_workers) == len(turn_workers) == 1 - assert json.dumps( - baseline_workers[0].to_dict(), - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") == json.dumps( - turn_workers[0].to_dict(), - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") - assert baseline_workers[0].fingerprint == turn_workers[0].fingerprint - assert baseline_bindings[0].private_fingerprint == ( - turn_bindings[0].private_fingerprint - ) - - -def test_exact_format_version_and_domain_separated_hmac( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - fixture = _fixture() - pane = fixture["pre_restore"]["pane_info"] - config = _config(tmp_path / "state") - key = bytes(range(32)) - config.data_dir.mkdir(mode=0o700) - config.installation_key_path.write_bytes(key) - os.chmod(config.installation_key_path, 0o600) - captured_messages: list[bytes] = [] - original_hmac_new = hmac.new - - def capture_hmac( - hmac_key: bytes, - message: bytes, - digestmod: Any, - ) -> hmac.HMAC: - captured_messages.append(message) - return original_hmac_new(hmac_key, message, digestmod) - - monkeypatch.setattr(worker_identity.hmac, "new", capture_hmac) - worker = _single_worker(config, pane) - message = ( - b'{"backend":"herdr","domain":"tendwire.worker-stable-key",' - b'"host_id":"stable-host","pane_id":"wR9:pA","version":1,' - b'"workspace_id":"wR9"}' - ) - expected = "wsk1_" + original_hmac_new(key, message, hashlib.sha256).hexdigest() - - assert _stable(worker) == expected - assert captured_messages == [message] - assert _STABLE_KEY.fullmatch(expected) - assert type(worker.meta["stable_key_version"]) is int - assert worker.meta["stable_key_version"] == 1 - public_meta = worker.to_dict()["meta"] - assert public_meta["stable_key"] == expected - assert type(public_meta["stable_key_version"]) is int - assert public_meta["stable_key_version"] == 1 - assert config.installation_key_marker_path.read_bytes() == hashlib.sha256(key).hexdigest().encode("ascii") - - -@pytest.mark.parametrize( - ("workspace_id", "pane_id", "expected"), - [ - ("w0", "w0:p0", ("w0", "w0:p0")), - ("w1", "w1:p1", ("w1", "w1:p1")), - ("wZ", "wZ:pZ", ("wZ", "wZ:pZ")), - ( - "w65383a2e877513", - "w65383a2e877513:p4", - ("w65383a2e877513", "w65383a2e877513:p4"), - ), - ( - "w653e50b41be581", - "w653e50b41be581:pC", - ("w653e50b41be581", "w653e50b41be581:pC"), - ), - ( - "wABCDEFGHJKMNPQRSTVWXYZ0123456789", - "wABCDEFGHJKMNPQRSTVWXYZ0123456789:" - "p9876543210ZYXWVTSRQPNMKJHGFEDCBA", - ( - "wABCDEFGHJKMNPQRSTVWXYZ0123456789", - "wABCDEFGHJKMNPQRSTVWXYZ0123456789:" - "p9876543210ZYXWVTSRQPNMKJHGFEDCBA", - ), - ), - (None, "wA:pA", None), - ("wA", None, None), - ("", ":pA", None), - ("w", "w:pA", None), - ("W1", "W1:p1", None), - ("wwA", "wwA:pA", None), - ("wa", "wa:pA", None), - ("w65383a2e87751", "w65383a2e87751:pA", None), - ("w65383a2e8775133", "w65383a2e8775133:pA", None), - ("w65383A2e877513", "w65383A2e877513:pA", None), - ("w65383g2e877513", "w65383g2e877513:pA", None), - ("wAa", "wAa:pA", None), - ("wA-B", "wA-B:pA", None), - ("wA_B", "wA_B:pA", None), - ("wI", "wI:pA", None), - ("wL", "wL:pA", None), - ("wO", "wO:pA", None), - ("wU", "wU:pA", None), - ("wΑ", "wΑ:pA", None), - ("wＡ", "wＡ:pA", None), - ("wA", "wA:p", None), - ("wA", "wA:PA", None), - ("wA", "wA:pa", None), - ("wA", "wA:pAa", None), - ("wA", "wA:pA-B", None), - ("wA", "wA:pA_B", None), - ("wA", "wA:pI", None), - ("wA", "wA:pL", None), - ("wA", "wA:pO", None), - ("wA", "wA:pU", None), - ("wA", "wA:pΑ", None), - ("wA", "wA:pＡ", None), - ("wA", "wA:pA:pB", None), - ("wA", " wA:pA", None), - ("wA", "wA:pA ", None), - ("wA", "wB:pA", None), - ("wA", "wAA:pA", None), - ], - ids=[ - "zero-boundary", - "one-boundary", - "uppercase-boundary", - "current-hex-workspace-numbered-pane", - "current-hex-workspace-uppercase-pane", - "full-authoritative-alphabet", - "missing-workspace", - "missing-pane", - "empty-workspace", - "empty-workspace-suffix", - "uppercase-structural-prefix", - "extra-workspace-prefix", - "lowercase-workspace-suffix", - "short-current-hex-workspace", - "long-current-hex-workspace", - "mixed-current-hex-workspace", - "nonhex-current-workspace", - "mixed-case-workspace-suffix", - "hyphenated-workspace-suffix", - "underscored-workspace-suffix", - "workspace-ascii-I-confusable", - "workspace-ascii-L-confusable", - "workspace-ascii-O-confusable", - "workspace-ascii-U", - "workspace-greek-alpha-confusable", - "workspace-fullwidth-alpha-confusable", - "empty-pane-suffix", - "uppercase-pane-structural-prefix", - "lowercase-pane-suffix", - "mixed-case-pane-suffix", - "hyphenated-pane-suffix", - "underscored-pane-suffix", - "pane-ascii-I-confusable", - "pane-ascii-L-confusable", - "pane-ascii-O-confusable", - "pane-ascii-U", - "pane-greek-alpha-confusable", - "pane-fullwidth-alpha-confusable", - "injected-pane-structure", - "leading-pane-whitespace", - "trailing-pane-whitespace", - "cross-workspace", - "cross-workspace-prefix", - ], -) -def test_canonical_herdr_pane_identity_uses_exact_authoritative_grammar( - workspace_id: str | None, - pane_id: str | None, - expected: tuple[str, str] | None, -) -> None: - assert worker_identity.canonical_herdr_pane_identity(workspace_id, pane_id) == expected - - -def test_worker_record_separates_canonical_identity_from_raw_observation() -> None: - canonical = _worker_record_from_item( - { - "workspaceId": "w65383a2e877513", - "paneId": "w65383a2e877513:pA", - "agent": "codex", - }, - pane_info_observed=True, - identity_source="event:pane.updated", - ) - assert canonical.workspace_id == "w65383a2e877513" - assert canonical.pane_id == "w65383a2e877513:pA" - assert canonical.observed_workspace_id == canonical.workspace_id - assert canonical.observed_pane_id == canonical.pane_id - assert canonical.identity_source == "event:pane.updated" - assert canonical.pane_info_observed is True - - raw_runtime_identity = _worker_record_from_item( - { - "workspace_id": 7, - "pane_id": 41, - "agent": "codex", - }, - pane_info_observed=True, - identity_source="event:pane.agent_status_changed", - ) - assert raw_runtime_identity.workspace_id is None - assert raw_runtime_identity.pane_id is None - assert raw_runtime_identity.observed_workspace_id == "7" - assert raw_runtime_identity.observed_pane_id == "41" - assert raw_runtime_identity.identity_source == "event:pane.agent_status_changed" - assert raw_runtime_identity.pane_info_observed is True - assert herdr_cli._stable_pane_identity(raw_runtime_identity) is None - - -@pytest.mark.parametrize( - ("workspace_id", "pane_id"), - [ - ("w0", "w0:p0"), - ("w1", "w1:p1"), - ("wZ", "wZ:pZ"), - ("w65383a2e877513", "w65383a2e877513:p4"), - ("w653e50b41be581", "w653e50b41be581:pC"), - ( - "wABCDEFGHJKMNPQRSTVWXYZ0123456789", - "wABCDEFGHJKMNPQRSTVWXYZ0123456789:" - "p9876543210ZYXWVTSRQPNMKJHGFEDCBA", - ), - ], -) -def test_every_supported_identity_form_is_restart_stable_and_private( - tmp_path: Path, - workspace_id: str, - pane_id: str, -) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item["workspace_id"] = workspace_id - item["pane_id"] = pane_id - data_dir = tmp_path / "state" - before = _single_worker(_config(data_dir), item) - - restarted_item = deepcopy(item) - restarted_item["terminal_id"] = "runtime-terminal-after-restart" - restarted_item["agent"] = "runtime-agent-after-restart" - restarted_item["agent_session"] = { - "source": "runtime-source-after-restart", - "agent": "runtime-agent-after-restart", - "kind": "id", - "value": "runtime-session-after-restart", - } - after = _single_worker(_config(data_dir), restarted_item) - - assert _stable(after) == _stable(before) - public = json.dumps(after.to_dict(), sort_keys=True) - assert pane_id not in public - assert restarted_item["terminal_id"] not in public - assert restarted_item["agent_session"]["source"] not in public - assert restarted_item["agent_session"]["value"] not in public - - -def test_restore_continuity_ignores_changed_runtime_terminal_agent_and_session(tmp_path: Path) -> None: - fixture = _fixture() - config = _config(tmp_path / "state") - - before = _single_worker(config, fixture["pre_restore"]["pane_info"]) - after = _single_worker(config, fixture["post_restore"]["pane_info"]) - - assert before.id != after.id - assert _stable(before) == _stable(after) - - -def test_split_panes_have_distinct_keys_and_survive_sibling_close_or_reorder(tmp_path: Path) -> None: - fixture = _fixture() - primary = fixture["post_restore"]["pane_info"] - sibling = fixture["post_restore"]["sibling_pane_info"] - config = _config(tmp_path / "state") - - _backend, first, _bindings, _records = _project(config, [], [primary, sibling]) - by_name = {worker.name: _stable(worker) for worker in first} - assert len(set(by_name.values())) == 2 - - _backend, reordered, _bindings, _records = _project(config, [], [sibling, primary]) - assert {worker.name: _stable(worker) for worker in reordered} == by_name - - primary_after_close = _single_worker(config, primary) - assert _stable(primary_after_close) == by_name[primary["agent"]] - - -def test_split_creation_adds_a_distinct_restart_stable_identity(tmp_path: Path) -> None: - fixture = _fixture() - primary = deepcopy(fixture["post_restore"]["pane_info"]) - created = deepcopy(fixture["split_creation"]["data"]["pane"]) - config = _config(tmp_path / "state") - backend, workers, bindings, records = _project(config, [], [primary]) - original = workers[0] - original_key = _stable(original) - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {primary["pane_id"]: primary["terminal_id"]} - backend._replace_ownership_maps(records, bindings) - - assert backend.queue_event_envelope(fixture["split_creation"], flush=True) - - assert len(backend._workers) == 2 - keys = {_stable(worker) for worker in backend._workers.values()} - assert original_key in keys - assert len(keys) == 2 - restarted_created = _single_worker(_config(config.data_dir), created) - assert _stable(restarted_created) in keys - public = json.dumps( - [worker.to_dict() for worker in backend._workers.values()], - sort_keys=True, - ) - assert created["pane_id"] not in public - assert created["terminal_id"] not in public - - -def test_same_pane_suffix_in_distinct_workspaces_has_distinct_keys(tmp_path: Path) -> None: - first_item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - first_item["workspace_id"] = "wA" - first_item["pane_id"] = "wA:p7" - second_item = deepcopy(first_item) - second_item["workspace_id"] = "wB" - second_item["pane_id"] = "wB:p7" - config = _config(tmp_path / "state") - - first = _single_worker(config, first_item) - second = _single_worker(config, second_item) - - assert _stable(first) != _stable(second) - assert "wA:p7" not in json.dumps(first.to_dict(), sort_keys=True) - assert "wB:p7" not in json.dumps(second.to_dict(), sort_keys=True) - - -def test_session_targeted_agent_adopts_matched_pane_identity_privately(tmp_path: Path) -> None: - fixture = _fixture() - pane = deepcopy(fixture["post_restore"]["pane_info"]) - agent = deepcopy(pane) - agent["agent"] = "codex" - agent.pop("workspace_id") - agent.pop("pane_id") - config = _config(tmp_path / "state") - - _backend, workers, _bindings, records = _project(config, [agent], [pane]) - - assert len(records) == len(workers) == 1 - assert records[0].workspace_id == "wR9" - assert records[0].pane_id == "wR9:pA" - assert _STABLE_KEY.fullmatch(_stable(workers[0])) - public = json.dumps(workers[0].to_dict(), sort_keys=True) - assert "wR9:pA" not in public - assert pane["terminal_id"] not in public - assert pane["agent_session"]["value"] not in public - - -def test_matched_pane_overrides_conflicting_agent_continuity_and_workspace( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" - agent = { - "worker_id": "public-conflicting-agent", - "agent_id": "agent-send-secret", - "terminal_id": pane["terminal_id"], - "agent": "codex", - "agent_status": "working", - "agent_session": { - "source": "conflicting-agent-source-secret", - "agent": "codex", - "kind": "id", - "value": "conflicting-agent-session-secret", - }, - "workspace_id": "wD2", - "pane_id": "wD2:pA", - } - config = _config(tmp_path / "state") - - _backend, pane_workers, _bindings, pane_records = _project(config, [], [pane]) - _backend, agent_workers, _bindings, agent_records = _project(config, [agent]) - _backend, merged_workers, merged_bindings, merged_records = _project(config, [agent], [pane]) - - assert len(pane_workers) == len(agent_workers) == len(merged_workers) == 1 - assert len(pane_records) == len(agent_records) == len(merged_records) == 1 - assert _stable(merged_workers[0]) == _stable(pane_workers[0]) - assert "stable_key" not in agent_workers[0].meta - assert "stable_key_version" not in agent_workers[0].meta - assert merged_records[0].workspace_id == pane_records[0].workspace_id == "wR9" - assert merged_records[0].pane_id == pane_records[0].pane_id == "wR9:pA" - assert merged_records[0].workspace_id != agent_records[0].workspace_id - assert merged_records[0].pane_id != agent_records[0].pane_id - assert merged_workers[0].space_id == pane_workers[0].space_id == "wR9" - assert merged_workers[0].space_id != agent_workers[0].space_id - assert merged_workers[0].backend_target == { - "kind": "agent_id", - "value": "agent-send-secret", - "sendable": True, - "reason": None, - } - assert len(merged_bindings) == 1 - assert merged_bindings[0].target_kind == "agent_id" - assert merged_bindings[0].target_value == "agent-send-secret" - assert merged_bindings[0].turn_target_kind is None - assert merged_bindings[0].turn_target_value is None - - public = json.dumps(merged_workers[0].to_dict(), sort_keys=True) - for private_value in ( - pane["pane_id"], - pane["terminal_id"], - pane["agent_session"]["source"], - pane["agent_session"]["value"], - agent["agent_id"], - agent["pane_id"], - agent["workspace_id"], - agent["agent_session"]["source"], - agent["agent_session"]["value"], - ): - assert private_value not in public - - -@pytest.mark.parametrize( - ("pane_workspace_id", "pane_id"), - [ - (None, "wR9:pA"), - ("wR9", None), - ("wR9", "wR9:pI"), - ], -) -def test_matched_incomplete_or_invalid_pane_suppresses_agent_identity_derivation( - tmp_path: Path, - pane_workspace_id: str | None, - pane_id: str | None, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" - if pane_workspace_id is None: - pane.pop("workspace_id") - else: - pane["workspace_id"] = pane_workspace_id - if pane_id is None: - pane.pop("pane_id") - else: - pane["pane_id"] = pane_id - agent = { - "worker_id": "public-conflicting-agent", - "agent_id": "agent-send-secret", - "terminal_id": pane["terminal_id"], +def _pane(**updates): + pane = { + "workspace_id": "wR9", + "pane_id": "wR9:pA", + "terminal_id": "term-one", "agent": "codex", - "agent_session": deepcopy(pane["agent_session"]), - "workspace_id": "wD2", - "pane_id": "wD2:pA", + "label": "Review", } - config = _config(tmp_path / f"state-{pane_workspace_id}-{pane_id}") - - _backend, pane_workers, _bindings, pane_records = _project(config, [], [pane]) - _backend, agent_workers, _bindings, _agent_records = _project(config, [agent]) - _backend, merged_workers, _bindings, merged_records = _project(config, [agent], [pane]) - - assert "stable_key" not in agent_workers[0].meta - assert "stable_key_version" not in agent_workers[0].meta - assert "stable_key" not in pane_workers[0].meta - assert "stable_key" not in merged_workers[0].meta - assert "stable_key_version" not in merged_workers[0].meta - assert merged_records[0].workspace_id == pane_records[0].workspace_id - assert merged_records[0].pane_id == pane_records[0].pane_id - assert merged_workers[0].space_id == pane_workers[0].space_id - assert merged_workers[0].backend_target is not None - assert merged_workers[0].backend_target["kind"] == "agent_id" - assert merged_workers[0].backend_target["value"] == "agent-send-secret" - - -def test_unmatched_agent_list_identity_never_authorizes_continuity( - tmp_path: Path, -) -> None: - agent = deepcopy(_fixture()["post_restore"]["pane_info"]) - agent["worker_id"] = "public-unmatched-agent" - agent["agent_id"] = "unmatched-agent-target-secret" - agent["agent"] = "codex" - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project(config, [agent]) + pane.update(updates) + return pane + + +def test_camel_snake_nested_envelopes_preserve_workspace_pane_agent_association(tmp_path) -> None: + spaces = _discovered_spaces( + {"result": {"data": {"workspaces": [{"workspaceId": "wR9", "title": "Repo"}]}}} + ) + workers, bindings = _discovered_workers( + _config(tmp_path), + {"result": {"payload": {"panes": [{ + "workspaceId": "wR9", + "paneId": "wR9:pA", + "terminalId": "term-one", + "agent": "pane fallback", + }]}}}, + {"data": {"agents": [{ + "agentId": "agent-private", + "paneId": "wR9:pA", + "name": "codex", + "agentStatus": "waiting", + }]}}, + OBSERVED_AT, + ) + assert [(space.id, space.name) for space in spaces] == [("wR9", "Repo")] + assert [(worker.name, worker.space_id, worker.status) for worker in workers] == [ + ("codex", "wR9", "waiting") + ] + assert bindings[0].target_kind == "agent_id" + assert bindings[0].target_value == "agent-private" - assert len(records) == len(workers) == len(bindings) == 1 - assert records[0].pane_info_observed is False - assert "stable_key" not in workers[0].meta - assert "stable_key_version" not in workers[0].meta - assert not config.installation_key_path.exists() - assert workers[0].backend_target == { - "kind": "agent_id", - "value": "unmatched-agent-target-secret", - "sendable": True, - "reason": None, - } - assert bindings[0].turn_target_kind is None - assert bindings[0].turn_target_value is None - public = json.dumps(workers[0].to_dict(), sort_keys=True) - for private_value in ( - agent["pane_id"], - agent["terminal_id"], - agent["agent_id"], - agent["agent_session"]["source"], - agent["agent_session"]["value"], - ): - assert private_value not in public +def test_discovery_derives_opaque_restart_stable_key_across_rename_and_reorder(tmp_path) -> None: + pane = _pane() + workers, bindings = _discover(tmp_path, [pane]) + first_key = workers[0].meta["stable_key"] + renamed = {**pane, "agent": "renamed", "label": "Renamed"} + reordered_workers, _ = _discover(tmp_path, [_pane(pane_id="wR9:pB"), renamed]) + renamed_worker = next(worker for worker in reordered_workers if worker.meta["stable_key"] == first_key) + assert renamed_worker.name == "renamed" + assert re.fullmatch(r"wsk1_[0-9a-f]{64}", first_key) + assert bindings[0].target_value == "term-one" + assert "term-one" not in json.dumps(workers[0].to_dict()) -def test_conflicting_match_keys_across_two_panes_fail_closed( - tmp_path: Path, -) -> None: - fixture = _fixture()["post_restore"] - first = deepcopy(fixture["pane_info"]) - second = deepcopy(fixture["sibling_pane_info"]) - second["agent"] = "codex" - second["agent_session"] = { - "source": "second-pane-source-secret", - "agent": "codex", - "kind": "id", - "value": "second-pane-session-secret", - } - agent = { - "worker_id": "public-ambiguous-agent", - "agent_id": "ambiguous-agent-target-secret", - "workspace_id": first["workspace_id"], - "pane_id": first["pane_id"], - "terminal_id": second["terminal_id"], - "agent": "codex", - "agent_session": deepcopy(second["agent_session"]), - } - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project( - config, - [agent], - [first, second], +def test_discovery_reuses_prior_public_worker_id_by_private_binding(tmp_path) -> None: + workers, bindings = _discover(tmp_path, [_pane(agent="old-name")]) + prior = replace( + bindings[0], + worker_id="durable-public-id", + worker_fingerprint=workers[0].fingerprint, ) - - assert len(records) == len(workers) == len(bindings) == 1 - assert records[0].pane_info_observed is False - assert "stable_key" not in workers[0].meta - assert "stable_key_version" not in workers[0].meta - assert workers[0].backend_target == { - "kind": "agent_id", - "value": "ambiguous-agent-target-secret", - "sendable": False, - "reason": "ambiguous_pane_match", - } - assert bindings[0].sendable is False - assert bindings[0].reason == "ambiguous_pane_match" - assert bindings[0].turn_target_kind is None - assert bindings[0].turn_target_value is None - - public = json.dumps(workers[0].to_dict(), sort_keys=True) - for private_value in ( - first["pane_id"], - first["terminal_id"], - first["agent_session"]["value"], - second["pane_id"], - second["terminal_id"], - second["agent_session"]["source"], - second["agent_session"]["value"], - agent["agent_id"], - ): - assert private_value not in public + current, current_bindings = _discover( + tmp_path, + [_pane(agent="new-name")], + prior_bindings=[prior], + ) + assert current[0].id == "durable-public-id" + assert current_bindings[0].worker_id == "durable-public-id" -def test_two_agents_claiming_one_pane_fail_closed_independent_of_order( - tmp_path: Path, +@pytest.mark.parametrize("match_mode", ["multi_key", "same_pane"]) +@pytest.mark.parametrize("reverse", [False, True]) +def test_conflicting_agent_matches_are_non_sendable_in_any_row_order( + tmp_path, match_mode, reverse ) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" + second_match = ( + {"terminal_id": "term-one"} + if match_mode == "multi_key" + else {"pane_id": "wR9:pA"} + ) agents = [ - { - "worker_id": f"public-agent-{suffix}", - "agent_id": f"agent-target-{suffix}-secret", - "workspace_id": pane["workspace_id"], - "pane_id": pane["pane_id"], - "terminal_id": pane["terminal_id"], - "agent": "codex", - "agent_session": deepcopy(pane["agent_session"]), - } - for suffix in ("a", "b") + {"agent_id": "by-pane", "pane_id": "wR9:pA", "name": "one"}, + {"agent_id": "second", "name": "two", **second_match}, ] - projections: list[tuple[tuple[Any, ...], ...]] = [] - - for index, ordered_agents in enumerate((agents, list(reversed(agents)))): - config = _config(tmp_path / f"state-{index}") - _backend, workers, bindings, records = _project( - config, - ordered_agents, - [pane], - ) + if reverse: + agents.reverse() + workers, bindings = _discover(tmp_path, [_pane()], agents) + assert len(workers) == len(bindings) == 1 + assert workers[0].backend_target["sendable"] is False + assert workers[0].backend_target["reason"] == "ambiguous_pane_match" + assert bindings[0].sendable is False + assert bindings[0].reason == "ambiguous_pane_match" - assert len(records) == len(workers) == len(bindings) == 2 - assert all(record.pane_info_observed is False for record in records) - assert all("stable_key" not in worker.meta for worker in workers) - assert all("stable_key_version" not in worker.meta for worker in workers) - assert all( - worker.backend_target is not None - and worker.backend_target["sendable"] is False - and worker.backend_target["reason"] == "ambiguous_pane_match" - for worker in workers - ) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert all(binding.turn_target_kind is None for binding in bindings) - assert all(binding.turn_target_value is None for binding in bindings) - assert not config.installation_key_path.exists() - projections.append( - tuple( - sorted( - ( - worker.id, - worker.name, - worker.space_id, - tuple(sorted((worker.backend_target or {}).items())), - ) - for worker in workers - ) - ) - ) - assert projections[0] == projections[1] +def test_duplicate_private_identity_fences_public_and_private_targets(tmp_path) -> None: + workers, bindings = _discover(tmp_path, [_pane(), _pane()]) + assert len({worker.id for worker in workers}) == 2 + assert all(worker.backend_target["sendable"] is False for worker in workers) + assert all(worker.backend_target["reason"] == "duplicate_backend_target" for worker in workers) + assert all(binding.sendable is False for binding in bindings) + assert all(binding.worker_fingerprint == worker.fingerprint for worker, binding in zip(workers, bindings)) -@pytest.mark.parametrize( - "shared_agent_owner", - ["backend_target", "turn_target", "send_token"], -) -def test_distinct_panes_with_shared_agent_owner_fail_closed_in_any_order( - tmp_path: Path, - shared_agent_owner: str, -) -> None: - fixture = _fixture()["post_restore"] +def test_duplicate_send_token_is_fenced_even_when_private_identities_differ(tmp_path) -> None: panes = [ - deepcopy(fixture["pane_info"]), - deepcopy(fixture["sibling_pane_info"]), + _pane(pane_id="wR9:pA", terminal_id="same", agent="A"), + _pane(pane_id="wR9:pB", terminal_id="same", agent="B"), ] - for pane in panes: - pane["agent"] = "codex" - pane.pop("agent_session", None) - if shared_agent_owner == "send_token": - panes[1]["terminal_id"] = "shared-agent-target-secret" - agents = [] - for index, pane in enumerate(panes): - agents.append( - { - "worker_id": f"public-owner-{index}", - "agent_id": ( - "shared-agent-target-secret" - if shared_agent_owner == "backend_target" - or ( - shared_agent_owner == "send_token" - and index == 0 - ) - else ( - None - if shared_agent_owner == "send_token" - else f"agent-target-{index}-secret" - ) - ), - "workspace_id": pane["workspace_id"], - "pane_id": pane["pane_id"], - "terminal_id": pane["terminal_id"], - "agent": "codex", - "agent_session": { - "source": f"source-{index}-secret", - "agent": "codex", - "kind": "id", - "value": ( - "shared-session-secret" - if shared_agent_owner == "turn_target" - else f"session-{index}-secret" - ), - }, - } - ) - - projections: list[tuple[tuple[Any, ...], ...]] = [] - for order, (agent_rows, pane_rows) in enumerate( - ( - (agents, panes), - (list(reversed(agents)), list(reversed(panes))), - ) - ): - config = _config(tmp_path / f"{shared_agent_owner}-{order}") - _backend, workers, bindings, records = _project( - config, - agent_rows, - pane_rows, - ) - - assert len(records) == len(workers) == len(bindings) == 2 - assert all(record.pane_info_observed is False for record in records) - assert all("stable_key" not in worker.meta for worker in workers) - assert all( - worker.backend_target is not None - and worker.backend_target["sendable"] is False - and worker.backend_target["reason"] == "ambiguous_pane_match" - for worker in workers - ) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert all(binding.turn_target_kind is None for binding in bindings) - assert all(binding.turn_target_value is None for binding in bindings) - projections.append( - tuple( - sorted( - ( - worker.id, - tuple(sorted((worker.backend_target or {}).items())), - ) - for worker in workers - ) - ) - ) - - assert projections[0] == projections[1] - - -@pytest.mark.parametrize( - "shared_owner_key", - ["terminal_id", "agent_session", "private_fingerprint"], -) -def test_conflicting_pane_owner_key_fails_closed_independent_of_row_order( - tmp_path: Path, - shared_owner_key: str, -) -> None: - fixture = _fixture()["post_restore"] - first = deepcopy(fixture["pane_info"]) - second = deepcopy(fixture["sibling_pane_info"]) - first["agent"] = "codex" - second["agent"] = "omp" - first["agent_session"] = { - "source": "first-source-secret", - "agent": "codex", - "kind": "id", - "value": "first-session-secret", - } - second["agent_session"] = { - "source": "second-source-secret", - "agent": "omp", - "kind": "id", - "value": "second-session-secret", - } - if shared_owner_key == "terminal_id": - first["terminal_id"] = second["terminal_id"] = "shared-terminal-secret" - elif shared_owner_key == "agent_session": - second["agent_session"]["value"] = first["agent_session"]["value"] - else: - first.pop("agent_session") - second.pop("agent_session") - first["agent_id"] = second["agent_id"] = "shared-agent-target-secret" - - projections: list[tuple[tuple[Any, ...], ...]] = [] - for index, panes in enumerate(([first, second], [second, first])): - config = _config(tmp_path / f"{shared_owner_key}-{index}") - _backend, workers, bindings, records = _project(config, [], panes) - - assert len(records) == len(workers) == len(bindings) == 2 - assert all(record.pane_info_observed is False for record in records) - assert all("stable_key" not in worker.meta for worker in workers) - assert all("stable_key_version" not in worker.meta for worker in workers) - assert all( - worker.backend_target is not None - and worker.backend_target["sendable"] is False - and worker.backend_target["reason"] == "ambiguous_pane_match" - for worker in workers - ) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert all(binding.turn_target_kind is None for binding in bindings) - assert all(binding.turn_target_value is None for binding in bindings) - assert not config.installation_key_path.exists() - projections.append( - tuple( - sorted( - ( - worker.id, - worker.name, - worker.space_id, - tuple(sorted((worker.backend_target or {}).items())), - ) - for worker in workers - ) - ) - ) - - assert projections[0] == projections[1] - - -def test_unmatched_agent_send_token_colliding_with_pane_fails_closed( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["terminal_id"] = "shared-send-token-secret" - agent = { - "worker_id": "public-unmatched-agent", - "agent_id": "shared-send-token-secret", - "workspace_id": "wD2", - "agent": "other-agent", - } - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project( - config, - [agent], - [pane], - ) - - assert len(records) == len(workers) == len(bindings) == 2 - assert all(record.pane_info_observed is False for record in records) - assert all( - worker.backend_target is not None - and worker.backend_target["sendable"] is False - and worker.backend_target["reason"] == "ambiguous_pane_match" - for worker in workers - ) + workers, bindings = _discover(tmp_path, panes) + assert all(worker.backend_target["sendable"] is False for worker in workers) assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert not config.installation_key_path.exists() - - -def test_exact_duplicate_pane_rows_collapse_without_losing_continuity( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project(config, [], [pane, deepcopy(pane)]) - - assert len(records) == len(workers) == len(bindings) == 1 - assert records[0].pane_info_observed is True - assert _STABLE_KEY.fullmatch(_stable(workers[0])) - assert bindings[0].sendable is True - - -def test_matched_pane_replaces_conflicting_pane_scoped_targets( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" - agent = { - "worker_id": "public-pane-target-conflict", - "workspace_id": "wD2", - "pane_id": "wD2:pA", - "agent": "other-agent", - "agent_session": deepcopy(pane["agent_session"]), - } - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project(config, [agent], [pane]) - assert len(records) == len(workers) == len(bindings) == 1 - assert records[0].pane_info_observed is True - assert _STABLE_KEY.fullmatch(_stable(workers[0])) - assert records[0].workspace_id == pane["workspace_id"] - assert records[0].pane_id == pane["pane_id"] - assert records[0].terminal_id == pane["terminal_id"] - assert workers[0].space_id == pane["workspace_id"] - assert workers[0].backend_target == { - "kind": "terminal_id", - "value": pane["terminal_id"], - "sendable": True, - "reason": None, - } - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == pane["terminal_id"] - assert bindings[0].turn_target_kind is None - assert bindings[0].turn_target_value is None - assert agent["pane_id"] != bindings[0].target_value - - public = json.dumps(workers[0].to_dict(), sort_keys=True) - for private_value in ( - pane["pane_id"], - pane["terminal_id"], - pane["agent_session"]["source"], - pane["agent_session"]["value"], - agent["pane_id"], - ): - assert private_value not in public - - -def test_cli_agent_success_always_enriches_identity_from_matching_pane( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" - agent = { - "terminal_id": pane["terminal_id"], - "agent": "codex", - "agent_status": "working", - "agent_session": deepcopy(pane["agent_session"]), - } - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): {"result": {"agents": [agent]}}, - ("pane", "list"): {"result": {"panes": [pane]}}, - } - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Any, config: Config) -> subprocess.CompletedProcess[str]: - del config - calls.append(tuple(args)) - response = responses.get(tuple(args)) - return subprocess.CompletedProcess( - args=list(args), - returncode=0 if response is not None else 1, - stdout=json.dumps(response) if response is not None else "", - stderr="", - ) - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _binary: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) +def test_missing_and_malformed_pane_identity_fail_closed(tmp_path) -> None: + workers, bindings = _discover( + tmp_path, + [ + {"workspace_id": "bad", "pane_id": "pane", "terminal_id": "term", "agent": "bad"}, + {"workspace_id": "wR9", "agent": "missing"}, + ], + ) + assert len(workers) == len(bindings) == 1 + assert bindings[0].reason == "invalid_pane_identity" + assert workers[0].backend_target["sendable"] is False - _spaces, workers = herdr_cli.fetch_herdr_state(_config(tmp_path / "state")) - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), +def test_closed_and_unknown_statuses_are_projected_canonically(tmp_path) -> None: + panes = [ + _pane(pane_id="wR9:pA", terminal_id="one", agent="A", status="closed"), + _pane(pane_id="wR9:pB", terminal_id="two", agent="B", status="mystery"), ] - assert len(workers) == 1 - assert _STABLE_KEY.fullmatch(_stable(workers[0])) - - -@pytest.mark.parametrize( - ("workspace_id", "pane_id"), - [ - (None, "wR9:pA"), - ("wR9", None), - ("wR9", "wOther:pA"), - ("wR9", "wR9:pI"), - ("wR9", "wR9:pa"), - ("wR9", "wR9:p"), - ("wR9", "wR9:A"), - ("not-herdr-id", "not-herdr-id:pA"), - ("wI", "wI:pA"), - ], -) -def test_identity_requires_both_canonical_membership_and_herdr_alphabet( - tmp_path: Path, - workspace_id: str | None, - pane_id: str | None, -) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item["terminal_id"] = "durable-terminal-is-not-a-fallback" - if workspace_id is None: - item.pop("workspace_id", None) - else: - item["workspace_id"] = workspace_id - if pane_id is None: - item.pop("pane_id", None) - else: - item["pane_id"] = pane_id - - worker = _single_worker(_config(tmp_path / "state"), item) - - assert "stable_key" not in worker.meta - assert "stable_key_version" not in worker.meta - - -def test_reconcile_and_identical_events_share_one_canonical_worker_fingerprint( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state", host_id="fingerprint-stability") - config.data_dir.mkdir(parents=True, mode=0o700) - init_store(Path(config.db_path)) - - class PaneClient: - def workspace_list(self, **_kwargs: Any) -> dict[str, Any]: - return {"workspaces": [{"id": pane["workspace_id"], "name": "Continuity"}]} - - def tab_list(self, **_kwargs: Any) -> dict[str, Any]: - return {"tabs": []} - - def pane_list(self, **_kwargs: Any) -> dict[str, Any]: - return {"panes": [deepcopy(pane)]} - - def agent_list(self, **_kwargs: Any) -> dict[str, Any]: - return {"agents": []} + workers, _ = _discover(tmp_path, panes) + assert {worker.name: worker.status for worker in workers} == {"A": "closed", "B": "unknown"} - backend = HerdrEventBackend(config, debounce_seconds=0) - client = PaneClient() - phases: list[tuple[Any, Any]] = [] - def capture() -> None: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") - assert snapshot is not None - assert len(snapshot.workers) == len(bindings) == 1 - phases.append((snapshot.workers[0], bindings[0])) - - backend.reconcile_once(client=client) - capture() - envelope = {"event": "pane.focused", "data": {"pane": deepcopy(pane)}} - assert backend.queue_event_envelope(envelope) is True - capture() - backend.reconcile_once(client=client) - capture() - assert backend.queue_event_envelope(envelope) is True - capture() - - workers = [worker for worker, _binding in phases] - bindings = [binding for _worker, binding in phases] - assert len({worker.id for worker in workers}) == 1 - assert len({_stable(worker) for worker in workers}) == 1 - assert len({worker.fingerprint for worker in workers}) == 1 - assert len({binding.worker_fingerprint for binding in bindings}) == 1 - assert all(binding.worker_id == worker.id for worker, binding in phases) - assert all(binding.worker_fingerprint == worker.fingerprint for worker, binding in phases) - assert len( - { - ( - binding.target_kind, - binding.target_value, - binding.turn_target_kind, - binding.turn_target_value, - binding.private_fingerprint, - binding.sendable, - binding.reason, - ) - for binding in bindings - } - ) == 1 - - worker = workers[0] - canonical = Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, +def test_authoritative_empty_discovery_expires_prior_binding(tmp_path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + workers, bindings = _discover(tmp_path, [_pane()]) + save_snapshot( + config.db_path, + Snapshot(host_id=config.host_id, updated_at=OBSERVED_AT, workers=workers), ) - pre_key_meta = { - key: value - for key, value in worker.meta.items() - if key not in {"stable_key", "stable_key_version"} - } - pre_key = Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=pre_key_meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - ) - assert worker.fingerprint == canonical.fingerprint - assert worker.fingerprint != pre_key.fingerprint - + upsert_worker_bindings(config.db_path, bindings) -def test_same_workspace_move_preserves_and_cross_workspace_move_changes_key(tmp_path: Path) -> None: - fixture = _fixture() - config = _config(tmp_path / "state") - initial = fixture["pre_restore"]["pane_info"] - backend, workers, bindings, _records = _project(config, [initial], [initial]) - original = workers[0] - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {initial["pane_id"]: initial["terminal_id"]} + class EmptyLifecycleClient: + def workspace_list(self, *, timeout): return [] + def pane_list(self, *, timeout): return [] + def agent_list(self, *, timeout): return [] + def close(self): return None - assert backend.queue_event_envelope(fixture["same_workspace_move"], flush=True) - same_workspace = backend._workers[original.id] - assert _stable(same_workspace) == _stable(original) - - assert backend.queue_event_envelope(fixture["cross_workspace_move"], flush=True) - cross_workspace = backend._workers[original.id] - assert _stable(cross_workspace) != _stable(original) - expected = _single_worker( + supervisor = AcpSupervisor( config, - fixture["cross_workspace_move"]["data"]["pane"], - ) - assert _stable(cross_workspace) == _stable(expected) - assert cross_workspace.space_id == "wD2" - - -def test_compatibility_only_partial_move_preserves_authenticated_local_key( - tmp_path: Path, -) -> None: - initial = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state") - backend, workers, bindings, _records = _project(config, [initial], [initial]) - original = workers[0] - original_key = _stable(original) - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {initial["pane_id"]: initial["terminal_id"]} - - assert backend.queue_event_envelope( - { - "event": "pane.moved", - "payload": { - "previous_pane_id": initial["pane_id"], - "new_pane_id": "wR9:pB", - }, - }, - flush=True, + threading.Event(), + endpoint_client_factory=lambda _config: object(), + discovery_client_factory=lambda _config: EmptyLifecycleClient(), + connection_factory=lambda *_args, **_kwargs: object(), ) + supervisor._discover_continuity() + snapshot = latest_snapshot(config.db_path, config.host_id) + assert snapshot is not None and snapshot.workers == [] + assert list_worker_bindings(config.db_path, config.host_id, backend="herdr") == [] - moved = backend._workers[original.id] - moved_binding = next(iter(backend._bindings.values())) - assert _stable(moved) == original_key - assert moved_binding.target_kind == "pane_id" - assert moved_binding.target_value == "wR9:pB" +def test_supervisor_reconcile_reuses_exact_binding_then_rebinds_target_churn(tmp_path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) -def test_complete_authoritative_move_retains_state_when_rederivation_fails( - tmp_path: Path, -) -> None: - initial = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state") - backend, workers, bindings, _records = _project(config, [initial], [initial]) - original = workers[0] - original_binding = bindings[0] - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {initial["pane_id"]: initial["terminal_id"]} - replacement = bytes( - byte ^ 0xFF for byte in config.installation_key_path.read_bytes() - ) - config.installation_key_path.write_bytes(replacement) - os.chmod(config.installation_key_path, 0o600) - moved_pane = deepcopy(initial) - moved_pane["workspace_id"] = "wD2" - moved_pane["pane_id"] = "wD2:p7" - moved_pane["terminal_id"] = "terminal-moved-secret" - - assert backend.queue_event_envelope( - { - "event": "pane.moved", - "data": { - "previous_pane_id": initial["pane_id"], - "pane": moved_pane, - }, - }, - flush=True, - ) - - assert backend._workers[original.id] == original - assert next(iter(backend._bindings.values())) == original_binding - assert backend.health.status == "degraded" - assert backend.health.outcome == "continuity_unavailable" - diagnostic = json.dumps(backend.health.to_backend_health().to_dict(), sort_keys=True) - for private_value in ( - initial["pane_id"], - initial["terminal_id"], - moved_pane["pane_id"], - moved_pane["terminal_id"], - moved_pane["agent_session"]["source"], - moved_pane["agent_session"]["value"], - ): - assert private_value not in diagnostic - + class MutableLifecycleClient: + pane = _pane(agent="original", terminal_id="term-one") -def test_partial_event_preserves_local_key_and_authoritative_failure_retains_state( - tmp_path: Path, -) -> None: - initial = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state") - backend, workers, bindings, _records = _project(config, [initial], [initial]) - original = workers[0] - original_key = _stable(original) - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {initial["pane_id"]: initial["terminal_id"]} - - replacement = bytes(byte ^ 0xFF for byte in config.installation_key_path.read_bytes()) - config.installation_key_path.write_bytes(replacement) - os.chmod(config.installation_key_path, 0o600) - - assert backend.queue_event_envelope( - { - "event": "pane_agent_status_changed", - "data": {"agent": initial["agent"], "status": "blocked"}, - }, - flush=True, - ) - partial = backend._workers[original.id] - assert _stable(partial) == original_key - - assert backend.queue_event_envelope( - { - "event": "pane_agent_status_changed", - "data": {"pane_id": initial["pane_id"], "status": "working"}, - }, - flush=True, - ) - pane_only = backend._workers[original.id] - binding_before_failure = next(iter(backend._bindings.values())) - assert _stable(pane_only) == original_key + def workspace_list(self, *, timeout): return [{"id": "wR9"}] + def pane_list(self, *, timeout): return [dict(self.pane)] + def agent_list(self, *, timeout): return [] + def close(self): return None - full = deepcopy(initial) - full["agent_status"] = "idle" - assert backend.queue_event_envelope( - { - "event": "pane_agent_status_changed", - "data": {"pane": full}, - }, - flush=True, - ) - - assert backend._workers[original.id] == pane_only - assert next(iter(backend._bindings.values())) == binding_before_failure - assert backend.health.status == "degraded" - assert backend.health.outcome == "continuity_unavailable" - - -@pytest.mark.parametrize( - ( - "observed_workspace_id", - "observed_pane_id", - ), - [ - ("wR9", "wR9:pA"), - ("wR9", "wR9:pB"), - ("wD2", "wR9:pA"), - ], -) -def test_key_loss_rejects_every_authoritative_pane_update( - tmp_path: Path, - observed_workspace_id: str, - observed_pane_id: str, -) -> None: - initial = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state") - backend, workers, bindings, records = _project(config, [initial], [initial]) - original = workers[0] - original_binding = bindings[0] - backend._workers = {original.id: original} - backend._bindings = { - binding.private_fingerprint: binding - for binding in bindings - } - backend._pane_terminals = { - initial["pane_id"]: initial["terminal_id"], - } - backend._replace_ownership_maps(records, bindings) - config.installation_key_marker_path.unlink() - - observed = deepcopy(initial) - observed["pane_id"] = observed_pane_id - observed["workspace_id"] = observed_workspace_id - observed["agent"] = "codex-runtime-after-key-loss" - assert backend.queue_event_envelope( - { - "event": "pane.created", - "data": {"pane": observed}, - }, - flush=True, - ) - - assert set(backend._workers) == {original.id} - current = backend._workers[original.id] - current_binding = next(iter(backend._bindings.values())) - assert current == original - assert current_binding == original_binding - assert _stable(current) == _stable(original) - assert backend.health.status == "degraded" - assert backend.health.outcome == "continuity_unavailable" - assert backend._pane_terminals == { - initial["pane_id"]: initial["terminal_id"], - } - assert backend._pane_owners == { - initial["pane_id"]: {original.id}, - } - - -def test_installations_are_unlinkable_and_host_is_part_of_message(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - first = _single_worker(_config(tmp_path / "one"), pane) - second = _single_worker(_config(tmp_path / "two"), pane) - other_host = _single_worker(_config(tmp_path / "one", host_id="other-host"), pane) - - assert _stable(first) != _stable(second) - assert _stable(first) != _stable(other_host) - - -def test_first_bootstrap_publishes_key_marker_and_initialization_sentinel( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "state" - candidate = bytes(range(32)) - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return candidate - - assert load_or_create_installation_key(data_dir, random_bytes=generated) == candidate - assert generated_sizes == [32] - assert (data_dir / "installation.key").read_bytes() == candidate - assert (data_dir / "installation.key.sha256").read_bytes() == hashlib.sha256( - candidate, - ).hexdigest().encode("ascii") - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_valid_pre_sentinel_pair_upgrades_only_after_validation(tmp_path: Path) -> None: - data_dir = tmp_path / "state" - data_dir.mkdir(mode=0o700) - key = b"p" * 32 - (data_dir / "installation.key").write_bytes(key) - (data_dir / "installation.key.sha256").write_bytes( - hashlib.sha256(key).hexdigest().encode("ascii"), - ) - os.chmod(data_dir / "installation.key", 0o600) - os.chmod(data_dir / "installation.key.sha256", 0o600) - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"n" * size - - assert load_or_create_installation_key(data_dir, random_bytes=generated) == key - assert generated_sizes == [] - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_complete_initialized_pair_loss_fails_without_replacement(tmp_path: Path) -> None: - data_dir = tmp_path / "state" - original = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"a" * size, - ) - (data_dir / "installation.key").unlink() - (data_dir / "installation.key.sha256").unlink() - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"b" * size - - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key(data_dir, random_bytes=generated) - - assert original == b"a" * 32 - assert generated_sizes == [] - assert not (data_dir / "installation.key").exists() - assert not (data_dir / "installation.key.sha256").exists() - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_explicit_acknowledged_reset_allows_offline_key_rotation(tmp_path: Path) -> None: - data_dir = tmp_path / "state" - original = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"a" * size, - ) - - with pytest.raises(InstallationKeyError, match="reset was not acknowledged"): - reset_installation_key(data_dir, acknowledge_continuity_break=False) - assert (data_dir / "installation.key").read_bytes() == original - - reset_installation_key(data_dir, acknowledge_continuity_break=True) - assert not (data_dir / "installation.key").exists() - assert not (data_dir / "installation.key.sha256").exists() - assert not (data_dir / "installation.key.initialized").exists() - - rotated = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"b" * size, - ) - assert rotated == b"b" * 32 - assert rotated != original - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_initialized_digest_loss_fails_without_rewriting_or_randomness(tmp_path: Path) -> None: - data_dir = tmp_path / "state" - key = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"k" * size, - ) - marker_path = data_dir / "installation.key.sha256" - marker_path.unlink() - key_path = data_dir / "installation.key" - sentinel_path = data_dir / "installation.key.initialized" - key_stat = os.lstat(key_path) - sentinel_stat = os.lstat(sentinel_path) - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"n" * size - - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key(data_dir, random_bytes=generated) - - assert generated_sizes == [] - assert key_path.read_bytes() == key - assert (os.lstat(key_path).st_ino, os.lstat(key_path).st_mtime_ns) == ( - key_stat.st_ino, - key_stat.st_mtime_ns, - ) - assert not marker_path.exists() - assert sentinel_path.read_bytes() == b"1" - assert (os.lstat(sentinel_path).st_ino, os.lstat(sentinel_path).st_mtime_ns) == ( - sentinel_stat.st_ino, - sentinel_stat.st_mtime_ns, - ) - - -def test_initialized_replaced_key_and_digest_loss_fails_without_rewriting_or_randomness( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "state" - original = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"a" * size, - ) - key_path = data_dir / "installation.key" - marker_path = data_dir / "installation.key.sha256" - sentinel_path = data_dir / "installation.key.initialized" - replacement = bytes(byte ^ 0xFF for byte in original) - key_path.write_bytes(replacement) - marker_path.unlink() - key_stat = os.lstat(key_path) - sentinel_stat = os.lstat(sentinel_path) - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"n" * size - - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key(data_dir, random_bytes=generated) - - assert generated_sizes == [] - assert key_path.read_bytes() == replacement - assert (os.lstat(key_path).st_ino, os.lstat(key_path).st_mtime_ns) == ( - key_stat.st_ino, - key_stat.st_mtime_ns, - ) - assert not marker_path.exists() - assert sentinel_path.read_bytes() == b"1" - assert (os.lstat(sentinel_path).st_ino, os.lstat(sentinel_path).st_mtime_ns) == ( - sentinel_stat.st_ino, - sentinel_stat.st_mtime_ns, - ) - - -def test_missing_key_with_marker_fails_closed_without_source_fallback(tmp_path: Path) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item["meta"] = {"stableKey": "source-fallback", "stable-key-version": 999} - config = _config(tmp_path / "state") - first = _single_worker(config, item) - assert _stable(first) - config.installation_key_path.unlink() - - after_loss = _single_worker(config, item) - - assert "stable_key" not in after_loss.meta - assert "stable_key_version" not in after_loss.meta - assert "source-fallback" not in json.dumps(after_loss.to_dict()) - - -def test_replaced_key_or_marker_mismatch_fails_closed(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - config = _config(tmp_path / "state") - assert _stable(_single_worker(config, pane)) - original_key = config.installation_key_path.read_bytes() - replacement = bytes(byte ^ 0xFF for byte in original_key) - config.installation_key_path.write_bytes(replacement) - os.chmod(config.installation_key_path, 0o600) - - rotated = _single_worker(config, pane) - assert "stable_key" not in rotated.meta - assert "stable_key_version" not in rotated.meta - - config.installation_key_path.write_bytes(original_key) - config.installation_key_marker_path.write_bytes(b"0" * 64) - mismatched_marker = _single_worker(config, pane) - assert "stable_key" not in mismatched_marker.meta - - -def test_exact_key_and_marker_content_is_required(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - - short_config = _config(tmp_path / "short") - short_config.data_dir.mkdir(mode=0o700) - short_config.installation_key_path.write_bytes(b"x" * 31) - os.chmod(short_config.installation_key_path, 0o600) - assert "stable_key" not in _single_worker(short_config, pane).meta - assert not short_config.installation_key_marker_path.exists() - - marker_config = _config(tmp_path / "marker") - marker_config.data_dir.mkdir(mode=0o700) - key = b"k" * 32 - marker_config.installation_key_path.write_bytes(key) - marker_config.installation_key_marker_path.write_bytes(hashlib.sha256(key).hexdigest().encode("ascii") + b"\n") - os.chmod(marker_config.installation_key_path, 0o600) - os.chmod(marker_config.installation_key_marker_path, 0o600) - assert "stable_key" not in _single_worker(marker_config, pane).meta - assert not marker_config.installation_key_sentinel_path.exists() - - -def test_permissive_umask_still_creates_private_modes(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - config = _config(tmp_path / "state") - previous_umask = os.umask(0) - try: - assert _stable(_single_worker(config, pane)) - finally: - os.umask(previous_umask) - - assert _mode(config.data_dir) == 0o700 - assert _mode(config.installation_key_path) == 0o600 - assert _mode(config.installation_key_marker_path) == 0o600 - assert _mode(config.installation_key_sentinel_path) == 0o600 - - -def test_existing_broad_identity_modes_are_narrowed_in_place_idempotently( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "state" - data_dir.mkdir(mode=0o700) - key = b"m" * 32 - paths = { - "installation.key": data_dir / "installation.key", - "installation.key.sha256": data_dir / "installation.key.sha256", - "installation.key.initialized": data_dir / "installation.key.initialized", - } - paths["installation.key"].write_bytes(key) - paths["installation.key.sha256"].write_bytes( - hashlib.sha256(key).hexdigest().encode("ascii"), - ) - paths["installation.key.initialized"].write_bytes(b"1") - os.chmod(data_dir, 0o755) - for path in paths.values(): - os.chmod(path, 0o644) - - data_dir_inode = os.lstat(data_dir).st_ino - before = { - name: (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for name, path in paths.items() - } - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"n" * size - - assert load_or_create_installation_key(data_dir, random_bytes=generated) == key - assert generated_sizes == [] - assert (os.lstat(data_dir).st_ino, _mode(data_dir)) == (data_dir_inode, 0o700) - assert { - name: (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for name, path in paths.items() - } == before - assert {_mode(path) for path in paths.values()} == {0o600} - - repaired = { - name: (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for name, path in paths.items() - } - assert load_or_create_installation_key(data_dir, random_bytes=generated) == key - assert generated_sizes == [] - assert (os.lstat(data_dir).st_ino, _mode(data_dir)) == (data_dir_inode, 0o700) - assert { - name: (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for name, path in paths.items() - } == repaired - assert {_mode(path) for path in paths.values()} == {0o600} - - -def test_equal_or_stricter_private_identity_modes_are_accepted_without_widening( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "strict" - data_dir.mkdir(mode=0o700) - key = b"m" * 32 - marker = hashlib.sha256(key).hexdigest().encode("ascii") - (data_dir / "installation.key").write_bytes(key) - (data_dir / "installation.key.sha256").write_bytes(marker) - (data_dir / "installation.key.initialized").write_bytes(b"1") - os.chmod(data_dir / "installation.key", 0o400) - os.chmod(data_dir / "installation.key.sha256", 0o400) - os.chmod(data_dir / "installation.key.initialized", 0o400) - os.chmod(data_dir, 0o500) - generated_sizes: list[int] = [] - - assert load_or_create_installation_key( - data_dir, - random_bytes=lambda size: generated_sizes.append(size) or b"n" * size, - ) == key - assert generated_sizes == [] - assert _mode(data_dir) == 0o500 - assert _mode(data_dir / "installation.key") == 0o400 - assert _mode(data_dir / "installation.key.sha256") == 0o400 - assert _mode(data_dir / "installation.key.initialized") == 0o400 - - -def test_symlink_identity_files_and_data_dir_fail_closed(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - - key_link_config = _config(tmp_path / "key-link") - key_link_config.data_dir.mkdir(mode=0o700) - outside_key = tmp_path / "outside.key" - outside_key.write_bytes(b"z" * 32) - key_link_config.installation_key_path.symlink_to(outside_key) - assert "stable_key" not in _single_worker(key_link_config, pane).meta - assert not key_link_config.installation_key_marker_path.exists() - - sentinel_link_config = _config(tmp_path / "sentinel-link") - sentinel_link_config.data_dir.mkdir(mode=0o700) - key = b"s" * 32 - sentinel_link_config.installation_key_path.write_bytes(key) - sentinel_link_config.installation_key_marker_path.write_bytes( - hashlib.sha256(key).hexdigest().encode("ascii"), - ) - outside_sentinel = tmp_path / "outside.initialized" - outside_sentinel.write_bytes(b"1") - sentinel_link_config.installation_key_sentinel_path.symlink_to(outside_sentinel) - assert "stable_key" not in _single_worker(sentinel_link_config, pane).meta - - - real_dir = tmp_path / "real-dir" - real_dir.mkdir(mode=0o700) - linked_dir = tmp_path / "linked-dir" - linked_dir.symlink_to(real_dir, target_is_directory=True) - with pytest.raises(InstallationKeyError) as raised: - load_or_create_installation_key(linked_dir) - assert str(raised.value) == "installation identity is unavailable" - - -@pytest.mark.parametrize("operation", ["load-or-create", "acknowledged-reset"]) -def test_identity_lifecycle_stays_on_pinned_data_directory_after_leaf_replacement( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - operation: str, -) -> None: - configured_parent = tmp_path / "configured" - configured_parent.mkdir() - data_dir = configured_parent / "state" - if operation == "acknowledged-reset": - assert load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"o" * size, - ) == b"o" * 32 - - replacement_target = tmp_path / "replacement-target" - replacement_target.mkdir(mode=0o750) - guard = replacement_target / "target.sentinel" - guard.write_bytes(b"must remain unchanged") - os.chmod(guard, 0o640) - target_before = _tree_snapshot(replacement_target) - detached_data_dir = configured_parent / "detached-state" - original_prepare = ( - worker_identity.local_state.prepare_and_open_private_directory - ) - prepare_calls = 0 - - def prepare_then_replace( - path: str | os.PathLike[str], - ) -> tuple[int, Any]: - nonlocal prepare_calls - prepare_calls += 1 - fd, result = original_prepare(path) - Path(path).rename(detached_data_dir) - Path(path).symlink_to(replacement_target, target_is_directory=True) - return fd, result - - monkeypatch.setattr( - worker_identity.local_state, - "prepare_and_open_private_directory", - prepare_then_replace, - ) - - if operation == "load-or-create": - candidate = b"c" * 32 - assert load_or_create_installation_key( - data_dir, - random_bytes=lambda size: candidate[:size], - ) == candidate - assert (detached_data_dir / "installation.key").read_bytes() == candidate - assert ( - detached_data_dir / "installation.key.sha256" - ).read_bytes() == hashlib.sha256(candidate).hexdigest().encode("ascii") - assert ( - detached_data_dir / "installation.key.initialized" - ).read_bytes() == b"1" - else: - reset_installation_key( - data_dir, - acknowledge_continuity_break=True, - ) - assert list(detached_data_dir.iterdir()) == [] - - assert prepare_calls == 1 - assert data_dir.is_symlink() - assert _tree_snapshot(replacement_target) == target_before - - -@pytest.mark.parametrize( - "relative", - [ - pytest.param(False, id="absolute"), - pytest.param(True, id="relative"), - ], -) -def test_intermediate_symlink_above_identity_data_dir_blocks_load_or_create( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - relative: bool, -) -> None: - protected_target = tmp_path / "protected-target" - protected_target.mkdir(mode=0o750) - guard = protected_target / "target.sentinel" - guard.write_bytes(b"must remain unchanged") - os.chmod(guard, 0o640) - - configured_root = tmp_path / "configured" - configured_root.mkdir() - (configured_root / "redirect").symlink_to( - protected_target, - target_is_directory=True, - ) - absolute_data_dir = configured_root / "redirect" / "one" / "two" / "state" - data_dir = ( - Path("configured") / "redirect" / "one" / "two" / "state" - if relative - else absolute_data_dir - ) - if relative: - monkeypatch.chdir(tmp_path) - - before = _tree_snapshot(protected_target) - generated_sizes: list[int] = [] - with pytest.raises(InstallationKeyError) as raised: - load_or_create_installation_key( - data_dir, - random_bytes=lambda size: generated_sizes.append(size) or b"x" * size, - ) - - assert str(raised.value) == "installation identity is unavailable" - assert "configured" not in str(raised.value) - assert "protected-target" not in str(raised.value) - assert generated_sizes == [] - assert _tree_snapshot(protected_target) == before - - -@pytest.mark.parametrize( - "relative", - [ - pytest.param(False, id="absolute"), - pytest.param(True, id="relative"), - ], -) -def test_intermediate_symlink_above_identity_data_dir_blocks_acknowledged_reset( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - relative: bool, -) -> None: - protected_target = tmp_path / "protected-target" - protected_target.mkdir(mode=0o750) - guard = protected_target / "target.sentinel" - guard.write_bytes(b"must remain unchanged") - os.chmod(guard, 0o640) - target_data_dir = protected_target / "one" / "two" / "state" - key = load_or_create_installation_key( - target_data_dir, - random_bytes=lambda size: b"r" * size, - ) - assert key == b"r" * 32 - preserved_temp = target_data_dir / ".tendwire-preserved.tmp" - preserved_temp.write_bytes(b"must not be removed") - os.chmod(preserved_temp, 0o640) - os.chmod(target_data_dir, 0o755) - for name in ( - "installation.key", - "installation.key.sha256", - "installation.key.initialized", - ): - os.chmod(target_data_dir / name, 0o644) - - configured_root = tmp_path / "configured" - configured_root.mkdir() - (configured_root / "redirect").symlink_to( - protected_target, - target_is_directory=True, - ) - absolute_data_dir = configured_root / "redirect" / "one" / "two" / "state" - data_dir = ( - Path("configured") / "redirect" / "one" / "two" / "state" - if relative - else absolute_data_dir - ) - if relative: - monkeypatch.chdir(tmp_path) - - before = _tree_snapshot(protected_target) - with pytest.raises(InstallationKeyError) as raised: - reset_installation_key( - data_dir, - acknowledge_continuity_break=True, - ) - - assert str(raised.value) == "installation identity is unavailable" - assert "configured" not in str(raised.value) - assert "protected-target" not in str(raised.value) - assert _tree_snapshot(protected_target) == before - - -@pytest.mark.parametrize( - "relative", - [ - pytest.param(False, id="absolute"), - pytest.param(True, id="relative"), - ], -) -def test_multi_level_missing_parent_identity_bootstrap_uses_resolved_directories( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - relative: bool, -) -> None: - absolute_data_dir = tmp_path / "bootstrap" / "one" / "two" / "state" - data_dir = ( - Path("bootstrap") / "one" / "two" / "state" - if relative - else absolute_data_dir - ) - if relative: - monkeypatch.chdir(tmp_path) - candidate = bytes(range(32)) - - assert load_or_create_installation_key( - data_dir, - random_bytes=lambda size: candidate[:size], - ) == candidate - - for directory in ( - tmp_path / "bootstrap", - tmp_path / "bootstrap" / "one", - tmp_path / "bootstrap" / "one" / "two", - absolute_data_dir, - ): - assert directory.is_dir() - assert _mode(directory) == 0o700 - assert (absolute_data_dir / "installation.key").read_bytes() == candidate - assert (absolute_data_dir / "installation.key.sha256").read_bytes() == ( - hashlib.sha256(candidate).hexdigest().encode("ascii") - ) - assert (absolute_data_dir / "installation.key.initialized").read_bytes() == b"1" - assert { - _mode(absolute_data_dir / name) - for name in ( - "installation.key", - "installation.key.sha256", - "installation.key.initialized", - ) - } == {0o600} - assert list(absolute_data_dir.glob(".tendwire-*.tmp")) == [] - - -@pytest.mark.parametrize( - "target_name", - [ - pytest.param(None, id="state-directory"), - pytest.param("installation.key", id="key"), - pytest.param("installation.key.sha256", id="digest"), - pytest.param("installation.key.initialized", id="sentinel"), - ], -) -def test_nonregular_identity_entries_fail_closed_without_randomness( - tmp_path: Path, - target_name: str | None, -) -> None: - data_dir = tmp_path / "state" - target = data_dir - if target_name is None: - data_dir.write_bytes(b"not a directory") - else: - data_dir.mkdir(mode=0o700) - key = b"r" * 32 - if target_name != "installation.key": - (data_dir / "installation.key").write_bytes(key) - os.chmod(data_dir / "installation.key", 0o600) - if target_name == "installation.key.initialized": - (data_dir / "installation.key.sha256").write_bytes( - hashlib.sha256(key).hexdigest().encode("ascii"), - ) - os.chmod(data_dir / "installation.key.sha256", 0o600) - target = data_dir / target_name - target.mkdir() - generated_sizes: list[int] = [] - - with pytest.raises(InstallationKeyError) as raised: - load_or_create_installation_key( - data_dir, - random_bytes=lambda size: generated_sizes.append(size) or b"n" * size, - ) - - assert str(raised.value) == "installation identity is unavailable" - assert generated_sizes == [] - if target_name is None: - assert target.read_bytes() == b"not a directory" - else: - assert target.is_dir() - - -def test_wrong_owner_identity_directory_fails_closed_without_changes( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - data_dir = tmp_path / "state" - key = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"o" * size, - ) - paths = ( - data_dir / "installation.key", - data_dir / "installation.key.sha256", - data_dir / "installation.key.initialized", - ) - before = tuple( - (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for path in paths - ) - actual_uid = os.geteuid() - monkeypatch.setattr( - worker_identity.local_state.os, - "geteuid", - lambda: actual_uid + 1, - ) - generated_sizes: list[int] = [] - - with pytest.raises(InstallationKeyError) as raised: - load_or_create_installation_key( - data_dir, - random_bytes=lambda size: generated_sizes.append(size) or b"n" * size, - ) - - assert str(raised.value) == "installation identity is unavailable" - assert generated_sizes == [] - assert tuple( - (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for path in paths - ) == before - assert paths[0].read_bytes() == key - - -def test_source_stable_key_family_is_recursively_stripped_then_replaced(tmp_path: Path) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item["Stable.Key.Future"] = "outer-injection" - item["meta"] = { - "stable_key": "snake-injection", - "StableKeyVersion": 999, - "stable-key-rotation": "kebab-injection", - "sTaBlEkEyFuture": "camel-injection", - "safe": {"stable.key.next": "nested-injection", "kept": "yes"}, - "items": [{"STABLE_KEY_NEXT": "list-injection", "kept": 1}], - } - - worker = _single_worker(_config(tmp_path / "state"), item) - - assert _STABLE_KEY.fullmatch(_stable(worker)) - assert worker.meta["stable_key_version"] == 1 - assert worker.meta["safe"] == {"kept": "yes"} - assert worker.meta["items"] == [{"kept": 1}] - assert set(_reserved_meta_keys(worker.meta)) == {"stable_key", "stable_key_version"} - public = json.dumps(worker.to_dict()) - for sentinel in ( - "outer-injection", - "snake-injection", - "kebab-injection", - "camel-injection", - "nested-injection", - "list-injection", - ): - assert sentinel not in public - - -def test_source_stable_key_injection_without_identity_is_never_preserved(tmp_path: Path) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item.pop("workspace_id") - item["stableKey"] = "top-source" - item["meta"] = { - "STABLE-KEY-VERSION": 22, - "nested": {"stable.key.future": "nested-source", "safe": True}, - } - - worker = _single_worker(_config(tmp_path / "state"), item) - - assert _reserved_meta_keys(worker.meta) == [] - assert worker.meta["nested"] == {"safe": True} - assert "source" not in json.dumps(worker.to_dict()) - - -def test_key_load_occurs_once_for_a_worker_batch(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - fixture = _fixture() - calls = 0 - original = herdr_cli.load_or_create_installation_key - - def counted(data_dir: Path) -> bytes: - nonlocal calls - calls += 1 - return original(data_dir) - - monkeypatch.setattr(herdr_cli, "load_or_create_installation_key", counted) - config = _config(tmp_path / "state") - _backend, workers, _bindings, _records = _project( + client = MutableLifecycleClient() + supervisor = AcpSupervisor( config, - [], - [fixture["pre_restore"]["pane_info"], fixture["pre_restore"]["sibling_pane_info"]], - ) - - assert len(workers) == 2 - assert calls == 1 - - -def test_atomic_publication_never_exposes_partial_final_file( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - data_dir = tmp_path / "state" - original_write_all = worker_identity.local_state._write_all - - def interrupted_write(fd: int, content: bytes) -> None: - del fd, content - raise OSError("simulated interrupted write") - - monkeypatch.setattr(worker_identity.local_state, "_write_all", interrupted_write) - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key(data_dir) - - assert not (data_dir / "installation.key").exists() - assert not (data_dir / "installation.key.sha256").exists() - assert not (data_dir / "installation.key.initialized").exists() - assert list(data_dir.glob(".tendwire-*.tmp")) == [] - - monkeypatch.setattr(worker_identity.local_state, "_write_all", original_write_all) - assert len(load_or_create_installation_key(data_dir)) == 32 - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_interrupted_sentinel_publication_recovers_without_key_rotation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - data_dir = tmp_path / "state" - candidate = b"c" * 32 - original_write_all = worker_identity.local_state._write_all - - def interrupt_sentinel(fd: int, content: bytes) -> None: - if content == b"1": - raise OSError("simulated interrupted sentinel write") - original_write_all(fd, content) - - monkeypatch.setattr(worker_identity.local_state, "_write_all", interrupt_sentinel) - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key( - data_dir, - random_bytes=lambda size: candidate, - ) - - assert (data_dir / "installation.key").read_bytes() == candidate - assert (data_dir / "installation.key.sha256").read_bytes() == hashlib.sha256( - candidate, - ).hexdigest().encode("ascii") - assert not (data_dir / "installation.key.initialized").exists() - assert list(data_dir.glob(".tendwire-*.tmp")) == [] - - monkeypatch.setattr(worker_identity.local_state, "_write_all", original_write_all) - recovered = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"replacement-that-must-not-be-used"[:size], - ) - assert recovered == candidate - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_concurrent_creators_publish_one_complete_key_marker_and_sentinel( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "nested" / "state" - with ThreadPoolExecutor(max_workers=8) as executor: - keys = list(executor.map(lambda _index: load_or_create_installation_key(data_dir), range(16))) - - assert len(set(keys)) == 1 - key = keys[0] - assert (data_dir / "installation.key").read_bytes() == key - assert (data_dir / "installation.key.sha256").read_bytes() == hashlib.sha256(key).hexdigest().encode("ascii") - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - assert list(data_dir.glob(".tendwire-*.tmp")) == [] - - -def test_public_output_excludes_private_identity_and_binding_fingerprint(tmp_path: Path) -> None: - fixture = _fixture() - before = fixture["pre_restore"] - item = before["pane_info"] - config = _config(tmp_path / "state") - installation_key = b"0123456789abcdef0123456789abcdef" - config.data_dir.mkdir(mode=0o700) - config.installation_key_path.write_bytes(installation_key) - os.chmod(config.installation_key_path, 0o600) - backend, workers, _bindings, records = _project(config, [], [item]) - del backend - worker = workers[0] - record = records[0] - expected_private = worker_binding_private_fingerprint( - host_id=config.host_id, - backend="herdr", - identity_material=_private_identity_material_from_item(item), - ) - public = json.dumps(worker.to_dict(), sort_keys=True) - - assert record.private_fingerprint == expected_private - assert record.private_fingerprint != _stable(worker) - assert record.private_fingerprint not in public - assert installation_key.decode("ascii") not in public - assert installation_key.hex() not in public - assert item["pane_id"] not in public - assert item["terminal_id"] not in public - assert item["agent_session"]["source"] not in public - assert item["agent_session"]["value"] not in public - for field in ("runtime_id", "worker_id", "agent_id"): - assert str(before[field]) not in public - - -def test_stable_derivation_does_not_use_binding_or_public_fingerprint_helpers( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - from tendwire.core.models import Worker - - record = herdr_cli._WorkerRecord( - worker=Worker(id="public", name="Public", status="working", space_id="wR9"), - private_fingerprint="existing-private-fingerprint", - workspace_id="wR9", - pane_id="wR9:pA", - pane_info_observed=True, - ) - - def forbidden(*args: Any, **kwargs: Any) -> str: - raise AssertionError("unrelated fingerprint helper was called") - - monkeypatch.setattr(herdr_cli, "worker_binding_private_fingerprint", forbidden) - monkeypatch.setattr(herdr_cli, "stable_fingerprint", forbidden) - workers, bindings = _workers_and_bindings_from_records(_config(tmp_path / "state"), [record]) - - assert bindings == [] - assert _STABLE_KEY.fullmatch(_stable(workers[0])) + threading.Event(), + endpoint_client_factory=lambda _config: object(), + discovery_client_factory=lambda _config: client, + connection_factory=lambda *_args, **_kwargs: object(), + ) + supervisor._discover_continuity() + first = latest_snapshot(config.db_path, config.host_id) + assert first is not None and first.workers[0].id == "original" + + first_stable_key = first.workers[0].meta["stable_key"] + client.pane = _pane(agent="renamed", terminal_id="term-one") + supervisor._discover_continuity() + second = latest_snapshot(config.db_path, config.host_id) + assert second is not None and second.workers[0].id == "original" + + client.pane = _pane(agent="renamed-again", terminal_id="term-two") + supervisor._discover_continuity() + third = latest_snapshot(config.db_path, config.host_id) + bindings = list_worker_bindings(config.db_path, config.host_id, backend="herdr") + assert third is not None + assert third.workers[0].meta["stable_key"] == first_stable_key + assert bindings[0].worker_id == third.workers[0].id + assert bindings[0].target_value == "term-two" + + +def test_public_projection_contains_no_raw_private_identifiers(tmp_path) -> None: + workers, _ = _discover( + tmp_path, + [_pane(cwd="/secret/private", terminal_id="term-private")], + [{"agent_id": "agent-private", "pane_id": "wR9:pA", "name": "codex"}], + ) + encoded = json.dumps(workers[0].to_dict(), sort_keys=True) + assert "term-private" not in encoded + assert "agent-private" not in encoded + assert "/secret/private" not in encoded From f393dd1ccc776dff98413de3f53f1c2f9884c227 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 22:16:36 +0800 Subject: [PATCH 75/83] fix: align socket lifecycle and daemon startup --- src/tendwire/backends/acp_coordinator.py | 15 +++++----- src/tendwire/backends/herdr_socket.py | 26 ++++++----------- src/tendwire/config.py | 5 +++- src/tendwire/daemon.py | 36 +++++++++++------------- tests/test_acp_coordinator.py | 9 ++++++ tests/test_config.py | 24 ++++++++++++++-- tests/test_daemon.py | 5 ++-- tests/test_daemon_acp.py | 23 ++++++++++++--- tests/test_herdr_socket.py | 35 +++++++++++++++++++---- 9 files changed, 118 insertions(+), 60 deletions(-) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py index a9a678d..35f57a8 100644 --- a/src/tendwire/backends/acp_coordinator.py +++ b/src/tendwire/backends/acp_coordinator.py @@ -10,6 +10,7 @@ from __future__ import annotations import json +import math import threading import time from collections import Counter @@ -254,14 +255,14 @@ def __init__( self._permission_callback = permission_callback self._require_permission_bridge = bool(require_permission_bridge) self._durable_permission_bridge = bool(durable_permission_bridge) - self._reconcile_interval = max( - 1.0, - float( - config.reconcile_interval_seconds - if reconcile_interval is None - else reconcile_interval - ), + resolved_reconcile_interval = float( + config.reconcile_interval_seconds + if reconcile_interval is None + else reconcile_interval ) + if not math.isfinite(resolved_reconcile_interval) or resolved_reconcile_interval <= 0: + raise ValueError("reconcile_interval must be finite and positive") + self._reconcile_interval = resolved_reconcile_interval self._lock = threading.RLock() # Endpoint minting, runtime publication, prompt lease validation, and # shutdown are one private generation transaction. Herdr diff --git a/src/tendwire/backends/herdr_socket.py b/src/tendwire/backends/herdr_socket.py index 6a2710b..75739b4 100644 --- a/src/tendwire/backends/herdr_socket.py +++ b/src/tendwire/backends/herdr_socket.py @@ -110,11 +110,15 @@ def request( timeout: float | None = None, ) -> Any: """Send one strictly correlated request and return its raw result payload.""" - request_id, deadline = self._send_request(method, params, timeout=timeout) - response = self._read_response(request_id, deadline=deadline) - if is_error_response(response): - raise HerdrErrorResponse(error_payload(response), request_id) - return result_payload(response) + try: + request_id, deadline = self._send_request(method, params, timeout=timeout) + response = self._read_response(request_id, deadline=deadline) + if is_error_response(response): + raise HerdrErrorResponse(error_payload(response), request_id) + return result_payload(response) + finally: + # Herdr accepts exactly one request on each Unix connection. + self.close() def workspace_list( self, @@ -229,18 +233,6 @@ def _write(self, payload: bytes, *, deadline: float) -> None: try: sock.settimeout(self._remaining(deadline)) sock.sendall(payload) - except (BrokenPipeError, ConnectionResetError) as exc: - self.close() - try: - self.connect() - sock = self._active_socket() - sock.settimeout(self._remaining(deadline)) - sock.sendall(payload) - except socket.timeout as retry_exc: - raise HerdrSocketTimeoutError("Herdr socket write timed out") from retry_exc - except OSError as retry_exc: - self.close() - raise HerdrSocketDisconnectedError("Herdr socket disconnected during write") from retry_exc except socket.timeout as exc: raise HerdrSocketTimeoutError("Herdr socket write timed out") from exc except OSError as exc: diff --git a/src/tendwire/config.py b/src/tendwire/config.py index 788bb7a..ce17f8b 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -161,7 +161,10 @@ def __post_init__(self) -> None: object.__setattr__( self, "reconcile_interval_seconds", - _non_negative_float(self.reconcile_interval_seconds, "reconcile_interval_seconds"), + _positive_finite_float( + self.reconcile_interval_seconds, + "reconcile_interval_seconds", + ), ) object.__setattr__( self, diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index 2a7690f..69d15db 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -544,17 +544,6 @@ def start(self) -> None: ) else: self.hooks.init_store(Path(self.config.db_path)) - self._connector_periodic_tick() - self._start_acp_supervisor() - from .store.sqlite import latest_snapshot - - self._snapshot = latest_snapshot( - Path(self.config.db_path), self.config.host_id - ) - if self._snapshot is None: - raise RuntimeError("ACP supervisor did not publish a lifecycle snapshot") - self._after_snapshot_saved() - api = TendwireDaemonAPI( get_snapshot=self.get_snapshot, get_health=self.get_health, @@ -574,12 +563,21 @@ def start(self) -> None: prepare_parent=self._prepare_socket_parent, periodic_callback=self._connector_periodic_tick, ) - self._server = server - # Bind before ingestion starts. Managed store connections and the - # socket publisher lock the same parent directory, so allowing an - # initial refresh first can make the daemon deadlock with itself. - # Requests are not served until start() returns successfully. + # Bind before ACP runtime/consumer threads can take store locks. + # No connections are accepted until serve_forever(), after + # this startup transaction has succeeded. server.start() + self._connector_periodic_tick() + self._start_acp_supervisor() + from .store.sqlite import latest_snapshot + + self._snapshot = latest_snapshot( + Path(self.config.db_path), self.config.host_id + ) + if self._snapshot is None: + raise RuntimeError("ACP supervisor did not publish a lifecycle snapshot") + self._after_snapshot_saved() + self._server = server except Exception: self.stop_event.set() @@ -959,6 +957,7 @@ def get_health(self) -> dict[str, Any]: "ready": acp_health["healthy"], "running": acp_health.get("state") == "running", "last_reconcile_at": acp_health.get("last_reconcile_at"), + "reconcile_enabled": True, } backend_maintenance = backend_runtime.get("automatic_maintenance") runtime_maintenance = ( @@ -1044,10 +1043,7 @@ def get_health(self) -> dict[str, Any]: "outcome": backend_runtime.get("outcome"), "ready": backend_runtime.get("ready"), "running": backend_runtime.get("running"), - "reconcile_enabled": backend_runtime.get( - "reconcile_enabled", - self.config.reconcile_interval_seconds > 0, - ), + "reconcile_enabled": backend_runtime["reconcile_enabled"], }, "acp": acp_health, "pending_ingestion": pending_ingestion, diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 7a94147..7cc8fc8 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -1958,6 +1958,15 @@ def close(self) -> None: coordinator.stop() +def test_coordinator_rejects_disabled_reconciliation(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="reconcile_interval must be finite and positive"): + AcpRuntimeCoordinator( + _config(tmp_path), + threading.Event(), + reconcile_interval=0, + ) + + def test_coordinator_forwards_explicit_permission_bridge(tmp_path: Path) -> None: config = _config(tmp_path) assert config.db_path is not None diff --git a/tests/test_config.py b/tests/test_config.py index 58673b8..f217944 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -203,7 +203,7 @@ def test_pr16_runtime_knobs_have_documented_defaults(monkeypatch) -> None: def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_RECONCILE_INTERVAL_SECONDS", "0") + monkeypatch.setenv("TENDWIRE_RECONCILE_INTERVAL_SECONDS", "2") monkeypatch.setenv("TENDWIRE_EVENT_RETENTION_DAYS", "14") monkeypatch.setenv("TENDWIRE_MAX_WORKERS", "64") monkeypatch.setenv("TENDWIRE_MAX_OUTBOX_ATTEMPTS", "3") @@ -227,7 +227,7 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: command_receipt_retention_seconds="691200", command_receipt_retention_count="12", ) - assert env_config.reconcile_interval_seconds == 0 + assert env_config.reconcile_interval_seconds == 2 assert env_config.event_retention_days == 14 assert env_config.max_workers == 64 assert env_config.max_outbox_attempts == 3 @@ -252,7 +252,16 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: @pytest.mark.parametrize( ("field", "value", "message"), [ - ("reconcile_interval_seconds", -1, "reconcile_interval_seconds must be non-negative"), + ( + "reconcile_interval_seconds", + 0, + "reconcile_interval_seconds must be a finite positive number", + ), + ( + "reconcile_interval_seconds", + -1, + "reconcile_interval_seconds must be a finite positive number", + ), ("event_retention_days", 0, "event_retention_days must be >= 1"), ("max_workers", 0, "max_workers must be >= 1"), ("max_outbox_attempts", 0, "max_outbox_attempts must be >= 1"), @@ -270,6 +279,15 @@ def test_pr16_runtime_knobs_reject_invalid_values(field: str, value: object, mes Config(**{field: value}) +def test_reconcile_interval_rejects_disabled_environment(monkeypatch) -> None: + monkeypatch.setenv("TENDWIRE_RECONCILE_INTERVAL_SECONDS", "0") + with pytest.raises( + ValueError, + match="reconcile_interval_seconds must be a finite positive number", + ): + load_config() + + ACKNOWLEDGED_FINAL_RETENTION_ENV_NAMES = ( "TENDWIRE_ACKNOWLEDGED_FINAL_RETENTION_DAYS", "TENDWIRE_ACKNOWLEDGED_FINAL_RETENTION_COUNT", diff --git a/tests/test_daemon.py b/tests/test_daemon.py index 77b9578..348d9f2 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -1345,7 +1345,7 @@ def test_daemon_health_exposes_public_operational_status_without_private_values( config = Config( host_id="health-host", db_path=db_path, - reconcile_interval_seconds=0, + reconcile_interval_seconds=2, event_retention_days=3, max_workers=8, max_outbox_attempts=4, @@ -1454,7 +1454,7 @@ def test_daemon_health_exposes_public_operational_status_without_private_values( "backlog": False, } assert health["limits"] == { - "reconcile_interval_seconds": 0, + "reconcile_interval_seconds": 2, "event_retention_days": 3, "max_workers": 8, "max_outbox_attempts": 4, @@ -1476,6 +1476,7 @@ def test_daemon_health_exposes_public_operational_status_without_private_values( "counts": {"fresh": 0, "stale": 0, "total": 0}, "bounds": {"stale_grace_seconds": 31.0}, } + assert health["backend"]["reconcile_enabled"] is True assert "health.db" not in encoded assert str(tmp_path) not in encoded assert "sentinel-private" not in encoded diff --git a/tests/test_daemon_acp.py b/tests/test_daemon_acp.py index 97aef58..89b1a4e 100644 --- a/tests/test_daemon_acp.py +++ b/tests/test_daemon_acp.py @@ -12,6 +12,7 @@ from tendwire.config import Config from tendwire.core.models import BackendHealth, Snapshot from tendwire.daemon import DaemonHooks, TendwireDaemon +from tendwire.daemon_api import UnixSocketJSONServer from tendwire.store.sqlite import init_store, save_snapshot @@ -96,25 +97,39 @@ def initialize(path: Path) -> None: ) -def test_daemon_requires_an_acp_supervisor_before_binding_socket(tmp_path: Path) -> None: +def test_daemon_binds_then_removes_socket_when_acp_supervisor_is_missing( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: config = _config(tmp_path) calls: list[str] = [] + original_start = UnixSocketJSONServer.start + + def record_bind(server: Any) -> None: + calls.append("socket_bind") + original_start(server) + + monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", record_bind) daemon = TendwireDaemon(config, hooks=_hooks(config, None, calls)) with pytest.raises(RuntimeError, match="ACP supervisor is required"): daemon.start() assert not config.socket_path.exists() - assert calls == ["init_store"] + assert calls == ["init_store", "socket_bind"] def test_daemon_starts_required_acp_and_exposes_only_public_health( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", lambda _self: None) config = _config(tmp_path) calls: list[str] = [] + + def record_bind(_server: Any) -> None: + calls.append("socket_bind") + + monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", record_bind) supervisor = _Supervisor(calls) daemon = TendwireDaemon( config, @@ -130,7 +145,7 @@ def test_daemon_starts_required_acp_and_exposes_only_public_health( assert health["acp"]["state"] == "running" assert health["acp"]["counters"]["updates_ingested"] == 7 assert "sentinel-private" not in json.dumps(health) - assert calls[:2] == ["init_store", "acp_start"] + assert calls[:3] == ["init_store", "socket_bind", "acp_start"] finally: daemon.stop() diff --git a/tests/test_herdr_socket.py b/tests/test_herdr_socket.py index ab9c88d..d25dd06 100644 --- a/tests/test_herdr_socket.py +++ b/tests/test_herdr_socket.py @@ -38,28 +38,51 @@ def run() -> None: return thread -def test_lifecycle_and_acp_methods_use_frozen_socket_shapes(tmp_path) -> None: +def test_discovery_lists_use_one_request_per_connection(tmp_path) -> None: path = tmp_path / "herdr.sock" requests: list[dict] = [] thread = _serve( path, [ + {"result": {"workspaces": []}}, {"result": {"panes": []}}, + {"result": {"agents": []}}, + ], + requests, + ) + client = HerdrSocketClient(str(path), timeout=1) + assert client.workspace_list() == {"workspaces": []} + assert client._socket is None + assert client.pane_list() == {"panes": []} + assert client._socket is None + assert client.agent_list() == {"agents": []} + assert client._socket is None + thread.join(2) + assert [item["method"] for item in requests] == [ + "workspace.list", + "pane.list", + "agent.list", + ] + + +def test_acp_methods_use_frozen_socket_shapes_without_connection_reuse(tmp_path) -> None: + path = tmp_path / "herdr.sock" + requests: list[dict] = [] + thread = _serve( + path, + [ {"result": {"type": "agent_acp_status"}}, {"result": {"type": "agent_acp_endpoint"}}, ], requests, ) client = HerdrSocketClient(str(path), timeout=1) - assert client.pane_list() == {"panes": []} - client.close() assert client.agent_acp_status("term") == {"type": "agent_acp_status"} - client.close() + assert client._socket is None assert client.agent_acp_endpoint("term") == {"type": "agent_acp_endpoint"} - client.close() + assert client._socket is None thread.join(2) assert [(item["method"], item["params"]) for item in requests] == [ - ("pane.list", {}), ("agent.acp_status", {"target": "term"}), ("agent.acp_endpoint", {"target": "term"}), ] From 261ed8a90e5bb6211754119ceadea3a09222940b Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 22:27:50 +0800 Subject: [PATCH 76/83] store: replace migrations with explicit current schema --- src/tendwire/backends/acp_ingestion.py | 6 +- src/tendwire/backends/acp_permissions.py | 2 - src/tendwire/cli.py | 4 +- src/tendwire/command_submission.py | 41 +- src/tendwire/config.py | 1 - src/tendwire/connectors/outbox.py | 5 +- src/tendwire/core/commands.py | 87 +- src/tendwire/daemon.py | 27 +- src/tendwire/store/sqlite.py | 5905 ++---------------- tests/store_helpers.py | 118 + tests/test_acp_atomic_ingestion.py | 81 - tests/test_acp_ingestion.py | 6 +- tests/test_acp_permissions.py | 25 - tests/test_agent_events.py | 697 +-- tests/test_commands.py | 186 +- tests/test_config.py | 2 - tests/test_connector_outbox.py | 395 -- tests/test_daemon.py | 71 - tests/test_delivery_retention.py | 12 +- tests/test_delivery_retention_hardening.py | 12 +- tests/test_delivery_retention_migration.py | 1792 ------ tests/test_delivery_retention_projection.py | 30 +- tests/test_delivery_retention_recovery.py | 375 +- tests/test_public_content_safety.py | 53 +- tests/test_release_readiness.py | 31 - tests/test_snapshot_sanitize_performance.py | 49 +- tests/test_store.py | 6065 ++++--------------- tests/test_turn_delta.py | 186 +- tests/test_turn_submissions.py | 1053 +--- tests/test_worker_label_and_model.py | 6 +- 30 files changed, 1803 insertions(+), 15520 deletions(-) create mode 100644 tests/store_helpers.py delete mode 100644 tests/test_delivery_retention_migration.py diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py index a8ff07a..d398271 100644 --- a/src/tendwire/backends/acp_ingestion.py +++ b/src/tendwire/backends/acp_ingestion.py @@ -14,7 +14,7 @@ from pathlib import Path from typing import Any -from ..config import DEFAULT_TURN_MODEL, Config +from ..config import Config from ..core.agent_events import AgentEvent, agent_event from ..core.models import WorkerBinding, stable_fingerprint from ..store.sqlite import ( @@ -416,8 +416,6 @@ def mark_prompt_complete( marker, expected_binding=self.binding, content=content, - observed_at=marker.observed_at, - turn_model=DEFAULT_TURN_MODEL, ) except BaseException: self._restore_speculation(checkpoint, prior_turn_state) @@ -508,8 +506,6 @@ def _accept( event, expected_binding=self.binding, content=projection, - observed_at=event.observed_at, - turn_model=DEFAULT_TURN_MODEL, ) except BaseException: self._restore_speculation(checkpoint, prior_turn_state) diff --git a/src/tendwire/backends/acp_permissions.py b/src/tendwire/backends/acp_permissions.py index 568eb91..91c58ad 100644 --- a/src/tendwire/backends/acp_permissions.py +++ b/src/tendwire/backends/acp_permissions.py @@ -148,7 +148,6 @@ def __call__(self, request: PermissionRequest) -> PermissionSelection | None: binding_private_fingerprint=binding.private_fingerprint, observed_turn_target_value=binding.turn_target_value, binding_authoritative=True, - route_kind="acp_permission", ) if not changed: raise AcpPermissionBrokerError("permission overlay was not published") @@ -256,7 +255,6 @@ def _clear_offer(self, offer: _Offer) -> None: PendingObservation("read_succeeded_no_prompt"), binding_private_fingerprint=offer.binding.private_fingerprint, observed_turn_target_value=offer.binding.turn_target_value, - route_kind="acp_permission", ) except Exception: pass diff --git a/src/tendwire/cli.py b/src/tendwire/cli.py index 80b2940..95dc522 100644 --- a/src/tendwire/cli.py +++ b/src/tendwire/cli.py @@ -962,7 +962,7 @@ def command_envelope_from_payload(config: Config, payload: str) -> CommandEnvelo if validation_error is not None: return CommandEnvelope.from_error(request, validation_error) - if request.action in {"send_instruction", "answer_pending", "answer_decision"}: + if request.action in {"send_instruction", "answer_decision"}: from .command_submission import submit_command return submit_command(config, payload) @@ -1038,7 +1038,7 @@ def cmd_command( and validation_error is None and parsed_request is not None and parsed_request.action - in {"send_instruction", "answer_pending", "answer_decision"} + in {"send_instruction", "answer_decision"} and parsed_request.dry_run ) daemon_eligible = ( diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index 36a17c3..870b20f 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -73,7 +73,7 @@ HERDR_BACKEND = "herdr" _MUTATING_ACTIONS = frozenset( - {"send_instruction", "answer_pending", "answer_decision"} + {"send_instruction", "answer_decision"} ) _DISALLOWED_SEND_STATUSES = frozenset({"closed", "failed", "unknown"}) _AMBIGUOUS_BINDING_REASONS = frozenset({"duplicate_backend_target", "not_unique"}) @@ -98,10 +98,9 @@ def prompt( on_send_start: Callable[[], None] | None = None, ) -> object: ... - # Production routes may expose a context manager that fences their exact - # generation from the final authority check through the prompt-frame - # acknowledgement. Test and third-party routes remain compatible without - # it; submit_acp_command probes this method dynamically. + # Routes may expose a context manager that fences their exact generation + # from the final authority check through the prompt-frame acknowledgement. + # Routes without it retain the final authority checks around submission. def prepare(self) -> Any: ... @property @@ -363,7 +362,6 @@ def _decision_claim_has_exact_route(claim: Any) -> bool: and not isinstance(claim.option_count, bool) and claim.option_count >= 1 and isinstance(getattr(claim, "option_refs", None), tuple) - and getattr(claim, "route_kind", None) in {"legacy", "acp_permission"} and ( (claim.text is None and bool(claim.option_refs)) or ( @@ -389,7 +387,6 @@ def _same_decision_route(left: Any, right: Any) -> bool: left.option_count, left.option_refs, left.text, - left.route_kind, ) == ( right.worker_id, @@ -401,16 +398,10 @@ def _same_decision_route(left: Any, right: Any) -> bool: right.option_count, right.option_refs, right.text, - right.route_kind, ) ) -def _decision_uses_acp_binding(_config: Config, decision: Any) -> bool: - """Classify from durable provenance, independent of current liveness.""" - return getattr(decision, "route_kind", "legacy") == "acp_permission" - - class PreSendCertainty(Enum): """How a pre-send failure must be classified before any external mutation. @@ -639,9 +630,8 @@ def _envelope_from_receipt( "state", "status", "result_json", - "legacy_collision", } - if not required.issubset(receipt) or receipt.get("legacy_collision") is not False: + if not required.issubset(receipt): return _backend_uncertain( request, "stored request receipt is malformed; not retrying mutation", @@ -705,7 +695,6 @@ def _reserve_canonical_request( public_worker_id=canonical.public_worker_id, pending_result_json=envelope_to_receipt_json(pending), selector_proof=_selector_proof(request), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), ) except Exception: # noqa: BLE001 try: @@ -902,7 +891,6 @@ def _reserve_terminal_replay( # proven equivalence with a different one, and overwriting it would # strand a later retry of the request as it was actually issued. selector_proof=_stored_selector_proof(previous_receipt), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), event_payload=_transition_payload( request, worker_id=canonical.public_worker_id, @@ -1569,8 +1557,6 @@ def _receipt_authority( stored reservation was abandoned before any send and the normal path may re-drive it. """ - if receipt.get("legacy_collision") is not False: - return _receipt_malformed(request) if receipt.get("action") != request.action: return _duplicate_request(request) @@ -1723,8 +1709,6 @@ def replay_command_receipt( return None if not isinstance(receipt, Mapping): return None - if receipt.get("legacy_collision") is not False: - return _receipt_malformed(request) if receipt.get("action") != request.action: return _duplicate_request(request) proven = _proven_replay_worker_id( @@ -2053,11 +2037,6 @@ def _submit_command_v2( return _execute_non_mutating(config, request) if request.dry_run: return _mutation_dry_run(request) - if request.action == "answer_pending": - return _backend_unavailable( - request, - "legacy pane choices are unavailable; use an ACP permission decision", - ) if request.action == "send_instruction": # Valid live instructions are consumed by submit_acp_command before this # shared parser path. Never expose a second command transport. @@ -2117,16 +2096,6 @@ def _submit_command_v2( return _answer_in_progress(request, receipt_reserved=True) if answer_pre_send is None: - if not _decision_uses_acp_binding(config, validated): - unavailable = _backend_unavailable( - request, - "legacy permission decisions are unavailable; ACP authority is required", - ) - return ( - _request_in_progress(request) - if takeover is not None - else unavailable - ) try: owns_decision = bool( acp_permission_router is not None diff --git a/src/tendwire/config.py b/src/tendwire/config.py index ce17f8b..60a4e68 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -15,7 +15,6 @@ ACP_THOUGHT_POLICIES = frozenset({"disabled", "private_summary", "private_all"}) ACP_CONSOLE_INPUT_POLICIES = frozenset({"preserve", "live_only"}) -DEFAULT_TURN_MODEL = "observed" DEFAULT_ACP_THOUGHT_POLICY = "disabled" DEFAULT_ACP_CONSOLE_INPUT_POLICY = "preserve" DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS = 30.0 diff --git a/src/tendwire/connectors/outbox.py b/src/tendwire/connectors/outbox.py index 8d332e2..dd3866d 100644 --- a/src/tendwire/connectors/outbox.py +++ b/src/tendwire/connectors/outbox.py @@ -10,7 +10,7 @@ from pathlib import Path from typing import Any -from ..config import DEFAULT_CONNECTOR_ACK_TTL_SECONDS, DEFAULT_TURN_MODEL +from ..config import DEFAULT_CONNECTOR_ACK_TTL_SECONDS from ..core.models import sanitize_public_mapping, sanitize_public_value from ..store.sqlite import ( ack_connector_delivery, @@ -217,7 +217,6 @@ def __init__( max_lease_seconds: int = 300, ack_ttl_seconds: int = DEFAULT_CONNECTOR_ACK_TTL_SECONDS, max_attempts: int = 10, - turn_model: str = DEFAULT_TURN_MODEL, ) -> None: self.db_path = Path(db_path) if db_path is not None else None self.host_id = str(host_id) @@ -225,7 +224,6 @@ def __init__( self.max_lease_seconds = max(1, int(max_lease_seconds)) self.ack_ttl_seconds = max(1, int(ack_ttl_seconds)) self.max_attempts = max(1, int(max_attempts)) - self.turn_model = str(turn_model or DEFAULT_TURN_MODEL).strip().lower() def _require_store(self, name: str = "") -> dict[str, Any] | None: if self.db_path is None: @@ -293,7 +291,6 @@ def prepare(self, params: Mapping[str, Any] | None = None) -> dict[str, Any]: presentation_version=version, part_count=part_count, source_ref=source_ref, - turn_model=self.turn_model, ) if action == "recover": diff --git a/src/tendwire/core/commands.py b/src/tendwire/core/commands.py index ff145f0..a37e1db 100644 --- a/src/tendwire/core/commands.py +++ b/src/tendwire/core/commands.py @@ -43,7 +43,6 @@ "read_snapshot", "resolve_target", "send_instruction", - "answer_pending", "answer_decision", } ) @@ -210,9 +209,6 @@ {"worker_id", "space_id", "name", "stable_key"} ) INSTRUCTION_ALLOWED_FIELDS = frozenset({"text"}) -ANSWER_PENDING_PARAM_FIELDS = frozenset( - {"pending_id", "pending_fingerprint", "choice_id"} -) ANSWER_DECISION_PARAM_FIELDS = frozenset({"decision_ref", "selection"}) # Connector, low-level terminal, routing, and private fields rejected anywhere in a request. @@ -388,8 +384,8 @@ def instruction_fingerprint(text: Any) -> str: Valid instruction text can consist entirely of whitespace even though its normalized form is empty. Preserve the normalized matching behavior for - ordinary text, but fingerprint the raw text in that edge case so shadow - ledger bookkeeping can never reject an otherwise valid legacy send. + ordinary text, but fingerprint the raw text in that edge case so submission + bookkeeping can never reject an otherwise valid send. """ normalized = normalize_instruction_text(text) fingerprint_text = normalized or str(text or "") @@ -538,16 +534,6 @@ def to_dict(self) -> dict[str, Any]: payload["response_schema_version"] = self.response_schema_version return payload - def payload_fingerprint(self) -> str: - """Return the legacy raw-request fingerprint used by compatibility callers. - - This includes request identity and unresolved selector spelling, so it is - not authoritative for mutating-command idempotency. New mutation - persistence must use :func:`build_canonical_mutation` after resolving the - public worker identity. - """ - return stable_fingerprint(self.to_dict()) - @classmethod def from_dict(cls, data: dict[str, Any]) -> "CommandRequest": return cls( @@ -592,10 +578,8 @@ def build_canonical_mutation( """ if not isinstance(request, CommandRequest): raise TypeError("request must be a CommandRequest") - if request.action not in {"send_instruction", "answer_pending", "answer_decision"}: - raise ValueError( - "canonical mutations require send_instruction, answer_pending, or answer_decision" - ) + if request.action not in {"send_instruction", "answer_decision"}: + raise ValueError("canonical mutations require send_instruction or answer_decision") if request.dry_run is not False: raise ValueError("canonical mutations require a non-dry-run request") request_error = validate_request(request) @@ -613,19 +597,6 @@ def build_canonical_mutation( "instruction": {"text": request.instruction["text"]}, "options": {}, } - elif request.action == "answer_pending": - assert request.params is not None - canonical_payload = { - "canonical_version": CANONICAL_MUTATION_VERSION, - "action": "answer_pending", - "target": {"worker_id": public_worker_id}, - "pending": { - "pending_id": request.params["pending_id"], - "pending_fingerprint": request.params["pending_fingerprint"], - "choice_id": request.params["choice_id"], - }, - "options": {}, - } else: assert request.params is not None selection = request.params["selection"] @@ -694,10 +665,8 @@ def build_selector_proof(request: CommandRequest) -> str: """ if not isinstance(request, CommandRequest): raise TypeError("request must be a CommandRequest") - if request.action not in {"send_instruction", "answer_pending", "answer_decision"}: - raise ValueError( - "selector proofs require send_instruction, answer_pending, or answer_decision" - ) + if request.action not in {"send_instruction", "answer_decision"}: + raise ValueError("selector proofs require send_instruction or answer_decision") if request.dry_run is not False: raise ValueError("selector proofs require a non-dry-run request") request_error = validate_request(request) @@ -780,7 +749,7 @@ def validate_request(request: CommandRequest) -> dict[str, Any] | None: details={"field": "action", "allowed": sorted(ALLOWED_ACTIONS)}, ) if ( - request.action in {"send_instruction", "answer_pending", "answer_decision"} + request.action in {"send_instruction", "answer_decision"} and request.dry_run is False and not is_valid_request_id(request.request_id) ): @@ -829,46 +798,6 @@ def validate_request(request: CommandRequest) -> dict[str, Any] | None: details={"field": "instruction.text"}, ) - if request.action == "answer_pending": - if request.target is not None: - return error_value( - STATUS_INVALID_REQUEST, - "answer_pending does not accept a target", - details={"field": "target"}, - ) - if request.instruction is not None: - return error_value( - STATUS_INVALID_REQUEST, - "answer_pending does not accept an instruction", - details={"field": "instruction"}, - ) - if not isinstance(request.params, dict): - return error_value( - STATUS_INVALID_REQUEST, - "answer_pending requires params", - details={"field": "params"}, - ) - actual_fields = set(request.params) - if actual_fields != ANSWER_PENDING_PARAM_FIELDS: - return error_value( - STATUS_INVALID_REQUEST, - "answer_pending params must contain exactly pending_id, pending_fingerprint, and choice_id", - details={ - "field": "params", - "required": sorted(ANSWER_PENDING_PARAM_FIELDS), - "missing": sorted(ANSWER_PENDING_PARAM_FIELDS - actual_fields), - "disallowed": sorted(actual_fields - ANSWER_PENDING_PARAM_FIELDS), - }, - ) - for field in sorted(ANSWER_PENDING_PARAM_FIELDS): - value = request.params.get(field) - if not isinstance(value, str) or not value.strip(): - return error_value( - STATUS_INVALID_REQUEST, - f"answer_pending requires nonblank params.{field}", - details={"field": f"params.{field}"}, - ) - if request.action == "answer_decision": if request.target is None or set(request.target) != {"worker_id"}: return error_value( @@ -1065,7 +994,6 @@ def __post_init__(self) -> None: mutating = self.action in { "send_instruction", - "answer_pending", "answer_decision", } live_mutation = mutating and self.dry_run is False @@ -1269,7 +1197,6 @@ def from_error(cls, request: CommandRequest | None, error: dict[str, Any]) -> "C ) mutating = request.action in { "send_instruction", - "answer_pending", "answer_decision", } valid_mutation_id = is_valid_request_id(request.request_id) diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index 69d15db..4a67e2a 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -11,7 +11,7 @@ from pathlib import Path from typing import Any -from .config import DEFAULT_TURN_MODEL, Config +from .config import Config from .core.commands import CommandEnvelope from .core.models import Snapshot, sanitize_public_mapping, utc_timestamp from .daemon_api import ( @@ -449,19 +449,10 @@ def default_socket_path(config: Config) -> Path: return Path(config.data_dir) / "tendwire.sock" -def _default_init_store( - db_path: Path, - *, - connector_ack_ttl_seconds: int | None = None, -) -> None: +def _default_init_store(db_path: Path) -> None: from .store.sqlite import init_store - kwargs = ( - {"connector_ack_ttl_seconds": connector_ack_ttl_seconds} - if connector_ack_ttl_seconds is not None - else {} - ) - init_store(db_path, **kwargs) + init_store(db_path) def _default_acp_supervisor_factory(config: Config, stop_event: threading.Event) -> Any: @@ -536,12 +527,7 @@ def start(self) -> None: socket_group=self.config.socket_group, ) if self.hooks.init_store is _default_init_store: - _default_init_store( - Path(self.config.db_path), - connector_ack_ttl_seconds=( - self.config.connector_ack_ttl_seconds - ), - ) + _default_init_store(Path(self.config.db_path)) else: self.hooks.init_store(Path(self.config.db_path)) api = TendwireDaemonAPI( @@ -763,7 +749,6 @@ def _after_snapshot_saved(self) -> None: policy=policy, agent_event_host_id=self.config.host_id, agent_event_retention_days=self.config.event_retention_days, - turn_model=DEFAULT_TURN_MODEL, acknowledged_final_retention_days=( self.config.acknowledged_final_retention_days ), @@ -1129,7 +1114,6 @@ def get_turns( limit=limit, cursor=cursor, since=since, - turn_model=DEFAULT_TURN_MODEL, ) def get_turn_content(self, params: Mapping[str, Any]) -> Mapping[str, Any]: @@ -1154,7 +1138,6 @@ def get_turn_content(self, params: Mapping[str, Any]) -> Mapping[str, Any]: field=params.get("field"), cursor=params.get("cursor"), schema_version=params.get("schema_version", 1), - turn_model=DEFAULT_TURN_MODEL, ) def get_turn_delta( @@ -1181,7 +1164,6 @@ def get_turn_delta( watermark=watermark, cursor=cursor, limit=limit, - turn_model=DEFAULT_TURN_MODEL, ) def connector_call(self, method: str, params: Mapping[str, Any]) -> Mapping[str, Any]: @@ -1206,7 +1188,6 @@ def connector_call(self, method: str, params: Mapping[str, Any]) -> Mapping[str, max_lease_seconds=self.config.connector_max_claim_ttl_seconds, ack_ttl_seconds=self.config.connector_ack_ttl_seconds, max_attempts=self.config.max_outbox_attempts, - turn_model=DEFAULT_TURN_MODEL, ).dispatch(method, params) def _connector_periodic_tick(self) -> None: diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index 76a07e3..9ed3588 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -31,7 +31,6 @@ DEFAULT_PENDING_STALE_GRACE_SECONDS, DEFAULT_SUBMISSION_HARD_TTL_SECONDS, DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS, - DEFAULT_TURN_MODEL, ) from ..local_state import ( EntryIdentity, @@ -96,7 +95,6 @@ utc_timestamp, ) from ..core.turns import ( - InteractionChoice, PendingInteraction, PendingObservation, PendingObservedChoice, @@ -145,19 +143,15 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS STORE_SCHEMA_VERSION = 28 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS -_LEGACY_TURN_CLAIM_HARD_TTL_SECONDS = 86_400 TURN_CHANGE_RETENTION_DAYS = 7 TURN_CHANGE_RETENTION_COUNT = 100_000 TURN_CHANGE_COMPACTION_BATCH_SIZE = 1_000 -HERDR_TURN_RETENTION_DAYS = 30 -HERDR_TURN_RETENTION_COUNT = 4096 -HERDR_TURN_RETENTION_BATCH_SIZE = 100 TURN_SUBMISSION_OBSERVATION_ADOPTION_WINDOW_SECONDS = 60.0 SUBMISSION_LINK_WINDOW_SECONDS = DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS SUBMISSION_HARD_TTL_SECONDS = DEFAULT_SUBMISSION_HARD_TTL_SECONDS # A send that has not produced an authoritative accepted/uncertain receipt # within the backend command timeout is no longer safe for instant linking. -# Keep this classification local to the shadow linker: receipts and the +# Keep this classification local to the observation linker: receipts and the # submission lifecycle remain authoritative and unchanged. SUBMISSION_SEND_ACK_TIMEOUT_SECONDS = 5.0 TURN_LEDGER_BACKFILL_BATCH_SIZE = 500 @@ -204,7 +198,6 @@ _LOGGER = logging.getLogger(__name__) _SUBMISSION_LINK_SWEEP_LAST_AT: dict[tuple[str, str, str], float] = {} _SUBMISSION_LINK_SWEEP_LOCK = threading.Lock() -TURN_MODELS = frozenset({"legacy", "dual", "shadow", "observed"}) DEFAULT_SUBMISSION_LINK_SWEEP_INTERVAL_SECONDS = 2.0 _SUBMISSION_LINK_BACKOFF: dict[ tuple[str, str, str, str], datetime | None @@ -220,14 +213,6 @@ _SUBMISSION_LINK_EMPTY_COMPONENT_RECHECK_SECONDS = 30.0 -def _submission_linking_enabled(turn_model: str) -> bool: - normalized = str(turn_model or "").strip().lower() - if normalized not in TURN_MODELS: - allowed = ", ".join(sorted(TURN_MODELS)) - raise ValueError(f"turn_model must be one of: {allowed}") - return True - - def _submission_link_backoff_key( db_path: Path | str, host_id: str, @@ -353,16 +338,6 @@ def is_valid_turn_submission_state_transition( ) -@dataclass(frozen=True) -class Migration: - """One exact, transaction-external schema transition.""" - - from_version: int - to_version: int - apply: Callable[[sqlite3.Connection], None] - - - @dataclass(frozen=True) class BackendPendingChoiceClaim: status: Literal[ @@ -421,7 +396,6 @@ class BackendPendingDecisionClaim: option_count: int | None = None option_refs: tuple[str, ...] = () text: str | None = None - route_kind: Literal["legacy", "acp_permission"] = "legacy" @dataclass(frozen=True) @@ -443,7 +417,6 @@ class BackendPendingDecisionSend: option_count: int | None = None option_refs: tuple[str, ...] = () text: str | None = None - route_kind: Literal["legacy", "acp_permission"] = "legacy" @dataclass(frozen=True) @@ -509,41 +482,9 @@ class AppendProjectedAgentEventResult: event: AppendBoundAgentEventResult turn: TurnRefreshApplyResult | None = None - -@dataclass(frozen=True) -class HerdrTurnWatermark: - """Durable replay position and retained completeness-break evidence.""" - - host_id: str - pane_id: str - turn_epoch: int - last_turn: int - completeness_break_count: int - last_completeness_break_reason: str | None - last_completeness_break_at: str | None - updated_at: str - - -@dataclass(frozen=True) -class HerdrTurnRefreshRetry: - """Durable local retry state for one Herdr completion refresh.""" - - host_id: str - pane_id: str - turn_epoch: int - turn: int - status: Literal["pending", "escalated"] - refresh_status: str - first_seen_at: str - last_attempt_at: str - next_attempt_at: str | None - attempt_count: int - escalated_at: str | None - - @dataclass(frozen=True) class _TurnContentMergeResult: - """Observation merge outcome and optional shadow-link settlement key.""" + """Observation merge outcome and optional submission settlement key.""" updated: int submission_link: tuple[str, str] | None = None @@ -621,14 +562,6 @@ def _record_response_size( ); """ -CREATE_LEGACY_SNAPSHOT_INDEXES = ( - "CREATE INDEX IF NOT EXISTS idx_snapshots_host_id ON snapshots(host_id)", - "CREATE INDEX IF NOT EXISTS idx_snapshots_created_at ON snapshots(created_at)", - ( - "CREATE INDEX IF NOT EXISTS idx_snapshots_content_fingerprint " - "ON snapshots(content_fingerprint)" - ), -) CREATE_SNAPSHOT_INDEXES = ( ( @@ -643,9 +576,6 @@ def _record_response_size( # Leading "~" sorts after every canonical timestamp under SQLite BINARY # collation, so raw indexed age comparisons never delete unknown history. _SNAPSHOT_CREATED_AT_QUARANTINE = "~invalid-snapshot-created-at" -_LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE = ( - "9999-12-31T23:59:59.999999+00:00" -) CREATE_STORE_MAINTENANCE_STATE_TABLE = """ CREATE TABLE IF NOT EXISTS store_maintenance_state ( @@ -675,20 +605,6 @@ def _record_response_size( ON CONFLICT(scope) DO NOTHING """ -CREATE_LEGACY_COMMAND_RECEIPTS_TABLE = """ -CREATE TABLE IF NOT EXISTS command_receipts ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - request_id TEXT NOT NULL, - action TEXT NOT NULL, - payload_fingerprint TEXT NOT NULL, - status TEXT NOT NULL, - result_json TEXT NOT NULL, - created_at TEXT NOT NULL, - completed_at TEXT, - uncertain INTEGER NOT NULL DEFAULT 0 -); -""" CREATE_COMMAND_RECEIPTS_TABLE = """ CREATE TABLE IF NOT EXISTS command_receipts ( @@ -713,13 +629,8 @@ def _record_response_size( send_started_at TEXT, terminal_at TEXT, updated_at TEXT NOT NULL, - legacy_collision INTEGER NOT NULL DEFAULT 0 CHECK (legacy_collision IN (0, 1)), - legacy_collision_count INTEGER NOT NULL DEFAULT 0 CHECK ( - legacy_collision_count >= 0 - ), -- Private evidence of the immutable selector this request was spelled with. - -- Empty means legacy evidence that cannot prove an alias retry. Declared - -- last so a v12 ALTER and a fresh CREATE agree on column order. + -- Empty is malformed fail-safe evidence that cannot prove an alias retry. selector_proof TEXT NOT NULL DEFAULT '', CHECK ( ( @@ -743,22 +654,10 @@ def _record_response_size( CHECK ( state != 'rejected' OR status NOT IN ('pending', 'accepted', 'request_state_uncertain') - ), - CHECK ( - legacy_collision = 0 - OR (state = 'uncertain' AND legacy_collision_count >= 2) ) ); """ -CREATE_LEGACY_COMMAND_RECEIPT_INDEXES = ( - "CREATE INDEX IF NOT EXISTS idx_command_receipts_host_request_action " - "ON command_receipts(host_id, request_id, action)", -) -CREATE_LEGACY_COMMAND_RECEIPT_UNIQUE_INDEX = ( - "CREATE UNIQUE INDEX IF NOT EXISTS ux_command_receipts_host_request_action " - "ON command_receipts(host_id, request_id, action)" -) CREATE_COMMAND_RECEIPT_INDEXES = ( "CREATE UNIQUE INDEX IF NOT EXISTS ux_command_receipts_host_request " "ON command_receipts(host_id, request_id)", @@ -1374,24 +1273,6 @@ def _record_response_size( ), ) -CREATE_LEGACY_COMMANDS_TABLE = """ -CREATE TABLE IF NOT EXISTS commands ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - request_id TEXT NOT NULL, - action TEXT NOT NULL, - payload_fingerprint TEXT NOT NULL, - status TEXT NOT NULL, - dry_run INTEGER NOT NULL DEFAULT 0, - uncertain INTEGER NOT NULL DEFAULT 0, - request_json TEXT NOT NULL DEFAULT '{}', - result_json TEXT NOT NULL DEFAULT '{}', - created_at TEXT NOT NULL, - reserved_at TEXT, - completed_at TEXT, - updated_at TEXT NOT NULL -); -""" CREATE_COMMANDS_TABLE = """ CREATE TABLE IF NOT EXISTS commands ( @@ -1413,10 +1294,6 @@ def _record_response_size( send_started_at TEXT, terminal_at TEXT, updated_at TEXT NOT NULL, - legacy_collision INTEGER NOT NULL DEFAULT 0 CHECK (legacy_collision IN (0, 1)), - legacy_collision_count INTEGER NOT NULL DEFAULT 0 CHECK ( - legacy_collision_count >= 0 - ), CHECK ( (state IN ('reserved', 'send_started') AND terminal_at IS NULL) OR (state IN ('accepted', 'rejected', 'uncertain') AND terminal_at IS NOT NULL) @@ -1430,10 +1307,6 @@ def _record_response_size( CHECK ( state != 'rejected' OR status NOT IN ('pending', 'accepted', 'request_state_uncertain') - ), - CHECK ( - legacy_collision = 0 - OR (state = 'uncertain' AND legacy_collision_count >= 2) ) ); """ @@ -1488,12 +1361,22 @@ def _record_response_size( """ -CREATE_LEGACY_BACKEND_PENDING_TABLE = """ +CREATE_BACKEND_PENDING_TABLE = """ CREATE TABLE IF NOT EXISTS backend_pending ( host_id TEXT NOT NULL, worker_id TEXT NOT NULL, payload_json TEXT NOT NULL, observed_at TEXT NOT NULL, + revision_digest TEXT NOT NULL DEFAULT '', + choice_routes_json TEXT NOT NULL DEFAULT '{}', + binding_private_fingerprint TEXT NOT NULL DEFAULT '', + observed_turn_target_value TEXT NOT NULL DEFAULT '', + observation_state TEXT NOT NULL DEFAULT 'open', + freshness TEXT NOT NULL DEFAULT 'fresh', + last_success_at TEXT, + last_failure_at TEXT, + grace_deadline TEXT, + updated_at TEXT NOT NULL DEFAULT '', PRIMARY KEY (host_id, worker_id) ); """ @@ -1517,83 +1400,10 @@ def _record_response_size( ); """ -CREATE_HERDR_TURN_WATERMARKS_TABLE = """ -CREATE TABLE IF NOT EXISTS herdr_turn_watermarks ( - host_id TEXT NOT NULL, - pane_id TEXT NOT NULL, - turn_epoch INTEGER NOT NULL CHECK (turn_epoch >= 0), - last_turn INTEGER NOT NULL CHECK (last_turn >= 0), - completeness_break_count INTEGER NOT NULL DEFAULT 0 - CHECK (completeness_break_count >= 0), - last_completeness_break_reason TEXT, - last_completeness_break_at TEXT, - updated_at TEXT NOT NULL, - PRIMARY KEY (host_id, pane_id) -); -""" -CREATE_HERDR_TURN_COMPLETIONS_TABLE = """ -CREATE TABLE IF NOT EXISTS herdr_turn_completions ( - host_id TEXT NOT NULL, - pane_id TEXT NOT NULL, - turn_epoch INTEGER NOT NULL CHECK (turn_epoch >= 0), - turn INTEGER NOT NULL CHECK (turn >= 0), - outcome TEXT NOT NULL CHECK (outcome IN ('completed', 'aborted')), - completed_unix_ms INTEGER NOT NULL CHECK (completed_unix_ms >= 0), - message TEXT, - message_truncated INTEGER NOT NULL DEFAULT 0 - CHECK (message_truncated IN (0, 1)), - agent_session_path TEXT, - worker_id TEXT, - refreshed_turn_id TEXT, - observed_at TEXT NOT NULL, - PRIMARY KEY (host_id, pane_id, turn_epoch, turn) -); -""" -CREATE_HERDR_TURN_REFRESH_RETRIES_TABLE = """ -CREATE TABLE IF NOT EXISTS herdr_turn_refresh_retries ( - host_id TEXT NOT NULL, - pane_id TEXT NOT NULL, - turn_epoch INTEGER NOT NULL CHECK (turn_epoch >= 0), - turn INTEGER NOT NULL CHECK (turn >= 0), - status TEXT NOT NULL CHECK (status IN ('pending', 'escalated')), - refresh_status TEXT NOT NULL, - first_seen_at TEXT NOT NULL, - last_attempt_at TEXT NOT NULL, - next_attempt_at TEXT, - attempt_count INTEGER NOT NULL CHECK (attempt_count >= 1), - escalated_at TEXT, - PRIMARY KEY (host_id, pane_id, turn_epoch, turn), - CHECK ( - ( - status = 'pending' - AND next_attempt_at IS NOT NULL - AND escalated_at IS NULL - ) - OR - ( - status = 'escalated' - AND next_attempt_at IS NULL - AND escalated_at IS NOT NULL - ) - ) -); -""" -CREATE_HERDR_TURN_INDEXES = ( - ( - "CREATE INDEX IF NOT EXISTS idx_herdr_turn_completions_worker " - "ON herdr_turn_completions(host_id, worker_id, turn_epoch, turn)" - ), -) -CREATE_HERDR_TURN_REFRESH_RETRY_INDEXES = ( - ( - "CREATE INDEX IF NOT EXISTS idx_herdr_turn_refresh_retries_due " - "ON herdr_turn_refresh_retries(host_id, status, next_attempt_at)" - ), -) CREATE_AGENT_EVENTS_TABLE = """ CREATE TABLE IF NOT EXISTS agent_events ( @@ -1727,58 +1537,7 @@ def _record_response_size( ), ) -CREATE_AGENT_EVENT_TOMBSTONES_TABLE = """ -CREATE TABLE IF NOT EXISTS agent_event_tombstones ( - host_id TEXT NOT NULL, - event_id TEXT NOT NULL, - sequence INTEGER NOT NULL CHECK (sequence >= 1), - replay_fingerprint TEXT NOT NULL CHECK (length(replay_fingerprint) = 64), - observed_at TEXT, - retired_at TEXT NOT NULL CHECK (length(retired_at) BETWEEN 20 AND 40), - PRIMARY KEY (host_id, event_id), - CHECK (length(host_id) BETWEEN 1 AND 2048), - CHECK (instr(host_id, char(0)) = 0), - CHECK (length(event_id) = 64), - CHECK (observed_at IS NULL OR length(observed_at) BETWEEN 20 AND 40) -); -""" - -# v24/v25 tombstones predate retained replay-authority time. Migration code -# must build the historical shape rather than whatever the current target DDL -# happens to contain. -CREATE_AGENT_EVENT_TOMBSTONES_V25_TABLE = """ -CREATE TABLE IF NOT EXISTS agent_event_tombstones ( - host_id TEXT NOT NULL, - event_id TEXT NOT NULL, - sequence INTEGER NOT NULL CHECK (sequence >= 1), - replay_fingerprint TEXT NOT NULL CHECK (length(replay_fingerprint) = 64), - retired_at TEXT NOT NULL CHECK (length(retired_at) BETWEEN 20 AND 40), - PRIMARY KEY (host_id, event_id), - CHECK (length(host_id) BETWEEN 1 AND 2048), - CHECK (instr(host_id, char(0)) = 0), - CHECK (length(event_id) = 64) -); -""" - -CREATE_AGENT_EVENT_TOMBSTONE_INDEXES = ( - ( - "CREATE INDEX IF NOT EXISTS idx_agent_event_tombstones_host_sequence " - "ON agent_event_tombstones(host_id, sequence)" - ), -) -CREATE_PR6_TABLES = ( - CREATE_EVENTS_TABLE, - CREATE_SPACES_TABLE, - CREATE_WORKERS_TABLE, - CREATE_TURNS_TABLE, - CREATE_PENDING_INTERACTIONS_TABLE, - CREATE_ATTENTION_ITEMS_TABLE, - CREATE_LEGACY_COMMANDS_TABLE, - CREATE_CONNECTOR_OUTBOX_TABLE, - CREATE_CONNECTOR_DELIVERIES_TABLE, - CREATE_BACKEND_HEALTH_TABLE, -) CREATE_CURRENT_PR6_TABLES = ( CREATE_EVENTS_TABLE, CREATE_SPACES_TABLE, @@ -1812,7 +1571,6 @@ def _record_response_size( "CREATE INDEX IF NOT EXISTS idx_pending_interactions_host_status " "ON pending_interactions(host_id, status)" ), - CREATE_LEGACY_BACKEND_PENDING_TABLE, ( "CREATE INDEX IF NOT EXISTS idx_attention_items_host_source " "ON attention_items(host_id, source)" @@ -1833,11 +1591,6 @@ def _record_response_size( "CREATE INDEX IF NOT EXISTS idx_attention_items_host_fingerprint " "ON attention_items(host_id, fingerprint)" ), - ( - "CREATE UNIQUE INDEX IF NOT EXISTS ux_commands_host_request_action " - "ON commands(host_id, request_id, action)" - ), - "CREATE INDEX IF NOT EXISTS idx_commands_host_status ON commands(host_id, status)", ( "CREATE UNIQUE INDEX IF NOT EXISTS ux_connector_outbox_host_connector_key " "ON connector_outbox(host_id, connector, delivery_key)" @@ -1860,7 +1613,7 @@ def _record_response_size( "ON commands(host_id, state, updated_at, id)", ) CREATE_CURRENT_PR6_INDEXES = ( - CREATE_PR6_INDEXES[:16] + CREATE_PR6_INDEXES[18:] + CREATE_COMMAND_INDEXES + CREATE_PR6_INDEXES + CREATE_COMMAND_INDEXES ) @@ -2962,8 +2715,8 @@ def _connector_reclaim_expired_awaiting_ack_conn( if delivery is not None else None ) - # A legacy/noncanonical awaiting_ack row without a deadline is already - # overdue. Every non-terminal state must be swept rather than wedged. + # A malformed awaiting_ack row without a deadline is already overdue. + # Every non-terminal state must be swept rather than wedged. if not _connector_deadline_due( deadline, next_attempt_at, @@ -3575,7 +3328,6 @@ def _final_ready_payload_conn( turn_id: str, content_revision_value: str, allow_unroutable: bool = False, - turn_model: str = DEFAULT_TURN_MODEL, ) -> dict[str, Any] | None: canonical_turn_id = _resolve_canonical_turn_id_conn( conn, @@ -4551,7 +4303,6 @@ def prepare_connector_plan_begin( presentation_version: str, part_count: int, source_ref: str | None = None, - turn_model: str = DEFAULT_TURN_MODEL, now: str | None = None, ) -> dict[str, Any]: """Idempotently begin one bounded range-only presentation plan.""" @@ -7432,7 +7183,7 @@ def _connector_update_ref( WHERE id = ? AND status = 'leased' """, ( - _migration_private_state( + _connector_group_private_state( sibling_private, group=migration_group, canonical=False, @@ -8530,9 +8281,7 @@ def _command_receipt_from_row(row: Any) -> dict[str, Any]: "send_started_at": row[16], "terminal_at": row[17], "updated_at": str(row[18]), - "legacy_collision": bool(row[19]), - "legacy_collision_count": int(row[20]), - "selector_proof": str(row[21]), + "selector_proof": str(row[19]), } @@ -8590,50 +8339,8 @@ def _worker_binding_from_row(row: Any) -> WorkerBinding: ) -def _dedupe_command_receipts(conn: sqlite3.Connection) -> None: - """Keep the latest legacy receipt per logical key using bounded batches.""" - while True: - delete_ids = [ - int(row[0]) - for row in conn.execute( - """ - WITH ranked AS ( - SELECT - id, - ROW_NUMBER() OVER ( - PARTITION BY host_id, request_id, action - ORDER BY - COALESCE(completed_at, created_at) DESC, - created_at DESC, - id DESC - ) AS receipt_rank - FROM command_receipts - ) - SELECT id - FROM ranked - WHERE receipt_rank > 1 - ORDER BY id - LIMIT 500 - """ - ).fetchall() - ] - if not delete_ids: - return - placeholders = ",".join("?" for _ in delete_ids) - conn.execute( - f"DELETE FROM command_receipts WHERE id IN ({placeholders})", - delete_ids, - ) -def _ensure_command_receipt_unique_index(conn: sqlite3.Connection) -> None: - for row in conn.execute("PRAGMA index_list(command_receipts)").fetchall(): - index_name = str(row[1]) - is_unique = int(row[2]) == 1 - if index_name == "ux_command_receipts_host_request_action" and not is_unique: - conn.execute("DROP INDEX ux_command_receipts_host_request_action") - break - conn.execute(CREATE_LEGACY_COMMAND_RECEIPT_UNIQUE_INDEX) def _command_request_row( @@ -8663,8 +8370,6 @@ def _command_request_row( send_started_at, terminal_at, updated_at, - legacy_collision, - legacy_collision_count, selector_proof FROM command_receipts WHERE host_id = ? AND request_id = ? @@ -8682,256 +8387,16 @@ def _snapshot_payload(data: Mapping[str, Any]) -> tuple[dict[str, Any], str]: return payload_data, fingerprint -def _table_columns(conn: sqlite3.Connection, table: str = "snapshots") -> set[str]: - rows = conn.execute(f"PRAGMA table_info({table})").fetchall() - return {str(row[1]) for row in rows} -def _ensure_columns( - conn: sqlite3.Connection, - table: str, - columns: Mapping[str, str], -) -> None: - existing = _table_columns(conn, table) - for column, definition in columns.items(): - if column not in existing: - conn.execute(f"ALTER TABLE {table} ADD COLUMN {column} {definition}") -def _backfill_content_fingerprints(conn: sqlite3.Connection) -> None: - rows = conn.execute( - """ - SELECT id, payload - FROM snapshots - WHERE content_fingerprint IS NULL OR content_fingerprint = '' - """ - ).fetchall() - for row_id, payload in rows: - try: - data = json.loads(payload) - except json.JSONDecodeError: - fingerprint = _content_fingerprint({"payload": payload}) - conn.execute( - "UPDATE snapshots SET content_fingerprint = ? WHERE id = ?", - (fingerprint, row_id), - ) - continue - if not isinstance(data, Mapping): - fingerprint = _content_fingerprint({"payload": data}) - conn.execute( - "UPDATE snapshots SET content_fingerprint = ? WHERE id = ?", - (fingerprint, row_id), - ) - continue - payload_data, fingerprint = _snapshot_payload( - Snapshot.from_dict(data).to_dict() - ) - conn.execute( - """ - UPDATE snapshots - SET content_fingerprint = ?, payload = ? - WHERE id = ? - """, - (fingerprint, _canonical_json(payload_data), row_id), - ) -def _ensure_command_receipt_columns(conn: sqlite3.Connection) -> None: - _ensure_columns( - conn, - "command_receipts", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "request_id": "TEXT NOT NULL DEFAULT ''", - "action": "TEXT NOT NULL DEFAULT ''", - "payload_fingerprint": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT ''", - "result_json": "TEXT NOT NULL DEFAULT '{}'", - "created_at": "TEXT NOT NULL DEFAULT ''", - "completed_at": "TEXT", - "uncertain": "INTEGER NOT NULL DEFAULT 0", - }, - ) -def _ensure_worker_binding_columns(conn: sqlite3.Connection) -> None: - _ensure_columns( - conn, - "worker_bindings", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "worker_id": "TEXT NOT NULL DEFAULT ''", - "worker_fingerprint": "TEXT NOT NULL DEFAULT ''", - "backend": "TEXT NOT NULL DEFAULT ''", - "target_kind": "TEXT NOT NULL DEFAULT ''", - "target_value": "TEXT NOT NULL DEFAULT ''", - "turn_target_kind": "TEXT", - "turn_target_value": "TEXT", - "sendable": "INTEGER NOT NULL DEFAULT 0", - "reason": "TEXT", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "expires_at": "TEXT NOT NULL DEFAULT '9999-12-31T23:59:59+00:00'", - "private_fingerprint": "TEXT NOT NULL DEFAULT ''", - }, - ) -def _ensure_pr6_columns(conn: sqlite3.Connection) -> None: - _ensure_columns( - conn, - "events", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "event_type": "TEXT NOT NULL DEFAULT ''", - "aggregate_type": "TEXT NOT NULL DEFAULT ''", - "aggregate_id": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "spaces", - { - "name": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "updated_at": "TEXT", - "fingerprint": "TEXT NOT NULL DEFAULT ''", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "workers", - { - "worker_fingerprint": "TEXT NOT NULL DEFAULT ''", - "space_id": "TEXT", - "name": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "last_seen_at": "TEXT", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "turns", - { - "worker_id": "TEXT NOT NULL DEFAULT ''", - "worker_fingerprint": "TEXT", - "space_id": "TEXT", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "kind": "TEXT NOT NULL DEFAULT 'unknown'", - "updated_at": "TEXT", - "fingerprint": "TEXT NOT NULL DEFAULT ''", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "pending_interactions", - { - "worker_id": "TEXT NOT NULL DEFAULT ''", - "worker_fingerprint": "TEXT", - "space_id": "TEXT", - "kind": "TEXT NOT NULL DEFAULT 'unknown'", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "updated_at": "TEXT", - "fingerprint": "TEXT NOT NULL DEFAULT ''", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "attention_items", - { - "source": "TEXT NOT NULL DEFAULT ''", - "kind": "TEXT NOT NULL DEFAULT 'unknown'", - "severity": "TEXT NOT NULL DEFAULT 'info'", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "updated_at": "TEXT", - "fingerprint": "TEXT NOT NULL DEFAULT ''", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "first_seen_at": "TEXT NOT NULL DEFAULT ''", - "last_seen_at": "TEXT NOT NULL DEFAULT ''", - "last_changed_at": "TEXT NOT NULL DEFAULT ''", - "resolved_at": "TEXT", - "lifecycle_status": "TEXT NOT NULL DEFAULT 'open'", - "resolved_reason": "TEXT", - "signal_count": "INTEGER NOT NULL DEFAULT 1", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "commands", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "request_id": "TEXT NOT NULL DEFAULT ''", - "action": "TEXT NOT NULL DEFAULT ''", - "payload_fingerprint": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT ''", - "dry_run": "INTEGER NOT NULL DEFAULT 0", - "uncertain": "INTEGER NOT NULL DEFAULT 0", - "request_json": "TEXT NOT NULL DEFAULT '{}'", - "result_json": "TEXT NOT NULL DEFAULT '{}'", - "created_at": "TEXT NOT NULL DEFAULT ''", - "reserved_at": "TEXT", - "completed_at": "TEXT", - "updated_at": "TEXT NOT NULL DEFAULT ''", - }, - ) - _ensure_columns( - conn, - "connector_outbox", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "connector": "TEXT NOT NULL DEFAULT ''", - "delivery_key": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - "private_state_json": "TEXT NOT NULL DEFAULT '{}'", - "created_at": "TEXT NOT NULL DEFAULT ''", - "updated_at": "TEXT NOT NULL DEFAULT ''", - "next_attempt_at": "TEXT", - }, - ) - _ensure_columns( - conn, - "connector_deliveries", - { - "outbox_id": "INTEGER", - "host_id": "TEXT NOT NULL DEFAULT ''", - "connector": "TEXT NOT NULL DEFAULT ''", - "delivery_key": "TEXT NOT NULL DEFAULT ''", - "attempt": "INTEGER NOT NULL DEFAULT 0", - "status": "TEXT NOT NULL DEFAULT ''", - "response_json": "TEXT NOT NULL DEFAULT '{}'", - "private_state_json": "TEXT NOT NULL DEFAULT '{}'", - "created_at": "TEXT NOT NULL DEFAULT ''", - "delivered_at": "TEXT", - }, - ) - _ensure_columns( - conn, - "backend_health", - { - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "outcome": "TEXT NOT NULL DEFAULT 'unknown'", - "observed_at": "TEXT", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) def _append_event_conn( @@ -9009,30 +8474,6 @@ def _repair_missing_final_ready_anchors_conn( ) -def append_event( - db_path: Path, - host_id: str, - event_type: str, - payload: Mapping[str, Any], - *, - aggregate_type: str = "", - aggregate_id: str = "", - observed_at: str | None = None, - content_fingerprint: str | None = None, -) -> int: - """Append a private store event and return its row id.""" - with _connect(db_path, prepare=True) as conn: - _ensure_schema(conn) - return _append_event_conn( - conn, - host_id=host_id, - event_type=event_type, - payload=payload, - aggregate_type=aggregate_type, - aggregate_id=aggregate_id, - observed_at=observed_at, - content_fingerprint=content_fingerprint, - ) def _prune_host_projection( @@ -9241,22 +8682,6 @@ def _strict_utc_timestamp(value: Any) -> str | None: return None -def _legacy_snapshot_created_at_is_authoritative( - created_at: Any, - payload: Any, -) -> bool: - """Distinguish a real year-9999 observation from the former sentinel.""" - canonical_created_at = _strict_utc_timestamp(created_at) - canonical_payload_at = _strict_utc_timestamp( - _json_object(payload).get("updated_at") - ) - return ( - canonical_created_at - == _LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE - and canonical_payload_at == canonical_created_at - ) - - def _attention_family_key(host_id: str, item: Mapping[str, Any]) -> str: source = _store_public_text(item.get("source"), default="unknown") kind = _store_public_label(item.get("kind")) @@ -10379,10 +9804,8 @@ def _project_command_request_conn(conn: sqlite3.Connection, row: Any) -> None: reserved_at, send_started_at, terminal_at, - updated_at, - legacy_collision, - legacy_collision_count - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(host_id, request_id) DO UPDATE SET action = excluded.action, canonical_version = excluded.canonical_version, @@ -10396,9 +9819,7 @@ def _project_command_request_conn(conn: sqlite3.Connection, row: Any) -> None: reserved_at = excluded.reserved_at, send_started_at = excluded.send_started_at, terminal_at = excluded.terminal_at, - updated_at = excluded.updated_at, - legacy_collision = excluded.legacy_collision, - legacy_collision_count = excluded.legacy_collision_count + updated_at = excluded.updated_at """, ( str(row[1]), @@ -10416,69 +9837,15 @@ def _project_command_request_conn(conn: sqlite3.Connection, row: Any) -> None: row[16], row[17], str(row[18]), - int(row[19]), - int(row[20]), ), ) -def _backfill_command_audit(conn: sqlite3.Connection) -> None: - rows = conn.execute( - """ - SELECT - host_id, - request_id, - action, - payload_fingerprint, - status, - result_json, - created_at, - completed_at, - uncertain - FROM command_receipts - WHERE request_id != '' - ORDER BY id - """ - ).fetchall() - for row in rows: - _upsert_command_audit_from_receipt_row(conn, row) -def _backfill_legacy_attention_columns(conn: sqlite3.Connection) -> None: - conn.execute( - """ - UPDATE attention_items - SET - first_seen_at = CASE - WHEN first_seen_at IS NULL OR first_seen_at = '' - THEN COALESCE(NULLIF(observed_at, ''), updated_at, '') - ELSE first_seen_at - END, - last_seen_at = CASE - WHEN last_seen_at IS NULL OR last_seen_at = '' - THEN COALESCE(NULLIF(observed_at, ''), updated_at, '') - ELSE last_seen_at - END, - last_changed_at = CASE - WHEN last_changed_at IS NULL OR last_changed_at = '' - THEN COALESCE(NULLIF(observed_at, ''), updated_at, '') - ELSE last_changed_at - END, - lifecycle_status = CASE - WHEN lifecycle_status IS NULL OR lifecycle_status = '' - THEN 'open' - ELSE lifecycle_status - END, - signal_count = CASE - WHEN signal_count IS NULL OR signal_count < 1 - THEN 1 - ELSE signal_count - END - """ - ) -def _migration_private_state( +def _connector_group_private_state( raw: Any, *, group: str, @@ -10495,673 +9862,6 @@ def _migration_private_state( return _canonical_json(state) -def _legacy_attention_job_identity( - row_host_id: str, - payload_json: Any, -) -> tuple[str, str, str, str, str, str | None] | None: - payload = sanitize_public_mapping(_json_object(payload_json), backend_neutral=True) - if str(payload.get("host_id") or "") != str(row_host_id): - return None - event_type = str(payload.get("event_type") or "") - if event_type not in {"attention_created", "attention_escalated"}: - return None - attention = payload.get("attention") - if not isinstance(attention, Mapping): - return None - source = _store_public_text(attention.get("source"), default="") - kind = _store_public_label(attention.get("kind")) - if not source or kind == "unknown": - return None - family_key = _attention_family_key(str(row_host_id), attention) - stage = ( - "initial" - if event_type == "attention_created" - else f"severity:{normalize_severity(attention.get('severity'))}" - ) - return ( - str(row_host_id), - family_key, - event_type, - stage, - _attention_id_from_item(attention), - _strict_utc_timestamp(payload.get("transition_at")), - ) - - -def _migrate_v4_attention_rows_conn(conn: sqlite3.Connection) -> None: - conn.execute("DROP TABLE IF EXISTS temp.attention_v5_rows") - conn.execute( - """ - CREATE TEMP TABLE attention_v5_rows ( - host_id TEXT NOT NULL, - family_key TEXT NOT NULL, - attention_id TEXT NOT NULL, - is_open INTEGER NOT NULL, - positive_at TEXT, - changed_at TEXT, - first_seen_at TEXT, - severity_rank INTEGER NOT NULL, - signal_count INTEGER NOT NULL - ) - """ - ) - cursor = conn.execute( - """ - SELECT host_id, attention_id, source, kind, severity, lifecycle_status, - updated_at, observed_at, first_seen_at, last_seen_at, - last_changed_at, signal_count - FROM attention_items - ORDER BY host_id, attention_id - """ - ) - while True: - batch = cursor.fetchmany(500) - if not batch: - break - values: list[tuple[Any, ...]] = [] - for row in batch: - host_id = str(row[0]) - item = {"source": row[2], "kind": row[3]} - positive_at = ( - _strict_utc_timestamp(row[9]) - or _strict_utc_timestamp(row[7]) - or _strict_utc_timestamp(row[6]) - ) - values.append( - ( - host_id, - _attention_family_key(host_id, item), - str(row[1]), - int(str(row[5] or "open") == ATTENTION_LIFECYCLE_OPEN), - positive_at, - _strict_utc_timestamp(row[10]), - _strict_utc_timestamp(row[8]), - _attention_severity_rank(row[4]), - max(1, int(row[11] or 1)), - ) - ) - conn.executemany( - """ - INSERT INTO attention_v5_rows ( - host_id, family_key, attention_id, is_open, positive_at, - changed_at, first_seen_at, severity_rank, signal_count - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - values, - ) - - family_cursor = conn.execute( - """ - SELECT host_id, family_key - FROM attention_v5_rows - GROUP BY host_id, family_key - ORDER BY host_id, family_key - """ - ) - while True: - families = family_cursor.fetchmany(500) - if not families: - break - for host_id_raw, family_key_raw in families: - host_id = str(host_id_raw) - family_key = str(family_key_raw) - candidates = conn.execute( - """ - SELECT attention_id, is_open, positive_at, changed_at, - first_seen_at, severity_rank, signal_count - FROM attention_v5_rows - WHERE host_id = ? AND family_key = ? - ORDER BY attention_id - """, - (host_id, family_key), - ) - winner: tuple[Any, ...] | None = None - earliest_first: str | None = None - latest_positive: str | None = None - latest_progress: str | None = None - total_signals = 0 - max_severity = -1 - while True: - candidate_batch = candidates.fetchmany(500) - if not candidate_batch: - break - for candidate in candidate_batch: - total_signals += max(1, int(candidate[6] or 1)) - max_severity = max(max_severity, int(candidate[5])) - if candidate[4] and ( - earliest_first is None or str(candidate[4]) < earliest_first - ): - earliest_first = str(candidate[4]) - if candidate[2] and ( - latest_positive is None or str(candidate[2]) > latest_positive - ): - latest_positive = str(candidate[2]) - progress_at = max( - str(candidate[2] or ""), - str(candidate[3] or ""), - ) - if progress_at and ( - latest_progress is None or progress_at > latest_progress - ): - latest_progress = progress_at - rank = ( - progress_at, - int(candidate[1]), - str(candidate[2] or ""), - str(candidate[3] or ""), - int(candidate[5]), - int(candidate[6]), - "".join( - chr(0x10FFFF - ord(ch)) for ch in str(candidate[0]) - ), - ) - if winner is None or rank > winner[0]: - winner = (rank, *candidate) - if winner is None or latest_positive is None: - continue - winner_attention_id = str(winner[1]) - is_open = bool(winner[2]) - first_seen_at = earliest_first or latest_positive - # The lifecycle watermark (last_accepted_at) must be the newest - # lifecycle progress — max(latest positive, latest change/resolve) — - # not merely the latest positive. A resolved episode whose resolution - # (t10) is newer than its last positive (t0) would otherwise seed the - # watermark at t0, letting a delayed positive at t5 (< the authoritative - # resolution) pass the observation guard and spuriously reopen - # generation 2 with a fresh notification. last_positive_at stays the - # actual latest positive; the observation key is anchored to the - # accepted progress so replaying the authoritative resolution is a no-op. - accepted_progress = latest_progress or latest_positive - observation_key = stable_fingerprint( - { - "domain": "tendwire.attention.observation.v1", - "host_id": host_id, - "authority": "migration", - "observed_at": accepted_progress, - "snapshot_content_fingerprint": family_key, - } - ) - conn.execute( - """ - INSERT OR IGNORE INTO attention_lifecycles ( - host_id, family_key, generation, lifecycle_status, - current_attention_id, first_seen_at, last_positive_at, - first_missing_at, missing_observation_count, last_accepted_at, - last_observation_key, max_notified_severity_rank - ) VALUES (?, ?, 1, ?, ?, ?, ?, NULL, 0, ?, ?, ?) - """, - ( - host_id, - family_key, - ( - ATTENTION_LIFECYCLE_OPEN - if is_open - else ATTENTION_LIFECYCLE_RESOLVED - ), - winner_attention_id if is_open else None, - first_seen_at, - latest_positive, - accepted_progress, - observation_key, - max_severity, - ), - ) - if is_open: - conn.execute( - """ - UPDATE attention_items - SET first_seen_at = ?, last_seen_at = ?, - signal_count = ?, lifecycle_status = 'open', - resolved_at = NULL, resolved_reason = NULL - WHERE host_id = ? AND attention_id = ? - """, - ( - first_seen_at, - latest_positive, - total_signals, - host_id, - winner_attention_id, - ), - ) - conn.execute( - """ - UPDATE attention_items - SET lifecycle_status = 'resolved', - resolved_at = COALESCE(NULLIF(resolved_at, ''), ?), - resolved_reason = ?, - last_changed_at = ? - WHERE host_id = ? AND lifecycle_status = 'open' - AND attention_id != ? - AND attention_id IN ( - SELECT attention_id FROM attention_v5_rows - WHERE host_id = ? AND family_key = ? AND is_open = 1 - ) - """, - ( - latest_positive, - ATTENTION_RESOLVED_REASON_SUPERSEDED, - latest_positive, - host_id, - winner_attention_id, - host_id, - family_key, - ), - ) - else: - conn.execute( - """ - UPDATE attention_items - SET lifecycle_status = 'resolved', - resolved_at = COALESCE(NULLIF(resolved_at, ''), ?), - resolved_reason = CASE - WHEN attention_id = ? THEN COALESCE(resolved_reason, 'gone') - ELSE ? - END, - last_changed_at = ? - WHERE host_id = ? AND lifecycle_status = 'open' - AND attention_id IN ( - SELECT attention_id FROM attention_v5_rows - WHERE host_id = ? AND family_key = ? AND is_open = 1 - ) - """, - ( - latest_positive, - winner_attention_id, - ATTENTION_RESOLVED_REASON_SUPERSEDED, - latest_positive, - host_id, - host_id, - family_key, - ), - ) - conn.execute("DROP TABLE temp.attention_v5_rows") - - -def _migrate_v4_attention_outbox_conn(conn: sqlite3.Connection) -> None: - conn.execute("DROP TABLE IF EXISTS temp.attention_v5_jobs") - conn.execute( - """ - CREATE TEMP TABLE attention_v5_jobs ( - outbox_id INTEGER PRIMARY KEY, - host_id TEXT NOT NULL, - family_key TEXT NOT NULL, - event_type TEXT NOT NULL, - stage TEXT NOT NULL, - attention_id TEXT NOT NULL, - transition_at TEXT, - group_key TEXT NOT NULL - ) - """ - ) - cursor = conn.execute( - """ - SELECT id, host_id, payload_json - FROM connector_outbox - WHERE connector = ? - ORDER BY id - """, - (ATTENTION_OUTBOX_CONNECTOR,), - ) - while True: - batch = cursor.fetchmany(500) - if not batch: - break - values: list[tuple[Any, ...]] = [] - for outbox_id, host_id, payload_json in batch: - identity = _legacy_attention_job_identity(str(host_id), payload_json) - if identity is None: - continue - ( - identity_host, - family_key, - event_type, - stage, - attention_id, - transition_at, - ) = identity - group_key = stable_fingerprint( - { - "domain": "tendwire.attention.migration-group.v1", - "host_id": identity_host, - "family_key": family_key, - "generation": 1, - "event_type": event_type, - "stage": stage, - } - ) - values.append( - ( - int(outbox_id), - identity_host, - family_key, - event_type, - stage, - attention_id, - transition_at, - group_key, - ) - ) - if event_type == "attention_escalated": - severity = stage.removeprefix("severity:") - conn.execute( - """ - UPDATE attention_lifecycles - SET max_notified_severity_rank = - MAX(max_notified_severity_rank, ?) - WHERE host_id = ? AND family_key = ? - """, - ( - _attention_severity_rank(severity), - identity_host, - family_key, - ), - ) - conn.executemany( - """ - INSERT INTO attention_v5_jobs ( - outbox_id, host_id, family_key, event_type, stage, - attention_id, transition_at, group_key - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - values, - ) - - group_cursor = conn.execute( - """ - SELECT host_id, family_key, event_type, stage, group_key - FROM attention_v5_jobs - GROUP BY host_id, family_key, event_type, stage, group_key - ORDER BY group_key - """ - ) - while True: - groups = group_cursor.fetchmany(500) - if not groups: - break - for host_id, family_key, event_type, stage, group_key in groups: - candidate_rows = conn.execute( - """ - SELECT o.id, o.status, o.payload_json, o.private_state_json, - j.attention_id, j.transition_at - FROM connector_outbox o - JOIN attention_v5_jobs j ON j.outbox_id = o.id - WHERE j.group_key = ? - AND o.status IN ('queued', 'retry', 'deferred', 'leased') - ORDER BY o.id - """, - (group_key,), - ).fetchall() - if not candidate_rows: - continue - lifecycle_row = conn.execute( - """ - SELECT l.lifecycle_status, l.current_attention_id, - l.first_seen_at, l.last_positive_at, l.last_accepted_at, - i.last_changed_at, i.last_seen_at, i.observed_at, - i.updated_at - FROM attention_lifecycles l - LEFT JOIN attention_items i - ON i.host_id = l.host_id - AND i.attention_id = l.current_attention_id - WHERE l.host_id = ? AND l.family_key = ? - """, - (host_id, family_key), - ).fetchone() - lifecycle_open = ( - lifecycle_row is not None - and str(lifecycle_row[0]) == ATTENTION_LIFECYCLE_OPEN - and lifecycle_row[1] is not None - ) - current_attention_id = ( - str(lifecycle_row[1]) if lifecycle_open else "" - ) - current_anchor_candidates = ( - [ - canonical - for canonical in ( - _strict_utc_timestamp(value) - # Include the lifecycle's persisted last_accepted_at - # (index 4) alongside the attention_items timestamps so - # the current-episode anchor reflects the authoritative - # accepted-progress watermark even when the row's own - # timestamps are skewed (e.g. a delayed positive). - for value in lifecycle_row[4:9] - ) - if canonical is not None - ] - if lifecycle_open - else [] - ) - current_episode_anchor = ( - max(current_anchor_candidates) - if current_anchor_candidates - else "" - ) - terminalized_current_episode = False - if lifecycle_open: - terminal_rows = conn.execute( - """ - SELECT j.attention_id, j.transition_at - FROM connector_outbox o - JOIN attention_v5_jobs j ON j.outbox_id = o.id - WHERE j.group_key = ? - AND o.status IN ('delivered', 'dead_letter') - """, - (group_key,), - ).fetchall() - terminalized_current_episode = bool(current_episode_anchor) and any( - str(terminal_attention_id) == current_attention_id - and bool(terminal_transition_at) - and str(terminal_transition_at) >= current_episode_anchor - for terminal_attention_id, terminal_transition_at in terminal_rows - ) - - leased_rows = [ - row - for row in candidate_rows - if str(row[1]) == _CONNECTOR_LEASE_STATUS - and conn.execute( - """ - SELECT 1 FROM connector_deliveries - WHERE outbox_id = ? AND status = 'leased' - LIMIT 1 - """, - (int(row[0]),), - ).fetchone() - is not None - ] - conn.execute( - """ - UPDATE connector_outbox - SET status = ?, next_attempt_at = NULL - WHERE id IN ( - SELECT j.outbox_id FROM attention_v5_jobs j - WHERE j.group_key = ? - ) AND ( - status IN ('queued', 'retry', 'deferred') - OR ( - status = 'leased' - AND id NOT IN ( - SELECT d.outbox_id FROM connector_deliveries d - WHERE d.status = 'leased' AND d.outbox_id IS NOT NULL - ) - ) - ) - """, - (_CONNECTOR_SUPERSEDED_OUTBOX_STATUS, group_key), - ) - - def active_rank(row: Any) -> tuple[int, str, int, int]: - return ( - int(str(row[4]) == current_attention_id), - str(row[5] or ""), - int(str(row[1]) == _CONNECTOR_LEASE_STATUS), - -int(row[0]), - ) - - if not lifecycle_open or terminalized_current_episode: - for leased_row in leased_rows: - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = ? - WHERE id = ? AND status = 'leased' - """, - ( - _migration_private_state( - leased_row[3], - group=str(group_key), - canonical=False, - terminal_after_lease=True, - ), - int(leased_row[0]), - ), - ) - continue - - pollable_candidates = [ - row - for row in candidate_rows - if str(row[1]) in _CONNECTOR_POLLABLE_STATUSES - ] - active_candidates = [*pollable_candidates, *leased_rows] - if not active_candidates: - continue - selected = max(active_candidates, key=active_rank) - selected_id = int(selected[0]) - selected_is_lease = ( - str(selected[1]) == _CONNECTOR_LEASE_STATUS - and any(int(row[0]) == selected_id for row in leased_rows) - ) - for leased_row in leased_rows: - leased_id = int(leased_row[0]) - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = ? - WHERE id = ? AND status = 'leased' - """, - ( - _migration_private_state( - leased_row[3], - group=str(group_key), - canonical=selected_is_lease and leased_id == selected_id, - terminal_after_lease=( - not selected_is_lease or leased_id != selected_id - ), - ), - leased_id, - ), - ) - if selected_is_lease: - continue - transition_key = stable_fingerprint( - { - "domain": "tendwire.attention.transition.v1", - "host_id": str(host_id), - "family_key": str(family_key), - "generation": 1, - "event_type": str(event_type), - "stage": str(stage), - } - ) - canonical_key = f"attention:{event_type}:{transition_key}" - payload = sanitize_public_mapping( - _json_object(selected[2]), backend_neutral=True - ) - transition_at = str(selected[5] or "") - if not transition_at: - transition_at = ( - str(lifecycle_row[4]) - if lifecycle_row is not None - else "1970-01-01T00:00:00+00:00" - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES (?, ?, ?, 'queued', ?, ?, ?, ?, NULL) - ON CONFLICT(host_id, connector, delivery_key) DO NOTHING - """, - ( - str(host_id), - ATTENTION_OUTBOX_CONNECTOR, - canonical_key, - _canonical_json(payload), - _migration_private_state( - {}, - group=str(group_key), - canonical=True, - ), - transition_at, - transition_at, - ), - ) - conn.execute("DROP TABLE temp.attention_v5_jobs") - - -def _migrate_v0_to_v1_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_SNAPSHOTS_TABLE) - if "content_fingerprint" not in _table_columns(conn): - conn.execute( - "ALTER TABLE snapshots ADD COLUMN " - "content_fingerprint TEXT NOT NULL DEFAULT ''" - ) - _backfill_content_fingerprints(conn) - for statement in CREATE_LEGACY_SNAPSHOT_INDEXES: - conn.execute(statement) - - -def _migrate_v1_to_v2_conn(conn: sqlite3.Connection) -> None: - _migrate_v0_to_v1_conn(conn) - conn.execute(CREATE_LEGACY_COMMAND_RECEIPTS_TABLE) - _ensure_command_receipt_columns(conn) - _dedupe_command_receipts(conn) - for statement in CREATE_LEGACY_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - _ensure_command_receipt_unique_index(conn) - conn.execute(CREATE_WORKER_BINDINGS_TABLE) - _ensure_worker_binding_columns(conn) - for statement in CREATE_WORKER_BINDING_INDEXES: - conn.execute(statement) - conn.execute(CREATE_WORKER_BINDING_UNIQUE_INDEX) - - -def _migrate_v2_to_v3_conn(conn: sqlite3.Connection) -> None: - _migrate_v1_to_v2_conn(conn) - for statement in CREATE_PR6_TABLES: - conn.execute(statement) - _ensure_pr6_columns(conn) - for statement in CREATE_PR6_INDEXES: - conn.execute(statement) - _backfill_command_audit(conn) - - -def _migrate_v3_to_v4_conn(conn: sqlite3.Connection) -> None: - _migrate_v2_to_v3_conn(conn) - _backfill_legacy_attention_columns(conn) - - -def _migrate_v4_to_v5_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_ATTENTION_LIFECYCLES_TABLE) - for statement in CREATE_ATTENTION_LIFECYCLE_INDEXES: - conn.execute(statement) - _migrate_v4_attention_rows_conn(conn) - _migrate_v4_attention_outbox_conn(conn) - - -_LEGACY_TRUNCATION_MARKER = "\n[truncated]" - - -def _legacy_canonical_field(value: Any) -> tuple[str | None, str]: - text = sanitize_canonical_turn_text(value) - if text is None or text == "": - return None, "absent" - state = "known_incomplete" if text.endswith(_LEGACY_TRUNCATION_MARKER) else "complete" - return text, state - - def _insert_turn_content_page_boundaries_conn( conn: sqlite3.Connection, *, @@ -11174,13 +9874,8 @@ def _insert_turn_content_page_boundaries_conn( conn.executemany( """ INSERT OR IGNORE INTO turn_content_page_boundaries ( - host_id, - turn_id, - content_revision, - field, - page_index, - start_char, - start_byte + host_id, turn_id, content_revision, field, + page_index, start_char, start_byte ) VALUES (?, ?, ?, ?, ?, ?, ?) """, ( @@ -11269,61 +9964,6 @@ def _insert_turn_content_revision_conn( ) return revision - -def _backfill_legacy_turn_content_conn(conn: sqlite3.Connection) -> None: - rows = conn.execute( - """ - SELECT host_id, turn_id, observed_at, payload_json - FROM turns - ORDER BY host_id, turn_id - """ - ).fetchall() - for host_id, turn_id, observed_at, payload_json in rows: - try: - payload = json.loads(str(payload_json or "{}")) - except (TypeError, json.JSONDecodeError): - payload = {} - if not isinstance(payload, dict): - payload = {} - user_text, user_state = _legacy_canonical_field(payload.get("user_text")) - final_text, final_state = _legacy_canonical_field( - payload.get("assistant_final_text") - ) - if user_state != "absent" or final_state != "absent": - _insert_turn_content_revision_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - user_text=user_text, - assistant_final_text=final_text, - user_state=user_state, - final_state=final_state, - created_at=str(observed_at or "1970-01-01T00:00:00+00:00"), - ) - for key in ( - "user_text", - "assistant_final_text", - "user_preview", - "assistant_final_preview", - "content", - ): - payload.pop(key, None) - encoded = _canonical_json(payload) - conn.execute( - """ - UPDATE turns - SET payload_json = ?, fingerprint = ? - WHERE host_id = ? AND turn_id = ? - """, - ( - encoded, - stable_fingerprint(payload), - str(host_id), - str(turn_id), - ), - ) - - def _ensure_payload_turn_content_revision_conn( conn: sqlite3.Connection, *, @@ -11372,7 +10012,6 @@ def _ensure_payload_turn_content_revision_conn( ) return True - def _ensure_absent_turn_content_revision_conn( conn: sqlite3.Connection, *, @@ -11426,252 +10065,190 @@ def _ensure_absent_turn_content_revision_conn( ) return bool(cursor.rowcount) - -def _backfill_missing_turn_content_revisions_conn( +def _resolve_canonical_turn_id_conn( conn: sqlite3.Connection, -) -> int: - """Give every stored turn one stable authoritative v2 content descriptor.""" - repaired = 0 - cursor = conn.execute( - """ - SELECT turns.host_id, turns.turn_id, turns.observed_at - FROM turns - WHERE NOT EXISTS ( - SELECT 1 - FROM turn_content_revisions AS revisions - WHERE revisions.host_id = turns.host_id - AND revisions.turn_id = turns.turn_id - AND revisions.is_current = 1 - ) - ORDER BY turns.host_id, turns.turn_id - """ - ) - while True: - rows = cursor.fetchmany(500) - if not rows: - return repaired - for host_id, turn_id, observed_at in rows: - if _ensure_absent_turn_content_revision_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - observed_at=str(observed_at) if observed_at else None, - ): - repaired += 1 + host_id: str, + turn_id: Any, +) -> str | None: + """Resolve a superseded public turn alias without guessing through cycles.""" + current = str(turn_id or "").strip() + if not current: + return None + seen: set[str] = set() + while current and current not in seen: + seen.add(current) + row = conn.execute( + """ + SELECT canonical_turn_id + FROM turn_supersessions + WHERE host_id = ? AND superseded_turn_id = ? + """, + (str(host_id), current), + ).fetchone() + if row is None: + return current + current = str(row[0] or "").strip() + return None -def _rebuild_v6_presentation_plans_conn(conn: sqlite3.Connection) -> None: - """Rebuild the two bounded plan tables with generation-aware v7 keys.""" - conn.execute( - """ - CREATE TABLE turn_presentation_plans_v7 ( - id INTEGER PRIMARY KEY, - host_id TEXT NOT NULL, - name TEXT NOT NULL, - plan_token TEXT NOT NULL, - turn_id TEXT NOT NULL, - content_revision TEXT NOT NULL, - presentation_version TEXT NOT NULL, - generation INTEGER NOT NULL DEFAULT 1 CHECK (generation >= 1), - part_count INTEGER NOT NULL CHECK (part_count > 0), - state TEXT NOT NULL - CHECK (state IN ( - 'preparing', - 'waiting_predecessor', - 'active', - 'completed', - 'superseded', - 'failed' - )), - replaces_plan_token TEXT, - recovers_plan_token TEXT, - created_at TEXT NOT NULL, - activated_at TEXT, - completed_at TEXT, - UNIQUE (host_id, name, plan_token), - UNIQUE ( - host_id, - name, - turn_id, - content_revision, - presentation_version, - generation - ), - FOREIGN KEY (host_id, turn_id, content_revision) - REFERENCES turn_content_revisions(host_id, turn_id, content_revision) - ON DELETE RESTRICT - ) - """ - ) - conn.execute( - """ - INSERT INTO turn_presentation_plans_v7 ( - id, host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - replaces_plan_token, recovers_plan_token, created_at, - activated_at, completed_at - ) - SELECT - id, host_id, name, plan_token, turn_id, content_revision, - presentation_version, 1, part_count, state, - replaces_plan_token, NULL, created_at, activated_at, completed_at - FROM turn_presentation_plans - ORDER BY id - """ - ) - conn.execute( - """ - CREATE TABLE turn_presentation_jobs_v7 ( - id INTEGER PRIMARY KEY, - plan_id INTEGER NOT NULL, - sequence_index INTEGER NOT NULL CHECK (sequence_index >= 0), - operation TEXT NOT NULL CHECK (operation IN ('upsert', 'retire')), - part_ordinal INTEGER NOT NULL CHECK (part_ordinal >= 0), - spans_json TEXT NOT NULL, - outbox_id INTEGER UNIQUE, - created_at TEXT NOT NULL, - UNIQUE (plan_id, sequence_index), - UNIQUE (plan_id, operation, part_ordinal), - FOREIGN KEY (plan_id) - REFERENCES turn_presentation_plans_v7(id) ON DELETE CASCADE, - FOREIGN KEY (outbox_id) - REFERENCES connector_outbox(id) ON DELETE RESTRICT - ) - """ - ) - conn.execute( - """ - INSERT INTO turn_presentation_jobs_v7 ( - id, plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - ) - SELECT - id, plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - FROM turn_presentation_jobs - ORDER BY id - """ - ) - conn.execute("DROP TABLE turn_presentation_jobs") - conn.execute("DROP TABLE turn_presentation_plans") + +def _create_current_schema_objects_conn(conn: sqlite3.Connection) -> None: + table_statements = ( + CREATE_SNAPSHOTS_TABLE, + CREATE_COMMAND_RECEIPTS_TABLE, + CREATE_WORKER_BINDINGS_TABLE, + *CREATE_CURRENT_PR6_TABLES, + CREATE_BACKEND_PENDING_TABLE, + CREATE_BACKEND_PENDING_CLAIMS_TABLE, + CREATE_ATTENTION_LIFECYCLES_TABLE, + CREATE_TURN_CONTENT_REVISIONS_TABLE, + CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE, + CREATE_TURN_PRESENTATION_PLANS_TABLE, + CREATE_TURN_PRESENTATION_JOBS_TABLE, + CREATE_TURN_PRESENTATION_RECOVERIES_TABLE, + CREATE_STORE_MAINTENANCE_STATE_TABLE, + CREATE_STORE_MAINTENANCE_CURSORS_TABLE, + CREATE_TURN_LIST_STATE_TABLE, + CREATE_TURN_LIST_HOSTS_TABLE, + CREATE_TURN_CHANGE_JOURNAL_TABLE, + CREATE_TURN_CHANGE_FLOOR_TABLE, + CREATE_TURN_CHANGE_STATE_TABLE, + CREATE_TURN_SUBMISSIONS_TABLE, + CREATE_TURN_SUPERSESSIONS_TABLE, + CREATE_AGENT_EVENTS_TABLE, + ) + index_statements = ( + *CREATE_COMMAND_RECEIPT_INDEXES, + *CREATE_WORKER_BINDING_INDEXES, + CREATE_WORKER_BINDING_UNIQUE_INDEX, + *CREATE_CURRENT_PR6_INDEXES, + *CREATE_TURN_LIST_INDEXES, + *CREATE_TURN_LIST_SEQUENCE_TRIGGERS, + *CREATE_TURN_CHANGE_INDEXES, + *CREATE_TURN_CHANGE_TRIGGERS, + *CREATE_TURN_SUBMISSION_INDEXES, + *CREATE_TURN_SUPERSESSION_INDEXES, + *CREATE_AGENT_EVENT_INDEXES, + *CREATE_ATTENTION_LIFECYCLE_INDEXES, + *CREATE_TURN_CONTENT_REVISION_INDEXES, + *CREATE_TURN_PRESENTATION_INDEXES, + *CREATE_FINAL_DELIVERY_INDEXES, + CREATE_CONNECTOR_ORDERING_INDEX, + *CREATE_SNAPSHOT_INDEXES, + ) + for statement in (*table_statements, *index_statements): + conn.execute(statement) + conn.execute(INSERT_STORE_MAINTENANCE_STATE) conn.execute( - "ALTER TABLE turn_presentation_plans_v7 RENAME TO turn_presentation_plans" + "INSERT INTO turn_list_state(scope, store_epoch) VALUES (?, ?)", + ("turn-list", secrets.token_urlsafe(32)), ) conn.execute( - "ALTER TABLE turn_presentation_jobs_v7 RENAME TO turn_presentation_jobs" + "INSERT INTO turn_change_state(scope, store_epoch) VALUES (?, ?)", + ("turn-delta", secrets.token_urlsafe(32)), ) + conn.execute(f"PRAGMA user_version = {STORE_SCHEMA_VERSION}") -def _migrate_v6_to_v7_conn(conn: sqlite3.Connection) -> None: - """Add explicit failed-plan generations and immutable recovery audit.""" - conn.execute(CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE) - _backfill_missing_turn_content_revisions_conn(conn) - _backfill_missing_turn_content_page_boundaries_conn(conn) - plan_columns = { - str(row[1]) - for row in conn.execute( - "PRAGMA table_info(turn_presentation_plans)" - ).fetchall() - } - if "generation" not in plan_columns: - _rebuild_v6_presentation_plans_conn(conn) - conn.execute(CREATE_TURN_PRESENTATION_RECOVERIES_TABLE) - for statement in CREATE_TURN_PRESENTATION_INDEXES: - conn.execute(statement) +def _create_current_schema_conn(conn: sqlite3.Connection) -> None: + """Create an empty database directly at the current schema.""" + if conn.in_transaction: + raise StoreSchemaError("schema_rebuild_in_transaction") + conn.execute("BEGIN IMMEDIATE") + try: + _create_current_schema_objects_conn(conn) + conn.commit() + except Exception: + conn.rollback() + raise -def _migrate_v5_to_v6_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_TURN_CONTENT_REVISIONS_TABLE) - conn.execute(CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE) - for statement in CREATE_TURN_CONTENT_REVISION_INDEXES: - conn.execute(statement) - conn.execute(CREATE_TURN_PRESENTATION_PLANS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_JOBS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_RECOVERIES_TABLE) - for statement in CREATE_TURN_PRESENTATION_INDEXES: - conn.execute(statement) - _backfill_legacy_turn_content_conn(conn) - _backfill_missing_turn_content_revisions_conn(conn) +def _application_schema_objects( + conn: sqlite3.Connection, +) -> tuple[tuple[str, str], ...]: + rows = conn.execute( + """ + SELECT type, name + FROM sqlite_master + WHERE name NOT LIKE 'sqlite_%' + AND type IN ('view', 'trigger', 'index', 'table') + ORDER BY CASE type + WHEN 'view' THEN 0 + WHEN 'trigger' THEN 1 + WHEN 'index' THEN 2 + ELSE 3 + END, name + """ + ).fetchall() + return tuple((str(row[0]), str(row[1])) for row in rows) + +def _quote_sqlite_identifier(value: str) -> str: + return '"' + str(value).replace('"', '""') + '"' -def _normalize_snapshot_created_at_v8_conn( + +def _rebuild_current_schema_conn( conn: sqlite3.Connection, + *, + previous_version: int, ) -> None: - """Canonicalize legacy ordering keys before the v8 age index is built.""" - last_id = 0 - while True: - rows = conn.execute( - """ - SELECT id, created_at, payload - FROM snapshots - WHERE id > ? - ORDER BY id - LIMIT 500 - """, - (last_id,), - ).fetchall() - if not rows: - return - updates: list[tuple[str, int]] = [] - for row_id, raw_created_at, raw_payload in rows: - raw_created_at_text = str(raw_created_at) - canonical = _strict_utc_timestamp(raw_created_at_text) - if ( - canonical == _LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE - and not _legacy_snapshot_created_at_is_authoritative( - raw_created_at, - raw_payload, + if conn.in_transaction: + raise StoreSchemaError("schema_rebuild_in_transaction") + objects = _application_schema_objects(conn) + _LOGGER.warning( + "store schema mismatch; discarding and recreating database " + "previous_version=%d target_version=%d discarded_objects=%s", + int(previous_version), + STORE_SCHEMA_VERSION, + ",".join(f"{kind}:{name}" for kind, name in objects) or "none", + ) + conn.execute("PRAGMA foreign_keys=OFF") + try: + conn.execute("BEGIN IMMEDIATE") + try: + for kind, name in objects: + conn.execute( + f"DROP {kind.upper()} IF EXISTS " + f"{_quote_sqlite_identifier(name)}" ) - ): - canonical = None - canonical = canonical or _SNAPSHOT_CREATED_AT_QUARANTINE - if str(raw_created_at) != canonical: - updates.append((canonical, int(row_id))) - last_id = int(row_id) - if updates: - conn.executemany( - "UPDATE snapshots SET created_at = ? WHERE id = ?", - updates, - ) + _create_current_schema_objects_conn(conn) + conn.commit() + except Exception: + conn.rollback() + raise + finally: + conn.execute("PRAGMA foreign_keys=ON") -def _migrate_v7_to_v8_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_STORE_MAINTENANCE_STATE_TABLE) - conn.execute(INSERT_STORE_MAINTENANCE_STATE) - _normalize_snapshot_created_at_v8_conn(conn) - for statement in CREATE_SNAPSHOT_INDEXES: - conn.execute(statement) - for index_name in ( - "idx_snapshots_host_id", - "idx_snapshots_created_at", - "idx_snapshots_content_fingerprint", - "idx_snapshots_host_created_id", - ): - conn.execute(f"DROP INDEX IF EXISTS {index_name}") +def ensure_schema(conn: sqlite3.Connection) -> None: + """Open v28 unchanged or explicitly replace any other schema.""" + version = int(conn.execute("PRAGMA user_version").fetchone()[0]) + if version == STORE_SCHEMA_VERSION: + return + if not isinstance(conn, _ClosingConnection): + raise local_state_error(LocalStateErrorCode.OPERATION_FAILED) + schema_authority = _schema_connection_authority(conn) + authority = ( + nullcontext() + if schema_authority.parent_fd is None + else _filesystem_schema_mutation_authority(conn) + ) + with authority: + version = int(conn.execute("PRAGMA user_version").fetchone()[0]) + if version == STORE_SCHEMA_VERSION: + return + with private_file_creation_umask(): + _configure_persistent_database_conn(conn) + objects = _application_schema_objects(conn) + if version == 0 and not objects: + _create_current_schema_conn(conn) + return + _rebuild_current_schema_conn( + conn, previous_version=version + ) -def _ensure_turn_list_state_conn(conn: sqlite3.Connection) -> str: - conn.execute(CREATE_TURN_LIST_STATE_TABLE) - row = conn.execute( - "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" - ).fetchone() - if row is not None and str(row[0]): - return str(row[0]) - epoch = secrets.token_urlsafe(32) - conn.execute( - """ - INSERT INTO turn_list_state (scope, store_epoch) - VALUES ('turn-list', ?) - ON CONFLICT(scope) DO NOTHING - """, - (epoch,), - ) - row = conn.execute( - "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" - ).fetchone() - if row is None or not str(row[0]): - raise StoreSchemaError("turn_list_state_unavailable") - return str(row[0]) +_ensure_schema = ensure_schema def _turn_list_store_epoch_conn(conn: sqlite3.Connection) -> str: @@ -11683,30 +10260,6 @@ def _turn_list_store_epoch_conn(conn: sqlite3.Connection) -> str: return str(row[0]) -def _ensure_turn_change_state_conn(conn: sqlite3.Connection) -> str: - conn.execute(CREATE_TURN_CHANGE_STATE_TABLE) - row = conn.execute( - "SELECT store_epoch FROM turn_change_state WHERE scope = 'turn-delta'" - ).fetchone() - if row is not None and str(row[0]): - return str(row[0]) - epoch = secrets.token_urlsafe(32) - conn.execute( - """ - INSERT INTO turn_change_state(scope, store_epoch) - VALUES ('turn-delta', ?) - ON CONFLICT(scope) DO NOTHING - """, - (epoch,), - ) - row = conn.execute( - "SELECT store_epoch FROM turn_change_state WHERE scope = 'turn-delta'" - ).fetchone() - if row is None or not str(row[0]): - raise StoreSchemaError("turn_change_state_unavailable") - return str(row[0]) - - def _turn_change_store_epoch_conn(conn: sqlite3.Connection) -> str: row = conn.execute( "SELECT store_epoch FROM turn_change_state WHERE scope = 'turn-delta'" @@ -11716,2258 +10269,63 @@ def _turn_change_store_epoch_conn(conn: sqlite3.Connection) -> str: return str(row[0]) -def _ensure_turn_list_host_states_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_TURN_LIST_HOSTS_TABLE) - conn.execute( - """ - INSERT INTO turn_list_hosts ( - host_id, - next_sequence, - traversal_generation - ) - SELECT host_id, COALESCE(MAX(list_sequence), 0) + 1, 1 - FROM turns - GROUP BY host_id - ON CONFLICT(host_id) DO NOTHING - """ - ) +def init_store( + db_path: Path, +) -> None: + """Initialize the sqlite store at the current schema.""" + with _connect(db_path, prepare=True) as conn: + ensure_schema(conn) -def _migrate_v8_to_v9_conn(conn: sqlite3.Connection) -> None: - columns = _table_columns(conn, "turns") - if "list_sequence" not in columns: - conn.execute( - "ALTER TABLE turns ADD COLUMN list_sequence " - "INTEGER NOT NULL DEFAULT 0" +def _agent_event_from_row(row: tuple[Any, ...]) -> StoredAgentEvent: + private_payload = _json_object(row[15]) + public_payload = _json_object(row[16]) + kind = str(row[3]) + if kind not in AGENT_EVENT_KINDS: + raise StoreSchemaError("invalid_agent_event_kind") + try: + normalized_host = normalize_agent_event_identifier( + row[1], "host_id", required=True ) - conn.execute( - """ - WITH ranked AS ( - SELECT - host_id, - turn_id, - ROW_NUMBER() OVER ( - PARTITION BY host_id - ORDER BY COALESCE(updated_at, observed_at, ''), turn_id - ) AS assigned_sequence - FROM turns + stored_event = AgentEvent( + event_id=str(row[2]), + kind=kind, # type: ignore[arg-type] + source=str(row[4]), + worker_id=str(row[5]), + visibility=str(row[6]), # type: ignore[arg-type] + source_session_id=str(row[7]) if row[7] is not None else None, + source_turn_id=str(row[8]) if row[8] is not None else None, + source_item_id=str(row[9]) if row[9] is not None else None, + source_message_id=str(row[10]) if row[10] is not None else None, + source_event_id=str(row[11]) if row[11] is not None else None, + source_sequence=int(row[12]) if row[12] is not None else None, + observed_at=str(row[13]), + payload_fingerprint=str(row[14]), + payload=private_payload, + public_payload=public_payload, ) - UPDATE turns - SET list_sequence = ( - SELECT assigned_sequence - FROM ranked - WHERE ranked.host_id = turns.host_id - AND ranked.turn_id = turns.turn_id + canonical = agent_event( + kind=stored_event.kind, + source=stored_event.source, + worker_id=stored_event.worker_id, + payload=stored_event.payload, + source_session_id=stored_event.source_session_id, + source_turn_id=stored_event.source_turn_id, + source_item_id=stored_event.source_item_id, + source_message_id=stored_event.source_message_id, + source_event_id=stored_event.source_event_id, + source_sequence=stored_event.source_sequence, + visibility=stored_event.visibility, + observed_at=stored_event.observed_at, ) - WHERE list_sequence <= 0 - """ - ) - for statement in CREATE_TURN_LIST_INDEXES: - conn.execute(statement) - _ensure_turn_list_state_conn(conn) - _ensure_turn_list_host_states_conn(conn) - - -def _migrate_v9_to_v10_conn(conn: sqlite3.Connection) -> None: - """Add explicit pending freshness, private routing, and two-phase claims.""" - conn.execute(CREATE_LEGACY_BACKEND_PENDING_TABLE) - columns = _table_columns(conn, "backend_pending") - additions = ( - ("revision_digest", "TEXT NOT NULL DEFAULT ''"), - ("choice_routes_json", "TEXT NOT NULL DEFAULT '{}'"), - ("binding_private_fingerprint", "TEXT NOT NULL DEFAULT ''"), - ("observed_turn_target_value", "TEXT NOT NULL DEFAULT ''"), - ("observation_state", "TEXT NOT NULL DEFAULT 'open'"), - ("freshness", "TEXT NOT NULL DEFAULT 'fresh'"), - ("last_success_at", "TEXT"), - ("last_failure_at", "TEXT"), - ("grace_deadline", "TEXT"), - ("updated_at", "TEXT NOT NULL DEFAULT ''"), - ) - for name, declaration in additions: - if name not in columns: - conn.execute( - f"ALTER TABLE backend_pending ADD COLUMN {name} {declaration}" - ) - rows = conn.execute( - """ - SELECT host_id, worker_id, payload_json, observed_at, - revision_digest, last_success_at, updated_at - FROM backend_pending - """ - ).fetchall() - for ( - host_id, - worker_id, - payload_json, - observed_at, - revision_digest, - last_success_at, - updated_at, - ) in rows: - timestamp = _strict_utc_timestamp(observed_at) or "1970-01-01T00:00:00+00:00" - digest = str(revision_digest or "") or stable_fingerprint( - {"legacy_backend_pending": str(payload_json)} - ) - conn.execute( - """ - UPDATE backend_pending - SET revision_digest = ?, - freshness = 'fresh', - last_success_at = ?, - updated_at = ? - WHERE host_id = ? AND worker_id = ? - """, - ( - digest, - str(last_success_at or timestamp), - str(updated_at or timestamp), - str(host_id), - str(worker_id), - ), - ) - conn.execute(CREATE_BACKEND_PENDING_CLAIMS_TABLE) - - -def _migration_plan_has_exact_coverage_conn( - conn: sqlite3.Connection, - *, - plan_id: int, -) -> bool: - plan = conn.execute( - """ - SELECT - host_id, name, turn_id, content_revision, - presentation_version, generation, part_count - FROM turn_presentation_plans - WHERE id = ? - """, - (int(plan_id),), - ).fetchone() - if plan is None: - return False - revision_row, revision_error = _current_presentation_revision_conn( - conn, - host_id=str(plan[0]), - turn_id=str(plan[2]), - content_revision_value=str(plan[3]), - ) - if revision_error is not None or revision_row is None: - return False - staged = conn.execute( - """ - WITH effective AS ( - SELECT - jobs.id, - jobs.part_ordinal, - jobs.spans_json, - ROW_NUMBER() OVER ( - PARTITION BY jobs.part_ordinal - ORDER BY lineage.generation DESC, lineage.id DESC, jobs.id DESC - ) AS effective_rank - FROM turn_presentation_plans AS lineage - JOIN turn_presentation_jobs AS jobs - ON jobs.plan_id = lineage.id - WHERE lineage.host_id = ? - AND lineage.name = ? - AND lineage.turn_id = ? - AND lineage.content_revision = ? - AND lineage.presentation_version = ? - AND lineage.generation <= ? - AND lineage.state IN ('completed', 'superseded') - AND jobs.operation = 'upsert' - ) - SELECT id, part_ordinal, spans_json - FROM effective - WHERE effective_rank = 1 - ORDER BY part_ordinal - """, - ( - str(plan[0]), - str(plan[1]), - str(plan[2]), - str(plan[3]), - str(plan[4]), - int(plan[5]), - ), - ).fetchall() - if ( - len(staged) != int(plan[6]) - or [int(row[1]) for row in staged] != list(range(int(plan[6]))) - ): - return False - try: - for row in staged: - spans = json.loads(str(row[2])) - if ( - not isinstance(spans, list) - or _validate_presentation_spans( - spans, - revision_row=revision_row, - ) - is None - ): - return False - return _presentation_exact_coverage(staged, revision_row=revision_row) - except (KeyError, TypeError, ValueError, json.JSONDecodeError): - return False - - -def _migration_plan_route_matches_conn( - conn: sqlite3.Connection, - *, - plan_id: int, - authoritative: Mapping[str, Any], -) -> bool: - rows = conn.execute( - """ - SELECT jobs.outbox_id, outbox.payload_json - FROM turn_presentation_jobs AS jobs - LEFT JOIN connector_outbox AS outbox ON outbox.id = jobs.outbox_id - WHERE jobs.plan_id = ? - ORDER BY jobs.id - """, - (int(plan_id),), - ).fetchall() - if not rows: - return False - expected = { - "schema_version": 2, - "turn_id": str(authoritative.get("turn_id") or ""), - "content_revision": str(authoritative.get("content_revision") or ""), - "final_identity": str(authoritative.get("final_identity") or ""), - "stable_key": str(authoritative.get("stable_key") or ""), - "stable_key_version": 1, - } - for outbox_id, payload_json in rows: - if outbox_id is None: - return False - payload = _json_object(payload_json) - route = payload.get("turn") - if not isinstance(route, Mapping): - return False - actual = { - "schema_version": route.get("schema_version"), - "turn_id": str(route.get("turn_id") or ""), - "content_revision": str(route.get("content_revision") or ""), - "final_identity": str(route.get("final_identity") or ""), - "stable_key": str(route.get("stable_key") or ""), - "stable_key_version": route.get("stable_key_version"), - } - if actual != expected: - return False - return True - - -def _migrate_v10_to_v11_conn(conn: sqlite3.Connection) -> None: - """Add typed final anchors and conservatively classify legacy finals.""" - required_legacy_tables = { - "connector_outbox", - "turn_presentation_plans", - "turn_presentation_jobs", - "turn_presentation_recoveries", - "turn_content_revisions", - } - present_legacy_tables = { - table for table in required_legacy_tables if _table_columns(conn, table) - } - if not present_legacy_tables: - conn.execute(CREATE_SNAPSHOTS_TABLE) - conn.execute(CREATE_LEGACY_COMMAND_RECEIPTS_TABLE) - conn.execute(CREATE_WORKER_BINDINGS_TABLE) - for statement in CREATE_PR6_TABLES: - conn.execute(statement) - conn.execute(CREATE_ATTENTION_LIFECYCLES_TABLE) - conn.execute(CREATE_TURN_CONTENT_REVISIONS_TABLE) - conn.execute(CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_PLANS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_JOBS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_RECOVERIES_TABLE) - conn.execute(CREATE_STORE_MAINTENANCE_STATE_TABLE) - conn.execute(CREATE_STORE_MAINTENANCE_CURSORS_TABLE) - conn.execute(CREATE_TURN_LIST_STATE_TABLE) - conn.execute(CREATE_TURN_LIST_HOSTS_TABLE) - for statements in ( - CREATE_LEGACY_COMMAND_RECEIPT_INDEXES, - CREATE_WORKER_BINDING_INDEXES, - CREATE_PR6_INDEXES, - CREATE_TURN_LIST_INDEXES, - CREATE_ATTENTION_LIFECYCLE_INDEXES, - CREATE_TURN_CONTENT_REVISION_INDEXES, - CREATE_TURN_PRESENTATION_INDEXES, - CREATE_FINAL_DELIVERY_INDEXES, - CREATE_SNAPSHOT_INDEXES, - ): - for statement in statements: - conn.execute(statement) - conn.execute(CREATE_LEGACY_COMMAND_RECEIPT_UNIQUE_INDEX) - conn.execute(CREATE_WORKER_BINDING_UNIQUE_INDEX) - conn.execute(INSERT_STORE_MAINTENANCE_STATE) - _ensure_turn_list_state_conn(conn) - return - if present_legacy_tables != required_legacy_tables: - raise StoreSchemaError("legacy_final_schema_incomplete") - conn.execute(CREATE_STORE_MAINTENANCE_CURSORS_TABLE) - _ensure_columns( - conn, - "connector_outbox", - { - "delivery_kind": "TEXT NOT NULL DEFAULT 'generic'", - "turn_id": "TEXT", - "content_revision": "TEXT", - }, - ) - _ensure_columns( - conn, - "turn_presentation_plans", - { - "source_outbox_id": ( - "INTEGER REFERENCES connector_outbox(id) ON DELETE RESTRICT" - ), - }, - ) - conn.execute( - """ - UPDATE connector_outbox AS outbox - SET delivery_kind = 'final_part', - turn_id = ( - SELECT plans.turn_id - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans ON plans.id = jobs.plan_id - WHERE jobs.outbox_id = outbox.id - AND plans.host_id = outbox.host_id - AND plans.name = outbox.connector - ), - content_revision = ( - SELECT plans.content_revision - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans ON plans.id = jobs.plan_id - WHERE jobs.outbox_id = outbox.id - AND plans.host_id = outbox.host_id - AND plans.name = outbox.connector - ) - WHERE EXISTS ( - SELECT 1 - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans ON plans.id = jobs.plan_id - WHERE jobs.outbox_id = outbox.id - AND plans.host_id = outbox.host_id - AND plans.name = outbox.connector - ) - """ - ) - dangling_recovery = conn.execute( - """ - SELECT 1 - FROM turn_presentation_recoveries AS recovery - LEFT JOIN turn_presentation_plans AS failed - ON failed.id = recovery.failed_plan_id - LEFT JOIN turn_presentation_plans AS recovered - ON recovered.id = recovery.recovered_plan_id - WHERE failed.id IS NULL - OR recovered.id IS NULL - OR failed.id = recovered.id - OR failed.host_id != recovered.host_id - OR failed.name != recovered.name - OR failed.turn_id != recovered.turn_id - OR failed.content_revision != recovered.content_revision - OR failed.presentation_version != recovered.presentation_version - OR recovered.generation <= failed.generation - LIMIT 1 - """ - ).fetchone() - if dangling_recovery is not None: - raise StoreSchemaError("legacy_final_recovery_invalid") - recovery_edges = conn.execute( - """ - SELECT failed_plan_id, recovered_plan_id, created_at - FROM turn_presentation_recoveries - ORDER BY generation DESC, id DESC - """ - ).fetchall() - for failed_plan_id, recovered_plan_id, recovered_at in recovery_edges: - _finalize_recovered_plan_materialization_conn( - conn, - failed_plan_id=int(failed_plan_id), - recovered_plan_id=int(recovered_plan_id), - now=str(recovered_at), - ) - current_finals = conn.execute( - """ - SELECT - revisions.host_id, - revisions.turn_id, - revisions.content_revision, - revisions.created_at, - turns.payload_json, - revisions.user_text, - revisions.assistant_final_text - FROM turn_content_revisions AS revisions - JOIN turns - ON turns.host_id = revisions.host_id - AND turns.turn_id = revisions.turn_id - WHERE revisions.is_current = 1 - AND revisions.final_state = 'complete' - ORDER BY revisions.host_id, revisions.turn_id - """ - ).fetchall() - for ( - host_id, - turn_id, - revision, - revision_created_at, - turn_payload_json, - revision_user_text, - revision_final_text, - ) in current_finals: - payload = _final_ready_payload_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - content_revision_value=str(revision), - allow_unroutable=True, - ) - if payload is None: - raise StoreSchemaError("legacy_final_descriptor_unavailable") - automation_payload = _json_object(turn_payload_json) - automation_payload["user_text"] = revision_user_text - automation_payload["assistant_final_text"] = revision_final_text - internal_automation = is_internal_automation_turn_payload( - automation_payload - ) - if internal_automation: - conn.execute( - """ - UPDATE connector_outbox - SET delivery_kind = 'final_migration_hold', - status = 'dead_letter', - next_attempt_at = NULL, - updated_at = ? - WHERE id IN ( - SELECT jobs.outbox_id - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans - ON plans.id = jobs.plan_id - WHERE plans.host_id = ? - AND plans.name = ? - AND plans.turn_id = ? - AND plans.content_revision = ? - AND jobs.outbox_id IS NOT NULL - ) - """, - ( - str(revision_created_at), - str(host_id), - _TURN_FINAL_NAME, - str(turn_id), - str(revision), - ), - ) - routable = ( - payload.get("schema_version") == 2 - and not bool(payload["content"]["known_incomplete"]) - and not internal_automation - ) - unresolved = conn.execute( - """ - SELECT id, state - FROM turn_presentation_plans - WHERE host_id = ? - AND name = ? - AND turn_id = ? - AND content_revision = ? - AND state IN ( - 'preparing', - 'waiting_predecessor', - 'active', - 'failed' - ) - AND NOT EXISTS ( - SELECT 1 - FROM turn_presentation_recoveries AS recovery - WHERE recovery.failed_plan_id = turn_presentation_plans.id - ) - ORDER BY id DESC - """, - ( - str(host_id), - _TURN_FINAL_NAME, - str(turn_id), - str(revision), - ), - ).fetchall() - proven = conn.execute( - """ - SELECT plans.id, plans.completed_at - FROM turn_presentation_plans AS plans - WHERE plans.host_id = ? - AND plans.name = ? - AND plans.turn_id = ? - AND plans.content_revision = ? - AND plans.state = 'completed' - AND plans.completed_at IS NOT NULL - AND ( - SELECT COUNT(DISTINCT jobs.part_ordinal) - FROM turn_presentation_plans AS lineage - JOIN turn_presentation_jobs AS jobs - ON jobs.plan_id = lineage.id - WHERE lineage.host_id = plans.host_id - AND lineage.name = plans.name - AND lineage.turn_id = plans.turn_id - AND lineage.content_revision = plans.content_revision - AND lineage.presentation_version = plans.presentation_version - AND lineage.generation <= plans.generation - AND lineage.state IN ('completed', 'superseded') - AND jobs.operation = 'upsert' - ) = plans.part_count - AND NOT EXISTS ( - SELECT 1 - FROM turn_presentation_plans AS lineage - JOIN turn_presentation_jobs AS jobs - ON jobs.plan_id = lineage.id - WHERE lineage.host_id = plans.host_id - AND lineage.name = plans.name - AND lineage.turn_id = plans.turn_id - AND lineage.content_revision = plans.content_revision - AND lineage.presentation_version = plans.presentation_version - AND lineage.generation <= plans.generation - AND lineage.state IN ('completed', 'superseded') - AND jobs.operation = 'upsert' - AND jobs.part_ordinal >= plans.part_count - ) - AND NOT EXISTS ( - SELECT 1 - FROM turn_presentation_plans AS lineage - JOIN turn_presentation_jobs AS jobs - ON jobs.plan_id = lineage.id - LEFT JOIN connector_outbox AS outbox - ON outbox.id = jobs.outbox_id - WHERE lineage.host_id = plans.host_id - AND lineage.name = plans.name - AND lineage.turn_id = plans.turn_id - AND lineage.content_revision = plans.content_revision - AND lineage.presentation_version = plans.presentation_version - AND lineage.generation <= plans.generation - AND lineage.state IN ('completed', 'superseded') - AND ( - outbox.id IS NULL - OR outbox.host_id != plans.host_id - OR outbox.connector != plans.name - OR outbox.turn_id != plans.turn_id - OR outbox.content_revision != plans.content_revision - OR outbox.delivery_kind != 'final_part' - OR outbox.status != 'delivered' - OR NOT EXISTS ( - SELECT 1 - FROM connector_deliveries AS delivered_attempt - WHERE delivered_attempt.outbox_id = outbox.id - AND delivered_attempt.host_id = outbox.host_id - AND delivered_attempt.connector = outbox.connector - AND delivered_attempt.delivery_key = outbox.delivery_key - AND delivered_attempt.status = 'delivered' - AND delivered_attempt.delivered_at IS NOT NULL - ) - ) - ) - ORDER BY plans.id DESC - LIMIT 1 - """, - ( - str(host_id), - _TURN_FINAL_NAME, - str(turn_id), - str(revision), - ), - ).fetchone() - if ( - proven is not None - and not _migration_plan_has_exact_coverage_conn( - conn, - plan_id=int(proven[0]), - ) - ): - proven = None - linkable = [ - int(row[0]) - for row in unresolved - if str(row[1]) in {"waiting_predecessor", "active", "failed"} - and _migration_plan_route_matches_conn( - conn, - plan_id=int(row[0]), - authoritative=payload, - ) - ] - if not routable: - linkable = [] - delivery_kind = "final_migration_hold" - status = "dead_letter" - classified_at = str(revision_created_at) - elif unresolved: - delivery_kind = "final_ready" if linkable else "final_migration_hold" - status = "awaiting_ack" if linkable else "dead_letter" - classified_at = str(revision_created_at) - elif proven is not None: - delivery_kind = "final_ready" - status = "delivered" - classified_at = str(proven[1]) - else: - delivery_kind = "final_migration_hold" - status = "dead_letter" - classified_at = str(revision_created_at) - final_identity = str(payload["final_identity"]) - delivery_key = f"{_TURN_FINAL_NAME}:revision:{final_identity}" - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, - connector, - delivery_key, - delivery_kind, - turn_id, - content_revision, - status, - payload_json, - private_state_json, - created_at, - updated_at, - next_attempt_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, '{}', ?, ?, NULL) - ON CONFLICT(host_id, connector, delivery_key) DO UPDATE SET - delivery_kind = excluded.delivery_kind, - turn_id = excluded.turn_id, - content_revision = excluded.content_revision, - status = excluded.status, - payload_json = excluded.payload_json, - updated_at = excluded.updated_at, - next_attempt_at = NULL - """, - ( - str(host_id), - _TURN_FINAL_NAME, - delivery_key, - delivery_kind, - str(turn_id), - str(revision), - status, - _canonical_json(payload), - str(revision_created_at), - classified_at, - ), - ) - source_row = conn.execute( - """ - SELECT id - FROM connector_outbox - WHERE host_id = ? AND connector = ? AND delivery_key = ? - """, - (str(host_id), _TURN_FINAL_NAME, delivery_key), - ).fetchone() - if source_row is None: - raise StoreSchemaError("legacy_final_anchor_unavailable") - source_outbox_id = int(source_row[0]) - if linkable: - placeholders = ",".join("?" for _ in linkable) - conn.execute( - f""" - UPDATE turn_presentation_plans - SET source_outbox_id = ? - WHERE id IN ({placeholders}) - """, - (source_outbox_id, *linkable), - ) - elif routable and proven is not None: - conn.execute( - """ - UPDATE turn_presentation_plans - SET source_outbox_id = ? - WHERE id = ? - """, - (source_outbox_id, int(proven[0])), - ) - for statement in CREATE_FINAL_DELIVERY_INDEXES: - conn.execute(statement) - - -def _legacy_command_timestamp( - values: Iterable[Any], - *, - latest: bool, -) -> str: - candidates = sorted(str(value) for value in values if str(value or "").strip()) - if not candidates: - return "1970-01-01T00:00:00+00:00" - return candidates[-1] if latest else candidates[0] - - -def _legacy_public_worker_id(request_json: Any) -> str: - request = _json_object(request_json) - target = request.get("target") - if not isinstance(target, Mapping): - return "" - return str(target.get("worker_id") or "") - - -def _migrate_v11_to_v12_conn(conn: sqlite3.Connection) -> None: - """Rebuild action-scoped legacy rows into one fail-closed host request.""" - receipt_columns = _table_columns(conn, "command_receipts") - command_columns = _table_columns(conn, "commands") - current_receipt_columns = { - "canonical_version", - "canonical_fingerprint", - "canonical_request_json", - "public_worker_id", - "state", - "owner_token_hash", - "owner_expires_at", - "binding_fingerprint", - "reserved_at", - "send_started_at", - "terminal_at", - "updated_at", - "legacy_collision", - "legacy_collision_count", - } - current_command_columns = { - "canonical_version", - "canonical_fingerprint", - "public_worker_id", - "state", - "send_started_at", - "terminal_at", - "legacy_collision", - "legacy_collision_count", - } - if current_receipt_columns <= receipt_columns: - if not current_command_columns <= command_columns: - raise StoreSchemaError("legacy_command_request_schema_ambiguous") - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - for statement in CREATE_COMMAND_INDEXES: - conn.execute(statement) - return - required_receipt_columns = { - "id", - "host_id", - "request_id", - "action", - "payload_fingerprint", - "status", - "result_json", - "created_at", - "completed_at", - "uncertain", - } - required_command_columns = { - "id", - "host_id", - "request_id", - "action", - "payload_fingerprint", - "status", - "uncertain", - "request_json", - "result_json", - "created_at", - "reserved_at", - "completed_at", - "updated_at", - } - if ( - not required_receipt_columns <= receipt_columns - or not required_command_columns <= command_columns - ): - raise StoreSchemaError("legacy_command_request_schema_ambiguous") - conflicting_tables = { - str(row[0]) - for row in conn.execute( - """ - SELECT name - FROM sqlite_master - WHERE type = 'table' - AND name IN ('command_receipts_v11', 'commands_v11') - """ - ).fetchall() - } - if conflicting_tables: - raise StoreSchemaError("legacy_command_request_schema_ambiguous") - - grouped: dict[tuple[str, str], list[dict[str, Any]]] = {} - receipt_rows = conn.execute( - """ - SELECT - id, host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - FROM command_receipts - WHERE TRIM(host_id) <> '' AND TRIM(request_id) <> '' - ORDER BY host_id, request_id, id - """ - ).fetchall() - for row in receipt_rows: - key = (str(row[1]), str(row[2])) - grouped.setdefault(key, []).append( - { - "source": "receipt", - "id": int(row[0]), - "action": str(row[3]), - "fingerprint": str(row[4]), - "status": str(row[5]), - "result_json": str(row[6]), - "created_at": str(row[7]), - "terminal_at": row[8], - "updated_at": row[8] or row[7], - "uncertain": bool(row[9]), - "public_worker_id": "", - } - ) - command_rows = conn.execute( - """ - SELECT - id, host_id, request_id, action, payload_fingerprint, status, - result_json, request_json, created_at, completed_at, updated_at, - uncertain - FROM commands - WHERE TRIM(host_id) <> '' AND TRIM(request_id) <> '' - ORDER BY host_id, request_id, id - """ - ).fetchall() - for row in command_rows: - key = (str(row[1]), str(row[2])) - grouped.setdefault(key, []).append( - { - "source": "command", - "id": int(row[0]), - "action": str(row[3]), - "fingerprint": str(row[4]), - "status": str(row[5]), - "result_json": str(row[6]), - "created_at": str(row[8]), - "terminal_at": row[9], - "updated_at": row[10] or row[9] or row[8], - "uncertain": bool(row[11]), - "public_worker_id": _legacy_public_worker_id(row[7]), - } - ) - - normalized: list[tuple[Any, ...]] = [] - for (host_id, request_id), rows in sorted(grouped.items()): - rows.sort(key=lambda item: (str(item["source"]), int(item["id"]))) - pairs = { - (str(item["action"]), str(item["fingerprint"])) - for item in rows - } - evidence = { - ( - str(item["status"]), - str(item["result_json"]), - bool(item["uncertain"]), - ) - for item in rows - } - public_worker_ids = { - str(item["public_worker_id"]) - for item in rows - if str(item["public_worker_id"]) - } - malformed = any( - not str(item["action"]).strip() - or not str(item["fingerprint"]).strip() - for item in rows - ) - collision = ( - malformed - or len(pairs) != 1 - or len(evidence) != 1 - or len(public_worker_ids) > 1 - ) - created_at = _legacy_command_timestamp( - (item["created_at"] for item in rows), - latest=False, - ) - terminal_at = _legacy_command_timestamp( - ( - item["terminal_at"] or item["updated_at"] or item["created_at"] - for item in rows - ), - latest=True, - ) - if collision: - action = "legacy_collision" - fingerprint = "legacy-collision" - state = "uncertain" - status = "request_state_uncertain" - result_json = ( - '{"ok":false,"status":"request_state_uncertain"}' - ) - collision_count = max(2, len(rows)) - public_worker_id = "" - else: - action, fingerprint = next(iter(pairs)) - first = rows[0] - legacy_status = str(first["status"]) - result_json = str(first["result_json"]) - uncertain = any(bool(item["uncertain"]) for item in rows) - if ( - uncertain - or legacy_status in {"pending", "request_state_uncertain"} - ): - state = "uncertain" - status = "request_state_uncertain" - elif legacy_status == "accepted": - state = "accepted" - status = "accepted" - else: - state = "rejected" - status = legacy_status or "legacy_rejected" - collision_count = 0 - public_worker_id = ( - next(iter(public_worker_ids)) if public_worker_ids else "" - ) - send_started_at = created_at if state == "accepted" else None - normalized.append( - ( - host_id, - request_id, - action, - 0, - fingerprint, - "{}", - public_worker_id, - state, - status, - result_json, - created_at, - created_at, - send_started_at, - terminal_at, - terminal_at, - int(collision), - collision_count, - ) - ) - - conn.execute("ALTER TABLE command_receipts RENAME TO command_receipts_v11") - conn.execute("ALTER TABLE commands RENAME TO commands_v11") - conn.execute(CREATE_COMMAND_RECEIPTS_TABLE) - conn.execute(CREATE_COMMANDS_TABLE) - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - for statement in CREATE_COMMAND_INDEXES: - conn.execute(statement) - for record in normalized: - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at, legacy_collision, - legacy_collision_count - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', NULL, NULL, ?, ?, ?, ?, ?, ?, ?) - """, - record, - ) - row = _command_request_row(conn, str(record[0]), str(record[1])) - if row is None: - raise StoreSchemaError("legacy_command_request_migration_failed") - _project_command_request_conn(conn, row) - conn.execute("DROP TABLE command_receipts_v11") - conn.execute("DROP TABLE commands_v11") - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - for statement in CREATE_COMMAND_INDEXES: - conn.execute(statement) - - -def _migrate_v12_to_v13_conn(conn: sqlite3.Connection) -> None: - """Add private selector-proof evidence without inventing it for old rows. - - A v12 receipt records the worker a request resolved to, never how the caller - spelled that target, so no existing row's selector can be reconstructed. Any - guess here would let a changed target replay an unrelated accepted result. - Every legacy row therefore keeps an empty proof, which the submission path - reads as "cannot prove an alias retry" and fails closed on. - """ - columns = _table_columns(conn, "command_receipts") - if not columns: - raise StoreSchemaError("legacy_command_request_schema_ambiguous") - if "selector_proof" not in columns: - conn.execute( - "ALTER TABLE command_receipts " - "ADD COLUMN selector_proof TEXT NOT NULL DEFAULT ''" - ) - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - - -def _migrate_v13_to_v14_conn(conn: sqlite3.Connection) -> None: - """Repair legacy nonpositive turn-list coordinates and reject recurrence.""" - columns = _table_columns(conn, "turns") - if not columns: - return - if "list_sequence" not in columns: - raise StoreSchemaError("legacy_turn_list_schema_ambiguous") - affected_hosts = [ - str(row[0]) - for row in conn.execute( - """ - SELECT DISTINCT host_id - FROM turns - WHERE list_sequence <= 0 - ORDER BY host_id - """ - ).fetchall() - ] - for host_id in affected_hosts: - _ensure_turn_list_host_state_conn(conn, host_id) - state = conn.execute( - """ - SELECT next_sequence - FROM turn_list_hosts - WHERE host_id = ? - """, - (host_id,), - ).fetchone() - if state is None: - raise StoreSchemaError("turn_list_host_state_unavailable") - high_row = conn.execute( - """ - SELECT COALESCE(MAX(list_sequence), 0) - FROM turns - WHERE host_id = ? AND list_sequence > 0 - """, - (host_id,), - ).fetchone() - next_sequence = max(int(state[0]), int(high_row[0]) + 1) - invalid_rows = conn.execute( - """ - SELECT turn_id - FROM turns - WHERE host_id = ? AND list_sequence <= 0 - ORDER BY COALESCE(updated_at, observed_at, ''), turn_id - """, - (host_id,), - ).fetchall() - for row in invalid_rows: - conn.execute( - """ - UPDATE turns - SET list_sequence = ? - WHERE host_id = ? AND turn_id = ? AND list_sequence <= 0 - """, - (next_sequence, host_id, str(row[0])), - ) - next_sequence += 1 - conn.execute( - """ - UPDATE turn_list_hosts - SET next_sequence = ?, - traversal_generation = traversal_generation + 1 - WHERE host_id = ? - """, - (next_sequence, host_id), - ) - for statement in CREATE_TURN_LIST_SEQUENCE_TRIGGERS: - conn.execute(statement) - - -def _migrate_v14_to_v15_conn(conn: sqlite3.Connection) -> None: - """Tombstone duplicate and stale command claims without rekeying turns.""" - if not _table_columns(conn, "turns"): - return - now = utc_timestamp() - now_dt = datetime.fromisoformat(now) - rows = conn.execute( - """ - SELECT turns.host_id, turns.turn_id, turns.worker_id, - turns.payload_json, turns.observed_at, - revisions.user_text, revisions.assistant_final_text, - revisions.user_state, revisions.final_state - FROM turns - LEFT JOIN turn_content_revisions AS revisions - ON revisions.host_id = turns.host_id - AND revisions.turn_id = turns.turn_id - AND revisions.is_current = 1 - """ - ).fetchall() - decoded = [] - for row in rows: - payload = _json_object(row[3]) - current = ( - { - "user_text": row[5], - "assistant_final_text": row[6], - "user_state": str(row[7]), - "final_state": str(row[8]), - } - if row[7] is not None - else None - ) - decoded.append( - ( - str(row[0]), - str(row[1]), - str(row[2]), - payload, - current, - str(row[4] or ""), - ) - ) - claims = [ - row - for row in decoded - if str(row[3].get("source") or "") == "command" - and not str(row[3].get("source_turn_id") or "").strip() - and not _turn_is_tombstoned(row[3]) - and row[3].get("complete") is not True - ] - done = [ - row - for row in decoded - if str(row[3].get("source_turn_id") or "").strip() - and not _turn_is_tombstoned(row[3]) - and ( - row[3].get("complete") is True - or row[4] is not None - and str(row[4].get("final_state") or "") == "complete" - ) - ] - affected_hosts: set[str] = set() - used_done = { - str(row[3].get("superseded_by_turn_id") or "") - for row in decoded - if _turn_is_tombstoned(row[3]) - and str(row[3].get("superseded_by_turn_id") or "").strip() - } - for claim in claims: - claim_view = _turn_with_current_content(claim[3], claim[4]) - matches = [ - observed - for observed in done - if observed[0] == claim[0] - and observed[2] == claim[2] - and observed[1] not in used_done - and _turn_content_matches_origin( - _turn_with_current_content(observed[3], observed[4]), - claim_view, - ) - ] - if len(matches) != 1: - continue - matching_claims = [ - candidate - for candidate in claims - if candidate[0] == claim[0] - and candidate[2] == claim[2] - and _turn_content_matches_origin( - _turn_with_current_content(matches[0][3], matches[0][4]), - _turn_with_current_content(candidate[3], candidate[4]), - ) - ] - if len(matching_claims) != 1: - continue - if _migrate_tombstone_command_turn_conn( - conn, - claim[0], - claim[1], - superseded_by_turn_id=matches[0][1], - superseded_at=now, - ): - affected_hosts.add(claim[0]) - used_done.add(matches[0][1]) - - configured_hard_ttl = _LEGACY_TURN_CLAIM_HARD_TTL_SECONDS - for claim in claims: - stored = conn.execute( - "SELECT payload_json FROM turns WHERE host_id = ? AND turn_id = ?", - (claim[0], claim[1]), - ).fetchone() - if stored is None or _turn_is_tombstoned(_json_object(stored[0])): - continue - claim_dt = _turn_row_time(claim[3], claim[5]) - if claim_dt is None or (now_dt - claim_dt).total_seconds() < configured_hard_ttl: - continue - if _migrate_tombstone_command_turn_conn( - conn, - claim[0], - claim[1], - superseded_by_turn_id=None, - superseded_at=now, - ): - affected_hosts.add(claim[0]) - for host_id in sorted(affected_hosts): - _increment_turn_list_generation_conn(conn, host_id) - - -def _legacy_outbox_ordering_key( - *, - outbox_id: int, - outbox_payload: Mapping[str, Any], - worker_id: Any, - turn_payload: Mapping[str, Any], -) -> str: - nested_turn = outbox_payload.get("turn") - route = dict(nested_turn) if isinstance(nested_turn, Mapping) else outbox_payload - route_meta = _json_object(route.get("meta")) - route_stable_key = route.get("stable_key") or route_meta.get("stable_key") - route_stable_key_version = ( - route.get("stable_key_version") - if route.get("stable_key") is not None - else route_meta.get("stable_key_version") - ) - if ( - _valid_final_stable_key(route_stable_key) - and type(route_stable_key_version) is int - and route_stable_key_version == 1 - ): - return str(route_stable_key) - meta = _json_object(turn_payload.get("meta")) - stable_key = meta.get("stable_key") - if ( - _valid_final_stable_key(stable_key) - and type(meta.get("stable_key_version")) is int - and meta.get("stable_key_version") == 1 - ): - return str(stable_key) - return str(worker_id or route.get("worker_id") or f"orphan:{outbox_id}") - - -def _migrate_v15_to_v16_conn( - conn: sqlite3.Connection, - *, - connector_ack_ttl_seconds: int = CONNECTOR_ACK_TTL_SECONDS, -) -> None: - """Partition final FIFO order and bound legacy awaiting-ack plans.""" - columns = _table_columns(conn, "connector_outbox") - if not columns: - conn.execute(CREATE_CONNECTOR_OUTBOX_TABLE) - elif "ordering_key" not in columns: - conn.execute( - "ALTER TABLE connector_outbox " - "ADD COLUMN ordering_key TEXT NOT NULL DEFAULT ''" - ) - if _table_columns(conn, "turns"): - rows = conn.execute( - """ - SELECT outbox.id, outbox.payload_json, turns.worker_id, turns.payload_json - FROM connector_outbox AS outbox - LEFT JOIN turns - ON turns.host_id = outbox.host_id - AND turns.turn_id = outbox.turn_id - WHERE outbox.ordering_key = '' - ORDER BY outbox.id - """ - ).fetchall() - else: - rows = [ - (row[0], row[1], None, None) - for row in conn.execute( - """ - SELECT id, payload_json - FROM connector_outbox - WHERE ordering_key = '' - ORDER BY id - """ - ).fetchall() - ] - for outbox_id, outbox_payload, worker_id, turn_payload in rows: - conn.execute( - "UPDATE connector_outbox SET ordering_key = ? WHERE id = ?", - ( - _legacy_outbox_ordering_key( - outbox_id=int(outbox_id), - outbox_payload=_json_object(outbox_payload), - worker_id=worker_id, - turn_payload=_json_object(turn_payload), - ), - int(outbox_id), - ), - ) - deadline = _connector_add_seconds( - utc_timestamp(), - max(1, int(connector_ack_ttl_seconds)), - ) - awaiting_rows = conn.execute( - "SELECT id, private_state_json FROM connector_outbox WHERE status = 'awaiting_ack'" - ).fetchall() - for outbox_id, private_state_json in awaiting_rows: - state = _json_object(private_state_json) - state["ack_deadline_at"] = deadline - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = ?, next_attempt_at = ? - WHERE id = ? - """, - (_canonical_json(state), deadline, int(outbox_id)), - ) - delivery_rows = ( - conn.execute( - """ - SELECT id, private_state_json - FROM connector_deliveries - WHERE outbox_id = ? AND status = 'awaiting_ack' - """, - (int(outbox_id),), - ).fetchall() - if _table_columns(conn, "connector_deliveries") - else [] - ) - for delivery_id, delivery_private in delivery_rows: - delivery_state = _json_object(delivery_private) - delivery_state["ack_deadline_at"] = deadline - conn.execute( - "UPDATE connector_deliveries SET private_state_json = ? WHERE id = ?", - (_canonical_json(delivery_state), int(delivery_id)), - ) - conn.execute(CREATE_CONNECTOR_ORDERING_INDEX) - - -def _migrate_v16_to_v17_conn(conn: sqlite3.Connection) -> None: - """Give unresolved legacy outbox rows independent FIFO partitions.""" - if _table_columns(conn, "turns"): - rows = conn.execute( - """ - SELECT outbox.id, outbox.payload_json, - turns.worker_id, turns.payload_json - FROM connector_outbox AS outbox - LEFT JOIN turns - ON turns.host_id = outbox.host_id - AND turns.turn_id = outbox.turn_id - WHERE outbox.ordering_key = '' - ORDER BY outbox.id - """ - ).fetchall() - else: - rows = [ - (row[0], row[1], None, None) - for row in conn.execute( - """ - SELECT id, payload_json - FROM connector_outbox - WHERE ordering_key = '' - ORDER BY id - """ - ).fetchall() - ] - for outbox_id, outbox_payload, worker_id, turn_payload in rows: - conn.execute( - "UPDATE connector_outbox SET ordering_key = ? WHERE id = ?", - ( - _legacy_outbox_ordering_key( - outbox_id=int(outbox_id), - outbox_payload=_json_object(outbox_payload), - worker_id=worker_id, - turn_payload=_json_object(turn_payload), - ), - int(outbox_id), - ), - ) - - -def _create_available_turn_change_triggers_conn(conn: sqlite3.Connection) -> None: - """Install capture triggers whose legacy source tables are present.""" - has_turns = bool(_table_columns(conn, "turns")) - has_revisions = bool(_table_columns(conn, "turn_content_revisions")) - for index, statement in enumerate(CREATE_TURN_CHANGE_TRIGGERS): - if index < 3 and not has_turns: - continue - if index in {3, 4} and not (has_turns and has_revisions): - continue - conn.execute(statement) - - -def _migrate_v17_to_v18_conn(conn: sqlite3.Connection) -> None: - """Install the empty, trigger-backed public turn change journal.""" - conn.execute(CREATE_TURN_CHANGE_JOURNAL_TABLE) - conn.execute(CREATE_TURN_CHANGE_FLOOR_TABLE) - conn.execute(CREATE_TURN_CHANGE_STATE_TABLE) - for statement in CREATE_TURN_CHANGE_INDEXES: - conn.execute(statement) - _ensure_turn_change_state_conn(conn) - _create_available_turn_change_triggers_conn(conn) - - -def _migrate_v18_to_v19_conn(conn: sqlite3.Connection) -> None: - """Install empty Phase 2 submission and supersession ledgers.""" - conn.execute(CREATE_TURN_SUBMISSIONS_TABLE) - conn.execute(CREATE_TURN_SUPERSESSIONS_TABLE) - for statement in CREATE_TURN_SUBMISSION_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_SUPERSESSION_INDEXES: - conn.execute(statement) - - -def _backfill_submission_state(receipt_state: Any, receipt_status: Any) -> str | None: - """Map a historical send receipt to the shadow-ledger state it earned.""" - state = str(receipt_state or "").strip().lower() - status = str(receipt_status or "").strip().lower() - if state == "purged" or status == "purged": - return None - if state in {"rejected", "cancelled", "canceled"} or status in { - "rejected", - "cancelled", - "canceled", - }: - return "cancelled" - if state == "send_started": - return "send_started" - if state == "accepted": - return "submitted" - if state == "uncertain": - return "uncertain" - # A reservation has not crossed the send boundary, so Stage 2 would not - # have created a submission row for it. Unknown legacy states fail closed. - return None - - -def _receipt_instruction_text(canonical_request_json: Any) -> str | None: - """Recover only a validated canonical send-instruction payload.""" - try: - payload = json.loads(str(canonical_request_json)) - except (TypeError, json.JSONDecodeError): - return None - if not isinstance(payload, Mapping) or payload.get("action") != "send_instruction": - return None - instruction = payload.get("instruction") - if not isinstance(instruction, Mapping): - return None - text = instruction.get("text") - if not isinstance(text, str) or validate_instruction_text(text) is not None: - return None - return text - - -def _backfill_turn_submissions_conn(conn: sqlite3.Connection) -> None: - """Backfill historical send receipts without changing live dual-write rows.""" - required_columns = { - "id", - "host_id", - "request_id", - "action", - "canonical_request_json", - "public_worker_id", - "state", - "status", - "created_at", - "reserved_at", - "send_started_at", - "terminal_at", - "updated_at", - } - if not required_columns <= _table_columns(conn, "command_receipts"): - return - hosts = [ - str(row[0]) - for row in conn.execute( - """ - SELECT DISTINCT host_id - FROM command_receipts - WHERE action = 'send_instruction' AND TRIM(host_id) <> '' - ORDER BY host_id - """ - ).fetchall() - ] - for host_id in hosts: - after_id = 0 - while True: - rows = conn.execute( - """ - SELECT id, request_id, canonical_request_json, - public_worker_id, state, status, created_at, - reserved_at, send_started_at, terminal_at, updated_at - FROM command_receipts - WHERE host_id = ? AND action = 'send_instruction' AND id > ? - ORDER BY id - LIMIT ? - """, - (host_id, after_id, TURN_LEDGER_BACKFILL_BATCH_SIZE), - ).fetchall() - if not rows: - break - after_id = int(rows[-1][0]) - for row in rows: - request_id = str(row[1] or "").strip() - public_worker_id = str(row[3] or "").strip() - ledger_state = _backfill_submission_state(row[4], row[5]) - instruction_text = _receipt_instruction_text(row[2]) - if ( - not request_id - or not public_worker_id - or ledger_state is None - or instruction_text is None - ): - continue - - anchor = _strict_utc_timestamp(row[8] or row[7] or row[6]) - updated_at = _strict_utc_timestamp(row[10]) - if anchor is None or updated_at is None: - continue - anchor_time = datetime.fromisoformat(anchor) - link_not_before = ( - anchor_time - - timedelta(seconds=SUBMISSION_LINK_WINDOW_SECONDS) - ).isoformat(timespec="seconds") - link_expires_at = ( - anchor_time - + timedelta(seconds=SUBMISSION_LINK_WINDOW_SECONDS) - ).isoformat(timespec="seconds") - hard_expires_at = ( - anchor_time - + timedelta(seconds=SUBMISSION_HARD_TTL_SECONDS) - ).isoformat(timespec="seconds") - terminal_at = ( - _strict_utc_timestamp(row[9]) - if ledger_state in {"submitted", "uncertain", "cancelled"} - else None - ) - if ( - ledger_state in {"submitted", "uncertain", "cancelled"} - and terminal_at is None - ): - continue - send_started_at = ( - _strict_utc_timestamp(row[8]) if row[8] is not None else None - ) - conn.execute( - """ - INSERT INTO turn_submissions ( - host_id, submission_id, request_id, owner_key, - owner_key_version, instruction_fingerprint, state, - linked_turn_id, link_not_before, link_expires_at, - hard_expires_at, linked_at, terminal_at, submitted_at, - send_started_at, updated_at - ) VALUES ( - ?, ?, ?, ?, 0, ?, ?, NULL, ?, ?, ?, NULL, ?, ?, ?, ? - ) - ON CONFLICT DO NOTHING - """, - ( - host_id, - turn_submission_id(host_id, request_id), - request_id, - f"legacy-worker:{public_worker_id}", - instruction_fingerprint(instruction_text), - ledger_state, - link_not_before, - link_expires_at, - hard_expires_at, - terminal_at, - terminal_at if ledger_state == "submitted" else None, - send_started_at, - updated_at, - ), - ) - - -def _linked_canonical_turn_id_conn( - conn: sqlite3.Connection, - host_id: str, - replacement_turn_id: Any, -) -> str | None: - """Follow only explicit tombstone links to a source-observed identity.""" - current = str(replacement_turn_id or "").strip() - seen: set[str] = set() - while current and current not in seen: - seen.add(current) - row = conn.execute( - """ - SELECT payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (str(host_id), current), - ).fetchone() - if row is None: - return None - payload = _json_object(row[0]) - if _turn_is_tombstoned(payload): - current = str(payload.get("superseded_by_turn_id") or "").strip() - continue - if not str(payload.get("source_turn_id") or "").strip(): - return None - # Phase 1 freezes an adopted command row's published turn_id instead - # of re-keying it after source_turn_id is learned. The row we just - # resolved is therefore the only canonical identity we can prove. - return current - return None - - -def _resolve_canonical_turn_id_conn( - conn: sqlite3.Connection, - host_id: str, - turn_id: Any, -) -> str | None: - """Resolve a public legacy turn alias without guessing through bad rows.""" - current = str(turn_id or "").strip() - if not current: - return None - if not { - "host_id", - "superseded_turn_id", - "canonical_turn_id", - } <= _table_columns(conn, "turn_supersessions"): - return current - seen: set[str] = set() - while current and current not in seen: - seen.add(current) - row = conn.execute( - """ - SELECT canonical_turn_id - FROM turn_supersessions - WHERE host_id = ? AND superseded_turn_id = ? - """, - (str(host_id), current), - ).fetchone() - if row is None: - return current - current = str(row[0] or "").strip() - return None - - -def _backfill_turn_supersessions_conn(conn: sqlite3.Connection) -> None: - """Alias only legacy command turns with deterministic Phase 1 linkage.""" - if not {"host_id", "turn_id", "payload_json", "observed_at", "list_sequence"} <= ( - _table_columns(conn, "turns") - ): - return - hosts = [ - str(row[0]) - for row in conn.execute( - """ - SELECT DISTINCT host_id - FROM turns - WHERE json_extract(payload_json, '$.source') = 'command' - ORDER BY host_id - """ - ).fetchall() - ] - for host_id in hosts: - after_sequence = 0 - while True: - rows = conn.execute( - """ - SELECT turn_id, payload_json, observed_at, list_sequence - FROM turns - WHERE host_id = ? - AND list_sequence > ? - AND json_extract(payload_json, '$.source') = 'command' - ORDER BY list_sequence - LIMIT ? - """, - (host_id, after_sequence, TURN_LEDGER_BACKFILL_BATCH_SIZE), - ).fetchall() - if not rows: - break - after_sequence = int(rows[-1][3]) - for turn_id, payload_json, observed_at, _sequence in rows: - legacy_turn_id = str(turn_id) - payload = _json_object(payload_json) - # A live adopted command turn was never superseded. Its row ID - # is deliberately frozen by Phase 1, so recomputing a Turn ID - # from its updated payload could only invent a dangling alias. - if not _turn_is_tombstoned(payload): - continue - canonical_turn_id = _linked_canonical_turn_id_conn( - conn, - host_id, - payload.get("superseded_by_turn_id"), - ) - if not canonical_turn_id or canonical_turn_id == legacy_turn_id: - continue - created_at = _strict_utc_timestamp( - payload.get("superseded_at") - or payload.get("updated_at") - or observed_at - ) - if created_at is None: - continue - conn.execute( - """ - INSERT INTO turn_supersessions ( - host_id, superseded_turn_id, canonical_turn_id, - reason, created_at - ) VALUES (?, ?, ?, 'phase1_migration', ?) - ON CONFLICT DO NOTHING - """, - ( - host_id, - legacy_turn_id, - canonical_turn_id, - created_at, - ), - ) - - -def _migrate_v19_to_v20_conn(conn: sqlite3.Connection) -> None: - """Backfill Phase 1 history into the non-authoritative Phase 2 ledgers.""" - # The pre-Phase-2 production lineage also used user_version 19, but did - # not contain either ledger table. Repair that valid legacy shape before - # backfilling instead of assuming every v19 store passed through this - # branch's v18 -> v19 migration. - conn.execute(CREATE_TURN_SUBMISSIONS_TABLE) - conn.execute(CREATE_TURN_SUPERSESSIONS_TABLE) - for statement in CREATE_TURN_SUBMISSION_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_SUPERSESSION_INDEXES: - conn.execute(statement) - _backfill_turn_submissions_conn(conn) - _backfill_turn_supersessions_conn(conn) - - -def _migrate_v20_to_v21_conn(conn: sqlite3.Connection) -> None: - """Add durable Herdr turn replay watermarks and completion provenance.""" - conn.execute(CREATE_HERDR_TURN_WATERMARKS_TABLE) - conn.execute(CREATE_HERDR_TURN_COMPLETIONS_TABLE) - for statement in CREATE_HERDR_TURN_INDEXES: - conn.execute(statement) - - -def _migrate_v21_to_v22_conn(conn: sqlite3.Connection) -> None: - """Add the append-only structured agent-event journal.""" - conn.execute(CREATE_AGENT_EVENTS_TABLE) - for statement in CREATE_AGENT_EVENT_INDEXES: - conn.execute(statement) - - -def _migrate_v22_to_v23_conn(conn: sqlite3.Connection) -> None: - """Harden event identity and rebuild v22 rows under the canonical contract.""" - columns = [ - "sequence", - "host_id", - "event_id", - "kind", - "source", - "worker_id", - "visibility", - "source_session_id", - "source_turn_id", - "source_item_id", - "source_message_id", - "source_event_id", - "source_sequence", - "observed_at", - "payload_fingerprint", - "private_payload_json", - "public_payload_json", - ] - rows = conn.execute( - "SELECT " + ", ".join(columns) + " FROM agent_events ORDER BY sequence" - ).fetchall() - conn.execute("ALTER TABLE agent_events RENAME TO agent_events_v22") - conn.execute(CREATE_AGENT_EVENTS_TABLE) - try: - for row in rows: - try: - private_payload = _json_object(row[15]) - public_payload = _json_object(row[16]) - if _canonical_json(private_payload) != row[15]: - raise StoreSchemaError("invalid_v22_agent_event_payload") - if _canonical_json(public_payload) != row[16]: - raise StoreSchemaError("invalid_v22_agent_event_projection") - host_id = normalize_agent_event_identifier( - row[1], "host_id", required=True - ) - # v22 allowed tool/plan rows to be public. The current - # constructor and table correctly reject that historical - # shape, so normalize it while it is still in the private - # migration transaction instead of rebuilding through the - # latest DDL and failing before v24 -> v25 can privatise it. - legacy_sensitive = str(row[3]) in { - "thought", - "tool_call", - "tool_call_update", - "plan", - "extension", - } - canonical = agent_event( - kind=row[3], - source=row[4], - worker_id=row[5], - visibility="private" if legacy_sensitive else row[6], - source_session_id=row[7], - source_turn_id=row[8], - source_item_id=row[9], - source_message_id=row[10], - source_event_id=row[11], - source_sequence=row[12], - observed_at=row[13], - payload=private_payload, - ) - legacy_identity = { - "schema_version": 1, - "source": canonical.source, - "session_id": canonical.source_session_id, - "event_id": canonical.source_event_id, - "sequence": canonical.source_sequence, - "kind": canonical.kind, - } - legacy_event_id = hashlib.sha256( - _canonical_json(legacy_identity).encode("utf-8") - ).hexdigest() - if str(row[2]) != legacy_event_id: - raise StoreSchemaError("invalid_v22_agent_event_identity") - if str(row[14]) != canonical.payload_fingerprint: - raise StoreSchemaError("invalid_v22_agent_event_fingerprint") - if not legacy_sensitive and public_payload != canonical.public_payload: - raise StoreSchemaError("invalid_v22_agent_event_projection") - except (TypeError, ValueError, OverflowError) as exc: - raise StoreSchemaError("invalid_v22_agent_event_row") from exc - conn.execute( - """ - INSERT INTO agent_events ( - sequence, host_id, event_id, kind, source, worker_id, - visibility, source_session_id, source_turn_id, - source_item_id, source_message_id, source_event_id, - source_sequence, observed_at, payload_fingerprint, - private_payload_json, public_payload_json - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - int(row[0]), - host_id, - canonical.event_id, - canonical.kind, - canonical.source, - canonical.worker_id, - canonical.visibility, - canonical.source_session_id, - canonical.source_turn_id, - canonical.source_item_id, - canonical.source_message_id, - canonical.source_event_id, - canonical.source_sequence, - canonical.observed_at, - canonical.payload_fingerprint, - _canonical_json(canonical.payload), - _canonical_json(canonical.public_payload), - ), - ) - except sqlite3.IntegrityError as exc: - raise StoreSchemaError("conflicting_v22_agent_event_identity") from exc - conn.execute("DROP TABLE agent_events_v22") - for statement in CREATE_AGENT_EVENT_INDEXES: - conn.execute(statement) - - -def _migrate_v23_to_v24_conn(conn: sqlite3.Connection) -> None: - """Raise private event bounds and add replay-preserving retention tombstones.""" - conn.execute("ALTER TABLE agent_events RENAME TO agent_events_v23") - conn.execute(CREATE_AGENT_EVENTS_TABLE) - conn.execute( - """ - INSERT INTO agent_events ( - sequence, host_id, event_id, kind, source, worker_id, visibility, - source_session_id, source_turn_id, source_item_id, - source_message_id, source_event_id, source_sequence, observed_at, - payload_fingerprint, private_payload_json, public_payload_json - ) - SELECT - sequence, host_id, event_id, kind, source, worker_id, - CASE - WHEN kind IN ('thought', 'tool_call', 'tool_call_update', 'plan', 'extension') - THEN 'private' - ELSE visibility - END, - source_session_id, source_turn_id, source_item_id, - source_message_id, source_event_id, source_sequence, observed_at, - payload_fingerprint, private_payload_json, - CASE - WHEN kind IN ('thought', 'tool_call', 'tool_call_update', 'plan', 'extension') - THEN '{}' - ELSE public_payload_json - END - FROM agent_events_v23 - ORDER BY sequence - """ - ) - conn.execute("DROP TABLE agent_events_v23") - for statement in CREATE_AGENT_EVENT_INDEXES: - conn.execute(statement) - conn.execute(CREATE_AGENT_EVENT_TOMBSTONES_V25_TABLE) - for statement in CREATE_AGENT_EVENT_TOMBSTONE_INDEXES: - conn.execute(statement) - - -def _migrate_v24_to_v25_conn(conn: sqlite3.Connection) -> None: - """Make thought, tool, plan, and extension journal rows private-only.""" - conn.execute("ALTER TABLE agent_events RENAME TO agent_events_v24") - conn.execute(CREATE_AGENT_EVENTS_TABLE) - conn.execute( - """ - INSERT INTO agent_events ( - sequence, host_id, event_id, kind, source, worker_id, visibility, - source_session_id, source_turn_id, source_item_id, - source_message_id, source_event_id, source_sequence, observed_at, - payload_fingerprint, private_payload_json, public_payload_json - ) - SELECT - sequence, host_id, event_id, kind, source, worker_id, - CASE - WHEN kind IN ('thought', 'tool_call', 'tool_call_update', 'plan', 'extension') - THEN 'private' - ELSE visibility - END, - source_session_id, source_turn_id, source_item_id, - source_message_id, source_event_id, source_sequence, observed_at, - payload_fingerprint, private_payload_json, - CASE - WHEN kind IN ('thought', 'tool_call', 'tool_call_update', 'plan', 'extension') - THEN '{}' - ELSE public_payload_json - END - FROM agent_events_v24 - ORDER BY sequence - """ - ) - conn.execute("DROP TABLE agent_events_v24") - for statement in CREATE_AGENT_EVENT_INDEXES: - conn.execute(statement) - - -def _migrate_v25_to_v26_conn(conn: sqlite3.Connection) -> None: - """Retain original event authority time for safe tombstone repair.""" - if "observed_at" in _table_columns(conn, "agent_event_tombstones"): - return - conn.execute( - """ - ALTER TABLE agent_event_tombstones - ADD COLUMN observed_at TEXT - CHECK (observed_at IS NULL OR length(observed_at) BETWEEN 20 AND 40) - """ - ) - - -def _migrate_v26_to_v27_conn(conn: sqlite3.Connection) -> None: - """Persist private pending-decision transport provenance.""" - pending_columns = _table_columns(conn, "backend_pending") - required_pending_columns = { - "revision_digest", - "choice_routes_json", - "binding_private_fingerprint", - "observed_turn_target_value", - "observation_state", - "freshness", - "updated_at", - } - if not required_pending_columns.issubset(pending_columns): - # Some old fixture databases intentionally contain only the v11 - # command tables. Reconstruct this unrelated family defensively. - _migrate_v9_to_v10_conn(conn) - pending_columns = _table_columns(conn, "backend_pending") - if not _table_columns(conn, "backend_pending_claims"): - conn.execute(CREATE_BACKEND_PENDING_CLAIMS_TABLE) - if "route_kind" not in pending_columns: - conn.execute( - "ALTER TABLE backend_pending ADD COLUMN route_kind " - "TEXT NOT NULL DEFAULT 'legacy' " - "CHECK (route_kind IN ('legacy', 'acp_permission'))" - ) - if "route_kind" not in _table_columns(conn, "backend_pending_claims"): - conn.execute( - "ALTER TABLE backend_pending_claims ADD COLUMN route_kind " - "TEXT NOT NULL DEFAULT 'legacy' " - "CHECK (route_kind IN ('legacy', 'acp_permission'))" - ) - - -def _migrate_v27_to_v28_conn(conn: sqlite3.Connection) -> None: - """Persist bounded Herdr completion-refresh retries and escalation.""" - conn.execute(CREATE_HERDR_TURN_REFRESH_RETRIES_TABLE) - for statement in CREATE_HERDR_TURN_REFRESH_RETRY_INDEXES: - conn.execute(statement) - - -MIGRATIONS: tuple[Migration, ...] = ( - Migration(0, 1, _migrate_v0_to_v1_conn), - Migration(1, 2, _migrate_v1_to_v2_conn), - Migration(2, 3, _migrate_v2_to_v3_conn), - Migration(3, 4, _migrate_v3_to_v4_conn), - Migration(4, 5, _migrate_v4_to_v5_conn), - Migration(5, 6, _migrate_v5_to_v6_conn), - Migration(6, 7, _migrate_v6_to_v7_conn), - Migration(7, 8, _migrate_v7_to_v8_conn), - Migration(8, 9, _migrate_v8_to_v9_conn), - Migration(9, 10, _migrate_v9_to_v10_conn), - Migration(10, 11, _migrate_v10_to_v11_conn), - Migration(11, 12, _migrate_v11_to_v12_conn), - Migration(12, 13, _migrate_v12_to_v13_conn), - Migration(13, 14, _migrate_v13_to_v14_conn), - Migration(14, 15, _migrate_v14_to_v15_conn), - Migration(15, 16, _migrate_v15_to_v16_conn), - Migration(16, 17, _migrate_v16_to_v17_conn), - Migration(17, 18, _migrate_v17_to_v18_conn), - Migration(18, 19, _migrate_v18_to_v19_conn), - Migration(19, 20, _migrate_v19_to_v20_conn), - Migration(20, 21, _migrate_v20_to_v21_conn), - Migration(21, 22, _migrate_v21_to_v22_conn), - Migration(22, 23, _migrate_v22_to_v23_conn), - Migration(23, 24, _migrate_v23_to_v24_conn), - Migration(24, 25, _migrate_v24_to_v25_conn), - Migration(25, 26, _migrate_v25_to_v26_conn), - Migration(26, 27, _migrate_v26_to_v27_conn), - Migration(27, 28, _migrate_v27_to_v28_conn), -) - - -def _validate_migration_registry( - migrations: tuple[Migration, ...] | None = None, - *, - target_version: int = STORE_SCHEMA_VERSION, -) -> None: - registry = MIGRATIONS if migrations is None else migrations - expected = 0 - for migration in registry: - if ( - migration.from_version != expected - or migration.to_version != expected + 1 - ): - raise RuntimeError("invalid migration registry") - expected = migration.to_version - if expected != STORE_SCHEMA_VERSION: - raise RuntimeError("invalid migration registry target") - if not 0 <= int(target_version) <= STORE_SCHEMA_VERSION: - raise RuntimeError("unsupported migration target") - - -def _create_current_schema_conn(conn: sqlite3.Connection) -> None: - """Create an empty database directly at the current schema.""" - if conn.in_transaction: - raise StoreSchemaError("schema_migration_in_transaction") - conn.execute("BEGIN IMMEDIATE") - try: - conn.execute(CREATE_SNAPSHOTS_TABLE) - conn.execute(CREATE_COMMAND_RECEIPTS_TABLE) - conn.execute(CREATE_WORKER_BINDINGS_TABLE) - for statement in CREATE_CURRENT_PR6_TABLES: - conn.execute(statement) - conn.execute(CREATE_LEGACY_BACKEND_PENDING_TABLE) - _migrate_v9_to_v10_conn(conn) - _migrate_v26_to_v27_conn(conn) - conn.execute(CREATE_ATTENTION_LIFECYCLES_TABLE) - conn.execute(CREATE_TURN_CONTENT_REVISIONS_TABLE) - conn.execute(CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_PLANS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_JOBS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_RECOVERIES_TABLE) - conn.execute(CREATE_STORE_MAINTENANCE_STATE_TABLE) - conn.execute(CREATE_STORE_MAINTENANCE_CURSORS_TABLE) - conn.execute(CREATE_TURN_LIST_STATE_TABLE) - conn.execute(CREATE_TURN_LIST_HOSTS_TABLE) - conn.execute(CREATE_TURN_CHANGE_JOURNAL_TABLE) - conn.execute(CREATE_TURN_CHANGE_FLOOR_TABLE) - conn.execute(CREATE_TURN_CHANGE_STATE_TABLE) - conn.execute(CREATE_TURN_SUBMISSIONS_TABLE) - conn.execute(CREATE_TURN_SUPERSESSIONS_TABLE) - conn.execute(CREATE_HERDR_TURN_WATERMARKS_TABLE) - conn.execute(CREATE_HERDR_TURN_COMPLETIONS_TABLE) - conn.execute(CREATE_HERDR_TURN_REFRESH_RETRIES_TABLE) - conn.execute(CREATE_AGENT_EVENTS_TABLE) - conn.execute(CREATE_AGENT_EVENT_TOMBSTONES_TABLE) - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - for statement in CREATE_WORKER_BINDING_INDEXES: - conn.execute(statement) - conn.execute(CREATE_WORKER_BINDING_UNIQUE_INDEX) - for statement in CREATE_CURRENT_PR6_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_LIST_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_LIST_SEQUENCE_TRIGGERS: - conn.execute(statement) - for statement in CREATE_TURN_CHANGE_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_CHANGE_TRIGGERS: - conn.execute(statement) - for statement in CREATE_TURN_SUBMISSION_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_SUPERSESSION_INDEXES: - conn.execute(statement) - for statement in CREATE_HERDR_TURN_INDEXES: - conn.execute(statement) - for statement in CREATE_HERDR_TURN_REFRESH_RETRY_INDEXES: - conn.execute(statement) - for statement in CREATE_AGENT_EVENT_INDEXES: - conn.execute(statement) - for statement in CREATE_AGENT_EVENT_TOMBSTONE_INDEXES: - conn.execute(statement) - for statement in CREATE_ATTENTION_LIFECYCLE_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_CONTENT_REVISION_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_PRESENTATION_INDEXES: - conn.execute(statement) - for statement in CREATE_FINAL_DELIVERY_INDEXES: - conn.execute(statement) - conn.execute(CREATE_CONNECTOR_ORDERING_INDEX) - for statement in CREATE_SNAPSHOT_INDEXES: - conn.execute(statement) - conn.execute(INSERT_STORE_MAINTENANCE_STATE) - _ensure_turn_list_state_conn(conn) - _ensure_turn_change_state_conn(conn) - conn.execute(f"PRAGMA user_version = {STORE_SCHEMA_VERSION}") - conn.commit() - except Exception: - conn.rollback() - raise - - -def _database_has_application_objects(conn: sqlite3.Connection) -> bool: - return ( - conn.execute( - """ - SELECT 1 - FROM sqlite_master - WHERE name NOT LIKE 'sqlite_%' - AND type IN ('table', 'index', 'view', 'trigger') - LIMIT 1 - """ - ).fetchone() - is not None - ) - - -def _run_migrations( - conn: sqlite3.Connection, - *, - target_version: int = STORE_SCHEMA_VERSION, - connector_ack_ttl_seconds: int = CONNECTOR_ACK_TTL_SECONDS, -) -> None: - """Run exact ordered transitions with one transaction per version.""" - if conn.in_transaction: - raise StoreSchemaError("schema_migration_in_transaction") - _validate_migration_registry(target_version=target_version) - current = int(conn.execute("PRAGMA user_version").fetchone()[0]) - while current < int(target_version): - migration = MIGRATIONS[current] - if migration.from_version != current: - raise RuntimeError("invalid migration registry dispatch") - conn.execute("BEGIN IMMEDIATE") - try: - if migration.apply is _migrate_v15_to_v16_conn: - _migrate_v15_to_v16_conn( - conn, - connector_ack_ttl_seconds=max( - 1, int(connector_ack_ttl_seconds) - ), - ) - else: - migration.apply(conn) - conn.execute(f"PRAGMA user_version = {migration.to_version}") - conn.commit() - except Exception: - conn.rollback() - raise - current = int(conn.execute("PRAGMA user_version").fetchone()[0]) - if current != migration.to_version: - raise StoreSchemaError("schema_version_not_advanced") - - -def ensure_schema( - conn: sqlite3.Connection, - *, - connector_ack_ttl_seconds: int = CONNECTOR_ACK_TTL_SECONDS, -) -> None: - """Gate the current schema cheaply, or initialize/migrate older stores.""" - version = int(conn.execute("PRAGMA user_version").fetchone()[0]) - if version == STORE_SCHEMA_VERSION: - return - if version > STORE_SCHEMA_VERSION: - raise StoreSchemaError("schema_too_new") - if not isinstance(conn, _ClosingConnection): - raise local_state_error(LocalStateErrorCode.OPERATION_FAILED) - schema_authority = _schema_connection_authority(conn) - authority = ( - nullcontext() - if schema_authority.parent_fd is None - else _filesystem_schema_mutation_authority(conn) - ) - with authority: - version = int(conn.execute("PRAGMA user_version").fetchone()[0]) - if version == STORE_SCHEMA_VERSION: - return - if version > STORE_SCHEMA_VERSION: - raise StoreSchemaError("schema_too_new") - with private_file_creation_umask(): - _configure_persistent_database_conn(conn) - if version == 0 and not _database_has_application_objects(conn): - _create_current_schema_conn(conn) - return - _run_migrations( - conn, - connector_ack_ttl_seconds=max( - 1, int(connector_ack_ttl_seconds) - ), - ) - - -_ensure_schema = ensure_schema - - -def init_store( - db_path: Path, - *, - connector_ack_ttl_seconds: int = CONNECTOR_ACK_TTL_SECONDS, -) -> None: - """Initialize or migrate the sqlite store to the current schema.""" - with _connect(db_path, prepare=True) as conn: - ensure_schema( - conn, - connector_ack_ttl_seconds=max( - 1, int(connector_ack_ttl_seconds) - ), - ) - - -def _agent_event_from_row(row: tuple[Any, ...]) -> StoredAgentEvent: - private_payload = _json_object(row[15]) - public_payload = _json_object(row[16]) - kind = str(row[3]) - if kind not in AGENT_EVENT_KINDS: - raise StoreSchemaError("invalid_agent_event_kind") - try: - normalized_host = normalize_agent_event_identifier( - row[1], "host_id", required=True - ) - stored_event = AgentEvent( - event_id=str(row[2]), - kind=kind, # type: ignore[arg-type] - source=str(row[4]), - worker_id=str(row[5]), - visibility=str(row[6]), # type: ignore[arg-type] - source_session_id=str(row[7]) if row[7] is not None else None, - source_turn_id=str(row[8]) if row[8] is not None else None, - source_item_id=str(row[9]) if row[9] is not None else None, - source_message_id=str(row[10]) if row[10] is not None else None, - source_event_id=str(row[11]) if row[11] is not None else None, - source_sequence=int(row[12]) if row[12] is not None else None, - observed_at=str(row[13]), - payload_fingerprint=str(row[14]), - payload=private_payload, - public_payload=public_payload, - ) - canonical = agent_event( - kind=stored_event.kind, - source=stored_event.source, - worker_id=stored_event.worker_id, - payload=stored_event.payload, - source_session_id=stored_event.source_session_id, - source_turn_id=stored_event.source_turn_id, - source_item_id=stored_event.source_item_id, - source_message_id=stored_event.source_message_id, - source_event_id=stored_event.source_event_id, - source_sequence=stored_event.source_sequence, - visibility=stored_event.visibility, - observed_at=stored_event.observed_at, - ) - except (TypeError, ValueError, OverflowError) as exc: - raise StoreSchemaError("invalid_agent_event_row") from exc - if canonical != stored_event: - raise StoreSchemaError("invalid_agent_event_row") - return StoredAgentEvent( - sequence=int(row[0]), - host_id=normalized_host or "", - event=stored_event, + except (TypeError, ValueError, OverflowError) as exc: + raise StoreSchemaError("invalid_agent_event_row") from exc + if canonical != stored_event: + raise StoreSchemaError("invalid_agent_event_row") + return StoredAgentEvent( + sequence=int(row[0]), + host_id=normalized_host or "", + event=stored_event, ) @@ -14000,60 +10358,8 @@ def _agent_event_conflicts(existing: StoredAgentEvent, incoming: AgentEvent) -> ) -def _agent_event_replay_contract_fingerprint( - *, - event_id: str, - kind: str, - source: str, - worker_id: str, - visibility: str, - source_session_id: str | None, - source_turn_id: str | None, - source_item_id: str | None, - source_message_id: str | None, - source_event_id: str | None, - source_sequence: int | None, - payload_fingerprint: str, - public_payload_json: str, -) -> str: - """Fingerprint compact replay metadata without reading private payload text.""" - contract = { - "event_id": event_id, - "kind": kind, - "source": source, - "worker_id": worker_id, - "visibility": visibility, - "source_session_id": source_session_id, - "source_turn_id": source_turn_id, - "source_item_id": source_item_id, - "source_message_id": source_message_id, - "source_event_id": source_event_id, - "source_sequence": source_sequence, - "payload_fingerprint": payload_fingerprint, - "public_payload_fingerprint": hashlib.sha256( - public_payload_json.encode("utf-8") - ).hexdigest(), - } - return hashlib.sha256(_canonical_json(contract).encode("utf-8")).hexdigest() -def _agent_event_replay_fingerprint(event: AgentEvent) -> str: - """Fingerprint the replay contract while excluding source observation time.""" - return _agent_event_replay_contract_fingerprint( - event_id=event.event_id, - kind=event.kind, - source=event.source, - worker_id=event.worker_id, - visibility=event.visibility, - source_session_id=event.source_session_id, - source_turn_id=event.source_turn_id, - source_item_id=event.source_item_id, - source_message_id=event.source_message_id, - source_event_id=event.source_event_id, - source_sequence=event.source_sequence, - payload_fingerprint=event.payload_fingerprint, - public_payload_json=_canonical_json(event.public_payload), - ) def _canonical_agent_event_for_append(event: AgentEvent) -> AgentEvent: @@ -14083,22 +10389,6 @@ def _append_agent_event_conn( host_id: str, event: AgentEvent, ) -> AppendAgentEventResult: - tombstone = conn.execute( - """ - SELECT sequence, replay_fingerprint - FROM agent_event_tombstones - WHERE host_id = ? AND event_id = ? - """, - (host_id, event.event_id), - ).fetchone() - if tombstone is not None: - if str(tombstone[1]) != _agent_event_replay_fingerprint(event): - raise AgentEventIdentityConflict(event.event_id) - return AppendAgentEventResult( - sequence=int(tombstone[0]), - event_id=event.event_id, - inserted=False, - ) private_json = _canonical_json(event.payload) public_json = _canonical_json(event.public_payload) cursor = conn.execute( @@ -14148,89 +10438,33 @@ def _append_agent_event_conn( ) -def append_agent_event( - db_path: Path | str, - host_id: str, - event: AgentEvent, -) -> AppendAgentEventResult: - """Append one structured event, or return its existing replay sequence. - - Reusing a deterministic event identity with different content is rejected - instead of silently mutating the journal or accepting source corruption. - """ - normalized_host = normalize_agent_event_identifier( - host_id, "host_id", required=True - ) - _canonical_agent_event_for_append(event) - with _connect(db_path, prepare=True) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - result = _append_agent_event_conn(conn, normalized_host or "", event) - conn.commit() - except Exception: - conn.rollback() - raise - return result - - -def append_agent_event_for_binding( + + + + +def record_agent_event( db_path: Path | str, host_id: str, - event: AgentEvent, - *, - expected_binding: WorkerBinding, -) -> AppendBoundAgentEventResult: - """Append only while the expected active worker binding remains current. - - The binding check and insert share one ``BEGIN IMMEDIATE`` transaction, so - a concurrent inventory refresh cannot invalidate the binding between the - check and journal mutation. - """ + **event_fields: Any, +) -> AppendAgentEventResult: + """Validate, normalize, and durably append one source event.""" normalized_host = normalize_agent_event_identifier( host_id, "host_id", required=True ) + event = agent_event(**event_fields) _canonical_agent_event_for_append(event) - if not isinstance(expected_binding, WorkerBinding): - raise ValueError("expected_binding must be a WorkerBinding") with _connect(db_path, prepare=True) as conn: _ensure_schema(conn) conn.execute("BEGIN IMMEDIATE") try: - if not _agent_event_binding_matches_conn( - conn, - normalized_host or "", - event.worker_id, - expected_binding, - ): - conn.rollback() - return AppendBoundAgentEventResult( - status="binding_changed", - event_id=event.event_id, - ) result = _append_agent_event_conn( - conn, - normalized_host or "", - event, + conn, normalized_host or "", event ) conn.commit() except Exception: conn.rollback() raise - return AppendBoundAgentEventResult( - status="inserted" if result.inserted else "replayed", - event_id=result.event_id, - sequence=result.sequence, - ) - - -def record_agent_event( - db_path: Path | str, - host_id: str, - **event_fields: Any, -) -> AppendAgentEventResult: - """Validate, normalize, and durably append one source event.""" - return append_agent_event(db_path, host_id, agent_event(**event_fields)) + return result def list_agent_events( @@ -14287,529 +10521,16 @@ def list_agent_events( parameters.append(int(limit)) with _connect(db_path) as conn: _ensure_schema(conn) - rows = conn.execute( - _AGENT_EVENT_SELECT - + " WHERE " - + " AND ".join(clauses) - + " ORDER BY sequence ASC LIMIT ?", - parameters, - ).fetchall() - return tuple(_agent_event_from_row(row) for row in rows) - - -def list_public_agent_events( - db_path: Path | str, - host_id: str, - *, - worker_id: str | None = None, - source: str | None = None, - session_id: str | None = None, - turn_id: str | None = None, - after_sequence: int = 0, - limit: int = AGENT_EVENT_QUERY_DEFAULT_LIMIT, -) -> tuple[dict[str, Any], ...]: - """List connector-safe projections without private source identifiers.""" - return tuple( - stored.public_dict() - for stored in list_agent_events( - db_path, - host_id, - worker_id=worker_id, - source=source, - session_id=session_id, - turn_id=turn_id, - visibility="public", - after_sequence=after_sequence, - limit=limit, - ) - ) - - -def _herdr_turn_counter(value: Any, field: str) -> int: - if ( - not isinstance(value, int) - or isinstance(value, bool) - or not 0 <= value <= _SQLITE_MAX_INTEGER - ): - raise ValueError(f"{field} must be a nonnegative SQLite integer") - return int(value) - - -def _herdr_turn_watermark_from_row(row: tuple[Any, ...]) -> HerdrTurnWatermark: - return HerdrTurnWatermark( - host_id=str(row[0]), - pane_id=str(row[1]), - turn_epoch=int(row[2]), - last_turn=int(row[3]), - completeness_break_count=int(row[4]), - last_completeness_break_reason=( - str(row[5]) if row[5] is not None else None - ), - last_completeness_break_at=( - str(row[6]) if row[6] is not None else None - ), - updated_at=str(row[7]), - ) - - -def get_herdr_turn_watermark( - db_path: Path | str, - host_id: str, - pane_id: str, -) -> HerdrTurnWatermark | None: - """Return one pane's durable Herdr replay watermark.""" - if not _sqlite_store_exists(db_path): - return None - with _connect(db_path) as conn: - _ensure_schema(conn) - row = conn.execute( - """ - SELECT - host_id, pane_id, turn_epoch, last_turn, - completeness_break_count, last_completeness_break_reason, - last_completeness_break_at, updated_at - FROM herdr_turn_watermarks - WHERE host_id = ? AND pane_id = ? - """, - (str(host_id), str(pane_id)), - ).fetchone() - return _herdr_turn_watermark_from_row(row) if row is not None else None - - -def set_herdr_turn_watermark( - db_path: Path | str, - host_id: str, - pane_id: str, - *, - turn_epoch: int, - last_turn: int, - observed_at: str | None = None, -) -> HerdrTurnWatermark: - """Create or re-baseline a watermark while retaining prior break evidence.""" - epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") - turn = _herdr_turn_counter(last_turn, "last_turn") - current = observed_at or utc_timestamp() - with _connect(db_path) as conn: - _ensure_schema(conn) - conn.execute( - """ - INSERT INTO herdr_turn_watermarks ( - host_id, pane_id, turn_epoch, last_turn, updated_at - ) VALUES (?, ?, ?, ?, ?) - ON CONFLICT(host_id, pane_id) DO UPDATE SET - turn_epoch = excluded.turn_epoch, - last_turn = excluded.last_turn, - updated_at = excluded.updated_at - """, - (str(host_id), str(pane_id), epoch, turn, current), - ) - watermark = get_herdr_turn_watermark(db_path, host_id, pane_id) - if watermark is None: - raise StoreSchemaError("herdr_turn_watermark_unavailable") - return watermark - - -def record_herdr_turn_completeness_break( - db_path: Path | str, - host_id: str, - pane_id: str, - *, - turn_epoch: int, - newest_turn: int, - reason: str, - observed_at: str | None = None, -) -> HerdrTurnWatermark: - """Persist an explicit replay gap and atomically re-baseline the pane.""" - epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") - newest = _herdr_turn_counter(newest_turn, "newest_turn") - break_reason = str(reason).strip() - if not break_reason: - raise ValueError("reason must not be empty") - current = observed_at or utc_timestamp() - with _connect(db_path) as conn: - _ensure_schema(conn) - conn.execute( - """ - INSERT INTO herdr_turn_watermarks ( - host_id, pane_id, turn_epoch, last_turn, - completeness_break_count, last_completeness_break_reason, - last_completeness_break_at, updated_at - ) VALUES (?, ?, ?, ?, 1, ?, ?, ?) - ON CONFLICT(host_id, pane_id) DO UPDATE SET - turn_epoch = excluded.turn_epoch, - last_turn = excluded.last_turn, - completeness_break_count = - herdr_turn_watermarks.completeness_break_count + 1, - last_completeness_break_reason = - excluded.last_completeness_break_reason, - last_completeness_break_at = - excluded.last_completeness_break_at, - updated_at = excluded.updated_at - """, - ( - str(host_id), - str(pane_id), - epoch, - newest, - break_reason, - current, - current, - ), - ) - watermark = get_herdr_turn_watermark(db_path, host_id, pane_id) - if watermark is None: - raise StoreSchemaError("herdr_turn_watermark_unavailable") - return watermark - - -def latest_turn_id_for_worker( - db_path: Path | str, - host_id: str, - worker_id: str, -) -> str | None: - """Return the newest live semantic turn row for provenance linking.""" - if not _sqlite_store_exists(db_path): - return None - with _connect(db_path) as conn: - _ensure_schema(conn) - row = conn.execute( - """ - SELECT turn_id - FROM turns - WHERE host_id = ? - AND worker_id = ? - AND COALESCE(json_extract(payload_json, '$.superseded_at'), '') = '' - ORDER BY observed_at DESC, list_sequence DESC - LIMIT 1 - """, - (str(host_id), str(worker_id)), - ).fetchone() - return str(row[0]) if row is not None else None - - -def _herdr_turn_refresh_retry_from_row( - row: tuple[Any, ...], -) -> HerdrTurnRefreshRetry: - return HerdrTurnRefreshRetry( - host_id=str(row[0]), - pane_id=str(row[1]), - turn_epoch=int(row[2]), - turn=int(row[3]), - status=str(row[4]), # type: ignore[arg-type] - refresh_status=str(row[5]), - first_seen_at=str(row[6]), - last_attempt_at=str(row[7]), - next_attempt_at=str(row[8]) if row[8] is not None else None, - attempt_count=int(row[9]), - escalated_at=str(row[10]) if row[10] is not None else None, - ) - - -def get_herdr_turn_refresh_retry( - db_path: Path | str, - host_id: str, - pane_id: str, - *, - turn_epoch: int, - turn: int, -) -> HerdrTurnRefreshRetry | None: - """Return durable completion-refresh retry state, if any.""" - epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") - turn_number = _herdr_turn_counter(turn, "turn") - if not _sqlite_store_exists(db_path): - return None - with _connect(db_path) as conn: - _ensure_schema(conn) - row = conn.execute( - """ - SELECT - host_id, pane_id, turn_epoch, turn, status, refresh_status, - first_seen_at, last_attempt_at, next_attempt_at, - attempt_count, escalated_at - FROM herdr_turn_refresh_retries - WHERE host_id = ? AND pane_id = ? AND turn_epoch = ? AND turn = ? - """, - (str(host_id), str(pane_id), epoch, turn_number), - ).fetchone() - return _herdr_turn_refresh_retry_from_row(row) if row is not None else None - - -def herdr_turn_refresh_retry_due( - db_path: Path | str, - host_id: str, - pane_id: str, - *, - turn_epoch: int, - turn: int, - now: str | None = None, -) -> bool: - """Return whether an absent/pending retry may run under the local clock. - - A backwards local-clock jump makes the retry immediately due. Combined - with the durable attempt ceiling this cannot leave a pane wedged waiting - for a wall clock to catch up. - """ - retry = get_herdr_turn_refresh_retry( - db_path, - host_id, - pane_id, - turn_epoch=turn_epoch, - turn=turn, - ) - if retry is None: - return True - if retry.status == "escalated" or retry.next_attempt_at is None: - return False - current = _connector_datetime(now or utc_timestamp()) - last_attempt = _connector_datetime(retry.last_attempt_at) - next_attempt = _connector_datetime(retry.next_attempt_at) - return current < last_attempt or current >= next_attempt - - -def record_herdr_turn_refresh_retry( - db_path: Path | str, - host_id: str, - pane_id: str, - *, - turn_epoch: int, - turn: int, - refresh_status: str, - now: str | None = None, - base_delay_seconds: int = 1, - max_delay_seconds: int = 30, - max_retry_age_seconds: int = 300, - max_attempts: int = 8, -) -> HerdrTurnRefreshRetry: - """Record one failed refresh attempt with bounded durable backoff.""" - epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") - turn_number = _herdr_turn_counter(turn, "turn") - normalized_status = str(refresh_status).strip() - if not normalized_status or len(normalized_status) > 128: - raise ValueError("refresh_status must contain at most 128 characters") - bounds = (base_delay_seconds, max_delay_seconds, max_retry_age_seconds) - if any( - isinstance(value, bool) or not isinstance(value, int) or value < 0 - for value in bounds - ): - raise ValueError("retry delays and age must be nonnegative integers") - if max_delay_seconds < base_delay_seconds: - raise ValueError("max_delay_seconds must be at least base_delay_seconds") - if ( - isinstance(max_attempts, bool) - or not isinstance(max_attempts, int) - or max_attempts < 1 - ): - raise ValueError("max_attempts must be a positive integer") - current_dt = _connector_datetime(now or utc_timestamp()) - current = current_dt.isoformat() - with _connect(db_path) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - row = conn.execute( - """ - SELECT - host_id, pane_id, turn_epoch, turn, status, - refresh_status, first_seen_at, last_attempt_at, - next_attempt_at, attempt_count, escalated_at - FROM herdr_turn_refresh_retries - WHERE host_id = ? AND pane_id = ? AND turn_epoch = ? AND turn = ? - """, - (str(host_id), str(pane_id), epoch, turn_number), - ).fetchone() - if row is not None and str(row[4]) == "escalated": - conn.commit() - return _herdr_turn_refresh_retry_from_row(row) - first_seen = str(row[6]) if row is not None else current - attempt_count = (int(row[9]) if row is not None else 0) + 1 - age_seconds = max( - 0.0, - (current_dt - _connector_datetime(first_seen)).total_seconds(), - ) - escalated = ( - attempt_count >= max_attempts - or age_seconds >= max_retry_age_seconds - ) - # Anchor backoff at the later of the current and last local sample. - # The due predicate explicitly detects rollback, while this avoids - # persisting decreasing attempt timestamps. - attempt_dt = current_dt - if row is not None: - attempt_dt = max(attempt_dt, _connector_datetime(str(row[7]))) - delay = min( - max_delay_seconds, - base_delay_seconds * (2 ** min(attempt_count - 1, 30)), - ) - next_attempt = ( - None - if escalated - else (attempt_dt + timedelta(seconds=delay)).isoformat() - ) - escalated_at = attempt_dt.isoformat() if escalated else None - conn.execute( - """ - INSERT INTO herdr_turn_refresh_retries ( - host_id, pane_id, turn_epoch, turn, status, - refresh_status, first_seen_at, last_attempt_at, - next_attempt_at, attempt_count, escalated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(host_id, pane_id, turn_epoch, turn) DO UPDATE SET - status = excluded.status, - refresh_status = excluded.refresh_status, - last_attempt_at = excluded.last_attempt_at, - next_attempt_at = excluded.next_attempt_at, - attempt_count = excluded.attempt_count, - escalated_at = excluded.escalated_at - """, - ( - str(host_id), - str(pane_id), - epoch, - turn_number, - "escalated" if escalated else "pending", - normalized_status, - first_seen, - attempt_dt.isoformat(), - next_attempt, - attempt_count, - escalated_at, - ), - ) - conn.commit() - except Exception: - conn.rollback() - raise - retry = get_herdr_turn_refresh_retry( - db_path, - host_id, - pane_id, - turn_epoch=epoch, - turn=turn_number, - ) - if retry is None: - raise StoreSchemaError("herdr_turn_refresh_retry_unavailable") - return retry - - -def record_herdr_turn_completion( - db_path: Path | str, - host_id: str, - pane_id: str, - *, - turn_epoch: int, - turn: int, - outcome: str, - completed_unix_ms: int, - message: str | None, - message_truncated: bool, - agent_session_path: str | None, - worker_id: str | None, - refreshed_turn_id: str | None, - observed_at: str | None = None, - preserve_refresh_retry: bool = False, -) -> HerdrTurnWatermark: - """Store completion provenance and advance its replay watermark atomically.""" - epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") - turn_number = _herdr_turn_counter(turn, "turn") - completed_ms = _herdr_turn_counter(completed_unix_ms, "completed_unix_ms") - normalized_outcome = str(outcome) - if normalized_outcome not in {"completed", "aborted"}: - raise ValueError("outcome must be completed or aborted") - if message is not None and not isinstance(message, str): - raise ValueError("message must be text or None") - if isinstance(message, str) and len(message.encode("utf-8")) > 8 * 1024: - raise ValueError("message must not exceed 8 KiB") - if not isinstance(message_truncated, bool): - raise ValueError("message_truncated must be a boolean") - if agent_session_path is not None and not isinstance( - agent_session_path, - str, - ): - raise ValueError("agent_session_path must be text or None") - if not isinstance(preserve_refresh_retry, bool): - raise ValueError("preserve_refresh_retry must be a boolean") - current = observed_at or utc_timestamp() - with _connect(db_path) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - existing = conn.execute( - """ - SELECT turn_epoch, last_turn - FROM herdr_turn_watermarks - WHERE host_id = ? AND pane_id = ? - """, - (str(host_id), str(pane_id)), - ).fetchone() - if existing is not None and int(existing[0]) != epoch: - raise StoreSchemaError("herdr_turn_epoch_changed") - conn.execute( - """ - INSERT INTO herdr_turn_completions ( - host_id, pane_id, turn_epoch, turn, outcome, - completed_unix_ms, message, message_truncated, - agent_session_path, worker_id, refreshed_turn_id, observed_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(host_id, pane_id, turn_epoch, turn) DO UPDATE SET - outcome = excluded.outcome, - completed_unix_ms = excluded.completed_unix_ms, - message = excluded.message, - message_truncated = excluded.message_truncated, - agent_session_path = excluded.agent_session_path, - worker_id = COALESCE( - excluded.worker_id, herdr_turn_completions.worker_id - ), - refreshed_turn_id = COALESCE( - excluded.refreshed_turn_id, - herdr_turn_completions.refreshed_turn_id - ), - observed_at = excluded.observed_at - """, - ( - str(host_id), - str(pane_id), - epoch, - turn_number, - normalized_outcome, - completed_ms, - message, - int(message_truncated), - agent_session_path, - str(worker_id) if worker_id else None, - str(refreshed_turn_id) if refreshed_turn_id else None, - current, - ), - ) - conn.execute( - """ - INSERT INTO herdr_turn_watermarks ( - host_id, pane_id, turn_epoch, last_turn, updated_at - ) VALUES (?, ?, ?, ?, ?) - ON CONFLICT(host_id, pane_id) DO UPDATE SET - last_turn = MAX( - herdr_turn_watermarks.last_turn, excluded.last_turn - ), - updated_at = excluded.updated_at - """, - (str(host_id), str(pane_id), epoch, turn_number, current), - ) - if not preserve_refresh_retry: - conn.execute( - """ - DELETE FROM herdr_turn_refresh_retries - WHERE host_id = ? AND pane_id = ? - AND turn_epoch = ? AND turn = ? AND status = 'pending' - """, - (str(host_id), str(pane_id), epoch, turn_number), - ) - conn.commit() - except Exception: - conn.rollback() - raise - watermark = get_herdr_turn_watermark(db_path, host_id, pane_id) - if watermark is None: - raise StoreSchemaError("herdr_turn_watermark_unavailable") - return watermark + rows = conn.execute( + _AGENT_EVENT_SELECT + + " WHERE " + + " AND ".join(clauses) + + " ORDER BY sequence ASC LIMIT ?", + parameters, + ).fetchall() + return tuple(_agent_event_from_row(row) for row in rows) + + def _normalized_command_request_policy( @@ -17458,7 +13179,6 @@ def maybe_run_automatic_store_maintenance( policy: SnapshotRetentionPolicy, agent_event_host_id: str | None = None, agent_event_retention_days: int | None = None, - turn_model: str = DEFAULT_TURN_MODEL, acknowledged_final_retention_days: int = ACKNOWLEDGED_FINAL_RETENTION_DAYS, acknowledged_final_retention_count: int = ACKNOWLEDGED_FINAL_RETENTION_COUNT, command_retry_horizon_seconds: int = COMMAND_RETRY_HORIZON_SECONDS, @@ -17527,9 +13247,7 @@ def maybe_run_automatic_store_maintenance( "batch_size": policy.batch_size, "examined": 0, "deleted": 0, - "tombstoned": 0, "remaining_candidates": False, - "replay_identity_retained": True, } empty_command_requests = _command_request_maintenance_summary( None, @@ -17538,16 +13256,6 @@ def maybe_run_automatic_store_maintenance( retention_count=command_receipt_retention_count, batch_size=policy.batch_size, ) - empty_herdr_turns = { - "examined": 0, - "deleted": 0, - "deleted_completions": 0, - "deleted_watermarks": 0, - "remaining_candidates": False, - "retention_days": policy.retention_days, - "retention_count": policy.retention_count, - "batch_size": policy.batch_size, - } if not _sqlite_store_exists(db_path): return dict(sanitize_public_value({ "schema_version": 1, @@ -17574,7 +13282,6 @@ def maybe_run_automatic_store_maintenance( ), }, "command_requests": empty_command_requests, - "herdr_turns": empty_herdr_turns, "batch_size": policy.batch_size, })) cutoff_at = _utc_cutoff(retention_days=policy.retention_days, now=current_at) @@ -17634,15 +13341,13 @@ def maybe_run_automatic_store_maintenance( ), }, "command_requests": empty_command_requests, - "herdr_turns": empty_herdr_turns, "batch_size": policy.batch_size, })) - if _submission_linking_enabled(turn_model): - _settle_due_submission_links_conn( - conn, - db_path=db_path, - now=current_at, - ) + _settle_due_submission_links_conn( + conn, + db_path=db_path, + now=current_at, + ) _expire_turn_submissions_conn( conn, current=current_at, @@ -17656,7 +13361,6 @@ def maybe_run_automatic_store_maintenance( cutoff_at=agent_event_cutoff_at, batch_size=policy.batch_size, dry_run=False, - retired_at=current_at, ) ) candidates, _ = _snapshot_retention_candidates_conn( @@ -17780,25 +13484,10 @@ def maybe_run_automatic_store_maintenance( retention_count=command_receipt_retention_count, batch_size=policy.batch_size, ) - herdr_turns = cleanup_herdr_turn_retention( - db_path, - retention_days=policy.retention_days, - retention_count=policy.retention_count, - batch_size=policy.batch_size, - now=current_at, - ) return dict(sanitize_public_value({ "schema_version": 1, - "ok": bool(command_requests["ok"]) and bool(herdr_turns["ok"]), - "status": ( - "ok" - if command_requests["ok"] and herdr_turns["ok"] - else ( - command_requests["status"] - if not command_requests["ok"] - else herdr_turns["status"] - ) - ), + "ok": bool(command_requests["ok"]), + "status": "ok" if command_requests["ok"] else command_requests["status"], "due": True, "last_completed_at": current_at, "next_due_at": _connector_add_seconds(current_at, cadence_seconds), @@ -17820,7 +13509,6 @@ def maybe_run_automatic_store_maintenance( ), }, "command_requests": command_requests, - "herdr_turns": herdr_turns, "batch_size": policy.batch_size, })) @@ -17915,46 +13603,11 @@ def cleanup_event_retention( _AGENT_EVENT_RETENTION_SELECT = """ -SELECT - sequence, host_id, event_id, kind, source, worker_id, visibility, - source_session_id, source_turn_id, source_item_id, source_message_id, - source_event_id, source_sequence, payload_fingerprint, public_payload_json, - observed_at +SELECT sequence FROM agent_events """ -def _agent_event_retention_candidate( - row: tuple[Any, ...], -) -> tuple[str, str, int, str, str]: - """Reduce one bounded metadata row to its durable tombstone fields.""" - public_payload_json = str(row[14]) - public_payload = _json_object(public_payload_json) - if _canonical_json(public_payload) != public_payload_json: - raise StoreSchemaError("invalid_agent_event_projection") - return ( - str(row[1]), - str(row[2]), - int(row[0]), - _agent_event_replay_contract_fingerprint( - event_id=str(row[2]), - kind=str(row[3]), - source=str(row[4]), - worker_id=str(row[5]), - visibility=str(row[6]), - source_session_id=str(row[7]) if row[7] is not None else None, - source_turn_id=str(row[8]) if row[8] is not None else None, - source_item_id=str(row[9]) if row[9] is not None else None, - source_message_id=str(row[10]) if row[10] is not None else None, - source_event_id=str(row[11]) if row[11] is not None else None, - source_sequence=int(row[12]) if row[12] is not None else None, - payload_fingerprint=str(row[13]), - public_payload_json=public_payload_json, - ), - str(row[15]), - ) - - def _cleanup_agent_event_retention_conn( conn: sqlite3.Connection, host_id: str, @@ -17962,56 +13615,18 @@ def _cleanup_agent_event_retention_conn( cutoff_at: str, batch_size: int, dry_run: bool, - retired_at: str, ) -> dict[str, Any]: - """Retire one batch using streamed metadata, never private payload values.""" - cursor = conn.execute( + """Delete one bounded batch of expired journal rows.""" + rows = conn.execute( _AGENT_EVENT_RETENTION_SELECT + " WHERE host_id = ? AND observed_at < ?" + " ORDER BY observed_at, sequence LIMIT ?", (str(host_id), cutoff_at, int(batch_size) + 1), - ) - candidates: list[tuple[str, str, int, str, str]] = [] - remaining = False - for row in cursor: - if len(candidates) >= batch_size: - remaining = True - break - if dry_run: - candidates.append( - (str(row[1]), str(row[2]), int(row[0]), "", str(row[15])) - ) - else: - candidates.append(_agent_event_retention_candidate(row)) - + ).fetchall() + sequences = [int(row[0]) for row in rows[:batch_size]] + remaining = len(rows) > len(sequences) deleted = 0 - if candidates and not dry_run: - conn.executemany( - """ - INSERT INTO agent_event_tombstones ( - host_id, event_id, sequence, replay_fingerprint, - observed_at, retired_at - ) VALUES (?, ?, ?, ?, ?, ?) - """, - ( - ( - candidate_host, - event_id, - sequence, - fingerprint, - observed_at, - retired_at, - ) - for ( - candidate_host, - event_id, - sequence, - fingerprint, - observed_at, - ) in candidates - ), - ) - sequences = [candidate[2] for candidate in candidates] + if sequences and not dry_run: placeholders = ",".join("?" for _ in sequences) deleted = int( conn.execute( @@ -18021,13 +13636,11 @@ def _cleanup_agent_event_retention_conn( ).rowcount or 0 ) - if deleted != len(candidates): + if deleted != len(sequences): raise StoreSchemaError("agent_event_retention_delete_mismatch") - retired = len(candidates) if dry_run else deleted return { - "examined": len(candidates), - "deleted": retired, - "tombstoned": retired, + "examined": len(sequences), + "deleted": len(sequences) if dry_run else deleted, "remaining_candidates": remaining, } @@ -18041,7 +13654,7 @@ def cleanup_agent_event_retention( dry_run: bool = False, batch_size: int = 100, ) -> dict[str, Any]: - """Retire event payloads while retaining compact replay identities.""" + """Delete a bounded batch of expired structured agent events.""" days = max(1, int(retention_days)) bounded_batch = max(1, min(int(batch_size), 1_000)) cutoff_at = _utc_cutoff(retention_days=days, now=now) @@ -18060,7 +13673,6 @@ def cleanup_agent_event_retention( "status": "store_unavailable", "examined": 0, "deleted": 0, - "tombstoned": 0, "remaining_candidates": False, })) with _connect(db_path, isolation_level=None) as conn: @@ -18077,7 +13689,6 @@ def cleanup_agent_event_retention( cutoff_at=cutoff_at, batch_size=bounded_batch, dry_run=bool(dry_run), - retired_at=utc_timestamp(), ) if dry_run: conn.rollback() @@ -19410,189 +15021,6 @@ def compact_turn_change_journal( } -def cleanup_herdr_turn_retention( - db_path: Path | str, - *, - host_id: str | None = None, - retention_days: int = HERDR_TURN_RETENTION_DAYS, - retention_count: int = HERDR_TURN_RETENTION_COUNT, - batch_size: int = HERDR_TURN_RETENTION_BATCH_SIZE, - now: str | None = None, - dry_run: bool = False, -) -> dict[str, Any]: - """Bound completion provenance and remove old watermarks for dead panes.""" - days = max(1, min(int(retention_days), _MAX_RETENTION_DAYS)) - count = max(1, min(int(retention_count), _SQLITE_MAX_INTEGER)) - bounded_batch = max(1, min(int(batch_size), 1_000)) - current_at = _connector_now(now) - cutoff_at = _utc_cutoff(retention_days=days, now=current_at) - base = { - "schema_version": 1, - "ok": False, - "status": "store_unavailable", - "scope": "host" if host_id is not None else "database", - "host_id": str(host_id) if host_id is not None else None, - "dry_run": bool(dry_run), - "retention_days": days, - "retention_count": count, - "cutoff_at": cutoff_at, - "batch_size": bounded_batch, - } - if not _sqlite_store_exists(db_path): - return dict(sanitize_public_value({ - **base, - "examined": 0, - "deleted": 0, - "deleted_completions": 0, - "deleted_watermarks": 0, - "remaining_candidates": False, - })) - host_clause = "AND host_id = :host_id" if host_id is not None else "" - watermark_host_clause = ( - "AND watermarks.host_id = :host_id" if host_id is not None else "" - ) - params: dict[str, Any] = { - "host_id": str(host_id) if host_id is not None else "", - "cutoff_at": cutoff_at, - "current_at": current_at, - "retention_count": count, - "limit": bounded_batch + 1, - } - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - completion_pool = [ - int(row[0]) - for row in conn.execute( - f""" - WITH ranked AS ( - SELECT - rowid AS completion_rowid, - observed_at, - ROW_NUMBER() OVER ( - PARTITION BY host_id - ORDER BY observed_at DESC, rowid DESC - ) AS retention_rank - FROM herdr_turn_completions - WHERE 1 = 1 {host_clause} - ) - SELECT completion_rowid - FROM ranked - WHERE observed_at < :cutoff_at - AND retention_rank > :retention_count - ORDER BY observed_at, completion_rowid - LIMIT :limit - """, - params, - ).fetchall() - ] - completion_ids = completion_pool[:bounded_batch] - completion_retry_keys = [] - if completion_ids: - placeholders = ",".join("?" for _ in completion_ids) - completion_retry_keys = conn.execute( - f""" - SELECT host_id, pane_id, turn_epoch, turn - FROM herdr_turn_completions - WHERE rowid IN ({placeholders}) - """, - completion_ids, - ).fetchall() - remaining_budget = bounded_batch - len(completion_ids) - watermark_params = { - **params, - # Query one extra row even when completions consume the whole - # batch so remaining_candidates still advertises dead panes. - "limit": remaining_budget + 1, - } - watermark_pool = [ - (str(row[0]), str(row[1])) - for row in conn.execute( - f""" - SELECT watermarks.host_id, watermarks.pane_id - FROM herdr_turn_watermarks AS watermarks - WHERE watermarks.updated_at < :cutoff_at - {watermark_host_clause} - AND NOT EXISTS ( - SELECT 1 - FROM worker_bindings AS bindings - WHERE bindings.host_id = watermarks.host_id - AND bindings.backend = 'herdr' - AND bindings.expires_at > :current_at - AND ( - ( - bindings.target_kind = 'pane_id' - AND bindings.target_value = watermarks.pane_id - ) - OR ( - bindings.turn_target_kind = 'pane_id' - AND bindings.turn_target_value = watermarks.pane_id - ) - ) - ) - ORDER BY watermarks.updated_at, watermarks.host_id, - watermarks.pane_id - LIMIT :limit - """, - watermark_params, - ).fetchall() - ] - watermark_keys = watermark_pool[:remaining_budget] - if not dry_run: - if completion_ids: - placeholders = ",".join("?" for _ in completion_ids) - conn.execute( - f""" - DELETE FROM herdr_turn_completions - WHERE rowid IN ({placeholders}) - """, - completion_ids, - ) - conn.executemany( - """ - DELETE FROM herdr_turn_refresh_retries - WHERE host_id = ? AND pane_id = ? - AND turn_epoch = ? AND turn = ? - """, - completion_retry_keys, - ) - if watermark_keys: - conn.executemany( - """ - DELETE FROM herdr_turn_watermarks - WHERE host_id = ? AND pane_id = ? - """, - watermark_keys, - ) - conn.executemany( - """ - DELETE FROM herdr_turn_refresh_retries - WHERE host_id = ? AND pane_id = ? - """, - watermark_keys, - ) - conn.commit() - else: - conn.rollback() - except Exception: - conn.rollback() - raise - examined = len(completion_ids) + len(watermark_keys) - remaining = ( - len(completion_pool) > len(completion_ids) - or len(watermark_pool) > len(watermark_keys) - ) - return dict(sanitize_public_value({ - **base, - "ok": True, - "status": "ok", - "examined": examined, - "deleted": examined, - "deleted_completions": len(completion_ids), - "deleted_watermarks": len(watermark_keys), - "remaining_candidates": remaining, - })) def run_store_maintenance( @@ -19616,9 +15044,6 @@ def run_store_maintenance( turn_change_retention_days: int = TURN_CHANGE_RETENTION_DAYS, turn_change_retention_count: int = TURN_CHANGE_RETENTION_COUNT, turn_change_batch_size: int = TURN_CHANGE_COMPACTION_BATCH_SIZE, - herdr_turn_retention_days: int = HERDR_TURN_RETENTION_DAYS, - herdr_turn_retention_count: int = HERDR_TURN_RETENTION_COUNT, - herdr_turn_batch_size: int = HERDR_TURN_RETENTION_BATCH_SIZE, ) -> dict[str, Any]: """Run one bounded batch for every online store-maintenance class.""" retention = cleanup_event_retention( @@ -19696,15 +15121,6 @@ def run_store_maintenance( now=now, dry_run=dry_run, ) - herdr_turns = cleanup_herdr_turn_retention( - db_path, - host_id=host_id, - retention_days=herdr_turn_retention_days, - retention_count=herdr_turn_retention_count, - batch_size=herdr_turn_batch_size, - now=now, - dry_run=dry_run, - ) ok = ( bool(retention.get("ok")) and bool(agent_events.get("ok")) @@ -19714,7 +15130,6 @@ def run_store_maintenance( and bool(turn_content.get("ok")) and bool(command_requests.get("ok")) and bool(turn_changes.get("ok")) - and bool(herdr_turns.get("ok")) ) return sanitize_public_value({ "schema_version": 1, @@ -19742,11 +15157,9 @@ def run_store_maintenance( ), "examined": int(agent_events.get("examined") or 0), "deleted": int(agent_events.get("deleted") or 0), - "tombstoned": int(agent_events.get("tombstoned") or 0), "remaining_candidates": bool( agent_events.get("remaining_candidates") ), - "replay_identity_retained": True, }, "snapshots": { "scope": "database", @@ -19816,30 +15229,6 @@ def run_store_maintenance( "deleted": int(turn_changes.get("deleted") or 0), "remaining_candidates": bool(turn_changes.get("remaining_candidates")), }, - "herdr_turns": { - "dry_run": bool(herdr_turns.get("dry_run")), - "retention_days": int( - herdr_turns.get("retention_days") or herdr_turn_retention_days - ), - "retention_count": int( - herdr_turns.get("retention_count") or herdr_turn_retention_count - ), - "cutoff_at": herdr_turns.get("cutoff_at"), - "batch_size": int( - herdr_turns.get("batch_size") or herdr_turn_batch_size - ), - "examined": int(herdr_turns.get("examined") or 0), - "deleted": int(herdr_turns.get("deleted") or 0), - "deleted_completions": int( - herdr_turns.get("deleted_completions") or 0 - ), - "deleted_watermarks": int( - herdr_turns.get("deleted_watermarks") or 0 - ), - "remaining_candidates": bool( - herdr_turns.get("remaining_candidates") - ), - }, "turn_content": { "dry_run": bool(turn_content.get("dry_run")), "retention_days": int( @@ -19920,8 +15309,8 @@ def _turn_continuity_identity(payload: Mapping[str, Any]) -> tuple[str, str, int return None -def _turn_submission_owner_identity(worker: Any) -> tuple[str, int]: - """Extract the Phase-1 continuity owner, with a legacy worker fallback.""" +def _turn_submission_owner_identity(worker: Any) -> tuple[str, int] | None: + """Extract the stable continuity owner required for submission linking.""" meta = getattr(worker, "meta", None) if meta is None and isinstance(worker, Mapping): meta = worker.get("meta") @@ -19931,34 +15320,7 @@ def _turn_submission_owner_identity(worker: Any) -> tuple[str, int]: if identity is not None: _kind, owner_key, owner_key_version = identity return owner_key, owner_key_version - - # Old snapshots and direct API callers can predate stable worker keys. The - # shadow ledger must never change their submission behavior, so isolate - # those rows by the existing public worker ID until Stage 4 migration. - worker_id = str(getattr(worker, "id", "") or "").strip() - if not worker_id and isinstance(worker, Mapping): - worker_id = str(worker.get("id") or worker.get("worker_id") or "").strip() - if not worker_id: - raise StoreSchemaError("turn_submission_owner_missing") - return f"legacy-worker:{worker_id}", 0 - - -def _turn_link_candidate_owner_identity(worker: Any) -> tuple[str, int]: - """Normalize prod turns that persisted a stable key without its version.""" - meta = getattr(worker, "meta", None) - if meta is None and isinstance(worker, Mapping): - meta = worker.get("meta") - if isinstance(meta, Mapping): - stable_key = meta.get("stable_key") - if ( - _valid_final_stable_key(stable_key) - and meta.get("stable_key_version") is None - ): - # The authenticated owner hash is the continuity identity. Some - # production observations omitted its v1 metadata marker, so - # normalize only that missing marker for candidate matching. - return str(stable_key), 1 - return _turn_submission_owner_identity(worker) + return None def _turn_submission_policy( @@ -19989,12 +15351,15 @@ def _insert_turn_submission_conn( link_window_seconds: int, hard_ttl_seconds: int, ) -> str: - """Insert the send-started shadow ledger row in the caller transaction.""" + """Insert the send-started submission row in the caller transaction.""" link_window, hard_ttl = _turn_submission_policy( link_window_seconds, hard_ttl_seconds, ) - owner_key, owner_key_version = _turn_submission_owner_identity(worker) + owner_identity = _turn_submission_owner_identity(worker) + if owner_identity is None: + raise StoreSchemaError("turn_submission_stable_owner_missing") + owner_key, owner_key_version = owner_identity submission_id = turn_submission_id(host_id, request_id) current_time = datetime.fromisoformat(current) link_not_before = (current_time - timedelta(seconds=link_window)).isoformat( @@ -20061,7 +15426,7 @@ def _terminalize_turn_submission_conn( terminal_state: str, current: str, ) -> bool: - """Advance an existing shadow row with its command receipt transaction.""" + """Advance a submission row with its command receipt transaction.""" next_state = { "accepted": "submitted", "rejected": "cancelled", @@ -20198,18 +15563,18 @@ def _submission_link_candidate_turns_conn( ) in rows: if not isinstance(source_turn_id, str) or not source_turn_id.strip(): continue - try: - candidate_owner, owner_version = _turn_link_candidate_owner_identity( - { - "id": str(worker_id), - "meta": { - "stable_key": stable_key, - "stable_key_version": stable_key_version, - }, - } - ) - except StoreSchemaError: + owner_identity = _turn_submission_owner_identity( + { + "id": str(worker_id), + "meta": { + "stable_key": stable_key, + "stable_key_version": stable_key_version, + }, + } + ) + if owner_identity is None: continue + candidate_owner, owner_version = owner_identity if owner_version != 1 or candidate_owner != str(owner_key): continue if user_text is None: @@ -20229,7 +15594,7 @@ def _submission_link_candidate_turns_conn( return candidates -def settle_submission_links_conn( +def _settle_submission_links_conn( conn: sqlite3.Connection, host_id: str, owner_key: str, @@ -20586,7 +15951,7 @@ def _settle_due_submission_links_conn( if not _submission_link_component_is_due(key, current_dt): continue next_eligible: list[datetime | None] = [] - component_changed = settle_submission_links_conn( + component_changed = _settle_submission_links_conn( conn, str(candidate_host), str(owner_key), @@ -21362,66 +16727,6 @@ def _apply_backend_pending_observation_conn( return changed -def _merge_backend_pending_conn( - conn: sqlite3.Connection, - host_id: str, - worker_id: str, - pending: Mapping[str, Any] | None, - *, - observed_at: str, -) -> bool: - """Compatibility entrypoint routed through the explicit transition helper.""" - if pending is None: - observation = PendingObservation("read_succeeded_no_prompt") - else: - clean = sanitize_public_mapping(pending) - raw_choices = clean.get("choices", []) if isinstance(clean, Mapping) else [] - normalized_choices = [ - InteractionChoice.from_dict(choice) - for choice in raw_choices - if isinstance(choice, Mapping) - ] - choices = tuple( - PendingObservedChoice( - choice_id=choice.choice_id, - label=choice.label, - picker_ordinal=ordinal, - ) - for ordinal, choice in enumerate(normalized_choices, 1) - ) - question = str(clean.get("question") or clean.get("kind") or "Pending action") - decision: dict[str, Any] | None = None - clean_meta = clean.get("meta") - if isinstance(clean_meta, Mapping) and "decision" in clean_meta: - decision = _normalize_pending_decision_meta(clean_meta["decision"]) - observation = PendingObservation( - "open_prompt", - question=question, - pending_kind=str(clean.get("kind") or "question"), - choices=choices, - revision_digest=stable_fingerprint({"legacy_pending_revision": clean}), - decision_kind=(decision or {}).get("kind"), - decision_options=tuple( - str(option["label"]) - for option in (decision or {}).get("options", []) - ), - decision_multi_select=bool( - (decision or {}).get("multi_select", False) - ), - decision_question_count=int( - (decision or {}).get("question_count", 0) - ), - ) - return _apply_backend_pending_observation_conn( - conn, - host_id, - worker_id, - observation, - observed_at=observed_at, - stale_grace_seconds=DEFAULT_PENDING_STALE_GRACE_SECONDS, - ) - - def apply_backend_pending_observation( db_path: Path | str, host_id: str, @@ -21433,13 +16738,10 @@ def apply_backend_pending_observation( binding_private_fingerprint: str | None = None, observed_turn_target_value: str | None = None, binding_authoritative: bool = False, - route_kind: Literal["legacy", "acp_permission"] = "legacy", ) -> bool: """Apply one explicit observation in a short writer transaction.""" if not _sqlite_store_exists(db_path): return False - if route_kind not in {"legacy", "acp_permission"}: - raise ValueError("invalid backend pending route kind") current_time, _ = _pending_observed_time(observed_at) with _connect(db_path, isolation_level=None) as conn: _ensure_schema(conn) @@ -21460,72 +16762,15 @@ def apply_backend_pending_observation( ), binding_authoritative=bool(binding_authoritative), ) - conn.execute( - "UPDATE backend_pending SET route_kind = ? " - "WHERE host_id = ? AND worker_id = ? " - "AND binding_private_fingerprint = ? " - "AND observed_turn_target_value = ?", - ( - route_kind, - str(host_id), - str(worker_id), - str(binding_private_fingerprint or ""), - str(observed_turn_target_value or ""), - ), - ) conn.commit() return changed except Exception: - conn.rollback() - raise - - -def merge_backend_pending( - db_path: Path | str, - host_id: str, - worker_id: str, - pending: Mapping[str, Any] | None, -) -> bool: - """Presence-sync one worker's backend-provided pending prompt.""" - if not _sqlite_store_exists(db_path): - return False - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - changed = _merge_backend_pending_conn( - conn, - host_id, - worker_id, - pending, - observed_at=utc_timestamp(), - ) - conn.commit() - return changed - except Exception: - conn.rollback() - raise - - -def list_backend_pending(db_path: Path | str, host_id: str) -> dict[str, dict[str, Any]]: - """worker_id -> normalized pending dict for every live backend-provided prompt.""" - out: dict[str, dict[str, Any]] = {} - if not _sqlite_store_exists(db_path): - return out - with _connect(db_path) as conn: - _ensure_schema(conn) - for worker_id, payload_json in conn.execute( - "SELECT worker_id, payload_json FROM backend_pending " - "WHERE host_id = ? AND observation_state = 'open'", - (host_id,), - ).fetchall(): - try: - payload = json.loads(payload_json) - except (TypeError, ValueError): - continue - if isinstance(payload, Mapping): - out[str(worker_id)] = sanitize_public_mapping(payload) - return sanitize_public_mapping(out) + conn.rollback() + raise + + + + def _backend_pending_health_from_rows( @@ -22170,8 +17415,7 @@ def claim_backend_pending_decision( """ SELECT payload_json, choice_routes_json, revision_digest, freshness, binding_private_fingerprint, - observed_turn_target_value, observation_state, - route_kind + observed_turn_target_value, observation_state FROM backend_pending WHERE host_id = ? AND worker_id = ? """, @@ -22205,11 +17449,7 @@ def claim_backend_pending_decision( return BackendPendingDecisionClaim("decision_not_pending") if context is None: conn.rollback() - return BackendPendingDecisionClaim( - "acp_authority_unavailable" - if str(row[7]) == "acp_permission" - else "decision_not_pending" - ) + return BackendPendingDecisionClaim("acp_authority_unavailable") if int(decision["question_count"]) > 1: conn.rollback() return BackendPendingDecisionClaim("unsupported_decision") @@ -22257,7 +17497,6 @@ def claim_backend_pending_decision( "option_count": option_count, "option_refs": option_refs, "text": text, - "route_kind": str(row[7]), } if not claim: conn.rollback() @@ -22269,14 +17508,13 @@ def claim_backend_pending_decision( host_id, worker_id, claim_token, revision_digest, choice_id, picker_ordinal, worker_fingerprint, binding_private_fingerprint, turn_target_value, state, - claimed_at, send_started_at, route_kind - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'claimed', ?, NULL, ?) + claimed_at, send_started_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'claimed', ?, NULL) """, ( str(host_id), str(worker_id), token, str(row[2]), _encode_decision_claim_selection(option_refs, text), picker_ordinal, context[2], context[1], context[3], current_time, - str(row[7]), ), ) conn.commit() @@ -22308,7 +17546,7 @@ def start_backend_pending_decision_send( """ SELECT worker_id, revision_digest, choice_id, worker_fingerprint, binding_private_fingerprint, - turn_target_value, state, claimed_at, route_kind + turn_target_value, state, claimed_at FROM backend_pending_claims WHERE host_id = ? AND claim_token = ? """, @@ -22336,7 +17574,7 @@ def start_backend_pending_decision_send( """ SELECT payload_json, choice_routes_json, revision_digest, freshness, binding_private_fingerprint, - observed_turn_target_value, route_kind + observed_turn_target_value FROM backend_pending WHERE host_id = ? AND worker_id = ? AND observation_state = 'open' @@ -22360,7 +17598,6 @@ def start_backend_pending_decision_send( str(current[2]) != str(row[1]) or str(current[4]) != str(row[4]) or str(current[5]) != str(row[5]) - or str(current[6]) != str(row[8]) or decision is None or validated_selection is None ): @@ -22377,7 +17614,6 @@ def start_backend_pending_decision_send( "option_count": len(decision["options"]), "option_refs": option_refs, "text": text, - "route_kind": str(row[8]), } if str(row[6]) == "send_started": conn.rollback() @@ -22671,78 +17907,6 @@ def effect(conn: sqlite3.Connection) -> None: return effect -def prune_backend_pending( - db_path: Path | str, - host_id: str, - live_binding_private_fingerprints: Iterable[str], - *, - deadline_monotonic: float | None = None, - cancelled: Callable[[], bool] | None = None, - observed_at: str | None = None, -) -> int: - """Delete state whose exact authoritative pane binding disappeared.""" - if not _sqlite_store_exists(db_path): - return 0 - live = { - str(private_fingerprint) - for private_fingerprint in live_binding_private_fingerprints - if str(private_fingerprint) - } - current_time, _ = _pending_observed_time(observed_at) - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - if not _begin_turn_refresh_transaction( - conn, - deadline_monotonic=deadline_monotonic, - cancelled=cancelled, - ): - return 0 - try: - if _turn_refresh_is_cancelled( - deadline_monotonic=deadline_monotonic, - cancelled=cancelled, - ): - conn.rollback() - return 0 - stored = [ - (str(row[0]), str(row[1]), str(row[2])) - for row in conn.execute( - """ - SELECT worker_id, binding_private_fingerprint, - observed_turn_target_value - FROM backend_pending - WHERE host_id = ? - """, - (str(host_id),), - ).fetchall() - ] - stale = [ - (worker_id, private_fingerprint, turn_target_value) - for worker_id, private_fingerprint, turn_target_value in stored - if private_fingerprint not in live - ] - for worker_id, private_fingerprint, turn_target_value in stale: - _apply_backend_pending_observation_conn( - conn, - str(host_id), - worker_id, - PendingObservation("worker_authoritatively_absent"), - observed_at=current_time, - stale_grace_seconds=DEFAULT_PENDING_STALE_GRACE_SECONDS, - binding_private_fingerprint=private_fingerprint, - observed_turn_target_value=turn_target_value, - ) - if _turn_refresh_is_cancelled( - deadline_monotonic=deadline_monotonic, - cancelled=cancelled, - ): - conn.rollback() - return 0 - conn.commit() - return len(stale) - except Exception: - conn.rollback() - raise def _decode_turn_content_rows( @@ -22986,7 +18150,6 @@ def _raw_owned_source_turn_candidates( raw_value: Any, *, owner_key: str, - source: str, kind: str, ) -> tuple[str, ...]: """Derive exact-source lookup tokens without invoking the public sanitizer. @@ -23017,15 +18180,7 @@ def _raw_owned_source_turn_candidates( }, } ) - legacy_token = "turnsrc-" + stable_fingerprint( - { - "seed": raw, - "public": {"source": str(source), "kind": str(kind)}, - } - ) - if owner_token == legacy_token: - return (owner_token,) - return owner_token, legacy_token + return (owner_token,) def _canonical_reobservation_text_matches( @@ -23052,7 +18207,6 @@ def _unchanged_owned_turn_reobservation_conn( content: Mapping[str, Any], *, observed_at: str, - turn_model: str, ) -> _TurnContentMergeResult | None: """Return a no-op merge result without decoding or sanitizing turn payloads. @@ -23112,7 +18266,6 @@ def _unchanged_owned_turn_reobservation_conn( source_candidates = _raw_owned_source_turn_candidates( content.get("source_turn_id"), owner_key=owner_key, - source=str(payload.get("source") or "snapshot"), kind=str(payload.get("kind") or "unknown"), ) if str(payload.get("source_turn_id") or "") in source_candidates: @@ -23571,22 +18724,13 @@ def _merge_observed_turn_content_conn( revision_repaired = False changed = metadata_changed or revision_changed or revision_repaired - owner_key, owner_key_version = _turn_submission_owner_identity( - current_worker_payload - ) + owner_identity = _turn_submission_owner_identity(current_worker_payload) submission_link = ( - (owner_key, instruction_fingerprint(incoming_user)) - if owner_key_version == 1 - else None - ) - candidate_owner_key, candidate_owner_key_version = ( - _turn_link_candidate_owner_identity(current_worker_payload) - ) - submission_link_rearm = ( - (candidate_owner_key, instruction_fingerprint(incoming_user)) - if candidate_owner_key_version == 1 + (owner_identity[0], instruction_fingerprint(incoming_user)) + if owner_identity is not None and owner_identity[1] == 1 else None ) + submission_link_rearm = submission_link current_revision = conn.execute( """ SELECT content_revision @@ -23618,7 +18762,6 @@ def _merge_turn_content_conn( content: Mapping[str, Any], *, observed_at: str, - turn_model: str, ) -> _TurnContentMergeResult: if not any(key in content for key in _TURN_CONTENT_FIELDS): return _TurnContentMergeResult(0) @@ -23640,7 +18783,6 @@ def _merge_turn_content_conn( current_worker_payload, content, observed_at=observed_at, - turn_model=turn_model, ) if unchanged is not None: return unchanged @@ -23778,21 +18920,11 @@ def _agent_event_authority_observed_at_conn( host_id: str, event_id: str, ) -> str | None: - """Return the immutable event time, including retained replay metadata. - - Tombstones written before schema v26 have no retained authority time. - They remain valid deduplication evidence but cannot authorize a repair. - """ + """Return the immutable authority time for a retained event.""" row = conn.execute( "SELECT observed_at FROM agent_events WHERE host_id = ? AND event_id = ?", (str(host_id), str(event_id)), ).fetchone() - if row is None: - row = conn.execute( - "SELECT observed_at FROM agent_event_tombstones " - "WHERE host_id = ? AND event_id = ?", - (str(host_id), str(event_id)), - ).fetchone() if row is None or row[0] is None: return None observed_at = _strict_utc_timestamp(row[0]) @@ -23870,8 +19002,6 @@ def append_agent_event_and_apply_turn_for_binding( *, expected_binding: WorkerBinding, content: Mapping[str, Any] | None = None, - observed_at: str | None = None, - turn_model: str = DEFAULT_TURN_MODEL, _fault_inject: Callable[[str], None] | None = None, ) -> AppendProjectedAgentEventResult: """Atomically journal an agent event and apply its text-only turn projection. @@ -23892,15 +19022,8 @@ def append_agent_event_and_apply_turn_for_binding( raise ValueError("content must be a mapping or None") if not str(content.get("source_turn_id") or "").strip(): raise ValueError("projected agent content requires source_turn_id") - normalized_turn_model = str(turn_model or "").strip().lower() - if normalized_turn_model not in TURN_MODELS: - allowed = ", ".join(sorted(TURN_MODELS)) - raise ValueError(f"turn_model must be one of: {allowed}") if _fault_inject is not None and not callable(_fault_inject): raise TypeError("_fault_inject must be callable or None") - # Validate the compatibility argument, but never grant it replay authority. - if observed_at is not None: - _pending_observed_time(observed_at) rearm_key: tuple[str, str] | None = None def fault(boundary: str) -> None: @@ -23967,7 +19090,6 @@ def fault(boundary: str) -> None: event.worker_id, content, observed_at=authority_time, - turn_model=normalized_turn_model, ) rearm_key = ( merge_result.submission_link_rearm @@ -23975,7 +19097,7 @@ def fault(boundary: str) -> None: ) if merge_result.submission_link is not None: owner_key, fingerprint = merge_result.submission_link - settle_submission_links_conn( + _settle_submission_links_conn( conn, normalized_host or "", owner_key, @@ -24012,28 +19134,17 @@ def apply_turn_refresh( worker_id: str, content: Mapping[str, Any], *, - backend_pending: Mapping[str, Any] | None | object = _UNSET, backend_pending_observation: PendingObservation | object = _UNSET, expected_binding: WorkerBinding | None = None, deadline_monotonic: float | None = None, cancelled: Callable[[], bool] | None = None, observed_at: str | None = None, pending_stale_grace_seconds: float = DEFAULT_PENDING_STALE_GRACE_SECONDS, - turn_model: str = DEFAULT_TURN_MODEL, ) -> TurnRefreshApplyResult: """Atomically apply one binding's turn observation and optional pending state.""" if not _sqlite_store_exists(db_path): return TurnRefreshApplyResult(0, False) - normalized_turn_model = str(turn_model or "").strip().lower() - if normalized_turn_model not in TURN_MODELS: - allowed = ", ".join(sorted(TURN_MODELS)) - raise ValueError(f"turn_model must be one of: {allowed}") current_time, _ = _pending_observed_time(observed_at) - if ( - backend_pending is not _UNSET - and backend_pending_observation is not _UNSET - ): - raise ValueError("provide only one backend pending observation") with _connect(db_path, isolation_level=None) as conn: _ensure_schema(conn) # Transaction acquisition must precede every merge-base read. Use @@ -24066,7 +19177,6 @@ def apply_turn_refresh( str(worker_id), content, observed_at=current_time, - turn_model=normalized_turn_model, ) updated = merge_result.updated rearm_key = ( @@ -24085,7 +19195,7 @@ def apply_turn_refresh( owner_key, fingerprint = merge_result.submission_link conn.execute("SAVEPOINT settle_submission_links") try: - settle_submission_links_conn( + _settle_submission_links_conn( conn, str(host_id), owner_key, @@ -24134,14 +19244,6 @@ def apply_turn_refresh( # pane binding for the same stable worker. binding_authoritative=expected_binding is not None, ) - elif backend_pending is not _UNSET: - pending_changed = _merge_backend_pending_conn( - conn, - str(host_id), - str(worker_id), - backend_pending, - observed_at=current_time, - ) else: pending_changed = False if _turn_refresh_is_cancelled( @@ -24157,24 +19259,6 @@ def apply_turn_refresh( raise -def merge_turn_content( - db_path: Path | str, - host_id: str, - worker_id: str, - content: Mapping[str, Any], - *, - observed_at: str | None = None, - turn_model: str = DEFAULT_TURN_MODEL, -) -> int: - """Compatibility wrapper for the transactional authoritative turn merge.""" - return apply_turn_refresh( - db_path, - host_id, - worker_id, - content, - observed_at=observed_at, - turn_model=turn_model, - ).updated def _update_turn_row( @@ -24297,16 +19381,6 @@ def _turn_delta_projection( final_byte_length=int(final_byte_length), final_page_count=int(final_page_count), final_inline=final_inline, final_preview=final_preview, )) - else: - legacy_user, legacy_user_state = _legacy_canonical_field(serialized.get("user_text")) - legacy_final, legacy_final_state = _legacy_canonical_field( - serialized.get("assistant_final_text") - ) - if legacy_user_state != "absent" or legacy_final_state != "absent": - item.update(project_turn_content( - str(turn_id), legacy_user, legacy_final, - user_state=legacy_user_state, final_state=legacy_final_state, - )) if submission_id is not None and submission_state == "linked": item["submission_id"] = str(submission_id) item["submission_state"] = "linked" @@ -24409,7 +19483,6 @@ def _turn_delta_payload_from_store( batch_sequence_ceiling: int = TURN_DELTA_MAX_BATCH_SEQUENCES, bootstrap_max_rows: int = TURN_DELTA_BOOTSTRAP_MAX_ROWS, bootstrap_max_pages: int = TURN_DELTA_BOOTSTRAP_MAX_PAGES, - turn_model: str = DEFAULT_TURN_MODEL, ) -> dict[str, Any]: """Return one atomic, frozen, byte-bounded public turn-delta page.""" started = time.perf_counter() @@ -24456,28 +19529,27 @@ def _turn_delta_payload_from_store( ), } - if _submission_linking_enabled(turn_model): - sweep_key, sweep_due = _reserve_lazy_submission_link_sweep( - db_path, - host, - purpose="submission_links", - current_clock=clock, - refresh_interval_seconds=DEFAULT_SUBMISSION_LINK_SWEEP_INTERVAL_SECONDS, - ) - if sweep_due: - try: - sweep_submission_links( - Path(db_path), - host_id=host, - now=datetime.fromtimestamp(clock, tz=timezone.utc).isoformat(), - ) - except Exception: - _release_failed_submission_link_sweep( - sweep_key, - current_clock=clock, - ) - # Delta remains available if opportunistic linkage is contended. - pass + sweep_key, sweep_due = _reserve_lazy_submission_link_sweep( + db_path, + host, + purpose="submission_links", + current_clock=clock, + refresh_interval_seconds=DEFAULT_SUBMISSION_LINK_SWEEP_INTERVAL_SECONDS, + ) + if sweep_due: + try: + _sweep_submission_links( + Path(db_path), + host_id=host, + now=datetime.fromtimestamp(clock, tz=timezone.utc).isoformat(), + ) + except Exception: + _release_failed_submission_link_sweep( + sweep_key, + current_clock=clock, + ) + # Delta remains available if opportunistic linkage is contended. + pass with _connect(db_path, isolation_level=None) as conn: _ensure_schema(conn) @@ -24761,7 +19833,6 @@ def turn_delta_payload_from_store( batch_sequence_ceiling: int = TURN_DELTA_MAX_BATCH_SEQUENCES, bootstrap_max_rows: int = TURN_DELTA_BOOTSTRAP_MAX_ROWS, bootstrap_max_pages: int = TURN_DELTA_BOOTSTRAP_MAX_PAGES, - turn_model: str = DEFAULT_TURN_MODEL, ) -> dict[str, Any]: """Fail closed to the documented public outcome for unavailable stores.""" try: @@ -24776,7 +19847,6 @@ def turn_delta_payload_from_store( batch_sequence_ceiling=batch_sequence_ceiling, bootstrap_max_rows=bootstrap_max_rows, bootstrap_max_pages=bootstrap_max_pages, - turn_model=turn_model, ) except (sqlite3.Error, StoreSchemaError, LocalStateError, OSError): return { @@ -24800,7 +19870,6 @@ def turns_payload_from_store( now: float | int | None = None, work_counters: TurnContentWorkCounters | None = None, turn_refresh_interval_seconds: float = 2.0, - turn_model: str = DEFAULT_TURN_MODEL, ) -> dict[str, Any]: """Return one insertion-stable, byte-bounded turn-list page.""" requested_schema = int(schema_version) @@ -24833,31 +19902,30 @@ def turns_payload_from_store( } current_clock = time.time() if now is None else float(now) refresh_interval = float(turn_refresh_interval_seconds) - if _submission_linking_enabled(turn_model): - submission_sweep_key, submission_sweep_due = _reserve_lazy_submission_link_sweep( - db_path, - str(host_id), - purpose="submission_links", - current_clock=current_clock, - refresh_interval_seconds=refresh_interval, - ) - if submission_sweep_due: - try: - sweep_submission_links( - Path(db_path), - host_id=str(host_id), - now=datetime.fromtimestamp( - current_clock, - tz=timezone.utc, - ).isoformat(), - ) - except Exception: - _release_failed_submission_link_sweep( - submission_sweep_key, - current_clock=current_clock, - ) - # Listing remains available if opportunistic maintenance is contended. - pass + submission_sweep_key, submission_sweep_due = _reserve_lazy_submission_link_sweep( + db_path, + str(host_id), + purpose="submission_links", + current_clock=current_clock, + refresh_interval_seconds=refresh_interval, + ) + if submission_sweep_due: + try: + _sweep_submission_links( + Path(db_path), + host_id=str(host_id), + now=datetime.fromtimestamp( + current_clock, + tz=timezone.utc, + ).isoformat(), + ) + except Exception: + _release_failed_submission_link_sweep( + submission_sweep_key, + current_clock=current_clock, + ) + # Listing remains available if opportunistic maintenance is contended. + pass try: decoded_cursor = ( decode_turn_list_cursor( @@ -25172,34 +20240,6 @@ def turns_payload_from_store( for descriptor in fields.values() ) item.update(projection) - else: - legacy_user, legacy_user_state = _legacy_canonical_field( - serialized.get("user_text") - ) - legacy_final, legacy_final_state = _legacy_canonical_field( - serialized.get("assistant_final_text") - ) - if requested_schema == TURN_LIST_SCHEMA_VERSION and ( - legacy_user_state != "absent" or legacy_final_state != "absent" - ): - item.update( - project_turn_content( - str(turn_id), - legacy_user, - legacy_final, - user_state=legacy_user_state, - final_state=legacy_final_state, - ) - ) - elif requested_schema == 1: - incompatible_v1 = incompatible_v1 or ( - legacy_user_state == "known_incomplete" - or legacy_final_state == "known_incomplete" - ) - if legacy_user_state == "complete": - item["user_text"] = legacy_user - if legacy_final_state == "complete": - item["assistant_final_text"] = legacy_final if requested_schema == 1: item["schema_version"] = 1 item.pop("content", None) @@ -25441,7 +20481,7 @@ def _ensure_turn_content_page_boundaries_conn( def _backfill_missing_turn_content_page_boundaries_conn( conn: sqlite3.Connection, ) -> int: - """Stream complete legacy fields once and persist exact non-content boundaries.""" + """Repair missing page boundaries for complete current revisions.""" repaired_fields = 0 cursor = conn.execute( """ @@ -25559,7 +20599,6 @@ def get_turn_content( field: str, cursor: str | None = None, schema_version: int = 1, - turn_model: str = DEFAULT_TURN_MODEL, work_counters: TurnContentWorkCounters | None = None, ) -> dict[str, Any]: """Read one bounded page directly from the canonical SQLite value.""" @@ -25960,15 +20999,7 @@ def _snapshot_projection_refresh_required( _snapshot_projection_freshness(payload_data) ): return True - retained_created_at = str(latest[2]) - retained_at = _strict_utc_timestamp(retained_created_at) - return retained_at is None or ( - retained_at == _LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE - and not _legacy_snapshot_created_at_is_authoritative( - retained_created_at, - latest[3], - ) - ) + return _strict_utc_timestamp(latest[2]) is None def save_snapshot( @@ -25980,13 +21011,8 @@ def save_snapshot( binding_backend: str | None = None, binding_observation_authoritative: bool = False, binding_workers_present: bool = True, - turn_model: str = DEFAULT_TURN_MODEL, ) -> bool: """Persist a canonical snapshot; return whether it became the host projection.""" - normalized_turn_model = str(turn_model or "").strip().lower() - if normalized_turn_model not in TURN_MODELS: - allowed = ", ".join(sorted(TURN_MODELS)) - raise ValueError(f"turn_model must be one of: {allowed}") context = observation or SnapshotObservationContext() binding_list = ( None @@ -26085,17 +21111,7 @@ def save_snapshot( if latest is not None else None ) - retained_is_unknown = ( - latest is None - or retained_at is None - or ( - retained_at == _LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE - and not _legacy_snapshot_created_at_is_authoritative( - retained_created_at, - latest[3], - ) - ) - ) + retained_is_unknown = latest is None or retained_at is None refresh_current = retained_is_unknown exact_replay = False if retained_at is not None and not retained_is_unknown: @@ -26352,23 +21368,6 @@ def _attention_item_from_row(row: Any) -> dict[str, Any]: ) -def list_attention_items( - db_path: Path, - host_id: str, - *, - include_resolved: bool = False, -) -> list[dict[str, Any]]: - """Return public-safe persisted attention items for a host.""" - if not _sqlite_store_exists(db_path): - return [] - with _connect(db_path) as conn: - _ensure_schema(conn) - rows = _attention_rows_conn( - conn, - host_id, - include_resolved=include_resolved, - ) - return sanitize_public_value([_attention_item_from_row(row) for row in rows]) def attention_payload_from_store( @@ -26467,16 +21466,6 @@ def attention_payload_from_store( return sanitize_public_value(payload) -def list_hosts(db_path: Path) -> list[str]: - """Return distinct host_ids seen in the store.""" - if not _sqlite_store_exists(db_path): - return [] - with _connect(db_path) as conn: - _ensure_schema(conn) - rows = conn.execute( - "SELECT DISTINCT host_id FROM snapshots ORDER BY host_id" - ).fetchall() - return sanitize_public_value([row[0] for row in rows]) def upsert_worker_bindings(db_path: Path, bindings: Iterable[WorkerBinding]) -> int: @@ -26545,56 +21534,6 @@ def list_worker_bindings( return [_worker_binding_from_row(row) for row in rows] -def resolve_worker_binding( - db_path: Path, - host_id: str, - worker_id: str, - *, - worker_fingerprint: str | None = None, - backend: str | None = None, - now: str | None = None, -) -> WorkerBinding | None: - """Resolve a single current, sendable private binding for a public worker.""" - if not _sqlite_store_exists(db_path): - return None - current_time = now or utc_timestamp() - clauses = ["host_id = ?", "worker_id = ?", "sendable = 1", "expires_at > ?"] - params: list[Any] = [str(host_id), str(worker_id), current_time] - if worker_fingerprint: - clauses.append("worker_fingerprint = ?") - params.append(str(worker_fingerprint)) - if backend is not None: - clauses.append("backend = ?") - params.append(str(backend)) - where = " AND ".join(clauses) - with _connect(db_path) as conn: - _ensure_schema(conn) - rows = conn.execute( - f""" - SELECT - host_id, - worker_id, - worker_fingerprint, - backend, - target_kind, - target_value, - turn_target_kind, - turn_target_value, - sendable, - reason, - observed_at, - expires_at, - private_fingerprint - FROM worker_bindings - WHERE {where} - ORDER BY observed_at DESC, id DESC - LIMIT 2 - """, - params, - ).fetchall() - if len(rows) != 1: - return None - return _worker_binding_from_row(rows[0]) def expire_worker_bindings( @@ -26731,15 +21670,9 @@ def _canonical_request_matches( canonical_fingerprint: str, canonical_request_json: str, public_worker_id: str, - legacy_raw_payload_fingerprint: str | None, ) -> bool: if str(row[3]) != str(action): return False - if int(row[4]) == 0: - return ( - legacy_raw_payload_fingerprint is not None - and str(row[5]) == str(legacy_raw_payload_fingerprint) - ) return ( int(row[4]) == int(canonical_version) and str(row[5]) == str(canonical_fingerprint) @@ -26774,7 +21707,6 @@ def reserve_command_request( public_worker_id: str, pending_result_json: str, selector_proof: str = "", - legacy_raw_payload_fingerprint: str | None = None, owner_lease_seconds: float = COMMAND_RECEIPT_OWNER_LEASE_SECONDS, now: str | None = None, ) -> dict[str, Any]: @@ -26819,9 +21751,6 @@ def reserve_command_request( conn.execute("BEGIN IMMEDIATE") row = _command_request_row(conn, values["host_id"], values["request_id"]) if row is not None: - if bool(row[19]): - conn.commit() - return _command_request_response("terminal", row) if not _canonical_request_matches( row, action=values["action"], @@ -26829,7 +21758,6 @@ def reserve_command_request( canonical_fingerprint=values["canonical_fingerprint"], canonical_request_json=str(canonical_request_json), public_worker_id=str(public_worker_id), - legacy_raw_payload_fingerprint=legacy_raw_payload_fingerprint, ): conn.commit() return _command_request_response("request_id_conflict", row) @@ -26883,11 +21811,10 @@ def reserve_command_request( public_worker_id, state, status, result_json, owner_token_hash, owner_expires_at, binding_fingerprint, created_at, reserved_at, send_started_at, terminal_at, - updated_at, legacy_collision, legacy_collision_count, - selector_proof + updated_at, selector_proof ) VALUES ( ?, ?, ?, ?, ?, ?, ?, 'reserved', 'pending', ?, ?, ?, NULL, - ?, ?, NULL, NULL, ?, 0, 0, ? + ?, ?, NULL, NULL, ?, ? ) """, ( @@ -26980,7 +21907,6 @@ def reserve_terminal_command_replay( status: str, result_json: str, selector_proof: str = "", - legacy_raw_payload_fingerprint: str | None = None, event_payload: Mapping[str, Any] | None = None, now: str | None = None, ) -> dict[str, Any]: @@ -27027,9 +21953,6 @@ def reserve_terminal_command_replay( conn.execute("BEGIN IMMEDIATE") row = _command_request_row(conn, values["host_id"], values["request_id"]) if row is not None: - if bool(row[19]): - conn.commit() - return _command_request_response("terminal", row) if not _canonical_request_matches( row, action=values["action"], @@ -27037,7 +21960,6 @@ def reserve_terminal_command_replay( canonical_fingerprint=values["canonical_fingerprint"], canonical_request_json=str(canonical_request_json), public_worker_id=str(public_worker_id), - legacy_raw_payload_fingerprint=legacy_raw_payload_fingerprint, ): conn.commit() return _command_request_response("request_id_conflict", row) @@ -27055,11 +21977,10 @@ def reserve_terminal_command_replay( public_worker_id, state, status, result_json, owner_token_hash, owner_expires_at, binding_fingerprint, created_at, reserved_at, send_started_at, terminal_at, - updated_at, legacy_collision, legacy_collision_count, - selector_proof + updated_at, selector_proof ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', NULL, NULL, - ?, ?, NULL, ?, ?, 0, 0, ? + ?, ?, NULL, ?, ?, ? ) """, ( @@ -27100,37 +22021,9 @@ def reserve_terminal_command_replay( conn.close() -def sweep_expired_turn_submissions( - db_path: Path, - *, - host_id: str | None = None, - now: str | None = None, -) -> int: - """Expire unlinked shadow submissions past their precomputed hard TTL. - - This Stage-2 store hook is intentionally caller-driven until a later - lifecycle stage wires submission maintenance into the daemon scheduler. - """ - if not _sqlite_store_exists(db_path): - return 0 - current = _command_request_now(now) - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - expired = _expire_turn_submissions_conn( - conn, - current=current, - host_id=None if host_id is None else str(host_id), - ) - conn.commit() - return expired - except Exception: - conn.rollback() - raise -def sweep_submission_links( +def _sweep_submission_links( db_path: Path, *, host_id: str | None = None, @@ -27189,7 +22082,7 @@ def settle_submission_link_for_request( conn.commit() return None owner_key, fingerprint = map(str, component) - changed = settle_submission_links_conn( + changed = _settle_submission_links_conn( conn, str(host_id), owner_key, @@ -27269,50 +22162,6 @@ def linked_turn_for_submission( return payload -def cancel_turn_submission( - db_path: Path, - *, - host_id: str, - request_id: str, - now: str | None = None, -) -> bool: - """Apply the shadow-ledger side of an authoritative request cancellation. - - This Stage-2 store hook is intentionally caller-driven until a later stage - introduces an authoritative production cancellation workflow. - """ - if not _sqlite_store_exists(db_path): - return False - current = _command_request_now(now) - source_states = _turn_submission_transition_sources("cancelled") - if not source_states: - return False - state_placeholders = ", ".join("?" for _ in source_states) - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - updated = conn.execute( - f""" - UPDATE turn_submissions - SET state = 'cancelled', terminal_at = ?, updated_at = ? - WHERE host_id = ? AND request_id = ? - AND linked_turn_id IS NULL - AND state IN ({state_placeholders}) - """, - ( - current, - current, - str(host_id), - str(request_id), - *source_states, - ), - ) - conn.commit() - return int(updated.rowcount or 0) == 1 - except Exception: - conn.rollback() - raise def mark_command_send_started( @@ -28096,7 +22945,7 @@ def cleanup_command_request_retention( public_worker_id, state, status, result_json, owner_token_hash, owner_expires_at, binding_fingerprint, created_at, reserved_at, send_started_at, terminal_at, - updated_at, legacy_collision, legacy_collision_count + updated_at, selector_proof FROM command_receipts WHERE id = ? """, diff --git a/tests/store_helpers.py b/tests/store_helpers.py new file mode 100644 index 0000000..0700831 --- /dev/null +++ b/tests/store_helpers.py @@ -0,0 +1,118 @@ +"""Test adapters onto supported store entry points.""" + +from __future__ import annotations + +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +from tendwire.core.agent_events import AgentEvent, AppendAgentEventResult +from tendwire.core.models import sanitize_public_mapping, stable_fingerprint +from tendwire.core.turns import PendingObservation, PendingObservedChoice +from tendwire.store.sqlite import ( + apply_turn_refresh, + attention_payload_from_store, + list_agent_events, + record_agent_event, +) + + +def apply_test_turn_refresh( + db_path: Path | str, + host_id: str, + worker_id: str, + content: Mapping[str, Any], + *, + observed_at: str | None = None, +) -> int: + return apply_turn_refresh( + db_path, + host_id, + worker_id, + content, + observed_at=observed_at, + ).updated + + +def apply_test_backend_pending( + db_path: Path | str, + host_id: str, + worker_id: str, + pending: Mapping[str, Any] | None, +) -> bool: + if pending is None: + observation = PendingObservation("read_succeeded_no_prompt") + else: + clean = sanitize_public_mapping(pending) + choices = tuple( + PendingObservedChoice( + choice_id=str(choice.get("choice_id") or choice.get("id") or ordinal), + label=str(choice.get("label") or "Option"), + picker_ordinal=ordinal, + ) + for ordinal, choice in enumerate(clean.get("choices", ()), 1) + if isinstance(choice, Mapping) + ) + observation = PendingObservation( + "open_prompt", + question=str(clean.get("question") or "Pending action"), + pending_kind=str(clean.get("kind") or "question"), + choices=choices, + revision_digest=stable_fingerprint( + {"domain": "test.pending-observation.v1", "payload": clean} + ), + ) + return apply_turn_refresh( + db_path, + host_id, + worker_id, + {}, + backend_pending_observation=observation, + ).pending_changed + + +def read_test_attention_items( + db_path: Path, + host_id: str, + *, + include_resolved: bool = False, +) -> list[dict[str, Any]]: + payload = attention_payload_from_store( + db_path, + host_id, + include_resolved=include_resolved, + ) + return [] if payload is None else list(payload["attention"]) + + +def record_test_agent_event( + db_path: Path | str, + host_id: str, + event: AgentEvent, +) -> AppendAgentEventResult: + return record_agent_event( + db_path, + host_id, + kind=event.kind, + source=event.source, + worker_id=event.worker_id, + payload=event.payload, + source_session_id=event.source_session_id, + source_turn_id=event.source_turn_id, + source_item_id=event.source_item_id, + source_message_id=event.source_message_id, + source_event_id=event.source_event_id, + source_sequence=event.source_sequence, + visibility=event.visibility, + observed_at=event.observed_at, + ) + + +def read_public_test_agent_events( + db_path: Path | str, + host_id: str, +) -> tuple[dict[str, Any], ...]: + return tuple( + item.public_dict() + for item in list_agent_events(db_path, host_id, visibility="public") + ) diff --git a/tests/test_acp_atomic_ingestion.py b/tests/test_acp_atomic_ingestion.py index e46f6f4..cd7eb4e 100644 --- a/tests/test_acp_atomic_ingestion.py +++ b/tests/test_acp_atomic_ingestion.py @@ -172,43 +172,10 @@ def test_stale_binding_writes_neither_journal_nor_projection(tmp_path: Path) -> assert _counts(config.db_path) == (0, 0) -def test_tombstoned_replay_can_repair_projection_without_reinserting_event( - tmp_path: Path, -) -> None: - config, binding = _store(tmp_path) - event = _event(binding, observed_at="2020-01-01T00:00:00+00:00") - inserted = append_agent_event_and_apply_turn_for_binding( - config.db_path, - config.host_id, - event, - expected_binding=binding, - ) - assert inserted.event.status == "inserted" - cleanup = cleanup_agent_event_retention( - config.db_path, - config.host_id, - retention_days=1, - now="2026-07-31T00:00:00+00:00", - ) - assert cleanup["tombstoned"] == 1 - assert len(list_agent_events(config.db_path, config.host_id)) == 0 - - repaired = append_agent_event_and_apply_turn_for_binding( - config.db_path, - config.host_id, - event, - expected_binding=binding, - content=_content(), - ) - assert repaired.event.status == "replayed" - assert repaired.turn is not None - assert _counts(config.db_path) == (0, 1) -@pytest.mark.parametrize("retire_original", (False, True)) def test_replay_cannot_supersede_newer_turn_or_requeue_connector( tmp_path: Path, - retire_original: bool, ) -> None: config, binding = _store(tmp_path, stable_owner=True) old = _event( @@ -227,14 +194,6 @@ def test_replay_cannot_supersede_newer_turn_or_requeue_connector( content=_content(complete=True, source_turn_id="turn-old", text="old"), ) assert first.event.status == "inserted" - if retire_original: - assert cleanup_agent_event_retention( - config.db_path, - config.host_id, - retention_days=1, - now="2021-01-01T00:00:00+00:00", - )["tombstoned"] == 1 - new = _event( binding, observed_at="2026-01-01T00:00:00+00:00", @@ -264,7 +223,6 @@ def test_replay_cannot_supersede_newer_turn_or_requeue_connector( replace(old, observed_at="2030-01-01T00:00:00+00:00"), expected_binding=binding, content=_content(complete=True, source_turn_id="turn-old", text="old"), - observed_at="2040-01-01T00:00:00+00:00", ) assert replayed.event.status == "replayed" @@ -277,10 +235,8 @@ def test_replay_cannot_supersede_newer_turn_or_requeue_connector( ).fetchall() == outbox_before -@pytest.mark.parametrize("retire_original", (False, True)) def test_replay_repairs_only_absent_projection_with_original_authority_time( tmp_path: Path, - retire_original: bool, ) -> None: config, binding = _store(tmp_path, stable_owner=True) event = _event(binding, observed_at="2020-01-01T00:00:00+00:00") @@ -291,21 +247,12 @@ def test_replay_repairs_only_absent_projection_with_original_authority_time( expected_binding=binding, ) assert inserted.event.status == "inserted" - if retire_original: - assert cleanup_agent_event_retention( - config.db_path, - config.host_id, - retention_days=1, - now="2021-01-01T00:00:00+00:00", - )["tombstoned"] == 1 - repaired = append_agent_event_and_apply_turn_for_binding( config.db_path, config.host_id, replace(event, observed_at="2030-01-01T00:00:00+00:00"), expected_binding=binding, content=_content(complete=True), - observed_at="2040-01-01T00:00:00+00:00", ) assert repaired.event.status == "replayed" assert repaired.turn is not None and repaired.turn.updated == 1 @@ -337,34 +284,6 @@ def test_replay_repairs_only_absent_projection_with_original_authority_time( ).fetchall() == outbox_before -def test_legacy_tombstone_without_authority_time_cannot_repair(tmp_path: Path) -> None: - config, binding = _store(tmp_path) - event = _event(binding, observed_at="2020-01-01T00:00:00+00:00") - append_agent_event_and_apply_turn_for_binding( - config.db_path, - config.host_id, - event, - expected_binding=binding, - ) - cleanup_agent_event_retention( - config.db_path, - config.host_id, - retention_days=1, - now="2021-01-01T00:00:00+00:00", - ) - with sqlite3.connect(config.db_path) as conn: - conn.execute("UPDATE agent_event_tombstones SET observed_at = NULL") - - replayed = append_agent_event_and_apply_turn_for_binding( - config.db_path, - config.host_id, - event, - expected_binding=binding, - content=_content(complete=True), - ) - assert replayed.event.status == "replayed" - assert replayed.turn is None - assert _counts(config.db_path) == (0, 0) def test_replay_repair_respects_binding_fence_and_rolls_back(tmp_path: Path) -> None: diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py index 107e72e..3d638a8 100644 --- a/tests/test_acp_ingestion.py +++ b/tests/test_acp_ingestion.py @@ -16,10 +16,10 @@ AppendProjectedAgentEventResult, TurnRefreshApplyResult, list_agent_events, - list_public_agent_events, save_snapshot, upsert_worker_bindings, ) +from .store_helpers import read_public_test_agent_events def _binding() -> WorkerBinding: @@ -434,7 +434,7 @@ def test_live_prompt_echo_is_suppressed_but_load_replay_user_message_is_retained assert [event.event.kind for event in events] == ["user_message", "user_message"] assert events[0].event.payload["assembled_text"] == "one question" assert events[1].event.payload["assembled_text"] == "historical question" - assert list_public_agent_events(db_path, "host-a") == () + assert read_public_test_agent_events(db_path, "host-a") == () def test_steering_prompt_appends_to_active_turn_without_resetting_identity( @@ -538,7 +538,7 @@ def test_stable_control_updates_persist_privately_and_replay_idempotently( assert events[0].event.payload["extension"] == ( f"acp.session_update.{update_kind}" ) - assert list_public_agent_events(db_path, "host-a") == () + assert read_public_test_agent_events(db_path, "host-a") == () def test_duplicate_durable_event_can_idempotently_repair_projection(tmp_path: Path) -> None: diff --git a/tests/test_acp_permissions.py b/tests/test_acp_permissions.py index 6acb602..58e53ea 100644 --- a/tests/test_acp_permissions.py +++ b/tests/test_acp_permissions.py @@ -422,31 +422,6 @@ def adapter_side() -> None: broker.close() -def test_v27_provenance_migration_preserves_stale_pending_state( - tmp_path: Path, -) -> None: - db_path = tmp_path / "v26.db" - with sqlite3.connect(db_path) as conn: - store_sqlite._run_migrations(conn, target_version=26) - conn.execute( - """ - INSERT INTO backend_pending ( - host_id, worker_id, payload_json, observed_at, - revision_digest, choice_routes_json, - binding_private_fingerprint, observed_turn_target_value, - observation_state, freshness, updated_at - ) VALUES ('host', 'worker', '{}', '2026-01-01T00:00:00+00:00', - '', '{}', '', '', 'failed', 'stale', - '2026-01-01T00:00:00+00:00') - """ - ) - conn.commit() - store_sqlite.init_store(db_path) - with sqlite3.connect(db_path) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == (28,) - assert conn.execute( - "SELECT freshness, route_kind FROM backend_pending" - ).fetchone() == ("stale", "legacy") def test_daemon_routes_acp_permission_answers_without_a_prompt_route( diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py index 74b47b6..851d690 100644 --- a/tests/test_agent_events.py +++ b/tests/test_agent_events.py @@ -1,6 +1,5 @@ from __future__ import annotations -import hashlib import sqlite3 import threading import time @@ -22,6 +21,8 @@ from tendwire.core.models import WorkerBinding from tendwire.store import sqlite as store_sqlite +from .store_helpers import record_test_agent_event, read_public_test_agent_events + # Exact production table shapes from schema v22 and v23. These fixtures do # not use the current target DDL, because doing so masks cross-version rebuild @@ -181,9 +182,9 @@ def test_append_is_ordered_and_replay_is_idempotent(tmp_path: Path) -> None: first = _message_event(sequence=10) second = _message_event(sequence=11, text="world") - inserted = store_sqlite.append_agent_event(db_path, "host-1", first) - replayed = store_sqlite.append_agent_event(db_path, "host-1", first) - later = store_sqlite.append_agent_event(db_path, "host-1", second) + inserted = record_test_agent_event(db_path, "host-1", first) + replayed = record_test_agent_event(db_path, "host-1", first) + later = record_test_agent_event(db_path, "host-1", second) assert inserted.inserted is True assert replayed.inserted is False @@ -201,10 +202,10 @@ def test_deterministic_identity_rejects_changed_replay(tmp_path: Path) -> None: original = _message_event(sequence=4, text="original") corrupt_replay = _message_event(sequence=4, text="changed") assert original.event_id == corrupt_replay.event_id - store_sqlite.append_agent_event(db_path, "host-1", original) + record_test_agent_event(db_path, "host-1", original) with pytest.raises(AgentEventIdentityConflict): - store_sqlite.append_agent_event(db_path, "host-1", corrupt_replay) + record_test_agent_event(db_path, "host-1", corrupt_replay) stored = store_sqlite.list_agent_events(db_path, "host-1") assert len(stored) == 1 @@ -235,9 +236,9 @@ def test_source_identity_reuse_cannot_evade_conflict_by_changing_kind_or_sequenc payload={"text": "hello"}, ) assert original_with_id.event_id == changed_kind.event_id - store_sqlite.append_agent_event(db_path, "host-1", original_with_id) + record_test_agent_event(db_path, "host-1", original_with_id) with pytest.raises(AgentEventIdentityConflict): - store_sqlite.append_agent_event(db_path, "host-1", changed_kind) + record_test_agent_event(db_path, "host-1", changed_kind) changed_sequence_kind = agent_event( kind="plan", @@ -248,9 +249,9 @@ def test_source_identity_reuse_cannot_evade_conflict_by_changing_kind_or_sequenc payload={"entries": []}, ) assert original.event_id == changed_sequence_kind.event_id - store_sqlite.append_agent_event(db_path, "host-2", original) + record_test_agent_event(db_path, "host-2", original) with pytest.raises(AgentEventIdentityConflict): - store_sqlite.append_agent_event(db_path, "host-2", changed_sequence_kind) + record_test_agent_event(db_path, "host-2", changed_sequence_kind) def test_private_ids_and_payload_never_enter_public_projection(tmp_path: Path) -> None: @@ -270,7 +271,7 @@ def test_private_ids_and_payload_never_enter_public_projection(tmp_path: Path) - "cwd": "/home/smith/private-repository", }, ) - store_sqlite.append_agent_event(db_path, "host-1", event) + record_test_agent_event(db_path, "host-1", event) private = store_sqlite.list_agent_events( db_path, @@ -279,7 +280,7 @@ def test_private_ids_and_payload_never_enter_public_projection(tmp_path: Path) - ) assert private[0].event.source_item_id == "item-secret" assert private[0].event.payload["cwd"] == "/home/smith/private-repository" - public = store_sqlite.list_public_agent_events(db_path, "host-1") + public = read_public_test_agent_events(db_path, "host-1") assert public[0]["payload"] == {"text": "safe status"} encoded = repr(public[0]) assert "session-secret" not in encoded @@ -310,8 +311,8 @@ def test_thought_events_are_private_and_not_publicly_listed(tmp_path: Path) -> N source_sequence=1, payload={"text": "reasoning summary"}, ) - store_sqlite.append_agent_event(db_path, "host-1", thought) - assert store_sqlite.list_public_agent_events(db_path, "host-1") == () + record_test_agent_event(db_path, "host-1", thought) + assert read_public_test_agent_events(db_path, "host-1") == () assert store_sqlite.list_agent_events(db_path, "host-1")[0].event.payload == { "text": "reasoning summary" } @@ -339,8 +340,8 @@ def test_tool_and_plan_events_are_private_only(kind: str, tmp_path: Path) -> Non payload={"content": [{"type": "text", "text": "private tool data"}]}, ) db_path = tmp_path / f"{kind}.db" - store_sqlite.append_agent_event(db_path, "host-1", event) - assert store_sqlite.list_public_agent_events(db_path, "host-1") == () + record_test_agent_event(db_path, "host-1", event) + assert read_public_test_agent_events(db_path, "host-1") == () assert "private tool data" in repr( store_sqlite.list_agent_events(db_path, "host-1") ) @@ -363,8 +364,8 @@ def test_queries_filter_worker_session_turn_and_cursor(tmp_path: Path) -> None: source_sequence=2, payload={"entries": [{"content": "test", "status": "pending"}]}, ) - first_result = store_sqlite.append_agent_event(db_path, "host-1", first) - store_sqlite.append_agent_event(db_path, "host-1", second) + first_result = record_test_agent_event(db_path, "host-1", first) + record_test_agent_event(db_path, "host-1", second) assert len( store_sqlite.list_agent_events(db_path, "host-1", worker_id="worker-1") @@ -463,7 +464,7 @@ def test_journal_payload_is_adapter_neutral_and_preserves_namespaced_extensions( source_event_id="extension-event", payload=payload, ) - store_sqlite.append_agent_event(tmp_path / "store.db", "host-1", event) + record_test_agent_event(tmp_path / "store.db", "host-1", event) stored = store_sqlite.list_agent_events(tmp_path / "store.db", "host-1") assert stored[0].event.source == "org.example.agent/v2" assert stored[0].event.payload == payload @@ -509,25 +510,18 @@ def test_observed_at_is_strict_aware_and_canonical_utc() -> None: ) -def test_store_rejects_noncanonical_public_projection(tmp_path: Path) -> None: - event = _message_event(sequence=1) - tampered = replace(event, public_payload={"session_id": "private-session"}) - with pytest.raises(ValueError, match="canonical agent event contract"): - store_sqlite.append_agent_event(tmp_path / "store.db", "host-1", tampered) - - def test_store_fails_closed_when_public_projection_is_corrupted( tmp_path: Path, ) -> None: db_path = tmp_path / "store.db" - store_sqlite.append_agent_event(db_path, "host-1", _message_event(sequence=1)) + record_test_agent_event(db_path, "host-1", _message_event(sequence=1)) with sqlite3.connect(db_path) as conn: conn.execute( "UPDATE agent_events SET public_payload_json = ?", ('{"cwd":"/home/private","text":"safe"}',), ) with pytest.raises(store_sqlite.StoreSchemaError, match="invalid_agent_event_row"): - store_sqlite.list_public_agent_events(db_path, "host-1") + read_public_test_agent_events(db_path, "host-1") def test_host_scoping_and_concurrent_replay_are_isolated(tmp_path: Path) -> None: @@ -538,14 +532,14 @@ def test_host_scoping_and_concurrent_replay_are_isolated(tmp_path: Path) -> None with ThreadPoolExecutor(max_workers=8) as executor: results = list( executor.map( - lambda _: store_sqlite.append_agent_event(db_path, "host-1", event), + lambda _: record_test_agent_event(db_path, "host-1", event), range(24), ) ) assert sum(result.inserted for result in results) == 1 assert len({result.sequence for result in results}) == 1 - other = store_sqlite.append_agent_event(db_path, "host-2", event) + other = record_test_agent_event(db_path, "host-2", event) assert other.inserted is True assert len(store_sqlite.list_agent_events(db_path, "host-1")) == 1 assert len(store_sqlite.list_agent_events(db_path, "host-2")) == 1 @@ -554,76 +548,13 @@ def test_host_scoping_and_concurrent_replay_are_isolated(tmp_path: Path) -> None store_sqlite.list_agent_events(db_path, " ") -def test_binding_guard_and_event_append_are_one_atomic_operation( - tmp_path: Path, -) -> None: - db_path = tmp_path / "store.db" - binding = WorkerBinding( - host_id="host-1", - worker_id="worker-1", - worker_fingerprint="worker-fingerprint-1", - backend="herdr", - target_kind="pane", - target_value="pane-1", - turn_target_kind="pane", - turn_target_value="pane-1", - sendable=True, - observed_at="2026-07-31T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-binding-generation-1", - ) - store_sqlite.upsert_worker_bindings(db_path, [binding]) - event = _message_event(sequence=1) - - inserted = store_sqlite.append_agent_event_for_binding( - db_path, - "host-1", - event, - expected_binding=binding, - ) - replayed = store_sqlite.append_agent_event_for_binding( - db_path, - "host-1", - event, - expected_binding=binding, - ) - assert (inserted.status, inserted.inserted, inserted.sequence) == ( - "inserted", - True, - replayed.sequence, - ) - assert (replayed.status, replayed.inserted) == ("replayed", False) - - replacement = replace( - binding, - worker_id="worker-replacement", - worker_fingerprint="worker-fingerprint-2", - observed_at="2026-07-31T00:00:01+00:00", - ) - store_sqlite.upsert_worker_bindings(db_path, [replacement]) - rejected_event = _message_event(sequence=2, text="must not persist") - rejected = store_sqlite.append_agent_event_for_binding( - db_path, - "host-1", - rejected_event, - expected_binding=binding, - ) - assert (rejected.status, rejected.sequence, rejected.inserted) == ( - "binding_changed", - None, - False, - ) - assert [ - stored.event.payload["text"] - for stored in store_sqlite.list_agent_events(db_path, "host-1") - ] == ["hello"] def test_database_constraints_and_indexes_cover_public_and_source_identity( tmp_path: Path, ) -> None: db_path = tmp_path / "store.db" - store_sqlite.append_agent_event(db_path, "host-1", _message_event(sequence=1)) + record_test_agent_event(db_path, "host-1", _message_event(sequence=1)) with sqlite3.connect(db_path) as conn: indexes = { str(row[1]) for row in conn.execute("PRAGMA index_list(agent_events)") @@ -645,7 +576,7 @@ def test_database_constraints_and_indexes_cover_public_and_source_identity( ) -def test_retention_removes_private_payload_but_preserves_replay_identity( +def test_retention_removes_private_payload_without_replay_state( tmp_path: Path, ) -> None: db_path = tmp_path / "store.db" @@ -657,8 +588,8 @@ def test_retention_removes_private_payload_but_preserves_replay_identity( _message_event(sequence=2, text="recent", visibility="private"), observed_at="2026-07-30T00:00:00+00:00", ) - inserted = store_sqlite.append_agent_event(db_path, "host-1", old) - store_sqlite.append_agent_event(db_path, "host-1", recent) + inserted = record_test_agent_event(db_path, "host-1", old) + record_test_agent_event(db_path, "host-1", recent) result = store_sqlite.cleanup_agent_event_retention( db_path, @@ -667,23 +598,19 @@ def test_retention_removes_private_payload_but_preserves_replay_identity( now="2026-07-31T00:00:00+00:00", ) - assert result["deleted"] == result["tombstoned"] == 1 + assert result["deleted"] == 1 + assert "tombstoned" not in result assert [item.event.payload["text"] for item in store_sqlite.list_agent_events(db_path, "host-1")] == ["recent"] with sqlite3.connect(db_path) as conn: - tombstone = conn.execute( - "SELECT sequence, length(replay_fingerprint) " - "FROM agent_event_tombstones WHERE host_id = ? AND event_id = ?", - ("host-1", old.event_id), - ).fetchone() encoded = "\n".join(conn.iterdump()) - assert tombstone == (inserted.sequence, 64) + assert "agent_event_tombstones" not in encoded assert "private historical payload" not in encoded - replay = store_sqlite.append_agent_event(db_path, "host-1", old) - assert replay.inserted is False - assert replay.sequence == inserted.sequence + replay = record_test_agent_event(db_path, "host-1", old) + assert replay.inserted is True + assert replay.sequence > inserted.sequence with pytest.raises(AgentEventIdentityConflict): - store_sqlite.append_agent_event( + record_test_agent_event( db_path, "host-1", _message_event(sequence=1, text="changed", visibility="private"), @@ -720,28 +647,7 @@ def test_retention_streams_metadata_for_exact_16_mib_private_row( payload=payload, observed_at="2026-06-01T00:00:00+00:00", ) - store_sqlite.append_agent_event(db_path, "host-1", event) - replay_contract = { - "event_id": event.event_id, - "kind": event.kind, - "source": event.source, - "worker_id": event.worker_id, - "visibility": event.visibility, - "source_session_id": event.source_session_id, - "source_turn_id": event.source_turn_id, - "source_item_id": event.source_item_id, - "source_message_id": event.source_message_id, - "source_event_id": event.source_event_id, - "source_sequence": event.source_sequence, - "payload_fingerprint": event.payload_fingerprint, - "public_payload_fingerprint": hashlib.sha256( - store_sqlite._canonical_json(event.public_payload).encode("utf-8") - ).hexdigest(), - } - expected_fingerprint = hashlib.sha256( - store_sqlite._canonical_json(replay_contract).encode("utf-8") - ).hexdigest() - + record_test_agent_event(db_path, "host-1", event) assert "private_payload_json" not in ( store_sqlite._AGENT_EVENT_RETENTION_SELECT.lower() ) @@ -760,14 +666,13 @@ def forbidden_full_row(_row: object) -> object: _current_bytes, peak_bytes = tracemalloc.get_traced_memory() tracemalloc.stop() - assert result["deleted"] == result["tombstoned"] == 1 + assert result["deleted"] == 1 assert peak_bytes < 8 * 1024 * 1024 with sqlite3.connect(db_path) as conn: assert conn.execute( - "SELECT replay_fingerprint FROM agent_event_tombstones " - "WHERE host_id = ? AND event_id = ?", - ("host-1", event.event_id), - ).fetchone() == (expected_fingerprint,) + "SELECT COUNT(*) FROM agent_events WHERE host_id = ?", + ("host-1",), + ).fetchone() == (0,) def test_automatic_maintenance_retires_agent_events_only_when_due( @@ -782,8 +687,8 @@ def test_automatic_maintenance_retires_agent_events_only_when_due( _message_event(sequence=2, text="recent", visibility="private"), observed_at="2026-01-31T23:00:00+00:00", ) - store_sqlite.append_agent_event(db_path, "host-1", old) - store_sqlite.append_agent_event(db_path, "host-1", recent) + record_test_agent_event(db_path, "host-1", old) + record_test_agent_event(db_path, "host-1", recent) policy = store_sqlite.SnapshotRetentionPolicy( retention_days=30, retention_count=100, @@ -802,7 +707,7 @@ def test_automatic_maintenance_retires_agent_events_only_when_due( _message_event(sequence=3, text="late old", visibility="private"), observed_at="2026-01-02T00:00:00+00:00", ) - store_sqlite.append_agent_event(db_path, "host-1", late_old) + record_test_agent_event(db_path, "host-1", late_old) not_due = store_sqlite.maybe_run_automatic_store_maintenance( db_path, policy=policy, @@ -828,525 +733,17 @@ def test_automatic_maintenance_retires_agent_events_only_when_due( item.event.payload["text"] for item in store_sqlite.list_agent_events(db_path, "host-1") ] == ["recent"] - replay = store_sqlite.append_agent_event(db_path, "host-1", old) - assert replay.inserted is False + replay = record_test_agent_event(db_path, "host-1", old) + assert replay.inserted is True with pytest.raises(AgentEventIdentityConflict): - store_sqlite.append_agent_event( + record_test_agent_event( db_path, "host-1", _message_event(sequence=1, text="changed", visibility="private"), ) -def test_automatic_agent_retention_failure_does_not_advance_cadence( - tmp_path: Path, -) -> None: - db_path = tmp_path / "automatic-agent-rollback.db" - old = replace( - _message_event(sequence=1, visibility="private"), - observed_at="2026-01-01T00:00:00+00:00", - ) - inserted = store_sqlite.append_agent_event(db_path, "host-1", old) - with sqlite3.connect(db_path) as conn: - conn.execute( - """ - INSERT INTO agent_event_tombstones ( - host_id, event_id, sequence, replay_fingerprint, retired_at - ) VALUES (?, ?, ?, ?, ?) - """, - ( - "host-1", - old.event_id, - inserted.sequence, - "0" * 64, - "2026-01-02T00:00:00+00:00", - ), - ) - - with pytest.raises(sqlite3.IntegrityError): - store_sqlite.maybe_run_automatic_store_maintenance( - db_path, - policy=store_sqlite.SnapshotRetentionPolicy( - retention_days=30, - retention_count=100, - batch_size=10, - ), - agent_event_host_id="host-1", - agent_event_retention_days=7, - now="2026-02-01T00:00:00+00:00", - ) - - with sqlite3.connect(db_path) as conn: - assert conn.execute( - "SELECT last_completed_at FROM store_maintenance_state " - "WHERE scope = 'automatic'" - ).fetchone() == (None,) - assert conn.execute("SELECT COUNT(*) FROM agent_events").fetchone() == (1,) - - -def test_retention_conflict_rolls_back_and_serializes_concurrent_append( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "retention-concurrency.db" - old = replace( - _message_event(sequence=1, visibility="private"), - observed_at="2026-01-01T00:00:00+00:00", - ) - store_sqlite.append_agent_event(db_path, "host-1", old) - entered = threading.Event() - release = threading.Event() - original = store_sqlite._agent_event_retention_candidate - - def blocking_candidate( - row: tuple[object, ...], - ) -> tuple[str, str, int, str, str]: - entered.set() - assert release.wait(timeout=5) - return original(row) - - monkeypatch.setattr( - store_sqlite, - "_agent_event_retention_candidate", - blocking_candidate, - ) - with ThreadPoolExecutor(max_workers=2) as executor: - cleanup = executor.submit( - store_sqlite.cleanup_agent_event_retention, - db_path, - "host-1", - retention_days=7, - now="2026-02-01T00:00:00+00:00", - ) - assert entered.wait(timeout=5) - append = executor.submit( - store_sqlite.append_agent_event, - db_path, - "host-1", - _message_event(sequence=2, text="new", visibility="private"), - ) - time.sleep(0.05) - assert append.done() is False - release.set() - assert cleanup.result(timeout=5)["deleted"] == 1 - assert append.result(timeout=5).inserted is True - - -def test_retention_tombstone_conflict_rolls_back_active_event(tmp_path: Path) -> None: - db_path = tmp_path / "retention-rollback.db" - old = replace( - _message_event(sequence=1, visibility="private"), - observed_at="2026-01-01T00:00:00+00:00", - ) - inserted = store_sqlite.append_agent_event(db_path, "host-1", old) - with sqlite3.connect(db_path) as conn: - conn.execute( - """ - INSERT INTO agent_event_tombstones ( - host_id, event_id, sequence, replay_fingerprint, retired_at - ) VALUES (?, ?, ?, ?, ?) - """, - ( - "host-1", - old.event_id, - inserted.sequence, - "0" * 64, - "2026-01-02T00:00:00+00:00", - ), - ) - - with pytest.raises(sqlite3.IntegrityError): - store_sqlite.cleanup_agent_event_retention( - db_path, - "host-1", - retention_days=7, - now="2026-02-01T00:00:00+00:00", - ) - - assert store_sqlite.list_agent_events(db_path, "host-1")[0].event == old - with sqlite3.connect(db_path) as conn: - assert conn.execute( - "SELECT replay_fingerprint FROM agent_event_tombstones " - "WHERE host_id = ? AND event_id = ?", - ("host-1", old.event_id), - ).fetchone() == ("0" * 64,) - - def test_journal_accepts_acp_sized_private_text(tmp_path: Path) -> None: event = _message_event(sequence=1, text="x" * (64 * 1024), visibility="private") - result = store_sqlite.append_agent_event(tmp_path / "store.db", "host-1", event) + result = record_test_agent_event(tmp_path / "store.db", "host-1", event) assert result.inserted is True - - -def test_v23_to_v24_preserves_populated_journal_sequence(tmp_path: Path) -> None: - db_path = tmp_path / "v23-populated.db" - event = _message_event(sequence=81, text="x" * (60 * 1024), visibility="private") - with sqlite3.connect(db_path) as conn: - store_sqlite._run_migrations(conn, target_version=23) - conn.execute( - """ - INSERT INTO agent_events ( - sequence, host_id, event_id, kind, source, worker_id, - visibility, source_session_id, source_turn_id, - source_item_id, source_message_id, source_event_id, - source_sequence, observed_at, payload_fingerprint, - private_payload_json, public_payload_json - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - 123, - "host-1", - event.event_id, - event.kind, - event.source, - event.worker_id, - event.visibility, - event.source_session_id, - event.source_turn_id, - event.source_item_id, - event.source_message_id, - event.source_event_id, - event.source_sequence, - event.observed_at, - event.payload_fingerprint, - store_sqlite._canonical_json(event.payload), - store_sqlite._canonical_json(event.public_payload), - ), - ) - conn.commit() - store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == ( - store_sqlite.STORE_SCHEMA_VERSION, - ) - assert conn.execute( - "SELECT sequence FROM agent_events WHERE event_id = ?", - (event.event_id,), - ).fetchone() == (123,) - assert conn.execute( - "SELECT COUNT(*) FROM agent_event_tombstones" - ).fetchone() == (0,) - - later = _message_event(sequence=82, text="later", visibility="private") - assert store_sqlite.append_agent_event(db_path, "host-1", later).sequence == 124 - - -def test_v24_to_v25_privatises_populated_tool_and_plan_rows(tmp_path: Path) -> None: - db_path = tmp_path / "v24-public-tools.db" - kinds = ("tool_call", "tool_call_update", "plan") - events = [ - agent_event( - kind=kind, - source="acp", - worker_id="worker-1", - source_session_id="private-session", - source_sequence=index, - payload={"content": [{"type": "text", "text": f"private-{kind}"}]}, - observed_at="2026-07-31T00:00:00+00:00", - ) - for index, kind in enumerate(kinds, 1) - ] - with sqlite3.connect(db_path) as conn: - store_sqlite._run_migrations(conn, target_version=24) - conn.execute("DROP TABLE IF EXISTS agent_events") - conn.execute( - store_sqlite.CREATE_AGENT_EVENTS_TABLE.replace( - """kind NOT IN ( - 'thought', 'tool_call', 'tool_call_update', 'plan', 'extension' - ) OR visibility = 'private'""", - "kind NOT IN ('thought', 'extension') OR visibility = 'private'", - ) - ) - for sequence, event in enumerate(events, 41): - conn.execute( - """ - INSERT INTO agent_events ( - sequence, host_id, event_id, kind, source, worker_id, - visibility, source_session_id, source_turn_id, - source_item_id, source_message_id, source_event_id, - source_sequence, observed_at, payload_fingerprint, - private_payload_json, public_payload_json - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - sequence, - "host-1", - event.event_id, - event.kind, - event.source, - event.worker_id, - "public", - event.source_session_id, - event.source_turn_id, - event.source_item_id, - event.source_message_id, - event.source_event_id, - event.source_sequence, - event.observed_at, - event.payload_fingerprint, - store_sqlite._canonical_json(event.payload), - store_sqlite._canonical_json(event.payload), - ), - ) - conn.commit() - store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == ( - store_sqlite.STORE_SCHEMA_VERSION, - ) - assert conn.execute( - "SELECT sequence, kind, visibility, public_payload_json " - "FROM agent_events ORDER BY sequence" - ).fetchall() == [ - (41, "tool_call", "private", "{}"), - (42, "tool_call_update", "private", "{}"), - (43, "plan", "private", "{}"), - ] - assert conn.execute("PRAGMA integrity_check").fetchone() == ("ok",) - - tmp_path.chmod(0o700) - db_path.chmod(0o600) - assert store_sqlite.list_public_agent_events(db_path, "host-1") == () - assert [ - stored.event.kind - for stored in store_sqlite.list_agent_events(db_path, "host-1") - ] == list(kinds) - - -@pytest.mark.parametrize( - ("source_version", "historical_ddl"), - ( - (22, _HISTORICAL_V22_AGENT_EVENTS_DDL), - (23, _HISTORICAL_V23_AGENT_EVENTS_DDL), - ), -) -def test_authentic_populated_public_tool_migrations_reach_current_private_schema( - tmp_path: Path, - source_version: int, - historical_ddl: str, -) -> None: - db_path = tmp_path / f"authentic-v{source_version}.db" - kinds = ("tool_call", "tool_call_update", "plan") - with sqlite3.connect(db_path) as conn: - store_sqlite._run_migrations(conn, target_version=source_version - 1) - conn.execute("DROP TABLE IF EXISTS agent_events") - conn.execute(historical_ddl) - for sequence, kind in enumerate(kinds, 51): - event = agent_event( - kind=kind, - source="acp", - worker_id="worker-1", - source_session_id="private-session", - source_sequence=sequence, - payload={"content": [{"type": "text", "text": f"private-{kind}"}]}, - observed_at="2026-07-31T00:00:00+00:00", - ) - event_id = event.event_id - if source_version == 22: - legacy_identity = { - "schema_version": 1, - "source": event.source, - "session_id": event.source_session_id, - "event_id": event.source_event_id, - "sequence": event.source_sequence, - "kind": event.kind, - } - event_id = hashlib.sha256( - store_sqlite._canonical_json(legacy_identity).encode("utf-8") - ).hexdigest() - conn.execute( - """ - INSERT INTO agent_events ( - sequence, host_id, event_id, kind, source, worker_id, - visibility, source_session_id, source_turn_id, - source_item_id, source_message_id, source_event_id, - source_sequence, observed_at, payload_fingerprint, - private_payload_json, public_payload_json - ) VALUES (?, ?, ?, ?, ?, ?, 'public', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - sequence, - "host-1", - event_id, - event.kind, - event.source, - event.worker_id, - event.source_session_id, - event.source_turn_id, - event.source_item_id, - event.source_message_id, - event.source_event_id, - event.source_sequence, - event.observed_at, - event.payload_fingerprint, - store_sqlite._canonical_json(event.payload), - store_sqlite._canonical_json(event.payload), - ), - ) - conn.execute(f"PRAGMA user_version = {source_version}") - conn.commit() - - store_sqlite._run_migrations(conn) - - assert conn.execute("PRAGMA user_version").fetchone() == ( - store_sqlite.STORE_SCHEMA_VERSION, - ) - assert conn.execute( - "SELECT sequence, kind, visibility, public_payload_json " - "FROM agent_events ORDER BY sequence" - ).fetchall() == [ - (51, "tool_call", "private", "{}"), - (52, "tool_call_update", "private", "{}"), - (53, "plan", "private", "{}"), - ] - assert conn.execute("PRAGMA integrity_check").fetchone() == ("ok",) - - -def test_v25_to_v26_retains_legacy_tombstones_as_dedup_only(tmp_path: Path) -> None: - db_path = tmp_path / "v25-tombstone.db" - legacy_event_id = "a" * 64 - with sqlite3.connect(db_path) as conn: - store_sqlite._run_migrations(conn, target_version=25) - assert "observed_at" not in { - str(row[1]) - for row in conn.execute("PRAGMA table_info(agent_event_tombstones)") - } - conn.execute( - """ - INSERT INTO agent_event_tombstones ( - host_id, event_id, sequence, replay_fingerprint, retired_at - ) VALUES ('host-1', ?, 1, ?, '2026-01-02T00:00:00+00:00') - """, - (legacy_event_id, "b" * 64), - ) - conn.commit() - store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == (28,) - assert conn.execute( - "SELECT observed_at FROM agent_event_tombstones WHERE event_id = ?", - (legacy_event_id,), - ).fetchone() == (None,) - - event = replace( - _message_event(sequence=901, visibility="private"), - observed_at="2020-01-01T00:00:00+00:00", - ) - store_sqlite.append_agent_event(db_path, "host-1", event) - assert store_sqlite.cleanup_agent_event_retention( - db_path, - "host-1", - retention_days=1, - now="2021-01-01T00:00:00+00:00", - )["tombstoned"] == 1 - with sqlite3.connect(db_path) as conn: - assert conn.execute( - "SELECT observed_at FROM agent_event_tombstones WHERE event_id = ?", - (event.event_id,), - ).fetchone() == ("2020-01-01T00:00:00+00:00",) - - -@pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) -def test_agent_event_schema_migrates_from_every_prior_version( - tmp_path: Path, - source_version: int, -) -> None: - db_path = tmp_path / f"v{source_version}.db" - with sqlite3.connect(db_path) as conn: - conn.execute("CREATE TABLE durable_sentinel (value TEXT NOT NULL)") - conn.execute("INSERT INTO durable_sentinel VALUES ('preserved')") - conn.commit() - store_sqlite._run_migrations(conn, target_version=source_version) - store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == ( - store_sqlite.STORE_SCHEMA_VERSION, - ) - assert conn.execute("SELECT value FROM durable_sentinel").fetchone() == ( - "preserved", - ) - assert conn.execute( - "SELECT COUNT(*) FROM sqlite_master " - "WHERE type = 'table' AND name = 'agent_events'" - ).fetchone() == (1,) - assert conn.execute("PRAGMA integrity_check").fetchone() == ("ok",) - - -def test_v21_migration_is_idempotent_and_preserves_existing_store( - tmp_path: Path, -) -> None: - db_path = tmp_path / "store.db" - store_sqlite.init_store(db_path) - with sqlite3.connect(db_path) as conn: - conn.execute("DROP TABLE agent_events") - conn.execute("PRAGMA user_version = 21") - - store_sqlite.init_store(db_path) - store_sqlite.init_store(db_path) - with sqlite3.connect(db_path) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == ( - store_sqlite.STORE_SCHEMA_VERSION, - ) - columns = { - str(row[1]) for row in conn.execute("PRAGMA table_info(agent_events)") - } - assert {"sequence", "event_id", "private_payload_json"} <= columns - - -def test_v22_migration_rekeys_legacy_event_identity_without_losing_sequence( - tmp_path: Path, -) -> None: - db_path = tmp_path / "v22-event.db" - event = _message_event(sequence=7) - legacy_identity = { - "schema_version": 1, - "source": event.source, - "session_id": event.source_session_id, - "event_id": event.source_event_id, - "sequence": event.source_sequence, - "kind": event.kind, - } - legacy_event_id = hashlib.sha256( - store_sqlite._canonical_json(legacy_identity).encode("utf-8") - ).hexdigest() - with sqlite3.connect(db_path) as conn: - store_sqlite._run_migrations(conn, target_version=22) - conn.execute( - """ - INSERT INTO agent_events ( - sequence, host_id, event_id, kind, source, worker_id, - visibility, source_session_id, source_turn_id, - source_item_id, source_message_id, source_event_id, - source_sequence, observed_at, payload_fingerprint, - private_payload_json, public_payload_json - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - 19, - "host-1", - legacy_event_id, - event.kind, - event.source, - event.worker_id, - event.visibility, - event.source_session_id, - event.source_turn_id, - event.source_item_id, - event.source_message_id, - event.source_event_id, - event.source_sequence, - event.observed_at, - event.payload_fingerprint, - store_sqlite._canonical_json(event.payload), - store_sqlite._canonical_json(event.public_payload), - ), - ) - conn.commit() - store_sqlite._run_migrations(conn) - row = conn.execute( - "SELECT sequence, event_id FROM agent_events" - ).fetchone() - assert row == (19, event.event_id) - assert conn.execute("PRAGMA user_version").fetchone() == ( - store_sqlite.STORE_SCHEMA_VERSION, - ) - - replay = store_sqlite.append_agent_event(db_path, "host-1", event) - assert replay.inserted is False - assert replay.sequence == 19 diff --git a/tests/test_commands.py b/tests/test_commands.py index 6a341bb..d81ea1b 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -193,7 +193,6 @@ def test_allowed_actions_frozen() -> None: "read_snapshot", "resolve_target", "send_instruction", - "answer_pending", "answer_decision", } @@ -405,51 +404,8 @@ def test_canonical_send_instruction_fingerprint_changes_for_semantics( assert changed.fingerprint != baseline.fingerprint -def test_build_canonical_answer_pending_has_hard_coded_v1_identity() -> None: - request = _answer_pending_request(request_id="not-canonical") - mutation = build_canonical_mutation(request, public_worker_id="worker-public-7") - - assert mutation.canonical_version == 1 - assert mutation.action == "answer_pending" - assert mutation.public_worker_id == "worker-public-7" - assert mutation.canonical_json == ( - '{"action":"answer_pending","canonical_version":1,"options":{},' - '"pending":{"choice_id":"choice-public",' - '"pending_fingerprint":"pending-revision","pending_id":"pending-public"},' - '"target":{"worker_id":"worker-public-7"}}' - ) - assert mutation.fingerprint == "1a88307fbc8afd0a1205eaca" - - -@pytest.mark.parametrize( - ("field", "value"), - [ - ("pending_id", "pending-public-changed"), - ("pending_fingerprint", "pending-revision-changed"), - ("choice_id", " choice-public "), - ], -) -def test_canonical_answer_pending_fingerprint_changes_for_exact_semantics( - field: str, - value: str, -) -> None: - baseline = build_canonical_mutation( - _answer_pending_request(request_id="baseline"), - public_worker_id="worker-public-7", - ) - params = { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - } - params[field] = value - changed = build_canonical_mutation( - _answer_pending_request(request_id="changed", params=params), - public_worker_id="worker-public-7", - ) - assert changed.fingerprint != baseline.fingerprint def test_command_envelope_shape_matches_contract() -> None: @@ -646,7 +602,7 @@ def test_command_envelope_rejects_inconsistent_receipt_tuples( @pytest.mark.parametrize( - "action", ["send_instruction", "answer_pending", "answer_decision"] + "action", ["send_instruction", "answer_decision"] ) @pytest.mark.parametrize("request_id", [None, "", "not canonical"]) def test_command_envelope_live_mutations_require_canonical_request_ids( @@ -657,23 +613,15 @@ def test_command_envelope_live_mutations_require_canonical_request_ids( action=action, request_id=request_id, dry_run=False, - target={"worker_id": "w-1"} if action != "answer_pending" else None, + target={"worker_id": "w-1"}, instruction={"text": "hello"} if action == "send_instruction" else None, params=( { - "pending_id": "pending-1", - "pending_fingerprint": "revision-1", - "choice_id": "choice-1", + "decision_ref": "decision-1", + "selection": {"option_refs": ["1"]}, } - if action == "answer_pending" - else ( - { - "decision_ref": "decision-1", - "selection": {"option_refs": ["1"]}, - } - if action == "answer_decision" - else None - ) + if action == "answer_decision" + else None ), ) @@ -802,7 +750,7 @@ def test_answer_in_progress_allows_only_retryable_dispositions() -> None: @pytest.mark.parametrize( - "action", ["send_instruction", "answer_pending", "answer_decision"] + "action", ["send_instruction", "answer_decision"] ) @pytest.mark.parametrize("ok", [False, True]) @pytest.mark.parametrize("status", sorted(VALID_STATUSES)) @@ -833,7 +781,7 @@ def test_command_envelope_from_dict_enforces_terminal_rejected_matrix( @pytest.mark.parametrize( - "action", ["send_instruction", "answer_pending", "answer_decision"] + "action", ["send_instruction", "answer_decision"] ) @pytest.mark.parametrize("dry_run", [False, True], ids=["live", "dry-run"]) @pytest.mark.parametrize("ok", [False, True]) @@ -1349,7 +1297,6 @@ def test_mutation_request_id_accepts_exact_ascii_tokens_and_roundtrips( target={"worker_id": "w-1"}, instruction={"text": "ok"}, ), - _answer_pending_request(request_id=request_id), ] assert is_valid_request_id(request_id) @@ -1417,7 +1364,6 @@ def test_mutation_request_id_rejects_everything_outside_exact_ascii_grammar( target={"worker_id": "w-1"}, instruction={"text": "ok"}, ), - _answer_pending_request(request_id=request_id), ] assert not is_valid_request_id(request_id) @@ -1437,130 +1383,14 @@ def test_mutation_request_id_rejects_everything_outside_exact_ascii_grammar( assert parsed_error["code"] == STATUS_INVALID_REQUEST -def _answer_pending_request( - *, - request_id: str | None = "answer-1", - dry_run: bool = False, - params: Any = None, -) -> CommandRequest: - return CommandRequest( - action="answer_pending", - request_id=request_id, - dry_run=dry_run, - params=params - if params is not None - else { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - }, - ) -def test_parse_answer_pending_accepts_exact_opaque_params() -> None: - payload = { - "schema_version": 1, - "action": "answer_pending", - "request_id": "answer-1", - "dry_run": False, - "params": { - "pending_id": " opaque pending ", - "pending_fingerprint": " opaque revision ", - "choice_id": " opaque choice ", - }, - } - request, parse_error = parse_command_request(json.dumps(payload)) - assert parse_error is None - assert request is not None - assert validate_request(request) is None - assert request.params == payload["params"] -@pytest.mark.parametrize( - "changes,field", - [ - ({"target": {"worker_id": "w-1"}}, "target"), - ({"instruction": {"text": "private"}}, "instruction"), - ({"params": None}, "params"), - ({"params": {}}, "params"), - ( - { - "params": { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - "extra": "no", - } - }, - "params", - ), - ( - { - "params": { - "pending_id": "", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - } - }, - "params.pending_id", - ), - ( - { - "params": { - "pending_id": "pending-public", - "pending_fingerprint": " \t", - "choice_id": "choice-public", - } - }, - "params.pending_fingerprint", - ), - ( - { - "params": { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": 1, - } - }, - "params.choice_id", - ), - ], -) -def test_validate_answer_pending_rejects_non_exact_shape( - changes: dict[str, Any], - field: str, -) -> None: - request = _answer_pending_request() - data = request.to_dict() - data.update(changes) - - error = validate_request(CommandRequest.from_dict(data)) - - assert error is not None - assert error["code"] == STATUS_INVALID_REQUEST - assert field in str(error) - - -@pytest.mark.parametrize( - "request_id", - [None, "", " \t", " leading", "trailing ", "\twrapped\t"], -) -def test_validate_answer_pending_non_dry_run_requires_canonical_request_id( - request_id: str | None, -) -> None: - error = validate_request(_answer_pending_request(request_id=request_id)) - - assert error is not None - assert error["code"] == STATUS_INVALID_REQUEST - assert "request_id" in error["message"] -def test_validate_answer_pending_dry_run_does_not_require_request_id() -> None: - assert validate_request( - _answer_pending_request(request_id=None, dry_run=True) - ) is None @pytest.mark.parametrize( diff --git a/tests/test_config.py b/tests/test_config.py index f217944..d6ce246 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -19,7 +19,6 @@ DEFAULT_COMMAND_RETRY_HORIZON_SECONDS, DEFAULT_SUBMISSION_HARD_TTL_SECONDS, DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS, - DEFAULT_TURN_MODEL, MAX_COMMAND_RETRY_HORIZON_SECONDS, MIN_COMMAND_RECEIPT_RETENTION_SECONDS, MAX_MAINTENANCE_CADENCE_SECONDS, @@ -114,7 +113,6 @@ def test_acp_bounds_reject_invalid_values(field: str, value: object) -> None: def test_runtime_turn_model_modes_are_removed(monkeypatch) -> None: monkeypatch.delenv("TENDWIRE_TURN_MODEL", raising=False) - assert DEFAULT_TURN_MODEL == "observed" assert not hasattr(load_config(), "turn_model") monkeypatch.setenv("TENDWIRE_TURN_MODEL", "shadow") diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index 97f3919..f450118 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -1281,165 +1281,6 @@ def test_lifecycle_delivery_key_survives_fail_defer_reclaim_and_ack(tmp_path: Pa assert [row[2] for row in delivery_rows] == ["failed", "deferred", "expired", "delivered"] -def test_migration_terminalizes_noncanonical_live_leases_through_public_helpers( - tmp_path: Path, -) -> None: - db_path = tmp_path / "leased-duplicate-migration.db" - host_id = "host-migration-leases" - observed_at = "2026-01-01T00:00:00+00:00" - snapshot = Snapshot( - host_id=host_id, - updated_at=observed_at, - attention=[ - AttentionSignal( - kind="worker_status", - severity="warning", - status="waiting", - reason="Review the worker", - source="worker:worker-1", - updated_at=observed_at, - host_id=host_id, - ) - ], - ) - save_snapshot( - db_path, - snapshot, - observation=SnapshotObservationContext( - authority="complete", - observed_at=observed_at, - ), - ) - - with sqlite3.connect(str(db_path)) as conn: - for suffix in range(1, 4): - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) - SELECT - host_id, connector, ?, 'queued', payload_json, - '{}', created_at, updated_at, NULL - FROM connector_outbox - WHERE host_id = ? AND connector = 'attention' - ORDER BY id - LIMIT 1 - """, - (f"legacy-duplicate-{suffix}", host_id), - ) - - canonical = poll_connector_outbox( - db_path, - host_id, - "attention", - lease_seconds=100, - now="2026-01-01T00:00:01+00:00", - )["items"][0] - failed_lease = poll_connector_outbox( - db_path, - host_id, - "attention", - lease_seconds=100, - now="2026-01-01T00:00:02+00:00", - )["items"][0] - deferred_lease = poll_connector_outbox( - db_path, - host_id, - "attention", - lease_seconds=100, - now="2026-01-01T00:00:03+00:00", - )["items"][0] - expiring_lease = poll_connector_outbox( - db_path, - host_id, - "attention", - lease_seconds=5, - now="2026-01-01T00:00:04+00:00", - )["items"][0] - - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA user_version = 4") - init_store(db_path) - - failed = fail_connector_delivery( - db_path, - host_id=host_id, - name="attention", - ref=failed_lease["ref"], - delay_seconds=0, - now="2026-01-01T00:00:05+00:00", - ) - deferred = defer_connector_delivery( - db_path, - host_id=host_id, - name="attention", - ref=deferred_lease["ref"], - delay_seconds=0, - now="2026-01-01T00:00:06+00:00", - ) - reclaimed = reclaim_expired_connector_leases( - db_path, - host_id, - "attention", - now="2026-01-01T00:00:09+00:00", - ) - acknowledged = ack_connector_delivery( - db_path, - host_id=host_id, - name="attention", - ref=canonical["ref"], - now="2026-01-01T00:00:10+00:00", - ) - after = poll_connector_outbox( - db_path, - host_id, - "attention", - now="2026-01-01T00:00:11+00:00", - ) - - assert failed["status"] == "superseded" - assert failed["key"] == failed_lease["key"] - assert deferred["status"] == "superseded" - assert deferred["key"] == deferred_lease["key"] - assert reclaimed["reclaimed"] == 1 - assert acknowledged["status"] == "acknowledged" - assert acknowledged["key"] == canonical["key"] - assert after["items"] == [] - - with sqlite3.connect(str(db_path)) as conn: - outbox_rows = conn.execute( - """ - SELECT delivery_key, status - FROM connector_outbox - WHERE host_id = ? AND connector = 'attention' - ORDER BY id - """, - (host_id,), - ).fetchall() - delivery_rows = conn.execute( - """ - SELECT delivery_key, status - FROM connector_deliveries - WHERE host_id = ? AND connector = 'attention' - ORDER BY id - """, - (host_id,), - ).fetchall() - - assert outbox_rows == [ - (canonical["key"], "delivered"), - (failed_lease["key"], "superseded"), - (deferred_lease["key"], "superseded"), - (expiring_lease["key"], "superseded"), - ] - assert delivery_rows == [ - (canonical["key"], "delivered"), - (failed_lease["key"], "failed"), - (deferred_lease["key"], "deferred"), - (expiring_lease["key"], "expired"), - ] def _canonical_turn( @@ -1765,83 +1606,6 @@ def _downgrade_presentation_schema_to_v6(db_path: Path) -> None: ) -def test_v6_to_current_plan_migration_is_bounded_atomic_and_preserves_jobs( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "presentation-v6.db" - turn_id, revision = _canonical_turn(db_path, final_text="abcdefgh") - api = ConnectorOutboxAPI(db_path, "host-a") - plan = _stage_final_plan( - api, - turn_id=turn_id, - revision=revision, - ranges=[(0, 4), (4, 8)], - ) - _downgrade_presentation_schema_to_v6(db_path) - - init_store(db_path) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - version = conn.execute("PRAGMA user_version").fetchone()[0] - plan_row = conn.execute( - """ - SELECT plan_token, generation, recovers_plan_token, state - FROM turn_presentation_plans - """ - ).fetchone() - job_count = conn.execute( - "SELECT COUNT(*) FROM turn_presentation_jobs" - ).fetchone()[0] - outbox_count = conn.execute( - "SELECT COUNT(*) FROM connector_outbox WHERE connector = 'turn-final'" - ).fetchone()[0] - audit_columns = { - row[1] - for row in conn.execute( - "PRAGMA table_info(turn_presentation_recoveries)" - ).fetchall() - } - foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert version == store_sqlite.STORE_SCHEMA_VERSION == 28 - assert plan_row == (plan["plan_token"], 1, None, "active") - assert job_count == 2 - assert outbox_count == 3 - assert { - "request_id", - "failed_plan_id", - "recovered_plan_id", - "generation", - "delivered_prefix_count", - "fresh_job_count", - "retained_failed_job_count", - "prior_attempt_count", - } <= audit_columns - assert foreign_keys == [] - - _downgrade_presentation_schema_to_v6(db_path) - - def fail_rebuild(_conn: sqlite3.Connection) -> None: - raise RuntimeError("controlled v7 migration failure") - - monkeypatch.setattr( - store_sqlite, - "_rebuild_v6_presentation_plans_conn", - fail_rebuild, - ) - with pytest.raises(RuntimeError, match="controlled v7 migration failure"): - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == 6 - assert "generation" not in { - row[1] - for row in conn.execute( - "PRAGMA table_info(turn_presentation_plans)" - ).fetchall() - } - assert conn.execute( - "SELECT COUNT(*) FROM turn_presentation_jobs" - ).fetchone()[0] == 2 def test_prepare_stages_idempotently_and_rejects_conflicts_or_incomplete_coverage( @@ -3639,162 +3403,3 @@ def test_awaiting_ack_without_plan_becomes_terminal_failed(tmp_path: Path) -> No "classification": "plan_unrecoverable", "terminalized_at": "2026-01-01T00:00:01+00:00", } - - -def test_v16_migration_backfills_ordering_and_awaiting_ack_deadlines( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "connector-v16-migration.db" - stable_key = "wsk1_" + ("c" * 64) - enqueue_stable_key = "wsk1_" + ("d" * 64) - tombstoned_turn, _ = _canonical_turn( - db_path, - worker_id="worker-tombstoned", - source_turn_id="source-tombstoned", - stable_key=stable_key, - final_text="done", - ) - fallback_turn, _ = _canonical_turn( - db_path, - worker_id="worker-fallback", - source_turn_id="source-fallback", - stable_key="invalid", - final_text="done", - ) - deleted_turns = [ - _canonical_turn( - db_path, - worker_id=f"worker-deleted-{suffix}", - source_turn_id=f"source-deleted-{suffix}", - final_text="done", - )[0] - for suffix in ("a", "b") - ] - with sqlite3.connect(str(db_path)) as conn: - assert store_sqlite._migrate_tombstone_command_turn_conn( - conn, - "host-a", - tombstoned_turn, - superseded_by_turn_id=None, - superseded_at="2026-01-02T00:00:00+00:00", - ) - for turn_id, status in ( - (tombstoned_turn, "awaiting_ack"), - (fallback_turn, "dead_letter"), - (fallback_turn, "superseded"), - ): - payload = ( - { - "stable_key": enqueue_stable_key, - "stable_key_version": 1, - } - if status == "dead_letter" - else {} - ) - cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, delivery_kind, turn_id, - ordering_key, status, payload_json, private_state_json, - created_at, updated_at - ) VALUES (?, 'turn-final', ?, 'final_ready', ?, '', ?, ?, '{}', ?, ?) - """, - ( - "host-a", - f"migration-{turn_id}-{status}", - turn_id, - status, - json.dumps(payload), - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - if status == "awaiting_ack": - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, - status, response_json, private_state_json, created_at - ) VALUES (?, 'host-a', 'turn-final', 'migration-awaiting', 1, - 'awaiting_ack', '{}', '{}', ?) - """, - (cursor.lastrowid, "2026-01-01T00:00:00+00:00"), - ) - for deleted_turn in deleted_turns: - conn.execute( - "DELETE FROM turn_content_revisions WHERE host_id = ? AND turn_id = ?", - ("host-a", deleted_turn), - ) - conn.execute( - "DELETE FROM turns WHERE host_id = ? AND turn_id = ?", - ("host-a", deleted_turn), - ) - orphan_ids: list[int] = [] - for suffix, deleted_turn in zip(("a", "b"), deleted_turns, strict=True): - cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, delivery_kind, turn_id, - ordering_key, status, payload_json, private_state_json, - created_at, updated_at - ) VALUES ( - 'host-a', 'turn-final', ?, 'final_ready', ?, '', - 'dead_letter', '{}', '{}', ?, ? - ) - """, - ( - f"migration-deleted-{suffix}", - deleted_turn, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - orphan_ids.append(int(cursor.lastrowid)) - conn.execute("DROP INDEX IF EXISTS idx_connector_outbox_final_ordering") - conn.execute("ALTER TABLE connector_outbox DROP COLUMN ordering_key") - conn.execute("PRAGMA user_version = 15") - migration_now = "2026-01-03T00:00:00+00:00" - monkeypatch.setenv("TENDWIRE_CONNECTOR_ACK_TTL_SECONDS", "999") - monkeypatch.setattr(store_sqlite, "utc_timestamp", lambda: migration_now) - init_store(db_path, connector_ack_ttl_seconds=123) - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - """ - SELECT delivery_key, turn_id, status, ordering_key, private_state_json - FROM connector_outbox - WHERE delivery_key LIKE 'migration-%' - ORDER BY id - """ - ).fetchall() - delivery_private = conn.execute( - "SELECT private_state_json FROM connector_deliveries WHERE status = 'awaiting_ack'" - ).fetchone()[0] - by_key = {row[0]: row[1:] for row in rows} - tombstoned_key = f"migration-{tombstoned_turn}-awaiting_ack" - dead_letter_key = f"migration-{fallback_turn}-dead_letter" - superseded_key = f"migration-{fallback_turn}-superseded" - assert by_key[tombstoned_key][0:3] == ( - tombstoned_turn, - "awaiting_ack", - stable_key, - ) - ack_deadline = json.loads(by_key[tombstoned_key][3])["ack_deadline_at"] - assert ( - datetime.fromisoformat(ack_deadline) - - datetime.fromisoformat(migration_now) - ).total_seconds() == 123 - assert by_key[dead_letter_key][0:3] == ( - fallback_turn, - "dead_letter", - enqueue_stable_key, - ) - assert by_key[superseded_key][0:3] == ( - fallback_turn, - "superseded", - "worker-fallback", - ) - assert by_key["migration-deleted-a"][2] == f"orphan:{orphan_ids[0]}" - assert by_key["migration-deleted-b"][2] == f"orphan:{orphan_ids[1]}" - assert by_key["migration-deleted-a"][2] != by_key["migration-deleted-b"][2] - assert json.loads(delivery_private)["ack_deadline_at"] == ack_deadline diff --git a/tests/test_daemon.py b/tests/test_daemon.py index 348d9f2..b661dc6 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -305,74 +305,6 @@ def test_daemon_api_required_methods_are_public_safe() -> None: _assert_no_public_json_forbidden(command_response) -def test_daemon_answer_pending_response_is_recursively_public_safe() -> None: - snapshot = _public_snapshot() - request = CommandRequest( - action="answer_pending", - request_id="answer-public", - dry_run=False, - params={ - "pending_id": "pending-" + ("a" * 24), - "pending_fingerprint": "b" * 24, - "choice_id": "choice-" + ("c" * 24), - }, - ) - api = TendwireDaemonAPI( - get_snapshot=lambda: snapshot, - get_health=lambda: {"schema_version": 1, "status": "ok"}, - submit_command=lambda _params: CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result={ - "target": { - "worker_id": "worker-public", - "pane_id": "sentinel-private-pane", - "private_binding": "sentinel-private-binding", - }, - "pending": { - "id": "pending-" + ("a" * 24), - "fingerprint": "b" * 24, - "decision_id": "sentinel-private-decision", - }, - "choice": { - "choice_id": "choice-" + ("c" * 24), - "tool_id": "sentinel-private-tool", - "raw_payload": "sentinel-private-option", - }, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - }, - ), - ) - - response = api.dispatch( - { - "method": "command.submit", - "params": request.to_dict(), - } - ) - result = response["result"]["result"] - - assert response["schema_version"] == 1 - assert response["ok"] is True - assert response["result"]["schema_version"] == 2 - assert response["result"]["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - assert result == { - "target": {"worker_id": "worker-public"}, - "pending": { - "id": "pending-" + ("a" * 24), - "fingerprint": "b" * 24, - }, - "choice": {"choice_id": "choice-" + ("c" * 24)}, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - } - assert "sentinel-private" not in json.dumps(response, sort_keys=True) - _assert_no_public_json_forbidden(response) @pytest.mark.parametrize( @@ -1000,7 +932,6 @@ def project( "limit": 17, "cursor": "twlist1.public", "since": None, - "turn_model": "observed", } for call in projection_calls ) @@ -2188,7 +2119,6 @@ def maintenance( policy: Any, agent_event_host_id: str | None = None, agent_event_retention_days: int | None = None, - turn_model: str = "legacy", acknowledged_final_retention_days: int = 30, acknowledged_final_retention_count: int = 4096, command_retry_horizon_seconds: int = 604_800, @@ -2198,7 +2128,6 @@ def maintenance( now: str | None = None, ) -> dict[str, Any]: assert now is None - assert turn_model == "observed" calls.append( ( path, diff --git a/tests/test_delivery_retention.py b/tests/test_delivery_retention.py index a39babc..75b7423 100644 --- a/tests/test_delivery_retention.py +++ b/tests/test_delivery_retention.py @@ -19,12 +19,12 @@ from tendwire.store.sqlite import ( cleanup_acknowledged_final_retention, init_store, - merge_turn_content, reclaim_expired_connector_leases, save_snapshot, store_status, turns_payload_from_store, ) +from .store_helpers import apply_test_turn_refresh HOST_ID = "retention-host" @@ -91,7 +91,7 @@ def _merge_final( observed_at: str, host_id: str = HOST_ID, ) -> dict[str, Any]: - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, WORKER_ID, @@ -369,7 +369,7 @@ def test_dead_letter_final_blocks_no_later_finals(tmp_path: Path) -> None: final_text="first owner later final", observed_at="2026-01-01T00:01:00+00:00", ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, second_worker_id, @@ -696,7 +696,7 @@ def test_new_authoritative_revision_supersedes_stale_lease_without_double_send( _finish_source(api, current_source, version="retention-current-revision") assert _anchor_state(db_path, current_source["key"])[0] == "delivered" - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -708,7 +708,7 @@ def test_new_authoritative_revision_supersedes_stale_lease_without_double_send( }, observed_at="2026-01-03T00:00:00+00:00", ) == 0 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -803,7 +803,7 @@ def test_concurrent_final_merges_create_unique_anchors_drained_in_durable_order( def merge_one(index: int) -> int: barrier.wait(timeout=10) - return merge_turn_content( + return apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, diff --git a/tests/test_delivery_retention_hardening.py b/tests/test_delivery_retention_hardening.py index fff2239..d6b8dea 100644 --- a/tests/test_delivery_retention_hardening.py +++ b/tests/test_delivery_retention_hardening.py @@ -18,13 +18,13 @@ cleanup_acknowledged_final_retention, init_store, inspect_connector_outbox, - merge_turn_content, maybe_run_automatic_store_maintenance, SnapshotRetentionPolicy, reclaim_expired_connector_leases, save_snapshot, store_status, ) +from .store_helpers import apply_test_turn_refresh FINAL_NAME = "turn-final" @@ -303,7 +303,7 @@ def _deliver_final( text: str, observed_at: str, ) -> str: - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -534,7 +534,7 @@ def test_turn_final_fail_and_defer_keep_public_reason_contract_private( db_path = tmp_path / "turn-final-reason-codes.db" host_id = "reason-code-host" _snapshot, api = _new_delivery_store(db_path, host_id) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -1065,7 +1065,7 @@ def owner_snapshot(stable_key: str, worker_id: str, space_id: str, second: int) worker_b = owner_snapshot(STABLE_KEY, "worker-tombstone-b", "space-tombstone-b", 2) assert worker_b.workers[0].fingerprint != worker_a.workers[0].fingerprint assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-tombstone-b", @@ -1105,7 +1105,7 @@ def owner_snapshot(stable_key: str, worker_id: str, space_id: str, second: int) init_store(db_path) assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-tombstone-b", @@ -1149,7 +1149,7 @@ def owner_snapshot(stable_key: str, worker_id: str, space_id: str, second: int) 4, ) assert save_snapshot(db_path, owner_k2) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-tombstone-b", diff --git a/tests/test_delivery_retention_migration.py b/tests/test_delivery_retention_migration.py deleted file mode 100644 index b7d92a0..0000000 --- a/tests/test_delivery_retention_migration.py +++ /dev/null @@ -1,1792 +0,0 @@ -"""Behavioral coverage for the schema-v11 final-delivery migration.""" - -from __future__ import annotations - -import json -import sqlite3 -from datetime import datetime, timezone -from pathlib import Path -from typing import Any - -import pytest - -from tendwire.config import Config -from tendwire.core.projector import project_from_raw -from tendwire.connectors import ConnectorOutboxAPI -from tendwire.store import sqlite as store_sqlite -from tendwire.store.sqlite import init_store, merge_turn_content, save_snapshot - - -_HOST_ID = "host-migration" -_CREATED_AT = "2026-01-01T00:00:00+00:00" -_RAW_USER_MARKER = "private-user-chat_id-telegram" -_RAW_FINAL_MARKER = "private-final-bot_token-herdres" -_PRIVATE_ROUTE_MARKER = "private-route-topic_id" -_V11_OUTBOX_COLUMNS = {"delivery_kind", "turn_id", "content_revision"} -_STABLE_KEY = "wsk1_" + ("c" * 64) -_STABLE_KEY_2 = "wsk1_" + ("d" * 64) -_HISTORICAL_RAW_SOURCE = "legacy-migration-source" -_HISTORICAL_SOURCE_TOKEN = "turnsrc-251c9c4adc3e3ad33ceb344b" -_HISTORICAL_TURN_ID = "turn-8f770eac5cd028d511a2b0f1" -_HISTORICAL_REVISION = ( - "twrev1.kiyTUuBR3pSWUjD5Vke4eAao7SB9y2r3P6P9eRzsPO8" -) -_HISTORICAL_FINAL_IDENTITY = ( - "twfinal1.j3LZtRHojHQKn8Fmae_OMdQ1TCunF51DpO_EYzxmDP4" -) -_HISTORICAL_FINAL_KEY = ( - "turn-final:revision:twfinal1.j3LZtRHojHQKn8Fmae_OMdQ1TCunF51DpO_EYzxmDP4" -) -_HISTORICAL_USER_TEXT = "migration continuity prompt" -_HISTORICAL_FINAL_TEXT = "migration continuity final" -_HISTORICAL_PART_OUTBOX_ID = 4101 -_HISTORICAL_ROOT_OUTBOX_ID = 4102 -_HISTORICAL_PLAN_ID = 5101 -_HISTORICAL_JOB_ID = 6101 -_HISTORICAL_DELIVERY_ID = 7101 -_HISTORICAL_LIST_SEQUENCE = 41 - - -def _columns(conn: sqlite3.Connection, table: str) -> set[str]: - return {str(row[1]) for row in conn.execute(f"PRAGMA table_info({table})")} - - -def _assert_v10_shape(conn: sqlite3.Connection) -> None: - assert int(conn.execute("PRAGMA user_version").fetchone()[0]) == 10 - assert _V11_OUTBOX_COLUMNS.isdisjoint(_columns(conn, "connector_outbox")) - assert "source_outbox_id" not in _columns(conn, "turn_presentation_plans") - - -def _create_v10_store(db_path: Path) -> None: - """Create the historical schema through migrations, then restore v10 table shapes. - - The migration registry creates every earlier schema programmatically. Current - CREATE constants contain additive v11 columns, so the two affected empty - tables are put back into their exact pre-v11 shape before fixture rows are - inserted. - """ - - with store_sqlite._connect( - db_path, - prepare=True, - isolation_level=None, - ) as conn: - store_sqlite._run_migrations(conn, target_version=10) - - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys = OFF") - conn.execute("DROP TABLE turn_presentation_jobs") - conn.execute("DROP TABLE turn_presentation_plans") - for column in sorted(_V11_OUTBOX_COLUMNS): - conn.execute(f"ALTER TABLE connector_outbox DROP COLUMN {column}") - conn.executescript( - """ - CREATE TABLE turn_presentation_plans ( - id INTEGER PRIMARY KEY, - host_id TEXT NOT NULL, - name TEXT NOT NULL, - plan_token TEXT NOT NULL, - turn_id TEXT NOT NULL, - content_revision TEXT NOT NULL, - presentation_version TEXT NOT NULL, - generation INTEGER NOT NULL DEFAULT 1 CHECK (generation >= 1), - part_count INTEGER NOT NULL CHECK (part_count > 0), - state TEXT NOT NULL CHECK (state IN ( - 'preparing', - 'waiting_predecessor', - 'active', - 'completed', - 'superseded', - 'failed' - )), - replaces_plan_token TEXT, - recovers_plan_token TEXT, - created_at TEXT NOT NULL, - activated_at TEXT, - completed_at TEXT, - UNIQUE (host_id, name, plan_token), - UNIQUE ( - host_id, - name, - turn_id, - content_revision, - presentation_version, - generation - ), - FOREIGN KEY (host_id, turn_id, content_revision) - REFERENCES turn_content_revisions( - host_id, - turn_id, - content_revision - ) ON DELETE RESTRICT - ); - CREATE TABLE turn_presentation_jobs ( - id INTEGER PRIMARY KEY, - plan_id INTEGER NOT NULL, - sequence_index INTEGER NOT NULL CHECK (sequence_index >= 0), - operation TEXT NOT NULL CHECK (operation IN ('upsert', 'retire')), - part_ordinal INTEGER NOT NULL CHECK (part_ordinal >= 0), - spans_json TEXT NOT NULL, - outbox_id INTEGER UNIQUE, - created_at TEXT NOT NULL, - UNIQUE (plan_id, sequence_index), - UNIQUE (plan_id, operation, part_ordinal), - FOREIGN KEY (plan_id) - REFERENCES turn_presentation_plans(id) ON DELETE CASCADE, - FOREIGN KEY (outbox_id) - REFERENCES connector_outbox(id) ON DELETE RESTRICT - ); - CREATE INDEX idx_turn_presentation_jobs_plan_sequence - ON turn_presentation_jobs(plan_id, sequence_index); - CREATE INDEX idx_turn_presentation_jobs_outbox - ON turn_presentation_jobs(outbox_id); - PRAGMA user_version = 10; - """ - ) - conn.execute("PRAGMA foreign_keys = ON") - _assert_v10_shape(conn) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - -def _insert_current_final( - conn: sqlite3.Connection, - *, - turn_id: str, - user_text: str, - final_text: str, -) -> str: - revision = store_sqlite.content_revision( - turn_id, - user_text, - final_text, - "complete", - "complete", - ) - user_segments = store_sqlite.segment_canonical_text(user_text) - final_segments = store_sqlite.segment_canonical_text(final_text) - conn.execute( - """ - INSERT INTO turns ( - host_id, - turn_id, - worker_id, - worker_fingerprint, - space_id, - status, - kind, - updated_at, - fingerprint, - snapshot_content_fingerprint, - observed_at, - payload_json, - list_sequence - ) VALUES ( - ?, ?, ?, NULL, NULL, 'complete', 'turn', ?, ?, ?, ?, ?, - (SELECT COALESCE(MAX(list_sequence), 0) + 1 FROM turns WHERE host_id = ?) - ) - """, - ( - _HOST_ID, - turn_id, - f"worker-{turn_id}", - _CREATED_AT, - f"fingerprint-{turn_id}", - f"snapshot-{turn_id}", - _CREATED_AT, - json.dumps( - { - "source_turn_id": f"source-{turn_id}", - "complete": True, - "meta": { - "stable_key": _STABLE_KEY, - "stable_key_version": 1, - }, - "chat_id": _PRIVATE_ROUTE_MARKER, - }, - sort_keys=True, - ), - _HOST_ID, - ), - ) - conn.execute( - """ - INSERT INTO turn_content_revisions ( - host_id, - turn_id, - content_revision, - user_text, - assistant_final_text, - user_state, - final_state, - user_char_length, - user_byte_length, - final_char_length, - final_byte_length, - user_page_count, - final_page_count, - is_current, - created_at, - superseded_at - ) VALUES (?, ?, ?, ?, ?, 'complete', 'complete', ?, ?, ?, ?, ?, ?, 1, ?, NULL) - """, - ( - _HOST_ID, - turn_id, - revision, - user_text, - final_text, - len(user_text), - len(user_text.encode("utf-8")), - len(final_text), - len(final_text.encode("utf-8")), - len(user_segments), - len(final_segments), - _CREATED_AT, - ), - ) - for field, segments in ( - ("user_text", user_segments), - ("assistant_final_text", final_segments), - ): - conn.executemany( - """ - INSERT INTO turn_content_page_boundaries ( - host_id, - turn_id, - content_revision, - field, - page_index, - start_char, - start_byte - ) VALUES (?, ?, ?, ?, ?, ?, ?) - """, - ( - ( - _HOST_ID, - turn_id, - revision, - field, - int(segment.index), - int(segment.start_char), - int(segment.start_byte), - ) - for segment in segments - ), - ) - return revision - - -def _seed_v10_finals(db_path: Path) -> dict[str, tuple[str, str]]: - finals: dict[str, tuple[str, str]] = {} - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys = ON") - for ordinal, label in enumerate(("delivered", "hold-a", "hold-b"), start=1): - turn_id = f"turn-{ordinal:02d}-{label}" - revision = _insert_current_final( - conn, - turn_id=turn_id, - user_text=f"{_RAW_USER_MARKER}-{label}", - final_text=f"{_RAW_FINAL_MARKER}-{label}", - ) - finals[label] = (turn_id, revision) - - delivered_turn, delivered_revision = finals["delivered"] - part_payload = { - "schema_version": 1, - "operation": "upsert", - "sequence_index": 0, - "spans": [ - { - "field": "user_text", - "start_char": 0, - "end_char": len(f"{_RAW_USER_MARKER}-delivered"), - }, - { - "field": "assistant_final_text", - "start_char": 0, - "end_char": len(f"{_RAW_FINAL_MARKER}-delivered"), - }, - ], - } - part_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, - connector, - delivery_key, - status, - payload_json, - private_state_json, - created_at, - updated_at, - next_attempt_at - ) VALUES (?, 'turn-final', ?, 'delivered', ?, ?, ?, ?, NULL) - """, - ( - _HOST_ID, - "turn-final:legacy-completed:000000", - json.dumps(part_payload, sort_keys=True), - json.dumps({"route": _PRIVATE_ROUTE_MARKER}, sort_keys=True), - _CREATED_AT, - _CREATED_AT, - ), - ) - part_outbox_id = int(part_cursor.lastrowid) - plan_cursor = conn.execute( - """ - INSERT INTO turn_presentation_plans ( - host_id, - name, - plan_token, - turn_id, - content_revision, - presentation_version, - generation, - part_count, - state, - replaces_plan_token, - recovers_plan_token, - created_at, - activated_at, - completed_at - ) VALUES ( - ?, 'turn-final', 'twplan1.legacy-completed', ?, ?, - 'turn-present-v10', 1, 1, 'completed', NULL, NULL, ?, ?, ? - ) - """, - ( - _HOST_ID, - delivered_turn, - delivered_revision, - _CREATED_AT, - _CREATED_AT, - _CREATED_AT, - ), - ) - plan_id = int(plan_cursor.lastrowid) - conn.execute( - """ - INSERT INTO turn_presentation_jobs ( - plan_id, - sequence_index, - operation, - part_ordinal, - spans_json, - outbox_id, - created_at - ) VALUES (?, 0, 'upsert', 0, ?, ?, ?) - """, - ( - plan_id, - json.dumps(part_payload["spans"], sort_keys=True), - part_outbox_id, - _CREATED_AT, - ), - ) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, - host_id, - connector, - delivery_key, - attempt, - status, - response_json, - private_state_json, - created_at, - delivered_at - ) VALUES (?, ?, 'turn-final', ?, 1, 'delivered', '{}', '{}', ?, ?) - """, - ( - part_outbox_id, - _HOST_ID, - "turn-final:legacy-completed:000000", - _CREATED_AT, - _CREATED_AT, - ), - ) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - return finals - - -def _seed_literal_v10_continuity_final( - db_path: Path, - *, - delivered_proof: bool, -) -> None: - """Insert one frozen pre-owner-continuity graph without constructing a Turn.""" - - attempt_status = "delivered" if delivered_proof else "failed" - delivered_at = _CREATED_AT if delivered_proof else None - turn_payload = { - "schema_version": 1, - "id": _HISTORICAL_TURN_ID, - "host_id": _HOST_ID, - "worker_id": "worker-a", - "worker_fingerprint": "worker-a-fingerprint", - "space_id": "space-a", - "status": "done", - "kind": "task", - "title": "Historical migration worker", - "summary": "Historical migration continuity", - "source": "snapshot", - "updated_at": _CREATED_AT, - "complete": True, - "has_open_turn": False, - "source_turn_id": _HISTORICAL_SOURCE_TOKEN, - "fingerprint": "historical-turn-fingerprint", - "meta": { - "stable_key": _STABLE_KEY, - "stable_key_version": 1, - }, - } - part_payload = { - "schema_version": 1, - "operation": "upsert", - "sequence_index": 0, - "spans": [ - { - "field": "user_text", - "start_char": 0, - "end_char": len(_HISTORICAL_USER_TEXT), - }, - { - "field": "assistant_final_text", - "start_char": 0, - "end_char": len(_HISTORICAL_FINAL_TEXT), - }, - ], - } - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys = ON") - conn.execute( - """ - INSERT INTO turn_list_hosts ( - host_id, next_sequence, traversal_generation - ) VALUES (?, ?, 1) - """, - (_HOST_ID, _HISTORICAL_LIST_SEQUENCE + 1), - ) - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, worker_fingerprint, space_id, - status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, - list_sequence - ) VALUES (?, ?, 'worker-a', 'worker-a-fingerprint', 'space-a', - 'done', 'task', ?, 'historical-turn-fingerprint', - 'historical-snapshot-fingerprint', ?, ?, ?) - """, - ( - _HOST_ID, - _HISTORICAL_TURN_ID, - _CREATED_AT, - _CREATED_AT, - json.dumps(turn_payload, sort_keys=True), - _HISTORICAL_LIST_SEQUENCE, - ), - ) - conn.execute( - """ - INSERT INTO turn_content_revisions ( - host_id, turn_id, content_revision, user_text, - assistant_final_text, user_state, final_state, - user_char_length, user_byte_length, final_char_length, - final_byte_length, user_page_count, final_page_count, - is_current, created_at, superseded_at - ) VALUES (?, ?, ?, ?, ?, 'complete', 'complete', ?, ?, ?, ?, - 1, 1, 1, ?, NULL) - """, - ( - _HOST_ID, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - _HISTORICAL_USER_TEXT, - _HISTORICAL_FINAL_TEXT, - len(_HISTORICAL_USER_TEXT), - len(_HISTORICAL_USER_TEXT.encode("utf-8")), - len(_HISTORICAL_FINAL_TEXT), - len(_HISTORICAL_FINAL_TEXT.encode("utf-8")), - _CREATED_AT, - ), - ) - conn.executemany( - """ - INSERT INTO turn_content_page_boundaries ( - host_id, turn_id, content_revision, field, page_index, - start_char, start_byte - ) VALUES (?, ?, ?, ?, 0, 0, 0) - """, - ( - ( - _HOST_ID, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - "user_text", - ), - ( - _HOST_ID, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - "assistant_final_text", - ), - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - id, host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES (?, ?, 'turn-final', - 'turn-final:legacy-continuity:000000', 'delivered', - ?, '{"historical_part":true}', ?, ?, NULL) - """, - ( - _HISTORICAL_PART_OUTBOX_ID, - _HOST_ID, - json.dumps(part_payload, sort_keys=True), - _CREATED_AT, - _CREATED_AT, - ), - ) - conn.execute( - """ - INSERT INTO turn_presentation_plans ( - id, host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - replaces_plan_token, recovers_plan_token, created_at, - activated_at, completed_at - ) VALUES (?, ?, 'turn-final', 'twplan1.literal-continuity', - ?, ?, 'turn-present-literal-v10', 1, 1, 'completed', - NULL, NULL, ?, ?, ?) - """, - ( - _HISTORICAL_PLAN_ID, - _HOST_ID, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - _CREATED_AT, - _CREATED_AT, - _CREATED_AT, - ), - ) - conn.execute( - """ - INSERT INTO turn_presentation_jobs ( - id, plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - ) VALUES (?, ?, 0, 'upsert', 0, ?, ?, ?) - """, - ( - _HISTORICAL_JOB_ID, - _HISTORICAL_PLAN_ID, - json.dumps(part_payload["spans"], sort_keys=True), - _HISTORICAL_PART_OUTBOX_ID, - _CREATED_AT, - ), - ) - conn.execute( - """ - INSERT INTO connector_deliveries ( - id, outbox_id, host_id, connector, delivery_key, attempt, - status, response_json, private_state_json, created_at, - delivered_at - ) VALUES (?, ?, ?, 'turn-final', - 'turn-final:legacy-continuity:000000', 3, ?, - '{"literal_response":true}', - '{"literal_attempt_state":true}', ?, ?) - """, - ( - _HISTORICAL_DELIVERY_ID, - _HISTORICAL_PART_OUTBOX_ID, - _HOST_ID, - attempt_status, - _CREATED_AT, - delivered_at, - ), - ) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - -def _continuity_snapshot( - db_path: Path, - *, - worker_id: str, - space_id: str, - stable_key: str, - second: int, -): - return project_from_raw( - Config(host_id=_HOST_ID, db_path=db_path), - workers=[ - { - "id": worker_id, - "name": f"Continuity worker {worker_id}", - "status": "done", - "space_id": space_id, - "meta": { - "stable_key": stable_key, - "stable_key_version": 1, - }, - } - ], - timestamp=datetime(2026, 1, 1, 0, 0, second, tzinfo=timezone.utc), - ) - - -def _observe_literal_continuity_final( - db_path: Path, - *, - worker_id: str, - space_id: str, - stable_key: str, - second: int, -) -> int: - snapshot = _continuity_snapshot( - db_path, - worker_id=worker_id, - space_id=space_id, - stable_key=stable_key, - second=second, - ) - assert save_snapshot(db_path, snapshot) is True - return merge_turn_content( - db_path, - _HOST_ID, - worker_id, - { - "source_turn_id": _HISTORICAL_RAW_SOURCE, - "user_text": _HISTORICAL_USER_TEXT, - "assistant_final_text": _HISTORICAL_FINAL_TEXT, - "complete": True, - "has_open_turn": False, - }, - observed_at=f"2026-01-01T00:00:{second + 1:02d}+00:00", - ) - - -def _historical_graph(db_path: Path) -> dict[str, list[tuple[Any, ...]]]: - with sqlite3.connect(str(db_path)) as conn: - turn_row = conn.execute( - """ - SELECT turn_id, list_sequence, payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchone() - assert turn_row is not None - payload = json.loads(str(turn_row[2])) - return { - "turn_identity": [ - ( - str(turn_row[0]), - int(turn_row[1]), - str(payload.get("id")), - str(payload.get("source_turn_id")), - json.dumps(payload.get("meta"), sort_keys=True), - ) - ], - "revisions": conn.execute( - """ - SELECT turn_id, content_revision, user_text, - assistant_final_text, user_state, final_state, - is_current, created_at, superseded_at - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? - ORDER BY content_revision - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "boundaries": conn.execute( - """ - SELECT turn_id, content_revision, field, page_index, - start_char, start_byte - FROM turn_content_page_boundaries - WHERE host_id = ? AND turn_id = ? - ORDER BY field, page_index - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "plans": conn.execute( - """ - SELECT id, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - replaces_plan_token, recovers_plan_token, - source_outbox_id, created_at, activated_at, completed_at - FROM turn_presentation_plans - WHERE host_id = ? AND turn_id = ? - ORDER BY id - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "jobs": conn.execute( - """ - SELECT jobs.id, jobs.plan_id, jobs.sequence_index, - jobs.operation, jobs.part_ordinal, jobs.spans_json, - jobs.outbox_id, jobs.created_at - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans ON plans.id = jobs.plan_id - WHERE plans.host_id = ? AND plans.turn_id = ? - ORDER BY jobs.id - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "recoveries": conn.execute( - """ - SELECT recoveries.* - FROM turn_presentation_recoveries AS recoveries - JOIN turn_presentation_plans AS plans - ON plans.id = recoveries.failed_plan_id - WHERE plans.host_id = ? AND plans.turn_id = ? - ORDER BY recoveries.id - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "outbox": conn.execute( - """ - SELECT id, delivery_key, delivery_kind, turn_id, - content_revision, status, payload_json, - private_state_json, created_at, updated_at, - next_attempt_at - FROM connector_outbox - WHERE id IN (?, ?) - ORDER BY id - """, - (_HISTORICAL_PART_OUTBOX_ID, _HISTORICAL_ROOT_OUTBOX_ID), - ).fetchall(), - "attempts": conn.execute( - """ - SELECT id, outbox_id, delivery_key, attempt, status, - response_json, private_state_json, created_at, - delivered_at - FROM connector_deliveries - WHERE id = ? - """, - (_HISTORICAL_DELIVERY_ID,), - ).fetchall(), - } - - -def _historical_root(db_path: Path) -> tuple[Any, ...]: - with sqlite3.connect(str(db_path)) as conn: - row = conn.execute( - """ - SELECT id, delivery_key, delivery_kind, turn_id, - content_revision, status, private_state_json, - next_attempt_at - FROM connector_outbox - WHERE id = ? - """, - (_HISTORICAL_ROOT_OUTBOX_ID,), - ).fetchone() - assert row is not None - return tuple(row) - - -def _assert_historical_route( - db_path: Path, - *, - worker_id: str, - worker_fingerprint: str, - space_id: str, -) -> None: - with sqlite3.connect(str(db_path)) as conn: - row = conn.execute( - """ - SELECT worker_id, worker_fingerprint, space_id, payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchone() - assert row is not None - payload = json.loads(str(row[3])) - assert tuple(row[:3]) == (worker_id, worker_fingerprint, space_id) - assert payload["id"] == _HISTORICAL_TURN_ID - assert payload["source_turn_id"] == _HISTORICAL_SOURCE_TOKEN - assert payload["worker_id"] == worker_id - assert payload["worker_fingerprint"] == worker_fingerprint - assert payload["space_id"] == space_id - - -def _final_key(turn_id: str, revision: str) -> str: - identity = store_sqlite.turn_final_delivery_identity( - _HOST_ID, - turn_id, - revision, - ) - return f"turn-final:revision:{identity}" - - -def _assert_no_private_or_raw(value: Any) -> None: - encoded = json.dumps(value, sort_keys=True).lower() - for forbidden in ( - _RAW_USER_MARKER, - _RAW_FINAL_MARKER, - _PRIVATE_ROUTE_MARKER, - "chat_id", - "topic_id", - "bot_token", - "telegram", - "herdres", - "private_state_json", - ): - assert forbidden.lower() not in encoded - - -def _anchor_rows(db_path: Path) -> list[tuple[Any, ...]]: - with sqlite3.connect(str(db_path)) as conn: - return conn.execute( - """ - SELECT - id, - delivery_key, - delivery_kind, - turn_id, - content_revision, - status, - payload_json, - private_state_json, - created_at, - updated_at, - next_attempt_at - FROM connector_outbox - WHERE connector = 'turn-final' - AND delivery_kind IN ('final_ready', 'final_migration_hold') - ORDER BY id - """ - ).fetchall() - - -def _database_dump(db_path: Path) -> tuple[int, str]: - with sqlite3.connect(str(db_path)) as conn: - version = int(conn.execute("PRAGMA user_version").fetchone()[0]) - return version, "\n".join(conn.iterdump()) - - -def test_v10_to_v12_migration_retains_finals_without_reposting_or_leaking( - tmp_path: Path, -) -> None: - db_path = tmp_path / "retention-v10.db" - _create_v10_store(db_path) - finals = _seed_v10_finals(db_path) - - init_store(db_path) - assert store_sqlite.STORE_SCHEMA_VERSION == 28 - - delivered_key = _final_key(*finals["delivered"]) - hold_keys = {_final_key(*finals[label]) for label in ("hold-a", "hold-b")} - rows_after_first_migration = _anchor_rows(db_path) - by_key = {str(row[1]): row for row in rows_after_first_migration} - assert set(by_key) == {delivered_key, *hold_keys} - assert by_key[delivered_key][2:6] == ( - "final_ready", - finals["delivered"][0], - finals["delivered"][1], - "delivered", - ) - for label in ("hold-a", "hold-b"): - key = _final_key(*finals[label]) - assert by_key[key][2:6] == ( - "final_migration_hold", - finals[label][0], - finals[label][1], - "dead_letter", - ) - for row in rows_after_first_migration: - assert row[1].startswith("turn-final:revision:twfinal1.") - root_payload = json.loads(str(row[6])) - assert root_payload["schema_version"] == 2 - assert root_payload["stable_key"] == _STABLE_KEY - assert root_payload["stable_key_version"] == 1 - assert "worker_fingerprint" not in root_payload - _assert_no_private_or_raw(root_payload) - - with sqlite3.connect(str(db_path)) as conn: - delivered_source_id = int(by_key[delivered_key][0]) - assert conn.execute( - """ - SELECT state, source_outbox_id - FROM turn_presentation_plans - WHERE plan_token = 'twplan1.legacy-completed' - """ - ).fetchone() == ("completed", delivered_source_id) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - init_store(db_path) - assert _anchor_rows(db_path) == rows_after_first_migration - - api = ConnectorOutboxAPI(db_path, _HOST_ID) - assert api.poll({"name": "turn-final", "limit": 100})["items"] == [] - inspected = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert inspected["ok"] is True - assert inspected["total"] == 2 - assert {item["key"] for item in inspected["items"]} == hold_keys - _assert_no_private_or_raw(inspected) - - with sqlite3.connect(str(db_path)) as conn: - new_turn = "turn-04-new-ready" - new_revision = _insert_current_final( - conn, - turn_id=new_turn, - user_text=f"{_RAW_USER_MARKER}-new", - final_text=f"{_RAW_FINAL_MARKER}-new", - ) - new_anchor_id = store_sqlite._ensure_final_ready_anchor_conn( - conn, - host_id=_HOST_ID, - turn_id=new_turn, - content_revision_value=new_revision, - now="2026-01-01T00:01:00+00:00", - ) - assert new_anchor_id is not None - new_key = _final_key(new_turn, new_revision) - - new_work = api.poll({"name": "turn-final", "limit": 100}) - assert [item["key"] for item in new_work["items"]] == [new_key] - _assert_no_private_or_raw(new_work) - assert delivered_key not in {item["key"] for item in new_work["items"]} - - selected_key = _final_key(*finals["hold-a"]) - untouched_key = _final_key(*finals["hold-b"]) - retried = api.retry( - { - "schema_version": 1, - "name": "turn-final", - "key": selected_key, - } - ) - assert retried["ok"] is True - assert retried["status"] == "requeued" - assert retried["key"] == selected_key - _assert_no_private_or_raw(retried) - - with sqlite3.connect(str(db_path)) as conn: - retry_states = dict( - conn.execute( - """ - SELECT delivery_key, delivery_kind || ':' || status - FROM connector_outbox - WHERE delivery_key IN (?, ?) - """, - (selected_key, untouched_key), - ).fetchall() - ) - assert retry_states == { - selected_key: "final_ready:queued", - untouched_key: "final_migration_hold:dead_letter", - } - - after_retry_inspect = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert after_retry_inspect["total"] == 1 - assert [item["key"] for item in after_retry_inspect["items"]] == [untouched_key] - _assert_no_private_or_raw(after_retry_inspect) - - released = api.poll({"name": "turn-final", "limit": 100}) - assert [item["key"] for item in released["items"]] == [selected_key] - _assert_no_private_or_raw(released) - assert delivered_key not in {item["key"] for item in released["items"]} - - -@pytest.mark.parametrize( - "invalid_meta", - [ - {}, - {"stable_key": _STABLE_KEY}, - {"stable_key": "wsk1_invalid", "stable_key_version": 1}, - {"stable_key": _STABLE_KEY, "stable_key_version": True}, - {"stable_key": _STABLE_KEY, "stable_key_version": 2}, - ], -) -def test_v10_to_v12_missing_stable_key_pair_becomes_nonroutable_hold( - tmp_path: Path, - invalid_meta: dict[str, object], -) -> None: - db_path = tmp_path / "retention-v10-stable-key-hold.db" - _create_v10_store(db_path) - finals = _seed_v10_finals(db_path) - turn_id, revision = finals["delivered"] - with sqlite3.connect(str(db_path)) as conn: - raw_payload = conn.execute( - """ - SELECT payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (_HOST_ID, turn_id), - ).fetchone()[0] - turn_payload = json.loads(str(raw_payload)) - turn_payload["meta"] = invalid_meta - conn.execute( - """ - UPDATE turns - SET payload_json = ? - WHERE host_id = ? AND turn_id = ? - """, - (json.dumps(turn_payload, sort_keys=True), _HOST_ID, turn_id), - ) - - init_store(db_path) - key = _final_key(turn_id, revision) - row = { - str(anchor[1]): anchor - for anchor in _anchor_rows(db_path) - }[key] - payload = json.loads(str(row[6])) - - assert row[2:6] == ( - "final_migration_hold", - turn_id, - revision, - "dead_letter", - ) - assert payload["schema_version"] == 1 - assert "stable_key" not in payload - assert "stable_key_version" not in payload - assert "worker_fingerprint" not in payload - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT source_outbox_id - FROM turn_presentation_plans - WHERE plan_token = 'twplan1.legacy-completed' - """ - ).fetchone() == (None,) - api = ConnectorOutboxAPI(db_path, _HOST_ID) - assert api.poll({"name": "turn-final", "limit": 100})["items"] == [] - inspected = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert key in {item.get("key") for item in inspected["items"]} - _assert_no_private_or_raw(inspected) - - -@pytest.mark.parametrize( - "proof_gap", - [ - "declared_part_missing", - "delivered_attempt_missing", - "delivered_attempt_contradicted", - "ack_time_missing", - "foreign_host_part", - ], -) -def test_v10_to_v12_migration_requires_complete_host_bound_ack_proof( - tmp_path: Path, - proof_gap: str, -) -> None: - db_path = tmp_path / f"retention-v10-{proof_gap}.db" - _create_v10_store(db_path) - finals = _seed_v10_finals(db_path) - with sqlite3.connect(str(db_path)) as conn: - if proof_gap == "declared_part_missing": - conn.execute( - """ - UPDATE turn_presentation_plans - SET part_count = 2 - WHERE plan_token = 'twplan1.legacy-completed' - """ - ) - elif proof_gap == "delivered_attempt_missing": - conn.execute( - """ - DELETE FROM connector_deliveries - WHERE delivery_key = 'turn-final:legacy-completed:000000' - """ - ) - elif proof_gap == "delivered_attempt_contradicted": - conn.execute( - """ - UPDATE connector_deliveries - SET status = 'failed', delivered_at = NULL - WHERE delivery_key = 'turn-final:legacy-completed:000000' - """ - ) - elif proof_gap == "ack_time_missing": - conn.execute( - """ - UPDATE turn_presentation_plans - SET completed_at = NULL - WHERE plan_token = 'twplan1.legacy-completed' - """ - ) - else: - conn.execute( - """ - UPDATE connector_outbox - SET host_id = 'foreign-host' - WHERE delivery_key = 'turn-final:legacy-completed:000000' - """ - ) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - init_store(db_path) - delivered_key = _final_key(*finals["delivered"]) - migrated = { - str(row[1]): row - for row in _anchor_rows(db_path) - }[delivered_key] - assert migrated[2:6] == ( - "final_migration_hold", - finals["delivered"][0], - finals["delivered"][1], - "dead_letter", - ) - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT source_outbox_id - FROM turn_presentation_plans - WHERE plan_token = 'twplan1.legacy-completed' - """ - ).fetchone() == (None,) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - api = ConnectorOutboxAPI(db_path, _HOST_ID) - assert api.poll({"name": "turn-final", "limit": 100})["items"] == [] - inspected = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert delivered_key in {item.get("key") for item in inspected["items"]} - _assert_no_private_or_raw(inspected) - -@pytest.mark.parametrize("owner_matches", [True, False]) -def test_v10_failed_plan_links_only_with_exact_immutable_job_route( - tmp_path: Path, - owner_matches: bool, -) -> None: - db_path = tmp_path / f"retention-v10-owner-{owner_matches}.db" - _create_v10_store(db_path) - turn_id = "turn-legacy-owner" - with sqlite3.connect(str(db_path)) as conn: - revision = _insert_current_final( - conn, - turn_id=turn_id, - user_text="legacy owner prompt", - final_text="legacy owner final", - ) - final_identity = store_sqlite.turn_final_delivery_identity( - _HOST_ID, - turn_id, - revision, - ) - route = { - "schema_version": 2, - "turn_id": turn_id, - "content_revision": revision, - "final_identity": final_identity, - "stable_key": _STABLE_KEY if owner_matches else "wsk1_" + ("b" * 64), - "stable_key_version": 1, - } - outbox_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES (?, 'turn-final', 'turn-final:legacy-owner:000000', - 'dead_letter', ?, '{}', ?, ?, NULL) - """, - ( - _HOST_ID, - json.dumps({"turn": route}, sort_keys=True), - _CREATED_AT, - _CREATED_AT, - ), - ) - plan_cursor = conn.execute( - """ - INSERT INTO turn_presentation_plans ( - host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - created_at, activated_at - ) VALUES (?, 'turn-final', 'twplan1.legacy-owner', ?, ?, - 'legacy-owner-v1', 1, 1, 'failed', ?, ?) - """, - (_HOST_ID, turn_id, revision, _CREATED_AT, _CREATED_AT), - ) - conn.execute( - """ - INSERT INTO turn_presentation_jobs ( - plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - ) VALUES (?, 0, 'upsert', 0, '[]', ?, ?) - """, - (int(plan_cursor.lastrowid), int(outbox_cursor.lastrowid), _CREATED_AT), - ) - - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - root = conn.execute( - """ - SELECT id, delivery_kind, status - FROM connector_outbox - WHERE delivery_kind IN ('final_ready', 'final_migration_hold') - """ - ).fetchone() - source_outbox_id = conn.execute( - """ - SELECT source_outbox_id - FROM turn_presentation_plans - WHERE plan_token = 'twplan1.legacy-owner' - """ - ).fetchone()[0] - - assert root is not None - if owner_matches: - assert root[1:3] == ("final_ready", "awaiting_ack") - assert source_outbox_id == root[0] - else: - assert root[1:3] == ("final_migration_hold", "dead_letter") - assert source_outbox_id is None - - - -def test_v10_to_v12_migration_failure_rolls_back_the_entire_transition( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "retention-v10-rollback.db" - _create_v10_store(db_path) - _seed_v10_finals(db_path) - before = _database_dump(db_path) - - real_payload = store_sqlite._final_ready_payload_conn - calls = 0 - - def fail_after_first_anchor(*args: Any, **kwargs: Any) -> dict[str, Any] | None: - nonlocal calls - calls += 1 - if calls == 2: - raise RuntimeError("controlled v11 migration failure") - return real_payload(*args, **kwargs) - - monkeypatch.setattr( - store_sqlite, - "_final_ready_payload_conn", - fail_after_first_anchor, - ) - - with pytest.raises(RuntimeError, match="controlled v11 migration failure"): - init_store(db_path) - - assert calls == 2 - assert _database_dump(db_path) == before - with sqlite3.connect(str(db_path)) as conn: - _assert_v10_shape(conn) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - assert conn.execute( - "SELECT COUNT(*) FROM connector_outbox WHERE connector = 'turn-final'" - ).fetchone()[0] == 1 - - -@pytest.mark.parametrize( - ( - "root_state", - "delivered_proof", - "expected_kind", - "expected_status", - "expected_attempt_status", - "expected_source_link", - ), - [ - ("queued", False, "final_ready", "queued", "failed", None), - ( - "delivered", - True, - "final_ready", - "delivered", - "delivered", - _HISTORICAL_ROOT_OUTBOX_ID, - ), - ( - "migration-hold", - False, - "final_migration_hold", - "dead_letter", - "failed", - None, - ), - ], -) -def test_v10_to_v12_history_stays_immutable_when_observed_turn_arrives( - tmp_path: Path, - root_state: str, - delivered_proof: bool, - expected_kind: str, - expected_status: str, - expected_attempt_status: str, - expected_source_link: int | None, -) -> None: - db_path = tmp_path / f"literal-v10-continuity-{root_state}.db" - _create_v10_store(db_path) - _seed_literal_v10_continuity_final( - db_path, - delivered_proof=delivered_proof, - ) - - init_store(db_path) - assert _historical_root(db_path)[:6] == ( - _HISTORICAL_ROOT_OUTBOX_ID, - _HISTORICAL_FINAL_KEY, - ( - "final_ready" - if delivered_proof - else "final_migration_hold" - ), - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - "delivered" if delivered_proof else "dead_letter", - ) - with sqlite3.connect(str(db_path)) as conn: - root_payload = json.loads( - str( - conn.execute( - "SELECT payload_json FROM connector_outbox WHERE id = ?", - (_HISTORICAL_ROOT_OUTBOX_ID,), - ).fetchone()[0] - ) - ) - assert root_payload["final_identity"] == _HISTORICAL_FINAL_IDENTITY - assert root_payload["turn_id"] == _HISTORICAL_TURN_ID - assert root_payload["content_revision"] == _HISTORICAL_REVISION - assert root_payload["stable_key"] == _STABLE_KEY - assert root_payload["stable_key_version"] == 1 - assert conn.execute( - """ - SELECT source_outbox_id - FROM turn_presentation_plans - WHERE id = ? - """, - (_HISTORICAL_PLAN_ID,), - ).fetchone() == (expected_source_link,) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - api = ConnectorOutboxAPI(db_path, _HOST_ID) - if root_state == "queued": - retried = api.retry( - { - "schema_version": 1, - "name": "turn-final", - "key": _HISTORICAL_FINAL_KEY, - } - ) - assert retried["ok"] is True - assert retried["status"] == "requeued" - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = ?, - next_attempt_at = ? - WHERE id = ? - """, - ( - json.dumps( - {"literal_root_state": root_state}, - sort_keys=True, - separators=(",", ":"), - ), - _CREATED_AT if root_state == "queued" else None, - _HISTORICAL_ROOT_OUTBOX_ID, - ), - ) - - expected_root = _historical_root(db_path) - assert expected_root[2:6] == ( - expected_kind, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - expected_status, - ) - assert expected_root[7] == ( - _CREATED_AT if root_state == "queued" else None - ) - expected_graph = _historical_graph(db_path) - assert expected_graph["turn_identity"] == [ - ( - _HISTORICAL_TURN_ID, - _HISTORICAL_LIST_SEQUENCE, - _HISTORICAL_TURN_ID, - _HISTORICAL_SOURCE_TOKEN, - json.dumps( - { - "stable_key": _STABLE_KEY, - "stable_key_version": 1, - }, - sort_keys=True, - ), - ) - ] - assert expected_graph["revisions"][0][1] == _HISTORICAL_REVISION - assert expected_graph["plans"][0][0] == _HISTORICAL_PLAN_ID - assert expected_graph["plans"][0][1] == "twplan1.literal-continuity" - assert expected_graph["plans"][0][10] == expected_source_link - assert expected_graph["jobs"][0][0] == _HISTORICAL_JOB_ID - assert expected_graph["jobs"][0][6] == _HISTORICAL_PART_OUTBOX_ID - assert expected_graph["attempts"][0][:5] == ( - _HISTORICAL_DELIVERY_ID, - _HISTORICAL_PART_OUTBOX_ID, - "turn-final:legacy-continuity:000000", - 3, - expected_attempt_status, - ) - assert expected_graph["recoveries"] == [] - - worker_b = _continuity_snapshot( - db_path, - worker_id="worker-b", - space_id="space-b", - stable_key=_STABLE_KEY, - second=2, - ) - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-b", - space_id="space-b", - stable_key=_STABLE_KEY, - second=2, - ) == 1 - _assert_historical_route( - db_path, - worker_id="worker-a", - worker_fingerprint="worker-a-fingerprint", - space_id="space-a", - ) - assert _historical_root(db_path) == expected_root - assert _historical_graph(db_path) == expected_graph - with sqlite3.connect(str(db_path)) as conn: - observed = conn.execute( - """ - SELECT turn_id, worker_id, worker_fingerprint, space_id, - json_extract(payload_json, '$.source_turn_id') - FROM turns - WHERE host_id = ? AND turn_id != ? - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchone() - assert observed is not None - observed_turn_id = str(observed[0]) - assert tuple(observed[1:4]) == ( - "worker-b", - worker_b.workers[0].fingerprint, - "space-b", - ) - assert str(observed[4]) != _HISTORICAL_SOURCE_TOKEN - observed_root = conn.execute( - """ - SELECT delivery_key, delivery_kind, status - FROM connector_outbox - WHERE turn_id = ? - """, - (observed_turn_id,), - ).fetchone() - assert observed_root is not None - observed_key = str(observed_root[0]) - assert tuple(observed_root[1:]) == ("final_ready", "queued") - - init_store(db_path) - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-b", - space_id="space-b", - stable_key=_STABLE_KEY, - second=4, - ) == 0 - assert _historical_root(db_path) == expected_root - assert _historical_graph(db_path) == expected_graph - - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT COUNT(*) - FROM connector_outbox - WHERE delivery_kind IN ('final_ready', 'final_migration_hold') - AND turn_id = ? - AND content_revision = ? - """, - (_HISTORICAL_TURN_ID, _HISTORICAL_REVISION), - ).fetchone() == (1,) - assert conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchone() == (1,) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - polled = api.poll({"name": "turn-final", "limit": 100}) - assert [item["key"] for item in polled["items"]] == ( - [_HISTORICAL_FINAL_KEY] - if root_state == "queued" - else [observed_key] - ) - if root_state == "migration-hold": - inspected = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert [item["key"] for item in inspected["items"]] == [ - _HISTORICAL_FINAL_KEY - ] - - -def test_v10_migration_hold_stays_immutable_under_observed_k2( - tmp_path: Path, -) -> None: - db_path = tmp_path / "literal-v10-continuity-k2.db" - _create_v10_store(db_path) - _seed_literal_v10_continuity_final(db_path, delivered_proof=False) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = '{"literal_root_state":"migration-hold"}' - WHERE id = ? - """, - (_HISTORICAL_ROOT_OUTBOX_ID,), - ) - - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-b", - space_id="space-b", - stable_key=_STABLE_KEY, - second=2, - ) == 1 - historical_graph = _historical_graph(db_path) - historical_root = _historical_root(db_path) - assert historical_root[2:6] == ( - "final_migration_hold", - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - "dead_letter", - ) - - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-c", - space_id="space-c", - stable_key=_STABLE_KEY_2, - second=4, - ) == 1 - assert _historical_graph(db_path) == historical_graph - assert _historical_root(db_path) == historical_root - - with sqlite3.connect(str(db_path)) as conn: - roots = [ - ( - int(root_id), - str(delivery_key), - str(delivery_kind), - str(turn_id), - str(status), - json.loads(str(payload_json)), - ) - for ( - root_id, - delivery_key, - delivery_kind, - turn_id, - status, - payload_json, - ) in conn.execute( - """ - SELECT id, delivery_key, delivery_kind, turn_id, status, - payload_json - FROM connector_outbox - WHERE delivery_kind IN ('final_ready', 'final_migration_hold') - ORDER BY id - """ - ).fetchall() - ] - source_turns = [ - (str(turn_id), json.loads(str(payload_json))) - for turn_id, payload_json in conn.execute( - """ - SELECT turn_id, payload_json - FROM turns - WHERE host_id = ? - ORDER BY turn_id - """, - (_HOST_ID,), - ).fetchall() - if json.loads(str(payload_json)).get("source_turn_id") - ] - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - assert len(roots) == 3 - historical = next(root for root in roots if root[0] == _HISTORICAL_ROOT_OUTBOX_ID) - assert historical[:5] == ( - _HISTORICAL_ROOT_OUTBOX_ID, - _HISTORICAL_FINAL_KEY, - "final_migration_hold", - _HISTORICAL_TURN_ID, - "dead_letter", - ) - observed_roots = [root for root in roots if root[0] != _HISTORICAL_ROOT_OUTBOX_ID] - by_owner = {root[5]["stable_key"]: root for root in observed_roots} - assert set(by_owner) == {_STABLE_KEY, _STABLE_KEY_2} - for owner_key in (_STABLE_KEY, _STABLE_KEY_2): - assert by_owner[owner_key][2] == "final_ready" - assert by_owner[owner_key][3] != _HISTORICAL_TURN_ID - assert by_owner[owner_key][4] == "queued" - assert len(source_turns) == 3 - historical_source = next( - row for row in source_turns if row[0] == _HISTORICAL_TURN_ID - ) - assert historical_source[1]["source_turn_id"] == _HISTORICAL_SOURCE_TOKEN - observed_source_by_owner = { - turn_payload["meta"]["stable_key"]: (turn_id, turn_payload) - for turn_id, turn_payload in source_turns - if turn_id != _HISTORICAL_TURN_ID - } - assert set(observed_source_by_owner) == {_STABLE_KEY, _STABLE_KEY_2} - assert all( - turn_payload["source_turn_id"] != _HISTORICAL_SOURCE_TOKEN - for _turn_id, turn_payload in observed_source_by_owner.values() - ) - same_owner_key = str(by_owner[_STABLE_KEY][1]) - new_key = str(by_owner[_STABLE_KEY_2][1]) - same_owner_turn_id = str(observed_source_by_owner[_STABLE_KEY][0]) - new_turn_id = str(observed_source_by_owner[_STABLE_KEY_2][0]) - source_identities = sorted( - ( - turn_id, - str(turn_payload["source_turn_id"]), - str(turn_payload["meta"]["stable_key"]), - ) - for turn_id, turn_payload in source_turns - ) - expected_source_revisions = { - (_HISTORICAL_TURN_ID, _HISTORICAL_REVISION), - ( - same_owner_turn_id, - str(by_owner[_STABLE_KEY][5]["content_revision"]), - ), - ( - new_turn_id, - str(by_owner[_STABLE_KEY_2][5]["content_revision"]), - ), - } - - init_store(db_path) - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-c", - space_id="space-c", - stable_key=_STABLE_KEY_2, - second=6, - ) in {0, 1} - assert _historical_graph(db_path) == historical_graph - assert _historical_root(db_path) == historical_root - with sqlite3.connect(str(db_path)) as conn: - roots_after_restart = [ - ( - int(root_id), - str(delivery_key), - str(delivery_kind), - str(turn_id), - str(status), - json.loads(str(payload_json)), - ) - for ( - root_id, - delivery_key, - delivery_kind, - turn_id, - status, - payload_json, - ) in conn.execute( - """ - SELECT id, delivery_key, delivery_kind, turn_id, status, - payload_json - FROM connector_outbox - WHERE delivery_kind IN ('final_ready', 'final_migration_hold') - ORDER BY id - """ - ).fetchall() - ] - source_identities_after_restart = sorted( - ( - str(turn_id), - str(source_turn_id), - str(stable_key), - ) - for turn_id, source_turn_id, stable_key in conn.execute( - """ - SELECT - turn_id, - json_extract(payload_json, '$.source_turn_id'), - json_extract(payload_json, '$.meta.stable_key') - FROM turns - WHERE host_id = ? - AND json_extract(payload_json, '$.source_turn_id') IS NOT NULL - ORDER BY turn_id - """, - (_HOST_ID,), - ).fetchall() - ) - source_revisions_after_restart = { - (str(turn_id), str(content_revision)) - for turn_id, content_revision in conn.execute( - """ - SELECT turns.turn_id, revisions.content_revision - FROM turns - JOIN turn_content_revisions AS revisions - ON revisions.host_id = turns.host_id - AND revisions.turn_id = turns.turn_id - AND revisions.is_current = 1 - WHERE turns.host_id = ? - AND json_extract( - turns.payload_json, - '$.source_turn_id' - ) IS NOT NULL - """, - (_HOST_ID,), - ).fetchall() - } - assert roots_after_restart == roots - assert source_identities_after_restart == source_identities - assert source_revisions_after_restart == expected_source_revisions - api = ConnectorOutboxAPI(db_path, _HOST_ID) - polled = api.poll({"name": "turn-final", "limit": 100}) - assert [item["key"] for item in polled["items"]] == [same_owner_key, new_key] - assert _HISTORICAL_FINAL_KEY not in { - item["key"] for item in polled["items"] - } - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] diff --git a/tests/test_delivery_retention_projection.py b/tests/test_delivery_retention_projection.py index 253b70f..ec32237 100644 --- a/tests/test_delivery_retention_projection.py +++ b/tests/test_delivery_retention_projection.py @@ -19,11 +19,11 @@ init_store, latest_snapshot, list_worker_bindings, - merge_turn_content, save_snapshot, turns_payload_from_store, upsert_worker_bindings, ) +from .store_helpers import apply_test_turn_refresh HOST_ID = "snapshot-retention-host" @@ -170,7 +170,7 @@ def _merge_continuity_final( worker_id: str, observed_at: str, ) -> dict[str, Any]: - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, worker_id, @@ -325,7 +325,7 @@ def _empty_snapshot(db_path: Path, *, second: int) -> Snapshot: def _seed_complete_final(db_path: Path, snapshot: Snapshot) -> tuple[str, str]: init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -489,7 +489,7 @@ def test_missing_identity_merge_persists_nonpollable_hold_on_current_placeholder init_store(db_path) assert save_snapshot(db_path, snapshot) is True - changed = merge_turn_content( + changed = apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -546,7 +546,7 @@ def test_worker_id_reuse_binds_each_root_to_immutable_stable_key( ) init_store(db_path) save_snapshot(db_path, first) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -577,7 +577,7 @@ def test_worker_id_reuse_binds_each_root_to_immutable_stable_key( ) assert second.workers[0].fingerprint != first.workers[0].fingerprint save_snapshot(db_path, second) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -660,7 +660,7 @@ def owner_snapshot(stable_key: str, second: int) -> Snapshot: "complete": True, "has_open_turn": False, } - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -689,7 +689,7 @@ def owner_snapshot(stable_key: str, second: int) -> Snapshot: second = owner_snapshot(STABLE_KEY_B, 2) assert save_snapshot(db_path, second) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -698,7 +698,7 @@ def owner_snapshot(stable_key: str, second: int) -> Snapshot: ) == 1 init_store(db_path) assert save_snapshot(db_path, second) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -757,7 +757,7 @@ def test_missing_owner_replay_holds_without_overwriting_existing_owner_root( "complete": True, "has_open_turn": False, } - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -777,7 +777,7 @@ def test_missing_owner_replay_holds_without_overwriting_existing_owner_root( timestamp=datetime(2026, 1, 1, 0, 0, 2, tzinfo=timezone.utc), ) assert save_snapshot(db_path, missing) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -786,7 +786,7 @@ def test_missing_owner_replay_holds_without_overwriting_existing_owner_root( ) == 1 init_store(db_path) assert save_snapshot(db_path, missing) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -934,7 +934,7 @@ def test_snapshot_omission_preserves_observed_source_provenance( original = _snapshot(db_path) init_store(db_path) save_snapshot(db_path, original) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -1094,7 +1094,7 @@ def test_same_owner_exact_source_survives_worker_fingerprint_space_and_source_ch assert first.workers[0].fingerprint != second.workers[0].fingerprint assert save_snapshot(db_path, second) is True upsert_worker_bindings(db_path, [_private_binding(second, PRIVATE_ROUTE_B)]) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-b", @@ -1142,7 +1142,7 @@ def test_same_owner_exact_source_survives_worker_fingerprint_space_and_source_ch init_store(db_path) assert save_snapshot(db_path, second) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-b", diff --git a/tests/test_delivery_retention_recovery.py b/tests/test_delivery_retention_recovery.py index 653b2f5..7961f92 100644 --- a/tests/test_delivery_retention_recovery.py +++ b/tests/test_delivery_retention_recovery.py @@ -17,17 +17,15 @@ from tendwire.store.sqlite import ( cleanup_acknowledged_final_retention, init_store, - merge_turn_content, save_snapshot, ) +from .store_helpers import apply_test_turn_refresh HOST_ID = "recovery-host" FINAL_NAME = "turn-final" CREATED_AT = "2026-01-01T00:00:00+00:00" STABLE_KEY = "wsk1_" + ("d" * 64) -RECOVERY_RAW_SOURCE = "legacy-recovery-backend-source" -RECOVERY_LEGACY_SOURCE_TOKEN = "turnsrc-422ef48fec1cfb0720da05bd" PRIVATE_ROUTE_SENTINEL = "PRIVATE-RECOVERY-ROUTE-SENTINEL" @@ -768,375 +766,10 @@ def test_repeated_recovery_history_is_hard_bounded_and_cumulative( assert source_status == "delivered" -def _seed_v10_recovered_lineage(db_path: Path) -> tuple[str, str, str]: - turn_id = "turn-v10-recovered" - revision = _insert_revision(db_path, turn_id=turn_id, final_text="abcdefghijkl") - failed_token = "twplan1.legacy-failed" - recovered_token = "twplan1.legacy-recovered" - authoritative_route = { - "schema_version": 2, - "turn_id": turn_id, - "content_revision": revision, - "final_identity": store_sqlite.turn_final_delivery_identity( - HOST_ID, - turn_id, - revision, - ), - "stable_key": STABLE_KEY, - "stable_key_version": 1, - } - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys = ON") - conn.execute( - """ - UPDATE turns - SET worker_id = 'worker-recovery-a', - worker_fingerprint = 'fingerprint-recovery-a', - space_id = 'space-recovery-a', - payload_json = ? - WHERE host_id = ? AND turn_id = ? - """, - ( - json.dumps( - { - "schema_version": 1, - "id": turn_id, - "host_id": HOST_ID, - "worker_id": "worker-recovery-a", - "worker_fingerprint": "fingerprint-recovery-a", - "space_id": "space-recovery-a", - "status": "complete", - "kind": "turn", - "source": "herdr-a", - "source_turn_id": RECOVERY_LEGACY_SOURCE_TOKEN, - "complete": True, - "has_open_turn": False, - "updated_at": CREATED_AT, - "meta": { - "stable_key": STABLE_KEY, - "stable_key_version": 1, - }, - "chat_id": PRIVATE_ROUTE_SENTINEL, - }, - sort_keys=True, - ), - HOST_ID, - turn_id, - ), - ) - failed_cursor = conn.execute( - """ - INSERT INTO turn_presentation_plans ( - host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - created_at, activated_at - ) VALUES (?, ?, ?, ?, ?, 'legacy-recovery-v10', 1, 3, 'failed', ?, ?) - """, - (HOST_ID, FINAL_NAME, failed_token, turn_id, revision, CREATED_AT, CREATED_AT), - ) - failed_plan_id = int(failed_cursor.lastrowid) - recovered_cursor = conn.execute( - """ - INSERT INTO turn_presentation_plans ( - host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - replaces_plan_token, recovers_plan_token, - created_at, activated_at, completed_at - ) VALUES ( - ?, ?, ?, ?, ?, 'legacy-recovery-v10', 2, 3, 'completed', - ?, ?, ?, ?, ? - ) - """, - ( - HOST_ID, - FINAL_NAME, - recovered_token, - turn_id, - revision, - failed_token, - failed_token, - CREATED_AT, - CREATED_AT, - CREATED_AT, - ), - ) - recovered_plan_id = int(recovered_cursor.lastrowid) - - def add_job( - plan_id: int, - sequence: int, - status: str, - key: str, - ) -> None: - outbox_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES (?, ?, ?, ?, ?, '{}', ?, ?, NULL) - """, - ( - HOST_ID, - FINAL_NAME, - key, - status, - json.dumps({"turn": authoritative_route}, sort_keys=True), - CREATED_AT, - CREATED_AT, - ), - ) - outbox_id = int(outbox_cursor.lastrowid) - spans_json = json.dumps( - [ - { - "field": "assistant_final_text", - "start_char": sequence * 4, - "end_char": (sequence + 1) * 4, - } - ], - sort_keys=True, - ) - conn.execute( - """ - INSERT INTO turn_presentation_jobs ( - plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - ) VALUES (?, ?, 'upsert', ?, ?, ?, ?) - """, - ( - plan_id, - sequence, - sequence, - spans_json, - outbox_id, - CREATED_AT, - ), - ) - if status in {"delivered", "dead_letter"}: - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, - status, response_json, private_state_json, - created_at, delivered_at - ) VALUES (?, ?, ?, ?, 1, ?, '{}', '{}', ?, ?) - """, - ( - outbox_id, - HOST_ID, - FINAL_NAME, - key, - "delivered" if status == "delivered" else "failed", - CREATED_AT, - CREATED_AT, - ), - ) - - add_job(failed_plan_id, 0, "delivered", "turn-final:legacy-root:000000") - add_job(failed_plan_id, 1, "dead_letter", "turn-final:legacy-root:000001") - add_job(failed_plan_id, 2, "queued", "turn-final:legacy-root:000002") - add_job(recovered_plan_id, 1, "delivered", "turn-final:legacy-recovered:000001") - add_job(recovered_plan_id, 2, "delivered", "turn-final:legacy-recovered:000002") - conn.execute( - """ - INSERT INTO turn_presentation_recoveries ( - host_id, name, request_id, failed_plan_id, recovered_plan_id, - failed_plan_token, recovered_plan_token, generation, - source_job_count, delivered_prefix_count, fresh_job_count, - retained_failed_job_count, prior_attempt_count, outcome, created_at - ) VALUES (?, ?, 'legacy-request', ?, ?, ?, ?, 2, 3, 1, 2, 1, 2, - 'recovered', ?) - """, - ( - HOST_ID, - FINAL_NAME, - failed_plan_id, - recovered_plan_id, - failed_token, - recovered_token, - CREATED_AT, - ), - ) - conn.execute("PRAGMA user_version = 10") - return turn_id, revision, failed_token -def test_v12_migration_uses_effective_recovery_lineage_without_repost_or_hold( - tmp_path: Path, -) -> None: - db_path = tmp_path / "v12-recovered-lineage.db" - turn_id, revision, failed_token = _seed_v10_recovered_lineage(db_path) - init_store(db_path) - key = _final_key(turn_id, revision) - api = ConnectorOutboxAPI(db_path, HOST_ID) - assert api.poll({"name": FINAL_NAME, "limit": 100})["items"] == [] - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 28 - anchor = conn.execute( - """ - SELECT delivery_kind, status - FROM connector_outbox - WHERE delivery_key = ? - """, - (key,), - ).fetchone() - failed_state = conn.execute( - "SELECT state FROM turn_presentation_plans WHERE plan_token = ?", - (failed_token,), - ).fetchone()[0] - linked_proof = conn.execute( - """ - SELECT plans.state - FROM turn_presentation_plans AS plans - JOIN connector_outbox AS source ON source.id = plans.source_outbox_id - WHERE source.delivery_key = ? - """, - (key,), - ).fetchone()[0] - assert anchor == ("final_ready", "delivered") - assert failed_state == "superseded" - assert linked_proof == "completed" - - cleanup = cleanup_acknowledged_final_retention( - db_path, - HOST_ID, - acknowledged_final_retention_days=1, - acknowledged_final_retention_count=1, - batch_size=100, - now="2099-01-01T00:00:00+00:00", - ) - assert cleanup["deleted"] == 1 - - -def test_v12_recovery_history_stays_immutable_when_observed_turn_arrives( - tmp_path: Path, -) -> None: - db_path = tmp_path / "v12-recovered-lineage-owner-churn.db" - turn_id, revision, failed_token = _seed_v10_recovered_lineage(db_path) - init_store(db_path) - - original_key = _final_key(turn_id, revision) - before = _turn_graph_snapshot(db_path, turn_id) - assert before["turn_identity"] == ( - turn_id, - turn_id, - RECOVERY_LEGACY_SOURCE_TOKEN, - 1, - ) - assert len(before["attempts"]) == 3 - assert [row[4] for row in before["attempts"]] == [ - "delivered", - "delivered", - "delivered", - ] - recovered_plan = next(row for row in before["plans"] if row[6] == "completed") - root = next(row for row in before["outbox"] if row[1] == original_key) - assert recovered_plan[9] == root[0] - assert before["recoveries"][0][4:6] == ( - failed_token, - recovered_plan[1], - ) - - worker_b = _owner_snapshot( - db_path, - worker_id="worker-recovery-b", - worker_name="Recovery Worker B", - space_id="space-recovery-b", - second=2, - ) - assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( - db_path, - HOST_ID, - "worker-recovery-b", - { - "source_turn_id": RECOVERY_RAW_SOURCE, - "assistant_final_text": "abcdefghijkl", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:03+00:00", - ) == 1 - - after_churn = _turn_graph_snapshot(db_path, turn_id) - assert after_churn == before - api = ConnectorOutboxAPI(db_path, HOST_ID) - observed_items = api.poll({"name": FINAL_NAME, "limit": 100})["items"] - assert len(observed_items) == 1 - assert observed_items[0]["key"] != original_key - with sqlite3.connect(str(db_path)) as conn: - current = conn.execute( - """ - SELECT worker_id, worker_fingerprint, space_id, payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (HOST_ID, turn_id), - ).fetchone() - assert current is not None - current_payload = json.loads(str(current[3])) - assert current[:3] == ( - "worker-recovery-a", - "fingerprint-recovery-a", - "space-recovery-a", - ) - assert current_payload["id"] == turn_id - assert current_payload["source_turn_id"] == RECOVERY_LEGACY_SOURCE_TOKEN - observed = conn.execute( - """ - SELECT turn_id, worker_id, worker_fingerprint, space_id, payload_json - FROM turns - WHERE host_id = ? AND turn_id != ? - """, - (HOST_ID, turn_id), - ).fetchone() - assert observed is not None - observed_payload = json.loads(str(observed[4])) - assert tuple(observed[1:4]) == ( - "worker-recovery-b", - worker_b.workers[0].fingerprint, - "space-recovery-b", - ) - assert observed_payload["id"] == str(observed[0]) - assert observed_payload["source_turn_id"] != RECOVERY_LEGACY_SOURCE_TOKEN - public_payloads = [ - str(row[0]) - for row in conn.execute( - """ - SELECT payload_json FROM turns - WHERE host_id = ? AND turn_id != ? - UNION ALL - SELECT payload_json FROM connector_outbox - WHERE host_id = ? AND turn_id != ? - """, - (HOST_ID, turn_id, HOST_ID, turn_id), - ).fetchall() - ] - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - encoded_public = "\n".join(public_payloads) - assert PRIVATE_ROUTE_SENTINEL not in encoded_public - assert RECOVERY_RAW_SOURCE not in encoded_public - - init_store(db_path) - assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( - db_path, - HOST_ID, - "worker-recovery-b", - { - "source_turn_id": RECOVERY_RAW_SOURCE, - "assistant_final_text": "abcdefghijkl", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:03+00:00", - ) == 0 - assert ConnectorOutboxAPI(db_path, HOST_ID).poll( - {"name": FINAL_NAME, "limit": 100} - )["items"] == [] - assert _turn_graph_snapshot(db_path, turn_id) == before def test_known_incomplete_final_is_hold_then_complete_revision_drains( @@ -1778,7 +1411,7 @@ def test_source_less_recovery_ids_survive_same_owner_worker_churn_and_ack_loss( ) init_store(db_path) assert save_snapshot(db_path, worker_a) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-source-less-a", @@ -1843,7 +1476,7 @@ def test_source_less_recovery_ids_survive_same_owner_worker_churn_and_ack_loss( ) assert worker_b.workers[0].fingerprint != worker_a.workers[0].fingerprint assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-source-less-b", @@ -1936,7 +1569,7 @@ def test_source_less_recovery_ids_survive_same_owner_worker_churn_and_ack_loss( assert replayed_recovery["failed_plan_token"] == failed_plan["plan_token"] assert replayed_recovery["plan_token"] == recovered["plan_token"] assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-source-less-b", diff --git a/tests/test_public_content_safety.py b/tests/test_public_content_safety.py index e94d7a9..1f13de7 100644 --- a/tests/test_public_content_safety.py +++ b/tests/test_public_content_safety.py @@ -39,11 +39,11 @@ attention_payload_from_store, init_store, get_turn_content, - merge_turn_content, save_snapshot, tail_event_metadata, turns_payload_from_store, ) +from .store_helpers import apply_test_turn_refresh def _sentinel_corpus() -> dict[str, str]: @@ -893,38 +893,22 @@ def save_complete(observed_at: str, attention: list[AttentionSignal]) -> None: {"name": "attention", "limit": 10} ) - # Copy the leased lifecycle jobs into a v4-shaped fixture. Migration must - # preserve live refs privately without putting its grouping markers into - # either the public attention projection or the connector payload. - migration_db_path = tmp_path / "migration-public-boundaries.db" - with ( - sqlite3.connect(str(db_path)) as source, - sqlite3.connect(str(migration_db_path)) as destination, - ): - source.backup(destination) - with sqlite3.connect(str(migration_db_path)) as conn: - conn.execute("DROP TABLE attention_lifecycles") - conn.execute("PRAGMA user_version = 4") - - init_store(migration_db_path) - migrated_feed = attention_payload_from_store(migration_db_path, host_id) - with sqlite3.connect(str(migration_db_path)) as conn: - migrated_attention_rows = [ + # Re-open the exact current schema and prove that every store-backed public + # edge re-sanitizes persisted data after initialization as well as in the + # original process. + init_store(db_path) + reloaded_feed = attention_payload_from_store(db_path, host_id) + with sqlite3.connect(str(db_path)) as conn: + reloaded_attention_rows = [ json.loads(row[0]) for row in conn.execute("SELECT payload_json FROM attention_items") ] - migrated_outbox_rows = [ + reloaded_outbox_rows = [ json.loads(row[0]) for row in conn.execute( "SELECT payload_json FROM connector_outbox ORDER BY id" ) ] - migration_private_states = [ - json.loads(row[0]) - for row in conn.execute( - "SELECT private_state_json FROM connector_outbox ORDER BY id" - ) - ] assert initial_feed is not None assert [item["severity"] for item in initial_feed["attention"]] == ["warning"] @@ -944,9 +928,9 @@ def save_complete(observed_at: str, attention: list[AttentionSignal]) -> None: "attention_created", ] - assert migrated_feed is not None - assert len(migrated_feed["attention"]) == 1 - assert migrated_feed["attention"][0]["reason"] == "Review the safe public result" + assert reloaded_feed is not None + assert len(reloaded_feed["attention"]) == 1 + assert reloaded_feed["attention"][0]["reason"] == "Review the safe public result" assert connector_payload["ok"] is True assert connector_payload["items"] assert all( @@ -956,9 +940,8 @@ def save_complete(observed_at: str, attention: list[AttentionSignal]) -> None: assert snapshot_rows assert event_rows assert attention_rows - assert migrated_attention_rows - assert migrated_outbox_rows - assert any("migration_group" in state for state in migration_private_states) + assert reloaded_attention_rows + assert reloaded_outbox_rows public_attention_surfaces = ( initial_feed, @@ -966,13 +949,13 @@ def save_complete(observed_at: str, attention: list[AttentionSignal]) -> None: pending_feed, resolved_feed, recurrence_feed, - migrated_feed, + reloaded_feed, attention_rows, - migrated_attention_rows, + reloaded_attention_rows, ) outbox_payload_surfaces = ( pre_migration_outbox_rows, - migrated_outbox_rows, + reloaded_outbox_rows, [item["payload"] for item in connector_payload["items"]], ) for surface in public_attention_surfaces + outbox_payload_surfaces: @@ -1102,7 +1085,7 @@ def test_boundary_secret_stays_redacted_through_store_pages_and_plan_outbox( ) init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, diff --git a/tests/test_release_readiness.py b/tests/test_release_readiness.py index afd1686..e23073b 100644 --- a/tests/test_release_readiness.py +++ b/tests/test_release_readiness.py @@ -273,9 +273,7 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( "batch_size": 5, "examined": 0, "deleted": 0, - "tombstoned": 0, "remaining_candidates": False, - "replay_identity_retained": True, }, "final_retention": { "examined": 0, @@ -298,23 +296,6 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( "deleted": 0, "remaining_candidates": False, }, - "herdr_turns": { - "schema_version": 1, - "ok": True, - "status": "ok", - "scope": "database", - "host_id": None, - "dry_run": False, - "retention_days": 36500, - "retention_count": 100, - "cutoff_at": "1926-02-04T00:00:00+00:00", - "batch_size": 5, - "examined": 0, - "deleted": 0, - "deleted_completions": 0, - "deleted_watermarks": 0, - "remaining_candidates": False, - }, "batch_size": 5, } assert status["counts"]["snapshots"] == 1 @@ -378,18 +359,6 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( assert cleanup["snapshots"]["examined"] == 0 assert cleanup["outbox"]["updated"] == 0 assert cleanup["turn_content"]["examined"] == 0 - assert cleanup["herdr_turns"] == { - "dry_run": True, - "retention_days": 30, - "retention_count": 4096, - "cutoff_at": "2025-12-11T00:00:00+00:00", - "batch_size": 100, - "examined": 0, - "deleted": 0, - "deleted_completions": 0, - "deleted_watermarks": 0, - "remaining_candidates": False, - } assert cleanup["command_requests"] == { "ok": True, "status": "ok", diff --git a/tests/test_snapshot_sanitize_performance.py b/tests/test_snapshot_sanitize_performance.py index 1eb26ca..46c7900 100644 --- a/tests/test_snapshot_sanitize_performance.py +++ b/tests/test_snapshot_sanitize_performance.py @@ -64,20 +64,15 @@ def _turn_content() -> dict[str, object]: } -@pytest.mark.parametrize("turn_model", ["legacy", "observed"]) -@pytest.mark.parametrize("compatibility_source_token", [False, True]) def test_unchanged_turn_reobservation_performs_no_forbidden_phrase_scans( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, - turn_model: str, - compatibility_source_token: bool, ) -> None: - db_path = tmp_path / f"unchanged-turn-{turn_model}.db" + db_path = tmp_path / "unchanged-turn.db" store_sqlite.init_store(db_path) store_sqlite.save_snapshot( db_path, _snapshot(1, [_worker(1)]), - turn_model=turn_model, ) content = _turn_content() first = store_sqlite.apply_turn_refresh( @@ -86,43 +81,9 @@ def test_unchanged_turn_reobservation_performs_no_forbidden_phrase_scans( "worker-1", content, observed_at="2026-07-20T00:00:02+00:00", - turn_model=turn_model, ) assert first.updated == 1 - if compatibility_source_token: - with sqlite3.connect(db_path) as conn: - row = conn.execute( - """ - SELECT turn_id, payload_json - FROM turns - WHERE host_id = ? - AND json_extract(payload_json, '$.source_turn_id') != '' - """, - (HOST_ID,), - ).fetchone() - assert row is not None - payload = store_sqlite._json_object(row[1]) - candidates = store_sqlite.turn_source_id_candidates( - content["source_turn_id"], - meta=payload["meta"], - source=payload["source"], - kind=payload["kind"], - ) - assert len(candidates) == 2 - payload["source_turn_id"] = candidates[1] - conn.execute( - """ - UPDATE turns SET payload_json = ? - WHERE host_id = ? AND turn_id = ? - """, - ( - store_sqlite._canonical_json(payload), - HOST_ID, - str(row[0]), - ), - ) - scans = 0 original_scan = models._is_forbidden_public_text_phrase @@ -142,7 +103,6 @@ def recording_scan(value: str) -> bool: "worker-1", content, observed_at="2026-07-20T00:00:03+00:00", - turn_model=turn_model, ) assert second.updated == 0 @@ -158,7 +118,6 @@ def test_unchanged_observed_turn_still_attempts_submission_settlement( store_sqlite.save_snapshot( db_path, _snapshot(1, [_worker(1)]), - turn_model="observed", ) content = _turn_content() store_sqlite.apply_turn_refresh( @@ -167,11 +126,10 @@ def test_unchanged_observed_turn_still_attempts_submission_settlement( "worker-1", content, observed_at="2026-07-20T00:00:02+00:00", - turn_model="observed", ) settle_calls = 0 - original_settle = store_sqlite.settle_submission_links_conn + original_settle = store_sqlite._settle_submission_links_conn def recording_settle(*args, **kwargs): nonlocal settle_calls @@ -180,7 +138,7 @@ def recording_settle(*args, **kwargs): monkeypatch.setattr( store_sqlite, - "settle_submission_links_conn", + "_settle_submission_links_conn", recording_settle, ) second = store_sqlite.apply_turn_refresh( @@ -189,7 +147,6 @@ def recording_settle(*args, **kwargs): "worker-1", content, observed_at="2026-07-20T00:00:03+00:00", - turn_model="observed", ) assert second.updated == 0 diff --git a/tests/test_store.py b/tests/test_store.py index f0b78c3..c538da8 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -6,6 +6,7 @@ import gc import hashlib import json +import logging import multiprocessing import os import sqlite3 @@ -41,7 +42,6 @@ CompactionOptions, ack_connector_delivery, backend_pending_choice_terminal_effect, - append_event, attention_payload_from_store, cleanup_event_retention, compact_store, @@ -55,23 +55,24 @@ get_command_request, init_store, latest_snapshot, - list_attention_items, - list_hosts, list_worker_bindings, reclaim_expired_connector_leases, poll_connector_outbox, mark_command_send_started, reserve_command_request, reserve_terminal_command_replay, - resolve_worker_binding, run_store_maintenance, save_snapshot, store_status, tail_event_metadata, - merge_turn_content, turns_payload_from_store, upsert_worker_bindings, ) +from .store_helpers import ( + read_test_attention_items, + apply_test_backend_pending, + apply_test_turn_refresh, +) _PR6_TABLES = { @@ -256,8 +257,6 @@ def _invoke_creation_capable_store_path(name: str, db_path: Path) -> None: ) if name == "init_store": init_store(db_path) - elif name == "append_event": - append_event(db_path, "host-a", "store.test", {"safe": True}) elif name == "save_snapshot": save_snapshot(db_path, snapshot) elif name == "upsert_worker_bindings": @@ -341,7 +340,7 @@ def broaden_then_validate(parent_fd: int, leaf: str) -> None: assert validated_live_family is True -def test_store_startup_repairs_broad_modes_idempotently_and_preserves_data( +def test_store_startup_repairs_broad_modes_before_explicit_schema_discard( tmp_path: Path, ) -> None: state_dir = tmp_path / "broad-state" @@ -358,20 +357,24 @@ def test_store_startup_repairs_broad_modes_idempotently_and_preserves_data( init_store(db_path) first_modes = (_mode(state_dir), _mode(db_path)) with closing(sqlite3.connect(str(db_path))) as conn, conn: - first_value = conn.execute("SELECT value FROM preserved").fetchone()[0] + first_preserved = conn.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'preserved'" + ).fetchone() first_version = _user_version(conn) conn.close() init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - second_value = conn.execute("SELECT value FROM preserved").fetchone()[0] + second_preserved = conn.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'preserved'" + ).fetchone() second_version = _user_version(conn) conn.close() assert first_modes == (0o700, 0o600) assert (_mode(state_dir), _mode(db_path)) == first_modes assert db_path.stat().st_ino == inode - assert (first_value, second_value) == ("kept", "kept") + assert (first_preserved, second_preserved) == (None, None) assert (first_version, second_version) == ( store_sqlite.STORE_SCHEMA_VERSION, store_sqlite.STORE_SCHEMA_VERSION, @@ -592,7 +595,6 @@ def reject_mutation(*_args: Any, **_kwargs: Any) -> Any: "entrypoint", [ "init_store", - "append_event", "save_snapshot", "upsert_worker_bindings", "expire_worker_bindings", @@ -1279,234 +1281,6 @@ def capture_connection(*args: Any, **kwargs: Any) -> sqlite3.Connection: assert {process.pid for process in multiprocessing.active_children()} == before_children -def test_store_initializes_v8_schema_with_companion_attention_lifecycle(tmp_path: Path) -> None: - db_path = tmp_path / "tendwire.db" - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _PR6_TABLES <= _table_names(conn) - columns = {row[1] for row in conn.execute("PRAGMA table_info(snapshots)")} - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - assert {"host_id", "created_at", "payload", "content_fingerprint"} <= columns - snapshot_indexes = { - str(row[1]) - for row in conn.execute("PRAGMA index_list(snapshots)").fetchall() - } - assert { - "idx_snapshots_host_newest", - "idx_snapshots_created_host_id", - } <= snapshot_indexes - assert { - "idx_snapshots_host_id", - "idx_snapshots_created_at", - "idx_snapshots_content_fingerprint", - "idx_snapshots_host_created_id", - }.isdisjoint(snapshot_indexes) - binding_columns = {row[1] for row in conn.execute("PRAGMA table_info(worker_bindings)")} - assert { - "host_id", - "worker_id", - "worker_fingerprint", - "backend", - "target_kind", - "target_value", - "turn_target_kind", - "turn_target_value", - "sendable", - "reason", - "observed_at", - "expires_at", - "private_fingerprint", - } <= binding_columns - binding_indexed = _indexed_columns(conn, "worker_bindings") - assert { - "worker_id", - "worker_fingerprint", - "private_fingerprint", - "target_kind", - "target_value", - "expires_at", - } <= binding_indexed - command_columns = { - row[1] for row in conn.execute("PRAGMA table_info(commands)") - } - assert { - "host_id", - "request_id", - "action", - "canonical_version", - "canonical_fingerprint", - "public_worker_id", - "state", - "status", - "request_json", - "result_json", - "reserved_at", - "send_started_at", - "terminal_at", - "updated_at", - "legacy_collision", - "legacy_collision_count", - } == command_columns - {"id", "created_at"} - receipt_columns = { - row[1] for row in conn.execute("PRAGMA table_info(command_receipts)") - } - assert { - "canonical_version", - "canonical_fingerprint", - "canonical_request_json", - "public_worker_id", - "state", - "owner_token_hash", - "owner_expires_at", - "binding_fingerprint", - "reserved_at", - "send_started_at", - "terminal_at", - "updated_at", - } <= receipt_columns - assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" - attention_columns = {row[1] for row in conn.execute("PRAGMA table_info(attention_items)")} - assert { - "attention_id", - "fingerprint", - "first_seen_at", - "last_seen_at", - "last_changed_at", - "resolved_at", - "lifecycle_status", - "resolved_reason", - "signal_count", - } <= attention_columns - attention_indexed = _indexed_columns(conn, "attention_items") - assert {"lifecycle_status", "last_seen_at", "fingerprint"} <= attention_indexed - lifecycle_columns = { - row[1] - for row in conn.execute("PRAGMA table_info(attention_lifecycles)") - } - assert lifecycle_columns == { - "host_id", - "family_key", - "generation", - "lifecycle_status", - "current_attention_id", - "first_seen_at", - "last_positive_at", - "first_missing_at", - "missing_observation_count", - "last_accepted_at", - "last_observation_key", - "max_notified_severity_rank", - } - assert {"lifecycle_status", "current_attention_id"} <= _indexed_columns( - conn, "attention_lifecycles" - ) - assert { - row[1] - for row in conn.execute("PRAGMA table_info(turn_content_revisions)") - } == { - "host_id", - "turn_id", - "content_revision", - "user_text", - "assistant_final_text", - "user_state", - "final_state", - "user_char_length", - "user_byte_length", - "final_char_length", - "final_byte_length", - "user_page_count", - "final_page_count", - "is_current", - "created_at", - "superseded_at", - } - revision_indexes = { - row[1] - for row in conn.execute("PRAGMA index_list(turn_content_revisions)") - } - assert {"ux_turn_content_current", "idx_turn_content_cleanup"} <= revision_indexes - assert { - row[1] - for row in conn.execute( - "PRAGMA table_info(turn_content_page_boundaries)" - ) - } == { - "host_id", - "turn_id", - "content_revision", - "field", - "page_index", - "start_char", - "start_byte", - } - assert { - row[1] - for row in conn.execute("PRAGMA table_info(turn_presentation_plans)") - } == { - "id", - "host_id", - "name", - "plan_token", - "turn_id", - "content_revision", - "presentation_version", - "generation", - "part_count", - "state", - "replaces_plan_token", - "recovers_plan_token", - "created_at", - "activated_at", - "completed_at", - "source_outbox_id", - } - assert { - row[1] - for row in conn.execute("PRAGMA table_info(turn_presentation_jobs)") - } == { - "id", - "plan_id", - "sequence_index", - "operation", - "part_ordinal", - "spans_json", - "outbox_id", - "created_at", - } - job_indexes = { - row[1] - for row in conn.execute("PRAGMA index_list(turn_presentation_jobs)") - } - assert { - "idx_turn_presentation_jobs_plan_sequence", - "idx_turn_presentation_jobs_outbox", - } <= job_indexes - assert { - row[1] - for row in conn.execute( - "PRAGMA table_info(turn_presentation_recoveries)" - ) - } == { - "id", - "host_id", - "name", - "request_id", - "failed_plan_id", - "recovered_plan_id", - "failed_plan_token", - "recovered_plan_token", - "generation", - "source_job_count", - "delivered_prefix_count", - "fresh_job_count", - "retained_failed_job_count", - "prior_attempt_count", - "outcome", - "created_at", - } def test_store_connections_apply_wal_busy_timeout_and_foreign_keys(tmp_path: Path) -> None: @@ -1553,29 +1327,23 @@ def test_store_command_receipts_have_unique_logical_key_index(tmp_path: Path) -> def test_store_status_tail_and_retention_cleanup_are_host_scoped_and_bounded(tmp_path: Path) -> None: db_path = tmp_path / "maintenance.db" config = Config(host_id="storehost", db_path=db_path) - snapshot = project_from_raw(config, workers=[{"id": "worker-1", "name": "Worker One"}]) - save_snapshot(db_path, snapshot) - append_event( - db_path, - "storehost", - "private.event", - {"pane_id": "sentinel-private-pane", "raw_payload": "sentinel-private-raw"}, - observed_at="2026-01-01T00:00:00+00:00", + old_snapshot = project_from_raw( + config, + workers=[{"id": "worker-1", "name": "Worker One"}], + timestamp=datetime.fromisoformat("2026-01-01T00:00:00+00:00"), ) - append_event( - db_path, - "storehost", - "public.event", - {"safe": "kept"}, - observed_at="2026-01-09T00:00:00+00:00", + new_snapshot = project_from_raw( + config, + workers=[{"id": "worker-2", "name": "Worker Two"}], + timestamp=datetime.fromisoformat("2026-01-09T00:00:00+00:00"), ) - append_event( - db_path, - "otherhost", - "other.event", - {"safe": "kept"}, - observed_at="2026-01-01T00:00:00+00:00", + other_snapshot = project_from_raw( + Config(host_id="otherhost", db_path=db_path), + workers=[{"id": "worker-3", "name": "Other Worker"}], + timestamp=datetime.fromisoformat("2026-01-01T00:00:00+00:00"), ) + for snapshot in (old_snapshot, new_snapshot, other_snapshot): + save_snapshot(db_path, snapshot) with sqlite3.connect(str(db_path)) as conn: conn.execute( """ @@ -1629,72 +1397,10 @@ def test_store_status_tail_and_retention_cleanup_are_host_scoped_and_bounded(tmp assert cleanup["deleted"] == 1 assert host_events == before["counts"]["events"] - 1 assert other_events == 1 - assert snapshots == 1 + assert snapshots == 2 assert outbox_rows == 1 -def test_store_operational_metadata_buckets_unsafe_labels(tmp_path: Path) -> None: - db_path = tmp_path / "unsafe-metadata.db" - init_store(db_path) - append_event( - db_path, - "storehost", - "telegram.delivery", - {"safe": "kept"}, - aggregate_type="raw_payload", - observed_at="2026-01-01T00:00:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "storehost", - "attention", - "job-unsafe", - "telegram_delivery", - '{"safe":"kept"}', - "{}", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "storehost", - "attention", - "job-queued", - "queued", - '{"safe":"kept"}', - "{}", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - - status = store_status(db_path, "storehost") - tail = tail_event_metadata(db_path, "storehost", limit=10) - encoded = json.dumps({"status": status, "tail": tail}, sort_keys=True).lower() - - assert status["outbox"]["pending"] == 1 - assert status["outbox"]["by_status"]["queued"] == 1 - assert status["outbox"]["by_status"]["unknown"] == 1 - assert "telegram_delivery" not in status["outbox"]["by_status"] - assert tail["events"][0]["event_type"] == "unknown" - assert tail["events"][0]["aggregate_type"] == "unknown" - assert "telegram" not in encoded - assert "raw_payload" not in encoded - assert "delivery" not in encoded def test_attention_payload_from_store_buckets_unsafe_row_text(tmp_path: Path) -> None: @@ -1840,19 +1546,6 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( ) _save_observation(db_path, snapshot, "positive", snapshot.updated_at) - append_event( - db_path, - "public-host", - "private.adapter", - { - "note": unsafe_value, - "pane_id": "private-pane", - "markdown": safe_markdown, - }, - aggregate_type="worker", - aggregate_id="worker-public", - observed_at="2026-01-01T00:01:00+00:00", - ) binding = WorkerBinding( host_id="public-host", worker_id="worker-public", @@ -1903,7 +1596,7 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( now="2026-01-01T00:01:02+00:00", ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "public-host", "worker-public", @@ -1915,7 +1608,7 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( }, observed_at="2026-01-01T00:02:00+00:00", ) == 1 - assert store_sqlite.merge_backend_pending( + assert apply_test_backend_pending( db_path, "public-host", "worker-public", @@ -1982,8 +1675,10 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( turns = turns_payload_from_store(db_path, "public-host", snapshot=restored) attention = attention_payload_from_store(db_path, "public-host") assert attention is not None - attention_items = list_attention_items(db_path, "public-host") - backend_pending = store_sqlite.list_backend_pending(db_path, "public-host") + attention_items = read_test_attention_items(db_path, "public-host") + backend_pending = store_sqlite.pending_payload_from_store( + db_path, "public-host" + ) tail = tail_event_metadata(db_path, "public-host", limit=20) public_readers = { "snapshot": restored.to_dict(), @@ -2035,18 +1730,6 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( assert values, f"{table}.{column} should be populated" assert all(unsafe_value not in value for value in values) - event_payload = json.loads( - conn.execute( - """ - SELECT payload_json - FROM events - WHERE host_id = ? AND event_type = ? - ORDER BY id DESC - LIMIT 1 - """, - ("public-host", "private.adapter"), - ).fetchone()[0] - ) outbox_private = json.loads( conn.execute( """ @@ -2060,8 +1743,6 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( ).fetchone()[0] ) - assert event_payload["note"] == unsafe_value - assert event_payload["pane_id"] == "private-pane" assert outbox_private["note"] == unsafe_value assert "lease_token" in outbox_private private_bindings = list_worker_bindings( @@ -2150,287 +1831,10 @@ def test_store_maintenance_dry_run_and_exhausted_outbox_status(tmp_path: Path) - assert json.loads(private_state) == {} -def test_store_maintenance_bounds_herdr_completions_and_drops_dead_watermarks( - tmp_path: Path, -) -> None: - db_path = tmp_path / "herdr-turn-maintenance.db" - init_store(db_path) - host_id = "herdr-retention-host" - dead_pane = "workspace:pDead" - active_pane = "workspace:pLive" - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - dead_pane, - turn_epoch=7, - last_turn=0, - observed_at="2026-01-01T00:00:00+00:00", - ) - for turn in range(1, 4): - store_sqlite.record_herdr_turn_completion( - db_path, - host_id, - dead_pane, - turn_epoch=7, - turn=turn, - outcome="completed", - completed_unix_ms=turn, - message=None, - message_truncated=False, - agent_session_path=None, - worker_id="dead-worker", - refreshed_turn_id=None, - observed_at=f"2026-01-0{turn}T00:00:00+00:00", - ) - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - active_pane, - turn_epoch=7, - last_turn=9, - observed_at="2026-01-01T00:00:00+00:00", - ) - store_sqlite.record_herdr_turn_completion( - db_path, - host_id, - active_pane, - turn_epoch=7, - turn=9, - outcome="completed", - completed_unix_ms=9, - message=None, - message_truncated=False, - agent_session_path=None, - worker_id="dead-worker", - refreshed_turn_id=None, - observed_at="2026-01-09T00:00:00+00:00", - ) - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=host_id, - # The same logical worker moved from dead_pane to active_pane. - # Only a current pane target keeps a watermark alive. - worker_id="dead-worker", - worker_fingerprint="live-fingerprint", - backend="herdr", - target_kind="terminal_id", - target_value="live-terminal", - turn_target_kind="pane_id", - turn_target_value=active_pane, - sendable=True, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="live-private-fingerprint", - ) - ], - ) - - result = run_store_maintenance( - db_path, - host_id, - retention_days=30, - max_outbox_attempts=3, - now="2026-03-15T00:00:00+00:00", - herdr_turn_retention_days=30, - herdr_turn_retention_count=1, - herdr_turn_batch_size=4, - ) - - assert result["ok"] is True - assert result["herdr_turns"]["deleted_completions"] == 3 - assert result["herdr_turns"]["deleted_watermarks"] == 1 - with closing(sqlite3.connect(str(db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - ORDER BY turn - """, - (host_id, dead_pane), - ).fetchall() == [] - assert conn.execute( - """ - SELECT pane_id, turn - FROM herdr_turn_completions - WHERE host_id = ? - ORDER BY pane_id, turn - """, - (host_id,), - ).fetchall() == [(active_pane, 9)] - assert conn.execute( - """ - SELECT pane_id - FROM herdr_turn_watermarks - WHERE host_id = ? - ORDER BY pane_id - """, - (host_id,), - ).fetchall() == [(active_pane,)] - - -def test_herdr_turn_maintenance_reports_watermark_deferred_by_full_batch( - tmp_path: Path, -) -> None: - db_path = tmp_path / "herdr-turn-deferred-watermark.db" - init_store(db_path) - host_id = "herdr-recent-retention-host" - active_pane = "workspace:pLive" - dead_pane = "workspace:pDead" - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - dead_pane, - turn_epoch=7, - last_turn=0, - observed_at="2026-01-01T00:00:00+00:00", - ) - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - active_pane, - turn_epoch=7, - last_turn=0, - observed_at="2026-01-01T00:00:00+00:00", - ) - for turn in range(1, 4): - store_sqlite.record_herdr_turn_completion( - db_path, - host_id, - active_pane, - turn_epoch=7, - turn=turn, - outcome="completed", - completed_unix_ms=turn, - message=None, - message_truncated=False, - agent_session_path=None, - worker_id="live-worker", - refreshed_turn_id=None, - observed_at=f"2026-01-0{turn}T00:00:00+00:00", - ) - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=host_id, - worker_id="live-worker", - worker_fingerprint="live-fingerprint", - backend="herdr", - target_kind="terminal_id", - target_value="live-terminal", - turn_target_kind="pane_id", - turn_target_value=active_pane, - sendable=True, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="live-private-fingerprint", - ) - ], - ) - - first = store_sqlite.cleanup_herdr_turn_retention( - db_path, - host_id=host_id, - retention_days=30, - retention_count=1, - batch_size=2, - now="2026-03-15T00:00:00+00:00", - ) - - assert first["deleted_completions"] == 2 - assert first["deleted_watermarks"] == 0 - assert first["remaining_candidates"] is True - - second = store_sqlite.cleanup_herdr_turn_retention( - db_path, - host_id=host_id, - retention_days=30, - retention_count=1, - batch_size=2, - now="2026-03-15T00:00:00+00:00", - ) - - assert second["deleted_completions"] == 0 - assert second["deleted_watermarks"] == 1 - assert second["remaining_candidates"] is False - with closing(sqlite3.connect(str(db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - ORDER BY turn - """, - (host_id, active_pane), - ).fetchall() == [(3,)] - assert conn.execute( - """ - SELECT pane_id - FROM herdr_turn_watermarks - WHERE host_id = ? - ORDER BY pane_id - """, - (host_id,), - ).fetchall() == [(active_pane,)] -def test_herdr_turn_maintenance_low_count_preserves_fresh_rows( - tmp_path: Path, -) -> None: - db_path = tmp_path / "herdr-turn-fresh-count-floor.db" - init_store(db_path) - host_id = "herdr-fresh-retention-host" - pane_id = "workspace:pFresh" - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - pane_id, - turn_epoch=7, - last_turn=0, - observed_at="2026-03-14T00:00:00+00:00", - ) - for turn in range(1, 4): - store_sqlite.record_herdr_turn_completion( - db_path, - host_id, - pane_id, - turn_epoch=7, - turn=turn, - outcome="completed", - completed_unix_ms=turn, - message=None, - message_truncated=False, - agent_session_path=None, - worker_id="fresh-worker", - refreshed_turn_id=None, - observed_at=f"2026-03-14T00:00:0{turn}+00:00", - ) - result = store_sqlite.cleanup_herdr_turn_retention( - db_path, - host_id=host_id, - retention_days=30, - retention_count=1, - batch_size=100, - now="2026-03-15T00:00:00+00:00", - ) - assert result["deleted_completions"] == 0 - assert result["deleted_watermarks"] == 0 - assert result["remaining_candidates"] is False - with closing(sqlite3.connect(str(db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - ORDER BY turn - """, - (host_id, pane_id), - ).fetchall() == [(1,), (2,), (3,)] def test_exhaust_connector_retries_reclaims_expired_leases_before_dead_letter(tmp_path: Path) -> None: @@ -2523,161 +1927,8 @@ def test_exhaust_connector_retries_reclaims_expired_leases_before_dead_letter(tm assert (attempt_count, max_attempt) == (2, 2) -def test_store_migrates_v1_schema_and_persists_content_fingerprint(tmp_path: Path) -> None: - db_path = tmp_path / "legacy.db" - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - """ - CREATE TABLE snapshots ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - created_at TEXT NOT NULL, - payload TEXT NOT NULL - ); - PRAGMA user_version = 1; - """ - ) - - init_store(db_path) - config = Config(host_id="storehost", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "Agent One", "status": "blocked"}], - ) - save_snapshot(db_path, snapshot) - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - row = conn.execute( - "SELECT host_id, content_fingerprint, payload FROM snapshots ORDER BY id DESC LIMIT 1" - ).fetchone() - assert row[0] == "storehost" - assert row[1] == snapshot.content_fingerprint - assert json.loads(row[2]) == json.loads(snapshot.to_json()) - restored = latest_snapshot(db_path) - assert restored is not None - assert restored.host_id == "storehost" - assert restored.content_fingerprint == snapshot.content_fingerprint - - -def test_store_migrates_partial_v3_db_with_legacy_data_idempotently(tmp_path: Path) -> None: - db_path = tmp_path / "partial-v3.db" - snapshot = project_empty(Config(host_id="legacy-host", db_path=db_path)) - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - """ - CREATE TABLE snapshots ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - created_at TEXT NOT NULL, - content_fingerprint TEXT NOT NULL DEFAULT '', - payload TEXT NOT NULL - ); - CREATE TABLE command_receipts ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - request_id TEXT NOT NULL, - action TEXT NOT NULL, - payload_fingerprint TEXT NOT NULL, - status TEXT NOT NULL, - result_json TEXT NOT NULL, - created_at TEXT NOT NULL, - completed_at TEXT, - uncertain INTEGER NOT NULL DEFAULT 0 - ); - CREATE TABLE worker_bindings ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - worker_id TEXT NOT NULL, - worker_fingerprint TEXT NOT NULL, - backend TEXT NOT NULL, - target_kind TEXT NOT NULL, - target_value TEXT NOT NULL, - turn_target_kind TEXT, - turn_target_value TEXT, - sendable INTEGER NOT NULL DEFAULT 0, - reason TEXT, - observed_at TEXT NOT NULL, - expires_at TEXT NOT NULL, - private_fingerprint TEXT NOT NULL - ); - PRAGMA user_version = 3; - """ - ) - conn.execute( - """ - INSERT INTO snapshots (host_id, created_at, content_fingerprint, payload) - VALUES (?, ?, ?, ?) - """, - ( - snapshot.host_id, - snapshot.updated_at, - snapshot.content_fingerprint, - snapshot.to_json(), - ), - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "legacy-host", - "legacy-req", - "send_instruction", - "legacy-fp", - STATUS_ACCEPTED, - '{"status":"accepted"}', - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - 0, - ), - ) - conn.execute( - """ - INSERT INTO worker_bindings ( - host_id, worker_id, worker_fingerprint, backend, target_kind, - target_value, sendable, reason, observed_at, expires_at, - private_fingerprint - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "legacy-host", - "worker-legacy", - "worker-fp", - "herdr", - "agent_id", - "agent-private", - 1, - None, - "2026-01-01T00:00:00+00:00", - "9999-12-31T23:59:59+00:00", - "legacy-private", - ), - ) - - init_store(db_path) - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _PR6_TABLES <= _table_names(conn) - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - assert conn.execute("SELECT COUNT(*) FROM snapshots").fetchone()[0] == 1 - assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 1 - assert conn.execute("SELECT COUNT(*) FROM worker_bindings").fetchone()[0] == 1 - command = conn.execute( - """ - SELECT status, canonical_fingerprint, result_json - FROM commands - WHERE host_id = 'legacy-host' - AND request_id = 'legacy-req' - """ - ).fetchone() - - assert command == (STATUS_ACCEPTED, "legacy-fp", '{"status":"accepted"}') def _snapshot_with_worker_status( @@ -2775,7 +2026,9 @@ def test_store_default_context_is_non_authoritative_and_snapshot_fallback_remain save_snapshot(db_path, snapshot) assert _lifecycle_rows(db_path) == [] - assert list_attention_items(db_path, "attention-host") == [] + assert read_test_attention_items(db_path, "attention-host")[0]["id"] == ( + snapshot.attention[0].id + ) payload = attention_payload_from_store(db_path, "attention-host") assert payload is not None assert payload["attention"][0]["id"] == snapshot.attention[0].id @@ -2845,8 +2098,8 @@ def test_store_resolution_requires_distinct_misses_and_120_seconds(tmp_path: Pat lifecycle = _lifecycle_rows(db_path)[0] assert lifecycle[3:5] == ("resolved", None) - assert list_attention_items(db_path, "attention-host") == [] - audit = list_attention_items( + assert read_test_attention_items(db_path, "attention-host") == [] + audit = read_test_attention_items( db_path, "attention-host", include_resolved=True ) assert audit[0]["resolved_reason"] == "gone" @@ -2894,7 +2147,7 @@ def test_store_none_authority_is_lifecycle_byte_equivalent( ) _save_observation(db_path, present, "complete", present.updated_at) before_lifecycle = _lifecycle_rows(db_path) - before_attention = list_attention_items( + before_attention = read_test_attention_items( db_path, "attention-host", include_resolved=True ) before_outbox = _connector_outbox_rows(db_path) @@ -2908,7 +2161,7 @@ def test_store_none_authority_is_lifecycle_byte_equivalent( _save_observation(db_path, degraded, "none", degraded.updated_at) assert _lifecycle_rows(db_path) == before_lifecycle - assert list_attention_items( + assert read_test_attention_items( db_path, "attention-host", include_resolved=True ) == before_attention assert _connector_outbox_rows(db_path) == before_outbox @@ -2991,10 +2244,10 @@ def test_store_escalation_downgrade_one_generation_one_current_pointer( _save_observation(db_path, snapshot, "positive", at) assert _lifecycle_rows(db_path)[0][2:4] == (1, "open") - current = list_attention_items(db_path, "attention-host") + current = read_test_attention_items(db_path, "attention-host") assert len(current) == 1 assert current[0]["status"] == "blocked" - audit = list_attention_items( + audit = read_test_attention_items( db_path, "attention-host", include_resolved=True ) assert len(audit) == 2 @@ -3149,7 +2402,7 @@ def test_store_same_family_variant_selection_is_order_independent(tmp_path: Path data["attention"] = list(reversed(variants)) if reverse else variants snapshot = store_sqlite.Snapshot.from_dict(data) _save_observation(db_path, snapshot, "complete", snapshot.updated_at) - current = list_attention_items(db_path, "attention-host") + current = read_test_attention_items(db_path, "attention-host") assert len(current) == 1 selected_ids.append(current[0]["id"]) assert selected_ids == ["critical-variant", "critical-variant"] @@ -3168,7 +2421,7 @@ def test_store_invalid_or_naive_lifecycle_time_is_noop( _save_observation(db_path, first, "complete", first.updated_at) before = ( _lifecycle_rows(db_path), - list_attention_items(db_path, "attention-host", include_resolved=True), + read_test_attention_items(db_path, "attention-host", include_resolved=True), _connector_outbox_rows(db_path), ) failed = _snapshot_with_worker_status( @@ -3177,7 +2430,7 @@ def test_store_invalid_or_naive_lifecycle_time_is_noop( _save_observation(db_path, failed, "positive", observed_at) after = ( _lifecycle_rows(db_path), - list_attention_items(db_path, "attention-host", include_resolved=True), + read_test_attention_items(db_path, "attention-host", include_resolved=True), _connector_outbox_rows(db_path), ) assert after == before @@ -3195,7 +2448,7 @@ def test_store_strict_order_preserves_subsecond_observations(tmp_path: Path) -> ) _save_observation(db_path, snapshot, "positive", at) lifecycle = _lifecycle_rows(db_path)[0] - current = list_attention_items(db_path, "attention-host")[0] + current = read_test_attention_items(db_path, "attention-host")[0] assert lifecycle[9] == "2026-01-01T00:00:00.000002+00:00" assert current["signal_count"] == 2 assert current["last_seen_at"] == lifecycle[9] @@ -3221,1398 +2474,204 @@ def test_store_deterministic_30_minute_flap_has_one_episode_and_initial( lifecycle = _lifecycle_rows(db_path)[0] assert lifecycle[2:4] == (1, "open") assert lifecycle[7:9] == (None, 0) - assert len(list_attention_items(db_path, "attention-host")) == 1 + assert len(read_test_attention_items(db_path, "attention-host")) == 1 assert len(_connector_outbox_rows(db_path)) == 1 -def _reset_store_to_v4(db_path: Path) -> None: - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - conn.execute("DROP TABLE attention_lifecycles") - conn.execute("PRAGMA user_version = 4") -def _insert_legacy_attention( - conn: sqlite3.Connection, + + + + +def _worker_binding( *, - attention_id: str, - source: str, - severity: str, - status: str, - lifecycle_status: str, - at: str, - first_seen_at: str | None = None, - signal_count: int = 1, - last_seen_at: str | None = None, - last_changed_at: str | None = None, -) -> None: - payload = { - "id": attention_id, - "source": source, - "kind": "worker_status", - "severity": severity, - "status": status, - "fingerprint": f"fp-{attention_id}", - } - # last_seen_at drives the migration's positive_at; last_changed_at drives its - # change/resolve progress. Allowing them to differ from `at` is what lets a - # test build a resolved episode whose resolution is newer than its last - # positive (the ordering the collapsed default hides). - seen_at = last_seen_at or at - changed_at = last_changed_at or at - resolved_at = (changed_at if lifecycle_status != "open" else None) - conn.execute( - """ - INSERT INTO attention_items ( - host_id, attention_id, source, kind, severity, status, updated_at, - fingerprint, snapshot_content_fingerprint, observed_at, - first_seen_at, last_seen_at, last_changed_at, resolved_at, - lifecycle_status, resolved_reason, signal_count, payload_json - ) VALUES ( - 'legacy-host', ?, ?, 'worker_status', ?, ?, ?, ?, 'snapshot-fp', ?, - ?, ?, ?, ?, ?, ?, ?, ? - ) - """, - ( - attention_id, - source, - severity, - status, - at, - f"fp-{attention_id}", - at, - first_seen_at or at, - seen_at, - changed_at, - resolved_at, - lifecycle_status, - "gone" if lifecycle_status != "open" else None, - signal_count, - json.dumps(payload, sort_keys=True), - ), + worker_id: str = "worker-1", + worker_fingerprint: str = "fp-1", + target_kind: str = "pane_id", + target_value: str = "pane-1", + private_fingerprint: str = "priv-1", + sendable: bool = True, + reason: str | None = None, + observed_at: str = "2026-01-01T00:00:00+00:00", + expires_at: str = "2026-01-02T00:00:00+00:00", +) -> WorkerBinding: + return WorkerBinding( + host_id="host-a", + worker_id=worker_id, + worker_fingerprint=worker_fingerprint, + backend="herdr", + target_kind=target_kind, + target_value=target_value, + turn_target_kind=None, + turn_target_value=None, + sendable=sendable, + reason=reason, + observed_at=observed_at, + expires_at=expires_at, + private_fingerprint=private_fingerprint, ) -def test_store_v4_collision_migration_is_deterministic_and_preserves_audit( - tmp_path: Path, -) -> None: - winners: list[tuple[Any, ...]] = [] - for reverse in (False, True): - db_path = tmp_path / f"collision-{reverse}.db" - _reset_store_to_v4(db_path) - candidates = [ - { - "attention_id": "attn-blocked", - "severity": "warning", - "status": "blocked", - "lifecycle_status": "open", - "at": "2026-01-01T00:01:00+00:00", - "first_seen_at": "2026-01-01T00:00:00+00:00", - "signal_count": 3, - }, - { - "attention_id": "attn-failed", - "severity": "critical", - "status": "failed", - "lifecycle_status": "open", - "at": "2026-01-01T00:02:00+00:00", - "first_seen_at": "2026-01-01T00:01:00+00:00", - "signal_count": 4, - }, - { - "attention_id": "attn-resolved", - "severity": "critical", - "status": "failed", - "lifecycle_status": "resolved", - "at": "2026-01-01T00:03:00+00:00", - "signal_count": 2, - }, - ] - with sqlite3.connect(str(db_path)) as conn: - for candidate in ( - reversed(candidates) if reverse else candidates - ): - _insert_legacy_attention( - conn, - source="worker:legacy", - **candidate, - ) - init_store(db_path) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - lifecycle = conn.execute( - """ - SELECT generation, lifecycle_status, current_attention_id, - first_seen_at, last_positive_at, - max_notified_severity_rank - FROM attention_lifecycles - """ - ).fetchone() - public_rows = conn.execute( - """ - SELECT attention_id, lifecycle_status, resolved_reason, - signal_count - FROM attention_items ORDER BY attention_id - """ - ).fetchall() - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - winners.append((lifecycle, public_rows)) +def test_store_worker_binding_upsert_list_resolve_and_expire(tmp_path: Path) -> None: + db_path = tmp_path / "bindings.db" + first = _worker_binding() + moved = _worker_binding( + target_value="pane-2", + observed_at="2026-01-01T00:10:00+00:00", + ) - assert winners[0] == winners[1] - lifecycle, public_rows = winners[0] - assert lifecycle == ( - 1, - "resolved", - None, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:03:00+00:00", - 2, + init_store(db_path) + assert upsert_worker_bindings(db_path, [first]) == 1 + assert upsert_worker_bindings(db_path, [moved]) == 1 + + current = list_worker_bindings( + db_path, + "host-a", + backend="herdr", + now="2026-01-01T00:30:00+00:00", ) - assert public_rows == [ - ("attn-blocked", "resolved", "superseded", 3), - ("attn-failed", "resolved", "superseded", 4), - ("attn-resolved", "resolved", "gone", 2), - ] + assert len(current) == 1 + assert current[0].target_value == "pane-2" + assert current[0].worker_id == "worker-1" + expired_count = expire_worker_bindings( + db_path, + "host-a", + backend="herdr", + private_fingerprints=["priv-1"], + now="2026-01-01T00:45:00+00:00", + reason="stale_target", + ) + assert expired_count == 1 + assert list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:46:00+00:00") == [] + history = list_worker_bindings( + db_path, + "host-a", + backend="herdr", + include_expired=True, + now="2026-01-01T00:46:00+00:00", + ) + assert len(history) == 1 + assert history[0].sendable is False + assert history[0].reason == "stale_target" -_MIG_T0 = "2026-01-01T00:00:00+00:00" -_MIG_T5 = "2026-01-01T00:05:00+00:00" -_MIG_T10 = "2026-01-01T00:10:00+00:00" -_MIG_T11 = "2026-01-01T00:11:00+00:00" +def test_store_worker_bindings_allow_duplicate_targets_and_expire_stale(tmp_path: Path) -> None: + db_path = tmp_path / "duplicate-bindings.db" + binding_a = _worker_binding( + worker_id="worker-a", + worker_fingerprint="fp-a", + private_fingerprint="priv-a", + target_value="same-pane", + sendable=False, + reason="duplicate_backend_target", + ) + binding_b = _worker_binding( + worker_id="worker-b", + worker_fingerprint="fp-b", + private_fingerprint="priv-b", + target_value="same-pane", + sendable=False, + reason="duplicate_backend_target", + ) + upsert_worker_bindings(db_path, [binding_a, binding_b]) + current = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:30:00+00:00") + assert len(current) == 2 + assert {binding.target_value for binding in current} == {"same-pane"} + assert {binding.reason for binding in current} == {"duplicate_backend_target"} -def _migrate_resolved_skewed_episode(db_path: Path) -> None: - """Legacy resolved episode whose resolution (t10) is newer than its last - positive (t0), migrated to v5.""" - _reset_store_to_v4(db_path) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-legacy", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="resolved", - at=_MIG_T0, - first_seen_at=_MIG_T0, - last_seen_at=_MIG_T0, - last_changed_at=_MIG_T10, - signal_count=2, - ) - init_store(db_path) + expired_count = expire_stale_worker_bindings( + db_path, + "host-a", + backend="herdr", + current_private_fingerprints=["priv-a"], + now="2026-01-01T00:40:00+00:00", + reason="stale_observation", + ) + assert expired_count == 1 + remaining = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:41:00+00:00") + assert [binding.private_fingerprint for binding in remaining] == ["priv-a"] -def _migrated_lifecycle_row(db_path: Path) -> tuple[Any, ...]: - with sqlite3.connect(str(db_path)) as conn: - return conn.execute( - """ - SELECT generation, lifecycle_status, current_attention_id, - last_positive_at, last_accepted_at - FROM attention_lifecycles - """ - ).fetchone() +def test_store_upsert_separates_colliding_duplicate_private_fingerprints(tmp_path: Path) -> None: + db_path = tmp_path / "colliding-bindings.db" + binding_a = _worker_binding( + worker_id="worker-a", + worker_fingerprint="fp-a", + target_value="same-agent", + private_fingerprint="colliding-private", + ) + binding_b = _worker_binding( + worker_id="worker-b", + worker_fingerprint="fp-b", + target_value="same-agent", + private_fingerprint="colliding-private", + ) + + assert upsert_worker_bindings(db_path, [binding_a, binding_b]) == 2 + + current = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:30:00+00:00") + assert len(current) == 2 + assert {binding.worker_id for binding in current} == {"worker-a", "worker-b"} + assert {binding.sendable for binding in current} == {False} + assert {binding.reason for binding in current} == {"duplicate_backend_target"} + assert "colliding-private" not in {binding.private_fingerprint for binding in current} + assert len({binding.private_fingerprint for binding in current}) == 2 + + +def test_store_snapshot_payload_does_not_contain_private_worker_bindings(tmp_path: Path) -> None: + db_path = tmp_path / "payload-clean.db" + config = Config(host_id="host-a", db_path=db_path) + snapshot = project_from_raw( + config, + workers=[{"id": "worker-1", "name": "Worker", "status": "active"}], + ) + binding = _worker_binding(target_value="pane-secret", private_fingerprint="priv-secret") + save_snapshot(db_path, snapshot) + upsert_worker_bindings(db_path, [binding]) -def _attention_outbox_job_count(db_path: Path) -> int: with sqlite3.connect(str(db_path)) as conn: - return int( - conn.execute( - "SELECT COUNT(*) FROM connector_outbox WHERE connector = 'attention'" - ).fetchone()[0] - ) + payload = conn.execute("SELECT payload FROM snapshots ORDER BY id DESC LIMIT 1").fetchone()[0] + target_value = conn.execute("SELECT target_value FROM worker_bindings LIMIT 1").fetchone()[0] + assert target_value == "pane-secret" + assert "pane-secret" not in payload + assert "priv-secret" not in payload + assert "target_kind" not in payload -def _legacy_worker_snapshot(status: str, timestamp: str): - from datetime import datetime - config = Config(host_id="legacy-host", db_path=Path("unused")) - return project_from_raw( - config, - workers=[{"id": "legacy", "name": "Legacy Worker", "status": status}], +def test_store_save_snapshot_updates_pr6_projections_and_prunes_by_host(tmp_path: Path) -> None: + db_path = tmp_path / "projections.db" + config_a = Config(host_id="host-a", db_path=db_path) + config_b = Config(host_id="host-b", db_path=db_path) + snapshot_a_old = project_from_raw( + config_a, + spaces=[{"id": "space-old", "name": "Old", "status": "active"}], + workers=[ + { + "id": "worker-old", + "name": "Old Worker", + "status": "active", + "space_id": "space-old", + "summary": "old", + } + ], backend_health=[ { "name": "herdr", "status": "healthy", "outcome": "healthy_non_empty", - "observed_at": timestamp, + "observed_at": "2026-01-01T00:00:00+00:00", "counts": {"workers": 1}, } ], - timestamp=datetime.fromisoformat(timestamp), - ) - - -def test_migration_resolved_episode_seeds_accepted_watermark_from_resolution( - tmp_path: Path, -) -> None: - """The migrated watermark is the resolution progress (t10), not the last - positive (t0), and a delayed positive at t5 cannot reopen the lifecycle.""" - db_path = tmp_path / "skewed-resolved.db" - _migrate_resolved_skewed_episode(db_path) - - assert _migrated_lifecycle_row(db_path) == ( - 1, - "resolved", - None, - _MIG_T0, # last_positive_at = actual latest positive - _MIG_T10, # last_accepted_at = newest lifecycle progress (resolution) ) - assert _attention_outbox_job_count(db_path) == 0 - - # A delayed positive observation timestamped t5 (< the authoritative t10 - # resolution) must be ignored: no reopen, no generation 2, no job. - save_snapshot( - db_path, - _legacy_worker_snapshot("blocked", _MIG_T5), - observation=SnapshotObservationContext(authority="positive", observed_at=_MIG_T5), - ) - assert _migrated_lifecycle_row(db_path) == (1, "resolved", None, _MIG_T0, _MIG_T10) - assert _attention_outbox_job_count(db_path) == 0 - - -def test_migration_resolved_episode_genuine_later_positive_opens_one_generation( - tmp_path: Path, -) -> None: - """A genuine positive after the resolution watermark opens generation 2 and - enqueues exactly one notification.""" - db_path = tmp_path / "genuine-reopen.db" - _migrate_resolved_skewed_episode(db_path) - assert _attention_outbox_job_count(db_path) == 0 - - save_snapshot( - db_path, - _legacy_worker_snapshot("blocked", _MIG_T11), - observation=SnapshotObservationContext(authority="positive", observed_at=_MIG_T11), - ) - generation, status, current, _positive, accepted = _migrated_lifecycle_row(db_path) - assert (generation, status) == (2, "open") - assert current is not None - assert accepted == _MIG_T11 - assert _attention_outbox_job_count(db_path) == 1 - - -def test_migration_resolved_episode_delayed_complete_miss_is_inert( - tmp_path: Path, -) -> None: - """A delayed 'complete' (missing) observation before the resolution watermark - cannot mutate the migrated resolved lifecycle.""" - db_path = tmp_path / "delayed-complete.db" - _migrate_resolved_skewed_episode(db_path) - - save_snapshot( - db_path, - _legacy_worker_snapshot("idle", _MIG_T5), - observation=SnapshotObservationContext(authority="complete", observed_at=_MIG_T5), - ) - assert _migrated_lifecycle_row(db_path) == (1, "resolved", None, _MIG_T0, _MIG_T10) - assert _attention_outbox_job_count(db_path) == 0 - - -def _legacy_attention_job_payload( - *, - event_type: str = "attention_created", - severity: str = "warning", - attention_id: str = "attn-legacy", - transition_at: str = "2026-01-01T00:00:00+00:00", -) -> str: - return json.dumps( - { - "schema_version": 1, - "event_type": event_type, - "host_id": "legacy-host", - "attention": { - "id": attention_id, - "source": "worker:legacy", - "kind": "worker_status", - "severity": severity, - "status": "blocked", - "fingerprint": "fp-attn-legacy", - }, - "transition_at": transition_at, - }, - sort_keys=True, - ) - - -def test_store_v4_migration_consolidates_duplicate_jobs_and_preserves_terminal_audit( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-jobs.db" - _reset_store_to_v4(db_path) - created_payload = _legacy_attention_job_payload() - escalation_payload = _legacy_attention_job_payload( - event_type="attention_escalated", severity="critical" - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-legacy", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:00:00+00:00", - ) - outbox_ids: dict[str, int] = {} - for index, status in enumerate( - ("queued", "retry", "deferred", "delivered", "dead_letter") - ): - cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, ?, ?, '{}', ?, ?, NULL) - """, - ( - f"legacy-created-{index}", - status, - created_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - outbox_ids[status] = int(cursor.lastrowid) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'legacy-created-3', 1, - 'delivered', '{}', '{}', ?, ?) - """, - ( - outbox_ids["delivered"], - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - leased_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'legacy-created-leased', - 'leased', ?, '{}', ?, ?, NULL) - """, - ( - created_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'legacy-created-leased', 1, - 'leased', '{}', '{}', ?, NULL) - """, - ( - int(leased_cursor.lastrowid), - "2026-01-01T00:00:00+00:00", - ), - ) - for index in range(8): - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, 'queued', ?, '{}', ?, ?, NULL) - """, - ( - f"legacy-escalation-{index}", - escalation_payload, - "2026-01-01T00:01:00+00:00", - "2026-01-01T00:01:00+00:00", - ), - ) - - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - created_statuses = conn.execute( - """ - SELECT status, private_state_json FROM connector_outbox - WHERE delivery_key LIKE 'legacy-created-%' - ORDER BY id - """ - ).fetchall() - escalation_statuses = conn.execute( - """ - SELECT status, delivery_key FROM connector_outbox - WHERE json_extract(payload_json, '$.event_type') = 'attention_escalated' - ORDER BY id - """ - ).fetchall() - delivered_count = conn.execute( - "SELECT COUNT(*) FROM connector_deliveries WHERE status = 'delivered'" - ).fetchone()[0] - assert created_statuses == [ - ("superseded", "{}"), - ("superseded", "{}"), - ("superseded", "{}"), - ("delivered", "{}"), - ("dead_letter", "{}"), - ( - "leased", - next( - private - for status, private in created_statuses - if status == "leased" - ), - ), - ] - leased_state = json.loads(created_statuses[-1][1]) - assert leased_state["migration_canonical"] is False - assert leased_state["terminal_after_lease"] is True - assert sum(status == "queued" for status, _ in escalation_statuses) == 1 - assert sum(status == "superseded" for status, _ in escalation_statuses) == 8 - assert delivered_count == 1 - - -def test_store_v4_delivered_old_does_not_suppress_active_current_recurrence( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-delivered-old.db" - _reset_store_to_v4(db_path) - old_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:00:00+00:00", - ) - current_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - first_seen_at="2026-01-01T00:00:00+00:00", - ) - delivered_ids: list[int] = [] - for key in ("old-delivered-audit", "old-delivered-outbox-only"): - cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, 'delivered', ?, '{}', - ?, ?, NULL) - """, - ( - key, - old_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - delivered_ids.append(int(cursor.lastrowid)) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'old-delivered-audit', 1, - 'delivered', '{}', '{}', ?, ?) - """, - ( - delivered_ids[0], - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'current-recurrence', - 'queued', ?, '{}', ?, ?, NULL) - """, - ( - current_payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - """ - SELECT delivery_key, status, payload_json - FROM connector_outbox ORDER BY id - """ - ).fetchall() - assert [row[1] for row in rows] == [ - "delivered", - "delivered", - "superseded", - "queued", - ] - assert json.loads(rows[-1][2])["attention"]["id"] == "attn-current" - assert rows[-1][0].startswith("attention:attention_created:") - - -@pytest.mark.parametrize("with_delivery_audit", [False, True]) -def test_store_v4_delivered_current_episode_suppresses_duplicate( - tmp_path: Path, - with_delivery_audit: bool, -) -> None: - db_path = tmp_path / f"migration-delivered-current-{with_delivery_audit}.db" - _reset_store_to_v4(db_path) - payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - ) - delivered_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'current-delivered', - 'delivered', ?, '{}', ?, ?, NULL) - """, - ( - payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - if with_delivery_audit: - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, - status, response_json, private_state_json, created_at, - delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'current-delivered', - 1, 'delivered', '{}', '{}', ?, ?) - """, - ( - int(delivered_cursor.lastrowid), - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'current-duplicate', - 'queued', ?, '{}', ?, ?, NULL) - """, - ( - payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - statuses = conn.execute( - "SELECT status FROM connector_outbox ORDER BY id" - ).fetchall() - assert statuses == [("delivered",), ("superseded",)] - - -@pytest.mark.parametrize( - ("dead_at", "expected_statuses"), - [ - ( - "2026-01-01T00:00:00+00:00", - [("dead_letter",), ("superseded",), ("queued",)], - ), - ( - "2026-01-01T00:10:00+00:00", - [("dead_letter",), ("superseded",)], - ), - ], -) -def test_store_v4_dead_letter_suppresses_only_proven_current_episode( - tmp_path: Path, - dead_at: str, - expected_statuses: list[tuple[str]], -) -> None: - db_path = tmp_path / f"migration-dead-{dead_at[14:19].replace(':', '-')}.db" - _reset_store_to_v4(db_path) - dead_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at=dead_at, - ) - current_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - first_seen_at="2026-01-01T00:00:00+00:00", - ) - for key, status, payload in ( - ("current-dead", "dead_letter", dead_payload), - ("current-active", "queued", current_payload), - ): - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, ?, ?, '{}', ?, ?, NULL) - """, - ( - key, - status, - payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - statuses = conn.execute( - "SELECT status FROM connector_outbox ORDER BY id" - ).fetchall() - assert statuses == expected_statuses - - -def test_store_v4_resolved_lifecycle_terminalizes_all_active_jobs( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-resolved-active.db" - _reset_store_to_v4(db_path) - payload = _legacy_attention_job_payload( - attention_id="attn-resolved", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-resolved", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="resolved", - at="2026-01-01T00:10:00+00:00", - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'resolved-active', 'queued', - ?, '{}', ?, ?, NULL) - """, - ( - payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - lifecycle_status = conn.execute( - "SELECT lifecycle_status FROM attention_lifecycles" - ).fetchone()[0] - outbox_status = conn.execute( - "SELECT status FROM connector_outbox" - ).fetchone()[0] - assert (lifecycle_status, outbox_status) == ("resolved", "superseded") - - - - -@pytest.mark.parametrize("terminal_action", ["fail", "defer", "expiry"]) -def test_store_v4_current_pollable_outranks_stale_live_lease( - tmp_path: Path, - terminal_action: str, -) -> None: - db_path = tmp_path / f"migration-stale-lease-{terminal_action}.db" - init_store(db_path) - old_payload = _legacy_attention_job_payload( - attention_id="attn-old", - transition_at="2026-01-01T00:00:00+00:00", - ) - current_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'old-leased', 'queued', - ?, '{}', ?, ?, NULL) - """, - ( - old_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - old_item = poll_connector_outbox( - db_path, - "legacy-host", - "attention", - lease_seconds=30, - now="2026-01-01T00:00:00+00:00", - )["items"][0] - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'current-queued', 'queued', - ?, '{}', ?, ?, NULL) - """, - ( - current_payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - conn.execute("DROP TABLE attention_lifecycles") - conn.execute("PRAGMA user_version = 4") - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - """ - SELECT delivery_key, status, private_state_json - FROM connector_outbox ORDER BY id - """ - ).fetchall() - assert [row[1] for row in rows] == ["leased", "superseded", "queued"] - stale_state = json.loads(rows[0][2]) - assert stale_state["migration_canonical"] is False - assert stale_state["terminal_after_lease"] is True - if terminal_action == "fail": - result = fail_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=old_item["ref"], - now="2026-01-01T00:00:10+00:00", - ) - assert result["status"] == "superseded" - elif terminal_action == "defer": - result = defer_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=old_item["ref"], - now="2026-01-01T00:00:10+00:00", - ) - assert result["status"] == "superseded" - else: - result = reclaim_expired_connector_leases( - db_path, - "legacy-host", - "attention", - now="2026-01-01T00:00:31+00:00", - ) - assert result["reclaimed"] == 1 - with sqlite3.connect(str(db_path)) as conn: - statuses = conn.execute( - "SELECT status, COUNT(*) FROM connector_outbox GROUP BY status" - ).fetchall() - assert dict(statuses) == {"queued": 1, "superseded": 2} - - -def test_store_v4_generated_flap_damage_migrates_bounded_and_idempotent( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-generated-flap.db" - _reset_store_to_v4(db_path) - current_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - old_payload = _legacy_attention_job_payload( - attention_id="attn-old", - transition_at="2026-01-01T00:00:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - for index in range(509): - _insert_legacy_attention( - conn, - attention_id=f"attn-flap-{index:04d}", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:00:00+00:00", - signal_count=2, - ) - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - signal_count=172, - ) - for index in range(600): - status = ("queued", "retry", "deferred")[index % 3] - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, ?, ?, '{}', ?, ?, NULL) - """, - ( - f"flap-active-{index:04d}", - status, - current_payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - delivered_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'flap-old-delivered', - 'delivered', ?, '{}', ?, ?, NULL) - """, - ( - old_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'flap-old-delivered', 1, - 'delivered', '{}', '{}', ?, ?) - """, - ( - int(delivered_cursor.lastrowid), - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'flap-old-dead', - 'dead_letter', ?, '{}', ?, ?, NULL) - """, - ( - old_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - - def migration_evidence() -> tuple[Any, ...]: - with sqlite3.connect(str(db_path)) as conn: - lifecycle = conn.execute( - """ - SELECT COUNT(*), lifecycle_status, current_attention_id - FROM attention_lifecycles - """ - ).fetchone() - attention = conn.execute( - """ - SELECT COUNT(*), - MAX(CASE WHEN lifecycle_status = 'open' - THEN signal_count ELSE 0 END) - FROM attention_items - """ - ).fetchone() - outbox = dict( - conn.execute( - "SELECT status, COUNT(*) FROM connector_outbox GROUP BY status" - ).fetchall() - ) - delivered_audit = conn.execute( - """ - SELECT COUNT(*) FROM connector_deliveries - WHERE status = 'delivered' - """ - ).fetchone()[0] - canonical_payload = conn.execute( - """ - SELECT payload_json FROM connector_outbox - WHERE status = 'queued' - """ - ).fetchall() - return ( - lifecycle, - attention, - outbox, - delivered_audit, - canonical_payload, - _user_version(conn), - ) - - init_store(db_path) - first = migration_evidence() - init_store(db_path) - second = migration_evidence() - with sqlite3.connect(str(db_path)) as conn: - integrity = conn.execute("PRAGMA integrity_check").fetchone()[0] - - assert first == second - lifecycle, attention, outbox, delivered_audit, canonical_payload, version = first - assert lifecycle == (1, "open", "attn-current") - assert attention == (510, 1190) - assert outbox == { - "dead_letter": 1, - "delivered": 1, - "queued": 1, - "superseded": 600, - } - assert delivered_audit == 1 - assert len(canonical_payload) == 1 - assert json.loads(canonical_payload[0][0])["attention"]["id"] == "attn-current" - assert version == store_sqlite.STORE_SCHEMA_VERSION - assert integrity == "ok" - - -def test_store_v4_migration_retains_single_and_multiple_live_leases_safely( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-leases.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-legacy", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:00:00+00:00", - ) - created_payload = _legacy_attention_job_payload() - escalation_payload = _legacy_attention_job_payload( - event_type="attention_escalated", severity="critical" - ) - for index in range(5): - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, 'queued', ?, '{}', ?, ?, NULL) - """, - ( - f"leased-created-{index}", - created_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'leased-escalation', 'queued', - ?, '{}', ?, ?, NULL) - """, - ( - escalation_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - leased = poll_connector_outbox( - db_path, - "legacy-host", - "attention", - limit=6, - lease_seconds=600, - now="2026-01-01T00:00:00+00:00", - )["items"] - assert len(leased) == 6 - with sqlite3.connect(str(db_path)) as conn: - conn.execute("DROP TABLE attention_lifecycles") - conn.execute("PRAGMA user_version = 4") - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - leased_rows = conn.execute( - """ - SELECT id, delivery_key, private_state_json - FROM connector_outbox WHERE status = 'leased' ORDER BY id - """ - ).fetchall() - pollable = conn.execute( - """ - SELECT COUNT(*) FROM connector_outbox - WHERE status IN ('queued', 'retry', 'deferred') - """ - ).fetchone()[0] - assert len(leased_rows) == 6 - assert pollable == 0 - created_rows = [row for row in leased_rows if "created" in row[1]] - created_states = [json.loads(row[2]) for row in created_rows] - assert sum(bool(state["migration_canonical"]) for state in created_states) == 1 - assert sum(bool(state.get("terminal_after_lease")) for state in created_states) == 4 - escalation_state = json.loads( - next(row[2] for row in leased_rows if row[1] == "leased-escalation") - ) - assert escalation_state["migration_canonical"] is True - assert "terminal_after_lease" not in escalation_state - - items_by_key = {item["key"]: item for item in leased} - canonical_created = next( - row for row in created_rows if json.loads(row[2])["migration_canonical"] - ) - duplicate_created = [ - row - for row in created_rows - if json.loads(row[2]).get("terminal_after_lease") - ] - canonical_ack = ack_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key[canonical_created[1]]["ref"], - now="2026-01-01T00:00:05+00:00", - ) - assert canonical_ack["status"] == "acknowledged" - with sqlite3.connect(str(db_path)) as conn: - sibling_states = [ - json.loads(row[0]) - for row in conn.execute( - """ - SELECT private_state_json FROM connector_outbox - WHERE status = 'leased' - AND delivery_key LIKE 'leased-created-%' - """ - ).fetchall() - ] - assert len(sibling_states) == 4 - assert all(state["terminal_after_lease"] for state in sibling_states) - - failed = fail_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key[duplicate_created[0][1]]["ref"], - now="2026-01-01T00:00:10+00:00", - ) - deferred = defer_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key[duplicate_created[1][1]]["ref"], - now="2026-01-01T00:00:20+00:00", - ) - assert failed["status"] == deferred["status"] == "superseded" - with sqlite3.connect(str(db_path)) as conn: - expiring = duplicate_created[2] - delivery = conn.execute( - """ - SELECT id, private_state_json FROM connector_deliveries - WHERE outbox_id = ? AND status = 'leased' - """, - (expiring[0],), - ).fetchone() - delivery_state = json.loads(delivery[1]) - delivery_state["lease_expires_at"] = "2026-01-01T00:00:30+00:00" - conn.execute( - "UPDATE connector_deliveries SET private_state_json = ? WHERE id = ?", - (json.dumps(delivery_state, sort_keys=True), delivery[0]), - ) - reclaimed = reclaim_expired_connector_leases( - db_path, - "legacy-host", - "attention", - now="2026-01-01T00:01:00+00:00", - ) - assert reclaimed["reclaimed"] == 1 - duplicate_ack = ack_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key[duplicate_created[3][1]]["ref"], - now="2026-01-01T00:01:30+00:00", - ) - assert duplicate_ack["status"] == "acknowledged" - single_ack = ack_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key["leased-escalation"]["ref"], - now="2026-01-01T00:02:00+00:00", - ) - assert single_ack["status"] == canonical_ack["status"] == "acknowledged" - with sqlite3.connect(str(db_path)) as conn: - statuses = conn.execute( - "SELECT status, COUNT(*) FROM connector_outbox GROUP BY status" - ).fetchall() - assert dict(statuses) == {"delivered": 3, "superseded": 3} - - -def _worker_binding( - *, - worker_id: str = "worker-1", - worker_fingerprint: str = "fp-1", - target_kind: str = "pane_id", - target_value: str = "pane-1", - private_fingerprint: str = "priv-1", - sendable: bool = True, - reason: str | None = None, - observed_at: str = "2026-01-01T00:00:00+00:00", - expires_at: str = "2026-01-02T00:00:00+00:00", -) -> WorkerBinding: - return WorkerBinding( - host_id="host-a", - worker_id=worker_id, - worker_fingerprint=worker_fingerprint, - backend="herdr", - target_kind=target_kind, - target_value=target_value, - turn_target_kind=None, - turn_target_value=None, - sendable=sendable, - reason=reason, - observed_at=observed_at, - expires_at=expires_at, - private_fingerprint=private_fingerprint, - ) - - -def test_store_worker_binding_upsert_list_resolve_and_expire(tmp_path: Path) -> None: - db_path = tmp_path / "bindings.db" - first = _worker_binding() - moved = _worker_binding( - target_value="pane-2", - observed_at="2026-01-01T00:10:00+00:00", - ) - - init_store(db_path) - assert upsert_worker_bindings(db_path, [first]) == 1 - assert upsert_worker_bindings(db_path, [moved]) == 1 - - current = list_worker_bindings( - db_path, - "host-a", - backend="herdr", - now="2026-01-01T00:30:00+00:00", - ) - assert len(current) == 1 - assert current[0].target_value == "pane-2" - assert current[0].worker_id == "worker-1" - resolved = resolve_worker_binding( - db_path, - "host-a", - "worker-1", - worker_fingerprint="fp-1", - backend="herdr", - now="2026-01-01T00:30:00+00:00", - ) - assert resolved is not None - assert resolved.target_value == "pane-2" - - expired_count = expire_worker_bindings( - db_path, - "host-a", - backend="herdr", - private_fingerprints=["priv-1"], - now="2026-01-01T00:45:00+00:00", - reason="stale_target", - ) - assert expired_count == 1 - assert list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:46:00+00:00") == [] - history = list_worker_bindings( - db_path, - "host-a", - backend="herdr", - include_expired=True, - now="2026-01-01T00:46:00+00:00", - ) - assert len(history) == 1 - assert history[0].sendable is False - assert history[0].reason == "stale_target" - assert resolve_worker_binding( - db_path, - "host-a", - "worker-1", - backend="herdr", - now="2026-01-01T00:46:00+00:00", - ) is None - - -def test_store_worker_bindings_allow_duplicate_targets_and_expire_stale(tmp_path: Path) -> None: - db_path = tmp_path / "duplicate-bindings.db" - binding_a = _worker_binding( - worker_id="worker-a", - worker_fingerprint="fp-a", - private_fingerprint="priv-a", - target_value="same-pane", - sendable=False, - reason="duplicate_backend_target", - ) - binding_b = _worker_binding( - worker_id="worker-b", - worker_fingerprint="fp-b", - private_fingerprint="priv-b", - target_value="same-pane", - sendable=False, - reason="duplicate_backend_target", - ) - upsert_worker_bindings(db_path, [binding_a, binding_b]) - - current = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:30:00+00:00") - assert len(current) == 2 - assert {binding.target_value for binding in current} == {"same-pane"} - assert {binding.reason for binding in current} == {"duplicate_backend_target"} - assert resolve_worker_binding( - db_path, - "host-a", - "worker-a", - backend="herdr", - now="2026-01-01T00:30:00+00:00", - ) is None - - expired_count = expire_stale_worker_bindings( - db_path, - "host-a", - backend="herdr", - current_private_fingerprints=["priv-a"], - now="2026-01-01T00:40:00+00:00", - reason="stale_observation", - ) - assert expired_count == 1 - remaining = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:41:00+00:00") - assert [binding.private_fingerprint for binding in remaining] == ["priv-a"] - - -def test_store_upsert_separates_colliding_duplicate_private_fingerprints(tmp_path: Path) -> None: - db_path = tmp_path / "colliding-bindings.db" - binding_a = _worker_binding( - worker_id="worker-a", - worker_fingerprint="fp-a", - target_value="same-agent", - private_fingerprint="colliding-private", - ) - binding_b = _worker_binding( - worker_id="worker-b", - worker_fingerprint="fp-b", - target_value="same-agent", - private_fingerprint="colliding-private", - ) - - assert upsert_worker_bindings(db_path, [binding_a, binding_b]) == 2 - - current = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:30:00+00:00") - assert len(current) == 2 - assert {binding.worker_id for binding in current} == {"worker-a", "worker-b"} - assert {binding.sendable for binding in current} == {False} - assert {binding.reason for binding in current} == {"duplicate_backend_target"} - assert "colliding-private" not in {binding.private_fingerprint for binding in current} - assert len({binding.private_fingerprint for binding in current}) == 2 - assert resolve_worker_binding( - db_path, - "host-a", - "worker-a", - backend="herdr", - now="2026-01-01T00:30:00+00:00", - ) is None - - -def test_store_snapshot_payload_does_not_contain_private_worker_bindings(tmp_path: Path) -> None: - db_path = tmp_path / "payload-clean.db" - config = Config(host_id="host-a", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "Worker", "status": "active"}], - ) - binding = _worker_binding(target_value="pane-secret", private_fingerprint="priv-secret") - - save_snapshot(db_path, snapshot) - upsert_worker_bindings(db_path, [binding]) - - with sqlite3.connect(str(db_path)) as conn: - payload = conn.execute("SELECT payload FROM snapshots ORDER BY id DESC LIMIT 1").fetchone()[0] - target_value = conn.execute("SELECT target_value FROM worker_bindings LIMIT 1").fetchone()[0] - - assert target_value == "pane-secret" - assert "pane-secret" not in payload - assert "priv-secret" not in payload - assert "target_kind" not in payload - - -def test_store_save_snapshot_updates_pr6_projections_and_prunes_by_host(tmp_path: Path) -> None: - db_path = tmp_path / "projections.db" - config_a = Config(host_id="host-a", db_path=db_path) - config_b = Config(host_id="host-b", db_path=db_path) - snapshot_a_old = project_from_raw( - config_a, - spaces=[{"id": "space-old", "name": "Old", "status": "active"}], - workers=[ - { - "id": "worker-old", - "name": "Old Worker", - "status": "active", - "space_id": "space-old", - "summary": "old", - } - ], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-01T00:00:00+00:00", - "counts": {"workers": 1}, - } - ], - ) - snapshot_b = project_from_raw( - config_b, - spaces=[{"id": "space-b", "name": "B", "status": "active"}], - workers=[{"id": "worker-b", "name": "Worker B", "status": "active"}], + snapshot_b = project_from_raw( + config_b, + spaces=[{"id": "space-b", "name": "B", "status": "active"}], + workers=[{"id": "worker-b", "name": "Worker B", "status": "active"}], ) snapshot_a_new = project_from_raw( config_a, @@ -4701,7 +2760,7 @@ def test_store_merges_public_turn_content_without_private_labels(tmp_path: Path) init_store(db_path) save_snapshot(db_path, snapshot) - updated = merge_turn_content( + updated = apply_test_turn_refresh( db_path, "turn-host", "worker-1", @@ -4731,7 +2790,7 @@ def test_store_merges_public_turn_content_without_private_labels(tmp_path: Path) -def test_store_save_latest_host_scope_and_list_hosts(tmp_path: Path) -> None: +def test_store_save_and_latest_snapshot_are_host_scoped(tmp_path: Path) -> None: db_path = tmp_path / "tendwire.db" config_a = Config(host_id="host-a", db_path=db_path) config_b = Config(host_id="host-b", db_path=db_path) @@ -4774,7 +2833,6 @@ def test_store_save_latest_host_scope_and_list_hosts(tmp_path: Path) -> None: assert restored_b.workers[0].id == "worker-b" assert latest_snapshot(db_path, "missing-host") is None - assert list_hosts(db_path) == ["host-a", "host-b"] def _reserve_test_request( @@ -4853,7 +2911,7 @@ def test_store_host_wide_request_identity_conflicts_across_actions_and_tombstone collision = _reserve_test_request( db_path, request_id="same-id", - action="answer_pending", + action="answer_decision", fingerprint="answer-fingerprint", ) assert collision["status"] == "request_id_conflict" @@ -5038,7 +3096,7 @@ def test_backend_pending_choice_terminal_effect_is_atomic_with_acceptance( reservation = _reserve_test_request( db_path, request_id="choice-effect", - action="answer_pending", + action="answer_decision", fingerprint="choice-effect-fingerprint", ) started = mark_command_send_started( @@ -5112,7 +3170,7 @@ def test_backend_pending_choice_terminal_effect_is_atomic_with_acceptance( missing_reservation = _reserve_test_request( db_path, request_id="missing-choice", - action="answer_pending", + action="answer_decision", fingerprint="missing-choice-fingerprint", now="2026-01-01T00:01:00+00:00", ) @@ -5520,256 +3578,19 @@ def attempt() -> None: assert not any(thread.is_alive() for thread in threads) assert sorted(result["status"] for result in results) == [ - "in_progress", - "reserved", - ] - assert sum(result["owner_token"] is not None for result in results) == 1 - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 1 - assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 1 - - -def test_store_v11_host_request_collision_migrates_to_uncertain_tombstone( - tmp_path: Path, -) -> None: - db_path = tmp_path / "legacy-collision.db" - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - for index, (action, fingerprint) in enumerate( - ( - ("send_instruction", "send-fingerprint"), - ("answer_pending", "answer-fingerprint"), - ) - ): - created = f"2026-01-01T00:00:0{index}+00:00" - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES (?, 'collision', ?, ?, 'accepted', ?, ?, ?, 0) - """, - ( - "host-a", - action, - fingerprint, - '{"ok":true,"private":"must-not-survive"}', - created, - created, - ), - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES (?, 'collision', ?, ?, 'accepted', 0, 0, ?, ?, ?, ?, ?, ?) - """, - ( - "host-a", - action, - fingerprint, - '{"target":{"worker_id":"public","worker_fingerprint":"private"}}', - '{"ok":true,"private":"must-not-survive"}', - created, - created, - created, - created, - ), - ) - conn.execute("PRAGMA user_version = 11") - - init_store(db_path) - receipt = get_command_request(db_path, "host-a", "collision") - assert receipt is not None - assert receipt["state"] == "uncertain" - assert receipt["status"] == "request_state_uncertain" - assert receipt["legacy_collision"] is True - assert receipt["legacy_collision_count"] == 4 - assert receipt["canonical_request_json"] == "{}" - assert "must-not-survive" not in receipt["result_json"] - assert _reserve_test_request( - db_path, - request_id="collision", - fingerprint="new-fingerprint", - )["status"] == "terminal" - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - "SELECT state, legacy_collision FROM commands" - ).fetchone() == ("uncertain", 1) - assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" - conn.execute(store_sqlite.CREATE_EVENTS_TABLE) - - _accept_test_request( - db_path, - request_id="newer-than-collision", - now="2026-02-02T00:00:00+00:00", - ) - cleanup = cleanup_command_request_retention( - db_path, - retry_horizon_seconds=604_800, - retention_seconds=2_592_000, - retention_count=1, - now="2026-03-05T00:00:00+00:00", - ) - assert cleanup["deleted"] == 1 - assert get_command_request(db_path, "host-a", "collision") is None - assert get_command_request( - db_path, "host-a", "newer-than-collision" - ) is not None - - - -def test_store_v11_noncollision_replays_only_exact_legacy_raw_fingerprint( - tmp_path: Path, -) -> None: - db_path = tmp_path / "legacy-noncollision.db" - legacy_fingerprint = "legacy-raw-payload-fingerprint" - canonical_json = '{"action":"send_instruction","worker_id":"worker-public"}' - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES ( - 'host-a', 'legacy-exact', 'send_instruction', ?, 'accepted', - '{"ok":true,"status":"accepted"}', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', 0 - ) - """, - (legacy_fingerprint,), - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES ( - 'host-a', 'legacy-exact', 'send_instruction', ?, 'accepted', - 0, 0, '{"target":{"worker_id":"worker-public"}}', - '{"ok":true,"status":"accepted"}', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', - '2026-01-01T00:00:01+00:00' - ) - """, - (legacy_fingerprint,), - ) - conn.execute("PRAGMA user_version = 11") - - init_store(db_path) - canonical_only = reserve_command_request( - db_path, - host_id="host-a", - request_id="legacy-exact", - action="send_instruction", - canonical_version=1, - canonical_fingerprint=legacy_fingerprint, - canonical_request_json=canonical_json, - public_worker_id="worker-public", - pending_result_json='{"ok":false,"status":"pending"}', - ) - assert canonical_only["status"] == "request_id_conflict" - - exact_replay = reserve_command_request( - db_path, - host_id="host-a", - request_id="legacy-exact", - action="send_instruction", - canonical_version=1, - canonical_fingerprint="new-canonical-fingerprint", - canonical_request_json=canonical_json, - public_worker_id="worker-public", - pending_result_json='{"ok":false,"status":"pending"}', - legacy_raw_payload_fingerprint=legacy_fingerprint, - ) - assert exact_replay["status"] == "terminal" - assert exact_replay["receipt"]["canonical_version"] == 0 - assert exact_replay["receipt"]["canonical_fingerprint"] == legacy_fingerprint - assert exact_replay["receipt"]["result_json"] == ( - '{"ok":true,"status":"accepted"}' - ) - - wrong_raw = reserve_command_request( - db_path, - host_id="host-a", - request_id="legacy-exact", - action="send_instruction", - canonical_version=1, - canonical_fingerprint="new-canonical-fingerprint", - canonical_request_json=canonical_json, - public_worker_id="worker-public", - pending_result_json='{"ok":false,"status":"pending"}', - legacy_raw_payload_fingerprint="different-legacy-raw-fingerprint", - ) - wrong_action = reserve_command_request( - db_path, - host_id="host-a", - request_id="legacy-exact", - action="answer_pending", - canonical_version=1, - canonical_fingerprint="new-canonical-fingerprint", - canonical_request_json=canonical_json, - public_worker_id="worker-public", - pending_result_json='{"ok":false,"status":"pending"}', - legacy_raw_payload_fingerprint=legacy_fingerprint, - ) - assert wrong_raw["status"] == "request_id_conflict" - assert wrong_action["status"] == "request_id_conflict" - -def test_store_v11_receipt_audit_disagreement_fails_closed(tmp_path: Path) -> None: - db_path = tmp_path / "legacy-disagreement.db" - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES ( - 'host-a', 'disagree', 'send_instruction', 'same-fingerprint', - 'accepted', '{"ok":true}', '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', 0 - ) - """ - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES ( - 'host-a', 'disagree', 'send_instruction', 'same-fingerprint', - 'backend_failed', 0, 0, '{}', '{"ok":false}', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', - '2026-01-01T00:00:01+00:00' - ) - """ - ) - conn.execute("PRAGMA user_version = 11") - init_store(db_path) - receipt = get_command_request(db_path, "host-a", "disagree") - assert receipt is not None - assert receipt["state"] == "uncertain" - assert receipt["legacy_collision"] is True + "in_progress", + "reserved", + ] + assert sum(result["owner_token"] is not None for result in results) == 1 + with sqlite3.connect(str(db_path)) as conn: + assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 1 + assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 1 + + + + + + def test_store_command_retention_obeys_age_count_host_and_batch_floors( @@ -6328,7 +4149,7 @@ def test_distinct_source_turns_mint_distinct_public_turn_ids(tmp_path: Path) -> [("first question", "first answer"), ("second question", "second answer")], start=1, ): - merge_turn_content( + apply_test_turn_refresh( db_path, "turn-host", "worker-1", @@ -6352,7 +4173,7 @@ def test_distinct_source_turns_mint_distinct_public_turn_ids(tmp_path: Path) -> assert all(not t.get("assistant_final_text") and not t.get("user_text") for t in base_rows) # Same source turn observed again updates its row, keeping the id stable. - merge_turn_content( + apply_test_turn_refresh( db_path, "turn-host", "worker-1", @@ -6413,481 +4234,103 @@ def test_twenty_offline_source_finals_are_retained_as_unique_ready_anchors( "space_id": "space-1", "meta": { "stable_key": "wsk1_" + ("2" * 64), - "stable_key_version": 1, - }, - } - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - for index in range(20): - assert merge_turn_content( - db_path, - "turn-host", - "worker-1", - { - "assistant_final_text": f"answer {index}", - "complete": True, - "source_turn_id": f"uuid-{index}", - }, - observed_at=f"2026-01-01T00:{index:02d}:00+00:00", - ) == 1 - - payload = turns_payload_from_store(db_path, "turn-host", snapshot=snapshot) - source_rows = [turn for turn in payload["turns"] if turn.get("source_turn_id")] - with sqlite3.connect(str(db_path)) as conn: - anchors = conn.execute( - """ - SELECT delivery_key, payload_json - FROM connector_outbox - WHERE host_id = ? - AND delivery_kind = 'final_ready' - AND status = 'queued' - ORDER BY id - """, - ("turn-host",), - ).fetchall() - - assert len(source_rows) == 20 - assert len(anchors) == len({row[0] for row in anchors}) == 20 - assert source_rows[0]["assistant_final_text"] == "answer 19" - assert all( - row[0].startswith("turn-final:revision:twfinal1.") - for row in anchors - ) - encoded_anchors = "\n".join(row[1] for row in anchors) - assert "answer 0" not in encoded_anchors - assert "answer 19" not in encoded_anchors - assert "source_turn_id" not in encoded_anchors - - - - - - - - - - - - - - - - - -def _reset_store_to_v5_with_legacy_turn( - db_path: Path, - *, - final_text: str, -) -> tuple[Any, str]: - config = Config(host_id="legacy-turn-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-1", - "name": "claude", - "status": "active", - "space_id": "space-1", - } - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - assert merge_turn_content( - db_path, - "legacy-turn-host", - "worker-1", - { - "source_turn_id": "v5-migration-source", - "user_text": "legacy prompt", - "assistant_final_text": final_text, - "complete": True, - "has_open_turn": False, - }, - ) == 1 - with sqlite3.connect(str(db_path)) as conn: - turn_id, payload_json = conn.execute( - """ - SELECT turn_id, payload_json - FROM turns - WHERE host_id = 'legacy-turn-host' - """ - ).fetchone() - payload = json.loads(payload_json) - payload["user_text"] = "legacy prompt" - payload["assistant_final_text"] = final_text - payload["complete"] = True - payload["has_open_turn"] = False - conn.execute( - "UPDATE turns SET payload_json = ? WHERE host_id = ? AND turn_id = ?", - ( - json.dumps(payload, sort_keys=True), - "legacy-turn-host", - str(turn_id), - ), - ) - conn.execute("DROP TABLE turn_presentation_recoveries") - conn.execute("DROP TABLE turn_presentation_jobs") - conn.execute("DROP TABLE turn_presentation_plans") - conn.execute("DROP TABLE turn_content_page_boundaries") - conn.execute("DROP TABLE turn_content_revisions") - conn.execute("CREATE TABLE preserved_v5 (value TEXT NOT NULL)") - conn.execute("INSERT INTO preserved_v5 (value) VALUES ('untouched')") - conn.execute("PRAGMA user_version = 5") - return snapshot, str(turn_id) - - -def _reconstruct_turn_content( - db_path: Path, - *, - host_id: str, - turn_id: str, - revision: str, - field: str, - work_counters: store_sqlite.TurnContentWorkCounters | None = None, -) -> tuple[str, list[dict[str, Any]]]: - cursor: str | None = None - pages: list[dict[str, Any]] = [] - while True: - page = store_sqlite.get_turn_content( - db_path, - host_id, - turn_id=turn_id, - content_revision=revision, - field=field, - cursor=cursor, - work_counters=work_counters, - ) - assert page.get("status") is None - pages.append(page) - cursor = page["next_cursor"] - if cursor is None: - break - return "".join(str(page["text"]) for page in pages), pages - - -def test_store_v5_to_v6_migration_is_atomic_idempotent_and_marks_incomplete( - tmp_path: Path, -) -> None: - db_path = tmp_path / "turn-v5.db" - fragment = ("x" * 11_988) + "\n[truncated]" - snapshot, turn_id = _reset_store_to_v5_with_legacy_turn( - db_path, - final_text=fragment, - ) - - init_store(db_path) - first_v2 = turns_payload_from_store( - db_path, - "legacy-turn-host", - snapshot=snapshot, - schema_version=2, - ) - init_store(db_path) - second_v2 = turns_payload_from_store( - db_path, - "legacy-turn-host", - snapshot=snapshot, - schema_version=2, - ) - - with sqlite3.connect(str(db_path)) as conn: - version = _user_version(conn) - tables = _table_names(conn) - revision_rows = conn.execute( - """ - SELECT - content_revision, user_text, assistant_final_text, - user_state, final_state, user_page_count, final_page_count, - is_current - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? - """, - ("legacy-turn-host", turn_id), - ).fetchall() - stored_payload = conn.execute( - "SELECT payload_json FROM turns WHERE host_id = ? AND turn_id = ?", - ("legacy-turn-host", turn_id), - ).fetchone()[0] - preserved = conn.execute("SELECT value FROM preserved_v5").fetchone()[0] - integrity = conn.execute("PRAGMA integrity_check").fetchone()[0] - foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - - assert version == store_sqlite.STORE_SCHEMA_VERSION - assert { - "turn_content_revisions", - "turn_presentation_plans", - "turn_presentation_jobs", - } <= tables - assert len(revision_rows) == 1 - revision = revision_rows[0] - assert revision[1:] == ( - "legacy prompt", - fragment, - "complete", - "known_incomplete", - 1, - 0, - 1, - ) - assert json.loads(stored_payload).get("assistant_final_text") is None - assert fragment not in stored_payload - assert preserved == "untouched" - assert integrity == "ok" - assert foreign_keys == [] - assert first_v2 == second_v2 - turn = next(item for item in first_v2["turns"] if item["id"] == turn_id) - assert turn["content"]["known_incomplete"] is True - assert turn["content"]["fields"]["assistant_final_text"] == { - "availability": "known_incomplete", - "inline": False, - "char_length": len(fragment), - "byte_length": len(fragment.encode("utf-8")), - "page_count": 0, - "first_cursor": None, - } - assert "assistant_final_text" not in turn - assert turn["assistant_final_preview"] == fragment[:1000] - assert turns_payload_from_store( - db_path, - "legacy-turn-host", - schema_version=1, - ) == { - "schema_version": 1, - "ok": False, - "status": "upgrade_required", - "required_turn_schema_version": 2, - } - assert store_sqlite.get_turn_content( - db_path, - "legacy-turn-host", - turn_id=turn_id, - content_revision=revision[0], - field="assistant_final_text", - ) == { - "schema_version": 1, - "ok": False, - "status": "content_known_incomplete", - } - -def test_store_v5_to_v6_migration_rolls_back_all_v6_ddl( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "turn-v5-rollback.db" - _reset_store_to_v5_with_legacy_turn(db_path, final_text="legacy answer") - - def fail_backfill(conn: sqlite3.Connection) -> None: - raise RuntimeError("controlled v6 migration failure") - - monkeypatch.setattr( - store_sqlite, - "_backfill_legacy_turn_content_conn", - fail_backfill, - ) - with pytest.raises(RuntimeError, match="controlled v6 migration failure"): - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == 5 - assert "turn_content_revisions" not in _table_names(conn) - assert "turn_content_page_boundaries" not in _table_names(conn) - assert "turn_presentation_plans" not in _table_names(conn) - assert "turn_presentation_jobs" not in _table_names(conn) - assert "turn_presentation_recoveries" not in _table_names(conn) - assert conn.execute("SELECT value FROM preserved_v5").fetchone()[0] == "untouched" - - -def test_v6_to_v7_repairs_mixed_turns_with_absent_content_descriptors( - tmp_path: Path, -) -> None: - db_path = tmp_path / "mixed-v6-turn-content.db" - host_id = "mixed-v6-host" - snapshot = project_from_raw( - Config(host_id=host_id, db_path=db_path), - workers=[ - {"id": "worker-complete", "name": "Complete", "status": "active"}, - {"id": "worker-working", "name": "Working", "status": "active"}, - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - assert merge_turn_content( - db_path, - host_id, - "worker-complete", - { - "source_turn_id": "complete-source", - "assistant_final_text": "complete final", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:00+00:00", - ) == 1 - assert merge_turn_content( - db_path, - host_id, - "worker-working", - { - "source_turn_id": "working-source", - "user_text": "observed working turn", - "complete": False, - "has_open_turn": True, - }, - observed_at="2026-01-01T00:01:00+00:00", - ) == 1 - - with sqlite3.connect(str(db_path)) as conn: - complete_turn_id = str( - conn.execute( - """ - SELECT turn_id - FROM turn_content_revisions - WHERE host_id = ? AND final_state = 'complete' AND is_current = 1 - """, - (host_id,), - ).fetchone()[0] - ) - missing_rows = conn.execute( - """ - SELECT turn_id, payload_json - FROM turns - WHERE host_id = ? AND turn_id != ? - ORDER BY turn_id - """, - (host_id, complete_turn_id), - ).fetchall() - assert len(missing_rows) == 1 - missing_turn_ids = [str(row[0]) for row in missing_rows] - for turn_id, payload_json in missing_rows: - payload = json.loads(payload_json) - payload["assistant_stream_text"] = "working progress" - conn.execute( - """ - UPDATE turns - SET payload_json = ? - WHERE host_id = ? AND turn_id = ? - """, - (json.dumps(payload, sort_keys=True), host_id, str(turn_id)), - ) - placeholders = ",".join("?" for _ in missing_turn_ids) - conn.execute( - f""" - DELETE FROM turn_content_page_boundaries - WHERE host_id = ? AND turn_id IN ({placeholders}) - """, - (host_id, *missing_turn_ids), - ) - conn.execute( - f""" - DELETE FROM turn_content_revisions - WHERE host_id = ? AND turn_id IN ({placeholders}) - """, - (host_id, *missing_turn_ids), - ) - conn.execute("DROP TABLE turn_content_page_boundaries") - conn.execute("PRAGMA user_version = 6") - - init_store(db_path) - first_v2 = turns_payload_from_store( - db_path, - host_id, - snapshot=snapshot, - schema_version=2, - ) - first_v1 = turns_payload_from_store( - db_path, - host_id, - snapshot=snapshot, - schema_version=1, + "stable_key_version": 1, + }, + } + ], ) - with sqlite3.connect(str(db_path)) as conn: - first_rows = conn.execute( - """ - SELECT turn_id, content_revision, user_state, final_state, is_current - FROM turn_content_revisions - WHERE host_id = ? AND turn_id IN ( - SELECT turn_id - FROM turns - WHERE host_id = ? AND turn_id != ? - ) - ORDER BY turn_id, content_revision - """, - (host_id, host_id, complete_turn_id), - ).fetchall() - version = conn.execute("PRAGMA user_version").fetchone()[0] init_store(db_path) - second_v2 = turns_payload_from_store( - db_path, - host_id, - snapshot=snapshot, - schema_version=2, - ) + save_snapshot(db_path, snapshot) + for index in range(20): + assert apply_test_turn_refresh( + db_path, + "turn-host", + "worker-1", + { + "assistant_final_text": f"answer {index}", + "complete": True, + "source_turn_id": f"uuid-{index}", + }, + observed_at=f"2026-01-01T00:{index:02d}:00+00:00", + ) == 1 + + payload = turns_payload_from_store(db_path, "turn-host", snapshot=snapshot) + source_rows = [turn for turn in payload["turns"] if turn.get("source_turn_id")] with sqlite3.connect(str(db_path)) as conn: - second_rows = conn.execute( + anchors = conn.execute( """ - SELECT turn_id, content_revision, user_state, final_state, is_current - FROM turn_content_revisions - WHERE host_id = ? AND turn_id IN ( - SELECT turn_id - FROM turns - WHERE host_id = ? AND turn_id != ? - ) - ORDER BY turn_id, content_revision + SELECT delivery_key, payload_json + FROM connector_outbox + WHERE host_id = ? + AND delivery_kind = 'final_ready' + AND status = 'queued' + ORDER BY id """, - (host_id, host_id, complete_turn_id), + ("turn-host",), ).fetchall() - absent_field = { - "availability": "absent", - "inline": False, - "char_length": 0, - "byte_length": 0, - "page_count": 0, - "first_cursor": None, - } - assert version == store_sqlite.STORE_SCHEMA_VERSION - assert first_v2 == second_v2 - assert first_rows == second_rows - assert len(first_rows) == len(missing_turn_ids) + assert len(source_rows) == 20 + assert len(anchors) == len({row[0] for row in anchors}) == 20 + assert source_rows[0]["assistant_final_text"] == "answer 19" assert all( - row - == ( - row[0], - store_sqlite.content_revision( - str(row[0]), - None, - None, - "absent", - "absent", - ), - "absent", - "absent", - 1, - ) - for row in first_rows + row[0].startswith("turn-final:revision:twfinal1.") + for row in anchors ) - assert len(first_v2["turns"]) >= 2 - for turn in first_v2["turns"]: - assert turn["content"]["schema_version"] == 1 - if turn["id"] not in missing_turn_ids: - continue - assert turn["assistant_stream_text"] == "working progress" - assert turn["content"]["known_incomplete"] is False - assert turn["content"]["fields"] == { - "user_text": absent_field, - "assistant_final_text": absent_field, - } - assert "user_text" not in turn - assert "assistant_final_text" not in turn - assert first_v1["schema_version"] == 1 - assert all("content" not in turn for turn in first_v1["turns"]) - for turn in first_v1["turns"]: - if turn["id"] in missing_turn_ids: - assert turn["user_text"] is None - assert turn["assistant_final_text"] is None + encoded_anchors = "\n".join(row[1] for row in anchors) + assert "answer 0" not in encoded_anchors + assert "answer 19" not in encoded_anchors + assert "source_turn_id" not in encoded_anchors + + + + + + + + + + + + + + + + + + + +def _reconstruct_turn_content( + db_path: Path, + *, + host_id: str, + turn_id: str, + revision: str, + field: str, + work_counters: store_sqlite.TurnContentWorkCounters | None = None, +) -> tuple[str, list[dict[str, Any]]]: + cursor: str | None = None + pages: list[dict[str, Any]] = [] + while True: + page = store_sqlite.get_turn_content( + db_path, + host_id, + turn_id=turn_id, + content_revision=revision, + field=field, + cursor=cursor, + work_counters=work_counters, + ) + assert page.get("status") is None + pages.append(page) + cursor = page["next_cursor"] + if cursor is None: + break + return "".join(str(page["text"]) for page in pages), pages + + + + + def _exact_utf8_fixture(byte_length: int) -> str: @@ -6917,7 +4360,7 @@ def test_store_list_is_preview_bounded_and_sequential_pages_are_linear( save_snapshot(db_path, snapshot) final = _exact_utf8_fixture(target_byte_length) assert len(final.encode("utf-8")) == target_byte_length - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -6989,153 +4432,6 @@ def test_store_list_is_preview_bounded_and_sequential_pages_are_linear( ) -def test_migrated_v6_boundaries_make_first_long_read_page_bounded( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migrated-v6-page-boundaries.db" - host_id = "legacy-boundary-host" - worker_id = "worker-boundary" - snapshot = project_from_raw( - Config(host_id=host_id, db_path=db_path), - workers=[{"id": worker_id, "name": "Boundary", "status": "active"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - final = _exact_utf8_fixture(1024 * 1024) - assert merge_turn_content( - db_path, - host_id, - worker_id, - { - "source_turn_id": "migrated-boundary-source", - "assistant_final_text": final, - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:00+00:00", - ) == 1 - listed = turns_payload_from_store( - db_path, - host_id, - snapshot=snapshot, - schema_version=2, - ) - turn = listed["turns"][0] - revision = turn["content"]["content_revision"] - page_count = turn["content"]["fields"]["assistant_final_text"]["page_count"] - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - DELETE FROM turn_content_page_boundaries - WHERE host_id = ? - AND turn_id = ? - AND content_revision = ? - AND field = 'assistant_final_text' - """, - (host_id, turn["id"], revision), - ) - conn.execute("PRAGMA user_version = 6") - - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - migrated_version = conn.execute("PRAGMA user_version").fetchone()[0] - migrated_boundaries = conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_page_boundaries - WHERE host_id = ? - AND turn_id = ? - AND content_revision = ? - AND field = 'assistant_final_text' - """, - (host_id, turn["id"], revision), - ).fetchone()[0] - first_counters = store_sqlite.TurnContentWorkCounters() - first_rebuilt, first_pages = _reconstruct_turn_content( - db_path, - host_id=host_id, - turn_id=turn["id"], - revision=revision, - field="assistant_final_text", - work_counters=first_counters, - ) - - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - DELETE FROM turn_content_page_boundaries - WHERE host_id = ? - AND turn_id = ? - AND content_revision = ? - AND field = 'assistant_final_text' - """, - (host_id, turn["id"], revision), - ) - init_store(db_path) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - current_boundaries = conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_page_boundaries - WHERE host_id = ? - AND turn_id = ? - AND content_revision = ? - AND field = 'assistant_final_text' - """, - (host_id, turn["id"], revision), - ).fetchone()[0] - incomplete_boundary_fields = conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_revisions AS revisions - WHERE ( - revisions.user_state = 'complete' - AND revisions.user_page_count != ( - SELECT COUNT(*) - FROM turn_content_page_boundaries AS boundaries - WHERE boundaries.host_id = revisions.host_id - AND boundaries.turn_id = revisions.turn_id - AND boundaries.content_revision = revisions.content_revision - AND boundaries.field = 'user_text' - ) - ) OR ( - revisions.final_state = 'complete' - AND revisions.final_page_count != ( - SELECT COUNT(*) - FROM turn_content_page_boundaries AS boundaries - WHERE boundaries.host_id = revisions.host_id - AND boundaries.turn_id = revisions.turn_id - AND boundaries.content_revision = revisions.content_revision - AND boundaries.field = 'assistant_final_text' - ) - ) - """ - ).fetchone()[0] - failed_page = store_sqlite.get_turn_content( - db_path, - host_id, - turn_id=turn["id"], - content_revision=revision, - field="assistant_final_text", - ) - - assert migrated_version == store_sqlite.STORE_SCHEMA_VERSION - assert migrated_boundaries == page_count - assert current_boundaries == 0 - assert incomplete_boundary_fields == 1 - assert first_rebuilt == final - assert len(first_pages) == page_count - assert first_counters.page_blob_reads == page_count - assert first_counters.page_chars_examined == len(final) - assert first_counters.page_bytes_examined <= ( - len(final.encode("utf-8")) + 3 * (page_count - 1) - ) - assert failed_page == { - "schema_version": 1, - "ok": False, - "status": "content_not_available", - } def test_many_long_turn_descriptors_do_one_bounded_list_query(tmp_path: Path) -> None: @@ -7152,7 +4448,7 @@ def test_many_long_turn_descriptors_do_one_bounded_list_query(tmp_path: Path) -> init_store(db_path) save_snapshot(db_path, snapshot) for index, worker_id in enumerate(worker_ids): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7214,7 +4510,7 @@ def test_store_canonical_pages_round_trip_long_content_without_duplicate_copy( final = "\n# Heading\n\n```\n" + ("🙂" * 270_000) + "\n```\n" assert len(final.encode("utf-8")) > 1024 * 1024 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "long-host", "worker-1", @@ -7361,7 +4657,7 @@ def test_store_canonical_pages_round_trip_long_content_without_duplicate_copy( assert final[:1000] not in payload_json assert revision_count == 1 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "long-host", "worker-1", @@ -7372,7 +4668,7 @@ def test_store_canonical_pages_round_trip_long_content_without_duplicate_copy( }, observed_at="2026-01-01T00:01:00+00:00", ) == 0 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "long-host", "worker-1", @@ -7404,7 +4700,7 @@ def test_store_merge_distinguishes_whitespace_content_from_empty_or_absent_field init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7416,14 +4712,14 @@ def test_store_merge_distinguishes_whitespace_content_from_empty_or_absent_field }, observed_at="2026-01-01T00:00:00+00:00", ) == 1 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, {"source_turn_id": "merge-precedence-source", "user_text": ""}, observed_at="2026-01-01T00:01:00+00:00", ) == 0 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7444,7 +4740,7 @@ def test_store_merge_distinguishes_whitespace_content_from_empty_or_absent_field assert preserved["assistant_final_text"] == "known final" whitespace = " \t\r\n " - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7463,7 +4759,7 @@ def test_store_merge_distinguishes_whitespace_content_from_empty_or_absent_field assert replaced["user_text"] == "known prompt" assert replaced["assistant_final_text"] == whitespace - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7492,7 +4788,7 @@ def test_store_revision_replacement_rolls_back_projection_and_current_flip( ) init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "rollback-host", "worker-1", @@ -7520,7 +4816,7 @@ def fail_insert(*args: Any, **kwargs: Any) -> str: monkeypatch.setattr(store_sqlite, "_insert_turn_content_revision_conn", fail_insert) with pytest.raises(RuntimeError, match="controlled revision insert failure"): - merge_turn_content( + apply_test_turn_refresh( db_path, "rollback-host", "worker-1", @@ -7573,7 +4869,7 @@ def _seed_superseded_content_revision( ) init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -7584,7 +4880,7 @@ def _seed_superseded_content_revision( }, observed_at="2026-01-01T00:00:00+00:00", ) == 1 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -8351,7 +5647,7 @@ def test_acknowledged_revision_count_bounds_one_frequently_revised_turn( api = ConnectorOutboxAPI(db_path, host_id) revisions: list[str] = [] for index in range(4): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -8557,7 +5853,6 @@ def test_turn_content_maintenance_preserves_live_and_young_final_sources( attempt_at if attempt_status == "delivered" else None, ), ) - result = run_store_maintenance( db_path, host_id, @@ -8798,7 +6093,7 @@ def test_turn_content_maintenance_preserves_all_reference_classes( workers=[{"id": "worker-1", "name": "claude", "status": "active"}], ), ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -9067,7 +6362,7 @@ def test_source_turn_history_pruning_retains_referenced_old_turn( init_store(db_path) save_snapshot(db_path, snapshot) for index in range(6): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "protected-host", "worker-1", @@ -9148,7 +6443,7 @@ def test_source_turn_history_pruning_retains_referenced_old_turn( ) for index in range(6, 10): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "protected-host", "worker-1", @@ -9218,7 +6513,7 @@ def test_source_turn_history_pruning_retains_referenced_old_turn( max_outbox_attempts=99, now="2026-01-20T00:00:00+00:00", ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "protected-host", "worker-1", @@ -9391,7 +6686,7 @@ def test_store_v8_maintenance_schema_singleton_and_ordered_indexes( assert created == ("created_at", "host_id", "id") -def test_current_v9_schema_gate_and_second_init_have_no_mutation_or_wal_setting( +def test_current_v28_schema_gate_and_second_init_have_no_mutation_or_wal_setting( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -9409,48 +6704,161 @@ def test_current_v9_schema_gate_and_second_init_have_no_mutation_or_wal_setting( traces: list[str] = [] original_pragmas = store_sqlite._apply_connection_pragmas - def traced_pragmas( - conn: sqlite3.Connection, - path: Path | str, - ) -> None: - conn.set_trace_callback(traces.append) - original_pragmas(conn, path) + def traced_pragmas( + conn: sqlite3.Connection, + path: Path | str, + ) -> None: + conn.set_trace_callback(traces.append) + original_pragmas(conn, path) + + monkeypatch.setattr(store_sqlite, "_apply_connection_pragmas", traced_pragmas) + original_flock = fcntl.flock + + def reject_parent_ex(fd: int, operation: int) -> None: + if operation & fcntl.LOCK_EX: + raise AssertionError("current schema attempted parent exclusivity") + original_flock(fd, operation) + + monkeypatch.setattr(store_sqlite.fcntl, "flock", reject_parent_ex) + init_store(db_path) + + normalized = [" ".join(statement.upper().split()) for statement in traces] + assert not any( + statement.startswith(("CREATE ", "ALTER ", "DROP ", "INSERT ", "UPDATE ", "DELETE ")) + for statement in normalized + ) + assert "PRAGMA JOURNAL_MODE=WAL" not in normalized + assert not any( + statement.startswith("PRAGMA USER_VERSION =") + for statement in normalized + ) + + +def _create_discarded_schema(db_path: Path, version: int) -> None: + with sqlite3.connect(str(db_path)) as conn: + conn.executescript( + """ + CREATE TABLE discarded_data (id INTEGER PRIMARY KEY, value TEXT); + CREATE INDEX discarded_index ON discarded_data(value); + CREATE TABLE discarded_audit (value TEXT); + CREATE TRIGGER discarded_trigger + AFTER INSERT ON discarded_data + BEGIN + INSERT INTO discarded_audit(value) VALUES (NEW.value); + END; + CREATE VIEW discarded_view AS + SELECT id, value FROM discarded_data; + INSERT INTO discarded_data(value) VALUES ('must be discarded'); + """ + ) + conn.execute(f"PRAGMA user_version = {int(version)}") + + +def _application_objects(db_path: Path) -> tuple[tuple[str, str], ...]: + with sqlite3.connect(str(db_path)) as conn: + return tuple( + (str(row[0]), str(row[1])) + for row in conn.execute( + """ + SELECT type, name FROM sqlite_master + WHERE name NOT LIKE 'sqlite_%' + ORDER BY type, name + """ + ).fetchall() + ) + + +def test_older_schema_is_loudly_discarded_and_recreated( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + db_path = tmp_path / "older.db" + _create_discarded_schema(db_path, store_sqlite.STORE_SCHEMA_VERSION - 1) + + with caplog.at_level(logging.WARNING, logger=store_sqlite.__name__): + init_store(db_path) + + with sqlite3.connect(str(db_path)) as conn: + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION + assert conn.execute( + "SELECT COUNT(*) FROM snapshots" + ).fetchone() == (0,) + names = {name for _kind, name in _application_objects(db_path)} + assert not any(name.startswith("discarded_") for name in names) + message = caplog.records[-1].getMessage() + for discarded in ( + "view:discarded_view", + "trigger:discarded_trigger", + "index:discarded_index", + "table:discarded_data", + "table:discarded_audit", + ): + assert discarded in message + + +def test_newer_schema_is_loudly_discarded_and_recreated( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + db_path = tmp_path / "newer.db" + _create_discarded_schema(db_path, store_sqlite.STORE_SCHEMA_VERSION + 1) + + with caplog.at_level(logging.WARNING, logger=store_sqlite.__name__): + init_store(db_path) + + with sqlite3.connect(str(db_path)) as conn: + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION + assert "previous_version=29" in caplog.records[-1].getMessage() + assert "table:discarded_data" in caplog.records[-1].getMessage() + + +def test_v0_with_application_objects_is_loudly_discarded_and_recreated( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + db_path = tmp_path / "v0-with-objects.db" + _create_discarded_schema(db_path, 0) + + with caplog.at_level(logging.WARNING, logger=store_sqlite.__name__): + init_store(db_path) - monkeypatch.setattr(store_sqlite, "_apply_connection_pragmas", traced_pragmas) - monkeypatch.setattr( - store_sqlite, - "MIGRATIONS", - tuple( - store_sqlite.Migration( - migration.from_version, - migration.to_version, - lambda _conn: (_ for _ in ()).throw( - AssertionError("current schema dispatched a migration") - ), - ) - for migration in store_sqlite.MIGRATIONS - ), - ) - original_flock = fcntl.flock + with sqlite3.connect(str(db_path)) as conn: + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION + assert conn.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'discarded_data'" + ).fetchone() is None + assert "previous_version=0" in caplog.records[-1].getMessage() - def reject_parent_ex(fd: int, operation: int) -> None: - if operation & fcntl.LOCK_EX: - raise AssertionError("current schema attempted parent exclusivity") - original_flock(fd, operation) - monkeypatch.setattr(store_sqlite.fcntl, "flock", reject_parent_ex) - init_store(db_path) +def test_schema_rebuild_failure_rolls_back_every_discarded_object( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "rebuild-rollback.db" + _create_discarded_schema(db_path, store_sqlite.STORE_SCHEMA_VERSION - 1) + before = _application_objects(db_path) - normalized = [" ".join(statement.upper().split()) for statement in traces] - assert not any( - statement.startswith(("CREATE ", "ALTER ", "DROP ", "INSERT ", "UPDATE ", "DELETE ")) - for statement in normalized - ) - assert "PRAGMA JOURNAL_MODE=WAL" not in normalized - assert not any( - statement.startswith("PRAGMA USER_VERSION =") - for statement in normalized + def fail_rebuild(_conn: sqlite3.Connection) -> None: + raise RuntimeError("injected rebuild failure") + + monkeypatch.setattr( + store_sqlite, + "_create_current_schema_objects_conn", + fail_rebuild, ) + with store_sqlite._connect(db_path, prepare=True) as conn: + assert conn.execute("PRAGMA foreign_keys").fetchone() == (1,) + with pytest.raises(RuntimeError, match="injected rebuild failure"): + store_sqlite.ensure_schema(conn) + assert conn.execute("PRAGMA foreign_keys").fetchone() == (1,) + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - 1 + assert conn.execute( + "SELECT value FROM discarded_data" + ).fetchone() == ("must be discarded",) + assert conn.execute( + "SELECT value FROM discarded_audit" + ).fetchone() == ("must be discarded",) + assert _application_objects(db_path) == before @pytest.mark.parametrize("version", (0, 1)) @pytest.mark.parametrize( @@ -9564,6 +6972,10 @@ def close() -> None: try: store_sqlite.ensure_schema(conn) assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION + assert conn.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'legacy_sentinel'" + ).fetchone() is None + assert "snapshots" in _table_names(conn) assert len(close_errors) == 1 assert isinstance(close_errors[0], sqlite3.ProgrammingError) finally: @@ -9571,261 +6983,30 @@ def close() -> None: def test_abandoned_store_connection_releases_parent_authority( - tmp_path: Path, -) -> None: - db_path = tmp_path / "abandoned-connection.db" - init_store(db_path) - - connection = store_sqlite._connect(db_path) - del connection - gc.collect() - - parent_fd = os.open( - db_path.parent, - os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), - ) - try: - fcntl.flock(parent_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - fcntl.flock(parent_fd, fcntl.LOCK_UN) - finally: - os.close(parent_fd) - - -def test_noncurrent_schema_promotion_preflight_preserves_callers_shared_authority( - tmp_path: Path, -) -> None: - db_path = tmp_path / "preflight-sh-legacy.db" - with sqlite3.connect(str(db_path)) as legacy: - legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") - legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") - legacy.execute("PRAGMA user_version = 1") - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - - before_fds = set(os.listdir("/proc/self/fd")) - before_threads = {id(thread) for thread in threading.enumerate()} - before_children = {process.pid for process in multiprocessing.active_children()} - connection: sqlite3.Connection | None = None - peer: sqlite3.Connection | None = None - lock_fd = -1 - try: - connection = store_sqlite._connect(db_path, prepare=True) - peer = store_sqlite._connect(db_path) - - with pytest.raises(LocalStateError) as caught: - store_sqlite.ensure_schema(connection) - assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED - assert _user_version(connection) == 1 - assert connection.execute( - "SELECT value FROM legacy_sentinel" - ).fetchone() == ("preserved",) - - peer.close() - peer = None - lock_fd = os.open( - db_path.parent, - os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), - ) - with pytest.raises(BlockingIOError): - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - - connection.close() - connection = None - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - fcntl.flock(lock_fd, fcntl.LOCK_UN) - finally: - if peer is not None: - peer.close() - if connection is not None: - connection.close() - if lock_fd >= 0: - os.close(lock_fd) - - assert set(os.listdir("/proc/self/fd")) - before_fds == set() - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children - - -def test_noncurrent_schema_external_upgrade_contention_restores_shared_authority( - tmp_path: Path, -) -> None: - db_path = tmp_path / "external-upgrade-sh-legacy.db" - with sqlite3.connect(str(db_path)) as legacy: - legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") - legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") - legacy.execute("PRAGMA user_version = 1") - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - before_threads = {id(thread) for thread in threading.enumerate()} - before_children = {process.pid for process in multiprocessing.active_children()} - context = multiprocessing.get_context("spawn") - acquired = context.Queue() - release = context.Event() - process = context.Process( - target=_cross_process_hold_parent_lock, - args=(str(db_path.parent), fcntl.LOCK_SH, acquired, release), - ) - connection: sqlite3.Connection | None = None - lock_fd = -1 - parent_fd = -1 - connection_id = -1 - started = False - try: - process.start() - started = True - assert acquired.get(timeout=5) is None - connection = store_sqlite._connect(db_path, prepare=True) - authority = store_sqlite._schema_connection_authority(connection) - assert authority.parent_fd is not None - parent_fd = authority.parent_fd - connection_id = id(connection) - - with pytest.raises(LocalStateError) as caught: - store_sqlite.ensure_schema(connection) - assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED - assert _user_version(connection) == 1 - - release.set() - process.join(timeout=15) - if process.is_alive(): - pytest.fail("external parent SH holder did not terminate") - assert process.exitcode == 0 - process.close() - started = False - - lock_fd = os.open( - db_path.parent, - os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), - ) - with pytest.raises(BlockingIOError): - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - - connection.close() - with pytest.raises(OSError): - os.fstat(parent_fd) - assert connection_id not in store_sqlite._SCHEMA_CONNECTION_AUTHORITIES - connection = None - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - fcntl.flock(lock_fd, fcntl.LOCK_UN) - finally: - release.set() - if started: - process.join(timeout=15) - if process.is_alive(): - process.terminate() - process.join(timeout=5) - process.close() - if connection is not None: - connection.close() - if lock_fd >= 0: - os.close(lock_fd) - acquired.close() - acquired.join_thread() - - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children - - -def test_noncurrent_schema_upgrade_recovery_retry_fails_closed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "upgrade-recovery-legacy.db" - with sqlite3.connect(str(db_path)) as legacy: - legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") - legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") - legacy.execute("PRAGMA user_version = 1") - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - - before_threads = {id(thread) for thread in threading.enumerate()} - before_children = {process.pid for process in multiprocessing.active_children()} - context = multiprocessing.get_context("spawn") - acquired = context.Queue() - release = context.Event() - process = context.Process( - target=_cross_process_hold_parent_lock, - args=(str(db_path.parent), fcntl.LOCK_EX, acquired, release), - ) - connection: sqlite3.Connection | None = None - lock_fd = -1 - started = False - connection_id = -1 - try: - connection = store_sqlite._connect(db_path, prepare=True) - authority = store_sqlite._schema_connection_authority(connection) - assert authority.parent_fd is not None - parent_fd = authority.parent_fd - connection_id = id(connection) - original_flock = fcntl.flock - restore_attempts = 0 - - def lose_upgrade_to_external_ex(fd: int, operation: int) -> None: - nonlocal restore_attempts, started - if fd == parent_fd and operation == (fcntl.LOCK_EX | fcntl.LOCK_NB): - original_flock(fd, fcntl.LOCK_UN) - assert acquired.get(timeout=5) is None - raise BlockingIOError() - if fd == parent_fd and operation == (fcntl.LOCK_SH | fcntl.LOCK_NB): - restore_attempts += 1 - try: - original_flock(fd, operation) - except BlockingIOError: - if restore_attempts == 1: - release.set() - process.join(timeout=15) - if process.is_alive(): - pytest.fail("external parent EX holder did not terminate") - assert process.exitcode == 0 - process.close() - started = False - raise - return - original_flock(fd, operation) + tmp_path: Path, +) -> None: + db_path = tmp_path / "abandoned-connection.db" + init_store(db_path) - process.start() - started = True - monkeypatch.setattr(store_sqlite.fcntl, "flock", lose_upgrade_to_external_ex) - with pytest.raises(LocalStateError) as caught: - store_sqlite.ensure_schema(connection) - assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED - assert restore_attempts == 2 - with pytest.raises(sqlite3.ProgrammingError): - connection.execute("SELECT 1") - with pytest.raises(OSError): - os.fstat(parent_fd) - assert connection_id not in store_sqlite._SCHEMA_CONNECTION_AUTHORITIES + connection = store_sqlite._connect(db_path) + del connection + gc.collect() - lock_fd = os.open( - db_path.parent, - os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), - ) - original_flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - original_flock(lock_fd, fcntl.LOCK_UN) + parent_fd = os.open( + db_path.parent, + os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), + ) + try: + fcntl.flock(parent_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + fcntl.flock(parent_fd, fcntl.LOCK_UN) finally: - release.set() - if started: - process.join(timeout=15) - if process.is_alive(): - process.terminate() - process.join(timeout=5) - process.close() - if connection is not None: - connection.close() - if lock_fd >= 0: - os.close(lock_fd) - acquired.close() - acquired.join_thread() - - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children + os.close(parent_fd) -def test_noncurrent_schema_downgrade_recovery_retry_fails_closed( +def test_noncurrent_schema_promotion_preflight_preserves_callers_shared_authority( tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, ) -> None: - db_path = tmp_path / "downgrade-recovery-legacy.db" + db_path = tmp_path / "preflight-sh-legacy.db" with sqlite3.connect(str(db_path)) as legacy: legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") @@ -9834,57 +7015,53 @@ def test_noncurrent_schema_downgrade_recovery_retry_fails_closed( os.chmod(db_path, 0o600) before_fds = set(os.listdir("/proc/self/fd")) + before_threads = {id(thread) for thread in threading.enumerate()} + before_children = {process.pid for process in multiprocessing.active_children()} connection: sqlite3.Connection | None = None + peer: sqlite3.Connection | None = None lock_fd = -1 try: connection = store_sqlite._connect(db_path, prepare=True) - authority = store_sqlite._schema_connection_authority(connection) - assert authority.parent_fd is not None - parent_fd = authority.parent_fd - connection_id = id(connection) - original_flock = fcntl.flock - restore_attempts = 0 - - def fail_first_shared_restore(fd: int, operation: int) -> None: - nonlocal restore_attempts - if fd == parent_fd and operation == (fcntl.LOCK_SH | fcntl.LOCK_NB): - restore_attempts += 1 - if restore_attempts == 1: - original_flock(fd, fcntl.LOCK_UN) - raise BlockingIOError() - original_flock(fd, operation) + peer = store_sqlite._connect(db_path) - monkeypatch.setattr(store_sqlite.fcntl, "flock", fail_first_shared_restore) with pytest.raises(LocalStateError) as caught: store_sqlite.ensure_schema(connection) assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED - assert restore_attempts == 2 - with pytest.raises(sqlite3.ProgrammingError): - connection.execute("SELECT 1") - with pytest.raises(OSError): - os.fstat(parent_fd) - assert connection_id not in store_sqlite._SCHEMA_CONNECTION_AUTHORITIES + assert _user_version(connection) == 1 + assert connection.execute( + "SELECT value FROM legacy_sentinel" + ).fetchone() == ("preserved",) + peer.close() + peer = None lock_fd = os.open( db_path.parent, os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), ) - original_flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - original_flock(lock_fd, fcntl.LOCK_UN) + with pytest.raises(BlockingIOError): + fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + + connection.close() + connection = None + fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + fcntl.flock(lock_fd, fcntl.LOCK_UN) finally: + if peer is not None: + peer.close() if connection is not None: connection.close() if lock_fd >= 0: os.close(lock_fd) assert set(os.listdir("/proc/self/fd")) - before_fds == set() + assert {id(thread) for thread in threading.enumerate()} == before_threads + assert {process.pid for process in multiprocessing.active_children()} == before_children -def test_noncurrent_schema_downgrade_external_ex_contention_fails_closed( +def test_noncurrent_schema_external_upgrade_contention_restores_shared_authority( tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, ) -> None: - db_path = tmp_path / "downgrade-ex-legacy.db" + db_path = tmp_path / "external-upgrade-sh-legacy.db" with sqlite3.connect(str(db_path)) as legacy: legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") @@ -9898,62 +7075,32 @@ def test_noncurrent_schema_downgrade_external_ex_contention_fails_closed( release = context.Event() process = context.Process( target=_cross_process_hold_parent_lock, - args=(str(db_path.parent), fcntl.LOCK_EX, acquired, release), + args=(str(db_path.parent), fcntl.LOCK_SH, acquired, release), ) connection: sqlite3.Connection | None = None lock_fd = -1 - started = False + parent_fd = -1 connection_id = -1 + started = False try: + process.start() + started = True + assert acquired.get(timeout=5) is None connection = store_sqlite._connect(db_path, prepare=True) authority = store_sqlite._schema_connection_authority(connection) assert authority.parent_fd is not None parent_fd = authority.parent_fd connection_id = id(connection) - original_run_migrations = store_sqlite._run_migrations - original_flock = fcntl.flock - failed_restores = 0 - - def start_external_ex_after_migration(*args: Any, **kwargs: Any) -> None: - nonlocal started - original_run_migrations(*args, **kwargs) - process.start() - started = True - - def lose_downgrade_to_external_ex(fd: int, operation: int) -> None: - nonlocal failed_restores - if fd == parent_fd and operation == (fcntl.LOCK_SH | fcntl.LOCK_NB): - failed_restores += 1 - original_flock(fd, fcntl.LOCK_UN) - if failed_restores == 1: - assert acquired.get(timeout=5) is None - raise BlockingIOError() - original_flock(fd, operation) - monkeypatch.setattr( - store_sqlite, - "_run_migrations", - start_external_ex_after_migration, - ) - monkeypatch.setattr( - store_sqlite.fcntl, - "flock", - lose_downgrade_to_external_ex, - ) with pytest.raises(LocalStateError) as caught: store_sqlite.ensure_schema(connection) assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED - assert failed_restores == store_sqlite._SCHEMA_PARENT_SHARED_LOCK_RECOVERY_ATTEMPTS - with pytest.raises(sqlite3.ProgrammingError): - connection.execute("SELECT 1") - with pytest.raises(OSError): - os.fstat(parent_fd) - assert connection_id not in store_sqlite._SCHEMA_CONNECTION_AUTHORITIES + assert _user_version(connection) == 1 release.set() process.join(timeout=15) if process.is_alive(): - pytest.fail("external parent EX holder did not terminate") + pytest.fail("external parent SH holder did not terminate") assert process.exitcode == 0 process.close() started = False @@ -9962,6 +7109,14 @@ def lose_downgrade_to_external_ex(fd: int, operation: int) -> None: db_path.parent, os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), ) + with pytest.raises(BlockingIOError): + fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + + connection.close() + with pytest.raises(OSError): + os.fstat(parent_fd) + assert connection_id not in store_sqlite._SCHEMA_CONNECTION_AUTHORITIES + connection = None fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) fcntl.flock(lock_fd, fcntl.LOCK_UN) finally: @@ -9983,73 +7138,11 @@ def lose_downgrade_to_external_ex(fd: int, operation: int) -> None: assert {process.pid for process in multiprocessing.active_children()} == before_children -def test_noncurrent_schema_live_shared_parent_fails_before_persistent_mutation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "shared-legacy.db" - with sqlite3.connect(str(db_path)) as legacy: - legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") - legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") - legacy.execute("PRAGMA user_version = 1") - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - - before_threads = {id(thread) for thread in threading.enumerate()} - before_children = {process.pid for process in multiprocessing.active_children()} - holder = store_sqlite._connect(db_path) - retained_fds = set(os.listdir("/proc/self/fd")) - traces: list[str] = [] - created: list[sqlite3.Connection] = [] - original_connect = sqlite3.connect - - def traced_connect(*args: Any, **kwargs: Any) -> sqlite3.Connection: - connection = original_connect(*args, **kwargs) - connection.set_trace_callback(traces.append) - created.append(connection) - return connection - - monkeypatch.setattr(store_sqlite.sqlite3, "connect", traced_connect) - try: - with pytest.raises(LocalStateError) as caught: - init_store(db_path) - assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED - assert holder.execute("SELECT value FROM legacy_sentinel").fetchone() == ( - "preserved", - ) - assert not Path(f"{db_path}-wal").exists() - assert not Path(f"{db_path}-shm").exists() - normalized = [" ".join(statement.upper().split()) for statement in traces] - assert "PRAGMA JOURNAL_MODE=WAL" not in normalized - assert "PRAGMA DATABASE_LIST" not in normalized - assert not any( - statement.startswith(("BEGIN", "CREATE ", "ALTER ", "DROP ")) - for statement in normalized - ) - assert len(created) == 1 - with pytest.raises(sqlite3.ProgrammingError): - created[0].execute("SELECT 1") - assert set(os.listdir("/proc/self/fd")) - retained_fds == set() - finally: - holder.close() - - with original_connect(str(db_path)) as inspection: - assert _user_version(inspection) == 1 - assert inspection.execute( - "SELECT value FROM legacy_sentinel" - ).fetchone() == ("preserved",) - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children - - -@pytest.mark.parametrize("change", ("unlink", "replace")) -def test_noncurrent_schema_finalization_rejects_changed_pinned_main_without_creation( +def test_noncurrent_schema_upgrade_recovery_retry_fails_closed( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, - change: str, ) -> None: - db_path = tmp_path / "unlinked-legacy.db" - replacement_bytes = b"replacement main must remain untouched" + db_path = tmp_path / "upgrade-recovery-legacy.db" with sqlite3.connect(str(db_path)) as legacy: legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") @@ -10057,94 +7150,94 @@ def test_noncurrent_schema_finalization_rejects_changed_pinned_main_without_crea os.chmod(tmp_path, 0o700) os.chmod(db_path, 0o600) - selected_identity = entry_identity(os.lstat(db_path)) - before_fds = set(os.listdir("/proc/self/fd")) before_threads = {id(thread) for thread in threading.enumerate()} before_children = {process.pid for process in multiprocessing.active_children()} - original_prepare = store_sqlite.prepare_sqlite_family_at - original_migrations = store_sqlite._run_migrations - prepare_calls = 0 - migrated = False - fired = False + context = multiprocessing.get_context("spawn") + acquired = context.Queue() + release = context.Event() + process = context.Process( + target=_cross_process_hold_parent_lock, + args=(str(db_path.parent), fcntl.LOCK_EX, acquired, release), + ) connection: sqlite3.Connection | None = None lock_fd = -1 + started = False + connection_id = -1 + try: + connection = store_sqlite._connect(db_path, prepare=True) + authority = store_sqlite._schema_connection_authority(connection) + assert authority.parent_fd is not None + parent_fd = authority.parent_fd + connection_id = id(connection) + original_flock = fcntl.flock + restore_attempts = 0 - def record_migration(*args: Any, **kwargs: Any) -> None: - nonlocal migrated - original_migrations(*args, **kwargs) - migrated = True - - def change_before_final_family_prepare( - parent_fd: int, - leaf: str, - **kwargs: Any, - ) -> tuple[PermissionResult, ...]: - nonlocal prepare_calls, fired - prepare_calls += 1 - if prepare_calls == 2: - assert migrated - assert kwargs["_parent_exclusive_lock_held"] is True - assert kwargs["_expected_main_identity"] == selected_identity - os.unlink(leaf, dir_fd=parent_fd) - if change == "replace": - replacement_fd = os.open( - leaf, - os.O_WRONLY | os.O_CREAT | os.O_EXCL, - 0o600, - dir_fd=parent_fd, - ) + def lose_upgrade_to_external_ex(fd: int, operation: int) -> None: + nonlocal restore_attempts, started + if fd == parent_fd and operation == (fcntl.LOCK_EX | fcntl.LOCK_NB): + original_flock(fd, fcntl.LOCK_UN) + assert acquired.get(timeout=5) is None + raise BlockingIOError() + if fd == parent_fd and operation == (fcntl.LOCK_SH | fcntl.LOCK_NB): + restore_attempts += 1 try: - os.write(replacement_fd, replacement_bytes) - os.fchmod(replacement_fd, 0o600) - finally: - os.close(replacement_fd) - fired = True - return original_prepare(parent_fd, leaf, **kwargs) + original_flock(fd, operation) + except BlockingIOError: + if restore_attempts == 1: + release.set() + process.join(timeout=15) + if process.is_alive(): + pytest.fail("external parent EX holder did not terminate") + assert process.exitcode == 0 + process.close() + started = False + raise + return + original_flock(fd, operation) - monkeypatch.setattr(store_sqlite, "_run_migrations", record_migration) - monkeypatch.setattr( - store_sqlite, - "prepare_sqlite_family_at", - change_before_final_family_prepare, - ) - try: - connection = store_sqlite._connect(db_path, prepare=True) + process.start() + started = True + monkeypatch.setattr(store_sqlite.fcntl, "flock", lose_upgrade_to_external_ex) with pytest.raises(LocalStateError) as caught: store_sqlite.ensure_schema(connection) + assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED + assert restore_attempts == 2 + with pytest.raises(sqlite3.ProgrammingError): + connection.execute("SELECT 1") + with pytest.raises(OSError): + os.fstat(parent_fd) + assert connection_id not in store_sqlite._SCHEMA_CONNECTION_AUTHORITIES - assert fired - assert prepare_calls == 2 - assert caught.value.code is LocalStateErrorCode.ENTRY_CHANGED - if change == "replace": - assert db_path.read_bytes() == replacement_bytes - assert _mode(db_path) == 0o600 - else: - assert not db_path.exists() lock_fd = os.open( db_path.parent, os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), ) - with pytest.raises(BlockingIOError): - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + original_flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + original_flock(lock_fd, fcntl.LOCK_UN) finally: + release.set() + if started: + process.join(timeout=15) + if process.is_alive(): + process.terminate() + process.join(timeout=5) + process.close() if connection is not None: connection.close() if lock_fd >= 0: - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - fcntl.flock(lock_fd, fcntl.LOCK_UN) os.close(lock_fd) + acquired.close() + acquired.join_thread() - if change == "replace": - assert db_path.read_bytes() == replacement_bytes - assert set(os.listdir("/proc/self/fd")) - before_fds == set() assert {id(thread) for thread in threading.enumerate()} == before_threads assert {process.pid for process in multiprocessing.active_children()} == before_children -def test_noncurrent_schema_keeps_sidecars_private_and_restores_shared_parent_lock( +def test_noncurrent_schema_downgrade_recovery_retry_fails_closed( tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, ) -> None: - db_path = tmp_path / "private-legacy.db" + db_path = tmp_path / "downgrade-recovery-legacy.db" with sqlite3.connect(str(db_path)) as legacy: legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") @@ -10152,780 +7245,265 @@ def test_noncurrent_schema_keeps_sidecars_private_and_restores_shared_parent_loc os.chmod(tmp_path, 0o700) os.chmod(db_path, 0o600) - before_threads = {id(thread) for thread in threading.enumerate()} - before_children = {process.pid for process in multiprocessing.active_children()} + before_fds = set(os.listdir("/proc/self/fd")) connection: sqlite3.Connection | None = None lock_fd = -1 - previous_umask = os.umask(0) try: connection = store_sqlite._connect(db_path, prepare=True) - store_sqlite.ensure_schema(connection) + authority = store_sqlite._schema_connection_authority(connection) + assert authority.parent_fd is not None + parent_fd = authority.parent_fd + connection_id = id(connection) + original_flock = fcntl.flock + restore_attempts = 0 - assert _user_version(connection) == store_sqlite.STORE_SCHEMA_VERSION - assert connection.execute( - "SELECT value FROM legacy_sentinel" - ).fetchone() == ("preserved",) - for suffix in ("", "-wal", "-shm"): - assert _mode(Path(f"{db_path}{suffix}")) == 0o600 + def fail_first_shared_restore(fd: int, operation: int) -> None: + nonlocal restore_attempts + if fd == parent_fd and operation == (fcntl.LOCK_SH | fcntl.LOCK_NB): + restore_attempts += 1 + if restore_attempts == 1: + original_flock(fd, fcntl.LOCK_UN) + raise BlockingIOError() + original_flock(fd, operation) + + monkeypatch.setattr(store_sqlite.fcntl, "flock", fail_first_shared_restore) + with pytest.raises(LocalStateError) as caught: + store_sqlite.ensure_schema(connection) + assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED + assert restore_attempts == 2 + with pytest.raises(sqlite3.ProgrammingError): + connection.execute("SELECT 1") + with pytest.raises(OSError): + os.fstat(parent_fd) + assert connection_id not in store_sqlite._SCHEMA_CONNECTION_AUTHORITIES lock_fd = os.open( db_path.parent, os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), ) - with pytest.raises(BlockingIOError): - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - - connection.close() - connection = None - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - fcntl.flock(lock_fd, fcntl.LOCK_UN) - finally: - if connection is not None: - connection.close() - if lock_fd >= 0: - os.close(lock_fd) - os.umask(previous_umask) - - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children - - -def test_direct_empty_creation_does_not_replay_migration_registry( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "direct-current.db" - monkeypatch.setattr( - store_sqlite, - "MIGRATIONS", - tuple( - store_sqlite.Migration( - migration.from_version, - migration.to_version, - lambda _conn: (_ for _ in ()).throw( - AssertionError("direct creation replayed history") - ), - ) - for migration in store_sqlite.MIGRATIONS - ), - ) - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - assert "turn_list_hosts" in _table_names(conn) - assert {"store_maintenance_state", "turn_list_state"} <= _table_names(conn) - assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - -def test_v13_migration_repairs_nonpositive_turn_sequences_and_blocks_recurrence( - tmp_path: Path, -) -> None: - db_path = tmp_path / "legacy-turn-sequence.db" - observed_at = "2026-07-15T00:00:00+00:00" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=13) - conn.execute("DROP TABLE turns") - conn.execute( - store_sqlite.CREATE_TURNS_TABLE.replace( - " CHECK (list_sequence > 0)", - "", - ) - ) - for statement in store_sqlite.CREATE_TURN_LIST_INDEXES: - conn.execute(statement) - for turn_id, sequence in (("turn-valid", 5), ("turn-invalid", 0)): - payload = { - "id": turn_id, - "worker_id": "worker-a", - "status": "complete", - "kind": "prompt", - "source": "snapshot", - "updated_at": observed_at, - } - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, worker_fingerprint, space_id, - status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, - list_sequence - ) VALUES (?, ?, 'worker-a', NULL, NULL, 'complete', 'prompt', - ?, '', '', ?, ?, ?) - """, - ( - "legacy-host", - turn_id, - observed_at, - observed_at, - json.dumps(payload), - sequence, - ), - ) - conn.execute( - """ - INSERT INTO turn_list_hosts ( - host_id, next_sequence, traversal_generation - ) VALUES ('legacy-host', 6, 7) - """ - ) - conn.commit() - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 28 - assert conn.execute( - """ - SELECT turn_id, list_sequence - FROM turns - WHERE host_id = 'legacy-host' - ORDER BY list_sequence - """ - ).fetchall() == [("turn-valid", 5), ("turn-invalid", 6)] - assert conn.execute( - """ - SELECT next_sequence, traversal_generation - FROM turn_list_hosts - WHERE host_id = 'legacy-host' - """ - ).fetchone() == (7, 8) - assert conn.execute( - """ - SELECT COUNT(*) - FROM sqlite_master - WHERE type = 'trigger' - AND name LIKE 'trg_turns_positive_list_sequence_%' - """ - ).fetchone() == (2,) - with pytest.raises(sqlite3.IntegrityError, match="invalid turn list sequence"): - conn.execute( - """ - UPDATE turns SET list_sequence = 0 - WHERE host_id = 'legacy-host' AND turn_id = 'turn-valid' - """ - ) - conn.rollback() - with pytest.raises(sqlite3.IntegrityError, match="invalid turn list sequence"): - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, status, kind, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, - list_sequence - ) VALUES ( - 'legacy-host', 'turn-recurrence', 'worker-a', 'complete', - 'prompt', '', '', ?, '{}', 0 - ) - """, - (observed_at,), - ) - conn.rollback() - - first = turns_payload_from_store( - db_path, - "legacy-host", - schema_version=2, - limit=1, - now=1_800_000_000, - ) - assert first["has_more"] is True - assert isinstance(first["next_cursor"], str) - second = turns_payload_from_store( - db_path, - "legacy-host", - schema_version=2, - limit=1, - cursor=first["next_cursor"], - now=1_800_000_001, - ) - assert second["has_more"] is False - assert [item["id"] for item in first["turns"] + second["turns"]] == [ - "turn-invalid", - "turn-valid", - ] - - -@pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) -def test_migration_registry_transition_rolls_back_resumes_and_reruns( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - source_version: int, -) -> None: - db_path = tmp_path / f"registry-{source_version}.db" - original_registry = store_sqlite.MIGRATIONS - assert tuple( - (migration.from_version, migration.to_version) - for migration in original_registry - ) == tuple( - (version, version + 1) - for version in range(store_sqlite.STORE_SCHEMA_VERSION) - ) - - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys=ON") - conn.execute("CREATE TABLE durable_sentinel (value TEXT NOT NULL)") - conn.execute("INSERT INTO durable_sentinel VALUES ('preserved')") - conn.commit() - store_sqlite._run_migrations(conn, target_version=source_version) - assert _user_version(conn) == source_version - - transition = original_registry[source_version] - - def apply_then_fail(current: sqlite3.Connection) -> None: - transition.apply(current) - raise RuntimeError("controlled migration interruption") - - interrupted = list(original_registry) - interrupted[source_version] = store_sqlite.Migration( - transition.from_version, - transition.to_version, - apply_then_fail, - ) - monkeypatch.setattr(store_sqlite, "MIGRATIONS", tuple(interrupted)) - with pytest.raises(RuntimeError, match="controlled migration interruption"): - store_sqlite._run_migrations( - conn, - target_version=source_version + 1, - ) - assert _user_version(conn) == source_version - assert conn.execute("SELECT value FROM durable_sentinel").fetchone() == ( - "preserved", - ) - - monkeypatch.setattr(store_sqlite, "MIGRATIONS", original_registry) - store_sqlite._run_migrations(conn, target_version=source_version + 1) - assert _user_version(conn) == source_version + 1 - conn.execute("BEGIN IMMEDIATE") - transition.apply(conn) - conn.commit() - assert conn.execute("SELECT value FROM durable_sentinel").fetchone() == ( - "preserved", - ) - assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - -def test_ten_thousand_adjacent_identical_saves_keep_one_row_and_event( - tmp_path: Path, -) -> None: - db_path = tmp_path / "identical-10000.db" - config = Config(host_id="identical-host", db_path=db_path) - first = datetime.fromisoformat("2026-01-01T00:00:00+00:00") - final_snapshot = None - for index in range(10_000): - final_snapshot = project_from_raw( - config, - timestamp=first + store_sqlite.timedelta(seconds=index), - ) - save_snapshot(db_path, final_snapshot) - - assert final_snapshot is not None - with sqlite3.connect(str(db_path)) as conn: - snapshot_rows = conn.execute( - "SELECT created_at, payload FROM snapshots" - ).fetchall() - saved_events = conn.execute( - "SELECT COUNT(*) FROM events WHERE event_type = 'snapshot.saved'" - ).fetchone()[0] - integrity = conn.execute("PRAGMA integrity_check").fetchone()[0] - foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - - assert len(snapshot_rows) == 1 - assert saved_events == 1 - assert snapshot_rows[0][0] == final_snapshot.updated_at - assert json.loads(snapshot_rows[0][1])["updated_at"] == final_snapshot.updated_at - assert integrity == "ok" - assert foreign_keys == [] - - -def test_adjacent_dedupe_is_host_local_and_a_b_a_appends_changed_history( - tmp_path: Path, -) -> None: - db_path = tmp_path / "aba-host-local.db" - config_a = Config(host_id="host-a", db_path=db_path) - config_b = Config(host_id="host-b", db_path=db_path) - at = datetime.fromisoformat("2026-01-01T00:00:00+00:00") - snapshot_a = project_from_raw( - config_a, - workers=[{"id": "worker-a", "name": "A", "status": "active"}], - timestamp=at, - ) - snapshot_b = project_from_raw( - config_a, - workers=[{"id": "worker-b", "name": "B", "status": "active"}], - timestamp=at + store_sqlite.timedelta(seconds=1), - ) - snapshot_a_again = project_from_raw( - config_a, - workers=[{"id": "worker-a", "name": "A", "status": "active"}], - timestamp=at + store_sqlite.timedelta(seconds=2), - ) - other_host = project_from_raw( - config_b, - workers=[{"id": "worker-a", "name": "A", "status": "active"}], - timestamp=at + store_sqlite.timedelta(seconds=3), - ) + original_flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + original_flock(lock_fd, fcntl.LOCK_UN) + finally: + if connection is not None: + connection.close() + if lock_fd >= 0: + os.close(lock_fd) - for snapshot in (snapshot_a, snapshot_b, snapshot_a_again, other_host, other_host): - save_snapshot(db_path, snapshot) + assert set(os.listdir("/proc/self/fd")) - before_fds == set() - with sqlite3.connect(str(db_path)) as conn: - history = conn.execute( - """ - SELECT host_id, content_fingerprint - FROM snapshots - ORDER BY id - """ - ).fetchall() - event_count = conn.execute( - "SELECT COUNT(*) FROM events WHERE event_type = 'snapshot.saved'" - ).fetchone()[0] - assert history == [ - ("host-a", snapshot_a.content_fingerprint), - ("host-a", snapshot_b.content_fingerprint), - ("host-a", snapshot_a_again.content_fingerprint), - ("host-b", other_host.content_fingerprint), - ] - assert event_count == 4 -def test_stale_same_fingerprint_save_does_not_regress_current_state( +def test_noncurrent_schema_live_shared_parent_fails_before_persistent_mutation( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - db_path = tmp_path / "stale-identical.db" - config = Config(host_id="stale-host", db_path=db_path) - fresh_at = "2026-01-01T00:10:00+00:00" - stale_at = "2026-01-01T00:05:00+00:00" - fresh = _snapshot_with_worker_status( - config, - status="blocked", - observed_at=fresh_at, - ) - stale = _snapshot_with_worker_status( - config, - status="blocked", - observed_at=stale_at, - ) - assert stale.content_fingerprint == fresh.content_fingerprint + db_path = tmp_path / "shared-legacy.db" + with sqlite3.connect(str(db_path)) as legacy: + legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") + legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") + legacy.execute("PRAGMA user_version = 1") + os.chmod(tmp_path, 0o700) + os.chmod(db_path, 0o600) - _save_observation(db_path, fresh, "positive", fresh_at) + before_threads = {id(thread) for thread in threading.enumerate()} + before_children = {process.pid for process in multiprocessing.active_children()} + holder = store_sqlite._connect(db_path) + retained_fds = set(os.listdir("/proc/self/fd")) + traces: list[str] = [] + created: list[sqlite3.Connection] = [] + original_connect = sqlite3.connect - def current_state() -> tuple[Any, ...]: - with sqlite3.connect(str(db_path)) as conn: - return ( - conn.execute( - """ - SELECT created_at, payload - FROM snapshots - WHERE host_id = ? - ORDER BY id DESC - LIMIT 1 - """, - ("stale-host",), - ).fetchone(), - conn.execute( - """ - SELECT observed_at, payload_json - FROM workers - WHERE host_id = ? AND worker_id = 'worker-1' - """, - ("stale-host",), - ).fetchone(), - conn.execute( - """ - SELECT observed_at, payload_json - FROM backend_health - WHERE host_id = ? AND backend_name = 'herdr' - """, - ("stale-host",), - ).fetchone(), - conn.execute( - """ - SELECT last_seen_at, last_changed_at, signal_count, payload_json - FROM attention_items - WHERE host_id = ? - """, - ("stale-host",), - ).fetchone(), - conn.execute( - """ - SELECT last_positive_at, last_accepted_at, last_observation_key - FROM attention_lifecycles - WHERE host_id = ? - """, - ("stale-host",), - ).fetchone(), - conn.execute( - "SELECT COUNT(*) FROM snapshots WHERE host_id = ?", - ("stale-host",), - ).fetchone()[0], - conn.execute( - """ - SELECT COUNT(*) - FROM events - WHERE host_id = ? AND event_type = 'snapshot.saved' - """, - ("stale-host",), - ).fetchone()[0], - ) + def traced_connect(*args: Any, **kwargs: Any) -> sqlite3.Connection: + connection = original_connect(*args, **kwargs) + connection.set_trace_callback(traces.append) + created.append(connection) + return connection - before = current_state() - attention_calls: list[str] = [] - original_attention = store_sqlite._apply_attention_observation_conn + monkeypatch.setattr(store_sqlite.sqlite3, "connect", traced_connect) + try: + with pytest.raises(LocalStateError) as caught: + init_store(db_path) + assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED + assert holder.execute("SELECT value FROM legacy_sentinel").fetchone() == ( + "preserved", + ) + assert not Path(f"{db_path}-wal").exists() + assert not Path(f"{db_path}-shm").exists() + normalized = [" ".join(statement.upper().split()) for statement in traces] + assert "PRAGMA JOURNAL_MODE=WAL" not in normalized + assert "PRAGMA DATABASE_LIST" not in normalized + assert not any( + statement.startswith(("BEGIN", "CREATE ", "ALTER ", "DROP ")) + for statement in normalized + ) + assert len(created) == 1 + with pytest.raises(sqlite3.ProgrammingError): + created[0].execute("SELECT 1") + assert set(os.listdir("/proc/self/fd")) - retained_fds == set() + finally: + holder.close() - def observed_attention( - conn: sqlite3.Connection, - **kwargs: Any, - ) -> None: - attention_calls.append(str(kwargs["observation"].observed_at)) - original_attention(conn, **kwargs) + with original_connect(str(db_path)) as inspection: + assert _user_version(inspection) == 1 + assert inspection.execute( + "SELECT value FROM legacy_sentinel" + ).fetchone() == ("preserved",) + assert {id(thread) for thread in threading.enumerate()} == before_threads + assert {process.pid for process in multiprocessing.active_children()} == before_children - monkeypatch.setattr( - store_sqlite, - "_apply_attention_observation_conn", - observed_attention, - ) - _save_observation(db_path, stale, "positive", stale_at) - after = current_state() - assert attention_calls == [stale_at] - assert after == before - assert after[0][0] == fresh_at - assert after[1][0] == fresh.updated_at - assert after[2][0] == fresh_at - assert after[3][0] == fresh_at - assert after[4][0:2] == (fresh_at, fresh_at) - assert after[5:] == (1, 1) -def test_snapshot_created_at_is_canonical_utc_and_invalid_input_fails_before_open( + +def test_noncurrent_schema_keeps_sidecars_private_and_restores_shared_parent_lock( tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, ) -> None: - db_path = tmp_path / "canonical-created-at.db" - config = Config(host_id="canonical-host", db_path=db_path) - first = project_from_raw( - config, - timestamp=datetime.fromisoformat("2026-07-01T05:30:00+05:30"), - ) - later = project_from_raw( - config, - timestamp=datetime.fromisoformat("2026-06-30T21:00:00-04:00"), - ) - assert first.content_fingerprint == later.content_fingerprint - - save_snapshot(db_path, first) - save_snapshot(db_path, later) - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - "SELECT created_at FROM snapshots" - ).fetchall() - assert rows == [("2026-07-01T01:00:00+00:00",)] + db_path = tmp_path / "private-legacy.db" + with sqlite3.connect(str(db_path)) as legacy: + legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") + legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") + legacy.execute("PRAGMA user_version = 1") + os.chmod(tmp_path, 0o700) + os.chmod(db_path, 0o600) - invalid_path = tmp_path / "invalid" / "store.db" + before_threads = {id(thread) for thread in threading.enumerate()} + before_children = {process.pid for process in multiprocessing.active_children()} + connection: sqlite3.Connection | None = None + lock_fd = -1 + previous_umask = os.umask(0) + try: + connection = store_sqlite._connect(db_path, prepare=True) + store_sqlite.ensure_schema(connection) - def forbidden_connect(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("invalid timestamp reached sqlite open") + assert _user_version(connection) == store_sqlite.STORE_SCHEMA_VERSION + assert connection.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'legacy_sentinel'" + ).fetchone() is None + assert "snapshots" in _table_names(connection) + for suffix in ("", "-wal", "-shm"): + assert _mode(Path(f"{db_path}{suffix}")) == 0o600 - monkeypatch.setattr(store_sqlite, "_connect", forbidden_connect) - for invalid_timestamp in ( - "malformed-timestamp", - "0001-01-01T00:00:00+14:00", - "9999-12-31T23:59:59-14:00", - ): - invalid = project_empty( - Config(host_id="invalid-host", db_path=invalid_path) + lock_fd = os.open( + db_path.parent, + os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), ) - object.__setattr__(invalid, "updated_at", invalid_timestamp) - with pytest.raises(ValueError, match="invalid snapshot updated_at"): - save_snapshot(invalid_path, invalid) - assert not invalid_path.parent.exists() + with pytest.raises(BlockingIOError): + fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + + connection.close() + connection = None + fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + fcntl.flock(lock_fd, fcntl.LOCK_UN) + finally: + if connection is not None: + connection.close() + if lock_fd >= 0: + os.close(lock_fd) + os.umask(previous_umask) + + assert {id(thread) for thread in threading.enumerate()} == before_threads + assert {process.pid for process in multiprocessing.active_children()} == before_children + + + + + -def test_v7_timestamp_normalization_is_transactional_idempotent_and_age_safe( + +def test_ten_thousand_adjacent_identical_saves_keep_one_row_and_event( tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, ) -> None: - db_path = tmp_path / "offset-migration.db" - init_store(db_path) - raw_rows = [ - ( - "2026-07-01T01:00:00+02:00", - "valid-old-positive-offset", - ), - ( - "2026-06-30T13:00:00-12:00", - "adversarial-newer-negative-offset", - ), - ("malformed-legacy-time", "malformed-quarantine"), - ( - "0001-01-01T00:00:00+14:00", - "underflow-quarantine", - ), - ( - "9999-12-31T23:59:59-14:00", - "overflow-quarantine", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-year-9999-quarantine", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-malformed-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-different-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-underflow-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-overflow-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legitimate-year-9999-observation", - ), - ( - "2026-07-01T00:30:00+02:00", - "old-but-latest", - ), - ] - with sqlite3.connect(str(db_path)) as conn: - conn.execute("DELETE FROM snapshots") - conn.executemany( - """ - INSERT INTO snapshots ( - host_id, created_at, content_fingerprint, payload - ) VALUES ('offset-host', ?, ?, '{}') - """, - raw_rows, - ) - conn.executemany( - """ - UPDATE snapshots - SET payload = ? - WHERE content_fingerprint = ? - """, - ( - ( - json.dumps({"updated_at": updated_at}, sort_keys=True), - fingerprint, - ) - for fingerprint, updated_at in ( - ("legacy-sentinel-malformed-payload", "not-a-time"), - ( - "legacy-sentinel-different-payload", - "2026-01-01T00:00:00+00:00", - ), - ( - "legacy-sentinel-underflow-payload", - "0001-01-01T00:00:00+14:00", - ), - ( - "legacy-sentinel-overflow-payload", - "9999-12-31T23:59:59-14:00", - ), - ( - "legitimate-year-9999-observation", - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - ), - ) - ), - ) - conn.execute("DROP TABLE store_maintenance_state") - conn.execute("DROP INDEX idx_snapshots_host_newest") - conn.execute("DROP INDEX idx_snapshots_created_host_id") - conn.execute( - "CREATE INDEX idx_snapshots_host_id ON snapshots(host_id)" - ) - conn.execute( - "CREATE INDEX idx_snapshots_created_at ON snapshots(created_at)" - ) - conn.execute( - """ - CREATE INDEX idx_snapshots_content_fingerprint - ON snapshots(content_fingerprint) - """ + db_path = tmp_path / "identical-10000.db" + config = Config(host_id="identical-host", db_path=db_path) + first = datetime.fromisoformat("2026-01-01T00:00:00+00:00") + final_snapshot = None + for index in range(10_000): + final_snapshot = project_from_raw( + config, + timestamp=first + store_sqlite.timedelta(seconds=index), ) - conn.execute("PRAGMA user_version = 7") - - original_registry = store_sqlite.MIGRATIONS - transition = original_registry[7] - - def normalize_then_fail(conn: sqlite3.Connection) -> None: - transition.apply(conn) - raise RuntimeError("controlled timestamp migration interruption") - - interrupted = list(original_registry) - interrupted[7] = store_sqlite.Migration(7, 8, normalize_then_fail) - monkeypatch.setattr(store_sqlite, "MIGRATIONS", tuple(interrupted)) - with pytest.raises( - RuntimeError, - match="controlled timestamp migration interruption", - ): - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == 7 - assert conn.execute( - """ - SELECT created_at, content_fingerprint - FROM snapshots - ORDER BY id - """ - ).fetchall() == raw_rows - assert "store_maintenance_state" not in _table_names(conn) + save_snapshot(db_path, final_snapshot) - monkeypatch.setattr(store_sqlite, "MIGRATIONS", original_registry) - init_store(db_path) + assert final_snapshot is not None with sqlite3.connect(str(db_path)) as conn: - normalized = conn.execute( - """ - SELECT created_at, content_fingerprint - FROM snapshots - ORDER BY id - """ - ).fetchall() - indexes = { - str(row[1]) - for row in conn.execute("PRAGMA index_list(snapshots)").fetchall() - } - conn.execute("BEGIN IMMEDIATE") - store_sqlite._migrate_v7_to_v8_conn(conn) - conn.commit() - rerun = conn.execute( - """ - SELECT created_at, content_fingerprint - FROM snapshots - ORDER BY id - """ + snapshot_rows = conn.execute( + "SELECT created_at, payload FROM snapshots" ).fetchall() + saved_events = conn.execute( + "SELECT COUNT(*) FROM events WHERE event_type = 'snapshot.saved'" + ).fetchone()[0] + integrity = conn.execute("PRAGMA integrity_check").fetchone()[0] + foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert normalized == [ - ( - "2026-06-30T23:00:00+00:00", - "valid-old-positive-offset", - ), - ( - "2026-07-01T01:00:00+00:00", - "adversarial-newer-negative-offset", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "malformed-quarantine", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "underflow-quarantine", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "overflow-quarantine", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-year-9999-quarantine", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-malformed-payload", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-different-payload", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-underflow-payload", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-overflow-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legitimate-year-9999-observation", - ), - ( - "2026-06-30T22:30:00+00:00", - "old-but-latest", - ), - ] - assert ( - store_sqlite._strict_utc_timestamp( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE - ) - is None + assert len(snapshot_rows) == 1 + assert saved_events == 1 + assert snapshot_rows[0][0] == final_snapshot.updated_at + assert json.loads(snapshot_rows[0][1])["updated_at"] == final_snapshot.updated_at + assert integrity == "ok" + assert foreign_keys == [] + + +def test_adjacent_dedupe_is_host_local_and_a_b_a_appends_changed_history( + tmp_path: Path, +) -> None: + db_path = tmp_path / "aba-host-local.db" + config_a = Config(host_id="host-a", db_path=db_path) + config_b = Config(host_id="host-b", db_path=db_path) + at = datetime.fromisoformat("2026-01-01T00:00:00+00:00") + snapshot_a = project_from_raw( + config_a, + workers=[{"id": "worker-a", "name": "A", "status": "active"}], + timestamp=at, ) - assert ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE - > store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE + snapshot_b = project_from_raw( + config_a, + workers=[{"id": "worker-b", "name": "B", "status": "active"}], + timestamp=at + store_sqlite.timedelta(seconds=1), ) - assert rerun == normalized - assert indexes == { - "idx_snapshots_host_newest", - "idx_snapshots_created_host_id", - } - - cleanup = store_sqlite.cleanup_snapshot_retention( - db_path, - retention_days=1, - retention_count=100, - batch_size=100, - now="2026-07-02T00:00:00Z", + snapshot_a_again = project_from_raw( + config_a, + workers=[{"id": "worker-a", "name": "A", "status": "active"}], + timestamp=at + store_sqlite.timedelta(seconds=2), + ) + other_host = project_from_raw( + config_b, + workers=[{"id": "worker-a", "name": "A", "status": "active"}], + timestamp=at + store_sqlite.timedelta(seconds=3), ) + + for snapshot in (snapshot_a, snapshot_b, snapshot_a_again, other_host, other_host): + save_snapshot(db_path, snapshot) + with sqlite3.connect(str(db_path)) as conn: - retained = conn.execute( + history = conn.execute( """ - SELECT content_fingerprint + SELECT host_id, content_fingerprint FROM snapshots ORDER BY id """ ).fetchall() - integrity = conn.execute("PRAGMA integrity_check").fetchone()[0] - foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() + event_count = conn.execute( + "SELECT COUNT(*) FROM events WHERE event_type = 'snapshot.saved'" + ).fetchone()[0] - assert cleanup["deleted"] == 1 - assert retained == [ - ("adversarial-newer-negative-offset",), - ("malformed-quarantine",), - ("underflow-quarantine",), - ("overflow-quarantine",), - ("legacy-year-9999-quarantine",), - ("legacy-sentinel-malformed-payload",), - ("legacy-sentinel-different-payload",), - ("legacy-sentinel-underflow-payload",), - ("legacy-sentinel-overflow-payload",), - ("legitimate-year-9999-observation",), - ("old-but-latest",), + assert history == [ + ("host-a", snapshot_a.content_fingerprint), + ("host-a", snapshot_b.content_fingerprint), + ("host-a", snapshot_a_again.content_fingerprint), + ("host-b", other_host.content_fingerprint), ] - assert integrity == "ok" - assert foreign_keys == [] + assert event_count == 4 -def test_legacy_sentinel_mismatched_payload_recovers_then_remains_monotonic( +def test_stale_same_fingerprint_save_does_not_regress_current_state( tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, ) -> None: - db_path = tmp_path / "quarantine-recovery.db" - config = Config(host_id="quarantine-host", db_path=db_path) - base_at = "2026-01-01T00:00:00+00:00" - fresh_at = "2027-01-01T00:00:00+00:00" - stale_at = "2026-06-01T00:00:00+00:00" - base = _snapshot_with_worker_status( - config, - status="blocked", - observed_at=base_at, - ) + db_path = tmp_path / "stale-identical.db" + config = Config(host_id="stale-host", db_path=db_path) + fresh_at = "2026-01-01T00:10:00+00:00" + stale_at = "2026-01-01T00:05:00+00:00" fresh = _snapshot_with_worker_status( config, status="blocked", @@ -10936,104 +7514,58 @@ def test_legacy_sentinel_mismatched_payload_recovers_then_remains_monotonic( status="blocked", observed_at=stale_at, ) - assert { - base.content_fingerprint, - fresh.content_fingerprint, - stale.content_fingerprint, - } == {base.content_fingerprint} - _save_observation(db_path, base, "positive", base_at) - - with sqlite3.connect(str(db_path)) as conn: - row_id = conn.execute( - """ - SELECT id - FROM snapshots - WHERE host_id = ? - """, - ("quarantine-host",), - ).fetchone()[0] - conn.execute( - """ - UPDATE snapshots - SET created_at = ? - WHERE id = ? - """, - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - row_id, - ), - ) - conn.execute( - """ - UPDATE workers - SET observed_at = 'malformed-legacy-time' - WHERE host_id = ? - """, - ("quarantine-host",), - ) - conn.execute( - """ - UPDATE backend_health - SET observed_at = 'malformed-legacy-time' - WHERE host_id = ? - """, - ("quarantine-host",), - ) - conn.execute("DROP TABLE store_maintenance_state") - conn.execute("DROP INDEX idx_snapshots_host_newest") - conn.execute("DROP INDEX idx_snapshots_created_host_id") - conn.execute("PRAGMA user_version = 7") - - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - "SELECT created_at FROM snapshots WHERE id = ?", - (row_id,), - ).fetchone() == (store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE,) + assert stale.content_fingerprint == fresh.content_fingerprint _save_observation(db_path, fresh, "positive", fresh_at) def current_state() -> tuple[Any, ...]: with sqlite3.connect(str(db_path)) as conn: - retained = conn.execute( - """ - SELECT id, created_at, payload - FROM snapshots - WHERE host_id = ? - """, - ("quarantine-host",), - ).fetchone() return ( - int(retained[0]), - str(retained[1]), - json.loads(retained[2])["updated_at"], conn.execute( """ - SELECT observed_at + SELECT created_at, payload + FROM snapshots + WHERE host_id = ? + ORDER BY id DESC + LIMIT 1 + """, + ("stale-host",), + ).fetchone(), + conn.execute( + """ + SELECT observed_at, payload_json FROM workers WHERE host_id = ? AND worker_id = 'worker-1' """, - ("quarantine-host",), - ).fetchone()[0], + ("stale-host",), + ).fetchone(), conn.execute( """ - SELECT observed_at + SELECT observed_at, payload_json FROM backend_health WHERE host_id = ? AND backend_name = 'herdr' """, - ("quarantine-host",), - ).fetchone()[0], + ("stale-host",), + ).fetchone(), + conn.execute( + """ + SELECT last_seen_at, last_changed_at, signal_count, payload_json + FROM attention_items + WHERE host_id = ? + """, + ("stale-host",), + ).fetchone(), conn.execute( """ - SELECT last_seen_at, signal_count - FROM attention_items + SELECT last_positive_at, last_accepted_at, last_observation_key + FROM attention_lifecycles WHERE host_id = ? """, - ("quarantine-host",), + ("stale-host",), ).fetchone(), conn.execute( "SELECT COUNT(*) FROM snapshots WHERE host_id = ?", - ("quarantine-host",), + ("stale-host",), ).fetchone()[0], conn.execute( """ @@ -11041,24 +7573,84 @@ def current_state() -> tuple[Any, ...]: FROM events WHERE host_id = ? AND event_type = 'snapshot.saved' """, - ("quarantine-host",), + ("stale-host",), ).fetchone()[0], ) - recovered = current_state() - assert recovered == ( - row_id, - fresh_at, - fresh_at, - fresh_at, - fresh_at, - (fresh_at, 2), - 1, - 1, - ) + before = current_state() + attention_calls: list[str] = [] + original_attention = store_sqlite._apply_attention_observation_conn + + def observed_attention( + conn: sqlite3.Connection, + **kwargs: Any, + ) -> None: + attention_calls.append(str(kwargs["observation"].observed_at)) + original_attention(conn, **kwargs) + monkeypatch.setattr( + store_sqlite, + "_apply_attention_observation_conn", + observed_attention, + ) _save_observation(db_path, stale, "positive", stale_at) - assert current_state() == recovered + after = current_state() + + assert attention_calls == [stale_at] + assert after == before + assert after[0][0] == fresh_at + assert after[1][0] == fresh.updated_at + assert after[2][0] == fresh_at + assert after[3][0] == fresh_at + assert after[4][0:2] == (fresh_at, fresh_at) + assert after[5:] == (1, 1) + +def test_snapshot_created_at_is_canonical_utc_and_invalid_input_fails_before_open( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "canonical-created-at.db" + config = Config(host_id="canonical-host", db_path=db_path) + first = project_from_raw( + config, + timestamp=datetime.fromisoformat("2026-07-01T05:30:00+05:30"), + ) + later = project_from_raw( + config, + timestamp=datetime.fromisoformat("2026-06-30T21:00:00-04:00"), + ) + assert first.content_fingerprint == later.content_fingerprint + + save_snapshot(db_path, first) + save_snapshot(db_path, later) + with sqlite3.connect(str(db_path)) as conn: + rows = conn.execute( + "SELECT created_at FROM snapshots" + ).fetchall() + assert rows == [("2026-07-01T01:00:00+00:00",)] + + invalid_path = tmp_path / "invalid" / "store.db" + + def forbidden_connect(*_args: Any, **_kwargs: Any) -> None: + raise AssertionError("invalid timestamp reached sqlite open") + + monkeypatch.setattr(store_sqlite, "_connect", forbidden_connect) + for invalid_timestamp in ( + "malformed-timestamp", + "0001-01-01T00:00:00+14:00", + "9999-12-31T23:59:59-14:00", + ): + invalid = project_empty( + Config(host_id="invalid-host", db_path=invalid_path) + ) + object.__setattr__(invalid, "updated_at", invalid_timestamp) + with pytest.raises(ValueError, match="invalid snapshot updated_at"): + save_snapshot(invalid_path, invalid) + assert not invalid_path.parent.exists() + + + + @@ -12471,546 +9063,104 @@ def interrupt_after_retention(phase: str) -> None: now="2026-02-01T00:00:00+00:00", phase_hook=interrupt_after_retention, ) - - assert result["status"] == "rollback_failed" - assert result["rollback"] == {"status": "failed"} - assert result["snapshots"]["deleted"] == 10 - assert restore_calls == 1 - assert checkpoint_calls == [None] - assert publish_calls == [] - assert backup_path.is_file() - current = db_path.stat() - assert (current.st_dev, current.st_ino) == ( - substitute_stat.st_dev, - substitute_stat.st_ino, - ) - assert hashlib.sha256(db_path.read_bytes()).hexdigest() == substitute_digest - verification = sqlite3.connect( - f"file:{db_path}?mode=ro&immutable=1", - uri=True, - ) - try: - assert verification.execute( - "SELECT value FROM substitute_sentinel" - ).fetchone() == ("must-not-be-overwritten",) - assert verification.execute( - "SELECT COUNT(*) FROM snapshots" - ).fetchone() == (0,) - finally: - verification.close() - assert displaced_source.is_file() - assert not any( - path.name.startswith(".tendwire-sqlite-") - for path in tmp_path.iterdir() - ) - assert set(os.listdir("/proc/self/fd")) == before_fds - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children - - -def test_compact_store_cleans_vacuum_output_when_sqlite_raises_after_creation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path, _private_payload = _seed_compaction_fixture(tmp_path) - backup_path = tmp_path / "vacuum-failure-backup.db" - original_connect = store_sqlite._connect - - class RaiseAfterVacuum: - def __init__(self, connection: sqlite3.Connection) -> None: - self.connection = connection - - def __enter__(self) -> "RaiseAfterVacuum": - self.connection.__enter__() - return self - - def __exit__(self, *args: Any) -> Any: - return self.connection.__exit__(*args) - - def __getattr__(self, name: str) -> Any: - return getattr(self.connection, name) - - def execute( - self, - sql: str, - parameters: Any = (), - ) -> Any: - result = self.connection.execute(sql, parameters) - if sql.lstrip().upper().startswith("VACUUM INTO"): - raise sqlite3.OperationalError("private-vacuum-failure") - return result - - def intercept_connect(*args: Any, **kwargs: Any) -> RaiseAfterVacuum: - return RaiseAfterVacuum(original_connect(*args, **kwargs)) - - monkeypatch.setattr(store_sqlite, "_connect", intercept_connect) - result = compact_store( - db_path, - options=CompactionOptions( - dry_run=False, - acknowledge_offline=True, - backup_path=backup_path, - snapshot_retention_days=14, - snapshot_retention_count=8, - batch_size=2, - ), - now="2026-02-01T00:00:00+00:00", - ) - - assert result["status"] == "rollback_completed" - assert result["rollback"] == {"status": "completed"} - assert backup_path.is_file() - assert not any( - path.name.startswith(".tendwire-sqlite-") - for path in tmp_path.iterdir() - ) - _assert_compaction_logical_evidence(db_path) - - -def test_v7_to_current_conservatively_classifies_legacy_final_and_preserves_state( - tmp_path: Path, -) -> None: - db_path = tmp_path / "complete-preservation.db" - backup_path = tmp_path / "complete-preservation-backup.db" - init_store(db_path) - old_rows = [ - ( - host_id, - f"2025-12-{sequence + 1:02d}T00:00:00+00:00", - f"old-{host_id}-{sequence}", - json.dumps({"old": sequence, "host_id": host_id}, sort_keys=True), - ) - for host_id in ("host-a", "host-b") - for sequence in range(3) - ] - with sqlite3.connect(str(db_path)) as conn: - conn.executemany( - """ - INSERT INTO snapshots ( - host_id, created_at, content_fingerprint, payload - ) VALUES (?, ?, ?, ?) - """, - old_rows, - ) - - host_a_config = Config(host_id="host-a", db_path=db_path) - host_b_config = Config(host_id="host-b", db_path=db_path) - host_a_snapshot = project_from_raw( - host_a_config, - spaces=[{"id": "space-a", "name": "Space A", "status": "active"}], - workers=[ - { - "id": "worker-1", - "name": "Worker One", - "status": "pending", - "space_id": "space-a", - "summary": "human approval required before continuing", - } - ], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-31T00:00:00+00:00", - "counts": {"workers": 1}, - } - ], - timestamp=datetime.fromisoformat("2026-01-31T00:00:00+00:00"), - ) - host_b_snapshot = project_from_raw( - host_b_config, - spaces=[{"id": "space-b", "name": "Space B", "status": "active"}], - workers=[ - { - "id": "worker-b", - "name": "Worker B", - "status": "active", - "space_id": "space-b", - } - ], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-31T00:01:00+00:00", - "counts": {"workers": 1}, - } - ], - timestamp=datetime.fromisoformat("2026-01-31T00:01:00+00:00"), - ) - _save_observation( - db_path, - host_a_snapshot, - "positive", - "2026-01-31T00:00:00+00:00", - ) - _save_observation( - db_path, - host_b_snapshot, - "positive", - "2026-01-31T00:01:00+00:00", - ) - assert merge_turn_content( - db_path, - "host-a", - "worker-1", - { - "user_text": "Preserve this prompt.", - "assistant_final_text": "Preserve this final.", - "complete": True, - "has_open_turn": False, - "source_turn_id": "complete-preservation-turn", - }, - observed_at="2026-01-31T00:02:00+00:00", - ) == 1 - assert upsert_worker_bindings( - db_path, - [ - _worker_binding( - observed_at="2026-01-31T00:00:00+00:00", - expires_at="2027-01-31T00:00:00+00:00", - ) - ], - ) == 1 - reserved = reserve_command_request( - db_path, - host_id="host-a", - request_id="preserved-request", - action="send_instruction", - canonical_version=1, - canonical_fingerprint="preserved-command-fingerprint", - canonical_request_json='{"action":"send_instruction"}', - public_worker_id="worker-1", - pending_result_json='{"status":"pending"}', - now="2026-01-31T00:02:01+00:00", - ) - assert reserved["status"] == "reserved" - started = mark_command_send_started( - db_path, - host_id="host-a", - request_id="preserved-request", - canonical_fingerprint="preserved-command-fingerprint", - owner_token=reserved["owner_token"], - binding_fingerprint="preserved-private-binding", - now="2026-01-31T00:02:02+00:00", - ) - finish_command_request( - db_path, - host_id="host-a", - request_id="preserved-request", - canonical_fingerprint="preserved-command-fingerprint", - owner_token=started["owner_token"], - expected_state="send_started", - terminal_state="accepted", - status=STATUS_ACCEPTED, - result_json='{"status":"accepted","result":"preserved"}', - now="2026-01-31T00:02:03+00:00", + + assert result["status"] == "rollback_failed" + assert result["rollback"] == {"status": "failed"} + assert result["snapshots"]["deleted"] == 10 + assert restore_calls == 1 + assert checkpoint_calls == [None] + assert publish_calls == [] + assert backup_path.is_file() + current = db_path.stat() + assert (current.st_dev, current.st_ino) == ( + substitute_stat.st_dev, + substitute_stat.st_ino, ) - assert store_sqlite.merge_backend_pending( - db_path, - "host-a", - "worker-1", - {"kind": "approval", "safe": "preserved"}, + assert hashlib.sha256(db_path.read_bytes()).hexdigest() == substitute_digest + verification = sqlite3.connect( + f"file:{db_path}?mode=ro&immutable=1", + uri=True, ) - leased = poll_connector_outbox( - db_path, - "host-a", - "attention", - now="2026-01-31T00:03:00+00:00", + try: + assert verification.execute( + "SELECT value FROM substitute_sentinel" + ).fetchone() == ("must-not-be-overwritten",) + assert verification.execute( + "SELECT COUNT(*) FROM snapshots" + ).fetchone() == (0,) + finally: + verification.close() + assert displaced_source.is_file() + assert not any( + path.name.startswith(".tendwire-sqlite-") + for path in tmp_path.iterdir() ) - assert len(leased["items"]) == 1 + assert set(os.listdir("/proc/self/fd")) == before_fds + assert {id(thread) for thread in threading.enumerate()} == before_threads + assert {process.pid for process in multiprocessing.active_children()} == before_children - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT OR REPLACE INTO pending_interactions ( - host_id, pending_id, worker_id, worker_fingerprint, space_id, - kind, status, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json - ) VALUES ( - 'host-a', 'durable-pending', 'worker-1', 'worker-fingerprint', - 'space-a', 'approval', 'pending', - '2026-01-31T00:00:00+00:00', 'pending-fingerprint', - ?, '2026-01-31T00:00:00+00:00', - '{"kind":"approval","safe":"preserved"}' - ) - """, - (host_a_snapshot.content_fingerprint,), - ) - conn.execute( - "CREATE TABLE unrelated_preservation_sentinel (value TEXT NOT NULL)" - ) - conn.execute( - "INSERT INTO unrelated_preservation_sentinel VALUES ('preserved')" - ) - conn.execute("DROP TABLE store_maintenance_state") - conn.execute("DROP INDEX idx_snapshots_host_newest") - conn.execute("DROP INDEX idx_snapshots_created_host_id") - conn.execute( - "CREATE INDEX idx_snapshots_host_id ON snapshots(host_id)" - ) - conn.execute( - "CREATE INDEX idx_snapshots_created_at ON snapshots(created_at)" - ) - conn.execute( - """ - CREATE INDEX idx_snapshots_content_fingerprint - ON snapshots(content_fingerprint) - """ - ) - conn.execute("PRAGMA user_version = 7") - preserved_tables = ( - "commands", - "command_receipts", - "worker_bindings", - "pending_interactions", - "backend_pending", - "attention_items", - "attention_lifecycles", - "spaces", - "workers", - "turns", - "turn_content_revisions", - "turn_content_page_boundaries", - "backend_health", - "connector_outbox", - "connector_deliveries", - "unrelated_preservation_sentinel", - ) - - def logical_evidence() -> dict[str, Any]: - with sqlite3.connect(str(db_path)) as conn: - table_rows = { - table: tuple( - sorted( - ( - tuple(row) - for row in conn.execute( - f"SELECT * FROM {table}" - ).fetchall() - ), - key=repr, - ) - ) - for table in preserved_tables - } - latest = tuple( - conn.execute( - """ - SELECT snapshot.host_id, snapshot.content_fingerprint, - snapshot.payload - FROM snapshots AS snapshot - WHERE snapshot.id = ( - SELECT MAX(newest.id) - FROM snapshots AS newest - WHERE newest.host_id = snapshot.host_id - ) - ORDER BY snapshot.host_id - """ - ).fetchall() - ) - snapshot_count = int( - conn.execute("SELECT COUNT(*) FROM snapshots").fetchone()[0] - ) - return { - "tables": table_rows, - "latest": latest, - "snapshot_count": snapshot_count, - "integrity": conn.execute( - "PRAGMA integrity_check" - ).fetchone()[0], - "foreign_keys": conn.execute( - "PRAGMA foreign_key_check" - ).fetchall(), - } +def test_compact_store_cleans_vacuum_output_when_sqlite_raises_after_creation( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path, _private_payload = _seed_compaction_fixture(tmp_path) + backup_path = tmp_path / "vacuum-failure-backup.db" + original_connect = store_sqlite._connect - before_migration = logical_evidence() - assert all(before_migration["tables"][table] for table in preserved_tables) - assert before_migration["snapshot_count"] == 8 - assert before_migration["integrity"] == "ok" - assert before_migration["foreign_keys"] == [] + class RaiseAfterVacuum: + def __init__(self, connection: sqlite3.Connection) -> None: + self.connection = connection - init_store(db_path) - after_migration = logical_evidence() - for table in preserved_tables: - if table != "connector_outbox": - assert after_migration["tables"][table] == before_migration["tables"][table] - assert after_migration["latest"] == before_migration["latest"] - assert after_migration["snapshot_count"] == before_migration["snapshot_count"] - with sqlite3.connect(str(db_path)) as conn: - legacy_final = conn.execute( - """ - SELECT delivery_kind, status, payload_json - FROM connector_outbox - WHERE host_id = 'host-a' AND connector = 'turn-final' - """ - ).fetchone() - assert legacy_final is not None - assert legacy_final[:2] == ("final_migration_hold", "dead_letter") - assert json.loads(legacy_final[2])["operation"] == "materialize" - assert poll_connector_outbox( - db_path, - "host-a", - "turn-final", - now="2026-01-31T00:04:00+00:00", - )["items"] == [] - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - assert conn.execute( - "SELECT scope FROM store_maintenance_state" - ).fetchone() == ("automatic",) + def __enter__(self) -> "RaiseAfterVacuum": + self.connection.__enter__() + return self - while True: - retention = store_sqlite.cleanup_snapshot_retention( - db_path, - retention_days=14, - retention_count=1, - batch_size=2, - now="2026-02-01T00:00:00+00:00", - ) - if not retention["remaining_candidates"]: - break - after_retention = logical_evidence() - assert after_retention["tables"] == after_migration["tables"] - assert after_retention["latest"] == after_migration["latest"] - assert after_retention["snapshot_count"] == 2 - assert after_retention["integrity"] == "ok" - assert after_retention["foreign_keys"] == [] + def __exit__(self, *args: Any) -> Any: + return self.connection.__exit__(*args) + + def __getattr__(self, name: str) -> Any: + return getattr(self.connection, name) + + def execute( + self, + sql: str, + parameters: Any = (), + ) -> Any: + result = self.connection.execute(sql, parameters) + if sql.lstrip().upper().startswith("VACUUM INTO"): + raise sqlite3.OperationalError("private-vacuum-failure") + return result + + def intercept_connect(*args: Any, **kwargs: Any) -> RaiseAfterVacuum: + return RaiseAfterVacuum(original_connect(*args, **kwargs)) - compacted = compact_store( + monkeypatch.setattr(store_sqlite, "_connect", intercept_connect) + result = compact_store( db_path, options=CompactionOptions( dry_run=False, acknowledge_offline=True, backup_path=backup_path, snapshot_retention_days=14, - snapshot_retention_count=1, + snapshot_retention_count=8, batch_size=2, ), now="2026-02-01T00:00:00+00:00", ) - after_compaction = logical_evidence() - assert compacted["status"] == "completed" - assert compacted["ok"] is True - assert after_compaction == after_retention + + assert result["status"] == "rollback_completed" + assert result["rollback"] == {"status": "completed"} assert backup_path.is_file() + assert not any( + path.name.startswith(".tendwire-sqlite-") + for path in tmp_path.iterdir() + ) + _assert_compaction_logical_evidence(db_path) -def test_store_v8_to_v9_backfills_host_local_sequences_and_paging_indexes( - tmp_path: Path, -) -> None: - db_path = tmp_path / "turn-v8-to-v9.db" - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - CREATE TABLE turns ( - host_id TEXT NOT NULL, - turn_id TEXT NOT NULL, - worker_id TEXT NOT NULL, - worker_fingerprint TEXT, - space_id TEXT, - status TEXT NOT NULL, - kind TEXT NOT NULL, - updated_at TEXT, - fingerprint TEXT NOT NULL, - snapshot_content_fingerprint TEXT NOT NULL, - observed_at TEXT NOT NULL, - payload_json TEXT NOT NULL, - PRIMARY KEY (host_id, turn_id) - ) - """ - ) - conn.execute("PRAGMA user_version=8") - rows = ( - ("host-a", "turn-c", "worker-a", "2026-01-02T00:00:00+00:00"), - ("host-a", "turn-b", "worker-a", "2026-01-01T00:00:00+00:00"), - ("host-a", "turn-a", "worker-a", "2026-01-01T00:00:00+00:00"), - ("host-b", "turn-z", "worker-z", "2026-01-03T00:00:00+00:00"), - ) - for host_id, turn_id, worker_id, observed_at in rows: - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, worker_fingerprint, space_id, - status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json - ) VALUES (?, ?, ?, NULL, NULL, 'active', 'task', ?, '', '', ?, '{}') - """, - (host_id, turn_id, worker_id, observed_at, observed_at), - ) - conn.commit() - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - first = conn.execute( - """ - SELECT host_id, turn_id, list_sequence - FROM turns - ORDER BY host_id, list_sequence - """ - ).fetchall() - indexes = { - str(row[1]): tuple( - str(column[2]) - for column in conn.execute( - f"PRAGMA index_info({row[1]})" - ).fetchall() - ) - for row in conn.execute("PRAGMA index_list(turns)").fetchall() - } - epoch = conn.execute( - "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" - ).fetchone()[0] - host_states = conn.execute( - """ - SELECT host_id, next_sequence, traversal_generation - FROM turn_list_hosts - ORDER BY host_id - """ - ).fetchall() - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - second = conn.execute( - """ - SELECT host_id, turn_id, list_sequence - FROM turns - ORDER BY host_id, list_sequence - """ - ).fetchall() - second_epoch = conn.execute( - "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" - ).fetchone()[0] - second_host_states = conn.execute( - """ - SELECT host_id, next_sequence, traversal_generation - FROM turn_list_hosts - ORDER BY host_id - """ - ).fetchall() - assert first == [ - ("host-a", "turn-a", 1), - ("host-a", "turn-b", 2), - ("host-a", "turn-c", 3), - ("host-b", "turn-z", 1), - ] - assert second == first - assert second_epoch == epoch - assert host_states == second_host_states == [ - ("host-a", 4, 1), - ("host-b", 2, 1), - ] - assert indexes["ux_turns_host_list_sequence"] == ("host_id", "list_sequence") - assert indexes["idx_turns_host_worker_list_sequence"] == ( - "host_id", - "worker_id", - "list_sequence", - "turn_id", - ) def test_all_api_turn_insertions_allocate_unique_immutable_sequences_concurrently( @@ -13030,7 +9180,7 @@ def test_all_api_turn_insertions_allocate_unique_immutable_sequences_concurrentl def insert_observation(index: int) -> None: try: barrier.wait(timeout=10) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -13097,7 +9247,7 @@ def test_turn_list_pagination_is_insert_stable_and_since_discovers_only_new_rows init_store(db_path) save_snapshot(db_path, snapshot) for index, worker in enumerate(snapshot.workers): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker.id, @@ -13123,7 +9273,7 @@ def test_turn_list_pagination_is_insert_stable_and_since_discovers_only_new_rows (host_id,), ).fetchall() } - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, snapshot.workers[0].id, @@ -13180,7 +9330,7 @@ def test_turn_list_tokens_distinguish_invalid_cursor_cursor_expiry_and_since_exp init_store(db_path) save_snapshot(db_path, snapshot) for index, worker in enumerate(snapshot.workers): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "expiry-host", worker.id, @@ -13273,6 +9423,17 @@ def test_turn_list_pages_remain_below_frame_cap_for_over_one_mib_logical_list( index + 1, ), ) + store_sqlite._insert_turn_content_revision_conn( + conn, + host_id=host_id, + turn_id=str(item["id"]), + user_text=None, + assistant_final_text=str(item["assistant_final_text"]), + user_state="absent", + final_state="complete", + created_at=str(item["updated_at"]), + is_current=True, + ) conn.commit() ids: list[str] = [] @@ -13319,7 +9480,7 @@ def test_same_source_completion_is_observation_monotonic_and_never_reopens( init_store(db_path) save_snapshot(db_path, snapshot) source = "same-source" - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13349,7 +9510,7 @@ def test_same_source_completion_is_observation_monotonic_and_never_reopens( "2029-12-31T23:59:59+00:00", "2030-01-01T00:00:00+00:00", ): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13361,7 +9522,7 @@ def test_same_source_completion_is_observation_monotonic_and_never_reopens( }, observed_at=observed_at, ) == 0 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13399,7 +9560,7 @@ def test_same_source_completion_is_observation_monotonic_and_never_reopens( assert revisions[0][1] == 0 assert revisions[1][1] == 1 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13443,23 +9604,27 @@ def test_apply_turn_refresh_rolls_back_turn_pending_and_rejects_stale_binding( turn_target_value="private-pane", ) upsert_worker_bindings(db_path, [binding]) - original_pending_apply = store_sqlite._merge_backend_pending_conn + original_pending_apply = store_sqlite._apply_backend_pending_observation_conn def fail_pending(*args: Any, **kwargs: Any) -> bool: raise RuntimeError("controlled pending failure") - monkeypatch.setattr(store_sqlite, "_merge_backend_pending_conn", fail_pending) + monkeypatch.setattr(store_sqlite, "_apply_backend_pending_observation_conn", fail_pending) with pytest.raises(RuntimeError, match="controlled pending failure"): store_sqlite.apply_turn_refresh( db_path, host_id, worker.id, {"assistant_final_text": "must roll back", "complete": True}, - backend_pending={"question": "must roll back"}, + backend_pending_observation=store_sqlite.PendingObservation( + "open_prompt", + question="must roll back", + revision_digest="must-roll-back", + ), expected_binding=binding, observed_at="2099-01-01T00:00:00+00:00", ) - assert not store_sqlite.list_backend_pending(db_path, host_id) + assert not store_sqlite.pending_payload_from_store(db_path, host_id)["pending_interactions"] assert all( turn.get("assistant_final_text") in (None, "") for turn in turns_payload_from_store(db_path, host_id)["turns"] @@ -13467,7 +9632,7 @@ def fail_pending(*args: Any, **kwargs: Any) -> bool: monkeypatch.setattr( store_sqlite, - "_merge_backend_pending_conn", + "_apply_backend_pending_observation_conn", original_pending_apply, ) with sqlite3.connect(str(db_path)) as conn: @@ -13484,12 +9649,16 @@ def fail_pending(*args: Any, **kwargs: Any) -> bool: host_id, worker.id, {"assistant_final_text": "stale result", "complete": True}, - backend_pending={"question": "stale pending"}, + backend_pending_observation=store_sqlite.PendingObservation( + "open_prompt", + question="stale pending", + revision_digest="stale-pending", + ), expected_binding=binding, observed_at="2099-01-01T00:00:01+00:00", ) assert stale == store_sqlite.TurnRefreshApplyResult(0, False, True) - assert not store_sqlite.list_backend_pending(db_path, host_id) + assert not store_sqlite.pending_payload_from_store(db_path, host_id)["pending_interactions"] assert all( turn.get("assistant_final_text") in (None, "") for turn in turns_payload_from_store(db_path, host_id)["turns"] @@ -13561,7 +9730,7 @@ def barrier_read( ) def write_final() -> None: - results["final"] = merge_turn_content( + results["final"] = apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13575,7 +9744,7 @@ def write_final() -> None: ) def write_late_working() -> None: - results["working"] = merge_turn_content( + results["working"] = apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13643,7 +9812,11 @@ def test_apply_turn_refresh_deadline_while_writer_locked_never_commits_later( host_id, worker_id, {"assistant_final_text": "must never commit", "complete": True}, - backend_pending={"question": "must never persist"}, + backend_pending_observation=store_sqlite.PendingObservation( + "open_prompt", + question="must never persist", + revision_digest="must-never-persist", + ), deadline_monotonic=started + 0.15, observed_at="2099-01-01T00:00:00+00:00", ) @@ -13659,7 +9832,7 @@ def test_apply_turn_refresh_deadline_while_writer_locked_never_commits_later( blocker.rollback() blocker.close() - assert not store_sqlite.list_backend_pending(db_path, host_id) + assert not store_sqlite.pending_payload_from_store(db_path, host_id)["pending_interactions"] assert all( turn.get("assistant_final_text") in (None, "") for turn in turns_payload_from_store(db_path, host_id)["turns"] @@ -13689,73 +9862,23 @@ def cancel_apply_before_commit() -> bool: host_id, worker_id, {"assistant_final_text": "rollback at commit seam", "complete": True}, - backend_pending={"question": "rollback at commit seam"}, + backend_pending_observation=store_sqlite.PendingObservation( + "open_prompt", + question="rollback at commit seam", + revision_digest="rollback-at-commit-seam", + ), cancelled=cancel_apply_before_commit, observed_at="2099-01-01T00:00:02+00:00", ) assert precommit_cancelled.cancelled is True assert apply_checks == 3 - assert not store_sqlite.list_backend_pending(db_path, host_id) + assert not store_sqlite.pending_payload_from_store(db_path, host_id)["pending_interactions"] assert all( turn.get("assistant_final_text") in (None, "") for turn in turns_payload_from_store(db_path, host_id)["turns"] ) -def test_prune_backend_pending_deadline_while_writer_locked_is_non_mutating( - tmp_path: Path, -) -> None: - db_path = tmp_path / "pending-prune-deadline.db" - init_store(db_path) - assert store_sqlite.merge_backend_pending( - db_path, - "prune-host", - "orphan-worker", - {"question": "still present"}, - ) - blocker = store_sqlite._connect(db_path, isolation_level=None) - try: - blocker.execute("BEGIN IMMEDIATE") - started = time.monotonic() - assert store_sqlite.prune_backend_pending( - db_path, - "prune-host", - (), - deadline_monotonic=started + 0.15, - ) == 0 - assert time.monotonic() - started < 1.0 - finally: - blocker.rollback() - blocker.close() - - assert "orphan-worker" in store_sqlite.list_backend_pending( - db_path, - "prune-host", - ) - prune_checks = 0 - - def cancel_prune_before_commit() -> bool: - nonlocal prune_checks - prune_checks += 1 - return prune_checks == 3 - - assert store_sqlite.prune_backend_pending( - db_path, - "prune-host", - (), - cancelled=cancel_prune_before_commit, - ) == 0 - assert prune_checks == 3 - assert "orphan-worker" in store_sqlite.list_backend_pending( - db_path, - "prune-host", - ) - assert store_sqlite.prune_backend_pending( - db_path, - "prune-host", - (), - ) == 1 - assert not store_sqlite.list_backend_pending(db_path, "prune-host") def test_turn_list_filtered_rows_advance_bounded_cursor_without_hiding_public_rows( @@ -13813,6 +9936,17 @@ def test_turn_list_filtered_rows_advance_bounded_cursor_without_hiding_public_ro """, (host_id, public_payload), ) + store_sqlite._insert_turn_content_revision_conn( + conn, + host_id=host_id, + turn_id="public-turn", + user_text="Public prompt", + assistant_final_text="Public answer", + user_state="complete", + final_state="complete", + created_at="1970-01-01T00:00:00+00:00", + is_current=True, + ) first = turns_payload_from_store( db_path, @@ -13852,7 +9986,7 @@ def test_turn_sequence_high_water_never_reuses_deleted_max_and_since_finds_inser init_store(db_path) save_snapshot(db_path, snapshot) for index, worker in enumerate(snapshot.workers): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker.id, @@ -13883,7 +10017,7 @@ def test_turn_sequence_high_water_never_reuses_deleted_max_and_since_finds_inser str(highest_turn), ) conn.commit() - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, snapshot.workers[0].id, @@ -13950,7 +10084,7 @@ def test_turn_list_interior_deletion_expires_cursor_but_not_since_watermark( init_store(db_path) save_snapshot(db_path, snapshot) for index, worker in enumerate(snapshot.workers): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker.id, @@ -14006,7 +10140,7 @@ def test_turn_list_interior_deletion_expires_cursor_but_not_since_watermark( assert insertion_poll["turns"] == [] -def test_source_reconciliation_isolates_stable_owners_and_legacy_no_owner( +def test_source_reconciliation_isolates_stable_owners( tmp_path: Path, ) -> None: db_path = tmp_path / "owner-source-isolation.db" @@ -14017,14 +10151,10 @@ def test_source_reconciliation_isolates_stable_owners_and_legacy_no_owner( def snapshot_for( worker_id: str, - stable_key: str | None, + stable_key: str, observed_at: str, ) -> Snapshot: - meta = ( - {"stable_key": stable_key, "stable_key_version": 1} - if stable_key is not None - else {} - ) + meta = {"stable_key": stable_key, "stable_key_version": 1} return Snapshot( host_id=host_id, updated_at=observed_at, @@ -14060,28 +10190,10 @@ def snapshot_for( "owner two final", "2026-07-13T03:01:01+00:00", ), - ( - snapshot_for( - "shared-worker", - None, - "2026-07-13T03:02:00+00:00", - ), - "legacy same-worker final", - "2026-07-13T03:02:01+00:00", - ), - ( - snapshot_for( - "legacy-worker-b", - None, - "2026-07-13T03:03:00+00:00", - ), - "legacy changed-worker final", - "2026-07-13T03:03:01+00:00", - ), ) for snapshot, final_text, observed_at in cases: save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, snapshot.workers[0].id, @@ -14103,65 +10215,14 @@ def snapshot_for( assert set(source_turns) == { "owner one final", "owner two final", - "legacy same-worker final", - "legacy changed-worker final", } ids = {str(turn["id"]) for turn in source_turns.values()} tokens = {str(turn["source_turn_id"]) for turn in source_turns.values()} - assert len(ids) == len(tokens) == 4 + assert len(ids) == len(tokens) == 2 assert ( source_turns["owner one final"]["meta"]["stable_key"], source_turns["owner two final"]["meta"]["stable_key"], ) == (stable_key_1, stable_key_2) - assert source_turns["legacy same-worker final"]["meta"].get("stable_key") is None - assert source_turns["legacy changed-worker final"]["meta"].get("stable_key") is None - assert ( - source_turns["legacy same-worker final"]["source_turn_id"] - == "turnsrc-fdc56cfa0289296df514b264" - ) - assert source_turns["legacy same-worker final"]["worker_id"] == "shared-worker" - assert source_turns["legacy changed-worker final"]["worker_id"] == "legacy-worker-b" assert raw_source not in json.dumps(payload, sort_keys=True) with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - -def test_v20_to_v21_adds_herdr_turn_watermark_and_provenance_tables( - tmp_path: Path, -) -> None: - db_path = tmp_path / "herdr-turn-v20.db" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=20) - assert conn.execute("PRAGMA user_version").fetchone() == (20,) - assert conn.execute( - """ - SELECT COUNT(*) FROM sqlite_master - WHERE type = 'table' - AND name IN ( - 'herdr_turn_watermarks', - 'herdr_turn_completions' - ) - """ - ).fetchone() == (0,) - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == ( - store_sqlite.STORE_SCHEMA_VERSION, - ) == (28,) - assert { - str(row[0]) - for row in conn.execute( - """ - SELECT name FROM sqlite_master - WHERE type = 'table' - AND name IN ( - 'herdr_turn_watermarks', - 'herdr_turn_completions' - ) - """ - ) - } == {"herdr_turn_watermarks", "herdr_turn_completions"} diff --git a/tests/test_turn_delta.py b/tests/test_turn_delta.py index f93a84e..e1a4f57 100644 --- a/tests/test_turn_delta.py +++ b/tests/test_turn_delta.py @@ -81,6 +81,7 @@ def _insert_turn( summary: str | None = None, updated_at: str = TS, extra: Mapping[str, Any] | None = None, + with_revision: bool = True, ) -> None: payload = _payload( turn_id, @@ -113,6 +114,84 @@ def _insert_turn( sequence, ), ) + if with_revision: + store_sqlite._insert_turn_content_revision_conn( + conn, + host_id=host_id, + turn_id=turn_id, + user_text=None, + assistant_final_text=None, + user_state="absent", + final_state="absent", + created_at=updated_at, + is_current=True, + ) + + +def _seed_current_store(db_path: Path, count: int) -> None: + init_store(db_path) + turns: list[tuple[object, ...]] = [] + revisions: list[tuple[object, ...]] = [] + for index in range(count): + turn_id = f"historical-{index:05d}" + worker_id = f"worker-{index % 8}" + status = "working" if index < 8 else "complete" + payload = _payload( + turn_id, + worker_id=worker_id, + status=status, + summary=f"retained public result {index}", + ) + turns.append( + ( + HOST, + turn_id, + worker_id, + status, + TS, + f"fingerprint-{index}", + f"snapshot-{index}", + TS, + stable_json_dumps(payload), + index + 1, + ) + ) + revisions.append( + ( + HOST, + turn_id, + f"empty-revision-{index}", + "absent", + "absent", + TS, + ) + ) + with sqlite3.connect(str(db_path)) as conn: + conn.executemany( + """ + INSERT INTO turns ( + host_id, turn_id, worker_id, status, kind, updated_at, + fingerprint, snapshot_content_fingerprint, observed_at, + payload_json, list_sequence + ) VALUES (?, ?, ?, ?, 'prompt', ?, ?, ?, ?, ?, ?) + """, + turns, + ) + conn.executemany( + """ + INSERT INTO turn_content_revisions ( + host_id, turn_id, content_revision, + user_text, assistant_final_text, user_state, final_state, + user_char_length, user_byte_length, + final_char_length, final_byte_length, + user_page_count, final_page_count, + is_current, created_at, superseded_at + ) VALUES (?, ?, ?, NULL, NULL, ?, ?, 0, 0, 0, 0, 0, 0, 1, ?, NULL) + """, + revisions, + ) + conn.execute("DELETE FROM turn_change_journal") + conn.commit() def _mutate_turn( @@ -160,47 +239,6 @@ def _tombstone_turn(db_path: Path, turn_id: str, replacement: str | None = None) conn.commit() -def _seed_pre_v18_store(db_path: Path, count: int) -> None: - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys=ON") - store_sqlite._run_migrations(conn, target_version=17) - rows = [] - for index in range(count): - turn_id = f"historical-{index:05d}" - worker_id = f"worker-{index % 8}" - status = "working" if index < 8 else "complete" - payload = _payload( - turn_id, - worker_id=worker_id, - status=status, - summary=f"retained public result {index}", - ) - rows.append( - ( - HOST, - turn_id, - worker_id, - status, - TS, - f"fingerprint-{index}", - f"snapshot-{index}", - TS, - stable_json_dumps(payload), - index + 1, - ) - ) - conn.executemany( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, status, kind, updated_at, - fingerprint, snapshot_content_fingerprint, observed_at, - payload_json, list_sequence - ) VALUES (?, ?, ?, ?, 'prompt', ?, ?, ?, ?, ?, ?) - """, - rows, - ) - conn.commit() - init_store(db_path) def _bootstrap_checkpoint(db_path: Path, *, limit: int = 100) -> str: @@ -259,7 +297,7 @@ def test_goal13_acceptance_1_to_3_ten_thousand_bootstrap_and_unchanged_polls( ) -> None: """10k bootstrap is stable/bounded; unchanged polls traverse no list/content.""" db_path = tmp_path / "ten-thousand.db" - _seed_pre_v18_store(db_path, 10_000) + _seed_current_store(db_path, 10_000) with sqlite3.connect(str(db_path)) as conn: assert conn.execute("SELECT COUNT(*) FROM turn_change_journal").fetchone() == (0,) @@ -331,7 +369,7 @@ def content_reader_spy(*args: Any, **kwargs: Any) -> dict[str, Any]: def test_bootstrap_size_gate_is_independent_of_client_limit_two(tmp_path: Path) -> None: db_path = tmp_path / "limit-two-bootstrap.db" - _seed_pre_v18_store(db_path, 5_000) + _seed_current_store(db_path, 5_000) first = turn_delta_payload_from_store( db_path, @@ -395,6 +433,13 @@ def test_goal13_acceptance_4_working_mutation_is_one_upsert_and_revision_only_ch """, (HOST, "working-turn"), ).fetchone()[0] + conn.execute( + """ + UPDATE turn_content_revisions SET is_current = 0, superseded_at = ? + WHERE host_id = ? AND turn_id = ? AND is_current = 1 + """, + (TS, HOST, "working-turn"), + ) conn.execute( """ UPDATE turn_content_revisions SET is_current = 1 @@ -421,7 +466,7 @@ def test_current_revision_insert_alone_emits_one_upsert(tmp_path: Path) -> None: db_path = tmp_path / "revision-insert.db" init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - _insert_turn(conn, "revision-insert", 1) + _insert_turn(conn, "revision-insert", 1, with_revision=False) conn.commit() checkpoint = _bootstrap_checkpoint(db_path) @@ -489,6 +534,7 @@ def test_single_oversized_change_degrades_and_advances_checkpoint(tmp_path: Path 1, summary="large public descriptor", extra={"meta": {"items": ["x" * 12_000 for _ in range(100)]}}, + with_revision=False, ) store_sqlite._insert_turn_content_revision_conn( conn, @@ -708,7 +754,7 @@ def test_goal13_acceptance_9_token_outcomes_compaction_and_store_epoch_rebuild( batch_size=10, now="2030-07-18T12:00:00+00:00", ) - assert compacted["deleted"] == 2 + assert compacted["deleted"] == 5 assert turn_delta_payload_from_store( db_path, HOST, watermark=checkpoint )["status"] == "expired_watermark" @@ -745,42 +791,6 @@ def test_goal13_acceptance_9_token_outcomes_compaction_and_store_epoch_rebuild( )["status"] == "invalid_watermark" -@pytest.mark.parametrize("source_version", range(18)) -def test_goal13_acceptance_11_every_prior_migration_installs_empty_v18_journal( - tmp_path: Path, - source_version: int, -) -> None: - db_path = tmp_path / f"migration-{source_version}.db" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=source_version) - assert conn.execute("PRAGMA user_version").fetchone() == (source_version,) - store_sqlite._run_migrations(conn, target_version=18) - assert conn.execute("PRAGMA user_version").fetchone() == (18,) - assert conn.execute("SELECT COUNT(*) FROM turn_change_journal").fetchone() == (0,) - columns = tuple( - row[1] for row in conn.execute("PRAGMA table_info(turn_change_journal)") - ) - assert columns == ("seq", "host_id", "turn_id", "op", "changed_at") - epoch = conn.execute( - "SELECT store_epoch FROM turn_change_state WHERE scope = 'turn-delta'" - ).fetchone() - assert epoch is not None and len(str(epoch[0])) >= 32 - trigger_names = { - str(row[0]) - for row in conn.execute( - "SELECT name FROM sqlite_master WHERE type = 'trigger'" - ) - } - assert { - "trg_turn_change_after_insert", - "trg_turn_change_after_update", - "trg_turn_change_after_delete", - "trg_turn_change_revision_current", - "trg_turn_change_revision_insert_current", - "trg_turn_change_journal_no_update", - } <= trigger_names - assert conn.execute("PRAGMA integrity_check").fetchone() == ("ok",) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] def test_goal13_capture_is_trigger_backed_immutable_and_public_minimal(tmp_path: Path) -> None: @@ -821,7 +831,13 @@ def test_delta_page_bytes_do_not_change_when_submission_sweep_fires_mid_request( ) -> None: db_path = tmp_path / "sweep-byte-stability.db" init_store(db_path) - worker = {"id": "worker-0"} + worker = { + "id": "worker-0", + "meta": { + "stable_key": "wsk1_" + ("a" * 64), + "stable_key_version": 1, + }, + } with sqlite3.connect(str(db_path)) as conn: store_sqlite._insert_turn_submission_conn( conn, @@ -861,7 +877,6 @@ def reserve_sweep(*_args: Any, **_kwargs: Any) -> tuple[tuple[str, str, str], bo db_path, HOST, now=current, - turn_model="observed", ) ).encode("utf-8") during = stable_json_dumps( @@ -869,7 +884,6 @@ def reserve_sweep(*_args: Any, **_kwargs: Any) -> tuple[tuple[str, str, str], bo db_path, HOST, now=current, - turn_model="observed", ) ).encode("utf-8") @@ -922,7 +936,7 @@ def journal_since(sequence: int) -> list[tuple[str, str]]: ] before = journal_high() - assert store_sqlite.merge_turn_content( + assert store_sqlite.apply_turn_refresh( db_path, HOST, "worker-0", @@ -934,7 +948,7 @@ def journal_since(sequence: int) -> list[tuple[str, str]]: "has_open_turn": True, }, observed_at="2026-01-01T00:01:00+00:00", - ) == 1 + ).updated == 1 first_rows = journal_since(before) assert first_rows and all(op == "upsert" for _turn_id, op in first_rows) diff --git a/tests/test_turn_submissions.py b/tests/test_turn_submissions.py index 9c1d2dd..c3d04d6 100644 --- a/tests/test_turn_submissions.py +++ b/tests/test_turn_submissions.py @@ -22,10 +22,8 @@ from tendwire.store import sqlite as store_sqlite from tendwire.store.sqlite import ( TURN_SUBMISSION_STATE_TRANSITIONS, - cancel_turn_submission, init_store, is_valid_turn_submission_state_transition, - sweep_expired_turn_submissions, turn_delta_payload_from_store, ) @@ -151,42 +149,8 @@ def test_fresh_v20_store_creates_empty_turn_ledgers_and_all_indexes( _assert_empty_v20_ledgers(conn) -def test_v18_to_v20_migration_matches_fresh_schema(tmp_path: Path) -> None: - fresh_path = tmp_path / "fresh.db" - upgrade_path = tmp_path / "upgrade.db" - init_store(fresh_path) - with sqlite3.connect(str(fresh_path)) as fresh: - fresh_schema = _ledger_schema(fresh) - with sqlite3.connect(str(upgrade_path)) as upgrade: - store_sqlite._run_migrations(upgrade, target_version=18) - assert not set(_LEDGER_TABLES) & { - str(row[0]) - for row in upgrade.execute( - "SELECT name FROM sqlite_master WHERE type = 'table'" - ).fetchall() - } - store_sqlite._run_migrations(upgrade, target_version=20) - _assert_empty_v20_ledgers(upgrade, expected_version=20) - assert _ledger_schema(upgrade) == fresh_schema - - -@pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) -def test_every_prior_schema_upgrades_to_identical_empty_v20_ledgers( - tmp_path: Path, - source_version: int, -) -> None: - fresh_path = tmp_path / f"fresh-{source_version}.db" - upgrade_path = tmp_path / f"upgrade-{source_version}.db" - init_store(fresh_path) - with sqlite3.connect(str(fresh_path)) as fresh: - fresh_schema = _ledger_schema(fresh) - with sqlite3.connect(str(upgrade_path)) as upgrade: - store_sqlite._run_migrations(upgrade, target_version=source_version) - store_sqlite._run_migrations(upgrade) - _assert_empty_v20_ledgers(upgrade) - assert _ledger_schema(upgrade) == fresh_schema def _insert_historical_send_receipt( @@ -218,12 +182,11 @@ def _insert_historical_send_receipt( canonical_fingerprint, canonical_request_json, public_worker_id, state, status, result_json, owner_token_hash, owner_expires_at, binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at, legacy_collision, - legacy_collision_count + terminal_at, updated_at ) VALUES ( 'host-a', ?, 'send_instruction', 1, ?, ?, 'worker-a', ?, ?, '{}', ?, ?, NULL, '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', ?, ?, ?, 0, 0 + '2026-01-01T00:00:01+00:00', ?, ?, ? ) """, ( @@ -245,196 +208,10 @@ def _insert_historical_send_receipt( ) -def test_v18_to_v20_backfills_historical_submission_receipt( - tmp_path: Path, -) -> None: - db_path = tmp_path / "submission-backfill.db" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=18) - _insert_historical_send_receipt( - conn, - request_id="historical-submit", - state="accepted", - status="accepted", - instruction_text=" historical prompt ", - ) - conn.commit() - store_sqlite._run_migrations(conn) - - assert conn.execute( - """ - SELECT owner_key, owner_key_version, instruction_fingerprint, - state, linked_turn_id - FROM turn_submissions - WHERE host_id = 'host-a' AND request_id = 'historical-submit' - """ - ).fetchone() == ( - "legacy-worker:worker-a", - 0, - instruction_fingerprint("historical prompt"), - "submitted", - None, - ) - - -def test_v19_to_v20_repairs_legacy_schema_without_phase2_ledgers( - tmp_path: Path, -) -> None: - db_path = tmp_path / "legacy-v19-without-phase2-ledgers.db" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=18) - _insert_historical_send_receipt( - conn, - request_id="legacy-v19-submit", - state="accepted", - status="accepted", - instruction_text="legacy v19 prompt", - ) - # The deployed pre-Phase-2 lineage used version 19 without creating - # the two Phase-2 ledgers. - conn.execute("PRAGMA user_version = 19") - conn.commit() - - assert not store_sqlite._table_columns(conn, "turn_submissions") - assert not store_sqlite._table_columns(conn, "turn_supersessions") - - store_sqlite._run_migrations(conn) - assert conn.execute("PRAGMA user_version").fetchone() == ( - store_sqlite.STORE_SCHEMA_VERSION, - ) - assert conn.execute( - """ - SELECT owner_key, state - FROM turn_submissions - WHERE host_id = 'host-a' AND request_id = 'legacy-v19-submit' - """ - ).fetchone() == ("legacy-worker:worker-a", "submitted") - assert store_sqlite._table_columns(conn, "turn_supersessions") - - -def test_v19_to_v20_backfills_legacy_tombstone_alias(tmp_path: Path) -> None: - db_path = tmp_path / "supersession-backfill.db" - _seed_link_worker(db_path) - canonical_turn_id = _observe_link_turn( - db_path, - source_turn_id="canonical-source", - ) - legacy_turn_id = "turn-" + ("2" * 24) - observed_at = "2026-02-01T12:00:01+00:00" - legacy_payload = { - "id": legacy_turn_id, - "host_id": "host-a", - "worker_id": "worker-a", - "status": "done", - "kind": "task", - "source": "command", - "origin_command_id": "historical-submit", - "complete": True, - "has_open_turn": False, - "updated_at": observed_at, - "superseded_at": observed_at, - "superseded_by_turn_id": canonical_turn_id, - } - with sqlite3.connect(str(db_path)) as conn: - next_sequence = int( - conn.execute( - "SELECT COALESCE(MAX(list_sequence), 0) + 1 FROM turns" - ).fetchone()[0] - ) - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, worker_fingerprint, space_id, - status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, - list_sequence - ) VALUES ( - 'host-a', ?, 'worker-a', NULL, NULL, 'done', 'task', ?, '', - '', ?, ?, ? - ) - """, - ( - legacy_turn_id, - observed_at, - observed_at, - json.dumps(legacy_payload, sort_keys=True, separators=(",", ":")), - next_sequence, - ), - ) - conn.execute("DELETE FROM turn_supersessions") - conn.execute("PRAGMA user_version = 19") - conn.commit() - store_sqlite._run_migrations(conn) - - assert conn.execute( - """ - SELECT canonical_turn_id, reason - FROM turn_supersessions - WHERE host_id = 'host-a' AND superseded_turn_id = ? - """, - (legacy_turn_id,), - ).fetchone() == (canonical_turn_id, "phase1_migration") - - -@pytest.mark.parametrize( - ("receipt_state", "receipt_status", "expected"), - ( - (None, None, None), - ({"state": "accepted"}, [], None), - ("unknown", "accepted", None), - ("reserved", "pending", None), - ("send_started", "pending", "send_started"), - ("accepted", "accepted", "submitted"), - ("uncertain", "request_state_uncertain", "uncertain"), - ("rejected", "cancelled", "cancelled"), - ("accepted", "purged", None), - ), -) -def test_backfill_submission_state_fails_closed_for_malformed_receipt_values( - receipt_state: object, - receipt_status: object, - expected: str | None, -) -> None: - assert ( - store_sqlite._backfill_submission_state(receipt_state, receipt_status) - == expected - ) - - -@pytest.mark.parametrize( - "canonical_request_json", - ( - None, - "not-json", - json.dumps([]), - json.dumps({"action": "observe", "instruction": {"text": "hello"}}), - json.dumps({"action": "send_instruction"}), - json.dumps({"action": "send_instruction", "instruction": []}), - json.dumps({"action": "send_instruction", "instruction": {}}), - json.dumps( - {"action": "send_instruction", "instruction": {"text": 42}} - ), - ), -) -def test_receipt_instruction_text_rejects_malformed_receipt_shapes( - canonical_request_json: object, -) -> None: - assert store_sqlite._receipt_instruction_text(canonical_request_json) is None -def test_receipt_instruction_text_accepts_valid_send_instruction() -> None: - canonical_request_json = json.dumps( - { - "action": "send_instruction", - "instruction": {"text": "ship the release"}, - } - ) - assert ( - store_sqlite._receipt_instruction_text(canonical_request_json) - == "ship the release" - ) def test_turn_submission_state_transition_table() -> None: @@ -538,75 +315,8 @@ def _insert_submission( ) -def test_submission_expiry_sweeper_expires_only_old_unlinked_rows( - tmp_path: Path, -) -> None: - db_path = tmp_path / "expiry.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - _insert_submission( - conn, - request_id="old", - state="submitted", - hard_expires_at="2026-01-02T00:00:00+00:00", - ) - _insert_submission( - conn, - request_id="future", - state="uncertain", - hard_expires_at="2026-03-01T00:00:00+00:00", - ) - - assert sweep_expired_turn_submissions( - db_path, - host_id="host-a", - now="2026-02-01T00:00:00+00:00", - ) == 1 - - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - """ - SELECT request_id, state, terminal_at - FROM turn_submissions ORDER BY request_id - """ - ).fetchall() - assert rows == [ - ("future", "uncertain", None), - ("old", "expired", "2026-02-01T00:00:00+00:00"), - ] -def test_submission_cancellation_is_terminal_and_idempotent(tmp_path: Path) -> None: - db_path = tmp_path / "cancel.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - _insert_submission( - conn, - request_id="cancel-me", - state="send_started", - hard_expires_at="2026-03-01T00:00:00+00:00", - ) - - assert cancel_turn_submission( - db_path, - host_id="host-a", - request_id="cancel-me", - now="2026-02-01T00:00:00+00:00", - ) - assert not cancel_turn_submission( - db_path, - host_id="host-a", - request_id="cancel-me", - now="2026-02-01T00:00:01+00:00", - ) - - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT state, terminal_at FROM turn_submissions - WHERE host_id = 'host-a' AND request_id = 'cancel-me' - """ - ).fetchone() == ("cancelled", "2026-02-01T00:00:00+00:00") def _seed_link_worker( @@ -679,30 +389,6 @@ def _insert_link_submission( ) -def _set_link_worker_prod_shape( - db_path: Path, - *, - worker_id: str = "worker-a", - host_id: str = "host-a", - explicit_null: bool = False, -) -> None: - version_update = ( - "json_set(payload_json, '$.meta.stable_key_version', NULL)" - if explicit_null - else "json_remove(payload_json, '$.meta.stable_key_version')" - ) - with sqlite3.connect(str(db_path)) as conn: - updated = conn.execute( - f""" - UPDATE workers - SET payload_json = {version_update} - WHERE host_id = ? AND worker_id = ? - """, - (host_id, worker_id), - ) - assert updated.rowcount == 1 - - def _observe_link_turn( db_path: Path, *, @@ -711,7 +397,6 @@ def _observe_link_turn( host_id: str = "host-a", instruction_text: str = "hello", observed_at: str = "2026-02-01T12:00:00+00:00", - turn_model: str = "dual", ) -> str: result = store_sqlite.apply_turn_refresh( db_path, @@ -725,7 +410,6 @@ def _observe_link_turn( "has_open_turn": False, }, observed_at=observed_at, - turn_model=turn_model, ) assert result.updated in {0, 1} with sqlite3.connect(str(db_path)) as conn: @@ -769,7 +453,7 @@ def _submission_rows(db_path: Path) -> list[tuple[str, str, str | None]]: ] -def test_shadow_linker_handles_both_race_directions_without_changing_turn_id( +def test_observed_submission_linker_handles_both_race_directions_without_changing_turn_id( tmp_path: Path, ) -> None: submission_first = tmp_path / "submission-first.db" @@ -815,198 +499,13 @@ def test_shadow_linker_handles_both_race_directions_without_changing_turn_id( ] -@pytest.mark.parametrize( - ("order", "explicit_null"), - (("submission-first", True), ("observation-first", False)), -) -def test_observed_linker_accepts_prod_shape_turn_owner_version( - tmp_path: Path, - order: str, - explicit_null: bool, -) -> None: - db_path = tmp_path / f"prod-shape-{order}.db" - owner_key = _seed_link_worker(db_path) - assert store_sqlite._turn_submission_owner_identity( - { - "id": "worker-a", - "meta": {"stable_key": owner_key, "stable_key_version": 1}, - } - ) == (owner_key, 1) - _set_link_worker_prod_shape(db_path, explicit_null=explicit_null) - - if order == "submission-first": - _insert_link_submission( - db_path, - request_id=order, - owner_key=owner_key, - ) - observed_turn_id = _observe_link_turn( - db_path, - source_turn_id=f"prod-shape-{order}-source", - turn_model="observed", - ) - if order == "observation-first": - _insert_link_submission( - db_path, - request_id=order, - owner_key=owner_key, - ) - - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - - assert _submission_rows(db_path) == [(order, "linked", observed_turn_id)] - with sqlite3.connect(str(db_path)) as conn: - turn_shape = conn.execute( - """ - SELECT json_extract(turns.payload_json, '$.meta.stable_key'), - json_extract(turns.payload_json, '$.meta.stable_key_version'), - json_extract(turns.payload_json, '$.stable_key_version'), - json_extract(turns.payload_json, '$.user_text'), - revisions.user_text - FROM turns - JOIN turn_content_revisions AS revisions - ON revisions.host_id = turns.host_id - AND revisions.turn_id = turns.turn_id - AND revisions.is_current = 1 - WHERE turns.host_id = 'host-a' AND turns.turn_id = ? - """, - (observed_turn_id,), - ).fetchone() - turn_payload = store_sqlite._json_object( - conn.execute( - """ - SELECT payload_json FROM turns - WHERE host_id = 'host-a' AND turn_id = ? - """, - (observed_turn_id,), - ).fetchone()[0] - ) - assert turn_shape == (owner_key, None, None, None, "hello") - turn_worker = {"id": "worker-a", "meta": turn_payload.get("meta")} - assert store_sqlite._turn_submission_owner_identity(turn_worker) == ( - "legacy-worker:worker-a", - 0, - ) - assert store_sqlite._turn_link_candidate_owner_identity(turn_worker) == ( - owner_key, - 1, - ) -@pytest.mark.parametrize( - ("submission_count", "observation_count"), - ((2, 1), (1, 2), (2, 2)), -) -def test_observed_linker_prod_shape_turns_still_fail_closed_on_ambiguity( - tmp_path: Path, - submission_count: int, - observation_count: int, -) -> None: - db_path = tmp_path / ( - f"prod-shape-ambiguous-{submission_count}-{observation_count}.db" - ) - owner_key = _seed_link_worker(db_path) - _set_link_worker_prod_shape(db_path) - for index in range(submission_count): - _insert_link_submission( - db_path, - request_id=f"prod-shape-{index}", - owner_key=owner_key, - ) - - for index in range(observation_count): - _observe_link_turn( - db_path, - source_turn_id=f"prod-shape-ambiguous-source-{index}", - turn_model="observed", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - - rows = _submission_rows(db_path) - assert [state for _request, state, _turn in rows] == [ - "ambiguous" - ] * submission_count - assert all(linked_turn_id is None for _request, _state, linked_turn_id in rows) - - -def test_observed_linker_prod_shape_turns_keep_owner_hash_isolated( - tmp_path: Path, -) -> None: - db_path = tmp_path / "prod-shape-owner-isolation.db" - first_owner = _seed_link_worker(db_path) - second_owner = "wsk1_" + ("b" * 64) - snapshot = project_from_raw( - Config(host_id="host-a", db_path=db_path), - workers=[ - { - "id": "worker-a", - "name": "worker-a", - "status": "active", - "meta": { - "stable_key": first_owner, - "stable_key_version": 1, - }, - }, - { - "id": "worker-b", - "name": "worker-b", - "status": "active", - "meta": { - "stable_key": second_owner, - "stable_key_version": 1, - }, - }, - ], - ) - store_sqlite.save_snapshot(db_path, snapshot) - _set_link_worker_prod_shape(db_path, worker_id="worker-a") - _set_link_worker_prod_shape(db_path, worker_id="worker-b", explicit_null=True) - _insert_link_submission( - db_path, - request_id="first-owner", - owner_key=first_owner, - link_expires_at="2026-02-01T12:01:00+00:00", - ) - _observe_link_turn( - db_path, - worker_id="worker-b", - source_turn_id="wrong-owner-source", - turn_model="observed", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - assert _submission_rows(db_path) == [("first-owner", "submitted", None)] - matching_turn_id = _observe_link_turn( - db_path, - worker_id="worker-a", - source_turn_id="first-owner-source", - observed_at="2026-02-01T12:00:02+00:00", - turn_model="observed", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:03+00:00", - ) - assert _submission_rows(db_path) == [ - ("first-owner", "linked", matching_turn_id) - ] -def test_shadow_linker_failure_keeps_observation_and_rolls_back_link_attempt( +def test_observed_submission_linker_failure_keeps_observation_and_rolls_back_link_attempt( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -1034,7 +533,7 @@ def fail_after_partial_settlement( monkeypatch.setattr( store_sqlite, - "settle_submission_links_conn", + "_settle_submission_links_conn", fail_after_partial_settlement, ) turn_id = _observe_link_turn( @@ -1067,7 +566,7 @@ def fail_after_partial_settlement( ) -def test_shadow_linker_never_uses_unverified_send_started_as_turn_evidence( +def test_observed_submission_linker_never_uses_unverified_send_started_as_turn_evidence( tmp_path: Path, ) -> None: linked_path = tmp_path / "send-started-linked.db" @@ -1105,7 +604,7 @@ def test_shadow_linker_never_uses_unverified_send_started_as_turn_evidence( ] -def test_shadow_linker_waits_for_window_close_before_failing_closed( +def test_observed_submission_linker_waits_for_window_close_before_failing_closed( tmp_path: Path, ) -> None: db_path = tmp_path / "delayed.db" @@ -1132,122 +631,11 @@ def test_shadow_linker_waits_for_window_close_before_failing_closed( assert _submission_rows(db_path) == [("delayed", "ambiguous", None)] -@pytest.mark.parametrize("turn_model", ("dual", "observed")) -def test_single_open_submission_links_on_first_sweep_after_observation( - tmp_path: Path, - turn_model: str, -) -> None: - db_path = tmp_path / f"instant-single-{turn_model}.db" - owner_key = _seed_link_worker(db_path) - _set_link_worker_prod_shape(db_path) - _insert_link_submission( - db_path, - request_id=f"instant-single-{turn_model}", - owner_key=owner_key, - link_expires_at="2026-02-01T12:01:00+00:00", - ) - observed_at = "2026-02-01T12:00:03+00:00" - turn_id = _observe_link_turn( - db_path, - source_turn_id=f"instant-single-{turn_model}-source", - observed_at=observed_at, - turn_model=turn_model, - ) - assert _submission_rows(db_path) == [ - (f"instant-single-{turn_model}", "submitted", None) - ] - - swept_at = "2026-02-01T12:00:05+00:00" - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now=swept_at, - ) - - assert _submission_rows(db_path) == [ - (f"instant-single-{turn_model}", "linked", turn_id) - ] - with sqlite3.connect(str(db_path)) as conn: - linked_at = conn.execute( - """ - SELECT linked_at FROM turn_submissions - WHERE host_id = 'host-a' AND request_id = ? - """, - (f"instant-single-{turn_model}",), - ).fetchone()[0] - assert linked_at == swept_at - assert ( - datetime.fromisoformat(linked_at) - - datetime.fromisoformat(observed_at) - ).total_seconds() == 2 - assert linked_at < "2026-02-01T12:01:00+00:00" - -def test_two_open_same_fingerprint_submissions_do_not_instant_link( - tmp_path: Path, -) -> None: - db_path = tmp_path / "two-open-no-instant.db" - owner_key = _seed_link_worker(db_path) - for index in range(2): - _insert_link_submission( - db_path, - request_id=f"two-open-{index}", - owner_key=owner_key, - link_expires_at="2026-02-01T12:01:00+00:00", - ) - _observe_link_turn( - db_path, - source_turn_id="two-open-source", - observed_at="2026-02-01T12:00:03+00:00", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:05+00:00", - ) - assert _submission_rows(db_path) == [ - ("two-open-0", "submitted", None), - ("two-open-1", "submitted", None), - ] -def test_disconnected_singleton_component_still_links_immediately( - tmp_path: Path, -) -> None: - db_path = tmp_path / "disconnected-singleton-instant.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="old-disconnected", - owner_key=owner_key, - link_not_before="2026-02-01T11:00:00+00:00", - link_expires_at="2026-02-01T11:01:00+00:00", - ) - _insert_link_submission( - db_path, - request_id="live-singleton", - owner_key=owner_key, - link_expires_at="2026-02-01T12:01:00+00:00", - ) - turn_id = _observe_link_turn( - db_path, - source_turn_id="live-singleton-source", - observed_at="2026-02-01T12:00:03+00:00", - ) - - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:05+00:00", - ) - - assert _submission_rows(db_path) == [ - ("live-singleton", "linked", turn_id), - ("old-disconnected", "expired", None), - ] - def test_manual_same_text_turn_links_single_open_submission( tmp_path: Path, @@ -1274,169 +662,12 @@ def test_manual_same_text_turn_links_single_open_submission( ] -def test_stale_send_started_submission_uses_windowed_settlement( - tmp_path: Path, -) -> None: - db_path = tmp_path / "stale-send-started.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="stale-send-started", - owner_key=owner_key, - state="send_started", - link_expires_at="2026-02-01T12:01:00+00:00", - ) - observed_at = ( - datetime.fromisoformat("2026-02-01T12:00:00+00:00") - + timedelta( - seconds=store_sqlite.SUBMISSION_SEND_ACK_TIMEOUT_SECONDS + 1 - ) - ).isoformat() - _observe_link_turn( - db_path, - source_turn_id="stale-send-started-source", - observed_at=observed_at, - ) - - assert _submission_rows(db_path) == [ - ("stale-send-started", "send_started", None) - ] - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:30+00:00", - ) - assert _submission_rows(db_path) == [ - ("stale-send-started", "send_started", None) - ] - - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:01:00+00:00", - ) - assert _submission_rows(db_path) == [ - ("stale-send-started", "expired", None) - ] - - -def test_ambiguous_component_is_stamped_at_window_close_not_hard_ttl( - tmp_path: Path, -) -> None: - db_path = tmp_path / "ambiguous-at-window-close.db" - owner_key = _seed_link_worker(db_path) - for index in range(2): - _insert_link_submission( - db_path, - request_id=f"ambiguous-at-close-{index}", - owner_key=owner_key, - link_expires_at="2026-02-01T12:01:00+00:00", - hard_expires_at="2026-02-02T12:00:00+00:00", - ) - _observe_link_turn( - db_path, - source_turn_id="ambiguous-at-close-source", - observed_at="2026-02-01T12:00:03+00:00", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:01:00+00:00", - ) - assert _submission_rows(db_path) == [ - ("ambiguous-at-close-0", "ambiguous", None), - ("ambiguous-at-close-1", "ambiguous", None), - ] - with sqlite3.connect(str(db_path)) as conn: - stamps = conn.execute( - """ - SELECT terminal_at, hard_expires_at - FROM turn_submissions - WHERE host_id = 'host-a' - ORDER BY request_id - """ - ).fetchall() - assert stamps == [ - ("2026-02-01T12:01:00+00:00", "2026-02-02T12:00:00+00:00"), - ("2026-02-01T12:01:00+00:00", "2026-02-02T12:00:00+00:00"), - ] - - -def test_lone_submission_without_candidate_expires_at_window_close( - tmp_path: Path, -) -> None: - db_path = tmp_path / "no-candidate-at-window-close.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="no-candidate-at-window-close", - owner_key=owner_key, - link_expires_at="2026-02-01T12:01:00+00:00", - hard_expires_at="2026-02-02T12:00:00+00:00", - ) - - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:30+00:00", - ) - assert _submission_rows(db_path) == [ - ("no-candidate-at-window-close", "submitted", None) - ] - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:01:00+00:00", - ) - - assert _submission_rows(db_path) == [ - ("no-candidate-at-window-close", "expired", None) - ] - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT terminal_at, hard_expires_at - FROM turn_submissions - WHERE host_id = 'host-a' - AND request_id = 'no-candidate-at-window-close' - """ - ).fetchone() == ( - "2026-02-01T12:01:00+00:00", - "2026-02-02T12:00:00+00:00", - ) -def test_each_no_candidate_component_expires_at_its_window_close( - tmp_path: Path, -) -> None: - db_path = tmp_path / "multiple-no-candidate-components.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="closed-no-candidate", - owner_key=owner_key, - link_expires_at="2026-02-01T12:00:10+00:00", - ) - _insert_link_submission( - db_path, - request_id="open-no-candidate", - owner_key=owner_key, - link_not_before="2026-02-01T12:01:00+00:00", - link_expires_at="2026-02-01T12:02:00+00:00", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:01:30+00:00", - ) - assert _submission_rows(db_path) == [ - ("closed-no-candidate", "expired", None), - ("open-no-candidate", "submitted", None), - ] def test_turn_observed_outside_link_window_never_links( @@ -1474,7 +705,7 @@ def test_turn_observed_outside_link_window_never_links( ("submission_count", "observation_count"), ((2, 1), (1, 2), (2, 2)), ) -def test_shadow_linker_marks_larger_identical_components_ambiguous( +def test_observed_submission_linker_marks_larger_identical_components_ambiguous( tmp_path: Path, submission_count: int, observation_count: int, @@ -1504,7 +735,7 @@ def test_shadow_linker_marks_larger_identical_components_ambiguous( assert all(linked_turn_id is None for _request, _state, linked_turn_id in rows) -def test_submission_linker_isolates_owners_and_legacy_alias_links( +def test_submission_linker_isolates_stable_owners( tmp_path: Path, ) -> None: first_path = tmp_path / "owners.db" @@ -1544,17 +775,6 @@ def test_submission_linker_isolates_owners_and_legacy_alias_links( ("owner-b", "linked", second_turn_id), ] - legacy_path = tmp_path / "legacy.db" - owner_key = _seed_link_worker(legacy_path) - _insert_link_submission(legacy_path, request_id="legacy", owner_key=owner_key) - legacy_turn_id = _observe_link_turn( - legacy_path, - source_turn_id="legacy-source", - turn_model="legacy", - ) - assert _submission_rows(legacy_path) == [("legacy", "linked", legacy_turn_id)] - - def test_goal13_delta_is_unperturbed_when_observed_turn_links_later( tmp_path: Path, ) -> None: @@ -1647,7 +867,6 @@ def test_idle_observation_first_submission_links_from_lazy_turn_read( now=datetime.fromisoformat( "2026-02-01T12:00:01+00:00" ).timestamp(), - turn_model="observed", ) assert payload["turns"] @@ -1656,120 +875,12 @@ def test_idle_observation_first_submission_links_from_lazy_turn_read( ] -def test_observed_prod_shape_sweep_never_runs_public_sanitizers( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "prod-shape-no-public-sanitize.db" - owner_key = _seed_link_worker(db_path) - _set_link_worker_prod_shape(db_path) - observed_turn_id = _observe_link_turn( - db_path, - source_turn_id="prod-shape-no-public-sanitize-source", - turn_model="observed", - ) - _insert_link_submission( - db_path, - request_id="prod-shape-no-public-sanitize", - owner_key=owner_key, - ) - contains_calls = 0 - sanitize_calls = 0 - original_contains = core_turns._contains_forbidden_public_text - original_sanitize = core_turns.sanitize_public_text - - def record_contains(value: str) -> bool: - nonlocal contains_calls - contains_calls += 1 - return original_contains(value) - - def record_sanitize(value: object, **kwargs: object) -> str: - nonlocal sanitize_calls - sanitize_calls += 1 - return original_sanitize(value, **kwargs) - - monkeypatch.setattr( - core_turns, - "_contains_forbidden_public_text", - record_contains, - ) - monkeypatch.setattr(core_turns, "sanitize_public_text", record_sanitize) - - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - - assert contains_calls == 0 - assert sanitize_calls == 0 - assert _submission_rows(db_path) == [ - ("prod-shape-no-public-sanitize", "linked", observed_turn_id) - ] - - -def test_submission_link_sweep_backs_off_until_matching_observation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "submission-link-backoff.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="submission-link-backoff", - owner_key=owner_key, - ) - candidate_calls = 0 - original_candidates = store_sqlite._submission_link_candidate_turns_conn - - def record_candidates(*args: object, **kwargs: object): - nonlocal candidate_calls - candidate_calls += 1 - return original_candidates(*args, **kwargs) - - monkeypatch.setattr( - store_sqlite, - "_submission_link_candidate_turns_conn", - record_candidates, - ) - - for _ in range(2): - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T11:59:30+00:00", - ) - assert candidate_calls == 1 - - # Production observations can persist the authenticated stable owner key - # without its version marker. That shape cannot use the observation-time - # direct settlement path, so this specifically proves that the observation - # re-arms the component for the next sweep. - _set_link_worker_prod_shape(db_path) - observed_turn_id = _observe_link_turn( - db_path, - source_turn_id="submission-link-backoff-source", - observed_at="2026-02-01T12:00:00+00:00", - turn_model="observed", - ) - assert candidate_calls == 1 - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - assert candidate_calls == 2 - assert _submission_rows(db_path) == [ - ("submission-link-backoff", "linked", observed_turn_id) - ] -@pytest.mark.parametrize("turn_model", sorted(store_sqlite.TURN_MODELS)) -def test_turn_alias_resolves_public_content_and_final_root_under_every_model( +def test_turn_alias_resolves_public_content_and_final_root( tmp_path: Path, - turn_model: str, ) -> None: db_path = tmp_path / "alias-lookup.db" _seed_link_worker(db_path) @@ -1808,7 +919,6 @@ def test_turn_alias_resolves_public_content_and_final_root_under_every_model( turn_id=legacy_turn_id, content_revision=revision, field="assistant_final_text", - turn_model=turn_model, ) assert page["turn_id"] == canonical_turn_id assert page["text"] == "answer for alias-source" @@ -1828,7 +938,6 @@ def test_turn_alias_resolves_public_content_and_final_root_under_every_model( presentation_version="alias-aware-v1", part_count=1, source_ref=leased["ref"], - turn_model=turn_model, now="2026-02-01T12:00:03+00:00", ) assert begun["ok"] is True @@ -1842,140 +951,6 @@ def test_turn_alias_resolves_public_content_and_final_root_under_every_model( ).fetchone() == (canonical_turn_id,) -def test_link_candidate_owner_identity_normalizes_only_missing_version() -> None: - # Guard-regression for the prod-shape normalization: ONLY a syntactically - # valid stable key whose version is exactly None is treated as v1; every - # other shape falls through to the strict submission-side identity. - valid_key = "wsk1_" + ("a" * 64) - normalized = store_sqlite._turn_link_candidate_owner_identity( - {"id": "worker-x", "meta": {"stable_key": valid_key, "stable_key_version": None}} - ) - assert normalized == (valid_key, 1) - - fallthrough_cases = [ - {"stable_key": valid_key, "stable_key_version": 2}, - {"stable_key": valid_key, "stable_key_version": 0}, - {"stable_key": valid_key, "stable_key_version": "1"}, - {"stable_key": valid_key, "stable_key_version": True}, - {"stable_key": "wsk1_short", "stable_key_version": None}, - {"stable_key": "not-a-key", "stable_key_version": None}, - {"stable_key": "", "stable_key_version": None}, - {"stable_key": None, "stable_key_version": None}, - ] - for meta in fallthrough_cases: - result = store_sqlite._turn_link_candidate_owner_identity( - {"id": "worker-x", "meta": meta} - ) - assert result == ("legacy-worker:worker-x", 0), meta - - -def test_observed_lazy_delta_sweep_links_first_poll_after_observation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Mirror the idle-pane production canary timeline through turn.delta.""" - db_path = tmp_path / "observed-live-timeline.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="observed-live-timeline", - owner_key=owner_key, - link_not_before="2026-07-22T12:01:23+00:00", - link_expires_at="2026-07-22T12:03:23+00:00", - hard_expires_at="2026-07-23T12:02:23+00:00", - ) - _set_link_worker_prod_shape(db_path) - - # Herdres polls turn.delta every ~5s. The early empty sweeps exercise the - # component backoff before the idle pane produces its first observation. - for second in range(23, 36, 5): - turn_delta_payload_from_store( - db_path, - "host-a", - now=datetime.fromisoformat( - f"2026-07-22T12:01:{second:02d}+00:00" - ).timestamp(), - turn_model="observed", - ) - for second in range(38, 60, 5): - turn_delta_payload_from_store( - db_path, - "host-a", - now=datetime.fromisoformat( - f"2026-07-22T12:01:{second:02d}+00:00" - ).timestamp(), - turn_model="observed", - ) - for second in range(3, 34, 5): - turn_delta_payload_from_store( - db_path, - "host-a", - now=datetime.fromisoformat( - f"2026-07-22T12:02:{second:02d}+00:00" - ).timestamp(), - turn_model="observed", - ) - - candidate_calls = 0 - original_candidates = store_sqlite._submission_link_candidate_turns_conn - - def record_candidates(*args: object, **kwargs: object): - nonlocal candidate_calls - candidate_calls += 1 - return original_candidates(*args, **kwargs) - - monkeypatch.setattr( - store_sqlite, - "_submission_link_candidate_turns_conn", - record_candidates, - ) - # The live observed prompt retained Herdr's trailing U+0001 framing byte. - # Submission input rejects that byte, so matching must ignore it only at - # the observation edge and re-arm the original component key. - observed_turn_id = _observe_link_turn( - db_path, - source_turn_id="turn-803b8be4224ccec08a20c794", - instruction_text="hello\x01", - observed_at="2026-07-22T12:02:36+00:00", - turn_model="observed", - ) - assert _submission_rows(db_path) == [ - ("observed-live-timeline", "submitted", None) - ] - candidate_calls_after_observation = candidate_calls - - linked_page = turn_delta_payload_from_store( - db_path, - "host-a", - now=datetime.fromisoformat( - "2026-07-22T12:02:38+00:00" - ).timestamp(), - turn_model="observed", - ) - assert candidate_calls == candidate_calls_after_observation + 1 - assert _submission_rows(db_path) == [ - ("observed-live-timeline", "linked", observed_turn_id) - ] - linked_turn = next( - change["turn"] - for change in linked_page["changes"] - if change.get("op") == "upsert" - and change.get("turn_id") == observed_turn_id - ) - assert linked_turn["submission_id"] == turn_submission_id( - "host-a", "observed-live-timeline" - ) - assert linked_turn["submission_state"] == "linked" - with sqlite3.connect(str(db_path)) as conn: - linked_at = conn.execute( - """ - SELECT linked_at FROM turn_submissions - WHERE host_id = 'host-a' - AND request_id = 'observed-live-timeline' - """ - ).fetchone()[0] - assert linked_at == "2026-07-22T12:02:38+00:00" - assert linked_at < "2026-07-22T12:03:23+00:00" def test_observed_link_rearm_uses_stable_owner_across_worker_renumber( @@ -1998,7 +973,6 @@ def test_observed_link_rearm_uses_stable_owner_across_worker_renumber( now=datetime.fromisoformat( "2026-07-22T12:02:33+00:00" ).timestamp(), - turn_model="observed", ) renumbered = project_from_raw( @@ -2019,7 +993,6 @@ def test_observed_link_rearm_uses_stable_owner_across_worker_renumber( assert store_sqlite.save_snapshot( db_path, renumbered, - turn_model="observed", ) rearmed_keys: list[tuple[str, str]] = [] @@ -2045,10 +1018,9 @@ def record_rearm( source_turn_id="renumbered-source", instruction_text="hello\x01", observed_at="2026-07-22T12:02:36+00:00", - turn_model="observed", ) assert (owner_key, instruction_fingerprint("hello")) in rearmed_keys - assert all(not owner.startswith("legacy-worker:") for owner, _ in rearmed_keys) + assert all(owner == owner_key for owner, _ in rearmed_keys) turn_delta_payload_from_store( db_path, @@ -2056,7 +1028,6 @@ def record_rearm( now=datetime.fromisoformat( "2026-07-22T12:03:23+00:00" ).timestamp(), - turn_model="observed", ) assert _submission_rows(db_path) == [ ("observed-renumber", "linked", observed_turn_id) @@ -2081,7 +1052,6 @@ def test_observed_busy_pane_completion_links_immediately( "has_open_turn": True, }, observed_at="2026-07-22T12:01:30+00:00", - turn_model="observed", ) assert started.updated == 1 _insert_link_submission( @@ -2096,7 +1066,6 @@ def test_observed_busy_pane_completion_links_immediately( db_path, source_turn_id="busy-pane-source", observed_at="2026-07-22T12:02:36+00:00", - turn_model="observed", ) assert _submission_rows(db_path) == [ ("observed-busy-pane", "linked", observed_turn_id) diff --git a/tests/test_worker_label_and_model.py b/tests/test_worker_label_and_model.py index fee7405..f513f4c 100644 --- a/tests/test_worker_label_and_model.py +++ b/tests/test_worker_label_and_model.py @@ -24,9 +24,11 @@ def test_pane_label_is_public_but_cwd_and_target_are_private(tmp_path) -> None: assert bindings[0].target_value == "term-private" -def test_turn_model_remains_content_not_identity() -> None: +def test_llm_model_round_trip_and_turn_id_stability() -> None: base = {"host_id": "h", "worker_id": "w", "kind": "turn", "source": "acp", "complete": True} plain = Turn.from_dict(base) - modeled = Turn.from_dict({**base, "model": "claude"}) + modeled = Turn.from_dict({**base, "model": "claude-fable-5"}) + assert modeled.model == "claude-fable-5" + assert modeled.to_dict()["model"] == "claude-fable-5" assert plain.id == modeled.id assert plain.fingerprint != modeled.fingerprint From 778cb2f95b6eb4f0fcd48b76568230668c0e9c5b Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 22:52:40 +0800 Subject: [PATCH 77/83] tests: align socket integration with direct store API --- tests/test_daemon.py | 11 +++++------ tests/test_store.py | 2 +- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/tests/test_daemon.py b/tests/test_daemon.py index b661dc6..e755f29 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -68,12 +68,11 @@ get_command_request, init_store, latest_snapshot, - merge_backend_pending, - merge_turn_content, pending_payload_from_store, save_snapshot, upsert_worker_bindings, ) +from .store_helpers import apply_test_backend_pending, apply_test_turn_refresh _PUBLIC_JSON_FORBIDDEN_KEYS = { @@ -469,7 +468,7 @@ def test_daemon_pending_matches_shared_durable_projection_and_fingerprint( recompute_pending_content_fingerprint(degraded) != baseline["content_fingerprint"] ) - merge_backend_pending( + apply_test_backend_pending( db_path, snapshot.host_id, "worker-1", @@ -657,7 +656,7 @@ def test_pending_store_projection_reads_snapshot_and_overlay_atomically( ) init_store(db_path) save_snapshot(db_path, snapshot_a) - merge_backend_pending( + apply_test_backend_pending( db_path, config.host_id, "worker-1", @@ -689,7 +688,7 @@ def publish_new_view() -> None: try: assert allow_writer.wait(timeout=5) save_snapshot(db_path, snapshot_b) - merge_backend_pending( + apply_test_backend_pending( db_path, config.host_id, "worker-1", @@ -4610,7 +4609,7 @@ def test_isolated_daemon_survives_deterministic_real_wal_retirement_without_reso ) ], ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, config.host_id, worker.id, diff --git a/tests/test_store.py b/tests/test_store.py index c538da8..83256de 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -8834,7 +8834,7 @@ def test_compact_store_dry_run_reports_low_headroom_without_mutating( assert _tree_metadata(tmp_path) == before -def test_compact_store_requires_current_v9_without_migrating( +def test_compact_store_requires_current_schema_without_rebuilding( tmp_path: Path, ) -> None: db_path, _private_payload = _seed_compaction_fixture(tmp_path) From 4dd9eadb278ea9c3156183f2c6e2467d9b8f5ca0 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 23:08:21 +0800 Subject: [PATCH 78/83] store: reset legacy v28 databases at schema 29 --- src/tendwire/store/sqlite.py | 4 +- tests/store_helpers.py | 12 +++++- tests/test_acp_coordinator.py | 4 ++ tests/test_store.py | 73 +++++++++++++++++++++++++++++++++-- 4 files changed, 87 insertions(+), 6 deletions(-) diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index 9ed3588..8008358 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -141,7 +141,7 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS -STORE_SCHEMA_VERSION = 28 +STORE_SCHEMA_VERSION = 29 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS TURN_CHANGE_RETENTION_DAYS = 7 TURN_CHANGE_RETENTION_COUNT = 100_000 @@ -10221,7 +10221,7 @@ def _rebuild_current_schema_conn( def ensure_schema(conn: sqlite3.Connection) -> None: - """Open v28 unchanged or explicitly replace any other schema.""" + """Open v29 unchanged or explicitly replace any other schema.""" version = int(conn.execute("PRAGMA user_version").fetchone()[0]) if version == STORE_SCHEMA_VERSION: return diff --git a/tests/store_helpers.py b/tests/store_helpers.py index 0700831..b2ca539 100644 --- a/tests/store_helpers.py +++ b/tests/store_helpers.py @@ -46,7 +46,17 @@ def apply_test_backend_pending( clean = sanitize_public_mapping(pending) choices = tuple( PendingObservedChoice( - choice_id=str(choice.get("choice_id") or choice.get("id") or ordinal), + choice_id=( + "choice-" + + stable_fingerprint( + { + "domain": "test.pending-choice.v1", + "ordinal": ordinal, + "choice": choice, + }, + length=24, + ) + ), label=str(choice.get("label") or "Option"), picker_ordinal=ordinal, ) diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py index 7cc8fc8..d7de916 100644 --- a/tests/test_acp_coordinator.py +++ b/tests/test_acp_coordinator.py @@ -1077,6 +1077,10 @@ def _seed(config: Config) -> Worker: name="worker", status="idle", fingerprint="worker-fingerprint", + meta={ + "stable_key": "wsk1_" + ("a" * 64), + "stable_key_version": 1, + }, ) save_snapshot( config.db_path, diff --git a/tests/test_store.py b/tests/test_store.py index 83256de..cd24708 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -6686,7 +6686,7 @@ def test_store_v8_maintenance_schema_singleton_and_ordered_indexes( assert created == ("created_at", "host_id", "id") -def test_current_v28_schema_gate_and_second_init_have_no_mutation_or_wal_setting( +def test_current_v29_schema_gate_and_second_init_have_no_mutation_or_wal_setting( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -6734,6 +6734,71 @@ def reject_parent_ex(fd: int, operation: int) -> None: ) +def test_base_v28_schema_is_loudly_reset_without_removed_store_objects( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + db_path = tmp_path / "base-v28-reset.db" + with sqlite3.connect(str(db_path)) as conn: + conn.executescript( + """ + CREATE TABLE herdr_turn_watermarks ( + host_id TEXT NOT NULL, + pane_id TEXT NOT NULL, + last_turn INTEGER NOT NULL + ); + INSERT INTO herdr_turn_watermarks VALUES ('host-a', 'pane-private', 7); + CREATE TABLE agent_event_tombstones ( + host_id TEXT NOT NULL, + event_id TEXT NOT NULL + ); + INSERT INTO agent_event_tombstones VALUES ('host-a', 'event-private'); + CREATE TABLE backend_pending ( + sentinel TEXT NOT NULL, + route_kind TEXT NOT NULL DEFAULT 'legacy' + ); + INSERT INTO backend_pending VALUES ('pending-private', 'legacy'); + CREATE TABLE command_receipts ( + sentinel TEXT NOT NULL, + legacy_collision INTEGER NOT NULL DEFAULT 0, + legacy_collision_count INTEGER NOT NULL DEFAULT 0 + ); + INSERT INTO command_receipts VALUES ('receipt-private', 1, 2); + PRAGMA user_version = 28; + """ + ) + + with caplog.at_level(logging.WARNING, logger=store_sqlite.__name__): + init_store(db_path) + + with sqlite3.connect(str(db_path)) as conn: + assert _user_version(conn) == 29 + object_names = { + str(row[0]) + for row in conn.execute( + "SELECT name FROM sqlite_master WHERE name NOT LIKE 'sqlite_%'" + ).fetchall() + } + assert "herdr_turn_watermarks" not in object_names + assert "agent_event_tombstones" not in object_names + assert "route_kind" not in { + str(row[1]) for row in conn.execute("PRAGMA table_info(backend_pending)") + } + receipt_columns = { + str(row[1]) for row in conn.execute("PRAGMA table_info(command_receipts)") + } + assert "legacy_collision" not in receipt_columns + assert "legacy_collision_count" not in receipt_columns + assert conn.execute("SELECT COUNT(*) FROM backend_pending").fetchone() == (0,) + assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone() == (0,) + + message = caplog.records[-1].getMessage() + assert "previous_version=28" in message + assert "target_version=29" in message + assert "table:herdr_turn_watermarks" in message + assert "table:agent_event_tombstones" in message + + def _create_discarded_schema(db_path: Path, version: int) -> None: with sqlite3.connect(str(db_path)) as conn: conn.executescript( @@ -6808,8 +6873,10 @@ def test_newer_schema_is_loudly_discarded_and_recreated( with sqlite3.connect(str(db_path)) as conn: assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - assert "previous_version=29" in caplog.records[-1].getMessage() - assert "table:discarded_data" in caplog.records[-1].getMessage() + message = caplog.records[-1].getMessage() + assert f"previous_version={store_sqlite.STORE_SCHEMA_VERSION + 1}" in message + assert f"target_version={store_sqlite.STORE_SCHEMA_VERSION}" in message + assert "table:discarded_data" in message def test_v0_with_application_objects_is_loudly_discarded_and_recreated( From f07a0a68851226c31593bb82c057d1f6549fde9a Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 23:18:04 +0800 Subject: [PATCH 79/83] tests: restore current store behavior coverage --- tests/test_agent_events.py | 96 +++++++++++++ tests/test_turn_submissions.py | 256 +++++++++++++++++++++++++++++++++ 2 files changed, 352 insertions(+) diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py index 851d690..64d6190 100644 --- a/tests/test_agent_events.py +++ b/tests/test_agent_events.py @@ -743,6 +743,102 @@ def test_automatic_maintenance_retires_agent_events_only_when_due( ) +def test_automatic_agent_retention_failure_does_not_advance_cadence( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "automatic-agent-rollback.db" + old = replace( + _message_event(sequence=1, visibility="private"), + observed_at="2026-01-01T00:00:00+00:00", + ) + record_test_agent_event(db_path, "host-1", old) + original_cleanup = store_sqlite._cleanup_agent_event_retention_conn + + def fail_after_cleanup(*args: object, **kwargs: object) -> dict[str, object]: + original_cleanup(*args, **kwargs) + raise RuntimeError("controlled retention failure") + + monkeypatch.setattr( + store_sqlite, + "_cleanup_agent_event_retention_conn", + fail_after_cleanup, + ) + + with pytest.raises(RuntimeError, match="controlled retention failure"): + store_sqlite.maybe_run_automatic_store_maintenance( + db_path, + policy=store_sqlite.SnapshotRetentionPolicy( + retention_days=30, + retention_count=100, + batch_size=10, + ), + agent_event_host_id="host-1", + agent_event_retention_days=7, + now="2026-02-01T00:00:00+00:00", + ) + + with sqlite3.connect(db_path) as conn: + assert conn.execute( + "SELECT last_completed_at FROM store_maintenance_state " + "WHERE scope = 'automatic'" + ).fetchone() == (None,) + assert conn.execute("SELECT COUNT(*) FROM agent_events").fetchone() == (1,) + + +def test_retention_cleanup_serializes_concurrent_append_and_both_commit( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "retention-concurrency.db" + old = replace( + _message_event(sequence=1, visibility="private"), + observed_at="2026-01-01T00:00:00+00:00", + ) + record_test_agent_event(db_path, "host-1", old) + entered = threading.Event() + release = threading.Event() + original_cleanup = store_sqlite._cleanup_agent_event_retention_conn + + def blocking_cleanup(*args: object, **kwargs: object) -> dict[str, object]: + entered.set() + assert release.wait(timeout=5) + return original_cleanup(*args, **kwargs) + + monkeypatch.setattr( + store_sqlite, + "_cleanup_agent_event_retention_conn", + blocking_cleanup, + ) + with ThreadPoolExecutor(max_workers=2) as executor: + cleanup = executor.submit( + store_sqlite.cleanup_agent_event_retention, + db_path, + "host-1", + retention_days=7, + now="2026-02-01T00:00:00+00:00", + ) + assert entered.wait(timeout=5) + append = executor.submit( + record_test_agent_event, + db_path, + "host-1", + _message_event(sequence=2, text="new", visibility="private"), + ) + time.sleep(0.05) + assert append.done() is False + release.set() + assert cleanup.result(timeout=5)["deleted"] == 1 + assert append.result(timeout=5).inserted is True + + assert [ + item.event.payload["text"] + for item in store_sqlite.list_agent_events(db_path, "host-1") + ] == ["new"] + with sqlite3.connect(db_path) as conn: + assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] + + def test_journal_accepts_acp_sized_private_text(tmp_path: Path) -> None: event = _message_event(sequence=1, text="x" * (64 * 1024), visibility="private") result = record_test_agent_event(tmp_path / "store.db", "host-1", event) diff --git a/tests/test_turn_submissions.py b/tests/test_turn_submissions.py index c3d04d6..cd3bef6 100644 --- a/tests/test_turn_submissions.py +++ b/tests/test_turn_submissions.py @@ -637,6 +637,144 @@ def test_observed_submission_linker_waits_for_window_close_before_failing_closed +@pytest.mark.parametrize( + ( + "scenario", + "initial_state", + "submission_count", + "observe_candidate", + "terminal_state", + ), + ( + ("no-candidate", "submitted", 1, False, "expired"), + ("stale-send-started", "send_started", 1, True, "expired"), + ("two-by-one", "submitted", 2, True, "ambiguous"), + ), +) +def test_link_window_close_settles_current_submission_components( + tmp_path: Path, + scenario: str, + initial_state: str, + submission_count: int, + observe_candidate: bool, + terminal_state: str, +) -> None: + db_path = tmp_path / f"window-close-{scenario}.db" + owner_key = _seed_link_worker(db_path) + for index in range(submission_count): + _insert_link_submission( + db_path, + request_id=f"{scenario}-{index}", + owner_key=owner_key, + state=initial_state, + link_expires_at="2026-02-01T12:01:00+00:00", + hard_expires_at="2026-02-02T12:00:00+00:00", + ) + if observe_candidate: + observed_at = "2026-02-01T12:00:03+00:00" + if initial_state == "send_started": + observed_at = ( + datetime.fromisoformat("2026-02-01T12:00:00+00:00") + + timedelta( + seconds=store_sqlite.SUBMISSION_SEND_ACK_TIMEOUT_SECONDS + 1 + ) + ).isoformat() + _observe_link_turn( + db_path, + source_turn_id=f"{scenario}-source", + observed_at=observed_at, + ) + + before_close = turn_delta_payload_from_store( + db_path, + "host-a", + now=datetime.fromisoformat( + "2026-02-01T12:00:30+00:00" + ).timestamp(), + ) + assert before_close["host_id"] == "host-a" + assert _submission_rows(db_path) == [ + (f"{scenario}-{index}", initial_state, None) + for index in range(submission_count) + ] + + at_close = turn_delta_payload_from_store( + db_path, + "host-a", + now=datetime.fromisoformat( + "2026-02-01T12:01:00+00:00" + ).timestamp(), + ) + assert at_close["host_id"] == "host-a" + assert _submission_rows(db_path) == [ + (f"{scenario}-{index}", terminal_state, None) + for index in range(submission_count) + ] + with sqlite3.connect(str(db_path)) as conn: + stamps = conn.execute( + """ + SELECT terminal_at, hard_expires_at + FROM turn_submissions + WHERE host_id = 'host-a' + ORDER BY request_id + """ + ).fetchall() + assert stamps == [ + ("2026-02-01T12:01:00+00:00", "2026-02-02T12:00:00+00:00") + ] * submission_count + + +def test_disconnected_submission_components_settle_independently( + tmp_path: Path, +) -> None: + db_path = tmp_path / "disconnected-components.db" + owner_key = _seed_link_worker(db_path) + _insert_link_submission( + db_path, + request_id="old-disconnected", + owner_key=owner_key, + link_not_before="2026-02-01T11:00:00+00:00", + link_expires_at="2026-02-01T11:01:00+00:00", + ) + _insert_link_submission( + db_path, + request_id="live-singleton", + owner_key=owner_key, + link_expires_at="2026-02-01T12:01:00+00:00", + ) + turn_id = _observe_link_turn( + db_path, + source_turn_id="live-singleton-source", + observed_at="2026-02-01T12:00:03+00:00", + ) + + payload = turn_delta_payload_from_store( + db_path, + "host-a", + now=datetime.fromisoformat( + "2026-02-01T12:00:05+00:00" + ).timestamp(), + ) + + assert payload["host_id"] == "host-a" + assert _submission_rows(db_path) == [ + ("live-singleton", "linked", turn_id), + ("old-disconnected", "expired", None), + ] + with sqlite3.connect(str(db_path)) as conn: + assert conn.execute( + """ + SELECT request_id, terminal_at, linked_at + FROM turn_submissions + WHERE host_id = 'host-a' + ORDER BY request_id + """ + ).fetchall() == [ + ("live-singleton", None, "2026-02-01T12:00:03+00:00"), + ("old-disconnected", "2026-02-01T12:00:03+00:00", None), + ] + + def test_manual_same_text_turn_links_single_open_submission( tmp_path: Path, ) -> None: @@ -953,6 +1091,124 @@ def test_turn_alias_resolves_public_content_and_final_root( +def test_stable_key_delta_lazy_sweep_rearms_component_backoff( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "stable-key-delta-rearm.db" + owner_key = _seed_link_worker(db_path) + _insert_link_submission( + db_path, + request_id="stable-key-delta-rearm", + owner_key=owner_key, + link_not_before="2026-07-22T12:01:23+00:00", + link_expires_at="2026-07-22T12:03:23+00:00", + hard_expires_at="2026-07-23T12:02:23+00:00", + ) + candidate_calls = 0 + rearmed_keys: list[tuple[str, str]] = [] + original_candidates = store_sqlite._submission_link_candidate_turns_conn + original_rearm = store_sqlite._rearm_submission_link_component + original_settle = store_sqlite._settle_submission_links_conn + fail_direct_settlement = False + + def record_candidates(*args: object, **kwargs: object): + nonlocal candidate_calls + candidate_calls += 1 + return original_candidates(*args, **kwargs) + + def record_rearm( + db: Path | str, + host: str, + owner: str, + fingerprint: str, + ) -> None: + rearmed_keys.append((owner, fingerprint)) + original_rearm(db, host, owner, fingerprint) + + def fail_one_direct_settlement(*args: object, **kwargs: object) -> int: + if fail_direct_settlement: + raise RuntimeError("controlled direct settlement failure") + return original_settle(*args, **kwargs) + + monkeypatch.setattr( + store_sqlite, + "_submission_link_candidate_turns_conn", + record_candidates, + ) + monkeypatch.setattr( + store_sqlite, + "_rearm_submission_link_component", + record_rearm, + ) + monkeypatch.setattr( + store_sqlite, + "_settle_submission_links_conn", + fail_one_direct_settlement, + ) + + for observed_at in ( + "2026-07-22T12:02:33+00:00", + "2026-07-22T12:02:38+00:00", + ): + payload = turn_delta_payload_from_store( + db_path, + "host-a", + now=datetime.fromisoformat(observed_at).timestamp(), + ) + assert payload["host_id"] == "host-a" + assert candidate_calls == 1 + assert _submission_rows(db_path) == [ + ("stable-key-delta-rearm", "submitted", None) + ] + + fail_direct_settlement = True + observed_turn_id = _observe_link_turn( + db_path, + source_turn_id="stable-key-delta-rearm-source", + instruction_text="hello\x01", + observed_at="2026-07-22T12:02:40+00:00", + ) + fail_direct_settlement = False + assert (owner_key, instruction_fingerprint("hello")) in rearmed_keys + assert all(owner == owner_key for owner, _fingerprint in rearmed_keys) + assert candidate_calls == 1 + assert _submission_rows(db_path) == [ + ("stable-key-delta-rearm", "submitted", None) + ] + + linked_page = turn_delta_payload_from_store( + db_path, + "host-a", + now=datetime.fromisoformat( + "2026-07-22T12:02:42+00:00" + ).timestamp(), + ) + assert linked_page["host_id"] == "host-a" + assert candidate_calls == 2 + assert _submission_rows(db_path) == [ + ("stable-key-delta-rearm", "linked", observed_turn_id) + ] + linked_turn = next( + change["turn"] + for change in linked_page["changes"] + if change.get("op") == "upsert" + and change.get("turn_id") == observed_turn_id + ) + assert linked_turn["submission_id"] == turn_submission_id( + "host-a", "stable-key-delta-rearm" + ) + assert linked_turn["submission_state"] == "linked" + with sqlite3.connect(str(db_path)) as conn: + assert conn.execute( + """ + SELECT linked_at FROM turn_submissions + WHERE host_id = 'host-a' + AND request_id = 'stable-key-delta-rearm' + """ + ).fetchone() == ("2026-07-22T12:02:42+00:00",) + + def test_observed_link_rearm_uses_stable_owner_across_worker_renumber( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, From af51027100f70fd0ecd0e8058c61ca690917d243 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Tue, 4 Aug 2026 23:25:50 +0800 Subject: [PATCH 80/83] tests: follow current store schema version --- tests/test_delivery_retention_projection.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_delivery_retention_projection.py b/tests/test_delivery_retention_projection.py index ec32237..9450dab 100644 --- a/tests/test_delivery_retention_projection.py +++ b/tests/test_delivery_retention_projection.py @@ -147,7 +147,7 @@ def _assert_continuity_integrity(db_path: Path) -> None: with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA user_version").fetchone() == ( store_sqlite.STORE_SCHEMA_VERSION, - ) == (28,) + ) assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] current_counts = conn.execute( """ From 406b1d7a7512d2eee8f46dc520b9a08776b10adf Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Wed, 5 Aug 2026 00:59:24 +0800 Subject: [PATCH 81/83] docs: freeze connector RPC contract v2 --- docs/connector-rpc-contract.md | 208 +++++++++++++++++++ tests/test_connector_daemon_cli.py | 314 ++++++++++++++++++++++++++++- 2 files changed, 520 insertions(+), 2 deletions(-) create mode 100644 docs/connector-rpc-contract.md diff --git a/docs/connector-rpc-contract.md b/docs/connector-rpc-contract.md new file mode 100644 index 0000000..f8361c9 --- /dev/null +++ b/docs/connector-rpc-contract.md @@ -0,0 +1,208 @@ +# Tendwire connector RPC contract v2 + +Status: authoritative for the connector boundary implemented at Tendwire baseline +`8bc8d8b` and descendants that contain this document. Contract revision `v2` is a +document revision; every request and response that has a `schema_version` field +still uses JSON schema version `1`. + +## Scope and ownership + +Tendwire is the sole owner of delivery durability, ordering, leases, retry state, +dead letters, receipts, and public payload projection. A connector such as Herdres +owns provider ingress, presentation, and its provider-message binding keyed by the +Tendwire delivery key. Herdr owns worker/pane lifecycle and ACP endpoint ownership; +it is not part of this delivery protocol. + +The supported cross-process boundary is Tendwire's local Unix socket. A connector +must not invoke a Tendwire CLI, spawn a subprocess, read Tendwire's database, or +read transcript/runtime files as a fallback. + +## Framing and response envelopes + +The socket carries one newline-terminated UTF-8 JSON request and response per +connection. The canonical request is: + +```json +{"method":"connector.poll","params":{"name":"attention"}} +``` + +`params` must be an object. The optional top-level `id` is echoed when it passes +the daemon's public-boundary validation. No other top-level request fields are +accepted. Frames are bounded by the daemon's configured request and response +limits (1 MiB by default). + +Every connector response has two distinct layers: + +```json +{ + "schema_version": 1, + "ok": true, + "status": "ok", + "error": null, + "result": { + "schema_version": 1, + "ok": false, + "status": "invalid_ref", + "error": {"code": "invalid_ref", "message": "..."} + } +} +``` + +The outer envelope reports daemon framing, method routing, and execution. The +inner `result` reports the connector operation. A caller must require both +`outer.ok` and `outer.result.ok`; outer `ok: true` alone never proves that an ACK +or other mutation succeeded. Invalid framing, invalid top-level parameters, an +unknown method, or an internal daemon failure produces outer `ok: false` with +`result: null`. Invalid or stale connector data produces outer `ok: true` and an +inner `ok: false` result. + +## Stable identities and opaque values + +- `key` is the durable delivery identity. It remains stable across lease expiry, + reclaim, release, defer, and retry. A presentation connector must use it as its + provider-message deduplication/binding key. +- `ref` is an attempt-scoped capability with prefix `twref1.`. It changes whenever + a delivery is leased again. Only the ref from the current live lease may mutate + that attempt; an older ref must be treated as stale or invalid. +- `attempt` increases when a delivery is leased again. +- `plan_token` (`twplan1.`), `content_revision` (`twrev1.`), and + `final_identity` (`twfinal1.`) are opaque, case-sensitive values. Clients must + preserve their exact UTF-8 bytes and must not parse, normalize, case-fold, or + regenerate them. The same rule applies when a token is nested in `payload`, + `turn`, `final`, or `content`. +- Connector payloads are already public, backend-neutral projections. Private + routing, terminal, provider, and credential fields are intentionally absent. + +## Required crash-recovery sequence + +The correct provider-delivery transaction is: + +1. Poll and receive `{key, ref, attempt, payload}`. +2. Look up the connector's durable provider-message binding by `key`. +3. If absent, send `payload` to the provider and durably store `key -> provider + message id` before ACKing Tendwire. +4. ACK the current `ref`. + +If the provider accepted the send but the ACK was lost, lease expiry followed by +`connector.poll` (or an explicit `connector.reclaim` first) returns the same +`key` and payload with a new `ref` and a higher `attempt`. The connector finds its +existing binding, does not send again, and ACKs the new ref. `connector.poll` +reclaims expired leases atomically before selecting work, so explicit reclaim is +an optional eager-maintenance operation, not a correctness requirement. + +This is the only supported ACK-loss recovery behavior. Payload mutation across +attempts or provider deduplication by `ref` would violate the contract. + +## Socket methods + +The complete connector method set is: + +```text +connector.prepare +connector.poll +connector.ack +connector.fail +connector.defer +connector.renew +connector.release +connector.reclaim +connector.retry +connector.inspect +``` + +There are no `turn_final_*`, `turn-final.*`, or similarly named socket methods. +Turn-final work uses the methods above with `name: "turn-final"`. + +### `connector.poll` + +Canonical params are `name`, with optional `limit` and `lease_seconds`. `limit` +defaults to 1 and is bounded to 1..100. The lease defaults to daemon configuration; +turn-final leases are bounded by the configured maximum, while other neutral +queues permit leases up to 86,400 seconds. + +A successful inner result contains `items`. Each item contains `ref`, `key`, +`attempt`, `leased_until`, `available_at`, and `payload`; turn-final items also +carry `created_at` when available. Polling is FIFO subject to Tendwire's stored +turn-final plan and ordering constraints. + +### `connector.ack` + +Params are `name`, live `ref`, and optional public `response`. Success is inner +status `acknowledged`; the delivery is terminal and is not polled again. An ACK +must use the newest ref obtained after any re-poll. + +### `connector.fail` + +Params are `name`, live `ref`, and optional `reason`, public `response`, +`available_at`, or `delay_seconds`. It records a failed attempt and either returns +the item to the retry schedule or exhausts it according to Tendwire's configured +attempt budget. Tendwire owns that budget and final state. + +### `connector.defer` + +Params have the same scheduling shape as `connector.fail`. Defer releases the +current lease onto the requested future schedule without classifying the provider +operation as a delivery failure. + +### `connector.renew` + +Params are `name`, live `ref`, and optional `lease_seconds`. It extends the current +lease within the same queue-specific bounds used by poll and returns status +`renewed`. It does not create a new delivery identity. + +### `connector.release` + +Params are `name` and live `ref`. It ends the current lease and makes the durable +delivery eligible for another poll, which creates a new attempt/ref while retaining +the key and payload. + +### `connector.reclaim` + +Params are `name`. It expires overdue leases for that queue and reports the +`reclaimed` count. Normal poll already performs this operation transactionally. + +### `connector.prepare` + +Prepare is valid only for `name: "turn-final"` and requires `schema_version: 1`. +It is a strict four-action protocol: + +- `begin`: `action`, `turn_id`, `content_revision`, `presentation_version`, + `part_count`, and optional live `source_ref`. +- `part`: `action`, `plan_token`, zero-based `ordinal`, and non-empty `spans`. + Each span is exactly `{field,start_char,end_char}`, where `field` is + `user_text` or `assistant_final_text` and the range is a valid non-empty slice. +- `commit`: `action`, `plan_token`, and optional live `source_ref`. Commit + materializes durable ordered delivery jobs. +- `recover`: `action`, `failed_plan_token`, and idempotency `request_id`. Recovery + retains the acknowledged prefix and creates executable work for the remaining + suffix according to Tendwire's stored plan state. + +Each action accepts only its declared fields. Callers must retain returned opaque +tokens exactly and honor the returned inner `ok` and `status`. + +### `connector.inspect` + +This is the strict dead-letter query for turn-final work. Params are exactly +`schema_version: 1`, `name: "turn-final"`, `status: "dead_letter"`, and `limit` +in 1..100. + +### `connector.retry` + +This is the strict operator retry for a turn-final dead letter. Params are exactly +`schema_version: 1`, `name: "turn-final"`, and one selector: either the durable +revision delivery `key` or its `final_identity`. Tendwire validates the +`turn-final:revision:twfinal1.*` identity form and owns the resulting transition. + +## Failure handling + +Known inner failures include `invalid_params`, `invalid_ref`, stale/expired-ref +variants, `store_unavailable`, revision/plan conflicts, missing plan or delivery +state, and exhausted attempts. Callers must branch on `result.ok` and +`result.status`, not error-message text. A timeout or broken socket after a +mutation is an unknown outcome: do not assume the mutation failed. Re-poll and use +the durable-key/provider-binding rule to converge safely. + +The daemon sanitizes all connector results at the public boundary. A connector +must not depend on private fields that happen to exist in Tendwire storage, and it +must reject any design that makes its own local cache authoritative for Tendwire +delivery state. diff --git a/tests/test_connector_daemon_cli.py b/tests/test_connector_daemon_cli.py index 196fd24..21bbe87 100644 --- a/tests/test_connector_daemon_cli.py +++ b/tests/test_connector_daemon_cli.py @@ -2,10 +2,13 @@ from __future__ import annotations -from contextlib import closing +from collections.abc import Iterator +from contextlib import closing, contextmanager +from datetime import datetime, timedelta, timezone import io import json import sqlite3 +import threading from pathlib import Path from typing import Any @@ -16,7 +19,11 @@ from tendwire.config import Config from tendwire.core.models import Snapshot from tendwire.daemon import TendwireDaemon -from tendwire.daemon_api import TendwireDaemonAPI +from tendwire.daemon_api import ( + DaemonAPIClient, + TendwireDaemonAPI, + UnixSocketJSONServer, +) from tendwire.store import sqlite as store_sqlite from tendwire.store.sqlite import init_store @@ -64,6 +71,64 @@ def _enqueue(db_path: Path, *, host_id: str = "host-a", key: str = "job-1") -> N ) +def _canonical_final_turn(db_path: Path, *, host_id: str) -> tuple[str, str]: + init_store(db_path) + turn_id = "turn-worker-contract-source-contract" + final_text = "abcdefgh" + revision = store_sqlite.content_revision( + turn_id, + None, + final_text, + "absent", + "complete", + ) + created_at = "2026-01-01T00:00:00+00:00" + with closing(sqlite3.connect(str(db_path))) as conn, conn: + conn.execute( + """ + INSERT INTO turns ( + host_id, turn_id, worker_id, status, kind, updated_at, + fingerprint, snapshot_content_fingerprint, observed_at, + payload_json, list_sequence + ) VALUES (?, ?, ?, 'complete', 'turn', ?, ?, ?, ?, ?, 1) + """, + ( + host_id, + turn_id, + "worker-contract", + created_at, + "fingerprint-contract", + "snapshot-contract", + created_at, + json.dumps( + { + "source_turn_id": "source-contract", + "complete": True, + "meta": { + "stable_key": "wsk1_" + ("c" * 64), + "stable_key_version": 1, + }, + } + ), + ), + ) + conn.execute( + """ + INSERT INTO turn_content_revisions ( + host_id, turn_id, content_revision, user_text, + assistant_final_text, user_state, final_state, + user_char_length, user_byte_length, + final_char_length, final_byte_length, + user_page_count, final_page_count, + is_current, created_at, superseded_at + ) VALUES (?, ?, ?, NULL, ?, 'absent', 'complete', 0, 0, 8, 8, + 0, 1, 1, ?, NULL) + """, + (host_id, turn_id, revision, final_text, created_at), + ) + return turn_id, revision + + def _assert_json_only_and_safe(payload: dict[str, Any]) -> None: encoded = json.dumps(payload, sort_keys=True).lower() for forbidden in ( @@ -86,6 +151,251 @@ def _assert_json_only_and_safe(payload: dict[str, Any]) -> None: assert forbidden not in encoded +@contextmanager +def _socket_client( + tmp_path: Path, + api: TendwireDaemonAPI, +) -> Iterator[DaemonAPIClient]: + socket_path = tmp_path / "s" + server = UnixSocketJSONServer(socket_path, api.dispatch) + server.start() + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + yield DaemonAPIClient(socket_path, timeout_seconds=2) + finally: + server.close() + thread.join(timeout=2) + assert not thread.is_alive() + + +@pytest.mark.skipif( + not hasattr(__import__("socket"), "AF_UNIX"), + reason="Unix sockets required", +) +def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "ack-lost.db" + turn_id, revision = _canonical_final_turn(db_path, host_id="daemon-host") + current = datetime(2026, 1, 1, tzinfo=timezone.utc) + monkeypatch.setattr( + store_sqlite, + "utc_timestamp", + lambda: current.isoformat(timespec="seconds"), + ) + outbox = ConnectorOutboxAPI(db_path, "daemon-host") + api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=outbox.dispatch, + ) + + provider_sends: list[str] = [] + provider_message_by_key: dict[str, str] = {} + with _socket_client(tmp_path, api) as client: + begun = client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "begin", + "name": "turn-final", + "turn_id": turn_id, + "content_revision": revision, + "presentation_version": "contract-v2", + "part_count": 2, + }, + ) + plan_token = begun["result"]["plan_token"] + for ordinal, start in enumerate((0, 4)): + staged = client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "part", + "name": "turn-final", + "plan_token": plan_token, + "ordinal": ordinal, + "spans": [ + { + "field": "assistant_final_text", + "start_char": start, + "end_char": start + 4, + } + ], + }, + ) + assert staged["ok"] is True + assert staged["result"]["ok"] is True + committed = client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "commit", + "name": "turn-final", + "plan_token": plan_token, + }, + ) + assert begun["ok"] is True + assert begun["result"]["ok"] is True + assert committed["ok"] is True + assert committed["result"]["ok"] is True + assert committed["result"]["job_count"] == 2 + + first_outer = client.request( + "connector.poll", + {"name": "turn-final", "lease_seconds": 5}, + ) + first = first_outer["result"]["items"][0] + + # The provider accepted the message and Herdres persisted the binding, + # but the connector ACK was lost before Tendwire received it. + provider_sends.append("provider-message-17") + provider_message_by_key[first["key"]] = provider_sends[-1] + current += timedelta(seconds=6) + + reclaimed = client.request("connector.reclaim", {"name": "turn-final"}) + second_outer = client.request( + "connector.poll", + {"name": "turn-final", "lease_seconds": 5}, + ) + second = second_outer["result"]["items"][0] + + assert first_outer["ok"] is True + assert first_outer["result"]["ok"] is True + assert reclaimed["ok"] is True + assert reclaimed["result"]["reclaimed"] == 1 + assert second["key"] == first["key"] + assert second["payload"] == first["payload"] + assert json.dumps( + second["payload"], sort_keys=True, separators=(",", ":") + ) == json.dumps(first["payload"], sort_keys=True, separators=(",", ":")) + assert second["ref"] != first["ref"] + assert second["attempt"] == first["attempt"] + 1 + stale_ack = client.request( + "connector.ack", + {"name": "turn-final", "ref": first["ref"]}, + ) + assert stale_ack["ok"] is True + assert stale_ack["result"]["ok"] is False + assert stale_ack["result"]["status"] in { + "expired_ref", + "invalid_ref", + "stale_ref", + } + + # A thin connector recognizes the durable key, does not resend, and + # acknowledges the current attempt rather than the stale first ref. + assert provider_message_by_key[second["key"]] == "provider-message-17" + acknowledged = client.request( + "connector.ack", + { + "name": "turn-final", + "ref": second["ref"], + "response": {"status": "deduplicated"}, + }, + ) + assert acknowledged["ok"] is True + assert acknowledged["result"]["ok"] is True + assert acknowledged["result"]["status"] == "acknowledged" + assert provider_sends == ["provider-message-17"] + next_item = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert next_item["key"] != second["key"] + assert next_item["payload"]["plan_token"] == plan_token + assert next_item["payload"]["sequence_index"] == 1 + assert client.request( + "connector.ack", + {"name": "turn-final", "ref": next_item["ref"]}, + )["result"]["ok"] is True + assert client.request("connector.poll", {"name": "turn-final"})["result"][ + "items" + ] == [] + + +@pytest.mark.skipif( + not hasattr(__import__("socket"), "AF_UNIX"), + reason="Unix sockets required", +) +def test_socket_keeps_transport_and_connector_errors_in_separate_envelopes( + tmp_path: Path, +) -> None: + db_path = tmp_path / "error-envelopes.db" + init_store(db_path) + outbox = ConnectorOutboxAPI(db_path, "daemon-host") + api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=outbox.dispatch, + ) + + with _socket_client(tmp_path, api) as client: + connector_error = client.request( + "connector.ack", + {"name": "attention", "ref": "not-a-live-ref"}, + ) + protocol_error = client.request("turn_final_ack", {}) + + assert connector_error["ok"] is True + assert connector_error["status"] == "ok" + assert connector_error["error"] is None + assert connector_error["result"]["ok"] is False + assert connector_error["result"]["status"] == "invalid_ref" + assert connector_error["result"]["error"]["code"] == "invalid_ref" + assert protocol_error["ok"] is False + assert protocol_error["status"] == "error" + assert protocol_error["result"] is None + assert protocol_error["error"]["code"] == "unknown_method" + + +@pytest.mark.skipif( + not hasattr(__import__("socket"), "AF_UNIX"), + reason="Unix sockets required", +) +def test_socket_preserves_opaque_connector_tokens_byte_for_byte( + tmp_path: Path, +) -> None: + expected = { + "plan_token": "twplan1.AaZz09_-ExactPlan", + "replaces_plan_token": "twplan1.Replaced_AaZz09-", + "content_revision": "twrev1.AaZz09_-ExactRevision", + "final_identity": "twfinal1.AaZz09_-ExactFinal", + } + key = f"turn-final:revision:{expected['final_identity']}" + api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=lambda _method, _params: { + "schema_version": 1, + "ok": True, + "status": "ok", + "host_id": "daemon-host", + "name": "turn-final", + "items": [ + { + "ref": "twref1.AaZz09_-ExactRef", + "key": key, + "attempt": 1, + "payload": {**expected, "operation": "upsert"}, + } + ], + }, + ) + + with _socket_client(tmp_path, api) as client: + response = client.request("connector.poll", {"name": "turn-final"}) + + item = response["result"]["items"][0] + assert item["key"] == key + for token_name, token in expected.items(): + assert item["payload"][token_name] == token + + def test_daemon_api_routes_connector_methods_safely(tmp_path: Path) -> None: db_path = tmp_path / "daemon-connector.db" _enqueue(db_path, host_id="daemon-host") From 59300ad10d55a8135bd60ab0f1723c86451515c0 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Wed, 5 Aug 2026 01:17:43 +0800 Subject: [PATCH 82/83] test: harden connector recovery contract --- docs/connector-rpc-contract.md | 47 +++-- tests/test_connector_daemon_cli.py | 322 +++++++++++++++++++++++++---- tests/test_connector_outbox.py | 15 +- 3 files changed, 330 insertions(+), 54 deletions(-) diff --git a/docs/connector-rpc-contract.md b/docs/connector-rpc-contract.md index f8361c9..a228db3 100644 --- a/docs/connector-rpc-contract.md +++ b/docs/connector-rpc-contract.md @@ -64,12 +64,14 @@ inner `ok: false` result. - `ref` is an attempt-scoped capability with prefix `twref1.`. It changes whenever a delivery is leased again. Only the ref from the current live lease may mutate that attempt; an older ref must be treated as stale or invalid. -- `attempt` increases when a delivery is leased again. +- `attempt` increases when a delivery is leased again within one retry generation. + It is not a lifetime counter: an explicit dead-letter `connector.retry` starts a + fresh generation whose first lease has `attempt: 1`. - `plan_token` (`twplan1.`), `content_revision` (`twrev1.`), and `final_identity` (`twfinal1.`) are opaque, case-sensitive values. Clients must preserve their exact UTF-8 bytes and must not parse, normalize, case-fold, or - regenerate them. The same rule applies when a token is nested in `payload`, - `turn`, `final`, or `content`. + regenerate them. The same rule applies to their supported positions in a polled + item payload and its nested `turn` and `content` objects. - Connector payloads are already public, backend-neutral projections. Private routing, terminal, provider, and credential fields are intentionally absent. @@ -85,10 +87,12 @@ The correct provider-delivery transaction is: If the provider accepted the send but the ACK was lost, lease expiry followed by `connector.poll` (or an explicit `connector.reclaim` first) returns the same -`key` and payload with a new `ref` and a higher `attempt`. The connector finds its -existing binding, does not send again, and ACKs the new ref. `connector.poll` -reclaims expired leases atomically before selecting work, so explicit reclaim is -an optional eager-maintenance operation, not a correctness requirement. +`key` and payload with a new `ref` and a higher attempt in the current retry +generation. This remains true after a connector process restart: the connector +reopens its durable provider binding, does not send again, and ACKs the new ref. +`connector.poll` reclaims expired leases atomically before selecting work, so +explicit reclaim is an optional eager-maintenance operation, not a correctness +requirement. This is the only supported ACK-loss recovery behavior. Payload mutation across attempts or provider deduplication by `ref` would violate the contract. @@ -136,13 +140,17 @@ must use the newest ref obtained after any re-poll. Params are `name`, live `ref`, and optional `reason`, public `response`, `available_at`, or `delay_seconds`. It records a failed attempt and either returns the item to the retry schedule or exhausts it according to Tendwire's configured -attempt budget. Tendwire owns that budget and final state. +attempt budget. If Tendwire marked a leased item `terminal_after_lease` because a +newer authoritative plan superseded it, fail instead returns `superseded`, makes +the item terminal, and does not schedule it. Tendwire owns the budget and final +state. ### `connector.defer` Params have the same scheduling shape as `connector.fail`. Defer releases the current lease onto the requested future schedule without classifying the provider -operation as a delivery failure. +operation as a delivery failure. For an item marked `terminal_after_lease`, it +instead returns `superseded`, makes the item terminal, and does not schedule it. ### `connector.renew` @@ -152,9 +160,10 @@ lease within the same queue-specific bounds used by poll and returns status ### `connector.release` -Params are `name` and live `ref`. It ends the current lease and makes the durable -delivery eligible for another poll, which creates a new attempt/ref while retaining -the key and payload. +Params are `name` and live `ref`. Normally it ends the current lease and makes the +durable delivery eligible for another poll, which creates a new attempt/ref while +retaining the key and payload. For an item marked `terminal_after_lease`, it +instead returns `superseded`, makes the item terminal, and does not requeue it. ### `connector.reclaim` @@ -180,6 +189,14 @@ It is a strict four-action protocol: Each action accepts only its declared fields. Callers must retain returned opaque tokens exactly and honor the returned inner `ok` and `status`. +The normal final delivery flow is source-bound. Poll the `final_ready` source, +pass its live `source_ref` to both `begin` and `commit`, and then deliver the +materialized parts. A successful source-bound commit atomically changes the source +and its attempt from `leased` to `awaiting_ack`. The source becomes `delivered` +only after all ordered parts are acknowledged. Source-less prepare exists for the +store's explicitly validated source-less/recovery cases; it is not a connector +shortcut around polling the normal source. + ### `connector.inspect` This is the strict dead-letter query for turn-final work. Params are exactly @@ -192,6 +209,12 @@ This is the strict operator retry for a turn-final dead letter. Params are exact `schema_version: 1`, `name: "turn-final"`, and one selector: either the durable revision delivery `key` or its `final_identity`. Tendwire validates the `turn-final:revision:twfinal1.*` identity form and owns the resulting transition. +For an exact dead-letter final anchor, success returns `status: "requeued"` and +`prior_attempt_count`, deletes the superseded per-attempt rows to keep history +bounded, and makes the next poll start at `attempt: 1` with the same key. A later +retry adds the new generation's attempts to `prior_attempt_count`. When the target +identifies one uniquely recoverable failed presentation plan, Tendwire may perform +the plan-recovery path instead and returns that recovery result. ## Failure handling diff --git a/tests/test_connector_daemon_cli.py b/tests/test_connector_daemon_cli.py index 21bbe87..84c8dfa 100644 --- a/tests/test_connector_daemon_cli.py +++ b/tests/test_connector_daemon_cli.py @@ -126,6 +126,14 @@ def _canonical_final_turn(db_path: Path, *, host_id: str) -> tuple[str, str]: """, (host_id, turn_id, revision, final_text, created_at), ) + source_id = store_sqlite._ensure_final_ready_anchor_conn( + conn, + host_id=host_id, + turn_id=turn_id, + content_revision_value=revision, + now=created_at, + ) + assert source_id is not None return turn_id, revision @@ -178,6 +186,7 @@ def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( monkeypatch: pytest.MonkeyPatch, ) -> None: db_path = tmp_path / "ack-lost.db" + binding_db = tmp_path / "provider-bindings.db" turn_id, revision = _canonical_final_turn(db_path, host_id="daemon-host") current = datetime(2026, 1, 1, tzinfo=timezone.utc) monkeypatch.setattr( @@ -185,17 +194,33 @@ def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( "utc_timestamp", lambda: current.isoformat(timespec="seconds"), ) - outbox = ConnectorOutboxAPI(db_path, "daemon-host") - api = TendwireDaemonAPI( + first_outbox = ConnectorOutboxAPI(db_path, "daemon-host") + first_api = TendwireDaemonAPI( get_snapshot=lambda: Snapshot(host_id="daemon-host"), get_health=lambda: {}, submit_command=lambda _params: {}, - connector_call=outbox.dispatch, + connector_call=first_outbox.dispatch, ) - provider_sends: list[str] = [] - provider_message_by_key: dict[str, str] = {} - with _socket_client(tmp_path, api) as client: + with closing(sqlite3.connect(str(binding_db))) as conn, conn: + conn.execute( + """ + CREATE TABLE provider_message_bindings ( + delivery_key TEXT PRIMARY KEY, + provider_message_id TEXT NOT NULL, + payload_json TEXT NOT NULL + ) + """ + ) + + with _socket_client(tmp_path, first_api) as client: + source_outer = client.request( + "connector.poll", + {"name": "turn-final", "lease_seconds": 5}, + ) + source = source_outer["result"]["items"][0] + assert source["payload"]["operation"] == "materialize" + assert source["payload"]["content_revision"] == revision begun = client.request( "connector.prepare", { @@ -206,6 +231,7 @@ def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( "content_revision": revision, "presentation_version": "contract-v2", "part_count": 2, + "source_ref": source["ref"], }, ) plan_token = begun["result"]["plan_token"] @@ -236,6 +262,7 @@ def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( "action": "commit", "name": "turn-final", "plan_token": plan_token, + "source_ref": source["ref"], }, ) assert begun["ok"] is True @@ -243,6 +270,18 @@ def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( assert committed["ok"] is True assert committed["result"]["ok"] is True assert committed["result"]["job_count"] == 2 + with closing(sqlite3.connect(str(db_path))) as conn: + source_state = conn.execute( + """ + SELECT outbox.status, attempts.status + FROM connector_outbox AS outbox + JOIN connector_deliveries AS attempts + ON attempts.outbox_id = outbox.id + WHERE outbox.delivery_key = ? + """, + (source["key"],), + ).fetchone() + assert source_state == ("awaiting_ack", "awaiting_ack") first_outer = client.request( "connector.poll", @@ -252,10 +291,36 @@ def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( # The provider accepted the message and Herdres persisted the binding, # but the connector ACK was lost before Tendwire received it. - provider_sends.append("provider-message-17") - provider_message_by_key[first["key"]] = provider_sends[-1] - current += timedelta(seconds=6) + with closing(sqlite3.connect(str(binding_db))) as conn, conn: + conn.execute( + """ + INSERT INTO provider_message_bindings ( + delivery_key, provider_message_id, payload_json + ) VALUES (?, ?, ?) + """, + ( + first["key"], + "provider-message-17", + json.dumps( + first["payload"], + sort_keys=True, + separators=(",", ":"), + ), + ), + ) + # The connector and its socket server have stopped. Advance beyond the lease, + # then construct fresh connector/server objects from the durable databases. + current += timedelta(seconds=6) + restarted_outbox = ConnectorOutboxAPI(db_path, "daemon-host") + restarted_api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=restarted_outbox.dispatch, + ) + + with _socket_client(tmp_path, restarted_api) as client: reclaimed = client.request("connector.reclaim", {"name": "turn-final"}) second_outer = client.request( "connector.poll", @@ -288,7 +353,35 @@ def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( # A thin connector recognizes the durable key, does not resend, and # acknowledges the current attempt rather than the stale first ref. - assert provider_message_by_key[second["key"]] == "provider-message-17" + with closing(sqlite3.connect(str(binding_db))) as conn, conn: + durable_binding = conn.execute( + """ + SELECT provider_message_id, payload_json + FROM provider_message_bindings + WHERE delivery_key = ? + """, + (second["key"],), + ).fetchone() + mutation_count_before = conn.total_changes + if durable_binding is None: + conn.execute( + """ + INSERT INTO provider_message_bindings ( + delivery_key, provider_message_id, payload_json + ) VALUES (?, ?, ?) + """, + (second["key"], "unexpected-second-send", "{}"), + ) + mutation_count_after = conn.total_changes + provider_row_count = conn.execute( + "SELECT COUNT(*) FROM provider_message_bindings" + ).fetchone()[0] + assert durable_binding == ( + "provider-message-17", + json.dumps(first["payload"], sort_keys=True, separators=(",", ":")), + ) + assert mutation_count_after == mutation_count_before + assert provider_row_count == 1 acknowledged = client.request( "connector.ack", { @@ -300,7 +393,6 @@ def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( assert acknowledged["ok"] is True assert acknowledged["result"]["ok"] is True assert acknowledged["result"]["status"] == "acknowledged" - assert provider_sends == ["provider-message-17"] next_item = client.request("connector.poll", {"name": "turn-final"})[ "result" ]["items"][0] @@ -314,6 +406,18 @@ def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( assert client.request("connector.poll", {"name": "turn-final"})["result"][ "items" ] == [] + with closing(sqlite3.connect(str(db_path))) as conn: + completed_source_state = conn.execute( + """ + SELECT outbox.status, attempts.status + FROM connector_outbox AS outbox + JOIN connector_deliveries AS attempts + ON attempts.outbox_id = outbox.id + WHERE outbox.delivery_key = ? + """, + (source["key"],), + ).fetchone() + assert completed_source_state == ("delivered", "delivered") @pytest.mark.skipif( @@ -352,6 +456,84 @@ def test_socket_keeps_transport_and_connector_errors_in_separate_envelopes( assert protocol_error["error"]["code"] == "unknown_method" +@pytest.mark.skipif( + not hasattr(__import__("socket"), "AF_UNIX"), + reason="Unix sockets required", +) +def test_socket_explicit_retry_resets_generation_attempt_and_bounds_history( + tmp_path: Path, +) -> None: + db_path = tmp_path / "retry-generation.db" + _canonical_final_turn(db_path, host_id="daemon-host") + outbox = ConnectorOutboxAPI(db_path, "daemon-host", max_attempts=2) + api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=outbox.dispatch, + ) + + with _socket_client(tmp_path, api) as client: + first = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert first["attempt"] == 1 + assert client.request( + "connector.fail", + {"name": "turn-final", "ref": first["ref"], "delay_seconds": 0}, + )["result"]["status"] == "retry_scheduled" + second = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert second["key"] == first["key"] + assert second["attempt"] == 2 + assert client.request( + "connector.fail", + {"name": "turn-final", "ref": second["ref"], "delay_seconds": 0}, + )["result"]["status"] == "attempts_exhausted" + inspected = client.request( + "connector.inspect", + { + "schema_version": 1, + "name": "turn-final", + "status": "dead_letter", + "limit": 10, + }, + )["result"] + assert inspected["items"][0]["attempt_count"] == 2 + retried = client.request( + "connector.retry", + { + "schema_version": 1, + "name": "turn-final", + "key": first["key"], + }, + )["result"] + assert retried["status"] == "requeued" + assert retried["prior_attempt_count"] == 2 + + with closing(sqlite3.connect(str(db_path))) as conn: + compacted = conn.execute( + """ + SELECT COUNT(attempts.id), outbox.private_state_json + FROM connector_outbox AS outbox + LEFT JOIN connector_deliveries AS attempts + ON attempts.outbox_id = outbox.id + WHERE outbox.delivery_key = ? + GROUP BY outbox.id + """, + (first["key"],), + ).fetchone() + assert compacted is not None + assert compacted[0] == 0 + assert json.loads(compacted[1])["prior_attempt_count"] == 2 + fresh = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert fresh["key"] == first["key"] + assert fresh["attempt"] == 1 + + @pytest.mark.skipif( not hasattr(__import__("socket"), "AF_UNIX"), reason="Unix sockets required", @@ -359,41 +541,105 @@ def test_socket_keeps_transport_and_connector_errors_in_separate_envelopes( def test_socket_preserves_opaque_connector_tokens_byte_for_byte( tmp_path: Path, ) -> None: - expected = { - "plan_token": "twplan1.AaZz09_-ExactPlan", - "replaces_plan_token": "twplan1.Replaced_AaZz09-", - "content_revision": "twrev1.AaZz09_-ExactRevision", - "final_identity": "twfinal1.AaZz09_-ExactFinal", - } - key = f"turn-final:revision:{expected['final_identity']}" + db_path = tmp_path / "token-preservation.db" + turn_id, revision = _canonical_final_turn(db_path, host_id="daemon-host") + with closing(sqlite3.connect(str(db_path))) as conn: + key, raw_source_payload = conn.execute( + """ + SELECT delivery_key, payload_json + FROM connector_outbox + WHERE delivery_kind = 'final_ready' + """ + ).fetchone() + expected_source = json.loads(raw_source_payload) + outbox = ConnectorOutboxAPI(db_path, "daemon-host") api = TendwireDaemonAPI( get_snapshot=lambda: Snapshot(host_id="daemon-host"), get_health=lambda: {}, submit_command=lambda _params: {}, - connector_call=lambda _method, _params: { - "schema_version": 1, - "ok": True, - "status": "ok", - "host_id": "daemon-host", - "name": "turn-final", - "items": [ - { - "ref": "twref1.AaZz09_-ExactRef", - "key": key, - "attempt": 1, - "payload": {**expected, "operation": "upsert"}, - } - ], - }, + connector_call=outbox.dispatch, ) with _socket_client(tmp_path, api) as client: - response = client.request("connector.poll", {"name": "turn-final"}) + source = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert source["key"].encode() == key.encode() + assert source["payload"]["final_identity"].encode() == expected_source[ + "final_identity" + ].encode() + assert source["payload"]["content_revision"].encode() == revision.encode() + assert source["payload"]["content"]["content_revision"].encode() == ( + expected_source["content"]["content_revision"].encode() + ) - item = response["result"]["items"][0] - assert item["key"] == key - for token_name, token in expected.items(): - assert item["payload"][token_name] == token + begun = client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "begin", + "name": "turn-final", + "turn_id": turn_id, + "content_revision": revision, + "presentation_version": "presentation-v2", + "part_count": 1, + "source_ref": source["ref"], + }, + )["result"] + assert begun["ok"] is True, begun + plan_token = begun["plan_token"] + assert client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "part", + "name": "turn-final", + "plan_token": plan_token, + "ordinal": 0, + "spans": [ + { + "field": "assistant_final_text", + "start_char": 0, + "end_char": 8, + } + ], + }, + )["result"]["ok"] is True + assert client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "commit", + "name": "turn-final", + "plan_token": plan_token, + "source_ref": source["ref"], + }, + )["result"]["ok"] is True + part = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + + with closing(sqlite3.connect(str(db_path))) as conn: + stored_part = json.loads( + conn.execute( + "SELECT payload_json FROM connector_outbox WHERE delivery_key = ?", + (part["key"],), + ).fetchone()[0] + ) + for path in ( + ("plan_token",), + ("content_revision",), + ("turn", "final_identity"), + ("turn", "content_revision"), + ("turn", "content", "content_revision"), + ): + actual: Any = part["payload"] + expected: Any = stored_part + for field in path: + actual = actual[field] + expected = expected[field] + assert actual.encode() == expected.encode() + assert part["payload"]["plan_token"].encode() == plan_token.encode() def test_daemon_api_routes_connector_methods_safely(tmp_path: Path) -> None: diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index f450118..90f1079 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -2234,8 +2234,10 @@ def test_unrelated_plans_poll_concurrently_but_never_colease_siblings( assert {item["payload"]["sequence_index"] for item in items} == {0} -def test_replacement_waits_for_old_lease_then_activates_without_requeue( +@pytest.mark.parametrize("terminal_action", ["release", "fail", "defer"]) +def test_replacement_terminalizes_old_lease_then_activates_without_requeue( tmp_path: Path, + terminal_action: str, ) -> None: db_path = tmp_path / "prepare-replacement-barrier.db" turn_id, revision = _canonical_turn(db_path, final_text="abcdefgh") @@ -2264,10 +2266,15 @@ def test_replacement_waits_for_old_lease_then_activates_without_requeue( assert new["state"] == "waiting_predecessor" assert api.poll({"name": "turn-final", "limit": 10})["items"] == [] - terminalized = api.fail( - {"name": "turn-final", "ref": leased_old["ref"], "delay_seconds": 0} - ) + terminal_params: dict[str, Any] = { + "name": "turn-final", + "ref": leased_old["ref"], + } + if terminal_action in {"fail", "defer"}: + terminal_params["delay_seconds"] = 0 + terminalized = getattr(api, terminal_action)(terminal_params) assert terminalized["status"] == "superseded" + assert "available_at" not in terminalized activated = api.poll({"name": "turn-final", "limit": 10})["items"] assert len(activated) == 1 assert activated[0]["payload"]["plan_token"] == new["plan_token"] From 738220bdcef4d0c455dec26a2f428a03d11f7e62 Mon Sep 17 00:00:00 2001 From: plotarmordev Date: Wed, 5 Aug 2026 10:58:18 +0800 Subject: [PATCH 83/83] docs: add wave 4 store design --- docs/wave4-store-design.md | 1235 ++++++++++++++++++++++++++++++++++++ 1 file changed, 1235 insertions(+) create mode 100644 docs/wave4-store-design.md diff --git a/docs/wave4-store-design.md b/docs/wave4-store-design.md new file mode 100644 index 0000000..732736f --- /dev/null +++ b/docs/wave4-store-design.md @@ -0,0 +1,1235 @@ +# Wave 4 T6 store design + +Status: design candidate. Production implementation is blocked until this note +passes adversarial review and the paired Herdres connector contract and tests +accept the same payload versions, recovery rules, and provider-binding model. + +Baseline: Tendwire `9ab5597b55bb918ac8e62c86651ccac7c03d18fb` and +Herdres `ec0e36a25b16469979402abfc8bfa1525db6f68a`. + +## Non-negotiable connector boundary + +T6 keeps the authoritative connector boundary already in production: + +- The connector name is exactly `turn-final`. +- The outer `connector.poll` item `key` is the sole durable replay and + provider-message binding identity. It is stable across lease expiry, reclaim, + release, defer, retry, daemon restart, and lost ACK responses. +- `ref` is an attempt-scoped `twref1.` capability. Only the newest live ref may + settle or renew an attempt. +- `attempt` starts at one and increases within a retry generation. Explicit + dead-letter retry starts a new generation at attempt one without changing the + outer key. +- The socket methods remain exactly `connector.prepare`, `connector.poll`, + `connector.ack`, `connector.fail`, `connector.defer`, `connector.renew`, + `connector.release`, `connector.reclaim`, `connector.retry`, and + `connector.inspect`. +- `connector.prepare` remains valid only for `name: "turn-final"`, request + `schema_version: 1`, and the existing `begin`, `part`, `commit`, and `recover` + action shapes. No required or optional request field is added. +- `connector.inspect` and `connector.retry` retain their existing strict request + field shapes. Contract-v3 inspect covers every retained `turn-final` dead + letter. Retry keeps the one `key` or `final_identity` selector: final identity + selects a final root, while key selects an exact retryable root, standalone + decision, or standalone retire as defined below. + +The AF_UNIX framing and RPC document remains connector transport contract v2; +top-level and inner operation envelopes continue to use JSON +`schema_version: 1`. The paired `turn-final` payload and operation-response +document revision is presentation contract v3. As with transport contract v2, +the document revision is not copied into every payload or result. The exact +kind-specific schema versions below identify the paired payload family. This +is not a new socket method, action, framing version, or replay identity. + +There is no `telegram-present` connector, alias connector, second presentation +queue, turn-list-driven final send, connector-specific socket method, inner +`job_key`, or copy of the current row's outer key as a parallel self-identity. +Lineage and retire payloads may reference another row's outer key only as an +explicit predecessor/replacement/target correlation; those references never +deduplicate or settle the current row. Herdres must continue to durably bind +the polled outer key to provider result before ACKing. That provider receipt +ledger remains required; the separate Herdres plan/pending-plan/recovery ledger +does not. + +## Scope, module budget, and gates + +T6 replaces `store/sqlite.py` with one fresh schema and concern-owned modules. +The target is 5,225 canonical SLOC, within the 4,500-5,500 gate. + +| Module | Responsibility | Budget | Public API | +| --- | --- | ---: | --- | +| `store/schema.py` | one DDL, exact-version cutover | 400 | `STORE_SCHEMA_VERSION`, `init_store`, `ensure_schema` | +| `store/db.py` | secure open, pragmas, read/write transactions, bounded health | 225 | `connect_read`, `read_transaction`, `write_transaction`, `store_status` | +| `store/events.py` | append, authoritative dedupe, bounded query | 300 | `record_agent_event`, `list_agent_events` | +| `store/turns.py` | atomic projection, content, list, delta, paging | 1,050 | append/apply result types, `append_agent_event_and_apply_turn_for_binding`, `apply_turn_refresh`, `turns_payload_from_store`, `turn_delta_payload_from_store`, `get_turn_content` | +| `store/projection.py` | snapshots, attention, bindings, route generation, health | 475 | snapshot context/save/latest, attention payload, binding upsert/list/expiry, backend-pending health | +| `store/pending.py` | pending observations and fenced decision claims | 425 | observation/payload, claim/start/abandon/terminal-effect functions | +| `store/receipts.py` | command receipts, submissions, replay and linking | 800 | current command reservation/send/finish/recovery/link functions | +| `store/outbox.py` | one queue, FIFO/DAG, leases, prepare, recovery, dead-letter | 1,350 | the current connector store functions only | +| `store/retention.py` | bounded cutoff deletion and WAL checkpoint | 200 | `RetentionPolicy`, `run_retention_cycle` | +| **Total** | | **5,225** | | + +The module rows still sum exactly 5,225; this correction adds no module or +second implementation path. Caller/configuration rewrites listed in file scope +replace existing lines and are measured in the repository-wide gate, not +double-counted as store-component SLOC. + +No function may exceed 150 lines. Orchestration functions target at most 130 +lines; transaction-local transition/query helpers target at most 60. There is +no compatibility re-export, generic repository/CRUD layer, migration chain, +authority registry, direct-SQLite CLI fallback, maintenance state machine, +automatic VACUUM, or compaction framework. + +## Fresh schema + +The application schema has 16 tables: + +- Projection: `turns`, `turn_content_revisions`, + `turn_content_page_boundaries`, `attention_items`, `pending_interactions`, + `snapshots`, and `agent_events`. +- Commands: `command_receipts`, `turn_submissions`, `turn_supersessions`, + `backend_pending`, `backend_pending_claims`, `worker_bindings`, and + `backend_health`. +- Delivery: `connector_outbox` and `connector_deliveries`. + +Explicitly absent are `events`, `spaces`, `workers`, `commands`, migration +tables, store-maintenance state/cursors, presentation plan/job/recovery tables, +turn-change state/floor tables, tombstone tables, and every `herdr_turn_*` +table. A retained removal row in `turns` is the delta tombstone. The greatest +turn/delta sequence row is retained as its allocation sentinel, and the +retained worker-binding allocator owns partition sequences, so a published +sequence is never reused. + +`turns` is keyed by `(host_id, turn_id)` and carries current ownership, +`route_generation`, insertion and change sequence, projection state, and +`removed_at`. There is no prepare-authority or source-less-presentation column. +Every final presentation plan is rooted in one polled `final_ready` outbox row. +`worker_bindings` carries exact columns `stable_key`, `stable_key_version`, +`route_generation`, `partition_key`, +`next_partition_sequence`, and `route_retain_until` alongside its private +backend binding. Content revisions are immutable, with one partial-unique +current row per turn. Page boundaries are revision/field/page-coordinate unique +and cascade only with a revision that retention has proved unreferenced. + +Command receipts preserve: + +```text +missing -> reserved -> send_started -> accepted | rejected | uncertain +``` + +Only an identical canonical request may take over an expired reservation; +terminal receipts are immutable. Turn submissions preserve the existing +send-started/submitted/uncertain/link/ambiguous/expired/cancelled semantics, +one-to-one linkage, owner/instruction matching, and fail-closed ambiguity. +Backend-pending claims remain fenced and are abandonable only before send +start. + +## Two delivery tables + +`connector_outbox` contains both immediately executable work and staged +`final_part` rows. Its logical columns are: + +```text +id, host_id, connector, key, kind, payload_version, status, +partition_key, partition_sequence, +turn_id, final_identity, content_revision, presentation_version, +plan_token, plan_generation, logical_sequence, logical_ordinal, +predecessor_outbox_id, replaces_outbox_id, target_outbox_id, +source_outbox_id, active_lineage_generation, +recovery_request_digest, recovered_from_plan_token, +terminal_after_lease, retry_generation, prior_attempt_count, +current_delivery_id, +payload_json, +created_at, updated_at, available_at +``` + +Required uniqueness is: + +```text +(host_id, connector, key) +(host_id, connector, partition_key, partition_sequence) +(host_id, connector, source_outbox_id, plan_generation, logical_sequence) +(host_id, connector, recovery_request_digest) WHERE recovery_request_digest IS NOT NULL +``` + +`status` has an exact database CHECK over `staged`, `blocked`, `queued`, +`leased`, `retry`, `deferred`, `awaiting_ack`, `delivered`, `superseded`, and +`dead_letter`. `kind` is CHECKed over `generic`, `working`, `final_ready`, +`final_part`, `retire`, and `decision`; `connector = 'turn-final'` requires one +of the five typed kinds, and every other connector requires `generic`. A +kind/status CHECK permits only the combinations in the matrix below, and a +kind/payload-version CHECK fixes working/retire/decision at one, final-part at +two, and final-ready at three. `predecessor_outbox_id`, `replaces_outbox_id`, `target_outbox_id`, and +`source_outbox_id` are nullable self-foreign keys with `ON DELETE RESTRICT`. +`current_delivery_id` is a nullable foreign key to `connector_deliveries(id)` +with `ON DELETE RESTRICT`; transition code additionally proves that the pointed +delivery's `outbox_id` is this row. Terminal outbox rows have a null current +delivery. Leased and awaiting-ACK rows have exactly one. + +`connector_deliveries` contains attempts only: + +```text +id, outbox_id, retry_generation, attempt, +ref_hash, status, leased_at, leased_until, ack_deadline_at, +public_response_json, private_reason_enum, +created_at, settled_at +``` + +`outbox_id` is non-null and references `connector_outbox(id) ON DELETE +RESTRICT`. Delivery `status` has an exact CHECK over `leased`, `awaiting_ack`, +`acknowledged`, `failed`, `deferred`, `released`, and `expired`. Required +uniqueness is `(outbox_id, retry_generation, attempt)` and `(ref_hash)`, plus a +partial unique live-attempt index on `outbox_id` where status is `leased` or +`awaiting_ack`. Raw refs are never stored. `public_response_json` is a bounded, +exact sanitized object; `private_reason_enum` is nullable and CHECKed against a +fixed enum: `temporary`, `rate_limited`, `provider_rejected`, +`provider_uncertain`, `invalid_payload`, `content_unavailable`, +`route_unavailable`, `provider_binding_unknown`, `lease_expired`, +`ack_deadline_expired`, `superseded`, `attempts_exhausted`, or +`operator_recovery`. There is no +`private_state_json` in either delivery table. + +`ack_deadline_at` is a canonical UTC timestamp or null. It is non-null exactly +when delivery status is `awaiting_ack`; it is null for ordinary leases and all +terminal attempts. The source-commit and recovery rules below are its only +initializers or mutators. + +Database constraints provide identity and deletion fencing; transaction code +provides the cross-row domain checks SQLite CHECKs cannot express. A staged or +executable `final_part` and every plan retire must reference exactly one +`final_ready` source of the same host, connector, turn, final identity, +revision, partition, and active lineage generation. Its predecessor, +replacement, and target must +have the same host/connector and the declared compatible route/revision. A +predecessor must have a lower logical sequence unless it is the delivered tail +of an inherited recovery prefix. Cycles, cross-root children, a root pointing +at its child as current delivery, or a delivery pointing at a different outbox +row abort the transaction. Root completion, supersession, retry, and recovery +update the root and every affected child in the same `BEGIN IMMEDIATE`. + +The exact nullability/correlation rules are also enforced on every write: + +- `working` and `decision` have no source, plan, logical-ordinal, target, or + recovery columns; only `replaces_outbox_id` may be set. +- `final_ready` has final identity/revision but no source, plan, predecessor, + target, or recovery parent. Its current delivery exists only while leased or + awaiting ACK. +- `final_part` has non-null plan token/generation/sequence/ordinal and a + non-null final-ready source. Its target is null; predecessor/replacement must + satisfy the declared lineage. +- A plan `retire` has plan coordinates, non-null target and predecessor, and the + same non-null final-ready source as its lineage. A standalone retire has no + plan/source columns, has a non-null decision target, and uses only the + decision-resolution predecessor rule. +- Only recovery-created executable rows have `recovered_from_plan_token`. + Exactly the first fresh row of a recovery generation carries the non-null + `recovery_request_digest`; every other row has null, so the declared unique + index is executable and that head is retained for the full lineage horizon. + +The outer key grammars are exact: + +```text +final_ready: turn-final:revision:twfinal1.<43 base64url characters> +final_part or plan retire: turn-final:twplan1.<1-256 base64url characters>: +working: turn-final:working:twwork1.<43 base64url characters> +decision: turn-final:decision:twdecision1.<43 base64url characters> +standalone retire: turn-final:retire:twretire1.<43 base64url characters> +``` + +Contract-v3 Tendwire final identities are always 32-byte digests encoded as 43 +unpadded base64url characters. A contract-v3 `final_ready` payload, outer key, +prepare source root, inspect selector, and retry selector reject every other +length. Existing non-v3 RPC positions that accept opaque `twfinal1.*` or +`twplan1.*` values may retain their transport-v2 bounded 1--256-character +validator, but that compatibility does not widen the five-kind contract. + +Producer identities are deterministic canonical SHA-256 digests, not random +per enqueue. `twwork1` hashes the domain, host, turn ID, content revision, and +route generation. `twdecision1` hashes the domain, host, decision ref, revision +digest, and route generation. `twretire1` hashes the domain, target decision +outer key, the fixed `decision_resolved` reason, resolving revision, and route +generation. There is no route-retirement producer. Replaying the same producer +transaction therefore finds the same outer key and must prove an identical +payload; a mismatch is a conflict. Plan tokens are the one persisted opaque +value allocated by idempotent begin, and plan child keys are deterministic from +that token and logical sequence. + +These tokens occur only as parts of the outer key or as non-replay correlation +coordinates declared below. No payload has `key`, `job_key`, `delivery_key`, +`ref`, or `attempt`. + +## Outbox status matrix + +| Kind | Legal statuses | Legal atomic transitions | +| --- | --- | --- | +| `generic` (non-`turn-final` queues only) | `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | preserves the existing neutral enqueue/poll/ACK/fail/defer/release/reclaim transitions; it cannot participate in prepare, route partitions, final lineage, or the five-kind payload contract | +| `working` | `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | enqueue to `queued`; due poll to `leased`; ACK to `delivered`; fail to `retry` or `dead_letter`; defer to `deferred`; release/expiry to `queued`; unleased supersession to `superseded`; leased supersession sets `terminal_after_lease`, after which ACK is `delivered` and fail/defer/release/expiry is `superseded` | +| `final_ready` | `queued`, `leased`, `retry`, `deferred`, `awaiting_ack`, `delivered`, `superseded`, `dead_letter` | ordinary lease transitions; first successful commit `leased` to `awaiting_ack`; complete effective lineage to `delivered`; ACK-deadline expiry to `dead_letter`; only an uncommitted root dead letter may retry to `queued` in a new retry generation; recovery replaces a failed suffix while the committed root remains live `awaiting_ack`; an uncommitted superseded root becomes `superseded`; a committed terminal root is never re-prepared from scratch | +| `final_part` | `staged`, `blocked`, `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | begin creates `staged` placeholders; part fills one staged row idempotently; commit freezes payloads and moves the head to `queued` and successors to `blocked`; predecessor ACK unblocks exactly one successor; ordinary lease transitions; recovery retains a delivered prefix, supersedes the failed suffix, and creates a new suffix | +| `retire` | `blocked`, `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | commit/resolution creates `blocked`; a delivered replacement normally moves it to `queued`, with the exact leased-decision exception below; ordinary lease transitions; supersession is legal only when Tendwire proves the target never had a delivery attempt and therefore could not have been provider-accepted; a mandatory retire for a leased, expired, failed, deferred, acknowledged, or otherwise possibly accepted target remains retryable/dead-letter work and is never superseded | +| `decision` | `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | pending projection/enqueue to `queued`; ordinary lease transitions; resolution atomically supersedes unleased work or marks leased work terminal-after-lease and creates the ordered retire | + +`staged`, retained `awaiting_ack` roots, and terminal rows do not block FIFO. +An executable row is pollable only when it is due, its explicit predecessor is +`delivered`, and no earlier executable row in the same partition is in +`queued`, `leased`, `retry`, or `deferred`. Poll reclaims expired leases before +selection in the same `BEGIN IMMEDIATE` transaction. + +The one explicit predecessor exception is a mandatory standalone retire for a +resolved decision that was already leased. It becomes eligible after the target +decision has no live attempt and is terminal as `delivered`, `superseded`, or +`dead_letter`, because any prior lease may have reached the provider even when +Tendwire did not receive an ACK. A decision superseded before its first lease +cannot have a provider object; its retire is provably unnecessary and may be +superseded. Replacement-driven working/final retires still require the +replacement key itself to be `delivered`. + +For that decision exception the retire payload has `predecessor_key` equal to +the target decision key. Herdres does not require a delivered provider job for +that key: it performs its immutable target lookup. If H7 has no provider job, +alias, or tombstone for a possibly accepted target, it returns the fixed +`provider_binding_unknown` failure. Tendwire immediately dead-letters the +mandatory retire, keeps its target/reason visible through inspect for the +30-day targetable horizon, and never converts absence into ACK, deletion, or +"unnecessary". Explicit retry may make the same target visible again but +cannot reconstruct an unknown provider coordinate. + +## Delivery-attempt and CAS matrix + +| Operation | Required current state | `connector_deliveries` transition | `connector_outbox` transition | +| --- | --- | --- | --- | +| poll | due eligible work, no blocker | insert `leased` at next attempt | executable work to `leased` | +| renew | newest unexpired ref; both rows `leased` | extend `leased_until` | remains `leased` | +| ACK | newest unexpired ref; both rows `leased` | `leased` to `acknowledged` | work to `delivered`; atomically unblock successor | +| fail, ambiguous new send | newest unexpired ref; exact `provider_uncertain` classification | `leased` to `failed` | immediate `dead_letter` regardless of attempt budget; effective root lineage updated atomically | +| fail, mandatory retire has unknown binding | newest unexpired ref; exact `provider_binding_unknown` classification | `leased` to `failed` | immediate visible `dead_letter`; target is retained and never inferred absent | +| fail, budget remains | newest unexpired ref; both rows `leased` | `leased` to `failed` | work to `retry` | +| fail, exhausted | newest unexpired ref; both rows `leased` | `leased` to `failed` | work/effective lineage to `dead_letter` | +| defer | newest unexpired ref; both rows `leased` | `leased` to `deferred` | work to `deferred` | +| release | newest unexpired ref; both rows `leased` | `leased` to `released` | work to `queued`, or `superseded` when terminal-after-lease | +| lease expiry | current `leased`, deadline due | `leased` to `expired` | work to `queued`, or `superseded` when terminal-after-lease | +| source commit | live final-ready source ref | `leased` to `awaiting_ack`; initialize `ack_deadline_at` once | root to `awaiting_ack`; head queued and tail blocked | +| lineage complete | every effective node delivered before the exact ACK deadline | `awaiting_ack` to `acknowledged`; clear deadline | root to `delivered` | +| lineage deadline | no live child lease; exact ACK deadline reached | `awaiting_ack` to `failed`; clear deadline | root and effective tail to `dead_letter` | +| explicit root retry | exact retryable root dead letter | prior generation remains terminal/aggregated | same root key to `queued`, retry generation +1; next attempt is one | +| recover | exact recoverable failed lineage under a live awaiting-ACK root | same attempt remains `awaiting_ack`; one new request digest replaces its deadline | prefix retained; failed non-retire tail superseded; possibly accepted mandatory retire audit rows retained dead-letter and linked to their fresh copies; fresh suffix created | + +Every ref mutation uses one CAS equivalent to: + +```text +outbox.id = delivery.outbox_id +AND outbox.host_id = requested host +AND outbox.connector = requested connector +AND outbox.status = expected outbox status +AND delivery.id = outbox.current_delivery_id +AND delivery.status = expected delivery status +AND delivery.ref_hash = H(presented ref) +AND delivery.leased_until > now +``` + +Zero changed rows returns the existing stale/invalid-ref result and commits no +partial transition. An old ref cannot mutate a later attempt. An explicit +retry never revives an old ref or attempt counter. + +The retained configuration is exactly +`TENDWIRE_CONNECTOR_ACK_TTL_SECONDS`/`connector_ack_ttl_seconds`, default 60. +Startup accepts only a non-Boolean integer in `1..86400`; it never clamps. The +first successful source commit computes `ack_deadline_at = commit_now + ttl` +inside the commit transaction. A byte-equivalent commit replay returns the +persisted deadline and never extends it. Lineage completion CASes the exact +root/current-delivery pair only while `ack_deadline_at > now`; deadline expiry +CASes that same pair only when `ack_deadline_at <= now` and no child has a live +lease. Exactly one successful recover request digest may replace the still-live +awaiting-ACK deadline with `recover_now + ttl`; an identical request replay +returns the persisted replacement deadline, and a different request must +create the next recovery generation or fail conflict. Recover after deadline +expiry is rejected. The committed expired root is terminal and +`not_retryable`; only a later producer revision may create unrelated new work. +Retention treats every non-null deadline and its effective lineage +as live, and cannot delete either until the root is terminal and the applicable +reference horizon has elapsed. + +Inspect request fields remain exactly `schema_version: 1`, +`name: "turn-final"`, `status: "dead_letter"`, and integer `limit` in `1..100`. +A successful inner result has exactly: + +```text +schema_version, ok, status, host_id, name, total, items +``` + +Its values are `schema_version=1`, `ok=true`, `status="ok"`, and `total` is the +full matching root/effective-lineage count before the limit. Every item has all +of these exact fields, with inapplicable correlations present as null: + +```text +kind, key, final_identity, failed_plan_token, decision_ref, target_key, +reason, attempt_count, prior_attempt_count, created_at, terminal_at, retryable, +recoverable +``` + +`reason` is one value from the fixed public reason enum corresponding to the +stored private classification. `retryable` is true only when the exact +`connector.retry` selector can currently act; `recoverable` is true only for an +exact failed plan whose source root is still awaiting ACK before its deadline +and can accept `connector.prepare(action="recover")`. Items contain no text, +payload, ref, provider fact/coordinate, private route, or raw diagnostic. Root +and plan-child failures are reported once at the root/effective-lineage level. +Inspect errors have exactly `schema_version`, `ok`, `status`, `host_id`, `name`, +`message`, and `items`, with `ok=false`, `items=[]`, and the common fixed error +rules below. + +Retry keeps its exact mutually exclusive selector shapes. `final_identity` +and a final-ready key requeue only an exact uncommitted dead-letter root in a +new retry generation; the root is then polled and prepared with its new live +ref. A committed terminal root is `not_retryable` and is never prepared from +scratch. `connector.retry` never invokes plan recovery. A still-awaiting-ACK +root with a recoverable failed suffix uses only +`connector.prepare(action="recover")`. +A standalone decision key or standalone +retire key starts a new retry generation on that same outer key, with attempt +one and accumulated prior-attempt count. A plan retire/final-part key is not +independently retried because root recovery owns its suffix. Working dead +letters are observationally replaced only by a newer deterministic producer +revision and are not operator-retried. Unknown, terminal-success, stale +revision, nonstandalone child, and ambiguous selectors return fixed +`not_retryable`/`stale_revision`/`invalid_params` outcomes without mutation. + +Provider-start classification is exact. A definitely-not-started provider call +uses `temporary` or `rate_limited` and follows the ordinary bounded retry/defer +path. An ambiguous new send for `working`, `final_part`, or `decision` uses +`provider_uncertain`; fail ignores remaining attempt budget and atomically moves +that row/effective lineage to `dead_letter` for inspection. Automatic retry is +forbidden because it can duplicate a message. Ambiguous known-target edit or +delete does not use `provider_uncertain`: Herdres defers the same immutable key +and target, then converges by exact not-modified or not-found handling. Invalid +reason/operation combinations are `invalid_params` without mutation. Explicit +operator retry/recover of a `provider_uncertain` row is allowed only with a +visible warning that the provider may already have accepted the send and the +operation can create a duplicate; recovery preserves the prior ambiguous audit +row and never claims the risk was repaired. + +## Exact additive payload contract + +The existing poll item envelope is unchanged: + +```json +{ + "key": "...", + "ref": "twref1....", + "attempt": 1, + "leased_until": "2026-08-05T00:00:00Z", + "available_at": "2026-08-05T00:00:00Z", + "created_at": "2026-08-05T00:00:00Z", + "payload": {} +} +``` + +Each payload has exactly these common keys: + +```text +schema_version, kind, created_at, worker, route +``` + +The kind-specific `schema_version` values below version that payload object; +they do not change the transport-v2 envelope's JSON schema version 1. + +`worker` is exactly: + +```text +worker_id, stable_key, stable_key_version, route_generation +``` + +`route` is exactly: + +```text +partition_key, partition_sequence +``` + +The kind-specific versions and keys are: + +### `working`, schema version 1 + +Adds only `turn`. `turn` is exactly: + +```text +turn_id, content_revision, replaces_key, text +``` + +`replaces_key` is an outer key or null. `text` is exactly: + +```text +assistant_stream_text, char_length, byte_length +``` + +The text is a bounded sanitized inline projection, and the two lengths must +match its exact Unicode scalar and UTF-8 byte lengths. + +### `final_ready`, schema version 3 + +Adds only `turn`. `turn` is exactly: + +```text +turn_id, final_identity, content_revision, replaces_key, content +``` + +`replaces_key` is an outer working/final key or null. `content` remains exact +content-schema-v1: + +```text +schema_version, content_revision, known_incomplete, fields +``` + +`fields` has exactly `user_text` and `assistant_final_text`; each descriptor is +exactly `availability`, `inline`, `char_length`, `byte_length`, `page_count`, +and `first_cursor`. The final-ready revision, nested revision, cursors, lengths, +and outer-key final identity must correlate exactly. + +### `final_part`, schema version 2 + +Adds exactly `turn`, `plan`, and `lineage`. + +```text +turn: turn_id, final_identity, content_revision +plan: plan_token, generation, presentation_version, ordinal, part_count, spans +lineage: recovered_from_plan_token, predecessor_key, replaces_key +``` + +Nullable lineage values are present as null, not omitted. Each span is exactly +`field`, `start_char`, and `end_char`; field is `user_text` or +`assistant_final_text`. Spans are nonempty, ordered, nonoverlapping exact +character slices of the retained revision. No final text is copied into a +plan row. + +### `retire`, schema version 1 + +Adds exactly `turn` and `retire`. + +```text +turn: turn_id, final_identity, content_revision +retire: target_key, target_kind, target_ordinal, predecessor_key, + plan_token, generation, reason +``` + +Nullable coordinates are present as null. `target_kind` is `working`, +`final_part`, or `decision`. Reason is exactly one of `working_replaced`, +`final_replaced`, `excess_part`, or `decision_resolved`. +There is no content or provider coordinate. + +### `decision`, schema version 1 + +Adds only `decision`, exactly: + +```text +decision_ref, revision_digest, mode, title, body, choices +``` + +`mode` is `single`, `multi`, or `plan`. Each choice is exactly `ordinal`, +`option_ref`, and `label`. `option_ref` is the public one-based ordinal from the +semantic decision contract; the private ACP option ID and backend route remain +only in `backend_pending`. + +Unknown keys, unknown versions/kinds, booleans used as integers, noncanonical +timestamps, malformed opaque tokens, duplicate ordinals, inconsistent lengths, +cross-revision cursors/spans, and correlation mismatch reject enqueue, prepare, +or poll before any connector-visible mutation. + +## Route generation and concurrency + +`route_generation` has exact grammar: + +```text +twroute1.<43 unpadded URL-safe base64 characters> +``` + +It encodes 32 cryptographically random bytes. In the atomic worker-binding +projection, Tendwire performs lookup-or-mint for this exact private tuple: + +```text +(host_id, stable_key_version=1, stable_key, backend, + binding_private_fingerprint) +``` + +An identical retained tuple reuses its generation. A changed stable identity, +backend/private binding fingerprint, deliberate installation-key rotation, or +resolved live-route collision mints a new generation. Ordinary daemon/Herdres +restart, worker label change, content update, lease attempt, or local Herdres +JSON revision does not rotate it. Multiple live route tuples claiming the same +stable identity fail closed. + +Route enrichment is part of the same snapshot/binding `BEGIN IMMEDIATE` that +accepts the authoritative worker observation. It validates the derived stable +key pair, selects or mints the route generation, derives the partition key, +stores all route columns on `worker_bindings`, and copies the exact public +stable-key/version/generation triplet into the worker and affected turn +projection before commit. No snapshot, turn, delta, pending decision, or +outbox producer may publish a partially enriched route. Direct binding upsert +uses the same helper and cannot invent a generation independently. + +`save_snapshot` and the binding transaction return the persisted enriched +snapshot, never the caller's pre-enrichment object. Before persistence and +return, every `Worker.fingerprint` is recomputed from the exact canonical +public Worker projection excluding the fingerprint itself and volatile +timestamps but including `stable_key`, `stable_key_version`, and the non-null +`route_generation`. Snapshot `content_fingerprint` is then recomputed from those +enriched workers. The coordinator replaces its in-memory snapshot with this +returned value before publishing, routing, or accepting a command. + +Post-T6 command reservation/send CASes the exact current `worker_id`, enriched +`worker_fingerprint`, stable key/version, route generation, and private binding +fingerprint in the same transaction. A mismatch is stale route and authorizes +no backend send. H8's one-time removal of `worker_fingerprint` from a stored +command is legal only before the paired cutover when the captured route +generation is null. Once a command carries a non-null `twroute1.*`, fingerprint +removal is forbidden; a mismatch quarantines rather than weakening the CAS. + +The generation is published in worker metadata, copied into current turn/list/ +delta ownership, and frozen in every queued payload. It is not a Telegram +topic generation and is never compared to Herdres's local state revision. + +`partition_key` has exact grammar `twpart1_` plus 64 lowercase hexadecimal +characters and is SHA-256 over canonical UTF-8 JSON: + +```json +{ + "domain": "tendwire.turn-final.partition.v1", + "host_id": "...", + "route_generation": "twroute1....", + "stable_key": "wsk1_...", + "stable_key_version": 1 +} +``` + +Different Tendwire partitions may lease concurrently. After resolving a +public route, Herdres must additionally serialize provider writes by its +private physical `(bot, chat, topic)` coordinate. Two route generations mapped +to one retained topic therefore cannot concurrently edit or delete it. Topic +creation and provider routing remain Herdres-owned; Tendwire sees only neutral +defer/fail/release/ACK outcomes. + +`worker_bindings.next_partition_sequence` is the durable allocator for its +exact partition. Every producer transaction atomically increments it with +`UPDATE ... RETURNING` and inserts the outbox row with the returned positive +sequence; `MAX(outbox.sequence)+1` is never used. The binding/allocator row is +retained while current/sendable, referenced by any Tendwire turn or outbox row, +or before its Tendwire-owned `route_retain_until`. + +The paired constants are exact minimum floors: terminal targetable Tendwire +outbox rows and their inspect/retry correlation are retained for 30 days; +referenced content revisions, page boundaries, worker bindings, route +generations, partition allocators, and installation-key derivation evidence are +retained for 45 days; H7 immutable provider jobs/messages/aliases/tombstones are +retained for 60 days. The corresponding names are +`TURN_FINAL_TARGETABLE_RETENTION_DAYS = 30`, +`TURN_FINAL_ROUTE_CONTENT_RETENTION_DAYS = 45`, and the cross-repository H7 +`PROVIDER_FACT_RETENTION_DAYS = 60`. Configuration may lengthen but never +shorten these floors. Live references, awaiting-ACK roots, current/recovered +lineage, active routes, and unresolved mandatory retires override every cutoff. +After references and the applicable horizon are both gone, retention may remove +the rows child-first. Tendwire neither reads nor infers Herdres provider +bindings and never shortens a horizon based on unknown provider state. + +## Prepare lease, idempotence, and response-loss recovery + +Prepare requests keep transport-v2 `schema_version: 1` and the four existing +action field sets. Contract v3 removes source-less preparation completely. +`begin` requires `source_ref`; the first `commit` requires `source_ref`; omission +or explicit null is `invalid_ref`. `part` remains token-fenced and has no +source-ref field. `recover` is the sole recovery action and accepts only its +unchanged `failed_plan_token` and `request_id` fields. No internal producer, +startup scan, retry path, or rediscovery path may call `begin` without a polled +source root. + +Successful action results have the following exact fields and no others: + +```text +begin: + schema_version, ok, status, host_id, name, plan_token, state, generation, + part_count, accepted_ordinals +part: + schema_version, ok, status, host_id, name, plan_token, state, generation, + part_count, ordinal, accepted_ordinals +commit: + schema_version, ok, status, host_id, name, plan_token, state, generation, + part_count, job_count, accepted_ordinals +recover: + schema_version, ok, status, host_id, name, failed_plan_token, plan_token, + generation, content_revision, state, acknowledged_prefix_count, + executable_job_count, retained_failed_job_count, prior_attempt_count, + idempotent_replay +``` + +`schema_version` is one. `ok` is true. Begin/part/commit `status` is `ok`; +recover status is `recovered`. `accepted_ordinals` is a sorted unique bounded +array of non-Boolean zero-based integers in `[0, part_count)`, with +`part_count` bounded to `1..10000`. Begin returns all already accepted ordinals, +part includes its exact `ordinal`, and commit returns every ordinal +`0..part_count-1`. `generation`, `part_count`, `ordinal`, and every count reject +Boolean values; generation and part count are positive and the other counts are +nonnegative. Begin/part state is exactly `preparing`, `active`, +`waiting_predecessor`, `completed`, `failed`, or `superseded`; commit state +excludes `preparing`, and recover state is exactly `active`. An identical replay +returns the same persisted fields and never recomputes a token or count. The +persisted ACK deadline is an internal lifecycle fence and is not added to the +transport-v2 result. Recover deliberately has no `accepted_ordinals` field +because it creates an executable suffix rather than a preparing upload set. + +Every prepare error has exactly `schema_version`, `ok`, `status`, `host_id`, +`name`, and `message`; `schema_version=1`, `ok=false`, and `message` is exactly +the same fixed enum token as `status`. The closed status/message set is +`invalid_params`, `invalid_ref`, `stale_ref`, `store_unavailable`, +`revision_not_found`, `stale_revision`, `content_unavailable`, `plan_not_found`, +`plan_conflict`, `part_conflict`, `plan_incomplete`, `plan_not_failed`, +`not_recoverable`, `request_conflict`, and `ack_deadline_expired`. Errors expose +no token, ordinal, count, deadline, payload, route, provider fact, or private +coordinate. + +Generation-one begin identity is exact: + +```text +(source final-ready outer key, final_identity, content_revision, + route_generation, presentation_version, part_count, plan_generation=1) +``` + +Begin validates the live `source_ref`, inserts one opaque +`twplan1.*`, and creates `part_count` staged `final_part` placeholders in one +transaction. Repeating the same begin returns the same token, plan state, part +count, and sorted exact `accepted_ordinals`. A differing version or count is +`plan_conflict`. A repeated begin may bind the same staged plan to a newly +leased attempt of the same source outer key, never to another root, revision, +or route. + +Part retains its current request shape. The first exact span set fills its +ordinal. An identical retry succeeds and returns the same accepted ordinals; a +different set is `part_conflict`. Although part carries no source ref, Tendwire +internally requires the plan to remain bound to the source's current live +attempt. Staged payloads become immutable when commit makes them executable. + +The first commit requires the current live `source_ref`, all ordinals, exact +full coverage, and a still-current revision. It atomically freezes children, +creates retire nodes, installs the predecessor DAG, moves source outbox and +attempt from `leased` to `awaiting_ack`, initializes the persisted ACK deadline, +and exposes only the head. A repeated commit after success returns the persisted +committed token/state/generation/part/job counts and accepted ordinals +even when the supplied old source ref is no longer live. If the first commit +did not happen, the plan remains preparing and a current source ref is required. + +Herdres renews the source lease immediately after root validation, after every +bounded content-page batch, before every part upload, and immediately before +commit. A failed or uncertain renew stops preparation. The same renew may be +retried while the ref can still be live; otherwise Herdres waits for expiry, +repolls the same outer key, repeats begin, and resumes missing ordinals. + +Consequently: + +- Lost begin response: repeat begin and rediscover token/accepted ordinals. +- Lost part response: repeat the identical part. +- Crash or source expiry before commit: the source requeues, staged rows remain, + repoll returns the same outer key with a new ref, and begin rediscovers them. +- Lost commit response: committed child keys become pollable; repeating commit + also returns the identical committed result. +- Failed lineage recovery: use only `recover` while the committed root remains + awaiting ACK before its deadline. Source-less `begin` does not exist, and an + expired committed root is terminal and `not_retryable`. +- Restart requires no Herdres plan token, pending-plan, accepted-ordinal, or + recovery ledger. Provider receipts keyed by outer key remain authoritative. + +## Recovery lineage + +Recover accepts its unchanged `failed_plan_token` and `request_id` request. +Tendwire requires one linear logical sequence: a contiguous acknowledged +prefix followed by a failed nonleased suffix, plus the exact source root and +current delivery still in `awaiting_ack` with `ack_deadline_at > now`. A gap, +live child lease, unexpired retry/defer, absent source, or noncontiguous prefix +returns `not_recoverable`; an expired root returns `ack_deadline_expired`. A `provider_uncertain` +dead-letter is recoverable only through explicit operator action acknowledging +the duplicate-send risk; it is never selected by automatic recovery. + +In one transaction Tendwire: + +1. stores a unique digest of the bounded public request ID; +2. returns the same result for an identical replay; +3. allocates the next generation and a new plan token; +4. retains old delivered prefix rows and their original outer keys unchanged; +5. creates fresh-key copies only for the suffix, including retire nodes; +6. points the first new node to the last old acknowledged key, or leaves its + executable predecessor null when the prefix is empty; +7. records the failed plan token and replaced old suffix row on each new node; +8. marks failed non-retire suffix rows `superseded`, but keeps every possibly + accepted mandatory retire as a terminal dead-letter audit row linked to its + fresh replacement rather than claiming it became unnecessary; and +9. replaces the live root delivery's ACK deadline once with + `recover_now + connector_ack_ttl_seconds`; and +10. defines root completion as old delivered prefix plus new suffix. + +Recovery may chain when a replacement suffix later fails. It never revives an +old ref or attempt and never erases provider-acceptance ambiguity. An explicit +operator recovery can duplicate an operation whose provider acceptance was not +durably bound. The retained old `provider_uncertain` audit item and its attempt +count remain inspectable even when the recovery suffix later succeeds. + +The prefix-empty head is ordered by its fresh partition sequence after the old +failed tail has atomically become nonblocking; it never points to the retained +`awaiting_ack` root. Making the root its predecessor would deadlock because the +root cannot become delivered until that head and suffix are delivered. + +## Retire DAG and provider-message alias safety + +A retire payload addresses a logical outer key, never a provider message ID. +Its predecessor is the accepted replacement key. Tendwire does not make retire +eligible until that predecessor is `delivered`. The sole exception is the +standalone `decision_resolved` retire defined above: its predecessor/target may +be a terminal `delivered`, `superseded`, or `dead_letter` decision after a +lease, because provider acceptance may have preceded the terminal Tendwire +state. + +Herdres provider/presentation ownership has three durable logical states: + +```text +owns(provider_message_id) +alias(provider_message_id, current_owner_key) +retired +``` + +When a replacement edits or reuses an existing provider message, Herdres first +binds the replacement key as owner, records an ownership alias from the old +logical slot to the replacement key, durably checkpoints both changes, and only +then ACKs the replacement. The old outer-key `job_binding` itself is immutable: +its accepted operation, payload fingerprint, provider coordinate, and outcome +are never rewritten or rebound. Alias/current-owner state belongs only to the +provider-message and presentation-slot records and is never an alternate replay +lookup. + +On retire: + +- `owns` and still current owner: delete, durably tombstone, then ACK; +- `alias` whose current owner is a different delivered replacement: perform a + logical retire only and never delete the reused message; +- already retired, or absent with a durable tombstone: idempotent no-op ACK; +- absent with no immutable job, alias, or tombstone for a possibly accepted + mandatory target: fail `provider_binding_unknown`, never ACK; and +- uncertain provider response: fail closed and never claim deletion. + +Thus a delayed retire for an old working/final key cannot delete a message that +has since been edited into the accepted replacement. + +Tendwire cannot observe whether Herdres retained, aliased, lost, or deleted a +provider binding. Its only evidence is connector settlement and attempt state. +It therefore never declares a retire unnecessary because of assumed Herdres +state and never shortens a root, route, content, or retry horizon based on such +an assumption. + +## Producer transactions and root invariants + +- ACP working ingestion atomically appends the authoritative event, applies the + turn/revision/delta projection, allocates a route sequence, supersedes older + unleased working work, and enqueues the new immutable working row. +- Completion atomically fixes the current content revision, prevents later + working enqueue for that turn/revision, supersedes outstanding working work, + and enqueues final-ready after it in the same partition. +- Prepare commit atomically creates the active child/retire DAG and retains the + source root in `awaiting_ack`. +- Pending-decision projection and decision enqueue are atomic. Resolution and + retire enqueue are atomic. +- A newer final supersedes an uncommitted old root immediately. For a committed + old root, its unaccepted tail is terminal-after-lease/superseded, but its + accepted prefix remains retained until the new replacement/retire DAG is + acknowledged. No delete-before-replace gap is possible. +- A root becomes `delivered` only when every node in its effective original or + recovered lineage is delivered. It becomes `superseded` or `dead_letter` + atomically with the relevant remaining lineage. + +Lifecycle snapshots reconcile workers/routes/topics but never create or replay +presentation content. Herdres has no `turn.delta`, `pending.list`, transcript, +snapshot-content, or other parallel authority for final delivery. + +## Production import and caller rewrite + +Every production `store.sqlite` import is rewritten directly; there is no +`store/__init__.py` façade. + +| Old symbol | New module or disposition | Current production callers after rewrite | +| --- | --- | --- | +| `init_store` | `store.schema` | daemon `_default_init_store`, `DaemonHooks`; smoke fixture setup | +| `record_agent_event` | `store.events` | ACP coordinator console bridge/cursor/outcome | +| `list_agent_events` | `store.events` | ACP coordinator console bridge/cursor/input loads | +| `tail_event_metadata` | **deleted** | only live production caller is deleted direct CLI `cmd_store`; no after-state caller | +| `AppendBoundAgentEventResult` | `core.agent_events` | ACP ingestion event result; never moved into a store module | +| `AppendProjectedAgentEventResult` | `store.turns` | ACP ingestion type/API | +| `TurnRefreshApplyResult` | `store.turns` | ACP ingestion result | +| `append_agent_event_and_apply_turn_for_binding` | `store.turns` | ACP ingestor default persistence path | +| `apply_turn_refresh` | `store.turns` | `scripts/sqlite_sidecar_race_benchmark.py`, `tests/store_helpers.py`, and focused turn/delta/submission/performance tests; no current `src` caller imports it directly, while ACP runtime uses the combined append/apply API | +| `turns_payload_from_store` | `store.turns` | daemon `get_turns` | +| `turn_delta_payload_from_store` | `store.turns` | daemon `get_turn_delta` | +| `get_turn_content` | `store.turns` | daemon `get_turn_content` | +| `SnapshotObservationContext` | `store.projection` | ACP discovery; daemon/tests | +| `save_snapshot` | `store.projection` | ACP coordinator discovery; returns the persisted route-enriched snapshot with recomputed worker/content fingerprints | +| `latest_snapshot` | `store.projection` | command submission; ACP coordinator; daemon start/snapshot; smoke | +| `attention_payload_from_store` | `store.projection` | daemon `get_attention` | +| `upsert_worker_bindings` | `store.projection` | ACP coordinator runtime/binding creation | +| `list_worker_bindings` | `store.projection` | ACP coordinator/runtime/permissions; smoke | +| `expire_worker_bindings` | `store.projection` | ACP coordinator/runtime | +| `expire_stale_worker_bindings` | `store.projection` | tests and projection API; no compatibility import | +| `backend_pending_health` | `store.projection` | daemon pending-ingestion health | +| `apply_backend_pending_observation` | `store.pending` | ACP permission broker | +| `pending_payload_from_store` | `store.pending` | ACP coordinator and daemon `get_pending` | +| `claim_backend_pending_decision` | `store.pending` | command submission decision validation/claim | +| `start_backend_pending_decision_send` | `store.pending` | command submission answer send | +| `abandon_backend_pending_choice_claim` | `store.pending` | command submission safe pre-send abandon | +| `backend_pending_choice_terminal_effect` | `store.pending` | command submission uncertain/terminal decision effect | +| `reserve_command_request` | `store.receipts` | command submission reservation | +| `reserve_terminal_command_replay` | `store.receipts` | command submission replay | +| `get_command_request` | `store.receipts` | command submission recovery/replay paths | +| `command_reservation_is_live` | `store.receipts` | command receipt authority | +| `abandon_command_request_reservation` | `store.receipts` | safe pre-transport abandon/retry | +| `mark_command_send_started` | `store.receipts` | command submission send fence | +| `finish_command_request` | `store.receipts` | command completion | +| `finish_queued_command_request` | `store.receipts` | queued receipt completion | +| `finish_unverified_queued_command_request` | `store.receipts` | uncertain queued completion | +| `recover_unresolved_command_send` | `store.receipts` | receipt authority recovery | +| `linked_turn_for_submission` | `store.receipts` | send/link/replay correlation | +| `settle_submission_link_for_request` | `store.receipts` | receipt and negotiated submission linkage | +| `envelope_to_receipt_json` | `store.receipts` | canonical receipt persistence | +| `poll_connector_outbox` | `store.outbox` | `ConnectorOutboxAPI.poll` | +| `prepare_connector_plan_begin` | `store.outbox` | `ConnectorOutboxAPI.prepare(begin)` | +| `prepare_connector_plan_part` | `store.outbox` | `ConnectorOutboxAPI.prepare(part)` | +| `prepare_connector_plan_commit` | `store.outbox` | `ConnectorOutboxAPI.prepare(commit)` | +| `prepare_connector_plan_recover` | `store.outbox` | `ConnectorOutboxAPI.prepare(recover)` | +| `ack_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.ack` | +| `fail_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.fail` | +| `defer_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.defer` | +| `renew_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.renew` | +| `release_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.release` | +| `reclaim_expired_connector_leases` | `store.outbox` | connector API reclaim; daemon periodic tick | +| `connector_reclaim_due` | `store.outbox` | daemon periodic connector tick | +| `inspect_connector_outbox` | `store.outbox` | `ConnectorOutboxAPI.inspect` | +| `retry_final_ready_delivery` | `store.outbox.retry_connector_dead_letter` | `ConnectorOutboxAPI.retry`; the old final-only name is deleted and the replacement handles exact retryable final roots, standalone decisions, and standalone retires | +| `store_status` | `store.db` | daemon health only | +| `SnapshotRetentionPolicy` | `store.retention.RetentionPolicy` | rewritten daemon retention cycle | +| `cleanup_*_retention` | private `store.retention` helpers | `run_retention_cycle` only | +| `CompactionOptions`, `compact_store`, `run_store_maintenance`, `maybe_run_automatic_store_maintenance`, `compact_turn_change_journal`, `exhaust_connector_retries` | **deleted as public APIs** | direct CLI/daemon maintenance paths removed; outbox exhaustion remains a private transition helper | + +The CLI rewrite removes the `store` parser subtree, `cmd_store`, direct status, +event-tail, maintenance, and compact options, and all five direct SQLite +imports. Operator reads use the daemon socket. There is no compaction RPC. + +Daemon `_after_snapshot_saved` removes maintenance-state and turn-journal +compaction calls and invokes one bounded `retention.run_retention_cycle` on an +in-memory cadence. Connector periodic reclaim remains independent. Health uses +`db.store_status`; it does not mutate or repair the store. + +`scripts/store_benchmark.py` is rewritten to import the owning schema, +projection, outbox, and retention APIs and to seed only the fresh schema; its +old direct assumptions about maintenance tables and `store.sqlite` are deleted. +`scripts/sqlite_sidecar_race_benchmark.py` is rewritten to import +`store.schema`/`store.db`, patch the new pinned-path connection authority rather +than a deleted module, and retain installed-wheel sidecar-race, inode, mode, +integrity, and descriptor evidence. Neither script is allowed a compatibility +import or private direct-SQL mutation that bypasses the invariant it claims to +benchmark. + +## Test-file disposition + +| File | Disposition | +| --- | --- | +| `tests/store_helpers.py` | rewrite direct imports to owning modules | +| `tests/test_acp_atomic_ingestion.py` | retain atomic journal/projection proof; rewrite imports | +| `tests/test_acp_coordinator.py` | rewrite imports; add route-generation publication/rotation/collision cases | +| `tests/test_acp_ingestion.py` | rewrite result-type and event/turn imports | +| `tests/test_acp_permissions.py` | replace module-wide sqlite monkeypatches with exact pending/projection patches | +| `tests/test_acp_runtime.py` | rewrite projection/event imports | +| `tests/test_agent_events.py` | rewrite to `store.events`; preserve authoritative dedupe/conflict, visibility, paging, and retention proofs | +| `tests/test_daemon.py` | rewrite imports/patch paths; delete automatic-maintenance-state assertions; add bounded retention and daemon-only health | +| `tests/test_daemon_acp.py` | rewrite schema/projection imports | +| `tests/test_connector_daemon_cli.py` | retain real socket RPC contract; delete direct CLI store fallback cases | +| `tests/test_connector_outbox.py` | replace with the five-kind status/CAS matrix, FIFO/DAG, lease, response-loss, recovery, and alias-retire tests | +| `tests/test_delivery_retention.py` | rewrite for kind-aware live/root/effective-lineage protection | +| `tests/test_delivery_retention_hardening.py` | delete maintenance-state tests; retain deadline/dead-letter/cutoff/security tests | +| `tests/test_delivery_retention_projection.py` | retain atomic projection/outbox proof; add route correlation | +| `tests/test_delivery_retention_recovery.py` | rewrite for retained old prefix plus fresh suffix lineage | +| `tests/test_local_state_permissions.py` | rewrite `init_store` import; retain secure-open proof | +| `tests/test_public_content_safety.py` | rewrite imports; add exact five-kind forbidden-key/value scans | +| `tests/test_release_readiness.py` | delete VACUUM/backup/compaction ceremony; retain fresh cutover, integrity, permissions, and retention proof | +| `tests/test_snapshot_sanitize_performance.py` | rewrite projection imports and preserve sanitizer/transaction performance bounds without a sqlite compatibility path | +| `tests/test_store.py` | split by owning module; delete migration-chain/generic CRUD/direct-maintenance internals | +| `tests/test_turn_delta.py` | rewrite turn imports; add route-generation publication/correlation | +| `tests/test_turn_submissions.py` | rewrite schema/turn imports; preserve state machine | +| `tests/test_worker_stable_key.py` | add stable/reused/rotated/collision route-generation tests | + +The paired Herdres H7/H6 table is authoritative here and in +`docs/wave4-presenter-state-design.md`; every baseline file has one matching +disposition: + +| Herdres baseline test file | Final disposition | +| --- | --- | +| `conftest.py` | retain; edit only bounded shared fixtures, never protocol or production logic | +| `test_accounts.py` | delete with pinned boards | +| `test_collapse_previous.py` | rewrite for immutable target keys, flattened one-hop aliases, and reuse-safe retire | +| `test_command_ingress_idempotency.py` | rewrite/retain H8 HMAC request identity, key security, and replay vectors | +| `test_gateway_cleanup.py` | delete old gateway cleanup; move known-ID topic cases to `test_topics.py` | +| `test_ingress_lanes.py` | delete with lanes; surviving FIFO/crash cases move to `test_ingress.py` | +| `test_ingress_requests.py` | delete with JSON request workers; surviving receipt/quarantine cases move to `test_ingress.py` | +| `test_lossless_turn_rendering.py` | rewrite for contract-v3 deterministic exact spans and no stored content | +| `test_model_in_pins.py` | delete with pinned boards | +| `test_offlock_delivery.py` | rewrite for shared guard, lock order, route revalidation, and uncertainty classes | +| `test_outbound_latency.py` | rewrite for one poll/presenter and bounded lease/guard budgets | +| `test_pane_topic_binding_integrity.py` | rewrite for typed lifecycle, current slots, aliases, and immutable provider facts | +| `test_pending_inputs.py` | delete pending-list/bare-number local presentation path | +| `test_release_readiness.py` | rewrite for exact scope, static gates, 8,920 SLOC target, security, and paired cutover | +| `test_remote_decisions.py` | rewrite for decision controls, composite phases, shared guard, and resolution retire | +| `test_restart_rekey_continuity.py` | delete old rekey machinery; route continuity moves to state/presenter tests | +| `test_rich_delivery.py` | rewrite for deterministic one-request materialization and no fallback mutation | +| `test_source_only.py` | rewrite surviving connector receipt/crash cases; delete local-source/source-less cases | +| `test_source_status_placeholders.py` | delete local source/status placeholder presentation | +| `test_speak_back.py` | delete with voice/TTS | +| `test_speech.py` | delete with voice/STT/TTS | +| `test_stable_generation_delivery.py` | rewrite for exact route token, enriched fingerprint, and stale-route fence | +| `test_stable_worker_key.py` | retain and extend stable identity, route reuse/rotation, and collision cases | +| `test_table_rendering.py` | retain supported renderer; remove pin/voice coupling if present | +| `test_telegram_backpressure.py` | rewrite for one guarded mutation, exact 429 defer, and no tight loop | +| `test_tendwire_client.py` | rewrite for five kinds, exact prepare/content responses, renew/release, and socket validation | +| `test_tendwire_socket_pairing.py` | rewrite for real paired five-kind/ingress/receipt/restart integration | +| `test_topic_lifecycle_cleanup.py` | rewrite as bounded known-ID cases in `test_topics.py` | +| `test_topic_names.py` | retain/rewrite for minimal supported topic naming only | +| `test_turn_delta_sync.py` | delete; final presentation has no turn-delta source | +| `test_turn_final_delivery.py` | replace with source-bound root, provider kinds, recovery, retry, and retire integration | +| `test_worker_topic_dedup.py` | rewrite for physical-owner serialization across route generations | + +The final stack adds exactly `test_ingress.py`, `test_state.py`, +`test_presenter.py`, `test_presentation.py`, and `test_topics.py`. No unlisted +Herdres test file may be added, deleted, or used as a compatibility dump. +Rewritten files may share fixtures through `tests/conftest.py`; production logic +and copied protocol validators are forbidden there. + +Tests assert public behavior and database constraints, not private helper names or +copied transition logic. Required real-daemon integration covers working to +final ordering, paged multipart prepare, lost begin/part/commit responses, +provider acceptance plus lost ACK, source lease expiry, recovered prefix/suffix, +supersession and message reuse, 429 defer, decision buttons/resolution, privacy +rejection, stale refs, dead-letter/retry generations, and concurrent logical +partitions sharing one physical provider route. + +A static AST/import gate scans `src`, `scripts`, and tests and fails on any +production import, attribute patch, string import, or compatibility re-export +of `tendwire.store.sqlite`/`store.sqlite`. It also proves each deleted public +maintenance symbol has no production caller. + +## Precise implementation file scope + +Tendwire T6 deletes/adds/edits only: + +```text +delete src/tendwire/store/sqlite.py +add src/tendwire/store/schema.py +add src/tendwire/store/db.py +add src/tendwire/store/events.py +add src/tendwire/store/turns.py +add src/tendwire/store/projection.py +add src/tendwire/store/pending.py +add src/tendwire/store/receipts.py +add src/tendwire/store/outbox.py +add src/tendwire/store/retention.py +edit src/tendwire/store/__init__.py # package marker only; no re-exports +edit src/tendwire/cli.py +edit src/tendwire/config.py # bounded ACK TTL and retention floors +edit src/tendwire/daemon.py +edit src/tendwire/daemon_api.py +edit src/tendwire/command_submission.py +edit src/tendwire/worker_identity.py +edit src/tendwire/core/models.py +edit src/tendwire/backends/acp_coordinator.py +edit src/tendwire/backends/acp_ingestion.py +edit src/tendwire/backends/acp_permissions.py +edit src/tendwire/backends/acp_runtime.py +edit src/tendwire/connectors/outbox.py +edit scripts/herdr_smoke.py +edit scripts/store_benchmark.py +edit scripts/sqlite_sidecar_race_benchmark.py +edit docs/connector-rpc-contract.md +edit docs/wave4-store-design.md +edit only the Tendwire tests listed in the disposition table +``` + +The reconciled Herdres scope is split at one safe merge boundary. + +H8 may be implemented, reviewed, and deployed independently against the current +Tendwire/current presenter pair. Its exact file disposition is: + +```text +add herdres_connector/ingress.py +add herdres_connector/ingress_queue.py +edit herdres_gateway.py # retained small executable wrapper +edit herdres.py # remove command-child ingress only +edit herdres_connector/state.py # frozen typed H8 seam +edit herdres_connector/decisions.py +edit herdres_connector/doctor.py +edit herdres_connector/config.py +edit herdres_connector/source_sync.py # remove JSON-ingress/receipt-working joins +edit herdres_connector/ingress_identity.py # trim retained key/identity code to 80--100 SLOC +edit herdres_connector/managed_bots.py # ephemeral receiver/policy typing only +edit herdres_connector/tendwire_client.py +keep herdres_connector/telegram_delivery.py # H8 uses existing bounded methods +delete herdres_connector/ingress_lanes.py +delete herdres_connector/ingress_requests.py +edit/delete only the H8 tests and docs declared by the approved H8 design +``` + +H8's frozen route result uses `route_generation: str | None`, and its ingress +queue column is nullable `TEXT`, never integer. Current routes publish null, +and stored null is immutable for that request. Contract-v3 H7 routes require a +non-null exact `twroute1.*`; open pre-T6 null-route requests are drained or +explicitly discarded at paired cutover. H8 does not depend on the five-kind +outbox. The old presenter continues the current observational working/final +behavior until the paired cutover. + +After H8 is deployed, corrected T6 and the H7/H6 presenter stack form one +inseparable compatibility and deployment unit. H7/H6's exact final disposition +is: + +```text +add herdres_connector/presenter.py +add herdres_connector/presentation.py +add herdres_connector/topics.py +edit herdres.py +retain herdres_gateway.py # small H8 compatibility wrapper +edit herdres_connector/__init__.py +edit herdres_connector/config.py +edit herdres_connector/decisions.py +edit herdres_connector/doctor.py +edit herdres_connector/ingress.py +edit herdres_connector/ingress_queue.py +keep herdres_connector/ingress_identity.py +edit herdres_connector/managed_bots.py +edit herdres_connector/rendering.py +edit herdres_connector/rich_delivery.py +edit herdres_connector/safe.py +edit herdres_connector/state.py +edit herdres_connector/telegram_delivery.py +edit herdres_connector/tendwire_client.py +delete herdres_connector/source_sync.py +delete herdres_connector/accounts.py # pinned-board deletion +delete herdres_connector/speech.py # voice/STT/TTS deletion +edit README.md +edit RELEASE.md +edit SECURITY.md +edit docs/connector-rpc-contract.md +edit docs/remote-decisions.md +edit docs/wave4-ingress-dependency-design.md +edit docs/wave4-presenter-state-design.md +edit docs/wave4-store-design.md +edit/delete/add only the paired Herdres tests in the exact table above +``` + +H7 state keeps immutable job bindings keyed only by Tendwire outer key and +provider/presentation ownership aliases separately; it contains no local plan, +accepted-ordinal, pending-retire, retry, or recovery ledger. Current Herdres +strictly accepts final-ready schema v2 and plan-job schema v1, so T6 cannot ship +alone. H7/H6 likewise cannot ship against current Tendwire. Only H8 has an +independent deployment boundary; the next deployable boundary is the complete +T6+H7/H6 paired contract-v3 unit. + +## Transactions, security, privacy, and retention + +Every mutation uses `BEGIN IMMEDIATE`. Frozen list/content/delta pages use a +read transaction. Event append, binding/route fence, turn and content revision, +delta sequence, command/submission effect, pending overlay, and outbox enqueue +are atomic where the producer invariant requires them. + +Connections are short-lived and use WAL, foreign keys, `FULL` synchronous, +bounded busy timeout, and `trusted_schema=OFF`. The configured database must be +an absolute leaf beneath the exact configured owner-private data directory; it +cannot be `:memory:`, a URI, a relative path, contain `..`, or select another +authority root. Code walks and pins every parent with directory fds and +`O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC`, rejects symlinks, wrong UID, group/world +writable directories, and parent identity changes, and opens/creates the leaf +relative to the pinned parent with restrictive umask and +`O_NOFOLLOW|O_CLOEXEC`, mode `0600`, regular-file, single-link, UID, and +device/inode checks. + +SQLite connects through the pinned `/proc/self/fd//` path while +the schema/sidecar authority is held. The implementation verifies the database +identity before and after connect and transaction use; validates or securely +creates `-wal` and `-shm` only as same-parent, same-UID, regular, single-link +0600 files; rejects unexpected journal/sidecar types or inode swaps; and pins +and revalidates the family across schema mutation, checkpoint, and close. A +separately pinned owner-only lock serializes schema/sidecar creation without +becoming a general authority registry. No fallback connect occurs after any +identity or permission failure. + +The connector-edge scanner rejects raw pane/session/terminal identities, +backend targets, private fingerprints, provider chat/topic/message IDs, +credentials/tokens, socket and absolute paths, command/argv/environment data, +ACP option IDs, stdout/stderr, and exception prose, including forbidden values +embedded in otherwise allowed strings. Only enumerated public errors and reason +codes cross the socket. Logs and cutover reports contain aggregate counts and +schema versions only, never IDs, refs, keys, payloads, or provider coordinates. + +Retention performs bounded child-first cutoff deletes. It never deletes a live +lease, staged plan, retained awaiting-ACK root, current/recovered lineage, +dead-letter inside its inspection cutoff, command replay floor, current content +revision, page referenced by work, active route generation, or maximum sequence +sentinel/partition allocator. A delivery with non-null `ack_deadline_at` is live +regardless of wall-clock age; after terminal settlement the deadline is cleared +and deletion still waits for the 30-day targetable floor. Targetable outbox +correlations use 30 days, route/content/key evidence uses 45 days, and the +paired H7 provider-fact floor is 60 days, with references overriding all three. +It retains only bounded terminal attempt detail plus aggregate prior attempt +counts, then runs `PRAGMA wal_checkpoint(TRUNCATE)`. It performs no +VACUUM, backup swap, direct CLI repair, or automatic schema migration. + +## Cutover and rollback + +H8 has the only independent deployment boundary. Its release stops the old +gateway writer, drains or explicitly discards both old ingress durability +stores, archives the old `inbound_spool.db` family and matching schema-2 +Herdres state, and preserves the one active H8 request-ID HMAC key byte-for-byte +at its configured owner-only path. It starts exactly one H8 writer on the fresh queue +with that key and proves one daemon-socket receipt. Snapshot rollback is +straightforward only before H8 accepts new work. After acceptance it requires the H8 design's +explicit cursor/request inventory and operator drain/discard decision; the old +snapshot is not described as lossless merely because it can be restored. + +T6+H7/H6 is the next single deployment boundary. Tendwire schema mismatch and +H7 state mismatch never migrate silently. Before that paired upgrade operators +must choose: + +1. drain old command reservations, claims, interactions, leases, + awaiting-ACK roots, presentation plans, and outbox work to zero; or +2. stop every old writer and explicitly acknowledge discarding all in-flight + state before fresh recreation. + +The quiescent cutover snapshot is one matched set: Tendwire database/WAL/SHM, +the active Tendwire installation key/marker/sentinel, H8 ingress +database/WAL/SHM, the same active H8 request-ID HMAC key used +before cutover, and old Herdres state/provider receipts. The paired release +must not rotate, regenerate, or silently replace the Tendwire key family or the +H8 request-ID key. Old/new +versions, discarded table names, and aggregate active counts are reported +without identities or payloads. Old and new presenters never run against the +same topics. + +Fresh H7 state starts behind a reconciliation barrier. While H8 submission and +all Telegram mutators remain stopped, one lifecycle writer reconciles every +current worker/private route, Tendwire durably publishes each non-null +`twroute1.*`, `save_snapshot` returns the enriched fingerprints, and H7 resolves +the same stable owner/generation pairs through the frozen seam. The barrier is +released only after one transactionally consistent inventory proves no live +route is missing/ambiguous and every retained H8 request either has the exact +non-null generation/fingerprint or was drained or explicitly discarded. Only +then may H8 and the contract-v3 presenter resume. Failure leaves the barrier +closed and authorizes no command or provider mutation. + +Rollback is mechanically safe before recreation and before any new provider +mutation. After the paired release has sent, edited, deleted, or created a +provider object, restoring old local snapshots alone is unsafe: it forgets +new provider facts and can replay or delete the wrong message. Rollback then +requires stopping all new writers, draining/ACKing or explicitly retiring every +new known provider mutation with the new release, resolving/reporting ambiguous +claims, and only then restoring the complete matched old snapshot. If that +drain/retire proof is unavailable, rollback is blocked rather than described as +safe. Installation-key or request-key replacement is a continuity break, not +recovery. + +No implementation, merge, deployment, or service restart is authorized by +this design. Production implementation remains blocked until the paired +Herdres payload, outer-key binding, no-plan-ledger recovery, physical-route +locking, and alias-retire tests pass against this exact contract.