diff --git a/.env.example b/.env.example index a643906..364664d 100644 --- a/.env.example +++ b/.env.example @@ -80,18 +80,24 @@ TENDWIRE_STORE_MAINTENANCE_CADENCE_SECONDS=3600 # Optional stable host label for the local machine. # TENDWIRE_HOST_ID=my-host -# Optional Herdr binary and timeout overrides. Turn-adapter reads use the same -# timeout. +# Optional Herdr binary and timeout overrides. Herdr supplies worker/pane +# lifecycle and owns ACP endpoints; it is not an agent transcript transport. +# Initial socket reconciliation has a separate whole-startup budget so a +# multi-call inventory can exceed one RPC without weakening normal deadlines. # TENDWIRE_HERDR_BIN=herdr TENDWIRE_HERDR_TIMEOUT_SECONDS=1.0 +TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS=120.0 -# Daemon-owned turn-ingestion cadence and dedicated worker pool. Defaults are -# 2.0 seconds and 4 workers; the worker count cannot exceed -# TENDWIRE_MAX_WORKERS. The internal refresh queue is fixed at 64. -TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS=2.0 -TENDWIRE_TURN_REFRESH_WORKERS=4 -# Compatibility flag: legacy|dual|shadow|observed all use the observed model. -TENDWIRE_TURN_MODEL=observed +# ACP is required for every supported agent. The daemon fails closed when a +# worker lacks a healthy Herdr-owned ACP endpoint; there is no transcript, +# shadow, preferred, or fallback mode. Stable ACP v1 does not distinguish raw +# reasoning from summaries. The safe +# default discards thought chunks. `private_summary` is an explicit trusted- +# adapter convention; `private_all` retains all thoughts for local diagnostics. +TENDWIRE_ACP_THOUGHT_POLICY=disabled +TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS=30 +TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS=5 +TENDWIRE_ACP_MAX_FRAME_BYTES=8388608 # Process-local bounded LRU for repeated public-text sanitization. Set to 0 to # disable; values above 65536 are capped. The default covers typical retained diff --git a/README.md b/README.md index da92142..fd826a2 100644 --- a/README.md +++ b/README.md @@ -29,11 +29,12 @@ background daemon; use [INSTALL.md](INSTALL.md) for persistent service setup. ## Relationship to Herdr, Herdres, and connectors -Herdr is the only concrete runtime backend documented here. Tendwire can observe -Herdr through the conservative CLI one-shot path or, when explicitly enabled, -through the Herdr socket/event backend. Both paths normalize Herdr state into -neutral Tendwire spaces, workers, attention, turns, pending interactions, -command results, connector jobs, and backend health. +Herdr supplies workspace, pane, and worker lifecycle plus ownership of private +ACP endpoints. Tendwire can observe that lifecycle through the conservative CLI +one-shot path or the Herdr socket/event backend. ACP is required for supported +agent semantics and commands; Herdr is not used to read transcripts or inject +pane input. The authority and privacy split is defined in +[docs/acp-migration.md](docs/acp-migration.md). Herdres can use Tendwire as its source/control plane while Herdres remains the Telegram connector. Tendwire owns Herdr observation, private bindings, @@ -175,6 +176,11 @@ not expose raw backend argv. The Herdr binary path, data directory, and database path expand `~`; each Herdr probe uses `TENDWIRE_HERDR_TIMEOUT_SECONDS` or `--herdr-timeout` when set, defaulting to 5.0 seconds. +Socket-daemon startup waits up to +`TENDWIRE_HERDR_INITIAL_RECONCILE_TIMEOUT_SECONDS` (default `120.0`) for the +entire initial reconciliation. This startup-only budget does not change the +normal per-RPC `TENDWIRE_HERDR_TIMEOUT_SECONDS` deadline. + When Herdr 0.7.0 is present, the adapter first tries the no-flag JSON envelopes (`herdr workspace list`, `herdr agent list`) that wrap records under `result.workspaces` and `result.agents`, then keeps `--json` list variants as a @@ -545,9 +551,10 @@ variables: | `snapshot_retention_count` | `TENDWIRE_SNAPSHOT_RETENTION_COUNT` | `4096` | positive integer; includes each host's latest row | | `snapshot_maintenance_batch_size` | `TENDWIRE_SNAPSHOT_MAINTENANCE_BATCH_SIZE` | `100` | integer from 1 through 1000 | | `store_maintenance_cadence_seconds` | `TENDWIRE_STORE_MAINTENANCE_CADENCE_SECONDS` | `3600` | positive integer | -| `turn_refresh_interval_seconds` | `TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS` | `2.0` | finite positive float | -| `turn_refresh_workers` | `TENDWIRE_TURN_REFRESH_WORKERS` | `4` | integer from 1 through 32 and no greater than `max_workers` | -| `turn_model` | `TENDWIRE_TURN_MODEL` | `observed` | `observed`; `legacy`, `dual`, and `shadow` are deprecated aliases with identical observed behavior | +| `acp_thought_policy` | `TENDWIRE_ACP_THOUGHT_POLICY` | `disabled` | `disabled`, `private_summary`, or `private_all`; never a public-delivery grant | +| `acp_request_timeout_seconds` | `TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS` | `30.0` | finite positive float | +| `acp_shutdown_timeout_seconds` | `TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS` | `5.0` | finite positive float | +| `acp_max_frame_bytes` | `TENDWIRE_ACP_MAX_FRAME_BYTES` | `8388608` | integer from 1 through 67108864 | The socket/event backend uses `event_debounce_seconds` for event batching and `reconcile_interval_seconds` for bounded periodic full reconciles. Set @@ -560,6 +567,34 @@ snapshot/projections instead of publishing a truncated authoritative snapshot. Incremental events that would add workers over the cap are ignored with the same public-safe degraded evidence. +The stock daemon requires ACP for supported agents. It uses Herdr's private +`agent.acp_endpoint` contract and accepts only workers explicitly marked +`acp_owned_ready`; ordinary PTY workers are never attached as sidecars. A +missing or unhealthy ACP endpoint degrades daemon health and fails command +submission closed. There is no transcript-reader scheduler, shadow/preferred +mode, or PTY fallback. +Production ACP uses a bounded per-worker permission broker. It publishes only +the sanitized tool title and numeric choices (option label and kind) through +the durable pending-decision surface; ACP option IDs, arguments, session IDs, +and adapter metadata remain private. `answer_decision` is fenced to the exact +worker binding, ACP session, and Herdr generation. A command is accepted only +after the complete JSON-RPC permission-response frame is written. Missing or +retired ACP authority fails closed without falling back to PTY input, and +concurrent answers can produce at most one response. None of this makes agent +thoughts public: thought events remain private diagnostic data unless a +separate, explicit sanitized projection is introduced. + +Store maintenance retires expired structured agent-event payloads in bounded +batches using `event_retention_days`. Compact identity tombstones remain so a +replayed source event cannot be reinserted or silently change content after its +private payload has expired. Due automatic daemon maintenance performs the same +host-scoped retirement as explicit `store cleanup`, using a metadata-only scan +that does not load private payloads. Tombstones intentionally retain bounded +per-event hashes and opaque identity only; they are not a recoverable copy of the +removed payload. Their count is permanent and grows with distinct source-event +identities. SQLite secure deletion is best-effort page hygiene, not a promise of +immediate physical erasure across WAL/checkpoints, snapshots, or backups. + Snapshot history defaults are sized for a five-minute observation rhythm: $14 \times 24 \times 12 = 4032$ observations, while the 4096-row count ceiling (including the latest row) leaves 64 rows of headroom. This is a @@ -1466,10 +1501,6 @@ metadata and does not change observation-derived turn identity, turn-list order, Goal 10 delivery, or Herdres consumption. Lazy and periodic settlement cover both submission-first and observation-first arrival order. -`TENDWIRE_TURN_MODEL` remains accepted for rollout compatibility. `legacy`, -`dual`, and `shadow` emit a warning and use the same observation-authoritative -behavior as `observed`. - `disposition`, not `status` alone, is the receipt-authority and finality contract: diff --git a/docs/acp-adapter-probe.md b/docs/acp-adapter-probe.md new file mode 100644 index 0000000..a717dce --- /dev/null +++ b/docs/acp-adapter-probe.md @@ -0,0 +1,67 @@ +# Black-box ACP adapter initialization probe + +Tendwire treats an ACP adapter as a separately installed executable. The +adapter is not vendored, imported, rebased, or coupled to a repository layout. +Run the initialization probe after installing or upgrading any adapter and +before promoting that executable into the Tendwire runtime: + +```console +python -m tendwire.backends.acp_probe -- /absolute/path/to/adapter adapter-arg +``` + +The command starts the supplied argv directly without a shell, negotiates ACP +v1 capabilities on a fresh process, closes the process, prints one bounded JSON +object, and exits zero only when negotiation and shutdown succeed. Use +`--timeout`, `--close-timeout`, and an absolute `--cwd` before the `--` marker +when needed. Probe timeouts have hard upper bounds. + +The probe scope is deliberately limited to `initialize`. It does not create a +session, send a prompt, spend model tokens, or claim that the adapter actually +implements ACP's mandatory baseline session methods. The output contains fixed +booleans for optional capabilities advertised by the agent, bounded counts for +schema-valid authentication methods and `_meta` capability-extension +namespaces, a process-reaped flag, and a fixed failure category. It never +contains: + +- adapter argv or executable paths; +- working directories or environment values; +- stderr, exception text, or raw JSON-RPC payloads; +- agent-provided names, versions, extension names, or extension values; +- session IDs, messages, thoughts, tool data, plans, or authentication values. + +Example successful shape: + +```json +{ + "authentication": {"method_count": 0, "method_count_capped": false}, + "advertised_capabilities": { + "additional_directories": true, + "auth_logout": false, + "mcp_http": false, + "mcp_sse": false, + "prompt_audio": false, + "prompt_embedded_context": false, + "prompt_image": false, + "session_close": true, + "session_delete": true, + "session_list": true, + "session_load": true, + "session_resume": true + }, + "initialization_compatible": true, + "extensions": {"capability_count": 0, "capability_count_capped": false}, + "failure": null, + "probe_scope": "initialize", + "process_reaped": true, + "protocol_version": 1, + "schema_version": 2 +} +``` + +An initialization-incompatible result exits with status 1 and reports only one +of these stable categories: `invalid_configuration`, `launch_failed`, `timeout`, +`protocol_version`, `protocol_error`, `transport_error`, `shutdown_failed`, or +`internal_error`. Keep the previously proven executable available for rollback; +the probe validates initialization and advertised capability parsing, not +account authentication, mandatory baseline method behavior, or a stateful agent +session. diff --git a/docs/acp-migration.md b/docs/acp-migration.md new file mode 100644 index 0000000..dde8263 --- /dev/null +++ b/docs/acp-migration.md @@ -0,0 +1,106 @@ +# ACP-required architecture + +ACP is Tendwire's required semantic and command protocol for every supported +agent. This is a release boundary, not a runtime rollout switch: rollback means +deploying an earlier release. There are no legacy, shadow, preferred, dual, or +fallback modes in the daemon. + +## Authority split + +| Concern | Authority | +| --- | --- | +| Workspace, pane, worker lifecycle, and ACP endpoint ownership | Herdr | +| Public stable worker identity | Tendwire's authenticated Herdr projection | +| Agent session, messages, tools, plans, usage, and permissions | ACP | +| Durable journal, public turn projection, privacy, and finality | Tendwire | +| Command idempotency and uncertain outcomes | Tendwire receipts | +| Connector retry and delivery state | Tendwire outbox | +| Telegram presentation | Herdres | + +Herdr is not a transcript or command-input transport. Tendwire consumes its +worker/pane lifecycle events, verifies the worker generation, and asks it to +mint a one-shot private ACP endpoint. A supported worker without an explicitly +Herdr-owned healthy ACP endpoint is unavailable and degrades ACP health. + +An ACP `sessionId`, endpoint ticket, adapter command, terminal ID, and pane ID +are private. Tendwire binds the session to the current private worker binding +and rejects updates or commands after that generation moves, expires, or is +replaced. + +## Runtime and protocol boundary + +The daemon supervises one ACP worker session for each eligible Herdr worker. +The supervisor owns reconciliation, generation fencing, reconnect, console +exchange, and command routing. The worker session owns initialize, session +open/load/resume, update draining, permission handling, cancellation, and +bounded shutdown. The bounded connection owns subprocess I/O, JSON-RPC request +correlation, framing limits, stderr limits, and backpressure. + +Tendwire uses the official `agent-client-protocol` Python package for generated +ACP schemas and validation. Tendwire retains its bounded stdio connection +because its hard frame-size, queue, write-deadline, shutdown, and privacy +requirements are stricter than the upstream convenience transport. Adapter +executables remain separately installed and replaceable; no adapter source tree +is imported or vendored. + +Before every prompt and during reconciliation, Tendwire verifies Herdr's +non-mutating ACP status for the exact worker generation. Reconnect always asks +Herdr for a fresh one-shot endpoint. A missing route fails before receipt +reservation; a failure after the durable `send_started` boundary is terminally +uncertain and is never retried through another transport. + +## Durable projection and privacy + +The structured journal accepts user and agent messages, thoughts, tool calls, +tool updates, plans, usage, session information, and private extension/control +updates. Producer identity and raw ACP payloads are retained only on the +private side. The public turn projection remains deliberately narrower and +continues to drive finality and connector eligibility. + +`TENDWIRE_ACP_THOUGHT_POLICY` controls private thought retention: + +- `disabled` discards thoughts before persistence. +- `private_summary` retains only updates carrying Tendwire's exact trusted + summary marker. +- `private_all` retains raw thought chunks for explicit local diagnostics. + +No thought policy grants public or outbox delivery. Raw reasoning, tool input, +tool output, session IDs, paths, terminal data, and adapter metadata must remain +outside every public API and connector payload. + +ACP permission requests use the durable pending-decision projection. Only a +sanitized title and numbered public choices are exposed. The private option ID, +tool call, arguments, ACP session, and metadata remain behind the boundary. +`answer_decision` is fenced to the exact worker binding and generation and is +accepted only after the full JSON-RPC response frame is written. + +## Retention and replay + +`event_retention_days` bounds private structured-event payloads. Cleanup +replaces expired payloads with compact identity tombstones so exact replays +remain idempotent and conflicting producer-identity reuse fails closed. +Tombstones preserve only bounded replay evidence and authority time; they are +not recoverable event content. + +Tendwire continues to own durable command receipts and its neutral connector +outbox. ACP adapter restarts therefore do not erase accepted-command evidence, +turn finality, acknowledgement state, or delivery retries. + +## Release and conformance gates + +An ACP-required release must pass: + +- initialization and official-schema validation against each supported + adapter; +- new/load/resume, prompt, steering, cancellation, and permission flows; +- generation fencing and reconnect with freshly minted Herdr endpoints; +- no missing or duplicated user/final messages across adapter restarts; +- deterministic replay deduplication and exactly-once final projection; +- complete tool and plan lifecycle after cancellation or permission denial; +- absence of raw thoughts, tool payloads, session IDs, and terminal data from + public APIs and the outbox; +- durable receipt behavior at every pre-send and post-send failure boundary; +- exact worker continuity across Herdr pane moves and agent recreation. + +If a release fails these gates, roll back the release. Do not reintroduce a +runtime fallback mode. diff --git a/docs/connector-rpc-contract.md b/docs/connector-rpc-contract.md new file mode 100644 index 0000000..a228db3 --- /dev/null +++ b/docs/connector-rpc-contract.md @@ -0,0 +1,231 @@ +# Tendwire connector RPC contract v2 + +Status: authoritative for the connector boundary implemented at Tendwire baseline +`8bc8d8b` and descendants that contain this document. Contract revision `v2` is a +document revision; every request and response that has a `schema_version` field +still uses JSON schema version `1`. + +## Scope and ownership + +Tendwire is the sole owner of delivery durability, ordering, leases, retry state, +dead letters, receipts, and public payload projection. A connector such as Herdres +owns provider ingress, presentation, and its provider-message binding keyed by the +Tendwire delivery key. Herdr owns worker/pane lifecycle and ACP endpoint ownership; +it is not part of this delivery protocol. + +The supported cross-process boundary is Tendwire's local Unix socket. A connector +must not invoke a Tendwire CLI, spawn a subprocess, read Tendwire's database, or +read transcript/runtime files as a fallback. + +## Framing and response envelopes + +The socket carries one newline-terminated UTF-8 JSON request and response per +connection. The canonical request is: + +```json +{"method":"connector.poll","params":{"name":"attention"}} +``` + +`params` must be an object. The optional top-level `id` is echoed when it passes +the daemon's public-boundary validation. No other top-level request fields are +accepted. Frames are bounded by the daemon's configured request and response +limits (1 MiB by default). + +Every connector response has two distinct layers: + +```json +{ + "schema_version": 1, + "ok": true, + "status": "ok", + "error": null, + "result": { + "schema_version": 1, + "ok": false, + "status": "invalid_ref", + "error": {"code": "invalid_ref", "message": "..."} + } +} +``` + +The outer envelope reports daemon framing, method routing, and execution. The +inner `result` reports the connector operation. A caller must require both +`outer.ok` and `outer.result.ok`; outer `ok: true` alone never proves that an ACK +or other mutation succeeded. Invalid framing, invalid top-level parameters, an +unknown method, or an internal daemon failure produces outer `ok: false` with +`result: null`. Invalid or stale connector data produces outer `ok: true` and an +inner `ok: false` result. + +## Stable identities and opaque values + +- `key` is the durable delivery identity. It remains stable across lease expiry, + reclaim, release, defer, and retry. A presentation connector must use it as its + provider-message deduplication/binding key. +- `ref` is an attempt-scoped capability with prefix `twref1.`. It changes whenever + a delivery is leased again. Only the ref from the current live lease may mutate + that attempt; an older ref must be treated as stale or invalid. +- `attempt` increases when a delivery is leased again within one retry generation. + It is not a lifetime counter: an explicit dead-letter `connector.retry` starts a + fresh generation whose first lease has `attempt: 1`. +- `plan_token` (`twplan1.`), `content_revision` (`twrev1.`), and + `final_identity` (`twfinal1.`) are opaque, case-sensitive values. Clients must + preserve their exact UTF-8 bytes and must not parse, normalize, case-fold, or + regenerate them. The same rule applies to their supported positions in a polled + item payload and its nested `turn` and `content` objects. +- Connector payloads are already public, backend-neutral projections. Private + routing, terminal, provider, and credential fields are intentionally absent. + +## Required crash-recovery sequence + +The correct provider-delivery transaction is: + +1. Poll and receive `{key, ref, attempt, payload}`. +2. Look up the connector's durable provider-message binding by `key`. +3. If absent, send `payload` to the provider and durably store `key -> provider + message id` before ACKing Tendwire. +4. ACK the current `ref`. + +If the provider accepted the send but the ACK was lost, lease expiry followed by +`connector.poll` (or an explicit `connector.reclaim` first) returns the same +`key` and payload with a new `ref` and a higher attempt in the current retry +generation. This remains true after a connector process restart: the connector +reopens its durable provider binding, does not send again, and ACKs the new ref. +`connector.poll` reclaims expired leases atomically before selecting work, so +explicit reclaim is an optional eager-maintenance operation, not a correctness +requirement. + +This is the only supported ACK-loss recovery behavior. Payload mutation across +attempts or provider deduplication by `ref` would violate the contract. + +## Socket methods + +The complete connector method set is: + +```text +connector.prepare +connector.poll +connector.ack +connector.fail +connector.defer +connector.renew +connector.release +connector.reclaim +connector.retry +connector.inspect +``` + +There are no `turn_final_*`, `turn-final.*`, or similarly named socket methods. +Turn-final work uses the methods above with `name: "turn-final"`. + +### `connector.poll` + +Canonical params are `name`, with optional `limit` and `lease_seconds`. `limit` +defaults to 1 and is bounded to 1..100. The lease defaults to daemon configuration; +turn-final leases are bounded by the configured maximum, while other neutral +queues permit leases up to 86,400 seconds. + +A successful inner result contains `items`. Each item contains `ref`, `key`, +`attempt`, `leased_until`, `available_at`, and `payload`; turn-final items also +carry `created_at` when available. Polling is FIFO subject to Tendwire's stored +turn-final plan and ordering constraints. + +### `connector.ack` + +Params are `name`, live `ref`, and optional public `response`. Success is inner +status `acknowledged`; the delivery is terminal and is not polled again. An ACK +must use the newest ref obtained after any re-poll. + +### `connector.fail` + +Params are `name`, live `ref`, and optional `reason`, public `response`, +`available_at`, or `delay_seconds`. It records a failed attempt and either returns +the item to the retry schedule or exhausts it according to Tendwire's configured +attempt budget. If Tendwire marked a leased item `terminal_after_lease` because a +newer authoritative plan superseded it, fail instead returns `superseded`, makes +the item terminal, and does not schedule it. Tendwire owns the budget and final +state. + +### `connector.defer` + +Params have the same scheduling shape as `connector.fail`. Defer releases the +current lease onto the requested future schedule without classifying the provider +operation as a delivery failure. For an item marked `terminal_after_lease`, it +instead returns `superseded`, makes the item terminal, and does not schedule it. + +### `connector.renew` + +Params are `name`, live `ref`, and optional `lease_seconds`. It extends the current +lease within the same queue-specific bounds used by poll and returns status +`renewed`. It does not create a new delivery identity. + +### `connector.release` + +Params are `name` and live `ref`. Normally it ends the current lease and makes the +durable delivery eligible for another poll, which creates a new attempt/ref while +retaining the key and payload. For an item marked `terminal_after_lease`, it +instead returns `superseded`, makes the item terminal, and does not requeue it. + +### `connector.reclaim` + +Params are `name`. It expires overdue leases for that queue and reports the +`reclaimed` count. Normal poll already performs this operation transactionally. + +### `connector.prepare` + +Prepare is valid only for `name: "turn-final"` and requires `schema_version: 1`. +It is a strict four-action protocol: + +- `begin`: `action`, `turn_id`, `content_revision`, `presentation_version`, + `part_count`, and optional live `source_ref`. +- `part`: `action`, `plan_token`, zero-based `ordinal`, and non-empty `spans`. + Each span is exactly `{field,start_char,end_char}`, where `field` is + `user_text` or `assistant_final_text` and the range is a valid non-empty slice. +- `commit`: `action`, `plan_token`, and optional live `source_ref`. Commit + materializes durable ordered delivery jobs. +- `recover`: `action`, `failed_plan_token`, and idempotency `request_id`. Recovery + retains the acknowledged prefix and creates executable work for the remaining + suffix according to Tendwire's stored plan state. + +Each action accepts only its declared fields. Callers must retain returned opaque +tokens exactly and honor the returned inner `ok` and `status`. + +The normal final delivery flow is source-bound. Poll the `final_ready` source, +pass its live `source_ref` to both `begin` and `commit`, and then deliver the +materialized parts. A successful source-bound commit atomically changes the source +and its attempt from `leased` to `awaiting_ack`. The source becomes `delivered` +only after all ordered parts are acknowledged. Source-less prepare exists for the +store's explicitly validated source-less/recovery cases; it is not a connector +shortcut around polling the normal source. + +### `connector.inspect` + +This is the strict dead-letter query for turn-final work. Params are exactly +`schema_version: 1`, `name: "turn-final"`, `status: "dead_letter"`, and `limit` +in 1..100. + +### `connector.retry` + +This is the strict operator retry for a turn-final dead letter. Params are exactly +`schema_version: 1`, `name: "turn-final"`, and one selector: either the durable +revision delivery `key` or its `final_identity`. Tendwire validates the +`turn-final:revision:twfinal1.*` identity form and owns the resulting transition. +For an exact dead-letter final anchor, success returns `status: "requeued"` and +`prior_attempt_count`, deletes the superseded per-attempt rows to keep history +bounded, and makes the next poll start at `attempt: 1` with the same key. A later +retry adds the new generation's attempts to `prior_attempt_count`. When the target +identifies one uniquely recoverable failed presentation plan, Tendwire may perform +the plan-recovery path instead and returns that recovery result. + +## Failure handling + +Known inner failures include `invalid_params`, `invalid_ref`, stale/expired-ref +variants, `store_unavailable`, revision/plan conflicts, missing plan or delivery +state, and exhausted attempts. Callers must branch on `result.ok` and +`result.status`, not error-message text. A timeout or broken socket after a +mutation is an unknown outcome: do not assume the mutation failed. Re-poll and use +the durable-key/provider-binding rule to converge safely. + +The daemon sanitizes all connector results at the public boundary. A connector +must not depend on private fields that happen to exist in Tendwire storage, and it +must reject any design that makes its own local cache authoritative for Tendwire +delivery state. diff --git a/docs/reduction-baseline.md b/docs/reduction-baseline.md new file mode 100644 index 0000000..6ff2976 --- /dev/null +++ b/docs/reduction-baseline.md @@ -0,0 +1,97 @@ +# Tendwire reduction baseline + +This baseline was measured at commit `91891bf` on branch `wave-0/baseline` on +2026-08-04. The production root is `src/`. + +## SLOC calibration + +Canonical production SLOC was measured with: + +```console +python3 /home/smith/acp-reduction-goal/tools/sloc_count.py \ + /home/smith/tendwire/.worktrees/wave-0-baseline/src +``` + +The counter reports **57,007 SLOC**, exactly matching the stated baseline of +57,007: a difference of **0 lines (0.00%)**. The canonical checkout at +`/home/smith/tendwire/src` independently reports the same total. + +Physical lines are newline-terminated lines reported by `wc -l` for the same +production Python files. Canonical SLOC excludes blank lines, full-line +comments, and module/class/function docstrings according to +`tools/sloc_count.py`. Tests, docs, tooling, generated code, and vendored code +are outside both tables. + +### Directory summary + +| Directory | Physical lines | Canonical SLOC | +|---|---:|---:| +| `tendwire/` (direct modules) | 12,337 | 10,978 | +| `tendwire/backends/` | 14,761 | 12,724 | +| `tendwire/connectors/` | 686 | 624 | +| `tendwire/core/` | 6,884 | 5,925 | +| `tendwire/store/` | 28,239 | 26,756 | +| **Total** | **62,907** | **57,007** | + +### Per-module measurements + +| Module | Physical lines | Canonical SLOC | +|---|---:|---:| +| `tendwire/__init__.py` | 5 | 2 | +| `tendwire/_version.py` | 3 | 1 | +| `tendwire/backends/__init__.py` | 1 | 0 | +| `tendwire/backends/acp_client.py` | 1,688 | 1,483 | +| `tendwire/backends/acp_coordinator.py` | 2,413 | 2,148 | +| `tendwire/backends/acp_ingestion.py` | 714 | 612 | +| `tendwire/backends/acp_permissions.py` | 295 | 263 | +| `tendwire/backends/acp_probe.py` | 360 | 295 | +| `tendwire/backends/acp_projection.py` | 1,299 | 1,117 | +| `tendwire/backends/acp_protocol.py` | 613 | 480 | +| `tendwire/backends/acp_runtime.py` | 1,270 | 1,036 | +| `tendwire/backends/herdr_cli.py` | 2,671 | 2,304 | +| `tendwire/backends/herdr_command.py` | 156 | 125 | +| `tendwire/backends/herdr_events.py` | 2,454 | 2,209 | +| `tendwire/backends/herdr_protocol.py` | 363 | 260 | +| `tendwire/backends/herdr_socket.py` | 464 | 392 | +| `tendwire/cli.py` | 1,856 | 1,669 | +| `tendwire/command_submission.py` | 2,306 | 2,016 | +| `tendwire/config.py` | 726 | 686 | +| `tendwire/connectors/__init__.py` | 5 | 2 | +| `tendwire/connectors/outbox.py` | 681 | 622 | +| `tendwire/core/__init__.py` | 1 | 0 | +| `tendwire/core/actions.py` | 219 | 179 | +| `tendwire/core/agent_events.py` | 393 | 328 | +| `tendwire/core/attention.py` | 196 | 155 | +| `tendwire/core/commands.py` | 1,385 | 1,141 | +| `tendwire/core/models.py` | 1,961 | 1,706 | +| `tendwire/core/projector.py` | 68 | 51 | +| `tendwire/core/turns.py` | 2,661 | 2,365 | +| `tendwire/daemon.py` | 1,361 | 1,226 | +| `tendwire/daemon_api.py` | 1,833 | 1,706 | +| `tendwire/local_state.py` | 3,868 | 3,342 | +| `tendwire/store/__init__.py` | 1 | 0 | +| `tendwire/store/sqlite.py` | 28,238 | 26,756 | +| `tendwire/worker_identity.py` | 379 | 330 | +| **Total** | **62,907** | **57,007** | + +## Test baseline + +The existing repository virtual environment was used without installing or +upgrading anything. This worktree's source tree was selected explicitly: + +```console +env PYTHONPATH=/home/smith/tendwire/.worktrees/wave-0-baseline/src \ + /home/smith/tendwire/.venv/bin/python -m pytest tests/ -x -q +``` + +Result: + +```text +2879 passed, 2 skipped in 496.88s (0:08:16) +``` + +The authoritative run was performed outside the managed filesystem sandbox. +Inside that sandbox, a hardened Unix-socket test cannot bind through its +`/proc/self/fd/...` path and stopped the first run after 428 passes. The same +isolated test passed outside the sandbox (`1 passed in 3.06s`), after which the +unchanged full-suite command above completed green. diff --git a/docs/wave4-store-design.md b/docs/wave4-store-design.md new file mode 100644 index 0000000..732736f --- /dev/null +++ b/docs/wave4-store-design.md @@ -0,0 +1,1235 @@ +# Wave 4 T6 store design + +Status: design candidate. Production implementation is blocked until this note +passes adversarial review and the paired Herdres connector contract and tests +accept the same payload versions, recovery rules, and provider-binding model. + +Baseline: Tendwire `9ab5597b55bb918ac8e62c86651ccac7c03d18fb` and +Herdres `ec0e36a25b16469979402abfc8bfa1525db6f68a`. + +## Non-negotiable connector boundary + +T6 keeps the authoritative connector boundary already in production: + +- The connector name is exactly `turn-final`. +- The outer `connector.poll` item `key` is the sole durable replay and + provider-message binding identity. It is stable across lease expiry, reclaim, + release, defer, retry, daemon restart, and lost ACK responses. +- `ref` is an attempt-scoped `twref1.` capability. Only the newest live ref may + settle or renew an attempt. +- `attempt` starts at one and increases within a retry generation. Explicit + dead-letter retry starts a new generation at attempt one without changing the + outer key. +- The socket methods remain exactly `connector.prepare`, `connector.poll`, + `connector.ack`, `connector.fail`, `connector.defer`, `connector.renew`, + `connector.release`, `connector.reclaim`, `connector.retry`, and + `connector.inspect`. +- `connector.prepare` remains valid only for `name: "turn-final"`, request + `schema_version: 1`, and the existing `begin`, `part`, `commit`, and `recover` + action shapes. No required or optional request field is added. +- `connector.inspect` and `connector.retry` retain their existing strict request + field shapes. Contract-v3 inspect covers every retained `turn-final` dead + letter. Retry keeps the one `key` or `final_identity` selector: final identity + selects a final root, while key selects an exact retryable root, standalone + decision, or standalone retire as defined below. + +The AF_UNIX framing and RPC document remains connector transport contract v2; +top-level and inner operation envelopes continue to use JSON +`schema_version: 1`. The paired `turn-final` payload and operation-response +document revision is presentation contract v3. As with transport contract v2, +the document revision is not copied into every payload or result. The exact +kind-specific schema versions below identify the paired payload family. This +is not a new socket method, action, framing version, or replay identity. + +There is no `telegram-present` connector, alias connector, second presentation +queue, turn-list-driven final send, connector-specific socket method, inner +`job_key`, or copy of the current row's outer key as a parallel self-identity. +Lineage and retire payloads may reference another row's outer key only as an +explicit predecessor/replacement/target correlation; those references never +deduplicate or settle the current row. Herdres must continue to durably bind +the polled outer key to provider result before ACKing. That provider receipt +ledger remains required; the separate Herdres plan/pending-plan/recovery ledger +does not. + +## Scope, module budget, and gates + +T6 replaces `store/sqlite.py` with one fresh schema and concern-owned modules. +The target is 5,225 canonical SLOC, within the 4,500-5,500 gate. + +| Module | Responsibility | Budget | Public API | +| --- | --- | ---: | --- | +| `store/schema.py` | one DDL, exact-version cutover | 400 | `STORE_SCHEMA_VERSION`, `init_store`, `ensure_schema` | +| `store/db.py` | secure open, pragmas, read/write transactions, bounded health | 225 | `connect_read`, `read_transaction`, `write_transaction`, `store_status` | +| `store/events.py` | append, authoritative dedupe, bounded query | 300 | `record_agent_event`, `list_agent_events` | +| `store/turns.py` | atomic projection, content, list, delta, paging | 1,050 | append/apply result types, `append_agent_event_and_apply_turn_for_binding`, `apply_turn_refresh`, `turns_payload_from_store`, `turn_delta_payload_from_store`, `get_turn_content` | +| `store/projection.py` | snapshots, attention, bindings, route generation, health | 475 | snapshot context/save/latest, attention payload, binding upsert/list/expiry, backend-pending health | +| `store/pending.py` | pending observations and fenced decision claims | 425 | observation/payload, claim/start/abandon/terminal-effect functions | +| `store/receipts.py` | command receipts, submissions, replay and linking | 800 | current command reservation/send/finish/recovery/link functions | +| `store/outbox.py` | one queue, FIFO/DAG, leases, prepare, recovery, dead-letter | 1,350 | the current connector store functions only | +| `store/retention.py` | bounded cutoff deletion and WAL checkpoint | 200 | `RetentionPolicy`, `run_retention_cycle` | +| **Total** | | **5,225** | | + +The module rows still sum exactly 5,225; this correction adds no module or +second implementation path. Caller/configuration rewrites listed in file scope +replace existing lines and are measured in the repository-wide gate, not +double-counted as store-component SLOC. + +No function may exceed 150 lines. Orchestration functions target at most 130 +lines; transaction-local transition/query helpers target at most 60. There is +no compatibility re-export, generic repository/CRUD layer, migration chain, +authority registry, direct-SQLite CLI fallback, maintenance state machine, +automatic VACUUM, or compaction framework. + +## Fresh schema + +The application schema has 16 tables: + +- Projection: `turns`, `turn_content_revisions`, + `turn_content_page_boundaries`, `attention_items`, `pending_interactions`, + `snapshots`, and `agent_events`. +- Commands: `command_receipts`, `turn_submissions`, `turn_supersessions`, + `backend_pending`, `backend_pending_claims`, `worker_bindings`, and + `backend_health`. +- Delivery: `connector_outbox` and `connector_deliveries`. + +Explicitly absent are `events`, `spaces`, `workers`, `commands`, migration +tables, store-maintenance state/cursors, presentation plan/job/recovery tables, +turn-change state/floor tables, tombstone tables, and every `herdr_turn_*` +table. A retained removal row in `turns` is the delta tombstone. The greatest +turn/delta sequence row is retained as its allocation sentinel, and the +retained worker-binding allocator owns partition sequences, so a published +sequence is never reused. + +`turns` is keyed by `(host_id, turn_id)` and carries current ownership, +`route_generation`, insertion and change sequence, projection state, and +`removed_at`. There is no prepare-authority or source-less-presentation column. +Every final presentation plan is rooted in one polled `final_ready` outbox row. +`worker_bindings` carries exact columns `stable_key`, `stable_key_version`, +`route_generation`, `partition_key`, +`next_partition_sequence`, and `route_retain_until` alongside its private +backend binding. Content revisions are immutable, with one partial-unique +current row per turn. Page boundaries are revision/field/page-coordinate unique +and cascade only with a revision that retention has proved unreferenced. + +Command receipts preserve: + +```text +missing -> reserved -> send_started -> accepted | rejected | uncertain +``` + +Only an identical canonical request may take over an expired reservation; +terminal receipts are immutable. Turn submissions preserve the existing +send-started/submitted/uncertain/link/ambiguous/expired/cancelled semantics, +one-to-one linkage, owner/instruction matching, and fail-closed ambiguity. +Backend-pending claims remain fenced and are abandonable only before send +start. + +## Two delivery tables + +`connector_outbox` contains both immediately executable work and staged +`final_part` rows. Its logical columns are: + +```text +id, host_id, connector, key, kind, payload_version, status, +partition_key, partition_sequence, +turn_id, final_identity, content_revision, presentation_version, +plan_token, plan_generation, logical_sequence, logical_ordinal, +predecessor_outbox_id, replaces_outbox_id, target_outbox_id, +source_outbox_id, active_lineage_generation, +recovery_request_digest, recovered_from_plan_token, +terminal_after_lease, retry_generation, prior_attempt_count, +current_delivery_id, +payload_json, +created_at, updated_at, available_at +``` + +Required uniqueness is: + +```text +(host_id, connector, key) +(host_id, connector, partition_key, partition_sequence) +(host_id, connector, source_outbox_id, plan_generation, logical_sequence) +(host_id, connector, recovery_request_digest) WHERE recovery_request_digest IS NOT NULL +``` + +`status` has an exact database CHECK over `staged`, `blocked`, `queued`, +`leased`, `retry`, `deferred`, `awaiting_ack`, `delivered`, `superseded`, and +`dead_letter`. `kind` is CHECKed over `generic`, `working`, `final_ready`, +`final_part`, `retire`, and `decision`; `connector = 'turn-final'` requires one +of the five typed kinds, and every other connector requires `generic`. A +kind/status CHECK permits only the combinations in the matrix below, and a +kind/payload-version CHECK fixes working/retire/decision at one, final-part at +two, and final-ready at three. `predecessor_outbox_id`, `replaces_outbox_id`, `target_outbox_id`, and +`source_outbox_id` are nullable self-foreign keys with `ON DELETE RESTRICT`. +`current_delivery_id` is a nullable foreign key to `connector_deliveries(id)` +with `ON DELETE RESTRICT`; transition code additionally proves that the pointed +delivery's `outbox_id` is this row. Terminal outbox rows have a null current +delivery. Leased and awaiting-ACK rows have exactly one. + +`connector_deliveries` contains attempts only: + +```text +id, outbox_id, retry_generation, attempt, +ref_hash, status, leased_at, leased_until, ack_deadline_at, +public_response_json, private_reason_enum, +created_at, settled_at +``` + +`outbox_id` is non-null and references `connector_outbox(id) ON DELETE +RESTRICT`. Delivery `status` has an exact CHECK over `leased`, `awaiting_ack`, +`acknowledged`, `failed`, `deferred`, `released`, and `expired`. Required +uniqueness is `(outbox_id, retry_generation, attempt)` and `(ref_hash)`, plus a +partial unique live-attempt index on `outbox_id` where status is `leased` or +`awaiting_ack`. Raw refs are never stored. `public_response_json` is a bounded, +exact sanitized object; `private_reason_enum` is nullable and CHECKed against a +fixed enum: `temporary`, `rate_limited`, `provider_rejected`, +`provider_uncertain`, `invalid_payload`, `content_unavailable`, +`route_unavailable`, `provider_binding_unknown`, `lease_expired`, +`ack_deadline_expired`, `superseded`, `attempts_exhausted`, or +`operator_recovery`. There is no +`private_state_json` in either delivery table. + +`ack_deadline_at` is a canonical UTC timestamp or null. It is non-null exactly +when delivery status is `awaiting_ack`; it is null for ordinary leases and all +terminal attempts. The source-commit and recovery rules below are its only +initializers or mutators. + +Database constraints provide identity and deletion fencing; transaction code +provides the cross-row domain checks SQLite CHECKs cannot express. A staged or +executable `final_part` and every plan retire must reference exactly one +`final_ready` source of the same host, connector, turn, final identity, +revision, partition, and active lineage generation. Its predecessor, +replacement, and target must +have the same host/connector and the declared compatible route/revision. A +predecessor must have a lower logical sequence unless it is the delivered tail +of an inherited recovery prefix. Cycles, cross-root children, a root pointing +at its child as current delivery, or a delivery pointing at a different outbox +row abort the transaction. Root completion, supersession, retry, and recovery +update the root and every affected child in the same `BEGIN IMMEDIATE`. + +The exact nullability/correlation rules are also enforced on every write: + +- `working` and `decision` have no source, plan, logical-ordinal, target, or + recovery columns; only `replaces_outbox_id` may be set. +- `final_ready` has final identity/revision but no source, plan, predecessor, + target, or recovery parent. Its current delivery exists only while leased or + awaiting ACK. +- `final_part` has non-null plan token/generation/sequence/ordinal and a + non-null final-ready source. Its target is null; predecessor/replacement must + satisfy the declared lineage. +- A plan `retire` has plan coordinates, non-null target and predecessor, and the + same non-null final-ready source as its lineage. A standalone retire has no + plan/source columns, has a non-null decision target, and uses only the + decision-resolution predecessor rule. +- Only recovery-created executable rows have `recovered_from_plan_token`. + Exactly the first fresh row of a recovery generation carries the non-null + `recovery_request_digest`; every other row has null, so the declared unique + index is executable and that head is retained for the full lineage horizon. + +The outer key grammars are exact: + +```text +final_ready: turn-final:revision:twfinal1.<43 base64url characters> +final_part or plan retire: turn-final:twplan1.<1-256 base64url characters>: +working: turn-final:working:twwork1.<43 base64url characters> +decision: turn-final:decision:twdecision1.<43 base64url characters> +standalone retire: turn-final:retire:twretire1.<43 base64url characters> +``` + +Contract-v3 Tendwire final identities are always 32-byte digests encoded as 43 +unpadded base64url characters. A contract-v3 `final_ready` payload, outer key, +prepare source root, inspect selector, and retry selector reject every other +length. Existing non-v3 RPC positions that accept opaque `twfinal1.*` or +`twplan1.*` values may retain their transport-v2 bounded 1--256-character +validator, but that compatibility does not widen the five-kind contract. + +Producer identities are deterministic canonical SHA-256 digests, not random +per enqueue. `twwork1` hashes the domain, host, turn ID, content revision, and +route generation. `twdecision1` hashes the domain, host, decision ref, revision +digest, and route generation. `twretire1` hashes the domain, target decision +outer key, the fixed `decision_resolved` reason, resolving revision, and route +generation. There is no route-retirement producer. Replaying the same producer +transaction therefore finds the same outer key and must prove an identical +payload; a mismatch is a conflict. Plan tokens are the one persisted opaque +value allocated by idempotent begin, and plan child keys are deterministic from +that token and logical sequence. + +These tokens occur only as parts of the outer key or as non-replay correlation +coordinates declared below. No payload has `key`, `job_key`, `delivery_key`, +`ref`, or `attempt`. + +## Outbox status matrix + +| Kind | Legal statuses | Legal atomic transitions | +| --- | --- | --- | +| `generic` (non-`turn-final` queues only) | `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | preserves the existing neutral enqueue/poll/ACK/fail/defer/release/reclaim transitions; it cannot participate in prepare, route partitions, final lineage, or the five-kind payload contract | +| `working` | `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | enqueue to `queued`; due poll to `leased`; ACK to `delivered`; fail to `retry` or `dead_letter`; defer to `deferred`; release/expiry to `queued`; unleased supersession to `superseded`; leased supersession sets `terminal_after_lease`, after which ACK is `delivered` and fail/defer/release/expiry is `superseded` | +| `final_ready` | `queued`, `leased`, `retry`, `deferred`, `awaiting_ack`, `delivered`, `superseded`, `dead_letter` | ordinary lease transitions; first successful commit `leased` to `awaiting_ack`; complete effective lineage to `delivered`; ACK-deadline expiry to `dead_letter`; only an uncommitted root dead letter may retry to `queued` in a new retry generation; recovery replaces a failed suffix while the committed root remains live `awaiting_ack`; an uncommitted superseded root becomes `superseded`; a committed terminal root is never re-prepared from scratch | +| `final_part` | `staged`, `blocked`, `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | begin creates `staged` placeholders; part fills one staged row idempotently; commit freezes payloads and moves the head to `queued` and successors to `blocked`; predecessor ACK unblocks exactly one successor; ordinary lease transitions; recovery retains a delivered prefix, supersedes the failed suffix, and creates a new suffix | +| `retire` | `blocked`, `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | commit/resolution creates `blocked`; a delivered replacement normally moves it to `queued`, with the exact leased-decision exception below; ordinary lease transitions; supersession is legal only when Tendwire proves the target never had a delivery attempt and therefore could not have been provider-accepted; a mandatory retire for a leased, expired, failed, deferred, acknowledged, or otherwise possibly accepted target remains retryable/dead-letter work and is never superseded | +| `decision` | `queued`, `leased`, `retry`, `deferred`, `delivered`, `superseded`, `dead_letter` | pending projection/enqueue to `queued`; ordinary lease transitions; resolution atomically supersedes unleased work or marks leased work terminal-after-lease and creates the ordered retire | + +`staged`, retained `awaiting_ack` roots, and terminal rows do not block FIFO. +An executable row is pollable only when it is due, its explicit predecessor is +`delivered`, and no earlier executable row in the same partition is in +`queued`, `leased`, `retry`, or `deferred`. Poll reclaims expired leases before +selection in the same `BEGIN IMMEDIATE` transaction. + +The one explicit predecessor exception is a mandatory standalone retire for a +resolved decision that was already leased. It becomes eligible after the target +decision has no live attempt and is terminal as `delivered`, `superseded`, or +`dead_letter`, because any prior lease may have reached the provider even when +Tendwire did not receive an ACK. A decision superseded before its first lease +cannot have a provider object; its retire is provably unnecessary and may be +superseded. Replacement-driven working/final retires still require the +replacement key itself to be `delivered`. + +For that decision exception the retire payload has `predecessor_key` equal to +the target decision key. Herdres does not require a delivered provider job for +that key: it performs its immutable target lookup. If H7 has no provider job, +alias, or tombstone for a possibly accepted target, it returns the fixed +`provider_binding_unknown` failure. Tendwire immediately dead-letters the +mandatory retire, keeps its target/reason visible through inspect for the +30-day targetable horizon, and never converts absence into ACK, deletion, or +"unnecessary". Explicit retry may make the same target visible again but +cannot reconstruct an unknown provider coordinate. + +## Delivery-attempt and CAS matrix + +| Operation | Required current state | `connector_deliveries` transition | `connector_outbox` transition | +| --- | --- | --- | --- | +| poll | due eligible work, no blocker | insert `leased` at next attempt | executable work to `leased` | +| renew | newest unexpired ref; both rows `leased` | extend `leased_until` | remains `leased` | +| ACK | newest unexpired ref; both rows `leased` | `leased` to `acknowledged` | work to `delivered`; atomically unblock successor | +| fail, ambiguous new send | newest unexpired ref; exact `provider_uncertain` classification | `leased` to `failed` | immediate `dead_letter` regardless of attempt budget; effective root lineage updated atomically | +| fail, mandatory retire has unknown binding | newest unexpired ref; exact `provider_binding_unknown` classification | `leased` to `failed` | immediate visible `dead_letter`; target is retained and never inferred absent | +| fail, budget remains | newest unexpired ref; both rows `leased` | `leased` to `failed` | work to `retry` | +| fail, exhausted | newest unexpired ref; both rows `leased` | `leased` to `failed` | work/effective lineage to `dead_letter` | +| defer | newest unexpired ref; both rows `leased` | `leased` to `deferred` | work to `deferred` | +| release | newest unexpired ref; both rows `leased` | `leased` to `released` | work to `queued`, or `superseded` when terminal-after-lease | +| lease expiry | current `leased`, deadline due | `leased` to `expired` | work to `queued`, or `superseded` when terminal-after-lease | +| source commit | live final-ready source ref | `leased` to `awaiting_ack`; initialize `ack_deadline_at` once | root to `awaiting_ack`; head queued and tail blocked | +| lineage complete | every effective node delivered before the exact ACK deadline | `awaiting_ack` to `acknowledged`; clear deadline | root to `delivered` | +| lineage deadline | no live child lease; exact ACK deadline reached | `awaiting_ack` to `failed`; clear deadline | root and effective tail to `dead_letter` | +| explicit root retry | exact retryable root dead letter | prior generation remains terminal/aggregated | same root key to `queued`, retry generation +1; next attempt is one | +| recover | exact recoverable failed lineage under a live awaiting-ACK root | same attempt remains `awaiting_ack`; one new request digest replaces its deadline | prefix retained; failed non-retire tail superseded; possibly accepted mandatory retire audit rows retained dead-letter and linked to their fresh copies; fresh suffix created | + +Every ref mutation uses one CAS equivalent to: + +```text +outbox.id = delivery.outbox_id +AND outbox.host_id = requested host +AND outbox.connector = requested connector +AND outbox.status = expected outbox status +AND delivery.id = outbox.current_delivery_id +AND delivery.status = expected delivery status +AND delivery.ref_hash = H(presented ref) +AND delivery.leased_until > now +``` + +Zero changed rows returns the existing stale/invalid-ref result and commits no +partial transition. An old ref cannot mutate a later attempt. An explicit +retry never revives an old ref or attempt counter. + +The retained configuration is exactly +`TENDWIRE_CONNECTOR_ACK_TTL_SECONDS`/`connector_ack_ttl_seconds`, default 60. +Startup accepts only a non-Boolean integer in `1..86400`; it never clamps. The +first successful source commit computes `ack_deadline_at = commit_now + ttl` +inside the commit transaction. A byte-equivalent commit replay returns the +persisted deadline and never extends it. Lineage completion CASes the exact +root/current-delivery pair only while `ack_deadline_at > now`; deadline expiry +CASes that same pair only when `ack_deadline_at <= now` and no child has a live +lease. Exactly one successful recover request digest may replace the still-live +awaiting-ACK deadline with `recover_now + ttl`; an identical request replay +returns the persisted replacement deadline, and a different request must +create the next recovery generation or fail conflict. Recover after deadline +expiry is rejected. The committed expired root is terminal and +`not_retryable`; only a later producer revision may create unrelated new work. +Retention treats every non-null deadline and its effective lineage +as live, and cannot delete either until the root is terminal and the applicable +reference horizon has elapsed. + +Inspect request fields remain exactly `schema_version: 1`, +`name: "turn-final"`, `status: "dead_letter"`, and integer `limit` in `1..100`. +A successful inner result has exactly: + +```text +schema_version, ok, status, host_id, name, total, items +``` + +Its values are `schema_version=1`, `ok=true`, `status="ok"`, and `total` is the +full matching root/effective-lineage count before the limit. Every item has all +of these exact fields, with inapplicable correlations present as null: + +```text +kind, key, final_identity, failed_plan_token, decision_ref, target_key, +reason, attempt_count, prior_attempt_count, created_at, terminal_at, retryable, +recoverable +``` + +`reason` is one value from the fixed public reason enum corresponding to the +stored private classification. `retryable` is true only when the exact +`connector.retry` selector can currently act; `recoverable` is true only for an +exact failed plan whose source root is still awaiting ACK before its deadline +and can accept `connector.prepare(action="recover")`. Items contain no text, +payload, ref, provider fact/coordinate, private route, or raw diagnostic. Root +and plan-child failures are reported once at the root/effective-lineage level. +Inspect errors have exactly `schema_version`, `ok`, `status`, `host_id`, `name`, +`message`, and `items`, with `ok=false`, `items=[]`, and the common fixed error +rules below. + +Retry keeps its exact mutually exclusive selector shapes. `final_identity` +and a final-ready key requeue only an exact uncommitted dead-letter root in a +new retry generation; the root is then polled and prepared with its new live +ref. A committed terminal root is `not_retryable` and is never prepared from +scratch. `connector.retry` never invokes plan recovery. A still-awaiting-ACK +root with a recoverable failed suffix uses only +`connector.prepare(action="recover")`. +A standalone decision key or standalone +retire key starts a new retry generation on that same outer key, with attempt +one and accumulated prior-attempt count. A plan retire/final-part key is not +independently retried because root recovery owns its suffix. Working dead +letters are observationally replaced only by a newer deterministic producer +revision and are not operator-retried. Unknown, terminal-success, stale +revision, nonstandalone child, and ambiguous selectors return fixed +`not_retryable`/`stale_revision`/`invalid_params` outcomes without mutation. + +Provider-start classification is exact. A definitely-not-started provider call +uses `temporary` or `rate_limited` and follows the ordinary bounded retry/defer +path. An ambiguous new send for `working`, `final_part`, or `decision` uses +`provider_uncertain`; fail ignores remaining attempt budget and atomically moves +that row/effective lineage to `dead_letter` for inspection. Automatic retry is +forbidden because it can duplicate a message. Ambiguous known-target edit or +delete does not use `provider_uncertain`: Herdres defers the same immutable key +and target, then converges by exact not-modified or not-found handling. Invalid +reason/operation combinations are `invalid_params` without mutation. Explicit +operator retry/recover of a `provider_uncertain` row is allowed only with a +visible warning that the provider may already have accepted the send and the +operation can create a duplicate; recovery preserves the prior ambiguous audit +row and never claims the risk was repaired. + +## Exact additive payload contract + +The existing poll item envelope is unchanged: + +```json +{ + "key": "...", + "ref": "twref1....", + "attempt": 1, + "leased_until": "2026-08-05T00:00:00Z", + "available_at": "2026-08-05T00:00:00Z", + "created_at": "2026-08-05T00:00:00Z", + "payload": {} +} +``` + +Each payload has exactly these common keys: + +```text +schema_version, kind, created_at, worker, route +``` + +The kind-specific `schema_version` values below version that payload object; +they do not change the transport-v2 envelope's JSON schema version 1. + +`worker` is exactly: + +```text +worker_id, stable_key, stable_key_version, route_generation +``` + +`route` is exactly: + +```text +partition_key, partition_sequence +``` + +The kind-specific versions and keys are: + +### `working`, schema version 1 + +Adds only `turn`. `turn` is exactly: + +```text +turn_id, content_revision, replaces_key, text +``` + +`replaces_key` is an outer key or null. `text` is exactly: + +```text +assistant_stream_text, char_length, byte_length +``` + +The text is a bounded sanitized inline projection, and the two lengths must +match its exact Unicode scalar and UTF-8 byte lengths. + +### `final_ready`, schema version 3 + +Adds only `turn`. `turn` is exactly: + +```text +turn_id, final_identity, content_revision, replaces_key, content +``` + +`replaces_key` is an outer working/final key or null. `content` remains exact +content-schema-v1: + +```text +schema_version, content_revision, known_incomplete, fields +``` + +`fields` has exactly `user_text` and `assistant_final_text`; each descriptor is +exactly `availability`, `inline`, `char_length`, `byte_length`, `page_count`, +and `first_cursor`. The final-ready revision, nested revision, cursors, lengths, +and outer-key final identity must correlate exactly. + +### `final_part`, schema version 2 + +Adds exactly `turn`, `plan`, and `lineage`. + +```text +turn: turn_id, final_identity, content_revision +plan: plan_token, generation, presentation_version, ordinal, part_count, spans +lineage: recovered_from_plan_token, predecessor_key, replaces_key +``` + +Nullable lineage values are present as null, not omitted. Each span is exactly +`field`, `start_char`, and `end_char`; field is `user_text` or +`assistant_final_text`. Spans are nonempty, ordered, nonoverlapping exact +character slices of the retained revision. No final text is copied into a +plan row. + +### `retire`, schema version 1 + +Adds exactly `turn` and `retire`. + +```text +turn: turn_id, final_identity, content_revision +retire: target_key, target_kind, target_ordinal, predecessor_key, + plan_token, generation, reason +``` + +Nullable coordinates are present as null. `target_kind` is `working`, +`final_part`, or `decision`. Reason is exactly one of `working_replaced`, +`final_replaced`, `excess_part`, or `decision_resolved`. +There is no content or provider coordinate. + +### `decision`, schema version 1 + +Adds only `decision`, exactly: + +```text +decision_ref, revision_digest, mode, title, body, choices +``` + +`mode` is `single`, `multi`, or `plan`. Each choice is exactly `ordinal`, +`option_ref`, and `label`. `option_ref` is the public one-based ordinal from the +semantic decision contract; the private ACP option ID and backend route remain +only in `backend_pending`. + +Unknown keys, unknown versions/kinds, booleans used as integers, noncanonical +timestamps, malformed opaque tokens, duplicate ordinals, inconsistent lengths, +cross-revision cursors/spans, and correlation mismatch reject enqueue, prepare, +or poll before any connector-visible mutation. + +## Route generation and concurrency + +`route_generation` has exact grammar: + +```text +twroute1.<43 unpadded URL-safe base64 characters> +``` + +It encodes 32 cryptographically random bytes. In the atomic worker-binding +projection, Tendwire performs lookup-or-mint for this exact private tuple: + +```text +(host_id, stable_key_version=1, stable_key, backend, + binding_private_fingerprint) +``` + +An identical retained tuple reuses its generation. A changed stable identity, +backend/private binding fingerprint, deliberate installation-key rotation, or +resolved live-route collision mints a new generation. Ordinary daemon/Herdres +restart, worker label change, content update, lease attempt, or local Herdres +JSON revision does not rotate it. Multiple live route tuples claiming the same +stable identity fail closed. + +Route enrichment is part of the same snapshot/binding `BEGIN IMMEDIATE` that +accepts the authoritative worker observation. It validates the derived stable +key pair, selects or mints the route generation, derives the partition key, +stores all route columns on `worker_bindings`, and copies the exact public +stable-key/version/generation triplet into the worker and affected turn +projection before commit. No snapshot, turn, delta, pending decision, or +outbox producer may publish a partially enriched route. Direct binding upsert +uses the same helper and cannot invent a generation independently. + +`save_snapshot` and the binding transaction return the persisted enriched +snapshot, never the caller's pre-enrichment object. Before persistence and +return, every `Worker.fingerprint` is recomputed from the exact canonical +public Worker projection excluding the fingerprint itself and volatile +timestamps but including `stable_key`, `stable_key_version`, and the non-null +`route_generation`. Snapshot `content_fingerprint` is then recomputed from those +enriched workers. The coordinator replaces its in-memory snapshot with this +returned value before publishing, routing, or accepting a command. + +Post-T6 command reservation/send CASes the exact current `worker_id`, enriched +`worker_fingerprint`, stable key/version, route generation, and private binding +fingerprint in the same transaction. A mismatch is stale route and authorizes +no backend send. H8's one-time removal of `worker_fingerprint` from a stored +command is legal only before the paired cutover when the captured route +generation is null. Once a command carries a non-null `twroute1.*`, fingerprint +removal is forbidden; a mismatch quarantines rather than weakening the CAS. + +The generation is published in worker metadata, copied into current turn/list/ +delta ownership, and frozen in every queued payload. It is not a Telegram +topic generation and is never compared to Herdres's local state revision. + +`partition_key` has exact grammar `twpart1_` plus 64 lowercase hexadecimal +characters and is SHA-256 over canonical UTF-8 JSON: + +```json +{ + "domain": "tendwire.turn-final.partition.v1", + "host_id": "...", + "route_generation": "twroute1....", + "stable_key": "wsk1_...", + "stable_key_version": 1 +} +``` + +Different Tendwire partitions may lease concurrently. After resolving a +public route, Herdres must additionally serialize provider writes by its +private physical `(bot, chat, topic)` coordinate. Two route generations mapped +to one retained topic therefore cannot concurrently edit or delete it. Topic +creation and provider routing remain Herdres-owned; Tendwire sees only neutral +defer/fail/release/ACK outcomes. + +`worker_bindings.next_partition_sequence` is the durable allocator for its +exact partition. Every producer transaction atomically increments it with +`UPDATE ... RETURNING` and inserts the outbox row with the returned positive +sequence; `MAX(outbox.sequence)+1` is never used. The binding/allocator row is +retained while current/sendable, referenced by any Tendwire turn or outbox row, +or before its Tendwire-owned `route_retain_until`. + +The paired constants are exact minimum floors: terminal targetable Tendwire +outbox rows and their inspect/retry correlation are retained for 30 days; +referenced content revisions, page boundaries, worker bindings, route +generations, partition allocators, and installation-key derivation evidence are +retained for 45 days; H7 immutable provider jobs/messages/aliases/tombstones are +retained for 60 days. The corresponding names are +`TURN_FINAL_TARGETABLE_RETENTION_DAYS = 30`, +`TURN_FINAL_ROUTE_CONTENT_RETENTION_DAYS = 45`, and the cross-repository H7 +`PROVIDER_FACT_RETENTION_DAYS = 60`. Configuration may lengthen but never +shorten these floors. Live references, awaiting-ACK roots, current/recovered +lineage, active routes, and unresolved mandatory retires override every cutoff. +After references and the applicable horizon are both gone, retention may remove +the rows child-first. Tendwire neither reads nor infers Herdres provider +bindings and never shortens a horizon based on unknown provider state. + +## Prepare lease, idempotence, and response-loss recovery + +Prepare requests keep transport-v2 `schema_version: 1` and the four existing +action field sets. Contract v3 removes source-less preparation completely. +`begin` requires `source_ref`; the first `commit` requires `source_ref`; omission +or explicit null is `invalid_ref`. `part` remains token-fenced and has no +source-ref field. `recover` is the sole recovery action and accepts only its +unchanged `failed_plan_token` and `request_id` fields. No internal producer, +startup scan, retry path, or rediscovery path may call `begin` without a polled +source root. + +Successful action results have the following exact fields and no others: + +```text +begin: + schema_version, ok, status, host_id, name, plan_token, state, generation, + part_count, accepted_ordinals +part: + schema_version, ok, status, host_id, name, plan_token, state, generation, + part_count, ordinal, accepted_ordinals +commit: + schema_version, ok, status, host_id, name, plan_token, state, generation, + part_count, job_count, accepted_ordinals +recover: + schema_version, ok, status, host_id, name, failed_plan_token, plan_token, + generation, content_revision, state, acknowledged_prefix_count, + executable_job_count, retained_failed_job_count, prior_attempt_count, + idempotent_replay +``` + +`schema_version` is one. `ok` is true. Begin/part/commit `status` is `ok`; +recover status is `recovered`. `accepted_ordinals` is a sorted unique bounded +array of non-Boolean zero-based integers in `[0, part_count)`, with +`part_count` bounded to `1..10000`. Begin returns all already accepted ordinals, +part includes its exact `ordinal`, and commit returns every ordinal +`0..part_count-1`. `generation`, `part_count`, `ordinal`, and every count reject +Boolean values; generation and part count are positive and the other counts are +nonnegative. Begin/part state is exactly `preparing`, `active`, +`waiting_predecessor`, `completed`, `failed`, or `superseded`; commit state +excludes `preparing`, and recover state is exactly `active`. An identical replay +returns the same persisted fields and never recomputes a token or count. The +persisted ACK deadline is an internal lifecycle fence and is not added to the +transport-v2 result. Recover deliberately has no `accepted_ordinals` field +because it creates an executable suffix rather than a preparing upload set. + +Every prepare error has exactly `schema_version`, `ok`, `status`, `host_id`, +`name`, and `message`; `schema_version=1`, `ok=false`, and `message` is exactly +the same fixed enum token as `status`. The closed status/message set is +`invalid_params`, `invalid_ref`, `stale_ref`, `store_unavailable`, +`revision_not_found`, `stale_revision`, `content_unavailable`, `plan_not_found`, +`plan_conflict`, `part_conflict`, `plan_incomplete`, `plan_not_failed`, +`not_recoverable`, `request_conflict`, and `ack_deadline_expired`. Errors expose +no token, ordinal, count, deadline, payload, route, provider fact, or private +coordinate. + +Generation-one begin identity is exact: + +```text +(source final-ready outer key, final_identity, content_revision, + route_generation, presentation_version, part_count, plan_generation=1) +``` + +Begin validates the live `source_ref`, inserts one opaque +`twplan1.*`, and creates `part_count` staged `final_part` placeholders in one +transaction. Repeating the same begin returns the same token, plan state, part +count, and sorted exact `accepted_ordinals`. A differing version or count is +`plan_conflict`. A repeated begin may bind the same staged plan to a newly +leased attempt of the same source outer key, never to another root, revision, +or route. + +Part retains its current request shape. The first exact span set fills its +ordinal. An identical retry succeeds and returns the same accepted ordinals; a +different set is `part_conflict`. Although part carries no source ref, Tendwire +internally requires the plan to remain bound to the source's current live +attempt. Staged payloads become immutable when commit makes them executable. + +The first commit requires the current live `source_ref`, all ordinals, exact +full coverage, and a still-current revision. It atomically freezes children, +creates retire nodes, installs the predecessor DAG, moves source outbox and +attempt from `leased` to `awaiting_ack`, initializes the persisted ACK deadline, +and exposes only the head. A repeated commit after success returns the persisted +committed token/state/generation/part/job counts and accepted ordinals +even when the supplied old source ref is no longer live. If the first commit +did not happen, the plan remains preparing and a current source ref is required. + +Herdres renews the source lease immediately after root validation, after every +bounded content-page batch, before every part upload, and immediately before +commit. A failed or uncertain renew stops preparation. The same renew may be +retried while the ref can still be live; otherwise Herdres waits for expiry, +repolls the same outer key, repeats begin, and resumes missing ordinals. + +Consequently: + +- Lost begin response: repeat begin and rediscover token/accepted ordinals. +- Lost part response: repeat the identical part. +- Crash or source expiry before commit: the source requeues, staged rows remain, + repoll returns the same outer key with a new ref, and begin rediscovers them. +- Lost commit response: committed child keys become pollable; repeating commit + also returns the identical committed result. +- Failed lineage recovery: use only `recover` while the committed root remains + awaiting ACK before its deadline. Source-less `begin` does not exist, and an + expired committed root is terminal and `not_retryable`. +- Restart requires no Herdres plan token, pending-plan, accepted-ordinal, or + recovery ledger. Provider receipts keyed by outer key remain authoritative. + +## Recovery lineage + +Recover accepts its unchanged `failed_plan_token` and `request_id` request. +Tendwire requires one linear logical sequence: a contiguous acknowledged +prefix followed by a failed nonleased suffix, plus the exact source root and +current delivery still in `awaiting_ack` with `ack_deadline_at > now`. A gap, +live child lease, unexpired retry/defer, absent source, or noncontiguous prefix +returns `not_recoverable`; an expired root returns `ack_deadline_expired`. A `provider_uncertain` +dead-letter is recoverable only through explicit operator action acknowledging +the duplicate-send risk; it is never selected by automatic recovery. + +In one transaction Tendwire: + +1. stores a unique digest of the bounded public request ID; +2. returns the same result for an identical replay; +3. allocates the next generation and a new plan token; +4. retains old delivered prefix rows and their original outer keys unchanged; +5. creates fresh-key copies only for the suffix, including retire nodes; +6. points the first new node to the last old acknowledged key, or leaves its + executable predecessor null when the prefix is empty; +7. records the failed plan token and replaced old suffix row on each new node; +8. marks failed non-retire suffix rows `superseded`, but keeps every possibly + accepted mandatory retire as a terminal dead-letter audit row linked to its + fresh replacement rather than claiming it became unnecessary; and +9. replaces the live root delivery's ACK deadline once with + `recover_now + connector_ack_ttl_seconds`; and +10. defines root completion as old delivered prefix plus new suffix. + +Recovery may chain when a replacement suffix later fails. It never revives an +old ref or attempt and never erases provider-acceptance ambiguity. An explicit +operator recovery can duplicate an operation whose provider acceptance was not +durably bound. The retained old `provider_uncertain` audit item and its attempt +count remain inspectable even when the recovery suffix later succeeds. + +The prefix-empty head is ordered by its fresh partition sequence after the old +failed tail has atomically become nonblocking; it never points to the retained +`awaiting_ack` root. Making the root its predecessor would deadlock because the +root cannot become delivered until that head and suffix are delivered. + +## Retire DAG and provider-message alias safety + +A retire payload addresses a logical outer key, never a provider message ID. +Its predecessor is the accepted replacement key. Tendwire does not make retire +eligible until that predecessor is `delivered`. The sole exception is the +standalone `decision_resolved` retire defined above: its predecessor/target may +be a terminal `delivered`, `superseded`, or `dead_letter` decision after a +lease, because provider acceptance may have preceded the terminal Tendwire +state. + +Herdres provider/presentation ownership has three durable logical states: + +```text +owns(provider_message_id) +alias(provider_message_id, current_owner_key) +retired +``` + +When a replacement edits or reuses an existing provider message, Herdres first +binds the replacement key as owner, records an ownership alias from the old +logical slot to the replacement key, durably checkpoints both changes, and only +then ACKs the replacement. The old outer-key `job_binding` itself is immutable: +its accepted operation, payload fingerprint, provider coordinate, and outcome +are never rewritten or rebound. Alias/current-owner state belongs only to the +provider-message and presentation-slot records and is never an alternate replay +lookup. + +On retire: + +- `owns` and still current owner: delete, durably tombstone, then ACK; +- `alias` whose current owner is a different delivered replacement: perform a + logical retire only and never delete the reused message; +- already retired, or absent with a durable tombstone: idempotent no-op ACK; +- absent with no immutable job, alias, or tombstone for a possibly accepted + mandatory target: fail `provider_binding_unknown`, never ACK; and +- uncertain provider response: fail closed and never claim deletion. + +Thus a delayed retire for an old working/final key cannot delete a message that +has since been edited into the accepted replacement. + +Tendwire cannot observe whether Herdres retained, aliased, lost, or deleted a +provider binding. Its only evidence is connector settlement and attempt state. +It therefore never declares a retire unnecessary because of assumed Herdres +state and never shortens a root, route, content, or retry horizon based on such +an assumption. + +## Producer transactions and root invariants + +- ACP working ingestion atomically appends the authoritative event, applies the + turn/revision/delta projection, allocates a route sequence, supersedes older + unleased working work, and enqueues the new immutable working row. +- Completion atomically fixes the current content revision, prevents later + working enqueue for that turn/revision, supersedes outstanding working work, + and enqueues final-ready after it in the same partition. +- Prepare commit atomically creates the active child/retire DAG and retains the + source root in `awaiting_ack`. +- Pending-decision projection and decision enqueue are atomic. Resolution and + retire enqueue are atomic. +- A newer final supersedes an uncommitted old root immediately. For a committed + old root, its unaccepted tail is terminal-after-lease/superseded, but its + accepted prefix remains retained until the new replacement/retire DAG is + acknowledged. No delete-before-replace gap is possible. +- A root becomes `delivered` only when every node in its effective original or + recovered lineage is delivered. It becomes `superseded` or `dead_letter` + atomically with the relevant remaining lineage. + +Lifecycle snapshots reconcile workers/routes/topics but never create or replay +presentation content. Herdres has no `turn.delta`, `pending.list`, transcript, +snapshot-content, or other parallel authority for final delivery. + +## Production import and caller rewrite + +Every production `store.sqlite` import is rewritten directly; there is no +`store/__init__.py` façade. + +| Old symbol | New module or disposition | Current production callers after rewrite | +| --- | --- | --- | +| `init_store` | `store.schema` | daemon `_default_init_store`, `DaemonHooks`; smoke fixture setup | +| `record_agent_event` | `store.events` | ACP coordinator console bridge/cursor/outcome | +| `list_agent_events` | `store.events` | ACP coordinator console bridge/cursor/input loads | +| `tail_event_metadata` | **deleted** | only live production caller is deleted direct CLI `cmd_store`; no after-state caller | +| `AppendBoundAgentEventResult` | `core.agent_events` | ACP ingestion event result; never moved into a store module | +| `AppendProjectedAgentEventResult` | `store.turns` | ACP ingestion type/API | +| `TurnRefreshApplyResult` | `store.turns` | ACP ingestion result | +| `append_agent_event_and_apply_turn_for_binding` | `store.turns` | ACP ingestor default persistence path | +| `apply_turn_refresh` | `store.turns` | `scripts/sqlite_sidecar_race_benchmark.py`, `tests/store_helpers.py`, and focused turn/delta/submission/performance tests; no current `src` caller imports it directly, while ACP runtime uses the combined append/apply API | +| `turns_payload_from_store` | `store.turns` | daemon `get_turns` | +| `turn_delta_payload_from_store` | `store.turns` | daemon `get_turn_delta` | +| `get_turn_content` | `store.turns` | daemon `get_turn_content` | +| `SnapshotObservationContext` | `store.projection` | ACP discovery; daemon/tests | +| `save_snapshot` | `store.projection` | ACP coordinator discovery; returns the persisted route-enriched snapshot with recomputed worker/content fingerprints | +| `latest_snapshot` | `store.projection` | command submission; ACP coordinator; daemon start/snapshot; smoke | +| `attention_payload_from_store` | `store.projection` | daemon `get_attention` | +| `upsert_worker_bindings` | `store.projection` | ACP coordinator runtime/binding creation | +| `list_worker_bindings` | `store.projection` | ACP coordinator/runtime/permissions; smoke | +| `expire_worker_bindings` | `store.projection` | ACP coordinator/runtime | +| `expire_stale_worker_bindings` | `store.projection` | tests and projection API; no compatibility import | +| `backend_pending_health` | `store.projection` | daemon pending-ingestion health | +| `apply_backend_pending_observation` | `store.pending` | ACP permission broker | +| `pending_payload_from_store` | `store.pending` | ACP coordinator and daemon `get_pending` | +| `claim_backend_pending_decision` | `store.pending` | command submission decision validation/claim | +| `start_backend_pending_decision_send` | `store.pending` | command submission answer send | +| `abandon_backend_pending_choice_claim` | `store.pending` | command submission safe pre-send abandon | +| `backend_pending_choice_terminal_effect` | `store.pending` | command submission uncertain/terminal decision effect | +| `reserve_command_request` | `store.receipts` | command submission reservation | +| `reserve_terminal_command_replay` | `store.receipts` | command submission replay | +| `get_command_request` | `store.receipts` | command submission recovery/replay paths | +| `command_reservation_is_live` | `store.receipts` | command receipt authority | +| `abandon_command_request_reservation` | `store.receipts` | safe pre-transport abandon/retry | +| `mark_command_send_started` | `store.receipts` | command submission send fence | +| `finish_command_request` | `store.receipts` | command completion | +| `finish_queued_command_request` | `store.receipts` | queued receipt completion | +| `finish_unverified_queued_command_request` | `store.receipts` | uncertain queued completion | +| `recover_unresolved_command_send` | `store.receipts` | receipt authority recovery | +| `linked_turn_for_submission` | `store.receipts` | send/link/replay correlation | +| `settle_submission_link_for_request` | `store.receipts` | receipt and negotiated submission linkage | +| `envelope_to_receipt_json` | `store.receipts` | canonical receipt persistence | +| `poll_connector_outbox` | `store.outbox` | `ConnectorOutboxAPI.poll` | +| `prepare_connector_plan_begin` | `store.outbox` | `ConnectorOutboxAPI.prepare(begin)` | +| `prepare_connector_plan_part` | `store.outbox` | `ConnectorOutboxAPI.prepare(part)` | +| `prepare_connector_plan_commit` | `store.outbox` | `ConnectorOutboxAPI.prepare(commit)` | +| `prepare_connector_plan_recover` | `store.outbox` | `ConnectorOutboxAPI.prepare(recover)` | +| `ack_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.ack` | +| `fail_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.fail` | +| `defer_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.defer` | +| `renew_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.renew` | +| `release_connector_delivery` | `store.outbox` | `ConnectorOutboxAPI.release` | +| `reclaim_expired_connector_leases` | `store.outbox` | connector API reclaim; daemon periodic tick | +| `connector_reclaim_due` | `store.outbox` | daemon periodic connector tick | +| `inspect_connector_outbox` | `store.outbox` | `ConnectorOutboxAPI.inspect` | +| `retry_final_ready_delivery` | `store.outbox.retry_connector_dead_letter` | `ConnectorOutboxAPI.retry`; the old final-only name is deleted and the replacement handles exact retryable final roots, standalone decisions, and standalone retires | +| `store_status` | `store.db` | daemon health only | +| `SnapshotRetentionPolicy` | `store.retention.RetentionPolicy` | rewritten daemon retention cycle | +| `cleanup_*_retention` | private `store.retention` helpers | `run_retention_cycle` only | +| `CompactionOptions`, `compact_store`, `run_store_maintenance`, `maybe_run_automatic_store_maintenance`, `compact_turn_change_journal`, `exhaust_connector_retries` | **deleted as public APIs** | direct CLI/daemon maintenance paths removed; outbox exhaustion remains a private transition helper | + +The CLI rewrite removes the `store` parser subtree, `cmd_store`, direct status, +event-tail, maintenance, and compact options, and all five direct SQLite +imports. Operator reads use the daemon socket. There is no compaction RPC. + +Daemon `_after_snapshot_saved` removes maintenance-state and turn-journal +compaction calls and invokes one bounded `retention.run_retention_cycle` on an +in-memory cadence. Connector periodic reclaim remains independent. Health uses +`db.store_status`; it does not mutate or repair the store. + +`scripts/store_benchmark.py` is rewritten to import the owning schema, +projection, outbox, and retention APIs and to seed only the fresh schema; its +old direct assumptions about maintenance tables and `store.sqlite` are deleted. +`scripts/sqlite_sidecar_race_benchmark.py` is rewritten to import +`store.schema`/`store.db`, patch the new pinned-path connection authority rather +than a deleted module, and retain installed-wheel sidecar-race, inode, mode, +integrity, and descriptor evidence. Neither script is allowed a compatibility +import or private direct-SQL mutation that bypasses the invariant it claims to +benchmark. + +## Test-file disposition + +| File | Disposition | +| --- | --- | +| `tests/store_helpers.py` | rewrite direct imports to owning modules | +| `tests/test_acp_atomic_ingestion.py` | retain atomic journal/projection proof; rewrite imports | +| `tests/test_acp_coordinator.py` | rewrite imports; add route-generation publication/rotation/collision cases | +| `tests/test_acp_ingestion.py` | rewrite result-type and event/turn imports | +| `tests/test_acp_permissions.py` | replace module-wide sqlite monkeypatches with exact pending/projection patches | +| `tests/test_acp_runtime.py` | rewrite projection/event imports | +| `tests/test_agent_events.py` | rewrite to `store.events`; preserve authoritative dedupe/conflict, visibility, paging, and retention proofs | +| `tests/test_daemon.py` | rewrite imports/patch paths; delete automatic-maintenance-state assertions; add bounded retention and daemon-only health | +| `tests/test_daemon_acp.py` | rewrite schema/projection imports | +| `tests/test_connector_daemon_cli.py` | retain real socket RPC contract; delete direct CLI store fallback cases | +| `tests/test_connector_outbox.py` | replace with the five-kind status/CAS matrix, FIFO/DAG, lease, response-loss, recovery, and alias-retire tests | +| `tests/test_delivery_retention.py` | rewrite for kind-aware live/root/effective-lineage protection | +| `tests/test_delivery_retention_hardening.py` | delete maintenance-state tests; retain deadline/dead-letter/cutoff/security tests | +| `tests/test_delivery_retention_projection.py` | retain atomic projection/outbox proof; add route correlation | +| `tests/test_delivery_retention_recovery.py` | rewrite for retained old prefix plus fresh suffix lineage | +| `tests/test_local_state_permissions.py` | rewrite `init_store` import; retain secure-open proof | +| `tests/test_public_content_safety.py` | rewrite imports; add exact five-kind forbidden-key/value scans | +| `tests/test_release_readiness.py` | delete VACUUM/backup/compaction ceremony; retain fresh cutover, integrity, permissions, and retention proof | +| `tests/test_snapshot_sanitize_performance.py` | rewrite projection imports and preserve sanitizer/transaction performance bounds without a sqlite compatibility path | +| `tests/test_store.py` | split by owning module; delete migration-chain/generic CRUD/direct-maintenance internals | +| `tests/test_turn_delta.py` | rewrite turn imports; add route-generation publication/correlation | +| `tests/test_turn_submissions.py` | rewrite schema/turn imports; preserve state machine | +| `tests/test_worker_stable_key.py` | add stable/reused/rotated/collision route-generation tests | + +The paired Herdres H7/H6 table is authoritative here and in +`docs/wave4-presenter-state-design.md`; every baseline file has one matching +disposition: + +| Herdres baseline test file | Final disposition | +| --- | --- | +| `conftest.py` | retain; edit only bounded shared fixtures, never protocol or production logic | +| `test_accounts.py` | delete with pinned boards | +| `test_collapse_previous.py` | rewrite for immutable target keys, flattened one-hop aliases, and reuse-safe retire | +| `test_command_ingress_idempotency.py` | rewrite/retain H8 HMAC request identity, key security, and replay vectors | +| `test_gateway_cleanup.py` | delete old gateway cleanup; move known-ID topic cases to `test_topics.py` | +| `test_ingress_lanes.py` | delete with lanes; surviving FIFO/crash cases move to `test_ingress.py` | +| `test_ingress_requests.py` | delete with JSON request workers; surviving receipt/quarantine cases move to `test_ingress.py` | +| `test_lossless_turn_rendering.py` | rewrite for contract-v3 deterministic exact spans and no stored content | +| `test_model_in_pins.py` | delete with pinned boards | +| `test_offlock_delivery.py` | rewrite for shared guard, lock order, route revalidation, and uncertainty classes | +| `test_outbound_latency.py` | rewrite for one poll/presenter and bounded lease/guard budgets | +| `test_pane_topic_binding_integrity.py` | rewrite for typed lifecycle, current slots, aliases, and immutable provider facts | +| `test_pending_inputs.py` | delete pending-list/bare-number local presentation path | +| `test_release_readiness.py` | rewrite for exact scope, static gates, 8,920 SLOC target, security, and paired cutover | +| `test_remote_decisions.py` | rewrite for decision controls, composite phases, shared guard, and resolution retire | +| `test_restart_rekey_continuity.py` | delete old rekey machinery; route continuity moves to state/presenter tests | +| `test_rich_delivery.py` | rewrite for deterministic one-request materialization and no fallback mutation | +| `test_source_only.py` | rewrite surviving connector receipt/crash cases; delete local-source/source-less cases | +| `test_source_status_placeholders.py` | delete local source/status placeholder presentation | +| `test_speak_back.py` | delete with voice/TTS | +| `test_speech.py` | delete with voice/STT/TTS | +| `test_stable_generation_delivery.py` | rewrite for exact route token, enriched fingerprint, and stale-route fence | +| `test_stable_worker_key.py` | retain and extend stable identity, route reuse/rotation, and collision cases | +| `test_table_rendering.py` | retain supported renderer; remove pin/voice coupling if present | +| `test_telegram_backpressure.py` | rewrite for one guarded mutation, exact 429 defer, and no tight loop | +| `test_tendwire_client.py` | rewrite for five kinds, exact prepare/content responses, renew/release, and socket validation | +| `test_tendwire_socket_pairing.py` | rewrite for real paired five-kind/ingress/receipt/restart integration | +| `test_topic_lifecycle_cleanup.py` | rewrite as bounded known-ID cases in `test_topics.py` | +| `test_topic_names.py` | retain/rewrite for minimal supported topic naming only | +| `test_turn_delta_sync.py` | delete; final presentation has no turn-delta source | +| `test_turn_final_delivery.py` | replace with source-bound root, provider kinds, recovery, retry, and retire integration | +| `test_worker_topic_dedup.py` | rewrite for physical-owner serialization across route generations | + +The final stack adds exactly `test_ingress.py`, `test_state.py`, +`test_presenter.py`, `test_presentation.py`, and `test_topics.py`. No unlisted +Herdres test file may be added, deleted, or used as a compatibility dump. +Rewritten files may share fixtures through `tests/conftest.py`; production logic +and copied protocol validators are forbidden there. + +Tests assert public behavior and database constraints, not private helper names or +copied transition logic. Required real-daemon integration covers working to +final ordering, paged multipart prepare, lost begin/part/commit responses, +provider acceptance plus lost ACK, source lease expiry, recovered prefix/suffix, +supersession and message reuse, 429 defer, decision buttons/resolution, privacy +rejection, stale refs, dead-letter/retry generations, and concurrent logical +partitions sharing one physical provider route. + +A static AST/import gate scans `src`, `scripts`, and tests and fails on any +production import, attribute patch, string import, or compatibility re-export +of `tendwire.store.sqlite`/`store.sqlite`. It also proves each deleted public +maintenance symbol has no production caller. + +## Precise implementation file scope + +Tendwire T6 deletes/adds/edits only: + +```text +delete src/tendwire/store/sqlite.py +add src/tendwire/store/schema.py +add src/tendwire/store/db.py +add src/tendwire/store/events.py +add src/tendwire/store/turns.py +add src/tendwire/store/projection.py +add src/tendwire/store/pending.py +add src/tendwire/store/receipts.py +add src/tendwire/store/outbox.py +add src/tendwire/store/retention.py +edit src/tendwire/store/__init__.py # package marker only; no re-exports +edit src/tendwire/cli.py +edit src/tendwire/config.py # bounded ACK TTL and retention floors +edit src/tendwire/daemon.py +edit src/tendwire/daemon_api.py +edit src/tendwire/command_submission.py +edit src/tendwire/worker_identity.py +edit src/tendwire/core/models.py +edit src/tendwire/backends/acp_coordinator.py +edit src/tendwire/backends/acp_ingestion.py +edit src/tendwire/backends/acp_permissions.py +edit src/tendwire/backends/acp_runtime.py +edit src/tendwire/connectors/outbox.py +edit scripts/herdr_smoke.py +edit scripts/store_benchmark.py +edit scripts/sqlite_sidecar_race_benchmark.py +edit docs/connector-rpc-contract.md +edit docs/wave4-store-design.md +edit only the Tendwire tests listed in the disposition table +``` + +The reconciled Herdres scope is split at one safe merge boundary. + +H8 may be implemented, reviewed, and deployed independently against the current +Tendwire/current presenter pair. Its exact file disposition is: + +```text +add herdres_connector/ingress.py +add herdres_connector/ingress_queue.py +edit herdres_gateway.py # retained small executable wrapper +edit herdres.py # remove command-child ingress only +edit herdres_connector/state.py # frozen typed H8 seam +edit herdres_connector/decisions.py +edit herdres_connector/doctor.py +edit herdres_connector/config.py +edit herdres_connector/source_sync.py # remove JSON-ingress/receipt-working joins +edit herdres_connector/ingress_identity.py # trim retained key/identity code to 80--100 SLOC +edit herdres_connector/managed_bots.py # ephemeral receiver/policy typing only +edit herdres_connector/tendwire_client.py +keep herdres_connector/telegram_delivery.py # H8 uses existing bounded methods +delete herdres_connector/ingress_lanes.py +delete herdres_connector/ingress_requests.py +edit/delete only the H8 tests and docs declared by the approved H8 design +``` + +H8's frozen route result uses `route_generation: str | None`, and its ingress +queue column is nullable `TEXT`, never integer. Current routes publish null, +and stored null is immutable for that request. Contract-v3 H7 routes require a +non-null exact `twroute1.*`; open pre-T6 null-route requests are drained or +explicitly discarded at paired cutover. H8 does not depend on the five-kind +outbox. The old presenter continues the current observational working/final +behavior until the paired cutover. + +After H8 is deployed, corrected T6 and the H7/H6 presenter stack form one +inseparable compatibility and deployment unit. H7/H6's exact final disposition +is: + +```text +add herdres_connector/presenter.py +add herdres_connector/presentation.py +add herdres_connector/topics.py +edit herdres.py +retain herdres_gateway.py # small H8 compatibility wrapper +edit herdres_connector/__init__.py +edit herdres_connector/config.py +edit herdres_connector/decisions.py +edit herdres_connector/doctor.py +edit herdres_connector/ingress.py +edit herdres_connector/ingress_queue.py +keep herdres_connector/ingress_identity.py +edit herdres_connector/managed_bots.py +edit herdres_connector/rendering.py +edit herdres_connector/rich_delivery.py +edit herdres_connector/safe.py +edit herdres_connector/state.py +edit herdres_connector/telegram_delivery.py +edit herdres_connector/tendwire_client.py +delete herdres_connector/source_sync.py +delete herdres_connector/accounts.py # pinned-board deletion +delete herdres_connector/speech.py # voice/STT/TTS deletion +edit README.md +edit RELEASE.md +edit SECURITY.md +edit docs/connector-rpc-contract.md +edit docs/remote-decisions.md +edit docs/wave4-ingress-dependency-design.md +edit docs/wave4-presenter-state-design.md +edit docs/wave4-store-design.md +edit/delete/add only the paired Herdres tests in the exact table above +``` + +H7 state keeps immutable job bindings keyed only by Tendwire outer key and +provider/presentation ownership aliases separately; it contains no local plan, +accepted-ordinal, pending-retire, retry, or recovery ledger. Current Herdres +strictly accepts final-ready schema v2 and plan-job schema v1, so T6 cannot ship +alone. H7/H6 likewise cannot ship against current Tendwire. Only H8 has an +independent deployment boundary; the next deployable boundary is the complete +T6+H7/H6 paired contract-v3 unit. + +## Transactions, security, privacy, and retention + +Every mutation uses `BEGIN IMMEDIATE`. Frozen list/content/delta pages use a +read transaction. Event append, binding/route fence, turn and content revision, +delta sequence, command/submission effect, pending overlay, and outbox enqueue +are atomic where the producer invariant requires them. + +Connections are short-lived and use WAL, foreign keys, `FULL` synchronous, +bounded busy timeout, and `trusted_schema=OFF`. The configured database must be +an absolute leaf beneath the exact configured owner-private data directory; it +cannot be `:memory:`, a URI, a relative path, contain `..`, or select another +authority root. Code walks and pins every parent with directory fds and +`O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC`, rejects symlinks, wrong UID, group/world +writable directories, and parent identity changes, and opens/creates the leaf +relative to the pinned parent with restrictive umask and +`O_NOFOLLOW|O_CLOEXEC`, mode `0600`, regular-file, single-link, UID, and +device/inode checks. + +SQLite connects through the pinned `/proc/self/fd//` path while +the schema/sidecar authority is held. The implementation verifies the database +identity before and after connect and transaction use; validates or securely +creates `-wal` and `-shm` only as same-parent, same-UID, regular, single-link +0600 files; rejects unexpected journal/sidecar types or inode swaps; and pins +and revalidates the family across schema mutation, checkpoint, and close. A +separately pinned owner-only lock serializes schema/sidecar creation without +becoming a general authority registry. No fallback connect occurs after any +identity or permission failure. + +The connector-edge scanner rejects raw pane/session/terminal identities, +backend targets, private fingerprints, provider chat/topic/message IDs, +credentials/tokens, socket and absolute paths, command/argv/environment data, +ACP option IDs, stdout/stderr, and exception prose, including forbidden values +embedded in otherwise allowed strings. Only enumerated public errors and reason +codes cross the socket. Logs and cutover reports contain aggregate counts and +schema versions only, never IDs, refs, keys, payloads, or provider coordinates. + +Retention performs bounded child-first cutoff deletes. It never deletes a live +lease, staged plan, retained awaiting-ACK root, current/recovered lineage, +dead-letter inside its inspection cutoff, command replay floor, current content +revision, page referenced by work, active route generation, or maximum sequence +sentinel/partition allocator. A delivery with non-null `ack_deadline_at` is live +regardless of wall-clock age; after terminal settlement the deadline is cleared +and deletion still waits for the 30-day targetable floor. Targetable outbox +correlations use 30 days, route/content/key evidence uses 45 days, and the +paired H7 provider-fact floor is 60 days, with references overriding all three. +It retains only bounded terminal attempt detail plus aggregate prior attempt +counts, then runs `PRAGMA wal_checkpoint(TRUNCATE)`. It performs no +VACUUM, backup swap, direct CLI repair, or automatic schema migration. + +## Cutover and rollback + +H8 has the only independent deployment boundary. Its release stops the old +gateway writer, drains or explicitly discards both old ingress durability +stores, archives the old `inbound_spool.db` family and matching schema-2 +Herdres state, and preserves the one active H8 request-ID HMAC key byte-for-byte +at its configured owner-only path. It starts exactly one H8 writer on the fresh queue +with that key and proves one daemon-socket receipt. Snapshot rollback is +straightforward only before H8 accepts new work. After acceptance it requires the H8 design's +explicit cursor/request inventory and operator drain/discard decision; the old +snapshot is not described as lossless merely because it can be restored. + +T6+H7/H6 is the next single deployment boundary. Tendwire schema mismatch and +H7 state mismatch never migrate silently. Before that paired upgrade operators +must choose: + +1. drain old command reservations, claims, interactions, leases, + awaiting-ACK roots, presentation plans, and outbox work to zero; or +2. stop every old writer and explicitly acknowledge discarding all in-flight + state before fresh recreation. + +The quiescent cutover snapshot is one matched set: Tendwire database/WAL/SHM, +the active Tendwire installation key/marker/sentinel, H8 ingress +database/WAL/SHM, the same active H8 request-ID HMAC key used +before cutover, and old Herdres state/provider receipts. The paired release +must not rotate, regenerate, or silently replace the Tendwire key family or the +H8 request-ID key. Old/new +versions, discarded table names, and aggregate active counts are reported +without identities or payloads. Old and new presenters never run against the +same topics. + +Fresh H7 state starts behind a reconciliation barrier. While H8 submission and +all Telegram mutators remain stopped, one lifecycle writer reconciles every +current worker/private route, Tendwire durably publishes each non-null +`twroute1.*`, `save_snapshot` returns the enriched fingerprints, and H7 resolves +the same stable owner/generation pairs through the frozen seam. The barrier is +released only after one transactionally consistent inventory proves no live +route is missing/ambiguous and every retained H8 request either has the exact +non-null generation/fingerprint or was drained or explicitly discarded. Only +then may H8 and the contract-v3 presenter resume. Failure leaves the barrier +closed and authorizes no command or provider mutation. + +Rollback is mechanically safe before recreation and before any new provider +mutation. After the paired release has sent, edited, deleted, or created a +provider object, restoring old local snapshots alone is unsafe: it forgets +new provider facts and can replay or delete the wrong message. Rollback then +requires stopping all new writers, draining/ACKing or explicitly retiring every +new known provider mutation with the new release, resolving/reporting ambiguous +claims, and only then restoring the complete matched old snapshot. If that +drain/retire proof is unavailable, rollback is blocked rather than described as +safe. Installation-key or request-key replacement is a continuity break, not +recovery. + +No implementation, merge, deployment, or service restart is authorized by +this design. Production implementation remains blocked until the paired +Herdres payload, outer-key binding, no-plan-ledger recovery, physical-route +locking, and alias-retire tests pass against this exact contract. diff --git a/pyproject.toml b/pyproject.toml index 0950545..b959794 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,7 +21,9 @@ classifiers = [ "Programming Language :: Python :: 3.13", "Topic :: Software Development :: Libraries :: Application Frameworks", ] -dependencies = [] +dependencies = [ + "agent-client-protocol>=0.11,<0.12", +] [project.scripts] tendwire = "tendwire.cli:main" diff --git a/scripts/codex_session_reader_benchmark.py b/scripts/codex_session_reader_benchmark.py deleted file mode 100755 index 0b5dc18..0000000 --- a/scripts/codex_session_reader_benchmark.py +++ /dev/null @@ -1,464 +0,0 @@ -#!/usr/bin/env python3 -"""Deterministic synthetic benchmark for the private Codex session reader. - -Run from a source checkout with ``PYTHONPATH=src``. The benchmark creates a -private memory-backed 20,000-file Codex fixture, prints one compact aggregate -JSON object, and never reports generated session identities or paths. Timing -ceilings are broad documented-host evidence gates; bounded work is the stable -contract. -""" - -from __future__ import annotations - -import argparse -import json -import os -import platform -import re -import sys -import tempfile -from collections.abc import Mapping -from pathlib import Path -from time import perf_counter_ns -from typing import Any -from uuid import UUID - -from tendwire.backends import herdr_turns - -REPORT_SCHEMA_VERSION = 1 -FIXTURE_FILE_COUNT = 20_000 -FIXTURE_SPARSE_BYTES = 20 * 1024 * 1024 -FIXTURE_DATE = "2026-07-03" -TARGET_ORDINAL = 10_000 -COLD_LOOKUP_BUDGET_NS = 30_000_000_000 -WARM_LOOKUP_BUDGET_NS = 1_000_000_000 -COLD_PARSE_BUDGET_NS = 2_000_000_000 -INCREMENTAL_POLL_BUDGET_NS = 1_000_000_000 -UNCHANGED_POLL_BUDGET_NS = 1_000_000_000 -_UUID_PATTERN = re.compile( - r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", - re.ASCII, -) - - -class _ArgumentError(Exception): - pass - - -class _Parser(argparse.ArgumentParser): - def error(self, message: str) -> None: - raise _ArgumentError(message) - - -def _canonical_json(value: Any) -> str: - return json.dumps( - value, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - ) - - -def _event(kind: str, turn_id: str, **extra: Any) -> dict[str, Any]: - return { - "type": "event_msg", - "payload": {"type": kind, "turn_id": turn_id, **extra}, - } - - -def _message( - turn_id: str, - role: str, - text: str, - *, - phase: str | None = None, -) -> dict[str, Any]: - payload: dict[str, Any] = { - "type": "message", - "role": role, - "content": [{"type": "output_text", "text": text}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - } - if phase is not None: - payload["phase"] = phase - return {"type": "response_item", "payload": payload} - - -def _jsonl(*records: Mapping[str, Any]) -> bytes: - return b"".join( - _canonical_json(record).encode("utf-8") + b"\n" for record in records - ) - - -def _rollout_name(session_id: str) -> str: - return f"rollout-{FIXTURE_DATE}T00-00-00-{session_id}.jsonl" - - -def _create_private_file(path: Path) -> None: - descriptor = os.open( - path, - os.O_CREAT | os.O_EXCL | os.O_WRONLY | getattr(os, "O_CLOEXEC", 0), - 0o600, - ) - os.close(descriptor) - - -def _create_fixture(home: Path) -> tuple[Path, str, str, str, bytes, bytes]: - sessions = home / "sessions" - year = sessions / "2026" - month = year / "07" - day = month / "03" - for directory in (home, sessions, year, month, day): - directory.mkdir(mode=0o700) - - target_id = str(UUID(int=TARGET_ORDINAL)) - for ordinal in range(1, FIXTURE_FILE_COUNT + 1): - session_id = str(UUID(int=ordinal)) - _create_private_file(day / _rollout_name(session_id)) - - turn_id = "synthetic-benchmark-turn" - user_text = "synthetic benchmark prompt" - stream_text = "synthetic benchmark incremental output" - tail = _jsonl( - _event("task_started", turn_id), - _message(turn_id, "user", user_text), - ) - append = _jsonl( - _message(turn_id, "assistant", stream_text, phase="commentary") - ) - target = day / _rollout_name(target_id) - with target.open("r+b", buffering=0) as handle: - handle.seek(FIXTURE_SPARSE_BYTES) - handle.write(b"\n") - handle.write(tail) - return target, target_id, turn_id, user_text, stream_text, append - - -def _reset_codex_state() -> None: - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - herdr_turns._CODEX_SESSION_CACHE_LIVE_KEYS = None - herdr_turns._CODEX_SESSION_CACHE_BINDING_GENERATIONS = {} - herdr_turns._CODEX_SESSION_CACHE_BINDING_FINGERPRINTS = {} - - -def _timed(call: Any) -> tuple[Any, int]: - started = perf_counter_ns() - value = call() - return value, perf_counter_ns() - started - - -def _contains_private_value(value: Any, forbidden: tuple[str, ...]) -> bool: - if isinstance(value, Mapping): - return any( - _contains_private_value(key, forbidden) - or _contains_private_value(item, forbidden) - for key, item in value.items() - ) - if isinstance(value, (list, tuple)): - return any(_contains_private_value(item, forbidden) for item in value) - if not isinstance(value, str): - return False - return bool(_UUID_PATTERN.search(value)) or any( - private and private in value for private in forbidden - ) - - -def _command_text() -> str: - return "PYTHONPATH=src python3 scripts/codex_session_reader_benchmark.py --json" - - -def _benchmark() -> dict[str, Any]: - benchmark_started = perf_counter_ns() - load_average = tuple(round(value, 2) for value in os.getloadavg()) - temporary_path: Path | None = None - index_observations: list[int] = [] - read_observations: list[int] = [] - prior_home = os.environ.get("CODEX_HOME") - prior_index_observer = herdr_turns._CODEX_INDEX_BUILD_OBSERVER - prior_read_observer = herdr_turns._CODEX_ISOLATED_READ_OBSERVER - report: dict[str, Any] | None = None - forbidden: tuple[str, ...] = () - - try: - with tempfile.TemporaryDirectory( - prefix="tendwire-codex-reader-benchmark-", - dir="/dev/shm", - ) as raw_root: - temporary_path = Path(raw_root) - root_private = temporary_path.stat().st_mode & 0o777 == 0o700 - home = temporary_path / "codex-home" - ( - target, - target_id, - turn_id, - user_text, - stream_text, - append, - ) = _create_fixture(home) - forbidden = ( - raw_root, - os.fspath(home), - os.fspath(target), - target_id, - turn_id, - user_text, - stream_text, - target.name, - ) - logical_file_bytes = target.stat().st_size - - os.environ["CODEX_HOME"] = os.fspath(home) - _reset_codex_state() - herdr_turns._CODEX_INDEX_BUILD_OBSERVER = index_observations.append - herdr_turns._CODEX_ISOLATED_READ_OBSERVER = read_observations.append - - wildcard_result, wildcard_ns = _timed( - lambda: herdr_turns._find_codex_session_file("*") - ) - builds_after_wildcard = len(index_observations) - - cold_result, cold_lookup_ns = _timed( - lambda: herdr_turns._find_codex_session_file(target_id) - ) - builds_after_cold_lookup = len(index_observations) - warm_result, warm_lookup_ns = _timed( - lambda: herdr_turns._find_codex_session_file(target_id) - ) - builds_after_warm_lookup = len(index_observations) - - with herdr_turns._CODEX_PATH_CACHE_LOCK: - index = herdr_turns._CODEX_INDEX_GENERATION - if index is None: - raise RuntimeError("index_generation_missing") - indexed_sessions = len(index.entries) - retained_index_bytes = index.retained_bytes - index_overflowed = index.overflowed - generation_visited = index.visited - - cold_content, cold_parse_ns = _timed( - lambda: herdr_turns._read_codex_session_turn(target_id) - ) - with target.open("ab", buffering=0) as handle: - handle.write(append) - incremental_content, incremental_poll_ns = _timed( - lambda: herdr_turns._read_codex_session_turn(target_id) - ) - unchanged_content, unchanged_poll_ns = _timed( - lambda: herdr_turns._read_codex_session_turn(target_id) - ) - - if len(read_observations) != 3: - raise RuntimeError("read_observation_count_mismatch") - cold_parse_bytes, incremental_poll_bytes, unchanged_poll_bytes = ( - read_observations - ) - builds_after_all_reads = len(index_observations) - observed_index_visits = sum(index_observations) - - checks = { - "append_sized_second_poll": incremental_poll_bytes == len(append), - "cold_lookup_budget_met": cold_lookup_ns <= COLD_LOOKUP_BUDGET_NS, - "cold_parse_budget_met": cold_parse_ns <= COLD_PARSE_BUDGET_NS, - "cold_parse_bounded": cold_parse_bytes - <= herdr_turns._CODEX_RESYNC_INITIAL_BYTES, - "exact_resolution": cold_result == target.resolve() - and warm_result == target.resolve(), - "fixture_file_count_exact": indexed_sessions == FIXTURE_FILE_COUNT, - "incremental_content_observed": isinstance(incremental_content, Mapping) - and incremental_content.get("assistant_stream_text") == stream_text, - "incremental_poll_budget_met": incremental_poll_ns - <= INCREMENTAL_POLL_BUDGET_NS, - "index_build_bounded": observed_index_visits - <= herdr_turns._CODEX_INDEX_MAX_VISITS, - "index_generation_complete": not index_overflowed, - "one_index_build": builds_after_cold_lookup == 1 - and builds_after_all_reads == 1, - "private_temporary_directory": root_private, - "sparse_large_fixture": logical_file_bytes > FIXTURE_SPARSE_BYTES, - "unchanged_poll_budget_met": unchanged_poll_ns - <= UNCHANGED_POLL_BUDGET_NS, - "unchanged_poll_no_read": unchanged_content is None - and unchanged_poll_bytes == 0, - "warm_lookup_budget_met": warm_lookup_ns <= WARM_LOOKUP_BUDGET_NS, - "warm_lookup_no_walk": builds_after_warm_lookup - == builds_after_cold_lookup, - "wildcard_no_match": wildcard_result is None, - "wildcard_no_walk": builds_after_wildcard == 0, - "cold_content_observed": isinstance(cold_content, Mapping) - and cold_content.get("user_text") == user_text, - } - - report = { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "validating", - "command": _command_text(), - "parameters": { - "fixture_files": FIXTURE_FILE_COUNT, - "sparse_prefix_bytes": FIXTURE_SPARSE_BYTES, - }, - "environment": { - "architecture": platform.machine(), - "fixture_storage": "memory_backed_tmpfs", - "load_average_1m_5m_15m": list(load_average), - "logical_cpus": os.cpu_count(), - "operating_system": platform.system(), - "platform": platform.platform(), - "platform_release": platform.release(), - "python_version": platform.python_version(), - "source_checkout_pythonpath": "src", - "timer": "perf_counter_ns", - }, - "fixture": { - "file_count": FIXTURE_FILE_COUNT, - "logical_session_file_bytes_before_append": logical_file_bytes, - "tail_bytes": logical_file_bytes - FIXTURE_SPARSE_BYTES - 1, - "append_bytes": len(append), - }, - "latency_ns": { - "wildcard_probe": wildcard_ns, - "cold_lookup": { - "elapsed_ns": cold_lookup_ns, - "documented_host_budget_ns": COLD_LOOKUP_BUDGET_NS, - "documented_host_budget_met": checks[ - "cold_lookup_budget_met" - ], - }, - "warm_lookup": { - "elapsed_ns": warm_lookup_ns, - "documented_host_budget_ns": WARM_LOOKUP_BUDGET_NS, - "documented_host_budget_met": checks[ - "warm_lookup_budget_met" - ], - }, - "cold_parse": { - "elapsed_ns": cold_parse_ns, - "documented_host_budget_ns": COLD_PARSE_BUDGET_NS, - "documented_host_budget_met": checks[ - "cold_parse_budget_met" - ], - }, - "incremental_poll": { - "elapsed_ns": incremental_poll_ns, - "documented_host_budget_ns": INCREMENTAL_POLL_BUDGET_NS, - "documented_host_budget_met": checks[ - "incremental_poll_budget_met" - ], - }, - "unchanged_poll": { - "elapsed_ns": unchanged_poll_ns, - "documented_host_budget_ns": UNCHANGED_POLL_BUDGET_NS, - "documented_host_budget_met": checks[ - "unchanged_poll_budget_met" - ], - }, - }, - "bounded_work": { - "index_builds": builds_after_all_reads, - "filesystem_entries_visited": observed_index_visits, - "filesystem_entry_visit_bound": herdr_turns._CODEX_INDEX_MAX_VISITS, - "generation_entries_visited": generation_visited, - "indexed_sessions": indexed_sessions, - "retained_index_bytes": retained_index_bytes, - "retained_index_byte_bound": herdr_turns._CODEX_INDEX_MAX_BYTES, - "wildcard_index_builds": builds_after_wildcard, - "warm_lookup_additional_index_builds": builds_after_warm_lookup - - builds_after_cold_lookup, - "cold_parse_bytes_read": cold_parse_bytes, - "cold_parse_byte_bound": herdr_turns._CODEX_RESYNC_INITIAL_BYTES, - "incremental_poll_bytes_read": incremental_poll_bytes, - "incremental_append_bytes": len(append), - "unchanged_poll_bytes_read": unchanged_poll_bytes, - }, - "checks": checks, - } - finally: - herdr_turns._CODEX_INDEX_BUILD_OBSERVER = prior_index_observer - herdr_turns._CODEX_ISOLATED_READ_OBSERVER = prior_read_observer - _reset_codex_state() - if prior_home is None: - os.environ.pop("CODEX_HOME", None) - else: - os.environ["CODEX_HOME"] = prior_home - - if report is None: - raise RuntimeError("report_not_created") - report["checks"]["temporary_artifacts_removed"] = bool( - temporary_path is not None and not temporary_path.exists() - ) - if _contains_private_value(report, forbidden): - raise RuntimeError("privacy_scan_failed") - report["checks"]["privacy_scan_passed"] = True - failed = sorted( - name for name, passed in report["checks"].items() if passed is not True - ) - if failed: - raise RuntimeError("benchmark_invariants_failed") - report["ok"] = True - report["status"] = "completed" - report["wall_time_ns"] = perf_counter_ns() - benchmark_started - return report - - -def _parser() -> argparse.ArgumentParser: - parser = _Parser( - add_help=False, - description="Run the deterministic synthetic Codex session-reader benchmark.", - ) - parser.add_argument( - "--json", - action="store_true", - help="Emit the aggregate report as one compact JSON object.", - ) - return parser - - -def main() -> int: - try: - args = _parser().parse_args() - except _ArgumentError: - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "invalid_arguments", - } - ) - ) - return 2 - if not args.json: - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "invalid_arguments", - } - ) - ) - return 2 - try: - report = _benchmark() - except Exception as exc: - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "benchmark_failed", - "error_type": type(exc).__name__, - } - ) - ) - return 1 - print(_canonical_json(report)) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/turn_ingestion_benchmark.py b/scripts/turn_ingestion_benchmark.py deleted file mode 100755 index 3568a7d..0000000 --- a/scripts/turn_ingestion_benchmark.py +++ /dev/null @@ -1,1549 +0,0 @@ -#!/usr/bin/env python3 -"""Deterministic synthetic benchmark for background turn ingestion. - -Run from a source checkout with ``PYTHONPATH=src``. The benchmark creates only -private temporary fixtures, exercises a real Unix-domain socket daemon, and -prints one aggregate JSON object. Documented-host latency budgets are evidence -gates for this benchmark run, not generic service-level guarantees. -""" - -from __future__ import annotations - -import argparse -import json -import math -import os -import platform -import sqlite3 -import stat -import sys -import tempfile -import threading -import time -from collections.abc import Callable, Mapping -from pathlib import Path -from time import perf_counter_ns -from typing import Any - -from tendwire.backends.herdr_turns import TurnIngestionScheduler -from tendwire.config import Config -from tendwire.core.commands import ( - DISPOSITION_NO_RECEIPT, - STATUS_NOOP, - CommandEnvelope, - CommandRequest, -) -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.core.turns import recompute_pending_content_fingerprint -from tendwire.daemon import DaemonHooks, TendwireDaemon -from tendwire.daemon_api import ( - MAX_REQUEST_BYTES, - MAX_RESPONSE_BYTES, - DaemonAPIClient, -) -from tendwire.store import sqlite as store - -REPORT_SCHEMA_VERSION = 1 -FIXTURE_HOST = "synthetic-turn-benchmark-host" -FIXTURE_TIMESTAMP = "2026-07-01T00:00:00+00:00" -SCHEDULER_REFRESH_SECONDS = 2.0 -SCHEDULER_WORKERS = 4 -SCHEDULER_QUEUE_CAPACITY = 64 -API_REQUEST_WORKERS = 8 -API_ADMISSION_CAPACITY = 32 -LIST_BUDGET_NS = 350_000_000 -HEALTH_BUDGET_NS = 350_000_000 -COMMAND_BUDGET_NS = 250_000_000 -SHUTDOWN_BOUND_NS = 2_000_000_000 -POLL_SECONDS = 0.005 - - -def _canonical_json(value: Any) -> str: - return json.dumps( - value, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - ) - - -def _nearest_rank(samples: list[int], percentile: float) -> int: - if not samples: - raise ValueError("samples_required") - ordered = sorted(samples) - rank = max(1, math.ceil(percentile * len(ordered))) - return ordered[rank - 1] - - -def _metric( - samples: list[int], - *, - warmups: int, - response_bytes: list[int], - budget_ns: int, -) -> dict[str, Any]: - p95_ns = _nearest_rank(samples, 0.95) - return { - "samples": len(samples), - "warmups": warmups, - "min_ns": min(samples), - "p50_ns": _nearest_rank(samples, 0.50), - "p95_ns": p95_ns, - "max_ns": max(samples), - "response_bytes_max": max(response_bytes), - "documented_host_budget_ns": budget_ns, - "documented_host_budget_met": p95_ns <= budget_ns, - } - - -def _wait_until(predicate: Callable[[], bool], timeout_seconds: float, code: str) -> None: - deadline = time.monotonic() + timeout_seconds - while time.monotonic() < deadline: - if predicate(): - return - time.sleep(POLL_SECONDS) - if not predicate(): - raise RuntimeError(code) - - -def _thread_ids(prefixes: tuple[str, ...]) -> set[int]: - return { - int(thread.ident) - for thread in threading.enumerate() - if thread.ident is not None and thread.name.startswith(prefixes) - } - - -def _process_alive(process_id: int) -> bool: - try: - os.kill(process_id, 0) - except ProcessLookupError: - return False - except PermissionError: - return True - return True - - -def _marker_records(marker_dir: Path) -> list[dict[str, int]]: - records: list[dict[str, int]] = [] - for marker in marker_dir.glob("done-*.json"): - try: - value = json.loads(marker.read_text(encoding="utf-8")) - except (OSError, UnicodeError, json.JSONDecodeError): - continue - if not isinstance(value, Mapping): - continue - try: - ordinal = int(value["ordinal"]) - process_id = int(value["process_id"]) - started_ns = int(value["started_ns"]) - finished_ns = int(value["finished_ns"]) - except (KeyError, TypeError, ValueError): - continue - if ordinal < 0 or process_id <= 0 or finished_ns < started_ns: - continue - records.append( - { - "ordinal": ordinal, - "process_id": process_id, - "started_ns": started_ns, - "finished_ns": finished_ns, - } - ) - return records - - -def _active_markers(marker_dir: Path) -> list[Path]: - return list(marker_dir.glob("active-*.json")) - - -def _source_call_count(marker_dir: Path) -> int: - return len(_active_markers(marker_dir)) + len(_marker_records(marker_dir)) - - -def _interval_maximum(records: list[dict[str, int]]) -> int: - events: list[tuple[int, int]] = [] - for record in records: - events.append((record["started_ns"], 1)) - events.append((record["finished_ns"], -1)) - active = 0 - maximum = 0 - for _at, delta in sorted(events, key=lambda item: (item[0], item[1])): - active += delta - maximum = max(maximum, active) - return maximum - - -def _first_call_overlap_ns( - records: list[dict[str, int]], - blocked_workers: int, -) -> int: - first: list[dict[str, int]] = [] - for ordinal in range(blocked_workers): - matching = sorted( - (record for record in records if record["ordinal"] == ordinal), - key=lambda record: record["started_ns"], - ) - if not matching: - return 0 - first.append(matching[0]) - return max( - 0, - min(record["finished_ns"] for record in first) - - max(record["started_ns"] for record in first), - ) - - -def _write_adapter( - adapter_path: Path, - marker_dir: Path, - release_path: Path, - state_path: Path, -) -> None: - source = f'''#!/usr/bin/env python3 -import json -import os -import pathlib -import sys -import time - -marker_dir = pathlib.Path({str(marker_dir)!r}) -release_path = pathlib.Path({str(release_path)!r}) -state_path = pathlib.Path({str(state_path)!r}) -target = sys.argv[3] if len(sys.argv) > 3 else "" -try: - ordinal = int(target.rsplit("-", 1)[-1]) -except ValueError: - raise SystemExit(2) -process_id = os.getpid() -started_ns = time.monotonic_ns() -active = marker_dir / f"active-{{ordinal}}-{{process_id}}.json" -done = marker_dir / f"done-{{ordinal}}-{{process_id}}.json" -active.write_text(json.dumps({{"ordinal": ordinal, "process_id": process_id, "started_ns": started_ns}}, sort_keys=True), encoding="utf-8") -os.chmod(active, 0o600) -with open(release_path, "rb", buffering=0) as release: - if release.read(1) != b"R": - raise SystemExit(3) -finished_ns = time.monotonic_ns() -done.write_text(json.dumps({{"ordinal": ordinal, "process_id": process_id, "started_ns": started_ns, "finished_ns": finished_ns}}, sort_keys=True), encoding="utf-8") -os.chmod(done, 0o600) -try: - active.unlink() -except FileNotFoundError: - pass -turn = {{"available": True, "user_text": "generated request", "assistant_final_text": "generated response", "complete": True, "has_open_turn": False, "model": "synthetic"}} -try: - pending_state = state_path.read_text(encoding="utf-8").strip() -except OSError: - pending_state = "none" -if pending_state == "open": - turn["pending_decision"] = {{ - "id": "synthetic-private-decision", - "prompt": "generated pending prompt", - "options": [ - {{"id": "approve", "label": "Approve generated choice"}}, - {{"id": "reject", "label": "Reject generated choice"}}, - ], - }} -print(json.dumps({{"result": {{"turn": turn}}}}, sort_keys=True, separators=(",", ":"))) -''' - adapter_path.write_text(source, encoding="utf-8") - adapter_path.chmod(0o700) - - -def _fixture( - blocked_workers: int, -) -> tuple[list[Worker], list[WorkerBinding], dict[str, Any]]: - workers: list[Worker] = [] - bindings: list[WorkerBinding] = [] - for ordinal in range(blocked_workers): - worker = Worker( - id=f"worker-benchmark-{ordinal}", - name=f"Generated Worker {ordinal + 1}", - status="active", - ) - workers.append(worker) - bindings.append( - WorkerBinding( - host_id=FIXTURE_HOST, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value=f"synthetic-agent-{ordinal}", - turn_target_kind="pane_id", - turn_target_value=f"synthetic-pane-{ordinal}", - sendable=True, - reason=None, - observed_at=FIXTURE_TIMESTAMP, - private_fingerprint=f"synthetic-private-binding-{ordinal}", - ) - ) - content = { - "user_text": "generated request", - "assistant_final_text": "generated response", - "complete": True, - "has_open_turn": False, - "model": "synthetic", - } - return workers, bindings, content - - -def _revision_state(db_path: Path) -> dict[str, int]: - with sqlite3.connect(str(db_path)) as conn: - rows = int( - conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - ) - current = int( - conn.execute( - "SELECT COUNT(*) FROM turn_content_revisions WHERE is_current = 1" - ).fetchone()[0] - ) - duplicate_groups = int( - conn.execute( - """ - SELECT COUNT(*) FROM ( - SELECT host_id, turn_id, content_revision - FROM turn_content_revisions - GROUP BY host_id, turn_id, content_revision - HAVING COUNT(*) > 1 - ) - """ - ).fetchone()[0] - ) - return { - "rows": rows, - "current_rows": current, - "duplicate_groups": duplicate_groups, - } - - -def _pending_row_state(db_path: Path) -> dict[str, int]: - with sqlite3.connect(str(db_path)) as conn: - rows = int(conn.execute("SELECT COUNT(*) FROM backend_pending").fetchone()[0]) - open_rows = int( - conn.execute( - "SELECT COUNT(*) FROM backend_pending WHERE observation_state = 'open'" - ).fetchone()[0] - ) - duplicate_groups = int( - conn.execute( - """ - SELECT COUNT(*) FROM ( - SELECT host_id, worker_id - FROM backend_pending - GROUP BY host_id, worker_id - HAVING COUNT(*) > 1 - ) - """ - ).fetchone()[0] - ) - return { - "rows": rows, - "open_rows": open_rows, - "duplicate_groups": duplicate_groups, - } - - -def _outbox_rows(db_path: Path) -> list[tuple[Any, ...]]: - with sqlite3.connect(str(db_path)) as conn: - return conn.execute( - "SELECT * FROM connector_outbox ORDER BY id" - ).fetchall() - - -def _seed_store( - db_path: Path, - workers: list[Worker], - bindings: list[WorkerBinding], - content: Mapping[str, Any], -) -> None: - snapshot = Snapshot( - host_id=FIXTURE_HOST, - updated_at=FIXTURE_TIMESTAMP, - workers=workers, - backend_health=[ - BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - observed_at=FIXTURE_TIMESTAMP, - counts={"workers": len(workers)}, - ) - ], - ) - store.save_snapshot(db_path, snapshot) - if store.upsert_worker_bindings(db_path, bindings) != len(bindings): - raise RuntimeError("binding_seed_failed") - for ordinal, binding in enumerate(bindings): - applied = store.apply_turn_refresh( - db_path, - FIXTURE_HOST, - binding.worker_id, - { - **content, - "source_turn_id": f"synthetic-source-turn-{ordinal}", - }, - expected_binding=binding, - observed_at=FIXTURE_TIMESTAMP, - ) - if applied.updated != 1: - raise RuntimeError("revision_seed_failed") - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - FIXTURE_HOST, - "attention", - "synthetic-delivery-key", - "queued", - '{"generated":true}', - '{"opaque":"generated"}', - FIXTURE_TIMESTAMP, - FIXTURE_TIMESTAMP, - ), - ) - - -class _FixtureEventBackend: - def __init__( - self, - db_path: Path, - workers: list[Worker], - bindings: list[WorkerBinding], - content: Mapping[str, Any], - ) -> None: - self._db_path = db_path - self._workers = workers - self._bindings = bindings - self._content = content - self._callback: Callable[[], None] | None = None - self.started = False - self.stopped = False - self.callback_detached = False - self.flush_calls = 0 - self.committed_events = 0 - self.event_rows_after = 0 - self.callback_notifications = 0 - - def start(self, *, wait_for_reconcile: bool = True) -> None: - if not wait_for_reconcile: - raise RuntimeError("event_reconcile_not_requested") - _seed_store( - self._db_path, - self._workers, - self._bindings, - self._content, - ) - with sqlite3.connect(str(self._db_path)) as conn: - before = int( - conn.execute( - "SELECT COUNT(*) FROM events WHERE host_id = ?", - (FIXTURE_HOST,), - ).fetchone()[0] - ) - for ordinal in range(2): - store.append_event( - self._db_path, - FIXTURE_HOST, - "pane.output_matched", - { - "schema_version": 1, - "generated": True, - "ordinal": ordinal, - }, - observed_at=FIXTURE_TIMESTAMP, - ) - with sqlite3.connect(str(self._db_path)) as conn: - self.event_rows_after = int( - conn.execute( - "SELECT COUNT(*) FROM events WHERE host_id = ?", - (FIXTURE_HOST,), - ).fetchone()[0] - ) - self.committed_events = self.event_rows_after - before - if self.committed_events != 2: - raise RuntimeError("event_commit_count_mismatch") - self.started = True - - def set_turn_refresh_callback( - self, - callback: Callable[[], None] | None, - ) -> None: - self._callback = callback - if callback is None: - self.callback_detached = True - - def emit_committed_burst(self, count: int) -> None: - if not self.started or self.stopped or count <= 0: - raise RuntimeError("event_backend_not_ready") - with sqlite3.connect(str(self._db_path)) as conn: - event_rows = int( - conn.execute( - "SELECT COUNT(*) FROM events WHERE host_id = ?", - (FIXTURE_HOST,), - ).fetchone()[0] - ) - if count != self.committed_events or event_rows != self.event_rows_after: - raise RuntimeError("event_commit_count_mismatch") - callback = self._callback - if callback is None: - raise RuntimeError("event_callback_missing") - self.callback_notifications += 1 - callback() - - def flush(self) -> None: - self.flush_calls += 1 - - def stop(self) -> None: - self.stopped = True - - @property - def operational_status(self) -> Mapping[str, Any]: - return { - "status": "healthy", - "outcome": "healthy_non_empty", - "ready": self.started and not self.stopped, - "running": self.started and not self.stopped, - "reconcile_enabled": False, - "last_event_at": FIXTURE_TIMESTAMP if self.committed_events else None, - "last_snapshot_at": FIXTURE_TIMESTAMP, - "last_reconcile_at": FIXTURE_TIMESTAMP, - } - - -class _APIConcurrency: - def __init__(self, workers: int) -> None: - self.workers = workers - self.lock = threading.Lock() - self.release = threading.Event() - self.active = 0 - self.maximum = 0 - self.probe_entered = 0 - self.dispatches = 0 - self.method_dispatches: dict[str, int] = {} - - def wrap( - self, - dispatcher: Callable[[Any], Mapping[str, Any]], - ) -> Callable[[Any], Mapping[str, Any]]: - def instrumented(request: Any) -> Mapping[str, Any]: - probe = bool( - isinstance(request, Mapping) - and request.get("method") == "ping" - and isinstance(request.get("params"), Mapping) - and request["params"].get("concurrency_probe") is True - ) - with self.lock: - self.active += 1 - self.maximum = max(self.maximum, self.active) - self.dispatches += 1 - method = ( - str(request.get("method")) - if isinstance(request, Mapping) - and isinstance(request.get("method"), str) - else "invalid" - ) - self.method_dispatches[method] = self.method_dispatches.get(method, 0) + 1 - if probe: - self.probe_entered += 1 - if self.probe_entered == self.workers: - self.release.set() - try: - if probe and not self.release.wait(timeout=2.0): - raise RuntimeError("api_probe_barrier_timeout") - return dispatcher(request) - finally: - with self.lock: - self.active -= 1 - - return instrumented - - -def _run_api_probe( - socket_path: Path, - workers: int, -) -> tuple[int, bool]: - ready = threading.Barrier(workers + 1) - results: list[dict[str, Any]] = [] - failures: list[str] = [] - lock = threading.Lock() - - def request() -> None: - try: - ready.wait(timeout=2.0) - response = DaemonAPIClient( - socket_path, - timeout_seconds=3.0, - ).request("ping", {"concurrency_probe": True}) - with lock: - results.append(response) - except BaseException as exc: # noqa: BLE001 - with lock: - failures.append(type(exc).__name__) - - clients = [ - threading.Thread( - target=request, - name=f"tendwire-benchmark-api-probe-{index}", - ) - for index in range(workers) - ] - started = perf_counter_ns() - for client in clients: - client.start() - ready.wait(timeout=2.0) - for client in clients: - client.join(timeout=4.0) - elapsed = perf_counter_ns() - started - clean = all(not client.is_alive() for client in clients) - ok = ( - clean - and not failures - and len(results) == workers - and all(response.get("ok") is True for response in results) - ) - return elapsed, ok - - -def _validate_list(response: Mapping[str, Any], blocked_workers: int) -> None: - result = response.get("result") - turns = result.get("turns") if isinstance(result, Mapping) else None - if ( - response.get("ok") is not True - or not isinstance(result, Mapping) - or result.get("schema_version") != 2 - or not isinstance(turns, list) - or sum( - isinstance(turn, Mapping) - and isinstance(turn.get("content"), Mapping) - and bool(turn["content"].get("content_revision")) - for turn in turns - ) - != blocked_workers - ): - raise RuntimeError("turn_list_contract_failed") - - -def _validate_pending(response: Mapping[str, Any], _blocked_workers: int) -> None: - result = response.get("result") - interactions = ( - result.get("pending_interactions") if isinstance(result, Mapping) else None - ) - health = result.get("pending_health") if isinstance(result, Mapping) else None - counts = health.get("counts") if isinstance(health, Mapping) else None - fingerprint = ( - result.get("content_fingerprint") if isinstance(result, Mapping) else None - ) - if ( - response.get("ok") is not True - or not isinstance(result, Mapping) - or result.get("schema_version") != 1 - or not isinstance(interactions, list) - or not all(isinstance(item, Mapping) for item in interactions) - or not isinstance(health, Mapping) - or health.get("status") not in {"healthy", "degraded"} - or not isinstance(counts, Mapping) - or set(counts) != {"fresh", "stale", "total"} - or any( - isinstance(counts.get(key), bool) - or not isinstance(counts.get(key), int) - or counts[key] < 0 - for key in ("fresh", "stale", "total") - ) - or counts["total"] != counts["fresh"] + counts["stale"] - or not isinstance(fingerprint, str) - or len(fingerprint) != 24 - or any(character not in "0123456789abcdef" for character in fingerprint) - or recompute_pending_content_fingerprint(result) != fingerprint - ): - raise RuntimeError("pending_list_contract_failed") - - -def _wait_for_pending_count( - socket_path: Path, - *, - expected_count: int, - blocked_workers: int, - timeout_seconds: float, - code: str, -) -> tuple[dict[str, Any], int]: - deadline = time.monotonic() + timeout_seconds - polls = 0 - while True: - response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("pending.list") - polls += 1 - _validate_pending(response, blocked_workers) - result = response["result"] - if len(result["pending_interactions"]) == expected_count: - return response, polls - if time.monotonic() >= deadline: - raise RuntimeError(code) - time.sleep(POLL_SECONDS) - - -def _validate_health(response: Mapping[str, Any], blocked_workers: int) -> None: - result = response.get("result") - ingestion = result.get("turn_ingestion") if isinstance(result, Mapping) else None - if ( - response.get("ok") is not True - or not isinstance(ingestion, Mapping) - or int(ingestion.get("active") or 0) < blocked_workers - ): - raise RuntimeError("health_contract_failed") - - -def _validate_command(response: Mapping[str, Any], _blocked_workers: int) -> None: - result = response.get("result") - try: - envelope = ( - CommandEnvelope.from_dict(dict(result)) - if isinstance(result, Mapping) - else None - ) - except (TypeError, ValueError): - envelope = None - if ( - response.get("ok") is not True - or envelope is None - or envelope.ok is not True - or envelope.status != STATUS_NOOP - or envelope.dry_run is not True - or envelope.disposition != DISPOSITION_NO_RECEIPT - ): - raise RuntimeError("command_contract_failed") - - -def _measure_requests( - socket_path: Path, - marker_dir: Path, - scheduler: TurnIngestionScheduler, - *, - method: str, - params: Mapping[str, Any], - validator: Callable[[Mapping[str, Any], int], None], - blocked_workers: int, - warmups: int, - samples: int, - budget_ns: int, -) -> dict[str, Any]: - timings: list[int] = [] - response_bytes: list[int] = [] - for index in range(warmups + samples): - if ( - int(scheduler.operational_status().get("active") or 0) < blocked_workers - or len(_active_markers(marker_dir)) < blocked_workers - ): - raise RuntimeError("adapters_not_blocked_during_request") - started = perf_counter_ns() - response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request(method, params) - elapsed = perf_counter_ns() - started - validator(response, blocked_workers) - if ( - int(scheduler.operational_status().get("active") or 0) < blocked_workers - or len(_active_markers(marker_dir)) < blocked_workers - ): - raise RuntimeError("adapter_block_ended_during_request") - if index >= warmups: - timings.append(elapsed) - response_bytes.append(len(_canonical_json(response).encode("utf-8")) + 1) - return _metric( - timings, - warmups=warmups, - response_bytes=response_bytes, - budget_ns=budget_ns, - ) - - -def _privacy_scan(report: Mapping[str, Any], forbidden_values: list[str]) -> bool: - encoded = _canonical_json(report) - return all(not value or value not in encoded for value in forbidden_values) - - -def _contains_raw_error_field(value: Any) -> bool: - if isinstance(value, Mapping): - return any( - str(key) in {"error", "error_type", "errors"} - or _contains_raw_error_field(item) - for key, item in value.items() - ) - if isinstance(value, list): - return any(_contains_raw_error_field(item) for item in value) - return False - - -def _command_text(args: argparse.Namespace) -> str: - return ( - "PYTHONPATH=src python3 scripts/turn_ingestion_benchmark.py " - f"--workers {args.workers} --blocked-workers {args.blocked_workers} " - f"--blocked-seconds {args.blocked_seconds:g} --warmups {args.warmups} " - f"--samples {args.samples} --json" - ) - - -def _benchmark(args: argparse.Namespace) -> dict[str, Any]: - temporary_path: Path | None = None - forbidden_values: list[str] = [ - FIXTURE_HOST, - "generated request", - "generated response", - "synthetic-delivery-key", - '{"opaque":"generated"}', - "generated pending prompt", - "Approve generated choice", - "Reject generated choice", - "synthetic-private-decision", - ] - report: dict[str, Any] | None = None - with tempfile.TemporaryDirectory( - prefix="tendwire-turn-benchmark-", - dir="/dev/shm", - ) as raw_root: - root = Path(raw_root) - temporary_path = root - db_path = root / "benchmark.db" - socket_path = root / "benchmark.sock" - marker_dir = root / "adapter-markers" - release_path = root / "release-adapters" - adapter_path = root / "generated-herdr" - state_path = root / "pending-state" - marker_dir.mkdir(mode=0o700) - os.mkfifo(release_path, mode=0o600) - _write_adapter(adapter_path, marker_dir, release_path, state_path) - state_path.write_text("none", encoding="utf-8") - state_path.chmod(0o600) - forbidden_values.extend( - [ - str(root), - str(db_path), - str(socket_path), - str(marker_dir), - str(release_path), - str(adapter_path), - str(state_path), - ] - ) - workers, bindings, content = _fixture(args.blocked_workers) - forbidden_values.extend( - [ - worker.id - for worker in workers - ] - ) - forbidden_values.extend( - value - for binding in bindings - for value in ( - binding.target_value, - str(binding.turn_target_value or ""), - binding.private_fingerprint, - ) - ) - - config = Config( - host_id=FIXTURE_HOST, - herdr_bin=str(adapter_path), - data_dir=root, - db_path=db_path, - socket_path=socket_path, - herdr_timeout_seconds=max(60.0, float(args.blocked_seconds) + 30.0), - herdr_backend="socket", - reconcile_interval_seconds=0.0, - turn_refresh_interval_seconds=SCHEDULER_REFRESH_SECONDS, - turn_refresh_workers=SCHEDULER_WORKERS, - ) - event_backend = _FixtureEventBackend( - db_path, - workers, - bindings, - content, - ) - scheduler: TurnIngestionScheduler | None = None - - def scheduler_factory(current: Config) -> TurnIngestionScheduler: - nonlocal scheduler - scheduler = TurnIngestionScheduler(current) - return scheduler - - command_calls = 0 - command_lock = threading.Lock() - - def submit_command(_config: Config, payload: str) -> Mapping[str, Any]: - nonlocal command_calls - parsed = json.loads(payload) - if ( - not isinstance(parsed, dict) - or parsed.get("schema_version") != 1 - or parsed.get("action") != "noop" - or parsed.get("dry_run") is not True - ): - raise RuntimeError("synthetic_command_invalid") - request = CommandRequest.from_dict(parsed) - with command_lock: - command_calls += 1 - return CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_NOOP, - result={}, - ).to_dict() - - baseline_threads = _thread_ids( - ( - "tendwire-turn-", - "tendwire-daemon-api", - "tendwire-benchmark-", - ) - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - event_backend_factory=lambda _config, _stop: event_backend, - turn_scheduler_factory=scheduler_factory, - submit_command=submit_command, - ), - ) - server_thread: threading.Thread | None = None - shutdown_ns = 0 - api_probe_elapsed_ns = 0 - api_probe_ok = False - api_concurrency = _APIConcurrency(args.workers) - initial_revisions: dict[str, int] = {} - initial_outbox: list[tuple[Any, ...]] = [] - during_block_health: dict[str, Any] = {} - final_health: dict[str, Any] = {} - latency: dict[str, Any] = {} - source_calls_before_requests = 0 - source_calls_after_requests = 0 - blocked_observation_started = 0 - release_fd: int | None = None - production_handlers_measured = False - production_pending_handler_measured = False - turn_list_calls_during_pending_measurement = 0 - production_event_callback_bound = False - pending_source_calls_before_measurement = 0 - pending_source_calls_after_measurement = 0 - pending_rows_before_requests: dict[str, int] = {} - pending_rows_after_requests: dict[str, int] = {} - final_pending_rows: dict[str, int] = {} - independent_pending_polls = 0 - independent_turn_list_calls = 0 - independent_prompt_count = 0 - independent_clear_count = -1 - independent_discovery_fingerprint_changed = False - independent_unchanged_fingerprint_stable = False - independent_clear_fingerprint_changed = False - independent_clear_restored_baseline = False - try: - daemon.start() - if scheduler is None or daemon.server is None: - raise RuntimeError("daemon_components_missing") - original_dispatcher = daemon.server.dispatcher - daemon.server.dispatcher = api_concurrency.wrap(original_dispatcher) - api_dispatch = getattr(original_dispatcher, "__self__", None) - production_handlers_measured = bool( - getattr(api_dispatch, "_get_turns", None) == daemon.get_turns - and getattr(api_dispatch, "_get_health", None) == daemon.get_health - ) - production_pending_handler_measured = bool( - getattr(api_dispatch, "_get_pending", None) == daemon.get_pending - ) - production_event_callback_bound = ( - event_backend._callback == scheduler.request_refresh - ) - if ( - not production_handlers_measured - or not production_pending_handler_measured - or not production_event_callback_bound - ): - raise RuntimeError("production_handlers_not_bound") - server_thread = threading.Thread( - target=daemon.serve_forever, - name="tendwire-benchmark-daemon", - ) - server_thread.start() - _wait_until( - lambda: len(_active_markers(marker_dir)) >= args.blocked_workers, - 5.0, - "blocked_adapters_not_entered", - ) - blocked_observation_started = perf_counter_ns() - initial_revisions = _revision_state(db_path) - initial_outbox = _outbox_rows(db_path) - source_calls_before_requests = _source_call_count(marker_dir) - pending_rows_before_requests = _pending_row_state(db_path) - if source_calls_before_requests != args.blocked_workers: - raise RuntimeError("unexpected_initial_source_calls") - - api_probe_elapsed_ns, api_probe_ok = _run_api_probe( - socket_path, - args.workers, - ) - event_backend.emit_committed_burst(2) - _wait_until( - lambda: int(scheduler.operational_status().get("coalesced") or 0) - >= args.blocked_workers, - SCHEDULER_REFRESH_SECONDS + 2.0, - "scheduler_coalescing_not_observed", - ) - during_response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("health.get") - _validate_health(during_response, args.blocked_workers) - during_block_health = dict( - during_response["result"]["turn_ingestion"] - ) - - latency["turn_list"] = _measure_requests( - socket_path, - marker_dir, - scheduler, - method="turn.list", - params={ - "schema_version": 2, - "limit": 100, - "cursor": None, - "since": None, - }, - validator=_validate_list, - blocked_workers=args.blocked_workers, - warmups=args.warmups, - samples=args.samples, - budget_ns=LIST_BUDGET_NS, - ) - pending_source_calls_before_measurement = _source_call_count(marker_dir) - turn_list_calls_before_pending = api_concurrency.method_dispatches.get( - "turn.list", - 0, - ) - latency["pending_list"] = _measure_requests( - socket_path, - marker_dir, - scheduler, - method="pending.list", - params={}, - validator=_validate_pending, - blocked_workers=args.blocked_workers, - warmups=args.warmups, - samples=args.samples, - budget_ns=LIST_BUDGET_NS, - ) - pending_source_calls_after_measurement = _source_call_count(marker_dir) - if ( - pending_source_calls_after_measurement - != pending_source_calls_before_measurement - ): - raise RuntimeError("pending_list_started_source_reads") - turn_list_calls_during_pending_measurement = ( - api_concurrency.method_dispatches.get("turn.list", 0) - - turn_list_calls_before_pending - ) - latency["health_get"] = _measure_requests( - socket_path, - marker_dir, - scheduler, - method="health.get", - params={}, - validator=_validate_health, - blocked_workers=args.blocked_workers, - warmups=args.warmups, - samples=args.samples, - budget_ns=HEALTH_BUDGET_NS, - ) - latency["command_submit"] = _measure_requests( - socket_path, - marker_dir, - scheduler, - method="command.submit", - params={ - "schema_version": 1, - "action": "noop", - "dry_run": True, - }, - validator=_validate_command, - blocked_workers=args.blocked_workers, - warmups=args.warmups, - samples=args.samples, - budget_ns=COMMAND_BUDGET_NS, - ) - source_calls_after_requests = _source_call_count(marker_dir) - if source_calls_after_requests != source_calls_before_requests: - raise RuntimeError("request_path_started_source_reads") - pending_rows_after_requests = _pending_row_state(db_path) - - remaining_ns = int(args.blocked_seconds * 1_000_000_000) - ( - perf_counter_ns() - blocked_observation_started - ) - if remaining_ns > 0: - time.sleep(remaining_ns / 1_000_000_000) - if len(_active_markers(marker_dir)) < args.blocked_workers: - raise RuntimeError("configured_block_not_held") - release_fd = os.open(release_path, os.O_WRONLY | os.O_NONBLOCK) - os.write(release_fd, b"R" * max(64, args.blocked_workers * 4)) - _wait_until( - lambda: ( - _source_call_count(marker_dir) >= args.blocked_workers * 2 - and int(scheduler.operational_status().get("active") or 0) == 0 - and int(scheduler.operational_status().get("queue_depth") or 0) == 0 - ), - 3.0, - "scheduler_did_not_drain", - ) - turn_calls_before_independent = api_concurrency.method_dispatches.get( - "turn.list", - 0, - ) - baseline_pending_response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("pending.list") - independent_pending_polls += 1 - _validate_pending(baseline_pending_response, args.blocked_workers) - baseline_pending = baseline_pending_response["result"] - baseline_fingerprint = str(baseline_pending["content_fingerprint"]) - - state_path.write_text("open", encoding="utf-8") - discovery_source_calls = _source_call_count(marker_dir) - event_backend.emit_committed_burst(2) - _wait_until( - lambda: ( - _source_call_count(marker_dir) - >= discovery_source_calls + args.blocked_workers - and int(scheduler.operational_status().get("active") or 0) == 0 - and int(scheduler.operational_status().get("queue_depth") or 0) == 0 - ), - 3.0, - "independent_pending_discovery_did_not_drain", - ) - open_pending_response, discovery_polls = _wait_for_pending_count( - socket_path, - expected_count=args.blocked_workers, - blocked_workers=args.blocked_workers, - timeout_seconds=3.0, - code="independent_pending_not_discovered", - ) - independent_pending_polls += discovery_polls - open_pending = open_pending_response["result"] - independent_prompt_count = len(open_pending["pending_interactions"]) - open_fingerprint = str(open_pending["content_fingerprint"]) - independent_discovery_fingerprint_changed = ( - open_fingerprint != baseline_fingerprint - ) - - unchanged_pending_response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("pending.list") - independent_pending_polls += 1 - _validate_pending(unchanged_pending_response, args.blocked_workers) - unchanged_fingerprint = str( - unchanged_pending_response["result"]["content_fingerprint"] - ) - independent_unchanged_fingerprint_stable = ( - unchanged_fingerprint == open_fingerprint - ) - - state_path.write_text("none", encoding="utf-8") - clearing_source_calls = _source_call_count(marker_dir) - event_backend.emit_committed_burst(2) - _wait_until( - lambda: ( - _source_call_count(marker_dir) - >= clearing_source_calls + args.blocked_workers - and int(scheduler.operational_status().get("active") or 0) == 0 - and int(scheduler.operational_status().get("queue_depth") or 0) == 0 - ), - 3.0, - "independent_pending_clear_did_not_drain", - ) - cleared_pending_response, clear_polls = _wait_for_pending_count( - socket_path, - expected_count=0, - blocked_workers=args.blocked_workers, - timeout_seconds=3.0, - code="independent_pending_not_cleared", - ) - independent_pending_polls += clear_polls - cleared_pending = cleared_pending_response["result"] - independent_clear_count = len(cleared_pending["pending_interactions"]) - cleared_fingerprint = str(cleared_pending["content_fingerprint"]) - independent_clear_fingerprint_changed = ( - cleared_fingerprint != open_fingerprint - ) - independent_clear_restored_baseline = ( - cleared_fingerprint == baseline_fingerprint - ) - independent_turn_list_calls = ( - api_concurrency.method_dispatches.get("turn.list", 0) - - turn_calls_before_independent - ) - final_pending_rows = _pending_row_state(db_path) - - daemon.server.dispatcher = api_concurrency.wrap(original_dispatcher) - final_response = DaemonAPIClient( - socket_path, - timeout_seconds=1.0, - ).request("health.get") - if final_response.get("ok") is not True: - raise RuntimeError("final_health_failed") - final_health = dict(final_response["result"]["turn_ingestion"]) - finally: - if release_fd is not None: - try: - os.close(release_fd) - except OSError: - pass - shutdown_started = perf_counter_ns() - daemon.stop() - if server_thread is not None: - server_thread.join(timeout=2.0) - shutdown_ns = perf_counter_ns() - shutdown_started - - if scheduler is None or server_thread is None: - raise RuntimeError("daemon_lifecycle_incomplete") - final_revisions = _revision_state(db_path) - final_outbox = _outbox_rows(db_path) - adapter_records = _marker_records(marker_dir) - process_ids = {record["process_id"] for record in adapter_records} - _wait_until( - lambda: all(not _process_alive(process_id) for process_id in process_ids), - 1.0, - "adapter_child_not_reaped", - ) - _wait_until( - lambda: not ( - _thread_ids( - ( - "tendwire-turn-", - "tendwire-daemon-api", - "tendwire-benchmark-", - ) - ) - - baseline_threads - ), - 2.0, - "benchmark_thread_not_reaped", - ) - remaining_threads = _thread_ids( - ( - "tendwire-turn-", - "tendwire-daemon-api", - "tendwire-benchmark-", - ) - ) - baseline_threads - source_calls_final = len(adapter_records) - forbidden_values.extend( - marker - for process_id in process_ids - for marker in ( - f'"process_id":{process_id}', - f'"process_id":"{process_id}"', - f'"pid":{process_id}', - f'"pid":"{process_id}"', - ) - ) - overlap_ns = _first_call_overlap_ns(adapter_records, args.blocked_workers) - response_bytes_max = max( - metric["response_bytes_max"] for metric in latency.values() - ) - scheduler_bounds = { - "refresh_interval_seconds": config.turn_refresh_interval_seconds, - "max_workers": config.turn_refresh_workers, - "queue_capacity": SCHEDULER_QUEUE_CAPACITY, - "adapter_timeout_seconds": config.herdr_timeout_seconds, - } - checks = { - "private_temporary_directory": stat.S_IMODE(root.stat().st_mode) == 0o700, - "private_marker_directory": stat.S_IMODE(marker_dir.stat().st_mode) == 0o700, - "private_adapter_executable": stat.S_IMODE(adapter_path.stat().st_mode) == 0o700, - "private_database_mode": stat.S_IMODE(db_path.stat().st_mode) & 0o077 == 0, - "real_unix_socket_removed": not os.path.lexists(socket_path), - "blocked_adapters_overlapped": overlap_ns >= int(args.blocked_seconds * 1_000_000_000), - "cached_requests_started_no_source_reads": source_calls_after_requests - == source_calls_before_requests, - "api_probe_completed": api_probe_ok, - "api_worker_bound_observed": api_concurrency.maximum == args.workers, - "production_list_health_handlers_measured": production_handlers_measured, - "production_pending_handler_measured": production_pending_handler_measured, - "production_event_callback_bound": production_event_callback_bound, - "pending_list_started_no_turn_reads": turn_list_calls_during_pending_measurement == 0, - "pending_list_started_no_source_reads": ( - pending_source_calls_after_measurement - == pending_source_calls_before_measurement - ), - "pending_list_store_rows_unchanged": ( - pending_rows_after_requests == pending_rows_before_requests - ), - "independent_pending_discovered": ( - independent_prompt_count == args.blocked_workers - ), - "independent_pending_cleared": independent_clear_count == 0, - "independent_pending_zero_turn_calls": independent_turn_list_calls == 0, - "independent_pending_discovery_fingerprint_changed": ( - independent_discovery_fingerprint_changed - ), - "independent_pending_unchanged_fingerprint_stable": ( - independent_unchanged_fingerprint_stable - ), - "independent_pending_clear_fingerprint_changed": ( - independent_clear_fingerprint_changed - ), - "independent_pending_clear_restored_baseline": ( - independent_clear_restored_baseline - ), - "no_duplicate_pending_rows": ( - final_pending_rows.get("duplicate_groups") == 0 - ), - "independent_pending_health_coherent": ( - open_pending["pending_health"] - == { - "status": "healthy", - "counts": { - "fresh": args.blocked_workers, - "stale": 0, - "total": args.blocked_workers, - }, - } - and cleared_pending["pending_health"] - == { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - } - ), - "independent_pending_rows_coherent_after_clear": ( - final_pending_rows.get("rows") == args.blocked_workers - and final_pending_rows.get("open_rows") == 0 - ), - "adapter_worker_bound_observed": _interval_maximum(adapter_records) - == args.blocked_workers, - "event_burst_committed_before_notification": event_backend.committed_events == 2 - and event_backend.callback_notifications == 3, - "scheduler_queue_drained": int(final_health.get("queue") or 0) == 0 - and int(final_health.get("active") or 0) == 0, - "scheduler_coalescing_observed": int(final_health.get("coalesced") or 0) - >= args.blocked_workers, - "scheduler_no_timeouts_or_queue_full": int( - final_health.get("timed_out") or 0 - ) - == 0 - and int(final_health.get("queue_full") or 0) == 0, - "revision_rows_unchanged": final_revisions == initial_revisions, - "no_duplicate_revisions": final_revisions.get("duplicate_groups") == 0, - "outbox_rows_unchanged": final_outbox == initial_outbox, - "expected_outbox_rows_preserved": ( - len(final_outbox) == args.blocked_workers + 1 - ), - "expected_command_calls": command_calls == args.warmups + args.samples, - "list_budget_met": bool(latency["turn_list"]["documented_host_budget_met"]), - "pending_list_budget_met": bool( - latency["pending_list"]["documented_host_budget_met"] - ), - "health_budget_met": bool(latency["health_get"]["documented_host_budget_met"]), - "command_budget_met": bool(latency["command_submit"]["documented_host_budget_met"]), - "shutdown_bounded": shutdown_ns <= SHUTDOWN_BOUND_NS, - "daemon_thread_reaped": not server_thread.is_alive(), - "adapter_children_reaped": all( - not _process_alive(process_id) for process_id in process_ids - ), - "benchmark_threads_reaped": not remaining_threads, - "event_callback_detached": event_backend.callback_detached, - "event_backend_stopped": event_backend.stopped, - } - report = { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "validating", - "command": _command_text(args), - "parameters": { - "api_probe_workers": args.workers, - "blocked_adapter_workers": args.blocked_workers, - "blocked_seconds": args.blocked_seconds, - "warmups_per_operation": args.warmups, - "samples_per_operation": args.samples, - }, - "environment": { - "python_version": platform.python_version(), - "sqlite_version": sqlite3.sqlite_version, - "operating_system": platform.system(), - "platform_release": platform.release(), - "platform": platform.platform(), - "architecture": platform.machine(), - "timer": "perf_counter_ns", - "percentiles": "nearest_rank", - "source_checkout_pythonpath": "src", - "fixture_storage": "memory_backed_tmpfs", - }, - "transport": { - "kind": "unix_stream_socket", - "request_workers": API_REQUEST_WORKERS, - "admission_capacity": API_ADMISSION_CAPACITY, - "request_frame_max_bytes": MAX_REQUEST_BYTES, - "response_frame_max_bytes": MAX_RESPONSE_BYTES, - "observed_max_api_concurrency": api_concurrency.maximum, - "probe_elapsed_ns": api_probe_elapsed_ns, - "dispatches": api_concurrency.dispatches, - "measured_response_bytes_max": response_bytes_max, - "method_dispatches": dict(sorted(api_concurrency.method_dispatches.items())), - "handler_mode": "production_store_backed", - }, - "ingestion": { - "scheduler_bounds": scheduler_bounds, - "source_calls_before_requests": source_calls_before_requests, - "source_calls_after_requests": source_calls_after_requests, - "source_calls_final": source_calls_final, - "turn_list_calls_during_pending_measurement": turn_list_calls_during_pending_measurement, - "pending_source_calls_before_measurement": pending_source_calls_before_measurement, - "pending_source_calls_after_measurement": pending_source_calls_after_measurement, - "independent_pending_polls": independent_pending_polls, - "independent_turn_list_calls": independent_turn_list_calls, - "independent_prompt_count": independent_prompt_count, - "independent_clear_count": independent_clear_count, - "observed_max_adapter_concurrency": _interval_maximum(adapter_records), - "first_call_overlap_ns": overlap_ns, - "event_committed_count": event_backend.committed_events, - "event_callback_notifications": event_backend.callback_notifications, - "during_block": { - "status": during_block_health.get("status"), - "queue": during_block_health.get("queue"), - "active": during_block_health.get("active"), - "refreshed": during_block_health.get("refreshed"), - "failed": during_block_health.get("failed"), - "timed_out": during_block_health.get("timed_out"), - "coalesced": during_block_health.get("coalesced"), - "queue_full": during_block_health.get("queue_full"), - }, - "final": { - "status": final_health.get("status"), - "queue": final_health.get("queue"), - "active": final_health.get("active"), - "refreshed": final_health.get("refreshed"), - "failed": final_health.get("failed"), - "timed_out": final_health.get("timed_out"), - "coalesced": final_health.get("coalesced"), - "queue_full": final_health.get("queue_full"), - }, - }, - "latency_ns": latency, - "store": { - "schema_version": store.STORE_SCHEMA_VERSION, - "event_rows_after": event_backend.event_rows_after, - "generated_event_rows": event_backend.committed_events, - "bindings": len(bindings), - "revision_rows_before": initial_revisions["rows"], - "revision_rows_after": final_revisions["rows"], - "current_revision_rows_after": final_revisions["current_rows"], - "duplicate_revision_groups_after": final_revisions["duplicate_groups"], - "pending_rows_before_requests": pending_rows_before_requests["rows"], - "pending_rows_after_requests": pending_rows_after_requests["rows"], - "pending_rows_after_independent_clear": final_pending_rows["rows"], - "pending_open_rows_after_independent_clear": final_pending_rows["open_rows"], - "duplicate_pending_groups_after": final_pending_rows["duplicate_groups"], - "outbox_rows_before": len(initial_outbox), - "outbox_rows_after": len(final_outbox), - }, - "cleanup": { - "shutdown_ns": shutdown_ns, - "shutdown_bound_ns": SHUTDOWN_BOUND_NS, - "adapter_child_count": len(process_ids), - "adapter_children_alive": sum( - _process_alive(process_id) for process_id in process_ids - ), - "benchmark_threads_alive": len(remaining_threads), - "socket_present_after_shutdown": os.path.lexists(socket_path), - "event_flush_calls": event_backend.flush_calls, - }, - "checks": checks, - } - - if report is None: - raise RuntimeError("report_not_created") - report["checks"]["temporary_artifacts_removed"] = bool( - temporary_path is not None and not temporary_path.exists() - ) - report["checks"]["raw_errors_absent"] = not _contains_raw_error_field(report) - if not _privacy_scan(report, forbidden_values): - raise RuntimeError("privacy_scan_failed") - report["checks"]["privacy_scan_passed"] = True - failed = sorted( - name - for name, passed in report["checks"].items() - if isinstance(passed, bool) and not passed - ) - if failed: - raise RuntimeError("benchmark_invariants_failed") - report["ok"] = True - report["status"] = "completed" - return report - - -def _parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - description="Run the deterministic synthetic turn-ingestion benchmark." - ) - parser.add_argument("--workers", type=int, default=8) - parser.add_argument("--blocked-workers", type=int, default=2) - parser.add_argument("--blocked-seconds", type=float, default=5.0) - parser.add_argument("--warmups", type=int, default=3) - parser.add_argument("--samples", type=int, default=21) - parser.add_argument( - "--json", - action="store_true", - help="Emit the aggregate report as one compact JSON object.", - ) - return parser - - -def main() -> int: - benchmark_started = perf_counter_ns() - args = _parser().parse_args() - if ( - not 1 <= args.workers <= API_REQUEST_WORKERS - or not 2 <= args.blocked_workers <= SCHEDULER_WORKERS - or not math.isfinite(args.blocked_seconds) - or args.blocked_seconds <= 0 - or args.warmups < 0 - or args.samples <= 0 - ): - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "invalid_arguments", - } - ) - ) - return 2 - try: - report = _benchmark(args) - report["wall_time_ns"] = perf_counter_ns() - benchmark_started - except Exception as exc: - print( - _canonical_json( - { - "schema_version": REPORT_SCHEMA_VERSION, - "ok": False, - "status": "benchmark_failed", - "error_type": type(exc).__name__, - } - ) - ) - return 1 - print(_canonical_json(report)) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/src/tendwire/backends/acp_client.py b/src/tendwire/backends/acp_client.py new file mode 100644 index 0000000..94a2ac1 --- /dev/null +++ b/src/tendwire/backends/acp_client.py @@ -0,0 +1,1688 @@ +"""Synchronous ACP v1 client over a supervised subprocess's stdio. + +The public client is intentionally independent from Tendwire's daemon and +persistence layers. A single reader thread owns stdout so concurrent requests, +streamed ``session/update`` notifications, and agent permission requests cannot +steal each other's frames. +""" + +from __future__ import annotations + +import math +import os +import queue +import select +import signal +import subprocess +import threading +import time +from collections import deque +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from enum import Enum +from pathlib import Path +from types import MappingProxyType +from typing import Any, Callable, TypeVar + +from acp.schema import ( + InitializeResponse as UpstreamInitializeResponse, + ListSessionsResponse as UpstreamListSessionsResponse, + LoadSessionResponse as UpstreamLoadSessionResponse, + NewSessionResponse as UpstreamNewSessionResponse, + PromptRequest as UpstreamPromptRequest, + PromptResponse as UpstreamPromptResponse, + ResumeSessionResponse as UpstreamResumeSessionResponse, +) +from pydantic import ValidationError + +from tendwire import __version__ + +from .acp_protocol import ( + ACP_PROTOCOL_VERSION, + DEFAULT_MAX_FRAME_BYTES, + AgentCapabilities, + AcpEnvelopeError, + AcpFramingError, + AcpProtocolError, + InitializeResult, + JsonRpcNotification, + JsonRpcRequest, + JsonRpcResponse, + PermissionRequest, + PromptResult, + RequestId, + SessionInfo, + SessionPage, + SessionResult, + SessionUpdate, + StopReason, + SteeringOutcome, + SteeringResult, + decode_json_line, + encode_message, + error_envelope, + notification_envelope, + parse_permission_request, + parse_session_update, + request_envelope, + result_envelope, +) + +_DEFAULT_REQUEST_TIMEOUT = 30.0 +_DEFAULT_PROMPT_TIMEOUT = 60.0 * 60.0 +_DEFAULT_CLOSE_TIMEOUT = 3.0 +_DEFAULT_QUEUE_SIZE = 4096 +_DEFAULT_STDERR_LIMIT = 64 * 1024 +_METHOD_NOT_FOUND = -32601 +_INTERNAL_ERROR = -32603 + + +class AcpClientError(AcpProtocolError): + """Base error for ACP client lifecycle and transport failures.""" + + +class AcpClientStateError(AcpClientError, RuntimeError): + """An operation is invalid in the client's current state.""" + + +class AcpTransportError(AcpClientError, ConnectionError): + """The ACP subprocess or stdio stream failed.""" + + +class AcpRequestTimeoutError(AcpClientError, TimeoutError): + """A request did not receive a response before its deadline.""" + + +class AcpCapabilityError(AcpClientError, RuntimeError): + """The requested optional method was not advertised by the agent.""" + + +class AcpProtocolVersionError(AcpClientError): + """The agent selected an ACP version this client does not support.""" + + +class AcpEventQueueFullError(AcpTransportError): + """The consumer fell behind the bounded lossless event queues.""" + + +class ClientState(str, Enum): + NEW = "new" + RUNNING = "running" + INITIALIZED = "initialized" + CLOSING = "closing" + CLOSED = "closed" + FAILED = "failed" + + +@dataclass(frozen=True, slots=True) +class RawNotification: + method: str + params: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class InboundRequest: + request_id: RequestId + method: str + params: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class ProcessExit: + returncode: int + stderr_tail: str + + +@dataclass(slots=True) +class _PendingRequest: + waiter: queue.Queue[JsonRpcResponse | BaseException] + method: str + session_id: str | None + + +_T = TypeVar("_T") +_END = object() +SessionEvent = SessionUpdate | PermissionRequest + + +class BoundedAcpConnection: + """Thread-safe ACP connection with bounded subprocess stdio framing.""" + + def __init__( + self, + argv: Sequence[str | os.PathLike[str]], + *, + cwd: str | os.PathLike[str] | None = None, + env: Mapping[str, str] | None = None, + request_timeout: float = _DEFAULT_REQUEST_TIMEOUT, + prompt_timeout: float = _DEFAULT_PROMPT_TIMEOUT, + close_timeout: float = _DEFAULT_CLOSE_TIMEOUT, + max_frame_bytes: int = DEFAULT_MAX_FRAME_BYTES, + max_pending_events: int = _DEFAULT_QUEUE_SIZE, + stderr_limit_bytes: int = _DEFAULT_STDERR_LIMIT, + supported_extension_notifications: Sequence[str] = (), + supported_extension_requests: Sequence[str] = (), + ) -> None: + command = tuple(os.fspath(item) for item in argv) + if not command or any( + not isinstance(item, str) or not item or "\x00" in item for item in command + ): + raise ValueError("argv must contain at least one non-empty argument") + self.argv = command + self.cwd = _absolute_path(cwd, "process cwd") if cwd is not None else None + self.env = _validated_env(env) + self.request_timeout = _positive_timeout(request_timeout, "request_timeout") + self.prompt_timeout = _positive_timeout(prompt_timeout, "prompt_timeout") + self.close_timeout = _positive_timeout(close_timeout, "close_timeout") + if max_frame_bytes <= 0: + raise ValueError("max_frame_bytes must be positive") + if max_pending_events <= 0: + raise ValueError("max_pending_events must be positive") + if stderr_limit_bytes <= 0: + raise ValueError("stderr_limit_bytes must be positive") + self.max_frame_bytes = max_frame_bytes + self.max_pending_events = max_pending_events + self.stderr_limit_bytes = stderr_limit_bytes + self.supported_extension_notifications = _extension_method_set( + supported_extension_notifications, + "supported_extension_notifications", + ) + self.supported_extension_requests = _extension_method_set( + supported_extension_requests, + "supported_extension_requests", + ) + + self._state = ClientState.NEW + self._process: subprocess.Popen[bytes] | None = None + self._state_lock = threading.RLock() + self._write_lock = threading.Lock() + self._initialize_lock = threading.Lock() + self._request_id_lock = threading.Lock() + self._next_id = 1 + self._pending: dict[RequestId, _PendingRequest] = {} + self._pending_lock = threading.Lock() + self._pending_permissions: dict[RequestId, PermissionRequest] = {} + self._cancelled_sessions: set[str] = set() + self._active_prompts: dict[str, int] = {} + self._permission_lock = threading.Lock() + # Updates and permission requests share one reader-ordered queue. A + # pair of duplicate queues can both reorder cross-kind events and fail + # the transport when an embedding consumes only one of them. + self._session_events: deque[SessionEvent] = deque() + self._session_event_condition = threading.Condition() + self._notifications: queue.Queue[RawNotification | object] = queue.Queue( + max_pending_events + ) + self._inbound_requests: queue.Queue[InboundRequest | object] = queue.Queue( + max_pending_events + ) + self._reader_thread: threading.Thread | None = None + self._stderr_thread: threading.Thread | None = None + self._stop = threading.Event() + self._closed = threading.Event() + self._stderr_chunks: deque[bytes] = deque() + self._stderr_size = 0 + self._stderr_lock = threading.Lock() + self._failure: BaseException | None = None + self._exit: ProcessExit | None = None + self._initialize_result: InitializeResult | None = None + + def __enter__(self) -> "BoundedAcpConnection": + self.start() + return self + + def __exit__(self, exc_type: object, exc: object, tb: object) -> None: + self.close() + + @property + def state(self) -> ClientState: + with self._state_lock: + return self._state + + @property + def process(self) -> subprocess.Popen[bytes] | None: + return self._process + + @property + def capabilities(self) -> AgentCapabilities | None: + result = self._initialize_result + return result.capabilities if result is not None else None + + @property + def initialize_result(self) -> InitializeResult | None: + return self._initialize_result + + @property + def steering_supported(self) -> bool: + """Whether the agent explicitly advertised the steering extension.""" + + initialized = self._initialize_result + if initialized is None: + return False + meta = initialized.raw.get("_meta") + steering = meta.get("steering") if isinstance(meta, Mapping) else None + return isinstance(steering, Mapping) and steering.get("supported") is True + + @property + def failure(self) -> BaseException | None: + return self._failure + + @property + def exit(self) -> ProcessExit | None: + process = self._process + if process is not None: + returncode = process.poll() + if returncode is not None: + self._exit = ProcessExit(returncode, self.stderr_tail()) + return self._exit + + def stderr_tail(self) -> str: + with self._stderr_lock: + data = b"".join(self._stderr_chunks) + return data.decode("utf-8", errors="replace") + + def start(self) -> "BoundedAcpConnection": + with self._state_lock: + if self._state in {ClientState.RUNNING, ClientState.INITIALIZED}: + return self + if self._state is not ClientState.NEW: + raise AcpClientStateError(f"cannot start ACP client in state {self._state.value}") + try: + process = subprocess.Popen( + self.argv, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + cwd=self.cwd, + env=self.env, + bufsize=0, + close_fds=True, + start_new_session=os.name == "posix", + ) + except OSError as exc: + self._state = ClientState.FAILED + self._failure = AcpTransportError( + f"could not start ACP agent {self.argv[0]!r}" + ) + raise self._failure from exc + self._process = process + assert process.stdin is not None + try: + os.set_blocking(process.stdin.fileno(), False) + except OSError as exc: + process.kill() + process.wait() + self._state = ClientState.FAILED + self._failure = AcpTransportError( + "could not configure ACP agent stdin" + ) + raise self._failure from exc + self._state = ClientState.RUNNING + self._reader_thread = threading.Thread( + target=self._reader_main, + name=f"acp-reader-{process.pid}", + daemon=True, + ) + self._stderr_thread = threading.Thread( + target=self._stderr_main, + name=f"acp-stderr-{process.pid}", + daemon=True, + ) + self._reader_thread.start() + self._stderr_thread.start() + return self + + def initialize( + self, + *, + client_capabilities: Mapping[str, Any] | None = None, + client_name: str = "tendwire", + client_version: str = __version__, + client_title: str = "Tendwire", + timeout: float | None = None, + ) -> InitializeResult: + """Perform ACP v1 capability negotiation. + + ACP v1 does not define a post-response ``initialized`` notification. + """ + self.start() + if not client_name or not client_version: + raise ValueError("client_name and client_version must be non-empty") + with self._initialize_lock: + with self._state_lock: + if self._state is ClientState.INITIALIZED: + assert self._initialize_result is not None + return self._initialize_result + if self._state is not ClientState.RUNNING: + self._raise_unusable() + client_info: dict[str, Any] = { + "name": client_name, + "version": client_version, + } + if client_title: + client_info["title"] = client_title + result = self.request( + "initialize", + { + "protocolVersion": ACP_PROTOCOL_VERSION, + "clientCapabilities": dict(client_capabilities or {}), + "clientInfo": client_info, + }, + timeout=timeout, + require_initialized=False, + ) + raw = _require_mapping(result, "initialize result") + _validate_upstream( + UpstreamInitializeResponse, + raw, + "initialize result", + ) + version = raw.get("protocolVersion") + if ( + not isinstance(version, int) + or isinstance(version, bool) + or version != ACP_PROTOCOL_VERSION + ): + failure = AcpProtocolVersionError( + f"agent selected unsupported ACP protocol version {version!r}" + ) + self._set_failed(failure) + # ACP v1 says clients should close a connection after an + # unsupported selection. Do the bounded cleanup here so direct + # callers cannot accidentally leave the adapter process alive. + try: + self.close() + except BaseException: + pass + raise failure + capabilities_value = raw.get("agentCapabilities", {}) + if not isinstance(capabilities_value, Mapping): + capabilities_value = {} + agent_info_value = raw.get("agentInfo") + agent_info = ( + MappingProxyType(dict(agent_info_value)) + if isinstance(agent_info_value, Mapping) + else None + ) + auth_methods_value = raw.get("authMethods", []) + auth_methods = ( + tuple( + MappingProxyType(dict(item)) + for item in auth_methods_value + if isinstance(item, Mapping) + ) + if isinstance(auth_methods_value, list) + else () + ) + parsed = InitializeResult( + protocol_version=version, + capabilities=AgentCapabilities.from_mapping(capabilities_value), + agent_info=agent_info, + auth_methods=auth_methods, + raw=MappingProxyType(dict(raw)), + ) + with self._state_lock: + if self._state is not ClientState.RUNNING: + self._raise_unusable() + self._initialize_result = parsed + self._state = ClientState.INITIALIZED + return parsed + + def request( + self, + method: str, + params: Mapping[str, Any] | None = None, + *, + timeout: float | None = None, + require_initialized: bool = True, + on_writing: Callable[[], None] | None = None, + on_written: Callable[[], None] | None = None, + ) -> Any: + if require_initialized: + self._require_initialized() + else: + self._require_running() + wait_timeout = self.request_timeout if timeout is None else _positive_timeout( + timeout, "timeout" + ) + deadline = time.monotonic() + wait_timeout + request_id = self._new_request_id() + waiter: queue.Queue[JsonRpcResponse | BaseException] = queue.Queue(maxsize=1) + session_id_value = (params or {}).get("sessionId") + pending = _PendingRequest( + waiter=waiter, + method=method, + session_id=( + session_id_value if isinstance(session_id_value, str) else None + ), + ) + with self._pending_lock: + self._pending[request_id] = pending + try: + self._write( + request_envelope(request_id, method, params), + deadline=deadline, + on_writing=on_writing, + ) + if on_written is not None: + on_written() + except BaseException: + with self._pending_lock: + if self._pending.get(request_id) is pending: + self._pending.pop(request_id, None) + raise + try: + response = waiter.get(timeout=max(0.0, deadline - time.monotonic())) + except queue.Empty as exc: + with self._pending_lock: + if self._pending.get(request_id) is pending: + self._pending.pop(request_id, None) + # A response dispatcher that won the lock must enqueue before + # releasing it. Recheck while no dispatcher can still claim us. + try: + response = waiter.get_nowait() + except queue.Empty: + response = None + if response is not None: + if isinstance(response, BaseException): + raise response + return response.result_or_raise() + raise AcpRequestTimeoutError( + f"ACP request {method!r} timed out after {wait_timeout:g}s" + ) from exc + with self._pending_lock: + if self._pending.get(request_id) is pending: + self._pending.pop(request_id, None) + if isinstance(response, BaseException): + raise response + return response.result_or_raise() + + def notify(self, method: str, params: Mapping[str, Any] | None = None) -> None: + self._require_running() + self._write(notification_envelope(method, params)) + + def new_session( + self, + cwd: str | os.PathLike[str], + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[str | os.PathLike[str]] = (), + timeout: float | None = None, + ) -> SessionResult: + params = self._session_setup_params( + cwd, + mcp_servers=mcp_servers, + additional_directories=additional_directories, + ) + result = self.request("session/new", params, timeout=timeout) + raw = _require_mapping(result, "session/new result") + _validate_upstream(UpstreamNewSessionResponse, raw, "session/new result") + return _parse_session_result(raw, require_session_id=True) + + def load_session( + self, + session_id: str, + cwd: str | os.PathLike[str], + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[str | os.PathLike[str]] = (), + timeout: float | None = None, + ) -> SessionResult: + self._require_capability("loadSession") + params = self._session_setup_params( + cwd, + mcp_servers=mcp_servers, + additional_directories=additional_directories, + ) + params["sessionId"] = _nonempty(session_id, "session_id") + result = self.request("session/load", params, timeout=timeout) + # ACP v1 defines the successful load response as null after all replay + # updates have been sent. Accept a mapping as a compatibility + # extension for agents that also return initial session state. + if result is None: + return SessionResult( + session_id, + None, + (), + MappingProxyType({}), + ) + raw = _require_mapping(result, "session/load result") + _validate_upstream(UpstreamLoadSessionResponse, raw, "session/load result") + parsed = _parse_session_result(raw, require_session_id=False) + return SessionResult(session_id, parsed.modes, parsed.config_options, parsed.raw) + + def resume_session( + self, + session_id: str, + cwd: str | os.PathLike[str], + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[str | os.PathLike[str]] = (), + timeout: float | None = None, + ) -> SessionResult: + self._require_capability("sessionResume") + params = self._session_setup_params( + cwd, + mcp_servers=mcp_servers, + additional_directories=additional_directories, + ) + params["sessionId"] = _nonempty(session_id, "session_id") + result = self.request("session/resume", params, timeout=timeout) + raw = _require_mapping(result, "session/resume result") + _validate_upstream(UpstreamResumeSessionResponse, raw, "session/resume result") + parsed = _parse_session_result(raw, require_session_id=False) + return SessionResult(session_id, parsed.modes, parsed.config_options, parsed.raw) + + def list_sessions( + self, + *, + cwd: str | os.PathLike[str] | None = None, + cursor: str | None = None, + timeout: float | None = None, + ) -> SessionPage: + self._require_capability("sessionList") + params: dict[str, Any] = {} + if cwd is not None: + params["cwd"] = _absolute_path(cwd, "cwd") + if cursor is not None: + params["cursor"] = _nonempty(cursor, "cursor") + result = self.request("session/list", params, timeout=timeout) + raw = _require_mapping(result, "session/list result") + _validate_upstream(UpstreamListSessionsResponse, raw, "session/list result") + raw_sessions = raw.get("sessions") + if not isinstance(raw_sessions, list): + raise AcpEnvelopeError("session/list result.sessions must be an array") + sessions = tuple(_parse_session_info(item) for item in raw_sessions) + next_cursor = raw.get("nextCursor") + if next_cursor is not None and not isinstance(next_cursor, str): + next_cursor = None + return SessionPage(sessions, next_cursor, MappingProxyType(dict(raw))) + + def close_session( + self, session_id: str, *, timeout: float | None = None + ) -> Mapping[str, Any]: + """Close an active session when advertised by the agent.""" + self._require_capability("sessionClose") + result = self.request( + "session/close", + {"sessionId": _nonempty(session_id, "session_id")}, + timeout=timeout, + ) + return _require_mapping(result, "session/close result") + + def delete_session( + self, session_id: str, *, timeout: float | None = None + ) -> Mapping[str, Any]: + """Delete a listed session when advertised by the agent.""" + self._require_capability("sessionDelete") + result = self.request( + "session/delete", + {"sessionId": _nonempty(session_id, "session_id")}, + timeout=timeout, + ) + return _require_mapping(result, "session/delete result") + + def prompt( + self, + session_id: str, + prompt: str | Sequence[Mapping[str, Any]], + *, + timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, + on_submitted: Callable[[], None] | None = None, + ) -> PromptResult: + content = list(self.prepare_prompt(prompt)) + session_id = _nonempty(session_id, "session_id") + with self._permission_lock: + if self._active_prompts.get(session_id, 0) == 0: + # A new turn supersedes an unconfirmed cancellation from a + # prior timed-out turn. + self._cancelled_sessions.discard(session_id) + self._active_prompts[session_id] = self._active_prompts.get(session_id, 0) + 1 + response_received = False + try: + _validate_upstream( + UpstreamPromptRequest, + {"sessionId": session_id, "prompt": content}, + "session/prompt params", + ) + result = self.request( + "session/prompt", + {"sessionId": session_id, "prompt": content}, + timeout=self.prompt_timeout if timeout is None else timeout, + on_writing=on_send_start, + on_written=on_submitted, + ) + response_received = True + finally: + with self._permission_lock: + active = self._active_prompts.get(session_id, 0) - 1 + if active > 0: + self._active_prompts[session_id] = active + else: + self._active_prompts.pop(session_id, None) + if response_received: + self._cancelled_sessions.discard(session_id) + raw = _require_mapping(result, "session/prompt result") + _validate_upstream(UpstreamPromptResponse, raw, "session/prompt result") + stop_reason = raw.get("stopReason") + try: + parsed_reason = StopReason(stop_reason) + except (ValueError, TypeError) as exc: + raise AcpEnvelopeError("session/prompt returned an invalid stopReason") from exc + return PromptResult(parsed_reason, MappingProxyType(dict(raw))) + + def steer_session( + self, + session_id: str, + prompt: str | Sequence[Mapping[str, Any]], + *, + timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, + on_submitted: Callable[[], None] | None = None, + ) -> SteeringResult: + """Inject input into an active turn through an advertised extension. + + ``codex-acp`` serializes these requests per session and either injects + into the live turn or starts a new turn after the prior one drains. + The method is never used unless the initialize response opted in. + """ + + if not self.steering_supported: + raise AcpCapabilityError("agent did not advertise steering capability") + content = list(self.prepare_prompt(prompt)) + result = self.request( + "_session/steering", + { + "sessionId": _nonempty(session_id, "session_id"), + "prompt": content, + }, + timeout=timeout, + on_writing=on_send_start, + on_written=on_submitted, + ) + raw = _require_mapping(result, "_session/steering result") + try: + outcome = SteeringOutcome(raw.get("outcome")) + except (TypeError, ValueError) as exc: + raise AcpEnvelopeError( + "_session/steering returned an invalid outcome" + ) from exc + return SteeringResult(outcome, MappingProxyType(dict(raw))) + + def prepare_prompt( + self, + prompt: str | Sequence[Mapping[str, Any]], + ) -> tuple[Mapping[str, Any], ...]: + """Validate prompt content without sending it to the agent.""" + + if isinstance(prompt, str): + content: list[Mapping[str, Any]] = [{"type": "text", "text": prompt}] + else: + content = list(prompt) + if not content: + raise ValueError("prompt must contain at least one content block") + self._require_initialized() + assert self.capabilities is not None + return tuple( + _validated_prompt_content_block(block, self.capabilities) + for block in content + ) + + def cancel(self, session_id: str) -> None: + """Cancel a turn and cancel all outstanding permissions for the session.""" + session_id = _nonempty(session_id, "session_id") + with self._permission_lock: + self._cancelled_sessions.add(session_id) + try: + self.notify("session/cancel", {"sessionId": session_id}) + except BaseException: + with self._permission_lock: + self._cancelled_sessions.discard(session_id) + raise + while True: + with self._permission_lock: + pending_ids = [ + request_id + for request_id, request in self._pending_permissions.items() + if request.session_id == session_id + ] + if not pending_ids: + break + for request_id in pending_ids: + try: + self.respond_permission(request_id, cancelled=True) + except AcpClientStateError: + # A consumer may have resolved it concurrently. + continue + with self._permission_lock: + if self._active_prompts.get(session_id, 0) == 0: + self._cancelled_sessions.discard(session_id) + + def respond_permission( + self, + request_id: RequestId, + *, + option_id: str | None = None, + cancelled: bool = False, + ) -> None: + if cancelled == (option_id is not None): + raise ValueError("select exactly one of option_id or cancelled=True") + with self._permission_lock: + request = self._pending_permissions.get(request_id) + if request is None: + raise AcpClientStateError("permission request is not pending") + if option_id is not None and option_id not in { + option.option_id for option in request.options + }: + raise ValueError("option_id was not offered by this permission request") + del self._pending_permissions[request_id] + if cancelled: + outcome = {"outcome": "cancelled"} + else: + outcome = {"outcome": "selected", "optionId": option_id} + try: + self._write(result_envelope(request_id, {"outcome": outcome})) + except BaseException: + # A partial frame may have reached the peer. Retrying the same + # JSON-RPC response is unsafe; transport failure is terminal. + raise + + def next_update(self, *, timeout: float | None = None) -> SessionUpdate: + return self._next_typed_session_event(SessionUpdate, timeout, "session update") + + def next_permission_request( + self, *, timeout: float | None = None + ) -> PermissionRequest: + return self._next_typed_session_event( + PermissionRequest, + timeout, + "permission request", + ) + + def next_session_event(self, *, timeout: float | None = None) -> SessionEvent: + """Return the next update or permission request in exact reader order.""" + + return self._next_typed_session_event(SessionEvent, timeout, "session event") + + def _next_typed_session_event( + self, + expected: type[_T] | object, + timeout: float | None, + description: str, + ) -> _T: + deadline = None + if timeout is not None: + deadline = time.monotonic() + _positive_timeout(timeout, "timeout") + with self._session_event_condition: + while True: + index = self._matching_session_event_index(expected) + if index is not None: + candidate = self._session_events[index] + del self._session_events[index] + self._session_event_condition.notify_all() + return candidate # type: ignore[return-value] + if self.state in { + ClientState.FAILED, + ClientState.CLOSING, + ClientState.CLOSED, + }: + self._raise_unusable() + remaining = None if deadline is None else deadline - time.monotonic() + if remaining is not None and remaining <= 0: + raise AcpRequestTimeoutError( + f"timed out waiting for ACP {description}" + ) + self._session_event_condition.wait(timeout=remaining) + + def _matching_session_event_index(self, expected: object) -> int | None: + index = 0 + removed_stale = False + while index < len(self._session_events): + candidate = self._session_events[index] + if isinstance(candidate, PermissionRequest): + with self._permission_lock: + pending = ( + self._pending_permissions.get(candidate.request_id) + is candidate + ) + if not pending: + del self._session_events[index] + removed_stale = True + continue + if self._session_event_matches(candidate, expected): + if removed_stale: + self._session_event_condition.notify_all() + return index + index += 1 + if removed_stale: + self._session_event_condition.notify_all() + return None + + @staticmethod + def _session_event_matches(candidate: SessionEvent, expected: object) -> bool: + if expected is SessionEvent: + return True + return isinstance(candidate, expected) # type: ignore[arg-type] + + def next_notification(self, *, timeout: float | None = None) -> RawNotification: + return self._queue_get(self._notifications, timeout, "notification") + + def next_inbound_request(self, *, timeout: float | None = None) -> InboundRequest: + return self._queue_get(self._inbound_requests, timeout, "inbound request") + + def reject_inbound_request( + self, + request_id: RequestId, + *, + code: int = _METHOD_NOT_FOUND, + message: str = "Method not supported by Tendwire ACP client", + ) -> None: + self._write(error_envelope(request_id, code, message)) + + def close(self) -> None: + wait_for_other_close = False + closed_without_process = False + with self._state_lock: + if self._state in {ClientState.CLOSED, ClientState.NEW}: + self._state = ClientState.CLOSED + self._closed.set() + closed_without_process = True + elif self._state is ClientState.CLOSING: + wait_for_other_close = True + else: + was_failed = self._state is ClientState.FAILED + self._state = ClientState.CLOSING + if closed_without_process: + self._signal_queues() + return + if wait_for_other_close: + self._closed.wait(timeout=self.close_timeout * 3) + return + process = self._process + try: + if process is not None: + acquired_write = self._write_lock.acquire(timeout=self.close_timeout) + if not acquired_write: + # A blocked frame cannot be completed safely during close. + # Terminating our private process group wakes the writer. + self._signal_process(process, signal.SIGTERM) + acquired_write = self._write_lock.acquire( + timeout=self.close_timeout + ) + if not acquired_write: + self._signal_process(process, signal.SIGKILL) + acquired_write = self._write_lock.acquire( + timeout=self.close_timeout + ) + if not acquired_write: + raise AcpTransportError("timed out closing ACP agent stdin") + try: + if process.stdin is not None: + try: + process.stdin.close() + except OSError: + pass + finally: + self._write_lock.release() + try: + process.wait(timeout=self.close_timeout) + except subprocess.TimeoutExpired: + self._signal_process(process, signal.SIGTERM) + try: + process.wait(timeout=self.close_timeout) + except subprocess.TimeoutExpired: + self._signal_process(process, signal.SIGKILL) + process.wait(timeout=self.close_timeout) + # The adapter can exit while leaving a spawned agent process + # holding its inherited stdio descriptors. The private process + # group makes those descendants safe to terminate as one unit. + self._signal_process(process, signal.SIGTERM) + self._stop.set() + for thread in (self._reader_thread, self._stderr_thread): + if thread is not None and thread is not threading.current_thread(): + thread.join(timeout=self.close_timeout) + if process is not None and any( + thread is not None and thread.is_alive() + for thread in (self._reader_thread, self._stderr_thread) + ): + self._signal_process(process, signal.SIGKILL) + for thread in (self._reader_thread, self._stderr_thread): + if thread is not None and thread is not threading.current_thread(): + thread.join(timeout=self.close_timeout) + if process is not None: + self._exit = ProcessExit(process.returncode, self.stderr_tail()) + self._fail_pending(AcpTransportError("ACP client closed")) + with self._permission_lock: + self._pending_permissions.clear() + self._cancelled_sessions.clear() + self._active_prompts.clear() + self._signal_queues() + with self._state_lock: + self._state = ClientState.FAILED if was_failed else ClientState.CLOSED + except BaseException as exc: + with self._state_lock: + if self._failure is None: + self._failure = AcpTransportError( + f"failed to stop ACP agent: {type(exc).__name__}" + ) + self._state = ClientState.FAILED + self._fail_pending(self._failure) + self._signal_queues() + raise + finally: + self._stop.set() + self._closed.set() + + def _session_setup_params( + self, + cwd: str | os.PathLike[str], + *, + mcp_servers: Sequence[Mapping[str, Any]], + additional_directories: Sequence[str | os.PathLike[str]], + ) -> dict[str, Any]: + self._require_initialized() + params: dict[str, Any] = { + "cwd": _absolute_path(cwd, "cwd"), + "mcpServers": [self._validated_mcp_server(server) for server in mcp_servers], + } + directories = [ + _absolute_path(directory, "additional directory") + for directory in additional_directories + ] + if directories: + self._require_capability("additionalDirectories") + params["additionalDirectories"] = directories + return params + + def _validated_mcp_server(self, server: Mapping[str, Any]) -> dict[str, Any]: + assert self.capabilities is not None + return _validated_mcp_server(server, self.capabilities) + + def _require_capability(self, name: str) -> None: + self._require_initialized() + assert self.capabilities is not None + supported = { + "loadSession": self.capabilities.load_session, + "sessionList": self.capabilities.session_list, + "sessionResume": self.capabilities.session_resume, + "sessionClose": self.capabilities.session_close, + "sessionDelete": self.capabilities.session_delete, + "additionalDirectories": self.capabilities.additional_directories, + }.get(name, False) + if not supported: + raise AcpCapabilityError(f"agent did not advertise {name} capability") + + def _new_request_id(self) -> int: + with self._request_id_lock: + request_id = self._next_id + if request_id > 2**63 - 1: + raise AcpClientStateError("ACP request ID space exhausted") + self._next_id += 1 + return request_id + + def _write( + self, + envelope: Mapping[str, Any], + *, + deadline: float | None = None, + on_writing: Callable[[], None] | None = None, + ) -> None: + payload = encode_message(envelope, max_frame_bytes=self.max_frame_bytes) + if deadline is None: + deadline = time.monotonic() + self.request_timeout + remaining_timeout = max(0.0, deadline - time.monotonic()) + if not self._write_lock.acquire(timeout=remaining_timeout): + raise AcpRequestTimeoutError("timed out waiting to write ACP frame") + try: + self._require_running() + process = self._process + if process is None or process.stdin is None: + raise AcpTransportError("ACP agent stdin is unavailable") + try: + fd = process.stdin.fileno() + remaining = memoryview(payload) + bytes_written = 0 + write_started = False + while remaining: + wait = deadline - time.monotonic() + if wait <= 0: + raise AcpRequestTimeoutError( + "timed out writing ACP frame to agent" + ) + if not self._wait_writable(fd, wait): + raise AcpRequestTimeoutError( + "timed out writing ACP frame to agent" + ) + # Cross the durable send boundary only after the write + # lock is held and the transport reports writable. A + # timeout waiting for either condition has written zero + # bytes and remains safely retryable. The callback still + # runs before the first write, so a process crash cannot + # leave an externally visible frame without a receipt. + if not write_started: + if on_writing is not None: + on_writing() + write_started = True + try: + written = self._write_chunk(fd, remaining) + except BlockingIOError: + continue + if written <= 0: + raise BrokenPipeError("zero-byte write to ACP agent stdin") + bytes_written += written + remaining = remaining[written:] + except AcpRequestTimeoutError as exc: + if bytes_written: + failure = AcpTransportError( + "ACP frame write timed out after a partial write" + ) + self._set_failed(failure) + raise failure from exc + raise + except (BrokenPipeError, OSError) as exc: + failure = AcpTransportError("ACP agent stdin disconnected") + self._set_failed(failure) + raise failure from exc + finally: + self._write_lock.release() + + @staticmethod + def _wait_writable(fd: int, timeout: float) -> bool: + _, writable, _ = select.select([], [fd], [], timeout) + return bool(writable) + + @staticmethod + def _write_chunk(fd: int, data: memoryview) -> int: + return os.write(fd, data) + + @staticmethod + def _signal_process(process: subprocess.Popen[bytes], signum: int) -> None: + try: + if os.name == "posix": + os.killpg(process.pid, signum) + elif signum == signal.SIGTERM: + process.terminate() + else: + process.kill() + except ProcessLookupError: + pass + + def _reader_main(self) -> None: + process = self._process + assert process is not None and process.stdout is not None + buffer = bytearray() + try: + while not self._stop.is_set(): + newline = buffer.find(b"\n") + if newline < 0: + room = self.max_frame_bytes + 1 - len(buffer) + chunk = os.read(process.stdout.fileno(), min(64 * 1024, room)) + if not chunk: + if self._stop.is_set() or self.state is ClientState.CLOSING: + return + if buffer: + raise AcpFramingError("ACP stdout ended during a JSON frame") + returncode = process.poll() + suffix = f" (exit {returncode})" if returncode is not None else "" + raise AcpTransportError(f"ACP agent stdout disconnected{suffix}") + buffer.extend(chunk) + if len(buffer) > self.max_frame_bytes and b"\n" not in buffer: + raise AcpFramingError( + "ACP stdout frame exceeds configured size limit" + ) + continue + line = bytes(buffer[: newline + 1]) + del buffer[: newline + 1] + if len(line) > self.max_frame_bytes: + raise AcpFramingError("ACP stdout frame exceeds configured size limit") + message = decode_json_line( + line, + max_frame_bytes=self.max_frame_bytes, + ) + self._dispatch(message) + except BaseException as exc: + if not self._stop.is_set(): + self._set_failed(exc) + + def _stderr_main(self) -> None: + process = self._process + assert process is not None and process.stderr is not None + try: + while True: + chunk = os.read(process.stderr.fileno(), 4096) + if not chunk: + return + with self._stderr_lock: + self._stderr_chunks.append(chunk) + self._stderr_size += len(chunk) + while self._stderr_size > self.stderr_limit_bytes: + excess = self._stderr_size - self.stderr_limit_bytes + first = self._stderr_chunks[0] + if len(first) <= excess: + self._stderr_size -= len(self._stderr_chunks.popleft()) + else: + self._stderr_chunks[0] = first[excess:] + self._stderr_size -= excess + except OSError: + return + + def _dispatch( + self, message: JsonRpcRequest | JsonRpcNotification | JsonRpcResponse + ) -> None: + if isinstance(message, JsonRpcResponse): + if message.request_id is None: + # JSON-RPC uses null for errors whose request ID could not be + # recovered. We never emit null request IDs, so this cannot + # correlate to local work and is safe to ignore. + return + with self._pending_lock: + pending = self._pending.get(message.request_id) + if pending is not None: + try: + pending.waiter.put_nowait(message) + except queue.Full as exc: + raise AcpEnvelopeError( + "duplicate ACP response for one request ID" + ) from exc + if pending is None: + # Late responses are expected after a local timeout. They have + # no consumer and must not be copied into a bounded event queue. + return + return + if isinstance(message, JsonRpcNotification): + if message.method == "session/update": + self._put_session_event(parse_session_update(message.params)) + elif message.method in self.supported_extension_notifications: + self._put_lossless( + self._notifications, + RawNotification(message.method, message.params), + ) + # ACP extension notifications are one-way and unrecognized ones + # should be ignored. Protocol-level $/ notifications are also + # explicitly optional, so they take the same bounded path. + return + + if message.method == "session/request_permission": + try: + parsed = parse_permission_request(message) + except AcpProtocolError as exc: + self._write( + error_envelope( + message.request_id, + -32602, + "Invalid permission request", + data=str(exc), + ) + ) + return + with self._permission_lock: + if message.request_id in self._pending_permissions: + raise AcpEnvelopeError("duplicate pending permission request id") + cancelled = parsed.session_id in self._cancelled_sessions + if not cancelled: + self._pending_permissions[message.request_id] = parsed + if cancelled: + self._write( + result_envelope( + message.request_id, + {"outcome": {"outcome": "cancelled"}}, + ) + ) + else: + self._put_session_event(parsed) + elif message.method in self.supported_extension_requests: + self._put_lossless( + self._inbound_requests, + InboundRequest(message.request_id, message.method, message.params), + ) + else: + self._write( + error_envelope( + message.request_id, + _METHOD_NOT_FOUND, + "Method not found", + ) + ) + + def _put_lossless(self, target: queue.Queue[Any], value: Any) -> None: + try: + # Brief backpressure lets an active ordered consumer drain LOAD + # replay bursts larger than the queue. A genuinely abandoned + # queue still fails closed within a bounded interval. + target.put(value, timeout=min(self.request_timeout, 0.5)) + except queue.Full as exc: + raise AcpEventQueueFullError( + "ACP event queue is full; refusing to drop protocol data" + ) from exc + + def _put_session_event(self, value: SessionEvent) -> None: + deadline = time.monotonic() + min(self.request_timeout, 0.5) + with self._session_event_condition: + while len(self._session_events) >= self.max_pending_events: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AcpEventQueueFullError( + "ACP event queue is full; refusing to drop protocol data" + ) + self._session_event_condition.wait(timeout=remaining) + self._session_events.append(value) + self._session_event_condition.notify_all() + + def _queue_get( + self, + source: queue.Queue[_T | object], + timeout: float | None, + description: str, + ) -> _T: + deadline = None + if timeout is not None: + deadline = time.monotonic() + _positive_timeout(timeout, "timeout") + while True: + if source.empty() and self.state in { + ClientState.FAILED, + ClientState.CLOSING, + ClientState.CLOSED, + }: + self._raise_unusable() + wait = 0.1 + if deadline is not None: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AcpRequestTimeoutError( + f"timed out waiting for ACP {description}" + ) + wait = min(wait, remaining) + try: + item = source.get(timeout=wait) + except queue.Empty: + continue + if item is _END: + # Preserve terminal visibility for all future consumers. + try: + source.put_nowait(_END) + except queue.Full: + pass + self._raise_unusable() + raise AcpTransportError("ACP event stream ended") + return item # type: ignore[return-value] + + def _set_failed(self, failure: BaseException) -> None: + if not isinstance(failure, AcpClientError | AcpProtocolError): + failure = AcpTransportError(str(failure)) + with self._state_lock: + if self._state in {ClientState.CLOSING, ClientState.CLOSED}: + return + self._failure = failure + self._state = ClientState.FAILED + self._stop.set() + self._fail_pending(failure) + self._signal_queues() + + def _fail_pending(self, failure: BaseException) -> None: + with self._pending_lock: + pending_requests = tuple(self._pending.values()) + self._pending.clear() + for pending in pending_requests: + try: + pending.waiter.put_nowait(failure) + except queue.Full: + pass + + def _signal_queues(self) -> None: + with self._session_event_condition: + self._session_event_condition.notify_all() + for target in ( + self._notifications, + self._inbound_requests, + ): + try: + target.put_nowait(_END) + except queue.Full: + # A full queue already has readable data; the recorded client + # state reports terminal failure once consumers drain it. + pass + + def _require_running(self) -> None: + with self._state_lock: + if self._state not in {ClientState.RUNNING, ClientState.INITIALIZED}: + self._raise_unusable() + + def _require_initialized(self) -> None: + with self._state_lock: + if self._state is not ClientState.INITIALIZED: + if self._state in {ClientState.FAILED, ClientState.CLOSED, ClientState.CLOSING}: + self._raise_unusable() + raise AcpClientStateError("ACP client has not been initialized") + + def _raise_unusable(self) -> None: + if self._failure is not None: + raise AcpTransportError(f"ACP client failed: {self._failure}") from self._failure + raise AcpClientStateError(f"ACP client is {self._state.value}") + + +def _positive_timeout(value: float | int, name: str) -> float: + result = float(value) + if not math.isfinite(result) or result <= 0: + raise ValueError(f"{name} must be finite and positive") + return result + + +def _nonempty(value: str, name: str) -> str: + if not isinstance(value, str) or not value: + raise ValueError(f"{name} must be a non-empty string") + return value + + +def _absolute_path(value: str | os.PathLike[str], name: str) -> str: + result = os.fspath(value) + if not isinstance(result, str) or not result or "\x00" in result: + raise ValueError(f"{name} must be a non-empty text path without NUL bytes") + if not Path(result).is_absolute(): + raise ValueError(f"{name} must be an absolute path") + return result + + +def _extension_method_set(values: Sequence[str], name: str) -> frozenset[str]: + if isinstance(values, str): + raise ValueError(f"{name} must be a sequence of extension method names") + result: set[str] = set() + for value in values: + if not isinstance(value, str) or not value.startswith("_") or len(value) == 1: + raise ValueError(f"{name} entries must be ACP methods starting with '_'") + result.add(value) + return frozenset(result) + + +def _validated_prompt_content_block( + value: Mapping[str, Any], + capabilities: AgentCapabilities, +) -> dict[str, Any]: + if not isinstance(value, Mapping): + raise ValueError("each prompt content block must be an object") + kind = value.get("type") + if kind == "text": + _reject_unknown_fields(value, {"type", "text", "annotations", "_meta"}, "text content") + _string_field(value, "text", "text content") + elif kind == "image": + if not capabilities.prompt_image: + raise AcpCapabilityError("agent did not advertise prompt image capability") + _reject_unknown_fields( + value, + {"type", "data", "mimeType", "uri", "annotations", "_meta"}, + "image content", + ) + _string_field(value, "data", "image content") + _string_field(value, "mimeType", "image content") + _optional_string_field(value, "uri", "image content") + elif kind == "audio": + if not capabilities.prompt_audio: + raise AcpCapabilityError("agent did not advertise prompt audio capability") + _reject_unknown_fields( + value, + {"type", "data", "mimeType", "annotations", "_meta"}, + "audio content", + ) + _string_field(value, "data", "audio content") + _string_field(value, "mimeType", "audio content") + elif kind == "resource_link": + _reject_unknown_fields( + value, + { + "type", + "name", + "uri", + "description", + "mimeType", + "size", + "title", + "annotations", + "_meta", + }, + "resource link", + ) + _string_field(value, "name", "resource link") + _string_field(value, "uri", "resource link") + for field in ("description", "mimeType", "title"): + _optional_string_field(value, field, "resource link") + size = value.get("size") + if size is not None and ( + not isinstance(size, int) + or isinstance(size, bool) + or not -(2**63) <= size <= 2**63 - 1 + ): + raise ValueError("resource link.size must be a signed 64-bit integer") + elif kind == "resource": + if not capabilities.prompt_embedded_context: + raise AcpCapabilityError( + "agent did not advertise prompt embeddedContext capability" + ) + _reject_unknown_fields( + value, + {"type", "resource", "annotations", "_meta"}, + "embedded resource", + ) + resource = value.get("resource") + if not isinstance(resource, Mapping): + raise ValueError("embedded resource.resource must be an object") + _string_field(resource, "uri", "embedded resource payload") + _optional_string_field(resource, "mimeType", "embedded resource payload") + has_text = "text" in resource + has_blob = "blob" in resource + if has_text == has_blob: + raise ValueError( + "embedded resource payload must contain exactly one of text or blob" + ) + payload_field = "text" if has_text else "blob" + _string_field(resource, payload_field, "embedded resource payload") + _reject_unknown_fields( + resource, + {"uri", "mimeType", payload_field, "_meta"}, + "embedded resource payload", + ) + _optional_mapping_field(resource, "_meta", "embedded resource payload") + else: + raise ValueError("prompt content block type is not valid ACP v1") + + _optional_mapping_field(value, "annotations", f"{kind} content") + _optional_mapping_field(value, "_meta", f"{kind} content") + return dict(value) + + +def _validated_mcp_server( + value: Mapping[str, Any], + capabilities: AgentCapabilities, +) -> dict[str, Any]: + if not isinstance(value, Mapping): + raise ValueError("each MCP server must be an object") + transport = value.get("type") + if transport is None: + _reject_unknown_fields( + value, + {"name", "command", "args", "env", "_meta"}, + "stdio MCP server", + ) + name = _string_field(value, "name", "stdio MCP server") + command = _absolute_path( + _string_field(value, "command", "stdio MCP server"), + "stdio MCP server.command", + ) + args = _string_array_field(value, "args", "stdio MCP server") + env = _name_value_array_field(value, "env", "stdio MCP server") + result: dict[str, Any] = { + "name": name, + "command": command, + "args": args, + "env": env, + } + elif transport in {"http", "sse"}: + if transport == "http" and not capabilities.mcp_http: + raise AcpCapabilityError("agent did not advertise MCP HTTP capability") + if transport == "sse" and not capabilities.mcp_sse: + raise AcpCapabilityError("agent did not advertise MCP SSE capability") + _reject_unknown_fields( + value, + {"type", "name", "url", "headers", "_meta"}, + f"{transport} MCP server", + ) + result = { + "type": transport, + "name": _string_field(value, "name", f"{transport} MCP server"), + "url": _string_field(value, "url", f"{transport} MCP server"), + "headers": _name_value_array_field( + value, + "headers", + f"{transport} MCP server", + ), + } + else: + raise ValueError("MCP server type is not valid stable ACP v1") + meta = _optional_mapping_field(value, "_meta", "MCP server") + if meta is not None: + result["_meta"] = dict(meta) + return result + + +def _reject_unknown_fields( + value: Mapping[str, Any], allowed: set[str], label: str +) -> None: + if any(key not in allowed for key in value): + raise ValueError(f"{label} contains fields outside the ACP v1 schema") + + +def _string_field(value: Mapping[str, Any], field: str, label: str) -> str: + result = value.get(field) + if not isinstance(result, str): + raise ValueError(f"{label}.{field} must be a string") + return result + + +def _optional_string_field(value: Mapping[str, Any], field: str, label: str) -> None: + result = value.get(field) + if result is not None and not isinstance(result, str): + raise ValueError(f"{label}.{field} must be a string or null") + + +def _optional_mapping_field( + value: Mapping[str, Any], field: str, label: str +) -> Mapping[str, Any] | None: + result = value.get(field) + if result is not None and not isinstance(result, Mapping): + raise ValueError(f"{label}.{field} must be an object or null") + return result + + +def _string_array_field( + value: Mapping[str, Any], field: str, label: str +) -> list[str]: + result = value.get(field) + if not isinstance(result, list) or any(not isinstance(item, str) for item in result): + raise ValueError(f"{label}.{field} must be an array of strings") + return list(result) + + +def _name_value_array_field( + value: Mapping[str, Any], field: str, label: str +) -> list[dict[str, Any]]: + result = value.get(field) + if not isinstance(result, list): + raise ValueError(f"{label}.{field} must be an array") + normalized: list[dict[str, Any]] = [] + for item in result: + if not isinstance(item, Mapping): + raise ValueError(f"{label}.{field} entries must be objects") + _reject_unknown_fields(item, {"name", "value", "_meta"}, f"{label}.{field} entry") + normalized_item: dict[str, Any] = { + "name": _string_field(item, "name", f"{label}.{field} entry"), + "value": _string_field(item, "value", f"{label}.{field} entry"), + } + meta = _optional_mapping_field(item, "_meta", f"{label}.{field} entry") + if meta is not None: + normalized_item["_meta"] = dict(meta) + normalized.append(normalized_item) + return normalized + + +def _validated_env(env: Mapping[str, str] | None) -> dict[str, str] | None: + if env is None: + return None + result: dict[str, str] = {} + for key, value in env.items(): + if ( + not isinstance(key, str) + or not key + or "=" in key + or "\x00" in key + or not isinstance(value, str) + or "\x00" in value + ): + raise ValueError("env must contain valid string names and values") + result[key] = value + return result + + +def _require_mapping(value: Any, name: str) -> Mapping[str, Any]: + if not isinstance(value, Mapping): + raise AcpEnvelopeError(f"{name} must be an object") + return value + + +def _validate_upstream(model: Any, value: Mapping[str, Any], name: str) -> None: + """Validate one stable ACP payload with the official generated schema.""" + + try: + model.model_validate(dict(value)) + except ValidationError as exc: + raise AcpEnvelopeError( + f"{name} does not match the upstream ACP schema" + ) from exc + + +def _parse_session_result( + raw: Mapping[str, Any], *, require_session_id: bool +) -> SessionResult: + session_id = raw.get("sessionId", "") + if require_session_id and (not isinstance(session_id, str) or not session_id): + raise AcpEnvelopeError("session/new result.sessionId must be a non-empty string") + modes_value = raw.get("modes") + modes = ( + MappingProxyType(dict(modes_value)) if isinstance(modes_value, Mapping) else None + ) + config_value = raw.get("configOptions", []) + config_options = tuple( + MappingProxyType(dict(item)) + for item in config_value + if isinstance(item, Mapping) + ) if isinstance(config_value, list) else () + return SessionResult( + session_id=session_id if isinstance(session_id, str) else "", + modes=modes, + config_options=config_options, + raw=MappingProxyType(dict(raw)), + ) + + +def _parse_session_info(value: Any) -> SessionInfo: + raw = _require_mapping(value, "session info") + session_id = raw.get("sessionId") + cwd = raw.get("cwd") + if not isinstance(session_id, str) or not session_id: + raise AcpEnvelopeError("session info.sessionId must be a non-empty string") + if not isinstance(cwd, str) or not Path(cwd).is_absolute(): + raise AcpEnvelopeError("session info.cwd must be an absolute path") + directories_value = raw.get("additionalDirectories", []) + directories = tuple( + item + for item in directories_value + if isinstance(item, str) and Path(item).is_absolute() + ) if isinstance(directories_value, list) else () + title = raw.get("title") + updated_at = raw.get("updatedAt") + return SessionInfo( + session_id=session_id, + cwd=cwd, + additional_directories=directories, + title=title if isinstance(title, str) else None, + updated_at=updated_at if isinstance(updated_at, str) else None, + raw=MappingProxyType(dict(raw)), + ) diff --git a/src/tendwire/backends/acp_coordinator.py b/src/tendwire/backends/acp_coordinator.py new file mode 100644 index 0000000..35f57a8 --- /dev/null +++ b/src/tendwire/backends/acp_coordinator.py @@ -0,0 +1,2777 @@ +"""Production Herdr-to-ACP worker discovery and runtime coordination. + +Herdr remains the authority for live worker identity and mints one-shot attach +endpoints. This module validates that private contract, supervises one ACP +runtime per worker generation, and exposes opaque prompt routes to the daemon. +No endpoint ticket, process argv, cwd, adapter identity, or ACP session ID is +part of the public health surface. +""" + +from __future__ import annotations + +import json +import math +import threading +import time +from collections import Counter +from collections.abc import Callable, Mapping, Sequence +from contextlib import contextmanager +from concurrent.futures import Future, ThreadPoolExecutor +from dataclasses import dataclass, field, replace +from pathlib import Path +from typing import Any + +from ..config import Config +from ..core.models import ( + BackendHealth, + Space, + Worker, + WorkerBinding, + normalize_status, + separate_duplicate_worker_bindings, + utc_timestamp, + worker_binding_private_fingerprint, +) +from ..core.commands import turn_submission_id +from ..core.models import stable_fingerprint +from ..store.sqlite import ( + expire_worker_bindings, + latest_snapshot, + list_agent_events, + list_worker_bindings, + pending_payload_from_store, + record_agent_event, + save_snapshot, + upsert_worker_bindings, +) +from ..core.projector import project_from_observations +from ..worker_identity import ( + STABLE_KEY_VERSION, + canonical_herdr_pane_identity, + load_or_create_installation_key, + stable_worker_key, +) +from .acp_client import BoundedAcpConnection +from .acp_permissions import AcpPermissionBroker +from .acp_runtime import ( + AcpWorkerSession, + PermissionCallback, + RuntimeState, + SessionOpenMode, +) +from .herdr_protocol import HerdrErrorResponse +from .herdr_socket import HerdrSocketClient + + +class AcpCoordinatorError(RuntimeError): + """The private Herdr ACP endpoint contract or supervisor failed.""" + + +class AcpPermissionBridgeUnavailable(AcpCoordinatorError): + """Production ACP cannot authorize tools without a durable user decision.""" + + +class AcpConsoleInputGap(AcpCoordinatorError): + """The bounded Herdr console queue lost unconsumed pane input.""" + + def __init__(self, message: str, *, recovery_after_sequence: int) -> None: + super().__init__(message) + self.recovery_after_sequence = recovery_after_sequence + + +class AcpVisibleConsoleUnavailable(AcpCoordinatorError): + """A worker cannot accept new prompts while its pane bridge is lost.""" + + +# Herdr's aggregate console queue admission is 768 KiB and its normal control +# frame is 2 MiB. Keep Tendwire's retained output contribution below 512 KiB +# so queued pane input and the echoed response retain explicit headroom. +_CONSOLE_OUTPUT_QUEUE_BUDGET_BYTES = 512 * 1024 +_CONSOLE_OUTPUT_ITEM_TEXT_BYTES = 128 * 1024 +# Three idle ACP panes previously produced roughly 30 control exchanges per +# second on the Raspberry Pi. A 500 ms cadence keeps pane input/output latency +# sub-second while leaving enough server capacity for delivery and health API +# traffic. Active prompts continue independently inside their ACP runtimes. +_CONSOLE_BRIDGE_INTERVAL_SECONDS = 0.5 + + +@dataclass(frozen=True, slots=True) +class HerdrAcpEndpoint: + command: tuple[str, ...] + cwd: Path + generation: str + session_mode: SessionOpenMode + session_id: str | None + console: HerdrAcpConsoleEndpoint | None = None + + +@dataclass(frozen=True, slots=True) +class HerdrAcpStatus: + generation: str + lifecycle: str + console_lifecycle: str + + +@dataclass(frozen=True, slots=True) +class HerdrAcpConsoleEndpoint: + generation: int + lease: str + + +@dataclass(slots=True) +class _SessionSlot: + continuity: WorkerBinding + generation: str + runtime: AcpWorkerSession + permission_broker: AcpPermissionBroker | None = None + console: HerdrAcpConsoleEndpoint | None = None + console_input_sequence: int = 0 + console_event_sequence: int = 0 + console_cursor_loaded: bool = False + console_local_turns: set[str] | None = None + console_executor: ThreadPoolExecutor | None = None + console_submissions: dict[int, Future[Any]] | None = None + console_failures: int = 0 + lock: threading.RLock = field(default_factory=threading.RLock, repr=False) + retired: bool = False + console_bridge_thread: threading.Thread | None = None + + +class _PromptRoute: + def __init__( + self, + owner: "AcpSupervisor", + worker: Worker, + slot: _SessionSlot, + ) -> None: + self._owner = owner + self._worker = worker + self._slot = slot + self._prepared = threading.local() + + @property + def binding_fingerprint(self) -> str: + return self._owner._route_binding_fingerprint(self._worker, self._slot) + + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Callable[[], None] | None = None, + ) -> object: + return self._owner._submit_prompt( + self._worker, + self._slot, + text, + producer_turn_id=producer_turn_id, + acknowledgement_timeout=timeout, + on_send_start=on_send_start, + generation_prepared=bool( + getattr(self._prepared, "depth", 0) + ), + ) + + @property + def supports_steering(self) -> bool: + return self._owner._supports_steering(self._worker, self._slot) + + def steer( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Callable[[], None] | None = None, + ) -> object: + return self._owner._submit_steering( + self._worker, + self._slot, + text, + producer_turn_id=producer_turn_id, + acknowledgement_timeout=timeout, + on_send_start=on_send_start, + generation_prepared=bool(getattr(self._prepared, "depth", 0)), + ) + + @contextmanager + def prepare(self): + """Fence one exact generation before its durable send receipt exists.""" + + with self._owner._reconcile_lock: + self._owner._require_reconcile_state(allow_starting=False) + if self._owner._current_slot(self._worker) is not self._slot: + raise AcpCoordinatorError("ACP worker route is stale") + self._owner._require_attached_generation(self._slot) + if self._owner._current_slot(self._worker) is not self._slot: + raise AcpCoordinatorError("ACP worker route is stale") + depth = int(getattr(self._prepared, "depth", 0)) + self._prepared.depth = depth + 1 + try: + yield self + finally: + self._prepared.depth = depth + + +EndpointClientFactory = Callable[[Config], Any] +WorkerSessionFactory = Callable[..., AcpWorkerSession] +ConnectionFactory = Callable[..., BoundedAcpConnection] + + +class AcpSupervisor: + """Reconcile Herdr endpoint ownership into per-worker ACP sessions.""" + + def __init__( + self, + config: Config, + stop_event: threading.Event, + *, + endpoint_client_factory: EndpointClientFactory | None = None, + discovery_client_factory: EndpointClientFactory | None = None, + session_factory: WorkerSessionFactory = AcpWorkerSession, + connection_factory: ConnectionFactory = BoundedAcpConnection, + reconcile_interval: float | None = None, + permission_callback: PermissionCallback | None = None, + require_permission_bridge: bool = False, + durable_permission_bridge: bool = False, + ) -> None: + if config.db_path is None: + raise ValueError("ACP coordinator requires a sqlite db path") + if durable_permission_bridge and permission_callback is not None: + raise ValueError( + "durable ACP permission bridge cannot use an embedding callback" + ) + self.config = config + self._daemon_stop = stop_event + self._endpoint_client_factory = ( + endpoint_client_factory or _default_endpoint_client_factory + ) + self._discovery_client_factory = discovery_client_factory + if self._discovery_client_factory is None and endpoint_client_factory is None: + self._discovery_client_factory = _default_endpoint_client_factory + self._session_factory = session_factory + self._connection_factory = connection_factory + self._permission_callback = permission_callback + self._require_permission_bridge = bool(require_permission_bridge) + self._durable_permission_bridge = bool(durable_permission_bridge) + resolved_reconcile_interval = float( + config.reconcile_interval_seconds + if reconcile_interval is None + else reconcile_interval + ) + if not math.isfinite(resolved_reconcile_interval) or resolved_reconcile_interval <= 0: + raise ValueError("reconcile_interval must be finite and positive") + self._reconcile_interval = resolved_reconcile_interval + self._lock = threading.RLock() + # Endpoint minting, runtime publication, prompt lease validation, and + # shutdown are one private generation transaction. Herdr + # tickets are one-shot, so overlapping reconciles cannot be repaired + # after the fact by selecting whichever runtime happened to attach. + self._reconcile_lock = threading.RLock() + self._stop = threading.Event() + self._slots: dict[str, _SessionSlot] = {} + self._retired_slots: list[_SessionSlot] = [] + self._thread: threading.Thread | None = None + self._console_thread: threading.Thread | None = None + self._state = RuntimeState.NEW + self._failure_type: str | None = None + self._required_degraded = False + self._console_degraded = False + self._console_failure_type: str | None = None + self._console_failed_workers: set[str] = set() + self._console_failed_claims: dict[str, str] = {} + # Exact ACP ownership survives runtime retirement so an outage remains + # distinguishable from a worker that Herdr has actually removed. + self._published_acp_claims: dict[str, str] = {} + self._last_discovery_at: str | None = None + self._worker_count = 0 + + def start(self) -> "AcpSupervisor": + with self._lock: + if self._state is RuntimeState.RUNNING: + return self + if self._state is not RuntimeState.NEW: + raise AcpCoordinatorError("ACP coordinator cannot be restarted") + if ( + self._require_permission_bridge + and self._permission_callback is None + and not self._durable_permission_bridge + ): + self._state = RuntimeState.FAILED + self._failure_type = "AcpPermissionBridgeUnavailable" + raise AcpPermissionBridgeUnavailable( + "durable ACP permission decisions are not configured" + ) + self._state = RuntimeState.STARTING + try: + # ACP adapter processes cannot survive this coordinator process. + # Revoke any process-owned rows left by an unclean prior exit before + # a fresh Herdr generation is allowed to attach. + self._expire_orphaned_bindings() + self._reconcile(strict=True) + except Exception as exc: + with self._lock: + self._state = RuntimeState.FAILED + self._failure_type = type(exc).__name__ + self._stop_all() + raise + with self._lock: + if self._state is not RuntimeState.STARTING or self._stop.is_set(): + self._state = RuntimeState.STOPPED + raise AcpCoordinatorError("ACP coordinator is stopping") + self._state = RuntimeState.RUNNING + thread = threading.Thread( + target=self._run, + name="tendwire-acp-coordinator", + daemon=True, + ) + self._thread = thread + thread.start() + console_thread = threading.Thread( + target=self._run_console_bridge, + name="tendwire-acp-console-bridge", + daemon=True, + ) + self._console_thread = console_thread + console_thread.start() + return self + + def stop(self, *, timeout: float | None = None) -> None: + limit = ( + self.config.acp_shutdown_timeout_seconds + if timeout is None + else float(timeout) + ) + if limit <= 0: + raise ValueError("stop timeout must be positive") + with self._lock: + if self._state is RuntimeState.STOPPED: + return + self._state = RuntimeState.STOPPING + slots = tuple(self._slots.values()) + tuple(self._retired_slots) + self._stop.set() + # A durable permission answer keeps the generation fence until the + # complete JSON-RPC response frame is written. Wake any broker waiters + # before waiting for that fence: otherwise a slow/stuck adapter write + # can hold ``_reconcile_lock`` for the request timeout (normally much + # longer than the coordinator's bounded shutdown deadline). Closing a + # broker is fail-closed; an answer that has not observed a complete + # frame becomes uncertain and releases the fence without selecting a + # second route. + for slot in slots: + if slot.permission_broker is not None: + slot.permission_broker.close() + deadline = time.monotonic() + limit + # Wait for endpoint mint/start or prompt lease validation to leave its + # critical section before clearing slots. A reconcile that observed + # STOPPING stops its provisional runtime instead of publishing it. + acquired = self._reconcile_lock.acquire(timeout=limit) + if not acquired: + with self._lock: + self._failure_type = "AcpCoordinatorError" + raise AcpCoordinatorError( + "ACP coordinator reconciliation did not stop within the deadline" + ) + try: + self._stop_all(timeout=max(0.001, deadline - time.monotonic())) + finally: + self._reconcile_lock.release() + thread = self._thread + if thread is not None and thread is not threading.current_thread(): + thread.join(timeout=max(0.0, deadline - time.monotonic())) + console_thread = self._console_thread + if console_thread is not None and console_thread is not threading.current_thread(): + console_thread.join(timeout=max(0.0, deadline - time.monotonic())) + unfinished = bool( + (thread is not None and thread.is_alive()) + or (console_thread is not None and console_thread.is_alive()) + ) + for slot in slots: + with slot.lock: + bridge_thread = slot.console_bridge_thread + futures = tuple((slot.console_submissions or {}).values()) + executor = slot.console_executor + if bridge_thread is not None and bridge_thread is not threading.current_thread(): + bridge_thread.join(timeout=max(0.0, deadline - time.monotonic())) + unfinished = unfinished or bridge_thread.is_alive() + slot_unfinished = False + for future in futures: + if future.done(): + continue + try: + future.result(timeout=max(0.0, deadline - time.monotonic())) + except Exception: + pass + slot_unfinished = slot_unfinished or not future.done() + unfinished = unfinished or slot_unfinished + if not slot_unfinished and executor is not None: + # The executor was already asked to shut down by retirement; + # once every task is terminal this join is immediate and + # ensures no non-daemon worker survives a successful stop. + executor.shutdown(wait=True, cancel_futures=True) + with self._lock: + if unfinished: + self._state = RuntimeState.FAILED + self._failure_type = "AcpRuntimeStopTimeout" + elif self._state is not RuntimeState.FAILED: + self._state = RuntimeState.STOPPED + + def join(self, timeout: float | None = None) -> bool: + deadline = None if timeout is None else time.monotonic() + timeout + threads = (self._thread, self._console_thread) + for thread in threads: + if thread is None or thread is threading.current_thread(): + continue + remaining = ( + None + if deadline is None + else max(0.0, deadline - time.monotonic()) + ) + thread.join(timeout=remaining) + return all( + thread is None + or thread is threading.current_thread() + or not thread.is_alive() + for thread in threads + ) + + def status(self) -> dict[str, Any]: + counters = { + "updates_ingested": 0, + "permissions_ingested": 0, + "permissions_selected": 0, + "permissions_cancelled": 0, + "invalid_permission_selections": 0, + "prompts_started": 0, + "prompts_completed": 0, + "prompts_failed": 0, + "cancellation_requests": 0, + } + with self._lock: + state = self._state + slots = tuple(self._slots.values()) + failure_type = self._failure_type + required_degraded = self._required_degraded + console_degraded = self._console_degraded + console_failure_type = self._console_failure_type + failure_type = failure_type or console_failure_type + healthy = ( + state is RuntimeState.RUNNING + and not required_degraded + and not console_degraded + ) + for slot in slots: + try: + status = slot.runtime.status() + except Exception as exc: # noqa: BLE001 + healthy = False + failure_type = failure_type or type(exc).__name__ + continue + healthy = healthy and status.healthy + for field in counters: + counters[field] += max(0, int(getattr(status, field, 0))) + failure_type = failure_type or status.failure_type + return { + "state": state.value, + "healthy": healthy, + "failure_type": failure_type, + "last_reconcile_at": self._last_discovery_at, + "worker_count": self._worker_count, + **counters, + } + + def prompt_route(self, worker: Worker) -> _PromptRoute | None: + try: + slot = self._current_slot(worker) + except AcpVisibleConsoleUnavailable: + # Console loss is a deliberate hard fence, not a stale runtime + # that reconciliation should remint synchronously on this command. + return None + except AcpCoordinatorError: + # A just-observed worker may not have reached the periodic pass. + try: + self._reconcile_worker(worker.id, strict=False) + slot = self._current_slot(worker) + except AcpVisibleConsoleUnavailable: + return None + except Exception: # noqa: BLE001 + return None + return _PromptRoute(self, worker, slot) + + def owns_permission_decision(self, decision: Any) -> bool: + """Return whether one pending decision belongs to an exact live slot.""" + worker_id = str(getattr(decision, "worker_id", "") or "") + with self._lock: + slot = self._slots.get(worker_id) + if slot is None or slot.permission_broker is None: + return False + try: + self._require_attached_generation(slot) + return slot.permission_broker.owns(decision) + except Exception: + return False + + def answer_permission_decision(self, decision: Any, *, timeout: float) -> None: + """Select an offered option and wait for its response-frame write.""" + worker_id = str(getattr(decision, "worker_id", "") or "") + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + with self._lock: + slot = self._slots.get(worker_id) + if slot is None or slot.permission_broker is None: + raise AcpCoordinatorError("ACP permission route is unavailable") + self._require_attached_generation(slot) + if not slot.permission_broker.owns(decision): + raise AcpCoordinatorError("ACP permission authority changed") + # Keep retirement fenced until respond_permission has acknowledged + # writing the complete JSON-RPC response frame. + slot.permission_broker.answer(decision, timeout=timeout) + + def _current_slot(self, worker: Worker) -> _SessionSlot: + with self._lock: + if self._state is not RuntimeState.RUNNING: + raise AcpCoordinatorError("ACP coordinator is not running") + slot = self._slots.get(worker.id) + console_failed = worker.id in self._console_failed_workers + if slot is None: + raise AcpCoordinatorError("ACP worker route is unavailable") + if console_failed: + # The visible pane is part of the required transport, not an + # optional observer. Fence every newly resolved route (and every + # use of a route resolved before the failure) on the first bridge + # loss. The in-flight ACP turn itself may continue to drain, but + # pane and Telegram callers cannot start another headless turn. + raise AcpVisibleConsoleUnavailable( + "ACP worker visible console is unavailable" + ) + if slot.continuity.worker_fingerprint != worker.fingerprint: + raise AcpCoordinatorError("ACP worker authority is stale") + if not slot.runtime.status().healthy: + raise AcpCoordinatorError("ACP worker runtime is unhealthy") + return slot + + def _run(self) -> None: + while not self._stop.wait(self._reconcile_interval): + if self._daemon_stop.is_set(): + return + try: + self._reconcile(strict=False) + except Exception as exc: # noqa: BLE001 + with self._lock: + self._failure_type = type(exc).__name__ + + def _run_console_bridge(self) -> None: + while not self._stop.wait(_CONSOLE_BRIDGE_INTERVAL_SECONDS): + if self._daemon_stop.is_set(): + return + self._bridge_console_slots() + + def _bridge_console_slots(self) -> None: + """Dispatch independent visible pane bridges without head-of-line blocking.""" + with self._lock: + self._retired_slots = [ + slot for slot in self._retired_slots if _slot_has_live_work(slot) + ] + slots = tuple(self._slots.values()) + for slot in slots: + if slot.console is None: + continue + with slot.lock: + if slot.retired: + continue + thread = slot.console_bridge_thread + if thread is not None and thread.is_alive(): + continue + thread = threading.Thread( + target=self._bridge_console_slot_supervised, + args=(slot,), + name=f"tendwire-acp-console-bridge-{slot.continuity.worker_id}", + daemon=True, + ) + slot.console_bridge_thread = thread + thread.start() + + def _bridge_console_slot_supervised(self, slot: _SessionSlot) -> None: + worker_id = slot.continuity.worker_id + try: + self._bridge_console_slot(slot) + with slot.lock: + if slot.retired: + return + slot.console_failures = 0 + with self._lock: + # A superseded bridge must not clear the fence for its + # replacement. Recovery requires a successful pass by the + # exact slot currently published for this worker. + if self._slots.get(worker_id) is not slot: + return + self._console_failed_workers.discard(worker_id) + self._console_failed_claims.pop(worker_id, None) + self._console_degraded = bool(self._console_failed_workers) + if not self._console_degraded: + self._console_failure_type = None + except Exception as exc: + with slot.lock: + if slot.retired: + return + slot.console_failures += 1 + failure_count = slot.console_failures + # Console availability is supervised independently of the ACP + # runtime. A later pass replays inputs and idempotent outputs. + with self._lock: + self._console_failed_workers.add(worker_id) + self._console_failed_claims[worker_id] = ( + slot.continuity.worker_fingerprint + ) + self._published_acp_claims[worker_id] = ( + slot.continuity.worker_fingerprint + ) + self._console_degraded = True + self._console_failure_type = type(exc).__name__ + if failure_count >= 3: + try: + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + self._retire_worker( + worker_id, + expected=slot, + preserve_console_failure=True, + ) + current, _ambiguities = self._continuity_bindings() + continuity = current.get(worker_id) + if continuity is None: + raise AcpCoordinatorError( + "worker has no unique Herdr authority" + ) + self._reconcile_binding(continuity) + except Exception: + # Keep the worker in the persistent failure set until a + # replacement slot completes a successful console pass. + pass + + def _bridge_console_slot(self, slot: _SessionSlot) -> None: + with slot.lock: + if slot.retired: + return + console = slot.console + if console is None or self.config.db_path is None: + return + binding = getattr(slot.runtime, "_binding", None) + if not isinstance(binding, WorkerBinding): + raise AcpCoordinatorError("ACP console session binding is unavailable") + with slot.lock: + cursor_loaded = slot.console_cursor_loaded + if not cursor_loaded: + persisted_cursor = _load_console_event_cursor( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + binding.turn_target_value, + ) + loaded_cursor = ( + persisted_cursor + if persisted_cursor is not None + else _initial_console_event_cursor( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + binding.turn_target_value, + ) + ) + with slot.lock: + if slot.retired: + return + slot.console_event_sequence = loaded_cursor + slot.console_cursor_loaded = True + with slot.lock: + event_sequence = slot.console_event_sequence + input_sequence = slot.console_input_sequence + submissions = slot.console_submissions or {} + slot.console_submissions = submissions + events = list_agent_events( + Path(self.config.db_path), + self.config.host_id, + worker_id=slot.continuity.worker_id, + source="acp", + session_id=binding.turn_target_value, + after_sequence=event_sequence, + limit=100, + ) + # Snapshot suppression after the event page. A console submission + # registers its deterministic turn id before it can emit the ACP user + # event, so this ordering closes the race where the bridge could copy + # an old set and then observe the newly emitted event in the same page. + with slot.lock: + if slot.retired: + return + local_turns = set(slot.console_local_turns or ()) + output: list[dict[str, str]] = [] + completed_submissions: list[int] = [] + for sequence, future in tuple(submissions.items()): + if not future.done(): + continue + completed_submissions.append(sequence) + try: + outcome = future.result() + durable_outcome = str(outcome) + except Exception: + # Keep the durable payload deterministic across a retry of the + # same request id; exception types can differ across a crash. + durable_outcome = "error" + _record_console_submission_outcome( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + binding.turn_target_value, + generation=console.generation, + input_sequence=sequence, + outcome=durable_outcome, + ) + # A terminal command receipt now exists (accepted or rejected), so + # the next exchange may acknowledge this one Herdr queue item. + record_agent_event( + Path(self.config.db_path), + self.config.host_id, + kind="extension", + source="tendwire-console", + worker_id=slot.continuity.worker_id, + payload={ + "extension": "tendwire.acp.console_input_cursor", + "generation": console.generation, + "input_sequence": sequence, + }, + source_session_id=binding.turn_target_value, + source_event_id=f"input:{console.generation}:{sequence}", + visibility="private", + ) + input_sequence = max(input_sequence, sequence) + pending = _console_pending_decision( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + ) + if pending is not None: + decision_ref, _options, prompt = pending + output.append( + _bounded_console_output( + "permission:" + stable_fingerprint( + { + "worker_id": slot.continuity.worker_id, + "decision_ref": decision_ref, + } + ), + "status", + prompt, + ) + ) + consumed_local_turns: set[str] = set() + processed_event_sequence = event_sequence + for stored in events: + event = stored.event + if event.kind == "user_message" and event.source_turn_id in local_turns: + if event.source_turn_id is not None: + consumed_local_turns.add(event.source_turn_id) + processed_event_sequence = stored.sequence + continue + rendered = _console_event_output(event.kind, event.payload) + if rendered is None: + processed_event_sequence = stored.sequence + continue + stream, text = rendered + item = _bounded_console_output(event.event_id, stream, text) + if not _console_output_fits(output, item, budget=_CONSOLE_OUTPUT_QUEUE_BUDGET_BYTES): + break + output.append(item) + processed_event_sequence = stored.sequence + client = self._endpoint_client_factory(self.config) + gap_error: AcpConsoleInputGap | None = None + inputs: tuple[tuple[int, str], ...] = () + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + # Probe the retained Herdr output queue before publishing. Herdr + # echoes that queue in its response, so publishing blind can create + # a response larger than its 2 MiB control-frame limit. + result = client.agent_acp_console_exchange( + slot.continuity.target_value, + generation=console.generation, + lease=console.lease, + after_input_sequence=input_sequence, + output=(), + timeout=self.config.herdr_timeout_seconds, + ) + inputs = _parse_console_exchange(result, input_sequence) + retained_bytes = _console_exchange_output_bytes(result) + publish = _fit_console_output_batch( + output, + budget=max(0, _CONSOLE_OUTPUT_QUEUE_BUDGET_BYTES - retained_bytes), + ) + if len(publish) < len(output): + # Only ACP events after this prefix must be replayed. Synthetic + # permission output is first and idempotent, so a full queue may + # delay it without advancing any ACP event cursor. + published_ids = {item["event_id"] for item in publish} + processed_event_sequence = event_sequence + consumed_local_turns.clear() + for stored in events: + event = stored.event + if event.kind == "user_message" and event.source_turn_id in local_turns: + if event.source_turn_id is not None: + consumed_local_turns.add(event.source_turn_id) + processed_event_sequence = stored.sequence + continue + rendered = _console_event_output(event.kind, event.payload) + if rendered is None: + processed_event_sequence = stored.sequence + continue + if event.event_id not in published_ids: + break + processed_event_sequence = stored.sequence + if publish: + result = client.agent_acp_console_exchange( + slot.continuity.target_value, + generation=console.generation, + lease=console.lease, + after_input_sequence=input_sequence, + output=publish, + timeout=self.config.herdr_timeout_seconds, + ) + inputs = _parse_console_exchange(result, input_sequence) + except AcpConsoleInputGap as exc: + gap_error = exc + finally: + close = getattr(client, "close", None) + if callable(close): + close() + if gap_error is not None: + if self.config.acp_console_input_policy == "live_only": + # The operator explicitly chose live traffic over recovery of + # a partially discarded backlog. Persist the tail watermark + # before acknowledging it, then make one fresh exchange so an + # input arriving after the probe is still delivered. + input_sequence = max( + input_sequence, + gap_error.recovery_after_sequence, + ) + record_agent_event( + Path(self.config.db_path), + self.config.host_id, + kind="extension", + source="tendwire-console", + worker_id=slot.continuity.worker_id, + payload={ + "extension": "tendwire.acp.console_input_cursor", + "generation": console.generation, + "input_sequence": input_sequence, + "recovery": "live_only", + }, + source_session_id=binding.turn_target_value, + source_event_id=( + f"input-live-baseline:{console.generation}:{input_sequence}" + ), + visibility="private", + ) + client = self._endpoint_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + result = client.agent_acp_console_exchange( + slot.continuity.target_value, + generation=console.generation, + lease=console.lease, + after_input_sequence=input_sequence, + output=(), + timeout=self.config.herdr_timeout_seconds, + ) + inputs = _parse_console_exchange(result, input_sequence) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + gap_error = None + if gap_error is not None: + gap_output = [{ + "event_id": f"console-gap:{console.generation}:{input_sequence}", + "stream": "error", + "text": "console input backlog overflowed; restart the pane console before retrying", + }] + client = self._endpoint_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + client.agent_acp_console_exchange( + slot.continuity.target_value, + generation=console.generation, + lease=console.lease, + after_input_sequence=input_sequence, + output=gap_output, + timeout=self.config.herdr_timeout_seconds, + ) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + raise gap_error + with slot.lock: + if slot.retired: + return + slot.console_input_sequence = input_sequence + for sequence in completed_submissions: + submissions.pop(sequence, None) + if processed_event_sequence > event_sequence: + # Herdr deduplicates output event_id values, so replaying a page + # after a crash between exchange and this cursor update is safe. + next_event_sequence = processed_event_sequence + record_agent_event( + Path(self.config.db_path), + self.config.host_id, + kind="extension", + source="tendwire-console", + worker_id=slot.continuity.worker_id, + payload={ + "extension": "tendwire.acp.console_cursor", + "sequence": next_event_sequence, + }, + source_session_id=binding.turn_target_value, + source_event_id=f"cursor:{next_event_sequence}", + visibility="private", + ) + with slot.lock: + if slot.retired: + return + slot.console_event_sequence = next_event_sequence + if slot.console_local_turns is not None: + slot.console_local_turns.difference_update(consumed_local_turns) + for sequence, text in inputs[:1]: + with slot.lock: + if slot.retired or sequence in submissions: + continue + executor = slot.console_executor + if executor is None: + raise AcpCoordinatorError("ACP console submission worker is unavailable") + submissions[sequence] = executor.submit( + self._submit_console_input, slot, sequence, text + ) + + def _submit_console_input( + self, slot: _SessionSlot, sequence: int, text: str + ) -> str: + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + with self._lock: + current = self._slots.get(slot.continuity.worker_id) + with slot.lock: + retired = slot.retired + if current is not slot or retired: + raise AcpCoordinatorError("ACP console input generation is stale") + return self._submit_console_input_fenced(slot, sequence, text) + + def _submit_console_input_fenced( + self, slot: _SessionSlot, sequence: int, text: str + ) -> str: + snapshot = latest_snapshot(Path(self.config.db_path), self.config.host_id) + worker = next( + ( + item + for item in (snapshot.workers if snapshot is not None else ()) + if item.id == slot.continuity.worker_id + and item.fingerprint == slot.continuity.worker_fingerprint + ), + None, + ) + stable_key = worker.meta.get("stable_key") if worker is not None else None + stable_version = ( + worker.meta.get("stable_key_version") if worker is not None else None + ) + if not isinstance(stable_key, str) or stable_version != 1: + raise AcpCoordinatorError("ACP console stable worker identity is unavailable") + request_id = "acpc." + stable_fingerprint( + { + "generation": slot.generation, + "input_sequence": sequence, + "worker_id": slot.continuity.worker_id, + } + ) + if text.strip() == "/cancel": + slot.runtime.cancel() + return "cancelled" + request: dict[str, Any] = { + "schema_version": 1, + "response_schema_version": 3, + "action": "send_instruction", + "request_id": request_id, + "dry_run": False, + "target": { + "stable_key": stable_key, + "stable_key_version": 1, + }, + "instruction": {"text": text}, + } + pending = _console_pending_decision( + Path(self.config.db_path), + self.config.host_id, + slot.continuity.worker_id, + ) + if pending is not None: + decision_ref, options, _prompt = pending + selected = _console_permission_selection(text, options) + if selected is None: + raise AcpCoordinatorError( + "console input is not a valid permission selection" + ) + request.pop("instruction", None) + request["action"] = "answer_decision" + request["params"] = { + "decision_ref": decision_ref, + "selection": {"option_refs": [selected]}, + } + from ..command_submission import submit_command + + local_turn_id: str | None = None + if request["action"] == "send_instruction": + submission_id = turn_submission_id(self.config.host_id, request_id) + session_id = str(getattr(slot.runtime, "_session_id", "") or "") + if session_id: + local_turn_id = "acpt_" + stable_fingerprint( + { + "source": "acp", + "session": session_id, + "producer_turn": submission_id, + } + ) + # Register before submit_command can emit the ACP user event; + # the independent bridge thread may observe it immediately. + with slot.lock: + if slot.retired: + raise AcpCoordinatorError( + "ACP console input generation is stale" + ) + local_turns = slot.console_local_turns + if local_turns is None: + local_turns = set() + slot.console_local_turns = local_turns + local_turns.add(local_turn_id) + try: + result = submit_command( + self.config, + json.dumps(request, sort_keys=True, separators=(",", ":")), + acp_prompt_router=self.prompt_route, + acp_permission_router=self, + ) + except Exception: + if local_turn_id is not None: + with slot.lock: + if slot.console_local_turns is not None: + slot.console_local_turns.discard(local_turn_id) + raise + if result.status not in {"accepted", "duplicate_request"}: + if local_turn_id is not None: + with slot.lock: + if slot.console_local_turns is not None: + slot.console_local_turns.discard(local_turn_id) + raise AcpCoordinatorError("ACP console input was not accepted") + if request["action"] == "answer_decision": + return "permission" + if result.status == "duplicate_request" and local_turn_id is not None: + # A duplicate receipt does not start a new ACP turn, so there is no + # corresponding user event left to suppress in this process. + with slot.lock: + if slot.console_local_turns is not None: + slot.console_local_turns.discard(local_turn_id) + return "instruction" + + def _require_reconcile_state(self, *, allow_starting: bool) -> None: + with self._lock: + allowed = {RuntimeState.RUNNING} + if allow_starting: + allowed.add(RuntimeState.STARTING) + if self._state not in allowed or self._stop.is_set(): + raise AcpCoordinatorError("ACP coordinator is stopping") + + def _continuity_bindings(self) -> tuple[dict[str, WorkerBinding], int]: + bindings = list_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="herdr", + ) + grouped: dict[str, list[WorkerBinding]] = {} + for binding in bindings: + if binding.sendable and binding.target_kind in { + "agent_id", + "agent", + "name", + "label", + "terminal_id", + "pane_id", + }: + grouped.setdefault(binding.worker_id, []).append(binding) + current = { + worker_id: rows[0] + for worker_id, rows in grouped.items() + if len(rows) == 1 + } + ambiguities = sum(1 for rows in grouped.values() if len(rows) != 1) + return current, ambiguities + + def _herdr_authority_claims(self) -> set[tuple[str, str]]: + """Return exact sendable identities without requiring unique routing.""" + + return { + (binding.worker_id, binding.worker_fingerprint) + for binding in list_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="herdr", + ) + if binding.sendable + } + + def _reconcile(self, *, strict: bool) -> None: + with self._reconcile_lock: + try: + self._require_reconcile_state(allow_starting=True) + self._reconcile_locked(strict=strict) + finally: + if self._stop.is_set(): + self._stop_all() + + def _reconcile_locked(self, *, strict: bool) -> None: + if self._discovery_client_factory is not None: + try: + self._discover_continuity() + except Exception as exc: + with self._lock: + self._required_degraded = True + self._failure_type = type(exc).__name__ + if strict: + raise + return + current, ambiguities = self._continuity_bindings() + with self._lock: + failed_claims = tuple(self._console_failed_claims.items()) + published_claims = tuple(self._published_acp_claims.items()) + exact_authorities = ( + self._herdr_authority_claims() + if failed_claims or published_claims + else set() + ) + with self._lock: + stale = [worker_id for worker_id in self._slots if worker_id not in current] + disappeared_failures = [ + worker_id + for worker_id, fingerprint in failed_claims + if (worker_id, fingerprint) not in exact_authorities + and worker_id not in self._slots + and self._console_failed_claims.get(worker_id) == fingerprint + ] + for worker_id in disappeared_failures: + self._console_failed_workers.discard(worker_id) + failed_fingerprint = self._console_failed_claims.pop(worker_id, None) + if self._published_acp_claims.get(worker_id) == failed_fingerprint: + self._published_acp_claims.pop(worker_id, None) + disappeared_published = [ + worker_id + for worker_id, fingerprint in published_claims + if (worker_id, fingerprint) not in exact_authorities + and worker_id not in self._slots + and self._published_acp_claims.get(worker_id) == fingerprint + ] + for worker_id in disappeared_published: + self._published_acp_claims.pop(worker_id, None) + self._console_degraded = bool(self._console_failed_workers) + if not self._console_degraded: + self._console_failure_type = None + for worker_id in stale: + self._retire_worker(worker_id) + failures: list[BaseException] = [ + AcpCoordinatorError("worker has ambiguous Herdr authority") + for _ in range(ambiguities) + ] + for worker_id, continuity in current.items(): + self._require_reconcile_state(allow_starting=True) + try: + self._reconcile_binding(continuity) + except Exception as exc: # noqa: BLE001 + failures.append(exc) + self._retire_worker(worker_id) + with self._lock: + self._required_degraded = bool(failures) + if failures: + self._failure_type = type(failures[0]).__name__ + elif not self._required_degraded: + self._failure_type = None + if strict and failures: + raise AcpCoordinatorError("one or more ACP workers failed to attach") + + def _discover_continuity(self) -> None: + """Refresh the one Herdr lifecycle projection consumed by ACP routing.""" + + client = self._discovery_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + workspaces = client.workspace_list(timeout=self.config.herdr_timeout_seconds) + panes = client.pane_list(timeout=self.config.herdr_timeout_seconds) + agents = client.agent_list(timeout=self.config.herdr_timeout_seconds) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + + observed_at = utc_timestamp() + spaces = _discovered_spaces(workspaces) + prior_bindings = list_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="herdr", + ) + workers, bindings = _discovered_workers( + self.config, + panes, + agents, + observed_at, + prior_bindings=prior_bindings, + ) + health = BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty" if spaces or workers else "empty_healthy", + observed_at=observed_at, + counts={"spaces": len(spaces), "workers": len(workers)}, + ) + snapshot = project_from_observations( + self.config, + spaces=spaces, + workers=workers, + backend_health=[health], + ) + from ..store.sqlite import SnapshotObservationContext + + save_snapshot( + Path(self.config.db_path), + snapshot, + observation=SnapshotObservationContext( + authority="complete", + observed_at=observed_at, + ), + worker_bindings=bindings, + binding_backend="herdr", + binding_observation_authoritative=True, + binding_workers_present=bool(workers), + ) + with self._lock: + self._last_discovery_at = observed_at + self._worker_count = len(workers) + + def _reconcile_worker(self, worker_id: str, *, strict: bool) -> None: + with self._reconcile_lock: + try: + self._require_reconcile_state(allow_starting=False) + current, _ambiguities = self._continuity_bindings() + continuity = current.get(worker_id) + if continuity is None: + with self._lock: + failed_fingerprint = self._console_failed_claims.get(worker_id) + published_fingerprint = self._published_acp_claims.get( + worker_id + ) + claimed_fingerprint = failed_fingerprint or published_fingerprint + authority_remains = True + if claimed_fingerprint is not None: + try: + authority_remains = ( + worker_id, + claimed_fingerprint, + ) in self._herdr_authority_claims() + except Exception: + # A failed ownership check cannot prove that the + # Herdr endpoint disappeared; retry periodically. + authority_remains = True + with self._lock: + if worker_id not in self._slots and not authority_remains: + self._console_failed_workers.discard(worker_id) + self._console_failed_claims.pop(worker_id, None) + self._published_acp_claims.pop(worker_id, None) + self._console_degraded = bool( + self._console_failed_workers + ) + if not self._console_degraded: + self._console_failure_type = None + if strict: + raise AcpCoordinatorError( + "worker has no unique Herdr authority" + ) + return + self._reconcile_binding(continuity) + finally: + if self._stop.is_set(): + self._stop_all() + + def _reconcile_binding(self, continuity: WorkerBinding) -> None: + with self._lock: + existing = self._slots.get(continuity.worker_id) + if ( + existing is not None + and _same_continuity(existing.continuity, continuity) + and existing.runtime.status().healthy + ): + try: + self._require_attached_generation(existing) + return + except Exception: + self._retire_worker(continuity.worker_id, expected=existing) + existing = None + if existing is not None: + self._retire_worker(continuity.worker_id, expected=existing) + endpoint = self._resolve_endpoint(continuity) + runtime, permission_broker = self._build_runtime(continuity, endpoint) + try: + runtime.start() + except Exception: + if permission_broker is not None: + permission_broker.close() + try: + runtime.stop(timeout=self.config.acp_shutdown_timeout_seconds) + except Exception: + pass + raise + try: + self._require_reconcile_state(allow_starting=True) + except Exception: + if permission_broker is not None: + permission_broker.close() + self._stop_runtime(runtime) + raise + runtime_binding = getattr(runtime, "_binding", None) + console_cursor = 0 + console_input_cursor = 0 + console_cursor_loaded = False + if ( + endpoint.console is not None + and isinstance(runtime_binding, WorkerBinding) + and runtime_binding.turn_target_value + ): + console_cursor = _prepare_console_event_cursor( + Path(self.config.db_path), + self.config.host_id, + continuity.worker_id, + runtime_binding.turn_target_value, + session_mode=endpoint.session_mode, + generation=endpoint.console.generation, + ) + console_cursor_loaded = True + console_input_cursor = _load_console_input_cursor( + Path(self.config.db_path), + self.config.host_id, + continuity.worker_id, + runtime_binding.turn_target_value, + endpoint.console.generation, + ) + slot = _SessionSlot( + continuity, + endpoint.generation, + runtime, + permission_broker, + console=endpoint.console, + console_input_sequence=console_input_cursor, + console_event_sequence=console_cursor, + console_cursor_loaded=console_cursor_loaded, + console_local_turns=set(), + console_executor=ThreadPoolExecutor( + max_workers=1, + thread_name_prefix=f"tendwire-acp-console-{continuity.worker_id}", + ), + console_submissions={}, + ) + with self._lock: + displaced = self._slots.get(continuity.worker_id) + self._slots[continuity.worker_id] = slot + self._published_acp_claims[continuity.worker_id] = ( + continuity.worker_fingerprint + ) + if displaced is not None: + self._stop_runtime(displaced.runtime) + + def _resolve_endpoint(self, continuity: WorkerBinding) -> HerdrAcpEndpoint: + client = self._endpoint_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + result = client.agent_acp_endpoint( + continuity.target_value, + timeout=self.config.herdr_timeout_seconds, + ) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + return _parse_endpoint(self.config, continuity, result) + + def _resolve_status(self, continuity: WorkerBinding) -> HerdrAcpStatus: + client = self._endpoint_client_factory(self.config) + try: + connect = getattr(client, "connect", None) + if callable(connect): + connect() + result = client.agent_acp_status( + continuity.target_value, + timeout=self.config.herdr_timeout_seconds, + ) + finally: + close = getattr(client, "close", None) + if callable(close): + close() + return _parse_status(continuity, result) + + def _require_attached_generation(self, slot: _SessionSlot) -> None: + try: + status = self._resolve_status(slot.continuity) + except Exception: + self._retire_worker(slot.continuity.worker_id, expected=slot) + raise + if ( + status.lifecycle != "acp_owned_attached" + or status.console_lifecycle != "attached" + or status.generation != slot.generation + ): + self._retire_worker(slot.continuity.worker_id, expected=slot) + raise AcpCoordinatorError("ACP worker generation lease is not current") + + def _submit_prompt( + self, + worker: Worker, + slot: _SessionSlot, + text: str, + *, + producer_turn_id: str, + acknowledgement_timeout: float, + on_send_start: Callable[[], None] | None = None, + generation_prepared: bool = False, + ) -> object: + """Write through the exact route generation used by the receipt.""" + + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + current = self._current_slot(worker) + if current is not slot: + raise AcpCoordinatorError("ACP worker route is stale") + if not generation_prepared: + self._require_attached_generation(slot) + if self._current_slot(worker) is not slot: + raise AcpCoordinatorError("ACP worker route is stale") + # The route lease covers the complete JSON-RPC request frame, not + # just status validation. Retirement can proceed as soon as the + # runtime acknowledges that the frame is written; turn completion + # remains supervised asynchronously by the runtime. + return slot.runtime.submit_prompt( + text, + producer_turn_id=producer_turn_id, + acknowledgement_timeout=acknowledgement_timeout, + on_send_start=on_send_start, + ) + + def _supports_steering(self, worker: Worker, slot: _SessionSlot) -> bool: + try: + return self._current_slot(worker) is slot and slot.runtime.can_steer() + except Exception: + return False + + def _submit_steering( + self, + worker: Worker, + slot: _SessionSlot, + text: str, + *, + producer_turn_id: str, + acknowledgement_timeout: float, + on_send_start: Callable[[], None] | None = None, + generation_prepared: bool = False, + ) -> object: + """Steer the exact attached generation used by the receipt.""" + + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + current = self._current_slot(worker) + if current is not slot: + raise AcpCoordinatorError("ACP worker route is stale") + if not generation_prepared: + self._require_attached_generation(slot) + if self._current_slot(worker) is not slot or not slot.runtime.can_steer(): + raise AcpCoordinatorError("ACP steering route is unavailable") + return slot.runtime.submit_steering( + text, + producer_turn_id=producer_turn_id, + acknowledgement_timeout=acknowledgement_timeout, + on_send_start=on_send_start, + ) + + def _route_binding_fingerprint( + self, + worker: Worker, + slot: _SessionSlot, + ) -> str: + """Return authority only while this exact route remains current.""" + + with self._reconcile_lock: + self._require_reconcile_state(allow_starting=False) + if self._current_slot(worker) is not slot: + raise AcpCoordinatorError("ACP worker route is stale") + binding = getattr(slot.runtime, "_binding", None) + return ( + str(binding.private_fingerprint) + if isinstance(binding, WorkerBinding) + else "" + ) + + def _build_runtime( + self, + continuity: WorkerBinding, + endpoint: HerdrAcpEndpoint, + ) -> tuple[AcpWorkerSession, AcpPermissionBroker | None]: + client = self._connection_factory( + endpoint.command, + cwd=endpoint.cwd, + request_timeout=self.config.acp_request_timeout_seconds, + prompt_timeout=float(self.config.submission_hard_ttl_seconds), + close_timeout=self.config.acp_shutdown_timeout_seconds, + max_frame_bytes=self.config.acp_max_frame_bytes, + ) + if endpoint.session_mode is SessionOpenMode.NEW: + binding = continuity + callback = self._bind_new_session + else: + assert endpoint.session_id is not None + binding = _derived_binding(continuity, endpoint.session_id) + upsert_worker_bindings(Path(self.config.db_path), [binding]) + callback = None + permission_broker = ( + AcpPermissionBroker( + self.config, + worker_id=continuity.worker_id, + worker_fingerprint=continuity.worker_fingerprint, + generation=endpoint.generation, + timeout=float(self.config.submission_hard_ttl_seconds), + ) + if self._durable_permission_bridge + else None + ) + try: + runtime = self._session_factory( + client, + config=self.config, + binding=binding, + cwd=endpoint.cwd, + session_mode=endpoint.session_mode, + session_id=endpoint.session_id, + # Herdr's generation authenticates the worker lease and can + # remain stable across several freshly minted adapter + # transports. AcpWorkerSession deliberately creates a new stream + # nonce when this argument is omitted; reusing the Herdr + # generation would make synthetic notification identities + # collide after a Tendwire restart. + session_binding_callback=callback, + permission_callback=(permission_broker or self._permission_callback), + poll_timeout=min(0.25, self.config.acp_request_timeout_seconds), + stop_timeout=self.config.acp_shutdown_timeout_seconds, + ) + return runtime, permission_broker + except Exception: + if permission_broker is not None: + permission_broker.close() + try: + client.close() + except Exception: + pass + if endpoint.session_mode is not SessionOpenMode.NEW: + _expire_derived_binding( + self.config, + binding, + reason="acp_runtime_construction_failed", + ) + raise + + def _bind_new_session( + self, + session_id: str, + continuity: WorkerBinding, + ) -> WorkerBinding: + bound = _derived_binding(continuity, session_id) + upsert_worker_bindings(Path(self.config.db_path), [bound]) + return bound + + def _retire_worker( + self, + worker_id: str, + *, + expected: _SessionSlot | None = None, + preserve_console_failure: bool = False, + ) -> None: + with self._reconcile_lock: + with self._lock: + slot = self._slots.get(worker_id) + if slot is None or (expected is not None and slot is not expected): + return + self._slots.pop(worker_id, None) + self._retired_slots.append(slot) + # A visible-console failure is an exact sticky ownership + # claim. Retirement, ambiguity, and failed reminting must not + # erase endpoint ownership while Herdr still publishes that + # identity. Only a successful current console pass or the + # positive-disappearance path in reconciliation may remove it. + if ( + not preserve_console_failure + and worker_id not in self._console_failed_claims + ): + self._console_failed_workers.discard(worker_id) + self._console_degraded = bool(self._console_failed_workers) + if not self._console_degraded: + self._console_failure_type = None + with slot.lock: + slot.retired = True + executor = slot.console_executor + if slot.permission_broker is not None: + slot.permission_broker.close() + if executor is not None: + executor.shutdown(wait=False, cancel_futures=True) + self._stop_runtime(slot.runtime) + + def _stop_runtime( + self, runtime: AcpWorkerSession, *, timeout: float | None = None + ) -> None: + binding = getattr(runtime, "_binding", None) + try: + runtime.stop( + timeout=( + self.config.acp_shutdown_timeout_seconds + if timeout is None + else timeout + ) + ) + except Exception: + pass + finally: + if isinstance(binding, WorkerBinding) and binding.backend == "acp": + _expire_derived_binding( + self.config, + binding, + reason="acp_runtime_retired", + ) + + def _expire_orphaned_bindings(self) -> None: + """Revoke ACP leases that no runtime in this process can own.""" + + expire_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="acp", + now=utc_timestamp(), + reason="acp_coordinator_restarted", + ) + + def _stop_all(self, *, timeout: float | None = None) -> None: + with self._lock: + slots = tuple(self._slots.values()) + self._slots.clear() + if not slots: + return + total = ( + self.config.acp_shutdown_timeout_seconds + if timeout is None + else timeout + ) + deadline = time.monotonic() + total + for slot in slots: + # Bridge-held slot critical sections are deliberately local and + # non-blocking; synchronize the retirement flag with every reader + # so shutdown cannot race a last console mutation or submission. + with slot.lock: + slot.retired = True + executor = slot.console_executor + if slot.permission_broker is not None: + slot.permission_broker.close() + if executor is not None: + executor.shutdown(wait=False, cancel_futures=True) + self._stop_runtime( + slot.runtime, + timeout=max(0.001, deadline - time.monotonic()), + ) + + +def _slot_has_live_work(slot: _SessionSlot) -> bool: + with slot.lock: + thread = slot.console_bridge_thread + futures = tuple((slot.console_submissions or {}).values()) + return bool( + (thread is not None and thread.is_alive()) + or any(not future.done() for future in futures) + ) + + +def _default_endpoint_client_factory(config: Config) -> HerdrSocketClient: + return HerdrSocketClient(timeout=config.herdr_timeout_seconds) + + +def _derived_binding( + continuity: WorkerBinding, + session_id: str, +) -> WorkerBinding: + return replace( + continuity, + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value=session_id, + # This row is established by the current Tendwire process, not by the + # Herdr observation that supplied continuity. Reusing the observer's + # timestamp can lose the upsert to a newer expired row left by the + # previous process and make an otherwise valid restart fail closed. + observed_at=utc_timestamp(), + # The ACP runtime owns this private lease until explicit stop/failure. + # Inheriting the observer's short Herdr lease would strand a healthy + # attached runtime after the next observation-expiry boundary. + expires_at=None, + private_fingerprint="", + ) + + +def _expire_derived_binding( + config: Config, + binding: WorkerBinding, + *, + reason: str, +) -> None: + if config.db_path is None: # pragma: no cover - coordinator invariant + return + expire_worker_bindings( + Path(config.db_path), + binding.host_id, + backend="acp", + private_fingerprints=[binding.private_fingerprint], + now=binding.observed_at, + reason=reason, + ) + + +def _same_continuity(left: WorkerBinding, right: WorkerBinding) -> bool: + """Compare authority identity while ignoring observation lease refreshes.""" + return ( + left.host_id, + left.worker_id, + left.worker_fingerprint, + left.backend, + left.target_kind, + left.target_value, + left.private_fingerprint, + left.sendable, + ) == ( + right.host_id, + right.worker_id, + right.worker_fingerprint, + right.backend, + right.target_kind, + right.target_value, + right.private_fingerprint, + right.sendable, + ) + + +def _field(item: Mapping[str, Any], name: str) -> Any: + expected = name.replace("_", "").lower() + for key, value in item.items(): + if str(key).replace("_", "").replace("-", "").lower() == expected: + return value + return None + + +def _text(item: Mapping[str, Any], *names: str) -> str | None: + for name in names: + value = _field(item, name) + if isinstance(value, (str, int)) and not isinstance(value, bool): + text = str(value).strip() + if text: + return text + return None + + +def _items(payload: Any, *names: str) -> list[dict[str, Any]]: + if isinstance(payload, list): + values = payload + elif isinstance(payload, Mapping): + values = [] + for name in (*names, "data", "payload"): + candidate = _field(payload, name) + if isinstance(candidate, list): + values = candidate + break + if isinstance(candidate, Mapping): + nested = _items(candidate, *names) + if nested: + return nested + else: + return [] + return [dict(value) for value in values if isinstance(value, Mapping)] + + +def _discovered_spaces(payload: Any) -> list[Space]: + spaces: list[Space] = [] + for item in _items(payload, "workspaces", "spaces", "items", "result"): + space_id = _text(item, "workspace_id", "space_id", "id", "name") + if space_id is None: + continue + spaces.append( + Space( + id=space_id, + name=_text(item, "label", "name", "title") or space_id, + status=normalize_status(_text(item, "status", "state")), + updated_at=_text(item, "updated_at", "observed_at"), + ) + ) + return spaces + + +def _agent_match( + pane: Mapping[str, Any], + agents: Sequence[Mapping[str, Any]], +) -> tuple[Mapping[str, Any] | None, bool]: + pane_id = _text(pane, "pane_id") + terminal_id = _text(pane, "terminal_id") + matches = [ + agent + for agent in agents + if (pane_id and _text(agent, "pane_id") == pane_id) + or (terminal_id and _text(agent, "terminal_id") == terminal_id) + ] + return (matches[0], False) if len(matches) == 1 else (None, len(matches) > 1) + + +def _assign_prior_worker_ids( + rows: list[dict[str, Any]], + prior_bindings: Sequence[WorkerBinding], +) -> None: + private_counts = Counter(str(row["private_fingerprint"]) for row in rows) + target_key_counts = Counter( + (str(row["target_kind"]), str(row["target_value"])) for row in rows + ) + stored_by_private: dict[str, list[WorkerBinding]] = {} + stored_by_target: dict[tuple[str, str], list[WorkerBinding]] = {} + for binding in prior_bindings: + if ( + binding.backend != "herdr" + or not binding.worker_id + or binding.reason in {"duplicate_backend_target", "ambiguous_pane_match"} + ): + continue + stored_by_private.setdefault(binding.private_fingerprint, []).append(binding) + stored_by_target.setdefault( + (binding.target_kind, binding.target_value), [] + ).append(binding) + for row in rows: + prior: WorkerBinding | None = None + private_fingerprint = str(row["private_fingerprint"]) + private_candidates = stored_by_private.get(private_fingerprint, []) + if private_counts[private_fingerprint] == 1 and len(private_candidates) == 1: + prior = private_candidates[0] + target_key = (str(row["target_kind"]), str(row["target_value"])) + target_candidates = stored_by_target.get(target_key, []) + if prior is None and target_key_counts[target_key] == 1 and len(target_candidates) == 1: + prior = target_candidates[0] + row["desired_id"] = prior.worker_id if prior is not None else row["base_id"] + + +def _materialize_discovered_workers( + config: Config, + rows: list[dict[str, Any]], + observed_at: str, +) -> tuple[list[Worker], list[WorkerBinding]]: + target_counts = Counter(str(row["target_value"]) for row in rows) + id_counts = Counter(str(row["desired_id"]) for row in rows) + workers: list[Worker] = [] + bindings: list[WorkerBinding] = [] + duplicate_indexes: dict[str, int] = {} + for row in sorted(rows, key=lambda value: str(value["private_fingerprint"])): + base_id = str(row["desired_id"]) + worker_id = base_id + if id_counts[base_id] > 1: + duplicate_indexes[base_id] = duplicate_indexes.get(base_id, 0) + 1 + worker_id = f"{base_id}-{duplicate_indexes[base_id]}" + pane = row["pane"] + agent = row["agent"] + meta: dict[str, Any] = {} + label = _text(pane, "label") + if label: + meta["label"] = label + stable_key = row["stable_key"] + if stable_key is not None: + meta["stable_key"] = stable_key + meta["stable_key_version"] = STABLE_KEY_VERSION + reason = row["reason"] + sendable = reason is None and target_counts[str(row["target_value"])] == 1 + if not sendable and reason is None: + reason = "duplicate_backend_target" + worker = Worker( + id=worker_id, + name=( + _text(agent, "agent", "name") + or _text(pane, "agent", "label", "name") + or worker_id + ), + status=normalize_status( + _text(agent, "status", "agent_status") + or _text(pane, "agent_status", "status") + ), + space_id=_text(pane, "workspace_id"), + meta=meta, + last_seen_at=observed_at, + backend_target={ + "kind": row["target_kind"], + "value": row["target_value"], + "sendable": sendable, + "reason": reason, + }, + ) + workers.append(worker) + bindings.append( + WorkerBinding( + host_id=config.host_id, + worker_id=worker.id, + worker_fingerprint=worker.fingerprint, + backend="herdr", + target_kind=str(row["target_kind"]), + target_value=str(row["target_value"]), + sendable=sendable, + reason=reason, + observed_at=observed_at, + expires_at=None, + private_fingerprint=str(row["private_fingerprint"]), + ) + ) + separated = separate_duplicate_worker_bindings(bindings) + final_workers: list[Worker] = [] + final_bindings: list[WorkerBinding] = [] + for worker, binding in zip(workers, separated, strict=True): + if worker.backend_target != binding.backend_target(): + worker = replace(worker, backend_target=binding.backend_target()) + binding = replace(binding, worker_fingerprint=worker.fingerprint) + final_workers.append(worker) + final_bindings.append(binding) + return final_workers, final_bindings + + +def _discovered_workers( + config: Config, + pane_payload: Any, + agent_payload: Any, + observed_at: str, + *, + prior_bindings: Sequence[WorkerBinding] = (), +) -> tuple[list[Worker], list[WorkerBinding]]: + panes = _items(pane_payload, "panes", "items", "result") + agents = _items(agent_payload, "agents", "workers", "items", "result") + rows: list[dict[str, Any]] = [] + needs_stable_key = False + for pane in panes: + agent, ambiguous_agent = _agent_match(pane, agents) + if agent is None and not ambiguous_agent and not _text(pane, "agent", "name", "label"): + continue + workspace_id = _text(pane, "workspace_id") + pane_id = _text(pane, "pane_id") + identity = canonical_herdr_pane_identity(workspace_id, pane_id) + needs_stable_key = needs_stable_key or identity is not None + target_kind = "" + target_value = "" + for kind, value in ( + ("agent_id", _text(agent or {}, "agent_id")), + ("terminal_id", _text(pane, "terminal_id")), + ("pane_id", pane_id), + ): + if value: + target_kind, target_value = kind, value + break + if not target_value: + continue + base_id = ( + _text(agent or {}, "agent", "name") + or _text(pane, "agent", "name", "label") + or "worker" + ) + private_fingerprint = worker_binding_private_fingerprint( + host_id=config.host_id, + backend="herdr", + identity_material={ + "workspace_id": workspace_id, + "pane_id": pane_id, + "terminal_id": _text(pane, "terminal_id"), + "agent_id": _text(agent or {}, "agent_id"), + }, + ) + rows.append( + { + "agent": agent or {}, + "base_id": base_id, + "identity": identity, + "pane": pane, + "private_fingerprint": private_fingerprint, + "target_kind": target_kind, + "target_value": target_value, + "reason": ( + "ambiguous_pane_match" + if ambiguous_agent + else ("invalid_pane_identity" if identity is None else None) + ), + } + ) + + installation_key = ( + load_or_create_installation_key(config.data_dir) if needs_stable_key else None + ) + for row in rows: + identity = row["identity"] + if installation_key is not None and identity is not None: + workspace_id, pane_id = identity + row["stable_key"] = stable_worker_key( + installation_key, + backend="herdr", + host_id=config.host_id, + workspace_id=workspace_id, + pane_id=pane_id, + ) + else: + row["stable_key"] = None + _assign_prior_worker_ids(rows, prior_bindings) + return _materialize_discovered_workers(config, rows, observed_at) + + +def _nonempty_text(value: Any, field: str) -> str: + if not isinstance(value, str) or not value or value.strip() != value: + raise AcpCoordinatorError(f"Herdr ACP endpoint {field} is invalid") + if len(value) > 4096 or "\x00" in value: + raise AcpCoordinatorError(f"Herdr ACP endpoint {field} is invalid") + return value + + +def _require_target_identity( + continuity: WorkerBinding, + worker: Mapping[str, Any], + *, + response: str, +) -> None: + """Prove the returned worker still owns the requested Herdr target.""" + + direct_fields = { + "terminal_id": "terminal_id", + "pane_id": "pane_id", + "name": "name", + "label": "name", + "agent": "agent", + } + field = direct_fields.get(continuity.target_kind) + if field is not None: + matched = worker.get(field) == continuity.target_value + else: + # Older Herdr projections can call the terminal identity `agent_id`. + # The v1 endpoint contract has no agent_id member, so require that the + # authority token still matches one of its immutable identity fields. + matched = continuity.target_value in { + worker.get("terminal_id"), + worker.get("pane_id"), + worker.get("name"), + worker.get("agent"), + } + if not matched: + raise AcpCoordinatorError(f"Herdr ACP {response} target changed") + + +def _parse_endpoint( + config: Config, + continuity: WorkerBinding, + value: Any, +) -> HerdrAcpEndpoint: + """Strictly validate the private, one-shot Herdr launch contract.""" + if not isinstance(value, Mapping) or set(value) != { + "type", + "endpoint", + "console", + "worker", + "adapter", + "session", + "cwd", + "lifecycle", + }: + raise AcpCoordinatorError("Herdr ACP endpoint response shape is invalid") + if ( + value.get("type") != "agent_acp_endpoint" + or value.get("lifecycle") != "acp_owned_ready" + ): + raise AcpCoordinatorError("Herdr ACP endpoint is not ready") + endpoint = value.get("endpoint") + console = value.get("console") + worker = value.get("worker") + adapter = value.get("adapter") + session = value.get("session") + if not all( + isinstance(item, Mapping) + for item in (endpoint, console, worker, adapter, session) + ): + raise AcpCoordinatorError("Herdr ACP endpoint nested shape is invalid") + assert isinstance(endpoint, Mapping) + assert isinstance(console, Mapping) + assert isinstance(worker, Mapping) + assert isinstance(adapter, Mapping) + assert isinstance(session, Mapping) + if set(endpoint) != {"transport", "command", "args", "protocol_version"}: + raise AcpCoordinatorError("Herdr ACP launch specification is invalid") + if ( + endpoint.get("transport") != "stdio" + or type(endpoint.get("protocol_version")) is not int + or endpoint.get("protocol_version") != 1 + ): + raise AcpCoordinatorError("Herdr ACP launch protocol is unsupported") + command = _nonempty_text(endpoint.get("command"), "command") + if command != "herdr" or Path(config.herdr_bin).name != "herdr": + raise AcpCoordinatorError("Herdr ACP endpoint executable is not configured Herdr") + raw_generation = worker.get("generation") + if ( + type(raw_generation) is not int + or raw_generation < 0 + or raw_generation > (1 << 64) - 1 + ): + raise AcpCoordinatorError("Herdr ACP endpoint generation is invalid") + generation = str(raw_generation) + if set(console) != {"generation", "lease"}: + raise AcpCoordinatorError("Herdr ACP console endpoint shape is invalid") + if console.get("generation") != raw_generation: + raise AcpCoordinatorError("Herdr ACP console generation is inconsistent") + console_lease = _nonempty_text(console.get("lease"), "console lease") + if len(console_lease) > 512: + raise AcpCoordinatorError("Herdr ACP console lease is invalid") + args = endpoint.get("args") + if ( + not isinstance(args, list) + or len(args) != 7 + or args[0:2] != ["agent", "acp-attach"] + or args[3] != "--generation" + or args[5] != "--ticket" + ): + raise AcpCoordinatorError("Herdr ACP attach arguments are invalid") + target = _nonempty_text(args[2], "target") + if target != continuity.target_value: + raise AcpCoordinatorError("Herdr ACP endpoint target changed") + if _nonempty_text(args[4], "argument generation") != generation: + raise AcpCoordinatorError("Herdr ACP endpoint generation is inconsistent") + _nonempty_text(args[6], "ticket") + if set(worker) != { + "terminal_id", + "workspace_id", + "tab_id", + "pane_id", + "name", + "agent", + "generation", + }: + raise AcpCoordinatorError("Herdr ACP worker identity shape is invalid") + _require_target_identity(continuity, worker, response="endpoint") + for field in ("terminal_id", "workspace_id", "tab_id", "name", "agent"): + _nonempty_text(worker.get(field), field) + _nonempty_text(worker.get("pane_id"), "pane_id") + if set(adapter) != {"name", "version"}: + raise AcpCoordinatorError("Herdr ACP adapter identity shape is invalid") + _nonempty_text(adapter.get("name"), "adapter name") + _nonempty_text(adapter.get("version"), "adapter version") + if set(session) not in ({"mode"}, {"id", "mode"}): + raise AcpCoordinatorError("Herdr ACP session shape is invalid") + try: + mode = SessionOpenMode(session.get("mode")) + except (TypeError, ValueError) as exc: + raise AcpCoordinatorError("Herdr ACP session mode is invalid") from exc + session_id_value = session.get("id") + if mode is SessionOpenMode.NEW: + if "id" in session and session_id_value is not None: + raise AcpCoordinatorError("new Herdr ACP endpoint already has a session") + session_id = None + else: + session_id = _nonempty_text(session_id_value, "session id") + cwd = Path(_nonempty_text(value.get("cwd"), "cwd")) + if not cwd.is_absolute(): + raise AcpCoordinatorError("Herdr ACP cwd must be absolute") + return HerdrAcpEndpoint( + command=( + config.herdr_bin, + *(_nonempty_text(item, "argument") for item in args), + ), + cwd=cwd, + generation=generation, + session_mode=mode, + session_id=session_id, + console=HerdrAcpConsoleEndpoint(raw_generation, console_lease), + ) + + +def _parse_status( + continuity: WorkerBinding, + value: Any, +) -> HerdrAcpStatus: + """Validate Herdr's non-ticketing ACP generation/lease status.""" + if not isinstance(value, Mapping) or set(value) != { + "type", + "worker", + "adapter", + "session", + "cwd", + "lifecycle", + "console_lifecycle", + }: + raise AcpCoordinatorError("Herdr ACP status response shape is invalid") + lifecycle = value.get("lifecycle") + console_lifecycle = value.get("console_lifecycle") + if value.get("type") != "agent_acp_status" or lifecycle not in { + "acp_owned_ready", + "acp_owned_attached", + }: + raise AcpCoordinatorError("Herdr ACP status lifecycle is invalid") + if console_lifecycle not in {"starting", "attached", "missing"}: + raise AcpCoordinatorError("Herdr ACP console lifecycle is invalid") + worker = value.get("worker") + adapter = value.get("adapter") + session = value.get("session") + if not all(isinstance(item, Mapping) for item in (worker, adapter, session)): + raise AcpCoordinatorError("Herdr ACP status nested shape is invalid") + assert isinstance(worker, Mapping) + assert isinstance(adapter, Mapping) + assert isinstance(session, Mapping) + if set(worker) != { + "terminal_id", + "workspace_id", + "tab_id", + "pane_id", + "name", + "agent", + "generation", + }: + raise AcpCoordinatorError("Herdr ACP status worker shape is invalid") + raw_generation = worker.get("generation") + if ( + type(raw_generation) is not int + or raw_generation < 0 + or raw_generation > (1 << 64) - 1 + ): + raise AcpCoordinatorError("Herdr ACP status generation is invalid") + for field in ( + "terminal_id", + "workspace_id", + "tab_id", + "pane_id", + "name", + "agent", + ): + _nonempty_text(worker.get(field), field) + _require_target_identity(continuity, worker, response="status") + if set(adapter) != {"name", "version"}: + raise AcpCoordinatorError("Herdr ACP status adapter shape is invalid") + _nonempty_text(adapter.get("name"), "adapter name") + _nonempty_text(adapter.get("version"), "adapter version") + if set(session) not in ({"mode"}, {"id", "mode"}): + raise AcpCoordinatorError("Herdr ACP status session shape is invalid") + try: + mode = SessionOpenMode(session.get("mode")) + except (TypeError, ValueError) as exc: + raise AcpCoordinatorError("Herdr ACP status session mode is invalid") from exc + if mode is SessionOpenMode.NEW: + if "id" in session and session.get("id") is not None: + raise AcpCoordinatorError("new Herdr ACP status already has a session") + else: + _nonempty_text(session.get("id"), "session id") + cwd = Path(_nonempty_text(value.get("cwd"), "cwd")) + if not cwd.is_absolute(): + raise AcpCoordinatorError("Herdr ACP status cwd must be absolute") + return HerdrAcpStatus( + str(raw_generation), str(lifecycle), str(console_lifecycle) + ) + + +def _parse_console_exchange( + value: Any, after_sequence: int +) -> tuple[tuple[int, str], ...]: + if not isinstance(value, Mapping) or set(value) != { + "type", + "inputs", + "outputs", + "input_floor_sequence", + "output_floor_sequence", + "next_input_sequence", + "next_output_sequence", + }: + raise AcpCoordinatorError("Herdr ACP console exchange shape is invalid") + if value.get("type") != "agent_acp_console_exchange": + raise AcpCoordinatorError("Herdr ACP console exchange type is invalid") + raw_inputs = value.get("inputs") + raw_outputs = value.get("outputs") + if not isinstance(raw_inputs, list) or not isinstance(raw_outputs, list): + raise AcpCoordinatorError("Herdr ACP console exchange lists are invalid") + input_floor = value.get("input_floor_sequence") + output_floor = value.get("output_floor_sequence") + next_input = value.get("next_input_sequence") + next_output = value.get("next_output_sequence") + if ( + type(after_sequence) is not int + or after_sequence < 0 + or after_sequence > (1 << 64) - 1 + or type(input_floor) is not int + or type(output_floor) is not int + or input_floor < 0 + or output_floor < 0 + or type(next_input) is not int + or type(next_output) is not int + or next_input < input_floor + or next_output < output_floor + or next_input > (1 << 64) - 1 + or next_output > (1 << 64) - 1 + ): + raise AcpCoordinatorError("Herdr ACP console exchange floors are invalid") + if input_floor > after_sequence + 1: + raise AcpConsoleInputGap( + "Herdr ACP console input floor has a gap", + recovery_after_sequence=next_input - 1, + ) + parsed: list[tuple[int, str]] = [] + expected = after_sequence + 1 + for item in raw_inputs: + if not isinstance(item, Mapping) or set(item) != {"sequence", "text"}: + raise AcpCoordinatorError("Herdr ACP console input shape is invalid") + sequence = item.get("sequence") + text = item.get("text") + if ( + type(sequence) is not int + or sequence != expected + or not isinstance(text, str) + or not text.strip() + ): + if type(sequence) is int and sequence > expected: + raise AcpConsoleInputGap( + "Herdr ACP console input sequence has a gap", + recovery_after_sequence=next_input - 1, + ) + raise AcpCoordinatorError("Herdr ACP console input sequence is invalid") + parsed.append((sequence, text)) + expected += 1 + if next_input != expected: + if next_input > expected: + raise AcpConsoleInputGap( + "Herdr ACP console input response is incomplete", + recovery_after_sequence=next_input - 1, + ) + raise AcpCoordinatorError( + "Herdr ACP console next input sequence is invalid" + ) + output_expected = output_floor + allowed_streams = { + "user", + "assistant", + "thought", + "tool", + "plan", + "status", + "error", + "turn_end", + } + for item in raw_outputs: + if not isinstance(item, Mapping) or set(item) != { + "sequence", + "event_id", + "stream", + "text", + }: + raise AcpCoordinatorError("Herdr ACP console output shape is invalid") + sequence = item.get("sequence") + event_id = item.get("event_id") + stream = item.get("stream") + text = item.get("text") + if ( + type(sequence) is not int + or sequence != output_expected + or not isinstance(event_id, str) + or not event_id + or len(event_id) > 512 + or stream not in allowed_streams + or not isinstance(text, str) + ): + raise AcpCoordinatorError("Herdr ACP console output item is invalid") + output_expected += 1 + if next_output != output_expected: + raise AcpCoordinatorError( + "Herdr ACP console next output sequence is invalid" + ) + return tuple(parsed) + + +def _console_event_output( + kind: str, payload: Mapping[str, Any] +) -> tuple[str, str] | None: + if kind in {"user_message", "agent_message", "thought"}: + delta = payload.get("text_delta") + if not isinstance(delta, str) or not delta: + return None + return ( + { + "user_message": "user", + "agent_message": "assistant", + "thought": "thought", + }[kind], + delta, + ) + if kind in {"tool_call", "tool_call_update"}: + snapshot = payload.get("snapshot") + if not isinstance(snapshot, Mapping): + return None + title = str(snapshot.get("title") or snapshot.get("kind") or "tool") + status = str(snapshot.get("status") or "updated") + details = [f"{title} [{status}]"] + content = snapshot.get("content") + if isinstance(content, list): + for item in content: + if not isinstance(item, Mapping): + continue + if item.get("type") == "content": + block = item.get("content") + if ( + isinstance(block, Mapping) + and block.get("type") == "text" + and isinstance(block.get("text"), str) + ): + details.append(str(block["text"])) + elif item.get("type") == "diff" and isinstance(item.get("path"), str): + details.append(f"diff {item['path']}") + if isinstance(item.get("oldText"), str): + details.append("- " + str(item["oldText"])) + if isinstance(item.get("newText"), str): + details.append("+ " + str(item["newText"])) + # Herdr also enforces a bounded console item. Bound before transport so + # a single verbose tool result cannot starve all later events. + return "tool", "\n".join(details)[: 256 * 1024] + if kind == "plan": + entries = payload.get("entries") + if not isinstance(entries, list): + return None + lines = [ + f"[{str(item.get('status') or 'pending')}] {str(item.get('content') or '').strip()}" + for item in entries + if isinstance(item, Mapping) and str(item.get("content") or "").strip() + ] + return ("plan", "\n".join(lines)) if lines else None + if kind == "usage": + return "status", "usage " + json.dumps( + dict(payload), sort_keys=True, separators=(",", ":") + ) + if kind == "session_info" and payload.get("title"): + return "status", f"session {payload['title']}" + if ( + kind == "extension" + and payload.get("extension") == "tendwire.acp.prompt_completion" + ): + return "status", f"turn {str(payload.get('outcome') or 'complete')}" + if ( + kind == "extension" + and payload.get("extension") == "tendwire.acp.console_submission_outcome" + ): + outcome = payload.get("outcome") + if outcome == "permission": + return "status", "permission selection accepted" + if outcome == "cancelled": + return "status", "active turn cancellation requested" + if outcome == "error": + return "error", "instruction failed" + return None + + +def _bounded_console_output(event_id: str, stream: str, text: str) -> dict[str, str]: + encoded = text.encode("utf-8") + if len(encoded) > _CONSOLE_OUTPUT_ITEM_TEXT_BYTES: + encoded = encoded[:_CONSOLE_OUTPUT_ITEM_TEXT_BYTES] + while True: + try: + text = encoded.decode("utf-8") + break + except UnicodeDecodeError as exc: + encoded = encoded[: exc.start] + text += "\n[console output truncated]" + return {"event_id": event_id, "stream": stream, "text": text} + + +def _console_output_wire_bytes(output: list[dict[str, str]]) -> int: + return len( + json.dumps(output, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + ) + + +def _console_output_fits( + output: list[dict[str, str]], item: dict[str, str], *, budget: int +) -> bool: + return _console_output_wire_bytes([*output, item]) <= budget + + +def _fit_console_output_batch( + output: list[dict[str, str]], *, budget: int +) -> list[dict[str, str]]: + fitted: list[dict[str, str]] = [] + for item in output: + if not _console_output_fits(fitted, item, budget=budget): + break + fitted.append(item) + return fitted + + +def _console_exchange_output_bytes(value: Any) -> int: + assert isinstance(value, Mapping) + outputs = value.get("outputs") + assert isinstance(outputs, list) + return len( + json.dumps(outputs, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + ) + + +def _load_console_event_cursor( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, +) -> int | None: + after = 0 + cursor = 0 + found = False + while True: + page = list_agent_events( + db_path, + host_id, + worker_id=worker_id, + source="tendwire-console", + session_id=session_id, + after_sequence=after, + limit=1000, + ) + if not page: + return cursor if found else None + for stored in page: + after = max(after, stored.sequence) + payload = stored.event.payload + value = payload.get("sequence") + if ( + payload.get("extension") == "tendwire.acp.console_cursor" + and type(value) is int + and value >= 0 + ): + found = True + cursor = max(cursor, value) + if len(page) < 1000: + return cursor if found else None + + +def _initial_console_event_cursor( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, +) -> int: + persisted = _load_console_event_cursor( + db_path, host_id, worker_id, session_id + ) + if persisted is not None: + return persisted + return 0 + + +def _record_console_event_cursor( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, + sequence: int, + *, + source_event_id: str, +) -> None: + record_agent_event( + db_path, + host_id, + kind="extension", + source="tendwire-console", + worker_id=worker_id, + payload={"extension": "tendwire.acp.console_cursor", "sequence": sequence}, + source_session_id=session_id, + source_event_id=source_event_id, + visibility="private", + ) + + +def _prepare_console_event_cursor( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, + *, + session_mode: SessionOpenMode, + generation: int, +) -> int: + persisted = _load_console_event_cursor(db_path, host_id, worker_id, session_id) + if persisted is not None: + return persisted + # A missing checkpoint is never evidence that any stored ACP update was + # displayed. Replay from zero for NEW setup updates as well as LOAD/RESUME; + # Herdr's stable event ids make the conservative replay idempotent. + baseline = 0 + _record_console_event_cursor( + db_path, + host_id, + worker_id, + session_id, + baseline, + source_event_id=f"baseline:{generation}:{baseline}", + ) + return baseline + + +def _record_console_submission_outcome( + db_path: Path, + host_id: str, + worker_id: str, + session_id: str, + *, + generation: int, + input_sequence: int, + outcome: str, +) -> None: + record_agent_event( + db_path, + host_id, + kind="extension", + source="acp", + worker_id=worker_id, + payload={ + "schema_version": 1, + "extension": "tendwire.acp.console_submission_outcome", + "generation": generation, + "input_sequence": input_sequence, + "outcome": outcome, + }, + source_session_id=session_id, + source_event_id=f"console-outcome:{generation}:{input_sequence}", + visibility="private", + ) + + +def _load_console_input_cursor( + db_path: Path, + host_id: str, + worker_id: str, + _session_id: str, + generation: int, +) -> int: + # Herdr owns the visible input queue at the worker generation, not at an + # adapter session. A NEW adapter session can be reminted after a bridge + # failure while Herdr retains the same console generation and sequence. + # Recovering only from the reminted session would reset this cursor to zero + # and turn every retained input into a permanent false gap. + after = 0 + cursor = 0 + while True: + page = list_agent_events( + db_path, + host_id, + worker_id=worker_id, + source="tendwire-console", + after_sequence=after, + limit=1000, + ) + if not page: + return cursor + for stored in page: + after = max(after, stored.sequence) + payload = stored.event.payload + if ( + payload.get("extension") == "tendwire.acp.console_input_cursor" + and payload.get("generation") == generation + and type(payload.get("input_sequence")) is int + ): + cursor = max(cursor, int(payload["input_sequence"])) + if len(page) < 1000: + return cursor + + +def _console_pending_decision( + db_path: Path, + host_id: str, + worker_id: str, +) -> tuple[str, tuple[tuple[str, str], ...], str] | None: + payload = pending_payload_from_store(db_path, host_id) + interactions = payload.get("pending_interactions") + if not isinstance(interactions, list): + return None + matches = [ + item + for item in interactions + if isinstance(item, Mapping) + and item.get("worker_id") == worker_id + and item.get("status") not in {"resolved", "closed", "stale"} + and isinstance(item.get("meta"), Mapping) + and isinstance(item["meta"].get("decision"), Mapping) + ] + if len(matches) != 1: + return None + item = matches[0] + decision = item["meta"]["decision"] + decision_ref = decision.get("decision_ref") + raw_options = decision.get("options") + if not isinstance(decision_ref, str) or not isinstance(raw_options, list): + return None + options = tuple( + (str(option.get("ref") or ""), str(option.get("label") or "")) + for option in raw_options + if isinstance(option, Mapping) + and str(option.get("ref") or "") + and str(option.get("label") or "") + ) + if not options: + return None + question = str(item.get("question") or "Permission required") + choices = " ".join(f"{ref}={label}" for ref, label in options) + return decision_ref, options, f"permission> {question} ({choices}); reply with a number or /cancel" + + +def _console_permission_selection( + text: str, options: tuple[tuple[str, str], ...] +) -> str | None: + value = text.strip().lower() + for ref, label in options: + if value == ref.lower() or value == label.lower(): + return ref + aliases = { + "allow": ("allow",), + "yes": ("allow",), + "deny": ("deny", "reject"), + "reject": ("deny", "reject"), + "no": ("deny", "reject"), + } + terms = aliases.get(value) + if terms is None: + return None + matches = [ + ref + for ref, label in options + if any(term in label.lower() for term in terms) + ] + return matches[0] if len(matches) == 1 else None + + +def production_acp_supervisor_factory( + config: Config, + stop_event: threading.Event, +) -> AcpSupervisor: + """Build the stock daemon's Herdr-backed ACP session supervisor.""" + return AcpSupervisor( + config, + stop_event, + require_permission_bridge=True, + durable_permission_bridge=True, + ) diff --git a/src/tendwire/backends/acp_ingestion.py b/src/tendwire/backends/acp_ingestion.py new file mode 100644 index 0000000..d398271 --- /dev/null +++ b/src/tendwire/backends/acp_ingestion.py @@ -0,0 +1,710 @@ +"""Durable ACP ingestion bound to one authenticated Tendwire worker. + +The transport, semantic projector, and SQLite journal are deliberately separate. +This module is the narrow authority bridge: it binds one ACP session generation +to one private Herdr worker binding, records every accepted semantic event, and +projects only user/assistant text into the existing turn model. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping, Sequence +from copy import deepcopy +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from ..config import Config +from ..core.agent_events import AgentEvent, agent_event +from ..core.models import WorkerBinding, stable_fingerprint +from ..store.sqlite import ( + AppendBoundAgentEventResult, + AppendProjectedAgentEventResult, + TurnRefreshApplyResult, + append_agent_event_and_apply_turn_for_binding, +) +from .acp_projection import AcpEventProjector, AcpProjectionCheckpoint +from .acp_protocol import StopReason + + +PersistEvent = Callable[..., AppendProjectedAgentEventResult] + + +@dataclass(frozen=True) +class AcpIngestionResult: + """Outcome of accepting, ignoring, or projecting one ACP event.""" + + kind: str | None + event: AppendBoundAgentEventResult | None = None + turn: TurnRefreshApplyResult | None = None + ignored_reason: str | None = None + + +class AcpSessionIngestor: + """Ingest one ACP session generation for one private worker binding. + + ``stream_generation`` must change whenever a transport is recreated. ACP v1 + does not require a stable event ID for notifications, so generation-scoped + synthetic IDs avoid corrupting the append-only journal. Notifications with + authoritative source IDs still deduplicate across reconnects. + """ + + def __init__( + self, + config: Config, + *, + session_id: str, + stream_generation: str, + binding: WorkerBinding, + projector: AcpEventProjector | None = None, + persist_event: PersistEvent = append_agent_event_and_apply_turn_for_binding, + ) -> None: + if config.db_path is None: + raise ValueError("ACP ingestion requires a sqlite db path") + if not isinstance(session_id, str) or not session_id.strip(): + raise ValueError("ACP session and stream generation are required") + if not isinstance(stream_generation, str) or not stream_generation.strip(): + raise ValueError("ACP session and stream generation are required") + if binding.host_id != config.host_id: + raise ValueError("ACP binding host does not match configuration") + if not binding.private_fingerprint: + raise ValueError("ACP ingestion requires an authenticated private binding") + if ( + binding.turn_target_kind != "acp_session_id" + or binding.turn_target_value != session_id.strip() + ): + raise ValueError("ACP session does not match the private worker binding") + self.config = config + self.session_id = session_id.strip() + self.stream_generation = stream_generation.strip() + self.binding = binding + self.projector = projector or AcpEventProjector() + self._persist_event = persist_event + self._turn_ordinal = 0 + self._source_turn_id: str | None = None + self._turn_complete = False + self._local_prompt_recorded = False + + @property + def source_turn_id(self) -> str | None: + """Return the opaque public-safe identity of the active ACP turn.""" + + return self._source_turn_id + + def start_turn(self, *, producer_turn_id: str | None = None) -> str: + """Reset message assembly and allocate one opaque turn identity.""" + + if producer_turn_id is not None and ( + not isinstance(producer_turn_id, str) or not producer_turn_id.strip() + ): + raise ValueError("producer_turn_id must be non-empty text or None") + self._turn_ordinal += 1 + self.projector.reset_turn(self.session_id) + # An authoritative producer turn ID retains identity across ACP + # transport recreation. The fallback exists only for unsolicited or + # historical inbound streams; outgoing prompts require producer + # identity before any durable or remote side effect. + identity = ( + { + "source": "acp", + "session": self.session_id, + "producer_turn": producer_turn_id.strip(), + } + if producer_turn_id is not None + else { + "source": "acp", + "session": self.session_id, + "turn": self._turn_ordinal, + } + ) + self._source_turn_id = f"acpt_{stable_fingerprint(identity)}" + self._turn_complete = False + self._local_prompt_recorded = False + return self._source_turn_id + + def ingest_update( + self, + notification: Mapping[str, Any], + *, + source_event_id: str | None = None, + replay: bool = False, + setup_replay: bool = False, + ) -> AcpIngestionResult: + """Normalize, journal, and conditionally project ``session/update``.""" + + mismatch = _notification_mismatch( + notification, + method="session/update", + session_id=self.session_id, + ) + if mismatch is not None: + return AcpIngestionResult(None, ignored_reason=mismatch) + update_kind = _session_update_kind(notification) + if self._turn_complete and update_kind in _TURN_SCOPED_UPDATES: + return AcpIngestionResult(None, ignored_reason="turn_already_complete") + if ( + update_kind == "user_message_chunk" + and self._local_prompt_recorded + and not replay + ): + # ACP agents commonly echo the prompt as a user-message update. + # begin_prompt() already journaled the complete producer-owned + # input, so accepting the echo would duplicate both the journal + # and the compatibility turn. Load replay has no local producer + # record and must continue to retain historical user messages. + return AcpIngestionResult("user_message", ignored_reason="prompt_echo") + thought_rejection = _thought_rejection_reason( + notification, + policy=self.config.acp_thought_policy, + ) + if thought_rejection is not None: + return AcpIngestionResult("thought", ignored_reason=thought_rejection) + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + if self._source_turn_id is None and update_kind in _TURN_SCOPED_UPDATES: + self.start_turn() + try: + canonical = self.projector.normalize_session_update( + notification, + source_event_id=source_event_id, + replay=replay, + ) + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + if canonical is None: + self._restore_speculation(checkpoint, prior_turn_state) + return AcpIngestionResult(None, ignored_reason="unsupported_or_duplicate") + return self._accept( + canonical, + checkpoint=checkpoint, + prior_turn_state=prior_turn_state, + project_turn=not setup_replay, + replay_namespace="load" if setup_replay else None, + ) + + def begin_prompt( + self, + prompt: Sequence[Mapping[str, Any]], + *, + producer_turn_id: str | None = None, + ) -> AcpIngestionResult: + """Durably record outgoing prompt content before transport send.""" + + if not isinstance(producer_turn_id, str) or not producer_turn_id.strip(): + raise ValueError("producer_turn_id must be non-empty text") + blocks = [dict(block) for block in prompt] + if not blocks: + raise ValueError("prompt must contain at least one content block") + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + source_turn_id = self.start_turn(producer_turn_id=producer_turn_id) + text = "\n".join( + str(block.get("text")) + for block in blocks + if block.get("type") == "text" and isinstance(block.get("text"), str) + ) + source_event_id = f"prompt-input:{source_turn_id}" + try: + canonical = self.projector.normalize_session_update( + { + "method": "session/update", + "params": { + "sessionId": self.session_id, + "update": { + "sessionUpdate": "user_message_chunk", + "messageId": source_event_id, + "content": {"type": "text", "text": text}, + }, + }, + }, + source_event_id=source_event_id, + replay=False, + ) + if canonical is None: # pragma: no cover - fresh turn invariant + raise RuntimeError("outgoing ACP prompt was unexpectedly duplicated") + payload = canonical.get("payload") + if not isinstance(payload, Mapping): # pragma: no cover - projector invariant + raise RuntimeError("outgoing ACP prompt projection is invalid") + canonical = { + **canonical, + "payload": { + **payload, + "prompt_content": deepcopy(blocks), + "outgoing": True, + }, + } + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + result = self._accept( + canonical, + checkpoint=checkpoint, + prior_turn_state=prior_turn_state, + ) + if result.event is not None and result.event.status != "binding_changed": + self._local_prompt_recorded = True + return result + + def can_append_prompt(self) -> bool: + """Return whether a steering input can join the current logical turn.""" + + return self._source_turn_id is not None and not self._turn_complete + + def append_prompt( + self, + prompt: Sequence[Mapping[str, Any]], + *, + producer_turn_id: str, + ) -> AcpIngestionResult: + """Durably append one steering input to the current ACP turn.""" + + if not isinstance(producer_turn_id, str) or not producer_turn_id.strip(): + raise ValueError("producer_turn_id must be non-empty text") + if not self.can_append_prompt(): + raise RuntimeError("ACP steering requires an active turn") + blocks = [dict(block) for block in prompt] + if not blocks: + raise ValueError("prompt must contain at least one content block") + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + text = "\n".join( + str(block.get("text")) + for block in blocks + if block.get("type") == "text" and isinstance(block.get("text"), str) + ) + producer = producer_turn_id.strip() + source_event_id = "steer-input:" + stable_fingerprint( + {"producer_turn": producer} + ) + try: + canonical = self.projector.normalize_session_update( + { + "method": "session/update", + "params": { + "sessionId": self.session_id, + "update": { + "sessionUpdate": "user_message_chunk", + "messageId": source_event_id, + "content": {"type": "text", "text": text}, + }, + }, + }, + source_event_id=source_event_id, + replay=False, + ) + if canonical is None: + raise RuntimeError("outgoing ACP steering input was duplicated") + payload = canonical.get("payload") + if not isinstance(payload, Mapping): + raise RuntimeError("outgoing ACP steering projection is invalid") + canonical = { + **canonical, + "payload": { + **payload, + "prompt_content": deepcopy(blocks), + "outgoing": True, + "steering": True, + }, + } + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + result = self._accept( + canonical, + checkpoint=checkpoint, + prior_turn_state=prior_turn_state, + ) + if result.event is not None and result.event.status != "binding_changed": + self._local_prompt_recorded = True + return result + + def reset_after_load(self) -> None: + """Drop replay turn assembly before accepting a new active prompt.""" + + self.projector.reset_turn(self.session_id) + self._source_turn_id = None + self._turn_complete = False + self._local_prompt_recorded = False + + def ingest_permission_request( + self, + request: Mapping[str, Any], + *, + source_event_id: str | None = None, + replay: bool = False, + setup_replay: bool = False, + ) -> AcpIngestionResult: + """Journal a permission request as a private tool lifecycle update.""" + + mismatch = _notification_mismatch( + request, + method="session/request_permission", + session_id=self.session_id, + ) + if mismatch is not None: + return AcpIngestionResult(None, ignored_reason=mismatch) + if self._turn_complete: + return AcpIngestionResult(None, ignored_reason="turn_already_complete") + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + if self._source_turn_id is None: + self.start_turn() + try: + canonical = self.projector.normalize_permission_request( + request, + source_event_id=source_event_id, + replay=replay, + ) + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + if canonical is None: + self._restore_speculation(checkpoint, prior_turn_state) + return AcpIngestionResult(None, ignored_reason="duplicate") + return self._accept( + canonical, + checkpoint=checkpoint, + prior_turn_state=prior_turn_state, + project_turn=not setup_replay, + replay_namespace="load" if setup_replay else None, + ) + + def mark_prompt_complete( + self, + stop_reason: StopReason | str = StopReason.END_TURN, + ) -> AcpIngestionResult: + """Durably finalize the current turn after ``session/prompt`` returns.""" + + if self._source_turn_id is None: + return AcpIngestionResult(None, ignored_reason="no_active_turn") + if self._turn_complete: + return AcpIngestionResult(None, ignored_reason="turn_already_complete") + checkpoint = self.projector.checkpoint_session(self.session_id) + prior_turn_state = self._turn_state() + try: + try: + normalized_reason = StopReason(stop_reason) + except ValueError as exc: + raise ValueError("unsupported ACP prompt stop reason") from exc + content = self.projector.mark_turn_complete(self.session_id) + content["source_turn_id"] = self._source_turn_id + content["assistant_final_text"] = _final_text_for_stop_reason( + str(content.get("assistant_final_text") or ""), + normalized_reason, + ) + marker = agent_event( + kind="extension", + source="acp", + worker_id=self.binding.worker_id, + payload={ + "schema_version": 1, + "extension": "tendwire.acp.prompt_completion", + "complete": True, + "stop_reason": normalized_reason.value, + "outcome": _STOP_REASON_OUTCOMES[normalized_reason], + "projection": content, + }, + source_session_id=self.session_id, + source_turn_id=self._source_turn_id, + source_event_id=f"prompt-complete:{self._source_turn_id}", + visibility="private", + ) + persisted = self._persist_event( + Path(self.config.db_path), + self.config.host_id, + marker, + expected_binding=self.binding, + content=content, + ) + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + if persisted.event.status == "binding_changed": + self._restore_speculation(checkpoint, prior_turn_state) + return AcpIngestionResult( + "extension", + event=persisted.event, + ignored_reason="stale_binding", + ) + self._turn_complete = True + self._local_prompt_recorded = False + return AcpIngestionResult( + "extension", + event=persisted.event, + turn=persisted.turn, + ignored_reason=( + "duplicate_event" + if persisted.event.status == "replayed" + else None + ), + ) + + def _accept( + self, + canonical: Mapping[str, Any], + *, + checkpoint: AcpProjectionCheckpoint, + prior_turn_state: tuple[int, str | None, bool, bool], + project_turn: bool = True, + replay_namespace: str | None = None, + ) -> AcpIngestionResult: + kind = str(canonical.get("kind") or "") + if kind == "thought" and self.config.acp_thought_policy == "disabled": + self._restore_speculation(checkpoint, prior_turn_state) + return AcpIngestionResult(kind, ignored_reason="thought_policy_disabled") + try: + payload = canonical.get("payload") + if not isinstance(payload, Mapping): + raise ValueError("canonical ACP event payload must be a mapping") + sequence = canonical.get("sequence") + if isinstance(sequence, bool) or not isinstance(sequence, int) or sequence < 0: + raise ValueError("canonical ACP event sequence must be nonnegative") + explicit_event_id = canonical.get("source_event_id") + source_id = ( + str(explicit_event_id) + if explicit_event_id is not None and str(explicit_event_id) + else ( + _replay_source_event_id( + replay_namespace, + session_id=self.session_id, + sequence=sequence, + kind=kind, + payload=payload, + ) + if replay_namespace is not None + else f"stream:{self.stream_generation}:{sequence}" + ) + ) + event = agent_event( + kind=kind, + source="acp", + worker_id=self.binding.worker_id, + payload=payload, + source_session_id=self.session_id, + source_turn_id=self._source_turn_id, + source_item_id=_source_item_id(kind, payload), + source_message_id=_source_message_id(kind, payload), + source_event_id=source_id, + source_sequence=sequence, + # The complete structured journal is private initially. Public and + # connector views require a separate explicit sanitizing projection. + visibility="private", + ) + projection: Mapping[str, Any] | None = None + if ( + kind in {"user_message", "agent_message"} + and project_turn + ): + content = self.projector.project_turn_content(self.session_id) + if self._source_turn_id is not None: + content["source_turn_id"] = self._source_turn_id + projection = content + persisted = self._persist_event( + Path(self.config.db_path), + self.config.host_id, + event, + expected_binding=self.binding, + content=projection, + ) + except BaseException: + self._restore_speculation(checkpoint, prior_turn_state) + raise + appended = persisted.event + # A durable replay on a newly constructed ingestor is also the + # reconstruction path for its in-memory projector. Keep that state so + # prompt completion can finalize the recovered text. Only a stale + # binding invalidates the speculative normalization. + if appended.status == "binding_changed": + self._restore_speculation(checkpoint, prior_turn_state) + turn = persisted.turn + return AcpIngestionResult( + kind, + event=appended, + turn=turn, + ignored_reason=( + "stale_binding" + if appended.status == "binding_changed" + or (turn is not None and turn.stale_binding) + else "duplicate_event" + if appended.status == "replayed" + else None + ), + ) + + def _turn_state(self) -> tuple[int, str | None, bool, bool]: + return ( + self._turn_ordinal, + self._source_turn_id, + self._turn_complete, + self._local_prompt_recorded, + ) + + def _restore_speculation( + self, + checkpoint: AcpProjectionCheckpoint, + prior_turn_state: tuple[int, str | None, bool, bool], + ) -> None: + self.projector.restore_session(checkpoint) + ( + self._turn_ordinal, + self._source_turn_id, + self._turn_complete, + self._local_prompt_recorded, + ) = prior_turn_state + + +_STOP_REASON_OUTCOMES = { + StopReason.END_TURN: "completed", + StopReason.MAX_TOKENS: "truncated_max_tokens", + StopReason.MAX_TURN_REQUESTS: "truncated_max_turn_requests", + StopReason.REFUSAL: "refused", + StopReason.CANCELLED: "cancelled", +} + +_STOP_REASON_NOTICES = { + StopReason.MAX_TOKENS: "[ACP response truncated: token limit reached]", + StopReason.MAX_TURN_REQUESTS: "[ACP response truncated: request limit reached]", + StopReason.REFUSAL: "[ACP agent refused the request]", + StopReason.CANCELLED: "[ACP prompt cancelled]", +} + + +def _final_text_for_stop_reason(text: str, stop_reason: StopReason) -> str: + notice = _STOP_REASON_NOTICES.get(stop_reason) + if notice is None: + return text + return f"{text}\n\n{notice}" if text else notice + + +def _replay_source_event_id( + namespace: str, + *, + session_id: str, + sequence: int, + kind: str, + payload: Mapping[str, Any], +) -> str: + fingerprint = stable_fingerprint( + { + "session": session_id, + "sequence": sequence, + "kind": kind, + "payload": payload, + } + ) + return f"{namespace}:{fingerprint}" + + +def _source_message_id(kind: str, payload: Mapping[str, Any]) -> str | None: + if kind not in {"user_message", "agent_message", "thought"}: + return None + value = payload.get("message_id") + return str(value) if value is not None and str(value) else None + + +def _source_item_id(kind: str, payload: Mapping[str, Any]) -> str | None: + if kind not in {"tool_call", "tool_call_update"}: + return None + value = payload.get("tool_call_id") + return str(value) if value is not None and str(value) else None + + +_TURN_SCOPED_UPDATES = frozenset( + { + "user_message_chunk", + "agent_message_chunk", + "agent_thought_chunk", + "tool_call", + "tool_call_update", + "plan", + } +) +_TRUSTED_THOUGHT_SUMMARY_KEY = "tendwire.dev/thought_kind" +_LEGACY_THOUGHT_CLASSIFICATION_KEYS = frozenset( + {"thought_kind", "thoughtKind", "reasoning_kind", "reasoningKind"} +) + + +def _params(value: Mapping[str, Any]) -> Mapping[str, Any]: + params = value.get("params") + return params if isinstance(params, Mapping) else value + + +def _notification_mismatch( + value: Mapping[str, Any], + *, + method: str, + session_id: str, +) -> str | None: + supplied_method = value.get("method") + if supplied_method is not None and supplied_method != method: + return "method_mismatch" + supplied_session = _params(value).get("sessionId") + if supplied_session != session_id: + return "session_mismatch" + return None + + +def _session_update(value: Mapping[str, Any]) -> Mapping[str, Any] | None: + update = _params(value).get("update") + return update if isinstance(update, Mapping) else None + + +def _session_update_kind(value: Mapping[str, Any]) -> str | None: + update = _session_update(value) + kind = update.get("sessionUpdate") if update is not None else None + return kind if isinstance(kind, str) else None + + +def _thought_classification(value: Mapping[str, Any]) -> str | None: + update = _session_update(value) + if update is None or update.get("sessionUpdate") != "agent_thought_chunk": + return None + update_meta = update.get("_meta") + trusted = ( + update_meta.get(_TRUSTED_THOUGHT_SUMMARY_KEY) + if isinstance(update_meta, Mapping) + else None + ) + if trusted != "summary": + return "unclassified" if trusted is None else "unknown" + + # The exact update-level marker is a Tendwire adapter convention, not an + # ACP classification guarantee. Contradictory legacy or content metadata + # therefore fails closed even when the trusted marker says "summary". + for container in (update, update.get("content")): + if not isinstance(container, Mapping): + continue + meta = container.get("_meta") + if not isinstance(meta, Mapping): + continue + marker = meta.get(_TRUSTED_THOUGHT_SUMMARY_KEY) + if marker is not None and marker != "summary": + return "conflicting" + if any(key in meta for key in _LEGACY_THOUGHT_CLASSIFICATION_KEYS): + return "conflicting" + legacy_namespace = meta.get("tendwire") + if isinstance(legacy_namespace, Mapping) and any( + key in legacy_namespace for key in _LEGACY_THOUGHT_CLASSIFICATION_KEYS + ): + return "conflicting" + return "summary" + + +def _thought_rejection_reason( + value: Mapping[str, Any], + *, + policy: str, +) -> str | None: + classification = _thought_classification(value) + if classification is None: + return None + if policy == "disabled": + return "thought_policy_disabled" + if policy == "private_summary" and classification != "summary": + return "thought_policy_requires_summary" + return None + + +__all__ = ["AcpIngestionResult", "AcpSessionIngestor"] diff --git a/src/tendwire/backends/acp_permissions.py b/src/tendwire/backends/acp_permissions.py new file mode 100644 index 0000000..91c58ad --- /dev/null +++ b/src/tendwire/backends/acp_permissions.py @@ -0,0 +1,293 @@ +"""Durable, privacy-preserving bridge for ACP permission decisions.""" + +from __future__ import annotations + +import secrets +import threading +import time +from collections.abc import Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from ..config import Config +from ..core.models import WorkerBinding, sanitize_public_text, stable_fingerprint +from ..core.turns import PendingObservation, PendingObservedChoice +from ..store.sqlite import ( + apply_backend_pending_observation, + list_worker_bindings, +) +from .acp_protocol import PermissionRequest +from .acp_runtime import PermissionSelection + + +class AcpPermissionBrokerError(RuntimeError): + """A permission could not be safely correlated or acknowledged.""" + + +_MAX_PERMISSION_OPTIONS = 64 + + +@dataclass(slots=True) +class _Offer: + decision_ref: str + binding: WorkerBinding + generation: str + option_ids: tuple[str, ...] + condition: threading.Condition + selected: int | None = None + response_state: str = "pending" + response_error: BaseException | None = None + answer_abandoned: bool = False + + +class AcpPermissionBroker: + """One bounded permission rendezvous for one worker generation.""" + + def __init__( + self, + config: Config, + *, + worker_id: str, + worker_fingerprint: str, + generation: str, + timeout: float, + ) -> None: + if config.db_path is None: + raise ValueError("ACP permission broker requires a sqlite db path") + if timeout <= 0: + raise ValueError("ACP permission broker timeout must be positive") + self.config = config + self.worker_id = worker_id + self.worker_fingerprint = worker_fingerprint + self.generation = generation + self.timeout = float(timeout) + self._lock = threading.RLock() + self._offer: _Offer | None = None + self._closed = False + + def __call__(self, request: PermissionRequest) -> PermissionSelection | None: + if not request.options or len(request.options) > _MAX_PERMISSION_OPTIONS: + return None + binding = self._exact_binding(request.session_id) + title = _tool_title(request.tool_call) + labels = tuple( + _option_label( + option.name, + str( + option.kind.value + if hasattr(option.kind, "value") + else option.kind + ), + ) + for option in request.options + ) + if not labels: + return None + nonce = secrets.token_urlsafe(24) + source_revision = stable_fingerprint( + { + "route": "acp_permission_v1", + "worker_id": self.worker_id, + "worker_fingerprint": self.worker_fingerprint, + "binding": binding.private_fingerprint, + "session": binding.turn_target_value, + "generation": self.generation, + "nonce": nonce, + } + ) + persisted_revision = stable_fingerprint( + { + "decision_revision": source_revision, + "binding_private_fingerprint": binding.private_fingerprint, + "observed_turn_target_value": binding.turn_target_value, + } + ) + offer = _Offer( + decision_ref=f"decision-{persisted_revision}", + binding=binding, + generation=self.generation, + option_ids=tuple(option.option_id for option in request.options), + condition=threading.Condition(self._lock), + ) + observation = PendingObservation( + "open_prompt", + question=title, + pending_kind="approval", + choices=tuple( + PendingObservedChoice( + choice_id="choice-" + + stable_fingerprint( + { + "revision": source_revision, + "ordinal": index, + "label": label, + } + ), + label=label, + picker_ordinal=index, + ) + for index, label in enumerate(labels, 1) + ), + revision_digest=source_revision, + decision_kind="single", + decision_options=labels, + decision_multi_select=False, + decision_question_count=1, + ) + try: + with self._lock: + if self._closed or self._offer is not None: + return None + self._offer = offer + changed = apply_backend_pending_observation( + Path(self.config.db_path), + self.config.host_id, + self.worker_id, + observation, + binding_private_fingerprint=binding.private_fingerprint, + observed_turn_target_value=binding.turn_target_value, + binding_authoritative=True, + ) + if not changed: + raise AcpPermissionBrokerError("permission overlay was not published") + deadline = time.monotonic() + self.timeout + with offer.condition: + while offer.selected is None and not self._closed: + remaining = deadline - time.monotonic() + if remaining <= 0: + break + offer.condition.wait(remaining) + if offer.selected is None: + self._clear_offer(offer) + return None + option_id = offer.option_ids[offer.selected - 1] + return PermissionSelection( + option_id, + response_written=lambda: self._response_complete(offer, None), + response_failed=lambda exc: self._response_complete(offer, exc), + ) + except BaseException: + self._clear_offer(offer) + raise + + def owns(self, decision: Any) -> bool: + with self._lock: + offer = self._offer + return bool( + not self._closed + and offer is not None + and decision.worker_id == self.worker_id + and decision.worker_fingerprint == self.worker_fingerprint + and decision.binding_private_fingerprint == offer.binding.private_fingerprint + and decision.turn_target_value == offer.binding.turn_target_value + and decision.decision_ref == offer.decision_ref + and offer.generation == self.generation + ) + + def answer(self, decision: Any, *, timeout: float) -> None: + clear_failed = False + uncertain = False + with self._lock: + offer = self._offer + if offer is None or not self.owns(decision): + raise AcpPermissionBrokerError("ACP permission authority changed") + if decision.text is not None or len(decision.option_refs) != 1: + raise AcpPermissionBrokerError("ACP permission selection is invalid") + ordinal = int(decision.option_refs[0]) + if ordinal < 1 or ordinal > len(offer.option_ids) or offer.selected is not None: + raise AcpPermissionBrokerError("ACP permission was already answered") + offer.selected = ordinal + offer.condition.notify_all() + deadline = time.monotonic() + max(0.1, float(timeout)) + while offer.response_state == "pending" and not self._closed: + remaining = deadline - time.monotonic() + if remaining <= 0: + break + offer.condition.wait(remaining) + if offer.response_state != "written": + # The command receipt is now terminally uncertain. A late + # transport callback must retire the stale public overlay and + # its send_started claim, but only after it proves that the + # response frame was written (or definitively failed). Until + # then the offer remains fail-closed and cannot be answered a + # second time. + offer.answer_abandoned = True + clear_failed = offer.response_state == "failed" + uncertain = True + else: + self._offer = None + if clear_failed: + self._clear_offer(offer) + if uncertain: + raise AcpPermissionBrokerError("ACP permission response state is uncertain") + + def close(self) -> None: + with self._lock: + self._closed = True + offer = self._offer + if offer is not None: + offer.condition.notify_all() + if offer is not None: + self._clear_offer(offer) + + def _response_complete(self, offer: _Offer, error: BaseException | None) -> None: + clear = False + with offer.condition: + if self._offer is offer: + offer.response_state = "failed" if error is not None else "written" + offer.response_error = error + clear = error is not None or offer.answer_abandoned + offer.condition.notify_all() + if clear: + self._clear_offer(offer) + + def _clear_offer(self, offer: _Offer) -> None: + with self._lock: + if self._offer is not offer: + return + self._offer = None + try: + apply_backend_pending_observation( + Path(self.config.db_path), + self.config.host_id, + self.worker_id, + PendingObservation("read_succeeded_no_prompt"), + binding_private_fingerprint=offer.binding.private_fingerprint, + observed_turn_target_value=offer.binding.turn_target_value, + ) + except Exception: + pass + + def _exact_binding(self, session_id: str) -> WorkerBinding: + rows = [ + row + for row in list_worker_bindings( + Path(self.config.db_path), + self.config.host_id, + backend="acp", + ) + if row.worker_id == self.worker_id + and row.worker_fingerprint == self.worker_fingerprint + and row.turn_target_kind == "acp_session_id" + and row.turn_target_value == session_id + and row.sendable + ] + if len(rows) != 1: + raise AcpPermissionBrokerError("ACP permission binding is not current") + return rows[0] + + +def _tool_title(tool_call: Any) -> str: + if isinstance(tool_call, Mapping): + value = tool_call.get("title") + if isinstance(value, str) and value.strip(): + clean = sanitize_public_text(value.strip()[:500]) + return clean or "Tool permission" + return "Tool permission" + + +def _option_label(name: str, kind: str) -> str: + clean_name = sanitize_public_text(name.strip()[:300]) or "Option" + clean_kind = sanitize_public_text(kind.strip()[:80]) or "unknown" + return f"{clean_name} ({clean_kind})" diff --git a/src/tendwire/backends/acp_probe.py b/src/tendwire/backends/acp_probe.py new file mode 100644 index 0000000..6b03f3d --- /dev/null +++ b/src/tendwire/backends/acp_probe.py @@ -0,0 +1,360 @@ +"""Bounded black-box compatibility probe for external ACP v1 adapters. + +The probe deliberately knows only the stable ACP wire protocol. It launches a +caller-supplied executable without a shell, negotiates a fresh connection, and +returns a fixed public-safe capability summary. Adapter output, argv, paths, +environment values, stderr, exception text, and extension names never enter the +report. + +Operator entry point:: + + python -m tendwire.backends.acp_probe -- adapter-command arg ... +""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import sys +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from enum import Enum +from types import MappingProxyType +from typing import Any + +from .acp_client import ( + BoundedAcpConnection, + AcpProtocolVersionError, + AcpRequestTimeoutError, + AcpTransportError, +) +from .acp_protocol import ( + ACP_PROTOCOL_VERSION, + AcpEnvelopeError, + AcpFramingError, + AcpRemoteError, + InitializeResult, +) + +PROBE_SCHEMA_VERSION = 2 +PROBE_SCOPE = "initialize" +DEFAULT_PROBE_TIMEOUT_SECONDS = 5.0 +DEFAULT_PROBE_CLOSE_TIMEOUT_SECONDS = 1.0 +MAX_PROBE_TIMEOUT_SECONDS = 30.0 +MAX_PROBE_CLOSE_TIMEOUT_SECONDS = 5.0 +PROBE_MAX_FRAME_BYTES = 1024 * 1024 +PROBE_MAX_PENDING_EVENTS = 16 +PROBE_STDERR_LIMIT_BYTES = 4096 +_MAX_REPORTED_COUNT = 1000 + +_ADVERTISED_CAPABILITY_KEYS = ( + "session_load", + "session_list", + "session_delete", + "session_resume", + "session_close", + "additional_directories", + "prompt_image", + "prompt_audio", + "prompt_embedded_context", + "mcp_http", + "mcp_sse", + "auth_logout", +) + + +class ProbeFailure(str, Enum): + """Fixed failure categories safe to expose to operators and automation.""" + + INVALID_CONFIGURATION = "invalid_configuration" + LAUNCH_FAILED = "launch_failed" + TIMEOUT = "timeout" + PROTOCOL_VERSION = "protocol_version" + PROTOCOL = "protocol_error" + TRANSPORT = "transport_error" + SHUTDOWN = "shutdown_failed" + INTERNAL = "internal_error" + + +@dataclass(frozen=True, slots=True) +class AcpAdapterProbeReport: + """Fixed-shape, bounded result that contains no adapter-controlled text.""" + + initialization_compatible: bool + protocol_version: int | None + advertised_capabilities: Mapping[str, bool] + authentication_method_count: int + authentication_method_count_capped: bool + extension_capability_count: int + extension_capability_count_capped: bool + process_reaped: bool + failure: ProbeFailure | None + schema_version: int = PROBE_SCHEMA_VERSION + + def to_payload(self) -> dict[str, Any]: + return { + "schema_version": self.schema_version, + "probe_scope": PROBE_SCOPE, + "initialization_compatible": self.initialization_compatible, + "protocol_version": self.protocol_version, + "advertised_capabilities": { + key: bool(self.advertised_capabilities.get(key, False)) + for key in _ADVERTISED_CAPABILITY_KEYS + }, + "authentication": { + "method_count": self.authentication_method_count, + "method_count_capped": self.authentication_method_count_capped, + }, + "extensions": { + "capability_count": self.extension_capability_count, + "capability_count_capped": self.extension_capability_count_capped, + }, + "process_reaped": self.process_reaped, + "failure": self.failure.value if self.failure is not None else None, + } + + +def probe_adapter( + argv: Sequence[str | os.PathLike[str]], + *, + cwd: str | os.PathLike[str] | None = None, + env: Mapping[str, str] | None = None, + timeout_seconds: float = DEFAULT_PROBE_TIMEOUT_SECONDS, + close_timeout_seconds: float = DEFAULT_PROBE_CLOSE_TIMEOUT_SECONDS, +) -> AcpAdapterProbeReport: + """Negotiate ACP v1 with one separately installed adapter executable. + + A new subprocess and initialization exchange are used for every call. No + session is created and no adapter source package is imported. All expected + failures become a fixed-category initialization-incompatible report; no raw + diagnostic text crosses this black-box boundary. + """ + + try: + timeout = _bounded_timeout( + timeout_seconds, + "timeout_seconds", + maximum=MAX_PROBE_TIMEOUT_SECONDS, + ) + close_timeout = _bounded_timeout( + close_timeout_seconds, + "close_timeout_seconds", + maximum=MAX_PROBE_CLOSE_TIMEOUT_SECONDS, + ) + client = BoundedAcpConnection( + argv, + cwd=cwd, + env=env, + request_timeout=timeout, + close_timeout=close_timeout, + max_frame_bytes=PROBE_MAX_FRAME_BYTES, + max_pending_events=PROBE_MAX_PENDING_EVENTS, + stderr_limit_bytes=PROBE_STDERR_LIMIT_BYTES, + ) + except Exception: + return _failure_report(ProbeFailure.INVALID_CONFIGURATION) + + initialized: InitializeResult | None = None + failure: ProbeFailure | None = None + started = False + try: + client.start() + started = True + initialized = client.initialize(timeout=timeout) + except Exception as exc: # exception text is intentionally never returned + failure = _failure_category(exc, started=started) + + process_reaped = False + try: + client.close() + except Exception: + if failure is None: + failure = ProbeFailure.SHUTDOWN + finally: + process = client.process + process_reaped = process is None or process.poll() is not None + if not process_reaped and failure is None: + failure = ProbeFailure.SHUTDOWN + + if initialized is None: + return _failure_report(failure or ProbeFailure.INTERNAL, process_reaped) + + capabilities = _capability_summary(initialized) + auth_count, auth_capped = _bounded_count( + sum(_valid_stable_auth_method(item) for item in initialized.auth_methods) + ) + extension_count, extension_capped = _bounded_count( + _extension_capability_count(initialized.capabilities.raw) + ) + initialization_compatible = ( + failure is None + and process_reaped + and initialized.protocol_version == ACP_PROTOCOL_VERSION + and client.failure is None + ) + return AcpAdapterProbeReport( + initialization_compatible=initialization_compatible, + protocol_version=initialized.protocol_version, + advertised_capabilities=MappingProxyType(capabilities), + authentication_method_count=auth_count, + authentication_method_count_capped=auth_capped, + extension_capability_count=extension_count, + extension_capability_count_capped=extension_capped, + process_reaped=process_reaped, + failure=failure, + ) + + +def _capability_summary(initialized: InitializeResult) -> dict[str, bool]: + raw = initialized.capabilities.raw + prompt = _mapping(raw.get("promptCapabilities")) + mcp = _mapping(raw.get("mcpCapabilities")) + session = _mapping(raw.get("sessionCapabilities")) + return { + "session_load": initialized.capabilities.load_session, + "session_list": initialized.capabilities.session_list, + "session_delete": initialized.capabilities.session_delete, + "session_resume": initialized.capabilities.session_resume, + "session_close": initialized.capabilities.session_close, + "additional_directories": initialized.capabilities.additional_directories, + "prompt_image": prompt.get("image") is True, + "prompt_audio": prompt.get("audio") is True, + "prompt_embedded_context": prompt.get("embeddedContext") is True, + "mcp_http": mcp.get("http") is True, + "mcp_sse": mcp.get("sse") is True, + "auth_logout": initialized.capabilities.auth_logout, + } + + +def _extension_capability_count(raw: Mapping[str, Any]) -> int: + """Count extension namespaces only where ACP v1 permits them: `_meta`.""" + count = _meta_entry_count(raw) + for section in ("promptCapabilities", "mcpCapabilities", "sessionCapabilities", "auth"): + nested = _mapping(raw.get(section)) + count += _meta_entry_count(nested) + session = _mapping(raw.get("sessionCapabilities")) + for capability in ("list", "delete", "additionalDirectories", "resume", "close"): + count += _meta_entry_count(_mapping(session.get(capability))) + auth = _mapping(raw.get("auth")) + count += _meta_entry_count(_mapping(auth.get("logout"))) + return count + + +def _meta_entry_count(value: Mapping[str, Any]) -> int: + meta = value.get("_meta") + return len(meta) if isinstance(meta, Mapping) else 0 + + +def _valid_stable_auth_method(value: Mapping[str, Any]) -> bool: + method_type = value.get("type", "agent") + return ( + method_type == "agent" + and isinstance(value.get("id"), str) + and isinstance(value.get("name"), str) + ) + + +def _mapping(value: Any) -> Mapping[str, Any]: + return value if isinstance(value, Mapping) else {} + + +def _bounded_count(value: int) -> tuple[int, bool]: + return min(value, _MAX_REPORTED_COUNT), value > _MAX_REPORTED_COUNT + + +def _failure_report( + failure: ProbeFailure, + process_reaped: bool = True, +) -> AcpAdapterProbeReport: + return AcpAdapterProbeReport( + initialization_compatible=False, + protocol_version=None, + advertised_capabilities=MappingProxyType( + {key: False for key in _ADVERTISED_CAPABILITY_KEYS} + ), + authentication_method_count=0, + authentication_method_count_capped=False, + extension_capability_count=0, + extension_capability_count_capped=False, + process_reaped=process_reaped, + failure=failure, + ) + + +def _failure_category(exc: Exception, *, started: bool) -> ProbeFailure: + if isinstance(exc, AcpRequestTimeoutError): + return ProbeFailure.TIMEOUT + if isinstance(exc, AcpProtocolVersionError): + return ProbeFailure.PROTOCOL_VERSION + if isinstance(exc, (AcpFramingError, AcpEnvelopeError, AcpRemoteError)): + return ProbeFailure.PROTOCOL + if isinstance(exc, AcpTransportError): + return ProbeFailure.TRANSPORT if started else ProbeFailure.LAUNCH_FAILED + if isinstance(exc, (TypeError, ValueError, OSError)): + return ProbeFailure.INVALID_CONFIGURATION + return ProbeFailure.INTERNAL + + +def _bounded_timeout(value: float, name: str, *, maximum: float) -> float: + result = float(value) + if not math.isfinite(result) or not 0 < result <= maximum: + raise ValueError(f"{name} must be positive and at most {maximum:g}") + return result + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="python -m tendwire.backends.acp_probe", + description="Probe a separately installed ACP v1 adapter executable.", + ) + parser.add_argument( + "--timeout", + type=float, + default=DEFAULT_PROBE_TIMEOUT_SECONDS, + help=f"initialization timeout in seconds (maximum {MAX_PROBE_TIMEOUT_SECONDS:g})", + ) + parser.add_argument( + "--close-timeout", + type=float, + default=DEFAULT_PROBE_CLOSE_TIMEOUT_SECONDS, + help=( + "shutdown stage timeout in seconds " + f"(maximum {MAX_PROBE_CLOSE_TIMEOUT_SECONDS:g})" + ), + ) + parser.add_argument( + "--cwd", + default=None, + help="absolute process working directory (never included in output)", + ) + parser.add_argument( + "adapter_argv", + nargs=argparse.REMAINDER, + help="adapter executable and arguments, conventionally after --", + ) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + parser = _build_parser() + args = parser.parse_args(argv) + adapter_argv = list(args.adapter_argv) + if adapter_argv and adapter_argv[0] == "--": + adapter_argv.pop(0) + if not adapter_argv: + parser.error("an adapter executable is required after --") + report = probe_adapter( + adapter_argv, + cwd=args.cwd, + timeout_seconds=args.timeout, + close_timeout_seconds=args.close_timeout, + ) + sys.stdout.write(json.dumps(report.to_payload(), sort_keys=True, separators=(",", ":"))) + sys.stdout.write("\n") + return 0 if report.initialization_compatible else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/tendwire/backends/acp_projection.py b/src/tendwire/backends/acp_projection.py new file mode 100644 index 0000000..231920b --- /dev/null +++ b/src/tendwire/backends/acp_projection.py @@ -0,0 +1,1299 @@ +"""Stateful, transport-independent projection of ACP session events. + +The projector deliberately does not own an ACP connection. It accepts the +JSON-shaped values carried by ``session/update`` notifications and +``session/request_permission`` requests, then emits stable mappings suitable +for a durable event journal. This keeps protocol transport, persistence, and +Tendwire's public-content boundary separate. + +ACP streams reasoning and tool data independently from assistant messages. +That distinction is preserved here: the compatibility turn projection only +contains user and assistant text and can never expose thought or tool payloads. +""" + +from __future__ import annotations + +import hashlib +import json +import math +import os +from collections.abc import Mapping +from copy import deepcopy +from dataclasses import dataclass, field +from typing import Any, Final + + +SUPPORTED_EVENT_KINDS: Final[frozenset[str]] = frozenset( + { + "user_message", + "agent_message", + "thought", + "tool_call", + "tool_call_update", + "plan", + "usage", + "session_info", + "extension", + } +) + +_UPDATE_KIND_MAP: Final[dict[str, str]] = { + "user_message_chunk": "user_message", + "agent_message_chunk": "agent_message", + "agent_thought_chunk": "thought", + "tool_call": "tool_call", + "tool_call_update": "tool_call_update", + "plan": "plan", + "available_commands_update": "extension", + "current_mode_update": "extension", + "config_option_update": "extension", + "usage_update": "usage", + "session_info_update": "session_info", +} +_RAW_TOOL_FIELDS: Final[tuple[str, str]] = ("rawInput", "rawOutput") +_MESSAGE_KINDS: Final[frozenset[str]] = frozenset( + {"user_message", "agent_message", "thought"} +) +_LEGACY_EMPTY: Final[dict[str, Any]] = { + "user_text": "", + "assistant_stream_text": "", + "assistant_final_text": "", + "complete": False, + "has_open_turn": False, +} +_MAX_IDENTIFIER_CHARS: Final[int] = 2048 +_MAX_SOURCE_ID_CHARS: Final[int] = 512 +_TOOL_KINDS: Final[frozenset[str]] = frozenset( + { + "read", + "edit", + "delete", + "move", + "search", + "execute", + "think", + "fetch", + "switch_mode", + "other", + } +) +_TOOL_STATUSES: Final[frozenset[str]] = frozenset( + {"pending", "in_progress", "completed", "failed"} +) +_PLAN_PRIORITIES: Final[frozenset[str]] = frozenset({"high", "medium", "low"}) +_PLAN_STATUSES: Final[frozenset[str]] = frozenset( + {"pending", "in_progress", "completed"} +) +_PERMISSION_KINDS: Final[frozenset[str]] = frozenset( + {"allow_once", "allow_always", "reject_once", "reject_always"} +) +_CONTENT_TYPES: Final[frozenset[str]] = frozenset( + {"text", "image", "audio", "resource_link", "resource"} +) +_ANNOTATION_ROLES: Final[frozenset[str]] = frozenset({"assistant", "user"}) + + +class AcpProjectionError(ValueError): + """Raised when an ACP value cannot be safely normalized.""" + + +@dataclass +class _MessageAssembly: + message_id: str + text: str = "" + explicit: bool = False + + +@dataclass +class _SessionState: + sequence: int = 0 + messages: dict[str, list[_MessageAssembly]] = field( + default_factory=lambda: {kind: [] for kind in _MESSAGE_KINDS} + ) + tools: dict[str, dict[str, Any]] = field(default_factory=dict) + plan: list[dict[str, Any]] = field(default_factory=list) + usage: dict[str, Any] = field(default_factory=dict) + info: dict[str, Any] = field(default_factory=dict) + # The digest lets us distinguish a harmless replay from a producer reusing + # one supposedly authoritative ID for different content. + seen_source_events: dict[str, str] = field(default_factory=dict) + retained_bytes: int = 0 + complete: bool = False + active_message: tuple[str, str] | None = None + last_update_name: str | None = None + implicit_message_ordinals: dict[str, int] = field(default_factory=dict) + replaced_state: _SessionState | None = field(default=None, repr=False) + + +@dataclass(frozen=True) +class AcpProjectionCheckpoint: + """Opaque rollback point for one session's in-memory projection state.""" + + session_id: str + state: _SessionState | None + + +class AcpEventProjector: + """Normalize ACP notifications while retaining per-session assembly state. + + The instance is intentionally in-memory. ``dedupe_hint`` is emitted on + every event so a durable caller can deduplicate replays across process + restarts. Within one instance, events with an explicit protocol/transport + identifier supplied by the transport as ``source_event_id`` are dropped + when repeated. ACP ``_meta`` is opaque and is never trusted as identity. + Content hashes are hints only: identical adjacent chunks can be legitimate + and are therefore never blindly discarded. + """ + + def __init__( + self, + *, + max_sessions: int = 64, + max_source_events_per_session: int = 4096, + max_messages_per_kind: int = 1024, + max_tool_calls_per_session: int = 4096, + max_state_fields: int = 1024, + max_plan_entries: int = 4096, + max_text_chars_per_message: int = 4 * 1024 * 1024, + max_event_bytes: int = 8 * 1024 * 1024, + max_json_depth: int = 128, + max_session_state_bytes: int = 8 * 1024 * 1024, + max_total_state_bytes: int = 128 * 1024 * 1024, + ) -> None: + limits = { + "max_sessions": max_sessions, + "max_source_events_per_session": max_source_events_per_session, + "max_messages_per_kind": max_messages_per_kind, + "max_tool_calls_per_session": max_tool_calls_per_session, + "max_state_fields": max_state_fields, + "max_plan_entries": max_plan_entries, + "max_text_chars_per_message": max_text_chars_per_message, + "max_event_bytes": max_event_bytes, + "max_json_depth": max_json_depth, + "max_session_state_bytes": max_session_state_bytes, + "max_total_state_bytes": max_total_state_bytes, + } + for name, value in limits.items(): + if isinstance(value, bool) or not isinstance(value, int) or value < 1: + raise ValueError(f"{name} must be a positive integer") + self._sessions: dict[str, _SessionState] = {} + self._max_sessions = max_sessions + self._max_source_events_per_session = max_source_events_per_session + self._max_messages_per_kind = max_messages_per_kind + self._max_tool_calls_per_session = max_tool_calls_per_session + self._max_state_fields = max_state_fields + self._max_plan_entries = max_plan_entries + self._max_text_chars_per_message = max_text_chars_per_message + self._max_event_bytes = max_event_bytes + self._max_json_depth = max_json_depth + self._max_session_state_bytes = max_session_state_bytes + self._max_total_state_bytes = max_total_state_bytes + + def normalize_session_update( + self, + notification: Mapping[str, Any], + *, + source_event_id: str | None = None, + replay: bool = False, + ) -> dict[str, Any] | None: + """Normalize one ACP ``session/update`` notification or its params. + + ``notification`` may be a full JSON-RPC notification, its ``params`` + object, or a direct object containing ``sessionId`` and ``update``. + Unknown ACP update variants are ignored for forward compatibility. + """ + + params = _unwrap_params(notification) + session_id = _required_string(params, "sessionId") + update = params.get("update") + if not isinstance(update, Mapping): + raise AcpProjectionError("ACP session/update is missing an object update") + + update_name = update.get("sessionUpdate") + if not isinstance(update_name, str): + raise AcpProjectionError("ACP update is missing sessionUpdate") + kind = _UPDATE_KIND_MAP.get(update_name) + if kind is None: + return None + + _bounded_json( + {"update": update, "_meta": params.get("_meta")}, + label="ACP session update", + max_bytes=self._max_event_bytes, + max_depth=self._max_json_depth, + ) + update = _normalized_supported_update(update_name, update) + state, _is_new_session = self._pending_session(session_id) + explicit_id = _explicit_source_event_id(source_event_id) + replay_digest = _event_digest(kind, update) + if explicit_id is not None: + previous_digest = state.seen_source_events.get(explicit_id) + if previous_digest == replay_digest: + return None + if previous_digest is not None: + raise AcpProjectionError( + "ACP source event ID was reused for different content" + ) + if len(state.seen_source_events) >= self._max_source_events_per_session: + raise AcpProjectionError("ACP source event replay window is full") + + if kind in _MESSAGE_KINDS: + if state.complete: + raise AcpProjectionError( + "ACP turn is complete; reset_turn is required before new message chunks" + ) + payload = self._normalize_message(state, kind, update) + elif kind in {"tool_call", "tool_call_update"}: + payload = self._normalize_tool(state, kind, update) + elif kind == "plan": + payload = self._normalize_plan(state, update) + elif kind == "usage": + payload = self._normalize_usage(state, update) + elif kind == "extension": + payload = { + "schema_version": 1, + "extension": f"acp.session_update.{update_name}", + "update": _without_discriminator(update), + } + else: + payload = self._normalize_session_info(state, update) + extension_meta = _scoped_metadata(params=params, update=update) + if extension_meta: + payload["extensions"] = extension_meta + + state.sequence += 1 + state.last_update_name = update_name + if kind not in _MESSAGE_KINDS and state.active_message is not None: + active_kind, active_id = state.active_message + active = next( + ( + item + for item in state.messages[active_kind] + if item.message_id == active_id + ), + None, + ) + if active is not None and not active.explicit: + state.active_message = None + if explicit_id is not None: + self._reserve_state( + state, + len(explicit_id.encode("utf-8")) + len(replay_digest.encode("ascii")), + ) + state.seen_source_events[explicit_id] = replay_digest + state.replaced_state = None + self._sessions[session_id] = state + return _canonical_event( + session_id=session_id, + sequence=state.sequence, + kind=kind, + payload=payload, + source_event_id=explicit_id, + replay=replay, + original_update=update, + ) + + def normalize_permission_request( + self, + request: Mapping[str, Any], + *, + source_event_id: str | None = None, + replay: bool = False, + ) -> dict[str, Any] | None: + """Project ``session/request_permission`` as a tool lifecycle update. + + ACP attaches permission requests to tool calls, so no synthetic ninth + event kind is introduced. Options are retained in the canonical tool + payload for a decision layer to consume. + """ + + params = _unwrap_params(request) + session_id = _required_string(params, "sessionId") + tool_call = params.get("toolCall") + if not isinstance(tool_call, Mapping): + raise AcpProjectionError("ACP permission request is missing toolCall") + tool_call_id = _required_string(tool_call, "toolCallId") + + _bounded_json( + params, + label="ACP permission request", + max_bytes=self._max_event_bytes, + max_depth=self._max_json_depth, + ) + tool_call = _normalized_tool_update( + tool_call, label="ACP permission toolCall" + ) + state, _is_new_session = self._pending_session(session_id) + explicit_id = _explicit_source_event_id(source_event_id) + options = params.get("options") + if not isinstance(options, list) or not options: + raise AcpProjectionError("ACP permission request options must be non-empty") + normalized_options = _permission_options(options) + request_material = {"toolCall": tool_call, "options": normalized_options} + replay_digest = _event_digest("tool_call_update", request_material) + if explicit_id is not None: + previous_digest = state.seen_source_events.get(explicit_id) + if previous_digest == replay_digest: + return None + if previous_digest is not None: + raise AcpProjectionError( + "ACP source event ID was reused for different content" + ) + if len(state.seen_source_events) >= self._max_source_events_per_session: + raise AcpProjectionError("ACP source event replay window is full") + + if ( + tool_call_id not in state.tools + and len(state.tools) >= self._max_tool_calls_per_session + ): + raise AcpProjectionError("ACP tool call state limit exceeded") + previous_tool = state.tools.get(tool_call_id) + snapshot = _merge_tool_snapshot(previous_tool, tool_call) + snapshot["permission"] = { + "required": True, + "options": deepcopy(normalized_options), + } + if len(snapshot) > self._max_state_fields: + raise AcpProjectionError("ACP tool snapshot field limit exceeded") + self._reserve_state( + state, + _json_size(snapshot) - (_json_size(previous_tool) if previous_tool else 0), + ) + state.tools[tool_call_id] = snapshot + state.sequence += 1 + state.last_update_name = "tool_call_update" + if state.active_message is not None: + active_kind, active_id = state.active_message + active = next( + ( + item + for item in state.messages[active_kind] + if item.message_id == active_id + ), + None, + ) + if active is not None and not active.explicit: + state.active_message = None + if explicit_id is not None: + self._reserve_state( + state, + len(explicit_id.encode("utf-8")) + len(replay_digest.encode("ascii")), + ) + state.seen_source_events[explicit_id] = replay_digest + state.replaced_state = None + self._sessions[session_id] = state + payload = { + "tool_call_id": tool_call_id, + "changes": _without_discriminator(tool_call), + "snapshot": deepcopy(snapshot), + "permission": deepcopy(snapshot["permission"]), + } + extension_meta = _scoped_metadata(params=params, toolCall=tool_call) + if extension_meta: + payload["extensions"] = extension_meta + return _canonical_event( + session_id=session_id, + sequence=state.sequence, + kind="tool_call_update", + payload=payload, + source_event_id=explicit_id, + replay=replay, + original_update={ + "sessionUpdate": "tool_call_update", + "toolCall": dict(tool_call), + "options": normalized_options, + }, + ) + + def project_turn_content( + self, + session_id: str, + *, + complete: bool | None = None, + ) -> dict[str, Any]: + """Return Tendwire's legacy text-only turn shape for one ACP session. + + Thought text, plans, tool content, raw inputs, raw outputs, and + permission details are structurally unreachable from this projection. + ``complete=True`` moves assembled assistant text from stream to final. + Callers should only set it after the ACP ``session/prompt`` response. + """ + + state = self._sessions.get(session_id) + if state is None: + return dict(_LEGACY_EMPTY) + # A read-only projection may promote a snapshot to final, but must + # never demote already-final state. Reopening requires reset_turn(). + is_complete = state.complete or ( + bool(complete) if complete is not None else False + ) + user_text = _joined_messages(state.messages["user_message"]) + assistant_text = _joined_messages(state.messages["agent_message"]) + return { + "user_text": user_text, + "assistant_stream_text": "" if is_complete else assistant_text, + "assistant_final_text": assistant_text if is_complete else "", + "complete": is_complete, + "has_open_turn": bool(user_text or assistant_text) and not is_complete, + } + + def mark_turn_complete(self, session_id: str) -> dict[str, Any]: + """Mark the current ACP prompt turn complete and return legacy content.""" + + state, _is_new_session = self._pending_session(session_id) + state.complete = True + state.replaced_state = None + self._sessions[session_id] = state + return self.project_turn_content(session_id) + + def reset_turn(self, session_id: str) -> None: + """Start a fresh prompt turn while preserving session-level ACP state.""" + + state, _is_new_session = self._pending_session(session_id) + state.retained_bytes = max( + 0, state.retained_bytes - _messages_state_bytes(state.messages) + ) + state.messages = {kind: [] for kind in _MESSAGE_KINDS} + state.complete = False + state.active_message = None + state.last_update_name = None + state.implicit_message_ordinals = {} + state.replaced_state = None + self._sessions[session_id] = state + + def checkpoint_session(self, session_id: str) -> AcpProjectionCheckpoint: + """Capture one session so a failed durable append can be rolled back.""" + + return AcpProjectionCheckpoint( + session_id=session_id, + state=self._sessions.get(session_id), + ) + + def restore_session(self, checkpoint: AcpProjectionCheckpoint) -> None: + """Restore a checkpoint created before a speculative normalization.""" + + if not isinstance(checkpoint, AcpProjectionCheckpoint): + raise TypeError("checkpoint must be an AcpProjectionCheckpoint") + if checkpoint.state is None: + self._sessions.pop(checkpoint.session_id, None) + else: + self._sessions[checkpoint.session_id] = checkpoint.state + + def drop_session(self, session_id: str) -> bool: + """Release all in-memory state after the owning ACP session is closed.""" + + return self._sessions.pop(session_id, None) is not None + + def session_snapshot(self, session_id: str) -> dict[str, Any] | None: + """Return a defensive snapshot for persistence or diagnostics.""" + + state = self._sessions.get(session_id) + if state is None: + return None + return { + "session_id": session_id, + "sequence": state.sequence, + "messages": { + kind: [ + {"message_id": message.message_id, "text": message.text} + for message in messages + ] + for kind, messages in state.messages.items() + }, + "tools": deepcopy(state.tools), + "plan": deepcopy(state.plan), + "usage": deepcopy(state.usage), + "session_info": deepcopy(state.info), + "complete": state.complete, + } + + def _normalize_message( + self, + state: _SessionState, + kind: str, + update: Mapping[str, Any], + ) -> dict[str, Any]: + content = update["content"] + assert isinstance(content, Mapping) + message_id_value = update.get("messageId") + assemblies = state.messages[kind] + if message_id_value is not None: + message_id = _identifier(message_id_value, "messageId") + explicit = True + else: + explicit = False + active = state.active_message + if active is not None and active[0] == kind: + candidate = next( + ( + item + for item in assemblies + if item.message_id == active[1] and not item.explicit + ), + None, + ) + else: + candidate = None + if state.last_update_name == update.get("sessionUpdate") and candidate is not None: + message_id = candidate.message_id + else: + ordinal = state.implicit_message_ordinals.get(kind, 0) + 1 + message_id = f"implicit-{kind}-{ordinal}" + active_key = state.active_message + assembly = None + if active_key == (kind, message_id): + assembly = next( + (item for item in assemblies if item.message_id == message_id), None + ) + elif explicit and any( + item.explicit and item.message_id == message_id + for items in state.messages.values() + for item in items + ): + raise AcpProjectionError("ACP messageId was reused after a message boundary") + text_delta = content.get("text") if content.get("type") == "text" else None + text_delta = text_delta if isinstance(text_delta, str) else "" + public_text_delta = text_delta if _content_is_user_visible(content) else "" + content_copy = _content_payload(content) + previous_text = assembly.text if assembly is not None else "" + assembled_text = previous_text + public_text_delta + if len(assembled_text) > self._max_text_chars_per_message: + raise AcpProjectionError("ACP assembled message text limit exceeded") + if assembly is None: + if len(assemblies) >= self._max_messages_per_kind: + raise AcpProjectionError("ACP message assembly limit exceeded") + self._reserve_state( + state, + len(message_id.encode("utf-8")) + + len(public_text_delta.encode("utf-8")), + ) + assembly = _MessageAssembly( + message_id=message_id, + text=assembled_text, + explicit=explicit, + ) + assemblies.append(assembly) + if not explicit: + state.implicit_message_ordinals[kind] = int(message_id.rsplit("-", 1)[1]) + else: + self._reserve_state(state, len(public_text_delta.encode("utf-8"))) + assembly.text = assembled_text + state.active_message = (kind, message_id) + return { + "message_id": message_id, + "content": content_copy, + "text_delta": text_delta, + "assembled_text": assembled_text, + "message_index": assemblies.index(assembly), + } + + def _normalize_tool( + self, + state: _SessionState, + kind: str, + update: Mapping[str, Any], + ) -> dict[str, Any]: + tool_call_id = _required_string(update, "toolCallId") + previous = state.tools.get(tool_call_id) + if previous is None and len(state.tools) >= self._max_tool_calls_per_session: + raise AcpProjectionError("ACP tool call state limit exceeded") + snapshot = _merge_tool_snapshot(previous, update) + if len(snapshot) > self._max_state_fields: + raise AcpProjectionError("ACP tool snapshot field limit exceeded") + payload: dict[str, Any] = { + "tool_call_id": tool_call_id, + "snapshot": deepcopy(snapshot), + } + if kind == "tool_call_update": + payload["changes"] = _without_discriminator(update) + self._reserve_state( + state, _json_size(snapshot) - (_json_size(previous) if previous else 0) + ) + state.tools[tool_call_id] = snapshot + return payload + + def _normalize_plan( + self, state: _SessionState, update: Mapping[str, Any] + ) -> dict[str, Any]: + entries = update.get("entries") + if not isinstance(entries, list): + raise AcpProjectionError("ACP plan update entries must be an array") + if len(entries) > self._max_plan_entries: + raise AcpProjectionError("ACP plan entry limit exceeded") + replacement = [_normalized_plan_entry(entry) for entry in entries] + payload: dict[str, Any] = {"entries": deepcopy(replacement), "snapshot": True} + self._reserve_state(state, _json_size(replacement) - _json_size(state.plan)) + state.plan = replacement + return payload + + def _normalize_usage( + self, state: _SessionState, update: Mapping[str, Any] + ) -> dict[str, Any]: + replacement = { + "used": update["used"], + "size": update["size"], + **({"cost": deepcopy(update["cost"])} if update.get("cost") is not None else {}), + } + if len(replacement) > self._max_state_fields: + raise AcpProjectionError("ACP usage state field limit exceeded") + self._reserve_state(state, _json_size(replacement) - _json_size(state.usage)) + state.usage = replacement + payload = deepcopy(state.usage) + return payload + + def _normalize_session_info( + self, state: _SessionState, update: Mapping[str, Any] + ) -> dict[str, Any]: + # Presence is meaningful: explicit null clears an existing property. + changes = { + key: deepcopy(update[key]) for key in ("title", "updatedAt") if key in update + } + replacement = {**state.info, **changes} + if len(replacement) > self._max_state_fields: + raise AcpProjectionError("ACP session info state field limit exceeded") + self._reserve_state(state, _json_size(replacement) - _json_size(state.info)) + state.info = replacement + payload = deepcopy(state.info) + return payload + + def _pending_session(self, session_id: str) -> tuple[_SessionState, bool]: + state = self._sessions.get(session_id) + if state is not None: + return _copy_session_state(state), False + if len(self._sessions) >= self._max_sessions: + raise AcpProjectionError("ACP projector session limit exceeded") + return _SessionState(), True + + def _reserve_state(self, state: _SessionState, retained_delta: int) -> None: + retained = max(0, state.retained_bytes + retained_delta) + if retained > self._max_session_state_bytes: + raise AcpProjectionError("ACP retained session state limit exceeded") + other_retained = sum( + item.retained_bytes + for item in self._sessions.values() + if item is not state and item is not state.replaced_state + ) + if other_retained + retained > self._max_total_state_bytes: + raise AcpProjectionError("ACP total retained state limit exceeded") + state.retained_bytes = retained + + +def _unwrap_params(value: Mapping[str, Any]) -> Mapping[str, Any]: + params = value.get("params") + if isinstance(params, Mapping): + return params + return value + + +def _copy_session_state(state: _SessionState) -> _SessionState: + """Copy mutable indexes while sharing immutable text and untouched snapshots.""" + + return _SessionState( + sequence=state.sequence, + messages={ + kind: [ + _MessageAssembly( + message_id=message.message_id, + text=message.text, + explicit=message.explicit, + ) + for message in messages + ] + for kind, messages in state.messages.items() + }, + tools=dict(state.tools), + plan=state.plan, + usage=state.usage, + info=state.info, + seen_source_events=dict(state.seen_source_events), + retained_bytes=state.retained_bytes, + complete=state.complete, + active_message=state.active_message, + last_update_name=state.last_update_name, + implicit_message_ordinals=dict(state.implicit_message_ordinals), + replaced_state=state, + ) + + +def _required_string(value: Mapping[str, Any], key: str) -> str: + item = value.get(key) + try: + return _identifier(item, key) + except AcpProjectionError: + raise AcpProjectionError(f"ACP value is missing non-empty {key}") from None + + +def _identifier(value: Any, label: str) -> str: + if not isinstance(value, str): + raise AcpProjectionError(f"ACP value has invalid {label}") + try: + value.encode("utf-8") + except UnicodeEncodeError as exc: + raise AcpProjectionError(f"ACP value has invalid {label}") from exc + if not value or len(value) > _MAX_IDENTIFIER_CHARS or "\x00" in value: + raise AcpProjectionError(f"ACP value has invalid {label}") + return value + + +def _explicit_source_event_id(value: Any) -> str | None: + if value is None: + return None + return _source_identifier(value, "source_event_id") + + +def _source_identifier(value: Any, label: str) -> str: + identifier = _identifier(value, label) + if len(identifier) > _MAX_SOURCE_ID_CHARS: + raise AcpProjectionError(f"ACP value has invalid {label}") + return identifier + + +def _without_discriminator(value: Mapping[str, Any]) -> dict[str, Any]: + return { + key: deepcopy(item) + for key, item in value.items() + if key not in {"sessionUpdate", "_meta"} + } + + +def _merge_tool_snapshot( + previous: Mapping[str, Any] | None, update: Mapping[str, Any] +) -> dict[str, Any]: + snapshot = deepcopy(dict(previous)) if previous is not None else {} + if update.get("sessionUpdate") == "tool_call": + snapshot.setdefault("kind", "other") + snapshot.setdefault("status", "pending") + snapshot.setdefault("content", []) + snapshot.setdefault("locations", []) + for key in ( + "toolCallId", + "title", + "kind", + "status", + "content", + "locations", + "rawInput", + "rawOutput", + ): + if key in update and update[key] is not None: + snapshot[key] = deepcopy(update[key]) + return snapshot + + +def _content_payload(content: Mapping[str, Any]) -> dict[str, Any]: + """Retain a validated content block, including opaque standard ``_meta``.""" + + return deepcopy(dict(content)) + + +def _content_is_user_visible(content: Mapping[str, Any]) -> bool: + annotations = content.get("annotations") + if not isinstance(annotations, Mapping) or "audience" not in annotations: + return True + audience = annotations.get("audience") + if audience is None: + return True + return isinstance(audience, list) and "user" in audience + + +def _extension_metadata(value: Mapping[str, Any]) -> dict[str, Any]: + meta = value.get("_meta") + if meta is None or not isinstance(meta, Mapping) or any( + not isinstance(key, str) for key in meta + ): + return {} + return _safe_deepcopy(dict(meta), label="ACP _meta") + + +def _scoped_metadata(**values: Mapping[str, Any]) -> dict[str, Any]: + """Preserve each protocol object's opaque metadata without key collisions.""" + + return { + scope: metadata + for scope, value in values.items() + if (metadata := _extension_metadata(value)) + } + + +def _normalized_supported_update( + update_name: str, update: Mapping[str, Any] +) -> dict[str, Any]: + """Apply ACP v1's tolerant decoding rules to optional/list fields.""" + + normalized = _safe_deepcopy(dict(update), label="ACP session update") + _salvage_meta(normalized) + if update_name in { + "user_message_chunk", + "agent_message_chunk", + "agent_thought_chunk", + }: + content = normalized.get("content") + if not isinstance(content, Mapping): + raise AcpProjectionError("ACP message update is missing object content") + normalized["content"] = _normalized_content_block( + content, label="ACP message content" + ) + if "messageId" in normalized and normalized["messageId"] is not None: + _identifier(normalized["messageId"], "messageId") + elif update_name == "tool_call": + normalized = _normalized_tool_call(normalized) + elif update_name == "tool_call_update": + normalized = _normalized_tool_update(normalized) + elif update_name == "plan": + if "entries" not in normalized: + raise AcpProjectionError("ACP plan update entries must be an array") + entries = normalized["entries"] + if not isinstance(entries, list): + normalized["entries"] = [] + else: + accepted: list[dict[str, Any]] = [] + for entry in entries: + try: + accepted.append(_normalized_plan_entry(entry)) + except AcpProjectionError: + continue + normalized["entries"] = accepted + elif update_name == "usage_update": + _validate_usage(normalized) + if normalized.get("cost") is not None: + try: + cost = normalized["cost"] + if not isinstance(cost, Mapping): + raise AcpProjectionError("ACP usage cost must be an object or null") + cost_copy = _safe_deepcopy(dict(cost), label="ACP usage cost") + _salvage_meta(cost_copy) + _validate_cost(cost_copy) + normalized["cost"] = cost_copy + except (AcpProjectionError, OverflowError): + normalized.pop("cost", None) + elif update_name == "session_info_update": + for key in ("title", "updatedAt"): + if key in normalized and normalized[key] is not None and not isinstance( + normalized[key], str + ): + normalized.pop(key) + return normalized + + +def _salvage_meta(value: dict[str, Any]) -> None: + meta = value.get("_meta") + if meta is not None and ( + not isinstance(meta, Mapping) or any(not isinstance(key, str) for key in meta) + ): + value.pop("_meta", None) + + +def _normalized_annotations(value: Any) -> dict[str, Any] | None: + if not isinstance(value, Mapping): + return None + normalized = _safe_deepcopy(dict(value), label="ACP annotations") + _salvage_meta(normalized) + audience = normalized.get("audience") + if isinstance(audience, list): + normalized["audience"] = [item for item in audience if item in _ANNOTATION_ROLES] + elif audience is not None: + normalized.pop("audience", None) + if normalized.get("lastModified") is not None and not isinstance( + normalized.get("lastModified"), str + ): + normalized.pop("lastModified", None) + priority = normalized.get("priority") + if priority is not None and not _is_finite_number(priority): + normalized.pop("priority", None) + return normalized + + +def _normalized_content_block( + content: Mapping[str, Any], *, label: str +) -> dict[str, Any]: + normalized = _safe_deepcopy(dict(content), label=label) + content_type = normalized.get("type") + if content_type not in _CONTENT_TYPES: + raise AcpProjectionError(f"{label} has unsupported type") + _salvage_meta(normalized) + annotations = _normalized_annotations(normalized.get("annotations")) + if annotations is None: + normalized.pop("annotations", None) + else: + normalized["annotations"] = annotations + if content_type == "text": + _required_text(normalized, "text", label=label) + elif content_type in {"image", "audio"}: + _required_text(normalized, "data", label=label) + _required_text(normalized, "mimeType", label=label) + if content_type == "image" and normalized.get("uri") is not None and not isinstance( + normalized.get("uri"), str + ): + normalized.pop("uri", None) + elif content_type == "resource_link": + _required_text(normalized, "name", label=label) + _required_text(normalized, "uri", label=label) + for key in ("description", "mimeType", "title"): + if normalized.get(key) is not None and not isinstance(normalized.get(key), str): + normalized.pop(key, None) + size = normalized.get("size") + if size is not None and ( + isinstance(size, bool) + or not isinstance(size, int) + or not -(2**63) <= size <= 2**63 - 1 + ): + normalized.pop("size", None) + else: + resource = normalized.get("resource") + if not isinstance(resource, Mapping): + raise AcpProjectionError(f"{label} resource must be an object") + resource_copy = _safe_deepcopy(dict(resource), label=f"{label} resource") + _salvage_meta(resource_copy) + _required_text(resource_copy, "uri", label=f"{label} resource") + has_text = "text" in resource_copy + has_blob = "blob" in resource_copy + if has_text == has_blob: + raise AcpProjectionError( + f"{label} resource must contain exactly one of text or blob" + ) + _required_text( + resource_copy, + "text" if has_text else "blob", + label=f"{label} resource", + ) + if resource_copy.get("mimeType") is not None and not isinstance( + resource_copy.get("mimeType"), str + ): + resource_copy.pop("mimeType", None) + normalized["resource"] = resource_copy + return normalized + + +def _permission_options(options: list[Any]) -> list[dict[str, Any]]: + normalized: list[dict[str, Any]] = [] + seen: set[str] = set() + for option in options: + if not isinstance(option, Mapping): + raise AcpProjectionError("ACP permission request option must be an object") + option_copy = _safe_deepcopy(dict(option), label="ACP permission option") + _salvage_meta(option_copy) + option_id = _required_string(option_copy, "optionId") + _required_string(option_copy, "name") + kind = _required_string(option_copy, "kind") + if kind not in _PERMISSION_KINDS: + raise AcpProjectionError("ACP permission option has invalid kind") + if option_id in seen: + raise AcpProjectionError("ACP permission option IDs must be unique") + seen.add(option_id) + normalized.append(option_copy) + return normalized + + +def _normalized_tool_call(update: Mapping[str, Any]) -> dict[str, Any]: + normalized = _safe_deepcopy(dict(update), label="ACP tool call") + _required_string(normalized, "toolCallId") + if not isinstance(normalized.get("title"), str): + raise AcpProjectionError("ACP tool_call is missing string title") + _salvage_tool_fields(normalized, creation=True, label="ACP tool call") + return normalized + + +def _normalized_tool_update( + update: Mapping[str, Any], *, label: str = "ACP tool_call_update" +) -> dict[str, Any]: + normalized = _safe_deepcopy(dict(update), label=label) + _required_string(normalized, "toolCallId") + _salvage_tool_fields(normalized, creation=False, label=label) + return normalized + + +def _salvage_tool_fields( + update: dict[str, Any], *, creation: bool, label: str +) -> None: + _salvage_meta(update) + kind = update.get("kind") + if kind is not None and (not isinstance(kind, str) or kind not in _TOOL_KINDS): + update.pop("kind", None) + status = update.get("status") + if status is not None and ( + not isinstance(status, str) or status not in _TOOL_STATUSES + ): + update.pop("status", None) + if not creation and update.get("title") is not None and not isinstance( + update.get("title"), str + ): + update.pop("title", None) + for key, normalizer in ( + ("content", _normalized_tool_content), + ("locations", _normalized_tool_location), + ): + value = update.get(key) + if not isinstance(value, list): + if value is not None: + update.pop(key, None) + continue + accepted: list[dict[str, Any]] = [] + for item in value: + try: + accepted.append(normalizer(item)) + except AcpProjectionError: + continue + update[key] = accepted + + +def _normalized_tool_content(value: Any) -> dict[str, Any]: + if not isinstance(value, Mapping): + raise AcpProjectionError("ACP tool content item must be an object") + normalized = _safe_deepcopy(dict(value), label="ACP tool content item") + _salvage_meta(normalized) + item_type = normalized.get("type") + if item_type == "content": + content = normalized.get("content") + if not isinstance(content, Mapping): + raise AcpProjectionError("ACP tool content is missing content block") + normalized["content"] = _normalized_content_block( + content, label="ACP tool content block" + ) + elif item_type == "diff": + path = _required_text(normalized, "path", label="ACP tool diff") + if not os.path.isabs(path): + raise AcpProjectionError("ACP tool diff path must be absolute") + _required_text(normalized, "newText", label="ACP tool diff") + if normalized.get("oldText") is not None and not isinstance( + normalized.get("oldText"), str + ): + normalized.pop("oldText", None) + elif item_type == "terminal": + _required_string(normalized, "terminalId") + else: + raise AcpProjectionError("ACP tool content item has unsupported type") + return normalized + + +def _normalized_tool_location(value: Any) -> dict[str, Any]: + if not isinstance(value, Mapping): + raise AcpProjectionError("ACP tool location must be an object") + normalized = _safe_deepcopy(dict(value), label="ACP tool location") + _salvage_meta(normalized) + path = _required_text(normalized, "path", label="ACP tool location") + if not os.path.isabs(path): + raise AcpProjectionError("ACP tool location path must be absolute") + line = normalized.get("line") + if line is not None and ( + isinstance(line, bool) + or not isinstance(line, int) + or not 0 <= line <= 2**32 - 1 + ): + normalized.pop("line", None) + return normalized + + +def _normalized_plan_entry(value: Any) -> dict[str, Any]: + if not isinstance(value, Mapping): + raise AcpProjectionError("ACP plan entry must be an object") + content = value.get("content") + priority = value.get("priority") + status = value.get("status") + if not isinstance(content, str): + raise AcpProjectionError("ACP plan entry is missing string content") + if priority not in _PLAN_PRIORITIES: + raise AcpProjectionError("ACP plan entry has invalid priority") + if status not in _PLAN_STATUSES: + raise AcpProjectionError("ACP plan entry has invalid status") + normalized = {"content": content, "priority": priority, "status": status} + meta = _extension_metadata(value) + if meta: + normalized["_meta"] = meta + return normalized + + +def _validate_usage(update: Mapping[str, Any]) -> None: + for key in ("used", "size"): + value = update.get(key) + if ( + isinstance(value, bool) + or not isinstance(value, int) + or not 0 <= value <= 2**64 - 1 + ): + raise AcpProjectionError(f"ACP usage {key} must be a u64") + + +def _validate_cost(cost: Any) -> None: + if not isinstance(cost, Mapping): + raise AcpProjectionError("ACP usage cost must be an object or null") + amount = cost.get("amount") + if not _is_finite_number(amount): + raise AcpProjectionError("ACP usage cost amount must be a finite number") + if not isinstance(cost.get("currency"), str): + raise AcpProjectionError("ACP usage cost currency must be text") + _extension_metadata(cost) + + +def _is_finite_number(value: Any) -> bool: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return False + try: + return math.isfinite(float(value)) + except (ValueError, OverflowError): + return False + + +def _required_text(value: Mapping[str, Any], key: str, *, label: str) -> str: + item = value.get(key) + if not isinstance(item, str): + raise AcpProjectionError(f"{label} is missing string {key}") + return item + + +def _bounded_json( + value: Any, *, label: str, max_bytes: int, max_depth: int +) -> bytes: + _validate_json_depth(value, label=label, max_depth=max_depth) + try: + encoded = json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + ).encode("utf-8") + except (TypeError, ValueError, RecursionError) as exc: + raise AcpProjectionError(f"{label} must be bounded JSON data") from exc + if len(encoded) > max_bytes: + raise AcpProjectionError(f"{label} exceeds the size limit") + return encoded + + +def _validate_json_depth(value: Any, *, label: str, max_depth: int) -> None: + stack: list[tuple[Any, int]] = [(value, 0)] + visited: set[int] = set() + while stack: + item, depth = stack.pop() + if depth > max_depth: + raise AcpProjectionError(f"{label} exceeds the nesting limit") + if not isinstance(item, (Mapping, list, tuple)): + continue + identity = id(item) + if identity in visited: + continue + visited.add(identity) + children = item.values() if isinstance(item, Mapping) else item + stack.extend((child, depth + 1) for child in children) + + +def _safe_deepcopy(value: Any, *, label: str) -> Any: + try: + return deepcopy(value) + except (TypeError, ValueError, RecursionError, OverflowError) as exc: + raise AcpProjectionError(f"{label} could not be copied safely") from exc + + +def _json_size(value: Any) -> int: + return len( + json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + ).encode("utf-8") + ) + + +def _messages_state_bytes( + messages: Mapping[str, list[_MessageAssembly]], +) -> int: + return sum( + len(message.message_id.encode("utf-8")) + len(message.text.encode("utf-8")) + for assemblies in messages.values() + for message in assemblies + ) + + +def _event_digest(kind: str, value: Mapping[str, Any]) -> str: + encoded = json.dumps( + {"kind": kind, "value": value}, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def _canonical_event( + *, + session_id: str, + sequence: int, + kind: str, + payload: Mapping[str, Any], + source_event_id: str | None, + replay: bool, + original_update: Mapping[str, Any], +) -> dict[str, Any]: + if kind not in SUPPORTED_EVENT_KINDS: + raise AcpProjectionError(f"unsupported canonical event kind: {kind}") + dedupe_material = { + "session_id": session_id, + "kind": kind, + "source_event_id": source_event_id, + "update": original_update, + } + encoded = json.dumps( + dedupe_material, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + ).encode("utf-8") + privacy = "session" + private_fields: list[str] = [] + if kind == "thought": + privacy = "private" + private_fields = ["payload"] + elif kind == "extension": + privacy = "private" + private_fields = ["payload"] + elif kind in {"tool_call", "tool_call_update"}: + privacy = "mixed" + for field_name in _RAW_TOOL_FIELDS: + snake_name = "raw_input" if field_name == "rawInput" else "raw_output" + private_fields.extend( + [ + f"payload.snapshot.{field_name}", + f"payload.snapshot.{snake_name}", + f"payload.changes.{field_name}", + f"payload.changes.{snake_name}", + ] + ) + if "permission" in payload: + private_fields.extend( + ["payload.permission", "payload.snapshot.permission"] + ) + event_id = ( + f"acp:{session_id}:{source_event_id}" + if source_event_id is not None + else f"acp:{session_id}:local:{sequence}" + ) + return { + "schema_version": 1, + "event_id": event_id, + "source": "acp", + "session_id": session_id, + "sequence": sequence, + "kind": kind, + "payload": deepcopy(dict(payload)), + "privacy": privacy, + "private_fields": private_fields, + "source_event_id": source_event_id, + "replay": bool(replay), + "dedupe_hint": hashlib.sha256(encoded).hexdigest(), + "dedupe_safe": source_event_id is not None, + } + + +def _joined_messages(messages: list[_MessageAssembly]) -> str: + return "\n\n".join(message.text for message in messages if message.text) + + +__all__ = [ + "AcpEventProjector", + "AcpProjectionCheckpoint", + "AcpProjectionError", + "SUPPORTED_EVENT_KINDS", +] diff --git a/src/tendwire/backends/acp_protocol.py b/src/tendwire/backends/acp_protocol.py new file mode 100644 index 0000000..4e69440 --- /dev/null +++ b/src/tendwire/backends/acp_protocol.py @@ -0,0 +1,613 @@ +"""Strict, stdlib-only primitives for the ACP v1 stdio protocol. + +ACP uses JSON-RPC 2.0 messages delimited by newlines. This module deliberately +does not know about processes or threads; :mod:`acp_client` owns that transport +lifecycle and uses the validated envelopes defined here. +""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from dataclasses import dataclass +from enum import Enum +from types import MappingProxyType +from typing import Any, TypeAlias + +from acp.schema import ( + RequestPermissionRequest as UpstreamRequestPermissionRequest, + SessionNotification as UpstreamSessionNotification, +) +from pydantic import ValidationError + +JSONRPC_VERSION = "2.0" +ACP_PROTOCOL_VERSION = 1 +DEFAULT_MAX_FRAME_BYTES = 8 * 1024 * 1024 + +RequestId: TypeAlias = str | int | None + +_MIN_REQUEST_NUMBER = -(2**63) +_MAX_REQUEST_NUMBER = 2**63 - 1 + + +class AcpProtocolError(Exception): + """Base class for ACP framing and envelope errors.""" + + +class AcpFramingError(AcpProtocolError, ValueError): + """A stdio frame is incomplete, oversized, or not valid JSON.""" + + +class AcpEnvelopeError(AcpProtocolError, ValueError): + """A decoded value is not a valid JSON-RPC 2.0 envelope.""" + + +class AcpRemoteError(AcpProtocolError): + """A correlated JSON-RPC error returned by the ACP agent.""" + + def __init__( + self, + code: int, + message: str, + *, + request_id: RequestId | None, + data: Any = None, + ) -> None: + self.code = code + self.message = message + self.request_id = request_id + self.data = data + super().__init__(f"ACP error {code}: {message}") + + +class MessageKind(str, Enum): + REQUEST = "request" + RESPONSE = "response" + NOTIFICATION = "notification" + + +class SessionUpdateKind(str, Enum): + USER_MESSAGE_CHUNK = "user_message_chunk" + AGENT_MESSAGE_CHUNK = "agent_message_chunk" + AGENT_THOUGHT_CHUNK = "agent_thought_chunk" + TOOL_CALL = "tool_call" + TOOL_CALL_UPDATE = "tool_call_update" + PLAN = "plan" + AVAILABLE_COMMANDS_UPDATE = "available_commands_update" + CURRENT_MODE_UPDATE = "current_mode_update" + CONFIG_OPTION_UPDATE = "config_option_update" + SESSION_INFO_UPDATE = "session_info_update" + USAGE_UPDATE = "usage_update" + + +class StopReason(str, Enum): + END_TURN = "end_turn" + MAX_TOKENS = "max_tokens" + MAX_TURN_REQUESTS = "max_turn_requests" + REFUSAL = "refusal" + CANCELLED = "cancelled" + + +class SteeringOutcome(str, Enum): + """Outcome returned by the capability-gated Codex ACP steering extension.""" + + INJECTED = "injected" + STARTED_NEW_TURN = "startedNewTurn" + FAILED = "failed" + + +class PermissionOptionKind(str, Enum): + ALLOW_ONCE = "allow_once" + ALLOW_ALWAYS = "allow_always" + REJECT_ONCE = "reject_once" + REJECT_ALWAYS = "reject_always" + + +@dataclass(frozen=True, slots=True) +class JsonRpcRequest: + request_id: RequestId + method: str + params: Mapping[str, Any] + + @property + def kind(self) -> MessageKind: + return MessageKind.REQUEST + + +@dataclass(frozen=True, slots=True) +class JsonRpcNotification: + method: str + params: Mapping[str, Any] + + @property + def kind(self) -> MessageKind: + return MessageKind.NOTIFICATION + + +@dataclass(frozen=True, slots=True) +class JsonRpcResponse: + request_id: RequestId | None + result: Any = None + error: Mapping[str, Any] | None = None + + @property + def kind(self) -> MessageKind: + return MessageKind.RESPONSE + + def result_or_raise(self) -> Any: + if self.error is None: + return self.result + raise AcpRemoteError( + self.error["code"], + self.error["message"], + request_id=self.request_id, + data=self.error.get("data"), + ) + + +JsonRpcMessage: TypeAlias = JsonRpcRequest | JsonRpcNotification | JsonRpcResponse + + +@dataclass(frozen=True, slots=True) +class AgentCapabilities: + """Captured ACP capabilities with convenient stable-v1 feature checks.""" + + raw: Mapping[str, Any] + + @classmethod + def from_mapping(cls, value: Mapping[str, Any] | None) -> "AgentCapabilities": + return cls(_freeze_mapping(value or {})) + + @property + def load_session(self) -> bool: + return self.raw.get("loadSession") is True + + @property + def session_list(self) -> bool: + return _is_capability_object(self._session_capabilities().get("list")) + + @property + def session_resume(self) -> bool: + return _is_capability_object(self._session_capabilities().get("resume")) + + @property + def session_close(self) -> bool: + return _is_capability_object(self._session_capabilities().get("close")) + + @property + def session_delete(self) -> bool: + return _is_capability_object(self._session_capabilities().get("delete")) + + @property + def additional_directories(self) -> bool: + return _is_capability_object( + self._session_capabilities().get("additionalDirectories") + ) + + @property + def prompt_image(self) -> bool: + return self._prompt_capabilities().get("image") is True + + @property + def prompt_audio(self) -> bool: + return self._prompt_capabilities().get("audio") is True + + @property + def prompt_embedded_context(self) -> bool: + return self._prompt_capabilities().get("embeddedContext") is True + + @property + def mcp_http(self) -> bool: + return self._mcp_capabilities().get("http") is True + + @property + def mcp_sse(self) -> bool: + return self._mcp_capabilities().get("sse") is True + + @property + def auth_logout(self) -> bool: + auth = self.raw.get("auth") + return isinstance(auth, Mapping) and _is_capability_object(auth.get("logout")) + + def _session_capabilities(self) -> Mapping[str, Any]: + value = self.raw.get("sessionCapabilities") + return value if isinstance(value, Mapping) else {} + + def _prompt_capabilities(self) -> Mapping[str, Any]: + value = self.raw.get("promptCapabilities") + return value if isinstance(value, Mapping) else {} + + def _mcp_capabilities(self) -> Mapping[str, Any]: + value = self.raw.get("mcpCapabilities") + return value if isinstance(value, Mapping) else {} + + +@dataclass(frozen=True, slots=True) +class InitializeResult: + protocol_version: int + capabilities: AgentCapabilities + agent_info: Mapping[str, Any] | None + auth_methods: tuple[Mapping[str, Any], ...] + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class SessionResult: + session_id: str + modes: Mapping[str, Any] | None + config_options: tuple[Mapping[str, Any], ...] + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class SessionInfo: + session_id: str + cwd: str + additional_directories: tuple[str, ...] + title: str | None + updated_at: str | None + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class SessionPage: + sessions: tuple[SessionInfo, ...] + next_cursor: str | None + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class PromptResult: + stop_reason: StopReason + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class SteeringResult: + outcome: SteeringOutcome + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class SessionUpdate: + session_id: str + update_kind: SessionUpdateKind | str + update: Mapping[str, Any] + meta: Mapping[str, Any] | None + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class PermissionOption: + option_id: str + name: str + kind: PermissionOptionKind | str + raw: Mapping[str, Any] + + +@dataclass(frozen=True, slots=True) +class PermissionRequest: + request_id: RequestId + session_id: str + tool_call: Mapping[str, Any] + options: tuple[PermissionOption, ...] + meta: Mapping[str, Any] | None + raw: Mapping[str, Any] + + +def _freeze_mapping(value: Mapping[str, Any]) -> Mapping[str, Any]: + # A shallow immutable copy is intentional: arbitrary extension payloads are + # retained verbatim and callers should treat all raw values as read-only. + return MappingProxyType(dict(value)) + + +def _is_capability_object(value: Any) -> bool: + # ACP advertises optional capabilities with an object (often simply {}). + return isinstance(value, Mapping) + + +def _valid_request_id(value: Any) -> bool: + # ACP v1 inherits JSON-RPC's String, integral Number, or Null request IDs. + # The official schema represents Number as a signed 64-bit integer. + return ( + value is None + or isinstance(value, str) + or ( + isinstance(value, int) + and not isinstance(value, bool) + and _MIN_REQUEST_NUMBER <= value <= _MAX_REQUEST_NUMBER + ) + ) + + +def _reject_constant(value: str) -> None: + raise ValueError(f"non-finite JSON number {value}") + + +def _object_without_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate JSON object key {key!r}") + result[key] = value + return result + + +def decode_json_line( + line: bytes | bytearray | memoryview | str, + *, + max_frame_bytes: int = DEFAULT_MAX_FRAME_BYTES, + require_newline: bool = True, +) -> JsonRpcMessage: + """Decode and validate exactly one newline-delimited JSON-RPC message.""" + if max_frame_bytes <= 0: + raise ValueError("max_frame_bytes must be positive") + if isinstance(line, str): + try: + raw = line.encode("utf-8") + except UnicodeEncodeError as exc: + raise AcpFramingError("ACP frame is not valid UTF-8") from exc + else: + raw = bytes(line) + if not raw: + raise AcpFramingError("empty ACP frame") + if len(raw) > max_frame_bytes: + raise AcpFramingError("ACP frame exceeds configured size limit") + if require_newline and not raw.endswith(b"\n"): + raise AcpFramingError("ACP frame is not newline terminated") + payload = raw[:-1] if raw.endswith(b"\n") else raw + if payload.endswith(b"\r"): + payload = payload[:-1] + if b"\n" in payload or b"\r" in payload: + raise AcpFramingError("ACP frame contains an embedded line break") + if not payload: + raise AcpFramingError("empty ACP JSON payload") + try: + text = payload.decode("utf-8", errors="strict") + value = json.loads( + text, + parse_constant=_reject_constant, + object_pairs_hook=_object_without_duplicates, + ) + except (UnicodeDecodeError, json.JSONDecodeError, ValueError) as exc: + raise AcpFramingError("ACP frame is not strict UTF-8 JSON") from exc + return validate_envelope(value) + + +def validate_envelope(value: Any) -> JsonRpcMessage: + """Validate a decoded JSON-RPC 2.0 object and return a typed envelope.""" + if not isinstance(value, Mapping): + raise AcpEnvelopeError("ACP JSON-RPC envelope must be an object") + if value.get("jsonrpc") != JSONRPC_VERSION: + raise AcpEnvelopeError("ACP envelope must declare jsonrpc '2.0'") + + has_method = "method" in value + has_id = "id" in value + has_result = "result" in value + has_error = "error" in value + + if has_method: + if has_result or has_error: + raise AcpEnvelopeError("JSON-RPC call cannot contain result or error") + method = value["method"] + if not isinstance(method, str) or not method: + raise AcpEnvelopeError("JSON-RPC method must be a non-empty string") + params = value.get("params", {}) + if not isinstance(params, Mapping): + raise AcpEnvelopeError("ACP method params must be an object") + frozen_params = _freeze_mapping(params) + if not has_id: + return JsonRpcNotification(method=method, params=frozen_params) + request_id = value["id"] + if not _valid_request_id(request_id): + raise AcpEnvelopeError( + "JSON-RPC request id must be a string, signed 64-bit integer, or null" + ) + return JsonRpcRequest( + request_id=request_id, + method=method, + params=frozen_params, + ) + + if not has_id: + raise AcpEnvelopeError("JSON-RPC response must contain an id") + request_id = value["id"] + if request_id is not None and not _valid_request_id(request_id): + raise AcpEnvelopeError("JSON-RPC response id is invalid") + if has_result == has_error: + raise AcpEnvelopeError("JSON-RPC response must contain exactly one of result or error") + if has_error: + error = value["error"] + if not isinstance(error, Mapping): + raise AcpEnvelopeError("JSON-RPC error must be an object") + code = error.get("code") + message = error.get("message") + if not isinstance(code, int) or isinstance(code, bool): + raise AcpEnvelopeError("JSON-RPC error code must be an integer") + if not isinstance(message, str): + raise AcpEnvelopeError("JSON-RPC error message must be a string") + return JsonRpcResponse( + request_id=request_id, + error=_freeze_mapping(error), + ) + return JsonRpcResponse(request_id=request_id, result=value["result"]) + + +def encode_message( + value: Mapping[str, Any], + *, + max_frame_bytes: int = DEFAULT_MAX_FRAME_BYTES, +) -> bytes: + """Validate and encode a JSON-RPC object as one bounded UTF-8 line.""" + validate_envelope(value) + try: + payload = json.dumps( + value, + ensure_ascii=False, + allow_nan=False, + separators=(",", ":"), + ).encode("utf-8") + b"\n" + except (TypeError, ValueError, UnicodeEncodeError) as exc: + raise AcpEnvelopeError("ACP envelope is not strict JSON serializable") from exc + if len(payload) > max_frame_bytes: + raise AcpFramingError("ACP frame exceeds configured size limit") + return payload + + +def request_envelope( + request_id: RequestId, + method: str, + params: Mapping[str, Any] | None = None, +) -> dict[str, Any]: + value = { + "jsonrpc": JSONRPC_VERSION, + "id": request_id, + "method": method, + "params": dict(params or {}), + } + validate_envelope(value) + return value + + +def notification_envelope( + method: str, + params: Mapping[str, Any] | None = None, +) -> dict[str, Any]: + value = { + "jsonrpc": JSONRPC_VERSION, + "method": method, + "params": dict(params or {}), + } + validate_envelope(value) + return value + + +def result_envelope(request_id: RequestId, result: Any) -> dict[str, Any]: + value = {"jsonrpc": JSONRPC_VERSION, "id": request_id, "result": result} + validate_envelope(value) + return value + + +def error_envelope( + request_id: RequestId | None, + code: int, + message: str, + *, + data: Any = None, +) -> dict[str, Any]: + error: dict[str, Any] = {"code": code, "message": message} + if data is not None: + error["data"] = data + value = {"jsonrpc": JSONRPC_VERSION, "id": request_id, "error": error} + validate_envelope(value) + return value + + +def parse_session_update(params: Mapping[str, Any]) -> SessionUpdate: + session_id = _required_string(params, "sessionId") + update = params.get("update") + if not isinstance(update, Mapping): + raise AcpEnvelopeError("session/update params.update must be an object") + kind_value = _required_string(update, "sessionUpdate") + try: + kind: SessionUpdateKind | str = SessionUpdateKind(kind_value) + except ValueError: + # ACP extensions and future stable revisions remain observable. + kind = kind_value + else: + try: + UpstreamSessionNotification.model_validate(dict(params)) + except ValidationError as exc: + raise AcpEnvelopeError( + "session/update params do not match the upstream ACP schema" + ) from exc + meta = params.get("_meta") + if meta is not None and not isinstance(meta, Mapping): + meta = None + return SessionUpdate( + session_id=session_id, + update_kind=kind, + update=_freeze_mapping(update), + meta=_freeze_mapping(meta) if isinstance(meta, Mapping) else None, + raw=_freeze_mapping(params), + ) + + +def parse_permission_request(request: JsonRpcRequest) -> PermissionRequest: + if request.method != "session/request_permission": + raise AcpEnvelopeError("request is not session/request_permission") + params = request.params + try: + UpstreamRequestPermissionRequest.model_validate(dict(params)) + except ValidationError as exc: + errors = exc.errors(include_url=False, include_input=False) + if any( + tuple(error.get("loc", ()))[-1:] == ("kind",) + for error in errors + ): + raise AcpEnvelopeError( + "permission option kind is not valid ACP v1" + ) from exc + locations = ", ".join( + ".".join(_upstream_alias(part) for part in error.get("loc", ())) + for error in errors + ) + suffix = f" ({locations})" if locations else "" + raise AcpEnvelopeError( + f"permission request does not match the upstream ACP schema{suffix}" + ) from exc + session_id = _required_string(params, "sessionId") + tool_call = params.get("toolCall") + if not isinstance(tool_call, Mapping): + raise AcpEnvelopeError("permission request toolCall must be an object") + _required_string(tool_call, "toolCallId") + raw_options = params.get("options") + if not isinstance(raw_options, list): + raise AcpEnvelopeError("permission request options must be an array") + options: list[PermissionOption] = [] + seen_option_ids: set[str] = set() + for raw in raw_options: + if not isinstance(raw, Mapping): + raise AcpEnvelopeError("permission option must be an object") + option_id = _required_string(raw, "optionId") + if option_id in seen_option_ids: + raise AcpEnvelopeError("permission option IDs must be unique") + seen_option_ids.add(option_id) + name = _required_string(raw, "name") + kind_value = _required_string(raw, "kind") + try: + kind: PermissionOptionKind | str = PermissionOptionKind(kind_value) + except ValueError as exc: + raise AcpEnvelopeError("permission option kind is not valid ACP v1") from exc + options.append( + PermissionOption( + option_id=option_id, + name=name, + kind=kind, + raw=_freeze_mapping(raw), + ) + ) + meta = params.get("_meta") + return PermissionRequest( + request_id=request.request_id, + session_id=session_id, + tool_call=_freeze_mapping(tool_call), + options=tuple(options), + meta=_freeze_mapping(meta) if isinstance(meta, Mapping) else None, + raw=_freeze_mapping(params), + ) + + +def _upstream_alias(value: object) -> str: + """Render safe validation locations using ACP wire aliases.""" + + text = str(value) + head, *tail = text.split("_") + return head + "".join(part[:1].upper() + part[1:] for part in tail) + + +def _required_string(value: Mapping[str, Any], key: str) -> str: + result = value.get(key) + if not isinstance(result, str) or not result: + raise AcpEnvelopeError(f"{key} must be a non-empty string") + return result diff --git a/src/tendwire/backends/acp_runtime.py b/src/tendwire/backends/acp_runtime.py new file mode 100644 index 0000000..36dd196 --- /dev/null +++ b/src/tendwire/backends/acp_runtime.py @@ -0,0 +1,1270 @@ +"""Supervised ACP session runtime bound to one Tendwire worker. + +The lower-level :mod:`acp_client` owns subprocess framing and JSON-RPC request +correlation. This module connects that transport to durable ACP ingestion, +keeps both inbound event queues drained, and exposes a deliberately redacted +health surface suitable for operator APIs. +""" + +from __future__ import annotations + +import threading +import time +import uuid +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from enum import Enum +from pathlib import Path +from typing import Any, Protocol + +from ..config import Config +from ..core.models import WorkerBinding, stable_fingerprint +from ..store.sqlite import expire_worker_bindings, list_worker_bindings +from .acp_ingestion import AcpSessionIngestor +from .acp_protocol import ( + PermissionRequest, + PromptResult, + RequestId, + SessionResult, + SessionUpdate, + StopReason, + SteeringOutcome, + SteeringResult, +) + + +class AcpRuntimeError(RuntimeError): + """Base error raised by the supervised ACP runtime.""" + + +class AcpRuntimeStateError(AcpRuntimeError): + """An operation is invalid for the runtime's current lifecycle state.""" + + +class AcpRuntimeProtocolError(AcpRuntimeError): + """The ACP client returned data that cannot be safely bound or finalized.""" + + +class AcpRuntimeBindingError(AcpRuntimeProtocolError): + """The runtime's authenticated worker binding is no longer current.""" + + +class AcpRuntimeStopTimeout(AcpRuntimeError, TimeoutError): + """The runtime could not stop all supervised work within its deadline.""" + + +class SessionOpenMode(str, Enum): + NEW = "new" + LOAD = "load" + RESUME = "resume" + + +class RuntimeState(str, Enum): + NEW = "new" + STARTING = "starting" + RUNNING = "running" + STOPPING = "stopping" + STOPPED = "stopped" + FAILED = "failed" + + +@dataclass(frozen=True, slots=True) +class AcpWorkerSessionStatus: + """Public-safe runtime health and counters. + + This type intentionally has no command, process, session, worker, target, + or exception-message fields. Those values are private routing material. + """ + + state: RuntimeState + healthy: bool + updates_ingested: int + permissions_ingested: int + permissions_selected: int + permissions_cancelled: int + invalid_permission_selections: int + prompts_started: int + prompts_completed: int + prompts_failed: int + cancellation_requests: int + failure_type: str | None + + +@dataclass(frozen=True, slots=True) +class PermissionSelection: + """One selected option plus transport acknowledgement hooks. + + The callback that selected an option must not report its durable command as + accepted until the JSON-RPC response frame has actually been written. + """ + + option_id: str + response_written: Callable[[], None] + response_failed: Callable[[BaseException], None] + + +PermissionCallback = Callable[[PermissionRequest], str | PermissionSelection | None] +IngestorFactory = Callable[..., AcpSessionIngestor] + + +class SessionBindingCallback(Protocol): + """Atomically persist an ACP binding derived from one continuity anchor.""" + + def __call__( + self, + session_id: str, + continuity_binding: WorkerBinding, + ) -> WorkerBinding: ... + + +class AcpSessionConnection(Protocol): + """Bounded ACP connection required by :class:`AcpWorkerSession`.""" + + def initialize( + self, + *, + client_capabilities: Mapping[str, Any] | None = None, + ) -> object: ... + + def new_session( + self, + cwd: Path, + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[Path] = (), + ) -> SessionResult: ... + + def load_session( + self, + session_id: str, + cwd: Path, + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[Path] = (), + ) -> SessionResult: ... + + def resume_session( + self, + session_id: str, + cwd: Path, + *, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[Path] = (), + ) -> SessionResult: ... + + def prompt( + self, + session_id: str, + prompt: str | Sequence[Mapping[str, Any]], + *, + timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, + on_submitted: Callable[[], None] | None = None, + ) -> PromptResult: ... + + def prepare_prompt( + self, + prompt: str | Sequence[Mapping[str, Any]], + ) -> tuple[Mapping[str, Any], ...]: ... + + @property + def steering_supported(self) -> bool: ... + + def steer_session( + self, + session_id: str, + prompt: str | Sequence[Mapping[str, Any]], + *, + timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, + on_submitted: Callable[[], None] | None = None, + ) -> SteeringResult: ... + + def cancel(self, session_id: str) -> None: ... + + def next_session_event( + self, + *, + timeout: float, + ) -> SessionUpdate | PermissionRequest: ... + + def respond_permission( + self, + request_id: RequestId, + *, + option_id: str | None = None, + cancelled: bool = False, + ) -> None: ... + + def close(self) -> None: ... + + +class AcpWorkerSession: + """Run and durably ingest exactly one ACP session for one worker binding. + + Permission requests fail closed: the default response is ``cancelled``. + A callback can authorize an action only by returning the ID of an option + present in that exact request. + """ + + def __init__( + self, + client: AcpSessionConnection, + *, + config: Config, + binding: WorkerBinding, + cwd: str | Path, + session_mode: SessionOpenMode | str = SessionOpenMode.NEW, + session_id: str | None = None, + stream_generation: str | None = None, + client_capabilities: Mapping[str, Any] | None = None, + mcp_servers: Sequence[Mapping[str, Any]] = (), + additional_directories: Sequence[str | Path] = (), + permission_callback: PermissionCallback | None = None, + session_binding_callback: SessionBindingCallback | None = None, + ingestor: AcpSessionIngestor | None = None, + ingestor_factory: IngestorFactory = AcpSessionIngestor, + poll_timeout: float = 0.05, + stop_timeout: float = 3.0, + ) -> None: + try: + mode = SessionOpenMode(session_mode) + except ValueError as exc: + raise ValueError(f"unsupported ACP session mode {session_mode!r}") from exc + if mode is not SessionOpenMode.NEW and not session_id: + raise ValueError(f"session_id is required for ACP {mode.value}") + if mode is SessionOpenMode.NEW and session_id is not None: + raise ValueError("session_id must be omitted when creating an ACP session") + if mode is SessionOpenMode.NEW and session_binding_callback is None: + raise ValueError("session_binding_callback is required for ACP new") + if mode is not SessionOpenMode.NEW and session_binding_callback is not None: + raise ValueError( + "session_binding_callback is only valid when creating an ACP session" + ) + if binding.host_id != config.host_id: + raise ValueError("ACP runtime binding host does not match configuration") + if not binding.private_fingerprint: + raise ValueError("ACP runtime requires an authenticated private binding") + if ( + mode is SessionOpenMode.NEW + and binding.turn_target_kind == "acp_session_id" + ): + raise ValueError( + "ACP new requires a non-ACP worker continuity binding" + ) + if mode is not SessionOpenMode.NEW: + if binding.backend != "acp": + raise ValueError("ACP load/resume requires an ACP backend binding") + if binding.turn_target_kind != "acp_session_id": + raise ValueError("ACP runtime requires an ACP session worker binding") + if binding.turn_target_value != session_id: + raise ValueError("ACP runtime session does not match the worker binding") + if config.db_path is None: + raise ValueError("ACP runtime requires a sqlite db path") + resolved_cwd = Path(cwd) + if not resolved_cwd.is_absolute(): + raise ValueError("ACP runtime cwd must be absolute") + if poll_timeout <= 0 or stop_timeout <= 0: + raise ValueError("ACP runtime timeouts must be positive") + + self._client = client + self._config = config + self._binding = binding + self._cwd = resolved_cwd + self._session_mode = mode + self._requested_session_id = session_id + self._stream_generation = stream_generation or uuid.uuid4().hex + self._client_capabilities = _runtime_client_capabilities( + client_capabilities + ) + self._mcp_servers = tuple(dict(server) for server in mcp_servers) + self._additional_directories = tuple(Path(path) for path in additional_directories) + self._permission_callback = permission_callback + self._session_binding_callback = session_binding_callback + self._provided_ingestor = ingestor + self._ingestor_factory = ingestor_factory + self._poll_timeout = float(poll_timeout) + self._stop_timeout = float(stop_timeout) + + self._state = RuntimeState.NEW + self._session_id: str | None = None + self._ingestor: AcpSessionIngestor | None = None + self._failure: BaseException | None = None + self._state_lock = threading.RLock() + # Session binders are embedding callbacks and may synchronously call + # stop(). Reentrancy must terminate startup, not deadlock on our own + # lifecycle lock. + self._lifecycle_lock = threading.RLock() + self._ingest_lock = threading.Lock() + self._prompt_lock = threading.Lock() + self._steering_lock = threading.Lock() + self._idle_condition = threading.Condition(self._state_lock) + self._stop_event = threading.Event() + self._threads: tuple[threading.Thread, ...] = () + self._event_idle_epoch = 0 + self._setup_replay = False + # NEW acquires authority through its binder during start. LOAD/RESUME + # are handed an existing live ACP lease and own releasing it once the + # runtime stops or fails. + self._provisional_binding: WorkerBinding | None = ( + binding if mode is not SessionOpenMode.NEW else None + ) + self._close_thread: threading.Thread | None = None + self._close_failures: list[BaseException] = [] + self._prompt_threads: set[threading.Thread] = set() + + self._updates_ingested = 0 + self._permissions_ingested = 0 + self._permissions_selected = 0 + self._permissions_cancelled = 0 + self._invalid_permission_selections = 0 + self._prompts_started = 0 + self._prompts_completed = 0 + self._prompts_failed = 0 + self._cancellation_requests = 0 + + def __enter__(self) -> "AcpWorkerSession": + return self.start() + + def __exit__(self, exc_type: object, exc: object, tb: object) -> None: + try: + self.stop() + except BaseException: + if exc is None: + raise + + def start(self) -> "AcpWorkerSession": + """Initialize capabilities, open one session, and start consumers.""" + + with self._lifecycle_lock: + with self._state_lock: + if self._state is RuntimeState.RUNNING: + return self + if self._state is not RuntimeState.NEW: + raise AcpRuntimeStateError( + f"cannot start ACP runtime in state {self._state.value}" + ) + self._state = RuntimeState.STARTING + try: + self._require_current_binding(self._binding) + self._client.initialize(client_capabilities=self._client_capabilities) + if self._session_mode is SessionOpenMode.LOAD: + assert self._requested_session_id is not None + # ACP load may synchronously replay more updates than the + # bounded client queue can hold before returning. Bind and + # drain the ordered stream before issuing the request. + self._session_id = self._requested_session_id + self._ingestor = self._make_ingestor(self._requested_session_id) + self._setup_replay = True + self._start_consumer() + session = self._open_session() + if not isinstance(session, SessionResult) or not session.session_id: + raise AcpRuntimeProtocolError( + "ACP session setup returned an invalid response" + ) + if self._session_mode is SessionOpenMode.NEW: + self._binding = self._bind_new_session(session.session_id) + else: + if session.session_id != self._binding.turn_target_value: + raise AcpRuntimeProtocolError( + "ACP session setup did not return the bound session" + ) + if session.session_id != self._requested_session_id: + raise AcpRuntimeProtocolError( + "ACP session setup returned an unexpected session" + ) + self._require_current_binding(self._binding) + if self._session_mode is SessionOpenMode.LOAD: + self._wait_for_event_idle( + self._stop_timeout, + allowed_state=RuntimeState.STARTING, + ) + with self._ingest_lock: + self._require_ingestor().reset_after_load() + self._setup_replay = False + else: + self._session_id = session.session_id + self._ingestor = self._make_ingestor(session.session_id) + self._start_consumer() + with self._state_lock: + if self._failure is not None: + raise self._failure + self._state = RuntimeState.RUNNING + except BaseException as exc: + self._release_derived_binding(reason="acp_startup_rollback") + self._record_failure(exc) + # ``__enter__`` is never completed when start fails, so no + # caller cleanup can be assumed. Bound shutdown prevents an + # initialized adapter or a partially started consumer leaking. + self._shutdown_transport(self._stop_timeout) + raise + return self + + def prompt( + self, + prompt: str | Sequence[Mapping[str, Any]], + *, + producer_turn_id: str | None = None, + timeout: float | None = None, + drain_timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, + on_submitted: Callable[[], None] | None = None, + ) -> PromptResult: + """Submit one prompt and finalize only after its prior updates drain.""" + + wait_limit = self._stop_timeout if drain_timeout is None else float(drain_timeout) + if wait_limit <= 0: + raise ValueError("drain_timeout must be positive") + with self._prompt_lock: + self.raise_if_failed() + session_id, ingestor = self._running_components() + if not isinstance(producer_turn_id, str) or not producer_turn_id.strip(): + raise ValueError("producer_turn_id must be non-empty text") + stable_producer_turn_id = producer_turn_id.strip() + with self._state_lock: + self._prompts_started += 1 + try: + self._require_current_binding(self._binding) + prepared_prompt = _prepare_prompt_content(self._client, prompt) + prompt_event = ingestor.begin_prompt( + prepared_prompt, + producer_turn_id=stable_producer_turn_id, + ) + _raise_for_binding_rejection(prompt_event) + except BaseException as exc: + with self._state_lock: + self._prompts_failed += 1 + self._record_failure(exc) + raise + try: + prompt_kwargs: dict[str, Any] = {"timeout": timeout} + if on_send_start is not None: + prompt_kwargs["on_send_start"] = on_send_start + if on_submitted is not None: + prompt_kwargs["on_submitted"] = on_submitted + result = self._client.prompt( + session_id, + prepared_prompt, + **prompt_kwargs, + ) + except BaseException as exc: + with self._state_lock: + self._prompts_failed += 1 + # Once a turn has been opened locally, a timed-out or failed + # prompt cannot be retried safely: late updates would otherwise + # be attributed to the next turn. Best-effort cancellation + # contains the remote work and the runtime becomes terminal. + self._close_failed_prompt(session_id, ingestor) + self._record_failure(exc) + raise + if not isinstance(result, PromptResult): + error = AcpRuntimeProtocolError( + "ACP prompt returned an invalid response" + ) + with self._state_lock: + self._prompts_failed += 1 + self._close_failed_prompt(session_id, ingestor) + self._record_failure(error) + raise error + + # The transport dispatches updates before the prompt response, but + # the consumer runs on another thread. Requiring a queue timeout + # after the response is a barrier: every earlier queued update has + # been durably ingested before the turn is marked complete. + try: + self._wait_for_event_idle(wait_limit) + with self._ingest_lock: + completion = ingestor.mark_prompt_complete(result.stop_reason) + _raise_for_binding_rejection(completion) + except BaseException as exc: + with self._state_lock: + self._prompts_failed += 1 + self._record_failure(exc) + raise + with self._state_lock: + self._prompts_completed += 1 + return result + + def submit_prompt( + self, + prompt: str | Sequence[Mapping[str, Any]], + *, + producer_turn_id: str, + acknowledgement_timeout: float, + completion_timeout: float | None = None, + on_send_start: Callable[[], None] | None = None, + ) -> None: + """Start a prompt and return after its complete frame is written. + + End-of-turn completion continues under runtime supervision. A caller + can therefore durably acknowledge delivery without blocking for the + agent's entire turn. If acknowledgement is not observed, delivery is + uncertain and the command layer must never retry it automatically. + """ + + if acknowledgement_timeout <= 0: + raise ValueError("acknowledgement_timeout must be positive") + acknowledged = threading.Event() + finished = threading.Event() + failures: list[BaseException] = [] + + def run_prompt() -> None: + try: + self.prompt( + prompt, + producer_turn_id=producer_turn_id, + timeout=completion_timeout, + on_send_start=on_send_start, + on_submitted=acknowledged.set, + ) + except BaseException as exc: + failures.append(exc) + finally: + finished.set() + with self._state_lock: + self._prompt_threads.discard(threading.current_thread()) + + thread = threading.Thread( + target=run_prompt, + name="tendwire-acp-prompt", + daemon=True, + ) + with self._state_lock: + self._prompt_threads.add(thread) + thread.start() + deadline = time.monotonic() + acknowledgement_timeout + while True: + if acknowledged.is_set(): + return + if finished.is_set(): + if failures: + raise failures[0] + raise AcpRuntimeProtocolError( + "ACP prompt completed without a submission acknowledgement" + ) + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AcpRuntimeStopTimeout( + "ACP prompt submission acknowledgement timed out" + ) + acknowledged.wait(min(remaining, 0.01)) + + def can_steer(self) -> bool: + """Return whether this runtime can append to a live ACP turn.""" + + try: + supported = getattr(self._client, "steering_supported", False) is True + except Exception: + supported = False + if not supported: + return False + with self._state_lock: + if self._state is not RuntimeState.RUNNING or self._failure is not None: + return False + with self._ingest_lock: + ingestor = self._ingestor + can_append = getattr(ingestor, "can_append_prompt", None) + return bool(callable(can_append) and can_append()) + + def submit_steering( + self, + prompt: str | Sequence[Mapping[str, Any]], + *, + producer_turn_id: str, + acknowledgement_timeout: float, + on_send_start: Callable[[], None] | None = None, + ) -> SteeringResult: + """Inject one input into the current turn through ACP steering.""" + + if acknowledgement_timeout <= 0: + raise ValueError("acknowledgement_timeout must be positive") + if not isinstance(producer_turn_id, str) or not producer_turn_id.strip(): + raise ValueError("producer_turn_id must be non-empty text") + with self._steering_lock: + self.raise_if_failed() + session_id, ingestor = self._running_components() + prepared = _prepare_prompt_content(self._client, prompt) + if not self.can_steer(): + raise AcpRuntimeStateError("ACP steering is unavailable") + + def start_and_record() -> None: + if on_send_start is not None: + on_send_start() + with self._ingest_lock: + result = ingestor.append_prompt( + prepared, + producer_turn_id=producer_turn_id.strip(), + ) + _raise_for_binding_rejection(result) + + deadline = time.monotonic() + acknowledgement_timeout + result = self._client.steer_session( + session_id, + prepared, + timeout=acknowledgement_timeout, + on_send_start=start_and_record, + ) + if result.outcome is not SteeringOutcome.FAILED: + return result + + # The steering extension defines ``failed`` as a definite + # non-application outcome. Retrying once is therefore safe and + # prevents a transient adapter/app-server race from turning a + # live Telegram message into a terminal drop. The durable input + # and transport-boundary callback were already recorded by the + # first attempt, so the retry deliberately omits the callback and + # stays inside the caller's original acknowledgement budget. + remaining = deadline - time.monotonic() + if remaining <= 0: + return result + return self._client.steer_session( + session_id, + prepared, + timeout=remaining, + ) + + def cancel(self) -> None: + """Cancel the active session and any permission requests pending in it.""" + + self.raise_if_failed() + session_id, _ = self._running_components() + try: + self._cancel_session(session_id) + except BaseException as exc: + self._record_failure(exc) + raise + + def status(self) -> AcpWorkerSessionStatus: + """Return redacted health and counters safe for a public status API.""" + + with self._state_lock: + return AcpWorkerSessionStatus( + state=self._state, + healthy=self._state is RuntimeState.RUNNING and self._failure is None, + updates_ingested=self._updates_ingested, + permissions_ingested=self._permissions_ingested, + permissions_selected=self._permissions_selected, + permissions_cancelled=self._permissions_cancelled, + invalid_permission_selections=self._invalid_permission_selections, + prompts_started=self._prompts_started, + prompts_completed=self._prompts_completed, + prompts_failed=self._prompts_failed, + cancellation_requests=self._cancellation_requests, + failure_type=( + type(self._failure).__name__ if self._failure is not None else None + ), + ) + + def raise_if_failed(self) -> None: + """Raise the original background/runtime failure without redaction.""" + + with self._state_lock: + failure = self._failure + if failure is not None: + raise failure + + def join(self, timeout: float | None = None) -> bool: + """Wait a bounded interval for consumer threads; return whether all exited.""" + + wait_limit = self._stop_timeout if timeout is None else float(timeout) + if wait_limit <= 0: + raise ValueError("join timeout must be positive") + deadline = time.monotonic() + wait_limit + with self._state_lock: + threads = (*self._threads, *self._prompt_threads) + for thread in threads: + if thread is threading.current_thread() or thread.ident is None: + continue + thread.join(timeout=max(0.0, deadline - time.monotonic())) + return all( + thread is threading.current_thread() + or thread.ident is None + or not thread.is_alive() + for thread in threads + ) + + def stop(self, *, timeout: float | None = None) -> None: + """Close transport and consumers without waiting beyond one deadline.""" + + wait_limit = self._stop_timeout if timeout is None else float(timeout) + if wait_limit <= 0: + raise ValueError("stop timeout must be positive") + deadline = time.monotonic() + wait_limit + if not self._lifecycle_lock.acquire(timeout=wait_limit): + error = AcpRuntimeStopTimeout( + "ACP runtime lifecycle did not stop within the configured deadline" + ) + self._record_failure(error) + raise error + try: + with self._idle_condition: + if self._state is RuntimeState.STOPPED: + return + if self._state is RuntimeState.NEW: + self._state = RuntimeState.STOPPED + return + if self._state is not RuntimeState.FAILED: + self._state = RuntimeState.STOPPING + self._stop_event.set() + self._idle_condition.notify_all() + + remaining = max(0.0, deadline - time.monotonic()) + if remaining <= 0 or not self._shutdown_transport(remaining): + error = AcpRuntimeStopTimeout( + "ACP runtime did not stop within the configured deadline" + ) + self._record_failure(error) + raise error + + self._release_derived_binding(reason="acp_runtime_stopped") + + with self._state_lock: + failure = self._failure + if self._close_failures and failure is None: + failure = self._close_failures[0] + self._record_failure(failure) + with self._state_lock: + if failure is None: + self._state = RuntimeState.STOPPED + if failure is not None: + raise failure + finally: + self._lifecycle_lock.release() + + def _shutdown_transport(self, timeout: float) -> bool: + """Start adapter close at most once and join all supervised work.""" + + if self._close_thread is None: + + def close_client() -> None: + try: + self._client.close() + except BaseException as exc: + self._close_failures.append(exc) + + self._close_thread = threading.Thread( + target=close_client, + name="tendwire-acp-close", + daemon=True, + ) + self._close_thread.start() + + deadline = time.monotonic() + timeout + self._close_thread.join(timeout=max(0.0, deadline - time.monotonic())) + remaining = max(0.0, deadline - time.monotonic()) + consumers_joined = ( + self.join(timeout=remaining) + if remaining > 0 + else all( + thread.ident is None or not thread.is_alive() + for thread in self._threads + ) + ) + return not self._close_thread.is_alive() and consumers_joined + + def _open_session(self) -> SessionResult: + options = { + "mcp_servers": self._mcp_servers, + "additional_directories": self._additional_directories, + } + if self._session_mode is SessionOpenMode.NEW: + return self._client.new_session(self._cwd, **options) + assert self._requested_session_id is not None + if self._session_mode is SessionOpenMode.LOAD: + return self._client.load_session( + self._requested_session_id, self._cwd, **options + ) + return self._client.resume_session( + self._requested_session_id, self._cwd, **options + ) + + def _make_ingestor(self, session_id: str) -> AcpSessionIngestor: + if self._provided_ingestor is not None: + existing_session = getattr(self._provided_ingestor, "session_id", session_id) + if existing_session != session_id: + raise ValueError("provided ACP ingestor is bound to another session") + return self._provided_ingestor + return self._ingestor_factory( + self._config, + session_id=session_id, + stream_generation=self._stream_generation, + binding=self._binding, + ) + + def _bind_new_session(self, session_id: str) -> WorkerBinding: + callback = self._session_binding_callback + if callback is None: # pragma: no cover - constructor invariant + raise AcpRuntimeBindingError("ACP session binding is unavailable") + continuity = self._binding + self._require_current_binding(continuity) + existing_acp = self._binding_fingerprints(continuity.host_id, backend="acp") + try: + bound = callback(session_id, continuity) + with self._state_lock: + if self._state is not RuntimeState.STARTING: + raise AcpRuntimeStateError( + "ACP runtime stopped during session binding" + ) + if not isinstance(bound, WorkerBinding): + raise AcpRuntimeBindingError( + "ACP session binder returned an invalid binding" + ) + if ( + bound.host_id != continuity.host_id + or bound.worker_id != continuity.worker_id + or bound.worker_fingerprint != continuity.worker_fingerprint + or bound.backend != "acp" + or bound.target_kind != continuity.target_kind + or bound.target_value != continuity.target_value + ): + raise AcpRuntimeBindingError( + "ACP session binder changed worker continuity" + ) + if ( + bound.turn_target_kind != "acp_session_id" + or bound.turn_target_value != session_id + ): + raise AcpRuntimeBindingError( + "ACP session binder returned the wrong session" + ) + if ( + not bound.private_fingerprint + or bound.private_fingerprint == continuity.private_fingerprint + ): + raise AcpRuntimeBindingError( + "ACP session binder did not establish a distinct private binding" + ) + # The callback must add a distinct ACP binding. It must not + # repurpose or overwrite the Herdr continuity row it was given. + self._require_current_binding(continuity) + self._require_current_binding(bound) + except BaseException: + self._expire_new_acp_bindings(continuity.host_id, existing_acp) + raise + if bound.private_fingerprint not in existing_acp: + self._provisional_binding = bound + return bound + + def _binding_fingerprints(self, host_id: str, *, backend: str) -> set[str]: + db_path = self._config.db_path + if db_path is None: # pragma: no cover - constructor invariant + return set() + return { + item.private_fingerprint + for item in list_worker_bindings(Path(db_path), host_id, backend=backend) + } + + def _expire_new_acp_bindings( + self, + host_id: str, + existing_fingerprints: set[str], + ) -> None: + db_path = self._config.db_path + if db_path is None: # pragma: no cover - constructor invariant + return + current = list_worker_bindings(Path(db_path), host_id, backend="acp") + created = [ + item + for item in current + if item.private_fingerprint not in existing_fingerprints + ] + for item in created: + expire_worker_bindings( + Path(db_path), + host_id, + backend="acp", + private_fingerprints=[item.private_fingerprint], + # Callback clocks are untrusted. Using the row's own + # observation instant guarantees a future-dated provisional + # lease is still revocable. + now=item.observed_at, + reason="acp_startup_rollback", + ) + + def _release_derived_binding(self, *, reason: str) -> None: + bound = self._provisional_binding + self._provisional_binding = None + if bound is None or self._config.db_path is None: + return + expire_worker_bindings( + Path(self._config.db_path), + bound.host_id, + backend="acp", + private_fingerprints=[bound.private_fingerprint], + now=bound.observed_at, + reason=reason, + ) + + def _require_current_binding(self, expected: WorkerBinding) -> None: + db_path = self._config.db_path + if db_path is None: # pragma: no cover - constructor invariant + raise AcpRuntimeBindingError("ACP binding store is unavailable") + current = list_worker_bindings( + Path(db_path), + expected.host_id, + backend=expected.backend, + ) + # Herdr may refresh only the observation lease while an ACP endpoint + # is being initialized. That does not change routing authority and + # must not invalidate the in-flight generation transaction. Every + # identity and routing field remains exact; process-owned ACP rows are + # still checked byte-for-byte so their revocation cannot be masked. + if expected.backend == "herdr": + present = any( + _same_binding_authority(item, expected) + for item in current + ) + else: + present = expected in current + if not present: + raise AcpRuntimeBindingError("ACP worker binding is not current") + + def _start_consumer(self) -> None: + if self._threads: + return + thread = threading.Thread( + target=self._consume_session_events, + name="tendwire-acp-session-events", + daemon=True, + ) + self._threads = (thread,) + thread.start() + + def _consume_session_events(self) -> None: + try: + while True: + try: + event = self._client.next_session_event(timeout=self._poll_timeout) + except TimeoutError: + with self._idle_condition: + self._event_idle_epoch += 1 + self._idle_condition.notify_all() + if self._stop_event.is_set(): + return + continue + setup_replay = self._setup_replay + if isinstance(event, SessionUpdate): + if event.session_id != self._session_id: + raise AcpRuntimeProtocolError( + "ACP update belongs to a different session" + ) + ingestor = self._require_ingestor() + with self._ingest_lock: + outcome = ingestor.ingest_update( + event.raw, + replay=setup_replay, + setup_replay=setup_replay, + ) + _raise_for_binding_rejection(outcome) + if _outcome_has_persisted_event(outcome): + with self._state_lock: + self._updates_ingested += 1 + elif isinstance(event, PermissionRequest): + self._handle_permission( + event, + setup_replay=setup_replay, + ) + else: # pragma: no cover - typed protocol invariant + raise AcpRuntimeProtocolError("ACP session event type is invalid") + except BaseException as exc: + if not self._stop_event.is_set(): + self._record_failure(exc) + + def _handle_permission( + self, + request: PermissionRequest, + *, + setup_replay: bool = False, + ) -> None: + """Journal then resolve one permission, failing closed before response.""" + + response_attempted = False + try: + if request.session_id != self._session_id: + raise AcpRuntimeProtocolError( + "ACP permission belongs to a different session" + ) + ingestor = self._require_ingestor() + with self._ingest_lock: + outcome = ingestor.ingest_permission_request( + request.raw, + source_event_id=_permission_source_event_id(request.request_id), + replay=setup_replay, + setup_replay=setup_replay, + ) + _raise_for_binding_rejection(outcome) + if _outcome_has_persisted_event(outcome): + with self._state_lock: + self._permissions_ingested += 1 + + selected: str | None = None + selection: PermissionSelection | None = None + callback_failure: BaseException | None = None + if self._permission_callback is not None: + try: + candidate = self._permission_callback(request) + candidate_id = ( + candidate.option_id + if isinstance(candidate, PermissionSelection) + else candidate + ) + if candidate_id is not None and candidate_id in { + option.option_id for option in request.options + }: + selected = candidate_id + selection = ( + candidate + if isinstance(candidate, PermissionSelection) + else None + ) + elif candidate_id is not None: + with self._state_lock: + self._invalid_permission_selections += 1 + except BaseException as exc: + callback_failure = exc + response_attempted = True + if selected is None: + self._client.respond_permission( + request.request_id, + cancelled=True, + ) + with self._state_lock: + self._permissions_cancelled += 1 + else: + try: + self._client.respond_permission( + request.request_id, + option_id=selected, + ) + except BaseException as exc: + if selection is not None: + selection.response_failed(exc) + raise + if selection is not None: + selection.response_written() + with self._state_lock: + self._permissions_selected += 1 + if callback_failure is not None: + raise callback_failure + except BaseException: + if not response_attempted: + # No response bytes have been attempted yet, so cancellation + # is safe. Never retry after respond_permission itself fails: + # a partial JSON-RPC frame may already have reached the agent. + try: + self._client.respond_permission( + request.request_id, + cancelled=True, + ) + except BaseException: + pass + else: + with self._state_lock: + self._permissions_cancelled += 1 + raise + + def _wait_for_event_idle( + self, + timeout: float, + *, + allowed_state: RuntimeState = RuntimeState.RUNNING, + ) -> None: + deadline = time.monotonic() + timeout + with self._idle_condition: + event_epoch = self._event_idle_epoch + while self._event_idle_epoch <= event_epoch: + if self._failure is not None: + raise self._failure + if self._state is not allowed_state: + raise AcpRuntimeStateError( + "ACP runtime stopped before prompt updates drained" + ) + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AcpRuntimeStopTimeout( + "ACP prompt updates did not drain before the deadline" + ) + self._idle_condition.wait(timeout=remaining) + + def _running_components(self) -> tuple[str, AcpSessionIngestor]: + with self._state_lock: + if self._state is not RuntimeState.RUNNING: + raise AcpRuntimeStateError( + f"ACP runtime is not running ({self._state.value})" + ) + session_id = self._session_id + ingestor = self._ingestor + if session_id is None or ingestor is None: # pragma: no cover - invariant guard + raise AcpRuntimeStateError("ACP runtime has no bound session") + return session_id, ingestor + + def _require_ingestor(self) -> AcpSessionIngestor: + ingestor = self._ingestor + if ingestor is None: # pragma: no cover - consumers start after binding + raise AcpRuntimeStateError("ACP runtime has no ingestor") + return ingestor + + def _cancel_session(self, session_id: str) -> None: + self._client.cancel(session_id) + with self._state_lock: + self._cancellation_requests += 1 + + def _close_failed_prompt( + self, + session_id: str, + ingestor: AcpSessionIngestor, + ) -> None: + """Best-effort cancel and durable closure after ``begin_prompt``. + + The original prompt exception remains authoritative even if either + cleanup operation fails. ``mark_prompt_complete`` is idempotent, so + this method cannot create a second completion for an already closed + turn. + """ + + try: + self._cancel_session(session_id) + except BaseException: + pass + try: + # The prompt response (including an invalid one) can overtake the + # consumer thread after earlier session/update frames were queued. + # Preserve those updates in the failed turn before writing its + # terminal marker. A late post-cancel update remains harmless: + # the ingestor rejects turn-scoped updates after completion. + self._wait_for_event_idle(self._stop_timeout) + except BaseException: + pass + try: + with self._ingest_lock: + completion = ingestor.mark_prompt_complete( + StopReason.CANCELLED + ) + _raise_for_binding_rejection(completion) + except BaseException: + pass + + def _record_failure(self, failure: BaseException) -> None: + self._release_derived_binding(reason="acp_runtime_failed") + with self._idle_condition: + if self._failure is None: + self._failure = failure + self._state = RuntimeState.FAILED + self._stop_event.set() + self._idle_condition.notify_all() + + +def _permission_source_event_id(request_id: RequestId) -> str: + """Return a bounded opaque ID while preserving JSON-RPC ID types.""" + + return f"permission:{stable_fingerprint({'request_id': request_id})}" + + +def _prepare_prompt_content( + client: AcpSessionConnection, + prompt: str | Sequence[Mapping[str, Any]], +) -> tuple[Mapping[str, Any], ...]: + """Validate before persistence when the transport exposes its validator.""" + + prepare = getattr(client, "prepare_prompt", None) + if callable(prepare): + prepared = tuple(prepare(prompt)) + elif isinstance(prompt, str): + prepared = ({"type": "text", "text": prompt},) + else: + prepared = tuple(dict(block) for block in prompt) + if not prepared or any(not isinstance(block, Mapping) for block in prepared): + raise ValueError("prompt must contain at least one content block") + return tuple(dict(block) for block in prepared) + + +def _runtime_client_capabilities( + capabilities: Mapping[str, Any] | None, +) -> dict[str, Any]: + """Return only client capabilities this runtime can actually service. + + The runtime handles ACP session updates and permission requests, neither of + which is advertised through ``clientCapabilities``. It has no handlers for + filesystem, terminal, elicitation, or session-config requests. Known keys + are therefore stripped even when supplied by an embedding caller. Unknown + top-level keys could advertise extension methods and are rejected. + """ + if capabilities is None: + return {} + if not isinstance(capabilities, Mapping): + raise ValueError("client_capabilities must be a mapping or None") + known = {"fs", "terminal", "session", "elicitation", "_meta"} + if any(not isinstance(key, str) or key not in known for key in capabilities): + raise ValueError( + "ACP runtime cannot advertise unsupported client capabilities" + ) + return {} + + +def _same_binding_authority(left: WorkerBinding, right: WorkerBinding) -> bool: + """Compare durable route authority while ignoring observer lease refreshes.""" + + return ( + left.host_id, + left.worker_id, + left.worker_fingerprint, + left.backend, + left.target_kind, + left.target_value, + left.turn_target_kind, + left.turn_target_value, + left.sendable, + left.reason, + left.private_fingerprint, + ) == ( + right.host_id, + right.worker_id, + right.worker_fingerprint, + right.backend, + right.target_kind, + right.target_value, + right.turn_target_kind, + right.turn_target_value, + right.sendable, + right.reason, + right.private_fingerprint, + ) + + +def _raise_for_binding_rejection(outcome: object) -> None: + """Make every stale durable-binding outcome terminal and public-safe.""" + if outcome is None: + return + reason = getattr(outcome, "ignored_reason", None) + event = getattr(outcome, "event", None) + turn = getattr(outcome, "turn", None) + event_status = getattr(event, "status", None) + turn_stale = getattr(turn, "stale_binding", False) + if ( + reason in {"stale_binding", "binding_changed"} + or event_status == "binding_changed" + or turn_stale is True + ): + raise AcpRuntimeBindingError("ACP worker binding is no longer current") + + +def _outcome_has_persisted_event(outcome: object) -> bool: + """Count only updates that reached the durable event boundary.""" + + return getattr(outcome, "event", None) is not None + + +__all__ = [ + "AcpWorkerSession", + "AcpRuntimeBindingError", + "AcpSessionConnection", + "AcpRuntimeError", + "AcpRuntimeProtocolError", + "AcpRuntimeStateError", + "AcpWorkerSessionStatus", + "AcpRuntimeStopTimeout", + "PermissionCallback", + "RuntimeState", + "SessionBindingCallback", + "SessionOpenMode", +] diff --git a/src/tendwire/backends/herdr_cli.py b/src/tendwire/backends/herdr_cli.py deleted file mode 100644 index bf120eb..0000000 --- a/src/tendwire/backends/herdr_cli.py +++ /dev/null @@ -1,2737 +0,0 @@ -"""Thin adapter boundary around the Herdr CLI. - -This module shells out read-only to a `herdr` binary when available and parses -output on a best-effort basis. If the binary is missing or fails, it returns -empty neutral data rather than blocking the snapshot contract. - -This module must not import Herdres code or leak delivery/routing state into -core models. -""" - -from __future__ import annotations - -import hashlib -import json -import shutil -import subprocess -import time -from collections import Counter -from collections.abc import Iterable, Mapping, Sequence -from dataclasses import dataclass, field, replace -from datetime import datetime, timedelta, timezone -from typing import Any - -from ..config import Config -from ..core.models import ( - BackendHealth, - Space, - Worker, - WorkerBinding, - normalize_status, - separate_duplicate_worker_bindings, - sanitize_public_text, - stable_fingerprint, - utc_timestamp, - worker_binding_private_fingerprint, -) -from ..local_state import ( - ConfigStateReport, - LocalStateErrorCode, - LocalStateKind, - PermissionState, - inspect_config_state, -) -from ..worker_identity import ( - InstallationKeyError, - STABLE_KEY_VERSION, - canonical_herdr_pane_identity, - load_or_create_installation_key, - stable_worker_key, -) - - -_HERDR_TIMEOUT_SECONDS = 5.0 -_BACKEND_NAME = "herdr" -_AMBIGUOUS_BINDING_REASONS = frozenset( - {"ambiguous_pane_match", "duplicate_backend_target", "not_unique"} -) - - -class HerdrContinuityUnavailableError(RuntimeError): - """A healthy-looking Herdr observation cannot authenticate worker continuity.""" - - -@dataclass(frozen=True) -class _WorkerRecord: - worker: Worker - private_fingerprint: str - turn_target_kind: str | None = None - turn_target_value: str | None = None - # Canonical public Herdr identity used exclusively for continuity. Raw - # observations remain private and separate so routing compatibility can - # never accidentally feed stable-key derivation. - workspace_id: str | None = None - pane_id: str | None = None - observed_workspace_id: str | None = None - observed_pane_id: str | None = None - identity_source: str = "unknown" - terminal_id: str | None = None - agent_session_id: str | None = None - # Continuity is authorized only by a PaneInfo observation, never by - # workspace/pane-shaped fields reported by agent.list. - pane_info_observed: bool = False - unmatched_agent_observation: bool = False - -_FORBIDDEN_CONNECTOR_FIELDS = { - "telegram", - "chat_id", - "topic_id", - "message_id", - "thread_id", - "token", - "bot_token", - "delivery", - "route", - "herdres_delivery", - "backend_target", -} - -_STATUS_KEYS = ( - "agent_status", - "status", - "state", - "phase", - "lifecycle", - "lifecycle_state", - "raw_status", -) - -_BACKEND_TARGET_KINDS = frozenset( - {"agent_id", "terminal_id", "pane_id", "agent", "name", "label"} -) -_AGENT_SCOPED_BACKEND_TARGET_KINDS = frozenset({"agent_id", "agent"}) -_SESSION_SCOPED_TURN_TARGET_KINDS = frozenset( - {"codex_session_id", "omp_session_path"} -) -_DEADLINE_EXHAUSTED_OUTCOMES = frozenset({"timeout", "deadline_exhausted"}) -_UNAVAILABLE_HEALTH_OUTCOMES = frozenset({"missing_binary", "launch_error", "socket_disconnected"}) -_DEGRADED_HEALTH_OUTCOMES = frozenset( - { - "timeout", - "deadline_exhausted", - "nonzero", - "malformed_json", - "protocol_error", - "worker_cap_exceeded", - "continuity_unavailable", - } -) - -_HEALTH_MESSAGES = { - "healthy_non_empty": "Herdr observation is healthy", - "empty_healthy": "Herdr observation is healthy but empty", - "missing_binary": "Herdr binary is unavailable", - "launch_error": "Herdr launch failed", - "timeout": "Herdr observation timed out", - "deadline_exhausted": "Herdr observation deadline was exhausted", - "nonzero": "Herdr command returned nonzero status", - "malformed_json": "Herdr command returned malformed JSON", - "protocol_error": "Herdr protocol returned an invalid envelope", - "socket_disconnected": "Herdr socket disconnected", - "worker_cap_exceeded": "Herdr observation exceeded the configured worker cap", - "continuity_unavailable": "Herdr continuity identity is unavailable", - "unknown": "Herdr observation state is unknown", -} - - -@dataclass(frozen=True) -class _ProbeBudget: - """Aggregate deadline for a read-only Herdr observation chain.""" - - started_at: float - per_probe_timeout_seconds: float - aggregate_deadline_seconds: float - - @classmethod - def from_config(cls, config: Config, *, planned_probes: int) -> "_ProbeBudget": - per_probe = float(config.herdr_timeout_seconds) - planned = max(1, int(planned_probes)) - return cls( - started_at=time.monotonic(), - per_probe_timeout_seconds=per_probe, - aggregate_deadline_seconds=per_probe * planned, - ) - - def remaining_seconds(self) -> float: - return self.aggregate_deadline_seconds - (time.monotonic() - self.started_at) - - def subprocess_timeout_seconds(self) -> float | None: - remaining = self.remaining_seconds() - if remaining <= 0: - return None - if remaining >= self.per_probe_timeout_seconds: - return self.per_probe_timeout_seconds - return max(0.001, remaining) - - -def herdr_health_status_for_outcome(outcome: str) -> str: - """Map a Herdr adapter outcome into the public backend health status.""" - normalized = str(outcome or "unknown").strip().lower().replace("-", "_") - if normalized in {"healthy_non_empty", "empty_healthy"}: - return "healthy" - if normalized in _UNAVAILABLE_HEALTH_OUTCOMES: - return "unavailable" - if normalized in _DEGRADED_HEALTH_OUTCOMES: - return "degraded" - return "unknown" - - -def herdr_backend_health( - outcome: str, - *, - observed_at: str | None = None, - message: str | None = None, - spaces: Sequence[Space] | None = None, - workers: Sequence[Worker] | None = None, -) -> BackendHealth: - """Return the fixed public-safe health object for a Herdr observation.""" - normalized_outcome = str(outcome or "unknown").strip().lower().replace("-", "_") - if normalized_outcome == "ok": - normalized_outcome = ( - "healthy_non_empty" - if (spaces and len(spaces) > 0) or (workers and len(workers) > 0) - else "empty_healthy" - ) - if normalized_outcome not in { - "healthy_non_empty", - "empty_healthy", - "missing_binary", - "launch_error", - "timeout", - "deadline_exhausted", - "nonzero", - "malformed_json", - "protocol_error", - "socket_disconnected", - "worker_cap_exceeded", - "continuity_unavailable", - "unknown", - }: - normalized_outcome = "unknown" - counts = { - "spaces": len(spaces or []), - "workers": len(workers or []), - } - return BackendHealth( - name=_BACKEND_NAME, - status=herdr_health_status_for_outcome(normalized_outcome), - outcome=normalized_outcome, - observed_at=observed_at or utc_timestamp(), - message=message if message is not None else _HEALTH_MESSAGES[normalized_outcome], - counts=counts, - ) - - -@dataclass(frozen=True) -class HerdrSnapshotObservation: - """Snapshot observation plus public backend health and private bindings.""" - - spaces: list[Space] - workers: list[Worker] - bindings: list[WorkerBinding] = field(default_factory=list) - backend_health: list[BackendHealth] = field(default_factory=list) - - def __post_init__(self) -> None: - spaces = list(self.spaces) - workers = list(self.workers) - bindings = list(self.bindings) - backend_health = list(self.backend_health) - if not backend_health: - outcome = "healthy_non_empty" if spaces or workers else "empty_healthy" - backend_health = [herdr_backend_health(outcome, spaces=spaces, workers=workers)] - object.__setattr__(self, "spaces", spaces) - object.__setattr__(self, "workers", workers) - object.__setattr__(self, "bindings", bindings) - object.__setattr__(self, "backend_health", backend_health) - - @property - def health(self) -> BackendHealth: - for item in self.backend_health: - if item.name == _BACKEND_NAME: - return item - return herdr_backend_health("unknown", spaces=self.spaces, workers=self.workers) - - @property - def authoritative(self) -> bool: - return self.health.status == "healthy" - - -@dataclass(frozen=True) -class HerdrCommandObservation: - """Command execution observation with health metadata.""" - - spaces: list[Space] - workers: list[Worker] - status: str - outcome: str - message: str = "" - bindings: list[WorkerBinding] = field(default_factory=list) - backend_health: list[BackendHealth] = field(default_factory=list) - - def __post_init__(self) -> None: - spaces = list(self.spaces) - workers = list(self.workers) - bindings = list(self.bindings) - backend_health = list(self.backend_health) - if not backend_health: - backend_health = [ - herdr_backend_health( - self.outcome, - message=self.message or None, - spaces=spaces, - workers=workers, - ) - ] - object.__setattr__(self, "spaces", spaces) - object.__setattr__(self, "workers", workers) - object.__setattr__(self, "bindings", bindings) - object.__setattr__(self, "backend_health", backend_health) - - @property - def healthy(self) -> bool: - return self.status == "healthy" and self.health.status == "healthy" - - @property - def health(self) -> BackendHealth: - for item in self.backend_health: - if item.name == _BACKEND_NAME: - return item - return herdr_backend_health(self.outcome, message=self.message or None, spaces=self.spaces, workers=self.workers) - - -_FORBIDDEN_CONNECTOR_FIELDS_COMPACT = {field.replace("_", "") for field in _FORBIDDEN_CONNECTOR_FIELDS} - - -def _compact_field_name(key: object) -> str: - """Normalize field names for conservative connector/status-key matching.""" - return str(key).lower().replace("-", "_").replace(".", "_").replace("_", "") - - -def _field_matches(key: object, expected: str) -> bool: - """Return True when a payload key matches snake_case or camelCase spelling.""" - return _compact_field_name(key) == _compact_field_name(expected) - -def _is_reserved_stable_key_field(key: object) -> bool: - """Match the entire current and future normalized stable-key family.""" - compact = str(key).lower().replace("_", "").replace("-", "").replace(".", "") - return compact.startswith("stablekey") - - -def _strip_stable_key_fields(value: Any) -> Any: - """Recursively remove every source-controlled stable-key family field.""" - if isinstance(value, Mapping): - return { - str(key): _strip_stable_key_fields(child) - for key, child in value.items() - if not _is_reserved_stable_key_field(key) - } - if isinstance(value, list): - return [_strip_stable_key_fields(item) for item in value] - if isinstance(value, tuple): - return [_strip_stable_key_fields(item) for item in value] - return value - - -_TURN_OBSERVATION_FIELD_NAMES = frozenset( - { - "turn", - "turnepoch", - "lastcompletedturn", - "outcome", - } -) - - -def _strip_turn_observation_fields(value: Any) -> Any: - """Remove turn counters and outcomes from every Worker identity surface.""" - if isinstance(value, Mapping): - return { - str(key): _strip_turn_observation_fields(child) - for key, child in value.items() - if _compact_field_name(key) not in _TURN_OBSERVATION_FIELD_NAMES - } - if isinstance(value, list): - return [_strip_turn_observation_fields(item) for item in value] - if isinstance(value, tuple): - return tuple(_strip_turn_observation_fields(item) for item in value) - return value - - -def _private_fingerprint(value: Any) -> str: - """Return a private adapter-only fingerprint without public sanitization.""" - encoded = json.dumps( - value, - sort_keys=True, - separators=(",", ":"), - default=str, - ).encode("utf-8") - return hashlib.sha256(encoded).hexdigest()[:24] - - -def _is_forbidden_connector_field(key: object) -> bool: - """Return True for forbidden connector fields, including common case variants.""" - normalized = str(key).lower().replace("-", "_").replace(".", "_") - compact = _compact_field_name(key) - segments = {part for part in normalized.split("_") if part} - return ( - normalized in _FORBIDDEN_CONNECTOR_FIELDS - or compact in _FORBIDDEN_CONNECTOR_FIELDS_COMPACT - or bool(segments & _FORBIDDEN_CONNECTOR_FIELDS) - ) - - -def _contains_forbidden_connector_text(value: object) -> bool: - """Return True when diagnostic text names connector/private delivery fields.""" - normalized = str(value).lower().replace("-", "_").replace(".", "_") - compact = normalized.replace("_", "") - return any(field in normalized for field in _FORBIDDEN_CONNECTOR_FIELDS) or any( - field in compact for field in _FORBIDDEN_CONNECTOR_FIELDS_COMPACT - ) - - -def _run_herdr( - args: Sequence[str], - config: Config, - *, - timeout_seconds: float | None = None, -) -> subprocess.CompletedProcess[str] | None: - """Run the Herdr CLI with read-only arguments; return None on launch failure.""" - timeout = config.herdr_timeout_seconds if timeout_seconds is None else timeout_seconds - try: - return subprocess.run( - [config.herdr_bin, *args], - capture_output=True, - text=True, - check=False, - timeout=timeout, - ) - except (OSError, UnicodeDecodeError, ValueError, TypeError): - return None - - -def _run_herdr_probe( - args: Sequence[str], - config: Config, - timeout_seconds: float | None, -) -> subprocess.CompletedProcess[str] | None: - """Call _run_herdr with timeout support while preserving simple test fakes.""" - if timeout_seconds is None: - return _run_herdr(args, config) - try: - return _run_herdr(args, config, timeout_seconds=timeout_seconds) - except TypeError: - return _run_herdr(args, config) - - -def _probe_herdr( - args: Sequence[str], - config: Config, - budget: _ProbeBudget | None = None, -) -> tuple[str, Any]: - """Run a read-only Herdr command and retain failure class for mutations.""" - timeout_seconds: float | None = None - if budget is not None: - timeout_seconds = budget.subprocess_timeout_seconds() - if timeout_seconds is None: - return "deadline_exhausted", None - try: - completed = _run_herdr_probe(args, config, timeout_seconds) - except subprocess.TimeoutExpired: - return "timeout", None - if completed is None: - return "launch_error", None - - if completed.returncode != 0: - return "nonzero", None - payload = _parse_json_output(completed.stdout) - if payload is None: - return "malformed_json", None - return "ok", payload - - -def _parse_json_output(stdout: str | None) -> Any: - """Best-effort parse of herdr JSON output; None on failure.""" - if not stdout or not stdout.strip(): - return None - try: - return json.loads(stdout) - except (json.JSONDecodeError, TypeError, ValueError): - return None - - -def _command_payload(args: Sequence[str], config: Config) -> Any: - """Return parsed JSON for a single herdr command, or None on any bad output.""" - outcome, payload = _probe_herdr(args, config) - if outcome != "ok": - return None - return payload - - -def _command_payload_variants(variants: Sequence[Sequence[str]], config: Config) -> Any: - """Try a sequence of herdr arg lists in order; return first successful payload.""" - outcome, payload = _probe_payload_variants(variants, config) - return payload if outcome == "ok" else None - - -def _safe_text_sample(value: str | None) -> str | None: - """Return a short diagnostic sample through the shared public sanitizer.""" - if not value or _contains_forbidden_connector_text(value): - return None - sanitized = sanitize_public_text( - value, - max_chars=200, - collapse_whitespace=True, - ) - return sanitized or None - - -_LOCAL_STATE_COMPLIANT_REMEDIATION = "No action required." -_LOCAL_STATE_UNINITIALIZED_REMEDIATION = ( - "No action required while local state is uninitialized." -) -_LOCAL_STATE_STOPPED_REMEDIATION = "No action required while the daemon is stopped." -_LOCAL_STATE_REPAIR_REMEDIATION = ( - "Restart Tendwire to repair local state permissions." -) -_LOCAL_STATE_UNSAFE_REMEDIATION = ( - "Move unsafe local state aside and restore from a trusted backup." -) -_LOCAL_STATE_CHECK_GROUPS = ( - ( - "state_directory_permissions", - frozenset({LocalStateKind.STATE_DIRECTORY}), - "not_initialized", - _LOCAL_STATE_UNINITIALIZED_REMEDIATION, - ), - ( - "database_permissions", - frozenset( - { - LocalStateKind.DATABASE, - LocalStateKind.DATABASE_WAL, - LocalStateKind.DATABASE_SHM, - LocalStateKind.DATABASE_JOURNAL, - } - ), - "not_initialized", - _LOCAL_STATE_UNINITIALIZED_REMEDIATION, - ), - ( - "identity_permissions", - frozenset({LocalStateKind.PRIVATE_FILE}), - "not_initialized", - _LOCAL_STATE_UNINITIALIZED_REMEDIATION, - ), - ( - "daemon_socket_permissions", - frozenset({LocalStateKind.SOCKET, LocalStateKind.SOCKET_GROUP}), - "not_running", - _LOCAL_STATE_STOPPED_REMEDIATION, - ), -) - - -def _local_state_check( - name: str, - kinds: frozenset[LocalStateKind], - neutral_outcome: str, - neutral_remediation: str, - *, - entries: Sequence[Any], - issues: Sequence[Any], -) -> dict[str, Any]: - """Fold one fixed local-state category into a path-free doctor record.""" - category_entries = [entry for entry in entries if entry.kind in kinds] - category_issues = [issue for issue in issues if issue.kind in kinds] - issue_codes = {issue.code for issue in category_issues} - if issue_codes - {LocalStateErrorCode.INSECURE_MODE}: - return { - "name": name, - "ok": False, - "outcome": "unsafe", - "remediation": _LOCAL_STATE_UNSAFE_REMEDIATION, - } - if ( - LocalStateErrorCode.INSECURE_MODE in issue_codes - or any( - entry.state is PermissionState.REPAIR_REQUIRED - for entry in category_entries - ) - ): - return { - "name": name, - "ok": False, - "outcome": "repair_required", - "remediation": _LOCAL_STATE_REPAIR_REMEDIATION, - } - if not category_entries or all( - entry.state is PermissionState.ABSENT for entry in category_entries - ): - return { - "name": name, - "ok": True, - "outcome": neutral_outcome, - "remediation": neutral_remediation, - } - return { - "name": name, - "ok": True, - "outcome": "compliant", - "remediation": _LOCAL_STATE_COMPLIANT_REMEDIATION, - } - - -def _inspect_local_state(config: Config) -> ConfigStateReport | None: - """Inspect configured local state once without creating or repairing it.""" - try: - if config.db_path is None: - raise ValueError - socket_path = config.socket_path or config.data_dir / "tendwire.sock" - return inspect_config_state( - config.data_dir, - config.db_path, - socket_path=socket_path, - private_files=( - config.installation_key_path, - config.installation_key_marker_path, - config.installation_key_sentinel_path, - ), - socket_group=config.socket_group, - ) - except Exception: - return None - - -def _local_state_checks( - report: ConfigStateReport | None, -) -> tuple[list[dict[str, Any]], bool]: - """Fold a path-free inspection report into the fixed local-state checks.""" - inspected = report - if inspected is None: - checks = [ - { - "name": name, - "ok": False, - "outcome": "unsafe", - "remediation": _LOCAL_STATE_UNSAFE_REMEDIATION, - } - for name, _kinds, _neutral, _remediation in _LOCAL_STATE_CHECK_GROUPS - ] - return checks, False - - checks = [ - _local_state_check( - name, - kinds, - neutral_outcome, - neutral_remediation, - entries=inspected.entries, - issues=inspected.issues, - ) - for name, kinds, neutral_outcome, neutral_remediation in _LOCAL_STATE_CHECK_GROUPS - ] - return checks, inspected.ok - - -def _public_herdr_bin(value: str) -> str: - """Retain the doctor key while redacting configured path-shaped values.""" - sanitized = sanitize_public_text( - value, - max_chars=200, - collapse_whitespace=True, - ) - return sanitized or "[redacted]" - - -_STORE_MAINTENANCE_REMEDIATION = { - "ok": "No action required.", - "overdue": "Keep Tendwire running to resume automatic store maintenance.", - "backlog": "Keep Tendwire running to drain the maintenance backlog.", - "not_initialized": "No action required while the store is uninitialized.", - "unsafe": "Move unsafe local state aside and restore from a trusted backup.", - "unavailable": "Check store availability before retrying diagnostics.", -} -_STORE_KINDS = frozenset( - { - LocalStateKind.STATE_DIRECTORY, - LocalStateKind.DATABASE, - LocalStateKind.DATABASE_WAL, - LocalStateKind.DATABASE_SHM, - LocalStateKind.DATABASE_JOURNAL, - } -) - - -def _store_maintenance_record( - config: Config, - outcome: str, - *, - snapshot_count: int = 0, - last_completed_at: str | None = None, -) -> dict[str, Any]: - """Build the fixed public-safe maintenance doctor record.""" - return { - "name": "store_maintenance", - "ok": outcome in {"ok", "not_initialized"}, - "outcome": outcome, - "remediation": _STORE_MAINTENANCE_REMEDIATION[outcome], - "snapshot_retention_days": config.snapshot_retention_days, - "snapshot_retention_count": config.snapshot_retention_count, - "maintenance_batch_size": config.snapshot_maintenance_batch_size, - "maintenance_cadence_seconds": config.store_maintenance_cadence_seconds, - "snapshot_count": snapshot_count, - "last_completed_at": last_completed_at, - } - - -def _public_utc_timestamp(value: Any) -> tuple[str | None, datetime | None]: - """Return a normalized public UTC timestamp, rejecting malformed/private text.""" - if not isinstance(value, str) or not value.strip(): - return None, None - raw = value.strip() - if raw.endswith("Z"): - raw = raw[:-1] + "+00:00" - try: - parsed = datetime.fromisoformat(raw) - except (TypeError, ValueError): - return None, None - if parsed.tzinfo is None: - return None, None - normalized = parsed.astimezone(timezone.utc) - return normalized.isoformat(), normalized - - - - -def _store_maintenance_check( - config: Config, - report: ConfigStateReport | None, -) -> dict[str, Any]: - """Return one fixed, path-free, non-mutating store-maintenance check.""" - if report is None: - return _store_maintenance_record(config, "unsafe") - relevant_issues = [issue for issue in report.issues if issue.kind in _STORE_KINDS] - relevant_entries = [entry for entry in report.entries if entry.kind in _STORE_KINDS] - if relevant_issues or any( - entry.state is PermissionState.REPAIR_REQUIRED for entry in relevant_entries - ): - return _store_maintenance_record(config, "unsafe") - database = next( - (entry for entry in relevant_entries if entry.kind is LocalStateKind.DATABASE), - None, - ) - if database is None or database.state is PermissionState.ABSENT: - return _store_maintenance_record(config, "not_initialized") - - try: - from ..store.sqlite import store_status - - status = store_status( - config.db_path, - config.host_id, - snapshot_retention_days=config.snapshot_retention_days, - snapshot_retention_count=config.snapshot_retention_count, - maintenance_batch_size=config.snapshot_maintenance_batch_size, - maintenance_cadence_seconds=config.store_maintenance_cadence_seconds, - require_current_schema=True, - ) - maintenance = status.get("maintenance") - if ( - status.get("ok") is not True - or not isinstance(maintenance, Mapping) - or isinstance(maintenance.get("snapshot_count"), bool) - or not isinstance(maintenance.get("snapshot_count"), int) - or int(maintenance["snapshot_count"]) < 0 - or not isinstance(maintenance.get("backlog"), bool) - or maintenance.get("status") not in {"never", "ok", "failed"} - ): - return _store_maintenance_record(config, "unavailable") - snapshot_count = int(maintenance["snapshot_count"]) - maintenance_status = str(maintenance["status"]) - last_value = maintenance.get("last_completed_at") - if maintenance_status == "failed" or ( - maintenance_status == "never" and last_value is not None - ): - return _store_maintenance_record(config, "unavailable") - if last_value is None: - last_completed_at = None - completed = None - else: - last_completed_at, completed = _public_utc_timestamp(last_value) - if completed is None: - return _store_maintenance_record(config, "unavailable") - if maintenance["backlog"]: - outcome = "backlog" - elif completed is None: - outcome = "overdue" - else: - _now_value, now = _public_utc_timestamp(utc_timestamp()) - if now is None: - return _store_maintenance_record(config, "unavailable") - due_at = completed + timedelta( - seconds=config.store_maintenance_cadence_seconds - ) - outcome = "overdue" if now >= due_at else "ok" - return _store_maintenance_record( - config, - outcome, - snapshot_count=snapshot_count, - last_completed_at=last_completed_at, - ) - except Exception: - return _store_maintenance_record(config, "unavailable") - - -def _pending_ingestion_check(config: Config) -> dict[str, Any]: - """Return one fixed non-mutating check from durable pending state only.""" - unavailable = { - "status": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - } - try: - from ..store.sqlite import backend_pending_health - - value = backend_pending_health(config.db_path, config.host_id) - except Exception: - value = unavailable - raw = value if isinstance(value, Mapping) else unavailable - raw_counts = raw.get("counts") - status = raw.get("status") - count_values = ( - tuple(raw_counts.get(key) for key in ("fresh", "stale", "total")) - if isinstance(raw_counts, Mapping) - else () - ) - valid_counts = len(count_values) == 3 and all( - isinstance(item, int) and not isinstance(item, bool) and item >= 0 - for item in count_values - ) - if ( - status not in {"healthy", "degraded", "store_unavailable"} - or not valid_counts - or count_values[2] != count_values[0] + count_values[1] - or (status == "healthy" and count_values[1] != 0) - or (status == "degraded" and count_values[1] == 0) - or (status == "store_unavailable" and count_values != (0, 0, 0)) - ): - status = "store_unavailable" - counts = dict(unavailable["counts"]) - else: - counts = dict(zip(("fresh", "stale", "total"), count_values, strict=True)) - return { - "name": "pending_ingestion", - "ok": status == "healthy", - "outcome": status, - "counts": counts, - "stale_grace_seconds": config.pending_stale_grace_seconds, - } - - -def _finish_diagnostics(result: dict[str, Any], config: Config) -> dict[str, Any]: - report = _inspect_local_state(config) - local_checks, local_ok = _local_state_checks(report) - maintenance = _store_maintenance_check(config, report) - pending_ingestion = _pending_ingestion_check(config) - result["checks"].extend(local_checks) - result["checks"].append(maintenance) - result["checks"].append(pending_ingestion) - if ( - (not local_ok or not maintenance["ok"] or not pending_ingestion["ok"]) - and result["status"] == "ok" - ): - result["status"] = "degraded" - return result - - -def _diagnostic_item_count(payload: Any, keys: Sequence[str]) -> int: - return len(_payload_items(payload, keys)) - - -def _diagnostic_check( - name: str, - args: Sequence[str], - config: Config, - keys: Sequence[str], - budget: _ProbeBudget, -) -> dict[str, Any]: - """Run one read-only Herdr command and return a sanitized diagnostic record.""" - check: dict[str, Any] = { - "name": name, - "ok": False, - "outcome": "unknown", - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": budget.aggregate_deadline_seconds, - } - timeout_seconds = budget.subprocess_timeout_seconds() - if timeout_seconds is None: - check["outcome"] = "deadline_exhausted" - return check - try: - completed = subprocess.run( - [config.herdr_bin, *args], - capture_output=True, - text=True, - check=False, - timeout=timeout_seconds, - ) - except subprocess.TimeoutExpired: - check["outcome"] = "timeout" - return check - except (OSError, UnicodeDecodeError, ValueError, TypeError): - check["outcome"] = "launch_error" - return check - - check["exit_code"] = int(completed.returncode) - if completed.returncode != 0: - check["outcome"] = "nonzero" - stdout_sample = _safe_text_sample(completed.stdout) - stderr_sample = _safe_text_sample(completed.stderr) - if stdout_sample is not None: - check["stdout_sample"] = stdout_sample - if stderr_sample is not None: - check["stderr_sample"] = stderr_sample - return check - - payload = _parse_json_output(completed.stdout) - if payload is None: - check["outcome"] = "malformed_json" - stdout_sample = _safe_text_sample(completed.stdout) - if stdout_sample is not None: - check["stdout_sample"] = stdout_sample - return check - - item_count = _diagnostic_item_count(payload, keys) - check["ok"] = True - check["item_count"] = item_count - check["outcome"] = "healthy_non_empty" if item_count else "empty_healthy" - return check - - -def diagnose_herdr(config: Config) -> dict[str, Any]: - """Return JSON-serializable read-only Herdr CLI diagnostics.""" - groups = [ - [ - ("workspace_list", ["workspace", "list"], ("workspaces", "spaces", "data", "items", "results", "result")), - ("workspace_list_json", ["workspace", "list", "--json"], ("workspaces", "spaces", "data", "items", "results", "result")), - ], - [ - ("agent_list", ["agent", "list"], ("agents", "workers", "data", "items", "results", "result")), - ("agent_list_json", ["agent", "list", "--json"], ("agents", "workers", "data", "items", "results", "result")), - ], - [ - ("pane_list", ["pane", "list"], ("panes", "items", "data", "results", "result")), - ("pane_list_json", ["pane", "list", "--json"], ("panes", "items", "data", "results", "result")), - ], - ] - planned = [check for group in groups for check in group] - result: dict[str, Any] = { - "schema_version": 1, - "command": "doctor", - "herdr_bin": _public_herdr_bin(config.herdr_bin), - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": config.herdr_timeout_seconds * len(planned), - "status": "ok", - "checks": [], - } - try: - binary_path = shutil.which(config.herdr_bin) - except (TypeError, ValueError, OSError): - binary_path = None - - if binary_path is None: - result["status"] = "unavailable" - result["checks"] = [ - { - "name": name, - "ok": False, - "outcome": "missing_binary", - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": result["aggregate_deadline_seconds"], - } - for name, args, _keys in planned - ] - return _finish_diagnostics(result, config) - - budget = _ProbeBudget.from_config(config, planned_probes=len(planned)) - checks: list[dict[str, Any]] = [] - stop_after_timeout = False - stop_after_outcome = "" - for group in groups: - if stop_after_timeout: - break - for index, (name, args, keys) in enumerate(group): - if index > 0 and checks[-1]["ok"]: - break - check = _diagnostic_check(name, args, config, keys, budget) - checks.append(check) - if check["outcome"] in _DEADLINE_EXHAUSTED_OUTCOMES: - stop_after_timeout = True - stop_after_outcome = str(check["outcome"]) - break - - names_seen = {str(check["name"]) for check in checks} - remaining_planned = [ - (name, args, keys) - for name, args, keys in planned - if name not in names_seen - ] - if stop_after_timeout: - skipped_outcome = ( - "skipped_after_deadline" - if stop_after_outcome == "deadline_exhausted" - else "skipped_after_timeout" - ) - for name, args, _keys in remaining_planned: - checks.append( - { - "name": name, - "ok": False, - "outcome": skipped_outcome, - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": budget.aggregate_deadline_seconds, - } - ) - else: - for name, args, _keys in remaining_planned: - checks.append( - { - "name": name, - "ok": True, - "outcome": "skipped_not_needed", - "timeout_seconds": config.herdr_timeout_seconds, - "aggregate_deadline_seconds": budget.aggregate_deadline_seconds, - } - ) - - result["checks"] = checks - if any(check["outcome"] in _DEADLINE_EXHAUSTED_OUTCOMES for check in checks): - result["status"] = "timeout" - elif any(not check["ok"] for check in checks): - result["status"] = "degraded" - return _finish_diagnostics(result, config) - - -def _strip_connector_fields(value: Any) -> Any: - """Recursively drop connector/delivery fields from arbitrary JSON-like values.""" - if isinstance(value, Mapping): - clean: dict[str, Any] = {} - for key, child in value.items(): - key_text = str(key) - if _is_forbidden_connector_field(key): - continue - clean[key_text] = _strip_connector_fields(child) - return clean - if isinstance(value, list): - return [_strip_connector_fields(item) for item in value] - if isinstance(value, tuple): - return [_strip_connector_fields(item) for item in value] - return value - - -def _strip_status_fields(value: Any) -> Any: - """Recursively drop raw status fields from metadata values.""" - if isinstance(value, Mapping): - clean: dict[str, Any] = {} - for key, child in value.items(): - if any(_field_matches(key, status_key) for status_key in _STATUS_KEYS): - continue - clean[str(key)] = _strip_status_fields(child) - return clean - if isinstance(value, list): - return [_strip_status_fields(item) for item in value] - if isinstance(value, tuple): - return [_strip_status_fields(item) for item in value] - return value - - -def _payload_items(payload: Any, keys: Sequence[str]) -> list[dict[str, Any]]: - """Extract object records from conservative herdr list payload shapes.""" - if isinstance(payload, list): - candidates: Iterable[Any] = payload - elif isinstance(payload, Mapping): - candidates = () - for key in keys: - value = _value_for_key(payload, key) - if isinstance(value, list): - candidates = value - break - if isinstance(value, Mapping): - nested = _payload_items(value, keys) - if nested: - return nested - else: - return [] - - items: list[dict[str, Any]] = [] - for item in candidates: - if isinstance(item, Mapping): - stripped = _strip_connector_fields(item) - if isinstance(stripped, dict): - items.append(stripped) - return items - - -def _first_text(item: Mapping[str, Any], keys: Sequence[str]) -> str | None: - """Return the first scalar string value for any key.""" - for key in keys: - value = _value_for_key(item, key) - if value is None: - continue - if isinstance(value, (str, int, float, bool)): - return str(value) - return None - - -def _nested_text(item: Mapping[str, Any], *path: str) -> str | None: - """Return the first scalar string value reachable via a dotted key path.""" - current: Any = item - for key in path: - if not isinstance(current, Mapping): - return None - current = _value_for_key(current, key) - if current is None: - return None - if isinstance(current, (str, int, float, bool)): - return str(current) - if isinstance(current, Mapping): - return _first_text(current, ("id", "value", "name", "label")) - return None - - -def _related_id(value: Any) -> str | None: - """Return a neutral related-object id from a scalar or mapping.""" - if value is None: - return None - if isinstance(value, Mapping): - return _first_text(value, ("id", "workspace_id", "space_id", "slug", "name")) - if isinstance(value, (str, int, float, bool)): - return str(value) - return None - - -def _normalize_status(raw_status: Any) -> tuple[str, str | None]: - """Return canonical status plus original raw string when normalization changed it.""" - if raw_status is None: - return "unknown", None - - raw_text = str(raw_status) - canonical = normalize_status(raw_text) - raw_key = raw_text.strip().lower().replace("_", "-") - raw_meta = raw_text if raw_text and raw_key != canonical else None - return canonical, raw_meta - - -def _value_for_key(item: Mapping[str, Any], expected_key: str) -> Any: - """Return a value by exact key or snake/camel-case equivalent.""" - if expected_key in item: - return item[expected_key] - for key, value in item.items(): - if _field_matches(key, expected_key): - return value - return None - - -def _status_from_item(item: Mapping[str, Any]) -> tuple[str, str | None]: - """Extract and normalize status-like fields from a herdr record.""" - for key in _STATUS_KEYS: - value = _value_for_key(item, key) - if value is not None: - return _normalize_status(value) - return "unknown", None - - -def _meta_from_item(item: Mapping[str, Any], excluded_keys: set[str], raw_status: str | None) -> dict[str, Any]: - """Build sanitized neutral metadata for a projected model.""" - item = _strip_stable_key_fields(item) - explicit_meta = _value_for_key(item, "meta") - meta = { - str(key): _strip_status_fields(value) - for key, value in item.items() - if not _field_matches(key, "meta") - and not any(_field_matches(key, excluded_key) for excluded_key in excluded_keys) - and not any(_field_matches(key, status_key) for status_key in _STATUS_KEYS) - } - if isinstance(explicit_meta, Mapping): - for key, value in explicit_meta.items(): - if _is_forbidden_connector_field(key): - continue - if any(_field_matches(key, status_key) for status_key in _STATUS_KEYS): - continue - meta[str(key)] = _strip_status_fields(value) - if raw_status is not None: - meta["raw_status"] = raw_status - return meta - - -def _space_id_from_item(item: Mapping[str, Any]) -> str: - """Resolve a stable space id, preferring explicit Herdr workspace_id.""" - return _first_text(item, ("workspace_id", "space_id", "id", "slug", "name")) or "unknown" - - -def _space_name_from_item(item: Mapping[str, Any], space_id: str) -> str: - """Resolve a space name, preferring label then workspace_id.""" - return _first_text(item, ("label", "name", "title", "workspace_id", "space_id")) or space_id - - -def _public_worker_id_base_from_item(item: Mapping[str, Any]) -> str: - """Resolve a neutral public worker id without terminal/session handles.""" - return ( - _first_text(item, ("worker_id", "id", "slug", "agent_id")) - or _first_text(item, ("agent", "name", "label", "title")) - or "unknown" - ) - - -def _worker_id_from_item(item: Mapping[str, Any]) -> str: - """Resolve a stable public worker id.""" - return _public_worker_id_base_from_item(item) - - -def _private_identity_material_from_item(item: Mapping[str, Any]) -> dict[str, Any]: - """Return private Herdr identity material that is never serialized publicly.""" - agent_id = _first_text(item, ("agent_id",)) - agent_session = _nested_text(item, "agent_session", "value") - session_id = _first_text(item, ("session_id",)) - if agent_id or agent_session or session_id: - return { - "agent_id": agent_id, - "agent_session": agent_session, - "session_id": session_id, - "space_id": _worker_space_id_from_item(item), - } - base = { - "public_id": _public_worker_id_base_from_item(item), - "name": _first_text(item, ("agent", "name", "label", "title")), - "space_id": _worker_space_id_from_item(item), - } - base["terminal_id"] = _first_text(item, ("terminal_id",)) - base["pane_id"] = _first_text(item, ("pane_id",)) - return base - - -def _private_identity_from_item(item: Mapping[str, Any], config: Config | None = None) -> str: - """Return a private identity used only to avoid collapsing distinct workers.""" - material = _private_identity_material_from_item(item) - if config is None: - return _private_fingerprint(material) - return worker_binding_private_fingerprint( - host_id=config.host_id, - backend=_BACKEND_NAME, - identity_material=material, - ) - - -def _private_backend_target(kind: str, value: str, *, sendable: bool = True, reason: str | None = None) -> dict[str, Any]: - """Return the internal backend target shape.""" - return { - "kind": kind, - "value": value, - "sendable": bool(sendable), - "reason": reason, - } - - -def _backend_target_from_item(item: Mapping[str, Any]) -> dict[str, Any] | None: - """Resolve the private Herdr send target from backend-observed fields.""" - candidates = ( - ("agent_id", _first_text(item, ("agent_id",))), - ("terminal_id", _first_text(item, ("terminal_id",))), - ("pane_id", _first_text(item, ("pane_id",))), - ("agent", _first_text(item, ("agent",))), - ("name", _first_text(item, ("name",))), - ("label", _first_text(item, ("label",))), - ) - for kind, value in candidates: - if value: - return _private_backend_target(kind, value) - return None - - -def _turn_target_from_item(item: Mapping[str, Any]) -> tuple[str, str] | None: - """Resolve the private Herdr structured-turn target from backend-observed fields.""" - agent_name = (_first_text(item, ("agent", "name")) or "").strip().lower() - agent_session = _nested_text(item, "agent_session", "value") - if agent_name == "codex" and agent_session: - return "codex_session_id", agent_session - if agent_name == "omp" and agent_session: - # oh-my-pi reports its native session file path (herdr:omp source). - return "omp_session_path", agent_session - pane_id = _first_text(item, ("pane_id",)) - if pane_id: - return "pane_id", pane_id - return None - - -def _worker_with_id(worker: Worker, worker_id: str) -> Worker: - """Return a worker copy with a disambiguated public id.""" - return Worker( - id=worker_id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - backend_target=worker.backend_target, - ) - - -def _worker_name_from_item(item: Mapping[str, Any], worker_id: str) -> str: - """Resolve a worker display name, preferring agent then label.""" - return _first_text(item, ("agent", "label", "name", "title")) or worker_id - - -def _worker_space_id_from_item(item: Mapping[str, Any]) -> str | None: - """Resolve a worker's parent space id, preferring workspace_id.""" - return ( - _first_text(item, ("workspace_id", "space_id", "spaceId", "workspaceId")) - or _related_id(_value_for_key(item, "space")) - or _related_id(_value_for_key(item, "workspace")) - ) - - -def _spaces_from_payload(payload: Any) -> list[Space]: - """Extract neutral Space objects from a herdr workspace-list payload.""" - spaces: list[Space] = [] - for item in _payload_items(payload, ("workspaces", "spaces", "data", "items", "results", "result")): - space_id = _space_id_from_item(item) - name = _space_name_from_item(item, space_id) - status, raw_status = _status_from_item(item) - updated_at = _first_text(item, ("updated_at", "last_seen_at", "observed_at", "timestamp")) - status_line = _first_text(item, ("status_line", "summary", "description")) - meta = _meta_from_item( - item, - { - "id", - "workspace_id", - "space_id", - "slug", - "name", - "title", - "label", - "meta", - "updated_at", - "last_seen_at", - "observed_at", - "timestamp", - "status_line", - "summary", - "description", - "fingerprint", - "agent_status", - }, - raw_status, - ) - spaces.append( - Space( - id=space_id, - name=name, - status=status, - meta=meta, - updated_at=updated_at, - status_line=status_line, - ) - ) - return spaces - - -def _worker_from_item(item: Mapping[str, Any]) -> Worker: - """Build a neutral Worker from a single herdr agent/pane record.""" - worker_id = _worker_id_from_item(item) - name = _worker_name_from_item(item, worker_id) - status, raw_status = _status_from_item(item) - last_seen_at = _first_text(item, ("last_seen_at", "updated_at", "observed_at", "timestamp")) - summary = _first_text(item, ("summary", "status_line", "description")) - space_id = _worker_space_id_from_item(item) - meta = _meta_from_item( - item, - { - "id", - "agent_id", - "worker_id", - "slug", - "name", - "title", - "agent", - "meta", - "space_id", - "workspace_id", - "spaceId", - "workspaceId", - "space", - "workspace", - "last_seen_at", - "updated_at", - "observed_at", - "timestamp", - "summary", - "status_line", - "description", - "fingerprint", - "agent_status", - "backend_target", - "terminal_id", - "pane_id", - "agent_session", - "session_id", - }, - raw_status, - ) - return Worker( - id=worker_id, - name=name, - status=status, - space_id=space_id, - meta=meta, - last_seen_at=last_seen_at, - summary=summary, - backend_target=_backend_target_from_item(item), - ) - - -def _output_excerpt_limit(config: Config | None) -> int | None: - if config is None: - return None - try: - limit = int(getattr(config, "output_excerpt_chars")) - except (TypeError, ValueError): - return None - return max(1, limit) - - -def _bounded_excerpt(value: str | None, limit: int | None) -> str | None: - if value is None or limit is None: - return value - text = str(value) - if len(text) <= limit: - return text - if limit <= 3: - return text[:limit] - return text[: limit - 3] + "..." - - -def _worker_with_summary(worker: Worker, summary: str | None) -> Worker: - if summary == worker.summary: - return worker - return Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=summary, - backend_target=worker.backend_target, - ) - - -def _worker_record_from_item( - item: Mapping[str, Any], - config: Config | None = None, - *, - pane_info_observed: bool = False, - identity_source: str = "unknown", -) -> _WorkerRecord: - item = _strip_turn_observation_fields(item) - worker = _worker_from_item(item) - worker = _worker_with_summary(worker, _bounded_excerpt(worker.summary, _output_excerpt_limit(config))) - turn_target = _turn_target_from_item(item) - observed_workspace_id = _first_text(item, ("workspace_id", "workspaceId")) - observed_pane_id = _first_text(item, ("pane_id", "paneId")) - canonical_identity = canonical_herdr_pane_identity( - observed_workspace_id, - observed_pane_id, - ) - workspace_id, pane_id = canonical_identity or (None, None) - return _WorkerRecord( - worker=worker, - private_fingerprint=_private_identity_from_item(item, config), - turn_target_kind=turn_target[0] if turn_target is not None else None, - turn_target_value=turn_target[1] if turn_target is not None else None, - workspace_id=workspace_id, - pane_id=pane_id, - observed_workspace_id=observed_workspace_id, - observed_pane_id=observed_pane_id, - identity_source=identity_source, - terminal_id=_first_text(item, ("terminal_id", "terminalId")), - agent_session_id=( - _nested_text(item, "agent_session", "value") - or _first_text(item, ("session_id", "sessionId")) - ), - pane_info_observed=pane_info_observed, - ) - - -def _worker_with_backend_target(worker: Worker, backend_target: dict[str, Any] | None) -> Worker: - return Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - fingerprint=worker.fingerprint, - backend_target=backend_target, - ) - - -def _agent_observation_record( - item: Mapping[str, Any], - config: Config | None = None, -) -> _WorkerRecord: - """Record agent.list provenance without authorizing pane continuity.""" - return replace( - _worker_record_from_item( - item, - config, - pane_info_observed=False, - identity_source="agent.list", - ), - unmatched_agent_observation=True, - ) - - -def _stable_pane_identity(record: _WorkerRecord) -> tuple[str, str] | None: - """Return a PaneInfo-verified workspace/public-pane pair, never a runtime id.""" - if not record.pane_info_observed: - return None - return canonical_herdr_pane_identity(record.workspace_id, record.pane_id) - - -def _worker_with_stable_key( - config: Config, - record: _WorkerRecord, - installation_key: bytes | None, -) -> Worker: - """Replace source continuity metadata with a locally authenticated key.""" - worker = record.worker - meta = _strip_stable_key_fields(worker.meta) - identity = _stable_pane_identity(record) - if installation_key is not None and identity is not None: - workspace_id, pane_id = identity - meta["stable_key"] = stable_worker_key( - installation_key, - backend=_BACKEND_NAME, - host_id=config.host_id, - workspace_id=workspace_id, - pane_id=pane_id, - ) - meta["stable_key_version"] = STABLE_KEY_VERSION - return Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - backend_target=worker.backend_target, - ) - - -def _backend_target_send_token(target: Mapping[str, Any] | None) -> str: - """Return the exact value passed as herdr agent send's target argv token.""" - if not isinstance(target, Mapping): - return "" - return str(target.get("value") or "") - - -def _mark_backend_sendability(workers: list[Worker]) -> list[Worker]: - """Mark unsupported or duplicate final Herdr send tokens as not sendable.""" - marked: list[Worker] = [] - for worker in workers: - target = worker.backend_target - if not isinstance(target, dict): - marked.append(worker) - continue - kind = str(target.get("kind") or "") - value = _backend_target_send_token(target) - if target.get("sendable") is False: - marked.append(worker) - continue - if kind not in _BACKEND_TARGET_KINDS or not value: - marked.append( - _worker_with_backend_target( - worker, - _private_backend_target(kind or "agent", value, sendable=False, reason="backend_unsupported"), - ) - ) - continue - marked.append(_worker_with_backend_target(worker, _private_backend_target(kind, value))) - - send_token_counts = Counter( - _backend_target_send_token(worker.backend_target) - for worker in marked - if isinstance(worker.backend_target, dict) and worker.backend_target.get("sendable") is True - ) - final: list[Worker] = [] - for worker in marked: - target = worker.backend_target - if not isinstance(target, dict) or target.get("sendable") is not True: - final.append(worker) - continue - kind = str(target.get("kind") or "") - value = _backend_target_send_token(target) - if send_token_counts[value] > 1: - final.append( - _worker_with_backend_target( - worker, - _private_backend_target(kind, value, sendable=False, reason="duplicate_backend_target"), - ) - ) - continue - final.append(worker) - return final - - -def assert_unique_sendable_backend_targets(workers: Iterable[Worker]) -> bool: - """Prove no sendable workers share the same final Herdr argv target token.""" - seen: set[str] = set() - for worker in workers: - target = worker.backend_target - if not isinstance(target, Mapping) or target.get("sendable") is not True: - continue - token = _backend_target_send_token(target) - if not token: - raise AssertionError("sendable backend target is missing a send token") - if token in seen: - raise AssertionError("duplicate sendable backend target token") - seen.add(token) - return True - - -def _binding_target_key(binding: WorkerBinding) -> tuple[str, str]: - return (binding.target_kind, binding.target_value) - - -def _safe_stored_binding_for_reuse(binding: WorkerBinding) -> bool: - return bool(binding.worker_id) and (binding.reason or "") not in _AMBIGUOUS_BINDING_REASONS - - -def _worker_target_key(worker: Worker) -> tuple[str, str] | None: - target = worker.backend_target - if not isinstance(target, Mapping): - return None - kind = str(target.get("kind") or "") - value = str(target.get("value") or "") - if not kind or not value: - return None - return kind, value - - -def _record_with_worker(record: _WorkerRecord, worker: Worker) -> _WorkerRecord: - return _WorkerRecord( - worker=worker, - private_fingerprint=record.private_fingerprint, - turn_target_kind=record.turn_target_kind, - turn_target_value=record.turn_target_value, - workspace_id=record.workspace_id, - pane_id=record.pane_id, - observed_workspace_id=record.observed_workspace_id, - observed_pane_id=record.observed_pane_id, - identity_source=record.identity_source, - terminal_id=record.terminal_id, - agent_session_id=record.agent_session_id, - pane_info_observed=record.pane_info_observed, - unmatched_agent_observation=record.unmatched_agent_observation, - ) - - -def _reuse_worker_ids_from_bindings( - records: list[_WorkerRecord], - stored_bindings: Sequence[WorkerBinding] | None, -) -> list[_WorkerRecord]: - """Reuse stable public ids from private binding matches when safe.""" - if not stored_bindings: - return records - - by_private: dict[str, list[WorkerBinding]] = {} - by_target: dict[tuple[str, str], list[WorkerBinding]] = {} - for binding in stored_bindings: - if binding.backend != _BACKEND_NAME: - continue - by_private.setdefault(binding.private_fingerprint, []).append(binding) - if _safe_stored_binding_for_reuse(binding): - key = _binding_target_key(binding) - if key[0] and key[1]: - by_target.setdefault(key, []).append(binding) - - current_private_counts = Counter(record.private_fingerprint for record in records) - current_target_counts = Counter( - key - for key in (_worker_target_key(record.worker) for record in records) - if key is not None - ) - - reused: list[_WorkerRecord] = [] - for record in records: - worker = record.worker - private_fingerprint = record.private_fingerprint - matched = None - private_candidates = [ - binding - for binding in by_private.get(private_fingerprint, []) - if _safe_stored_binding_for_reuse(binding) - ] - if current_private_counts[private_fingerprint] == 1 and len(private_candidates) == 1: - matched = private_candidates[0] - if matched is None: - key = _worker_target_key(worker) - candidates = by_target.get(key or ("", ""), []) - if key is not None and current_target_counts[key] == 1 and len(candidates) == 1: - matched = candidates[0] - if matched is not None and matched.worker_id: - reused.append(_record_with_worker(record, _worker_with_id(worker, matched.worker_id))) - else: - reused.append(record) - return reused - - -def _deduplicated_worker_records( - records: list[_WorkerRecord], - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[_WorkerRecord]: - """Drop exact duplicates, then disambiguate duplicate public ids.""" - records = _reuse_worker_ids_from_bindings(records, stored_bindings) - seen: set[tuple[str, str, str | None, str, str, str]] = set() - unique: list[_WorkerRecord] = [] - for record in records: - worker = record.worker - backend_kind = "" - backend_value = "" - if worker.backend_target: - backend_kind = str(worker.backend_target.get("kind", "")) - backend_value = str(worker.backend_target.get("value", "")) - key = (worker.id, worker.name, worker.space_id, backend_kind, backend_value, record.private_fingerprint) - if key in seen: - continue - seen.add(key) - unique.append(record) - - groups: dict[str, list[_WorkerRecord]] = {} - for record in unique: - groups.setdefault(record.worker.id, []).append(record) - - disambiguated: list[_WorkerRecord] = [] - for worker_id, group in groups.items(): - if len(group) == 1: - disambiguated.append(group[0]) - continue - ordered = sorted( - group, - key=lambda record: ( - record.worker.name, - record.worker.space_id or "", - str((record.worker.backend_target or {}).get("kind", "")), - str((record.worker.backend_target or {}).get("value", "")), - record.private_fingerprint, - stable_fingerprint( - { - "id": record.worker.id, - "name": record.worker.name, - "space_id": record.worker.space_id, - "status": record.worker.status, - "summary": record.worker.summary, - } - ), - ), - ) - for index, record in enumerate(ordered, start=1): - disambiguated.append(_record_with_worker(record, _worker_with_id(record.worker, f"{worker_id}-{index}"))) - - disambiguated = sorted(disambiguated, key=lambda record: record.worker.id) - workers = _mark_backend_sendability([record.worker for record in disambiguated]) - assert_unique_sendable_backend_targets(workers) - return [ - _record_with_worker(record, worker) - for record, worker in zip(disambiguated, workers, strict=True) - ] - - -def _deduplicate_worker_records( - records: list[_WorkerRecord], - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[Worker]: - return [record.worker for record in _deduplicated_worker_records(records, stored_bindings)] - - -def _deduplicate_workers(workers: list[Worker]) -> list[Worker]: - return _deduplicate_worker_records( - [_WorkerRecord(worker=worker, private_fingerprint=worker.fingerprint) for worker in workers] - ) - - -def _binding_from_worker_record( - config: Config, - record: _WorkerRecord, - observed_at: str, -) -> WorkerBinding | None: - worker = record.worker - target = worker.backend_target - if not isinstance(target, Mapping): - return None - target_kind = str(target.get("kind") or "") - target_value = str(target.get("value") or "") - if not target_kind or not target_value: - return None - reason = target.get("reason") - return WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend=_BACKEND_NAME, - target_kind=target_kind, - target_value=target_value, - turn_target_kind=record.turn_target_kind, - turn_target_value=record.turn_target_value, - sendable=target.get("sendable") is True, - reason=str(reason) if reason is not None else None, - observed_at=observed_at, - expires_at=None, - private_fingerprint=record.private_fingerprint, - ) - - -def _workers_and_bindings_from_records( - config: Config, - records: list[_WorkerRecord], - *, - stored_bindings: Sequence[WorkerBinding] | None = None, - require_authenticated_continuity: bool = False, -) -> tuple[list[Worker], list[WorkerBinding]]: - observed_at = utc_timestamp() - deduplicated = _deduplicated_worker_records(records, stored_bindings) - if require_authenticated_continuity and any( - record.unmatched_agent_observation for record in deduplicated - ): - raise HerdrContinuityUnavailableError( - "Herdr agent observation has no authoritative pane owner" - ) - installation_key = None - if any( - _stable_pane_identity(record) is not None - or (require_authenticated_continuity and record.pane_info_observed) - for record in deduplicated - ): - try: - installation_key = load_or_create_installation_key(config.data_dir) - except InstallationKeyError: - if require_authenticated_continuity: - raise - installation_key = None - finalized = [ - _record_with_worker( - record, - _worker_with_stable_key(config, record, installation_key), - ) - for record in deduplicated - ] - workers = [record.worker for record in finalized] - bindings = [ - binding - for record in finalized - if (binding := _binding_from_worker_record(config, record, observed_at)) is not None - ] - return workers, separate_duplicate_worker_bindings(bindings) - - -def bindings_from_workers( - config: Config, - workers: Sequence[Worker], - *, - observed_at: str | None = None, -) -> list[WorkerBinding]: - """Build private Herdr bindings from in-memory workers when raw records are absent.""" - timestamp = observed_at or utc_timestamp() - workers = _mark_backend_sendability(list(workers)) - bindings: list[WorkerBinding] = [] - for worker in workers: - target = worker.backend_target - if not isinstance(target, Mapping): - continue - target_kind = str(target.get("kind") or "") - target_value = str(target.get("value") or "") - if not target_kind or not target_value: - continue - private_fingerprint = worker_binding_private_fingerprint( - host_id=config.host_id, - backend=_BACKEND_NAME, - identity_material={ - "worker_id": worker.id, - "worker_fingerprint": worker.fingerprint, - "target_kind": target_kind, - "target_value": target_value, - }, - ) - record = _WorkerRecord(worker=worker, private_fingerprint=private_fingerprint) - binding = _binding_from_worker_record(config, record, timestamp) - if binding is not None: - bindings.append(binding) - return separate_duplicate_worker_bindings(bindings) - - -def _binding_for_worker(worker: Worker, bindings: Sequence[WorkerBinding]) -> WorkerBinding | None: - candidates = [binding for binding in bindings if binding.worker_id == worker.id] - if not candidates: - return None - exact = [binding for binding in candidates if binding.worker_fingerprint == worker.fingerprint] - if len(exact) == 1: - return exact[0] - if exact: - return None - if len(candidates) == 1: - return candidates[0] - return None - - -def rehydrate_workers_from_bindings( - workers: Sequence[Worker], - current_bindings: Sequence[WorkerBinding] | None = None, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[Worker]: - """Attach private backend targets to workers from current, then stored bindings.""" - current = list(current_bindings or []) - stored = list(stored_bindings or []) - rehydrated: list[Worker] = [] - for worker in workers: - binding = _binding_for_worker(worker, current) - if binding is not None: - rehydrated.append(_worker_with_backend_target(worker, binding.backend_target())) - continue - if isinstance(worker.backend_target, Mapping): - rehydrated.append(worker) - continue - binding = _binding_for_worker(worker, stored) - if binding is None: - rehydrated.append(worker) - else: - rehydrated.append(_worker_with_backend_target(worker, binding.backend_target())) - return rehydrated - - -def _workers_from_payload( - payload: Any, - config: Config | None = None, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[Worker]: - """Extract neutral Worker objects from a herdr agent-list payload.""" - records: list[_WorkerRecord] = [] - for item in _payload_items(payload, ("agents", "workers", "data", "items", "results", "result")): - records.append(_agent_observation_record(item, config)) - return _deduplicate_worker_records(records, stored_bindings) - - -def _workers_and_bindings_from_payload( - payload: Any, - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> tuple[list[Worker], list[WorkerBinding]]: - records: list[_WorkerRecord] = [] - for item in _payload_items(payload, ("agents", "workers", "data", "items", "results", "result")): - records.append(_agent_observation_record(item, config)) - return _workers_and_bindings_from_records(config, records, stored_bindings=stored_bindings) - - -def _pane_has_agent(item: Mapping[str, Any]) -> bool: - """Return True when a pane record carries an agent or explicit agent marker.""" - if _value_for_key(item, "agent") is not None: - return True - if _value_for_key(item, "agent_session") is not None: - return True - if _nested_text(item, "agent_session", "value"): - return True - markers = _value_for_key(item, "state_labels") or _value_for_key(item, "labels") or [] - if isinstance(markers, list): - for marker in markers: - if isinstance(marker, str) and "agent" in marker.lower(): - return True - return False - - -def _record_match_keys( - item: Mapping[str, Any], - record: _WorkerRecord, - *, - pane_shaped: bool = False, -) -> list[tuple[str, str]]: - keys: list[tuple[str, str]] = [] - pane_id_keys = ("pane_id", "paneId", "id") if pane_shaped else ("pane_id", "paneId") - agent_session = ( - _nested_text(item, "agent_session", "value") - or _first_text(item, ("session_id", "sessionId")) - ) - for kind, value in ( - ("pane_id", _first_text(item, pane_id_keys)), - ("terminal_id", _first_text(item, ("terminal_id", "terminalId"))), - ("agent_session", agent_session), - ("private_fingerprint", str(record.private_fingerprint)), - ): - if value and (kind, value) not in keys: - keys.append((kind, value)) - return keys - - -def _backend_target_present(target: Any) -> bool: - return ( - isinstance(target, Mapping) - and bool(str(target.get("kind") or "")) - and bool(str(target.get("value") or "")) - ) - - -def _compatible_backend_target( - agent_record: _WorkerRecord, - pane_record: _WorkerRecord, -) -> dict[str, Any] | None: - """Retain only verified agent-scoped targets; PaneInfo owns pane targets.""" - agent_target = agent_record.worker.backend_target - pane_target = pane_record.worker.backend_target - if not _backend_target_present(agent_target): - return pane_target if _backend_target_present(pane_target) else None - - kind = str(agent_target.get("kind") or "") - value = str(agent_target.get("value") or "") - if ( - kind == "agent_id" - and _backend_target_present(pane_target) - and str(pane_target.get("kind") or "") == "agent_id" - and str(pane_target.get("value") or "") != value - ): - return pane_target - if kind in _AGENT_SCOPED_BACKEND_TARGET_KINDS and ( - kind != "agent" or value == pane_record.worker.name - ): - return agent_target - return pane_target if _backend_target_present(pane_target) else None - - -def _compatible_turn_target( - agent_record: _WorkerRecord, - pane_record: _WorkerRecord, -) -> tuple[str | None, str | None]: - """Prefer PaneInfo targets unless a compatible session target is more precise.""" - agent_kind = agent_record.turn_target_kind - agent_value = agent_record.turn_target_value - pane_kind = pane_record.turn_target_kind - pane_value = pane_record.turn_target_value - if agent_kind in _SESSION_SCOPED_TURN_TARGET_KINDS and agent_value: - pane_session_id = pane_record.agent_session_id - if pane_session_id and pane_session_id != agent_value: - if pane_kind and pane_value: - return pane_kind, pane_value - return None, None - if pane_kind in _SESSION_SCOPED_TURN_TARGET_KINDS and pane_value: - return pane_kind, pane_value - return agent_kind, agent_value - if pane_kind and pane_value: - return pane_kind, pane_value - return None, None - - -def _ambiguous_agent_record( - record: _WorkerRecord, - *, - config: Config | None = None, - observation: Mapping[str, Any] | None = None, -) -> _WorkerRecord: - """Fail closed and retain a unique auditable row for ambiguous ownership.""" - worker = record.worker - target = worker.backend_target - if _backend_target_present(target): - worker = _worker_with_backend_target( - worker, - _private_backend_target( - str(target.get("kind") or ""), - str(target.get("value") or ""), - sendable=False, - reason="ambiguous_pane_match", - ), - ) - private_fingerprint = record.private_fingerprint - if config is not None and observation is not None: - private_fingerprint = worker_binding_private_fingerprint( - host_id=config.host_id, - backend=_BACKEND_NAME, - identity_material={ - "ambiguous_pane_ownership": dict(observation), - "source_private_fingerprint": record.private_fingerprint, - }, - ) - return _WorkerRecord( - worker=worker, - private_fingerprint=private_fingerprint, - workspace_id=record.workspace_id, - pane_id=record.pane_id, - observed_workspace_id=record.observed_workspace_id, - observed_pane_id=record.observed_pane_id, - identity_source=record.identity_source, - terminal_id=record.terminal_id, - agent_session_id=record.agent_session_id, - pane_info_observed=False, - ) - - -def _merge_agent_pane_record( - agent_record: _WorkerRecord, - pane_record: _WorkerRecord | None, -) -> _WorkerRecord: - if pane_record is None: - return agent_record - pane_meta = pane_record.worker.meta - merged_meta = dict(agent_record.worker.meta) - label = pane_meta.get("label") - if isinstance(label, str) and label.strip(): - merged_meta["label"] = label - for key in ("foreground_cwd", "cwd"): - value = pane_meta.get(key) - if isinstance(value, str) and value.strip() and not merged_meta.get(key): - merged_meta[key] = value - - backend_target = _compatible_backend_target(agent_record, pane_record) - worker = agent_record.worker - pane_space_id = pane_record.worker.space_id - if ( - merged_meta != worker.meta - or backend_target != worker.backend_target - or pane_space_id != worker.space_id - ): - worker = Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=pane_space_id, - meta=merged_meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - backend_target=backend_target, - ) - - turn_target_kind, turn_target_value = _compatible_turn_target( - agent_record, - pane_record, - ) - - workspace_id = pane_record.workspace_id - pane_id = pane_record.pane_id - terminal_id = pane_record.terminal_id - if ( - worker == agent_record.worker - and turn_target_kind == agent_record.turn_target_kind - and turn_target_value == agent_record.turn_target_value - and workspace_id == agent_record.workspace_id - and pane_id == agent_record.pane_id - and terminal_id == agent_record.terminal_id - and agent_record.pane_info_observed - ): - return agent_record - return _WorkerRecord( - worker=worker, - private_fingerprint=agent_record.private_fingerprint, - turn_target_kind=turn_target_kind, - turn_target_value=turn_target_value, - workspace_id=workspace_id, - pane_id=pane_id, - observed_workspace_id=pane_record.observed_workspace_id, - observed_pane_id=pane_record.observed_pane_id, - identity_source=pane_record.identity_source, - terminal_id=terminal_id, - agent_session_id=pane_record.agent_session_id, - pane_info_observed=True, - ) - - -def _collapse_exact_observation_items( - items: Sequence[Mapping[str, Any]], -) -> list[dict[str, Any]]: - """Collapse byte-equivalent JSON observations before ownership cardinality.""" - seen: set[str] = set() - unique: list[dict[str, Any]] = [] - for item in items: - signature = json.dumps( - item, - sort_keys=True, - separators=(",", ":"), - default=str, - ) - if signature in seen: - continue - seen.add(signature) - unique.append(dict(item)) - return unique - - -def _record_ownership_keys( - item: Mapping[str, Any], - record: _WorkerRecord, - *, - pane_shaped: bool = False, -) -> set[tuple[str, str]]: - keys = set( - _record_match_keys( - item, - record, - pane_shaped=pane_shaped, - ) - ) - backend_target = record.worker.backend_target - if _backend_target_present(backend_target): - keys.add( - ( - f"backend:{backend_target.get('kind')}", - str(backend_target.get("value") or ""), - ) - ) - keys.add( - ( - "backend_send_token", - str(backend_target.get("value") or ""), - ) - ) - if record.turn_target_kind and record.turn_target_value: - keys.add( - ( - f"turn:{record.turn_target_kind}", - record.turn_target_value, - ) - ) - return keys - - -def _pane_ownership_graph( - agent_items: Sequence[Mapping[str, Any]], - agent_records: Sequence[_WorkerRecord], - pane_items: Sequence[Mapping[str, Any]], - pane_records: Sequence[_WorkerRecord], -) -> tuple[list[set[int]], set[int], set[int]]: - """Return agent-to-pane edges and every ambiguous ownership component.""" - agent_nodes = [("agent", index) for index in range(len(agent_records))] - pane_nodes = [("pane", index) for index in range(len(pane_records))] - adjacency: dict[tuple[str, int], set[tuple[str, int]]] = { - node: set() for node in [*agent_nodes, *pane_nodes] - } - ambiguous_seeds: set[tuple[str, int]] = set() - - pane_indices_by_match: dict[tuple[str, str], set[int]] = {} - pane_indices_by_owner: dict[tuple[str, str], set[int]] = {} - for pane_index, (item, record) in enumerate( - zip(pane_items, pane_records, strict=True) - ): - match_keys = _record_match_keys(item, record, pane_shaped=True) - for key in match_keys: - pane_indices_by_match.setdefault(key, set()).add(pane_index) - for key in _record_ownership_keys(item, record, pane_shaped=True): - pane_indices_by_owner.setdefault(key, set()).add(pane_index) - - # Each private pane, terminal, session, or agent identity may own only one - # PaneInfo row. Exact duplicate rows were already collapsed above. - for pane_indices in pane_indices_by_owner.values(): - if len(pane_indices) < 2: - continue - ordered = sorted(pane_indices) - first_node = ("pane", ordered[0]) - ambiguous_seeds.update(("pane", index) for index in ordered) - for pane_index in ordered[1:]: - pane_node = ("pane", pane_index) - adjacency[first_node].add(pane_node) - adjacency[pane_node].add(first_node) - - agent_matches: list[set[int]] = [] - pane_claimants: dict[int, set[int]] = {} - agent_indices_by_owner: dict[tuple[str, str], set[int]] = {} - for agent_index, (item, record) in enumerate( - zip(agent_items, agent_records, strict=True) - ): - matches: set[int] = set() - for key in _record_match_keys(item, record): - matches.update(pane_indices_by_match.get(key, ())) - agent_matches.append(matches) - agent_node = ("agent", agent_index) - owner_keys = _record_ownership_keys(item, record) - for key in owner_keys: - agent_indices_by_owner.setdefault(key, set()).add(agent_index) - if key[0] != "backend_send_token": - continue - for pane_index in pane_indices_by_owner.get(key, ()): - if pane_index in matches: - continue - pane_node = ("pane", pane_index) - adjacency[agent_node].add(pane_node) - adjacency[pane_node].add(agent_node) - ambiguous_seeds.add(agent_node) - ambiguous_seeds.add(pane_node) - for pane_index in matches: - pane_node = ("pane", pane_index) - adjacency[agent_node].add(pane_node) - adjacency[pane_node].add(agent_node) - pane_claimants.setdefault(pane_index, set()).add(agent_index) - if len(matches) > 1: - ambiguous_seeds.add(agent_node) - ambiguous_seeds.update(("pane", index) for index in matches) - - for agent_indices in agent_indices_by_owner.values(): - if len(agent_indices) < 2: - continue - if not any(agent_matches[index] for index in agent_indices): - continue - ordered = sorted(agent_indices) - first_node = ("agent", ordered[0]) - ambiguous_seeds.update(("agent", index) for index in ordered) - for agent_index in ordered[1:]: - agent_node = ("agent", agent_index) - adjacency[first_node].add(agent_node) - adjacency[agent_node].add(first_node) - - for pane_index, claimants in pane_claimants.items(): - if len(claimants) < 2: - continue - ambiguous_seeds.add(("pane", pane_index)) - ambiguous_seeds.update(("agent", index) for index in claimants) - - ambiguous_nodes = set(ambiguous_seeds) - pending = list(ambiguous_seeds) - while pending: - node = pending.pop() - for adjacent in adjacency[node]: - if adjacent in ambiguous_nodes: - continue - ambiguous_nodes.add(adjacent) - pending.append(adjacent) - - return ( - agent_matches, - {index for kind, index in ambiguous_nodes if kind == "agent"}, - {index for kind, index in ambiguous_nodes if kind == "pane"}, - ) - - -def _records_from_agent_and_pane_payloads( - config: Config | None, - agent_payload: Any, - pane_payload: Any, - *, - include_unmatched_panes: bool = True, -) -> list[_WorkerRecord]: - """Merge PaneInfo identity only across a one-to-one ownership graph.""" - pane_items = _collapse_exact_observation_items( - [ - item - for item in _payload_items( - pane_payload, - ("panes", "items", "data", "results", "result"), - ) - if _pane_has_agent(item) - ] - ) - pane_records = [ - _worker_record_from_item( - item, - config, - pane_info_observed=True, - identity_source="pane.list", - ) - for item in pane_items - ] - agent_items = _collapse_exact_observation_items( - _payload_items( - agent_payload, - ("agents", "workers", "data", "items", "results", "result"), - ) - ) - agent_records = [ - _agent_observation_record(item, config) - for item in agent_items - ] - agent_matches, ambiguous_agents, ambiguous_panes = _pane_ownership_graph( - agent_items, - agent_records, - pane_items, - pane_records, - ) - - records: list[_WorkerRecord] = [] - consumed_pane_indices: set[int] = set() - for agent_index, record in enumerate(agent_records): - matched_pane_indices = agent_matches[agent_index] - consumed_pane_indices.update(matched_pane_indices) - if agent_index in ambiguous_agents: - records.append( - _ambiguous_agent_record( - record, - config=config, - observation=agent_items[agent_index], - ) - ) - elif len(matched_pane_indices) == 1: - matched_pane_index = next(iter(matched_pane_indices)) - records.append( - _merge_agent_pane_record(record, pane_records[matched_pane_index]) - ) - else: - records.append(record) - if include_unmatched_panes: - records.extend( - _ambiguous_agent_record( - record, - config=config, - observation=pane_items[index], - ) - if index in ambiguous_panes - else record - for index, record in enumerate(pane_records) - if index not in consumed_pane_indices - ) - return records - - -def _workers_from_pane_payload( - payload: Any, - config: Config | None = None, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> list[Worker]: - """Extract pane workers through the ownership graph.""" - records = _records_from_agent_and_pane_payloads( - config, - None, - payload, - ) - return _deduplicate_worker_records(records, stored_bindings) - - -def _workers_and_bindings_from_pane_payload( - payload: Any, - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> tuple[list[Worker], list[WorkerBinding]]: - records = _records_from_agent_and_pane_payloads( - config, - None, - payload, - ) - return _workers_and_bindings_from_records( - config, - records, - stored_bindings=stored_bindings, - ) - - -def _probe_payload_variants( - variants: Sequence[Sequence[str]], - config: Config, - budget: _ProbeBudget | None = None, -) -> tuple[str, Any]: - outcomes: list[str] = [] - for args in variants: - if budget is None: - outcome, payload = _probe_herdr(args, config) - else: - try: - outcome, payload = _probe_herdr(args, config, budget) - except TypeError: - outcome, payload = _probe_herdr(args, config) - if outcome == "ok": - return outcome, payload - if outcome in _DEADLINE_EXHAUSTED_OUTCOMES: - return outcome, None - outcomes.append(outcome) - if outcomes and all(outcome == "launch_error" for outcome in outcomes): - return "launch_error", None - if "malformed_json" in outcomes: - return "malformed_json", None - if "nonzero" in outcomes: - return "nonzero", None - return outcomes[-1] if outcomes else "nonzero", None - - -def _degraded_observation( - outcome: str, - message: str, - *, - spaces: Sequence[Space] | None = None, - workers: Sequence[Worker] | None = None, -) -> HerdrCommandObservation: - observed_spaces = list(spaces or []) - observed_workers = list(workers or []) - health = herdr_backend_health( - outcome, - message=message, - spaces=observed_spaces, - workers=observed_workers, - ) - return HerdrCommandObservation( - spaces=observed_spaces, - workers=observed_workers, - status=health.status, - outcome=health.outcome, - message=health.message, - backend_health=[health], - ) - - -def _snapshot_observation( - spaces: list[Space], - workers: list[Worker], - bindings: list[WorkerBinding], - outcome: str, - *, - message: str | None = None, -) -> HerdrSnapshotObservation: - health = herdr_backend_health( - outcome, - message=message, - spaces=spaces, - workers=workers, - ) - return HerdrSnapshotObservation( - spaces=spaces, - workers=workers, - bindings=bindings, - backend_health=[health], - ) - - -def fetch_herdr_command_observation( - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, -) -> HerdrCommandObservation: - """Return Herdr observations plus health metadata for mutation safety.""" - try: - if shutil.which(config.herdr_bin) is None: - return _degraded_observation("missing_binary", "Herdr binary is unavailable") - except (TypeError, ValueError, OSError): - return _degraded_observation("launch_error", "Herdr binary could not be inspected") - - budget = _ProbeBudget.from_config(config, planned_probes=5) - workspace_outcome, workspace_payload = _probe_payload_variants( - [ - ["workspace", "list"], - ["workspace", "list", "--json"], - ], - config, - budget, - ) - if workspace_outcome != "ok": - return _degraded_observation( - workspace_outcome, - "Herdr workspace observation is not healthy", - ) - - agent_outcome, agent_payload = _probe_payload_variants( - [ - ["agent", "list"], - ["agent", "list", "--json"], - ], - config, - budget, - ) - if agent_outcome != "ok": - return _degraded_observation( - agent_outcome, - "Herdr agent observation is not healthy", - ) - - spaces = _spaces_from_payload(workspace_payload) - try: - pane_outcome, pane_payload = _probe_herdr(["pane", "list"], config, budget) - except TypeError: - pane_outcome, pane_payload = _probe_herdr(["pane", "list"], config) - records = _records_from_agent_and_pane_payloads( - config, - agent_payload, - pane_payload if pane_outcome == "ok" else None, - include_unmatched_panes=not bool( - _payload_items( - agent_payload, - ("agents", "workers", "data", "items", "results", "result"), - ) - ), - ) - try: - workers, bindings = _workers_and_bindings_from_records( - config, - records, - stored_bindings=stored_bindings, - require_authenticated_continuity=pane_outcome == "ok", - ) - except (HerdrContinuityUnavailableError, InstallationKeyError): - return _degraded_observation( - "continuity_unavailable", - _HEALTH_MESSAGES["continuity_unavailable"], - spaces=spaces, - ) - if pane_outcome != "ok": - return _degraded_observation( - pane_outcome, - "Herdr pane continuity observation is not healthy", - spaces=spaces, - workers=workers, - ) - - return HerdrCommandObservation( - spaces=spaces, - workers=workers, - status="healthy", - outcome="healthy_non_empty" if spaces or workers else "empty_healthy", - bindings=bindings, - backend_health=[ - herdr_backend_health( - "healthy_non_empty" if spaces or workers else "empty_healthy", - spaces=spaces, - workers=workers, - ) - ], - ) - - -def _state_result( - spaces: list[Space], - workers: list[Worker], - bindings: list[WorkerBinding], - include_bindings: bool, -) -> tuple[list[Space], list[Worker]] | tuple[list[Space], list[Worker], list[WorkerBinding]]: - if include_bindings: - return spaces, workers, bindings - return spaces, workers - - -def fetch_herdr_snapshot_observation( - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, - *, - require_authenticated_continuity: bool = True, -) -> HerdrSnapshotObservation: - """Return Herdr snapshot observations plus public backend health.""" - try: - if shutil.which(config.herdr_bin) is None: - return _snapshot_observation( - [], - [], - [], - "missing_binary", - message=_HEALTH_MESSAGES["missing_binary"], - ) - except (TypeError, ValueError, OSError): - return _snapshot_observation( - [], - [], - [], - "launch_error", - message="Herdr binary could not be inspected", - ) - - budget = _ProbeBudget.from_config(config, planned_probes=5) - workspace_outcome, workspace_payload = _probe_payload_variants( - [ - ["workspace", "list"], - ["workspace", "list", "--json"], - ], - config, - budget, - ) - if workspace_outcome in _DEADLINE_EXHAUSTED_OUTCOMES: - return _snapshot_observation( - [], - [], - [], - workspace_outcome, - message=_HEALTH_MESSAGES[workspace_outcome], - ) - - agent_outcome, agent_payload = _probe_payload_variants( - [ - ["agent", "list"], - ["agent", "list", "--json"], - ], - config, - budget, - ) - - spaces = _spaces_from_payload(workspace_payload) - if agent_outcome in _DEADLINE_EXHAUSTED_OUTCOMES: - return _snapshot_observation( - spaces, - [], - [], - agent_outcome, - message=_HEALTH_MESSAGES[agent_outcome], - ) - - try: - pane_outcome, pane_payload = _probe_herdr(["pane", "list"], config, budget) - except TypeError: - pane_outcome, pane_payload = _probe_herdr(["pane", "list"], config) - records = _records_from_agent_and_pane_payloads( - config, - agent_payload, - pane_payload if pane_outcome == "ok" else None, - include_unmatched_panes=not bool( - _payload_items( - agent_payload, - ("agents", "workers", "data", "items", "results", "result"), - ) - ), - ) - try: - workers, bindings = _workers_and_bindings_from_records( - config, - records, - stored_bindings=stored_bindings, - require_authenticated_continuity=( - require_authenticated_continuity and pane_outcome == "ok" - ), - ) - except (HerdrContinuityUnavailableError, InstallationKeyError): - return _snapshot_observation( - spaces, - [], - [], - "continuity_unavailable", - message=_HEALTH_MESSAGES["continuity_unavailable"], - ) - - failed_outcomes = [ - outcome - for outcome in (workspace_outcome, agent_outcome, pane_outcome) - if outcome not in {"ok"} - ] - if failed_outcomes: - outcome = failed_outcomes[0] - return _snapshot_observation( - spaces, - workers, - bindings, - outcome, - message=_HEALTH_MESSAGES.get(outcome, _HEALTH_MESSAGES["unknown"]), - ) - - return _snapshot_observation( - spaces, - workers, - bindings, - "healthy_non_empty" if spaces or workers else "empty_healthy", - ) - - -def fetch_herdr_state( - config: Config, - stored_bindings: Sequence[WorkerBinding] | None = None, - *, - include_bindings: bool = False, -) -> tuple[list[Space], list[Worker]] | tuple[list[Space], list[Worker], list[WorkerBinding]]: - """Return neutral spaces and workers from the Herdr CLI, or empty lists.""" - observation = fetch_herdr_snapshot_observation( - config, - stored_bindings=stored_bindings, - require_authenticated_continuity=False, - ) - return _state_result(observation.spaces, observation.workers, observation.bindings, include_bindings) diff --git a/src/tendwire/backends/herdr_command.py b/src/tendwire/backends/herdr_command.py deleted file mode 100644 index 3fb6d46..0000000 --- a/src/tendwire/backends/herdr_command.py +++ /dev/null @@ -1,156 +0,0 @@ -"""Narrow mutating command adapter for Herdr. - -Only the high-level ``herdr agent send `` API is used here. -This module must not fall back to pane control, key sending, shell commands, -PTY control, signals, paste buffers, raw argv, or client-provided backend -parameters. -""" - -from __future__ import annotations - -from dataclasses import dataclass - -import shutil -import subprocess -from typing import Any - -from ..config import Config -from ..core.commands import ( - STATUS_ACCEPTED, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_FAILED, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_REQUEST_STATE_UNCERTAIN, - error_value, - sanitize_command_result, -) -from ..core.models import _string_value - -_BACKEND_TARGET_KINDS = frozenset( - {"agent_id", "terminal_id", "pane_id", "agent", "name", "label"} -) - - -@dataclass(frozen=True) -class _HerdrCommandResult: - """Internal backend outcome, never a public command receipt envelope.""" - - ok: bool - status: str - result: dict[str, Any] | None = None - error: dict[str, Any] | None = None - - def to_dict(self) -> dict[str, Any]: - return { - "ok": self.ok, - "status": self.status, - "result": sanitize_command_result(self.result), - "error": sanitize_command_result(self.error), - } - - -def _run_agent_send( - config: Config, - target_value: str, - instruction_text: str, -) -> subprocess.CompletedProcess[str]: - """Run the single allowed Herdr send surface with an argv list.""" - return subprocess.run( - [config.herdr_bin, "agent", "send", target_value, instruction_text], - capture_output=True, - text=True, - check=False, - timeout=config.herdr_timeout_seconds, - ) - - -def _backend_error( - status: str, - message: str, - details: dict[str, Any] | None = None, -) -> _HerdrCommandResult: - return _HerdrCommandResult( - ok=False, - status=status, - error=error_value(status, message, details=details), - ) - - -def send_instruction( - config: Config, - target: dict[str, Any], - instruction: dict[str, Any], -) -> _HerdrCommandResult: - """Send instruction text to the backend-resolved private Herdr target.""" - backend_target = target.get("backend_target") - target_value = "" - target_kind = "" - target_reason = "" - if isinstance(backend_target, dict): - target_value = _string_value(backend_target.get("value")) - target_kind = _string_value(backend_target.get("kind")) - target_reason = _string_value(backend_target.get("reason")) - public_worker_id = _string_value(target.get("worker_id")) - instruction_text = instruction.get("text") - - if not isinstance(instruction_text, str) or not instruction_text: - return _backend_error( - STATUS_BACKEND_FAILED, - "instruction text is missing after validation", - ) - - try: - if shutil.which(config.herdr_bin) is None: - return _backend_error( - STATUS_BACKEND_UNAVAILABLE, - "Herdr binary is unavailable", - ) - except (OSError, TypeError, ValueError): - return _backend_error( - STATUS_BACKEND_UNAVAILABLE, - "Herdr binary is unavailable", - ) - - if ( - not isinstance(backend_target, dict) - or backend_target.get("sendable") is not True - or target_kind not in _BACKEND_TARGET_KINDS - or not target_value - ): - if target_reason in {"duplicate_backend_target", "not_unique"}: - return _backend_error( - STATUS_AMBIGUOUS_BACKEND_TARGET, - "resolved target is ambiguous for backend send", - ) - return _backend_error( - STATUS_BACKEND_UNSUPPORTED, - "resolved target has no backend-owned sendable target", - ) - - try: - completed = _run_agent_send(config, target_value, instruction_text) - except subprocess.TimeoutExpired: - return _backend_error( - STATUS_REQUEST_STATE_UNCERTAIN, - "Herdr agent send timed out after starting", - details={"timeout_seconds": config.herdr_timeout_seconds}, - ) - except (OSError, UnicodeDecodeError, ValueError, TypeError): - return _backend_error( - STATUS_BACKEND_UNAVAILABLE, - "Herdr agent send could not be launched", - ) - - if completed.returncode == 0: - return _HerdrCommandResult( - ok=True, - status=STATUS_ACCEPTED, - result={"target": {"worker_id": public_worker_id}}, - ) - - return _backend_error( - STATUS_BACKEND_FAILED, - "Herdr agent send exited non-zero", - details={"exit_code": int(completed.returncode)}, - ) diff --git a/src/tendwire/backends/herdr_decision.py b/src/tendwire/backends/herdr_decision.py deleted file mode 100644 index c6199bd..0000000 --- a/src/tendwire/backends/herdr_decision.py +++ /dev/null @@ -1,124 +0,0 @@ -"""Translate semantic Claude decisions into private Herdr pane input. - -Calibration assumptions are deliberately confined to this backend module, and -every mapping below was LIVE-VERIFIED against Claude Code 2.1.211 on a real -pane (2026-07-16): - -* Single-choice and plan rows carry 1-based decimal shortcuts, and typing the - ordinal alone SELECTS AND SUBMITS the row instantly — no Enter follows. A - trailing Enter would leak into whatever UI appears next, so none is sent. -* The single-choice write-in row ("Type something", at position N + 1) does - NOT respond to its digit. It is reached by pressing Down exactly N times - from the initial cursor on row 1; the focused row is itself a text input, - so Tendwire then sends the write-in prose and submits with Enter. -* Multi-select digits toggle their row ABSOLUTELY without moving the cursor, - Right switches to the Submit tab (a review screen whose default focus is - "Submit answers"), and Enter there submits the selection set. -* Every driven ordinal must stay a single keystroke, so decisions expose at - most 9 real options (PENDING_DECISION_MAX_OPTIONS in herdr_turns). -* Herdr's private ``pane.send_keys`` accepts decimal character keys plus - ``Down``, ``Up``, ``Right``, and ``Enter``; write-in prose uses - ``pane.send_input`` so Herdr owns terminal text encoding and appends the - final Enter atomically. - -These steps are internal calibration data. They are never accepted from a -connector and there is intentionally no public raw-key command action. -""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Literal - - -MULTI_SELECT_CALIBRATION = { - "submit_tab": "Right", - "submit": "Enter", -} - - -@dataclass(frozen=True) -class HerdrDecisionStep: - """One private, already-calibrated Herdr pane operation.""" - - operation: Literal["keys", "text", "input"] - keys: tuple[str, ...] = () - text: str | None = None - - def __post_init__(self) -> None: - if self.operation == "keys": - if not self.keys or self.text is not None: - raise ValueError("key calibration step requires only keys") - elif self.operation == "text": - raise ValueError("text calibration steps are no longer produced") - elif self.operation == "input": - if not isinstance(self.text, str) or not self.text or self.keys != ("Enter",): - raise ValueError("input calibration step requires text plus Enter") - else: - raise ValueError("unsupported decision calibration operation") - - -def _digit_keys(value: int | str) -> tuple[str, ...]: - text = str(value) - if not text.isdigit() or int(text) < 1: - raise ValueError("decision ordinal must be a positive decimal") - return tuple(text) - - -def calibrate_decision_steps( - *, - kind: Literal["single", "multi", "plan"], - option_count: int, - option_refs: tuple[str, ...] = (), - text: str | None = None, -) -> tuple[HerdrDecisionStep, ...]: - """Return private pane operations for one validated semantic selection.""" - if ( - kind not in {"single", "multi", "plan"} - or not isinstance(option_count, int) - or isinstance(option_count, bool) - or option_count < 1 - ): - raise ValueError("invalid decision calibration context") - if text is not None: - if kind != "single" or option_refs or not isinstance(text, str) or not text: - raise ValueError("invalid decision write-in calibration") - # The write-in row ignores digits; reach it with Down x N from row 1, - # where the focused row is itself the text input. - return ( - HerdrDecisionStep("keys", keys=("Down",) * option_count), - HerdrDecisionStep("input", keys=("Enter",), text=text), - ) - if not option_refs or len(option_refs) != len(set(option_refs)): - raise ValueError("decision option refs must be nonempty and unique") - ordinals: list[int] = [] - for ref in option_refs: - if not isinstance(ref, str) or not ref.isdigit(): - raise ValueError("decision option ref must be a decimal ordinal") - ordinal = int(ref) - if not 1 <= ordinal <= option_count: - raise ValueError("decision option ref is out of range") - ordinals.append(ordinal) - if kind in {"single", "plan"}: - if len(ordinals) != 1: - raise ValueError("single and plan decisions require one option") - # The digit alone selects AND submits; a trailing Enter would leak into - # the next UI (composer, or worse, a modal). - return (HerdrDecisionStep("keys", keys=_digit_keys(ordinals[0])),) - - # Digits toggle rows absolutely (cursor-independent); Right reaches the - # Submit tab whose default focus is "Submit answers"; Enter submits. - steps: list[HerdrDecisionStep] = [ - HerdrDecisionStep("keys", keys=_digit_keys(ordinal)) - for ordinal in sorted(ordinals) - ] - steps.append( - HerdrDecisionStep( - "keys", - keys=( - MULTI_SELECT_CALIBRATION["submit_tab"], - MULTI_SELECT_CALIBRATION["submit"], - ), - ) - ) - return tuple(steps) diff --git a/src/tendwire/backends/herdr_events.py b/src/tendwire/backends/herdr_events.py deleted file mode 100644 index b207a18..0000000 --- a/src/tendwire/backends/herdr_events.py +++ /dev/null @@ -1,3077 +0,0 @@ -"""Opt-in Herdr socket event backend and reconciliation layer. - -This module is intentionally imported only from the explicit socket backend -path. It reuses the PR8 socket client for transport and the Herdr CLI adapter's -projection helpers for Tendwire model normalization. -""" - -from __future__ import annotations - -import inspect -import logging -import threading -import time -from collections import OrderedDict -from collections.abc import Callable, Iterable, Mapping, Sequence -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -from ..config import Config -from ..core.models import ( - BackendHealth, - Snapshot, - Space, - Worker, - WorkerBinding, - normalize_status, - utc_timestamp, -) -from ..worker_identity import ( - InstallationKeyError, - STABLE_KEY_VERSION, - canonical_herdr_pane_identity, - is_stable_worker_key, -) -from ..core.projector import project_from_observations -from ..store.sqlite import ( - HerdrTurnWatermark, - SnapshotObservationContext, - SnapshotRetentionPolicy, - expire_stale_worker_bindings, - expire_worker_bindings, - get_herdr_turn_watermark, - latest_snapshot, - list_worker_bindings, - maybe_run_automatic_store_maintenance, - record_herdr_turn_completeness_break, - record_herdr_turn_completion, - save_snapshot, - set_herdr_turn_watermark, - upsert_worker_bindings, -) -from .herdr_cli import ( - HerdrContinuityUnavailableError, - _pane_has_agent, - _payload_items, - _spaces_from_payload, - _records_from_agent_and_pane_payloads, - _worker_record_from_item, - _workers_and_bindings_from_records, - _strip_stable_key_fields, - herdr_backend_health, -) -from .herdr_protocol import ( - HERDR_EVENTS_SUBSCRIBE_METHOD, - HERDR_OFFICIAL_EVENT_NAMES, - HERDR_TURN_COMPLETED_EVENT_NAME, - HerdrEnvelopeError, - HerdrErrorResponse, - HerdrMalformedLineError, - HerdrProtocolError, - build_events_subscribe_params, -) -from .herdr_socket import ( - HerdrSocketClient, - HerdrSocketConnectionError, - HerdrSocketDisconnectedError, - HerdrSocketTimeoutError, -) - - -BACKEND_NAME = "herdr" -DEFAULT_SUBSCRIBE_METHOD = HERDR_EVENTS_SUBSCRIBE_METHOD -DEFAULT_DEBOUNCE_SECONDS = 0.05 -DEFAULT_DEDUPE_SIZE = 512 -DEFAULT_MAX_BATCH_SIZE = 64 -DEFAULT_RECONNECT_DELAY_SECONDS = 0.25 -_LOGGER = logging.getLogger(__name__) - -_AGENT_PAYLOAD_KEYS = ("agents", "workers", "data", "items", "results", "result") -_PANE_PAYLOAD_KEYS = ("panes", "items", "data", "results", "result") -_SUPPORTED_EVENT_NAMES = ( - *HERDR_OFFICIAL_EVENT_NAMES, - HERDR_TURN_COMPLETED_EVENT_NAME, -) -_SUPPORTED_EVENT_NAME_SET = frozenset(_SUPPORTED_EVENT_NAMES) -_HERDR_074_EVENT_NAMES = tuple( - event_name - for event_name in _SUPPORTED_EVENT_NAMES - if event_name not in {"pane.updated", "pane.turn_completed"} -) -_HERDR_074_PANE_SCOPED_REPLAY_EVENT_NAMES = frozenset( - { - "workspace.focused", - "pane.focused", - "pane.agent_detected", - "pane.output_matched", - } -) -_HERDR_074_PANE_SCOPED_FALLBACK_EVENT_NAMES = tuple( - event_name - for event_name in _HERDR_074_EVENT_NAMES - if event_name not in _HERDR_074_PANE_SCOPED_REPLAY_EVENT_NAMES -) -_PARAMETERIZED_EVENT_NAMES = frozenset( - { - "pane.agent_status_changed", - "pane.output_matched", - "pane.turn_completed", - } -) -_GLOBAL_EVENT_NAMES = tuple( - event_name - for event_name in _SUPPORTED_EVENT_NAMES - if event_name not in _PARAMETERIZED_EVENT_NAMES -) -_CLOSED_EVENT_NAMES = frozenset({"pane.closed", "pane.exited"}) -_SPACE_EVENT_NAMES = frozenset( - { - "workspace.created", - "workspace.updated", - "workspace.renamed", - "workspace.closed", - "workspace.focused", - } -) -_WORKTREE_EVENT_NAMES = frozenset({"worktree.created", "worktree.opened", "worktree.removed"}) -# ``pane.updated`` is normalized from Herdr 0.7.5's scalar -# ``PaneOutputChanged`` event. It is a turn-refresh notification, not a -# PaneInfo observation, and therefore must never rebuild worker identity. -_PANE_WORKER_EVENT_NAMES = frozenset({"pane.created", "pane.focused"}) -_TURN_REFRESH_EVENT_NAMES = frozenset( - { - "pane.created", - "pane.updated", - "pane.focused", - "pane.moved", - "pane.closed", - "pane.exited", - "pane.agent_detected", - "pane.agent_status_changed", - "pane.output_matched", - } -) -_COMPLETED_TURN_REFRESH_STATUSES = frozenset({"updated", "unchanged", "missing"}) - - -class HerdrEventBackendError(Exception): - """Base error for the opt-in Herdr socket event backend.""" - - -@dataclass(frozen=True) -class HerdrEventBackendHealth: - """Small in-memory health state for the Herdr socket backend.""" - - status: str - outcome: str - observed_at: str - message: str - - def to_backend_health( - self, - *, - spaces: Sequence[Space] | None = None, - workers: Sequence[Worker] | None = None, - ) -> BackendHealth: - return herdr_backend_health( - self.outcome, - observed_at=self.observed_at, - message=self.message, - spaces=spaces or [], - workers=workers or [], - ) - - -@dataclass(frozen=True) -class HerdrEventId: - """Forward-compatible authoritative producer event identifier.""" - - value: str - - -@dataclass(frozen=True) -class HerdrProducerSequence: - """Forward-compatible producer-scoped sequence identifier.""" - - producer_id: str - sequence: str - - -HerdrProducerIdentity = HerdrEventId | HerdrProducerSequence - - -@dataclass(frozen=True) -class NormalizedHerdrEvent: - """A validated Herdr event with optional durable producer identity.""" - - name: str - payload: Mapping[str, Any] - producer_identity: HerdrProducerIdentity | None - - -@dataclass(frozen=True) -class HerdrTurnCompletionRecord: - """Validated completion metadata; semantic content still comes from adapters.""" - - pane_id: str - turn: int - turn_epoch: int - outcome: str - completed_unix_ms: int - message: str | None = None - message_truncated: bool = False - agent_session_path: str | None = None - - -@dataclass(frozen=True) -class HerdrPaneTurnsReplay: - pane_id: str - turn_epoch: int - records: tuple[HerdrTurnCompletionRecord, ...] - truncated: bool - oldest_available: int | None - - @property - def newest_turn(self) -> int: - return max((record.turn for record in self.records), default=0) - - -def _compact_key(value: object) -> str: - return str(value).strip().lower().replace("-", "_").replace(".", "_").replace(":", "_") - - -def _field_value(item: Mapping[str, Any], expected_key: str) -> Any: - expected = _compact_key(expected_key) - for key, value in item.items(): - if _compact_key(key) == expected: - return value - return None - - -def _first_text(item: Mapping[str, Any], keys: Iterable[str]) -> str | None: - for key in keys: - value = _field_value(item, key) - if value is None: - continue - if isinstance(value, Mapping): - nested = _first_text(value, ("id", "value", "name", "label")) - if nested: - return nested - continue - if isinstance(value, (str, int, float, bool)): - text = str(value) - if text: - return text - return None - - -def _safe_mapping(value: Any) -> dict[str, Any]: - return dict(value) if isinstance(value, Mapping) else {} - - -def _call_with_optional_keywords( - callback: Callable[..., Any], - args: tuple[Any, ...], - kwargs: Mapping[str, Any], -) -> Any: - """Invoke once, omitting optional keywords only when the signature requires it.""" - try: - signature = inspect.signature(callback) - except (TypeError, ValueError): - return callback(*args, **dict(kwargs)) - try: - signature.bind(*args, **dict(kwargs)) - except TypeError: - return callback(*args) - return callback(*args, **dict(kwargs)) - - -def _nonnegative_protocol_integer(value: Any, field: str) -> int: - if ( - not isinstance(value, int) - or isinstance(value, bool) - or value < 0 - or value > (1 << 63) - 1 - ): - raise HerdrEnvelopeError(f"invalid Herdr {field}") - return int(value) - - -def _turn_completion_record( - value: Any, - *, - pane_id: str | None = None, -) -> HerdrTurnCompletionRecord: - if not isinstance(value, Mapping): - raise HerdrEnvelopeError("invalid Herdr turn completion record") - pane = value.get("pane") - resolved_pane_id = pane_id or ( - _first_text(pane, ("pane_id", "paneId", "id")) - if isinstance(pane, Mapping) - else None - ) or _first_text(value, ("pane_id", "paneId")) - if not resolved_pane_id: - raise HerdrEnvelopeError("Herdr turn completion is missing pane_id") - outcome = value.get("outcome") - if outcome not in {"completed", "aborted"}: - raise HerdrEnvelopeError("invalid Herdr turn completion outcome") - message = value.get("message") - if message is not None and not isinstance(message, str): - raise HerdrEnvelopeError("invalid Herdr turn completion message") - if isinstance(message, str) and len(message.encode("utf-8")) > 8 * 1024: - raise HerdrEnvelopeError("Herdr turn completion message is too large") - message_truncated = value.get("message_truncated", False) - if not isinstance(message_truncated, bool): - raise HerdrEnvelopeError("invalid Herdr message_truncated") - agent_session_path = value.get("agent_session_path") - if agent_session_path is not None and not isinstance(agent_session_path, str): - raise HerdrEnvelopeError("invalid Herdr agent_session_path") - return HerdrTurnCompletionRecord( - pane_id=str(resolved_pane_id), - turn=_nonnegative_protocol_integer(value.get("turn"), "turn"), - turn_epoch=_nonnegative_protocol_integer( - value.get("turn_epoch"), - "turn_epoch", - ), - outcome=str(outcome), - completed_unix_ms=_nonnegative_protocol_integer( - value.get("completed_unix_ms"), - "completed_unix_ms", - ), - message=message, - message_truncated=message_truncated, - agent_session_path=agent_session_path, - ) - - -def _pane_turns_replay(value: Any, pane_id: str) -> HerdrPaneTurnsReplay: - if not isinstance(value, Mapping): - raise HerdrEnvelopeError("invalid pane.turns response") - turns = value.get("turns") - if isinstance(turns, Mapping): - value = turns - response_pane_id = value.get("pane_id") - if not isinstance(response_pane_id, str) or response_pane_id != pane_id: - raise HerdrEnvelopeError("pane.turns returned the wrong pane") - epoch = _nonnegative_protocol_integer(value.get("turn_epoch"), "turn_epoch") - raw_records = value.get("records") - if not isinstance(raw_records, list): - raise HerdrEnvelopeError("pane.turns records must be an array") - records = tuple( - _turn_completion_record(record, pane_id=pane_id) - for record in raw_records - ) - if any(record.turn_epoch != epoch for record in records): - raise HerdrEnvelopeError("pane.turns mixed turn epochs") - if tuple(record.turn for record in records) != tuple( - sorted({record.turn for record in records}) - ): - raise HerdrEnvelopeError("pane.turns records are not strictly ordered") - truncated = value.get("truncated", False) - if not isinstance(truncated, bool): - raise HerdrEnvelopeError("invalid pane.turns truncated marker") - raw_oldest = value.get("oldest_available") - oldest = ( - None - if raw_oldest is None - else _nonnegative_protocol_integer(raw_oldest, "oldest_available") - ) - return HerdrPaneTurnsReplay( - pane_id=pane_id, - turn_epoch=epoch, - records=records, - truncated=truncated, - oldest_available=oldest, - ) - - -def _entity_payload_with_source(payload: Mapping[str, Any], *entity_names: str) -> tuple[dict[str, Any], str | None]: - """Return an event entity object plus the nested entity name selected.""" - merged: dict[str, Any] = {} - nested_entity_keys: set[str] = set() - selected_entity: str | None = None - for entity_name in entity_names: - nested = _field_value(payload, entity_name) - if isinstance(nested, Mapping): - merged.update(dict(nested)) - compact_name = _compact_key(entity_name) - nested_entity_keys.add(compact_name) - selected_entity = compact_name - break - for key, value in payload.items(): - if _compact_key(key) in nested_entity_keys: - continue - merged.setdefault(str(key), value) - return merged, selected_entity - - -def _entity_payload(payload: Mapping[str, Any], *entity_names: str) -> dict[str, Any]: - """Return a single object for an event entity while preserving scalar hints.""" - item, _selected_entity = _entity_payload_with_source(payload, *entity_names) - return item - - -def _privatize_pane_event_id(item: dict[str, Any]) -> dict[str, Any]: - """Treat generic ``id`` on pane events as a private pane identifier.""" - raw_id = _first_text(item, ("id",)) - if raw_id and _first_text(item, ("pane_id", "paneId")) is None: - item["pane_id"] = raw_id - for key in list(item): - if _compact_key(key) == "id": - item.pop(key, None) - return item - - -def _pane_event_payload_with_provenance( - payload: Mapping[str, Any], - *entity_names: str, - allow_top_level_pane_info: bool = False, -) -> tuple[dict[str, Any], bool]: - """Return a pane-event item and whether a full PaneInfo authorizes it.""" - item, selected_entity = _entity_payload_with_source(payload, *entity_names) - # EventData's internally tagged discriminator is envelope metadata, not a - # PaneInfo field. Keeping it would make event and pane.list projections - # differ even after their identity pair was canonicalized. - for key in list(item): - if _compact_key(key) == "type": - item.pop(key, None) - # Scalar agent/status events and nested agent/worker objects are not - # PaneInfo, even when they repeat workspace_id/pane_id. Treating them as - # PaneInfo was the observation-layer path that admitted alternate identity - # representations. Pane lifecycle events historically also carry a full - # top-level PaneInfo, so their caller opts into that established shape. - top_level_identity = canonical_herdr_pane_identity( - _first_text(item, ("workspace_id", "workspaceId")), - _first_text(item, ("pane_id", "paneId")), - ) - pane_info_observed = selected_entity == "pane" or ( - selected_entity is None - and allow_top_level_pane_info - and top_level_identity is not None - ) - return _privatize_pane_event_id(item), pane_info_observed - - -def _pane_event_payload(payload: Mapping[str, Any], *entity_names: str) -> dict[str, Any]: - """Return a pane-event item without exposing pane ``id`` as public worker id.""" - item, _pane_info_observed = _pane_event_payload_with_provenance( - payload, - *entity_names, - ) - return item - - -def _event_alias_key(name: str) -> str: - return "_".join(part for part in _compact_key(name).split("_") if part) - - -def _canonical_event_name(raw_name: Any) -> str | None: - if not isinstance(raw_name, str) or not raw_name.strip(): - return None - event_name = raw_name.strip() - if event_name in _SUPPORTED_EVENT_NAME_SET: - return event_name - aliases = {_event_alias_key(name): name for name in _SUPPORTED_EVENT_NAMES} - aliases.update( - { - "agent_detected": "pane.agent_detected", - "agent_observed": "pane.agent_detected", - "agent_status_changed": "pane.agent_status_changed", - "agent_status_updated": "pane.agent_status_changed", - "pane_output_changed": "pane.updated", - "pane_observed": "pane.created", - "pane_detected": "pane.created", - "workspace_observed": "workspace.updated", - "workspace_detected": "workspace.created", - "worktree_observed": "worktree.opened", - "worktree_detected": "worktree.created", - "worktree_updated": "worktree.opened", - "worktree_changed": "worktree.opened", - "worktree_closed": "worktree.removed", - "worktree_deleted": "worktree.removed", - } - ) - return aliases.get(_event_alias_key(event_name)) - - -def _strict_producer_id(value: object) -> str | None: - if type(value) is not str or not value or any(character.isspace() for character in value): - return None - return value - - -def _strict_producer_sequence(value: object) -> str | None: - if type(value) is not int or value < 0: - return None - return str(value) - - -def _producer_identity(envelope: Mapping[str, Any]) -> HerdrProducerIdentity | None: - """Return only valid explicit top-level producer identity when present. - - Herdr's confirmed EventEnvelope has only ``event`` and ``data``. The - identity fields handled here are forward-compatible optional metadata; - malformed metadata leaves the event idless, and entity fields inside - ``data`` are never durable event identity. - """ - if "event_id" in envelope: - event_id = _strict_producer_id(envelope.get("event_id")) - return HerdrEventId(event_id) if event_id is not None else None - server_present = "server_id" in envelope - sequence_present = "sequence" in envelope - if not server_present and not sequence_present: - return None - producer_id = _strict_producer_id(envelope.get("server_id")) - sequence = _strict_producer_sequence(envelope.get("sequence")) - if not server_present or not sequence_present or producer_id is None or sequence is None: - return None - return HerdrProducerSequence(producer_id, sequence) - - -def normalize_event(envelope: Mapping[str, Any]) -> NormalizedHerdrEvent | None: - """Normalize a Herdr event envelope; unsupported events return ``None``. - - Confirmed Herdr envelopes use ``event`` and ``data`` and are intentionally - idless. ``payload`` remains receive-only compatibility for older clients. - """ - name = _canonical_event_name(envelope.get("event")) - if name is None: - return None - payload = envelope.get("data") if "data" in envelope else envelope.get("payload", {}) - if payload is None: - payload = {} - if not isinstance(payload, Mapping): - return None - return NormalizedHerdrEvent( - name=name, - payload=dict(payload), - producer_identity=_producer_identity(envelope), - ) - - -def _worker_copy( - worker: Worker, - *, - worker_id: str | None = None, - name: str | None = None, - status: str | None = None, - space_id: str | None = None, - meta: Mapping[str, Any] | None = None, - last_seen_at: str | None = None, - summary: str | None = None, - backend_target: Mapping[str, Any] | None = None, -) -> Worker: - return Worker( - id=worker_id if worker_id is not None else worker.id, - name=name if name is not None else worker.name, - status=status if status is not None else worker.status, - space_id=space_id if space_id is not None else worker.space_id, - meta=dict(meta) if meta is not None else dict(worker.meta), - last_seen_at=last_seen_at if last_seen_at is not None else worker.last_seen_at, - summary=summary if summary is not None else worker.summary, - backend_target=dict(backend_target) if isinstance(backend_target, Mapping) else worker.backend_target, - ) - - -def _merge_worker_update( - existing: Worker | None, - observed: Worker, - *, - status: str | None = None, - preserve_existing_continuity: bool = False, -) -> Worker: - if existing is None: - if status is not None: - return _worker_copy(observed, status=status) - return observed - merged_meta = _strip_stable_key_fields(existing.meta) - if ( - preserve_existing_continuity - and is_stable_worker_key(existing.meta.get("stable_key")) - and type(existing.meta.get("stable_key_version")) is int - and existing.meta.get("stable_key_version") == STABLE_KEY_VERSION - ): - merged_meta["stable_key"] = existing.meta["stable_key"] - merged_meta["stable_key_version"] = STABLE_KEY_VERSION - merged_meta.update(observed.meta) - observed_name_is_identity = observed.name in {observed.id, "unknown"} - resolved_status = status if status is not None else observed.status - if resolved_status == "unknown" and existing.status != "unknown": - resolved_status = existing.status - return Worker( - id=existing.id, - name=existing.name if observed_name_is_identity else observed.name, - status=resolved_status, - space_id=observed.space_id or existing.space_id, - meta=merged_meta, - last_seen_at=observed.last_seen_at or utc_timestamp(), - summary=observed.summary or existing.summary, - backend_target=observed.backend_target or existing.backend_target, - ) - - -def _closed_worker(worker: Worker) -> Worker: - return _worker_copy(worker, status="closed", last_seen_at=worker.last_seen_at or utc_timestamp()) - - -def _observed_worker_count(workers: Sequence[Worker]) -> int: - return len([worker for worker in workers if worker.status != "closed"]) - - -def _binding_target(binding: WorkerBinding) -> tuple[str, str]: - return (binding.target_kind, binding.target_value) - -def _worker_state_equal(left: Worker, right: Worker) -> bool: - """Compare effective worker state while ignoring observation timestamps.""" - return left.fingerprint == right.fingerprint and left.backend_target == right.backend_target - - -def _binding_state_equal(left: WorkerBinding, right: WorkerBinding) -> bool: - """Compare effective private routing while ignoring observation timestamps.""" - return ( - left.host_id, - left.worker_id, - left.worker_fingerprint, - left.backend, - left.target_kind, - left.target_value, - left.turn_target_kind, - left.turn_target_value, - left.sendable, - left.reason, - left.expires_at, - left.private_fingerprint, - ) == ( - right.host_id, - right.worker_id, - right.worker_fingerprint, - right.backend, - right.target_kind, - right.target_value, - right.turn_target_kind, - right.turn_target_value, - right.sendable, - right.reason, - right.expires_at, - right.private_fingerprint, - ) - - - - -def _target_pairs_from_item(item: Mapping[str, Any], *, old_first: bool = False) -> list[tuple[str, str]]: - old_pairs = [ - ("pane_id", _first_text(item, ("old_pane_id", "previous_pane_id", "from_pane_id", "source_pane_id"))), - ( - "terminal_id", - _first_text(item, ("old_terminal_id", "previous_terminal_id", "from_terminal_id", "source_terminal_id")), - ), - ] - current_pairs = [ - ("agent_id", _first_text(item, ("agent_id", "agentId"))), - ("terminal_id", _first_text(item, ("terminal_id", "terminalId"))), - ("pane_id", _first_text(item, ("pane_id", "paneId", "id"))), - ("agent", _first_text(item, ("agent",))), - ("name", _first_text(item, ("name", "label"))), - ] - pairs = [*old_pairs, *current_pairs] if old_first else [*current_pairs, *old_pairs] - seen: set[tuple[str, str]] = set() - result: list[tuple[str, str]] = [] - for kind, value in pairs: - if not value: - continue - pair = (kind, value) - if pair in seen: - continue - seen.add(pair) - result.append(pair) - return result - - -def _new_move_target(item: Mapping[str, Any]) -> tuple[str, str] | None: - candidates = ( - ("pane_id", ("new_pane_id", "to_pane_id", "target_pane_id", "pane_id", "paneId", "id")), - ("terminal_id", ("new_terminal_id", "to_terminal_id", "target_terminal_id", "terminal_id", "terminalId")), - ("agent_id", ("agent_id", "agentId")), - ) - for kind, keys in candidates: - value = _first_text(item, keys) - if value: - return kind, value - return None - - -def _has_public_worker_identity(item: Mapping[str, Any]) -> bool: - return ( - _first_text(item, ("worker_id", "id", "slug", "agent_id", "agent", "name", "label", "title")) - is not None - ) - -def _has_authoritative_identity_tuple(item: Mapping[str, Any]) -> bool: - return ( - _field_value(item, "workspace_id") is not None - and _field_value(item, "pane_id") is not None - ) - - -def _has_authoritative_binding_target(item: Mapping[str, Any]) -> bool: - agent_session = _safe_mapping(_field_value(item, "agent_session")) - return ( - _first_text(item, ("agent_id", "terminal_id")) is not None - or _first_text(agent_session, ("value", "id")) is not None - ) - - -def _authenticated_local_stable_key(worker: Worker) -> str | None: - value = worker.meta.get("stable_key") - version = worker.meta.get("stable_key_version") - if ( - is_stable_worker_key(value) - and type(version) is int - and version == STABLE_KEY_VERSION - ): - return str(value) - return None - - -class HerdrEventBackend: - """Maintain Tendwire projections from Herdr socket reconcile and events.""" - - def __init__( - self, - config: Config, - *, - client_factory: Callable[[Config], HerdrSocketClient] | None = None, - subscribe_method: str = DEFAULT_SUBSCRIBE_METHOD, - debounce_seconds: float | None = None, - reconcile_interval_seconds: float | None = None, - max_workers: int | None = None, - output_excerpt_chars: int | None = None, - dedupe_size: int = DEFAULT_DEDUPE_SIZE, - max_batch_size: int = DEFAULT_MAX_BATCH_SIZE, - reconnect_delay_seconds: float = DEFAULT_RECONNECT_DELAY_SECONDS, - stop_event: threading.Event | None = None, - turn_completion_processor: Callable[..., Any] | None = None, - ) -> None: - self.config = config - self.client_factory = client_factory or self._default_client_factory - requested_subscribe_method = str(subscribe_method or DEFAULT_SUBSCRIBE_METHOD) - if requested_subscribe_method != HERDR_EVENTS_SUBSCRIBE_METHOD: - raise HerdrEventBackendError("Herdr event backend requires events.subscribe") - self.subscribe_method = HERDR_EVENTS_SUBSCRIBE_METHOD - configured_debounce = config.event_debounce_seconds if debounce_seconds is None else debounce_seconds - configured_reconcile = ( - config.reconcile_interval_seconds - if reconcile_interval_seconds is None - else reconcile_interval_seconds - ) - self.debounce_seconds = max(0.0, float(configured_debounce)) - self.reconcile_interval_seconds = max(0.0, float(configured_reconcile)) - self.max_workers = max(1, int(config.max_workers if max_workers is None else max_workers)) - self.output_excerpt_chars = max( - 1, - int(config.output_excerpt_chars if output_excerpt_chars is None else output_excerpt_chars), - ) - self.dedupe_size = max(1, int(dedupe_size)) - self.max_batch_size = max(1, int(max_batch_size)) - self.reconnect_delay_seconds = max(0.0, float(reconnect_delay_seconds)) - self.stop_event = stop_event or threading.Event() - self.turn_completion_processor = ( - turn_completion_processor - or self._default_turn_completion_processor - ) - self._lock = threading.RLock() - self._turn_refresh_callback_lock = threading.Lock() - self._turn_refresh_callback: Callable[[], None] | None = None - self._ready = threading.Event() - self._thread: threading.Thread | None = None - self._producer_dedupe: OrderedDict[HerdrProducerIdentity, None] = OrderedDict() - self._pending_events: list[NormalizedHerdrEvent] = [] - self._spaces: dict[str, Space] = {} - self._workers: dict[str, Worker] = {} - self._bindings: dict[str, WorkerBinding] = {} - # pane_id -> terminal_id, remembered from reconcile pane lists so that - # pane-id-only events (pane.agent_status_changed carries no terminal id) - # can still resolve to the terminal-targeted stored binding. - self._pane_terminals: dict[str, str] = {} - self._pane_owners: dict[str, set[str]] = {} - self._terminal_owners: dict[str, set[str]] = {} - self._session_owners: dict[str, set[str]] = {} - self._event_continuity_revalidated = False - self._health = self._health_for("unknown") - self._last_event_at: str | None = None - self._last_reconcile_at: str | None = None - self._last_snapshot_at: str | None = None - self._last_cap_status_at: str | None = None - self._automatic_maintenance_status: dict[str, Any] | None = None - self._next_reconcile_monotonic: float | None = None - self._subscription_pane_ids: list[str] = [] - self._turn_api_probed = False - self._turn_api_supported = False - self._turn_completion_diagnostic_counts: dict[str, int] = {} - self._load_existing_state() - - @staticmethod - def _default_client_factory(config: Config) -> HerdrSocketClient: - return HerdrSocketClient(timeout=config.herdr_timeout_seconds) - - @staticmethod - def _default_turn_completion_processor( - config: Config, - pane_id: str, - *, - terminal_id: str | None = None, - binding_private_fingerprint: str | None = None, - ) -> Any: - from .herdr_turns import refresh_completed_pane_turn - - return refresh_completed_pane_turn( - config, - pane_id, - terminal_id=terminal_id, - binding_private_fingerprint=binding_private_fingerprint, - adapter_timeout_seconds=config.herdr_timeout_seconds, - ) - - @property - def db_path(self) -> Path: - if self.config.db_path is None: - raise HerdrEventBackendError("socket event backend requires a sqlite db path") - return Path(self.config.db_path) - - @property - def health(self) -> HerdrEventBackendHealth: - with self._lock: - return self._health - - @property - def operational_status(self) -> dict[str, Any]: - with self._lock: - return { - "status": self._health.status, - "outcome": self._health.outcome, - "ready": self.ready, - "running": self.running, - "last_event_at": self._last_event_at, - "last_reconcile_at": self._last_reconcile_at, - "last_snapshot_at": self._last_snapshot_at, - "last_cap_status_at": self._last_cap_status_at, - "reconcile_enabled": self.reconcile_interval_seconds > 0, - "automatic_maintenance": ( - dict(self._automatic_maintenance_status) - if self._automatic_maintenance_status is not None - else None - ), - "turn_completion_diagnostics": dict( - self._turn_completion_diagnostic_counts - ), - } - - @property - def ready(self) -> bool: - return self._ready.is_set() - - @property - def running(self) -> bool: - thread = self._thread - return thread is not None and thread.is_alive() - - def set_turn_refresh_callback(self, callback: Callable[[], None] | None) -> None: - """Set the post-persistence turn refresh signal.""" - if callback is not None and not callable(callback): - raise TypeError("turn refresh callback must be callable or None") - with self._turn_refresh_callback_lock: - self._turn_refresh_callback = callback - - def _notify_turn_refresh(self) -> None: - with self._turn_refresh_callback_lock: - callback = self._turn_refresh_callback - if callback is None: - return - try: - callback() - except Exception: - # Scheduling is best-effort after the durable backend commit. - pass - - def _health_for(self, outcome: str) -> HerdrEventBackendHealth: - health = herdr_backend_health(outcome) - return HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or utc_timestamp(), - message=health.message, - ) - - def _save_snapshot( - self, - snapshot: Snapshot, - *, - observation: SnapshotObservationContext, - worker_bindings: Iterable[WorkerBinding] | None = None, - binding_observation_authoritative: bool = False, - binding_workers_present: bool = True, - ) -> None: - if not save_snapshot( - self.db_path, - snapshot, - turn_model=self.config.turn_model, - observation=observation, - worker_bindings=worker_bindings, - binding_backend=BACKEND_NAME if worker_bindings is not None else None, - binding_observation_authoritative=binding_observation_authoritative, - binding_workers_present=binding_workers_present, - ): - raise RuntimeError("snapshot rejected by store ordering") - policy = SnapshotRetentionPolicy( - retention_days=self.config.snapshot_retention_days, - retention_count=self.config.snapshot_retention_count, - batch_size=self.config.snapshot_maintenance_batch_size, - ) - try: - result = maybe_run_automatic_store_maintenance( - self.db_path, - policy=policy, - turn_model=self.config.turn_model, - acknowledged_final_retention_days=( - self.config.acknowledged_final_retention_days - ), - acknowledged_final_retention_count=( - self.config.acknowledged_final_retention_count - ), - command_retry_horizon_seconds=( - self.config.command_retry_horizon_seconds - ), - command_receipt_retention_seconds=( - self.config.command_receipt_retention_seconds - ), - command_receipt_retention_count=( - self.config.command_receipt_retention_count - ), - cadence_seconds=self.config.store_maintenance_cadence_seconds, - ) - snapshot_result = result.get("snapshot") - snapshot_counts = snapshot_result if isinstance(snapshot_result, Mapping) else {} - maintenance_status = { - "ok": bool(result.get("ok")), - "status": str(result.get("status") or "unknown"), - "due": bool(result.get("due")), - "examined": int(snapshot_counts.get("examined") or 0), - "deleted": int(snapshot_counts.get("deleted") or 0), - "remaining_candidates": bool(snapshot_counts.get("remaining_candidates")), - } - except Exception: - self._automatic_maintenance_status = { - "ok": False, - "status": "failed", - "due": False, - "examined": 0, - "deleted": 0, - "remaining_candidates": False, - } - else: - self._automatic_maintenance_status = maintenance_status - - def _load_existing_state(self) -> None: - try: - snapshot = latest_snapshot(self.db_path, self.config.host_id) - except Exception: - snapshot = None - if snapshot is not None: - self._spaces = {space.id: space for space in snapshot.spaces} - self._workers = {worker.id: worker for worker in snapshot.workers} - self._last_snapshot_at = snapshot.updated_at - for health in snapshot.backend_health: - if health.name == BACKEND_NAME: - self._health = HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or utc_timestamp(), - message=health.message, - ) - break - try: - bindings = list_worker_bindings(self.db_path, self.config.host_id, backend=BACKEND_NAME) - except Exception: - bindings = [] - self._bindings = {binding.private_fingerprint: binding for binding in bindings} - self._replace_ownership_maps([], bindings) - - def start(self, *, wait_for_reconcile: bool = True, timeout_seconds: float | None = None) -> None: - if self._thread is not None: - return - self.stop_event.clear() - self._ready.clear() - thread = threading.Thread(target=self.run_forever, name="tendwire-herdr-events", daemon=True) - self._thread = thread - thread.start() - if wait_for_reconcile: - timeout = self.config.herdr_timeout_seconds if timeout_seconds is None else timeout_seconds - self._ready.wait(max(0.001, float(timeout))) - - def stop(self) -> None: - self.stop_event.set() - self.flush() - thread = self._thread - if thread is not None and thread is not threading.current_thread(): - thread.join(timeout=max(1.0, self.config.herdr_timeout_seconds)) - self._thread = None - - def run_forever(self) -> None: - while not self.stop_event.is_set(): - reconciled = False - try: - client = self.client_factory(self.config) - try: - self._turn_api_probed = False - self._turn_api_supported = False - self.reconcile_once(client=client) - reconciled = True - self._replay_turns_after_reconcile(client) - if self.stop_event.is_set(): - break - if hasattr(client, "connect"): - client.connect() - stream = self._subscribe_event_stream(client) - # Herdr subscriptions intentionally start at the server's - # current sequence and deliver no backlog. The first - # replay above capability-gates the new subscription kind; - # this second replay closes the probe-to-subscribe race. - # Events concurrent with it are buffered by the socket - # client and become harmless watermark-deduped duplicates. - if self._turn_api_supported: - self._replay_turns_after_reconcile(client) - self._ready.set() - self._read_event_stream(client, stream.subscription_id) - finally: - if hasattr(client, "close"): - client.close() - except HerdrSocketTimeoutError: - self._mark_unhealthy_safe("timeout") - except (HerdrSocketDisconnectedError, HerdrSocketConnectionError): - # A complete list reconciliation is authoritative. The event - # stream only accelerates later observations, so its closure - # must not replace that healthy snapshot with unavailable. - if not reconciled: - self._mark_unhealthy_safe("socket_disconnected") - except (HerdrMalformedLineError, HerdrEnvelopeError, HerdrProtocolError, ValueError, TypeError): - self._mark_unhealthy_safe("protocol_error") - except Exception: - self._mark_unhealthy_safe("unknown") - if self.stop_event.is_set(): - break - delay = self._reconnect_delay_seconds() - if delay: - self.stop_event.wait(delay) - - def _reconnect_delay_seconds(self) -> float: - delay = self.reconnect_delay_seconds - with self._lock: - outcome = self._health.outcome - if outcome == "protocol_error" and self.ready and self.reconcile_interval_seconds > 0: - return max(delay, min(self.reconcile_interval_seconds, 60.0)) - return delay - - def _read_event_stream(self, client: Any, subscription_id: str) -> None: - while not self.stop_event.is_set(): - self._run_periodic_reconcile_if_due() - try: - envelope = client.read_event(subscription_id, timeout=self.config.herdr_timeout_seconds) - except HerdrSocketTimeoutError: - self._run_periodic_reconcile_if_due() - continue - self.queue_event_envelope(envelope) - disconnected = False - deadline = time.monotonic() + self.debounce_seconds - while ( - self.debounce_seconds > 0 - and self._pending_event_count() < self.max_batch_size - and not self.stop_event.is_set() - ): - remaining = deadline - time.monotonic() - if remaining <= 0: - break - try: - extra = client.read_event(subscription_id, timeout=max(0.001, remaining)) - except HerdrSocketTimeoutError: - break - except HerdrSocketDisconnectedError: - disconnected = True - break - self.queue_event_envelope(extra, flush=False) - self.flush() - self._run_periodic_reconcile_if_due() - if disconnected: - raise HerdrSocketDisconnectedError("Herdr socket disconnected during event drain") - - def _schedule_next_reconcile(self) -> None: - if self.reconcile_interval_seconds <= 0: - self._next_reconcile_monotonic = None - return - self._next_reconcile_monotonic = time.monotonic() + self.reconcile_interval_seconds - - def _run_periodic_reconcile_if_due(self, client: Any | None = None) -> None: - if self.reconcile_interval_seconds <= 0: - return - due_at = self._next_reconcile_monotonic - if due_at is None: - self._schedule_next_reconcile() - return - if time.monotonic() < due_at: - return - self.reconcile_once(client=client) - - def _pending_event_count(self) -> int: - with self._lock: - return len(self._pending_events) - - def _current_bindings(self) -> list[WorkerBinding]: - return list(self._bindings.values()) - - def _records_from_reconcile_payloads(self, agent_payload: Any, pane_payload: Any) -> list[Any]: - return _records_from_agent_and_pane_payloads( - self.config, - agent_payload, - pane_payload, - ) - - def _pane_subscription_ids(self, pane_payload: Any) -> list[str]: - pane_ids: list[str] = [] - seen: set[str] = set() - for item in _payload_items(pane_payload, _PANE_PAYLOAD_KEYS): - if not _pane_has_agent(item): - continue - pane_id = _first_text(item, ("pane_id", "paneId", "id")) - if not pane_id or pane_id in seen: - continue - seen.add(pane_id) - pane_ids.append(pane_id) - if len(pane_ids) >= self.max_workers: - break - return pane_ids - - def reconcile_once(self, *, client: Any | None = None) -> Snapshot: - """Perform a full Herdr list reconcile and persist Tendwire projections.""" - owns_client = client is None - if client is None: - client = self.client_factory(self.config) - try: - if owns_client and hasattr(client, "connect"): - client.connect() - payloads = { - "workspace.list": self._call_list_method(client, "workspace_list"), - "tab.list": self._call_list_method(client, "tab_list"), - "pane.list": self._call_list_method(client, "pane_list"), - "agent.list": self._call_list_method(client, "agent_list"), - } - # tab.list is intentionally part of the authoritative reconcile barrier; - # Tendwire has no current public tab model to project into. - _ = payloads["tab.list"] - with self._lock: - stored_bindings = list_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - ) - spaces = _spaces_from_payload(payloads["workspace.list"]) - records = self._records_from_reconcile_payloads( - payloads["agent.list"], - payloads["pane.list"], - ) - subscription_pane_ids = self._pane_subscription_ids(payloads["pane.list"]) - workers, bindings = _workers_and_bindings_from_records( - self.config, - records, - stored_bindings=stored_bindings, - require_authenticated_continuity=True, - ) - if _observed_worker_count(workers) > self.max_workers: - return self._mark_worker_cap_exceeded_locked( - _observed_worker_count(workers) - ) - outcome = "healthy_non_empty" if spaces or workers else "empty_healthy" - health = herdr_backend_health(outcome, spaces=spaces, workers=workers) - previous = latest_snapshot(self.db_path, self.config.host_id) - snapshot_workers = self._workers_with_closed_missing( - previous.workers if previous is not None else [], - workers, - bound_worker_ids={binding.worker_id for binding in stored_bindings}, - ) - snapshot = project_from_observations( - self.config, - spaces=spaces, - workers=snapshot_workers, - backend_health=[health], - ) - self._save_snapshot( - snapshot, - observation=SnapshotObservationContext( - authority="complete", - observed_at=health.observed_at or snapshot.updated_at, - ), - worker_bindings=bindings, - binding_observation_authoritative=True, - binding_workers_present=bool(workers), - ) - self._last_reconcile_at = snapshot.updated_at - self._last_snapshot_at = snapshot.updated_at - self._schedule_next_reconcile() - self._spaces = {space.id: space for space in snapshot.spaces} - self._workers = {worker.id: worker for worker in snapshot.workers} - self._bindings = {binding.private_fingerprint: binding for binding in bindings} - self._replace_ownership_maps(records, bindings) - self._subscription_pane_ids = subscription_pane_ids - self._health = HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or snapshot.updated_at, - message=health.message, - ) - self._notify_turn_refresh() - return snapshot - except (HerdrContinuityUnavailableError, InstallationKeyError): - snapshot = self._mark_unhealthy("continuity_unavailable") - with self._lock: - self._last_reconcile_at = snapshot.updated_at - self._schedule_next_reconcile() - return snapshot - except Exception: - self._mark_unhealthy("unknown") - raise - finally: - if owns_client and hasattr(client, "close"): - client.close() - - def _call_list_method(self, client: Any, method_name: str) -> Any: - method = getattr(client, method_name) - try: - try: - return method(timeout=self.config.herdr_timeout_seconds) - except TypeError: - return method() - finally: - # Herdr 0.7.x may close a list connection immediately after its - # response. Reconnect between authoritative read-only probes so a - # late close cannot race the next request after its write. - if hasattr(client, "close"): - client.close() - - @staticmethod - def _herdr_error_code(exc: HerdrErrorResponse) -> str: - error = exc.error - if isinstance(error, Mapping): - code = error.get("code") - if isinstance(code, str): - return code - return "" - - @staticmethod - def _herdr_error_message(exc: HerdrErrorResponse) -> str: - error = exc.error - if isinstance(error, Mapping): - message = error.get("message") - if isinstance(message, str): - return message - return "" - - def _call_pane_turns( - self, - client: Any, - pane_id: str, - *, - since: int, - expected_epoch: int | None, - allow_uncorrelated: bool = False, - ) -> HerdrPaneTurnsReplay: - params: dict[str, Any] = { - "pane_id": str(pane_id), - "since": int(since), - } - if expected_epoch is not None: - params["expected_epoch"] = int(expected_epoch) - optional_keywords = {"timeout": self.config.herdr_timeout_seconds} - if allow_uncorrelated: - optional_keywords["allow_uncorrelated"] = True - method = getattr(client, "pane_turns", None) - if callable(method): - value = _call_with_optional_keywords( - method, - (params,), - optional_keywords, - ) - else: - request = getattr(client, "request", None) - if not callable(request): - raise AttributeError("client does not expose pane.turns") - value = _call_with_optional_keywords( - request, - ("pane.turns", params), - optional_keywords, - ) - return _pane_turns_replay(value, pane_id) - - def _record_turn_diagnostic( - self, - code: str, - pane_id: str, - *, - status: str | None = None, - ) -> None: - count_key = f"{code}:{status}" if status else code - with self._lock: - self._turn_completion_diagnostic_counts[count_key] = ( - self._turn_completion_diagnostic_counts.get(count_key, 0) + 1 - ) - diagnostic = { - "code": code, - "host_id": self.config.host_id, - "pane_id": pane_id, - } - if status: - diagnostic["status"] = status - _LOGGER.warning( - code, - extra={"tendwire_diagnostic": diagnostic}, - ) - - def _record_completeness_break( - self, - replay: HerdrPaneTurnsReplay, - reason: str, - ) -> None: - record_herdr_turn_completeness_break( - self.db_path, - self.config.host_id, - replay.pane_id, - turn_epoch=replay.turn_epoch, - newest_turn=replay.newest_turn, - reason=reason, - ) - _LOGGER.warning( - "herdr_turn_completeness_break", - extra={ - "tendwire_diagnostic": { - "code": "herdr_turn_completeness_break", - "host_id": self.config.host_id, - "pane_id": replay.pane_id, - "reason": reason, - } - }, - ) - - def _completion_processor_result( - self, - record: HerdrTurnCompletionRecord, - ) -> tuple[str, str | None, str | None]: - terminal_id = self._pane_terminals.get(record.pane_id) - owner_ids = self._pane_owners.get(record.pane_id, set()) - owner_bindings = [ - binding - for binding in self._bindings.values() - if binding.worker_id in owner_ids - ] - binding_private_fingerprint = ( - owner_bindings[0].private_fingerprint - if len(owner_bindings) == 1 - else None - ) - result = _call_with_optional_keywords( - self.turn_completion_processor, - (self.config, record.pane_id), - { - "terminal_id": terminal_id, - "binding_private_fingerprint": binding_private_fingerprint, - }, - ) - if isinstance(result, Mapping): - status = str(result.get("status") or "") - worker_id = result.get("worker_id") - refreshed_turn_id = result.get("refreshed_turn_id") - else: - status = str(getattr(result, "status", "")) - worker_id = getattr(result, "worker_id", None) - refreshed_turn_id = getattr(result, "refreshed_turn_id", None) - return ( - status, - str(worker_id) if worker_id else None, - str(refreshed_turn_id) if refreshed_turn_id else None, - ) - - def _process_turn_record( - self, - record: HerdrTurnCompletionRecord, - ) -> None: - watermark = get_herdr_turn_watermark( - self.db_path, - self.config.host_id, - record.pane_id, - ) - if watermark is None: - self._record_completeness_break( - HerdrPaneTurnsReplay( - pane_id=record.pane_id, - turn_epoch=record.turn_epoch, - records=(record,), - truncated=False, - oldest_available=record.turn, - ), - "live_without_baseline", - ) - return - if watermark.turn_epoch != record.turn_epoch: - self._record_completeness_break( - HerdrPaneTurnsReplay( - pane_id=record.pane_id, - turn_epoch=record.turn_epoch, - records=(record,), - truncated=False, - oldest_available=record.turn, - ), - "turn_epoch_mismatch", - ) - return - if record.turn <= watermark.last_turn: - return - if record.turn != watermark.last_turn + 1: - self._record_completeness_break( - HerdrPaneTurnsReplay( - pane_id=record.pane_id, - turn_epoch=record.turn_epoch, - records=(record,), - truncated=False, - oldest_available=record.turn, - ), - "live_gap", - ) - return - status, worker_id, refreshed_turn_id = self._completion_processor_result( - record - ) - if status not in _COMPLETED_TURN_REFRESH_STATUSES: - self._record_turn_diagnostic( - "herdr_turn_completion_refresh_skipped", - record.pane_id, - status=status or "unknown", - ) - refreshed_turn_id = None - record_herdr_turn_completion( - self.db_path, - self.config.host_id, - record.pane_id, - turn_epoch=record.turn_epoch, - turn=record.turn, - outcome=record.outcome, - completed_unix_ms=record.completed_unix_ms, - message=record.message, - message_truncated=record.message_truncated, - agent_session_path=record.agent_session_path, - worker_id=worker_id, - refreshed_turn_id=refreshed_turn_id, - ) - - def _consume_replay( - self, - replay: HerdrPaneTurnsReplay, - watermark: HerdrTurnWatermark | None, - ) -> None: - if watermark is None: - set_herdr_turn_watermark( - self.db_path, - self.config.host_id, - replay.pane_id, - turn_epoch=replay.turn_epoch, - last_turn=replay.newest_turn, - ) - return - if replay.turn_epoch != watermark.turn_epoch: - self._record_completeness_break(replay, "turn_epoch_mismatch") - return - if replay.truncated: - self._record_completeness_break(replay, "replay_truncated") - return - expected = watermark.last_turn + 1 - actual_turns = tuple(record.turn for record in replay.records) - expected_turns = tuple(range(expected, expected + len(replay.records))) - if actual_turns != expected_turns: - self._record_completeness_break(replay, "replay_gap") - return - for record in replay.records: - self._process_turn_record(record) - - @classmethod - def _turn_api_method_unsupported(cls, exc: HerdrErrorResponse) -> bool: - code = cls._herdr_error_code(exc).strip().lower().replace("-", "_") - raw_message = cls._herdr_error_message(exc).lower() - message = raw_message.strip() - if code in { - "method_not_found", - "unknown_method", - "unsupported_method", - "not_implemented", - }: - return True - if code == "invalid_request" and raw_message.startswith( - "invalid request: unknown variant" - ): - return True - return code == "invalid_params" and any( - marker in message - for marker in ( - "unknown method", - "method not found", - "unsupported method", - "pane.turns is not supported", - ) - ) - - def _probe_turn_api( - self, - client: Any, - pane_id: str, - watermark: HerdrTurnWatermark | None, - ) -> tuple[ - bool, - HerdrPaneTurnsReplay | None, - HerdrErrorResponse | AttributeError | None, - ]: - """Probe pane.turns once without treating pane-scoped errors as absence.""" - if not callable(getattr(client, "pane_turns", None)) and not callable( - getattr(client, "request", None) - ): - return False, None, None - try: - replay = self._call_pane_turns( - client, - pane_id, - since=watermark.last_turn if watermark is not None else 0, - expected_epoch=watermark.turn_epoch if watermark is not None else None, - allow_uncorrelated=True, - ) - except HerdrErrorResponse as exc: - if self._turn_api_method_unsupported(exc): - return False, None, None - return True, None, exc - except AttributeError as exc: - return True, None, exc - return True, replay, None - - def _consume_pane_replay( - self, - client: Any, - pane_id: str, - watermark: HerdrTurnWatermark | None, - *, - replay: HerdrPaneTurnsReplay | None = None, - error: HerdrErrorResponse | AttributeError | None = None, - ) -> None: - try: - if error is not None: - raise error - current_replay = replay or self._call_pane_turns( - client, - pane_id, - since=watermark.last_turn if watermark is not None else 0, - expected_epoch=watermark.turn_epoch if watermark is not None else None, - ) - except HerdrErrorResponse as exc: - code = self._herdr_error_code(exc) - message = self._herdr_error_message(exc) - if code == "turn_epoch_mismatch": - try: - current_replay = self._call_pane_turns( - client, - pane_id, - since=0, - expected_epoch=None, - ) - except (HerdrErrorResponse, AttributeError) as retry_exc: - self._record_turn_diagnostic( - "herdr_turn_replay_pane_skipped", - pane_id, - status=( - self._herdr_error_code(retry_exc) - if isinstance(retry_exc, HerdrErrorResponse) - else "method_unavailable" - ), - ) - return - self._record_completeness_break( - current_replay, - "turn_epoch_mismatch", - ) - return - if code == "invalid_params" and "newer than current turn" in message: - try: - current_replay = self._call_pane_turns( - client, - pane_id, - since=0, - expected_epoch=None, - ) - except (HerdrErrorResponse, AttributeError) as retry_exc: - self._record_turn_diagnostic( - "herdr_turn_replay_pane_skipped", - pane_id, - status=( - self._herdr_error_code(retry_exc) - if isinstance(retry_exc, HerdrErrorResponse) - else "method_unavailable" - ), - ) - return - self._record_completeness_break(current_replay, "watermark_ahead") - return - self._record_turn_diagnostic( - "herdr_turn_replay_pane_skipped", - pane_id, - status=code or "pane_error", - ) - return - except AttributeError: - self._record_turn_diagnostic( - "herdr_turn_replay_pane_skipped", - pane_id, - status="method_unavailable", - ) - return - self._consume_replay(current_replay, watermark) - - def _replay_turns_after_reconcile(self, client: Any) -> None: - """Probe pane.turns once, then replay each pane independently.""" - pane_ids = tuple(self._subscription_pane_ids) - if not pane_ids: - self._turn_api_probed = True - self._turn_api_supported = False - return - watermarks = { - pane_id: get_herdr_turn_watermark( - self.db_path, - self.config.host_id, - pane_id, - ) - for pane_id in pane_ids - } - probe_pane_id: str | None = None - probe_replay: HerdrPaneTurnsReplay | None = None - probe_error: HerdrErrorResponse | AttributeError | None = None - if not self._turn_api_probed: - probe_pane_id = pane_ids[0] - supported, probe_replay, probe_error = self._probe_turn_api( - client, - probe_pane_id, - watermarks[probe_pane_id], - ) - self._turn_api_probed = True - self._turn_api_supported = supported - if not self._turn_api_supported: - return - for pane_id in pane_ids: - self._consume_pane_replay( - client, - pane_id, - watermarks[pane_id], - replay=probe_replay if pane_id == probe_pane_id else None, - error=probe_error if pane_id == probe_pane_id else None, - ) - - def _subscribe_event_stream(self, client: Any) -> Any: - # Herdr 0.7.5 strictly validates pane-scoped status subscriptions and - # added the general pane.updated event. Use one bounded mixed - # subscription: one global entry per lifecycle/update type and one - # status entry per pane. - # pane.output_matched is intentionally absent because 0.7.5 requires a - # caller-provided match expression; pane.updated is the generic turn - # and stream refresh signal. - subscriptions = [{"type": event_name} for event_name in _GLOBAL_EVENT_NAMES] - subscriptions.extend( - {"type": "pane.agent_status_changed", "pane_id": pane_id} - for pane_id in self._subscription_pane_ids - ) - if self._turn_api_supported: - subscriptions.extend( - {"type": "pane.turn_completed", "pane_id": pane_id} - for pane_id in self._subscription_pane_ids - ) - params = {"subscriptions": subscriptions} - if hasattr(client, "subscribe"): - try: - return client.subscribe( - self.subscribe_method, - params, - timeout=self.config.herdr_timeout_seconds, - event_timeout=self.config.herdr_timeout_seconds, - ) - except TypeError: - return client.subscribe(self.subscribe_method, params) - except HerdrErrorResponse as exc: - if not exc.uncorrelated: - raise - if hasattr(client, "close"): - client.close() - if hasattr(client, "connect"): - client.connect() - return self._subscribe_legacy_event_stream(client) - if hasattr(client, "events_subscribe"): - try: - return client.events_subscribe( - _HERDR_074_EVENT_NAMES, - timeout=self.config.herdr_timeout_seconds, - event_timeout=self.config.herdr_timeout_seconds, - ) - except TypeError: - return client.events_subscribe(_HERDR_074_EVENT_NAMES) - try: - return client.subscribe( - self.subscribe_method, - params, - timeout=self.config.herdr_timeout_seconds, - event_timeout=self.config.herdr_timeout_seconds, - ) - except TypeError: - return client.subscribe(self.subscribe_method, params) - - def _subscribe_legacy_event_stream(self, client: Any) -> Any: - # Preserve 0.7.4's proven compatibility request: its status event is - # parameterized by pane, while unrelated global event variants tolerate - # the same pane_id field. Replay-only events remain excluded exactly as - # before so reconnecting does not synthesize focus/detection activity. - if self._subscription_pane_ids: - subscriptions = [ - {"pane_id": pane_id, "type": event_name} - for pane_id in self._subscription_pane_ids - for event_name in _HERDR_074_PANE_SCOPED_FALLBACK_EVENT_NAMES - ] - else: - # An empty installation has no pane id with which to build the - # pane-scoped compatibility request. Fall back to the pre-0.7.5 - # global shape instead of retrying the rejected mixed shape on - # every reconnect. - subscriptions = [ - {"type": event_name} for event_name in _HERDR_074_EVENT_NAMES - ] - params = {"subscriptions": subscriptions} - try: - return client.subscribe( - self.subscribe_method, - params, - timeout=self.config.herdr_timeout_seconds, - event_timeout=self.config.herdr_timeout_seconds, - ) - except TypeError: - return client.subscribe(self.subscribe_method, params) - - def _workers_with_closed_missing( - self, - previous_workers: Sequence[Worker], - current_workers: Sequence[Worker], - *, - bound_worker_ids: set[str] | None = None, - ) -> list[Worker]: - current_by_id = {worker.id: worker for worker in current_workers} - merged = list(current_workers) - for worker in previous_workers: - if worker.id in current_by_id: - continue - if bound_worker_ids is not None and worker.id not in bound_worker_ids: - # A missing worker with no live binding is a phantom (event - # projections that never matched a binding); dropping it here - # keeps it from riding along as "closed" forever. - continue - merged.append(_closed_worker(worker)) - return merged - - def queue_event_envelope(self, envelope: Mapping[str, Any], *, flush: bool | None = None) -> bool: - event = normalize_event(envelope) - if event is None: - return False - with self._lock: - if ( - event.producer_identity is not None - and self._is_duplicate_producer_identity(event.producer_identity) - ): - return False - # Confirmed Herdr envelopes are idless. Every such event is queued; - # current-state idempotence, not historical content, prevents side effects. - self._pending_events.append(event) - self._last_event_at = utc_timestamp() - should_flush = self.debounce_seconds <= 0 if flush is None else flush - if should_flush: - self.flush() - return True - - def _is_duplicate_producer_identity(self, identity: HerdrProducerIdentity) -> bool: - if identity in self._producer_dedupe: - self._producer_dedupe.move_to_end(identity) - return True - return any(event.producer_identity == identity for event in self._pending_events) - - def _commit_producer_identities(self, events: Sequence[NormalizedHerdrEvent]) -> None: - for event in events: - identity = event.producer_identity - if identity is None: - continue - self._producer_dedupe[identity] = None - self._producer_dedupe.move_to_end(identity) - while len(self._producer_dedupe) > self.dedupe_size: - self._producer_dedupe.popitem(last=False) - - def flush(self) -> None: - # Draining, application, persistence, and producer-ID commitment share - # one lock scope so later batches cannot overtake an earlier flush. - notify_turn_refresh = False - completed_turns: list[HerdrTurnCompletionRecord] = [] - with self._lock: - if not self._pending_events: - return - events = list(self._pending_events) - accepted_at = utc_timestamp() - self._pending_events.clear() - has_producer_identity = any(event.producer_identity is not None for event in events) - has_turn_refresh_event = any( - event.name in _TURN_REFRESH_EVENT_NAMES for event in events - ) - notify_turn_refresh = has_turn_refresh_event - for event in events: - if event.name != "pane.turn_completed": - continue - try: - completed_turns.append(_turn_completion_record(event.payload)) - except HerdrEnvelopeError: - self._record_turn_diagnostic( - "herdr_turn_completion_record_quarantined", - _first_text(event.payload, ("pane_id", "paneId")) - or "unknown", - ) - try: - self._event_continuity_revalidated = False - changed = False - for event in events: - changed = self._apply_event(event) or changed - # Producer identities become durable only after this barrier. - # It also persists dirty memory when a failed first attempt made - # a retry appear idempotent before any snapshot reached storage. - if changed or has_producer_identity: - self._persist_current_state(observed_at=accepted_at) - self._commit_producer_identities(events) - except (HerdrContinuityUnavailableError, InstallationKeyError): - self._mark_unhealthy("continuity_unavailable") - finally: - self._event_continuity_revalidated = False - for record in completed_turns: - self._process_turn_record(record) - if notify_turn_refresh: - self._notify_turn_refresh() - - def _apply_event(self, event: NormalizedHerdrEvent) -> bool: - if event.name in _SPACE_EVENT_NAMES: - status = "closed" if event.name == "workspace.closed" else None - return self._apply_space_event(event.payload, status=status) - if event.name in _WORKTREE_EVENT_NAMES: - status = "closed" if event.name == "worktree.removed" else None - return self._apply_worktree_event(event.payload, status=status) - if event.name in _PANE_WORKER_EVENT_NAMES: - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "pane", - "agent", - "worker", - allow_top_level_pane_info=True, - ) - if not _pane_has_agent(item) and self._match_binding(item) is None: - return False - return self._upsert_worker_from_item( - item, - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name == "pane.agent_detected": - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "agent", - "worker", - "pane", - ) - return self._upsert_worker_from_item( - item, - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name == "pane.agent_status_changed": - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "agent", - "worker", - "pane", - ) - raw_status = _first_text(item, ("status", "agent_status", "state", "phase")) - return self._upsert_worker_from_item( - item, - status=normalize_status(raw_status), - update_binding=( - pane_info_observed - and _has_authoritative_identity_tuple(item) - and _has_authoritative_binding_target(item) - ), - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name == "pane.moved": - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "pane", - allow_top_level_pane_info=True, - ) - return self._apply_pane_moved( - item, - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name in _CLOSED_EVENT_NAMES: - item, pane_info_observed = _pane_event_payload_with_provenance( - event.payload, - "pane", - allow_top_level_pane_info=True, - ) - return self._apply_pane_closed( - item, - reason=event.name.replace(".", "_"), - pane_info_observed=pane_info_observed, - identity_source=f"event:{event.name}", - ) - if event.name == "pane.output_matched": - return False - return False - - def _apply_space_event(self, payload: Mapping[str, Any], *, status: str | None = None) -> bool: - item = _entity_payload(payload, "workspace", "space") - direct_name_hint = _first_text(item, ("label", "name", "title")) - rename_hint = _first_text(item, ("new_name", "newName")) - if _first_text(item, ("workspace_id", "space_id", "id", "slug", "name", "label", "title")) is None: - return False - name_hint = direct_name_hint or rename_hint - if rename_hint and direct_name_hint is None: - item["name"] = rename_hint - if status is not None: - item["status"] = status - spaces = _spaces_from_payload([item] if item else []) - if not spaces: - return False - space = spaces[0] - existing = self._spaces.get(space.id) - if existing is not None and status is None: - meta = dict(existing.meta) - meta.update(space.meta) - space = Space( - id=existing.id, - name=space.name if name_hint else existing.name, - status=space.status, - meta=meta, - updated_at=space.updated_at or utc_timestamp(), - status_line=space.status_line or existing.status_line, - ) - if existing is not None and space.fingerprint == existing.fingerprint: - return False - self._spaces[space.id] = space - return True - - def _apply_worktree_event(self, payload: Mapping[str, Any], *, status: str | None = None) -> bool: - item = _entity_payload(payload, "workspace", "space", "worktree") - workspace_id = _first_text(item, ("workspace_id", "space_id")) - if workspace_id is None or workspace_id not in self._spaces: - return False - item.setdefault("id", workspace_id) - if status is not None: - item["status"] = status - return self._apply_space_event({"workspace": item}, status=status) - - def _event_worker_and_binding( - self, - item: Mapping[str, Any], - *, - status: str | None = None, - pane_info_observed: bool = False, - identity_source: str = "event", - ) -> tuple[Worker | None, WorkerBinding | None, WorkerBinding | None]: - try: - record = _worker_record_from_item( - item, - self.config, - pane_info_observed=pane_info_observed, - identity_source=identity_source, - ) - if pane_info_observed and canonical_herdr_pane_identity( - record.workspace_id, - record.pane_id, - ) is None: - matched_owner = self._match_binding(item) - existing_owner = ( - self._workers.get(matched_owner.worker_id) - if matched_owner is not None - else None - ) - if ( - existing_owner is not None - and _authenticated_local_stable_key(existing_owner) is not None - ): - raise HerdrContinuityUnavailableError( - "Herdr event PaneInfo has no canonical public pane identity" - ) - workers, bindings = _workers_and_bindings_from_records( - self.config, - [record], - stored_bindings=self._current_bindings(), - require_authenticated_continuity=True, - ) - except (HerdrContinuityUnavailableError, InstallationKeyError): - raise - except Exception: - return None, None, self._match_binding(item) - worker = workers[0] if workers else None - binding = bindings[0] if bindings else None - matched_binding = self._match_binding(item) - if matched_binding is not None and worker is not None: - worker = _worker_copy(worker, worker_id=matched_binding.worker_id) - if worker is not None and status is not None: - worker = _worker_copy(worker, status=status) - return worker, binding, matched_binding - - @staticmethod - def _add_owner( - owners: dict[str, set[str]], - value: str | None, - worker_id: str, - ) -> None: - if value: - owners.setdefault(value, set()).add(worker_id) - - def _remove_owner(self, worker_id: str) -> None: - for owners in ( - self._pane_owners, - self._terminal_owners, - self._session_owners, - ): - for value in list(owners): - owner_ids = owners[value] - owner_ids.discard(worker_id) - if not owner_ids: - owners.pop(value, None) - - def _remember_item_owner( - self, - item: Mapping[str, Any], - worker_id: str, - *, - replace: bool, - ) -> None: - if replace: - self._remove_owner(worker_id) - agent_session = _safe_mapping(_field_value(item, "agent_session")) - session_id = ( - _first_text(agent_session, ("value", "id")) - or _first_text(item, ("session_id", "sessionId")) - ) - self._add_owner( - self._pane_owners, - _first_text(item, ("pane_id", "paneId")), - worker_id, - ) - self._add_owner( - self._terminal_owners, - _first_text(item, ("terminal_id", "terminalId")), - worker_id, - ) - self._add_owner(self._session_owners, session_id, worker_id) - - def _replace_ownership_maps( - self, - records: Sequence[Any], - bindings: Sequence[WorkerBinding], - ) -> None: - self._pane_terminals = {} - self._pane_owners = {} - self._terminal_owners = {} - self._session_owners = {} - worker_ids_by_private: dict[str, set[str]] = {} - for binding in bindings: - worker_ids_by_private.setdefault( - binding.private_fingerprint, - set(), - ).add(binding.worker_id) - if binding.target_kind == "pane_id": - self._add_owner( - self._pane_owners, - binding.target_value, - binding.worker_id, - ) - if binding.target_kind == "terminal_id": - self._add_owner( - self._terminal_owners, - binding.target_value, - binding.worker_id, - ) - if binding.turn_target_value: - self._add_owner( - self._session_owners, - binding.turn_target_value, - binding.worker_id, - ) - for record in records: - if not record.pane_info_observed: - continue - owner_ids = worker_ids_by_private.get( - record.private_fingerprint, - set(), - ) - if len(owner_ids) != 1: - continue - worker_id = next(iter(owner_ids)) - observed_pane_id = record.observed_pane_id or record.pane_id - if observed_pane_id and record.terminal_id: - self._pane_terminals[observed_pane_id] = record.terminal_id - self._add_owner( - self._pane_owners, - observed_pane_id, - worker_id, - ) - self._add_owner( - self._terminal_owners, - record.terminal_id, - worker_id, - ) - self._add_owner( - self._session_owners, - record.agent_session_id, - worker_id, - ) - - def _ownership_worker_ids( - self, - item: Mapping[str, Any], - observed_binding: WorkerBinding | None = None, - ) -> set[str]: - owner_ids = { - binding.worker_id - for binding in self._matching_bindings(item) - if binding.worker_id - } - if observed_binding is not None: - owner_ids.update( - binding.worker_id - for binding in self._bindings.values() - if binding.worker_id - and binding.target_value - == observed_binding.target_value - ) - pane_id = _first_text(item, ("pane_id", "paneId")) - terminal_id = _first_text(item, ("terminal_id", "terminalId")) - agent_session = _safe_mapping(_field_value(item, "agent_session")) - session_id = ( - _first_text(agent_session, ("value", "id")) - or _first_text(item, ("session_id", "sessionId")) - ) - owner_ids.update(self._pane_owners.get(pane_id or "", ())) - owner_ids.update(self._terminal_owners.get(terminal_id or "", ())) - owner_ids.update(self._session_owners.get(session_id or "", ())) - return owner_ids - - def _target_worker_ids( - self, - target_kind: str, - target_value: str, - ) -> set[str]: - owner_ids = { - binding.worker_id - for binding in self._bindings.values() - if binding.worker_id - and ( - binding.target_value == target_value - or ( - binding.turn_target_kind == target_kind - and binding.turn_target_value == target_value - ) - ) - } - if target_kind == "pane_id": - owner_ids.update(self._pane_owners.get(target_value, ())) - if target_kind == "terminal_id": - owner_ids.update(self._terminal_owners.get(target_value, ())) - return owner_ids - - - def _matching_bindings( - self, - item: Mapping[str, Any], - *, - old_first: bool = False, - ) -> list[WorkerBinding]: - pairs = _target_pairs_from_item(item, old_first=old_first) - mapped_pairs = [ - ("terminal_id", self._pane_terminals[value]) - for kind, value in pairs - if kind == "pane_id" and value in self._pane_terminals - ] - agent_session = _safe_mapping(_field_value(item, "agent_session")) - session_id = ( - _first_text(agent_session, ("value", "id")) - or _first_text(item, ("session_id", "sessionId")) - ) - keys = set([*pairs, *mapped_pairs]) - if session_id: - keys.add(("agent_session", session_id)) - if not keys: - return [] - return [ - binding - for binding in self._bindings.values() - if _binding_target(binding) in keys - or ( - str(binding.turn_target_kind or ""), - str(binding.turn_target_value or ""), - ) - in keys - or ( - "agent_session", - str(binding.turn_target_value or ""), - ) - in keys - ] - - def _fail_closed_ownership(self, worker_ids: set[str]) -> bool: - """Remove continuity and routing from every current ambiguous owner.""" - if not worker_ids: - return False - if ( - self._health.outcome == "continuity_unavailable" - and not self._event_continuity_revalidated - ): - return False - changed = False - binding_updates: list[WorkerBinding] = [] - bindings_by_worker: dict[str, list[WorkerBinding]] = {} - for private_fingerprint, binding in list(self._bindings.items()): - if binding.worker_id not in worker_ids: - continue - bindings_by_worker.setdefault(binding.worker_id, []).append(binding) - ambiguous = WorkerBinding( - host_id=binding.host_id, - worker_id=binding.worker_id, - worker_fingerprint=binding.worker_fingerprint, - backend=binding.backend, - target_kind=binding.target_kind, - target_value=binding.target_value, - turn_target_kind=None, - turn_target_value=None, - sendable=False, - reason="ambiguous_pane_match", - observed_at=utc_timestamp(), - expires_at=binding.expires_at, - private_fingerprint=binding.private_fingerprint, - ) - if _binding_state_equal(binding, ambiguous): - continue - self._bindings[private_fingerprint] = ambiguous - binding_updates.append(ambiguous) - changed = True - - for worker_id in worker_ids: - worker = self._workers.get(worker_id) - if worker is None: - continue - target = worker.backend_target - if not isinstance(target, Mapping): - candidates = bindings_by_worker.get(worker_id, []) - target = candidates[0].backend_target() if candidates else None - ambiguous_target = None - if isinstance(target, Mapping): - kind = str(target.get("kind") or "") - value = str(target.get("value") or "") - if kind and value: - ambiguous_target = { - "kind": kind, - "value": value, - "sendable": False, - "reason": "ambiguous_pane_match", - } - ambiguous_worker = _worker_copy( - worker, - meta=_strip_stable_key_fields(worker.meta), - backend_target=ambiguous_target, - ) - if _worker_state_equal(worker, ambiguous_worker): - continue - self._workers[worker_id] = ambiguous_worker - changed = True - if binding_updates: - upsert_worker_bindings(self.db_path, binding_updates) - return changed - - - def _upsert_worker_from_item( - self, - item: Mapping[str, Any], - *, - status: str | None = None, - update_binding: bool = True, - pane_info_observed: bool = False, - identity_source: str = "event", - ) -> bool: - if not item: - return False - if not _has_public_worker_identity(item) and self._match_binding(item) is None: - return False - worker, binding, matched_binding = self._event_worker_and_binding( - item, - status=status, - pane_info_observed=pane_info_observed, - identity_source=identity_source, - ) - if worker is None: - return False - - authoritative_identity = ( - pane_info_observed and _has_authoritative_identity_tuple(item) - ) - stable_owner_reused = False - if authoritative_identity: - observed_stable_key = _authenticated_local_stable_key(worker) - stable_owner_ids = ( - { - current.id - for current in self._workers.values() - if _authenticated_local_stable_key(current) - == observed_stable_key - } - if observed_stable_key is not None - else set() - ) - target_owner_ids = self._ownership_worker_ids( - item, - binding, - ) - conflicting_owner_ids: set[str] = set() - if len(stable_owner_ids) > 1 or len(target_owner_ids) > 1: - conflicting_owner_ids.update(stable_owner_ids) - conflicting_owner_ids.update(target_owner_ids) - elif len(stable_owner_ids) == 1: - stable_owner_id = next(iter(stable_owner_ids)) - other_target_owners = target_owner_ids - {stable_owner_id} - if other_target_owners: - conflicting_owner_ids.add(stable_owner_id) - conflicting_owner_ids.update(other_target_owners) - else: - worker = _worker_copy( - worker, - worker_id=stable_owner_id, - ) - stable_owner_reused = True - elif len(target_owner_ids) == 1: - target_owner_id = next(iter(target_owner_ids)) - target_owner = self._workers.get(target_owner_id) - target_stable_key = ( - _authenticated_local_stable_key(target_owner) - if target_owner is not None - else None - ) - observed_pane_id = _first_text( - item, - ("pane_id", "paneId"), - ) - observed_canonical_identity = canonical_herdr_pane_identity( - _first_text(item, ("workspace_id", "workspaceId")), - observed_pane_id, - ) - same_pane_owner_ids = self._pane_owners.get( - observed_pane_id or "", - set(), - ) - target_is_ambiguous = any( - current_binding.worker_id == target_owner_id - and current_binding.reason == "ambiguous_pane_match" - for current_binding in self._bindings.values() - ) - if target_is_ambiguous or ( - observed_stable_key is None - and ( - observed_canonical_identity is None - or target_owner_id not in same_pane_owner_ids - ) - ) or ( - observed_stable_key is not None - and target_stable_key is not None - and target_stable_key != observed_stable_key - ): - conflicting_owner_ids.add(target_owner_id) - else: - worker = _worker_copy( - worker, - worker_id=target_owner_id, - ) - stable_owner_reused = True - if conflicting_owner_ids: - return self._fail_closed_ownership( - conflicting_owner_ids - ) - else: - compatibility_owner_ids = self._ownership_worker_ids( - item, - binding, - ) - compatibility_identity = canonical_herdr_pane_identity( - _first_text(item, ("workspace_id", "workspaceId")), - _first_text(item, ("pane_id", "paneId")), - ) - pane_owner_ids = ( - self._pane_owners.get(compatibility_identity[1], set()) - if compatibility_identity is not None - else set() - ) - if ( - matched_binding is None - and len(compatibility_owner_ids) == 1 - and compatibility_owner_ids <= pane_owner_ids - ): - compatibility_owner_id = next(iter(compatibility_owner_ids)) - owner_bindings = [ - current_binding - for current_binding in self._bindings.values() - if current_binding.worker_id == compatibility_owner_id - ] - if len(owner_bindings) == 1: - # Scalar events can locate an already-authenticated owner - # through the private pane map, but cannot replace that - # owner's identity or binding with event-only fields. - matched_binding = owner_bindings[0] - matched_owner_ids = ( - {matched_binding.worker_id} - if matched_binding is not None - else set() - ) - if compatibility_owner_ids - matched_owner_ids: - return self._fail_closed_ownership( - compatibility_owner_ids - ) - - existing = self._workers.get(worker.id) - if existing is None and matched_binding is not None: - existing = self._workers.get(matched_binding.worker_id) - if matched_binding is not None and not stable_owner_reused: - worker = _worker_copy(worker, worker_id=matched_binding.worker_id) - worker = _merge_worker_update( - existing, - worker, - status=status, - preserve_existing_continuity=not authoritative_identity, - ) - if not authoritative_identity and matched_binding is not None: - worker = _worker_copy( - worker, - backend_target=matched_binding.backend_target(), - ) - if self._would_exceed_worker_cap(worker, existing=existing): - self._mark_worker_cap_exceeded_locked(_observed_worker_count(list(self._workers.values())) + 1) - return False - changed = existing is None or not _worker_state_equal(existing, worker) - if changed: - self._workers[worker.id] = worker - else: - assert existing is not None - worker = existing - if update_binding and binding is not None: - if stable_owner_reused: - stale_private_fingerprints = [ - private_fingerprint - for private_fingerprint, current_binding in self._bindings.items() - if current_binding.worker_id == worker.id - and private_fingerprint != binding.private_fingerprint - ] - if stale_private_fingerprints: - expire_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - private_fingerprints=stale_private_fingerprints, - reason="identity_replaced", - ) - for private_fingerprint in stale_private_fingerprints: - self._bindings.pop(private_fingerprint, None) - changed = True - binding = self._binding_with_worker(binding, worker) - elif matched_binding is not None and ( - not authoritative_identity - or binding.private_fingerprint - != matched_binding.private_fingerprint - ): - binding = self._binding_with_worker(matched_binding, worker) - else: - binding = self._binding_with_worker(binding, worker) - current_binding = self._bindings.get(binding.private_fingerprint) - if current_binding is None or not _binding_state_equal(current_binding, binding): - self._bindings[binding.private_fingerprint] = binding - upsert_worker_bindings(self.db_path, [binding]) - changed = True - if authoritative_identity: - self._remember_item_owner( - item, - worker.id, - replace=update_binding, - ) - self._note_pane_terminal(item) - if _authenticated_local_stable_key(worker) is not None: - if self._health.outcome == "continuity_unavailable": - changed = True - self._event_continuity_revalidated = True - return changed - - def _binding_with_worker(self, binding: WorkerBinding, worker: Worker) -> WorkerBinding: - return WorkerBinding( - host_id=binding.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend=binding.backend, - target_kind=binding.target_kind, - target_value=binding.target_value, - turn_target_kind=binding.turn_target_kind, - turn_target_value=binding.turn_target_value, - sendable=binding.sendable, - reason=binding.reason, - observed_at=utc_timestamp(), - expires_at=None, - private_fingerprint=binding.private_fingerprint, - ) - - def _match_binding(self, item: Mapping[str, Any], *, old_first: bool = False) -> WorkerBinding | None: - pairs = _target_pairs_from_item(item, old_first=old_first) - if not pairs: - return None - binding_by_target = {_binding_target(binding): binding for binding in self._bindings.values()} - # Event payloads often carry only a pane id while stored bindings target - # a terminal id; fall back to the turn target so a known pane never - # spawns a duplicate re-lettered worker. - for binding in self._bindings.values(): - turn_key = (str(binding.turn_target_kind or ""), str(binding.turn_target_value or "")) - if turn_key[0] and turn_key[1]: - binding_by_target.setdefault(turn_key, binding) - # Translate pane ids to the terminal ids remembered from the last - # reconcile: agent kinds whose turn target is not a pane id (codex - # session ids) would otherwise never match a pane-id-only event. - mapped_pairs = [ - ("terminal_id", self._pane_terminals[value]) - for kind, value in pairs - if kind == "pane_id" and value in self._pane_terminals - ] - for pair in [*pairs, *mapped_pairs]: - binding = binding_by_target.get(pair) - if binding is not None: - return binding - return None - - def _previous_ownership_worker_ids( - self, - item: Mapping[str, Any], - ) -> set[str]: - previous_pane_id = _first_text( - item, - ( - "old_pane_id", - "previous_pane_id", - "from_pane_id", - "source_pane_id", - ), - ) - previous_terminal_id = _first_text( - item, - ( - "old_terminal_id", - "previous_terminal_id", - "from_terminal_id", - "source_terminal_id", - ), - ) - owner_ids = set(self._pane_owners.get(previous_pane_id or "", ())) - owner_ids.update( - self._terminal_owners.get(previous_terminal_id or "", ()) - ) - if previous_pane_id and previous_pane_id in self._pane_terminals: - previous_terminal_id = self._pane_terminals[previous_pane_id] - owner_ids.update( - self._terminal_owners.get(previous_terminal_id, ()) - ) - previous_keys = { - ("pane_id", previous_pane_id or ""), - ("terminal_id", previous_terminal_id or ""), - } - owner_ids.update( - binding.worker_id - for binding in self._bindings.values() - if binding.worker_id - and ( - _binding_target(binding) in previous_keys - or ( - str(binding.turn_target_kind or ""), - str(binding.turn_target_value or ""), - ) - in previous_keys - ) - ) - return owner_ids - - - def _note_pane_terminal(self, item: Mapping[str, Any]) -> None: - pane_id = _first_text(item, ("pane_id", "paneId")) - terminal_id = _first_text(item, ("terminal_id", "terminalId")) - if pane_id and terminal_id: - self._pane_terminals[pane_id] = terminal_id - - def _apply_pane_moved( - self, - item: Mapping[str, Any], - *, - pane_info_observed: bool = False, - identity_source: str = "event:pane.moved", - ) -> bool: - authoritative_identity = ( - pane_info_observed and _has_authoritative_identity_tuple(item) - ) - observed_worker, observed_binding, _matched = self._event_worker_and_binding( - item, - pane_info_observed=pane_info_observed, - identity_source=identity_source, - ) - source_owner_ids = self._previous_ownership_worker_ids(item) - if len(source_owner_ids) > 1: - return self._fail_closed_ownership(source_owner_ids) - if not source_owner_ids: - return False - source_owner_id = next(iter(source_owner_ids)) - existing = self._workers.get(source_owner_id) - if existing is None: - return False - source_bindings = [ - binding - for binding in self._bindings.values() - if binding.worker_id == source_owner_id - ] - new_target = _new_move_target(item) - if new_target is not None: - destination_owner_ids = self._target_worker_ids(*new_target) - conflicting_destination_ids = destination_owner_ids - { - source_owner_id - } - if conflicting_destination_ids: - return self._fail_closed_ownership( - {source_owner_id, *conflicting_destination_ids} - ) - - if authoritative_identity: - if observed_worker is None: - return False - observed_stable_key = _authenticated_local_stable_key( - observed_worker - ) - destination_owner_ids = self._ownership_worker_ids( - item, - observed_binding, - ) - if observed_stable_key is not None: - destination_owner_ids.update( - current.id - for current in self._workers.values() - if _authenticated_local_stable_key(current) - == observed_stable_key - ) - conflicting_owner_ids = destination_owner_ids - { - source_owner_id - } - if conflicting_owner_ids: - return self._fail_closed_ownership( - {source_owner_id, *conflicting_owner_ids} - ) - worker = _merge_worker_update( - existing, - _worker_copy( - observed_worker, - worker_id=source_owner_id, - ), - preserve_existing_continuity=False, - ) - else: - worker = existing - if observed_worker is not None: - worker = _merge_worker_update( - existing, - _worker_copy( - observed_worker, - worker_id=source_owner_id, - ), - preserve_existing_continuity=True, - ) - - if self._would_exceed_worker_cap(worker, existing=existing): - self._mark_worker_cap_exceeded_locked( - _observed_worker_count(list(self._workers.values())) + 1 - ) - return False - - if authoritative_identity and observed_binding is not None: - if len(source_bindings) != 1: - return self._fail_closed_ownership({source_owner_id}) - old_binding = source_bindings[0] - moved_binding = WorkerBinding( - host_id=observed_binding.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend=observed_binding.backend, - target_kind=observed_binding.target_kind, - target_value=observed_binding.target_value, - turn_target_kind=observed_binding.turn_target_kind, - turn_target_value=observed_binding.turn_target_value, - sendable=observed_binding.sendable, - reason=observed_binding.reason, - observed_at=utc_timestamp(), - expires_at=None, - private_fingerprint=old_binding.private_fingerprint, - ) - else: - if new_target is None or len(source_bindings) != 1: - return False - old_binding = source_bindings[0] - target_kind, target_value = new_target - moved_binding = WorkerBinding( - host_id=old_binding.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend=old_binding.backend, - target_kind=target_kind, - target_value=target_value, - turn_target_kind=old_binding.turn_target_kind, - turn_target_value=( - target_value - if old_binding.turn_target_kind == "pane_id" - and target_kind == "pane_id" - else old_binding.turn_target_value - ), - sendable=old_binding.sendable, - reason=old_binding.reason, - observed_at=utc_timestamp(), - expires_at=None, - private_fingerprint=old_binding.private_fingerprint, - ) - - stale_private_fingerprints = [ - binding.private_fingerprint - for binding in source_bindings - if binding.private_fingerprint - != moved_binding.private_fingerprint - ] - if stale_private_fingerprints: - expire_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - private_fingerprints=stale_private_fingerprints, - reason="identity_replaced", - ) - for private_fingerprint in stale_private_fingerprints: - self._bindings.pop(private_fingerprint, None) - - worker = _worker_copy( - worker, - backend_target=moved_binding.backend_target(), - ) - moved_binding = self._binding_with_worker(moved_binding, worker) - self._workers[worker.id] = worker - self._bindings[moved_binding.private_fingerprint] = moved_binding - upsert_worker_bindings(self.db_path, [moved_binding]) - - previous_pane_ids: set[str] = set() - previous_pane_id = _first_text( - item, - ( - "old_pane_id", - "previous_pane_id", - "from_pane_id", - "source_pane_id", - ), - ) - if previous_pane_id: - previous_pane_ids.add(previous_pane_id) - previous_terminal_id = _first_text( - item, - ( - "old_terminal_id", - "previous_terminal_id", - "from_terminal_id", - "source_terminal_id", - ), - ) - if previous_terminal_id: - previous_pane_ids.update( - pane_id - for pane_id, terminal_id in self._pane_terminals.items() - if terminal_id == previous_terminal_id - and source_owner_id in self._pane_owners.get(pane_id, ()) - ) - current_pane_id = _first_text(item, ("pane_id", "paneId")) - for source_pane_id in previous_pane_ids: - if source_pane_id != current_pane_id: - self._pane_terminals.pop(source_pane_id, None) - - self._remove_owner(worker.id) - if authoritative_identity: - self._remember_item_owner( - item, - worker.id, - replace=False, - ) - self._note_pane_terminal(item) - if _authenticated_local_stable_key(worker) is not None: - self._event_continuity_revalidated = True - else: - if moved_binding.target_kind == "pane_id": - self._add_owner( - self._pane_owners, - moved_binding.target_value, - worker.id, - ) - if moved_binding.target_kind == "terminal_id": - self._add_owner( - self._terminal_owners, - moved_binding.target_value, - worker.id, - ) - if moved_binding.turn_target_value: - self._add_owner( - self._session_owners, - moved_binding.turn_target_value, - worker.id, - ) - return True - - def _apply_pane_closed( - self, - item: Mapping[str, Any], - *, - reason: str, - pane_info_observed: bool = False, - identity_source: str = "event", - ) -> bool: - observed_worker: Worker | None = None - matched_binding: WorkerBinding | None = None - observed_binding: WorkerBinding | None = None - if pane_info_observed: - observed_worker, observed_binding, matched_binding = self._event_worker_and_binding( - item, - status="closed", - pane_info_observed=True, - identity_source=identity_source, - ) - observed_stable_key = ( - _authenticated_local_stable_key(observed_worker) - if observed_worker is not None - else None - ) - if observed_stable_key is not None: - stable_owner_ids = { - current.id - for current in self._workers.values() - if _authenticated_local_stable_key(current) == observed_stable_key - } - target_owner_ids = self._ownership_worker_ids(item, observed_binding) - if len(stable_owner_ids | target_owner_ids) > 1: - return False - binding = matched_binding or self._match_binding(item) - worker: Worker | None = None - if binding is not None: - worker = self._workers.get(binding.worker_id) - if worker is None: - if binding is None and not _has_public_worker_identity(item): - return False - if observed_worker is None: - observed_worker, _event_binding, matched_binding = self._event_worker_and_binding( - item, - status="closed", - pane_info_observed=False, - identity_source=identity_source, - ) - if matched_binding is not None: - binding = matched_binding - worker = observed_worker - if worker is None: - return False - if binding is None and self._would_exceed_worker_cap(worker, existing=self._workers.get(worker.id)): - self._mark_worker_cap_exceeded_locked(_observed_worker_count(list(self._workers.values())) + 1) - return False - closed = _closed_worker(worker) - current = self._workers.get(closed.id) - changed = current is None or not _worker_state_equal(current, closed) - if changed: - self._workers[closed.id] = closed - if binding is not None: - expire_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - private_fingerprints=[binding.private_fingerprint], - now=utc_timestamp(), - reason=reason, - ) - self._bindings.pop(binding.private_fingerprint, None) - changed = True - elif changed: - expire_worker_bindings( - self.db_path, - self.config.host_id, - backend=BACKEND_NAME, - worker_id=closed.id, - now=utc_timestamp(), - reason=reason, - ) - if ( - pane_info_observed - and observed_worker is not None - and _authenticated_local_stable_key(observed_worker) is not None - ): - if self._health.outcome == "continuity_unavailable": - changed = True - self._event_continuity_revalidated = True - return changed - - def _persist_current_state(self, *, observed_at: str | None = None) -> Snapshot: - accepted_at = observed_at or utc_timestamp() - spaces = list(self._spaces.values()) - workers = list(self._workers.values()) - if ( - self._health.outcome == "continuity_unavailable" - and not self._event_continuity_revalidated - ): - health = self._health.to_backend_health(spaces=spaces, workers=workers) - else: - outcome = ( - "healthy_non_empty" - if spaces or _observed_worker_count(workers) - else "empty_healthy" - ) - health = herdr_backend_health( - outcome, - observed_at=accepted_at, - spaces=spaces, - workers=workers, - ) - snapshot = project_from_observations( - self.config, - spaces=spaces, - workers=workers, - backend_health=[health], - ) - self._save_snapshot( - snapshot, - observation=SnapshotObservationContext( - authority="positive" if health.status == "healthy" else "none", - observed_at=health.observed_at or accepted_at, - ), - ) - self._last_snapshot_at = snapshot.updated_at - self._health = HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or snapshot.updated_at, - message=health.message, - ) - return snapshot - - def _would_exceed_worker_cap(self, worker: Worker, *, existing: Worker | None = None) -> bool: - if worker.status == "closed": - return False - previous = existing if existing is not None else self._workers.get(worker.id) - if previous is not None and previous.status != "closed": - return False - return _observed_worker_count(list(self._workers.values())) + 1 > self.max_workers - - def _mark_worker_cap_exceeded_locked(self, observed_workers: int) -> Snapshot: - now = utc_timestamp() - previous = latest_snapshot(self.db_path, self.config.host_id) - spaces = list(previous.spaces) if previous is not None else list(self._spaces.values()) - workers = list(previous.workers) if previous is not None else list(self._workers.values()) - if ( - self._health.outcome == "continuity_unavailable" - and not self._event_continuity_revalidated - ): - health = self._health.to_backend_health(spaces=spaces, workers=workers) - else: - health = herdr_backend_health( - "worker_cap_exceeded", - observed_at=now, - message="Herdr observation exceeded the configured worker cap", - spaces=spaces, - workers=workers, - ) - snapshot = project_from_observations( - self.config, - spaces=spaces, - workers=workers, - backend_health=[health], - ) - self._save_snapshot( - snapshot, - observation=SnapshotObservationContext( - authority="none", - observed_at=health.observed_at or now, - ), - ) - self._spaces = {space.id: space for space in snapshot.spaces} - self._workers = {worker.id: worker for worker in snapshot.workers} - self._health = HerdrEventBackendHealth( - status=health.status, - outcome=health.outcome, - observed_at=health.observed_at or now, - message=health.message, - ) - self._last_cap_status_at = now - self._last_reconcile_at = now - self._last_snapshot_at = snapshot.updated_at - self._schedule_next_reconcile() - return snapshot - - def _mark_unhealthy(self, outcome: str) -> Snapshot: - with self._lock: - if ( - self._health.outcome == "continuity_unavailable" - and not self._event_continuity_revalidated - ): - health_state = self._health - else: - health_state = self._health_for(outcome) - self._health = health_state - self._event_continuity_revalidated = False - spaces = list(self._spaces.values()) - workers = list(self._workers.values()) - if not spaces and not workers: - snapshot = latest_snapshot(self.db_path, self.config.host_id) - if snapshot is not None: - spaces = list(snapshot.spaces) - workers = list(snapshot.workers) - health = health_state.to_backend_health(spaces=spaces, workers=workers) - snapshot = project_from_observations( - self.config, - spaces=spaces, - workers=workers, - backend_health=[health], - ) - self._save_snapshot( - snapshot, - observation=SnapshotObservationContext( - authority="none", - observed_at=health.observed_at or snapshot.updated_at, - ), - ) - self._last_snapshot_at = snapshot.updated_at - self._spaces = {space.id: space for space in snapshot.spaces} - self._workers = {worker.id: worker for worker in snapshot.workers} - return snapshot - - def _mark_unhealthy_safe(self, outcome: str) -> Snapshot | None: - try: - return self._mark_unhealthy(outcome) - finally: - self._ready.set() diff --git a/src/tendwire/backends/herdr_protocol.py b/src/tendwire/backends/herdr_protocol.py index a035b58..ff68069 100644 --- a/src/tendwire/backends/herdr_protocol.py +++ b/src/tendwire/backends/herdr_protocol.py @@ -9,7 +9,7 @@ import json import os import uuid -from collections.abc import Iterable, Mapping +from collections.abc import Mapping from pathlib import Path from typing import Any @@ -22,30 +22,6 @@ "HERDR_SESSION", ) -HERDR_EVENTS_SUBSCRIBE_METHOD = "events.subscribe" -HERDR_TURN_COMPLETED_EVENT_NAME = "pane.turn_completed" -HERDR_OFFICIAL_EVENT_NAMES = ( - "workspace.created", - "workspace.updated", - "workspace.renamed", - "workspace.closed", - "workspace.focused", - "pane.created", - "pane.closed", - "pane.updated", - "pane.focused", - "pane.moved", - "pane.exited", - "pane.agent_detected", - "pane.output_matched", - "pane.agent_status_changed", - "worktree.created", - "worktree.opened", - "worktree.removed", -) -HERDR_OFFICIAL_EVENT_NAME_SET = frozenset(HERDR_OFFICIAL_EVENT_NAMES) - - class HerdrProtocolError(Exception): """Base error for Herdr socket protocol failures.""" @@ -62,6 +38,10 @@ class HerdrEnvelopeError(HerdrProtocolError, ValueError): """Raised when a decoded JSON object is not a valid protocol envelope.""" +class HerdrFrameTooLargeError(HerdrProtocolError, ValueError): + """Raised when one JSON-line frame exceeds the fixed transport bound.""" + + class HerdrRequestIdMismatchError(HerdrEnvelopeError): """Raised when a server envelope is not correlated to the expected id.""" @@ -73,12 +53,9 @@ def __init__( self, error: Any, request_id: str, - *, - uncorrelated: bool = False, ) -> None: self.error = error self.request_id = request_id - self.uncorrelated = uncorrelated message = "Herdr returned an error response" if isinstance(error, Mapping): raw_message = error.get("message") @@ -170,43 +147,6 @@ def build_request( return {"id": request_id, "method": method, "params": dict(params)} -def _validate_event_subscription_name(name: Any) -> str: - if not isinstance(name, str): - raise HerdrEnvelopeError("Herdr event subscription names must be strings") - if not name: - raise HerdrEnvelopeError("Herdr event subscription names must not be empty") - if name.strip() != name or name not in HERDR_OFFICIAL_EVENT_NAME_SET: - raise HerdrEnvelopeError(f"unsupported Herdr event subscription {name!r}") - return name - - -def build_events_subscribe_params(event_names: Iterable[str] | str | None = None) -> dict[str, Any]: - """Return official events.subscribe params for validated Herdr event names.""" - if event_names is None: - names = HERDR_OFFICIAL_EVENT_NAMES - elif isinstance(event_names, str): - names = (event_names,) - else: - try: - names = tuple(event_names) - except TypeError as exc: - raise HerdrEnvelopeError("Herdr event subscriptions must be iterable") from exc - return {"subscriptions": [{"type": _validate_event_subscription_name(name)} for name in names]} - - -def build_events_subscribe_request( - event_names: Iterable[str] | str | None = None, - *, - request_id: str | None = None, -) -> dict[str, Any]: - """Build an official Herdr event subscription request envelope.""" - return build_request( - HERDR_EVENTS_SUBSCRIBE_METHOD, - build_events_subscribe_params(event_names), - request_id=request_id, - ) - - def frame_request(request: Mapping[str, Any]) -> bytes: """Encode one request object as UTF-8 JSON Lines.""" try: @@ -274,87 +214,20 @@ def is_response(envelope: Mapping[str, Any]) -> bool: return is_result_response(envelope) or is_error_response(envelope) -def is_event(envelope: Mapping[str, Any]) -> bool: - return "event" in envelope and "result" not in envelope and "error" not in envelope - - -def validate_response( - envelope: Mapping[str, Any], - *, - allow_uncorrelated_error: bool = False, - allow_uncorrelated_method_error: bool = False, -) -> dict[str, Any]: +def validate_response(envelope: Mapping[str, Any]) -> dict[str, Any]: """Validate a response envelope while tolerating unknown fields.""" if not is_response(envelope): raise HerdrEnvelopeError("Herdr response must contain exactly one of result or error") - # Herdr 0.7.5 emits ``{"id":"", "error":...}`` when subscription - # parameters fail schema validation. Only the subscription negotiation - # path may opt into that compatibility exception; ordinary requests remain - # strictly correlated. - error = envelope.get("error") - uncorrelated_subscription_error = ( - allow_uncorrelated_error - and is_error_response(envelope) - and envelope.get("id") == "" - and isinstance(error, Mapping) - and error.get("code") == "invalid_request" - and isinstance(error.get("message"), str) - and error["message"].startswith("invalid request:") - ) - # Stock Herdr 0.7.5 omits the id entirely when an internally tagged RPC - # variant is unknown. Only an explicit capability probe may opt into this - # narrower exception; ordinary requests remain strictly correlated. - uncorrelated_method_error = ( - allow_uncorrelated_method_error - and is_error_response(envelope) - and ("id" not in envelope or envelope.get("id") == "") - and isinstance(error, Mapping) - and error.get("code") == "invalid_request" - and isinstance(error.get("message"), str) - and error["message"].startswith("invalid request: unknown variant") - and "pane.turns" in error["message"] - ) - if not (uncorrelated_subscription_error or uncorrelated_method_error): - _validated_id(envelope) - return dict(envelope) - - -def validate_event(envelope: Mapping[str, Any]) -> dict[str, Any]: - """Validate an event envelope while preserving its raw data. - - The confirmed Herdr ``EventEnvelope`` consists of ``event`` and ``data``. - A generic ``id`` is tolerated as subscription correlation only; it is not - authoritative producer event identity. Unknown top-level fields remain - available for forward-compatible consumers. - """ - request_id = envelope.get("id") - if request_id is not None and (not isinstance(request_id, str) or not request_id): - raise HerdrEnvelopeError("Herdr event id must be a non-empty string when present") - event_name = envelope.get("event") - if not isinstance(event_name, str) or not event_name: - raise HerdrEnvelopeError("Herdr event name must be a non-empty string") - if not is_event(envelope): - raise HerdrEnvelopeError("Herdr event must not contain result or error fields") + _validated_id(envelope) return dict(envelope) -def validate_server_envelope( - envelope: Mapping[str, Any], - *, - allow_uncorrelated_error: bool = False, - allow_uncorrelated_method_error: bool = False, -) -> dict[str, Any]: - """Validate a decoded server response or event envelope.""" +def validate_server_envelope(envelope: Mapping[str, Any]) -> dict[str, Any]: + """Validate a decoded server response envelope.""" if is_response(envelope): - return validate_response( - envelope, - allow_uncorrelated_error=allow_uncorrelated_error, - allow_uncorrelated_method_error=allow_uncorrelated_method_error, - ) - if is_event(envelope): - return validate_event(envelope) + return validate_response(envelope) _validated_id(envelope) - raise HerdrEnvelopeError("Herdr envelope is neither a response nor an event") + raise HerdrEnvelopeError("Herdr envelope is not a response") def ensure_response_id(envelope: Mapping[str, Any], expected_id: str) -> None: diff --git a/src/tendwire/backends/herdr_socket.py b/src/tendwire/backends/herdr_socket.py index 1ed1b51..75739b4 100644 --- a/src/tendwire/backends/herdr_socket.py +++ b/src/tendwire/backends/herdr_socket.py @@ -1,31 +1,22 @@ -"""Inactive stdlib Herdr Unix socket client. - -This module is additive and is not imported by Tendwire's production -observation or CLI paths. It exposes a low-level JSON-line client plus thin -wrappers for the PR8-allowed Herdr methods only. -""" +"""Synchronous client for Herdr lifecycle discovery and ACP ownership.""" from __future__ import annotations import socket import time -from collections import deque -from collections.abc import Iterable, Iterator, Mapping +from collections.abc import Iterable, Mapping from typing import Any from .herdr_protocol import ( HerdrEnvelopeError, HerdrErrorResponse, + HerdrFrameTooLargeError, HerdrProtocolError, - HerdrRequestIdMismatchError, - HERDR_EVENTS_SUBSCRIBE_METHOD, - build_events_subscribe_params, build_request, ensure_response_id, error_payload, frame_request, is_error_response, - is_event, is_result_response, parse_json_line, resolve_socket_path, @@ -35,7 +26,7 @@ _DEFAULT_TIMEOUT_SECONDS = 5.0 _RECV_SIZE = 4096 -_MAX_PENDING_EVENTS = 1024 +_MAX_FRAME_BYTES = 8 * 1024 * 1024 class HerdrSocketError(HerdrProtocolError): @@ -54,39 +45,6 @@ class HerdrSocketConnectionError(HerdrSocketError, ConnectionError): """Raised when the Unix socket cannot be opened.""" -class HerdrEventStream(Iterator[dict[str, Any]]): - """Iterator over events correlated to a subscription request id.""" - - def __init__( - self, - client: "HerdrSocketClient", - subscription_id: str, - ack: Any, - *, - timeout: float | None = None, - ) -> None: - self.client = client - self.subscription_id = subscription_id - self.ack = ack - self.timeout = timeout - self._closed = False - - def __iter__(self) -> "HerdrEventStream": - return self - - def __next__(self) -> dict[str, Any]: - if self._closed: - raise StopIteration - try: - return self.client.read_event(self.subscription_id, timeout=self.timeout) - except HerdrSocketDisconnectedError: - self._closed = True - raise StopIteration from None - - def close(self) -> None: - self._closed = True - - class HerdrSocketClient: """Synchronous Herdr JSON-line client over a Unix domain socket.""" @@ -100,7 +58,6 @@ def __init__( self.timeout = self._validate_timeout(timeout) self._socket: socket.socket | None = None self._buffer = bytearray() - self._pending_events: deque[dict[str, Any]] = deque() def __enter__(self) -> "HerdrSocketClient": self.connect() @@ -137,7 +94,6 @@ def close(self) -> None: sock = self._socket self._socket = None self._buffer.clear() - self._pending_events.clear() if sock is None: return try: @@ -152,75 +108,17 @@ def request( params: Mapping[str, Any] | None = None, *, timeout: float | None = None, - allow_uncorrelated: bool = False, ) -> Any: - """Send one request and return its raw result payload. - - ``allow_uncorrelated`` is reserved for an inert method-capability - probe; the default keeps every ordinary request strictly correlated. - """ - request_id, deadline = self._send_request(method, params, timeout=timeout) - response = self._read_response( - request_id, - deadline=deadline, - allow_uncorrelated_error=False, - allow_uncorrelated_method_error=allow_uncorrelated, - ) - if is_error_response(response): - raise HerdrErrorResponse(error_payload(response), request_id) - return result_payload(response) - - def subscribe( - self, - method: str, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> HerdrEventStream: - """Send a subscription request and return an iterator over its events.""" - request_id, deadline = self._send_request(method, params, timeout=timeout) - response = self._read_response( - request_id, - deadline=deadline, - allow_uncorrelated_error=True, - allow_uncorrelated_method_error=False, - ) - if is_error_response(response): - raise HerdrErrorResponse(error_payload(response), request_id) - return HerdrEventStream( - self, - request_id, - result_payload(response), - timeout=self.timeout if event_timeout is None else event_timeout, - ) - - def events_subscribe( - self, - event_names: Iterable[str] | str | None = None, - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> HerdrEventStream: - """Subscribe to the official Herdr event stream.""" - return self.subscribe( - HERDR_EVENTS_SUBSCRIBE_METHOD, - build_events_subscribe_params(event_names), - timeout=timeout, - event_timeout=event_timeout, - ) - - def read_event(self, subscription_id: str, *, timeout: float | None = None) -> dict[str, Any]: - envelope = ( - self._pending_events.popleft() - if self._pending_events - else self._read_server_envelope(deadline=self._deadline(timeout)) - ) - if not is_event(envelope): - raise HerdrEnvelopeError("expected Herdr event envelope") - if envelope.get("id") is not None: - ensure_response_id(envelope, subscription_id) - return envelope + """Send one strictly correlated request and return its raw result payload.""" + try: + request_id, deadline = self._send_request(method, params, timeout=timeout) + response = self._read_response(request_id, deadline=deadline) + if is_error_response(response): + raise HerdrErrorResponse(error_payload(response), request_id) + return result_payload(response) + finally: + # Herdr accepts exactly one request on each Unix connection. + self.close() def workspace_list( self, @@ -230,14 +128,6 @@ def workspace_list( ) -> Any: return self.request("workspace.list", params, timeout=timeout) - def tab_list( - self, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - ) -> Any: - return self.request("tab.list", params, timeout=timeout) - def pane_list( self, params: Mapping[str, Any] | None = None, @@ -254,51 +144,56 @@ def agent_list( ) -> Any: return self.request("agent.list", params, timeout=timeout) - def pane_get( - self, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - ) -> Any: - return self.request("pane.get", params, timeout=timeout) - - def agent_get( + def agent_acp_endpoint( self, - params: Mapping[str, Any] | None = None, + target: str, *, timeout: float | None = None, ) -> Any: - return self.request("agent.get", params, timeout=timeout) - - def pane_read( - self, - params: Mapping[str, Any] | None = None, - *, - timeout: float | None = None, - ) -> Any: - return self.request("pane.read", params, timeout=timeout) + """Mint a one-shot, private ACP attach endpoint for one live agent.""" + return self.request( + "agent.acp_endpoint", + {"target": target}, + timeout=timeout, + ) - def pane_turns( + def agent_acp_status( self, - params: Mapping[str, Any] | None = None, + target: str, *, timeout: float | None = None, - allow_uncorrelated: bool = False, ) -> Any: + """Read ACP ownership/generation without minting an attach ticket.""" return self.request( - "pane.turns", - params, + "agent.acp_status", + {"target": target}, timeout=timeout, - allow_uncorrelated=allow_uncorrelated, ) - def agent_send( + def agent_acp_console_exchange( self, - params: Mapping[str, Any] | None = None, + target: str, *, + generation: int, + lease: str, + after_input_sequence: int = 0, + output: Iterable[Mapping[str, Any]] = (), timeout: float | None = None, ) -> Any: - return self.request("agent.send", params, timeout=timeout) + """Exchange pane input and idempotent ACP output as coordinator.""" + return self.request( + "agent.acp_console_exchange", + { + "target": target, + "generation": generation, + "lease": lease, + "role": "coordinator", + "output": [dict(item) for item in output], + "after_input_sequence": after_input_sequence, + "after_output_sequence": 0, + }, + timeout=timeout, + ) def _send_request( self, @@ -311,7 +206,10 @@ def _send_request( request = build_request(method, params) request_id = str(request["id"]) deadline = self._deadline(timeout) - self._write(frame_request(request), deadline=deadline) + frame = frame_request(request) + if len(frame) > _MAX_FRAME_BYTES: + raise HerdrFrameTooLargeError("Herdr request frame is too large") + self._write(frame, deadline=deadline) return request_id, deadline def _deadline(self, timeout: float | None) -> float: @@ -335,18 +233,6 @@ def _write(self, payload: bytes, *, deadline: float) -> None: try: sock.settimeout(self._remaining(deadline)) sock.sendall(payload) - except (BrokenPipeError, ConnectionResetError) as exc: - self.close() - try: - self.connect() - sock = self._active_socket() - sock.settimeout(self._remaining(deadline)) - sock.sendall(payload) - except socket.timeout as retry_exc: - raise HerdrSocketTimeoutError("Herdr socket write timed out") from retry_exc - except OSError as retry_exc: - self.close() - raise HerdrSocketDisconnectedError("Herdr socket disconnected during write") from retry_exc except socket.timeout as exc: raise HerdrSocketTimeoutError("Herdr socket write timed out") from exc except OSError as exc: @@ -358,74 +244,25 @@ def _read_response( request_id: str, *, deadline: float, - allow_uncorrelated_error: bool, - allow_uncorrelated_method_error: bool, ) -> dict[str, Any]: - while True: - envelope = self._read_server_envelope( - deadline=deadline, - allow_uncorrelated_error=allow_uncorrelated_error, - allow_uncorrelated_method_error=allow_uncorrelated_method_error, - ) - if is_event(envelope): - if len(self._pending_events) >= _MAX_PENDING_EVENTS: - raise HerdrEnvelopeError( - "too many Herdr events arrived before the response" - ) - self._pending_events.append(envelope) - continue - if ( - (allow_uncorrelated_error or allow_uncorrelated_method_error) - and is_error_response(envelope) - and ( - ( - allow_uncorrelated_error - and envelope.get("id") == "" - ) - or ( - allow_uncorrelated_method_error - and ( - "id" not in envelope - or envelope.get("id") == "" - ) - ) - ) - ): - # These narrowly validated Herdr 0.7.5 errors belong to the - # only in-flight synchronous request and must reach the caller - # as server errors rather than poison the connection. - payload = envelope.get("error") - if not isinstance(payload, Mapping): - raise HerdrEnvelopeError("Herdr error payload must be an object") - raise HerdrErrorResponse( - dict(payload), - request_id, - uncorrelated=True, - ) - ensure_response_id(envelope, request_id) - if not (is_result_response(envelope) or is_error_response(envelope)): - raise HerdrEnvelopeError("expected Herdr response envelope") - return envelope + envelope = self._read_server_envelope(deadline=deadline) + ensure_response_id(envelope, request_id) + if not (is_result_response(envelope) or is_error_response(envelope)): + raise HerdrEnvelopeError("expected Herdr response envelope") + return envelope - def _read_server_envelope( - self, - *, - deadline: float, - allow_uncorrelated_error: bool = False, - allow_uncorrelated_method_error: bool = False, - ) -> dict[str, Any]: + def _read_server_envelope(self, *, deadline: float) -> dict[str, Any]: line = self._read_line(deadline=deadline) envelope = parse_json_line(line) - return validate_server_envelope( - envelope, - allow_uncorrelated_error=allow_uncorrelated_error, - allow_uncorrelated_method_error=allow_uncorrelated_method_error, - ) + return validate_server_envelope(envelope) def _read_line(self, *, deadline: float) -> bytes: while True: newline_index = self._buffer.find(b"\n") if newline_index >= 0: + if newline_index + 1 > _MAX_FRAME_BYTES: + self.close() + raise HerdrFrameTooLargeError("Herdr response frame is too large") line = bytes(self._buffer[: newline_index + 1]) del self._buffer[: newline_index + 1] return line @@ -446,3 +283,6 @@ def _read_line(self, *, deadline: float) -> bytes: "Herdr socket disconnected before a complete line was received" ) self._buffer.extend(chunk) + if len(self._buffer) > _MAX_FRAME_BYTES: + self.close() + raise HerdrFrameTooLargeError("Herdr response frame is too large") diff --git a/src/tendwire/backends/herdr_turns.py b/src/tendwire/backends/herdr_turns.py deleted file mode 100644 index 268a1d5..0000000 --- a/src/tendwire/backends/herdr_turns.py +++ /dev/null @@ -1,4808 +0,0 @@ -"""Structured turn ingestion through Tendwire-owned private Herdr bindings.""" - -from __future__ import annotations - -import base64 -import hashlib -import json -import multiprocessing -import os -import re -import secrets -import select -import shlex -import socket -import stat -import struct -import subprocess -import threading -import time -from collections import OrderedDict, deque -from collections.abc import Callable, Mapping -from concurrent.futures import FIRST_COMPLETED, Future, ThreadPoolExecutor, wait -from dataclasses import dataclass, field, replace -from datetime import datetime -from pathlib import Path -from typing import Any, Literal -from uuid import UUID - -from ..config import Config -from ..core.models import WorkerBinding, stable_fingerprint, utc_timestamp -from ..core.turns import ( - InteractionChoice, - PendingObservation, - PendingObservedChoice, - is_internal_automation_turn_payload, - redact_private_prompt_text, -) -from ..store.sqlite import ( - apply_turn_refresh, - latest_turn_id_for_worker, - list_worker_bindings, - prune_backend_pending, -) - - -_TURN_CONTENT_KEYS = ( - "user_text", - "assistant_final_text", - "assistant_stream_text", - "model", - "complete", - "has_open_turn", - "awaiting_input", - "pending_decision", -) -_CODEX_SESSION_TURN_KIND = "codex_session_id" -_OMP_SESSION_TURN_KIND = "omp_session_path" -_PANE_TURN_KIND = "pane_id" -_MAX_CODEX_STREAM_MESSAGES = 4 -_CODEX_RECORD_MAX_BYTES = 8 * 1024 * 1024 -_CODEX_TURN_ID_MAX_BYTES = 1024 -_CODEX_READ_CHUNK_BYTES = 64 * 1024 -_CODEX_RESYNC_INITIAL_BYTES = 64 * 1024 -_CODEX_RESYNC_MAX_BYTES = 16 * 1024 * 1024 -_CODEX_RESYNC_MAX_RECORDS = 65_536 -_CODEX_INDEX_MAX_DEPTH = 4 -_CODEX_INDEX_MAX_VISITS = 100_000 -_CODEX_INDEX_MAX_ENTRIES = _CODEX_INDEX_MAX_VISITS -_CODEX_INDEX_MAX_BYTES = 16 * 1024 * 1024 -_CODEX_PATH_CACHE_CAPACITY = 256 -_CODEX_PATH_CACHE_MAX_BYTES = 256 * 1024 -_CODEX_NEGATIVE_TTL_SECONDS = 2.0 -# A found path is inode-validated on every hit. Duplicate discovery is -# intentionally bounded-stale by this complete-index refresh interval so the -# daemon never walks a 20k tree on each two-second poll. -_CODEX_POSITIVE_TTL_SECONDS = 60.0 -_CODEX_SESSION_CACHE_CAPACITY = 64 -_CODEX_SESSION_CACHE_MAX_BYTES = 16 * 1024 * 1024 -_CODEX_STATE_IPC_MAX_BYTES = 12 * 1024 * 1024 -_CODEX_IPC_FRAME_MAX_BYTES = 64 * 1024 * 1024 -_CODEX_POLL_MAX_BYTES = 64 * 1024 * 1024 -_OMP_IPC_RESPONSE_CHUNK_BYTES = 1024 * 1024 -_OMP_TAIL_BYTES = 786432 -_OMP_TOOL_SNIPPET_CHARS = 160 -_OMP_SESSION_CACHE_CAPACITY = 64 -_OMP_SESSION_CACHE_MAX_BYTES = 64 * 1024 -_OMP_REQUEST_MAX_BYTES = 16 * 1024 -_OMP_TARGET_MAX_CHARS = 4096 -_OMP_TEARDOWN_GRACE_SECONDS = 0.25 -_OMP_FRAME_HEADER = struct.Struct("!Q") -_UNCHANGED_TURN = object() -_PROMPTLESS_STATUS_FINAL_RE = re.compile( - r"\b(?:" - r"standing by|" - r"waiting(?: quietly)?|" - r"wait for|" - r"no new state|" - r"repeat tick|" - r"startup line|" - r"initial state|" - r"current state|" - r"monitor(?: reports|ing)?|" - r"review in progress|" - r"gate (?:phase|verdict|held|reached)|" - r"not yet (?:reached|materialized)|" - r"handoff|" - r"phase" - r")\b", - re.IGNORECASE, -) - - -@dataclass(frozen=True) -class _CodexPathResolution: - status: str - root: str - root_file_id: tuple[int, int] - session_id: str - canonical_path: str | None - relative_path: str | None - file_id: tuple[int, int] | None - generation: int - expires_at: float - - -@dataclass(frozen=True) -class _CodexIndexGeneration: - root: str - root_signature: tuple[int, int, int, int] - generation: int - built_at: float - entries: Mapping[str, tuple[str, ...]] - retained_bytes: int - visited: int - overflowed: bool - - -@dataclass(frozen=True) -class _CodexRecordSpan: - start: int - end: int - - -@dataclass(frozen=True) -class _CodexSessionState: - resolver_generation: int - root: str - session_id: str - canonical_path: str - file_id: tuple[int, int] | None - observed_size: int - mtime_ns: int - ctime_ns: int - committed_offset: int - partial_record: bytes - active_turn_id: str - last_content_turn_id: str - turn_open: bool - final_seen: bool - complete: bool - stream_spans: tuple[_CodexRecordSpan, ...] - internal_turn: bool = False - root_file_id: tuple[int, int] | None = None - - -@dataclass(frozen=True) -class _CodexSemanticEvent: - kind: str - turn_id: str - text: str = "" - - -@dataclass -class _CodexWorkState: - resolver_generation: int - root: str - session_id: str - canonical_path: str - file_id: tuple[int, int] - observed_size: int - mtime_ns: int - ctime_ns: int - committed_offset: int = 0 - partial_record: bytes = b"" - active_turn_id: str = "" - last_content_turn_id: str = "" - turn_open: bool = False - final_seen: bool = False - complete: bool = False - internal_turn: bool = False - root_file_id: tuple[int, int] | None = None - stream_items: list[tuple[_CodexRecordSpan, str]] = field(default_factory=list) - user_text: str | None = None - final_text: str | None = None - public_changed: bool = False - - -_CODEX_ROLLOUT_RE = re.compile( - r"^rollout-(\d{4})-(\d{2})-(\d{2})T" - r"(\d{2})-(\d{2})-(\d{2})-" - r"([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})" - r"\.jsonl$", - re.ASCII, -) -_CODEX_PATH_CACHE: OrderedDict[tuple[str, str], _CodexPathResolution] = OrderedDict() -_CODEX_PATH_CACHE_LOCK = threading.RLock() -_CODEX_INDEX_GENERATION: _CodexIndexGeneration | None = None -_CODEX_INDEX_GENERATION_COUNTER = 0 -_CODEX_RESOLUTION_GENERATION_COUNTER = 0 -_CODEX_INDEX_BUILD_OBSERVER: Callable[[int], None] | None = None -_CODEX_SESSION_CACHE: OrderedDict[tuple[str, str], _CodexSessionState] = OrderedDict() -_CODEX_SESSION_CACHE_LOCK = threading.RLock() -_CODEX_SESSION_CACHE_LIVE_KEYS: set[tuple[str, str]] | None = None -_CODEX_SESSION_CACHE_BINDING_GENERATIONS: dict[tuple[str, str], int] = {} -_CODEX_SESSION_CACHE_BINDING_FINGERPRINTS: dict[tuple[str, str], tuple[str, ...]] = {} -_CODEX_SESSION_CACHE_GENERATION_COUNTER = 0 -_CODEX_ISOLATED_READ_OBSERVER: Callable[[int], None] | None = None - - -@dataclass -class _OmpSessionState: - """Constant-size parser coordinates retained between polls.""" - - offset: int = 0 - observed_size: int = 0 - file_id: tuple[int, int] | None = None - mtime_ns: int = 0 - ctime_ns: int = 0 - replay_offset: int = 0 - turn_open: bool = False - project_root: Path | None = None - - -@dataclass -class _OmpTurnState: - """Canonical turn data confined to one parse and one response.""" - - prompt_id: str = "" - user_text: str = "" - stream_parts: list[str] = field(default_factory=list) - final_text: str = "" - tool_count: int = 0 - project_root: Path | None = None - - -@dataclass(frozen=True) -class _PublicOmpToolProgress: - """Progress assembled only from constants and proven-safe relative paths.""" - - action: str - subject: str | None = None - - def render(self, step: int) -> str: - body = f"{self.action}: {self.subject}" if self.subject else self.action - return f"step {step} · {body}" - - - - -_OMP_SESSION_CACHE: dict[str, _OmpSessionState] = {} -_OMP_SESSION_CACHE_LOCK = threading.RLock() -_OMP_SESSION_CACHE_LIVE_KEYS: set[str] | None = None -_OMP_SESSION_CACHE_BINDING_GENERATIONS: dict[str, int] = {} -_OMP_SESSION_CACHE_BINDING_FINGERPRINTS: dict[str, tuple[str, ...]] = {} -_OMP_SESSION_CACHE_GENERATION_COUNTER = 0 -_OMP_ISOLATED_READ_OBSERVER: Callable[[int], None] | None = None - - -def _omp_cache_get_locked(cache_key: str) -> _OmpSessionState | None: - state = _OMP_SESSION_CACHE.pop(cache_key, None) - if state is not None: - _OMP_SESSION_CACHE[cache_key] = state - return state - - -def _omp_cache_store_locked(cache_key: str, state: _OmpSessionState) -> None: - _OMP_SESSION_CACHE.pop(cache_key, None) - _OMP_SESSION_CACHE[cache_key] = state - while _OMP_SESSION_CACHE and ( - len(_OMP_SESSION_CACHE) > _OMP_SESSION_CACHE_CAPACITY - or _omp_cache_weight_locked() > _OMP_SESSION_CACHE_MAX_BYTES - ): - del _OMP_SESSION_CACHE[next(iter(_OMP_SESSION_CACHE))] - - -def _omp_cache_weight_locked() -> int: - return sum( - len(cache_key.encode("utf-8")) - + len( - json.dumps( - _serialize_omp_state(state), - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - ) - for cache_key, state in _OMP_SESSION_CACHE.items() - ) - - -def _omp_cache_binding_generation_locked(cache_key: str) -> int | None: - if _OMP_SESSION_CACHE_LIVE_KEYS is None: - return None - return _OMP_SESSION_CACHE_BINDING_GENERATIONS.get(cache_key) - - -def _serialize_omp_state(state: _OmpSessionState | None) -> dict[str, Any] | None: - if state is None: - return None - return { - "offset": state.offset, - "observed_size": state.observed_size, - "file_id": list(state.file_id) if state.file_id is not None else None, - "mtime_ns": state.mtime_ns, - "ctime_ns": state.ctime_ns, - "replay_offset": state.replay_offset, - "turn_open": state.turn_open, - "project_root": os.fspath(state.project_root) if state.project_root is not None else None, - } - - -def _deserialize_omp_state(value: Any) -> _OmpSessionState | None: - if value is None: - return None - if not isinstance(value, Mapping) or set(value) != { - "offset", - "observed_size", - "file_id", - "mtime_ns", - "ctime_ns", - "replay_offset", - "turn_open", - "project_root", - }: - raise ValueError("invalid OMP parser state") - offset = value["offset"] - observed_size = value["observed_size"] - file_id_value = value["file_id"] - mtime_ns = value["mtime_ns"] - ctime_ns = value["ctime_ns"] - replay_offset = value["replay_offset"] - turn_open = value["turn_open"] - project_root_value = value["project_root"] - if ( - type(offset) is not int - or offset < 0 - or type(observed_size) is not int - or observed_size < offset - or type(replay_offset) is not int - or replay_offset < 0 - or replay_offset > offset - or type(turn_open) is not bool - or type(mtime_ns) is not int - or mtime_ns < 0 - or type(ctime_ns) is not int - or ctime_ns < 0 - ): - raise ValueError("invalid OMP coordinates") - if ( - file_id_value is not None - and ( - not isinstance(file_id_value, (list, tuple)) - or len(file_id_value) != 2 - or any(type(part) is not int or part < 0 for part in file_id_value) - ) - ): - raise ValueError("invalid OMP file identity") - if project_root_value is not None and type(project_root_value) is not str: - raise ValueError("invalid OMP project root") - return _OmpSessionState( - offset=offset, - observed_size=observed_size, - file_id=tuple(file_id_value) if file_id_value is not None else None, - mtime_ns=mtime_ns, - ctime_ns=ctime_ns, - replay_offset=replay_offset, - turn_open=turn_open, - project_root=Path(project_root_value) if project_root_value is not None else None, - ) - - -def _extract_turn_payload(value: Any) -> Mapping[str, Any] | None: - if not isinstance(value, Mapping): - return None - result = value.get("result") - if isinstance(result, Mapping) and isinstance(result.get("turn"), Mapping): - return result["turn"] - if isinstance(value.get("turn"), Mapping): - return value["turn"] - return value - - -# Every driven ordinal must stay a single keystroke (digits select/toggle -# absolutely in the pane, live-verified on Claude Code 2.1.211), so 9 is the -# hard bound; larger decisions fail closed to the read-only interaction. -PENDING_DECISION_MAX_OPTIONS = 9 -_PENDING_TEXT_MAX = 2000 -_SINGLE_WRITE_IN_OPTION_IDS = frozenset( - {"custom", "other", "writein", "write_in", "write-in"} -) - - -def _private_pending_revision(value: Mapping[str, Any]) -> str: - encoded = json.dumps( - value, - sort_keys=True, - separators=(",", ":"), - ensure_ascii=False, - default=str, - ).encode("utf-8") - return hashlib.sha256(encoded).hexdigest()[:24] - - -def _pending_observation_from_turn(turn: Mapping[str, Any]) -> PendingObservation: - """Build one explicit private-neutral pending observation from a pane read.""" - decision = turn.get("pending_decision") - if decision is not None: - if not isinstance(decision, Mapping): - return PendingObservation("read_succeeded_invalid_prompt") - revision = _private_pending_revision(decision) - question = redact_private_prompt_text( - decision.get("prompt") or decision.get("question"), - max_chars=_PENDING_TEXT_MAX, - ) - options = decision.get("options") - if options is None: - options = [] - if not isinstance(options, list): - return PendingObservation("read_succeeded_invalid_prompt") - # A single-choice Claude prompt may have one trailing write-in row in - # addition to the digit-addressable options. Validate the effective - # selectable rows after the decision kind and write-in shape are known. - if len(options) > PENDING_DECISION_MAX_OPTIONS + 1: - return PendingObservation("read_succeeded_unsupported_decision") - choices: list[PendingObservedChoice] = [] - for ordinal, option in enumerate(options, 1): - if not isinstance(option, Mapping): - return PendingObservation("read_succeeded_invalid_prompt") - label = redact_private_prompt_text( - option.get("label"), - max_chars=_PENDING_TEXT_MAX, - ) - if not label: - return PendingObservation("read_succeeded_invalid_prompt") - label = InteractionChoice(label=label).label - choices.append( - PendingObservedChoice( - choice_id=f"choice-{stable_fingerprint({'revision': revision, 'ordinal': ordinal, 'label': label})}", - label=label, - picker_ordinal=ordinal, - ) - ) - if not question: - return PendingObservation("read_succeeded_invalid_prompt") - option_ids = { - str(option.get("id") or "") - for option in options - if isinstance(option, Mapping) - } - raw_kind = str( - decision.get("kind") - or decision.get("tool_name") - or decision.get("name") - or "" - ).strip().lower().replace("-", "_") - compact_kind = raw_kind.replace("_", "") - raw_mode = ( - str(decision.get("mode") or "") - .strip() - .lower() - .replace("-", "_") - ) - compact_mode = raw_mode.replace("_", "") - if compact_kind not in { - "", - "askuserquestion", - "single", - "multi", - "multiselect", - "exitplanmode", - "plan", - } or compact_mode not in { - "", - "buttons", - "single", - "multi", - "multiselect", - "plan", - }: - return PendingObservation("read_succeeded_unsupported_decision") - raw_multi_select = decision.get( - "multi_select", - decision.get("multiSelect", False), - ) - if not isinstance(raw_multi_select, bool): - return PendingObservation("read_succeeded_invalid_prompt") - if ( - compact_mode == "plan" - or compact_kind in {"exitplanmode", "plan"} - or (not compact_kind and "approve" in option_ids) - ): - decision_kind: Literal["single", "multi", "plan"] = "plan" - elif ( - compact_mode in {"multi", "multiselect"} - or raw_multi_select - or compact_kind in {"multi", "multiselect"} - ): - decision_kind = "multi" - else: - decision_kind = "single" - if raw_multi_select is not (decision_kind == "multi"): - return PendingObservation("read_succeeded_unsupported_decision") - raw_question_count = decision.get("question_count") - if raw_question_count is None: - raw_questions = decision.get("questions") - raw_question_count = ( - len(raw_questions) if isinstance(raw_questions, list) else 1 - ) - if ( - not isinstance(raw_question_count, int) - or isinstance(raw_question_count, bool) - or raw_question_count < 1 - ): - return PendingObservation("read_succeeded_invalid_prompt") - decision_option_labels = [choice.label for choice in choices] - if ( - decision_kind == "single" - and options - and isinstance(options[len(decision_option_labels) - 1], Mapping) - and str( - options[len(decision_option_labels) - 1].get("id") or "" - ).strip().lower() - in _SINGLE_WRITE_IN_OPTION_IDS - ): - decision_option_labels.pop() - decision_options = tuple(decision_option_labels) - if not decision_options: - return PendingObservation("read_succeeded_invalid_prompt") - if len(decision_options) > PENDING_DECISION_MAX_OPTIONS: - return PendingObservation("read_succeeded_unsupported_decision") - return PendingObservation( - "open_prompt", - question=question, - pending_kind="approval" if "approve" in option_ids else "question", - choices=tuple(choices), - revision_digest=revision, - decision_kind=decision_kind, - decision_options=decision_options, - decision_multi_select=decision_kind == "multi", - decision_question_count=raw_question_count, - ) - interaction = turn.get("pending_interaction") - if interaction is not None: - if not isinstance(interaction, Mapping): - return PendingObservation("read_succeeded_invalid_prompt") - questions = interaction.get("questions") - if questions is None: - questions = [] - if not isinstance(questions, list): - return PendingObservation("read_succeeded_invalid_prompt") - parts: list[str] = [] - for item in questions[:4]: - if not isinstance(item, Mapping): - return PendingObservation("read_succeeded_invalid_prompt") - part = redact_private_prompt_text(item.get("question")) - if part: - parts.append(part) - if not parts: - return PendingObservation("read_succeeded_invalid_prompt") - return PendingObservation( - "open_prompt", - question=" / ".join(parts)[:_PENDING_TEXT_MAX], - pending_kind="review", - revision_digest=_private_pending_revision(interaction), - ) - return PendingObservation("read_succeeded_no_prompt") - - -def _backend_pending_from_turn(turn: Mapping[str, Any]) -> dict[str, Any] | None: - """Compatibility public projection of the explicit observation.""" - observation = _pending_observation_from_turn(turn) - if observation.kind != "open_prompt": - return None - meta: dict[str, Any] = {"source": "backend"} - if observation.decision_kind is not None: - meta["decision"] = { - "decision_ref": ( - "decision-" - + stable_fingerprint( - {"decision_revision": observation.revision_digest} - ) - ), - "kind": observation.decision_kind, - "prompt": observation.question, - "options": [ - {"ref": str(ordinal), "label": label} - for ordinal, label in enumerate(observation.decision_options, 1) - ], - "multi_select": observation.decision_multi_select, - "question_count": observation.decision_question_count, - } - return { - "question": observation.question, - "kind": observation.pending_kind or "question", - "choices": [ - {"choice_id": choice.choice_id, "label": choice.label} - for choice in observation.choices - ], - "meta": meta, - } - - -def _public_turn_pending_projection( - observation: PendingObservation, -) -> dict[str, Any]: - """Return the private-neutral decision fields persisted with an open turn.""" - if observation.kind != "open_prompt" or observation.decision_kind is None: - return {} - mode = { - "single": "buttons", - "multi": "multi", - "plan": "plan", - }[observation.decision_kind] - return { - "awaiting_input": True, - "pending_decision": { - "prompt": observation.question, - "mode": mode, - "options": [ - {"id": str(ordinal), "label": label} - for ordinal, label in enumerate(observation.decision_options, 1) - ], - "multi_select": observation.decision_multi_select, - "question_count": observation.decision_question_count, - }, - } - - -class _TurnReadTimeout(Exception): - """Fixed internal timeout signal; never serialized with private details.""" - - -class _TurnReadFailed(Exception): - """Fixed internal adapter failure signal; never serialized with raw errors.""" - - -def _read_private_turn( - config: Config, - pane_id: str, - *, - timeout_seconds: float | None = None, - raise_timeout: bool = False, - cancel_event: threading.Event | None = None, -) -> Mapping[str, Any] | None: - argv = [ - config.herdr_bin, - "pane", - "turn", - pane_id, - "--last", - "--format", - "json", - ] - timeout = config.herdr_timeout_seconds if timeout_seconds is None else timeout_seconds - try: - if cancel_event is None: - completed = subprocess.run( - argv, - capture_output=True, - text=True, - check=False, - timeout=timeout, - ) - else: - process = subprocess.Popen( - argv, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - text=True, - ) - deadline = time.monotonic() + float(timeout) - while True: - try: - stdout, stderr = process.communicate( - timeout=max(0.001, min(0.05, deadline - time.monotonic())) - ) - completed = subprocess.CompletedProcess( - argv, - process.returncode, - stdout, - stderr, - ) - break - except subprocess.TimeoutExpired: - if not cancel_event.is_set() and time.monotonic() < deadline: - continue - process.terminate() - try: - process.wait(0.25) - except subprocess.TimeoutExpired: - process.kill() - process.communicate() - raise _TurnReadTimeout from None - except subprocess.TimeoutExpired: - if raise_timeout: - raise _TurnReadTimeout from None - return None - except _TurnReadTimeout: - if raise_timeout: - raise - return None - except (OSError, UnicodeDecodeError, ValueError): - if raise_timeout: - raise _TurnReadFailed from None - return None - if completed.returncode != 0: - if raise_timeout: - raise _TurnReadFailed - return None - try: - payload = json.loads(completed.stdout) - except (json.JSONDecodeError, TypeError, ValueError): - if raise_timeout: - raise _TurnReadFailed from None - return None - turn = _extract_turn_payload(payload) - if not isinstance(turn, Mapping): - if raise_timeout: - raise _TurnReadFailed - return None - pending_observation = _pending_observation_from_turn(turn) - pending_projection = _public_turn_pending_projection(pending_observation) - if turn.get("available") is False: - return ( - {"_backend_pending_observation": pending_observation} - if raise_timeout - else None - ) - open_user_text = turn.get("open_user_text") - open_turn_id = str(turn.get("open_turn_id") or "").strip() - if turn.get("has_open_turn") and (open_user_text or open_turn_id): - # An in-progress turn is reported alongside the last completed one via - # open_* fields. Emit it as its own open turn keyed by the stable prompt - # id so the connector streams a live "working" card that later edits - # into the final once this same id completes. - opened = _open_turn_content( - open_user_text, - turn.get("assistant_stream_text"), - open_turn_id, - ) - if raise_timeout: - opened_data = dict(opened or {}) - opened_data.update(pending_projection) - opened_data["_backend_pending_observation"] = pending_observation - return opened_data - - return opened - - content = {key: turn.get(key) for key in _TURN_CONTENT_KEYS if key in turn} - content.update(pending_projection) - if raise_timeout: - content["_backend_pending_observation"] = pending_observation - # Prefer the stable prompt-scoped id so a turn keeps one identity from - # open through complete; fall back to turn_id for backends without it. - source_turn_id = str(turn.get("source_turn_id") or turn.get("turn_id") or "").strip() - if source_turn_id: - content["source_turn_id"] = source_turn_id[:160] - if _is_internal_turn_content(content): - return ( - {"_backend_pending_observation": pending_observation} - if raise_timeout - else None - ) - # Never clobber stored text with an empty value: notification-triggered - # turns legitimately carry no prompt, but the previous real prompt and - # stream must survive the merge. - for key in ("user_text", "assistant_final_text", "assistant_stream_text"): - if key in content and not (content.get(key) or "").strip(): - content.pop(key) - if not any(value not in (None, "", False) for value in content.values()): - return None - return content - - -def _pending_public_payload( - observation: PendingObservation, -) -> dict[str, Any] | None: - if observation.kind != "open_prompt": - return None - return { - "question": observation.question, - "kind": observation.pending_kind or "question", - "choices": [ - {"choice_id": choice.choice_id, "label": choice.label} - for choice in observation.choices - ], - "meta": {"source": "backend"}, - } - - -def _pop_backend_pending_observation( - content: Mapping[str, Any] | None, -) -> tuple[dict[str, Any] | None, PendingObservation | None]: - if content is None: - return None, None - data = dict(content) - observation = data.pop("_backend_pending_observation", None) - data.pop("_backend_pending", None) - if not any(value not in (None, "", False) for value in data.values()): - data = None - return ( - data, - observation if isinstance(observation, PendingObservation) else None, - ) - - -def _pop_backend_pending(content: Mapping[str, Any] | None) -> tuple[dict[str, Any] | None, dict[str, Any] | None]: - """Split the reserved _backend_pending key off a turn-content read. Returns (content, pending); - content becomes None when nothing else remains.""" - if content is None: - return None, None - data = dict(content) - pending = data.pop("_backend_pending", None) - if not any(value not in (None, "", False) for value in data.values()): - data = None - return data, pending if isinstance(pending, dict) else None - - -def _open_turn_content( - open_user_text: Any, - stream_text: Any, - open_turn_id: str, -) -> Mapping[str, Any] | None: - if isinstance(open_user_text, str) and _is_internal_user_text(open_user_text): - return None - content: dict[str, Any] = { - "assistant_final_text": None, - "complete": False, - "has_open_turn": True, - } - if isinstance(open_user_text, str) and open_user_text.strip(): - content["user_text"] = open_user_text - if isinstance(stream_text, str) and stream_text.strip(): - content["assistant_stream_text"] = stream_text - if open_turn_id: - content["source_turn_id"] = open_turn_id[:160] - if _is_internal_turn_content(content): - return None - if not (content.get("user_text") or content.get("assistant_stream_text")): - return None - return content - - -def _codex_home() -> Path: - raw = os.environ.get("CODEX_HOME") - if raw: - return Path(raw).expanduser() - return Path.home() / ".codex" - - -def _canonical_codex_session_id(value: Any) -> str | None: - if type(value) is not str or len(value) != 36 or not value.isascii(): - return None - try: - parsed = UUID(value) - except (ValueError, AttributeError): - return None - if parsed.int == 0 or str(parsed) != value: - return None - return value - - -def _codex_rollout_identity( - relative_date: tuple[str, str, str], - basename: str, -) -> str | None: - if len(relative_date) != 3: - return None - match = _CODEX_ROLLOUT_RE.fullmatch(basename) - if match is None: - return None - year, month, day, hour, minute, second, session_id = match.groups() - if relative_date != (year, month, day): - return None - try: - datetime( - int(year), - int(month), - int(day), - int(hour), - int(minute), - int(second), - ) - except ValueError: - return None - return _canonical_codex_session_id(session_id) - - -class _CodexIndexLimit(Exception): - pass - - -def _resolve_codex_sessions_root() -> Path: - lexical_root = _codex_home() / "sessions" - flags = ( - os.O_RDONLY - | getattr(os, "O_DIRECTORY", 0) - | getattr(os, "O_CLOEXEC", 0) - | getattr(os, "O_NOFOLLOW", 0) - ) - try: - descriptor = os.open(lexical_root, flags) - except OSError as exc: - raise OSError("Codex sessions root is unavailable") from exc - try: - opened = os.fstat(descriptor) - root = lexical_root.resolve(strict=True) - current = root.lstat() - if ( - not stat.S_ISDIR(opened.st_mode) - or not stat.S_ISDIR(current.st_mode) - or (int(opened.st_dev), int(opened.st_ino)) - != (int(current.st_dev), int(current.st_ino)) - ): - raise OSError("Codex sessions root changed during resolution") - return root - finally: - os.close(descriptor) - - -def _codex_root_signature(root: Path) -> tuple[int, int, int, int]: - value = root.lstat() - if not stat.S_ISDIR(value.st_mode): - raise OSError("Codex sessions root is not a directory") - return ( - int(value.st_dev), - int(value.st_ino), - int(value.st_mtime_ns), - int(value.st_ctime_ns), - ) - - - - - - -def _build_codex_index(root: Path) -> _CodexIndexGeneration: - global _CODEX_INDEX_GENERATION_COUNTER - entries: dict[str, tuple[str, ...]] = {} - retained_bytes = 0 - visited = 0 - - def visit() -> None: - nonlocal visited - visited += 1 - if visited > _CODEX_INDEX_MAX_VISITS: - raise _CodexIndexLimit - def scan(path: str | os.PathLike[str]) -> list[Any]: - iterator = os.scandir(path) - items: list[Any] = [] - try: - for item in iterator: - visit() - items.append(item) - finally: - close = getattr(iterator, "close", None) - if callable(close): - close() - items.sort(key=lambda item: item.name) - return items - - - try: - for year_entry in scan(root): - if ( - len(year_entry.name) != 4 - or not year_entry.name.isascii() - or not year_entry.name.isdecimal() - or not year_entry.is_dir(follow_symlinks=False) - ): - continue - for month_entry in scan(year_entry.path): - if ( - len(month_entry.name) != 2 - or not month_entry.name.isascii() - or not month_entry.name.isdecimal() - or not month_entry.is_dir(follow_symlinks=False) - ): - continue - for day_entry in scan(month_entry.path): - if ( - len(day_entry.name) != 2 - or not day_entry.name.isascii() - or not day_entry.name.isdecimal() - or not day_entry.is_dir(follow_symlinks=False) - ): - continue - date_parts = ( - year_entry.name, - month_entry.name, - day_entry.name, - ) - for file_entry in scan(day_entry.path): - if not file_entry.is_file(follow_symlinks=False): - continue - session_id = _codex_rollout_identity( - date_parts, - file_entry.name, - ) - if session_id is None: - continue - relative_path = "/".join((*date_parts, file_entry.name)) - previous = entries.get(session_id, ()) - if relative_path in previous: - continue - retained = (*previous, relative_path) - added_weight = len(relative_path.encode("utf-8")) - if not previous: - added_weight += len(session_id) - if ( - len(entries) + (0 if previous else 1) - > _CODEX_INDEX_MAX_ENTRIES - or retained_bytes + added_weight - > _CODEX_INDEX_MAX_BYTES - ): - raise _CodexIndexLimit - entries[session_id] = retained - retained_bytes += added_weight - overflowed = False - except _CodexIndexLimit: - entries = {} - retained_bytes = 0 - overflowed = True - _CODEX_INDEX_GENERATION_COUNTER += 1 - generation = _CodexIndexGeneration( - root=os.fspath(root), - root_signature=_codex_root_signature(root), - generation=_CODEX_INDEX_GENERATION_COUNTER, - built_at=time.monotonic(), - entries=entries, - retained_bytes=retained_bytes, - visited=visited, - overflowed=overflowed, - ) - observer = _CODEX_INDEX_BUILD_OBSERVER - if observer is not None: - observer(visited) - return generation - - -def _codex_path_cache_weight_locked() -> int: - return sum( - len(root.encode("utf-8")) - + len(session_id) - + len((entry.relative_path or "").encode("utf-8")) - + 96 - for (root, session_id), entry in _CODEX_PATH_CACHE.items() - ) - - -def _codex_path_cache_store_locked( - key: tuple[str, str], - resolution: _CodexPathResolution, -) -> None: - _CODEX_PATH_CACHE.pop(key, None) - _CODEX_PATH_CACHE[key] = resolution - while _CODEX_PATH_CACHE and ( - len(_CODEX_PATH_CACHE) > _CODEX_PATH_CACHE_CAPACITY - or _codex_path_cache_weight_locked() > _CODEX_PATH_CACHE_MAX_BYTES - ): - _CODEX_PATH_CACHE.popitem(last=False) - - -def _codex_resolution_from_index_locked( - root: Path, - session_id: str, - index: _CodexIndexGeneration, - previous: _CodexPathResolution | None, -) -> _CodexPathResolution: - global _CODEX_RESOLUTION_GENERATION_COUNTER - relative_paths = index.entries.get(session_id, ()) - status = ( - "index_limit" - if index.overflowed - else "missing" - if not relative_paths - else "ambiguous" - if len(relative_paths) != 1 - else "found" - ) - canonical_path: str | None = None - relative_path: str | None = None - file_id: tuple[int, int] | None = None - if status == "found": - relative_path = relative_paths[0] - candidate = root.joinpath(*relative_path.split("/")) - try: - path_stat = candidate.lstat() - resolved = candidate.resolve(strict=True) - resolved.relative_to(root) - if ( - not stat.S_ISREG(path_stat.st_mode) - or resolved != candidate - or (int(path_stat.st_dev), int(path_stat.st_ino)) - != (int(resolved.stat().st_dev), int(resolved.stat().st_ino)) - ): - status = "unsafe_path" - else: - canonical_path = os.fspath(resolved) - file_id = (int(path_stat.st_dev), int(path_stat.st_ino)) - except (OSError, ValueError): - status = "unsafe_path" - same_resolution = bool( - previous is not None - and previous.status == status - and previous.root_file_id == index.root_signature[:2] - and previous.canonical_path == canonical_path - and previous.relative_path == relative_path - and previous.file_id == file_id - ) - if same_resolution: - generation = previous.generation - else: - _CODEX_RESOLUTION_GENERATION_COUNTER += 1 - generation = _CODEX_RESOLUTION_GENERATION_COUNTER - return _CodexPathResolution( - status=status, - root=os.fspath(root), - root_file_id=(index.root_signature[0], index.root_signature[1]), - session_id=session_id, - canonical_path=canonical_path, - relative_path=relative_path, - file_id=file_id, - generation=generation, - expires_at=time.monotonic() - + ( - _CODEX_POSITIVE_TTL_SECONDS - if status == "found" - else _CODEX_NEGATIVE_TTL_SECONDS - ), - ) - - -def _resolve_codex_session(session_id: Any) -> _CodexPathResolution | None: - canonical_id = _canonical_codex_session_id(session_id) - if canonical_id is None: - return None - try: - root = _resolve_codex_sessions_root() - root_signature = _codex_root_signature(root) - except OSError: - return None - key = (os.fspath(root), canonical_id) - now = time.monotonic() - global _CODEX_INDEX_GENERATION - with _CODEX_PATH_CACHE_LOCK: - cached = _CODEX_PATH_CACHE.get(key) - index = _CODEX_INDEX_GENERATION - invalidated = False - current_root_id = root_signature[:2] - root_changed = bool( - (cached is not None and cached.root_file_id != current_root_id) - or ( - index is not None - and index.root == os.fspath(root) - and index.root_signature[:2] != current_root_id - ) - ) - if root_changed: - for path_key in tuple(_CODEX_PATH_CACHE): - if path_key[0] == os.fspath(root): - del _CODEX_PATH_CACHE[path_key] - _CODEX_INDEX_GENERATION = None - cached = None - index = None - invalidated = True - if cached is not None and cached.expires_at > now: - if cached.status != "found": - _CODEX_PATH_CACHE.move_to_end(key) - return cached - try: - current = Path(cached.canonical_path or "").lstat() - except OSError: - invalidated = True - else: - if ( - stat.S_ISREG(current.st_mode) - and (int(current.st_dev), int(current.st_ino)) == cached.file_id - ): - _CODEX_PATH_CACHE.move_to_end(key) - return cached - invalidated = True - rebuild = bool( - invalidated - or index is None - or index.root != os.fspath(root) - or index.root_signature != root_signature - or now - index.built_at >= _CODEX_POSITIVE_TTL_SECONDS - ) - if rebuild: - index = _build_codex_index(root) - _CODEX_INDEX_GENERATION = index - assert index is not None - resolution = _codex_resolution_from_index_locked( - root, - canonical_id, - index, - cached, - ) - _codex_path_cache_store_locked(key, resolution) - return resolution - - -def _find_codex_session_file(session_id: Any) -> Path | None: - resolution = _resolve_codex_session(session_id) - if resolution is None or resolution.status != "found": - return None - return Path(resolution.canonical_path or "") - - -def _payload_turn_id(payload: Mapping[str, Any]) -> str: - raw = payload.get("turn_id") - if isinstance(raw, str) and raw.strip(): - return raw.strip() - metadata = payload.get("internal_chat_message_metadata_passthrough") - if isinstance(metadata, Mapping): - raw = metadata.get("turn_id") - if isinstance(raw, str) and raw.strip(): - return raw.strip() - return "" - - -def _message_text(payload: Mapping[str, Any]) -> str: - content = payload.get("content") - if isinstance(content, list): - parts: list[str] = [] - for item in content: - if not isinstance(item, Mapping): - continue - text = item.get("text") - if isinstance(text, str) and text.strip(): - parts.append(text) - return "\n".join(parts) - text = payload.get("message") - if isinstance(text, str): - return text - return "" - - -_INTERNAL_USER_TEXT_PREFIXES = ( - "", - "", - "", - "", - "", - "", - "", - "", - "Caveat: The messages below were generated by the user while running local commands.", -) - -def _is_internal_user_text(text: str) -> bool: - clean = text.lstrip().replace("\r\n", "\n") - return clean.startswith(_INTERNAL_USER_TEXT_PREFIXES) or is_internal_automation_turn_payload( - {"user_text": clean} - ) - - -def _is_internal_turn_content(content: Mapping[str, Any]) -> bool: - user_text = content.get("user_text") - if isinstance(user_text, str) and _is_internal_user_text(user_text): - return True - if _is_promptless_status_final(content): - return True - return is_internal_automation_turn_payload(content) - - -def _is_promptless_status_final(content: Mapping[str, Any]) -> bool: - if str(content.get("user_text") or "").strip(): - return False - if content.get("complete") is False or content.get("has_open_turn") is True: - return False - final_text = str(content.get("assistant_final_text") or "").strip() - if not final_text or len(final_text) > 800: - return False - return bool(_PROMPTLESS_STATUS_FINAL_RE.search(final_text)) - - -def _append_unique_recent(items: list[str], text: str) -> None: - clean = text.strip() - if not clean: - return - if clean in items: - items.remove(clean) - items.append(clean) - if len(items) > _MAX_CODEX_STREAM_MESSAGES: - del items[: len(items) - _MAX_CODEX_STREAM_MESSAGES] - - -def _open_verified_codex_file( - resolution: _CodexPathResolution, -) -> tuple[int, os.stat_result]: - if ( - resolution.status != "found" - or resolution.canonical_path is None - or resolution.file_id is None - ): - raise _TurnReadFailed - root = Path(resolution.root) - candidate = Path(resolution.canonical_path) - try: - before = candidate.lstat() - resolved = candidate.resolve(strict=True) - resolved.relative_to(root) - except (OSError, ValueError) as exc: - raise _TurnReadFailed from exc - before_id = (int(before.st_dev), int(before.st_ino)) - if ( - not stat.S_ISREG(before.st_mode) - or resolved != candidate - or before_id != resolution.file_id - ): - raise _TurnReadFailed - if resolution.relative_path is None: - raise _TurnReadFailed - relative_parts = tuple(resolution.relative_path.split("/")) - if len(relative_parts) != _CODEX_INDEX_MAX_DEPTH: - raise _TurnReadFailed - nofollow = getattr(os, "O_NOFOLLOW", 0) - close_on_exec = getattr(os, "O_CLOEXEC", 0) - directory_flags = ( - os.O_RDONLY - | getattr(os, "O_DIRECTORY", 0) - | close_on_exec - | nofollow - ) - file_flags = os.O_RDONLY | close_on_exec | nofollow - directory_fd: int | None = None - try: - directory_fd = os.open(root, directory_flags) - root_stat = os.fstat(directory_fd) - if (int(root_stat.st_dev), int(root_stat.st_ino)) != resolution.root_file_id: - raise _TurnReadFailed - for component in relative_parts[:-1]: - next_fd = os.open( - component, - directory_flags, - dir_fd=directory_fd, - ) - os.close(directory_fd) - directory_fd = next_fd - descriptor = os.open( - relative_parts[-1], - file_flags, - dir_fd=directory_fd, - ) - except OSError as exc: - raise _TurnReadFailed from exc - finally: - if directory_fd is not None: - os.close(directory_fd) - try: - opened = os.fstat(descriptor) - after = candidate.stat() - opened_id = (int(opened.st_dev), int(opened.st_ino)) - if ( - not stat.S_ISREG(opened.st_mode) - or opened_id != before_id - or opened_id != (int(after.st_dev), int(after.st_ino)) - ): - raise _TurnReadFailed - return descriptor, opened - except BaseException: - os.close(descriptor) - raise - - -def _serialize_codex_state(state_value: _CodexSessionState | None) -> dict[str, Any] | None: - if state_value is None: - return None - value = { - "resolver_generation": state_value.resolver_generation, - "root": state_value.root, - "root_file_id": list(state_value.root_file_id) if state_value.root_file_id is not None else None, - "session_id": state_value.session_id, - "canonical_path": state_value.canonical_path, - "file_id": list(state_value.file_id) if state_value.file_id is not None else None, - "observed_size": state_value.observed_size, - "mtime_ns": state_value.mtime_ns, - "ctime_ns": state_value.ctime_ns, - "committed_offset": state_value.committed_offset, - "partial_record_b64": base64.b64encode(state_value.partial_record).decode("ascii"), - "active_turn_id": state_value.active_turn_id, - "last_content_turn_id": state_value.last_content_turn_id, - "turn_open": state_value.turn_open, - "final_seen": state_value.final_seen, - "complete": state_value.complete, - "internal_turn": state_value.internal_turn, - "stream_spans": [ - [span.start, span.end] - for span in state_value.stream_spans - ], - } - encoded = json.dumps( - value, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - if len(encoded) > _CODEX_STATE_IPC_MAX_BYTES: - raise ValueError("Codex parser state exceeds IPC limit") - return value - - -def _deserialize_codex_state(value: Any) -> _CodexSessionState | None: - if value is None: - return None - expected = { - "resolver_generation", - "root", - "root_file_id", - "session_id", - "canonical_path", - "file_id", - "observed_size", - "mtime_ns", - "ctime_ns", - "committed_offset", - "partial_record_b64", - "active_turn_id", - "last_content_turn_id", - "turn_open", - "final_seen", - "complete", - "internal_turn", - "stream_spans", - } - if not isinstance(value, Mapping) or set(value) != expected: - raise ValueError("invalid Codex parser state") - encoded = json.dumps( - value, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - if len(encoded) > _CODEX_STATE_IPC_MAX_BYTES: - raise ValueError("oversized Codex parser state") - resolver_generation = value["resolver_generation"] - root = value["root"] - root_file_id_value = value["root_file_id"] - session_id = value["session_id"] - canonical_path = value["canonical_path"] - file_id_value = value["file_id"] - observed_size = value["observed_size"] - mtime_ns = value["mtime_ns"] - ctime_ns = value["ctime_ns"] - committed_offset = value["committed_offset"] - partial_value = value["partial_record_b64"] - active_turn_id = value["active_turn_id"] - last_content_turn_id = value["last_content_turn_id"] - stream_value = value["stream_spans"] - if ( - type(resolver_generation) is not int - or resolver_generation <= 0 - or type(root) is not str - or not root - or type(session_id) is not str - or _canonical_codex_session_id(session_id) != session_id - or type(canonical_path) is not str - or not canonical_path - or type(observed_size) is not int - or observed_size < 0 - or type(mtime_ns) is not int - or mtime_ns < 0 - or type(ctime_ns) is not int - or ctime_ns < 0 - or type(committed_offset) is not int - or committed_offset < 0 - or type(partial_value) is not str - or type(active_turn_id) is not str - or type(last_content_turn_id) is not str - or type(value["turn_open"]) is not bool - or type(value["final_seen"]) is not bool - or type(value["complete"]) is not bool - or type(value["internal_turn"]) is not bool - or not isinstance(stream_value, list) - or len(stream_value) > _MAX_CODEX_STREAM_MESSAGES - ): - raise ValueError("invalid Codex parser state values") - for turn_id in (active_turn_id, last_content_turn_id): - if len(turn_id.encode("utf-8")) > _CODEX_TURN_ID_MAX_BYTES: - raise ValueError("oversized Codex turn identity") - try: - partial_record = base64.b64decode(partial_value, validate=True) - except (ValueError, base64.binascii.Error) as exc: - raise ValueError("invalid Codex partial record") from exc - if ( - base64.b64encode(partial_record).decode("ascii") != partial_value - or len(partial_record) > _CODEX_RECORD_MAX_BYTES - or committed_offset + len(partial_record) > observed_size - ): - raise ValueError("invalid Codex partial coordinates") - if ( - not isinstance(root_file_id_value, (list, tuple)) - or len(root_file_id_value) != 2 - or any(type(part) is not int or part < 0 for part in root_file_id_value) - ): - raise ValueError("invalid Codex root identity") - if ( - file_id_value is not None - and ( - not isinstance(file_id_value, (list, tuple)) - or len(file_id_value) != 2 - or any(type(part) is not int or part < 0 for part in file_id_value) - ) - ): - raise ValueError("invalid Codex file identity") - if file_id_value is None and ( - observed_size != 0 - or committed_offset != 0 - or partial_record - or mtime_ns != 0 - or ctime_ns != 0 - or active_turn_id - or last_content_turn_id - or value["internal_turn"] - or stream_value - ): - raise ValueError("invalid Codex bootstrap state") - spans: list[_CodexRecordSpan] = [] - prior_end = -1 - for raw_span in stream_value: - if ( - not isinstance(raw_span, (list, tuple)) - or len(raw_span) != 2 - or any(type(part) is not int or part < 0 for part in raw_span) - ): - raise ValueError("invalid Codex record span") - start, end = raw_span - if start >= end or end > committed_offset or end - start > _CODEX_RECORD_MAX_BYTES: - raise ValueError("invalid Codex record coordinates") - if start <= prior_end: - raise ValueError("overlapping Codex record spans") - spans.append(_CodexRecordSpan(start, end)) - prior_end = end - try: - root_path = Path(root) - path = Path(canonical_path) - relative = path.relative_to(root_path) - except ValueError as exc: - raise ValueError("Codex path outside root") from exc - if ( - len(relative.parts) != _CODEX_INDEX_MAX_DEPTH - or _codex_rollout_identity( - (relative.parts[0], relative.parts[1], relative.parts[2]), - relative.parts[3], - ) - != session_id - ): - raise ValueError("invalid Codex rollout path") - return _CodexSessionState( - resolver_generation=resolver_generation, - root=root, - root_file_id=tuple(root_file_id_value), - session_id=session_id, - canonical_path=canonical_path, - file_id=tuple(file_id_value) if file_id_value is not None else None, - observed_size=observed_size, - mtime_ns=mtime_ns, - ctime_ns=ctime_ns, - committed_offset=committed_offset, - partial_record=partial_record, - active_turn_id=active_turn_id, - last_content_turn_id=last_content_turn_id, - turn_open=value["turn_open"], - final_seen=value["final_seen"], - complete=value["complete"], - internal_turn=value["internal_turn"], - stream_spans=tuple(spans), - ) - - -def _codex_cache_weight_locked() -> int: - total = 0 - for (root, session_id), state_value in _CODEX_SESSION_CACHE.items(): - serialized = _serialize_codex_state(state_value) - total += ( - len(root.encode("utf-8")) - + len(session_id) - + len( - json.dumps( - serialized, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - ) - ) - return total - - -def _codex_cache_get_locked( - cache_key: tuple[str, str], -) -> _CodexSessionState | None: - state_value = _CODEX_SESSION_CACHE.pop(cache_key, None) - if state_value is not None: - _CODEX_SESSION_CACHE[cache_key] = state_value - return state_value - - -def _codex_cache_store_locked( - cache_key: tuple[str, str], - state_value: _CodexSessionState, -) -> bool: - encoded = _serialize_codex_state(state_value) - entry_weight = ( - len(cache_key[0].encode("utf-8")) - + len(cache_key[1]) - + len( - json.dumps( - encoded, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - ) - ) - if entry_weight > _CODEX_SESSION_CACHE_MAX_BYTES: - return False - _CODEX_SESSION_CACHE.pop(cache_key, None) - _CODEX_SESSION_CACHE[cache_key] = state_value - while _CODEX_SESSION_CACHE and ( - len(_CODEX_SESSION_CACHE) > _CODEX_SESSION_CACHE_CAPACITY - or _codex_cache_weight_locked() > _CODEX_SESSION_CACHE_MAX_BYTES - ): - _CODEX_SESSION_CACHE.popitem(last=False) - return cache_key in _CODEX_SESSION_CACHE - - -def _codex_binding_cache_key(value: Any) -> tuple[str, str] | None: - session_id = _canonical_codex_session_id(value) - if session_id is None: - return None - try: - root = _resolve_codex_sessions_root() - _codex_root_signature(root) - except OSError: - return None - return (os.fspath(root), session_id) - - -def _codex_cache_binding_generation_locked( - cache_key: tuple[str, str], -) -> int | None: - if _CODEX_SESSION_CACHE_LIVE_KEYS is None: - return None - return _CODEX_SESSION_CACHE_BINDING_GENERATIONS.get(cache_key) - - -def _prune_codex_cache_for_bindings(bindings: list[WorkerBinding]) -> None: - live_fingerprint_sets: dict[tuple[str, str], set[str]] = {} - for binding in bindings: - if ( - binding.turn_target_kind != _CODEX_SESSION_TURN_KIND - or not _eligible_turn_binding(binding) - ): - continue - cache_key = _codex_binding_cache_key(binding.turn_target_value) - if cache_key is not None: - live_fingerprint_sets.setdefault(cache_key, set()).add( - binding.private_fingerprint - ) - live_keys = set(live_fingerprint_sets) - live_fingerprints = { - key: tuple(sorted(values)) - for key, values in live_fingerprint_sets.items() - } - global _CODEX_SESSION_CACHE_GENERATION_COUNTER - global _CODEX_SESSION_CACHE_BINDING_FINGERPRINTS - global _CODEX_SESSION_CACHE_BINDING_GENERATIONS - global _CODEX_SESSION_CACHE_LIVE_KEYS - with _CODEX_SESSION_CACHE_LOCK: - changed = { - key - for key in live_keys - if ( - _CODEX_SESSION_CACHE_LIVE_KEYS is not None - and key in _CODEX_SESSION_CACHE_LIVE_KEYS - and _CODEX_SESSION_CACHE_BINDING_FINGERPRINTS.get(key) - != live_fingerprints[key] - ) - } - generations: dict[tuple[str, str], int] = {} - for key in live_keys: - if ( - _CODEX_SESSION_CACHE_LIVE_KEYS is not None - and key in _CODEX_SESSION_CACHE_LIVE_KEYS - and _CODEX_SESSION_CACHE_BINDING_FINGERPRINTS.get(key) - == live_fingerprints[key] - ): - generations[key] = _CODEX_SESSION_CACHE_BINDING_GENERATIONS[key] - else: - _CODEX_SESSION_CACHE_GENERATION_COUNTER += 1 - generations[key] = _CODEX_SESSION_CACHE_GENERATION_COUNTER - _CODEX_SESSION_CACHE_LIVE_KEYS = live_keys - _CODEX_SESSION_CACHE_BINDING_FINGERPRINTS = live_fingerprints - _CODEX_SESSION_CACHE_BINDING_GENERATIONS = generations - for key in tuple(_CODEX_SESSION_CACHE): - if key not in live_keys or key in changed: - del _CODEX_SESSION_CACHE[key] - - -def _codex_record_event(record: Mapping[str, Any]) -> _CodexSemanticEvent | None: - payload = record.get("payload") - if not isinstance(payload, Mapping): - return None - payload_type = str(payload.get("type") or "") - turn_id = _payload_turn_id(payload) - if turn_id and len(turn_id.encode("utf-8")) > _CODEX_TURN_ID_MAX_BYTES: - raise ValueError("oversized Codex turn identity") - if record.get("type") == "event_msg" and payload_type == "task_started": - return _CodexSemanticEvent("start", turn_id) - if record.get("type") == "event_msg" and payload_type == "task_complete": - text = str(payload.get("last_agent_message") or "") - if not text: - return _CodexSemanticEvent("complete_empty", turn_id) - return _CodexSemanticEvent("final", turn_id, text) - if record.get("type") != "response_item" or payload_type != "message": - return None - text = _message_text(payload) - if not text: - return None - role = str(payload.get("role") or "") - if role == "user": - return _CodexSemanticEvent("user", turn_id, text) - if role != "assistant": - return None - if str(payload.get("phase") or "") == "commentary": - return _CodexSemanticEvent("commentary", turn_id, text) - return _CodexSemanticEvent("final", turn_id, text) - - -def _apply_codex_event( - state_value: _CodexWorkState, - event: _CodexSemanticEvent | None, - span: _CodexRecordSpan, -) -> None: - if event is None: - return - if event.kind == "start": - if not event.turn_id: - return - state_value.active_turn_id = event.turn_id - state_value.last_content_turn_id = "" - state_value.turn_open = True - state_value.final_seen = False - state_value.complete = False - state_value.internal_turn = False - state_value.stream_items.clear() - state_value.user_text = None - state_value.final_text = None - state_value.public_changed = True - return - turn_id = event.turn_id or state_value.active_turn_id - if not turn_id or event.kind == "complete_empty": - return - selected = state_value.active_turn_id or state_value.last_content_turn_id - if state_value.active_turn_id and turn_id != state_value.active_turn_id: - return - if not state_value.active_turn_id and selected and turn_id != selected: - state_value.stream_items.clear() - state_value.final_seen = False - state_value.complete = False - state_value.internal_turn = False - state_value.user_text = None - state_value.final_text = None - if event.kind == "user": - if _is_internal_user_text(event.text): - # Codex may emit environment or command context before the real - # user message under the same turn ID. Suppress that context - # without allowing later internal metadata to erase a user turn - # that has already become public. - if state_value.user_text is None: - state_value.internal_turn = True - state_value.last_content_turn_id = turn_id - state_value.stream_items.clear() - state_value.final_text = None - return - state_value.internal_turn = False - state_value.user_text = event.text - state_value.last_content_turn_id = turn_id - state_value.turn_open = not state_value.final_seen - state_value.public_changed = True - return - if state_value.internal_turn: - return - if event.kind == "commentary": - clean = event.text.strip() - if not clean: - return - state_value.stream_items = [ - item for item in state_value.stream_items if item[1] != clean - ] - state_value.stream_items.append((span, clean)) - if len(state_value.stream_items) > _MAX_CODEX_STREAM_MESSAGES: - del state_value.stream_items[ - : len(state_value.stream_items) - _MAX_CODEX_STREAM_MESSAGES - ] - state_value.last_content_turn_id = turn_id - state_value.turn_open = not state_value.final_seen - state_value.public_changed = True - return - if event.kind == "final": - state_value.final_text = event.text - state_value.last_content_turn_id = turn_id - state_value.final_seen = True - state_value.complete = True - state_value.turn_open = False - state_value.stream_items.clear() - state_value.public_changed = True - - -def _codex_decode_record(raw: bytes) -> Mapping[str, Any]: - if len(raw) > _CODEX_RECORD_MAX_BYTES: - raise ValueError("oversized Codex record") - try: - decoded = raw.decode("utf-8") - value = json.loads(decoded) - except (UnicodeError, json.JSONDecodeError) as exc: - raise ValueError("invalid Codex record") from exc - if not isinstance(value, Mapping): - raise ValueError("invalid Codex record shape") - return value - - -def _codex_materialize_stream_items( - descriptor: int, - spans: tuple[_CodexRecordSpan, ...], -) -> tuple[list[tuple[_CodexRecordSpan, str]], int]: - items: list[tuple[_CodexRecordSpan, str]] = [] - bytes_read = 0 - for span in spans: - length = span.end - span.start - raw = os.pread(descriptor, length, span.start) - bytes_read += len(raw) - if len(raw) != length: - raise _TurnReadFailed - event = _codex_record_event(_codex_decode_record(raw)) - if event is None or event.kind != "commentary": - raise _TurnReadFailed - clean = event.text.strip() - items = [item for item in items if item[1] != clean] - items.append((span, clean)) - return items, bytes_read - - -def _codex_content_from_work( - state_value: _CodexWorkState, -) -> Mapping[str, Any] | None: - if state_value.internal_turn: - return None - if not state_value.public_changed: - return None - turn_id = state_value.active_turn_id or state_value.last_content_turn_id - if not turn_id: - return None - stream_text = "\n\n".join(text for _span, text in state_value.stream_items) or None - content = { - "user_text": state_value.user_text, - "assistant_stream_text": None if state_value.final_seen else stream_text, - "assistant_final_text": state_value.final_text, - "complete": state_value.complete if state_value.final_seen else False, - "has_open_turn": not state_value.final_seen, - "source_turn_id": turn_id[:160], - } - if _is_internal_turn_content(content): - return None - if not any(item not in (None, "", False) for item in content.values()): - return None - return content - - -def _codex_freeze_work(state_value: _CodexWorkState) -> _CodexSessionState: - return _CodexSessionState( - resolver_generation=state_value.resolver_generation, - root=state_value.root, - root_file_id=state_value.root_file_id, - session_id=state_value.session_id, - canonical_path=state_value.canonical_path, - file_id=state_value.file_id, - observed_size=state_value.observed_size, - mtime_ns=state_value.mtime_ns, - ctime_ns=state_value.ctime_ns, - committed_offset=state_value.committed_offset, - partial_record=state_value.partial_record, - active_turn_id=state_value.active_turn_id, - last_content_turn_id=state_value.last_content_turn_id, - turn_open=state_value.turn_open, - final_seen=state_value.final_seen, - complete=state_value.complete, - internal_turn=state_value.internal_turn, - stream_spans=tuple(span for span, _text in state_value.stream_items), - ) - - -def _codex_work_from_prior( - descriptor: int, - prior: _CodexSessionState, - opened: os.stat_result, -) -> tuple[_CodexWorkState, int]: - stream_items, bytes_read = _codex_materialize_stream_items( - descriptor, - prior.stream_spans, - ) - return ( - _CodexWorkState( - resolver_generation=prior.resolver_generation, - root=prior.root, - root_file_id=prior.root_file_id, - session_id=prior.session_id, - canonical_path=prior.canonical_path, - file_id=(int(opened.st_dev), int(opened.st_ino)), - observed_size=int(opened.st_size), - mtime_ns=int(opened.st_mtime_ns), - ctime_ns=int(opened.st_ctime_ns), - committed_offset=prior.committed_offset, - partial_record=prior.partial_record, - active_turn_id=prior.active_turn_id, - last_content_turn_id=prior.last_content_turn_id, - turn_open=prior.turn_open, - final_seen=prior.final_seen, - complete=prior.complete, - internal_turn=prior.internal_turn, - stream_items=stream_items, - ), - bytes_read, - ) - - -def _codex_apply_complete_line( - state_value: _CodexWorkState, - raw: bytes, - start: int, - end: int, -) -> _CodexSemanticEvent | None: - event = None - if raw.strip(): - record = _codex_decode_record(raw) - event = _codex_record_event(record) - _apply_codex_event( - state_value, - event, - _CodexRecordSpan(start, end), - ) - state_value.committed_offset = end + 1 - return event - - -def _read_codex_incremental( - descriptor: int, - prior: _CodexSessionState, - opened: os.stat_result, -) -> tuple[Mapping[str, Any] | None, _CodexSessionState, int]: - state_value, bytes_read = _codex_work_from_prior(descriptor, prior, opened) - buffer = bytearray(prior.partial_record) - line_start = prior.committed_offset - physical_offset = prior.committed_offset + len(prior.partial_record) - target_size = int(opened.st_size) - if bytes_read + (target_size - physical_offset) > _CODEX_POLL_MAX_BYTES: - raise ValueError("Codex poll byte limit exceeded") - while physical_offset < target_size: - amount = min(_CODEX_READ_CHUNK_BYTES, target_size - physical_offset) - chunk = os.pread(descriptor, amount, physical_offset) - if not chunk: - raise _TurnReadFailed - physical_offset += len(chunk) - bytes_read += len(chunk) - buffer.extend(chunk) - while True: - newline = buffer.find(b"\n") - if newline < 0: - if len(buffer) > _CODEX_RECORD_MAX_BYTES: - raise ValueError("oversized Codex record") - break - if newline > _CODEX_RECORD_MAX_BYTES: - raise ValueError("oversized Codex record") - raw = bytes(buffer[:newline]) - was_final = state_value.final_seen - event = _codex_apply_complete_line( - state_value, - raw, - line_start, - line_start + newline, - ) - del buffer[: newline + 1] - line_start = state_value.committed_offset - if ( - event is not None - and event.kind == "final" - and not was_final - and state_value.final_seen - ): - # Publish each newly completed turn before consuming a later - # turn already present in the same append batch. Bytes after - # this record may already have been read into ``buffer``; do - # not checkpoint them. The next refresh rereads from the exact - # committed newline and advances to the following turn. - state_value.partial_record = b"" - state_value.observed_size = state_value.committed_offset - state_value.mtime_ns = int(opened.st_mtime_ns) - state_value.ctime_ns = int(opened.st_ctime_ns) - return ( - _codex_content_from_work(state_value), - _codex_freeze_work(state_value), - bytes_read, - ) - state_value.partial_record = bytes(buffer) - state_value.observed_size = target_size - state_value.mtime_ns = int(opened.st_mtime_ns) - state_value.ctime_ns = int(opened.st_ctime_ns) - return ( - _codex_content_from_work(state_value), - _codex_freeze_work(state_value), - bytes_read, - ) - - -def _codex_tail_candidate( - data: bytes, - absolute_start: int, - file_size: int, -) -> tuple[int, list[tuple[int, int, Mapping[str, Any], _CodexSemanticEvent | None]], bytes] | None: - aligned_start = absolute_start - if absolute_start > 0: - first_lf = data.find(b"\n") - if first_lf < 0: - return None - aligned_start += first_lf + 1 - data = data[first_lf + 1 :] - last_lf = data.rfind(b"\n") - if last_lf < 0: - return None - complete = data[: last_lf + 1] - partial = data[last_lf + 1 :] - if len(partial) > _CODEX_RECORD_MAX_BYTES: - raise ValueError("oversized Codex record") - records: list[ - tuple[int, int, Mapping[str, Any], _CodexSemanticEvent | None] - ] = [] - cursor = 0 - while cursor < len(complete): - newline = complete.find(b"\n", cursor) - if newline < 0: - break - raw = complete[cursor:newline] - start = aligned_start + cursor - end = aligned_start + newline - cursor = newline + 1 - if not raw.strip(): - continue - if len(records) >= _CODEX_RESYNC_MAX_RECORDS: - raise ValueError("too many Codex resync records") - try: - record = _codex_decode_record(raw) - event = _codex_record_event(record) - except ValueError: - records.append((start, end, {}, _CodexSemanticEvent("invalid", ""))) - continue - records.append((start, end, record, event)) - latest_start: int | None = None - for index, (_start, _end, _record, event) in enumerate(records): - if event is not None and event.kind == "start" and event.turn_id: - latest_start = index - boundary = latest_start - if boundary is None: - last_identity = "" - for _start, _end, _record, event in records: - if ( - event is not None - and event.kind in {"user", "commentary", "final"} - and event.turn_id - ): - last_identity = event.turn_id - if last_identity: - boundary = next( - index - for index, (_start, _end, _record, event) in enumerate(records) - if ( - event is not None - and event.turn_id == last_identity - and event.kind in {"user", "commentary", "final"} - ) - ) - if boundary is None: - return None - if any( - event is not None and event.kind == "invalid" - for _start, _end, _record, event in records[boundary:] - ): - raise ValueError("invalid Codex record") - return boundary, records, partial - - -def _resync_codex( - descriptor: int, - bootstrap: _CodexSessionState, - opened: os.stat_result, -) -> tuple[Mapping[str, Any] | None, _CodexSessionState, int] | None: - file_size = int(opened.st_size) - if file_size == 0: - return None - chunks: deque[bytes] = deque() - total = 0 - next_amount = _CODEX_RESYNC_INITIAL_BYTES - candidate = None - absolute_start = file_size - while total < min(file_size, _CODEX_RESYNC_MAX_BYTES): - amount = min( - next_amount, - file_size - total, - _CODEX_RESYNC_MAX_BYTES - total, - ) - absolute_start = file_size - total - amount - chunk = os.pread(descriptor, amount, absolute_start) - if len(chunk) != amount: - raise _TurnReadFailed - chunks.appendleft(chunk) - total += amount - data = b"".join(chunks) - candidate = _codex_tail_candidate(data, absolute_start, file_size) - if candidate is not None: - boundary, records, _partial = candidate - boundary_event = records[boundary][3] - if ( - boundary_event is not None - and boundary_event.kind == "start" - ) or absolute_start == 0 or total >= min(file_size, _CODEX_RESYNC_MAX_BYTES): - break - candidate = None - next_amount = min(next_amount * 2, _CODEX_RESYNC_MAX_BYTES - total) - if next_amount <= 0: - break - if candidate is None: - return None - boundary, records, partial = candidate - first_start = records[boundary][0] - state_value = _CodexWorkState( - resolver_generation=bootstrap.resolver_generation, - root=bootstrap.root, - root_file_id=bootstrap.root_file_id, - session_id=bootstrap.session_id, - canonical_path=bootstrap.canonical_path, - file_id=(int(opened.st_dev), int(opened.st_ino)), - observed_size=file_size, - mtime_ns=int(opened.st_mtime_ns), - ctime_ns=int(opened.st_ctime_ns), - committed_offset=first_start, - ) - for start, end, _record, event in records[boundary:]: - if event is not None and event.kind == "invalid": - raise ValueError("invalid Codex record") - _apply_codex_event( - state_value, - event, - _CodexRecordSpan(start, end), - ) - state_value.committed_offset = end + 1 - state_value.partial_record = partial - if state_value.committed_offset + len(partial) != file_size: - raise _TurnReadFailed - return ( - _codex_content_from_work(state_value), - _codex_freeze_work(state_value), - total, - ) - - -def _codex_resolution_for_state( - state_value: _CodexSessionState, -) -> _CodexPathResolution: - try: - configured_root = _resolve_codex_sessions_root() - configured_signature = _codex_root_signature(configured_root) - state_root = Path(state_value.root) - path = Path(state_value.canonical_path) - relative = path.relative_to(state_root) - except (OSError, ValueError) as exc: - raise _TurnReadFailed from exc - if ( - configured_root != state_root - or state_value.root_file_id != configured_signature[:2] - or len(relative.parts) != _CODEX_INDEX_MAX_DEPTH - ): - raise _TurnReadFailed - parsed_id = _codex_rollout_identity( - (relative.parts[0], relative.parts[1], relative.parts[2]), - relative.parts[3], - ) - if parsed_id != state_value.session_id: - raise _TurnReadFailed - try: - current = path.lstat() - except OSError as exc: - raise _TurnReadFailed from exc - return _CodexPathResolution( - status="found", - root=state_value.root, - root_file_id=(configured_signature[0], configured_signature[1]), - session_id=state_value.session_id, - canonical_path=state_value.canonical_path, - relative_path="/".join(relative.parts), - file_id=(int(current.st_dev), int(current.st_ino)), - generation=state_value.resolver_generation, - expires_at=0.0, - ) - - -def _read_codex_session_turn_with_state( - session_id: str, - supplied_state: _CodexSessionState, -) -> tuple[Mapping[str, Any] | None, _CodexSessionState, int] | None: - if _canonical_codex_session_id(session_id) != supplied_state.session_id: - raise _TurnReadFailed - prior: _CodexSessionState | None = supplied_state - for _attempt in range(2): - resolution = _codex_resolution_for_state(supplied_state) - descriptor, opened = _open_verified_codex_file(resolution) - try: - file_id = (int(opened.st_dev), int(opened.st_ino)) - reusable = bool( - prior is not None - and prior.file_id == file_id - and prior.resolver_generation == supplied_state.resolver_generation - and prior.canonical_path == supplied_state.canonical_path - and int(opened.st_size) - >= prior.committed_offset + len(prior.partial_record) - and ( - int(opened.st_size) > prior.observed_size - or ( - int(opened.st_size) == prior.observed_size - and int(opened.st_mtime_ns) == prior.mtime_ns - and int(opened.st_ctime_ns) == prior.ctime_ns - ) - ) - ) - if reusable: - parsed = _read_codex_incremental(descriptor, prior, opened) - else: - parsed = _resync_codex(descriptor, supplied_state, opened) - after_fd = os.fstat(descriptor) - after_path = Path(supplied_state.canonical_path).stat() - if ( - (int(after_fd.st_dev), int(after_fd.st_ino)) != file_id - or (int(after_path.st_dev), int(after_path.st_ino)) != file_id - or int(after_fd.st_size) != int(opened.st_size) - or int(after_fd.st_mtime_ns) != int(opened.st_mtime_ns) - or int(after_fd.st_ctime_ns) != int(opened.st_ctime_ns) - ): - raise _TurnReadFailed - return parsed - except (_TurnReadFailed, OSError): - prior = None - if _attempt: - raise _TurnReadFailed - finally: - os.close(descriptor) - raise _TurnReadFailed - - -def _codex_checkpoint_matches_stat( - state_value: _CodexSessionState, - resolution: _CodexPathResolution, -) -> bool: - if ( - state_value.file_id is None - or resolution.status != "found" - or resolution.canonical_path != state_value.canonical_path - or resolution.root_file_id != state_value.root_file_id - or resolution.generation != state_value.resolver_generation - ): - return False - try: - current = Path(state_value.canonical_path).stat() - except OSError: - return False - return bool( - stat.S_ISREG(current.st_mode) - and (int(current.st_dev), int(current.st_ino)) == state_value.file_id - and int(current.st_size) == state_value.observed_size - and int(current.st_mtime_ns) == state_value.mtime_ns - and int(current.st_ctime_ns) == state_value.ctime_ns - ) - - -def _publish_codex_cache_state( - cache_key: tuple[str, str], - prior_state_value: Mapping[str, Any] | None, - updated_state: _CodexSessionState, - content: Mapping[str, Any] | None, - binding_generation: int | None, -) -> Mapping[str, Any] | None: - with _CODEX_SESSION_CACHE_LOCK: - current_generation = _codex_cache_binding_generation_locked(cache_key) - if ( - binding_generation != current_generation - or ( - _CODEX_SESSION_CACHE_LIVE_KEYS is not None - and cache_key not in _CODEX_SESSION_CACHE_LIVE_KEYS - ) - ): - return None - current = _CODEX_SESSION_CACHE.get(cache_key) - current_value = _serialize_codex_state(current) - exact_prior = current_value == prior_state_value - monotone = bool( - current is not None - and current.resolver_generation == updated_state.resolver_generation - and current.root_file_id == updated_state.root_file_id - and current.canonical_path == updated_state.canonical_path - and current.file_id == updated_state.file_id - and updated_state.committed_offset > current.committed_offset - ) - if not exact_prior and not monotone: - return None - if prior_state_value is not None and current is None: - return None - if not _codex_cache_store_locked(cache_key, updated_state): - return None - return content - - -def _read_codex_session_turn(session_id: str) -> Mapping[str, Any] | None: - resolution = _resolve_codex_session(session_id) - if resolution is None or resolution.status != "found": - return None - cache_key = (resolution.root, resolution.session_id) - with _CODEX_SESSION_CACHE_LOCK: - prior = _codex_cache_get_locked(cache_key) - if ( - prior is not None - and ( - prior.resolver_generation != resolution.generation - or prior.root_file_id != resolution.root_file_id - or prior.canonical_path != resolution.canonical_path - or prior.file_id != resolution.file_id - ) - ): - _CODEX_SESSION_CACHE.pop(cache_key, None) - prior = None - prior_value = _serialize_codex_state(prior) - binding_generation = _codex_cache_binding_generation_locked(cache_key) - if prior is not None and _codex_checkpoint_matches_stat(prior, resolution): - observer = _CODEX_ISOLATED_READ_OBSERVER - if observer is not None: - observer(0) - return None - supplied = prior - if ( - supplied is None - or supplied.resolver_generation != resolution.generation - or supplied.root_file_id != resolution.root_file_id - or supplied.canonical_path != resolution.canonical_path - ): - supplied = _CodexSessionState( - resolver_generation=resolution.generation, - root=resolution.root, - root_file_id=resolution.root_file_id, - session_id=resolution.session_id, - canonical_path=resolution.canonical_path or "", - file_id=None, - observed_size=0, - mtime_ns=0, - ctime_ns=0, - committed_offset=0, - partial_record=b"", - active_turn_id="", - last_content_turn_id="", - turn_open=False, - final_seen=False, - complete=False, - stream_spans=(), - ) - prior_value = None - parsed = _read_codex_session_turn_with_state(session_id, supplied) - if parsed is None: - return None - content, updated_state, bytes_read = parsed - observer = _CODEX_ISOLATED_READ_OBSERVER - if observer is not None: - observer(bytes_read) - return _publish_codex_cache_state( - cache_key, - prior_value, - updated_state, - content, - binding_generation, - ) - - -def _omp_sessions_root() -> Path: - raw = os.environ.get("OMP_SESSIONS_DIR") - if raw: - return Path(raw).expanduser() - return Path.home() / ".omp" / "agent" / "sessions" - - -def _safe_omp_session_path(value: str) -> Path | None: - candidate = Path(str(value or "")).expanduser() - root = _omp_sessions_root() - try: - candidate.resolve().relative_to(root.resolve()) - except (ValueError, OSError): - return None - if candidate.suffix != ".jsonl" or not candidate.is_file(): - return None - return candidate - - -def _omp_valid_git_directory(path: Path) -> bool: - """Require bounded, parseable Git HEAD evidence inside a git directory.""" - try: - if not path.is_dir(): - return False - with open(path / "HEAD", encoding="ascii") as handle: - head_text = handle.read(257) - except (OSError, UnicodeError): - return False - if len(head_text) > 256: - return False - lines = head_text.splitlines() - if len(lines) != 1: - return False - head = lines[0] - if re.fullmatch(r"[0-9a-fA-F]{40}|[0-9a-fA-F]{64}", head): - return True - if not head.startswith("ref: refs/"): - return False - reference = head[len("ref: ") :] - return bool(reference) and all( - part not in {"", ".", ".."} for part in reference.split("/") - ) and all(character.isalnum() or character in "._-/" for character in reference) - - -def _omp_project_root(session_file: Path) -> Path | None: - """Return the nearest repository root proven by the session cwd.""" - try: - with open(session_file, encoding="utf-8", errors="replace") as handle: - for _index in range(32): - line = handle.readline() - if not line or handle.tell() > 65536: - break - try: - entry = json.loads(line) - except (TypeError, json.JSONDecodeError): - continue - if not isinstance(entry, Mapping) or entry.get("type") != "session": - continue - cwd = entry.get("cwd") - if type(cwd) is not str or not cwd.strip(): - return None - resolved_cwd = Path(cwd).expanduser().resolve(strict=True) - if not resolved_cwd.is_dir(): - return None - for root in (resolved_cwd, *resolved_cwd.parents): - git_marker = root / ".git" - if git_marker.is_dir(): - return root if _omp_valid_git_directory(git_marker) else None - if not git_marker.exists(): - continue - if not git_marker.is_file(): - return None - with open(git_marker, encoding="utf-8") as marker_handle: - marker_text = marker_handle.read(4097) - if len(marker_text) > 4096: - return None - marker_lines = marker_text.splitlines() - if len(marker_lines) != 1 or not marker_lines[0].startswith("gitdir:"): - return None - reference = marker_lines[0][len("gitdir:") :].strip() - if not reference: - return None - git_dir = Path(reference) - if not git_dir.is_absolute(): - git_dir = git_marker.parent / git_dir - resolved_git_dir = git_dir.resolve(strict=True) - return root if _omp_valid_git_directory(resolved_git_dir) else None - return None - except (OSError, RuntimeError, UnicodeError, ValueError): - return None - return None - - -def _omp_file_id(stat_result: os.stat_result) -> tuple[int, int]: - return (int(stat_result.st_dev), int(stat_result.st_ino)) - - -def _omp_checkpoint_matches_stat( - state: _OmpSessionState | None, - stat_result: os.stat_result, -) -> bool: - return bool( - state is not None - and state.file_id == _omp_file_id(stat_result) - and state.observed_size == int(stat_result.st_size) - and state.mtime_ns == int(stat_result.st_mtime_ns) - and state.ctime_ns == int(stat_result.st_ctime_ns) - ) - - -class _OmpFileChanged(Exception): - pass - - -def _omp_binding_cache_key(path_value: str) -> str | None: - candidate = Path(str(path_value or "")).expanduser() - if candidate.suffix != ".jsonl": - return None - try: - resolved = candidate.resolve() - resolved.relative_to(_omp_sessions_root().resolve()) - except (ValueError, OSError): - return None - return os.fspath(resolved) - - -def _omp_cache_key(path_value: str) -> str | None: - if _safe_omp_session_path(path_value) is None: - return None - return _omp_binding_cache_key(path_value) - - -def _prune_omp_cache_for_bindings(bindings: list[WorkerBinding]) -> None: - live_fingerprint_sets: dict[str, set[str]] = {} - live_keys: set[str] = set() - for binding in bindings: - if ( - binding.turn_target_kind != _OMP_SESSION_TURN_KIND - or not _eligible_turn_binding(binding) - ): - continue - cache_key = _omp_binding_cache_key(str(binding.turn_target_value or "")) - if cache_key is not None: - live_keys.add(cache_key) - live_fingerprint_sets.setdefault(cache_key, set()).add( - binding.private_fingerprint - ) - global _OMP_SESSION_CACHE_GENERATION_COUNTER - global _OMP_SESSION_CACHE_BINDING_FINGERPRINTS - global _OMP_SESSION_CACHE_BINDING_GENERATIONS - global _OMP_SESSION_CACHE_LIVE_KEYS - with _OMP_SESSION_CACHE_LOCK: - live_fingerprints = { - cache_key: tuple(sorted(fingerprints)) - for cache_key, fingerprints in live_fingerprint_sets.items() - } - changed_keys = { - cache_key - for cache_key in live_keys - if ( - _OMP_SESSION_CACHE_LIVE_KEYS is not None - and cache_key in _OMP_SESSION_CACHE_LIVE_KEYS - and _OMP_SESSION_CACHE_BINDING_FINGERPRINTS.get(cache_key) - != live_fingerprints.get(cache_key) - ) - } - generations: dict[str, int] = {} - for cache_key in live_keys: - if ( - _OMP_SESSION_CACHE_LIVE_KEYS is not None - and cache_key in _OMP_SESSION_CACHE_LIVE_KEYS - and _OMP_SESSION_CACHE_BINDING_FINGERPRINTS.get(cache_key) - == live_fingerprints.get(cache_key) - ): - generations[cache_key] = _OMP_SESSION_CACHE_BINDING_GENERATIONS[cache_key] - else: - _OMP_SESSION_CACHE_GENERATION_COUNTER += 1 - generations[cache_key] = _OMP_SESSION_CACHE_GENERATION_COUNTER - _OMP_SESSION_CACHE_LIVE_KEYS = live_keys - _OMP_SESSION_CACHE_BINDING_FINGERPRINTS = live_fingerprints - _OMP_SESSION_CACHE_BINDING_GENERATIONS = generations - for cache_key in tuple(_OMP_SESSION_CACHE): - if cache_key not in live_keys or cache_key in changed_keys: - del _OMP_SESSION_CACHE[cache_key] - - -def _read_omp_jsonl_lines( - session_file: Path, - *, - start_offset: int, - drop_first_partial: bool, - expected_file_id: tuple[int, int], -) -> tuple[list[str], int, int]: - try: - with open(session_file, "rb") as handle: - opened_stat = os.fstat(handle.fileno()) - if ( - _omp_file_id(opened_stat) != expected_file_id - or int(opened_stat.st_size) < start_offset - ): - raise _OmpFileChanged - handle.seek(start_offset) - blob = handle.read() - completed_stat = os.fstat(handle.fileno()) - current_stat = session_file.stat() - opened_signature = ( - _omp_file_id(opened_stat), - int(opened_stat.st_size), - int(opened_stat.st_mtime_ns), - int(opened_stat.st_ctime_ns), - ) - completed_signature = ( - _omp_file_id(completed_stat), - int(completed_stat.st_size), - int(completed_stat.st_mtime_ns), - int(completed_stat.st_ctime_ns), - ) - current_signature = ( - _omp_file_id(current_stat), - int(current_stat.st_size), - int(current_stat.st_mtime_ns), - int(current_stat.st_ctime_ns), - ) - if ( - completed_signature != opened_signature - or current_signature != completed_signature - ): - raise _OmpFileChanged - except _OmpFileChanged: - raise - except OSError as exc: - raise _TurnReadFailed from exc - if not blob: - return [], start_offset, 0 - - offset = start_offset - segments = blob.splitlines(keepends=True) - if drop_first_partial and segments: - offset += len(segments[0]) - segments = segments[1:] - - lines: list[str] = [] - for index, segment in enumerate(segments): - line_bytes = segment.rstrip(b"\r\n") - if not line_bytes: - offset += len(segment) - continue - text = line_bytes.decode("utf-8", "replace") - has_line_end = segment.endswith(b"\n") or segment.endswith(b"\r") - if not has_line_end and index == len(segments) - 1: - try: - json.loads(text) - except (TypeError, json.JSONDecodeError): - break - lines.append(text) - offset += len(segment) - return lines, offset, len(blob) - - -def _omp_message_entry_from_line(line: str) -> tuple[Mapping[str, Any], Mapping[str, Any]] | None: - try: - entry = json.loads(line) - except (TypeError, json.JSONDecodeError): - return None - if not isinstance(entry, Mapping) or entry.get("type") != "message": - return None - message = entry.get("message") - if not isinstance(message, Mapping): - return None - return entry, message - - -def _is_omp_user_message(message: Mapping[str, Any]) -> bool: - return str(message.get("role") or "") == "user" and str(message.get("attribution") or "") == "user" - - -def _last_omp_user_line_index(lines: list[str]) -> int | None: - found: int | None = None - for index, line in enumerate(lines): - parsed = _omp_message_entry_from_line(line) - if parsed is None: - continue - _entry, message = parsed - text = _omp_message_text(message) - if _is_omp_user_message(message) and text and not _is_internal_user_text(text): - found = index - return found - - -def _omp_thinking_snippet(message: Mapping[str, Any]) -> str: - """Compact progress line from an omp thinking block: its bold headline, - falling back to a trimmed first line.""" - content = message.get("content") - if not isinstance(content, list): - return "" - for item in content: - if not isinstance(item, Mapping) or item.get("type") != "thinking": - continue - text = str(item.get("thinking") or "").strip() - if not text: - continue - first = text.splitlines()[0].strip() - if first.startswith("**") and first.endswith("**") and len(first) > 4: - return first.strip("*").strip() - return first[:120] - return "" - - -_OMP_FILE_ACTIONS = { - "read": "read", - "read_file": "read", - "write": "write", - "write_file": "write", - "edit": "edit", - "edit_file": "edit", - "apply_patch": "edit", -} -_OMP_ACTIONS = { - "grep": "search", - "search": "search", - "ast_grep": "search", - "glob": "list files", - "list_files": "list files", - "browser": "browse", - "web_search": "search web", - "task": "delegate", - "agent": "delegate", - "lsp": "inspect code", -} -_OMP_PRIVATE_PATH_SEGMENTS = frozenset( - { - ".git", - ".netrc", - ".npmrc", - ".pypirc", - ".ssh", - "auth.json", - "credential", - "credentials", - "credentials.json", - "id_dsa", - "id_ecdsa", - "id_ed25519", - "id_rsa", - "secret", - "secrets", - "secrets.json", - "token", - "tokens", - } -) - - -def _omp_path_segment_is_private(value: str) -> bool: - lowered = value.lower() - return ( - lowered in _OMP_PRIVATE_PATH_SEGMENTS - or lowered.startswith(".env") - or lowered.endswith(".key") - ) -_OMP_SHELL_TOOLS = frozenset({"bash", "shell", "sh", "exec", "execute", "run"}) - - -def _omp_tool_name(item: Mapping[str, Any]) -> str: - raw = item.get("name") - if raw is None: - raw = item.get("toolName") - if raw is None: - raw = item.get("tool") - if type(raw) is not str: - return "" - return re.sub(r"[^a-z0-9]+", "_", raw.strip().lower()).strip("_") - - -def _omp_tool_arguments(item: Mapping[str, Any]) -> Mapping[str, Any]: - for key in ("arguments", "input", "args"): - value = item.get(key) - if isinstance(value, Mapping): - return value - return {} - - -def _omp_repo_relative_path( - arguments: Mapping[str, Any], - project_root: Path | None, -) -> str | None: - if project_root is None: - return None - raw: Any = None - for key in ("path", "file_path", "filepath", "file"): - if key in arguments: - raw = arguments.get(key) - break - if type(raw) is not str: - return None - text = raw.strip() - if not text or text.startswith("~") or any(ord(character) < 32 for character in text): - return None - try: - root = project_root.resolve(strict=True) - candidate = Path(text) - if ".." in candidate.parts: - return None - resolved = candidate.resolve(strict=False) if candidate.is_absolute() else (root / candidate).resolve(strict=False) - relative = resolved.relative_to(root) - except (OSError, RuntimeError, ValueError): - return None - if any(_omp_path_segment_is_private(part) for part in relative.parts): - return None - public_path = relative.as_posix() - if not public_path or public_path == "." or len(public_path) > 120: - return None - allowed = frozenset("._-+@/ ") - if any(not (character.isascii() and (character.isalnum() or character in allowed)) for character in public_path): - return None - return public_path - - -def _omp_shell_progress(arguments: Mapping[str, Any]) -> _PublicOmpToolProgress: - command = arguments.get("command") - if command is None: - command = arguments.get("cmd") - if type(command) is not str or any(character in command for character in "\r\n;&|`$<>"): - return _PublicOmpToolProgress("run command") - try: - tokens = shlex.split(command) - except ValueError: - return _PublicOmpToolProgress("run command") - if not tokens: - return _PublicOmpToolProgress("run command") - if tokens[:2] == ["git", "status"] and all( - token in {"--short", "--porcelain", "--branch", "-s", "-sb"} - for token in tokens[2:] - ): - return _PublicOmpToolProgress("git status") - if tokens[0] == "pytest" or tokens[:3] in (["python", "-m", "pytest"], ["python3", "-m", "pytest"]): - return _PublicOmpToolProgress("test", "pytest") - if tokens[:3] == ["uv", "run", "pytest"]: - return _PublicOmpToolProgress("test", "pytest") - if len(tokens) >= 2 and tokens[0] in {"bun", "cargo", "go", "npm"} and tokens[1] == "test": - return _PublicOmpToolProgress("test", tokens[0]) - if len(tokens) >= 2 and tokens[0] in {"cargo", "go"} and tokens[1] == "build": - return _PublicOmpToolProgress("build", tokens[0]) - if tokens[:3] == ["npm", "run", "build"]: - return _PublicOmpToolProgress("build", "npm") - if tokens[0] == "make": - return _PublicOmpToolProgress("build", "make") - if tokens[:3] in (["python", "-m", "build"], ["python3", "-m", "build"]): - return _PublicOmpToolProgress("build", "python") - return _PublicOmpToolProgress("run command") - - -def _omp_public_tool_progress( - item: Mapping[str, Any], - project_root: Path | None, -) -> _PublicOmpToolProgress: - name = _omp_tool_name(item) - arguments = _omp_tool_arguments(item) - if name in _OMP_FILE_ACTIONS: - action = _OMP_FILE_ACTIONS[name] - subject = _omp_repo_relative_path(arguments, project_root) - return _PublicOmpToolProgress(action, subject) if subject else _PublicOmpToolProgress(f"{action} file") - if name in _OMP_SHELL_TOOLS: - return _omp_shell_progress(arguments) - return _PublicOmpToolProgress(_OMP_ACTIONS.get(name, "tool")) - - -def _omp_tool_snippet( - item: Mapping[str, Any], - step: int, - project_root: Path | None = None, -) -> str: - return _omp_public_tool_progress(item, project_root).render(step) - - -def _apply_omp_progress_message( - state: _OmpTurnState, - message: Mapping[str, Any], -) -> bool: - contributed = False - text = _omp_message_text(message) - if text: - _append_unique_recent(state.stream_parts, text) - contributed = True - - content = message.get("content") - if not isinstance(content, list): - return contributed - for item in content: - if not isinstance(item, Mapping): - continue - kind = str(item.get("type") or "") - if kind == "thinking": - snippet = _omp_thinking_snippet({"content": [item]}) - if snippet: - _append_unique_recent(state.stream_parts, snippet) - contributed = True - continue - if kind == "toolCall": - state.tool_count += 1 - snippet = _omp_tool_snippet(item, state.tool_count, state.project_root) - if snippet: - _append_unique_recent(state.stream_parts, snippet) - contributed = True - return contributed - - -def _omp_message_text(message: Mapping[str, Any]) -> str: - content = message.get("content") - if isinstance(content, str): - return content - if isinstance(content, list): - return "\n".join( - str(item.get("text") or "") - for item in content - if isinstance(item, Mapping) - and item.get("type") == "text" - and str(item.get("text") or "").strip() - ) - return "" - - -def _apply_omp_lines_to_state(state: _OmpTurnState, lines: list[str]) -> None: - for line in lines: - parsed = _omp_message_entry_from_line(line) - if parsed is None: - continue - entry, message = parsed - role = str(message.get("role") or "") - text = _omp_message_text(message) - if role == "user": - if not _is_omp_user_message(message): - continue - if not text or _is_internal_user_text(text): - continue - state.prompt_id = str(entry.get("id") or "") - state.user_text = text - state.stream_parts = [] - state.final_text = "" - state.tool_count = 0 - continue - if role != "assistant" or not state.prompt_id: - continue - if state.final_text: - continue - if str(message.get("stopReason") or "") == "stop" and text: - state.final_text = text - state.stream_parts = [] - continue - _apply_omp_progress_message(state, message) - - -def _read_omp_state_from_recent( - session_file: Path, - size: int, - file_id: tuple[int, int], - project_root: Path | None, -) -> tuple[_OmpTurnState, int, int, int]: - turn = _OmpTurnState(project_root=project_root) - if size <= 0: - return turn, 0, 0, 0 - - window = min(size, max(1, _OMP_TAIL_BYTES)) - selected_lines: list[str] = [] - selected_offset = size - replay_offset = size - bytes_read = 0 - while True: - start = max(0, size - window) - lines, next_offset, consumed = _read_omp_jsonl_lines( - session_file, - start_offset=start, - drop_first_partial=start > 0, - expected_file_id=file_id, - ) - bytes_read += consumed - user_index = _last_omp_user_line_index(lines) - if user_index is not None: - selected_lines = lines[user_index:] - selected_offset = next_offset - replay_offset = start - break - if start == 0: - selected_lines = lines - selected_offset = next_offset - replay_offset = 0 - break - window = min(size, window * 2) - - _apply_omp_lines_to_state(turn, selected_lines) - return turn, selected_offset, replay_offset, bytes_read - - -def _omp_state_to_content(state: _OmpTurnState) -> Mapping[str, Any] | None: - if not state.prompt_id: - return None - has_final = bool(state.final_text) - content: dict[str, Any] = { - "user_text": state.user_text or None, - "assistant_stream_text": None if has_final else ("\n\n".join(state.stream_parts) or None), - "assistant_final_text": state.final_text or None, - "complete": has_final, - "has_open_turn": not has_final, - "source_turn_id": state.prompt_id[:160], - } - if _is_internal_turn_content(content): - return None - if not (content.get("user_text") or content.get("assistant_stream_text") or content.get("assistant_final_text")): - return None - return content - - -def _publish_omp_cache_state( - cache_key: str, - prior_state_value: Mapping[str, Any] | None, - updated_state: _OmpSessionState, - content: Mapping[str, Any] | None, - binding_generation: int | None = None, -) -> Mapping[str, Any] | None: - """Atomically publish a winning parser state and never return a losing view.""" - with _OMP_SESSION_CACHE_LOCK: - current_generation = _omp_cache_binding_generation_locked(cache_key) - if ( - binding_generation != current_generation - or ( - _OMP_SESSION_CACHE_LIVE_KEYS is not None - and cache_key not in _OMP_SESSION_CACHE_LIVE_KEYS - ) - ): - return None - current_state = _OMP_SESSION_CACHE.get(cache_key) - current_value = _serialize_omp_state(current_state) - if ( - current_value == prior_state_value - or current_state is None - or ( - current_state.file_id == updated_state.file_id - and updated_state.offset > current_state.offset - ) - ): - _omp_cache_store_locked(cache_key, updated_state) - return content - return None - - -def _read_omp_session_turn_with_state( - path_value: str, - prior_state: _OmpSessionState | None, -) -> tuple[Mapping[str, Any] | None, _OmpSessionState, int] | None: - """Parse one OMP session while returning only compact checkpoint coordinates.""" - session_file = _safe_omp_session_path(path_value) - if session_file is None: - return None - for _attempt in range(2): - try: - stat_result = session_file.stat() - except OSError as exc: - raise _TurnReadFailed from exc - size = int(stat_result.st_size) - file_id = _omp_file_id(stat_result) - reusable = bool( - prior_state is not None - and prior_state.file_id == file_id - and size >= prior_state.offset - and ( - size > prior_state.observed_size - or ( - size == prior_state.observed_size - and prior_state.mtime_ns == int(stat_result.st_mtime_ns) - and prior_state.ctime_ns == int(stat_result.st_ctime_ns) - ) - ) - ) - try: - if reusable: - start_offset = ( - prior_state.replay_offset if prior_state.turn_open else prior_state.offset - ) - lines, next_offset, bytes_read = _read_omp_jsonl_lines( - session_file, - start_offset=start_offset, - drop_first_partial=False, - expected_file_id=file_id, - ) - turn = _OmpTurnState(project_root=prior_state.project_root) - _apply_omp_lines_to_state(turn, lines) - replay_offset = start_offset - else: - project_root = _omp_project_root(session_file) - turn, next_offset, replay_offset, bytes_read = _read_omp_state_from_recent( - session_file, - size, - file_id, - project_root, - ) - content = _omp_state_to_content(turn) - turn_open = bool(content is not None and content.get("has_open_turn") is True) - checkpoint = _OmpSessionState( - offset=next_offset, - observed_size=max(size, next_offset), - file_id=file_id, - mtime_ns=int(stat_result.st_mtime_ns), - ctime_ns=int(stat_result.st_ctime_ns), - replay_offset=replay_offset if turn_open else next_offset, - turn_open=turn_open, - project_root=turn.project_root, - ) - return content, checkpoint, bytes_read - except _OmpFileChanged: - prior_state = None - raise _TurnReadFailed - - -def _read_omp_session_turn(path_value: str) -> Mapping[str, Any] | None: - """Parse an OMP session while retaining private state in the local process.""" - cache_key = _omp_cache_key(path_value) - if cache_key is None: - return None - with _OMP_SESSION_CACHE_LOCK: - prior_state = _omp_cache_get_locked(cache_key) - prior_state_value = _serialize_omp_state(prior_state) - binding_generation = _omp_cache_binding_generation_locked(cache_key) - try: - unchanged = prior_state is not None and _omp_checkpoint_matches_stat( - prior_state, - Path(cache_key).stat(), - ) - except OSError as exc: - raise _TurnReadFailed from exc - if unchanged: - return None - parsed = _read_omp_session_turn_with_state(path_value, prior_state) - if parsed is None: - return None - content, state, _bytes_read = parsed - return _publish_omp_cache_state( - cache_key, - prior_state_value, - state, - content, - binding_generation, - ) - - -@dataclass(frozen=True) -class TurnRefreshKey: - """Opaque scheduler identity for one durable private binding.""" - - private_fingerprint: str - - -@dataclass(frozen=True) -class _TurnRefreshItem: - key: TurnRefreshKey - worker_id: str - worker_fingerprint: str - turn_target_kind: str - turn_target_value: str - - @classmethod - def from_binding(cls, binding: WorkerBinding) -> "_TurnRefreshItem": - return cls( - key=TurnRefreshKey(binding.private_fingerprint), - worker_id=binding.worker_id, - worker_fingerprint=binding.worker_fingerprint, - turn_target_kind=str(binding.turn_target_kind or ""), - turn_target_value=str(binding.turn_target_value or ""), - ) - - -TurnRefreshStatus = Literal[ - "updated", - "unchanged", - "missing", - "timeout", - "failed", - "stale_binding", -] - - -class _BindingLookupFailed(Exception): - pass - - -@dataclass(frozen=True) -class TurnRefreshResult: - """Public-safe result of one binding refresh.""" - - status: TurnRefreshStatus - updated: int - pending_changed: bool = False - retry_binding_lookup: bool = False - binding_validated: bool = False - - -@dataclass(frozen=True) -class CompletedPaneTurnRefreshResult: - """Result of one completion-authoritative, pane-targeted semantic refresh.""" - - status: str - worker_id: str | None = None - refreshed_turn_id: str | None = None - - -_ELIGIBLE_TURN_TARGET_KINDS = frozenset( - {_CODEX_SESSION_TURN_KIND, _OMP_SESSION_TURN_KIND, _PANE_TURN_KIND} -) -_TURN_INGESTION_QUEUE_CAPACITY = 64 - - -def _eligible_turn_binding(binding: WorkerBinding) -> bool: - return bool( - binding.turn_target_kind in _ELIGIBLE_TURN_TARGET_KINDS - and binding.turn_target_value - and binding.private_fingerprint - ) - - -def _isolated_content_is_valid(value: Any) -> bool: - if not isinstance(value, Mapping): - return False - if not set(value).issubset({*_TURN_CONTENT_KEYS, "source_turn_id"}): - return False - for key in ("user_text", "assistant_final_text", "assistant_stream_text", "model", "source_turn_id"): - if key in value and value[key] is not None and type(value[key]) is not str: - return False - for key in ("complete", "has_open_turn"): - if key in value and type(value[key]) is not bool: - return False - return True - - -@dataclass(frozen=True) -class _OmpCachePublication: - cache_key: str - prior_state_value: Mapping[str, Any] | None - updated_state: _OmpSessionState - binding_generation: int | None -@dataclass(frozen=True) -class _CodexCachePublication: - cache_key: tuple[str, str] - prior_state_value: Mapping[str, Any] | None - updated_state: _CodexSessionState - binding_generation: int | None - resolver_generation: int - - - - -@dataclass(frozen=True) -class _ObservedFileTurn: - content: Mapping[str, Any] | None - publication: _OmpCachePublication | _CodexCachePublication - - -def _blocking_recv_frame(sock: socket.socket, maximum: int) -> bytes: - header = bytearray() - while len(header) < _OMP_FRAME_HEADER.size: - chunk = sock.recv(_OMP_FRAME_HEADER.size - len(header)) - if not chunk: - raise EOFError - header.extend(chunk) - length = _OMP_FRAME_HEADER.unpack(header)[0] - if length > maximum: - raise ValueError("oversized IPC frame") - payload = bytearray() - while len(payload) < length: - chunk = sock.recv(min(65536, length - len(payload))) - if not chunk: - raise EOFError - payload.extend(chunk) - return bytes(payload) - - -def _blocking_send_frame(sock: socket.socket, payload: bytes) -> None: - sock.sendall(_OMP_FRAME_HEADER.pack(len(payload)) + payload) - - -def _check_ipc_deadline( - deadline: float, - cancel_event: threading.Event | None, -) -> float: - if cancel_event is not None and cancel_event.is_set(): - raise _TurnReadTimeout - remaining = deadline - time.monotonic() - if remaining <= 0: - raise _TurnReadTimeout - return remaining - - -def _send_frame_until( - sock: socket.socket, - payload: bytes, - deadline: float, - cancel_event: threading.Event | None, -) -> None: - framed = _OMP_FRAME_HEADER.pack(len(payload)) + payload - view = memoryview(framed) - while view: - remaining = _check_ipc_deadline(deadline, cancel_event) - _readable, writable, _exceptional = select.select( - [], - [sock], - [], - min(0.05, remaining), - ) - if not writable: - continue - sent = sock.send(view) - if sent <= 0: - raise OSError("IPC socket closed") - view = view[sent:] - - -def _recv_frame_until( - sock: socket.socket, - deadline: float, - cancel_event: threading.Event | None, - maximum: int = _CODEX_IPC_FRAME_MAX_BYTES, -) -> bytes: - framed = bytearray() - expected: int | None = None - while expected is None or len(framed) < _OMP_FRAME_HEADER.size + expected: - remaining = _check_ipc_deadline(deadline, cancel_event) - readable, _writable, _exceptional = select.select( - [sock], - [], - [], - min(0.05, remaining), - ) - if not readable: - continue - if expected is None: - remaining_bytes = _OMP_FRAME_HEADER.size - len(framed) - else: - remaining_bytes = _OMP_FRAME_HEADER.size + expected - len(framed) - chunk = sock.recv(min(65536, remaining_bytes)) - if not chunk: - raise EOFError - framed.extend(chunk) - if expected is None and len(framed) >= _OMP_FRAME_HEADER.size: - expected = _OMP_FRAME_HEADER.unpack(framed[: _OMP_FRAME_HEADER.size])[0] - if expected > maximum: - raise ValueError("oversized IPC frame") - assert expected is not None - return bytes(framed[_OMP_FRAME_HEADER.size : _OMP_FRAME_HEADER.size + expected]) -def _blocking_send_streamed_omp_response( - sock: socket.socket, - payload: bytes, - nonce: str, - *, - chunk_bytes: int = _OMP_IPC_RESPONSE_CHUNK_BYTES, -) -> None: - if ( - type(chunk_bytes) is not int - or chunk_bytes <= 0 - or chunk_bytes > _OMP_IPC_RESPONSE_CHUNK_BYTES - ): - raise ValueError("invalid OMP IPC chunk bound") - chunk_count = (len(payload) + chunk_bytes - 1) // chunk_bytes - manifest = { - "protocol": 1, - "nonce": nonce, - "stream": "omp_response", - "chunks": chunk_count, - "total_bytes": len(payload), - "chunk_bytes": chunk_bytes, - } - _blocking_send_frame( - sock, - json.dumps(manifest, separators=(",", ":")).encode("utf-8"), - ) - view = memoryview(payload) - for offset in range(0, len(payload), chunk_bytes): - _blocking_send_frame(sock, bytes(view[offset : offset + chunk_bytes])) - end = { - "protocol": 1, - "nonce": nonce, - "stream": "omp_response_end", - } - _blocking_send_frame( - sock, - json.dumps(end, separators=(",", ":")).encode("utf-8"), - ) - - -def _recv_streamed_omp_response_until( - sock: socket.socket, - first_payload: bytes, - nonce: str, - target_kind: str, - deadline: float, - cancel_event: threading.Event | None, -) -> bytes: - try: - manifest = json.loads(first_payload.decode("utf-8")) - except (UnicodeError, json.JSONDecodeError): - return first_payload - if not isinstance(manifest, Mapping) or "stream" not in manifest: - return first_payload - if ( - target_kind != _OMP_SESSION_TURN_KIND - or set(manifest) - != { - "protocol", - "nonce", - "stream", - "chunks", - "total_bytes", - "chunk_bytes", - } - or manifest["protocol"] != 1 - or manifest["stream"] != "omp_response" - or type(manifest["nonce"]) is not str - or not secrets.compare_digest(manifest["nonce"], nonce) - or type(manifest["chunks"]) is not int - or manifest["chunks"] <= 0 - or type(manifest["total_bytes"]) is not int - or manifest["total_bytes"] <= 0 - or type(manifest["chunk_bytes"]) is not int - or manifest["chunk_bytes"] <= 0 - or manifest["chunk_bytes"] > _OMP_IPC_RESPONSE_CHUNK_BYTES - or manifest["chunks"] - != ( - manifest["total_bytes"] + manifest["chunk_bytes"] - 1 - ) - // manifest["chunk_bytes"] - ): - raise _TurnReadFailed - assembled = bytearray() - for index in range(manifest["chunks"]): - chunk = _recv_frame_until( - sock, - deadline, - cancel_event, - manifest["chunk_bytes"], - ) - expected = ( - manifest["chunk_bytes"] - if index + 1 < manifest["chunks"] - else manifest["total_bytes"] - len(assembled) - ) - if len(chunk) != expected: - raise _TurnReadFailed - assembled.extend(chunk) - if len(assembled) != manifest["total_bytes"]: - raise _TurnReadFailed - end_payload = _recv_frame_until(sock, deadline, cancel_event, 1024) - try: - end = json.loads(end_payload.decode("utf-8")) - except (UnicodeError, json.JSONDecodeError) as exc: - raise _TurnReadFailed from exc - if ( - not isinstance(end, Mapping) - or set(end) != {"protocol", "nonce", "stream"} - or end["protocol"] != 1 - or end["stream"] != "omp_response_end" - or type(end["nonce"]) is not str - or not secrets.compare_digest(end["nonce"], nonce) - ): - raise _TurnReadFailed - while True: - remaining = _check_ipc_deadline(deadline, cancel_event) - readable, _writable, _exceptional = select.select( - [sock], - [], - [], - min(0.05, remaining), - ) - if not readable: - continue - if sock.recv(1): - raise _TurnReadFailed - break - return bytes(assembled) - - - - -def _omp_publication_commit( - publication: _OmpCachePublication, - content: Mapping[str, Any] | None, -) -> Mapping[str, Any] | None: - return _publish_omp_cache_state( - publication.cache_key, - publication.prior_state_value, - publication.updated_state, - content, - publication.binding_generation, - ) -def _codex_publication_commit( - publication: _CodexCachePublication, - content: Mapping[str, Any] | None, -) -> Mapping[str, Any] | None: - resolution = _resolve_codex_session(publication.cache_key[1]) - if ( - resolution is None - or resolution.status != "found" - or resolution.root != publication.cache_key[0] - or resolution.root_file_id != publication.updated_state.root_file_id - or resolution.generation != publication.resolver_generation - or resolution.canonical_path != publication.updated_state.canonical_path - or resolution.file_id != publication.updated_state.file_id - ): - return None - return _publish_codex_cache_state( - publication.cache_key, - publication.prior_state_value, - publication.updated_state, - content, - publication.binding_generation, - ) - - -def _file_publication_commit( - publication: _OmpCachePublication | _CodexCachePublication, - content: Mapping[str, Any] | None, -) -> Mapping[str, Any] | None: - if isinstance(publication, _CodexCachePublication): - return _codex_publication_commit(publication, content) - return _omp_publication_commit(publication, content) - - - - - - -def _file_turn_child(channel: socket.socket) -> None: - """Spawn entry point using one bounded, source-tagged private frame.""" - nonce = "" - try: - payload = _blocking_recv_frame(channel, _CODEX_STATE_IPC_MAX_BYTES) - request = json.loads(payload.decode("utf-8")) - if not isinstance(request, Mapping) or set(request) != { - "protocol", - "nonce", - "target_kind", - "target_value", - "parser_state", - }: - raise ValueError("invalid turn reader request") - nonce = request["nonce"] - target_kind = request["target_kind"] - target_value = request["target_value"] - parser_state = request["parser_state"] - if ( - request["protocol"] != 1 - or type(nonce) is not str - or not nonce - or type(target_kind) is not str - or type(target_value) is not str - or not isinstance(parser_state, Mapping) - or set(parser_state) != {"source", "state"} - ): - raise ValueError("invalid turn reader request values") - source = parser_state["source"] - content: Mapping[str, Any] | None - response_state: dict[str, Any] | None = None - bytes_read = 0 - disposition = "ok" - if target_kind == _CODEX_SESSION_TURN_KIND: - if source != "codex": - raise ValueError("wrong parser state source") - supplied = _deserialize_codex_state(parser_state["state"]) - if supplied is None: - raise ValueError("missing Codex parser state") - parsed = _read_codex_session_turn_with_state(target_value, supplied) - if parsed is None: - content = None - disposition = "missing" - else: - content, updated_state, bytes_read = parsed - response_state = { - "source": "codex", - "state": _serialize_codex_state(updated_state), - } - elif target_kind == _OMP_SESSION_TURN_KIND: - if source != "omp": - raise ValueError("wrong parser state source") - prior_state = _deserialize_omp_state(parser_state["state"]) - parsed = _read_omp_session_turn_with_state(target_value, prior_state) - if parsed is None: - content = None - disposition = "missing" - else: - content, updated_state, bytes_read = parsed - response_state = { - "source": "omp", - "state": _serialize_omp_state(updated_state), - } - else: - raise ValueError("unsupported turn reader target") - response = { - "protocol": 1, - "nonce": nonce, - "disposition": disposition, - "content": dict(content) if content is not None else None, - "parser_state": response_state, - "bytes_read": bytes_read, - } - encoded = json.dumps( - response, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - if target_kind == _CODEX_SESSION_TURN_KIND: - if len(encoded) > _CODEX_IPC_FRAME_MAX_BYTES: - raise ValueError("oversized Codex turn reader response") - _blocking_send_frame(channel, encoded) - elif len(encoded) > _OMP_IPC_RESPONSE_CHUNK_BYTES: - _blocking_send_streamed_omp_response(channel, encoded, nonce) - else: - _blocking_send_frame(channel, encoded) - except BaseException: - try: - response = { - "protocol": 1, - "nonce": nonce, - "disposition": "failed", - "content": None, - "parser_state": None, - "bytes_read": 0, - } - _blocking_send_frame( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - ) - except BaseException: - pass - finally: - channel.close() - - -def _terminate_and_reap(process: multiprocessing.Process) -> None: - """Reap within a fixed grace after the request deadline under normal POSIX scheduling.""" - if process.pid is None: - return - teardown_deadline = time.monotonic() + _OMP_TEARDOWN_GRACE_SECONDS - if not process.is_alive(): - process.join(0) - return - process.terminate() - process.join(min(0.05, max(0.0, teardown_deadline - time.monotonic()))) - if process.is_alive(): - process.kill() - process.join(max(0.0, teardown_deadline - time.monotonic())) - if process.is_alive(): - # Never trade the caller's hard teardown bound for an unbounded join. - # A SIGKILL-resistant kernel task remains an OS-level exceptional case. - process.kill() - - -def _read_file_turn_isolated( - target_kind: str, - target_value: str, - *, - timeout_seconds: float, - cancel_event: threading.Event | None = None, - defer_cache: bool = False, -) -> Mapping[str, Any] | _ObservedFileTurn | object | None: - """Read through one bounded source-tagged IPC request.""" - deadline = time.monotonic() + float(timeout_seconds) - if ( - type(target_kind) is not str - or type(target_value) is not str - or len(target_value) > _OMP_TARGET_MAX_CHARS - ): - raise _TurnReadFailed - codex_resolution: _CodexPathResolution | None = None - codex_cache_key: tuple[str, str] | None = None - codex_prior: _CodexSessionState | None = None - omp_cache_key: str | None = None - omp_prior: _OmpSessionState | None = None - prior_state_value: dict[str, Any] | None = None - binding_generation: int | None = None - parser_state: dict[str, Any] - if target_kind == _CODEX_SESSION_TURN_KIND: - canonical_id = _canonical_codex_session_id(target_value) - if canonical_id is None: - return None - codex_resolution = _resolve_codex_session(canonical_id) - if codex_resolution is None or codex_resolution.status != "found": - return None - codex_cache_key = (codex_resolution.root, canonical_id) - with _CODEX_SESSION_CACHE_LOCK: - codex_prior = _codex_cache_get_locked(codex_cache_key) - if ( - codex_prior is not None - and ( - codex_prior.resolver_generation != codex_resolution.generation - or codex_prior.root_file_id != codex_resolution.root_file_id - or codex_prior.canonical_path != codex_resolution.canonical_path - or codex_prior.file_id != codex_resolution.file_id - ) - ): - _CODEX_SESSION_CACHE.pop(codex_cache_key, None) - codex_prior = None - prior_state_value = _serialize_codex_state(codex_prior) - binding_generation = _codex_cache_binding_generation_locked( - codex_cache_key - ) - if ( - codex_prior is not None - and _codex_checkpoint_matches_stat(codex_prior, codex_resolution) - ): - _check_ipc_deadline(deadline, cancel_event) - observer = _CODEX_ISOLATED_READ_OBSERVER - if observer is not None: - observer(0) - return _UNCHANGED_TURN - supplied = codex_prior or _CodexSessionState( - resolver_generation=codex_resolution.generation, - root=codex_resolution.root, - root_file_id=codex_resolution.root_file_id, - session_id=canonical_id, - canonical_path=codex_resolution.canonical_path or "", - file_id=None, - observed_size=0, - mtime_ns=0, - ctime_ns=0, - committed_offset=0, - partial_record=b"", - active_turn_id="", - last_content_turn_id="", - turn_open=False, - final_seen=False, - complete=False, - stream_spans=(), - ) - parser_state = { - "source": "codex", - "state": _serialize_codex_state(supplied), - } - elif target_kind == _OMP_SESSION_TURN_KIND: - omp_cache_key = _omp_cache_key(target_value) - if omp_cache_key is not None: - with _OMP_SESSION_CACHE_LOCK: - omp_prior = _omp_cache_get_locked(omp_cache_key) - prior_state_value = _serialize_omp_state(omp_prior) - binding_generation = _omp_cache_binding_generation_locked( - omp_cache_key - ) - try: - unchanged = ( - omp_prior is not None - and _omp_checkpoint_matches_stat( - omp_prior, - Path(omp_cache_key).stat(), - ) - ) - except OSError as exc: - raise _TurnReadFailed from exc - if unchanged: - _check_ipc_deadline(deadline, cancel_event) - observer = _OMP_ISOLATED_READ_OBSERVER - if observer is not None: - observer(0) - return _UNCHANGED_TURN - parser_state = {"source": "omp", "state": prior_state_value} - else: - raise _TurnReadFailed - nonce = secrets.token_urlsafe(32) - request = { - "protocol": 1, - "nonce": nonce, - "target_kind": target_kind, - "target_value": target_value, - "parser_state": parser_state, - } - request_payload = json.dumps( - request, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - maximum_request = ( - _OMP_REQUEST_MAX_BYTES - if target_kind == _OMP_SESSION_TURN_KIND - else _CODEX_STATE_IPC_MAX_BYTES - ) - if len(request_payload) > maximum_request: - raise _TurnReadFailed - _check_ipc_deadline(deadline, cancel_event) - context = multiprocessing.get_context("spawn") - parent_channel, child_channel = socket.socketpair() - process: multiprocessing.Process | None = None - try: - parent_channel.setblocking(False) - process = context.Process( - target=_file_turn_child, - args=(child_channel,), - name="tendwire-turn-reader", - ) - process.start() - child_channel.close() - _check_ipc_deadline(deadline, cancel_event) - _send_frame_until(parent_channel, request_payload, deadline, cancel_event) - response_payload = _recv_frame_until( - parent_channel, - deadline, - cancel_event, - ( - _OMP_IPC_RESPONSE_CHUNK_BYTES - if target_kind == _OMP_SESSION_TURN_KIND - else _CODEX_IPC_FRAME_MAX_BYTES - ), - ) - response_payload = _recv_streamed_omp_response_until( - parent_channel, - response_payload, - nonce, - target_kind, - deadline, - cancel_event, - ) - _check_ipc_deadline(deadline, cancel_event) - response = json.loads(response_payload.decode("utf-8")) - _check_ipc_deadline(deadline, cancel_event) - process.join(max(0.0, deadline - time.monotonic())) - if process.is_alive(): - raise _TurnReadTimeout - if not isinstance(response, Mapping) or set(response) != { - "protocol", - "nonce", - "disposition", - "content", - "parser_state", - "bytes_read", - }: - raise _TurnReadFailed - response_nonce = response["nonce"] - disposition = response["disposition"] - content = response["content"] - bytes_read = response["bytes_read"] - response_state = response["parser_state"] - if ( - response["protocol"] != 1 - or type(response_nonce) is not str - or not secrets.compare_digest(response_nonce, nonce) - or disposition not in {"ok", "missing"} - or type(bytes_read) is not int - or bytes_read < 0 - or ( - target_kind == _CODEX_SESSION_TURN_KIND - and bytes_read > _CODEX_POLL_MAX_BYTES - ) - ): - raise _TurnReadFailed - if disposition == "missing": - if content is not None or response_state is not None or bytes_read != 0: - raise _TurnReadFailed - return None - if content is not None and not _isolated_content_is_valid(content): - raise _TurnReadFailed - if ( - not isinstance(response_state, Mapping) - or set(response_state) != {"source", "state"} - ): - raise _TurnReadFailed - if target_kind == _OMP_SESSION_TURN_KIND: - if response_state["source"] != "omp" or omp_cache_key is None: - raise _TurnReadFailed - updated_omp = _deserialize_omp_state(response_state["state"]) - if updated_omp is None: - raise _TurnReadFailed - observer = _OMP_ISOLATED_READ_OBSERVER - if observer is not None: - observer(bytes_read) - publication: _OmpCachePublication | _CodexCachePublication = ( - _OmpCachePublication( - omp_cache_key, - prior_state_value, - updated_omp, - binding_generation, - ) - ) - else: - if ( - response_state["source"] != "codex" - or codex_cache_key is None - or codex_resolution is None - ): - raise _TurnReadFailed - updated_codex = _deserialize_codex_state(response_state["state"]) - if ( - updated_codex is None - or updated_codex.root != codex_resolution.root - or updated_codex.root_file_id != codex_resolution.root_file_id - or updated_codex.session_id != codex_resolution.session_id - or updated_codex.canonical_path - != codex_resolution.canonical_path - or updated_codex.resolver_generation - != codex_resolution.generation - ): - raise _TurnReadFailed - observer = _CODEX_ISOLATED_READ_OBSERVER - if observer is not None: - observer(bytes_read) - publication = _CodexCachePublication( - codex_cache_key, - prior_state_value, - updated_codex, - binding_generation, - codex_resolution.generation, - ) - if defer_cache: - return _ObservedFileTurn(content, publication) - return _file_publication_commit(publication, content) - except _TurnReadTimeout: - raise - except ( - OSError, - EOFError, - BrokenPipeError, - UnicodeError, - ValueError, - json.JSONDecodeError, - ): - raise _TurnReadFailed from None - finally: - parent_channel.close() - child_channel.close() - if process is not None and process.pid is not None: - _terminate_and_reap(process) - - -def _read_turn_for_binding( - config: Config, - binding: WorkerBinding, - *, - timeout_seconds: float | None = None, - cancel_event: threading.Event | None = None, -) -> Mapping[str, Any] | _ObservedFileTurn | object | None: - target_kind = str(binding.turn_target_kind or "") - target_value = str(binding.turn_target_value or "") - if not target_value: - return None - deadline = config.herdr_timeout_seconds if timeout_seconds is None else float(timeout_seconds) - if target_kind in {_CODEX_SESSION_TURN_KIND, _OMP_SESSION_TURN_KIND}: - return _read_file_turn_isolated( - target_kind, - target_value, - timeout_seconds=deadline, - cancel_event=cancel_event, - defer_cache=True, - ) - if target_kind == _PANE_TURN_KIND: - return _read_private_turn( - config, - target_value, - timeout_seconds=deadline, - raise_timeout=True, - cancel_event=cancel_event, - ) - return None - - -def _binding_still_matches( - config: Config, - item: _TurnRefreshItem, -) -> bool: - if config.db_path is None: - return False - try: - current = list_worker_bindings(config.db_path, config.host_id, backend="herdr") - except Exception: - return False - return any( - binding.private_fingerprint == item.key.private_fingerprint - and binding.worker_id == item.worker_id - and binding.worker_fingerprint == item.worker_fingerprint - and str(binding.turn_target_kind or "") == item.turn_target_kind - and str(binding.turn_target_value or "") == item.turn_target_value - for binding in current - ) - - -def _refresh_turn_binding( - config: Config, - binding: WorkerBinding, - *, - pane_target_override: str | None = None, - adapter_timeout_seconds: float | None = None, - cancel_event: threading.Event | None = None, - apply_deadline_monotonic: float | None = None, - observed_at: str | None = None, -) -> TurnRefreshResult: - """Read and atomically apply exactly one immutable private binding. - - Completion-authoritative callers may read through the live pane while the - immutable stored binding remains the ownership and transaction guard. This - prevents a prior agent-session generation from choosing the content source - after Herdr has reported completion for the current pane. - """ - if config.db_path is None or not _eligible_turn_binding(binding): - return TurnRefreshResult("missing", 0) - timeout_seconds = ( - config.herdr_timeout_seconds - if adapter_timeout_seconds is None - else float(adapter_timeout_seconds) - ) - if timeout_seconds <= 0: - return TurnRefreshResult("failed", 0) - item = _TurnRefreshItem.from_binding(binding) - read_binding = binding - if pane_target_override is not None: - live_pane_id = str(pane_target_override).strip() - if not live_pane_id: - return TurnRefreshResult("missing", 0) - read_binding = replace( - binding, - turn_target_kind=_PANE_TURN_KIND, - turn_target_value=live_pane_id, - ) - read_target_kind = str(read_binding.turn_target_kind or "") - current_time = observed_at or utc_timestamp() - grace_seconds = float(config.pending_stale_grace_seconds) - def failed_pending_result(status: Literal["failed", "timeout"]) -> TurnRefreshResult: - if read_target_kind != _PANE_TURN_KIND: - return TurnRefreshResult(status, 0) - if not _binding_still_matches(config, item): - return TurnRefreshResult("stale_binding", 0) - try: - applied_failure = apply_turn_refresh( - config.db_path, - config.host_id, - item.worker_id, - {}, - backend_pending_observation=PendingObservation("read_failed"), - expected_binding=binding, - deadline_monotonic=apply_deadline_monotonic, - cancelled=cancel_event.is_set if cancel_event is not None else None, - observed_at=current_time, - pending_stale_grace_seconds=grace_seconds, - turn_model=config.turn_model, - ) - except Exception: - return TurnRefreshResult(status, 0) - if applied_failure.stale_binding: - return TurnRefreshResult("stale_binding", 0) - return TurnRefreshResult( - status, - 0, - pending_changed=applied_failure.pending_changed, - ) - try: - observed = _read_turn_for_binding( - config, - read_binding, - timeout_seconds=timeout_seconds, - cancel_event=cancel_event, - ) - except _TurnReadTimeout: - return failed_pending_result("timeout") - except Exception: - return failed_pending_result("failed") - if observed is _UNCHANGED_TURN: - return TurnRefreshResult("unchanged", 0) - publication: _OmpCachePublication | _CodexCachePublication | None = None - if isinstance(observed, _ObservedFileTurn): - publication = observed.publication - observed = observed.content - if observed is None: - if not _binding_still_matches(config, item): - return TurnRefreshResult("stale_binding", 0) - _file_publication_commit(publication, None) - return TurnRefreshResult("unchanged", 0) - if observed is None: - return TurnRefreshResult("missing", 0) - if read_target_kind == _PANE_TURN_KIND: - content, pending_observation = _pop_backend_pending_observation(observed) - else: - content, pending_observation = dict(observed), None - if not _binding_still_matches(config, item): - return TurnRefreshResult("stale_binding", 0) - try: - if read_target_kind == _PANE_TURN_KIND: - applied = apply_turn_refresh( - config.db_path, - config.host_id, - item.worker_id, - content or {}, - backend_pending_observation=( - pending_observation - or PendingObservation("read_succeeded_no_prompt") - ), - expected_binding=binding, - deadline_monotonic=apply_deadline_monotonic, - cancelled=cancel_event.is_set if cancel_event is not None else None, - observed_at=current_time, - pending_stale_grace_seconds=grace_seconds, - turn_model=config.turn_model, - ) - elif content is not None: - applied = apply_turn_refresh( - config.db_path, - config.host_id, - item.worker_id, - content, - expected_binding=binding, - deadline_monotonic=apply_deadline_monotonic, - cancelled=cancel_event.is_set if cancel_event is not None else None, - observed_at=current_time, - turn_model=config.turn_model, - ) - else: - return TurnRefreshResult("missing", 0) - except Exception: - return TurnRefreshResult("failed", 0) - if applied.cancelled: - return TurnRefreshResult("timeout", 0) - if applied.stale_binding: - return TurnRefreshResult("stale_binding", 0) - if ( - pending_observation is not None - and pending_observation.kind == "read_failed" - ): - return TurnRefreshResult( - "failed", - int(applied.updated), - bool(applied.pending_changed), - ) - if publication is not None: - _file_publication_commit(publication, content) - updated = int(applied.updated) - return TurnRefreshResult( - "updated" if updated else "unchanged", - updated, - bool(applied.pending_changed), - ) - - -def refresh_turn_binding( - config: Config, - binding: WorkerBinding, - *, - adapter_timeout_seconds: float | None = None, -) -> TurnRefreshResult: - return _refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=adapter_timeout_seconds, - ) - - -def refresh_completed_pane_turn( - config: Config, - pane_id: str, - *, - terminal_id: str | None = None, - binding_private_fingerprint: str | None = None, - adapter_timeout_seconds: float | None = None, -) -> CompletedPaneTurnRefreshResult: - """Capture one completed turn through its current pane adapter target.""" - if config.db_path is None: - return CompletedPaneTurnRefreshResult("store_unavailable") - try: - bindings = list_worker_bindings( - config.db_path, - config.host_id, - backend="herdr", - ) - except Exception: - return CompletedPaneTurnRefreshResult("store_unavailable") - eligible = [binding for binding in bindings if _eligible_turn_binding(binding)] - candidate_tiers: list[list[WorkerBinding]] = [] - if binding_private_fingerprint is not None: - candidate_tiers.append( - [ - binding - for binding in eligible - if binding.private_fingerprint == binding_private_fingerprint - ] - ) - candidate_tiers.append( - [ - binding - for binding in eligible - if binding.turn_target_kind == _PANE_TURN_KIND - and str(binding.turn_target_value or "") == str(pane_id) - ] - ) - if terminal_id is not None: - candidate_tiers.append( - [ - binding - for binding in eligible - if str(binding.target_kind or "") == "terminal_id" - and str(binding.target_value or "") == str(terminal_id) - ] - ) - binding = None - for candidates in candidate_tiers: - if len(candidates) > 1: - return CompletedPaneTurnRefreshResult("binding_ambiguous") - if candidates: - binding = candidates[0] - break - if binding is None: - return CompletedPaneTurnRefreshResult("binding_missing") - refreshed = _refresh_turn_binding( - config, - binding, - pane_target_override=pane_id, - adapter_timeout_seconds=adapter_timeout_seconds, - ) - if refreshed.status not in {"updated", "unchanged", "missing"}: - return CompletedPaneTurnRefreshResult( - refreshed.status, - worker_id=binding.worker_id, - ) - try: - turn_id = latest_turn_id_for_worker( - config.db_path, - config.host_id, - binding.worker_id, - ) - except Exception: - return CompletedPaneTurnRefreshResult( - "store_unavailable", - worker_id=binding.worker_id, - ) - return CompletedPaneTurnRefreshResult( - refreshed.status, - worker_id=binding.worker_id, - refreshed_turn_id=turn_id, - ) - - -def refresh_structured_turn_content( - config: Config, - *, - adapter_timeout_seconds: float | None = None, - max_workers: int | None = None, - total_timeout_seconds: float | None = None, -) -> dict[str, Any]: - """Run one bounded unavailable-only fallback scan with active feeding.""" - if config.db_path is None: - return {"ok": False, "status": "store_unavailable", "updated": 0, "attempted": 0} - try: - bindings = list_worker_bindings(config.db_path, config.host_id, backend="herdr") - except Exception: - return {"ok": False, "status": "store_unavailable", "updated": 0, "attempted": 0} - _prune_omp_cache_for_bindings(bindings) - _prune_codex_cache_for_bindings(bindings) - turn_bindings = [binding for binding in bindings if _eligible_turn_binding(binding)] - worker_limit = max( - 1, - min( - 32, - int( - max_workers - if max_workers is not None - else getattr(config, "turn_refresh_workers", 4) - ), - ), - ) - adapter_deadline = ( - config.herdr_timeout_seconds - if adapter_timeout_seconds is None - else float(adapter_timeout_seconds) - ) - waves = (len(turn_bindings) + worker_limit - 1) // worker_limit - total_deadline = ( - max(1.0, (waves * adapter_deadline) + 1.0) - if total_timeout_seconds is None - else max(0.0, float(total_timeout_seconds)) - ) - deadline = time.monotonic() + total_deadline - shutdown_reserve = min(0.75, total_deadline / 2.0) - work_deadline = deadline - shutdown_reserve - pending_bindings = deque(turn_bindings) - active: dict[Future[TurnRefreshResult], WorkerBinding] = {} - submitted: list[Future[TurnRefreshResult]] = [] - updated = 0 - deadline_reached = False - pool = ThreadPoolExecutor( - max_workers=worker_limit, - thread_name_prefix="tendwire-turn-fallback", - ) - def fallback_cancelled() -> bool: - return time.monotonic() >= deadline - - - def collect(done: set[Future[TurnRefreshResult]]) -> None: - nonlocal updated - for future in done: - active.pop(future, None) - try: - updated += future.result().updated - except Exception: - pass - - while pending_bindings or active: - while ( - pending_bindings - and len(active) < worker_limit - and time.monotonic() < work_deadline - ): - binding = pending_bindings.popleft() - remaining_for_binding = max( - 0.001, - min(adapter_deadline, work_deadline - time.monotonic()), - ) - future = pool.submit( - _refresh_turn_binding, - config, - binding, - adapter_timeout_seconds=remaining_for_binding, - apply_deadline_monotonic=work_deadline, - ) - active[future] = binding - submitted.append(future) - if not active: - deadline_reached = bool(pending_bindings) - break - remaining = max(0.0, work_deadline - time.monotonic()) - done, _ = wait(active, timeout=remaining, return_when=FIRST_COMPLETED) - if not done: - deadline_reached = True - break - collect(done) - if time.monotonic() >= work_deadline and (pending_bindings or active): - deadline_reached = True - break - if deadline_reached: - for future in active: - future.cancel() - remaining = max(0.0, deadline - time.monotonic()) - done, _ = wait(active, timeout=remaining) - collect(done) - # Supported readers terminate and reap at their per-binding deadlines - # inside the reserved shutdown window. Never use an executor context or - # wait=True: the absolute fallback deadline owns return latency. - pool.shutdown(wait=False, cancel_futures=True) - if not deadline_reached: - try: - prune_backend_pending( - config.db_path, - config.host_id, - { - binding.private_fingerprint - for binding in bindings - if binding.turn_target_kind == _PANE_TURN_KIND - }, - deadline_monotonic=deadline, - cancelled=fallback_cancelled, - observed_at=utc_timestamp(), - ) - except Exception: - pass - deadline_reached = fallback_cancelled() - return { - "ok": not deadline_reached, - "status": "deadline_exceeded" if deadline_reached else "ok", - "updated": updated, - "attempted": len(submitted), - } - - -class TurnIngestionScheduler: - """One bounded coordinator and fixed worker pool for turn ingestion.""" - - def __init__( - self, - config: Config, - *, - refresh_interval_seconds: float | None = None, - max_workers: int | None = None, - queue_capacity: int = _TURN_INGESTION_QUEUE_CAPACITY, - adapter_timeout_seconds: float | None = None, - clock: Callable[[], float] = time.monotonic, - utc_clock: Callable[[], str] = utc_timestamp, - reader: Callable[..., TurnRefreshResult] = refresh_turn_binding, - ) -> None: - self.config = config - self.refresh_interval_seconds = float( - refresh_interval_seconds - if refresh_interval_seconds is not None - else getattr(config, "turn_refresh_interval_seconds", 2.0) - ) - self.max_workers = int( - max_workers - if max_workers is not None - else getattr(config, "turn_refresh_workers", 4) - ) - self.queue_capacity = int(queue_capacity) - self.adapter_timeout_seconds = float( - config.herdr_timeout_seconds - if adapter_timeout_seconds is None - else adapter_timeout_seconds - ) - if self.refresh_interval_seconds <= 0: - raise ValueError("refresh_interval_seconds must be positive") - if not 1 <= self.max_workers <= 32: - raise ValueError("max_workers must be between 1 and 32") - if self.queue_capacity <= 0: - raise ValueError("queue_capacity must be positive") - if self.adapter_timeout_seconds <= 0: - raise ValueError("adapter_timeout_seconds must be positive") - self._clock = clock - self._utc_clock = utc_clock - self._reader = reader - self._uses_default_reader = reader is refresh_turn_binding - self._cancel_event = threading.Event() - self._condition = threading.Condition(threading.RLock()) - self._queue: deque[_TurnRefreshItem] = deque() - self._queued: set[TurnRefreshKey] = set() - self._running: dict[TurnRefreshKey, tuple[_TurnRefreshItem, Future[TurnRefreshResult], float]] = {} - self._dirty: set[TurnRefreshKey] = set() - self._latest: dict[TurnRefreshKey, _TurnRefreshItem] = {} - self._deferred_reruns: dict[TurnRefreshKey, _TurnRefreshItem] = {} - self._executor: ThreadPoolExecutor | None = None - self._coordinator: threading.Thread | None = None - self._accepting = False - self._started = False - self._force_exit = False - self._rescan_requested = False - self._next_scan = 0.0 - self._scan_cursor = 0 - self._stopping = False - self._refreshed = 0 - self._failed = 0 - self._timed_out = 0 - self._coalesced = 0 - self._queue_full = 0 - self._last_success: str | None = None - self._last_success_clock: float | None = None - self._last_duration_ms: float | None = None - self._last_completed_outcome: str | None = None - self._consecutive_failures = 0 - self._scan_failed = False - self._scan_retry_remaining = 1 - self._binding_retry_remaining: dict[TurnRefreshKey, int] = {} - self._binding_retry_due: dict[ - TurnRefreshKey, - tuple[_TurnRefreshItem, float], - ] = {} - - def start(self) -> None: - with self._condition: - if self._started: - return - if self._stopping: - raise RuntimeError("turn ingestion scheduler cannot restart") - self._executor = ThreadPoolExecutor( - max_workers=self.max_workers, - thread_name_prefix="tendwire-turn-ingestion", - ) - self._accepting = True - self._started = True - self._rescan_requested = True - self._next_scan = self._clock() - self._coordinator = threading.Thread( - target=self._coordinate, - name="tendwire-turn-coordinator", - daemon=False, - ) - self._coordinator.start() - - def request_refresh(self) -> None: - with self._condition: - if not self._accepting: - return - if self._rescan_requested: - self._coalesced += 1 - self._rescan_requested = True - self._condition.notify() - - def _enqueue_locked(self, item: _TurnRefreshItem) -> None: - key = item.key - self._latest[key] = item - if key in self._running: - if key in self._dirty: - self._coalesced += 1 - else: - self._dirty.add(key) - return - if key in self._binding_retry_due: - _previous, due = self._binding_retry_due[key] - self._binding_retry_due[key] = (item, due) - self._coalesced += 1 - return - if key in self._deferred_reruns: - self._deferred_reruns[key] = item - self._coalesced += 1 - return - if key in self._queued: - self._coalesced += 1 - return - if len(self._queue) >= self.queue_capacity: - self._queue_full += 1 - self._latest.pop(key, None) - return - self._queue.append(item) - self._queued.add(key) - - def _scan_bindings(self) -> None: - if self.config.db_path is None: - with self._condition: - self._failed += 1 - self._scan_failed = True - return - try: - bindings = list_worker_bindings( - self.config.db_path, - self.config.host_id, - backend="herdr", - ) - except Exception: - with self._condition: - self._failed += 1 - self._scan_failed = True - if self._accepting and self._scan_retry_remaining > 0: - self._scan_retry_remaining -= 1 - self._next_scan = min( - self._next_scan, - self._clock() + 0.05, - ) - self._condition.notify() - return - _prune_omp_cache_for_bindings(bindings) - _prune_codex_cache_for_bindings(bindings) - items = [ - _TurnRefreshItem.from_binding(binding) - for binding in bindings - if _eligible_turn_binding(binding) - ] - with self._condition: - self._scan_failed = False - self._scan_retry_remaining = 1 - live_keys = {item.key for item in items} - for key in tuple(self._binding_retry_remaining): - if key not in live_keys: - self._binding_retry_remaining.pop(key, None) - self._binding_retry_due.pop(key, None) - if self._accepting: - if items: - offset = self._scan_cursor % len(items) - items = items[offset:] + items[:offset] - # Rotate each full scan so bindings dropped at the explicit - # queue cap are first on a later cadence instead of starving. - self._scan_cursor = (offset + self.queue_capacity) % len(items) - for item in items: - self._enqueue_locked(item) - self._dispatch_locked() - self._condition.notify() - # Lock order: no scheduler condition is held during store reads/writes, - # adapter/process waits, SQLite transactions, or future waits. - try: - prune_backend_pending( - self.config.db_path, - self.config.host_id, - { - binding.private_fingerprint - for binding in bindings - if binding.turn_target_kind == _PANE_TURN_KIND - }, - cancelled=self._cancel_event.is_set, - observed_at=self._utc_clock(), - ) - except Exception: - pass - - def _binding_for_item(self, item: _TurnRefreshItem) -> WorkerBinding | None: - if self.config.db_path is None: - return None - try: - bindings = list_worker_bindings( - self.config.db_path, - self.config.host_id, - backend="herdr", - ) - except Exception as exc: - raise _BindingLookupFailed from exc - return next( - ( - binding - for binding in bindings - if binding.private_fingerprint == item.key.private_fingerprint - and binding.worker_id == item.worker_id - and binding.worker_fingerprint == item.worker_fingerprint - and str(binding.turn_target_kind or "") == item.turn_target_kind - and str(binding.turn_target_value or "") == item.turn_target_value - ), - None, - ) - - def _run_item(self, item: _TurnRefreshItem) -> TurnRefreshResult: - try: - binding = self._binding_for_item(item) - except _BindingLookupFailed: - return TurnRefreshResult("failed", 0, retry_binding_lookup=True) - if binding is None: - return TurnRefreshResult("stale_binding", 0) - try: - if self._uses_default_reader: - result = _refresh_turn_binding( - self.config, - binding, - adapter_timeout_seconds=self.adapter_timeout_seconds, - cancel_event=self._cancel_event, - observed_at=self._utc_clock(), - ) - else: - result = self._reader( - self.config, - binding, - adapter_timeout_seconds=self.adapter_timeout_seconds, - ) - except Exception: - result = TurnRefreshResult("failed", 0) - return TurnRefreshResult( - result.status, - result.updated, - result.pending_changed, - binding_validated=True, - ) - - def _future_finished(self, _future: Future[TurnRefreshResult]) -> None: - with self._condition: - self._condition.notify() - - def _collect_finished_locked(self) -> None: - now = self._clock() - for key, (item, future, started_at) in list(self._running.items()): - if not future.done(): - continue - self._running.pop(key, None) - self._last_duration_ms = max(0.0, (now - started_at) * 1000.0) - try: - result = future.result() - except Exception: - result = TurnRefreshResult("failed", 0) - self._last_completed_outcome = result.status - if result.binding_validated or result.status == "stale_binding": - self._binding_retry_remaining.pop(key, None) - self._binding_retry_due.pop(key, None) - if result.status == "timeout": - self._timed_out += 1 - self._consecutive_failures += 1 - elif result.status == "failed": - self._failed += 1 - self._consecutive_failures += 1 - elif result.status == "stale_binding": - self._failed += 1 - else: - self._refreshed += 1 - self._last_success = utc_timestamp() - self._last_success_clock = now - self._consecutive_failures = 0 - if result.retry_binding_lookup: - retry_item = self._latest.get(key, item) - self._dirty.discard(key) - self._latest.pop(key, None) - remaining = self._binding_retry_remaining.get(key, 1) - if self._accepting and remaining > 0: - self._binding_retry_remaining[key] = remaining - 1 - self._binding_retry_due[key] = (retry_item, now + 0.05) - continue - if key in self._dirty: - self._dirty.discard(key) - rerun = self._latest.get(key, item) - if self._accepting and len(self._queue) < self.queue_capacity: - self._queue.append(rerun) - self._queued.add(key) - elif self._accepting: - self._deferred_reruns[key] = rerun - self._queue_full += 1 - else: - self._latest.pop(key, None) - else: - self._latest.pop(key, None) - - def _promote_deferred_locked(self) -> None: - while ( - self._accepting - and self._deferred_reruns - and len(self._queue) < self.queue_capacity - ): - key, item = next(iter(self._deferred_reruns.items())) - del self._deferred_reruns[key] - self._queue.append(item) - self._queued.add(key) - - def _promote_binding_retries_locked(self, now: float) -> None: - for key, (item, due) in tuple(self._binding_retry_due.items()): - if due > now: - continue - self._binding_retry_due.pop(key, None) - self._enqueue_locked(item) - - - def _dispatch_locked(self) -> None: - executor = self._executor - if executor is None: - return - self._promote_deferred_locked() - self._promote_binding_retries_locked(self._clock()) - while self._accepting and self._queue and len(self._running) < self.max_workers: - item = self._queue.popleft() - self._queued.discard(item.key) - self._binding_retry_due.pop(item.key, None) - self._promote_deferred_locked() - future = executor.submit(self._run_item, item) - self._running[item.key] = (item, future, self._clock()) - future.add_done_callback(self._future_finished) - - def _coordinate(self) -> None: - while True: - run_scan = False - with self._condition: - self._collect_finished_locked() - if self._force_exit: - return - if not self._accepting and not self._running: - return - now = self._clock() - self._promote_binding_retries_locked(now) - if self._accepting and ( - self._rescan_requested or now >= self._next_scan - ): - run_scan = True - self._rescan_requested = False - self._next_scan = now + self.refresh_interval_seconds - self._dispatch_locked() - if not run_scan: - timeout: float | None = None - if self._accepting: - timeout = max(0.0, self._next_scan - self._clock()) - if self._binding_retry_due: - retry_timeout = max( - 0.0, - min(due for _item, due in self._binding_retry_due.values()) - - self._clock(), - ) - timeout = min(timeout, retry_timeout) - self._condition.wait(timeout) - continue - self._scan_bindings() - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - timeout = ( - self.adapter_timeout_seconds + 1.0 - if flush_timeout_seconds is None - else max(0.0, float(flush_timeout_seconds)) - ) - with self._condition: - if self._stopping: - return - self._stopping = True - self._accepting = False - self._rescan_requested = False - self._queue.clear() - self._queued.clear() - self._dirty.clear() - self._deferred_reruns.clear() - self._latest.clear() - self._binding_retry_due.clear() - self._binding_retry_remaining.clear() - self._cancel_event.set() - self._condition.notify_all() - coordinator = self._coordinator - executor = self._executor - coordinator_clean = True - if coordinator is not None: - coordinator.join(timeout) - if coordinator.is_alive(): - coordinator_clean = False - with self._condition: - self._force_exit = True - self._condition.notify_all() - coordinator.join(0.25) - coordinator_clean = not coordinator.is_alive() - if executor is not None: - executor.shutdown(wait=coordinator_clean, cancel_futures=True) - - def operational_status(self) -> Mapping[str, Any]: - with self._condition: - now = self._clock() - stale_age = ( - None - if self._last_success_clock is None - else max(0.0, now - self._last_success_clock) - ) - if self._stopping: - status = "stopping" - elif self._scan_failed or self._consecutive_failures > 0: - status = "degraded" - elif self._last_success_clock is None: - status = "stale" - elif stale_age is not None and stale_age > max( - self.refresh_interval_seconds * 3.0, - self.adapter_timeout_seconds * 2.0, - ): - status = "stale" - else: - status = "healthy" - return { - "status": status, - "queue_depth": len(self._queue), - "active": len(self._running), - "refreshed": self._refreshed, - "failed": self._failed, - "timed_out": self._timed_out, - "coalesced": self._coalesced, - "queue_full": self._queue_full, - "last_success": self._last_success, - "last_duration_ms": self._last_duration_ms, - "stale_age_seconds": stale_age, - "max_workers": self.max_workers, - "queue_capacity": self.queue_capacity, - "refresh_interval_seconds": self.refresh_interval_seconds, - "adapter_timeout_seconds": self.adapter_timeout_seconds, - } diff --git a/src/tendwire/cli.py b/src/tendwire/cli.py index 6e1d64e..95dc522 100644 --- a/src/tendwire/cli.py +++ b/src/tendwire/cli.py @@ -15,18 +15,8 @@ from pathlib import Path from typing import Any, Mapping -from .backends.herdr_cli import ( - bindings_from_workers, - diagnose_herdr, - fetch_herdr_snapshot_observation, - fetch_herdr_state, - herdr_backend_health, - rehydrate_workers_from_bindings, -) -from .backends.herdr_turns import refresh_structured_turn_content from .config import Config, load_config from .core.actions import CommandContext, execute_command -from .core.attention import attention_payload_from_snapshot from .core.commands import ( STATUS_BACKEND_UNAVAILABLE, CommandEnvelope, @@ -34,42 +24,26 @@ parse_command_request, validate_request, ) -from .core.projector import project_from_observations from .core.models import ( - BackendHealth, - WorkerBinding, public_json_dumps, sanitize_public_mapping, - separate_duplicate_worker_bindings, ) from .core.turns import ( TURN_DELTA_DEFAULT_LIMIT, TURN_DELTA_MAX_LIMIT, TURN_LIST_DEFAULT_LIMIT, TURN_LIST_MAX_LIMIT, - turns_payload_from_snapshot, ) from .local_state import repair_config_state from .store.sqlite import ( CompactionOptions, - attention_payload_from_store, compact_store, - expire_stale_worker_bindings, - latest_healthy_backend_snapshot, - latest_snapshot, - list_worker_bindings, - pending_payload_from_store, run_store_maintenance, store_status, tail_event_metadata, - turns_payload_from_store, - turn_delta_payload_from_store, - upsert_worker_bindings, ) -_HERDR_BACKEND = "herdr" -_DEFAULT_FETCH_HERDR_STATE = fetch_herdr_state _DAEMON_FAST_CLIENT_TIMEOUT_SECONDS = 2.0 _DAEMON_CONTENT_CLIENT_TIMEOUT_SECONDS = 10.0 _DAEMON_CONNECTOR_CLIENT_TIMEOUT_SECONDS = 30.0 @@ -153,13 +127,13 @@ def _build_parser() -> argparse.ArgumentParser: "--herdr-timeout", dest="herdr_timeout_seconds", default=None, - help="Seconds to wait for each Herdr CLI probe (default: 5.0).", + help="Seconds to wait for each Herdr socket request (default: 5.0).", ) parser.add_argument( "--socket-path", dest="socket_path", default=None, - help="Unix socket path for daemon-backed requests when explicitly enabled.", + help="Unix socket path for daemon requests (default: data-dir/tendwire.sock).", ) subparsers = parser.add_subparsers(dest="command") @@ -175,19 +149,6 @@ def _build_parser() -> argparse.ArgumentParser: default=True, help="Print snapshot as JSON (default).", ) - snapshot_parser.add_argument( - "--store", - dest="store_snapshot", - action="store_true", - default=False, - help="Persist the snapshot to the sqlite store without changing stdout.", - ) - snapshot_parser.add_argument( - "--db-path", - dest="db_path", - default=None, - help="SQLite database path to use with --store (default: config path).", - ) attention_parser = subparsers.add_parser( "attention", @@ -200,19 +161,6 @@ def _build_parser() -> argparse.ArgumentParser: default=True, help="Print attention as JSON (default).", ) - attention_parser.add_argument( - "--store", - dest="store_snapshot", - action="store_true", - default=False, - help="Persist a fresh snapshot before listing store-backed attention.", - ) - attention_parser.add_argument( - "--db-path", - dest="db_path", - default=None, - help="SQLite database path for store-backed attention (default: config path).", - ) turns_parser = subparsers.add_parser( "turns", @@ -538,184 +486,6 @@ def add_common(action_parser: argparse.ArgumentParser) -> None: action_parser.add_argument("--lease-seconds", dest="lease_seconds", type=int, default=None) -def _load_worker_bindings(config: Config) -> list[WorkerBinding]: - if config.db_path is None: - return [] - return list_worker_bindings( - config.db_path, - config.host_id, - backend=_HERDR_BACKEND, - ) - - -def _fetch_state_with_bindings( - config: Config, - stored_bindings: list[WorkerBinding], -) -> tuple[list[Any], list[Any], list[WorkerBinding]]: - try: - result = fetch_herdr_state( - config, - stored_bindings=stored_bindings, - include_bindings=True, - ) - except TypeError: - spaces, workers = fetch_herdr_state(config) - return spaces, workers, bindings_from_workers(config, workers) - - if len(result) == 3: - spaces, workers, bindings = result - return spaces, workers, bindings - spaces, workers = result - return spaces, workers, bindings_from_workers(config, workers) - - -def _legacy_backend_health(spaces: list[Any], workers: list[Any]) -> list[BackendHealth]: - return [ - herdr_backend_health( - "healthy_non_empty" if spaces or workers else "unknown", - spaces=spaces, - workers=workers, - ) - ] - - -def _fetch_snapshot_observation_with_bindings( - config: Config, - stored_bindings: list[WorkerBinding], -) -> tuple[list[Any], list[Any], list[WorkerBinding], list[BackendHealth], bool]: - complete_barrier = False - if fetch_herdr_state is not _DEFAULT_FETCH_HERDR_STATE: - spaces, workers, bindings = _fetch_state_with_bindings(config, stored_bindings) - backend_health = _legacy_backend_health(spaces, workers) - else: - try: - observation = fetch_herdr_snapshot_observation( - config, - stored_bindings=stored_bindings, - ) - except TypeError: - spaces, workers, bindings = _fetch_state_with_bindings(config, stored_bindings) - backend_health = _legacy_backend_health(spaces, workers) - else: - spaces = list(getattr(observation, "spaces", []) or []) - workers = list(getattr(observation, "workers", []) or []) - bindings = list(getattr(observation, "bindings", []) or []) - backend_health = list(getattr(observation, "backend_health", []) or []) - complete_barrier = bool(backend_health) - if not backend_health: - backend_health = _legacy_backend_health(spaces, workers) - - health = _herdr_health_from_items(backend_health) - if health.status == "healthy": - return spaces, workers, bindings, backend_health, complete_barrier - - # Failed observations are not an authority for routing or continuity. - # Never persist their bindings, and retain the last authenticated public - # state when one has already been stored. - bindings = [] - if config.db_path is None: - return spaces, workers, bindings, backend_health, complete_barrier - - db_path = Path(config.db_path) - latest = latest_snapshot(db_path, config.host_id) - if latest is not None: - latest_health = _herdr_health_from_items(list(latest.backend_health)) - if latest_health.outcome == "continuity_unavailable": - health = latest_health - - previous = latest_healthy_backend_snapshot( - db_path, - config.host_id, - backend=_HERDR_BACKEND, - ) - if previous is not None: - spaces = list(previous.spaces) - workers = list(previous.workers) - - retained_health = herdr_backend_health( - health.outcome, - observed_at=health.observed_at, - message=health.message, - spaces=spaces, - workers=workers, - ) - backend_health = [ - retained_health if item.name == _HERDR_BACKEND else item - for item in backend_health - ] - if not any(item.name == _HERDR_BACKEND for item in backend_health): - backend_health.append(retained_health) - return spaces, workers, bindings, backend_health, complete_barrier - - - - -def _herdr_health_from_items(items: list[BackendHealth]) -> BackendHealth: - for item in items: - if getattr(item, "name", "") == _HERDR_BACKEND: - return item - return herdr_backend_health("unknown") - - - - -def observe_public_snapshot( - config: Config, - *, - store_snapshot: bool = False, -) -> Any: - """Build the public snapshot through the existing one-shot observation path.""" - # Always seed observation with stored bindings: they are what keeps public - # worker ids stable across snapshots. Skipping them re-letters duplicate - # worker names (claude, claude-1, ...) from scratch on every observation. - stored_bindings = _load_worker_bindings(config) - spaces, workers, bindings, backend_health, complete_barrier = ( - _fetch_snapshot_observation_with_bindings( - config, - stored_bindings, - ) - ) - snapshot = project_from_observations( - config, - spaces=spaces, - workers=workers, - backend_health=backend_health, - ) - - if store_snapshot: - from .store.sqlite import SnapshotObservationContext, save_snapshot - - if config.db_path is None: - raise RuntimeError("snapshot persistence requires a db path") - health = _herdr_health_from_items(backend_health) - authority = ( - "complete" - if complete_barrier - and health.status == "healthy" - and health.outcome in {"healthy_non_empty", "empty_healthy"} - else "none" - ) - save_snapshot( - config.db_path, - snapshot, - turn_model=config.turn_model, - observation=SnapshotObservationContext( - authority=authority, - observed_at=health.observed_at, - ), - worker_bindings=bindings, - binding_backend=_HERDR_BACKEND, - binding_observation_authoritative=health.status == "healthy", - binding_workers_present=bool(workers), - ) - - return snapshot - - -def _current_public_snapshot(config: Config) -> Any: - return observe_public_snapshot(config) - - def _try_daemon_attempt( config: Config, method: str, @@ -723,10 +493,10 @@ def _try_daemon_attempt( *, preserve_content_text: bool = False, ) -> _DaemonAttempt: - """Return a daemon result only when a daemon socket was explicitly selected.""" - if config.socket_path is None: - return _DaemonAttempt(error_kind="unavailable", request_started=False) - socket_path = config.socket_path + """Return one result from the daemon's authoritative socket API.""" + from .daemon import default_socket_path + + socket_path = default_socket_path(config) try: from .daemon_api import ( @@ -801,61 +571,20 @@ def _try_daemon_attempt( return _DaemonAttempt(error_kind="protocol", request_started=True) -def _try_daemon_result( - config: Config, - method: str, - params: dict[str, Any] | None = None, -) -> dict[str, Any] | None: - """Return only a daemon result, preserving read-only fallback behavior.""" - return _try_daemon_attempt(config, method, params).result - - -def _persist_binding_observation( - config: Config, - bindings: list[WorkerBinding], - *, - observed_at: str, - workers_present: bool, - authoritative: bool = True, -) -> list[WorkerBinding]: - bindings = separate_duplicate_worker_bindings(bindings) - if config.db_path is None: - return bindings - if bindings: - upsert_worker_bindings(config.db_path, bindings) - if authoritative and (bindings or not workers_present): - expire_stale_worker_bindings( - config.db_path, - config.host_id, - backend=_HERDR_BACKEND, - current_private_fingerprints=[binding.private_fingerprint for binding in bindings], - now=observed_at, - ) - return bindings - - def cmd_snapshot( config: Config, *, json_output: bool = True, - store_snapshot: bool = False, ) -> int: - """Build and print a neutral snapshot.""" + """Read the daemon's current neutral snapshot.""" if json_output: - daemon_attempt = ( - _DaemonAttempt(error_kind="unavailable", request_started=False) - if store_snapshot - else _try_daemon_attempt(config, "snapshot.get") - ) + daemon_attempt = _try_daemon_attempt(config, "snapshot.get") if daemon_attempt.result is not None: payload = daemon_attempt.result code = 0 elif daemon_attempt.response_error is not None: payload = daemon_attempt.response_error code = 1 - elif daemon_attempt.request_started is False: - payload = observe_public_snapshot(config, store_snapshot=store_snapshot).to_dict() - code = 0 elif daemon_attempt.error_kind == "timeout": payload = { "schema_version": 2, @@ -871,10 +600,22 @@ def cmd_snapshot( payload = { "schema_version": 2, "ok": False, - "status": "daemon_protocol_error", + "status": ( + "daemon_unavailable" + if daemon_attempt.request_started is False + else "daemon_protocol_error" + ), "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", + "code": ( + "daemon_unavailable" + if daemon_attempt.request_started is False + else "daemon_protocol_error" + ), + "message": ( + "Tendwire daemon is unavailable" + if daemon_attempt.request_started is False + else "Tendwire daemon returned an invalid response" + ), }, } code = 1 @@ -984,6 +725,11 @@ def _restore_cli_content_text( and original.get("availability") == "complete" ): sanitized["text"] = text + content_revision = original.get("content_revision") + if isinstance(content_revision, str) and re.fullmatch( + r"twrev1\.[A-Za-z0-9_-]+", content_revision + ): + sanitized["content_revision"] = content_revision def _restore_cli_plan_token( @@ -1007,12 +753,17 @@ def _restore_cli_plan_token( r"twfinal1\.[A-Za-z0-9_-]+", final_identity ): sanitized["final_identity"] = final_identity + content_revision = original.get("content_revision") + if isinstance(content_revision, str) and re.fullmatch( + r"twrev1\.[A-Za-z0-9_-]+", content_revision + ): + sanitized["content_revision"] = content_revision delivery_key = original.get("key") if isinstance(delivery_key, str) and re.fullmatch( r"turn-final:revision:twfinal1\.[A-Za-z0-9_-]+", delivery_key ): sanitized["key"] = delivery_key - for nested_key in ("turn", "final", "payload"): + for nested_key in ("turn", "final", "payload", "content"): nested_original = original.get(nested_key) nested_sanitized = sanitized.get(nested_key) if isinstance(nested_original, dict) and isinstance(nested_sanitized, dict): @@ -1053,6 +804,41 @@ def _content_payload_json(payload: dict[str, Any], *, indent: int | None = None) ) +def _daemon_read_payload( + attempt: _DaemonAttempt, + *, + schema_version: int = 1, + extra: Mapping[str, Any] | None = None, +) -> dict[str, Any]: + if attempt.result is not None: + return attempt.result + if attempt.response_error is not None: + return attempt.response_error + status = ( + "daemon_timeout" + if attempt.error_kind == "timeout" + else "daemon_unavailable" + if attempt.request_started is False + else "daemon_protocol_error" + ) + payload: dict[str, Any] = { + "schema_version": schema_version, + "ok": False, + "status": status, + "error": { + "code": status, + "message": { + "daemon_timeout": "Tendwire daemon request timed out", + "daemon_unavailable": "Tendwire daemon is unavailable", + "daemon_protocol_error": "Tendwire daemon returned an invalid response", + }[status], + }, + } + if extra: + payload.update(extra) + return payload + + def cmd_turns( config: Config, *, @@ -1072,66 +858,17 @@ def cmd_turns( "cursor": cursor, "since": since, } - daemon_attempt = _try_daemon_attempt(config, "turn.list", params) - if daemon_attempt.result is not None: - payload = daemon_attempt.result - elif daemon_attempt.response_error is not None: - payload = daemon_attempt.response_error - elif ( - daemon_attempt.error_kind in {"unavailable", "timeout"} - and daemon_attempt.request_started is False - ): - if config.db_path is None: - payload = { - "schema_version": schema_version, - "host_id": config.host_id, - "ok": False, - "status": "store_unavailable", - } - else: - if cursor is None and since is None: - refresh_structured_turn_content( - config, - adapter_timeout_seconds=config.herdr_timeout_seconds, - max_workers=config.turn_refresh_workers, - total_timeout_seconds=config.herdr_timeout_seconds + 1.0, - ) - payload = turns_payload_from_store( - config.db_path, - config.host_id, - schema_version=schema_version, - limit=limit, - cursor=cursor, - since=since, - turn_refresh_interval_seconds=config.turn_refresh_interval_seconds, - turn_model=config.turn_model, - ) - elif daemon_attempt.error_kind == "timeout": - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_timeout", - "error": { - "code": "daemon_timeout", - "message": "Tendwire daemon request timed out", - }, - } - else: - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", - }, - } + payload = _daemon_read_payload( + _try_daemon_attempt(config, "turn.list", params), + schema_version=schema_version, + extra={"host_id": config.host_id}, + ) print(_turn_list_payload_json(payload, indent=2)) return 0 if payload.get("ok") is not False else 1 def cmd_turn_content_get(config: Config, args: argparse.Namespace) -> int: - """Fetch one bounded canonical content page with daemon/store parity.""" + """Fetch one bounded canonical content page from the daemon.""" params: dict[str, Any] = { "schema_version": 1, "turn_id": args.turn_id, @@ -1140,53 +877,14 @@ def cmd_turn_content_get(config: Config, args: argparse.Namespace) -> int: } if args.cursor is not None: params["cursor"] = args.cursor - daemon_attempt = _try_daemon_attempt( - config, - "turn.content.get", - params, - preserve_content_text=True, - ) - if daemon_attempt.result is not None: - payload = daemon_attempt.result - elif daemon_attempt.response_error is not None: - payload = daemon_attempt.response_error - elif daemon_attempt.error_kind not in {"unavailable", "timeout"}: - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "daemon returned an invalid response", - }, - } - elif config.db_path is None: - payload = { - "schema_version": 1, - "ok": False, - "status": "store_unavailable", - "error": { - "code": "store_unavailable", - "message": "command requires --db-path or a reachable daemon", - }, - } - else: - from .store.sqlite import get_turn_content, init_store - - init_store( - config.db_path, - connector_ack_ttl_seconds=config.connector_ack_ttl_seconds, - ) - payload = get_turn_content( - config.db_path, - config.host_id, - turn_id=args.turn_id, - content_revision=args.content_revision, - field=args.field, - cursor=args.cursor, - schema_version=1, - turn_model=config.turn_model, + payload = _daemon_read_payload( + _try_daemon_attempt( + config, + "turn.content.get", + params, + preserve_content_text=True, ) + ) print(_content_payload_json(payload, indent=2)) return 0 if payload.get("ok") is not False and isinstance(payload.get("text"), str) else 1 @@ -1195,27 +893,14 @@ def cmd_attention( config: Config, *, json_output: bool = True, - store_snapshot: bool = False, ) -> int: """Print neutral public attention items.""" if not json_output: print("error: only --json output is supported", file=sys.stderr) return 2 - if not store_snapshot: - daemon_result = _try_daemon_result(config, "attention.list") - if daemon_result is not None: - print(public_json_dumps(daemon_result, indent=2)) - return 0 - if store_snapshot: - observe_public_snapshot(config, store_snapshot=True) - if config.db_path is not None: - payload = attention_payload_from_store(config.db_path, config.host_id) - if payload is not None: - print(public_json_dumps(payload, indent=2)) - return 0 - snapshot = _current_public_snapshot(config) - print(public_json_dumps(attention_payload_from_snapshot(snapshot), indent=2)) - return 0 + payload = _daemon_read_payload(_try_daemon_attempt(config, "attention.list")) + print(public_json_dumps(payload, indent=2)) + return 0 if payload.get("ok") is not False else 1 def cmd_pending( @@ -1223,84 +908,27 @@ def cmd_pending( *, json_output: bool = True, ) -> int: - """Print pending interactions from one daemon attempt or durable fallback.""" + """Print pending interactions from the daemon.""" if not json_output: print("error: only --json output is supported", file=sys.stderr) return 2 - daemon_attempt = _try_daemon_attempt(config, "pending.list") - if daemon_attempt.result is not None: - payload = daemon_attempt.result - elif daemon_attempt.response_error is not None: - payload = daemon_attempt.response_error - elif ( - daemon_attempt.error_kind == "unavailable" - and daemon_attempt.request_started is False - ): - payload = pending_payload_from_store(config.db_path, config.host_id) - elif daemon_attempt.error_kind == "timeout": - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_timeout", - "error": { - "code": "daemon_timeout", - "message": "Tendwire daemon request timed out", - }, - } - else: - payload = { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", - }, - } + payload = _daemon_read_payload(_try_daemon_attempt(config, "pending.list")) print(public_json_dumps(payload, indent=2)) return 0 if payload.get("ok") is not False else 1 def cmd_turn_delta(config: Config, args: argparse.Namespace) -> int: - """Read one delta page via daemon, with read-only store fallback.""" + """Read one delta page from the daemon.""" params = { "limit": args.limit, "watermark": args.watermark, "cursor": args.cursor, } - daemon_attempt = _try_daemon_attempt(config, "turn.delta", params) - if daemon_attempt.result is not None: - payload = daemon_attempt.result - elif daemon_attempt.response_error is not None: - payload = daemon_attempt.response_error - elif ( - daemon_attempt.error_kind in {"unavailable", "timeout"} - and daemon_attempt.request_started is False - and config.db_path is not None - ): - payload = turn_delta_payload_from_store( - config.db_path, - config.host_id, - watermark=args.watermark, - cursor=args.cursor, - limit=args.limit, - turn_model=config.turn_model, - ) - elif daemon_attempt.error_kind == "timeout": - payload = { - "schema_version": 1, - "projection_schema_version": 2, - "ok": False, - "status": "daemon_timeout", - } - else: - payload = { - "schema_version": 1, - "projection_schema_version": 2, - "ok": False, - "status": "store_unavailable" if config.db_path is None else "daemon_protocol_error", - } + payload = _daemon_read_payload( + _try_daemon_attempt(config, "turn.delta", params), + extra={"projection_schema_version": 2}, + ) print(_turn_delta_payload_json(payload, indent=2)) return 0 if payload.get("ok") is not False else 1 @@ -1310,13 +938,13 @@ def cmd_doctor( *, json_output: bool = True, ) -> int: - """Run read-only backend diagnostics and print a JSON result.""" + """Read daemon and lifecycle diagnostics.""" if not json_output: print("error: only --json output is supported", file=sys.stderr) return 2 - payload = diagnose_herdr(config) + payload = _daemon_read_payload(_try_daemon_attempt(config, "health.get")) print(json.dumps(payload, indent=2, sort_keys=True)) - return 0 if payload.get("status") == "ok" else 1 + return 0 if payload.get("status") == "ok" and payload.get("ok") is not False else 1 def command_envelope_from_payload(config: Config, payload: str) -> CommandEnvelope: @@ -1334,7 +962,7 @@ def command_envelope_from_payload(config: Config, payload: str) -> CommandEnvelo if validation_error is not None: return CommandEnvelope.from_error(request, validation_error) - if request.action in {"send_instruction", "answer_pending", "answer_decision"}: + if request.action in {"send_instruction", "answer_decision"}: from .command_submission import submit_command return submit_command(config, payload) @@ -1345,31 +973,9 @@ def command_envelope_from_payload(config: Config, payload: str) -> CommandEnvelo CommandContext(host_id=config.host_id, workers=[]), ) - stored_bindings = _load_worker_bindings(config) - spaces, workers, current_bindings, backend_health, _complete_barrier = ( - _fetch_snapshot_observation_with_bindings( - config, - stored_bindings, - ) - ) - workers = rehydrate_workers_from_bindings( - workers, - current_bindings, - stored_bindings, - ) - snapshot = project_from_observations( - config, - spaces=spaces, - workers=workers, - backend_health=backend_health, - ) - return execute_command( + return CommandEnvelope.from_error( request, - CommandContext( - host_id=config.host_id, - workers=workers, - snapshot=snapshot, - ), + error_value(STATUS_BACKEND_UNAVAILABLE, "Tendwire daemon backend is unavailable"), ) @@ -1377,24 +983,6 @@ def _command_exit_code(envelope: CommandEnvelope) -> int: return 0 if envelope.ok else 1 -def _requires_daemon_for_mutating_command(config: Config, payload: str) -> Any | None: - """Return a live mutating request that must not fall back from the daemon.""" - if config.socket_path is None and config.herdr_backend != "socket": - return None - request, parse_error = parse_command_request(payload) - if parse_error is not None or request is None: - return None - validation_error = validate_request(request) - if validation_error is not None: - return None - if ( - request.action in {"send_instruction", "answer_pending", "answer_decision"} - and not request.dry_run - ): - return request - return None - - def _daemon_backend_failure_envelope( request: Any, attempt: _DaemonAttempt, @@ -1427,15 +1015,6 @@ def _strict_daemon_command_envelope( return envelope -def _replay_daemon_command_receipt( - config: Config, - payload: str, -) -> CommandEnvelope | None: - from .command_submission import replay_command_receipt - - return replay_command_receipt(config, payload) - - def cmd_command( config: Config, *, @@ -1459,10 +1038,9 @@ def cmd_command( and validation_error is None and parsed_request is not None and parsed_request.action - in {"send_instruction", "answer_pending", "answer_decision"} + in {"send_instruction", "answer_decision"} and parsed_request.dry_run ) - daemon_required_request = _requires_daemon_for_mutating_command(config, payload) daemon_eligible = ( isinstance(request_payload, dict) and parse_error is None @@ -1484,18 +1062,14 @@ def cmd_command( error_kind="protocol", request_started=True, ) - if daemon_required_request is not None: + if parsed_request is not None and not parsed_request.dry_run: if daemon_attempt.request_started is False: envelope = _daemon_backend_failure_envelope( - daemon_required_request, + parsed_request, daemon_attempt, ) print(envelope.to_json(indent=2)) return _command_exit_code(envelope) - envelope = _replay_daemon_command_receipt(config, payload) - if envelope is not None: - print(envelope.to_json(indent=2)) - return _command_exit_code(envelope) print( "error: Tendwire daemon command result is unresolved", file=sys.stderr, @@ -1609,38 +1183,12 @@ def cmd_connector(config: Config, args: argparse.Namespace) -> int: } print(_connector_payload_json(payload, indent=2)) return 1 - if config.db_path is None: - payload = { - "schema_version": 1, - "ok": False, - "status": "store_unavailable", - "host_id": config.host_id, - "name": params.get("name", ""), - "error": { - "code": "store_unavailable", - "message": "command requires --db-path or a reachable daemon", - }, - } - print(public_json_dumps(payload, indent=2)) - return 1 - from .connectors import ConnectorOutboxAPI - from .store.sqlite import init_store - - init_store( - config.db_path, - connector_ack_ttl_seconds=config.connector_ack_ttl_seconds, + payload = _daemon_read_payload( + daemon_attempt, + extra={"host_id": config.host_id, "name": params.get("name", "")}, ) - payload = ConnectorOutboxAPI( - config.db_path, - config.host_id, - default_lease_seconds=config.connector_claim_ttl_seconds, - max_lease_seconds=config.connector_max_claim_ttl_seconds, - ack_ttl_seconds=config.connector_ack_ttl_seconds, - max_attempts=config.max_outbox_attempts, - turn_model=config.turn_model, - ).dispatch(method, params) print(_connector_payload_json(payload, indent=2)) - return 0 if payload.get("ok") is not False else 1 + return 1 def cmd_store(config: Config, args: argparse.Namespace) -> int: @@ -1787,7 +1335,7 @@ def main(argv: list[str] | None = None) -> int: socket_group=getattr(args, "socket_group", None), herdr_timeout_seconds=args.herdr_timeout_seconds, ) - if args.command not in {"daemon", "doctor"} and not ( + if args.command in {"daemon", "store"} and not ( args.command == "store" and args.store_action == "compact" ): repair_config_state( @@ -1804,14 +1352,12 @@ def main(argv: list[str] | None = None) -> int: return cmd_snapshot( config, json_output=args.json_output, - store_snapshot=args.store_snapshot, ) if args.command == "attention": return cmd_attention( config, json_output=args.json_output, - store_snapshot=args.store_snapshot, ) if args.command == "turns": diff --git a/src/tendwire/command_submission.py b/src/tendwire/command_submission.py index ebdf472..870b20f 100644 --- a/src/tendwire/command_submission.py +++ b/src/tendwire/command_submission.py @@ -6,7 +6,7 @@ from collections.abc import Callable, Mapping from dataclasses import dataclass from enum import Enum -from typing import Any +from typing import Any, Protocol from .config import Config from .core.actions import CommandContext, execute_command @@ -20,11 +20,8 @@ DISPOSITION_TERMINAL_UNCERTAIN, STATUS_ACCEPTED, STATUS_ANSWER_IN_PROGRESS, - STATUS_AMBIGUOUS_BACKEND_TARGET, STATUS_AMBIGUOUS_TARGET, - STATUS_BACKEND_FAILED, STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, STATUS_DRY_RUN, STATUS_DECISION_NOT_PENDING, STATUS_DUPLICATE_REQUEST, @@ -50,14 +47,12 @@ validate_request, worker_candidate, ) -from .core.models import BackendHealth, Snapshot, Worker, WorkerBinding +from .core.models import BackendHealth, Snapshot, Worker from .core.projector import project_from_observations -from .backends.herdr_decision import calibrate_decision_steps from .store.sqlite import ( abandon_backend_pending_choice_claim, abandon_command_request_reservation, backend_pending_choice_terminal_effect, - claim_backend_pending_choice, claim_backend_pending_decision, command_reservation_is_live, envelope_to_receipt_json, @@ -67,70 +62,79 @@ get_command_request, linked_turn_for_submission, latest_snapshot, - list_worker_bindings, mark_command_send_started, - record_command_send_queued, recover_unresolved_command_send, reserve_command_request, reserve_terminal_command_replay, settle_submission_link_for_request, - start_backend_pending_choice_send, start_backend_pending_decision_send, ) HERDR_BACKEND = "herdr" _MUTATING_ACTIONS = frozenset( - {"send_instruction", "answer_pending", "answer_decision"} + {"send_instruction", "answer_decision"} ) -_LEGACY_V0_REPLAY_WORKER_ID = "legacy-v0-replay-only" -_PENDING_CHANGED_MESSAGE = "pending interaction changed or is no longer answerable" _DISALLOWED_SEND_STATUSES = frozenset({"closed", "failed", "unknown"}) _AMBIGUOUS_BINDING_REASONS = frozenset({"duplicate_backend_target", "not_unique"}) -_PRIVATE_PANE_CLEAR_KEY_SEQUENCES = ( - ("ctrl+u",), - ("ctrl+a", "ctrl+k"), - ("ctrl+a", "backspace"), -) -_PANE_SUBMIT_TARGET_KINDS = frozenset( - { - "agent_id", - "agent", - "name", - "label", - "terminal_id", - "pane_id", - } -) -SocketClientFactory = Callable[[Config], Any] +class AcpPromptRoute(Protocol): + """One live, authority-checked ACP prompt route owned by the daemon. -@dataclass(frozen=True) -class ResolvedCommandTarget: - worker: Worker - binding: WorkerBinding + The route deliberately exposes neither adapter argv nor session identity. + Its binding fingerprint is private durable evidence used only by the + command receipt state machine. + """ + binding_fingerprint: str -def _raw_payload_from_mapping(params: Mapping[str, Any]) -> str: - return json.dumps( - dict(params), - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - ) + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Callable[[], None] | None = None, + ) -> object: ... + + # Routes may expose a context manager that fences their exact generation + # from the final authority check through the prompt-frame acknowledgement. + # Routes without it retain the final authority checks around submission. + def prepare(self) -> Any: ... + + @property + def supports_steering(self) -> bool: ... + def steer( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Callable[[], None] | None = None, + ) -> object: ... +AcpPromptRouter = Callable[[Worker], AcpPromptRoute | None] -def _default_socket_client_factory(config: Config) -> Any: - from .backends.herdr_socket import HerdrSocketClient - return HerdrSocketClient(timeout=config.herdr_timeout_seconds) +class AcpPermissionDecisionRouter(Protocol): + """Private daemon-owned bridge for one exact ACP permission decision.""" + def owns_permission_decision(self, decision: Any) -> bool: ... + def answer_permission_decision(self, decision: Any, *, timeout: float) -> None: ... +def _raw_payload_from_mapping(params: Mapping[str, Any]) -> str: + return json.dumps( + dict(params), + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ) + def _backend_health(snapshot: Snapshot) -> BackendHealth: for health in snapshot.backend_health: if health.name == HERDR_BACKEND: @@ -166,11 +170,6 @@ def _backend_unavailable( def _backend_health_error(config: Config, request: CommandRequest, snapshot: Snapshot) -> CommandEnvelope | None: - if config.herdr_backend != "socket": - return _backend_unavailable( - request, - "Herdr socket backend is not enabled", - ) health = _backend_health(snapshot) if health.status != "healthy": return _backend_unavailable( @@ -225,67 +224,6 @@ def _target_resolution_error( ) -def _binding_error(request: CommandRequest, status: str, message: str) -> CommandEnvelope: - return CommandEnvelope.from_result( - request, - ok=False, - status=status, - error=error_value(status, message), - ) - - -def _binding_for_worker( - request: CommandRequest, - worker: Worker, - bindings: list[WorkerBinding], -) -> ResolvedCommandTarget | CommandEnvelope: - worker_bindings = [ - binding - for binding in bindings - if binding.backend == HERDR_BACKEND and binding.worker_id == worker.id - ] - if not worker_bindings: - return _binding_error( - request, - STATUS_BACKEND_UNSUPPORTED, - "target has no backend-owned sendable private binding", - ) - - exact = [binding for binding in worker_bindings if binding.worker_fingerprint == worker.fingerprint] - if not exact: - return _binding_error( - request, - STATUS_STALE_TARGET, - "target private binding is stale for the current worker", - ) - if len(exact) != 1: - return _binding_error( - request, - STATUS_AMBIGUOUS_BACKEND_TARGET, - "target resolves to an ambiguous backend send target", - ) - - binding = exact[0] - if ( - not binding.sendable - or not binding.target_value - or binding.target_kind not in _PANE_SUBMIT_TARGET_KINDS - ): - if (binding.reason or "") in _AMBIGUOUS_BINDING_REASONS: - return _binding_error( - request, - STATUS_AMBIGUOUS_BACKEND_TARGET, - "target resolves to an ambiguous backend send target", - ) - return _binding_error( - request, - STATUS_BACKEND_UNSUPPORTED, - "target has no backend-owned sendable private binding", - ) - - return ResolvedCommandTarget(worker=worker, binding=binding) - - def _resolve_authoritative_worker( request: CommandRequest, snapshot: Snapshot, @@ -321,125 +259,6 @@ def _worker_status_error( ) -def _socket_request(client: Any, method: str, params: Mapping[str, Any], *, timeout: float) -> Any: - if not hasattr(client, "request"): - raise TypeError("socket client does not expose generic request") - return client.request(method, params, timeout=timeout) - - -def _pane_id_from_agent_info(value: Any) -> str: - if not isinstance(value, Mapping): - return "" - result = value.get("result") - agent = result.get("agent") if isinstance(result, Mapping) else None - if not isinstance(agent, Mapping): - agent = value.get("agent") - if not isinstance(agent, Mapping): - return "" - pane_id = agent.get("pane_id") or agent.get("paneId") - return str(pane_id or "").strip() - - -def _pane_id_from_terminal_listing(value: Any, terminal_id: str) -> str: - if not isinstance(value, Mapping): - raise ValueError("invalid agent.list response") - agents = value.get("agents") - if not isinstance(agents, list): - raise ValueError("invalid agent.list agents") - matches: set[str] = set() - for agent in agents: - if not isinstance(agent, Mapping): - raise ValueError("invalid agent.list entry") - listed_terminal_id = agent.get("terminal_id") - listed_pane_id = agent.get("pane_id") - if ( - not isinstance(listed_terminal_id, str) - or not listed_terminal_id - or not isinstance(listed_pane_id, str) - or not listed_pane_id.strip() - ): - raise ValueError("invalid agent.list entry") - if listed_terminal_id == terminal_id: - matches.add(listed_pane_id.strip()) - if len(matches) > 1: - raise ValueError("ambiguous agent.list terminal_id match") - return next(iter(matches), "") - - -def _private_pane_id_for_binding(client: Any, binding: WorkerBinding, *, timeout: float) -> str: - if binding.target_kind == "pane_id": - return str(binding.target_value or "").strip() - try: - response = _socket_request( - client, - "agent.get", - {"target": binding.target_value}, - timeout=timeout, - ) - except Exception as exc: # noqa: BLE001 - from .backends.herdr_protocol import HerdrErrorResponse - - # Herdr 0.7.5 stopped resolving terminal-id targets through agent.get - # while agent.list still publishes the terminal_id -> pane_id mapping, - # so a definite target-lookup error falls back to the listing before - # the caller terminalizes the request. - if not isinstance(exc, HerdrErrorResponse) or binding.target_kind != "terminal_id": - raise - listing = _socket_request(client, "agent.list", {}, timeout=timeout) - pane_id = _pane_id_from_terminal_listing( - listing, - str(binding.target_value or ""), - ) - if pane_id: - return pane_id - raise - return _pane_id_from_agent_info(response) - - -def _submit_private_pane_input(client: Any, pane_id: str, instruction_text: str, *, timeout: float) -> None: - # A single ctrl+u is not reliable across all foreground TUIs. Clear stale - # input first, then submit text and Enter in one Herdr operation so the - # foreground application cannot observe a staged prompt between requests. - try: - for keys in _PRIVATE_PANE_CLEAR_KEY_SEQUENCES: - _socket_request( - client, - "pane.send_keys", - {"pane_id": pane_id, "keys": list(keys)}, - timeout=timeout, - ) - except Exception as exc: - raise _PaneInputNotStartedError from exc - _socket_request( - client, - "pane.send_input", - {"pane_id": pane_id, "text": instruction_text, "keys": ["Enter"]}, - timeout=timeout, - ) - - -class _PaneInputNotStartedError(RuntimeError): - """The instruction input operation was never attempted.""" - - -def _agent_prompt_delivery(value: Any) -> str: - if not isinstance(value, Mapping): - return "" - result = value.get("result") - candidate = result if isinstance(result, Mapping) else value - delivery = candidate.get("delivery") - return delivery if isinstance(delivery, str) else "" - - -def _herdr_error_code(exc: BaseException) -> str: - from .backends.herdr_protocol import HerdrErrorResponse - - if not isinstance(exc, HerdrErrorResponse) or not isinstance(exc.error, Mapping): - return "" - code = exc.error.get("code") - return code if isinstance(code, str) else "" - - def _target_state_at_send(worker: Worker) -> str: status = str(worker.status or "").strip().lower().replace("-", "_") return status or "unknown" @@ -453,15 +272,6 @@ def _instruction_text(request: CommandRequest) -> str: -def _backend_failure(request: CommandRequest, message: str) -> CommandEnvelope: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_BACKEND_FAILED, - error=error_value(STATUS_BACKEND_FAILED, message), - ) - - def _backend_uncertain(request: CommandRequest, message: str) -> CommandEnvelope: return CommandEnvelope.from_result( request, @@ -516,73 +326,6 @@ def _duplicate_request(request: CommandRequest) -> CommandEnvelope: ) -def _pending_changed_envelope(request: CommandRequest) -> CommandEnvelope: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_STALE_TARGET, - error=error_value(STATUS_STALE_TARGET, _PENDING_CHANGED_MESSAGE), - ) - - -def _pending_public_result( - request: CommandRequest, - claim: Any, - *, - delivery_state: str, -) -> dict[str, Any]: - params = request.params or {} - result: dict[str, Any] = { - "target": {"worker_id": claim.worker_id}, - "pending": { - "id": params.get("pending_id"), - "fingerprint": params.get("pending_fingerprint"), - }, - "choice": {"choice_id": params.get("choice_id")}, - "delivery_state": delivery_state, - } - if delivery_state == "submitted": - result.update( - { - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - } - ) - return result - - -def _pending_claim_has_exact_route(claim: Any) -> bool: - return ( - isinstance(getattr(claim, "worker_id", None), str) - and bool(claim.worker_id) - and isinstance(getattr(claim, "worker_fingerprint", None), str) - and bool(claim.worker_fingerprint) - and isinstance(getattr(claim, "binding_private_fingerprint", None), str) - and bool(claim.binding_private_fingerprint) - and isinstance(getattr(claim, "turn_target_value", None), str) - and bool(claim.turn_target_value.strip()) - and not isinstance(getattr(claim, "picker_ordinal", None), bool) - and isinstance(claim.picker_ordinal, int) - and claim.picker_ordinal >= 1 - ) - - -def _same_pending_route(left: Any, right: Any) -> bool: - return _pending_claim_has_exact_route(left) and _pending_claim_has_exact_route(right) and ( - left.worker_id, - left.worker_fingerprint, - left.binding_private_fingerprint, - left.turn_target_value, - left.picker_ordinal, - ) == ( - right.worker_id, - right.worker_fingerprint, - right.binding_private_fingerprint, - right.turn_target_value, - right.picker_ordinal, - ) - - def _decision_failure_envelope( request: CommandRequest, status: str, @@ -700,15 +443,6 @@ def _safe_transient_pre_send(envelope: CommandEnvelope) -> PreSendFailure: return PreSendFailure(envelope=envelope, certainty=PreSendCertainty.SAFE_TRANSIENT) -def _close_socket_client(client: Any | None) -> None: - if client is None or not hasattr(client, "close"): - return - try: - client.close() - except Exception: - pass - - def _abandon_pending_claim(config: Config, claim_token: str | None) -> bool: if config.db_path is None or not claim_token: return False @@ -741,100 +475,6 @@ def _abandon_request_reservation( return False -def _connect_socket( - config: Config, - request: CommandRequest, - socket_client_factory: SocketClientFactory | None, -) -> Any | CommandEnvelope: - factory = socket_client_factory or _default_socket_client_factory - client: Any | None = None - try: - client = factory(config) - if not hasattr(client, "request"): - raise TypeError("socket client does not expose generic request") - if hasattr(client, "connect"): - client.connect() - return client - except Exception: # noqa: BLE001 - _close_socket_client(client) - return _backend_unavailable(request, "Herdr socket could not be reached") - - -def _resolve_private_pane( - config: Config, - request: CommandRequest, - client: Any, - binding: WorkerBinding, -) -> str | PreSendFailure: - try: - pane_id = _private_pane_id_for_binding( - client, - binding, - timeout=config.herdr_timeout_seconds, - ) - except Exception as exc: # noqa: BLE001 - from .backends.herdr_protocol import HerdrErrorResponse, HerdrProtocolError - from .backends.herdr_socket import ( - HerdrSocketConnectionError, - HerdrSocketDisconnectedError, - HerdrSocketTimeoutError, - ) - - # A definite error response from Herdr is an authoritative answer that - # this target cannot be resolved; a same-ID retry would get the same - # answer, so it terminalizes rather than looping to the retry horizon. - # It is checked before the transport branch because HerdrErrorResponse - # subclasses HerdrProtocolError and must not be mistaken for framing loss. - if isinstance(exc, HerdrErrorResponse): - return _permanent_pre_send( - _backend_failure( - request, - "Herdr socket could not resolve the private send target", - ) - ) - # A transport read that could not complete -- timeout, disconnect, - # connection loss, protocol framing, or an OS-level socket error -- - # proves nothing about the target and never began a send, so it stays - # retryable. - if isinstance( - exc, - HerdrSocketConnectionError - | HerdrSocketTimeoutError - | HerdrSocketDisconnectedError - | HerdrProtocolError, - ) or isinstance(exc, OSError): - return _safe_transient_pre_send( - _backend_unavailable( - request, - "Herdr socket could not resolve the private send target", - ) - ) - # A malformed or unsupported resolution response is a proven target - # property, not a transient operation failure. - if isinstance(exc, (TypeError, ValueError)): - return _permanent_pre_send( - _backend_failure( - request, - "Herdr socket private send target is unsupported", - ) - ) - # An unclassifiable resolution error is not proven safe to retry, so it - # retains the prior terminal behavior rather than looping indefinitely. - return _permanent_pre_send( - _backend_failure( - request, - "Herdr socket private send target resolution failed", - ) - ) - if not pane_id: - # Herdr answered, and the answer is that the target has no resolvable - # pane. That is authoritative target unsuitability, not a read failure. - return _permanent_pre_send( - _backend_failure(request, "Herdr socket private send target has no pane") - ) - return pane_id - - def _transition_payload( request: CommandRequest, *, @@ -866,11 +506,6 @@ def _receipt_is_canonical( ) if not common_identity: return False - if version == 0: - return ( - receipt.get("legacy_collision") is False - and receipt.get("canonical_fingerprint") == request.payload_fingerprint() - ) return ( version == canonical.canonical_version and receipt.get("canonical_fingerprint") == canonical.fingerprint @@ -911,33 +546,9 @@ def _stored_terminal_envelope( try: if type(schema_version) is not int: raise ValueError("stored envelope schema_version must be an exact integer") - if schema_version == COMMAND_ENVELOPE_SCHEMA_VERSION: - envelope = CommandEnvelope.from_dict(data) - elif schema_version == 1: - legacy_fields = { - "schema_version", - "action", - "request_id", - "ok", - "dry_run", - "status", - "result", - "error", - "warnings", - } - if set(data) != legacy_fields: - raise ValueError("legacy envelope has an invalid field set") - upgraded = dict(data) - upgraded["schema_version"] = COMMAND_ENVELOPE_SCHEMA_VERSION - upgraded["disposition"] = expected_disposition - envelope = CommandEnvelope.from_dict(upgraded) - roundtrip = envelope.to_dict() - roundtrip.pop("disposition") - roundtrip["schema_version"] = 1 - if roundtrip != data: - raise ValueError("legacy envelope is not an exact public roundtrip") - else: + if schema_version != COMMAND_ENVELOPE_SCHEMA_VERSION: raise ValueError("unsupported stored envelope schema") + envelope = CommandEnvelope.from_dict(data) except (TypeError, ValueError): return _backend_uncertain(request, malformed) @@ -1019,9 +630,8 @@ def _envelope_from_receipt( "state", "status", "result_json", - "legacy_collision", } - if not required.issubset(receipt) or receipt.get("legacy_collision") is not False: + if not required.issubset(receipt): return _backend_uncertain( request, "stored request receipt is malformed; not retrying mutation", @@ -1065,76 +675,6 @@ class ReservedCommandMutation: canonical: CanonicalMutation owner_token: str -@dataclass(frozen=True) -class PreparedInstructionMutation: - client: Any - pane_id: str - target_value: str - binding_fingerprint: str - - -def _prepare_instruction( - config: Config, - request: CommandRequest, - worker: Worker, - *, - socket_client_factory: SocketClientFactory | None, -) -> PreparedInstructionMutation | PreSendFailure: - assert config.db_path is not None - try: - bindings = list_worker_bindings( - config.db_path, - config.host_id, - backend=HERDR_BACKEND, - ) - except Exception: - # The binding store raised. This is an operation failure, not a proven - # target property, and no send began: stay retryable under the same ID. - return _safe_transient_pre_send( - _backend_unavailable(request, "private binding store is unavailable") - ) - resolved = _binding_for_worker(request, worker, bindings) - if isinstance(resolved, CommandEnvelope): - # A missing, stale, or ambiguous binding read from current data is a - # proven target property; a same-ID retry would resolve it the same way. - return _permanent_pre_send(resolved) - - binding_fingerprint = str(resolved.binding.private_fingerprint or "").strip() - if not binding_fingerprint: - return _permanent_pre_send( - _binding_error( - request, - STATUS_BACKEND_UNSUPPORTED, - "target private binding has no durable identity", - ) - ) - - # A socket that cannot be reached is an operation failure before any send, - # so it stays retryable rather than burning the request ID. - client_or_error = _connect_socket(config, request, socket_client_factory) - if isinstance(client_or_error, CommandEnvelope): - return _safe_transient_pre_send(client_or_error) - client = client_or_error - # Pane resolution classifies its own outcome: a transport read failure is a - # safe transient, while a definite backend answer (rejection, unsupported, - # or no pane) is a proven-permanent target failure. - pane_or_error = _resolve_private_pane( - config, - request, - client, - resolved.binding, - ) - if isinstance(pane_or_error, PreSendFailure): - _close_socket_client(client) - return pane_or_error - return PreparedInstructionMutation( - client=client, - pane_id=pane_or_error, - target_value=str(resolved.binding.target_value), - binding_fingerprint=binding_fingerprint, - ) - - def _reserve_canonical_request( config: Config, request: CommandRequest, @@ -1155,7 +695,6 @@ def _reserve_canonical_request( public_worker_id=canonical.public_worker_id, pending_result_json=envelope_to_receipt_json(pending), selector_proof=_selector_proof(request), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), ) except Exception: # noqa: BLE001 try: @@ -1352,7 +891,6 @@ def _reserve_terminal_replay( # proven equivalence with a different one, and overwriting it would # strand a later retry of the request as it was actually issued. selector_proof=_stored_selector_proof(previous_receipt), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), event_payload=_transition_payload( request, worker_id=canonical.public_worker_id, @@ -1471,31 +1009,6 @@ def _accepted_send_envelope( ) -def _queued_send_envelope( - request: CommandRequest, - worker: Worker, -) -> CommandEnvelope: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_PENDING, - disposition=DISPOSITION_IN_PROGRESS, - result={ - "target": {"worker_id": worker.id}, - "delivery_state": "queued", - "transport_state": "queued", - "target_state_at_send": _target_state_at_send(worker), - "turn_id": None, - "observed_turn_state": "pending_observation", - "submission_verdict": "written_to_pty", - }, - error=error_value( - STATUS_PENDING, - "instruction is queued for the next agent turn boundary", - ), - ) - - def _instruction_rejected_envelope( request: CommandRequest, worker: Worker, @@ -1626,401 +1139,54 @@ def _unverified_queued_send_envelope( ) -def _record_queued_send( +def _uncertain_pending_effect( config: Config, - request: CommandRequest, - worker: Worker, - reservation: ReservedCommandMutation, - envelope: CommandEnvelope, -) -> CommandEnvelope: - assert config.db_path is not None + claim_token: str, +) -> Callable[[Any], Any] | None: try: - queued = record_command_send_queued( - config.db_path, + return backend_pending_choice_terminal_effect( host_id=config.host_id, - request_id=request.request_id or "", - canonical_fingerprint=reservation.canonical.fingerprint, - owner_token=reservation.owner_token, - result_json=envelope_to_receipt_json(envelope), - event_payload=_transition_payload( - request, - worker_id=worker.id, - envelope=envelope, - ), + claim_token=claim_token, + accepted=False, ) - except Exception: # noqa: BLE001 - return _recover_request(config, request, reservation.canonical) - if not isinstance(queued, Mapping): - return _recover_request(config, request, reservation.canonical) - return _envelope_from_receipt( - request, - reservation.canonical, - queued.get("receipt"), - ) + except Exception: + return None -def _submit_instruction( +def _validate_pending_decision( config: Config, request: CommandRequest, - worker: Worker, - reservation: ReservedCommandMutation, - prepared: PreparedInstructionMutation, -) -> CommandEnvelope: - assert config.db_path is not None +) -> Any | PreSendFailure: + if config.db_path is None: + return _safe_transient_pre_send( + _backend_unavailable(request, "pending state store is unavailable") + ) + params = request.params or {} + target = request.target or {} try: - send_started = _mark_request_send_started( - config, - request, - reservation, - binding_fingerprint=prepared.binding_fingerprint, - worker=worker, - instruction_text=_instruction_text(request), - ) - if isinstance(send_started, CommandEnvelope): - return send_started - if not isinstance(send_started, Mapping): - return _recover_request( - config, - request, - reservation.canonical, - ) - observed_turn = send_started - - try: - response = _socket_request( - prepared.client, - "agent.prompt", - { - "target": prepared.target_value, - "text": _instruction_text(request), - "wait": { - "until": ["working"], - "timeout_ms": max( - 1, - int(config.herdr_timeout_seconds * 1000), - ), - }, - }, - timeout=config.herdr_timeout_seconds + 1.0, - ) - except Exception as exc: # noqa: BLE001 - verdict = _herdr_error_code(exc) - if verdict in { - "agent_not_ready", - "agent_target_ambiguous", - "agent_prompt_not_received", - "agent_prompt_unsubmitted", - "agent_input_pending", - }: - envelope = _instruction_rejected_envelope( - request, - worker, - verdict=verdict, - ) - return _finish_request( - config, - request, - reservation, - envelope, - expected_state="send_started", - terminal_state="rejected", - ) - if verdict == "agent_prompt_stalled": - envelope = _instruction_uncertain_envelope( - request, - worker, - verdict=verdict, - ) - else: - envelope = _instruction_uncertain_envelope( - request, - worker, - verdict="unknown", - ) - return _finish_request( - config, - request, - reservation, - envelope, - expected_state="send_started", - terminal_state="uncertain", - ) - - verdict = _agent_prompt_delivery(response) - if verdict == "written_to_pty": - return _record_queued_send( - config, - request, - worker, - reservation, - _queued_send_envelope(request, worker), - ) - if verdict != "submitted": - return _finish_request( - config, - request, - reservation, - _instruction_uncertain_envelope( - request, - worker, - verdict="unknown", - ), - expected_state="send_started", - terminal_state="uncertain", - ) - - # The observation may arrive while the pane call is in flight. Re-read - # the durable link so the accepted envelope can report it immediately. - try: - refreshed_turn = linked_turn_for_submission( - config.db_path, - host_id=config.host_id, - request_id=request.request_id or "", - ) - except Exception: # noqa: BLE001 - refreshed_turn = None - if isinstance(refreshed_turn, Mapping): - observed_turn = refreshed_turn - finally: - _close_socket_client(prepared.client) - - accepted = _accepted_send_envelope( - request, - worker, - observed_turn, - submission_verdict="submitted", - ) - return _finish_request( - config, - request, - reservation, - accepted, - expected_state="send_started", - terminal_state="accepted", - ) - - -def _validate_pending_choice( - config: Config, - request: CommandRequest, -) -> Any | PreSendFailure: - if config.db_path is None: - return _safe_transient_pre_send( - _backend_unavailable(request, "pending state store is unavailable") - ) - params = request.params or {} - try: - validated = claim_backend_pending_choice( - config.db_path, - config.host_id, - str(params.get("pending_id") or ""), - str(params.get("pending_fingerprint") or ""), - str(params.get("choice_id") or ""), - claim=False, + validated = claim_backend_pending_decision( + config.db_path, + config.host_id, + str(target.get("worker_id") or ""), + str(params.get("decision_ref") or ""), + params.get("selection") + if isinstance(params.get("selection"), Mapping) + else {}, + claim=False, ) except Exception: - # The pending store raised; nothing was claimed or sent. return _safe_transient_pre_send( _backend_unavailable(request, "pending state store is unavailable") ) - if validated.status != "validated" or not _pending_claim_has_exact_route(validated): - # The pending interaction provably changed or is no longer answerable. - return _permanent_pre_send(_pending_changed_envelope(request)) - return validated - - -def _claim_pending_choice( - config: Config, - request: CommandRequest, - validated: Any, -) -> Any | CommandEnvelope: - assert config.db_path is not None - params = request.params or {} - try: - claim = claim_backend_pending_choice( - config.db_path, - config.host_id, - str(params.get("pending_id") or ""), - str(params.get("pending_fingerprint") or ""), - str(params.get("choice_id") or ""), - claim=True, - ) - except Exception: - return _backend_uncertain(request, "pending answer claim state is uncertain") - if ( - claim.status != "claimed" - or not isinstance(getattr(claim, "claim_token", None), str) - or not claim.claim_token - or not _same_pending_route(validated, claim) - ): - return _pending_changed_envelope(request) - return claim - - -def _uncertain_pending_effect( - config: Config, - claim_token: str, -) -> Callable[[Any], Any] | None: - try: - return backend_pending_choice_terminal_effect( - host_id=config.host_id, - claim_token=claim_token, - accepted=False, - ) - except Exception: - return None - - -def _answer_pending( - config: Config, - request: CommandRequest, - validated: Any, - reservation: ReservedCommandMutation, - client: Any, -) -> CommandEnvelope: - assert config.db_path is not None - claim = _claim_pending_choice(config, request, validated) - if isinstance(claim, CommandEnvelope): - _close_socket_client(client) - return _finish_before_send(config, request, reservation, claim) - claim_token = claim.claim_token - - send_start_error = _mark_request_send_started( - config, - request, - reservation, - binding_fingerprint=claim.binding_private_fingerprint, - ) - if send_start_error is not None: - _close_socket_client(client) - claim_released = _abandon_pending_claim(config, claim_token) - if send_start_error.status == STATUS_PENDING and not claim_released: - return _finish_before_send( - config, + if validated.status == "validated" and _decision_claim_has_exact_route(validated): + return validated + if validated.status == "acp_authority_unavailable": + return _safe_transient_pre_send( + _backend_unavailable( request, - reservation, - _backend_uncertain( - request, - "pending answer claim could not be safely released", - ), + "ACP permission authority is temporarily unavailable", ) - return send_start_error - - try: - started = start_backend_pending_choice_send( - config.db_path, - config.host_id, - claim_token, - ) - except Exception: - _close_socket_client(client) - _abandon_pending_claim(config, claim_token) - return _finish_request( - config, - request, - reservation, - _backend_uncertain(request, "pending answer start state is uncertain"), - expected_state="send_started", - terminal_state="uncertain", - terminal_effect=_uncertain_pending_effect(config, claim_token), - ) - if getattr(started, "status", None) != "started" or not _same_pending_route(claim, started): - _close_socket_client(client) - _abandon_pending_claim(config, claim_token) - return _finish_request( - config, - request, - reservation, - _backend_uncertain( - request, - "pending answer state is uncertain after send start", - ), - expected_state="send_started", - terminal_state="uncertain", - terminal_effect=_uncertain_pending_effect(config, claim_token), - ) - - try: - _submit_private_pane_input( - client, - started.turn_target_value.strip(), - str(started.picker_ordinal), - timeout=config.herdr_timeout_seconds, - ) - except Exception: # noqa: BLE001 - uncertain = _backend_uncertain( - request, - "Herdr socket pane input state is uncertain after send start", - ) - return _finish_request( - config, - request, - reservation, - uncertain, - expected_state="send_started", - terminal_state="uncertain", - terminal_effect=_uncertain_pending_effect(config, claim_token), - ) - finally: - _close_socket_client(client) - - accepted = CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result=_pending_public_result(request, started, delivery_state="submitted"), - ) - try: - effect = backend_pending_choice_terminal_effect( - host_id=config.host_id, - claim_token=claim_token, - accepted=True, - ) - except Exception: - return _recover_request( - config, - request, - reservation.canonical, - ) - return _finish_request( - config, - request, - reservation, - accepted, - expected_state="send_started", - terminal_state="accepted", - terminal_effect=effect, - ) - - -def _validate_pending_decision( - config: Config, - request: CommandRequest, -) -> Any | PreSendFailure: - if config.db_path is None: - return _safe_transient_pre_send( - _backend_unavailable(request, "pending state store is unavailable") - ) - params = request.params or {} - target = request.target or {} - try: - validated = claim_backend_pending_decision( - config.db_path, - config.host_id, - str(target.get("worker_id") or ""), - str(params.get("decision_ref") or ""), - params.get("selection") - if isinstance(params.get("selection"), Mapping) - else {}, - claim=False, ) - except Exception: - return _safe_transient_pre_send( - _backend_unavailable(request, "pending state store is unavailable") - ) - if validated.status == "validated" and _decision_claim_has_exact_route(validated): - return validated status = { "already_claimed": STATUS_ANSWER_IN_PROGRESS, "unknown_worker": STATUS_UNKNOWN_WORKER, @@ -2058,6 +1224,11 @@ def _claim_pending_decision( ): if _same_decision_route(validated, claim): return claim + if claim.status == "acp_authority_unavailable": + return _backend_unavailable( + request, + "ACP permission authority is temporarily unavailable", + ) status = { "already_claimed": STATUS_ANSWER_IN_PROGRESS, "unknown_worker": STATUS_UNKNOWN_WORKER, @@ -2067,43 +1238,6 @@ def _claim_pending_decision( return _decision_failure_envelope(request, status) -def _submit_decision_calibration( - client: Any, - pane_id: str, - decision: Any, - *, - timeout: float, -) -> None: - steps = calibrate_decision_steps( - kind=decision.decision_kind, - option_count=decision.option_count, - option_refs=decision.option_refs, - text=decision.text, - ) - for step in steps: - if step.operation == "keys": - _socket_request( - client, - "pane.send_keys", - {"pane_id": pane_id, "keys": list(step.keys)}, - timeout=timeout, - ) - elif step.operation == "text": - _socket_request( - client, - "pane.send_text", - {"pane_id": pane_id, "text": step.text}, - timeout=timeout, - ) - else: - _socket_request( - client, - "pane.send_input", - {"pane_id": pane_id, "text": step.text, "keys": list(step.keys)}, - timeout=timeout, - ) - - def _decision_public_result( request: CommandRequest, claim: Any, @@ -2122,15 +1256,27 @@ def _answer_decision( request: CommandRequest, validated: Any, reservation: ReservedCommandMutation, - client: Any, + *, + acp_permission_router: AcpPermissionDecisionRouter | None = None, ) -> CommandEnvelope: assert config.db_path is not None claim = _claim_pending_decision(config, request, validated) if isinstance(claim, CommandEnvelope): - _close_socket_client(client) if claim.status == STATUS_ANSWER_IN_PROGRESS: _abandon_request_reservation(config, request, reservation) return _answer_in_progress(request, receipt_reserved=True) + if claim.status == STATUS_BACKEND_UNAVAILABLE: + if _abandon_request_reservation(config, request, reservation): + return claim + return _finish_before_send( + config, + request, + reservation, + _backend_uncertain( + request, + "ACP permission reservation state is uncertain", + ), + ) return _finish_before_send(config, request, reservation, claim) claim_token = claim.claim_token @@ -2141,7 +1287,6 @@ def _answer_decision( binding_fingerprint=claim.binding_private_fingerprint, ) if send_start_error is not None: - _close_socket_client(client) claim_released = _abandon_pending_claim(config, claim_token) if send_start_error.status == STATUS_PENDING and not claim_released: return _finish_before_send( @@ -2162,7 +1307,6 @@ def _answer_decision( claim_token, ) except Exception: - _close_socket_client(client) _abandon_pending_claim(config, claim_token) return _finish_request( config, @@ -2174,7 +1318,6 @@ def _answer_decision( terminal_effect=_uncertain_pending_effect(config, claim_token), ) if getattr(started, "status", None) != "started" or not _same_decision_route(claim, started): - _close_socket_client(client) _abandon_pending_claim(config, claim_token) return _finish_request( config, @@ -2190,11 +1333,11 @@ def _answer_decision( ) try: - _submit_decision_calibration( - client, - started.turn_target_value.strip(), + if acp_permission_router is None: + raise RuntimeError("ACP permission bridge is unavailable") + acp_permission_router.answer_permission_decision( started, - timeout=config.herdr_timeout_seconds, + timeout=config.acp_request_timeout_seconds, ) except Exception: # noqa: BLE001 return _finish_request( @@ -2203,15 +1346,12 @@ def _answer_decision( reservation, _backend_uncertain( request, - "Herdr decision input state is uncertain after send start", + "permission decision state is uncertain after send start", ), expected_state="send_started", terminal_state="uncertain", terminal_effect=_uncertain_pending_effect(config, claim_token), ) - finally: - _close_socket_client(client) - accepted = CommandEnvelope.from_result( request, ok=True, @@ -2359,11 +1499,6 @@ def _proven_replay_worker_id( stored = receipt.get("public_worker_id") stored_worker_id = stored if isinstance(stored, str) and stored else "" - # A v0 receipt is validated against the exact raw request payload, which - # already pins the original selector spelling. It needs no resolution, and - # a changed payload fails its canonical check rather than replaying here. - if version == 0: - return stored_worker_id or _LEGACY_V0_REPLAY_WORKER_ID if not stored_worker_id: return _receipt_malformed(request) if request.action == "answer_decision": @@ -2390,7 +1525,7 @@ def _proven_replay_worker_id( # 3. Only a current, healthy observation can prove that a different spelling # names the same canonical worker. A degraded one proves nothing, and a - # legacy receipt carries no proof to fall back on. + # receipt without a proof has no fallback evidence. if not allow_current_authority: return None try: @@ -2422,8 +1557,6 @@ def _receipt_authority( stored reservation was abandoned before any send and the normal path may re-drive it. """ - if receipt.get("legacy_collision") is not False: - return _receipt_malformed(request) if receipt.get("action") != request.action: return _duplicate_request(request) @@ -2444,10 +1577,6 @@ def _receipt_authority( if replay.status == STATUS_DUPLICATE_REQUEST: # A changed canonical mutation never rewrites the original receipt. return replay - if receipt.get("canonical_version") == 0: - # Legacy evidence cannot be re-expressed as a canonical v1 row, so - # replay it as read instead of inventing one. - return replay return _reserve_terminal_replay(config, request, canonical, receipt, replay) if replay.status != STATUS_PENDING: return replay @@ -2580,8 +1709,6 @@ def replay_command_receipt( return None if not isinstance(receipt, Mapping): return None - if receipt.get("legacy_collision") is not False: - return _receipt_malformed(request) if receipt.get("action") != request.action: return _duplicate_request(request) proven = _proven_replay_worker_id( @@ -2608,28 +1735,29 @@ def replay_command_receipt( ) -def _submit_command_v2( +def submit_acp_command( config: Config, params: Mapping[str, Any] | str, *, - socket_client_factory: SocketClientFactory | None = None, -) -> CommandEnvelope: - """Submit one command through the authoritative daemon/socket path.""" - payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) - request, parse_error = parse_command_request(payload) - if parse_error is not None: - if request is not None: - return CommandEnvelope.from_error(request, parse_error) - return CommandEnvelope.from_error(None, parse_error) + prompt_router: AcpPromptRouter, +) -> CommandEnvelope | None: + """Submit ``send_instruction`` through the required ACP worker route. - validation_error = validate_request(request) - if validation_error is not None: - return CommandEnvelope.from_error(request, validation_error) + ``None`` is reserved for input that is not an executable instruction, so + the shared parser/dry-run/non-instruction path can handle it. A valid live + instruction always produces an ACP result and never crosses transports. + """ - if request.action not in _MUTATING_ACTIONS: - return _execute_non_mutating(config, request) + payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) + request, parse_error = parse_command_request(payload) + if parse_error is not None or request is None: + return None + if validate_request(request) is not None: + return None if request.dry_run: - return _mutation_dry_run(request) + return None + if request.action != "send_instruction": + return None existing_receipt: Mapping[str, Any] | None = None if config.db_path is not None: @@ -2644,154 +1772,350 @@ def _submit_command_v2( if isinstance(candidate, Mapping): existing_receipt = candidate - # An existing receipt is the authority for its request ID. It decides the - # retry from stored evidence before any mutable worker snapshot is read, so - # a vanished, renamed, or recycled worker can never downgrade a live receipt - # to a no-receipt failure or drive a second backend mutation. Only an - # abandoned reservation returns here, to be re-driven by the normal path. takeover: _ReceiptTakeover | None = None if existing_receipt is not None: decided = _receipt_authority(config, request, existing_receipt) if isinstance(decided, CommandEnvelope): - return decided + return _negotiated_submission_envelope(config, request, decided) takeover = decided try: snapshot = _current_snapshot(config) except Exception: # noqa: BLE001 - # No external mutation has begun. Store/open contention while reading - # current authority is safely retryable when no receipt exists. An - # abandoned reservation remains authoritative and stays in progress. if takeover is not None: return _request_in_progress(request) return _backend_unavailable( request, "Current worker authority is temporarily unavailable", ) + health_error = _backend_health_error(config, request, snapshot) + worker = _resolve_authoritative_worker(request, snapshot) + if isinstance(worker, CommandEnvelope): + if takeover is not None: + return _request_in_progress(request) + if health_error is not None: + return health_error + return worker + if takeover is not None and worker.id != takeover.public_worker_id: + return _duplicate_request(request) - if request.action == "send_instruction": - worker = _resolve_authoritative_worker(request, snapshot) - if isinstance(worker, CommandEnvelope): - if takeover is not None: - # The receipt says this request is reserved and unsent. Mutable - # authority may not restate that as a no-receipt failure. - return _request_in_progress(request) - # An unhealthy observation cannot authoritatively establish that a - # selector is absent, stale, or ambiguous, so keep the request ID - # retryable until a canonical worker can be proven. - if health_error is not None: - return health_error - return worker - if takeover is not None and worker.id != takeover.public_worker_id: - # The abandoned reservation named a different worker, so this is a - # changed target. Fail before any socket or backend work. - return _duplicate_request(request) + route: AcpPromptRoute | None = None + permanent_error = _worker_status_error(request, worker) or health_error + if permanent_error is not None: canonical = build_canonical_mutation(request, public_worker_id=worker.id) - # A disallowed worker status or an unavailable backend is an authoritative - # observation of proven target unsuitability: a durable rejection is - # justified, and a same-ID retry replays it. - permanent_error = _worker_status_error(request, worker) or health_error - prepared: PreparedInstructionMutation | PreSendFailure | None = None - if permanent_error is None: - prepared = _prepare_instruction( - config, - request, - worker, - socket_client_factory=socket_client_factory, - ) - # A safe transient preparation failure never began a send and never - # created durable authority. Keep the request ID retryable without - # reserving, so a command that was never sent is never silently dropped. - if isinstance(prepared, PreSendFailure) and prepared.is_transient: + reservation = _reserve_canonical_request(config, request, canonical) + if isinstance(reservation, CommandEnvelope): + return reservation + return _finish_before_send(config, request, reservation, permanent_error) + + try: + route = prompt_router(worker) + except Exception: # noqa: BLE001 + route = None + if route is None: + if takeover is not None: + return _request_in_progress(request) + return _backend_unavailable(request, "ACP worker route is unavailable") + def submit_through(active_route: AcpPromptRoute) -> CommandEnvelope | None: + try: + binding_fingerprint = str( + getattr(active_route, "binding_fingerprint", "") or "" + ).strip() + except Exception: # noqa: BLE001 + binding_fingerprint = "" + if not binding_fingerprint: if takeover is not None: return _request_in_progress(request) - return prepared.envelope + return _backend_unavailable( + request, "ACP worker route has no durable authority" + ) + + canonical = build_canonical_mutation(request, public_worker_id=worker.id) reservation = _reserve_canonical_request(config, request, canonical) if isinstance(reservation, CommandEnvelope): - if isinstance(prepared, PreparedInstructionMutation): - _close_socket_client(prepared.client) return reservation - if permanent_error is not None: - return _finish_before_send( + + send_started: Mapping[str, Any] | None = None + send_start_outcome: CommandEnvelope | None = None + + class SendStartRejected(RuntimeError): + pass + + def mark_send_started_at_transport_boundary() -> None: + nonlocal send_started, send_start_outcome + started = _mark_request_send_started( config, request, reservation, - permanent_error, + binding_fingerprint=binding_fingerprint, + worker=worker, + instruction_text=_instruction_text(request), ) - if isinstance(prepared, PreSendFailure): - return _finish_before_send(config, request, reservation, prepared.envelope) - assert isinstance(prepared, PreparedInstructionMutation) - return _submit_instruction( - config, + if isinstance(started, CommandEnvelope): + send_start_outcome = started + raise SendStartRejected + if not isinstance(started, Mapping): + send_start_outcome = _recover_request( + config, request, reservation.canonical + ) + raise SendStartRejected + send_started = started + + def retryable_before_transport() -> CommandEnvelope: + abandoned = False + if config.db_path is not None: + try: + abandoned = abandon_command_request_reservation( + config.db_path, + host_id=config.host_id, + request_id=request.request_id or "", + canonical_fingerprint=reservation.canonical.fingerprint, + owner_token=reservation.owner_token, + ) + except Exception: # noqa: BLE001 + abandoned = False + if abandoned: + return _backend_unavailable( + request, + "ACP prompt did not reach its transport boundary", + ) + return _recover_request(config, request, reservation.canonical) + + try: + use_steering = False + steer = getattr(active_route, "steer", None) + if callable(steer): + try: + # The ACP runtime is authoritative about whether this + # exact session currently has an appendable prompt. A + # Herdr snapshot can briefly remain idle after the ACP + # prompt has started; requiring both signals opens a + # second session/prompt that the adapter serializes behind + # the live turn, so its submission acknowledgement can + # never arrive in time. An actually idle runtime reports + # supports_steering=False and keeps the normal prompt path. + use_steering = ( + getattr(active_route, "supports_steering", False) is True + ) + except Exception: + use_steering = False + submit = steer if use_steering else active_route.prompt + route_result = submit( + _instruction_text(request), + producer_turn_id=turn_submission_id( + config.host_id, + request.request_id or "", + ), + timeout=config.acp_request_timeout_seconds, + on_send_start=mark_send_started_at_transport_boundary, + ) + if use_steering: + raw_outcome = getattr(route_result, "outcome", None) + steering_outcome = getattr(raw_outcome, "value", raw_outcome) + if steering_outcome == "failed": + return _finish_request( + config, + request, + reservation, + _instruction_rejected_envelope( + request, + worker, + verdict="steering_failed", + ), + expected_state="send_started", + terminal_state="rejected", + ) + if steering_outcome not in {"injected", "startedNewTurn"}: + return _finish_request( + config, + request, + reservation, + _instruction_uncertain_envelope( + request, + worker, + verdict="unknown", + ), + expected_state="send_started", + terminal_state="uncertain", + ) + except SendStartRejected: + return send_start_outcome or _recover_request( + config, request, reservation.canonical + ) + except Exception: # noqa: BLE001 + if send_started is None: + return retryable_before_transport() + return _finish_request( + config, + request, + reservation, + _instruction_uncertain_envelope( + request, + worker, + verdict="unknown", + ), + expected_state="send_started", + terminal_state="uncertain", + ) + + if send_started is None: + # A route that claims success without crossing the durable + # transport boundary is not an accepted implementation. + return retryable_before_transport() + + observed_turn: Mapping[str, Any] | None = send_started + if config.db_path is not None: + try: + refreshed = linked_turn_for_submission( + config.db_path, + host_id=config.host_id, + request_id=request.request_id or "", + ) + except Exception: # noqa: BLE001 + refreshed = None + if isinstance(refreshed, Mapping): + observed_turn = refreshed + accepted = _accepted_send_envelope( request, worker, + observed_turn, + submission_verdict="submitted", + ) + return _finish_request( + config, + request, reservation, - prepared, + accepted, + expected_state="send_started", + terminal_state="accepted", + ) + + prepare_route = getattr(route, "prepare", None) + if not callable(prepare_route): + return submit_through(route) + + # Enter the generation fence before reserving a receipt. A failed status + # check is therefore a retryable no-receipt outcome, never a fabricated + # send_started ambiguity. Keep the context active until the ACP client has + # acknowledged writing the complete prompt frame. + try: + preparation = prepare_route() + active_route = preparation.__enter__() + except Exception: # noqa: BLE001 - no receipt or transport exists yet + if takeover is not None: + return _request_in_progress(request) + return _backend_unavailable( + request, "ACP worker route could not be prepared" ) + if active_route is None: + active_route = route + try: + return submit_through(active_route) + finally: + preparation.__exit__(None, None, None) + + +def _submit_command_v2( + config: Config, + params: Mapping[str, Any] | str, + *, + acp_permission_router: AcpPermissionDecisionRouter | None = None, +) -> CommandEnvelope: + """Handle non-prompt commands and ACP permission decisions.""" + + payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) + request, parse_error = parse_command_request(payload) + if parse_error is not None: + return CommandEnvelope.from_error(request, parse_error) + + validation_error = validate_request(request) + if validation_error is not None: + return CommandEnvelope.from_error(request, validation_error) + + if request.action not in _MUTATING_ACTIONS: + return _execute_non_mutating(config, request) + if request.dry_run: + return _mutation_dry_run(request) + if request.action == "send_instruction": + # Valid live instructions are consumed by submit_acp_command before this + # shared parser path. Never expose a second command transport. + return _backend_unavailable(request, "ACP worker route is unavailable") + + existing_receipt: Mapping[str, Any] | None = None + if config.db_path is not None: + try: + candidate = get_command_request( + config.db_path, + config.host_id, + request.request_id or "", + ) + except Exception: + candidate = None + if isinstance(candidate, Mapping): + existing_receipt = candidate + + takeover: _ReceiptTakeover | None = None + if existing_receipt is not None: + decided = _receipt_authority(config, request, existing_receipt) + if isinstance(decided, CommandEnvelope): + return decided + takeover = decided answer_pre_send: PreSendFailure | None = None - validate_answer = ( - _validate_pending_decision - if request.action == "answer_decision" - else _validate_pending_choice - ) if takeover is not None: - # Re-driving an abandoned answer reservation: the receipt already fixed - # which worker this request answers, so a pending interaction that now - # routes elsewhere is a changed target, not a new one. existing_worker_id = takeover.public_worker_id canonical = build_canonical_mutation( request, public_worker_id=existing_worker_id, ) - validated = validate_answer(config, request) + validated = _validate_pending_decision(config, request) if isinstance(validated, PreSendFailure): answer_pre_send = validated elif validated.worker_id != existing_worker_id: answer_pre_send = _permanent_pre_send(_duplicate_request(request)) else: - validated = validate_answer(config, request) + validated = _validate_pending_decision(config, request) if isinstance(validated, PreSendFailure): - # No reservation exists yet, so neither a transient nor a permanent - # validation failure writes a receipt here. Return it directly. - if health_error is not None and request.action != "answer_decision": - return health_error return validated.envelope canonical = build_canonical_mutation( request, public_worker_id=validated.worker_id, ) - # A safe transient pre-send failure (the pending store raised) never began a - # send. Keep it retryable under the same request ID without reserving. if answer_pre_send is not None and answer_pre_send.is_transient: - if takeover is not None: - return _request_in_progress(request) - return answer_pre_send.envelope + return ( + _request_in_progress(request) + if takeover is not None + else answer_pre_send.envelope + ) if ( answer_pre_send is not None and answer_pre_send.envelope.status == STATUS_ANSWER_IN_PROGRESS ): - # Another request owns the still-live decision claim. Keep this - # abandoned reservation nonterminal so it can take over after that - # claim is released or expires. return _answer_in_progress(request, receipt_reserved=True) - client_or_error: Any | CommandEnvelope | None = None - if answer_pre_send is None and health_error is None: - client_or_error = _connect_socket(config, request, socket_client_factory) - if isinstance(client_or_error, CommandEnvelope): - # The socket could not be reached before any transmission -> safe - # transient. Stay retryable rather than reserving a durable rejection. - if takeover is not None: - return _request_in_progress(request) - return client_or_error + if answer_pre_send is None: + try: + owns_decision = bool( + acp_permission_router is not None + and acp_permission_router.owns_permission_decision(validated) + ) + except Exception: + owns_decision = False + if not owns_decision: + unavailable = _backend_unavailable( + request, + "ACP permission authority is temporarily unavailable", + ) + return ( + _request_in_progress(request) + if takeover is not None + else unavailable + ) reservation = _reserve_canonical_request(config, request, canonical) if isinstance(reservation, CommandEnvelope): - if client_or_error is not None and not isinstance(client_or_error, CommandEnvelope): - _close_socket_client(client_or_error) return reservation if answer_pre_send is not None: return _finish_before_send( @@ -2800,28 +2124,12 @@ def _submit_command_v2( reservation, answer_pre_send.envelope, ) - if health_error is not None: - return _finish_before_send( - config, - request, - reservation, - health_error, - ) - assert client_or_error is not None - if request.action == "answer_decision": - return _answer_decision( - config, - request, - validated, - reservation, - client_or_error, - ) - return _answer_pending( + return _answer_decision( config, request, validated, reservation, - client_or_error, + acp_permission_router=acp_permission_router, ) @@ -2888,13 +2196,33 @@ def submit_command( config: Config, params: Mapping[str, Any] | str, *, - socket_client_factory: SocketClientFactory | None = None, + acp_prompt_router: AcpPromptRouter | None = None, + acp_permission_router: AcpPermissionDecisionRouter | None = None, ) -> CommandEnvelope: - """Submit one command and apply optional response-envelope negotiation.""" + """Submit one command with ACP as the only instruction transport.""" + acp_envelope = submit_acp_command( + config, + params, + prompt_router=acp_prompt_router or (lambda _worker: None), + ) + if acp_envelope is not None: + payload = ( + params + if isinstance(params, str) + else _raw_payload_from_mapping(params) + ) + request, parse_error = parse_command_request(payload) + if parse_error is None and request is not None: + return _negotiated_submission_envelope( + config, + request, + acp_envelope, + ) + return acp_envelope envelope = _submit_command_v2( config, params, - socket_client_factory=socket_client_factory, + acp_permission_router=acp_permission_router, ) payload = params if isinstance(params, str) else _raw_payload_from_mapping(params) request, parse_error = parse_command_request(payload) diff --git a/src/tendwire/config.py b/src/tendwire/config.py index 64ee04c..60a4e68 100644 --- a/src/tendwire/config.py +++ b/src/tendwire/config.py @@ -6,7 +6,6 @@ from __future__ import annotations -import logging import math import os import platform @@ -14,16 +13,16 @@ from dataclasses import dataclass, field from pathlib import Path -HERDR_BACKENDS = frozenset({"cli", "socket"}) -TURN_MODELS = frozenset({"legacy", "dual", "shadow", "observed"}) -DEFAULT_TURN_MODEL = "observed" -DEFAULT_EVENT_DEBOUNCE_SECONDS = 0.05 -DEFAULT_RECONCILE_INTERVAL_SECONDS = 300.0 +ACP_THOUGHT_POLICIES = frozenset({"disabled", "private_summary", "private_all"}) +ACP_CONSOLE_INPUT_POLICIES = frozenset({"preserve", "live_only"}) +DEFAULT_ACP_THOUGHT_POLICY = "disabled" +DEFAULT_ACP_CONSOLE_INPUT_POLICY = "preserve" +DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS = 30.0 +DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS = 5.0 +DEFAULT_ACP_MAX_FRAME_BYTES = 8 * 1024 * 1024 +DEFAULT_RECONCILE_INTERVAL_SECONDS = 15.0 DEFAULT_EVENT_RETENTION_DAYS = 7 -DEFAULT_OUTPUT_EXCERPT_CHARS = 200 DEFAULT_MAX_WORKERS = 512 -DEFAULT_TURN_REFRESH_INTERVAL_SECONDS = 2.0 -DEFAULT_TURN_REFRESH_WORKERS = 4 DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS = 60 DEFAULT_SUBMISSION_HARD_TTL_SECONDS = 86_400 DEFAULT_PENDING_STALE_GRACE_SECONDS = 30.0 @@ -49,7 +48,6 @@ MAX_RETENTION_DAYS = 365_000 MAX_SQLITE_INTEGER = (1 << 63) - 1 MAX_MAINTENANCE_CADENCE_SECONDS = MAX_RETENTION_DAYS * 24 * 60 * 60 -_LOGGER = logging.getLogger(__name__) @dataclass(frozen=True) @@ -62,15 +60,14 @@ class Config: db_path: Path | None = None socket_path: Path | None = None herdr_timeout_seconds: float = 5.0 - herdr_backend: str = "cli" - turn_model: str = DEFAULT_TURN_MODEL - event_debounce_seconds: float = DEFAULT_EVENT_DEBOUNCE_SECONDS + acp_thought_policy: str = DEFAULT_ACP_THOUGHT_POLICY + acp_console_input_policy: str = DEFAULT_ACP_CONSOLE_INPUT_POLICY + acp_request_timeout_seconds: float = DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS + acp_shutdown_timeout_seconds: float = DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS + acp_max_frame_bytes: int = DEFAULT_ACP_MAX_FRAME_BYTES reconcile_interval_seconds: float = DEFAULT_RECONCILE_INTERVAL_SECONDS event_retention_days: int = DEFAULT_EVENT_RETENTION_DAYS - output_excerpt_chars: int = DEFAULT_OUTPUT_EXCERPT_CHARS max_workers: int = DEFAULT_MAX_WORKERS - turn_refresh_interval_seconds: float = DEFAULT_TURN_REFRESH_INTERVAL_SECONDS - turn_refresh_workers: int = DEFAULT_TURN_REFRESH_WORKERS submission_link_window_seconds: int = DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS submission_hard_ttl_seconds: int = DEFAULT_SUBMISSION_HARD_TTL_SECONDS pending_stale_grace_seconds: float = DEFAULT_PENDING_STALE_GRACE_SECONDS @@ -109,71 +106,79 @@ def __post_init__(self) -> None: normalized_socket_group = str(self.socket_group).strip() object.__setattr__(self, "socket_group", normalized_socket_group or None) - if self.herdr_timeout_seconds <= 0: - raise ValueError("herdr_timeout_seconds must be positive") - backend = str(self.herdr_backend or "").strip().lower() - if backend not in HERDR_BACKENDS: - allowed = ", ".join(sorted(HERDR_BACKENDS)) - raise ValueError(f"herdr_backend must be one of: {allowed}") - object.__setattr__(self, "herdr_backend", backend) - turn_model = str(self.turn_model or "").strip().lower() - if turn_model not in TURN_MODELS: - allowed = ", ".join(sorted(TURN_MODELS)) - raise ValueError(f"turn_model must be one of: {allowed}") - object.__setattr__(self, "turn_model", turn_model) - if turn_model != "observed": - _LOGGER.warning( - "turn_model=%s is a compatibility alias and behaves as observed", - turn_model, - ) object.__setattr__( self, - "event_debounce_seconds", - _non_negative_float(self.event_debounce_seconds, "event_debounce_seconds"), + "herdr_timeout_seconds", + _positive_finite_float( + self.herdr_timeout_seconds, + "herdr_timeout_seconds", + ), ) + acp_thought_policy = str(self.acp_thought_policy or "").strip().lower() + if acp_thought_policy not in ACP_THOUGHT_POLICIES: + allowed = ", ".join(sorted(ACP_THOUGHT_POLICIES)) + raise ValueError(f"acp_thought_policy must be one of: {allowed}") + object.__setattr__(self, "acp_thought_policy", acp_thought_policy) + acp_console_input_policy = str( + self.acp_console_input_policy or "" + ).strip().lower() + if acp_console_input_policy not in ACP_CONSOLE_INPUT_POLICIES: + allowed = ", ".join(sorted(ACP_CONSOLE_INPUT_POLICIES)) + raise ValueError( + f"acp_console_input_policy must be one of: {allowed}" + ) object.__setattr__( self, - "reconcile_interval_seconds", - _non_negative_float(self.reconcile_interval_seconds, "reconcile_interval_seconds"), + "acp_console_input_policy", + acp_console_input_policy, ) object.__setattr__( self, - "event_retention_days", - _bounded_positive_int( - self.event_retention_days, - "event_retention_days", - maximum=MAX_RETENTION_DAYS, + "acp_request_timeout_seconds", + _positive_finite_float( + self.acp_request_timeout_seconds, + "acp_request_timeout_seconds", ), ) object.__setattr__( self, - "output_excerpt_chars", - _positive_int(self.output_excerpt_chars, "output_excerpt_chars", minimum=1), + "acp_shutdown_timeout_seconds", + _positive_finite_float( + self.acp_shutdown_timeout_seconds, + "acp_shutdown_timeout_seconds", + ), ) object.__setattr__( self, - "max_workers", - _positive_int(self.max_workers, "max_workers", minimum=1), + "acp_max_frame_bytes", + _bounded_positive_int( + self.acp_max_frame_bytes, + "acp_max_frame_bytes", + maximum=64 * 1024 * 1024, + ), ) object.__setattr__( self, - "turn_refresh_interval_seconds", + "reconcile_interval_seconds", _positive_finite_float( - self.turn_refresh_interval_seconds, - "turn_refresh_interval_seconds", + self.reconcile_interval_seconds, + "reconcile_interval_seconds", ), ) object.__setattr__( self, - "turn_refresh_workers", + "event_retention_days", _bounded_positive_int( - self.turn_refresh_workers, - "turn_refresh_workers", - maximum=32, + self.event_retention_days, + "event_retention_days", + maximum=MAX_RETENTION_DAYS, ), ) - if self.turn_refresh_workers > self.max_workers: - raise ValueError("turn_refresh_workers must be <= max_workers") + object.__setattr__( + self, + "max_workers", + _positive_int(self.max_workers, "max_workers", minimum=1), + ) object.__setattr__( self, "submission_link_window_seconds", @@ -441,15 +446,14 @@ def load_config( socket_path: str | Path | None = None, socket_group: str | None = None, herdr_timeout_seconds: float | str | None = None, - herdr_backend: str | None = None, - turn_model: str | None = None, - event_debounce_seconds: float | str | None = None, + acp_thought_policy: str | None = None, + acp_console_input_policy: str | None = None, + acp_request_timeout_seconds: float | str | None = None, + acp_shutdown_timeout_seconds: float | str | None = None, + acp_max_frame_bytes: int | str | None = None, reconcile_interval_seconds: float | str | None = None, event_retention_days: int | str | None = None, - output_excerpt_chars: int | str | None = None, max_workers: int | str | None = None, - turn_refresh_interval_seconds: float | str | None = None, - turn_refresh_workers: int | str | None = None, submission_link_window_seconds: int | str | None = None, submission_hard_ttl_seconds: int | str | None = None, pending_stale_grace_seconds: float | str | None = None, @@ -478,7 +482,6 @@ def load_config( env_socket_path = os.environ.get("TENDWIRE_SOCKET_PATH") env_socket_group = os.environ.get("TENDWIRE_SOCKET_GROUP") env_herdr_timeout_seconds = os.environ.get("TENDWIRE_HERDR_TIMEOUT_SECONDS") - env_herdr_backend = os.environ.get("TENDWIRE_HERDR_BACKEND") resolved_host_id = host_id or env_host_id or (platform.node() or "unknown") resolved_herdr_bin = herdr_bin or env_herdr_bin or "herdr" @@ -519,12 +522,6 @@ def load_config( except (TypeError, ValueError) as exc: raise ValueError("herdr timeout must be a positive number") from exc - resolved_herdr_backend = herdr_backend - if resolved_herdr_backend is None: - resolved_herdr_backend = env_herdr_backend - if resolved_herdr_backend is None: - resolved_herdr_backend = "cli" - return Config( host_id=resolved_host_id, herdr_bin=resolved_herdr_bin, @@ -532,16 +529,30 @@ def load_config( db_path=resolved_db_path, socket_path=resolved_socket_path, herdr_timeout_seconds=resolved_herdr_timeout_seconds, - herdr_backend=resolved_herdr_backend, - turn_model=_resolve_value( - turn_model, - "TENDWIRE_TURN_MODEL", - DEFAULT_TURN_MODEL, + acp_thought_policy=_resolve_value( + acp_thought_policy, + "TENDWIRE_ACP_THOUGHT_POLICY", + DEFAULT_ACP_THOUGHT_POLICY, + ), + acp_console_input_policy=_resolve_value( + acp_console_input_policy, + "TENDWIRE_ACP_CONSOLE_INPUT_POLICY", + DEFAULT_ACP_CONSOLE_INPUT_POLICY, + ), + acp_request_timeout_seconds=_resolve_value( + acp_request_timeout_seconds, + "TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS", + DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS, ), - event_debounce_seconds=_resolve_value( - event_debounce_seconds, - "TENDWIRE_EVENT_DEBOUNCE_SECONDS", - DEFAULT_EVENT_DEBOUNCE_SECONDS, + acp_shutdown_timeout_seconds=_resolve_value( + acp_shutdown_timeout_seconds, + "TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS", + DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS, + ), + acp_max_frame_bytes=_resolve_value( + acp_max_frame_bytes, + "TENDWIRE_ACP_MAX_FRAME_BYTES", + DEFAULT_ACP_MAX_FRAME_BYTES, ), reconcile_interval_seconds=_resolve_value( reconcile_interval_seconds, @@ -553,26 +564,11 @@ def load_config( "TENDWIRE_EVENT_RETENTION_DAYS", DEFAULT_EVENT_RETENTION_DAYS, ), - output_excerpt_chars=_resolve_value( - output_excerpt_chars, - "TENDWIRE_OUTPUT_EXCERPT_CHARS", - DEFAULT_OUTPUT_EXCERPT_CHARS, - ), max_workers=_resolve_value( max_workers, "TENDWIRE_MAX_WORKERS", DEFAULT_MAX_WORKERS, ), - turn_refresh_interval_seconds=_resolve_value( - turn_refresh_interval_seconds, - "TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS", - DEFAULT_TURN_REFRESH_INTERVAL_SECONDS, - ), - turn_refresh_workers=_resolve_value( - turn_refresh_workers, - "TENDWIRE_TURN_REFRESH_WORKERS", - DEFAULT_TURN_REFRESH_WORKERS, - ), submission_link_window_seconds=_resolve_value( submission_link_window_seconds, "TENDWIRE_SUBMISSION_LINK_WINDOW_SECONDS", diff --git a/src/tendwire/connectors/outbox.py b/src/tendwire/connectors/outbox.py index fbb6888..dd3866d 100644 --- a/src/tendwire/connectors/outbox.py +++ b/src/tendwire/connectors/outbox.py @@ -10,7 +10,7 @@ from pathlib import Path from typing import Any -from ..config import DEFAULT_CONNECTOR_ACK_TTL_SECONDS, DEFAULT_TURN_MODEL +from ..config import DEFAULT_CONNECTOR_ACK_TTL_SECONDS from ..core.models import sanitize_public_mapping, sanitize_public_value from ..store.sqlite import ( ack_connector_delivery, @@ -124,6 +124,9 @@ def _restore_plan_tokens(clean: dict[str, Any], original: Mapping[str, Any]) -> ) if final_identity: clean["final_identity"] = final_identity + content_revision = _revision(original.get("content_revision")) + if content_revision: + clean["content_revision"] = content_revision turn_id = _text(original.get("turn_id")) if ( turn_id.startswith("turn-") @@ -131,15 +134,17 @@ def _restore_plan_tokens(clean: dict[str, Any], original: Mapping[str, Any]) -> and all(char in _CONNECTOR_NAME_CHARS for char in turn_id) ): clean["turn_id"] = turn_id - nested_turn = original.get("turn") - if isinstance(nested_turn, Mapping): - clean_nested = clean.get("turn") + for nested_key in ("turn", "content"): + nested_turn = original.get(nested_key) + if not isinstance(nested_turn, Mapping): + continue + clean_nested = clean.get(nested_key) if not isinstance(clean_nested, dict): clean_nested = sanitize_public_mapping( nested_turn, backend_neutral=True, ) - clean["turn"] = clean_nested + clean[nested_key] = clean_nested _restore_plan_tokens(clean_nested, nested_turn) return clean @@ -212,7 +217,6 @@ def __init__( max_lease_seconds: int = 300, ack_ttl_seconds: int = DEFAULT_CONNECTOR_ACK_TTL_SECONDS, max_attempts: int = 10, - turn_model: str = DEFAULT_TURN_MODEL, ) -> None: self.db_path = Path(db_path) if db_path is not None else None self.host_id = str(host_id) @@ -220,7 +224,6 @@ def __init__( self.max_lease_seconds = max(1, int(max_lease_seconds)) self.ack_ttl_seconds = max(1, int(ack_ttl_seconds)) self.max_attempts = max(1, int(max_attempts)) - self.turn_model = str(turn_model or DEFAULT_TURN_MODEL).strip().lower() def _require_store(self, name: str = "") -> dict[str, Any] | None: if self.db_path is None: @@ -288,7 +291,6 @@ def prepare(self, params: Mapping[str, Any] | None = None) -> dict[str, Any]: presentation_version=version, part_count=part_count, source_ref=source_ref, - turn_model=self.turn_model, ) if action == "recover": @@ -432,16 +434,18 @@ def poll(self, params: Mapping[str, Any] | None = None) -> dict[str, Any]: if not ref: continue clean_payload = _clean_mapping(item.get("payload")) - clean_item = sanitize_public_value( - { - "ref": ref, - "key": str(item.get("key") or ""), - "attempt": int(item.get("attempt") or 0), - "leased_until": str(item.get("leased_until") or ""), - "available_at": str(item.get("available_at") or ""), - "payload": clean_payload, - } - ) + public_item = { + "ref": ref, + "key": str(item.get("key") or ""), + "attempt": int(item.get("attempt") or 0), + "leased_until": str(item.get("leased_until") or ""), + "available_at": str(item.get("available_at") or ""), + "payload": clean_payload, + } + created_at = str(item.get("created_at") or "") + if name == _TURN_FINAL_NAME and created_at: + public_item["created_at"] = created_at + clean_item = sanitize_public_value(public_item) if isinstance(clean_item, dict): item_key = str(item.get("key") or "") if item_key.startswith(_FINAL_KEY_PREFIX): diff --git a/src/tendwire/core/actions.py b/src/tendwire/core/actions.py index 373633b..06b7713 100644 --- a/src/tendwire/core/actions.py +++ b/src/tendwire/core/actions.py @@ -13,9 +13,6 @@ from ..config import Config from .commands import ( STATUS_AMBIGUOUS_TARGET, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DRY_RUN, STATUS_NOOP, STATUS_NOT_FOUND, STATUS_REJECTED, @@ -30,7 +27,6 @@ resolve_target, snapshot_result, validate_request, - worker_candidate, ) from .projector import project_from_observations @@ -110,84 +106,6 @@ def _resolve_target_result(request: CommandRequest, workers: list[Worker]) -> Co ) -def _send_instruction_result(request: CommandRequest, context: CommandContext) -> CommandEnvelope: - resolved, candidates, status = resolve_target( - request.target, - context.workers, - allow_disallowed_status=True, - include_backend_target=True, - ) - if status != STATUS_RESOLVED: - return _resolve_target_result(request, context.workers) - - # Even though resolve_target succeeded, send_instruction must reject workers - # whose current status is closed, failed, or unknown. - resolved_worker = next( - (w for w in context.workers if w.id == (resolved or {}).get("worker_id")), - None, - ) - disallowed = {"closed", "failed", "unknown"} - if resolved_worker is not None and resolved_worker.status in disallowed: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_REJECTED, - result={"candidates": [worker_candidate(resolved_worker)]}, - error=error_value( - STATUS_REJECTED, - f"target worker status does not allow instructions: {resolved_worker.status!r}", - ), - ) - - instruction = request.instruction or {} - target = resolved or {} - text = instruction.get("text", "") - - if request.dry_run: - public_target = worker_candidate(resolved_worker) if resolved_worker is not None else target - return CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_DRY_RUN, - result={"target": public_target, "instruction": {"text": text}}, - ) - - backend_target = target.get("backend_target") - backend_reason = "" - if isinstance(backend_target, dict): - backend_reason = str(backend_target.get("reason") or "") - if not isinstance(backend_target, dict) or backend_target.get("sendable") is not True: - if backend_reason in {"duplicate_backend_target", "not_unique"}: - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_AMBIGUOUS_BACKEND_TARGET, - error=error_value( - STATUS_AMBIGUOUS_BACKEND_TARGET, - "target resolves to an ambiguous backend send target", - ), - ) - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_BACKEND_UNSUPPORTED, - error=error_value( - STATUS_BACKEND_UNSUPPORTED, - "target has no backend-owned sendable private binding", - ), - ) - - return CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_BACKEND_UNSUPPORTED, - error=error_value( - STATUS_BACKEND_UNSUPPORTED, - "live mutations require the authoritative command submission path", - ), - ) - - def execute_command(request: CommandRequest, context: CommandContext) -> CommandEnvelope: """Execute a validated command request and return a neutral envelope.""" validation_error = validate_request(request) @@ -210,9 +128,6 @@ def execute_command(request: CommandRequest, context: CommandContext) -> Command if request.action == "resolve_target": return _resolve_target_result(request, context.workers) - if request.action == "send_instruction": - return _send_instruction_result(request, context) - return CommandEnvelope.from_error( request, error_value(STATUS_REJECTED, f"unknown action {request.action!r}"), diff --git a/src/tendwire/core/agent_events.py b/src/tendwire/core/agent_events.py new file mode 100644 index 0000000..f20a789 --- /dev/null +++ b/src/tendwire/core/agent_events.py @@ -0,0 +1,393 @@ +"""Canonical structured events emitted by agent-protocol backends. + +The event contract deliberately keeps source identifiers and the unsanitized +payload on the private side of Tendwire's trust boundary. A separate public +projection is produced at construction time so connector code never has to +guess which source fields are safe to expose. +""" + +from __future__ import annotations + +import hashlib +import json +import math +from collections.abc import Mapping +from copy import deepcopy +from dataclasses import dataclass +from datetime import datetime, timezone +from typing import Any, Literal + +from .models import sanitize_public_mapping, utc_timestamp + +AgentEventKind = Literal[ + "user_message", + "agent_message", + "thought", + "tool_call", + "tool_call_update", + "plan", + "usage", + "session_info", + "extension", +] +AgentEventVisibility = Literal["private", "public"] + +AGENT_EVENT_KINDS = frozenset( + { + "user_message", + "agent_message", + "thought", + "tool_call", + "tool_call_update", + "plan", + "usage", + "session_info", + "extension", + } +) +AGENT_EVENT_VISIBILITIES = frozenset({"private", "public"}) +AGENT_EVENT_PRIVATE_KINDS = frozenset( + {"thought", "tool_call", "tool_call_update", "plan", "extension"} +) +AGENT_EVENT_MAX_PAYLOAD_BYTES = 16 * 1024 * 1024 +AGENT_EVENT_MAX_PUBLIC_PAYLOAD_BYTES = 64 * 1024 +AGENT_EVENT_MAX_TEXT_CHARS = 4 * 1024 * 1024 +AGENT_EVENT_MAX_COLLECTION_ITEMS = 4096 +AGENT_EVENT_MAX_TOTAL_ITEMS = 32768 +AGENT_EVENT_MAX_DEPTH = 12 +AGENT_EVENT_MAX_IDENTIFIER_CHARS = 2048 +AGENT_EVENT_QUERY_DEFAULT_LIMIT = 100 +AGENT_EVENT_QUERY_MAX_LIMIT = 1000 +AGENT_EVENT_SCHEMA_VERSION = 1 + + +def _canonical_json(value: Any) -> str: + return json.dumps( + value, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ) + + +def _fingerprint(value: Any) -> str: + return hashlib.sha256(_canonical_json(value).encode("utf-8")).hexdigest() + + +def normalize_agent_event_identifier( + value: Any, + field: str, + *, + required: bool = False, +) -> str | None: + """Validate an opaque identifier without changing its source identity. + + Protocol identifiers are byte-significant. Compatibility normalization or + trimming would make distinct source IDs collide (for example ``"1"`` and + ``"①"`` under NFKC), defeating the journal's replay-conflict checks. + """ + if value is None: + if required: + raise ValueError(f"{field} must not be empty") + return None + if not isinstance(value, str): + raise ValueError(f"{field} must be text or None") + if not value.strip(): + raise ValueError(f"{field} must not be empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain NUL") + if len(value) > AGENT_EVENT_MAX_IDENTIFIER_CHARS: + raise ValueError(f"{field} is too long") + try: + value.encode("utf-8") + except UnicodeEncodeError as exc: + raise ValueError(f"{field} must contain valid Unicode") from exc + return value + + +def _timestamp(value: str | datetime | None) -> str: + if value is None: + return utc_timestamp() + if isinstance(value, datetime): + parsed = value + elif isinstance(value, str): + raw = value.strip() + if not raw or len(raw) > 64: + raise ValueError("observed_at must be an aware ISO-8601 timestamp") + if raw.endswith(("Z", "z")): + raw = raw[:-1] + "+00:00" + try: + parsed = datetime.fromisoformat(raw) + except ValueError as exc: + raise ValueError( + "observed_at must be an aware ISO-8601 timestamp" + ) from exc + else: + raise ValueError("observed_at must be an aware ISO-8601 timestamp") + if parsed.tzinfo is None or parsed.utcoffset() is None: + raise ValueError("observed_at must be an aware ISO-8601 timestamp") + return parsed.astimezone(timezone.utc).isoformat() + + +def _valid_unicode(value: str) -> str: + try: + value.encode("utf-8") + except UnicodeEncodeError as exc: + raise ValueError( + "agent event payload contains invalid Unicode" + ) from exc + return value + + +def _normalize_payload_value( + value: Any, + *, + depth: int = 0, + item_budget: list[int] | None = None, +) -> Any: + if item_budget is None: + item_budget = [AGENT_EVENT_MAX_TOTAL_ITEMS] + if depth > AGENT_EVENT_MAX_DEPTH: + raise ValueError("agent event payload is nested too deeply") + if value is None or isinstance(value, bool | int): + return value + if isinstance(value, float): + if not math.isfinite(value): + raise ValueError("agent event payload contains a non-finite number") + return value + if isinstance(value, datetime): + if value.tzinfo is None or value.utcoffset() is None: + raise ValueError("agent event payload contains a naive datetime") + return utc_timestamp(value) + if isinstance(value, str): + if len(value) > AGENT_EVENT_MAX_TEXT_CHARS: + raise ValueError("agent event payload text is too long") + return _valid_unicode(value) + if isinstance(value, Mapping): + if len(value) > AGENT_EVENT_MAX_COLLECTION_ITEMS: + raise ValueError("agent event payload mapping has too many entries") + item_budget[0] -= len(value) + if item_budget[0] < 0: + raise ValueError("agent event payload has too many total items") + result: dict[str, Any] = {} + for raw_key, item in value.items(): + if not isinstance(raw_key, str): + raise ValueError("agent event payload keys must be text") + key = _valid_unicode(raw_key) + if not key or len(key) > 256: + raise ValueError("agent event payload contains an invalid key") + result[key] = _normalize_payload_value( + item, + depth=depth + 1, + item_budget=item_budget, + ) + return result + if isinstance(value, tuple | list): + if len(value) > AGENT_EVENT_MAX_COLLECTION_ITEMS: + raise ValueError("agent event payload sequence has too many entries") + item_budget[0] -= len(value) + if item_budget[0] < 0: + raise ValueError("agent event payload has too many total items") + return [ + _normalize_payload_value( + item, + depth=depth + 1, + item_budget=item_budget, + ) + for item in value + ] + raise ValueError("agent event payload must contain only JSON-safe values") + + +def normalize_agent_event_payload(payload: Mapping[str, Any]) -> dict[str, Any]: + """Return a bounded, deterministic, JSON-safe private payload.""" + if not isinstance(payload, Mapping): + raise ValueError("agent event payload must be a mapping") + normalized = _normalize_payload_value(payload) + if not isinstance(normalized, dict): # Defensive; mappings normalize to dicts. + raise ValueError("agent event payload must be a mapping") + payload_size = len(_canonical_json(normalized).encode("utf-8")) + if payload_size > AGENT_EVENT_MAX_PAYLOAD_BYTES: + raise ValueError("agent event payload is too large") + return normalized + + +def public_agent_event_payload( + payload: Mapping[str, Any], + *, + visibility: AgentEventVisibility, +) -> dict[str, Any]: + """Build the bounded public projection for an event payload.""" + if visibility == "private": + return {} + public = sanitize_public_mapping(payload, backend_neutral=True) + if ( + len(_canonical_json(public).encode("utf-8")) + > AGENT_EVENT_MAX_PUBLIC_PAYLOAD_BYTES + ): + raise ValueError("public agent event payload is too large") + return public + + +@dataclass(frozen=True) +class AgentEvent: + """One immutable structured source event before durable sequencing.""" + + event_id: str + kind: AgentEventKind + source: str + worker_id: str + visibility: AgentEventVisibility + observed_at: str + payload: dict[str, Any] + public_payload: dict[str, Any] + payload_fingerprint: str + source_session_id: str | None = None + source_turn_id: str | None = None + source_item_id: str | None = None + source_message_id: str | None = None + source_event_id: str | None = None + source_sequence: int | None = None + + def public_dict(self, *, sequence: int | None = None) -> dict[str, Any]: + """Return a connector-safe view with all source identifiers omitted.""" + result: dict[str, Any] = { + "schema_version": AGENT_EVENT_SCHEMA_VERSION, + "event_id": self.event_id, + "kind": self.kind, + "worker_id": self.worker_id, + "visibility": self.visibility, + "observed_at": self.observed_at, + "payload": deepcopy(self.public_payload), + } + if sequence is not None: + result["sequence"] = int(sequence) + return result + + +def agent_event( + *, + kind: AgentEventKind | str, + source: str, + worker_id: str, + payload: Mapping[str, Any], + source_session_id: str | None = None, + source_turn_id: str | None = None, + source_item_id: str | None = None, + source_message_id: str | None = None, + source_event_id: str | None = None, + source_sequence: int | None = None, + visibility: AgentEventVisibility | str = "private", + observed_at: str | datetime | None = None, +) -> AgentEvent: + """Validate and construct an event with deterministic retry identity. + + Sources must provide either their event identifier or a session-local + monotonically assigned sequence. The source timestamp is intentionally + excluded from identity so replaying the same notification deduplicates. + """ + normalized_kind = str(kind).strip().lower() + if normalized_kind not in AGENT_EVENT_KINDS: + raise ValueError("unsupported agent event kind") + normalized_visibility = str(visibility).strip().lower() + if normalized_visibility not in AGENT_EVENT_VISIBILITIES: + raise ValueError("visibility must be private or public") + if ( + normalized_kind in AGENT_EVENT_PRIVATE_KINDS + and normalized_visibility != "private" + ): + raise ValueError(f"{normalized_kind} events must remain private") + normalized_source = normalize_agent_event_identifier( + source, "source", required=True + ) + normalized_worker = normalize_agent_event_identifier( + worker_id, "worker_id", required=True + ) + session_id = normalize_agent_event_identifier( + source_session_id, "source_session_id" + ) + turn_id = normalize_agent_event_identifier(source_turn_id, "source_turn_id") + item_id = normalize_agent_event_identifier(source_item_id, "source_item_id") + message_id = normalize_agent_event_identifier( + source_message_id, "source_message_id" + ) + event_id = normalize_agent_event_identifier(source_event_id, "source_event_id") + if source_sequence is not None and ( + isinstance(source_sequence, bool) + or not isinstance(source_sequence, int) + or source_sequence < 0 + or source_sequence > (1 << 63) - 1 + ): + raise ValueError("source_sequence must be a nonnegative SQLite integer") + if event_id is None and source_sequence is None: + raise ValueError("source_event_id or source_sequence is required") + if event_id is None and session_id is None: + raise ValueError("source_session_id is required with source_sequence") + normalized_payload = normalize_agent_event_payload(payload) + public_payload = public_agent_event_payload( + normalized_payload, + visibility=normalized_visibility, # type: ignore[arg-type] + ) + identity: dict[str, Any] = { + "schema_version": AGENT_EVENT_SCHEMA_VERSION, + "source": normalized_source, + "session_id": session_id, + } + if event_id is not None: + identity.update({"identity": "event_id", "value": event_id}) + else: + identity.update({"identity": "sequence", "value": source_sequence}) + return AgentEvent( + event_id=_fingerprint(identity), + kind=normalized_kind, # type: ignore[arg-type] + source=normalized_source or "", + worker_id=normalized_worker or "", + visibility=normalized_visibility, # type: ignore[arg-type] + observed_at=_timestamp(observed_at), + payload=normalized_payload, + public_payload=public_payload, + payload_fingerprint=_fingerprint(normalized_payload), + source_session_id=session_id, + source_turn_id=turn_id, + source_item_id=item_id, + source_message_id=message_id, + source_event_id=event_id, + source_sequence=source_sequence, + ) + + +@dataclass(frozen=True) +class StoredAgentEvent: + """One durably sequenced structured agent event.""" + + sequence: int + host_id: str + event: AgentEvent + + def public_dict(self) -> dict[str, Any]: + return self.event.public_dict(sequence=self.sequence) + + +@dataclass(frozen=True) +class AppendAgentEventResult: + sequence: int + event_id: str + inserted: bool + + +@dataclass(frozen=True) +class AppendBoundAgentEventResult: + """Atomic binding check and journal append outcome.""" + + status: Literal["inserted", "replayed", "binding_changed"] + event_id: str + sequence: int | None = None + + @property + def inserted(self) -> bool: + return self.status == "inserted" + + +class AgentEventIdentityConflict(RuntimeError): + """The same deterministic source identity was reused for other content.""" diff --git a/src/tendwire/core/commands.py b/src/tendwire/core/commands.py index 1c9a3a0..a37e1db 100644 --- a/src/tendwire/core/commands.py +++ b/src/tendwire/core/commands.py @@ -43,7 +43,6 @@ "read_snapshot", "resolve_target", "send_instruction", - "answer_pending", "answer_decision", } ) @@ -189,7 +188,16 @@ ) # Neutral target fields permitted in command requests. -TARGET_ALLOWED_FIELDS = frozenset({"worker_id", "worker_fingerprint", "space_id", "name"}) +TARGET_ALLOWED_FIELDS = frozenset( + { + "worker_id", + "worker_fingerprint", + "space_id", + "name", + "stable_key", + "stable_key_version", + } +) # Selectors that name a worker durably. A worker_fingerprint is a mutable # observation precondition -- "proceed only if the worker still looks like this" @@ -197,11 +205,10 @@ # fingerprints would then be indistinguishable to any identity-based idempotency # key, letting one request ID claim another worker's stored result. Every target # must carry at least one of these. -TARGET_STABLE_SELECTOR_FIELDS = frozenset({"worker_id", "space_id", "name"}) -INSTRUCTION_ALLOWED_FIELDS = frozenset({"text"}) -ANSWER_PENDING_PARAM_FIELDS = frozenset( - {"pending_id", "pending_fingerprint", "choice_id"} +TARGET_STABLE_SELECTOR_FIELDS = frozenset( + {"worker_id", "space_id", "name", "stable_key"} ) +INSTRUCTION_ALLOWED_FIELDS = frozenset({"text"}) ANSWER_DECISION_PARAM_FIELDS = frozenset({"decision_ref", "selection"}) # Connector, low-level terminal, routing, and private fields rejected anywhere in a request. @@ -212,6 +219,7 @@ MAX_INSTRUCTION_LENGTH = 4096 _REQUEST_ID_RE = re.compile(r"[A-Za-z0-9._-]{1,128}", re.ASCII) +_STABLE_WORKER_KEY_RE = re.compile(r"wsk1_[0-9a-f]{64}", re.ASCII) _TURN_SUBMISSION_ID_RE = re.compile(r"twsub1\.[0-9a-f]{64}", re.ASCII) _INSTRUCTION_FINGERPRINT_DOMAIN = b"tendwire.instruction-fingerprint.v1" _TURN_SUBMISSION_ID_DOMAIN = b"tendwire.turn-submission-id.v1" @@ -376,8 +384,8 @@ def instruction_fingerprint(text: Any) -> str: Valid instruction text can consist entirely of whitespace even though its normalized form is empty. Preserve the normalized matching behavior for - ordinary text, but fingerprint the raw text in that edge case so shadow - ledger bookkeeping can never reject an otherwise valid legacy send. + ordinary text, but fingerprint the raw text in that edge case so submission + bookkeeping can never reject an otherwise valid send. """ normalized = normalize_instruction_text(text) fingerprint_text = normalized or str(text or "") @@ -422,6 +430,27 @@ def _validate_target_shape(target: dict[str, Any] | None) -> dict[str, Any] | No f"target contains disallowed fields: {sorted(extra)}", details={"field": "target", "disallowed": sorted(extra)}, ) + stable_key_present = "stable_key" in target + stable_version_present = "stable_key_version" in target + stable_key = target.get("stable_key") + stable_version = target.get("stable_key_version") + if stable_key_present != stable_version_present: + return error_value( + STATUS_INVALID_REQUEST, + "target stable_key and stable_key_version must be supplied together", + details={"field": "target"}, + ) + if stable_key_present and ( + not isinstance(stable_key, str) + or _STABLE_WORKER_KEY_RE.fullmatch(stable_key) is None + or type(stable_version) is not int + or stable_version != 1 + ): + return error_value( + STATUS_INVALID_REQUEST, + "target stable worker identity is invalid or unsupported", + details={"field": "target"}, + ) if _string_value(target.get("worker_fingerprint")) and not _target_has_stable_selector( target ): @@ -505,16 +534,6 @@ def to_dict(self) -> dict[str, Any]: payload["response_schema_version"] = self.response_schema_version return payload - def payload_fingerprint(self) -> str: - """Return the legacy raw-request fingerprint used by compatibility callers. - - This includes request identity and unresolved selector spelling, so it is - not authoritative for mutating-command idempotency. New mutation - persistence must use :func:`build_canonical_mutation` after resolving the - public worker identity. - """ - return stable_fingerprint(self.to_dict()) - @classmethod def from_dict(cls, data: dict[str, Any]) -> "CommandRequest": return cls( @@ -559,10 +578,8 @@ def build_canonical_mutation( """ if not isinstance(request, CommandRequest): raise TypeError("request must be a CommandRequest") - if request.action not in {"send_instruction", "answer_pending", "answer_decision"}: - raise ValueError( - "canonical mutations require send_instruction, answer_pending, or answer_decision" - ) + if request.action not in {"send_instruction", "answer_decision"}: + raise ValueError("canonical mutations require send_instruction or answer_decision") if request.dry_run is not False: raise ValueError("canonical mutations require a non-dry-run request") request_error = validate_request(request) @@ -580,19 +597,6 @@ def build_canonical_mutation( "instruction": {"text": request.instruction["text"]}, "options": {}, } - elif request.action == "answer_pending": - assert request.params is not None - canonical_payload = { - "canonical_version": CANONICAL_MUTATION_VERSION, - "action": "answer_pending", - "target": {"worker_id": public_worker_id}, - "pending": { - "pending_id": request.params["pending_id"], - "pending_fingerprint": request.params["pending_fingerprint"], - "choice_id": request.params["choice_id"], - }, - "options": {}, - } else: assert request.params is not None selection = request.params["selection"] @@ -661,10 +665,8 @@ def build_selector_proof(request: CommandRequest) -> str: """ if not isinstance(request, CommandRequest): raise TypeError("request must be a CommandRequest") - if request.action not in {"send_instruction", "answer_pending", "answer_decision"}: - raise ValueError( - "selector proofs require send_instruction, answer_pending, or answer_decision" - ) + if request.action not in {"send_instruction", "answer_decision"}: + raise ValueError("selector proofs require send_instruction or answer_decision") if request.dry_run is not False: raise ValueError("selector proofs require a non-dry-run request") request_error = validate_request(request) @@ -684,6 +686,8 @@ def build_selector_proof(request: CommandRequest) -> str: "worker_id": _string_value(target.get("worker_id")), "name": _string_value(target.get("name")), "space_id": _optional_string(target.get("space_id")), + "stable_key": _string_value(target.get("stable_key")), + "stable_key_version": target.get("stable_key_version"), } payload = { "proof_version": SELECTOR_PROOF_VERSION, @@ -745,7 +749,7 @@ def validate_request(request: CommandRequest) -> dict[str, Any] | None: details={"field": "action", "allowed": sorted(ALLOWED_ACTIONS)}, ) if ( - request.action in {"send_instruction", "answer_pending", "answer_decision"} + request.action in {"send_instruction", "answer_decision"} and request.dry_run is False and not is_valid_request_id(request.request_id) ): @@ -794,46 +798,6 @@ def validate_request(request: CommandRequest) -> dict[str, Any] | None: details={"field": "instruction.text"}, ) - if request.action == "answer_pending": - if request.target is not None: - return error_value( - STATUS_INVALID_REQUEST, - "answer_pending does not accept a target", - details={"field": "target"}, - ) - if request.instruction is not None: - return error_value( - STATUS_INVALID_REQUEST, - "answer_pending does not accept an instruction", - details={"field": "instruction"}, - ) - if not isinstance(request.params, dict): - return error_value( - STATUS_INVALID_REQUEST, - "answer_pending requires params", - details={"field": "params"}, - ) - actual_fields = set(request.params) - if actual_fields != ANSWER_PENDING_PARAM_FIELDS: - return error_value( - STATUS_INVALID_REQUEST, - "answer_pending params must contain exactly pending_id, pending_fingerprint, and choice_id", - details={ - "field": "params", - "required": sorted(ANSWER_PENDING_PARAM_FIELDS), - "missing": sorted(ANSWER_PENDING_PARAM_FIELDS - actual_fields), - "disallowed": sorted(actual_fields - ANSWER_PENDING_PARAM_FIELDS), - }, - ) - for field in sorted(ANSWER_PENDING_PARAM_FIELDS): - value = request.params.get(field) - if not isinstance(value, str) or not value.strip(): - return error_value( - STATUS_INVALID_REQUEST, - f"answer_pending requires nonblank params.{field}", - details={"field": f"params.{field}"}, - ) - if request.action == "answer_decision": if request.target is None or set(request.target) != {"worker_id"}: return error_value( @@ -1030,7 +994,6 @@ def __post_init__(self) -> None: mutating = self.action in { "send_instruction", - "answer_pending", "answer_decision", } live_mutation = mutating and self.dry_run is False @@ -1234,7 +1197,6 @@ def from_error(cls, request: CommandRequest | None, error: dict[str, Any]) -> "C ) mutating = request.action in { "send_instruction", - "answer_pending", "answer_decision", } valid_mutation_id = is_valid_request_id(request.request_id) @@ -1296,6 +1258,8 @@ def resolve_target( name = _string_value(target.get("name")) space_id = _optional_string(target.get("space_id")) fingerprint = _optional_string(target.get("worker_fingerprint")) + stable_key = _string_value(target.get("stable_key")) + stable_key_version = target.get("stable_key_version") # First match by identity/name/space, excluding fingerprint. identity_matches: list[Worker] = [] @@ -1306,6 +1270,11 @@ def resolve_target( continue if space_id is not None and worker.space_id != space_id: continue + if stable_key and ( + worker.meta.get("stable_key") != stable_key + or worker.meta.get("stable_key_version") != stable_key_version + ): + continue identity_matches.append(worker) # If a fingerprint was supplied, filter further. A non-empty identity match diff --git a/src/tendwire/core/models.py b/src/tendwire/core/models.py index f1aa12a..32157c3 100644 --- a/src/tendwire/core/models.py +++ b/src/tendwire/core/models.py @@ -489,6 +489,7 @@ "agent_prompt_unsubmitted", "agent_input_pending", "agent_prompt_stalled", + "steering_failed", "unknown", } ) diff --git a/src/tendwire/daemon.py b/src/tendwire/daemon.py index a92af95..4a67e2a 100644 --- a/src/tendwire/daemon.py +++ b/src/tendwire/daemon.py @@ -47,6 +47,15 @@ def _nonnegative_float(value: Any) -> float | None: return converted +def _public_failure_type(value: Any) -> str | None: + """Return a bounded exception type label, never arbitrary failure text.""" + if not isinstance(value, str) or not value or len(value) > 128: + return None + if any(not (character.isalnum() or character in "._") for character in value): + return None + return value + + _STORE_COUNT_FIELDS = ( "snapshots", "events", @@ -385,41 +394,6 @@ def _command_requests_health( }, True -def _turn_ingestion_health(config: Config, scheduler: Any | None) -> dict[str, Any]: - raw: Mapping[str, Any] = {} - if scheduler is not None: - try: - status_value = scheduler.operational_status() - except Exception: - status_value = {} - if isinstance(status_value, Mapping): - raw = status_value - status = raw.get("status") - if status not in {"healthy", "stale", "degraded", "stopping"}: - status = "stale" if scheduler is None else "degraded" - return { - "status": status, - "queue": _nonnegative_int(raw.get("queue_depth")), - "active": _nonnegative_int(raw.get("active")), - "refreshed": _nonnegative_int(raw.get("refreshed")), - "failed": _nonnegative_int(raw.get("failed")), - "timed_out": _nonnegative_int(raw.get("timed_out")), - "coalesced": _nonnegative_int(raw.get("coalesced")), - "queue_full": _nonnegative_int(raw.get("queue_full")), - "last_success": _valid_observation_timestamp( - raw.get("last_success") if isinstance(raw.get("last_success"), str) else None - ), - "last_duration_ms": _nonnegative_float(raw.get("last_duration_ms")), - "stale_age": _nonnegative_float(raw.get("stale_age_seconds")), - "bounds": { - "refresh_interval_seconds": config.turn_refresh_interval_seconds, - "max_workers": config.turn_refresh_workers, - "queue_capacity": _nonnegative_int(raw.get("queue_capacity")), - "adapter_timeout_seconds": config.herdr_timeout_seconds, - }, - } - - def _pending_ingestion_health(config: Config) -> dict[str, Any]: """Return the fixed durable pending aggregate without exposing row identity.""" unavailable = { @@ -475,37 +449,16 @@ def default_socket_path(config: Config) -> Path: return Path(config.data_dir) / "tendwire.sock" -def _default_init_store( - db_path: Path, - *, - connector_ack_ttl_seconds: int | None = None, -) -> None: +def _default_init_store(db_path: Path) -> None: from .store.sqlite import init_store - kwargs = ( - {"connector_ack_ttl_seconds": connector_ack_ttl_seconds} - if connector_ack_ttl_seconds is not None - else {} - ) - init_store(db_path, **kwargs) - - -def _default_observe_initial_snapshot(config: Config) -> Snapshot: - from .cli import observe_public_snapshot - - return observe_public_snapshot(config, store_snapshot=True) - - -def _default_submit_command(config: Config, payload: str) -> CommandEnvelope: - from .command_submission import submit_command + init_store(db_path) - return submit_command(config, payload) +def _default_acp_supervisor_factory(config: Config, stop_event: threading.Event) -> Any: + from .backends.acp_coordinator import production_acp_supervisor_factory -def _default_turn_scheduler_factory(config: Config) -> Any: - from .backends.herdr_turns import TurnIngestionScheduler - - return TurnIngestionScheduler(config) + return production_acp_supervisor_factory(config, stop_event) @dataclass(frozen=True) @@ -513,10 +466,9 @@ class DaemonHooks: """Dependency injection points for deterministic daemon tests.""" init_store: Callable[[Path], None] = _default_init_store - observe_initial_snapshot: Callable[[Config], Snapshot] = _default_observe_initial_snapshot - submit_command: Callable[[Config, str], CommandEnvelope | Mapping[str, Any]] = _default_submit_command - event_backend_factory: Callable[[Config, threading.Event], Any] | None = None - turn_scheduler_factory: Callable[[Config], Any] = _default_turn_scheduler_factory + acp_supervisor_factory: Callable[[Config, threading.Event], Any | None] | None = ( + _default_acp_supervisor_factory + ) class TendwireDaemon: @@ -538,8 +490,8 @@ def __init__( self.started_at = utc_timestamp() self._snapshot: Snapshot | None = None self._server: UnixSocketJSONServer | None = None - self._event_backend: Any | None = None - self._turn_scheduler: Any | None = None + self._acp_supervisor: Any | None = None + self._acp_startup_failure_type: str | None = None self._stop_lock = threading.Lock() self._automatic_maintenance_status: dict[str, Any] | None = None @@ -575,24 +527,9 @@ def start(self) -> None: socket_group=self.config.socket_group, ) if self.hooks.init_store is _default_init_store: - _default_init_store( - Path(self.config.db_path), - connector_ack_ttl_seconds=( - self.config.connector_ack_ttl_seconds - ), - ) + _default_init_store(Path(self.config.db_path)) else: self.hooks.init_store(Path(self.config.db_path)) - self._connector_periodic_tick() - if self.config.herdr_backend == "socket": - self._snapshot = self._start_socket_event_backend() - else: - self._snapshot = self.hooks.observe_initial_snapshot(self.config) - self._after_snapshot_saved() - - scheduler = self.hooks.turn_scheduler_factory(self.config) - self._turn_scheduler = scheduler - api = TendwireDaemonAPI( get_snapshot=self.get_snapshot, get_health=self.get_health, @@ -612,51 +549,27 @@ def start(self) -> None: prepare_parent=self._prepare_socket_parent, periodic_callback=self._connector_periodic_tick, ) - self._server = server - # Bind before ingestion starts. Managed store connections and the - # socket publisher lock the same parent directory, so allowing an - # initial refresh first can make the daemon deadlock with itself. - # Requests are not served until start() returns successfully. + # Bind before ACP runtime/consumer threads can take store locks. + # No connections are accepted until serve_forever(), after + # this startup transaction has succeeded. server.start() + self._connector_periodic_tick() + self._start_acp_supervisor() + from .store.sqlite import latest_snapshot - backend = self._event_backend - callback_setter = ( - getattr(backend, "set_turn_refresh_callback", None) - if backend is not None - else None + self._snapshot = latest_snapshot( + Path(self.config.db_path), self.config.host_id ) - if callable(callback_setter): - callback_setter(scheduler.request_refresh) - scheduler.start() - scheduler.request_refresh() + if self._snapshot is None: + raise RuntimeError("ACP supervisor did not publish a lifecycle snapshot") + self._after_snapshot_saved() + self._server = server + except Exception: self.stop_event.set() - backend = self._event_backend - callback_setter = ( - getattr(backend, "set_turn_refresh_callback", None) - if backend is not None - else None - ) - if callable(callback_setter): - try: - callback_setter(None) - except Exception: - pass - scheduler = self._turn_scheduler - self._turn_scheduler = None - if scheduler is not None: - try: - scheduler.stop( - flush_timeout_seconds=self.config.herdr_timeout_seconds + 1.0 - ) - except Exception: - pass - self._event_backend = None - if backend is not None: - try: - backend.stop() - except Exception: - pass + supervisor = self._acp_supervisor + self._acp_supervisor = None + self._stop_acp_supervisor(supervisor) self._server = None if server is not None: try: @@ -690,11 +603,9 @@ def stop(self) -> None: with self._stop_lock: self.stop_event.set() server = self._server - backend = self._event_backend - scheduler = self._turn_scheduler + supervisor = self._acp_supervisor self._server = None - self._event_backend = None - self._turn_scheduler = None + self._acp_supervisor = None if server is not None: try: @@ -702,34 +613,122 @@ def stop(self) -> None: except Exception: pass - if backend is not None: - flush = getattr(backend, "flush", None) - if callable(flush): - try: - flush() - except Exception: - pass - callback_setter = getattr(backend, "set_turn_refresh_callback", None) - if callable(callback_setter): - try: - callback_setter(None) - except Exception: - pass + self._stop_acp_supervisor(supervisor) - if scheduler is not None: + def _start_acp_supervisor(self) -> None: + """Start the required ACP supervisor and fail the daemon closed.""" + self._acp_startup_failure_type = None + + factory = self.hooks.acp_supervisor_factory + if factory is None: + raise RuntimeError("ACP supervisor is required but unavailable") + + supervisor: Any | None = None + try: + supervisor = factory(self.config, self.stop_event) + if supervisor is None: + raise RuntimeError("ACP supervisor is required but unavailable") + self._acp_supervisor = supervisor + supervisor.start() + health = self._acp_supervisor_health() + if health["healthy"] is not True: + failure_type = health.get("failure_type") + self._acp_startup_failure_type = _public_failure_type(failure_type) + raise RuntimeError("ACP supervisor did not become healthy") + except Exception as exc: + self._acp_startup_failure_type = ( + self._acp_startup_failure_type or type(exc).__name__ + ) + if supervisor is not None: + self._stop_acp_supervisor(supervisor) + self._acp_supervisor = None + raise RuntimeError( + "ACP supervisor is required but failed to start " + f"({self._acp_startup_failure_type})" + ) from None + + def _stop_acp_supervisor(self, supervisor: Any | None) -> None: + """Best-effort bounded shutdown for the ACP supervisor.""" + if supervisor is None: + return + timeout = self.config.acp_shutdown_timeout_seconds + stop = getattr(supervisor, "stop", None) + if callable(stop): try: - scheduler.stop( - flush_timeout_seconds=self.config.herdr_timeout_seconds + 1.0 - ) + stop(timeout=timeout) except Exception: pass - - if backend is not None: + join = getattr(supervisor, "join", None) + if callable(join): try: - backend.stop() + join(timeout=timeout) except Exception: pass + def _acp_supervisor_health(self) -> dict[str, Any]: + """Return a fixed, public-safe ACP lifecycle aggregate.""" + counters = { + "updates_ingested": 0, + "permissions_ingested": 0, + "permissions_selected": 0, + "permissions_cancelled": 0, + "invalid_permission_selections": 0, + "prompts_started": 0, + "prompts_completed": 0, + "prompts_failed": 0, + "cancellation_requests": 0, + } + supervisor = self._acp_supervisor + if supervisor is None: + return { + "required": True, + "status": "unavailable", + "healthy": False, + "state": "unavailable", + "failure_type": self._acp_startup_failure_type, + "counters": counters, + } + + status_method = getattr(supervisor, "status", None) + try: + raw = status_method() if callable(status_method) else None + except Exception as exc: + return { + "required": True, + "status": "degraded", + "healthy": False, + "state": "failed", + "failure_type": type(exc).__name__, + "counters": counters, + } + + def field(name: str) -> Any: + if isinstance(raw, Mapping): + return raw.get(name) + return getattr(raw, name, None) + + state_value = field("state") + state = getattr(state_value, "value", state_value) + if state not in {"new", "starting", "running", "stopping", "stopped", "failed"}: + state = "unknown" + healthy = field("healthy") is True and state == "running" + for key in counters: + counters[key] = _nonnegative_int(field(key)) + failure_type_value = field("failure_type") + failure_type = _public_failure_type(failure_type_value) + return { + "required": True, + "status": "healthy" if healthy else "degraded", + "healthy": healthy, + "state": state, + "failure_type": failure_type, + "last_reconcile_at": _valid_observation_timestamp( + field("last_reconcile_at") + ), + "worker_count": _nonnegative_int(field("worker_count")), + "counters": counters, + } + def _after_snapshot_saved(self) -> None: if self.config.db_path is None: return @@ -748,7 +747,8 @@ def _after_snapshot_saved(self) -> None: result = maybe_run_automatic_store_maintenance( Path(self.config.db_path), policy=policy, - turn_model=self.config.turn_model, + agent_event_host_id=self.config.host_id, + agent_event_retention_days=self.config.event_retention_days, acknowledged_final_retention_days=( self.config.acknowledged_final_retention_days ), @@ -777,6 +777,12 @@ def _after_snapshot_saved(self) -> None: ) snapshot_result = result.get("snapshot") snapshot_counts = snapshot_result if isinstance(snapshot_result, Mapping) else {} + agent_event_result = result.get("agent_events") + agent_event_counts = ( + agent_event_result + if isinstance(agent_event_result, Mapping) + else {} + ) maintenance_status = { "ok": bool(result.get("ok")) and bool(turn_change_result.get("ok")), "status": ( @@ -788,6 +794,15 @@ def _after_snapshot_saved(self) -> None: "examined": int(snapshot_counts.get("examined") or 0), "deleted": int(snapshot_counts.get("deleted") or 0), "remaining_candidates": bool(snapshot_counts.get("remaining_candidates")), + "agent_events_examined": int( + agent_event_counts.get("examined") or 0 + ), + "agent_events_deleted": int( + agent_event_counts.get("deleted") or 0 + ), + "agent_events_remaining_candidates": bool( + agent_event_counts.get("remaining_candidates") + ), } except Exception: self._automatic_maintenance_status = { @@ -797,48 +812,13 @@ def _after_snapshot_saved(self) -> None: "examined": 0, "deleted": 0, "remaining_candidates": False, + "agent_events_examined": 0, + "agent_events_deleted": 0, + "agent_events_remaining_candidates": False, } else: self._automatic_maintenance_status = maintenance_status - def _start_socket_event_backend(self) -> Snapshot: - if self.hooks.event_backend_factory is None: - from .backends.herdr_events import HerdrEventBackend - - backend = HerdrEventBackend(self.config, stop_event=self.stop_event) - else: - backend = self.hooks.event_backend_factory(self.config, self.stop_event) - self._event_backend = backend - backend.start(wait_for_reconcile=True) - from .store.sqlite import SnapshotObservationContext, latest_snapshot, save_snapshot - - snapshot = latest_snapshot(Path(self.config.db_path), self.config.host_id) - if snapshot is not None: - return snapshot - from .backends.herdr_cli import herdr_backend_health - from .core.projector import project_from_observations - - backend_health = ( - backend.health.to_backend_health() - if hasattr(backend, "health") - else herdr_backend_health("unknown") - ) - snapshot = project_from_observations( - self.config, - backend_health=[backend_health], - ) - save_snapshot( - Path(self.config.db_path), - snapshot, - turn_model=self.config.turn_model, - observation=SnapshotObservationContext( - authority="none", - observed_at=_valid_observation_timestamp(backend_health.observed_at), - ), - ) - self._after_snapshot_saved() - return snapshot - def get_snapshot(self) -> Snapshot: if self.config.db_path is not None: from .store.sqlite import latest_snapshot @@ -955,11 +935,15 @@ def get_health(self) -> dict[str, Any]: and command_requests_valid and maintenance_valid ) - backend_runtime: dict[str, Any] = {} - if self._event_backend is not None and hasattr(self._event_backend, "operational_status"): - status_value = getattr(self._event_backend, "operational_status") - if isinstance(status_value, Mapping): - backend_runtime = dict(status_value) + acp_health = self._acp_supervisor_health() + backend_runtime: dict[str, Any] = { + "status": "healthy" if acp_health["healthy"] else "degraded", + "outcome": "healthy_non_empty" if acp_health.get("worker_count", 0) else "empty_healthy", + "ready": acp_health["healthy"], + "running": acp_health.get("state") == "running", + "last_reconcile_at": acp_health.get("last_reconcile_at"), + "reconcile_enabled": True, + } backend_maintenance = backend_runtime.get("automatic_maintenance") runtime_maintenance = ( backend_maintenance @@ -1006,10 +990,10 @@ def get_health(self) -> dict[str, Any]: if store_ok and not maintenance_degraded and pending_ingestion["status"] == "healthy" + and acp_health["healthy"] is True else "degraded" ), "host_id": self.config.host_id, - "turn_model": self.config.turn_model, "daemon": { "status": "healthy", "started_at": self.started_at, @@ -1044,21 +1028,13 @@ def get_health(self) -> dict[str, Any]: "outcome": backend_runtime.get("outcome"), "ready": backend_runtime.get("ready"), "running": backend_runtime.get("running"), - "reconcile_enabled": backend_runtime.get( - "reconcile_enabled", - self.config.reconcile_interval_seconds > 0, - ), + "reconcile_enabled": backend_runtime["reconcile_enabled"], }, - "turn_ingestion": _turn_ingestion_health( - self.config, - self._turn_scheduler, - ), + "acp": acp_health, "pending_ingestion": pending_ingestion, "limits": { - "event_debounce_seconds": self.config.event_debounce_seconds, "reconcile_interval_seconds": self.config.reconcile_interval_seconds, "event_retention_days": self.config.event_retention_days, - "output_excerpt_chars": self.config.output_excerpt_chars, "max_workers": self.config.max_workers, "max_outbox_attempts": self.config.max_outbox_attempts, "outbox_claim_ttl_seconds": self.config.connector_claim_ttl_seconds, @@ -1138,8 +1114,6 @@ def get_turns( limit=limit, cursor=cursor, since=since, - turn_refresh_interval_seconds=self.config.turn_refresh_interval_seconds, - turn_model=self.config.turn_model, ) def get_turn_content(self, params: Mapping[str, Any]) -> Mapping[str, Any]: @@ -1164,7 +1138,6 @@ def get_turn_content(self, params: Mapping[str, Any]) -> Mapping[str, Any]: field=params.get("field"), cursor=params.get("cursor"), schema_version=params.get("schema_version", 1), - turn_model=self.config.turn_model, ) def get_turn_delta( @@ -1191,7 +1164,6 @@ def get_turn_delta( watermark=watermark, cursor=cursor, limit=limit, - turn_model=self.config.turn_model, ) def connector_call(self, method: str, params: Mapping[str, Any]) -> Mapping[str, Any]: @@ -1216,7 +1188,6 @@ def connector_call(self, method: str, params: Mapping[str, Any]) -> Mapping[str, max_lease_seconds=self.config.connector_max_claim_ttl_seconds, ack_ttl_seconds=self.config.connector_ack_ttl_seconds, max_attempts=self.config.max_outbox_attempts, - turn_model=self.config.turn_model, ).dispatch(method, params) def _connector_periodic_tick(self) -> None: @@ -1254,7 +1225,21 @@ def submit_command(self, params: Mapping[str, Any]) -> CommandEnvelope | Mapping sort_keys=True, separators=(",", ":"), ) - return self.hooks.submit_command(self.config, payload) + supervisor = self._acp_supervisor + route = getattr(supervisor, "prompt_route", None) + permission_router = ( + supervisor + if callable(getattr(supervisor, "answer_permission_decision", None)) + else None + ) + from .command_submission import submit_command + + return submit_command( + self.config, + payload, + acp_prompt_router=route if callable(route) else None, + acp_permission_router=permission_router, + ) def run_daemon( diff --git a/src/tendwire/daemon_api.py b/src/tendwire/daemon_api.py index 7096e07..123b4f5 100644 --- a/src/tendwire/daemon_api.py +++ b/src/tendwire/daemon_api.py @@ -727,6 +727,13 @@ def _restore_content_page_text( result = response.get("result") if isinstance(result, dict): result["text"] = text + content_revision = original_result.get("content_revision") + if ( + isinstance(content_revision, str) + and re.fullmatch(r"twrev1\.[A-Za-z0-9_-]+", content_revision) + is not None + ): + result["content_revision"] = content_revision def _restore_turn_delta_text( @@ -784,6 +791,13 @@ def restore(target: dict[str, Any], original: Mapping[str, Any]) -> None: is not None ): target["final_identity"] = final_identity + content_revision = original.get("content_revision") + if ( + isinstance(content_revision, str) + and re.fullmatch(r"twrev1\.[A-Za-z0-9_-]+", content_revision) + is not None + ): + target["content_revision"] = content_revision delivery_key = original.get("key") if ( isinstance(delivery_key, str) @@ -794,7 +808,7 @@ def restore(target: dict[str, Any], original: Mapping[str, Any]) -> None: is not None ): target["key"] = delivery_key - for nested_key in ("turn", "final", "payload"): + for nested_key in ("turn", "final", "payload", "content"): nested_original = original.get(nested_key) nested_target = target.get(nested_key) if isinstance(nested_original, Mapping) and isinstance(nested_target, dict): diff --git a/src/tendwire/store/sqlite.py b/src/tendwire/store/sqlite.py index af45530..8008358 100644 --- a/src/tendwire/store/sqlite.py +++ b/src/tendwire/store/sqlite.py @@ -31,9 +31,6 @@ DEFAULT_PENDING_STALE_GRACE_SECONDS, DEFAULT_SUBMISSION_HARD_TTL_SECONDS, DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS, - DEFAULT_TURN_MODEL, - DEFAULT_TURN_REFRESH_INTERVAL_SECONDS, - TURN_MODELS, ) from ..local_state import ( EntryIdentity, @@ -63,6 +60,18 @@ verify_created_private_sqlite_replacement_at, verify_entry_identity, ) +from ..core.agent_events import ( + AGENT_EVENT_KINDS, + AGENT_EVENT_QUERY_DEFAULT_LIMIT, + AGENT_EVENT_QUERY_MAX_LIMIT, + AgentEvent, + AgentEventIdentityConflict, + AppendAgentEventResult, + AppendBoundAgentEventResult, + StoredAgentEvent, + agent_event, + normalize_agent_event_identifier, +) from ..core.commands import ( CommandEnvelope, instruction_fingerprint, @@ -86,7 +95,6 @@ utc_timestamp, ) from ..core.turns import ( - InteractionChoice, PendingInteraction, PendingObservation, PendingObservedChoice, @@ -133,21 +141,17 @@ FINGERPRINT_HEX_LENGTH = FINGERPRINT_HEX_CHARS -STORE_SCHEMA_VERSION = 21 +STORE_SCHEMA_VERSION = 29 CONNECTOR_ACK_TTL_SECONDS = DEFAULT_CONNECTOR_ACK_TTL_SECONDS -_LEGACY_TURN_CLAIM_HARD_TTL_SECONDS = 86_400 TURN_CHANGE_RETENTION_DAYS = 7 TURN_CHANGE_RETENTION_COUNT = 100_000 TURN_CHANGE_COMPACTION_BATCH_SIZE = 1_000 -HERDR_TURN_RETENTION_DAYS = 30 -HERDR_TURN_RETENTION_COUNT = 4096 -HERDR_TURN_RETENTION_BATCH_SIZE = 100 TURN_SUBMISSION_OBSERVATION_ADOPTION_WINDOW_SECONDS = 60.0 SUBMISSION_LINK_WINDOW_SECONDS = DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS SUBMISSION_HARD_TTL_SECONDS = DEFAULT_SUBMISSION_HARD_TTL_SECONDS # A send that has not produced an authoritative accepted/uncertain receipt # within the backend command timeout is no longer safe for instant linking. -# Keep this classification local to the shadow linker: receipts and the +# Keep this classification local to the observation linker: receipts and the # submission lifecycle remain authoritative and unchanged. SUBMISSION_SEND_ACK_TIMEOUT_SECONDS = 5.0 TURN_LEDGER_BACKFILL_BATCH_SIZE = 500 @@ -194,6 +198,7 @@ _LOGGER = logging.getLogger(__name__) _SUBMISSION_LINK_SWEEP_LAST_AT: dict[tuple[str, str, str], float] = {} _SUBMISSION_LINK_SWEEP_LOCK = threading.Lock() +DEFAULT_SUBMISSION_LINK_SWEEP_INTERVAL_SECONDS = 2.0 _SUBMISSION_LINK_BACKOFF: dict[ tuple[str, str, str, str], datetime | None ] = {} @@ -208,14 +213,6 @@ _SUBMISSION_LINK_EMPTY_COMPONENT_RECHECK_SECONDS = 30.0 -def _submission_linking_enabled(turn_model: str) -> bool: - normalized = str(turn_model or "").strip().lower() - if normalized not in TURN_MODELS: - allowed = ", ".join(sorted(TURN_MODELS)) - raise ValueError(f"turn_model must be one of: {allowed}") - return True - - def _submission_link_backoff_key( db_path: Path | str, host_id: str, @@ -341,16 +338,6 @@ def is_valid_turn_submission_state_transition( ) -@dataclass(frozen=True) -class Migration: - """One exact, transaction-external schema transition.""" - - from_version: int - to_version: int - apply: Callable[[sqlite3.Connection], None] - - - @dataclass(frozen=True) class BackendPendingChoiceClaim: status: Literal[ @@ -396,6 +383,7 @@ class BackendPendingDecisionClaim: "decision_not_pending", "invalid_selection", "unsupported_decision", + "acp_authority_unavailable", "already_claimed", ] claim_token: str | None = None @@ -488,22 +476,15 @@ class TurnRefreshApplyResult: @dataclass(frozen=True) -class HerdrTurnWatermark: - """Durable replay position and retained completeness-break evidence.""" - - host_id: str - pane_id: str - turn_epoch: int - last_turn: int - completeness_break_count: int - last_completeness_break_reason: str | None - last_completeness_break_at: str | None - updated_at: str +class AppendProjectedAgentEventResult: + """One binding-fenced journal append and optional turn projection outcome.""" + event: AppendBoundAgentEventResult + turn: TurnRefreshApplyResult | None = None @dataclass(frozen=True) class _TurnContentMergeResult: - """Observation merge outcome and optional shadow-link settlement key.""" + """Observation merge outcome and optional submission settlement key.""" updated: int submission_link: tuple[str, str] | None = None @@ -581,14 +562,6 @@ def _record_response_size( ); """ -CREATE_LEGACY_SNAPSHOT_INDEXES = ( - "CREATE INDEX IF NOT EXISTS idx_snapshots_host_id ON snapshots(host_id)", - "CREATE INDEX IF NOT EXISTS idx_snapshots_created_at ON snapshots(created_at)", - ( - "CREATE INDEX IF NOT EXISTS idx_snapshots_content_fingerprint " - "ON snapshots(content_fingerprint)" - ), -) CREATE_SNAPSHOT_INDEXES = ( ( @@ -603,9 +576,6 @@ def _record_response_size( # Leading "~" sorts after every canonical timestamp under SQLite BINARY # collation, so raw indexed age comparisons never delete unknown history. _SNAPSHOT_CREATED_AT_QUARANTINE = "~invalid-snapshot-created-at" -_LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE = ( - "9999-12-31T23:59:59.999999+00:00" -) CREATE_STORE_MAINTENANCE_STATE_TABLE = """ CREATE TABLE IF NOT EXISTS store_maintenance_state ( @@ -635,20 +605,6 @@ def _record_response_size( ON CONFLICT(scope) DO NOTHING """ -CREATE_LEGACY_COMMAND_RECEIPTS_TABLE = """ -CREATE TABLE IF NOT EXISTS command_receipts ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - request_id TEXT NOT NULL, - action TEXT NOT NULL, - payload_fingerprint TEXT NOT NULL, - status TEXT NOT NULL, - result_json TEXT NOT NULL, - created_at TEXT NOT NULL, - completed_at TEXT, - uncertain INTEGER NOT NULL DEFAULT 0 -); -""" CREATE_COMMAND_RECEIPTS_TABLE = """ CREATE TABLE IF NOT EXISTS command_receipts ( @@ -673,13 +629,8 @@ def _record_response_size( send_started_at TEXT, terminal_at TEXT, updated_at TEXT NOT NULL, - legacy_collision INTEGER NOT NULL DEFAULT 0 CHECK (legacy_collision IN (0, 1)), - legacy_collision_count INTEGER NOT NULL DEFAULT 0 CHECK ( - legacy_collision_count >= 0 - ), -- Private evidence of the immutable selector this request was spelled with. - -- Empty means legacy evidence that cannot prove an alias retry. Declared - -- last so a v12 ALTER and a fresh CREATE agree on column order. + -- Empty is malformed fail-safe evidence that cannot prove an alias retry. selector_proof TEXT NOT NULL DEFAULT '', CHECK ( ( @@ -703,22 +654,10 @@ def _record_response_size( CHECK ( state != 'rejected' OR status NOT IN ('pending', 'accepted', 'request_state_uncertain') - ), - CHECK ( - legacy_collision = 0 - OR (state = 'uncertain' AND legacy_collision_count >= 2) ) ); """ -CREATE_LEGACY_COMMAND_RECEIPT_INDEXES = ( - "CREATE INDEX IF NOT EXISTS idx_command_receipts_host_request_action " - "ON command_receipts(host_id, request_id, action)", -) -CREATE_LEGACY_COMMAND_RECEIPT_UNIQUE_INDEX = ( - "CREATE UNIQUE INDEX IF NOT EXISTS ux_command_receipts_host_request_action " - "ON command_receipts(host_id, request_id, action)" -) CREATE_COMMAND_RECEIPT_INDEXES = ( "CREATE UNIQUE INDEX IF NOT EXISTS ux_command_receipts_host_request " "ON command_receipts(host_id, request_id)", @@ -1334,24 +1273,6 @@ def _record_response_size( ), ) -CREATE_LEGACY_COMMANDS_TABLE = """ -CREATE TABLE IF NOT EXISTS commands ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - request_id TEXT NOT NULL, - action TEXT NOT NULL, - payload_fingerprint TEXT NOT NULL, - status TEXT NOT NULL, - dry_run INTEGER NOT NULL DEFAULT 0, - uncertain INTEGER NOT NULL DEFAULT 0, - request_json TEXT NOT NULL DEFAULT '{}', - result_json TEXT NOT NULL DEFAULT '{}', - created_at TEXT NOT NULL, - reserved_at TEXT, - completed_at TEXT, - updated_at TEXT NOT NULL -); -""" CREATE_COMMANDS_TABLE = """ CREATE TABLE IF NOT EXISTS commands ( @@ -1373,10 +1294,6 @@ def _record_response_size( send_started_at TEXT, terminal_at TEXT, updated_at TEXT NOT NULL, - legacy_collision INTEGER NOT NULL DEFAULT 0 CHECK (legacy_collision IN (0, 1)), - legacy_collision_count INTEGER NOT NULL DEFAULT 0 CHECK ( - legacy_collision_count >= 0 - ), CHECK ( (state IN ('reserved', 'send_started') AND terminal_at IS NULL) OR (state IN ('accepted', 'rejected', 'uncertain') AND terminal_at IS NOT NULL) @@ -1390,10 +1307,6 @@ def _record_response_size( CHECK ( state != 'rejected' OR status NOT IN ('pending', 'accepted', 'request_state_uncertain') - ), - CHECK ( - legacy_collision = 0 - OR (state = 'uncertain' AND legacy_collision_count >= 2) ) ); """ @@ -1448,12 +1361,22 @@ def _record_response_size( """ -CREATE_LEGACY_BACKEND_PENDING_TABLE = """ +CREATE_BACKEND_PENDING_TABLE = """ CREATE TABLE IF NOT EXISTS backend_pending ( host_id TEXT NOT NULL, worker_id TEXT NOT NULL, payload_json TEXT NOT NULL, observed_at TEXT NOT NULL, + revision_digest TEXT NOT NULL DEFAULT '', + choice_routes_json TEXT NOT NULL DEFAULT '{}', + binding_private_fingerprint TEXT NOT NULL DEFAULT '', + observed_turn_target_value TEXT NOT NULL DEFAULT '', + observation_state TEXT NOT NULL DEFAULT 'open', + freshness TEXT NOT NULL DEFAULT 'fresh', + last_success_at TEXT, + last_failure_at TEXT, + grace_deadline TEXT, + updated_at TEXT NOT NULL DEFAULT '', PRIMARY KEY (host_id, worker_id) ); """ @@ -1477,59 +1400,144 @@ def _record_response_size( ); """ -CREATE_HERDR_TURN_WATERMARKS_TABLE = """ -CREATE TABLE IF NOT EXISTS herdr_turn_watermarks ( - host_id TEXT NOT NULL, - pane_id TEXT NOT NULL, - turn_epoch INTEGER NOT NULL CHECK (turn_epoch >= 0), - last_turn INTEGER NOT NULL CHECK (last_turn >= 0), - completeness_break_count INTEGER NOT NULL DEFAULT 0 - CHECK (completeness_break_count >= 0), - last_completeness_break_reason TEXT, - last_completeness_break_at TEXT, - updated_at TEXT NOT NULL, - PRIMARY KEY (host_id, pane_id) -); -""" -CREATE_HERDR_TURN_COMPLETIONS_TABLE = """ -CREATE TABLE IF NOT EXISTS herdr_turn_completions ( + + + + +CREATE_AGENT_EVENTS_TABLE = """ +CREATE TABLE IF NOT EXISTS agent_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, host_id TEXT NOT NULL, - pane_id TEXT NOT NULL, - turn_epoch INTEGER NOT NULL CHECK (turn_epoch >= 0), - turn INTEGER NOT NULL CHECK (turn >= 0), - outcome TEXT NOT NULL CHECK (outcome IN ('completed', 'aborted')), - completed_unix_ms INTEGER NOT NULL CHECK (completed_unix_ms >= 0), - message TEXT, - message_truncated INTEGER NOT NULL DEFAULT 0 - CHECK (message_truncated IN (0, 1)), - agent_session_path TEXT, - worker_id TEXT, - refreshed_turn_id TEXT, + event_id TEXT NOT NULL, + kind TEXT NOT NULL CHECK ( + kind IN ( + 'user_message', 'agent_message', 'thought', 'tool_call', + 'tool_call_update', 'plan', 'usage', 'session_info', 'extension' + ) + ), + source TEXT NOT NULL, + worker_id TEXT NOT NULL, + visibility TEXT NOT NULL CHECK (visibility IN ('private', 'public')), + source_session_id TEXT, + source_turn_id TEXT, + source_item_id TEXT, + source_message_id TEXT, + source_event_id TEXT, + source_sequence INTEGER CHECK (source_sequence >= 0), observed_at TEXT NOT NULL, - PRIMARY KEY (host_id, pane_id, turn_epoch, turn) + payload_fingerprint TEXT NOT NULL, + private_payload_json TEXT NOT NULL, + public_payload_json TEXT NOT NULL, + UNIQUE (host_id, event_id), + CHECK (length(host_id) BETWEEN 1 AND 2048), + CHECK (instr(host_id, char(0)) = 0), + CHECK (length(event_id) = 64), + CHECK (length(source) BETWEEN 1 AND 2048), + CHECK (instr(source, char(0)) = 0), + CHECK (length(worker_id) BETWEEN 1 AND 2048), + CHECK (instr(worker_id, char(0)) = 0), + CHECK ( + source_session_id IS NULL OR ( + length(source_session_id) BETWEEN 1 AND 2048 + AND instr(source_session_id, char(0)) = 0 + ) + ), + CHECK ( + source_turn_id IS NULL OR ( + length(source_turn_id) BETWEEN 1 AND 2048 + AND instr(source_turn_id, char(0)) = 0 + ) + ), + CHECK ( + source_item_id IS NULL OR ( + length(source_item_id) BETWEEN 1 AND 2048 + AND instr(source_item_id, char(0)) = 0 + ) + ), + CHECK ( + source_message_id IS NULL OR ( + length(source_message_id) BETWEEN 1 AND 2048 + AND instr(source_message_id, char(0)) = 0 + ) + ), + CHECK ( + source_event_id IS NULL OR ( + length(source_event_id) BETWEEN 1 AND 2048 + AND instr(source_event_id, char(0)) = 0 + ) + ), + CHECK (length(observed_at) BETWEEN 20 AND 40), + CHECK (length(payload_fingerprint) = 64), + CHECK ( + CASE WHEN json_valid(private_payload_json) + THEN json_type(private_payload_json) = 'object' ELSE 0 END + ), + CHECK ( + length(CAST(private_payload_json AS BLOB)) <= 16777216 + ), + CHECK ( + CASE WHEN json_valid(public_payload_json) + THEN json_type(public_payload_json) = 'object' ELSE 0 END + ), + CHECK ( + length(CAST(public_payload_json AS BLOB)) <= 65536 + ), + CHECK (visibility != 'private' OR public_payload_json = '{}'), + CHECK (source_event_id IS NOT NULL OR source_sequence IS NOT NULL), + CHECK ( + source_event_id IS NOT NULL + OR (source_session_id IS NOT NULL AND source_sequence IS NOT NULL) + ), + CHECK ( + kind NOT IN ( + 'thought', 'tool_call', 'tool_call_update', 'plan', 'extension' + ) OR visibility = 'private' + ) ); """ -CREATE_HERDR_TURN_INDEXES = ( +CREATE_AGENT_EVENT_INDEXES = ( + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_worker_sequence " + "ON agent_events(host_id, worker_id, sequence)" + ), + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_session_sequence " + "ON agent_events(host_id, source_session_id, sequence)" + ), + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_turn_sequence " + "ON agent_events(host_id, source_turn_id, sequence)" + ), + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_source_sequence " + "ON agent_events(host_id, source, sequence)" + ), + ( + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_visibility_sequence " + "ON agent_events(host_id, visibility, sequence)" + ), ( - "CREATE INDEX IF NOT EXISTS idx_herdr_turn_completions_worker " - "ON herdr_turn_completions(host_id, worker_id, turn_epoch, turn)" + "CREATE INDEX IF NOT EXISTS idx_agent_events_host_observed_sequence " + "ON agent_events(host_id, observed_at, sequence)" + ), + ( + "CREATE UNIQUE INDEX IF NOT EXISTS " + "idx_agent_events_source_event_identity " + "ON agent_events(" + "host_id, source, COALESCE(source_session_id, ''), source_event_id" + ") WHERE source_event_id IS NOT NULL" + ), + ( + "CREATE UNIQUE INDEX IF NOT EXISTS " + "idx_agent_events_source_sequence_identity " + "ON agent_events(host_id, source, source_session_id, source_sequence) " + "WHERE source_event_id IS NULL" ), ) -CREATE_PR6_TABLES = ( - CREATE_EVENTS_TABLE, - CREATE_SPACES_TABLE, - CREATE_WORKERS_TABLE, - CREATE_TURNS_TABLE, - CREATE_PENDING_INTERACTIONS_TABLE, - CREATE_ATTENTION_ITEMS_TABLE, - CREATE_LEGACY_COMMANDS_TABLE, - CREATE_CONNECTOR_OUTBOX_TABLE, - CREATE_CONNECTOR_DELIVERIES_TABLE, - CREATE_BACKEND_HEALTH_TABLE, -) + CREATE_CURRENT_PR6_TABLES = ( CREATE_EVENTS_TABLE, CREATE_SPACES_TABLE, @@ -1563,7 +1571,6 @@ def _record_response_size( "CREATE INDEX IF NOT EXISTS idx_pending_interactions_host_status " "ON pending_interactions(host_id, status)" ), - CREATE_LEGACY_BACKEND_PENDING_TABLE, ( "CREATE INDEX IF NOT EXISTS idx_attention_items_host_source " "ON attention_items(host_id, source)" @@ -1584,11 +1591,6 @@ def _record_response_size( "CREATE INDEX IF NOT EXISTS idx_attention_items_host_fingerprint " "ON attention_items(host_id, fingerprint)" ), - ( - "CREATE UNIQUE INDEX IF NOT EXISTS ux_commands_host_request_action " - "ON commands(host_id, request_id, action)" - ), - "CREATE INDEX IF NOT EXISTS idx_commands_host_status ON commands(host_id, status)", ( "CREATE UNIQUE INDEX IF NOT EXISTS ux_connector_outbox_host_connector_key " "ON connector_outbox(host_id, connector, delivery_key)" @@ -1611,7 +1613,7 @@ def _record_response_size( "ON commands(host_id, state, updated_at, id)", ) CREATE_CURRENT_PR6_INDEXES = ( - CREATE_PR6_INDEXES[:16] + CREATE_PR6_INDEXES[18:] + CREATE_COMMAND_INDEXES + CREATE_PR6_INDEXES + CREATE_COMMAND_INDEXES ) @@ -2713,8 +2715,8 @@ def _connector_reclaim_expired_awaiting_ack_conn( if delivery is not None else None ) - # A legacy/noncanonical awaiting_ack row without a deadline is already - # overdue. Every non-terminal state must be swept rather than wedged. + # A malformed awaiting_ack row without a deadline is already overdue. + # Every non-terminal state must be swept rather than wedged. if not _connector_deadline_due( deadline, next_attempt_at, @@ -3326,7 +3328,6 @@ def _final_ready_payload_conn( turn_id: str, content_revision_value: str, allow_unroutable: bool = False, - turn_model: str = DEFAULT_TURN_MODEL, ) -> dict[str, Any] | None: canonical_turn_id = _resolve_canonical_turn_id_conn( conn, @@ -4164,12 +4165,17 @@ def _restore_presentation_tokens( final_identity = original.get("final_identity") if _valid_presentation_opaque(final_identity, "twfinal1."): sanitized["final_identity"] = str(final_identity) + content_revision_value = original.get("content_revision") + if _valid_presentation_opaque(content_revision_value, "twrev1."): + sanitized["content_revision"] = str(content_revision_value) turn_id = original.get("turn_id") if _valid_presentation_label(turn_id, prefix="turn-"): sanitized["turn_id"] = str(turn_id) - nested_turn = original.get("turn") - if isinstance(nested_turn, Mapping): - clean_nested = sanitized.get("turn") + for nested_key in ("turn", "content"): + nested_turn = original.get(nested_key) + if not isinstance(nested_turn, Mapping): + continue + clean_nested = sanitized.get(nested_key) if not isinstance(clean_nested, dict): clean_nested = dict( sanitize_public_mapping( @@ -4177,7 +4183,7 @@ def _restore_presentation_tokens( backend_neutral=True, ) ) - sanitized["turn"] = clean_nested + sanitized[nested_key] = clean_nested _restore_presentation_tokens(clean_nested, nested_turn) return sanitized @@ -4297,7 +4303,6 @@ def prepare_connector_plan_begin( presentation_version: str, part_count: int, source_ref: str | None = None, - turn_model: str = DEFAULT_TURN_MODEL, now: str | None = None, ) -> dict[str, Any]: """Idempotently begin one bounded range-only presentation plan.""" @@ -5367,12 +5372,27 @@ def _mark_exhausted_presentation_plans_conn( SET state = 'failed' WHERE plans.host_id = ? {connector_clause} - AND plans.state IN ('active', 'waiting_predecessor') - AND EXISTS ( - SELECT 1 - FROM turn_presentation_jobs AS jobs - JOIN connector_outbox AS outbox ON outbox.id = jobs.outbox_id - WHERE jobs.plan_id = plans.id AND outbox.status = 'dead_letter' + AND ( + ( + plans.state IN ('active', 'waiting_predecessor') + AND EXISTS ( + SELECT 1 + FROM turn_presentation_jobs AS jobs + JOIN connector_outbox AS outbox + ON outbox.id = jobs.outbox_id + WHERE jobs.plan_id = plans.id + AND outbox.status = 'dead_letter' + ) + ) + OR ( + plans.state = 'preparing' + AND EXISTS ( + SELECT 1 + FROM connector_outbox AS source + WHERE source.id = plans.source_outbox_id + AND source.status = 'dead_letter' + ) + ) ) """, params, @@ -5524,6 +5544,25 @@ def prepare_connector_plan_commit( "generation": int(plan[7]), } ) + if plan_state in {"failed", "superseded"}: + # A source final can exhaust before commit materializes any + # child jobs. Preserve terminal idempotency so a recovering + # connector can abandon that zero-job plan instead of retrying + # a now-stale source reference forever. + conn.commit() + return _presentation_response( + { + "schema_version": _PRESENTATION_SCHEMA_VERSION, + "ok": True, + "status": "ok", + "host_id": str(host_id), + "name": str(name), + "plan_token": str(plan_token), + "state": plan_state, + "job_count": 0, + "generation": int(plan[7]), + } + ) _, early_revision_error = _current_presentation_revision_conn( conn, host_id=str(host_id), @@ -6491,7 +6530,8 @@ def poll_connector_outbox( outbox.id, outbox.delivery_key, outbox.payload_json, - outbox.private_state_json + outbox.private_state_json, + outbox.created_at FROM connector_outbox AS outbox WHERE outbox.host_id = ? AND outbox.connector = ? @@ -6749,6 +6789,7 @@ def poll_connector_outbox( "leased_until": lease_expires_at, "ref": public_ref, "available_at": current_time, + "created_at": str(row[4] or ""), "payload": _restore_presentation_tokens( sanitize_public_mapping( _json_object(row[2]), @@ -7142,7 +7183,7 @@ def _connector_update_ref( WHERE id = ? AND status = 'leased' """, ( - _migration_private_state( + _connector_group_private_state( sibling_private, group=migration_group, canonical=False, @@ -7285,6 +7326,15 @@ def _connector_update_ref( outbox_status=outbox_status, now=current_time, ) + if outbox_status == _CONNECTOR_EXHAUSTED_OUTBOX_STATUS: + # The exhausted row can be either a materialized plan job or + # the final-ready source anchor of a still-preparing plan. + _mark_exhausted_presentation_plans_conn( + conn, + host_id=str(host_id), + name=str(name), + now=current_time, + ) conn.commit() return _connector_response( ok=True, @@ -8231,9 +8281,7 @@ def _command_receipt_from_row(row: Any) -> dict[str, Any]: "send_started_at": row[16], "terminal_at": row[17], "updated_at": str(row[18]), - "legacy_collision": bool(row[19]), - "legacy_collision_count": int(row[20]), - "selector_proof": str(row[21]), + "selector_proof": str(row[19]), } @@ -8291,50 +8339,8 @@ def _worker_binding_from_row(row: Any) -> WorkerBinding: ) -def _dedupe_command_receipts(conn: sqlite3.Connection) -> None: - """Keep the latest legacy receipt per logical key using bounded batches.""" - while True: - delete_ids = [ - int(row[0]) - for row in conn.execute( - """ - WITH ranked AS ( - SELECT - id, - ROW_NUMBER() OVER ( - PARTITION BY host_id, request_id, action - ORDER BY - COALESCE(completed_at, created_at) DESC, - created_at DESC, - id DESC - ) AS receipt_rank - FROM command_receipts - ) - SELECT id - FROM ranked - WHERE receipt_rank > 1 - ORDER BY id - LIMIT 500 - """ - ).fetchall() - ] - if not delete_ids: - return - placeholders = ",".join("?" for _ in delete_ids) - conn.execute( - f"DELETE FROM command_receipts WHERE id IN ({placeholders})", - delete_ids, - ) -def _ensure_command_receipt_unique_index(conn: sqlite3.Connection) -> None: - for row in conn.execute("PRAGMA index_list(command_receipts)").fetchall(): - index_name = str(row[1]) - is_unique = int(row[2]) == 1 - if index_name == "ux_command_receipts_host_request_action" and not is_unique: - conn.execute("DROP INDEX ux_command_receipts_host_request_action") - break - conn.execute(CREATE_LEGACY_COMMAND_RECEIPT_UNIQUE_INDEX) def _command_request_row( @@ -8364,8 +8370,6 @@ def _command_request_row( send_started_at, terminal_at, updated_at, - legacy_collision, - legacy_collision_count, selector_proof FROM command_receipts WHERE host_id = ? AND request_id = ? @@ -8383,256 +8387,16 @@ def _snapshot_payload(data: Mapping[str, Any]) -> tuple[dict[str, Any], str]: return payload_data, fingerprint -def _table_columns(conn: sqlite3.Connection, table: str = "snapshots") -> set[str]: - rows = conn.execute(f"PRAGMA table_info({table})").fetchall() - return {str(row[1]) for row in rows} -def _ensure_columns( - conn: sqlite3.Connection, - table: str, - columns: Mapping[str, str], -) -> None: - existing = _table_columns(conn, table) - for column, definition in columns.items(): - if column not in existing: - conn.execute(f"ALTER TABLE {table} ADD COLUMN {column} {definition}") -def _backfill_content_fingerprints(conn: sqlite3.Connection) -> None: - rows = conn.execute( - """ - SELECT id, payload - FROM snapshots - WHERE content_fingerprint IS NULL OR content_fingerprint = '' - """ - ).fetchall() - for row_id, payload in rows: - try: - data = json.loads(payload) - except json.JSONDecodeError: - fingerprint = _content_fingerprint({"payload": payload}) - conn.execute( - "UPDATE snapshots SET content_fingerprint = ? WHERE id = ?", - (fingerprint, row_id), - ) - continue - if not isinstance(data, Mapping): - fingerprint = _content_fingerprint({"payload": data}) - conn.execute( - "UPDATE snapshots SET content_fingerprint = ? WHERE id = ?", - (fingerprint, row_id), - ) - continue - payload_data, fingerprint = _snapshot_payload( - Snapshot.from_dict(data).to_dict() - ) - conn.execute( - """ - UPDATE snapshots - SET content_fingerprint = ?, payload = ? - WHERE id = ? - """, - (fingerprint, _canonical_json(payload_data), row_id), - ) -def _ensure_command_receipt_columns(conn: sqlite3.Connection) -> None: - _ensure_columns( - conn, - "command_receipts", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "request_id": "TEXT NOT NULL DEFAULT ''", - "action": "TEXT NOT NULL DEFAULT ''", - "payload_fingerprint": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT ''", - "result_json": "TEXT NOT NULL DEFAULT '{}'", - "created_at": "TEXT NOT NULL DEFAULT ''", - "completed_at": "TEXT", - "uncertain": "INTEGER NOT NULL DEFAULT 0", - }, - ) -def _ensure_worker_binding_columns(conn: sqlite3.Connection) -> None: - _ensure_columns( - conn, - "worker_bindings", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "worker_id": "TEXT NOT NULL DEFAULT ''", - "worker_fingerprint": "TEXT NOT NULL DEFAULT ''", - "backend": "TEXT NOT NULL DEFAULT ''", - "target_kind": "TEXT NOT NULL DEFAULT ''", - "target_value": "TEXT NOT NULL DEFAULT ''", - "turn_target_kind": "TEXT", - "turn_target_value": "TEXT", - "sendable": "INTEGER NOT NULL DEFAULT 0", - "reason": "TEXT", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "expires_at": "TEXT NOT NULL DEFAULT '9999-12-31T23:59:59+00:00'", - "private_fingerprint": "TEXT NOT NULL DEFAULT ''", - }, - ) -def _ensure_pr6_columns(conn: sqlite3.Connection) -> None: - _ensure_columns( - conn, - "events", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "event_type": "TEXT NOT NULL DEFAULT ''", - "aggregate_type": "TEXT NOT NULL DEFAULT ''", - "aggregate_id": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "spaces", - { - "name": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "updated_at": "TEXT", - "fingerprint": "TEXT NOT NULL DEFAULT ''", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "workers", - { - "worker_fingerprint": "TEXT NOT NULL DEFAULT ''", - "space_id": "TEXT", - "name": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "last_seen_at": "TEXT", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "turns", - { - "worker_id": "TEXT NOT NULL DEFAULT ''", - "worker_fingerprint": "TEXT", - "space_id": "TEXT", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "kind": "TEXT NOT NULL DEFAULT 'unknown'", - "updated_at": "TEXT", - "fingerprint": "TEXT NOT NULL DEFAULT ''", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "pending_interactions", - { - "worker_id": "TEXT NOT NULL DEFAULT ''", - "worker_fingerprint": "TEXT", - "space_id": "TEXT", - "kind": "TEXT NOT NULL DEFAULT 'unknown'", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "updated_at": "TEXT", - "fingerprint": "TEXT NOT NULL DEFAULT ''", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "attention_items", - { - "source": "TEXT NOT NULL DEFAULT ''", - "kind": "TEXT NOT NULL DEFAULT 'unknown'", - "severity": "TEXT NOT NULL DEFAULT 'info'", - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "updated_at": "TEXT", - "fingerprint": "TEXT NOT NULL DEFAULT ''", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "observed_at": "TEXT NOT NULL DEFAULT ''", - "first_seen_at": "TEXT NOT NULL DEFAULT ''", - "last_seen_at": "TEXT NOT NULL DEFAULT ''", - "last_changed_at": "TEXT NOT NULL DEFAULT ''", - "resolved_at": "TEXT", - "lifecycle_status": "TEXT NOT NULL DEFAULT 'open'", - "resolved_reason": "TEXT", - "signal_count": "INTEGER NOT NULL DEFAULT 1", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) - _ensure_columns( - conn, - "commands", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "request_id": "TEXT NOT NULL DEFAULT ''", - "action": "TEXT NOT NULL DEFAULT ''", - "payload_fingerprint": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT ''", - "dry_run": "INTEGER NOT NULL DEFAULT 0", - "uncertain": "INTEGER NOT NULL DEFAULT 0", - "request_json": "TEXT NOT NULL DEFAULT '{}'", - "result_json": "TEXT NOT NULL DEFAULT '{}'", - "created_at": "TEXT NOT NULL DEFAULT ''", - "reserved_at": "TEXT", - "completed_at": "TEXT", - "updated_at": "TEXT NOT NULL DEFAULT ''", - }, - ) - _ensure_columns( - conn, - "connector_outbox", - { - "host_id": "TEXT NOT NULL DEFAULT ''", - "connector": "TEXT NOT NULL DEFAULT ''", - "delivery_key": "TEXT NOT NULL DEFAULT ''", - "status": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - "private_state_json": "TEXT NOT NULL DEFAULT '{}'", - "created_at": "TEXT NOT NULL DEFAULT ''", - "updated_at": "TEXT NOT NULL DEFAULT ''", - "next_attempt_at": "TEXT", - }, - ) - _ensure_columns( - conn, - "connector_deliveries", - { - "outbox_id": "INTEGER", - "host_id": "TEXT NOT NULL DEFAULT ''", - "connector": "TEXT NOT NULL DEFAULT ''", - "delivery_key": "TEXT NOT NULL DEFAULT ''", - "attempt": "INTEGER NOT NULL DEFAULT 0", - "status": "TEXT NOT NULL DEFAULT ''", - "response_json": "TEXT NOT NULL DEFAULT '{}'", - "private_state_json": "TEXT NOT NULL DEFAULT '{}'", - "created_at": "TEXT NOT NULL DEFAULT ''", - "delivered_at": "TEXT", - }, - ) - _ensure_columns( - conn, - "backend_health", - { - "status": "TEXT NOT NULL DEFAULT 'unknown'", - "outcome": "TEXT NOT NULL DEFAULT 'unknown'", - "observed_at": "TEXT", - "snapshot_content_fingerprint": "TEXT NOT NULL DEFAULT ''", - "payload_json": "TEXT NOT NULL DEFAULT '{}'", - }, - ) def _append_event_conn( @@ -8710,36 +8474,12 @@ def _repair_missing_final_ready_anchors_conn( ) -def append_event( - db_path: Path, - host_id: str, - event_type: str, - payload: Mapping[str, Any], - *, - aggregate_type: str = "", - aggregate_id: str = "", - observed_at: str | None = None, - content_fingerprint: str | None = None, -) -> int: - """Append a private store event and return its row id.""" - with _connect(db_path, prepare=True) as conn: - _ensure_schema(conn) - return _append_event_conn( - conn, - host_id=host_id, - event_type=event_type, - payload=payload, - aggregate_type=aggregate_type, - aggregate_id=aggregate_id, - observed_at=observed_at, - content_fingerprint=content_fingerprint, - ) - - -def _prune_host_projection( - conn: sqlite3.Connection, - table: str, - key_column: str, + + +def _prune_host_projection( + conn: sqlite3.Connection, + table: str, + key_column: str, host_id: str, keep_ids: Iterable[str], ) -> None: @@ -8942,22 +8682,6 @@ def _strict_utc_timestamp(value: Any) -> str | None: return None -def _legacy_snapshot_created_at_is_authoritative( - created_at: Any, - payload: Any, -) -> bool: - """Distinguish a real year-9999 observation from the former sentinel.""" - canonical_created_at = _strict_utc_timestamp(created_at) - canonical_payload_at = _strict_utc_timestamp( - _json_object(payload).get("updated_at") - ) - return ( - canonical_created_at - == _LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE - and canonical_payload_at == canonical_created_at - ) - - def _attention_family_key(host_id: str, item: Mapping[str, Any]) -> str: source = _store_public_text(item.get("source"), default="unknown") kind = _store_public_label(item.get("kind")) @@ -10080,10 +9804,8 @@ def _project_command_request_conn(conn: sqlite3.Connection, row: Any) -> None: reserved_at, send_started_at, terminal_at, - updated_at, - legacy_collision, - legacy_collision_count - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(host_id, request_id) DO UPDATE SET action = excluded.action, canonical_version = excluded.canonical_version, @@ -10097,9 +9819,7 @@ def _project_command_request_conn(conn: sqlite3.Connection, row: Any) -> None: reserved_at = excluded.reserved_at, send_started_at = excluded.send_started_at, terminal_at = excluded.terminal_at, - updated_at = excluded.updated_at, - legacy_collision = excluded.legacy_collision, - legacy_collision_count = excluded.legacy_collision_count + updated_at = excluded.updated_at """, ( str(row[1]), @@ -10117,69 +9837,15 @@ def _project_command_request_conn(conn: sqlite3.Connection, row: Any) -> None: row[16], row[17], str(row[18]), - int(row[19]), - int(row[20]), ), ) -def _backfill_command_audit(conn: sqlite3.Connection) -> None: - rows = conn.execute( - """ - SELECT - host_id, - request_id, - action, - payload_fingerprint, - status, - result_json, - created_at, - completed_at, - uncertain - FROM command_receipts - WHERE request_id != '' - ORDER BY id - """ - ).fetchall() - for row in rows: - _upsert_command_audit_from_receipt_row(conn, row) -def _backfill_legacy_attention_columns(conn: sqlite3.Connection) -> None: - conn.execute( - """ - UPDATE attention_items - SET - first_seen_at = CASE - WHEN first_seen_at IS NULL OR first_seen_at = '' - THEN COALESCE(NULLIF(observed_at, ''), updated_at, '') - ELSE first_seen_at - END, - last_seen_at = CASE - WHEN last_seen_at IS NULL OR last_seen_at = '' - THEN COALESCE(NULLIF(observed_at, ''), updated_at, '') - ELSE last_seen_at - END, - last_changed_at = CASE - WHEN last_changed_at IS NULL OR last_changed_at = '' - THEN COALESCE(NULLIF(observed_at, ''), updated_at, '') - ELSE last_changed_at - END, - lifecycle_status = CASE - WHEN lifecycle_status IS NULL OR lifecycle_status = '' - THEN 'open' - ELSE lifecycle_status - END, - signal_count = CASE - WHEN signal_count IS NULL OR signal_count < 1 - THEN 1 - ELSE signal_count - END - """ - ) -def _migration_private_state( +def _connector_group_private_state( raw: Any, *, group: str, @@ -10196,2819 +9862,218 @@ def _migration_private_state( return _canonical_json(state) -def _legacy_attention_job_identity( - row_host_id: str, - payload_json: Any, -) -> tuple[str, str, str, str, str, str | None] | None: - payload = sanitize_public_mapping(_json_object(payload_json), backend_neutral=True) - if str(payload.get("host_id") or "") != str(row_host_id): - return None - event_type = str(payload.get("event_type") or "") - if event_type not in {"attention_created", "attention_escalated"}: - return None - attention = payload.get("attention") - if not isinstance(attention, Mapping): - return None - source = _store_public_text(attention.get("source"), default="") - kind = _store_public_label(attention.get("kind")) - if not source or kind == "unknown": - return None - family_key = _attention_family_key(str(row_host_id), attention) - stage = ( - "initial" - if event_type == "attention_created" - else f"severity:{normalize_severity(attention.get('severity'))}" - ) - return ( - str(row_host_id), - family_key, - event_type, - stage, - _attention_id_from_item(attention), - _strict_utc_timestamp(payload.get("transition_at")), +def _insert_turn_content_page_boundaries_conn( + conn: sqlite3.Connection, + *, + host_id: str, + turn_id: str, + content_revision_value: str, + field: str, + segments: Iterable[Any], +) -> None: + conn.executemany( + """ + INSERT OR IGNORE INTO turn_content_page_boundaries ( + host_id, turn_id, content_revision, field, + page_index, start_char, start_byte + ) VALUES (?, ?, ?, ?, ?, ?, ?) + """, + ( + ( + str(host_id), + str(turn_id), + str(content_revision_value), + str(field), + int(segment.index), + int(segment.start_char), + int(segment.start_byte), + ) + for segment in segments + ), ) -def _migrate_v4_attention_rows_conn(conn: sqlite3.Connection) -> None: - conn.execute("DROP TABLE IF EXISTS temp.attention_v5_rows") +def _insert_turn_content_revision_conn( + conn: sqlite3.Connection, + *, + host_id: str, + turn_id: str, + user_text: str | None, + assistant_final_text: str | None, + user_state: str, + final_state: str, + created_at: str, + is_current: bool = True, +) -> str: + revision = content_revision( + str(turn_id), + user_text, + assistant_final_text, + user_state, + final_state, + ) + user_segments = segment_canonical_text(user_text or "") if user_state == "complete" else () + final_segments = ( + segment_canonical_text(assistant_final_text or "") + if final_state == "complete" + else () + ) conn.execute( """ - CREATE TEMP TABLE attention_v5_rows ( - host_id TEXT NOT NULL, - family_key TEXT NOT NULL, - attention_id TEXT NOT NULL, - is_open INTEGER NOT NULL, - positive_at TEXT, - changed_at TEXT, - first_seen_at TEXT, - severity_rank INTEGER NOT NULL, - signal_count INTEGER NOT NULL - ) - """ + INSERT INTO turn_content_revisions ( + host_id, turn_id, content_revision, user_text, assistant_final_text, + user_state, final_state, user_char_length, user_byte_length, + final_char_length, final_byte_length, user_page_count, + final_page_count, is_current, created_at, superseded_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL) + ON CONFLICT(host_id, turn_id, content_revision) DO NOTHING + """, + ( + str(host_id), + str(turn_id), + revision, + user_text, + assistant_final_text, + user_state, + final_state, + len(user_text or ""), + len((user_text or "").encode("utf-8")), + len(assistant_final_text or ""), + len((assistant_final_text or "").encode("utf-8")), + len(user_segments), + len(final_segments), + int(bool(is_current)), + str(created_at), + ), ) - cursor = conn.execute( - """ - SELECT host_id, attention_id, source, kind, severity, lifecycle_status, - updated_at, observed_at, first_seen_at, last_seen_at, - last_changed_at, signal_count - FROM attention_items - ORDER BY host_id, attention_id + _insert_turn_content_page_boundaries_conn( + conn, + host_id=str(host_id), + turn_id=str(turn_id), + content_revision_value=revision, + field="user_text", + segments=user_segments, + ) + _insert_turn_content_page_boundaries_conn( + conn, + host_id=str(host_id), + turn_id=str(turn_id), + content_revision_value=revision, + field="assistant_final_text", + segments=final_segments, + ) + return revision + +def _ensure_payload_turn_content_revision_conn( + conn: sqlite3.Connection, + *, + host_id: str, + turn_id: str, + payload: Mapping[str, Any], + observed_at: str | None, +) -> bool: + current = conn.execute( """ + SELECT 1 + FROM turn_content_revisions + WHERE host_id = ? AND turn_id = ? AND is_current = 1 + LIMIT 1 + """, + (str(host_id), str(turn_id)), + ).fetchone() + if current is not None: + return False + user_text = sanitize_canonical_turn_text(payload.get("user_text")) + final_text = sanitize_canonical_turn_text( + payload.get("assistant_final_text") ) - while True: - batch = cursor.fetchmany(500) - if not batch: - break - values: list[tuple[Any, ...]] = [] - for row in batch: - host_id = str(row[0]) - item = {"source": row[2], "kind": row[3]} - positive_at = ( - _strict_utc_timestamp(row[9]) - or _strict_utc_timestamp(row[7]) - or _strict_utc_timestamp(row[6]) - ) - values.append( - ( - host_id, - _attention_family_key(host_id, item), - str(row[1]), - int(str(row[5] or "open") == ATTENTION_LIFECYCLE_OPEN), - positive_at, - _strict_utc_timestamp(row[10]), - _strict_utc_timestamp(row[8]), - _attention_severity_rank(row[4]), - max(1, int(row[11] or 1)), - ) - ) - conn.executemany( - """ - INSERT INTO attention_v5_rows ( - host_id, family_key, attention_id, is_open, positive_at, - changed_at, first_seen_at, severity_rank, signal_count - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - values, + if user_text == "": + user_text = None + if final_text == "": + final_text = None + user_state = "complete" if user_text else "absent" + final_state = "complete" if final_text else "absent" + if user_state == "absent" and final_state == "absent": + return _ensure_absent_turn_content_revision_conn( + conn, + host_id=str(host_id), + turn_id=str(turn_id), + observed_at=observed_at, ) + _insert_turn_content_revision_conn( + conn, + host_id=str(host_id), + turn_id=str(turn_id), + user_text=user_text, + assistant_final_text=final_text, + user_state=user_state, + final_state=final_state, + created_at=str(observed_at or "1970-01-01T00:00:00+00:00"), + ) + return True - family_cursor = conn.execute( +def _ensure_absent_turn_content_revision_conn( + conn: sqlite3.Connection, + *, + host_id: str, + turn_id: str, + observed_at: str | None, +) -> bool: + current = conn.execute( """ - SELECT host_id, family_key - FROM attention_v5_rows - GROUP BY host_id, family_key - ORDER BY host_id, family_key + SELECT 1 + FROM turn_content_revisions + WHERE host_id = ? AND turn_id = ? AND is_current = 1 + LIMIT 1 + """, + (str(host_id), str(turn_id)), + ).fetchone() + if current is not None: + return False + revision = _insert_turn_content_revision_conn( + conn, + host_id=str(host_id), + turn_id=str(turn_id), + user_text=None, + assistant_final_text=None, + user_state="absent", + final_state="absent", + created_at=str(observed_at or "1970-01-01T00:00:00+00:00"), + ) + cursor = conn.execute( """ + UPDATE turn_content_revisions + SET is_current = 1, superseded_at = NULL + WHERE host_id = ? + AND turn_id = ? + AND content_revision = ? + AND NOT EXISTS ( + SELECT 1 + FROM turn_content_revisions AS current_revision + WHERE current_revision.host_id = ? + AND current_revision.turn_id = ? + AND current_revision.is_current = 1 + ) + """, + ( + str(host_id), + str(turn_id), + revision, + str(host_id), + str(turn_id), + ), ) - while True: - families = family_cursor.fetchmany(500) - if not families: - break - for host_id_raw, family_key_raw in families: - host_id = str(host_id_raw) - family_key = str(family_key_raw) - candidates = conn.execute( - """ - SELECT attention_id, is_open, positive_at, changed_at, - first_seen_at, severity_rank, signal_count - FROM attention_v5_rows - WHERE host_id = ? AND family_key = ? - ORDER BY attention_id - """, - (host_id, family_key), - ) - winner: tuple[Any, ...] | None = None - earliest_first: str | None = None - latest_positive: str | None = None - latest_progress: str | None = None - total_signals = 0 - max_severity = -1 - while True: - candidate_batch = candidates.fetchmany(500) - if not candidate_batch: - break - for candidate in candidate_batch: - total_signals += max(1, int(candidate[6] or 1)) - max_severity = max(max_severity, int(candidate[5])) - if candidate[4] and ( - earliest_first is None or str(candidate[4]) < earliest_first - ): - earliest_first = str(candidate[4]) - if candidate[2] and ( - latest_positive is None or str(candidate[2]) > latest_positive - ): - latest_positive = str(candidate[2]) - progress_at = max( - str(candidate[2] or ""), - str(candidate[3] or ""), - ) - if progress_at and ( - latest_progress is None or progress_at > latest_progress - ): - latest_progress = progress_at - rank = ( - progress_at, - int(candidate[1]), - str(candidate[2] or ""), - str(candidate[3] or ""), - int(candidate[5]), - int(candidate[6]), - "".join( - chr(0x10FFFF - ord(ch)) for ch in str(candidate[0]) - ), - ) - if winner is None or rank > winner[0]: - winner = (rank, *candidate) - if winner is None or latest_positive is None: - continue - winner_attention_id = str(winner[1]) - is_open = bool(winner[2]) - first_seen_at = earliest_first or latest_positive - # The lifecycle watermark (last_accepted_at) must be the newest - # lifecycle progress — max(latest positive, latest change/resolve) — - # not merely the latest positive. A resolved episode whose resolution - # (t10) is newer than its last positive (t0) would otherwise seed the - # watermark at t0, letting a delayed positive at t5 (< the authoritative - # resolution) pass the observation guard and spuriously reopen - # generation 2 with a fresh notification. last_positive_at stays the - # actual latest positive; the observation key is anchored to the - # accepted progress so replaying the authoritative resolution is a no-op. - accepted_progress = latest_progress or latest_positive - observation_key = stable_fingerprint( - { - "domain": "tendwire.attention.observation.v1", - "host_id": host_id, - "authority": "migration", - "observed_at": accepted_progress, - "snapshot_content_fingerprint": family_key, - } - ) - conn.execute( - """ - INSERT OR IGNORE INTO attention_lifecycles ( - host_id, family_key, generation, lifecycle_status, - current_attention_id, first_seen_at, last_positive_at, - first_missing_at, missing_observation_count, last_accepted_at, - last_observation_key, max_notified_severity_rank - ) VALUES (?, ?, 1, ?, ?, ?, ?, NULL, 0, ?, ?, ?) - """, - ( - host_id, - family_key, - ( - ATTENTION_LIFECYCLE_OPEN - if is_open - else ATTENTION_LIFECYCLE_RESOLVED - ), - winner_attention_id if is_open else None, - first_seen_at, - latest_positive, - accepted_progress, - observation_key, - max_severity, - ), - ) - if is_open: - conn.execute( - """ - UPDATE attention_items - SET first_seen_at = ?, last_seen_at = ?, - signal_count = ?, lifecycle_status = 'open', - resolved_at = NULL, resolved_reason = NULL - WHERE host_id = ? AND attention_id = ? - """, - ( - first_seen_at, - latest_positive, - total_signals, - host_id, - winner_attention_id, - ), - ) - conn.execute( - """ - UPDATE attention_items - SET lifecycle_status = 'resolved', - resolved_at = COALESCE(NULLIF(resolved_at, ''), ?), - resolved_reason = ?, - last_changed_at = ? - WHERE host_id = ? AND lifecycle_status = 'open' - AND attention_id != ? - AND attention_id IN ( - SELECT attention_id FROM attention_v5_rows - WHERE host_id = ? AND family_key = ? AND is_open = 1 - ) - """, - ( - latest_positive, - ATTENTION_RESOLVED_REASON_SUPERSEDED, - latest_positive, - host_id, - winner_attention_id, - host_id, - family_key, - ), - ) - else: - conn.execute( - """ - UPDATE attention_items - SET lifecycle_status = 'resolved', - resolved_at = COALESCE(NULLIF(resolved_at, ''), ?), - resolved_reason = CASE - WHEN attention_id = ? THEN COALESCE(resolved_reason, 'gone') - ELSE ? - END, - last_changed_at = ? - WHERE host_id = ? AND lifecycle_status = 'open' - AND attention_id IN ( - SELECT attention_id FROM attention_v5_rows - WHERE host_id = ? AND family_key = ? AND is_open = 1 - ) - """, - ( - latest_positive, - winner_attention_id, - ATTENTION_RESOLVED_REASON_SUPERSEDED, - latest_positive, - host_id, - host_id, - family_key, - ), - ) - conn.execute("DROP TABLE temp.attention_v5_rows") - - -def _migrate_v4_attention_outbox_conn(conn: sqlite3.Connection) -> None: - conn.execute("DROP TABLE IF EXISTS temp.attention_v5_jobs") - conn.execute( - """ - CREATE TEMP TABLE attention_v5_jobs ( - outbox_id INTEGER PRIMARY KEY, - host_id TEXT NOT NULL, - family_key TEXT NOT NULL, - event_type TEXT NOT NULL, - stage TEXT NOT NULL, - attention_id TEXT NOT NULL, - transition_at TEXT, - group_key TEXT NOT NULL - ) - """ - ) - cursor = conn.execute( - """ - SELECT id, host_id, payload_json - FROM connector_outbox - WHERE connector = ? - ORDER BY id - """, - (ATTENTION_OUTBOX_CONNECTOR,), - ) - while True: - batch = cursor.fetchmany(500) - if not batch: - break - values: list[tuple[Any, ...]] = [] - for outbox_id, host_id, payload_json in batch: - identity = _legacy_attention_job_identity(str(host_id), payload_json) - if identity is None: - continue - ( - identity_host, - family_key, - event_type, - stage, - attention_id, - transition_at, - ) = identity - group_key = stable_fingerprint( - { - "domain": "tendwire.attention.migration-group.v1", - "host_id": identity_host, - "family_key": family_key, - "generation": 1, - "event_type": event_type, - "stage": stage, - } - ) - values.append( - ( - int(outbox_id), - identity_host, - family_key, - event_type, - stage, - attention_id, - transition_at, - group_key, - ) - ) - if event_type == "attention_escalated": - severity = stage.removeprefix("severity:") - conn.execute( - """ - UPDATE attention_lifecycles - SET max_notified_severity_rank = - MAX(max_notified_severity_rank, ?) - WHERE host_id = ? AND family_key = ? - """, - ( - _attention_severity_rank(severity), - identity_host, - family_key, - ), - ) - conn.executemany( - """ - INSERT INTO attention_v5_jobs ( - outbox_id, host_id, family_key, event_type, stage, - attention_id, transition_at, group_key - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - values, - ) - - group_cursor = conn.execute( - """ - SELECT host_id, family_key, event_type, stage, group_key - FROM attention_v5_jobs - GROUP BY host_id, family_key, event_type, stage, group_key - ORDER BY group_key - """ - ) - while True: - groups = group_cursor.fetchmany(500) - if not groups: - break - for host_id, family_key, event_type, stage, group_key in groups: - candidate_rows = conn.execute( - """ - SELECT o.id, o.status, o.payload_json, o.private_state_json, - j.attention_id, j.transition_at - FROM connector_outbox o - JOIN attention_v5_jobs j ON j.outbox_id = o.id - WHERE j.group_key = ? - AND o.status IN ('queued', 'retry', 'deferred', 'leased') - ORDER BY o.id - """, - (group_key,), - ).fetchall() - if not candidate_rows: - continue - lifecycle_row = conn.execute( - """ - SELECT l.lifecycle_status, l.current_attention_id, - l.first_seen_at, l.last_positive_at, l.last_accepted_at, - i.last_changed_at, i.last_seen_at, i.observed_at, - i.updated_at - FROM attention_lifecycles l - LEFT JOIN attention_items i - ON i.host_id = l.host_id - AND i.attention_id = l.current_attention_id - WHERE l.host_id = ? AND l.family_key = ? - """, - (host_id, family_key), - ).fetchone() - lifecycle_open = ( - lifecycle_row is not None - and str(lifecycle_row[0]) == ATTENTION_LIFECYCLE_OPEN - and lifecycle_row[1] is not None - ) - current_attention_id = ( - str(lifecycle_row[1]) if lifecycle_open else "" - ) - current_anchor_candidates = ( - [ - canonical - for canonical in ( - _strict_utc_timestamp(value) - # Include the lifecycle's persisted last_accepted_at - # (index 4) alongside the attention_items timestamps so - # the current-episode anchor reflects the authoritative - # accepted-progress watermark even when the row's own - # timestamps are skewed (e.g. a delayed positive). - for value in lifecycle_row[4:9] - ) - if canonical is not None - ] - if lifecycle_open - else [] - ) - current_episode_anchor = ( - max(current_anchor_candidates) - if current_anchor_candidates - else "" - ) - terminalized_current_episode = False - if lifecycle_open: - terminal_rows = conn.execute( - """ - SELECT j.attention_id, j.transition_at - FROM connector_outbox o - JOIN attention_v5_jobs j ON j.outbox_id = o.id - WHERE j.group_key = ? - AND o.status IN ('delivered', 'dead_letter') - """, - (group_key,), - ).fetchall() - terminalized_current_episode = bool(current_episode_anchor) and any( - str(terminal_attention_id) == current_attention_id - and bool(terminal_transition_at) - and str(terminal_transition_at) >= current_episode_anchor - for terminal_attention_id, terminal_transition_at in terminal_rows - ) - - leased_rows = [ - row - for row in candidate_rows - if str(row[1]) == _CONNECTOR_LEASE_STATUS - and conn.execute( - """ - SELECT 1 FROM connector_deliveries - WHERE outbox_id = ? AND status = 'leased' - LIMIT 1 - """, - (int(row[0]),), - ).fetchone() - is not None - ] - conn.execute( - """ - UPDATE connector_outbox - SET status = ?, next_attempt_at = NULL - WHERE id IN ( - SELECT j.outbox_id FROM attention_v5_jobs j - WHERE j.group_key = ? - ) AND ( - status IN ('queued', 'retry', 'deferred') - OR ( - status = 'leased' - AND id NOT IN ( - SELECT d.outbox_id FROM connector_deliveries d - WHERE d.status = 'leased' AND d.outbox_id IS NOT NULL - ) - ) - ) - """, - (_CONNECTOR_SUPERSEDED_OUTBOX_STATUS, group_key), - ) - - def active_rank(row: Any) -> tuple[int, str, int, int]: - return ( - int(str(row[4]) == current_attention_id), - str(row[5] or ""), - int(str(row[1]) == _CONNECTOR_LEASE_STATUS), - -int(row[0]), - ) - - if not lifecycle_open or terminalized_current_episode: - for leased_row in leased_rows: - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = ? - WHERE id = ? AND status = 'leased' - """, - ( - _migration_private_state( - leased_row[3], - group=str(group_key), - canonical=False, - terminal_after_lease=True, - ), - int(leased_row[0]), - ), - ) - continue - - pollable_candidates = [ - row - for row in candidate_rows - if str(row[1]) in _CONNECTOR_POLLABLE_STATUSES - ] - active_candidates = [*pollable_candidates, *leased_rows] - if not active_candidates: - continue - selected = max(active_candidates, key=active_rank) - selected_id = int(selected[0]) - selected_is_lease = ( - str(selected[1]) == _CONNECTOR_LEASE_STATUS - and any(int(row[0]) == selected_id for row in leased_rows) - ) - for leased_row in leased_rows: - leased_id = int(leased_row[0]) - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = ? - WHERE id = ? AND status = 'leased' - """, - ( - _migration_private_state( - leased_row[3], - group=str(group_key), - canonical=selected_is_lease and leased_id == selected_id, - terminal_after_lease=( - not selected_is_lease or leased_id != selected_id - ), - ), - leased_id, - ), - ) - if selected_is_lease: - continue - transition_key = stable_fingerprint( - { - "domain": "tendwire.attention.transition.v1", - "host_id": str(host_id), - "family_key": str(family_key), - "generation": 1, - "event_type": str(event_type), - "stage": str(stage), - } - ) - canonical_key = f"attention:{event_type}:{transition_key}" - payload = sanitize_public_mapping( - _json_object(selected[2]), backend_neutral=True - ) - transition_at = str(selected[5] or "") - if not transition_at: - transition_at = ( - str(lifecycle_row[4]) - if lifecycle_row is not None - else "1970-01-01T00:00:00+00:00" - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES (?, ?, ?, 'queued', ?, ?, ?, ?, NULL) - ON CONFLICT(host_id, connector, delivery_key) DO NOTHING - """, - ( - str(host_id), - ATTENTION_OUTBOX_CONNECTOR, - canonical_key, - _canonical_json(payload), - _migration_private_state( - {}, - group=str(group_key), - canonical=True, - ), - transition_at, - transition_at, - ), - ) - conn.execute("DROP TABLE temp.attention_v5_jobs") - - -def _migrate_v0_to_v1_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_SNAPSHOTS_TABLE) - if "content_fingerprint" not in _table_columns(conn): - conn.execute( - "ALTER TABLE snapshots ADD COLUMN " - "content_fingerprint TEXT NOT NULL DEFAULT ''" - ) - _backfill_content_fingerprints(conn) - for statement in CREATE_LEGACY_SNAPSHOT_INDEXES: - conn.execute(statement) - - -def _migrate_v1_to_v2_conn(conn: sqlite3.Connection) -> None: - _migrate_v0_to_v1_conn(conn) - conn.execute(CREATE_LEGACY_COMMAND_RECEIPTS_TABLE) - _ensure_command_receipt_columns(conn) - _dedupe_command_receipts(conn) - for statement in CREATE_LEGACY_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - _ensure_command_receipt_unique_index(conn) - conn.execute(CREATE_WORKER_BINDINGS_TABLE) - _ensure_worker_binding_columns(conn) - for statement in CREATE_WORKER_BINDING_INDEXES: - conn.execute(statement) - conn.execute(CREATE_WORKER_BINDING_UNIQUE_INDEX) - - -def _migrate_v2_to_v3_conn(conn: sqlite3.Connection) -> None: - _migrate_v1_to_v2_conn(conn) - for statement in CREATE_PR6_TABLES: - conn.execute(statement) - _ensure_pr6_columns(conn) - for statement in CREATE_PR6_INDEXES: - conn.execute(statement) - _backfill_command_audit(conn) - - -def _migrate_v3_to_v4_conn(conn: sqlite3.Connection) -> None: - _migrate_v2_to_v3_conn(conn) - _backfill_legacy_attention_columns(conn) - - -def _migrate_v4_to_v5_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_ATTENTION_LIFECYCLES_TABLE) - for statement in CREATE_ATTENTION_LIFECYCLE_INDEXES: - conn.execute(statement) - _migrate_v4_attention_rows_conn(conn) - _migrate_v4_attention_outbox_conn(conn) - - -_LEGACY_TRUNCATION_MARKER = "\n[truncated]" - - -def _legacy_canonical_field(value: Any) -> tuple[str | None, str]: - text = sanitize_canonical_turn_text(value) - if text is None or text == "": - return None, "absent" - state = "known_incomplete" if text.endswith(_LEGACY_TRUNCATION_MARKER) else "complete" - return text, state - - -def _insert_turn_content_page_boundaries_conn( - conn: sqlite3.Connection, - *, - host_id: str, - turn_id: str, - content_revision_value: str, - field: str, - segments: Iterable[Any], -) -> None: - conn.executemany( - """ - INSERT OR IGNORE INTO turn_content_page_boundaries ( - host_id, - turn_id, - content_revision, - field, - page_index, - start_char, - start_byte - ) VALUES (?, ?, ?, ?, ?, ?, ?) - """, - ( - ( - str(host_id), - str(turn_id), - str(content_revision_value), - str(field), - int(segment.index), - int(segment.start_char), - int(segment.start_byte), - ) - for segment in segments - ), - ) - - -def _insert_turn_content_revision_conn( - conn: sqlite3.Connection, - *, - host_id: str, - turn_id: str, - user_text: str | None, - assistant_final_text: str | None, - user_state: str, - final_state: str, - created_at: str, - is_current: bool = True, -) -> str: - revision = content_revision( - str(turn_id), - user_text, - assistant_final_text, - user_state, - final_state, - ) - user_segments = segment_canonical_text(user_text or "") if user_state == "complete" else () - final_segments = ( - segment_canonical_text(assistant_final_text or "") - if final_state == "complete" - else () - ) - conn.execute( - """ - INSERT INTO turn_content_revisions ( - host_id, turn_id, content_revision, user_text, assistant_final_text, - user_state, final_state, user_char_length, user_byte_length, - final_char_length, final_byte_length, user_page_count, - final_page_count, is_current, created_at, superseded_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL) - ON CONFLICT(host_id, turn_id, content_revision) DO NOTHING - """, - ( - str(host_id), - str(turn_id), - revision, - user_text, - assistant_final_text, - user_state, - final_state, - len(user_text or ""), - len((user_text or "").encode("utf-8")), - len(assistant_final_text or ""), - len((assistant_final_text or "").encode("utf-8")), - len(user_segments), - len(final_segments), - int(bool(is_current)), - str(created_at), - ), - ) - _insert_turn_content_page_boundaries_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - content_revision_value=revision, - field="user_text", - segments=user_segments, - ) - _insert_turn_content_page_boundaries_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - content_revision_value=revision, - field="assistant_final_text", - segments=final_segments, - ) - return revision - - -def _backfill_legacy_turn_content_conn(conn: sqlite3.Connection) -> None: - rows = conn.execute( - """ - SELECT host_id, turn_id, observed_at, payload_json - FROM turns - ORDER BY host_id, turn_id - """ - ).fetchall() - for host_id, turn_id, observed_at, payload_json in rows: - try: - payload = json.loads(str(payload_json or "{}")) - except (TypeError, json.JSONDecodeError): - payload = {} - if not isinstance(payload, dict): - payload = {} - user_text, user_state = _legacy_canonical_field(payload.get("user_text")) - final_text, final_state = _legacy_canonical_field( - payload.get("assistant_final_text") - ) - if user_state != "absent" or final_state != "absent": - _insert_turn_content_revision_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - user_text=user_text, - assistant_final_text=final_text, - user_state=user_state, - final_state=final_state, - created_at=str(observed_at or "1970-01-01T00:00:00+00:00"), - ) - for key in ( - "user_text", - "assistant_final_text", - "user_preview", - "assistant_final_preview", - "content", - ): - payload.pop(key, None) - encoded = _canonical_json(payload) - conn.execute( - """ - UPDATE turns - SET payload_json = ?, fingerprint = ? - WHERE host_id = ? AND turn_id = ? - """, - ( - encoded, - stable_fingerprint(payload), - str(host_id), - str(turn_id), - ), - ) - - -def _ensure_payload_turn_content_revision_conn( - conn: sqlite3.Connection, - *, - host_id: str, - turn_id: str, - payload: Mapping[str, Any], - observed_at: str | None, -) -> bool: - current = conn.execute( - """ - SELECT 1 - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - LIMIT 1 - """, - (str(host_id), str(turn_id)), - ).fetchone() - if current is not None: - return False - user_text = sanitize_canonical_turn_text(payload.get("user_text")) - final_text = sanitize_canonical_turn_text( - payload.get("assistant_final_text") - ) - if user_text == "": - user_text = None - if final_text == "": - final_text = None - user_state = "complete" if user_text else "absent" - final_state = "complete" if final_text else "absent" - if user_state == "absent" and final_state == "absent": - return _ensure_absent_turn_content_revision_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - observed_at=observed_at, - ) - _insert_turn_content_revision_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - user_text=user_text, - assistant_final_text=final_text, - user_state=user_state, - final_state=final_state, - created_at=str(observed_at or "1970-01-01T00:00:00+00:00"), - ) - return True - - -def _ensure_absent_turn_content_revision_conn( - conn: sqlite3.Connection, - *, - host_id: str, - turn_id: str, - observed_at: str | None, -) -> bool: - current = conn.execute( - """ - SELECT 1 - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - LIMIT 1 - """, - (str(host_id), str(turn_id)), - ).fetchone() - if current is not None: - return False - revision = _insert_turn_content_revision_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - user_text=None, - assistant_final_text=None, - user_state="absent", - final_state="absent", - created_at=str(observed_at or "1970-01-01T00:00:00+00:00"), - ) - cursor = conn.execute( - """ - UPDATE turn_content_revisions - SET is_current = 1, superseded_at = NULL - WHERE host_id = ? - AND turn_id = ? - AND content_revision = ? - AND NOT EXISTS ( - SELECT 1 - FROM turn_content_revisions AS current_revision - WHERE current_revision.host_id = ? - AND current_revision.turn_id = ? - AND current_revision.is_current = 1 - ) - """, - ( - str(host_id), - str(turn_id), - revision, - str(host_id), - str(turn_id), - ), - ) - return bool(cursor.rowcount) - - -def _backfill_missing_turn_content_revisions_conn( - conn: sqlite3.Connection, -) -> int: - """Give every stored turn one stable authoritative v2 content descriptor.""" - repaired = 0 - cursor = conn.execute( - """ - SELECT turns.host_id, turns.turn_id, turns.observed_at - FROM turns - WHERE NOT EXISTS ( - SELECT 1 - FROM turn_content_revisions AS revisions - WHERE revisions.host_id = turns.host_id - AND revisions.turn_id = turns.turn_id - AND revisions.is_current = 1 - ) - ORDER BY turns.host_id, turns.turn_id - """ - ) - while True: - rows = cursor.fetchmany(500) - if not rows: - return repaired - for host_id, turn_id, observed_at in rows: - if _ensure_absent_turn_content_revision_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - observed_at=str(observed_at) if observed_at else None, - ): - repaired += 1 - - -def _rebuild_v6_presentation_plans_conn(conn: sqlite3.Connection) -> None: - """Rebuild the two bounded plan tables with generation-aware v7 keys.""" - conn.execute( - """ - CREATE TABLE turn_presentation_plans_v7 ( - id INTEGER PRIMARY KEY, - host_id TEXT NOT NULL, - name TEXT NOT NULL, - plan_token TEXT NOT NULL, - turn_id TEXT NOT NULL, - content_revision TEXT NOT NULL, - presentation_version TEXT NOT NULL, - generation INTEGER NOT NULL DEFAULT 1 CHECK (generation >= 1), - part_count INTEGER NOT NULL CHECK (part_count > 0), - state TEXT NOT NULL - CHECK (state IN ( - 'preparing', - 'waiting_predecessor', - 'active', - 'completed', - 'superseded', - 'failed' - )), - replaces_plan_token TEXT, - recovers_plan_token TEXT, - created_at TEXT NOT NULL, - activated_at TEXT, - completed_at TEXT, - UNIQUE (host_id, name, plan_token), - UNIQUE ( - host_id, - name, - turn_id, - content_revision, - presentation_version, - generation - ), - FOREIGN KEY (host_id, turn_id, content_revision) - REFERENCES turn_content_revisions(host_id, turn_id, content_revision) - ON DELETE RESTRICT - ) - """ - ) - conn.execute( - """ - INSERT INTO turn_presentation_plans_v7 ( - id, host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - replaces_plan_token, recovers_plan_token, created_at, - activated_at, completed_at - ) - SELECT - id, host_id, name, plan_token, turn_id, content_revision, - presentation_version, 1, part_count, state, - replaces_plan_token, NULL, created_at, activated_at, completed_at - FROM turn_presentation_plans - ORDER BY id - """ - ) - conn.execute( - """ - CREATE TABLE turn_presentation_jobs_v7 ( - id INTEGER PRIMARY KEY, - plan_id INTEGER NOT NULL, - sequence_index INTEGER NOT NULL CHECK (sequence_index >= 0), - operation TEXT NOT NULL CHECK (operation IN ('upsert', 'retire')), - part_ordinal INTEGER NOT NULL CHECK (part_ordinal >= 0), - spans_json TEXT NOT NULL, - outbox_id INTEGER UNIQUE, - created_at TEXT NOT NULL, - UNIQUE (plan_id, sequence_index), - UNIQUE (plan_id, operation, part_ordinal), - FOREIGN KEY (plan_id) - REFERENCES turn_presentation_plans_v7(id) ON DELETE CASCADE, - FOREIGN KEY (outbox_id) - REFERENCES connector_outbox(id) ON DELETE RESTRICT - ) - """ - ) - conn.execute( - """ - INSERT INTO turn_presentation_jobs_v7 ( - id, plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - ) - SELECT - id, plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - FROM turn_presentation_jobs - ORDER BY id - """ - ) - conn.execute("DROP TABLE turn_presentation_jobs") - conn.execute("DROP TABLE turn_presentation_plans") - conn.execute( - "ALTER TABLE turn_presentation_plans_v7 RENAME TO turn_presentation_plans" - ) - conn.execute( - "ALTER TABLE turn_presentation_jobs_v7 RENAME TO turn_presentation_jobs" - ) - - -def _migrate_v6_to_v7_conn(conn: sqlite3.Connection) -> None: - """Add explicit failed-plan generations and immutable recovery audit.""" - conn.execute(CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE) - _backfill_missing_turn_content_revisions_conn(conn) - _backfill_missing_turn_content_page_boundaries_conn(conn) - plan_columns = { - str(row[1]) - for row in conn.execute( - "PRAGMA table_info(turn_presentation_plans)" - ).fetchall() - } - if "generation" not in plan_columns: - _rebuild_v6_presentation_plans_conn(conn) - conn.execute(CREATE_TURN_PRESENTATION_RECOVERIES_TABLE) - for statement in CREATE_TURN_PRESENTATION_INDEXES: - conn.execute(statement) - - -def _migrate_v5_to_v6_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_TURN_CONTENT_REVISIONS_TABLE) - conn.execute(CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE) - for statement in CREATE_TURN_CONTENT_REVISION_INDEXES: - conn.execute(statement) - conn.execute(CREATE_TURN_PRESENTATION_PLANS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_JOBS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_RECOVERIES_TABLE) - for statement in CREATE_TURN_PRESENTATION_INDEXES: - conn.execute(statement) - _backfill_legacy_turn_content_conn(conn) - _backfill_missing_turn_content_revisions_conn(conn) - - -def _normalize_snapshot_created_at_v8_conn( - conn: sqlite3.Connection, -) -> None: - """Canonicalize legacy ordering keys before the v8 age index is built.""" - last_id = 0 - while True: - rows = conn.execute( - """ - SELECT id, created_at, payload - FROM snapshots - WHERE id > ? - ORDER BY id - LIMIT 500 - """, - (last_id,), - ).fetchall() - if not rows: - return - updates: list[tuple[str, int]] = [] - for row_id, raw_created_at, raw_payload in rows: - raw_created_at_text = str(raw_created_at) - canonical = _strict_utc_timestamp(raw_created_at_text) - if ( - canonical == _LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE - and not _legacy_snapshot_created_at_is_authoritative( - raw_created_at, - raw_payload, - ) - ): - canonical = None - canonical = canonical or _SNAPSHOT_CREATED_AT_QUARANTINE - if str(raw_created_at) != canonical: - updates.append((canonical, int(row_id))) - last_id = int(row_id) - if updates: - conn.executemany( - "UPDATE snapshots SET created_at = ? WHERE id = ?", - updates, - ) - - -def _migrate_v7_to_v8_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_STORE_MAINTENANCE_STATE_TABLE) - conn.execute(INSERT_STORE_MAINTENANCE_STATE) - _normalize_snapshot_created_at_v8_conn(conn) - for statement in CREATE_SNAPSHOT_INDEXES: - conn.execute(statement) - for index_name in ( - "idx_snapshots_host_id", - "idx_snapshots_created_at", - "idx_snapshots_content_fingerprint", - "idx_snapshots_host_created_id", - ): - conn.execute(f"DROP INDEX IF EXISTS {index_name}") - - -def _ensure_turn_list_state_conn(conn: sqlite3.Connection) -> str: - conn.execute(CREATE_TURN_LIST_STATE_TABLE) - row = conn.execute( - "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" - ).fetchone() - if row is not None and str(row[0]): - return str(row[0]) - epoch = secrets.token_urlsafe(32) - conn.execute( - """ - INSERT INTO turn_list_state (scope, store_epoch) - VALUES ('turn-list', ?) - ON CONFLICT(scope) DO NOTHING - """, - (epoch,), - ) - row = conn.execute( - "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" - ).fetchone() - if row is None or not str(row[0]): - raise StoreSchemaError("turn_list_state_unavailable") - return str(row[0]) - - -def _turn_list_store_epoch_conn(conn: sqlite3.Connection) -> str: - row = conn.execute( - "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" - ).fetchone() - if row is None or not str(row[0]): - raise StoreSchemaError("turn_list_state_unavailable") - return str(row[0]) - - -def _ensure_turn_change_state_conn(conn: sqlite3.Connection) -> str: - conn.execute(CREATE_TURN_CHANGE_STATE_TABLE) - row = conn.execute( - "SELECT store_epoch FROM turn_change_state WHERE scope = 'turn-delta'" - ).fetchone() - if row is not None and str(row[0]): - return str(row[0]) - epoch = secrets.token_urlsafe(32) - conn.execute( - """ - INSERT INTO turn_change_state(scope, store_epoch) - VALUES ('turn-delta', ?) - ON CONFLICT(scope) DO NOTHING - """, - (epoch,), - ) - row = conn.execute( - "SELECT store_epoch FROM turn_change_state WHERE scope = 'turn-delta'" - ).fetchone() - if row is None or not str(row[0]): - raise StoreSchemaError("turn_change_state_unavailable") - return str(row[0]) - - -def _turn_change_store_epoch_conn(conn: sqlite3.Connection) -> str: - row = conn.execute( - "SELECT store_epoch FROM turn_change_state WHERE scope = 'turn-delta'" - ).fetchone() - if row is None or not str(row[0]): - raise StoreSchemaError("turn_change_state_unavailable") - return str(row[0]) - - -def _ensure_turn_list_host_states_conn(conn: sqlite3.Connection) -> None: - conn.execute(CREATE_TURN_LIST_HOSTS_TABLE) - conn.execute( - """ - INSERT INTO turn_list_hosts ( - host_id, - next_sequence, - traversal_generation - ) - SELECT host_id, COALESCE(MAX(list_sequence), 0) + 1, 1 - FROM turns - GROUP BY host_id - ON CONFLICT(host_id) DO NOTHING - """ - ) - - -def _migrate_v8_to_v9_conn(conn: sqlite3.Connection) -> None: - columns = _table_columns(conn, "turns") - if "list_sequence" not in columns: - conn.execute( - "ALTER TABLE turns ADD COLUMN list_sequence " - "INTEGER NOT NULL DEFAULT 0" - ) - conn.execute( - """ - WITH ranked AS ( - SELECT - host_id, - turn_id, - ROW_NUMBER() OVER ( - PARTITION BY host_id - ORDER BY COALESCE(updated_at, observed_at, ''), turn_id - ) AS assigned_sequence - FROM turns - ) - UPDATE turns - SET list_sequence = ( - SELECT assigned_sequence - FROM ranked - WHERE ranked.host_id = turns.host_id - AND ranked.turn_id = turns.turn_id - ) - WHERE list_sequence <= 0 - """ - ) - for statement in CREATE_TURN_LIST_INDEXES: - conn.execute(statement) - _ensure_turn_list_state_conn(conn) - _ensure_turn_list_host_states_conn(conn) - - -def _migrate_v9_to_v10_conn(conn: sqlite3.Connection) -> None: - """Add explicit pending freshness, private routing, and two-phase claims.""" - conn.execute(CREATE_LEGACY_BACKEND_PENDING_TABLE) - columns = _table_columns(conn, "backend_pending") - additions = ( - ("revision_digest", "TEXT NOT NULL DEFAULT ''"), - ("choice_routes_json", "TEXT NOT NULL DEFAULT '{}'"), - ("binding_private_fingerprint", "TEXT NOT NULL DEFAULT ''"), - ("observed_turn_target_value", "TEXT NOT NULL DEFAULT ''"), - ("observation_state", "TEXT NOT NULL DEFAULT 'open'"), - ("freshness", "TEXT NOT NULL DEFAULT 'fresh'"), - ("last_success_at", "TEXT"), - ("last_failure_at", "TEXT"), - ("grace_deadline", "TEXT"), - ("updated_at", "TEXT NOT NULL DEFAULT ''"), - ) - for name, declaration in additions: - if name not in columns: - conn.execute( - f"ALTER TABLE backend_pending ADD COLUMN {name} {declaration}" - ) - rows = conn.execute( - """ - SELECT host_id, worker_id, payload_json, observed_at, - revision_digest, last_success_at, updated_at - FROM backend_pending - """ - ).fetchall() - for ( - host_id, - worker_id, - payload_json, - observed_at, - revision_digest, - last_success_at, - updated_at, - ) in rows: - timestamp = _strict_utc_timestamp(observed_at) or "1970-01-01T00:00:00+00:00" - digest = str(revision_digest or "") or stable_fingerprint( - {"legacy_backend_pending": str(payload_json)} - ) - conn.execute( - """ - UPDATE backend_pending - SET revision_digest = ?, - freshness = 'fresh', - last_success_at = ?, - updated_at = ? - WHERE host_id = ? AND worker_id = ? - """, - ( - digest, - str(last_success_at or timestamp), - str(updated_at or timestamp), - str(host_id), - str(worker_id), - ), - ) - conn.execute(CREATE_BACKEND_PENDING_CLAIMS_TABLE) - - -def _migration_plan_has_exact_coverage_conn( - conn: sqlite3.Connection, - *, - plan_id: int, -) -> bool: - plan = conn.execute( - """ - SELECT - host_id, name, turn_id, content_revision, - presentation_version, generation, part_count - FROM turn_presentation_plans - WHERE id = ? - """, - (int(plan_id),), - ).fetchone() - if plan is None: - return False - revision_row, revision_error = _current_presentation_revision_conn( - conn, - host_id=str(plan[0]), - turn_id=str(plan[2]), - content_revision_value=str(plan[3]), - ) - if revision_error is not None or revision_row is None: - return False - staged = conn.execute( - """ - WITH effective AS ( - SELECT - jobs.id, - jobs.part_ordinal, - jobs.spans_json, - ROW_NUMBER() OVER ( - PARTITION BY jobs.part_ordinal - ORDER BY lineage.generation DESC, lineage.id DESC, jobs.id DESC - ) AS effective_rank - FROM turn_presentation_plans AS lineage - JOIN turn_presentation_jobs AS jobs - ON jobs.plan_id = lineage.id - WHERE lineage.host_id = ? - AND lineage.name = ? - AND lineage.turn_id = ? - AND lineage.content_revision = ? - AND lineage.presentation_version = ? - AND lineage.generation <= ? - AND lineage.state IN ('completed', 'superseded') - AND jobs.operation = 'upsert' - ) - SELECT id, part_ordinal, spans_json - FROM effective - WHERE effective_rank = 1 - ORDER BY part_ordinal - """, - ( - str(plan[0]), - str(plan[1]), - str(plan[2]), - str(plan[3]), - str(plan[4]), - int(plan[5]), - ), - ).fetchall() - if ( - len(staged) != int(plan[6]) - or [int(row[1]) for row in staged] != list(range(int(plan[6]))) - ): - return False - try: - for row in staged: - spans = json.loads(str(row[2])) - if ( - not isinstance(spans, list) - or _validate_presentation_spans( - spans, - revision_row=revision_row, - ) - is None - ): - return False - return _presentation_exact_coverage(staged, revision_row=revision_row) - except (KeyError, TypeError, ValueError, json.JSONDecodeError): - return False - - -def _migration_plan_route_matches_conn( - conn: sqlite3.Connection, - *, - plan_id: int, - authoritative: Mapping[str, Any], -) -> bool: - rows = conn.execute( - """ - SELECT jobs.outbox_id, outbox.payload_json - FROM turn_presentation_jobs AS jobs - LEFT JOIN connector_outbox AS outbox ON outbox.id = jobs.outbox_id - WHERE jobs.plan_id = ? - ORDER BY jobs.id - """, - (int(plan_id),), - ).fetchall() - if not rows: - return False - expected = { - "schema_version": 2, - "turn_id": str(authoritative.get("turn_id") or ""), - "content_revision": str(authoritative.get("content_revision") or ""), - "final_identity": str(authoritative.get("final_identity") or ""), - "stable_key": str(authoritative.get("stable_key") or ""), - "stable_key_version": 1, - } - for outbox_id, payload_json in rows: - if outbox_id is None: - return False - payload = _json_object(payload_json) - route = payload.get("turn") - if not isinstance(route, Mapping): - return False - actual = { - "schema_version": route.get("schema_version"), - "turn_id": str(route.get("turn_id") or ""), - "content_revision": str(route.get("content_revision") or ""), - "final_identity": str(route.get("final_identity") or ""), - "stable_key": str(route.get("stable_key") or ""), - "stable_key_version": route.get("stable_key_version"), - } - if actual != expected: - return False - return True - - -def _migrate_v10_to_v11_conn(conn: sqlite3.Connection) -> None: - """Add typed final anchors and conservatively classify legacy finals.""" - required_legacy_tables = { - "connector_outbox", - "turn_presentation_plans", - "turn_presentation_jobs", - "turn_presentation_recoveries", - "turn_content_revisions", - } - present_legacy_tables = { - table for table in required_legacy_tables if _table_columns(conn, table) - } - if not present_legacy_tables: - conn.execute(CREATE_SNAPSHOTS_TABLE) - conn.execute(CREATE_LEGACY_COMMAND_RECEIPTS_TABLE) - conn.execute(CREATE_WORKER_BINDINGS_TABLE) - for statement in CREATE_PR6_TABLES: - conn.execute(statement) - conn.execute(CREATE_ATTENTION_LIFECYCLES_TABLE) - conn.execute(CREATE_TURN_CONTENT_REVISIONS_TABLE) - conn.execute(CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_PLANS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_JOBS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_RECOVERIES_TABLE) - conn.execute(CREATE_STORE_MAINTENANCE_STATE_TABLE) - conn.execute(CREATE_STORE_MAINTENANCE_CURSORS_TABLE) - conn.execute(CREATE_TURN_LIST_STATE_TABLE) - conn.execute(CREATE_TURN_LIST_HOSTS_TABLE) - for statements in ( - CREATE_LEGACY_COMMAND_RECEIPT_INDEXES, - CREATE_WORKER_BINDING_INDEXES, - CREATE_PR6_INDEXES, - CREATE_TURN_LIST_INDEXES, - CREATE_ATTENTION_LIFECYCLE_INDEXES, - CREATE_TURN_CONTENT_REVISION_INDEXES, - CREATE_TURN_PRESENTATION_INDEXES, - CREATE_FINAL_DELIVERY_INDEXES, - CREATE_SNAPSHOT_INDEXES, - ): - for statement in statements: - conn.execute(statement) - conn.execute(CREATE_LEGACY_COMMAND_RECEIPT_UNIQUE_INDEX) - conn.execute(CREATE_WORKER_BINDING_UNIQUE_INDEX) - conn.execute(INSERT_STORE_MAINTENANCE_STATE) - _ensure_turn_list_state_conn(conn) - return - if present_legacy_tables != required_legacy_tables: - raise StoreSchemaError("legacy_final_schema_incomplete") - conn.execute(CREATE_STORE_MAINTENANCE_CURSORS_TABLE) - _ensure_columns( - conn, - "connector_outbox", - { - "delivery_kind": "TEXT NOT NULL DEFAULT 'generic'", - "turn_id": "TEXT", - "content_revision": "TEXT", - }, - ) - _ensure_columns( - conn, - "turn_presentation_plans", - { - "source_outbox_id": ( - "INTEGER REFERENCES connector_outbox(id) ON DELETE RESTRICT" - ), - }, - ) - conn.execute( - """ - UPDATE connector_outbox AS outbox - SET delivery_kind = 'final_part', - turn_id = ( - SELECT plans.turn_id - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans ON plans.id = jobs.plan_id - WHERE jobs.outbox_id = outbox.id - AND plans.host_id = outbox.host_id - AND plans.name = outbox.connector - ), - content_revision = ( - SELECT plans.content_revision - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans ON plans.id = jobs.plan_id - WHERE jobs.outbox_id = outbox.id - AND plans.host_id = outbox.host_id - AND plans.name = outbox.connector - ) - WHERE EXISTS ( - SELECT 1 - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans ON plans.id = jobs.plan_id - WHERE jobs.outbox_id = outbox.id - AND plans.host_id = outbox.host_id - AND plans.name = outbox.connector - ) - """ - ) - dangling_recovery = conn.execute( - """ - SELECT 1 - FROM turn_presentation_recoveries AS recovery - LEFT JOIN turn_presentation_plans AS failed - ON failed.id = recovery.failed_plan_id - LEFT JOIN turn_presentation_plans AS recovered - ON recovered.id = recovery.recovered_plan_id - WHERE failed.id IS NULL - OR recovered.id IS NULL - OR failed.id = recovered.id - OR failed.host_id != recovered.host_id - OR failed.name != recovered.name - OR failed.turn_id != recovered.turn_id - OR failed.content_revision != recovered.content_revision - OR failed.presentation_version != recovered.presentation_version - OR recovered.generation <= failed.generation - LIMIT 1 - """ - ).fetchone() - if dangling_recovery is not None: - raise StoreSchemaError("legacy_final_recovery_invalid") - recovery_edges = conn.execute( - """ - SELECT failed_plan_id, recovered_plan_id, created_at - FROM turn_presentation_recoveries - ORDER BY generation DESC, id DESC - """ - ).fetchall() - for failed_plan_id, recovered_plan_id, recovered_at in recovery_edges: - _finalize_recovered_plan_materialization_conn( - conn, - failed_plan_id=int(failed_plan_id), - recovered_plan_id=int(recovered_plan_id), - now=str(recovered_at), - ) - current_finals = conn.execute( - """ - SELECT - revisions.host_id, - revisions.turn_id, - revisions.content_revision, - revisions.created_at, - turns.payload_json, - revisions.user_text, - revisions.assistant_final_text - FROM turn_content_revisions AS revisions - JOIN turns - ON turns.host_id = revisions.host_id - AND turns.turn_id = revisions.turn_id - WHERE revisions.is_current = 1 - AND revisions.final_state = 'complete' - ORDER BY revisions.host_id, revisions.turn_id - """ - ).fetchall() - for ( - host_id, - turn_id, - revision, - revision_created_at, - turn_payload_json, - revision_user_text, - revision_final_text, - ) in current_finals: - payload = _final_ready_payload_conn( - conn, - host_id=str(host_id), - turn_id=str(turn_id), - content_revision_value=str(revision), - allow_unroutable=True, - ) - if payload is None: - raise StoreSchemaError("legacy_final_descriptor_unavailable") - automation_payload = _json_object(turn_payload_json) - automation_payload["user_text"] = revision_user_text - automation_payload["assistant_final_text"] = revision_final_text - internal_automation = is_internal_automation_turn_payload( - automation_payload - ) - if internal_automation: - conn.execute( - """ - UPDATE connector_outbox - SET delivery_kind = 'final_migration_hold', - status = 'dead_letter', - next_attempt_at = NULL, - updated_at = ? - WHERE id IN ( - SELECT jobs.outbox_id - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans - ON plans.id = jobs.plan_id - WHERE plans.host_id = ? - AND plans.name = ? - AND plans.turn_id = ? - AND plans.content_revision = ? - AND jobs.outbox_id IS NOT NULL - ) - """, - ( - str(revision_created_at), - str(host_id), - _TURN_FINAL_NAME, - str(turn_id), - str(revision), - ), - ) - routable = ( - payload.get("schema_version") == 2 - and not bool(payload["content"]["known_incomplete"]) - and not internal_automation - ) - unresolved = conn.execute( - """ - SELECT id, state - FROM turn_presentation_plans - WHERE host_id = ? - AND name = ? - AND turn_id = ? - AND content_revision = ? - AND state IN ( - 'preparing', - 'waiting_predecessor', - 'active', - 'failed' - ) - AND NOT EXISTS ( - SELECT 1 - FROM turn_presentation_recoveries AS recovery - WHERE recovery.failed_plan_id = turn_presentation_plans.id - ) - ORDER BY id DESC - """, - ( - str(host_id), - _TURN_FINAL_NAME, - str(turn_id), - str(revision), - ), - ).fetchall() - proven = conn.execute( - """ - SELECT plans.id, plans.completed_at - FROM turn_presentation_plans AS plans - WHERE plans.host_id = ? - AND plans.name = ? - AND plans.turn_id = ? - AND plans.content_revision = ? - AND plans.state = 'completed' - AND plans.completed_at IS NOT NULL - AND ( - SELECT COUNT(DISTINCT jobs.part_ordinal) - FROM turn_presentation_plans AS lineage - JOIN turn_presentation_jobs AS jobs - ON jobs.plan_id = lineage.id - WHERE lineage.host_id = plans.host_id - AND lineage.name = plans.name - AND lineage.turn_id = plans.turn_id - AND lineage.content_revision = plans.content_revision - AND lineage.presentation_version = plans.presentation_version - AND lineage.generation <= plans.generation - AND lineage.state IN ('completed', 'superseded') - AND jobs.operation = 'upsert' - ) = plans.part_count - AND NOT EXISTS ( - SELECT 1 - FROM turn_presentation_plans AS lineage - JOIN turn_presentation_jobs AS jobs - ON jobs.plan_id = lineage.id - WHERE lineage.host_id = plans.host_id - AND lineage.name = plans.name - AND lineage.turn_id = plans.turn_id - AND lineage.content_revision = plans.content_revision - AND lineage.presentation_version = plans.presentation_version - AND lineage.generation <= plans.generation - AND lineage.state IN ('completed', 'superseded') - AND jobs.operation = 'upsert' - AND jobs.part_ordinal >= plans.part_count - ) - AND NOT EXISTS ( - SELECT 1 - FROM turn_presentation_plans AS lineage - JOIN turn_presentation_jobs AS jobs - ON jobs.plan_id = lineage.id - LEFT JOIN connector_outbox AS outbox - ON outbox.id = jobs.outbox_id - WHERE lineage.host_id = plans.host_id - AND lineage.name = plans.name - AND lineage.turn_id = plans.turn_id - AND lineage.content_revision = plans.content_revision - AND lineage.presentation_version = plans.presentation_version - AND lineage.generation <= plans.generation - AND lineage.state IN ('completed', 'superseded') - AND ( - outbox.id IS NULL - OR outbox.host_id != plans.host_id - OR outbox.connector != plans.name - OR outbox.turn_id != plans.turn_id - OR outbox.content_revision != plans.content_revision - OR outbox.delivery_kind != 'final_part' - OR outbox.status != 'delivered' - OR NOT EXISTS ( - SELECT 1 - FROM connector_deliveries AS delivered_attempt - WHERE delivered_attempt.outbox_id = outbox.id - AND delivered_attempt.host_id = outbox.host_id - AND delivered_attempt.connector = outbox.connector - AND delivered_attempt.delivery_key = outbox.delivery_key - AND delivered_attempt.status = 'delivered' - AND delivered_attempt.delivered_at IS NOT NULL - ) - ) - ) - ORDER BY plans.id DESC - LIMIT 1 - """, - ( - str(host_id), - _TURN_FINAL_NAME, - str(turn_id), - str(revision), - ), - ).fetchone() - if ( - proven is not None - and not _migration_plan_has_exact_coverage_conn( - conn, - plan_id=int(proven[0]), - ) - ): - proven = None - linkable = [ - int(row[0]) - for row in unresolved - if str(row[1]) in {"waiting_predecessor", "active", "failed"} - and _migration_plan_route_matches_conn( - conn, - plan_id=int(row[0]), - authoritative=payload, - ) - ] - if not routable: - linkable = [] - delivery_kind = "final_migration_hold" - status = "dead_letter" - classified_at = str(revision_created_at) - elif unresolved: - delivery_kind = "final_ready" if linkable else "final_migration_hold" - status = "awaiting_ack" if linkable else "dead_letter" - classified_at = str(revision_created_at) - elif proven is not None: - delivery_kind = "final_ready" - status = "delivered" - classified_at = str(proven[1]) - else: - delivery_kind = "final_migration_hold" - status = "dead_letter" - classified_at = str(revision_created_at) - final_identity = str(payload["final_identity"]) - delivery_key = f"{_TURN_FINAL_NAME}:revision:{final_identity}" - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, - connector, - delivery_key, - delivery_kind, - turn_id, - content_revision, - status, - payload_json, - private_state_json, - created_at, - updated_at, - next_attempt_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, '{}', ?, ?, NULL) - ON CONFLICT(host_id, connector, delivery_key) DO UPDATE SET - delivery_kind = excluded.delivery_kind, - turn_id = excluded.turn_id, - content_revision = excluded.content_revision, - status = excluded.status, - payload_json = excluded.payload_json, - updated_at = excluded.updated_at, - next_attempt_at = NULL - """, - ( - str(host_id), - _TURN_FINAL_NAME, - delivery_key, - delivery_kind, - str(turn_id), - str(revision), - status, - _canonical_json(payload), - str(revision_created_at), - classified_at, - ), - ) - source_row = conn.execute( - """ - SELECT id - FROM connector_outbox - WHERE host_id = ? AND connector = ? AND delivery_key = ? - """, - (str(host_id), _TURN_FINAL_NAME, delivery_key), - ).fetchone() - if source_row is None: - raise StoreSchemaError("legacy_final_anchor_unavailable") - source_outbox_id = int(source_row[0]) - if linkable: - placeholders = ",".join("?" for _ in linkable) - conn.execute( - f""" - UPDATE turn_presentation_plans - SET source_outbox_id = ? - WHERE id IN ({placeholders}) - """, - (source_outbox_id, *linkable), - ) - elif routable and proven is not None: - conn.execute( - """ - UPDATE turn_presentation_plans - SET source_outbox_id = ? - WHERE id = ? - """, - (source_outbox_id, int(proven[0])), - ) - for statement in CREATE_FINAL_DELIVERY_INDEXES: - conn.execute(statement) - - -def _legacy_command_timestamp( - values: Iterable[Any], - *, - latest: bool, -) -> str: - candidates = sorted(str(value) for value in values if str(value or "").strip()) - if not candidates: - return "1970-01-01T00:00:00+00:00" - return candidates[-1] if latest else candidates[0] - - -def _legacy_public_worker_id(request_json: Any) -> str: - request = _json_object(request_json) - target = request.get("target") - if not isinstance(target, Mapping): - return "" - return str(target.get("worker_id") or "") - - -def _migrate_v11_to_v12_conn(conn: sqlite3.Connection) -> None: - """Rebuild action-scoped legacy rows into one fail-closed host request.""" - receipt_columns = _table_columns(conn, "command_receipts") - command_columns = _table_columns(conn, "commands") - current_receipt_columns = { - "canonical_version", - "canonical_fingerprint", - "canonical_request_json", - "public_worker_id", - "state", - "owner_token_hash", - "owner_expires_at", - "binding_fingerprint", - "reserved_at", - "send_started_at", - "terminal_at", - "updated_at", - "legacy_collision", - "legacy_collision_count", - } - current_command_columns = { - "canonical_version", - "canonical_fingerprint", - "public_worker_id", - "state", - "send_started_at", - "terminal_at", - "legacy_collision", - "legacy_collision_count", - } - if current_receipt_columns <= receipt_columns: - if not current_command_columns <= command_columns: - raise StoreSchemaError("legacy_command_request_schema_ambiguous") - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - for statement in CREATE_COMMAND_INDEXES: - conn.execute(statement) - return - required_receipt_columns = { - "id", - "host_id", - "request_id", - "action", - "payload_fingerprint", - "status", - "result_json", - "created_at", - "completed_at", - "uncertain", - } - required_command_columns = { - "id", - "host_id", - "request_id", - "action", - "payload_fingerprint", - "status", - "uncertain", - "request_json", - "result_json", - "created_at", - "reserved_at", - "completed_at", - "updated_at", - } - if ( - not required_receipt_columns <= receipt_columns - or not required_command_columns <= command_columns - ): - raise StoreSchemaError("legacy_command_request_schema_ambiguous") - conflicting_tables = { - str(row[0]) - for row in conn.execute( - """ - SELECT name - FROM sqlite_master - WHERE type = 'table' - AND name IN ('command_receipts_v11', 'commands_v11') - """ - ).fetchall() - } - if conflicting_tables: - raise StoreSchemaError("legacy_command_request_schema_ambiguous") - - grouped: dict[tuple[str, str], list[dict[str, Any]]] = {} - receipt_rows = conn.execute( - """ - SELECT - id, host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - FROM command_receipts - WHERE TRIM(host_id) <> '' AND TRIM(request_id) <> '' - ORDER BY host_id, request_id, id - """ - ).fetchall() - for row in receipt_rows: - key = (str(row[1]), str(row[2])) - grouped.setdefault(key, []).append( - { - "source": "receipt", - "id": int(row[0]), - "action": str(row[3]), - "fingerprint": str(row[4]), - "status": str(row[5]), - "result_json": str(row[6]), - "created_at": str(row[7]), - "terminal_at": row[8], - "updated_at": row[8] or row[7], - "uncertain": bool(row[9]), - "public_worker_id": "", - } - ) - command_rows = conn.execute( - """ - SELECT - id, host_id, request_id, action, payload_fingerprint, status, - result_json, request_json, created_at, completed_at, updated_at, - uncertain - FROM commands - WHERE TRIM(host_id) <> '' AND TRIM(request_id) <> '' - ORDER BY host_id, request_id, id - """ - ).fetchall() - for row in command_rows: - key = (str(row[1]), str(row[2])) - grouped.setdefault(key, []).append( - { - "source": "command", - "id": int(row[0]), - "action": str(row[3]), - "fingerprint": str(row[4]), - "status": str(row[5]), - "result_json": str(row[6]), - "created_at": str(row[8]), - "terminal_at": row[9], - "updated_at": row[10] or row[9] or row[8], - "uncertain": bool(row[11]), - "public_worker_id": _legacy_public_worker_id(row[7]), - } - ) - - normalized: list[tuple[Any, ...]] = [] - for (host_id, request_id), rows in sorted(grouped.items()): - rows.sort(key=lambda item: (str(item["source"]), int(item["id"]))) - pairs = { - (str(item["action"]), str(item["fingerprint"])) - for item in rows - } - evidence = { - ( - str(item["status"]), - str(item["result_json"]), - bool(item["uncertain"]), - ) - for item in rows - } - public_worker_ids = { - str(item["public_worker_id"]) - for item in rows - if str(item["public_worker_id"]) - } - malformed = any( - not str(item["action"]).strip() - or not str(item["fingerprint"]).strip() - for item in rows - ) - collision = ( - malformed - or len(pairs) != 1 - or len(evidence) != 1 - or len(public_worker_ids) > 1 - ) - created_at = _legacy_command_timestamp( - (item["created_at"] for item in rows), - latest=False, - ) - terminal_at = _legacy_command_timestamp( - ( - item["terminal_at"] or item["updated_at"] or item["created_at"] - for item in rows - ), - latest=True, - ) - if collision: - action = "legacy_collision" - fingerprint = "legacy-collision" - state = "uncertain" - status = "request_state_uncertain" - result_json = ( - '{"ok":false,"status":"request_state_uncertain"}' - ) - collision_count = max(2, len(rows)) - public_worker_id = "" - else: - action, fingerprint = next(iter(pairs)) - first = rows[0] - legacy_status = str(first["status"]) - result_json = str(first["result_json"]) - uncertain = any(bool(item["uncertain"]) for item in rows) - if ( - uncertain - or legacy_status in {"pending", "request_state_uncertain"} - ): - state = "uncertain" - status = "request_state_uncertain" - elif legacy_status == "accepted": - state = "accepted" - status = "accepted" - else: - state = "rejected" - status = legacy_status or "legacy_rejected" - collision_count = 0 - public_worker_id = ( - next(iter(public_worker_ids)) if public_worker_ids else "" - ) - send_started_at = created_at if state == "accepted" else None - normalized.append( - ( - host_id, - request_id, - action, - 0, - fingerprint, - "{}", - public_worker_id, - state, - status, - result_json, - created_at, - created_at, - send_started_at, - terminal_at, - terminal_at, - int(collision), - collision_count, - ) - ) - - conn.execute("ALTER TABLE command_receipts RENAME TO command_receipts_v11") - conn.execute("ALTER TABLE commands RENAME TO commands_v11") - conn.execute(CREATE_COMMAND_RECEIPTS_TABLE) - conn.execute(CREATE_COMMANDS_TABLE) - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - for statement in CREATE_COMMAND_INDEXES: - conn.execute(statement) - for record in normalized: - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at, legacy_collision, - legacy_collision_count - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', NULL, NULL, ?, ?, ?, ?, ?, ?, ?) - """, - record, - ) - row = _command_request_row(conn, str(record[0]), str(record[1])) - if row is None: - raise StoreSchemaError("legacy_command_request_migration_failed") - _project_command_request_conn(conn, row) - conn.execute("DROP TABLE command_receipts_v11") - conn.execute("DROP TABLE commands_v11") - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - for statement in CREATE_COMMAND_INDEXES: - conn.execute(statement) - - -def _migrate_v12_to_v13_conn(conn: sqlite3.Connection) -> None: - """Add private selector-proof evidence without inventing it for old rows. - - A v12 receipt records the worker a request resolved to, never how the caller - spelled that target, so no existing row's selector can be reconstructed. Any - guess here would let a changed target replay an unrelated accepted result. - Every legacy row therefore keeps an empty proof, which the submission path - reads as "cannot prove an alias retry" and fails closed on. - """ - columns = _table_columns(conn, "command_receipts") - if not columns: - raise StoreSchemaError("legacy_command_request_schema_ambiguous") - if "selector_proof" not in columns: - conn.execute( - "ALTER TABLE command_receipts " - "ADD COLUMN selector_proof TEXT NOT NULL DEFAULT ''" - ) - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - - -def _migrate_v13_to_v14_conn(conn: sqlite3.Connection) -> None: - """Repair legacy nonpositive turn-list coordinates and reject recurrence.""" - columns = _table_columns(conn, "turns") - if not columns: - return - if "list_sequence" not in columns: - raise StoreSchemaError("legacy_turn_list_schema_ambiguous") - affected_hosts = [ - str(row[0]) - for row in conn.execute( - """ - SELECT DISTINCT host_id - FROM turns - WHERE list_sequence <= 0 - ORDER BY host_id - """ - ).fetchall() - ] - for host_id in affected_hosts: - _ensure_turn_list_host_state_conn(conn, host_id) - state = conn.execute( - """ - SELECT next_sequence - FROM turn_list_hosts - WHERE host_id = ? - """, - (host_id,), - ).fetchone() - if state is None: - raise StoreSchemaError("turn_list_host_state_unavailable") - high_row = conn.execute( - """ - SELECT COALESCE(MAX(list_sequence), 0) - FROM turns - WHERE host_id = ? AND list_sequence > 0 - """, - (host_id,), - ).fetchone() - next_sequence = max(int(state[0]), int(high_row[0]) + 1) - invalid_rows = conn.execute( - """ - SELECT turn_id - FROM turns - WHERE host_id = ? AND list_sequence <= 0 - ORDER BY COALESCE(updated_at, observed_at, ''), turn_id - """, - (host_id,), - ).fetchall() - for row in invalid_rows: - conn.execute( - """ - UPDATE turns - SET list_sequence = ? - WHERE host_id = ? AND turn_id = ? AND list_sequence <= 0 - """, - (next_sequence, host_id, str(row[0])), - ) - next_sequence += 1 - conn.execute( - """ - UPDATE turn_list_hosts - SET next_sequence = ?, - traversal_generation = traversal_generation + 1 - WHERE host_id = ? - """, - (next_sequence, host_id), - ) - for statement in CREATE_TURN_LIST_SEQUENCE_TRIGGERS: - conn.execute(statement) - - -def _migrate_v14_to_v15_conn(conn: sqlite3.Connection) -> None: - """Tombstone duplicate and stale command claims without rekeying turns.""" - if not _table_columns(conn, "turns"): - return - now = utc_timestamp() - now_dt = datetime.fromisoformat(now) - rows = conn.execute( - """ - SELECT turns.host_id, turns.turn_id, turns.worker_id, - turns.payload_json, turns.observed_at, - revisions.user_text, revisions.assistant_final_text, - revisions.user_state, revisions.final_state - FROM turns - LEFT JOIN turn_content_revisions AS revisions - ON revisions.host_id = turns.host_id - AND revisions.turn_id = turns.turn_id - AND revisions.is_current = 1 - """ - ).fetchall() - decoded = [] - for row in rows: - payload = _json_object(row[3]) - current = ( - { - "user_text": row[5], - "assistant_final_text": row[6], - "user_state": str(row[7]), - "final_state": str(row[8]), - } - if row[7] is not None - else None - ) - decoded.append( - ( - str(row[0]), - str(row[1]), - str(row[2]), - payload, - current, - str(row[4] or ""), - ) - ) - claims = [ - row - for row in decoded - if str(row[3].get("source") or "") == "command" - and not str(row[3].get("source_turn_id") or "").strip() - and not _turn_is_tombstoned(row[3]) - and row[3].get("complete") is not True - ] - done = [ - row - for row in decoded - if str(row[3].get("source_turn_id") or "").strip() - and not _turn_is_tombstoned(row[3]) - and ( - row[3].get("complete") is True - or row[4] is not None - and str(row[4].get("final_state") or "") == "complete" - ) - ] - affected_hosts: set[str] = set() - used_done = { - str(row[3].get("superseded_by_turn_id") or "") - for row in decoded - if _turn_is_tombstoned(row[3]) - and str(row[3].get("superseded_by_turn_id") or "").strip() - } - for claim in claims: - claim_view = _turn_with_current_content(claim[3], claim[4]) - matches = [ - observed - for observed in done - if observed[0] == claim[0] - and observed[2] == claim[2] - and observed[1] not in used_done - and _turn_content_matches_origin( - _turn_with_current_content(observed[3], observed[4]), - claim_view, - ) - ] - if len(matches) != 1: - continue - matching_claims = [ - candidate - for candidate in claims - if candidate[0] == claim[0] - and candidate[2] == claim[2] - and _turn_content_matches_origin( - _turn_with_current_content(matches[0][3], matches[0][4]), - _turn_with_current_content(candidate[3], candidate[4]), - ) - ] - if len(matching_claims) != 1: - continue - if _migrate_tombstone_command_turn_conn( - conn, - claim[0], - claim[1], - superseded_by_turn_id=matches[0][1], - superseded_at=now, - ): - affected_hosts.add(claim[0]) - used_done.add(matches[0][1]) - - configured_hard_ttl = _LEGACY_TURN_CLAIM_HARD_TTL_SECONDS - for claim in claims: - stored = conn.execute( - "SELECT payload_json FROM turns WHERE host_id = ? AND turn_id = ?", - (claim[0], claim[1]), - ).fetchone() - if stored is None or _turn_is_tombstoned(_json_object(stored[0])): - continue - claim_dt = _turn_row_time(claim[3], claim[5]) - if claim_dt is None or (now_dt - claim_dt).total_seconds() < configured_hard_ttl: - continue - if _migrate_tombstone_command_turn_conn( - conn, - claim[0], - claim[1], - superseded_by_turn_id=None, - superseded_at=now, - ): - affected_hosts.add(claim[0]) - for host_id in sorted(affected_hosts): - _increment_turn_list_generation_conn(conn, host_id) - - -def _legacy_outbox_ordering_key( - *, - outbox_id: int, - outbox_payload: Mapping[str, Any], - worker_id: Any, - turn_payload: Mapping[str, Any], -) -> str: - nested_turn = outbox_payload.get("turn") - route = dict(nested_turn) if isinstance(nested_turn, Mapping) else outbox_payload - route_meta = _json_object(route.get("meta")) - route_stable_key = route.get("stable_key") or route_meta.get("stable_key") - route_stable_key_version = ( - route.get("stable_key_version") - if route.get("stable_key") is not None - else route_meta.get("stable_key_version") - ) - if ( - _valid_final_stable_key(route_stable_key) - and type(route_stable_key_version) is int - and route_stable_key_version == 1 - ): - return str(route_stable_key) - meta = _json_object(turn_payload.get("meta")) - stable_key = meta.get("stable_key") - if ( - _valid_final_stable_key(stable_key) - and type(meta.get("stable_key_version")) is int - and meta.get("stable_key_version") == 1 - ): - return str(stable_key) - return str(worker_id or route.get("worker_id") or f"orphan:{outbox_id}") - - -def _migrate_v15_to_v16_conn( - conn: sqlite3.Connection, - *, - connector_ack_ttl_seconds: int = CONNECTOR_ACK_TTL_SECONDS, -) -> None: - """Partition final FIFO order and bound legacy awaiting-ack plans.""" - columns = _table_columns(conn, "connector_outbox") - if not columns: - conn.execute(CREATE_CONNECTOR_OUTBOX_TABLE) - elif "ordering_key" not in columns: - conn.execute( - "ALTER TABLE connector_outbox " - "ADD COLUMN ordering_key TEXT NOT NULL DEFAULT ''" - ) - if _table_columns(conn, "turns"): - rows = conn.execute( - """ - SELECT outbox.id, outbox.payload_json, turns.worker_id, turns.payload_json - FROM connector_outbox AS outbox - LEFT JOIN turns - ON turns.host_id = outbox.host_id - AND turns.turn_id = outbox.turn_id - WHERE outbox.ordering_key = '' - ORDER BY outbox.id - """ - ).fetchall() - else: - rows = [ - (row[0], row[1], None, None) - for row in conn.execute( - """ - SELECT id, payload_json - FROM connector_outbox - WHERE ordering_key = '' - ORDER BY id - """ - ).fetchall() - ] - for outbox_id, outbox_payload, worker_id, turn_payload in rows: - conn.execute( - "UPDATE connector_outbox SET ordering_key = ? WHERE id = ?", - ( - _legacy_outbox_ordering_key( - outbox_id=int(outbox_id), - outbox_payload=_json_object(outbox_payload), - worker_id=worker_id, - turn_payload=_json_object(turn_payload), - ), - int(outbox_id), - ), - ) - deadline = _connector_add_seconds( - utc_timestamp(), - max(1, int(connector_ack_ttl_seconds)), - ) - awaiting_rows = conn.execute( - "SELECT id, private_state_json FROM connector_outbox WHERE status = 'awaiting_ack'" - ).fetchall() - for outbox_id, private_state_json in awaiting_rows: - state = _json_object(private_state_json) - state["ack_deadline_at"] = deadline - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = ?, next_attempt_at = ? - WHERE id = ? - """, - (_canonical_json(state), deadline, int(outbox_id)), - ) - delivery_rows = ( - conn.execute( - """ - SELECT id, private_state_json - FROM connector_deliveries - WHERE outbox_id = ? AND status = 'awaiting_ack' - """, - (int(outbox_id),), - ).fetchall() - if _table_columns(conn, "connector_deliveries") - else [] - ) - for delivery_id, delivery_private in delivery_rows: - delivery_state = _json_object(delivery_private) - delivery_state["ack_deadline_at"] = deadline - conn.execute( - "UPDATE connector_deliveries SET private_state_json = ? WHERE id = ?", - (_canonical_json(delivery_state), int(delivery_id)), - ) - conn.execute(CREATE_CONNECTOR_ORDERING_INDEX) - - -def _migrate_v16_to_v17_conn(conn: sqlite3.Connection) -> None: - """Give unresolved legacy outbox rows independent FIFO partitions.""" - if _table_columns(conn, "turns"): - rows = conn.execute( - """ - SELECT outbox.id, outbox.payload_json, - turns.worker_id, turns.payload_json - FROM connector_outbox AS outbox - LEFT JOIN turns - ON turns.host_id = outbox.host_id - AND turns.turn_id = outbox.turn_id - WHERE outbox.ordering_key = '' - ORDER BY outbox.id - """ - ).fetchall() - else: - rows = [ - (row[0], row[1], None, None) - for row in conn.execute( - """ - SELECT id, payload_json - FROM connector_outbox - WHERE ordering_key = '' - ORDER BY id - """ - ).fetchall() - ] - for outbox_id, outbox_payload, worker_id, turn_payload in rows: - conn.execute( - "UPDATE connector_outbox SET ordering_key = ? WHERE id = ?", - ( - _legacy_outbox_ordering_key( - outbox_id=int(outbox_id), - outbox_payload=_json_object(outbox_payload), - worker_id=worker_id, - turn_payload=_json_object(turn_payload), - ), - int(outbox_id), - ), - ) - - -def _create_available_turn_change_triggers_conn(conn: sqlite3.Connection) -> None: - """Install capture triggers whose legacy source tables are present.""" - has_turns = bool(_table_columns(conn, "turns")) - has_revisions = bool(_table_columns(conn, "turn_content_revisions")) - for index, statement in enumerate(CREATE_TURN_CHANGE_TRIGGERS): - if index < 3 and not has_turns: - continue - if index in {3, 4} and not (has_turns and has_revisions): - continue - conn.execute(statement) - - -def _migrate_v17_to_v18_conn(conn: sqlite3.Connection) -> None: - """Install the empty, trigger-backed public turn change journal.""" - conn.execute(CREATE_TURN_CHANGE_JOURNAL_TABLE) - conn.execute(CREATE_TURN_CHANGE_FLOOR_TABLE) - conn.execute(CREATE_TURN_CHANGE_STATE_TABLE) - for statement in CREATE_TURN_CHANGE_INDEXES: - conn.execute(statement) - _ensure_turn_change_state_conn(conn) - _create_available_turn_change_triggers_conn(conn) - - -def _migrate_v18_to_v19_conn(conn: sqlite3.Connection) -> None: - """Install empty Phase 2 submission and supersession ledgers.""" - conn.execute(CREATE_TURN_SUBMISSIONS_TABLE) - conn.execute(CREATE_TURN_SUPERSESSIONS_TABLE) - for statement in CREATE_TURN_SUBMISSION_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_SUPERSESSION_INDEXES: - conn.execute(statement) - - -def _backfill_submission_state(receipt_state: Any, receipt_status: Any) -> str | None: - """Map a historical send receipt to the shadow-ledger state it earned.""" - state = str(receipt_state or "").strip().lower() - status = str(receipt_status or "").strip().lower() - if state == "purged" or status == "purged": - return None - if state in {"rejected", "cancelled", "canceled"} or status in { - "rejected", - "cancelled", - "canceled", - }: - return "cancelled" - if state == "send_started": - return "send_started" - if state == "accepted": - return "submitted" - if state == "uncertain": - return "uncertain" - # A reservation has not crossed the send boundary, so Stage 2 would not - # have created a submission row for it. Unknown legacy states fail closed. - return None - - -def _receipt_instruction_text(canonical_request_json: Any) -> str | None: - """Recover only a validated canonical send-instruction payload.""" - try: - payload = json.loads(str(canonical_request_json)) - except (TypeError, json.JSONDecodeError): - return None - if not isinstance(payload, Mapping) or payload.get("action") != "send_instruction": - return None - instruction = payload.get("instruction") - if not isinstance(instruction, Mapping): - return None - text = instruction.get("text") - if not isinstance(text, str) or validate_instruction_text(text) is not None: - return None - return text - - -def _backfill_turn_submissions_conn(conn: sqlite3.Connection) -> None: - """Backfill historical send receipts without changing live dual-write rows.""" - required_columns = { - "id", - "host_id", - "request_id", - "action", - "canonical_request_json", - "public_worker_id", - "state", - "status", - "created_at", - "reserved_at", - "send_started_at", - "terminal_at", - "updated_at", - } - if not required_columns <= _table_columns(conn, "command_receipts"): - return - hosts = [ - str(row[0]) - for row in conn.execute( - """ - SELECT DISTINCT host_id - FROM command_receipts - WHERE action = 'send_instruction' AND TRIM(host_id) <> '' - ORDER BY host_id - """ - ).fetchall() - ] - for host_id in hosts: - after_id = 0 - while True: - rows = conn.execute( - """ - SELECT id, request_id, canonical_request_json, - public_worker_id, state, status, created_at, - reserved_at, send_started_at, terminal_at, updated_at - FROM command_receipts - WHERE host_id = ? AND action = 'send_instruction' AND id > ? - ORDER BY id - LIMIT ? - """, - (host_id, after_id, TURN_LEDGER_BACKFILL_BATCH_SIZE), - ).fetchall() - if not rows: - break - after_id = int(rows[-1][0]) - for row in rows: - request_id = str(row[1] or "").strip() - public_worker_id = str(row[3] or "").strip() - ledger_state = _backfill_submission_state(row[4], row[5]) - instruction_text = _receipt_instruction_text(row[2]) - if ( - not request_id - or not public_worker_id - or ledger_state is None - or instruction_text is None - ): - continue - - anchor = _strict_utc_timestamp(row[8] or row[7] or row[6]) - updated_at = _strict_utc_timestamp(row[10]) - if anchor is None or updated_at is None: - continue - anchor_time = datetime.fromisoformat(anchor) - link_not_before = ( - anchor_time - - timedelta(seconds=SUBMISSION_LINK_WINDOW_SECONDS) - ).isoformat(timespec="seconds") - link_expires_at = ( - anchor_time - + timedelta(seconds=SUBMISSION_LINK_WINDOW_SECONDS) - ).isoformat(timespec="seconds") - hard_expires_at = ( - anchor_time - + timedelta(seconds=SUBMISSION_HARD_TTL_SECONDS) - ).isoformat(timespec="seconds") - terminal_at = ( - _strict_utc_timestamp(row[9]) - if ledger_state in {"submitted", "uncertain", "cancelled"} - else None - ) - if ( - ledger_state in {"submitted", "uncertain", "cancelled"} - and terminal_at is None - ): - continue - send_started_at = ( - _strict_utc_timestamp(row[8]) if row[8] is not None else None - ) - conn.execute( - """ - INSERT INTO turn_submissions ( - host_id, submission_id, request_id, owner_key, - owner_key_version, instruction_fingerprint, state, - linked_turn_id, link_not_before, link_expires_at, - hard_expires_at, linked_at, terminal_at, submitted_at, - send_started_at, updated_at - ) VALUES ( - ?, ?, ?, ?, 0, ?, ?, NULL, ?, ?, ?, NULL, ?, ?, ?, ? - ) - ON CONFLICT DO NOTHING - """, - ( - host_id, - turn_submission_id(host_id, request_id), - request_id, - f"legacy-worker:{public_worker_id}", - instruction_fingerprint(instruction_text), - ledger_state, - link_not_before, - link_expires_at, - hard_expires_at, - terminal_at, - terminal_at if ledger_state == "submitted" else None, - send_started_at, - updated_at, - ), - ) - - -def _linked_canonical_turn_id_conn( - conn: sqlite3.Connection, - host_id: str, - replacement_turn_id: Any, -) -> str | None: - """Follow only explicit tombstone links to a source-observed identity.""" - current = str(replacement_turn_id or "").strip() - seen: set[str] = set() - while current and current not in seen: - seen.add(current) - row = conn.execute( - """ - SELECT payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (str(host_id), current), - ).fetchone() - if row is None: - return None - payload = _json_object(row[0]) - if _turn_is_tombstoned(payload): - current = str(payload.get("superseded_by_turn_id") or "").strip() - continue - if not str(payload.get("source_turn_id") or "").strip(): - return None - # Phase 1 freezes an adopted command row's published turn_id instead - # of re-keying it after source_turn_id is learned. The row we just - # resolved is therefore the only canonical identity we can prove. - return current - return None - + return bool(cursor.rowcount) def _resolve_canonical_turn_id_conn( conn: sqlite3.Connection, host_id: str, turn_id: Any, ) -> str | None: - """Resolve a public legacy turn alias without guessing through bad rows.""" + """Resolve a superseded public turn alias without guessing through cycles.""" current = str(turn_id or "").strip() if not current: return None - if not { - "host_id", - "superseded_turn_id", - "canonical_turn_id", - } <= _table_columns(conn, "turn_supersessions"): - return current seen: set[str] = set() while current and current not in seen: seen.add(current) @@ -13026,283 +10091,140 @@ def _resolve_canonical_turn_id_conn( return None -def _backfill_turn_supersessions_conn(conn: sqlite3.Connection) -> None: - """Alias only legacy command turns with deterministic Phase 1 linkage.""" - if not {"host_id", "turn_id", "payload_json", "observed_at", "list_sequence"} <= ( - _table_columns(conn, "turns") - ): - return - hosts = [ - str(row[0]) - for row in conn.execute( - """ - SELECT DISTINCT host_id - FROM turns - WHERE json_extract(payload_json, '$.source') = 'command' - ORDER BY host_id - """ - ).fetchall() - ] - for host_id in hosts: - after_sequence = 0 - while True: - rows = conn.execute( - """ - SELECT turn_id, payload_json, observed_at, list_sequence - FROM turns - WHERE host_id = ? - AND list_sequence > ? - AND json_extract(payload_json, '$.source') = 'command' - ORDER BY list_sequence - LIMIT ? - """, - (host_id, after_sequence, TURN_LEDGER_BACKFILL_BATCH_SIZE), - ).fetchall() - if not rows: - break - after_sequence = int(rows[-1][3]) - for turn_id, payload_json, observed_at, _sequence in rows: - legacy_turn_id = str(turn_id) - payload = _json_object(payload_json) - # A live adopted command turn was never superseded. Its row ID - # is deliberately frozen by Phase 1, so recomputing a Turn ID - # from its updated payload could only invent a dangling alias. - if not _turn_is_tombstoned(payload): - continue - canonical_turn_id = _linked_canonical_turn_id_conn( - conn, - host_id, - payload.get("superseded_by_turn_id"), - ) - if not canonical_turn_id or canonical_turn_id == legacy_turn_id: - continue - created_at = _strict_utc_timestamp( - payload.get("superseded_at") - or payload.get("updated_at") - or observed_at - ) - if created_at is None: - continue - conn.execute( - """ - INSERT INTO turn_supersessions ( - host_id, superseded_turn_id, canonical_turn_id, - reason, created_at - ) VALUES (?, ?, ?, 'phase1_migration', ?) - ON CONFLICT DO NOTHING - """, - ( - host_id, - legacy_turn_id, - canonical_turn_id, - created_at, - ), - ) - - -def _migrate_v19_to_v20_conn(conn: sqlite3.Connection) -> None: - """Backfill Phase 1 history into the non-authoritative Phase 2 ledgers.""" - # The pre-Phase-2 production lineage also used user_version 19, but did - # not contain either ledger table. Repair that valid legacy shape before - # backfilling instead of assuming every v19 store passed through this - # branch's v18 -> v19 migration. - conn.execute(CREATE_TURN_SUBMISSIONS_TABLE) - conn.execute(CREATE_TURN_SUPERSESSIONS_TABLE) - for statement in CREATE_TURN_SUBMISSION_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_SUPERSESSION_INDEXES: - conn.execute(statement) - _backfill_turn_submissions_conn(conn) - _backfill_turn_supersessions_conn(conn) - -def _migrate_v20_to_v21_conn(conn: sqlite3.Connection) -> None: - """Add durable Herdr turn replay watermarks and completion provenance.""" - conn.execute(CREATE_HERDR_TURN_WATERMARKS_TABLE) - conn.execute(CREATE_HERDR_TURN_COMPLETIONS_TABLE) - for statement in CREATE_HERDR_TURN_INDEXES: +def _create_current_schema_objects_conn(conn: sqlite3.Connection) -> None: + table_statements = ( + CREATE_SNAPSHOTS_TABLE, + CREATE_COMMAND_RECEIPTS_TABLE, + CREATE_WORKER_BINDINGS_TABLE, + *CREATE_CURRENT_PR6_TABLES, + CREATE_BACKEND_PENDING_TABLE, + CREATE_BACKEND_PENDING_CLAIMS_TABLE, + CREATE_ATTENTION_LIFECYCLES_TABLE, + CREATE_TURN_CONTENT_REVISIONS_TABLE, + CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE, + CREATE_TURN_PRESENTATION_PLANS_TABLE, + CREATE_TURN_PRESENTATION_JOBS_TABLE, + CREATE_TURN_PRESENTATION_RECOVERIES_TABLE, + CREATE_STORE_MAINTENANCE_STATE_TABLE, + CREATE_STORE_MAINTENANCE_CURSORS_TABLE, + CREATE_TURN_LIST_STATE_TABLE, + CREATE_TURN_LIST_HOSTS_TABLE, + CREATE_TURN_CHANGE_JOURNAL_TABLE, + CREATE_TURN_CHANGE_FLOOR_TABLE, + CREATE_TURN_CHANGE_STATE_TABLE, + CREATE_TURN_SUBMISSIONS_TABLE, + CREATE_TURN_SUPERSESSIONS_TABLE, + CREATE_AGENT_EVENTS_TABLE, + ) + index_statements = ( + *CREATE_COMMAND_RECEIPT_INDEXES, + *CREATE_WORKER_BINDING_INDEXES, + CREATE_WORKER_BINDING_UNIQUE_INDEX, + *CREATE_CURRENT_PR6_INDEXES, + *CREATE_TURN_LIST_INDEXES, + *CREATE_TURN_LIST_SEQUENCE_TRIGGERS, + *CREATE_TURN_CHANGE_INDEXES, + *CREATE_TURN_CHANGE_TRIGGERS, + *CREATE_TURN_SUBMISSION_INDEXES, + *CREATE_TURN_SUPERSESSION_INDEXES, + *CREATE_AGENT_EVENT_INDEXES, + *CREATE_ATTENTION_LIFECYCLE_INDEXES, + *CREATE_TURN_CONTENT_REVISION_INDEXES, + *CREATE_TURN_PRESENTATION_INDEXES, + *CREATE_FINAL_DELIVERY_INDEXES, + CREATE_CONNECTOR_ORDERING_INDEX, + *CREATE_SNAPSHOT_INDEXES, + ) + for statement in (*table_statements, *index_statements): conn.execute(statement) - - -MIGRATIONS: tuple[Migration, ...] = ( - Migration(0, 1, _migrate_v0_to_v1_conn), - Migration(1, 2, _migrate_v1_to_v2_conn), - Migration(2, 3, _migrate_v2_to_v3_conn), - Migration(3, 4, _migrate_v3_to_v4_conn), - Migration(4, 5, _migrate_v4_to_v5_conn), - Migration(5, 6, _migrate_v5_to_v6_conn), - Migration(6, 7, _migrate_v6_to_v7_conn), - Migration(7, 8, _migrate_v7_to_v8_conn), - Migration(8, 9, _migrate_v8_to_v9_conn), - Migration(9, 10, _migrate_v9_to_v10_conn), - Migration(10, 11, _migrate_v10_to_v11_conn), - Migration(11, 12, _migrate_v11_to_v12_conn), - Migration(12, 13, _migrate_v12_to_v13_conn), - Migration(13, 14, _migrate_v13_to_v14_conn), - Migration(14, 15, _migrate_v14_to_v15_conn), - Migration(15, 16, _migrate_v15_to_v16_conn), - Migration(16, 17, _migrate_v16_to_v17_conn), - Migration(17, 18, _migrate_v17_to_v18_conn), - Migration(18, 19, _migrate_v18_to_v19_conn), - Migration(19, 20, _migrate_v19_to_v20_conn), - Migration(20, 21, _migrate_v20_to_v21_conn), -) - - -def _validate_migration_registry( - migrations: tuple[Migration, ...] | None = None, - *, - target_version: int = STORE_SCHEMA_VERSION, -) -> None: - registry = MIGRATIONS if migrations is None else migrations - expected = 0 - for migration in registry: - if ( - migration.from_version != expected - or migration.to_version != expected + 1 - ): - raise RuntimeError("invalid migration registry") - expected = migration.to_version - if expected != STORE_SCHEMA_VERSION: - raise RuntimeError("invalid migration registry target") - if not 0 <= int(target_version) <= STORE_SCHEMA_VERSION: - raise RuntimeError("unsupported migration target") + conn.execute(INSERT_STORE_MAINTENANCE_STATE) + conn.execute( + "INSERT INTO turn_list_state(scope, store_epoch) VALUES (?, ?)", + ("turn-list", secrets.token_urlsafe(32)), + ) + conn.execute( + "INSERT INTO turn_change_state(scope, store_epoch) VALUES (?, ?)", + ("turn-delta", secrets.token_urlsafe(32)), + ) + conn.execute(f"PRAGMA user_version = {STORE_SCHEMA_VERSION}") def _create_current_schema_conn(conn: sqlite3.Connection) -> None: """Create an empty database directly at the current schema.""" if conn.in_transaction: - raise StoreSchemaError("schema_migration_in_transaction") + raise StoreSchemaError("schema_rebuild_in_transaction") conn.execute("BEGIN IMMEDIATE") try: - conn.execute(CREATE_SNAPSHOTS_TABLE) - conn.execute(CREATE_COMMAND_RECEIPTS_TABLE) - conn.execute(CREATE_WORKER_BINDINGS_TABLE) - for statement in CREATE_CURRENT_PR6_TABLES: - conn.execute(statement) - conn.execute(CREATE_LEGACY_BACKEND_PENDING_TABLE) - _migrate_v9_to_v10_conn(conn) - conn.execute(CREATE_ATTENTION_LIFECYCLES_TABLE) - conn.execute(CREATE_TURN_CONTENT_REVISIONS_TABLE) - conn.execute(CREATE_TURN_CONTENT_PAGE_BOUNDARIES_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_PLANS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_JOBS_TABLE) - conn.execute(CREATE_TURN_PRESENTATION_RECOVERIES_TABLE) - conn.execute(CREATE_STORE_MAINTENANCE_STATE_TABLE) - conn.execute(CREATE_STORE_MAINTENANCE_CURSORS_TABLE) - conn.execute(CREATE_TURN_LIST_STATE_TABLE) - conn.execute(CREATE_TURN_LIST_HOSTS_TABLE) - conn.execute(CREATE_TURN_CHANGE_JOURNAL_TABLE) - conn.execute(CREATE_TURN_CHANGE_FLOOR_TABLE) - conn.execute(CREATE_TURN_CHANGE_STATE_TABLE) - conn.execute(CREATE_TURN_SUBMISSIONS_TABLE) - conn.execute(CREATE_TURN_SUPERSESSIONS_TABLE) - conn.execute(CREATE_HERDR_TURN_WATERMARKS_TABLE) - conn.execute(CREATE_HERDR_TURN_COMPLETIONS_TABLE) - for statement in CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - for statement in CREATE_WORKER_BINDING_INDEXES: - conn.execute(statement) - conn.execute(CREATE_WORKER_BINDING_UNIQUE_INDEX) - for statement in CREATE_CURRENT_PR6_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_LIST_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_LIST_SEQUENCE_TRIGGERS: - conn.execute(statement) - for statement in CREATE_TURN_CHANGE_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_CHANGE_TRIGGERS: - conn.execute(statement) - for statement in CREATE_TURN_SUBMISSION_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_SUPERSESSION_INDEXES: - conn.execute(statement) - for statement in CREATE_HERDR_TURN_INDEXES: - conn.execute(statement) - for statement in CREATE_ATTENTION_LIFECYCLE_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_CONTENT_REVISION_INDEXES: - conn.execute(statement) - for statement in CREATE_TURN_PRESENTATION_INDEXES: - conn.execute(statement) - for statement in CREATE_FINAL_DELIVERY_INDEXES: - conn.execute(statement) - conn.execute(CREATE_CONNECTOR_ORDERING_INDEX) - for statement in CREATE_SNAPSHOT_INDEXES: - conn.execute(statement) - conn.execute(INSERT_STORE_MAINTENANCE_STATE) - _ensure_turn_list_state_conn(conn) - _ensure_turn_change_state_conn(conn) - conn.execute(f"PRAGMA user_version = {STORE_SCHEMA_VERSION}") + _create_current_schema_objects_conn(conn) conn.commit() except Exception: conn.rollback() raise -def _database_has_application_objects(conn: sqlite3.Connection) -> bool: - return ( - conn.execute( - """ - SELECT 1 - FROM sqlite_master - WHERE name NOT LIKE 'sqlite_%' - AND type IN ('table', 'index', 'view', 'trigger') - LIMIT 1 - """ - ).fetchone() - is not None - ) +def _application_schema_objects( + conn: sqlite3.Connection, +) -> tuple[tuple[str, str], ...]: + rows = conn.execute( + """ + SELECT type, name + FROM sqlite_master + WHERE name NOT LIKE 'sqlite_%' + AND type IN ('view', 'trigger', 'index', 'table') + ORDER BY CASE type + WHEN 'view' THEN 0 + WHEN 'trigger' THEN 1 + WHEN 'index' THEN 2 + ELSE 3 + END, name + """ + ).fetchall() + return tuple((str(row[0]), str(row[1])) for row in rows) + +def _quote_sqlite_identifier(value: str) -> str: + return '"' + str(value).replace('"', '""') + '"' -def _run_migrations( + +def _rebuild_current_schema_conn( conn: sqlite3.Connection, *, - target_version: int = STORE_SCHEMA_VERSION, - connector_ack_ttl_seconds: int = CONNECTOR_ACK_TTL_SECONDS, + previous_version: int, ) -> None: - """Run exact ordered transitions with one transaction per version.""" if conn.in_transaction: - raise StoreSchemaError("schema_migration_in_transaction") - _validate_migration_registry(target_version=target_version) - current = int(conn.execute("PRAGMA user_version").fetchone()[0]) - while current < int(target_version): - migration = MIGRATIONS[current] - if migration.from_version != current: - raise RuntimeError("invalid migration registry dispatch") + raise StoreSchemaError("schema_rebuild_in_transaction") + objects = _application_schema_objects(conn) + _LOGGER.warning( + "store schema mismatch; discarding and recreating database " + "previous_version=%d target_version=%d discarded_objects=%s", + int(previous_version), + STORE_SCHEMA_VERSION, + ",".join(f"{kind}:{name}" for kind, name in objects) or "none", + ) + conn.execute("PRAGMA foreign_keys=OFF") + try: conn.execute("BEGIN IMMEDIATE") try: - if migration.apply is _migrate_v15_to_v16_conn: - _migrate_v15_to_v16_conn( - conn, - connector_ack_ttl_seconds=max( - 1, int(connector_ack_ttl_seconds) - ), + for kind, name in objects: + conn.execute( + f"DROP {kind.upper()} IF EXISTS " + f"{_quote_sqlite_identifier(name)}" ) - else: - migration.apply(conn) - conn.execute(f"PRAGMA user_version = {migration.to_version}") + _create_current_schema_objects_conn(conn) conn.commit() except Exception: conn.rollback() raise - current = int(conn.execute("PRAGMA user_version").fetchone()[0]) - if current != migration.to_version: - raise StoreSchemaError("schema_version_not_advanced") + finally: + conn.execute("PRAGMA foreign_keys=ON") -def ensure_schema( - conn: sqlite3.Connection, - *, - connector_ack_ttl_seconds: int = CONNECTOR_ACK_TTL_SECONDS, -) -> None: - """Gate the current schema cheaply, or initialize/migrate older stores.""" +def ensure_schema(conn: sqlite3.Connection) -> None: + """Open v29 unchanged or explicitly replace any other schema.""" version = int(conn.execute("PRAGMA user_version").fetchone()[0]) if version == STORE_SCHEMA_VERSION: return - if version > STORE_SCHEMA_VERSION: - raise StoreSchemaError("schema_too_new") if not isinstance(conn, _ClosingConnection): raise local_state_error(LocalStateErrorCode.OPERATION_FAILED) schema_authority = _schema_connection_authority(conn) @@ -13315,308 +10237,300 @@ def ensure_schema( version = int(conn.execute("PRAGMA user_version").fetchone()[0]) if version == STORE_SCHEMA_VERSION: return - if version > STORE_SCHEMA_VERSION: - raise StoreSchemaError("schema_too_new") with private_file_creation_umask(): _configure_persistent_database_conn(conn) - if version == 0 and not _database_has_application_objects(conn): + objects = _application_schema_objects(conn) + if version == 0 and not objects: _create_current_schema_conn(conn) return - _run_migrations( - conn, - connector_ack_ttl_seconds=max( - 1, int(connector_ack_ttl_seconds) - ), + _rebuild_current_schema_conn( + conn, previous_version=version ) _ensure_schema = ensure_schema +def _turn_list_store_epoch_conn(conn: sqlite3.Connection) -> str: + row = conn.execute( + "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" + ).fetchone() + if row is None or not str(row[0]): + raise StoreSchemaError("turn_list_state_unavailable") + return str(row[0]) + + +def _turn_change_store_epoch_conn(conn: sqlite3.Connection) -> str: + row = conn.execute( + "SELECT store_epoch FROM turn_change_state WHERE scope = 'turn-delta'" + ).fetchone() + if row is None or not str(row[0]): + raise StoreSchemaError("turn_change_state_unavailable") + return str(row[0]) + + def init_store( db_path: Path, - *, - connector_ack_ttl_seconds: int = CONNECTOR_ACK_TTL_SECONDS, ) -> None: - """Initialize or migrate the sqlite store to the current schema.""" + """Initialize the sqlite store at the current schema.""" with _connect(db_path, prepare=True) as conn: - ensure_schema( - conn, - connector_ack_ttl_seconds=max( - 1, int(connector_ack_ttl_seconds) - ), - ) + ensure_schema(conn) -def _herdr_turn_counter(value: Any, field: str) -> int: - if ( - not isinstance(value, int) - or isinstance(value, bool) - or not 0 <= value <= _SQLITE_MAX_INTEGER - ): - raise ValueError(f"{field} must be a nonnegative SQLite integer") - return int(value) +def _agent_event_from_row(row: tuple[Any, ...]) -> StoredAgentEvent: + private_payload = _json_object(row[15]) + public_payload = _json_object(row[16]) + kind = str(row[3]) + if kind not in AGENT_EVENT_KINDS: + raise StoreSchemaError("invalid_agent_event_kind") + try: + normalized_host = normalize_agent_event_identifier( + row[1], "host_id", required=True + ) + stored_event = AgentEvent( + event_id=str(row[2]), + kind=kind, # type: ignore[arg-type] + source=str(row[4]), + worker_id=str(row[5]), + visibility=str(row[6]), # type: ignore[arg-type] + source_session_id=str(row[7]) if row[7] is not None else None, + source_turn_id=str(row[8]) if row[8] is not None else None, + source_item_id=str(row[9]) if row[9] is not None else None, + source_message_id=str(row[10]) if row[10] is not None else None, + source_event_id=str(row[11]) if row[11] is not None else None, + source_sequence=int(row[12]) if row[12] is not None else None, + observed_at=str(row[13]), + payload_fingerprint=str(row[14]), + payload=private_payload, + public_payload=public_payload, + ) + canonical = agent_event( + kind=stored_event.kind, + source=stored_event.source, + worker_id=stored_event.worker_id, + payload=stored_event.payload, + source_session_id=stored_event.source_session_id, + source_turn_id=stored_event.source_turn_id, + source_item_id=stored_event.source_item_id, + source_message_id=stored_event.source_message_id, + source_event_id=stored_event.source_event_id, + source_sequence=stored_event.source_sequence, + visibility=stored_event.visibility, + observed_at=stored_event.observed_at, + ) + except (TypeError, ValueError, OverflowError) as exc: + raise StoreSchemaError("invalid_agent_event_row") from exc + if canonical != stored_event: + raise StoreSchemaError("invalid_agent_event_row") + return StoredAgentEvent( + sequence=int(row[0]), + host_id=normalized_host or "", + event=stored_event, + ) + + +_AGENT_EVENT_SELECT = """ +SELECT + sequence, host_id, event_id, kind, source, worker_id, visibility, + source_session_id, source_turn_id, source_item_id, source_message_id, + source_event_id, source_sequence, observed_at, payload_fingerprint, + private_payload_json, public_payload_json +FROM agent_events +""" -def _herdr_turn_watermark_from_row(row: tuple[Any, ...]) -> HerdrTurnWatermark: - return HerdrTurnWatermark( - host_id=str(row[0]), - pane_id=str(row[1]), - turn_epoch=int(row[2]), - last_turn=int(row[3]), - completeness_break_count=int(row[4]), - last_completeness_break_reason=( - str(row[5]) if row[5] is not None else None - ), - last_completeness_break_at=( - str(row[6]) if row[6] is not None else None - ), - updated_at=str(row[7]), +def _agent_event_conflicts(existing: StoredAgentEvent, incoming: AgentEvent) -> bool: + stored = existing.event + return ( + stored.kind != incoming.kind + or stored.source != incoming.source + or stored.worker_id != incoming.worker_id + or stored.visibility != incoming.visibility + or stored.source_session_id != incoming.source_session_id + or stored.source_turn_id != incoming.source_turn_id + or stored.source_item_id != incoming.source_item_id + or stored.source_message_id != incoming.source_message_id + or stored.source_event_id != incoming.source_event_id + or stored.source_sequence != incoming.source_sequence + or stored.payload_fingerprint != incoming.payload_fingerprint + or stored.payload != incoming.payload + or stored.public_payload != incoming.public_payload ) -def get_herdr_turn_watermark( - db_path: Path | str, - host_id: str, - pane_id: str, -) -> HerdrTurnWatermark | None: - """Return one pane's durable Herdr replay watermark.""" - if not _sqlite_store_exists(db_path): - return None - with _connect(db_path) as conn: - _ensure_schema(conn) - row = conn.execute( - """ - SELECT - host_id, pane_id, turn_epoch, last_turn, - completeness_break_count, last_completeness_break_reason, - last_completeness_break_at, updated_at - FROM herdr_turn_watermarks - WHERE host_id = ? AND pane_id = ? - """, - (str(host_id), str(pane_id)), - ).fetchone() - return _herdr_turn_watermark_from_row(row) if row is not None else None - - -def set_herdr_turn_watermark( - db_path: Path | str, - host_id: str, - pane_id: str, - *, - turn_epoch: int, - last_turn: int, - observed_at: str | None = None, -) -> HerdrTurnWatermark: - """Create or re-baseline a watermark while retaining prior break evidence.""" - epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") - turn = _herdr_turn_counter(last_turn, "last_turn") - current = observed_at or utc_timestamp() - with _connect(db_path) as conn: - _ensure_schema(conn) - conn.execute( - """ - INSERT INTO herdr_turn_watermarks ( - host_id, pane_id, turn_epoch, last_turn, updated_at - ) VALUES (?, ?, ?, ?, ?) - ON CONFLICT(host_id, pane_id) DO UPDATE SET - turn_epoch = excluded.turn_epoch, - last_turn = excluded.last_turn, - updated_at = excluded.updated_at - """, - (str(host_id), str(pane_id), epoch, turn, current), - ) - watermark = get_herdr_turn_watermark(db_path, host_id, pane_id) - if watermark is None: - raise StoreSchemaError("herdr_turn_watermark_unavailable") - return watermark -def record_herdr_turn_completeness_break( - db_path: Path | str, - host_id: str, - pane_id: str, - *, - turn_epoch: int, - newest_turn: int, - reason: str, - observed_at: str | None = None, -) -> HerdrTurnWatermark: - """Persist an explicit replay gap and atomically re-baseline the pane.""" - epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") - newest = _herdr_turn_counter(newest_turn, "newest_turn") - break_reason = str(reason).strip() - if not break_reason: - raise ValueError("reason must not be empty") - current = observed_at or utc_timestamp() - with _connect(db_path) as conn: - _ensure_schema(conn) - conn.execute( - """ - INSERT INTO herdr_turn_watermarks ( - host_id, pane_id, turn_epoch, last_turn, - completeness_break_count, last_completeness_break_reason, - last_completeness_break_at, updated_at - ) VALUES (?, ?, ?, ?, 1, ?, ?, ?) - ON CONFLICT(host_id, pane_id) DO UPDATE SET - turn_epoch = excluded.turn_epoch, - last_turn = excluded.last_turn, - completeness_break_count = - herdr_turn_watermarks.completeness_break_count + 1, - last_completeness_break_reason = - excluded.last_completeness_break_reason, - last_completeness_break_at = - excluded.last_completeness_break_at, - updated_at = excluded.updated_at - """, - ( - str(host_id), - str(pane_id), - epoch, - newest, - break_reason, - current, - current, - ), - ) - watermark = get_herdr_turn_watermark(db_path, host_id, pane_id) - if watermark is None: - raise StoreSchemaError("herdr_turn_watermark_unavailable") - return watermark -def latest_turn_id_for_worker( - db_path: Path | str, - host_id: str, - worker_id: str, -) -> str | None: - """Return the newest live semantic turn row for provenance linking.""" - if not _sqlite_store_exists(db_path): - return None - with _connect(db_path) as conn: - _ensure_schema(conn) - row = conn.execute( - """ - SELECT turn_id - FROM turns - WHERE host_id = ? - AND worker_id = ? - AND COALESCE(json_extract(payload_json, '$.superseded_at'), '') = '' - ORDER BY observed_at DESC, list_sequence DESC - LIMIT 1 - """, - (str(host_id), str(worker_id)), - ).fetchone() - return str(row[0]) if row is not None else None +def _canonical_agent_event_for_append(event: AgentEvent) -> AgentEvent: + if not isinstance(event, AgentEvent): + raise ValueError("event must be an AgentEvent") + canonical_event = agent_event( + kind=event.kind, + source=event.source, + worker_id=event.worker_id, + payload=event.payload, + source_session_id=event.source_session_id, + source_turn_id=event.source_turn_id, + source_item_id=event.source_item_id, + source_message_id=event.source_message_id, + source_event_id=event.source_event_id, + source_sequence=event.source_sequence, + visibility=event.visibility, + observed_at=event.observed_at, + ) + if canonical_event != event: + raise ValueError("event must use the canonical agent event contract") + return canonical_event -def record_herdr_turn_completion( - db_path: Path | str, +def _append_agent_event_conn( + conn: sqlite3.Connection, host_id: str, - pane_id: str, - *, - turn_epoch: int, - turn: int, - outcome: str, - completed_unix_ms: int, - message: str | None, - message_truncated: bool, - agent_session_path: str | None, - worker_id: str | None, - refreshed_turn_id: str | None, - observed_at: str | None = None, -) -> HerdrTurnWatermark: - """Store completion provenance and advance its replay watermark atomically.""" - epoch = _herdr_turn_counter(turn_epoch, "turn_epoch") - turn_number = _herdr_turn_counter(turn, "turn") - completed_ms = _herdr_turn_counter(completed_unix_ms, "completed_unix_ms") - normalized_outcome = str(outcome) - if normalized_outcome not in {"completed", "aborted"}: - raise ValueError("outcome must be completed or aborted") - if message is not None and not isinstance(message, str): - raise ValueError("message must be text or None") - if isinstance(message, str) and len(message.encode("utf-8")) > 8 * 1024: - raise ValueError("message must not exceed 8 KiB") - if not isinstance(message_truncated, bool): - raise ValueError("message_truncated must be a boolean") - if agent_session_path is not None and not isinstance( - agent_session_path, - str, - ): - raise ValueError("agent_session_path must be text or None") - current = observed_at or utc_timestamp() - with _connect(db_path) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - existing = conn.execute( - """ - SELECT turn_epoch, last_turn - FROM herdr_turn_watermarks - WHERE host_id = ? AND pane_id = ? - """, - (str(host_id), str(pane_id)), - ).fetchone() - if existing is not None and int(existing[0]) != epoch: - raise StoreSchemaError("herdr_turn_epoch_changed") - conn.execute( - """ - INSERT INTO herdr_turn_completions ( - host_id, pane_id, turn_epoch, turn, outcome, - completed_unix_ms, message, message_truncated, - agent_session_path, worker_id, refreshed_turn_id, observed_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(host_id, pane_id, turn_epoch, turn) DO UPDATE SET - outcome = excluded.outcome, - completed_unix_ms = excluded.completed_unix_ms, - message = excluded.message, - message_truncated = excluded.message_truncated, - agent_session_path = excluded.agent_session_path, - worker_id = COALESCE( - excluded.worker_id, herdr_turn_completions.worker_id - ), - refreshed_turn_id = COALESCE( - excluded.refreshed_turn_id, - herdr_turn_completions.refreshed_turn_id - ), - observed_at = excluded.observed_at - """, - ( - str(host_id), - str(pane_id), - epoch, - turn_number, - normalized_outcome, - completed_ms, - message, - int(message_truncated), - agent_session_path, - str(worker_id) if worker_id else None, - str(refreshed_turn_id) if refreshed_turn_id else None, - current, - ), - ) - conn.execute( - """ - INSERT INTO herdr_turn_watermarks ( - host_id, pane_id, turn_epoch, last_turn, updated_at - ) VALUES (?, ?, ?, ?, ?) - ON CONFLICT(host_id, pane_id) DO UPDATE SET - last_turn = MAX( - herdr_turn_watermarks.last_turn, excluded.last_turn - ), - updated_at = excluded.updated_at - """, - (str(host_id), str(pane_id), epoch, turn_number, current), + event: AgentEvent, +) -> AppendAgentEventResult: + private_json = _canonical_json(event.payload) + public_json = _canonical_json(event.public_payload) + cursor = conn.execute( + """ + INSERT INTO agent_events ( + host_id, event_id, kind, source, worker_id, visibility, + source_session_id, source_turn_id, source_item_id, + source_message_id, source_event_id, source_sequence, + observed_at, payload_fingerprint, private_payload_json, + public_payload_json + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(host_id, event_id) DO NOTHING + """, + ( + host_id, + event.event_id, + event.kind, + event.source, + event.worker_id, + event.visibility, + event.source_session_id, + event.source_turn_id, + event.source_item_id, + event.source_message_id, + event.source_event_id, + event.source_sequence, + event.observed_at, + event.payload_fingerprint, + private_json, + public_json, + ), + ) + inserted = cursor.rowcount == 1 + row = conn.execute( + _AGENT_EVENT_SELECT + " WHERE host_id = ? AND event_id = ?", + (host_id, event.event_id), + ).fetchone() + if row is None: + raise StoreSchemaError("agent_event_append_failed") + stored = _agent_event_from_row(row) + if _agent_event_conflicts(stored, event): + raise AgentEventIdentityConflict(event.event_id) + return AppendAgentEventResult( + sequence=stored.sequence, + event_id=event.event_id, + inserted=inserted, + ) + + + + + + +def record_agent_event( + db_path: Path | str, + host_id: str, + **event_fields: Any, +) -> AppendAgentEventResult: + """Validate, normalize, and durably append one source event.""" + normalized_host = normalize_agent_event_identifier( + host_id, "host_id", required=True + ) + event = agent_event(**event_fields) + _canonical_agent_event_for_append(event) + with _connect(db_path, prepare=True) as conn: + _ensure_schema(conn) + conn.execute("BEGIN IMMEDIATE") + try: + result = _append_agent_event_conn( + conn, normalized_host or "", event ) conn.commit() except Exception: conn.rollback() raise - watermark = get_herdr_turn_watermark(db_path, host_id, pane_id) - if watermark is None: - raise StoreSchemaError("herdr_turn_watermark_unavailable") - return watermark + return result + + +def list_agent_events( + db_path: Path | str, + host_id: str, + *, + worker_id: str | None = None, + source: str | None = None, + session_id: str | None = None, + turn_id: str | None = None, + visibility: Literal["private", "public"] | None = None, + after_sequence: int = 0, + limit: int = AGENT_EVENT_QUERY_DEFAULT_LIMIT, +) -> tuple[StoredAgentEvent, ...]: + """List private structured events in append order with bounded work.""" + if ( + isinstance(after_sequence, bool) + or not isinstance(after_sequence, int) + or after_sequence < 0 + or after_sequence > (1 << 63) - 1 + ): + raise ValueError("after_sequence must be a nonnegative integer") + if ( + isinstance(limit, bool) + or not isinstance(limit, int) + or not 1 <= limit <= AGENT_EVENT_QUERY_MAX_LIMIT + ): + raise ValueError( + f"limit must be between 1 and {AGENT_EVENT_QUERY_MAX_LIMIT}" + ) + normalized_host = normalize_agent_event_identifier( + host_id, "host_id", required=True + ) + normalized_filters: list[tuple[str, str | None]] = [] + for column, value, field in ( + ("worker_id", worker_id, "worker_id"), + ("source", source, "source"), + ("source_session_id", session_id, "source_session_id"), + ("source_turn_id", turn_id, "source_turn_id"), + ): + normalized_filters.append( + (column, normalize_agent_event_identifier(value, field)) + ) + if visibility is not None and visibility not in {"private", "public"}: + raise ValueError("visibility must be private, public, or None") + if not _sqlite_store_exists(db_path): + return () + clauses = ["host_id = ?", "sequence > ?"] + parameters: list[Any] = [normalized_host, int(after_sequence)] + for column, value in (*normalized_filters, ("visibility", visibility)): + if value is not None: + clauses.append(f"{column} = ?") + parameters.append(value) + parameters.append(int(limit)) + with _connect(db_path) as conn: + _ensure_schema(conn) + rows = conn.execute( + _AGENT_EVENT_SELECT + + " WHERE " + + " AND ".join(clauses) + + " ORDER BY sequence ASC LIMIT ?", + parameters, + ).fetchall() + return tuple(_agent_event_from_row(row) for row in rows) + + def _normalized_command_request_policy( @@ -16263,7 +13177,8 @@ def maybe_run_automatic_store_maintenance( db_path: Path, *, policy: SnapshotRetentionPolicy, - turn_model: str = DEFAULT_TURN_MODEL, + agent_event_host_id: str | None = None, + agent_event_retention_days: int | None = None, acknowledged_final_retention_days: int = ACKNOWLEDGED_FINAL_RETENTION_DAYS, acknowledged_final_retention_count: int = ACKNOWLEDGED_FINAL_RETENTION_COUNT, command_retry_horizon_seconds: int = COMMAND_RETRY_HORIZON_SECONDS, @@ -16295,7 +13210,45 @@ def maybe_run_automatic_store_maintenance( or acknowledged_final_retention_count > _SQLITE_MAX_INTEGER ): raise ValueError("acknowledged final retention values are too large") + if agent_event_host_id is None and agent_event_retention_days is not None: + raise ValueError("agent_event_host_id is required for event retention") + normalized_agent_event_host = ( + normalize_agent_event_identifier( + agent_event_host_id, + "agent_event_host_id", + required=True, + ) + if agent_event_host_id is not None + else None + ) + agent_event_days = ( + policy.retention_days + if agent_event_retention_days is None + else agent_event_retention_days + ) + if ( + normalized_agent_event_host is not None + and ( + isinstance(agent_event_days, bool) + or not isinstance(agent_event_days, int) + or not 1 <= agent_event_days <= _MAX_RETENTION_DAYS + ) + ): + raise ValueError("agent_event_retention_days must be positive") current_at = _connector_now(now) + agent_event_cutoff_at = _utc_cutoff( + retention_days=agent_event_days, + now=current_at, + ) + empty_agent_events = { + "host_id": normalized_agent_event_host, + "retention_days": agent_event_days, + "cutoff_at": agent_event_cutoff_at, + "batch_size": policy.batch_size, + "examined": 0, + "deleted": 0, + "remaining_candidates": False, + } empty_command_requests = _command_request_maintenance_summary( None, retry_horizon_seconds=command_retry_horizon_seconds, @@ -16303,16 +13256,6 @@ def maybe_run_automatic_store_maintenance( retention_count=command_receipt_retention_count, batch_size=policy.batch_size, ) - empty_herdr_turns = { - "examined": 0, - "deleted": 0, - "deleted_completions": 0, - "deleted_watermarks": 0, - "remaining_candidates": False, - "retention_days": policy.retention_days, - "retention_count": policy.retention_count, - "batch_size": policy.batch_size, - } if not _sqlite_store_exists(db_path): return dict(sanitize_public_value({ "schema_version": 1, @@ -16326,6 +13269,7 @@ def maybe_run_automatic_store_maintenance( "deleted": 0, "remaining_candidates": False, }, + "agent_events": empty_agent_events, "final_retention": { "examined": 0, "deleted": 0, @@ -16338,7 +13282,6 @@ def maybe_run_automatic_store_maintenance( ), }, "command_requests": empty_command_requests, - "herdr_turns": empty_herdr_turns, "batch_size": policy.batch_size, })) cutoff_at = _utc_cutoff(retention_days=policy.retention_days, now=current_at) @@ -16348,6 +13291,8 @@ def maybe_run_automatic_store_maintenance( ) with _connect(db_path, isolation_level=None) as conn: _ensure_schema(conn) + if normalized_agent_event_host is not None: + conn.execute("PRAGMA secure_delete=ON") conn.execute("BEGIN IMMEDIATE") try: state = conn.execute( @@ -16383,6 +13328,7 @@ def maybe_run_automatic_store_maintenance( "deleted": 0, "remaining_candidates": False, }, + "agent_events": empty_agent_events, "final_retention": { "examined": 0, "deleted": 0, @@ -16395,19 +13341,28 @@ def maybe_run_automatic_store_maintenance( ), }, "command_requests": empty_command_requests, - "herdr_turns": empty_herdr_turns, "batch_size": policy.batch_size, })) - if _submission_linking_enabled(turn_model): - _settle_due_submission_links_conn( - conn, - db_path=db_path, - now=current_at, - ) + _settle_due_submission_links_conn( + conn, + db_path=db_path, + now=current_at, + ) _expire_turn_submissions_conn( conn, current=current_at, ) + agent_events = dict(empty_agent_events) + if normalized_agent_event_host is not None: + agent_events.update( + _cleanup_agent_event_retention_conn( + conn, + normalized_agent_event_host, + cutoff_at=agent_event_cutoff_at, + batch_size=policy.batch_size, + dry_run=False, + ) + ) candidates, _ = _snapshot_retention_candidates_conn( conn, cutoff_at=cutoff_at, @@ -16529,25 +13484,10 @@ def maybe_run_automatic_store_maintenance( retention_count=command_receipt_retention_count, batch_size=policy.batch_size, ) - herdr_turns = cleanup_herdr_turn_retention( - db_path, - retention_days=policy.retention_days, - retention_count=policy.retention_count, - batch_size=policy.batch_size, - now=current_at, - ) return dict(sanitize_public_value({ "schema_version": 1, - "ok": bool(command_requests["ok"]) and bool(herdr_turns["ok"]), - "status": ( - "ok" - if command_requests["ok"] and herdr_turns["ok"] - else ( - command_requests["status"] - if not command_requests["ok"] - else herdr_turns["status"] - ) - ), + "ok": bool(command_requests["ok"]), + "status": "ok" if command_requests["ok"] else command_requests["status"], "due": True, "last_completed_at": current_at, "next_due_at": _connector_add_seconds(current_at, cadence_seconds), @@ -16556,6 +13496,7 @@ def maybe_run_automatic_store_maintenance( "deleted": deleted, "remaining_candidates": bool(remaining_ids), }, + "agent_events": agent_events, "final_retention": { "examined": final_examined, "deleted": final_deleted, @@ -16568,7 +13509,6 @@ def maybe_run_automatic_store_maintenance( ), }, "command_requests": command_requests, - "herdr_turns": herdr_turns, "batch_size": policy.batch_size, })) @@ -16662,6 +13602,109 @@ def cleanup_event_retention( })) +_AGENT_EVENT_RETENTION_SELECT = """ +SELECT sequence +FROM agent_events +""" + + +def _cleanup_agent_event_retention_conn( + conn: sqlite3.Connection, + host_id: str, + *, + cutoff_at: str, + batch_size: int, + dry_run: bool, +) -> dict[str, Any]: + """Delete one bounded batch of expired journal rows.""" + rows = conn.execute( + _AGENT_EVENT_RETENTION_SELECT + + " WHERE host_id = ? AND observed_at < ?" + + " ORDER BY observed_at, sequence LIMIT ?", + (str(host_id), cutoff_at, int(batch_size) + 1), + ).fetchall() + sequences = [int(row[0]) for row in rows[:batch_size]] + remaining = len(rows) > len(sequences) + deleted = 0 + if sequences and not dry_run: + placeholders = ",".join("?" for _ in sequences) + deleted = int( + conn.execute( + f"DELETE FROM agent_events WHERE host_id = ? " + f"AND sequence IN ({placeholders})", + (str(host_id), *sequences), + ).rowcount + or 0 + ) + if deleted != len(sequences): + raise StoreSchemaError("agent_event_retention_delete_mismatch") + return { + "examined": len(sequences), + "deleted": len(sequences) if dry_run else deleted, + "remaining_candidates": remaining, + } + + +def cleanup_agent_event_retention( + db_path: Path, + host_id: str, + *, + retention_days: int, + now: str | None = None, + dry_run: bool = False, + batch_size: int = 100, +) -> dict[str, Any]: + """Delete a bounded batch of expired structured agent events.""" + days = max(1, int(retention_days)) + bounded_batch = max(1, min(int(batch_size), 1_000)) + cutoff_at = _utc_cutoff(retention_days=days, now=now) + base = { + "schema_version": 1, + "host_id": str(host_id), + "dry_run": bool(dry_run), + "retention_days": days, + "cutoff_at": cutoff_at, + "batch_size": bounded_batch, + } + if not _sqlite_store_exists(db_path): + return dict(sanitize_public_value({ + **base, + "ok": False, + "status": "store_unavailable", + "examined": 0, + "deleted": 0, + "remaining_candidates": False, + })) + with _connect(db_path, isolation_level=None) as conn: + _ensure_schema(conn) + # Ask SQLite to scrub deleted cells in modified pages. WAL frames, + # checkpoints, filesystem snapshots, and backups have independent + # lifecycles, so this is not an immediate physical-erasure guarantee. + conn.execute("PRAGMA secure_delete=ON") + conn.execute("BEGIN IMMEDIATE") + try: + result = _cleanup_agent_event_retention_conn( + conn, + str(host_id), + cutoff_at=cutoff_at, + batch_size=bounded_batch, + dry_run=bool(dry_run), + ) + if dry_run: + conn.rollback() + else: + conn.commit() + except Exception: + conn.rollback() + raise + return dict(sanitize_public_value({ + **base, + "ok": True, + "status": "ok", + **result, + })) + + def _turn_content_retention_candidates_conn( conn: sqlite3.Connection, *, @@ -17928,213 +14971,56 @@ def compact_turn_change_journal( """, (str(host_id), count - 1), ).fetchone() - candidates = [] if boundary is None else conn.execute( - """ - SELECT seq FROM turn_change_journal - WHERE host_id = ? AND seq < ? - AND julianday(changed_at) IS NOT NULL - AND julianday(changed_at) < julianday(?) - ORDER BY seq ASC LIMIT ? - """, - (str(host_id), int(boundary[0]), cutoff_at, bounded_batch), - ).fetchall() - sequences = [int(row[0]) for row in candidates] - if sequences: - placeholders = ",".join("?" for _ in sequences) - conn.execute( - f"DELETE FROM turn_change_journal WHERE host_id = ? AND seq IN ({placeholders})", - (str(host_id), *sequences), - ) - conn.execute( - """ - INSERT INTO turn_change_floor(host_id, floor_seq) VALUES (?, ?) - ON CONFLICT(host_id) DO UPDATE SET - floor_seq = MAX(turn_change_floor.floor_seq, excluded.floor_seq) - """, - (str(host_id), max(sequences)), - ) - remaining = bool(boundary is not None and conn.execute( - """ - SELECT 1 FROM turn_change_journal - WHERE host_id = ? AND seq < ? - AND julianday(changed_at) IS NOT NULL - AND julianday(changed_at) < julianday(?) LIMIT 1 - """, - (str(host_id), int(boundary[0]), cutoff_at), - ).fetchone()) - if dry_run: - conn.rollback() - else: - conn.commit() - except Exception: - conn.rollback() - raise - return { - "schema_version": 1, "ok": True, "status": "ok", - "host_id": str(host_id), "dry_run": bool(dry_run), - "retention_days": days, "retention_count": count, - "batch_size": bounded_batch, "examined": len(sequences), - "deleted": len(sequences), "remaining_candidates": remaining, - } - - -def cleanup_herdr_turn_retention( - db_path: Path | str, - *, - host_id: str | None = None, - retention_days: int = HERDR_TURN_RETENTION_DAYS, - retention_count: int = HERDR_TURN_RETENTION_COUNT, - batch_size: int = HERDR_TURN_RETENTION_BATCH_SIZE, - now: str | None = None, - dry_run: bool = False, -) -> dict[str, Any]: - """Bound completion provenance and remove old watermarks for dead panes.""" - days = max(1, min(int(retention_days), _MAX_RETENTION_DAYS)) - count = max(1, min(int(retention_count), _SQLITE_MAX_INTEGER)) - bounded_batch = max(1, min(int(batch_size), 1_000)) - current_at = _connector_now(now) - cutoff_at = _utc_cutoff(retention_days=days, now=current_at) - base = { - "schema_version": 1, - "ok": False, - "status": "store_unavailable", - "scope": "host" if host_id is not None else "database", - "host_id": str(host_id) if host_id is not None else None, - "dry_run": bool(dry_run), - "retention_days": days, - "retention_count": count, - "cutoff_at": cutoff_at, - "batch_size": bounded_batch, - } - if not _sqlite_store_exists(db_path): - return dict(sanitize_public_value({ - **base, - "examined": 0, - "deleted": 0, - "deleted_completions": 0, - "deleted_watermarks": 0, - "remaining_candidates": False, - })) - host_clause = "AND host_id = :host_id" if host_id is not None else "" - watermark_host_clause = ( - "AND watermarks.host_id = :host_id" if host_id is not None else "" - ) - params: dict[str, Any] = { - "host_id": str(host_id) if host_id is not None else "", - "cutoff_at": cutoff_at, - "current_at": current_at, - "retention_count": count, - "limit": bounded_batch + 1, - } - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - completion_pool = [ - int(row[0]) - for row in conn.execute( - f""" - WITH ranked AS ( - SELECT - rowid AS completion_rowid, - observed_at, - ROW_NUMBER() OVER ( - PARTITION BY host_id - ORDER BY observed_at DESC, rowid DESC - ) AS retention_rank - FROM herdr_turn_completions - WHERE 1 = 1 {host_clause} - ) - SELECT completion_rowid - FROM ranked - WHERE observed_at < :cutoff_at - AND retention_rank > :retention_count - ORDER BY observed_at, completion_rowid - LIMIT :limit - """, - params, - ).fetchall() - ] - completion_ids = completion_pool[:bounded_batch] - remaining_budget = bounded_batch - len(completion_ids) - watermark_params = { - **params, - # Query one extra row even when completions consume the whole - # batch so remaining_candidates still advertises dead panes. - "limit": remaining_budget + 1, - } - watermark_pool = [ - (str(row[0]), str(row[1])) - for row in conn.execute( - f""" - SELECT watermarks.host_id, watermarks.pane_id - FROM herdr_turn_watermarks AS watermarks - WHERE watermarks.updated_at < :cutoff_at - {watermark_host_clause} - AND NOT EXISTS ( - SELECT 1 - FROM worker_bindings AS bindings - WHERE bindings.host_id = watermarks.host_id - AND bindings.backend = 'herdr' - AND bindings.expires_at > :current_at - AND ( - ( - bindings.target_kind = 'pane_id' - AND bindings.target_value = watermarks.pane_id - ) - OR ( - bindings.turn_target_kind = 'pane_id' - AND bindings.turn_target_value = watermarks.pane_id - ) - ) - ) - ORDER BY watermarks.updated_at, watermarks.host_id, - watermarks.pane_id - LIMIT :limit + candidates = [] if boundary is None else conn.execute( + """ + SELECT seq FROM turn_change_journal + WHERE host_id = ? AND seq < ? + AND julianday(changed_at) IS NOT NULL + AND julianday(changed_at) < julianday(?) + ORDER BY seq ASC LIMIT ? + """, + (str(host_id), int(boundary[0]), cutoff_at, bounded_batch), + ).fetchall() + sequences = [int(row[0]) for row in candidates] + if sequences: + placeholders = ",".join("?" for _ in sequences) + conn.execute( + f"DELETE FROM turn_change_journal WHERE host_id = ? AND seq IN ({placeholders})", + (str(host_id), *sequences), + ) + conn.execute( + """ + INSERT INTO turn_change_floor(host_id, floor_seq) VALUES (?, ?) + ON CONFLICT(host_id) DO UPDATE SET + floor_seq = MAX(turn_change_floor.floor_seq, excluded.floor_seq) """, - watermark_params, - ).fetchall() - ] - watermark_keys = watermark_pool[:remaining_budget] - if not dry_run: - if completion_ids: - placeholders = ",".join("?" for _ in completion_ids) - conn.execute( - f""" - DELETE FROM herdr_turn_completions - WHERE rowid IN ({placeholders}) - """, - completion_ids, - ) - if watermark_keys: - conn.executemany( - """ - DELETE FROM herdr_turn_watermarks - WHERE host_id = ? AND pane_id = ? - """, - watermark_keys, - ) - conn.commit() - else: + (str(host_id), max(sequences)), + ) + remaining = bool(boundary is not None and conn.execute( + """ + SELECT 1 FROM turn_change_journal + WHERE host_id = ? AND seq < ? + AND julianday(changed_at) IS NOT NULL + AND julianday(changed_at) < julianday(?) LIMIT 1 + """, + (str(host_id), int(boundary[0]), cutoff_at), + ).fetchone()) + if dry_run: conn.rollback() + else: + conn.commit() except Exception: conn.rollback() raise - examined = len(completion_ids) + len(watermark_keys) - remaining = ( - len(completion_pool) > len(completion_ids) - or len(watermark_pool) > len(watermark_keys) - ) - return dict(sanitize_public_value({ - **base, - "ok": True, - "status": "ok", - "examined": examined, - "deleted": examined, - "deleted_completions": len(completion_ids), - "deleted_watermarks": len(watermark_keys), - "remaining_candidates": remaining, - })) + return { + "schema_version": 1, "ok": True, "status": "ok", + "host_id": str(host_id), "dry_run": bool(dry_run), + "retention_days": days, "retention_count": count, + "batch_size": bounded_batch, "examined": len(sequences), + "deleted": len(sequences), "remaining_candidates": remaining, + } + + def run_store_maintenance( @@ -18158,9 +15044,6 @@ def run_store_maintenance( turn_change_retention_days: int = TURN_CHANGE_RETENTION_DAYS, turn_change_retention_count: int = TURN_CHANGE_RETENTION_COUNT, turn_change_batch_size: int = TURN_CHANGE_COMPACTION_BATCH_SIZE, - herdr_turn_retention_days: int = HERDR_TURN_RETENTION_DAYS, - herdr_turn_retention_count: int = HERDR_TURN_RETENTION_COUNT, - herdr_turn_batch_size: int = HERDR_TURN_RETENTION_BATCH_SIZE, ) -> dict[str, Any]: """Run one bounded batch for every online store-maintenance class.""" retention = cleanup_event_retention( @@ -18171,6 +15054,14 @@ def run_store_maintenance( dry_run=dry_run, batch_size=event_batch_size, ) + agent_events = cleanup_agent_event_retention( + db_path, + host_id, + retention_days=retention_days, + now=now, + dry_run=dry_run, + batch_size=event_batch_size, + ) snapshots = cleanup_snapshot_retention( db_path, retention_days=snapshot_retention_days, @@ -18230,24 +15121,15 @@ def run_store_maintenance( now=now, dry_run=dry_run, ) - herdr_turns = cleanup_herdr_turn_retention( - db_path, - host_id=host_id, - retention_days=herdr_turn_retention_days, - retention_count=herdr_turn_retention_count, - batch_size=herdr_turn_batch_size, - now=now, - dry_run=dry_run, - ) ok = ( bool(retention.get("ok")) + and bool(agent_events.get("ok")) and bool(snapshots.get("ok")) and bool(outbox.get("ok")) and bool(final_retention.get("ok")) and bool(turn_content.get("ok")) and bool(command_requests.get("ok")) and bool(turn_changes.get("ok")) - and bool(herdr_turns.get("ok")) ) return sanitize_public_value({ "schema_version": 1, @@ -18265,6 +15147,20 @@ def run_store_maintenance( retention.get("remaining_candidates") ), }, + "agent_events": { + "retention_days": int( + agent_events.get("retention_days") or retention_days + ), + "cutoff_at": agent_events.get("cutoff_at"), + "batch_size": int( + agent_events.get("batch_size") or event_batch_size + ), + "examined": int(agent_events.get("examined") or 0), + "deleted": int(agent_events.get("deleted") or 0), + "remaining_candidates": bool( + agent_events.get("remaining_candidates") + ), + }, "snapshots": { "scope": "database", "retention_days": int( @@ -18333,30 +15229,6 @@ def run_store_maintenance( "deleted": int(turn_changes.get("deleted") or 0), "remaining_candidates": bool(turn_changes.get("remaining_candidates")), }, - "herdr_turns": { - "dry_run": bool(herdr_turns.get("dry_run")), - "retention_days": int( - herdr_turns.get("retention_days") or herdr_turn_retention_days - ), - "retention_count": int( - herdr_turns.get("retention_count") or herdr_turn_retention_count - ), - "cutoff_at": herdr_turns.get("cutoff_at"), - "batch_size": int( - herdr_turns.get("batch_size") or herdr_turn_batch_size - ), - "examined": int(herdr_turns.get("examined") or 0), - "deleted": int(herdr_turns.get("deleted") or 0), - "deleted_completions": int( - herdr_turns.get("deleted_completions") or 0 - ), - "deleted_watermarks": int( - herdr_turns.get("deleted_watermarks") or 0 - ), - "remaining_candidates": bool( - herdr_turns.get("remaining_candidates") - ), - }, "turn_content": { "dry_run": bool(turn_content.get("dry_run")), "retention_days": int( @@ -18437,8 +15309,8 @@ def _turn_continuity_identity(payload: Mapping[str, Any]) -> tuple[str, str, int return None -def _turn_submission_owner_identity(worker: Any) -> tuple[str, int]: - """Extract the Phase-1 continuity owner, with a legacy worker fallback.""" +def _turn_submission_owner_identity(worker: Any) -> tuple[str, int] | None: + """Extract the stable continuity owner required for submission linking.""" meta = getattr(worker, "meta", None) if meta is None and isinstance(worker, Mapping): meta = worker.get("meta") @@ -18448,34 +15320,7 @@ def _turn_submission_owner_identity(worker: Any) -> tuple[str, int]: if identity is not None: _kind, owner_key, owner_key_version = identity return owner_key, owner_key_version - - # Old snapshots and direct API callers can predate stable worker keys. The - # shadow ledger must never change their submission behavior, so isolate - # those rows by the existing public worker ID until Stage 4 migration. - worker_id = str(getattr(worker, "id", "") or "").strip() - if not worker_id and isinstance(worker, Mapping): - worker_id = str(worker.get("id") or worker.get("worker_id") or "").strip() - if not worker_id: - raise StoreSchemaError("turn_submission_owner_missing") - return f"legacy-worker:{worker_id}", 0 - - -def _turn_link_candidate_owner_identity(worker: Any) -> tuple[str, int]: - """Normalize prod turns that persisted a stable key without its version.""" - meta = getattr(worker, "meta", None) - if meta is None and isinstance(worker, Mapping): - meta = worker.get("meta") - if isinstance(meta, Mapping): - stable_key = meta.get("stable_key") - if ( - _valid_final_stable_key(stable_key) - and meta.get("stable_key_version") is None - ): - # The authenticated owner hash is the continuity identity. Some - # production observations omitted its v1 metadata marker, so - # normalize only that missing marker for candidate matching. - return str(stable_key), 1 - return _turn_submission_owner_identity(worker) + return None def _turn_submission_policy( @@ -18506,12 +15351,15 @@ def _insert_turn_submission_conn( link_window_seconds: int, hard_ttl_seconds: int, ) -> str: - """Insert the send-started shadow ledger row in the caller transaction.""" + """Insert the send-started submission row in the caller transaction.""" link_window, hard_ttl = _turn_submission_policy( link_window_seconds, hard_ttl_seconds, ) - owner_key, owner_key_version = _turn_submission_owner_identity(worker) + owner_identity = _turn_submission_owner_identity(worker) + if owner_identity is None: + raise StoreSchemaError("turn_submission_stable_owner_missing") + owner_key, owner_key_version = owner_identity submission_id = turn_submission_id(host_id, request_id) current_time = datetime.fromisoformat(current) link_not_before = (current_time - timedelta(seconds=link_window)).isoformat( @@ -18578,7 +15426,7 @@ def _terminalize_turn_submission_conn( terminal_state: str, current: str, ) -> bool: - """Advance an existing shadow row with its command receipt transaction.""" + """Advance a submission row with its command receipt transaction.""" next_state = { "accepted": "submitted", "rejected": "cancelled", @@ -18715,18 +15563,18 @@ def _submission_link_candidate_turns_conn( ) in rows: if not isinstance(source_turn_id, str) or not source_turn_id.strip(): continue - try: - candidate_owner, owner_version = _turn_link_candidate_owner_identity( - { - "id": str(worker_id), - "meta": { - "stable_key": stable_key, - "stable_key_version": stable_key_version, - }, - } - ) - except StoreSchemaError: + owner_identity = _turn_submission_owner_identity( + { + "id": str(worker_id), + "meta": { + "stable_key": stable_key, + "stable_key_version": stable_key_version, + }, + } + ) + if owner_identity is None: continue + candidate_owner, owner_version = owner_identity if owner_version != 1 or candidate_owner != str(owner_key): continue if user_text is None: @@ -18746,7 +15594,7 @@ def _submission_link_candidate_turns_conn( return candidates -def settle_submission_links_conn( +def _settle_submission_links_conn( conn: sqlite3.Connection, host_id: str, owner_key: str, @@ -19103,7 +15951,7 @@ def _settle_due_submission_links_conn( if not _submission_link_component_is_due(key, current_dt): continue next_eligible: list[datetime | None] = [] - component_changed = settle_submission_links_conn( + component_changed = _settle_submission_links_conn( conn, str(candidate_host), str(owner_key), @@ -19879,66 +16727,6 @@ def _apply_backend_pending_observation_conn( return changed -def _merge_backend_pending_conn( - conn: sqlite3.Connection, - host_id: str, - worker_id: str, - pending: Mapping[str, Any] | None, - *, - observed_at: str, -) -> bool: - """Compatibility entrypoint routed through the explicit transition helper.""" - if pending is None: - observation = PendingObservation("read_succeeded_no_prompt") - else: - clean = sanitize_public_mapping(pending) - raw_choices = clean.get("choices", []) if isinstance(clean, Mapping) else [] - normalized_choices = [ - InteractionChoice.from_dict(choice) - for choice in raw_choices - if isinstance(choice, Mapping) - ] - choices = tuple( - PendingObservedChoice( - choice_id=choice.choice_id, - label=choice.label, - picker_ordinal=ordinal, - ) - for ordinal, choice in enumerate(normalized_choices, 1) - ) - question = str(clean.get("question") or clean.get("kind") or "Pending action") - decision: dict[str, Any] | None = None - clean_meta = clean.get("meta") - if isinstance(clean_meta, Mapping) and "decision" in clean_meta: - decision = _normalize_pending_decision_meta(clean_meta["decision"]) - observation = PendingObservation( - "open_prompt", - question=question, - pending_kind=str(clean.get("kind") or "question"), - choices=choices, - revision_digest=stable_fingerprint({"legacy_pending_revision": clean}), - decision_kind=(decision or {}).get("kind"), - decision_options=tuple( - str(option["label"]) - for option in (decision or {}).get("options", []) - ), - decision_multi_select=bool( - (decision or {}).get("multi_select", False) - ), - decision_question_count=int( - (decision or {}).get("question_count", 0) - ), - ) - return _apply_backend_pending_observation_conn( - conn, - host_id, - worker_id, - observation, - observed_at=observed_at, - stale_grace_seconds=DEFAULT_PENDING_STALE_GRACE_SECONDS, - ) - - def apply_backend_pending_observation( db_path: Path | str, host_id: str, @@ -19949,6 +16737,7 @@ def apply_backend_pending_observation( stale_grace_seconds: float = DEFAULT_PENDING_STALE_GRACE_SECONDS, binding_private_fingerprint: str | None = None, observed_turn_target_value: str | None = None, + binding_authoritative: bool = False, ) -> bool: """Apply one explicit observation in a short writer transaction.""" if not _sqlite_store_exists(db_path): @@ -19971,6 +16760,7 @@ def apply_backend_pending_observation( observed_turn_target_value=str( observed_turn_target_value or "" ), + binding_authoritative=bool(binding_authoritative), ) conn.commit() return changed @@ -19979,52 +16769,8 @@ def apply_backend_pending_observation( raise -def merge_backend_pending( - db_path: Path | str, - host_id: str, - worker_id: str, - pending: Mapping[str, Any] | None, -) -> bool: - """Presence-sync one worker's backend-provided pending prompt.""" - if not _sqlite_store_exists(db_path): - return False - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - changed = _merge_backend_pending_conn( - conn, - host_id, - worker_id, - pending, - observed_at=utc_timestamp(), - ) - conn.commit() - return changed - except Exception: - conn.rollback() - raise -def list_backend_pending(db_path: Path | str, host_id: str) -> dict[str, dict[str, Any]]: - """worker_id -> normalized pending dict for every live backend-provided prompt.""" - out: dict[str, dict[str, Any]] = {} - if not _sqlite_store_exists(db_path): - return out - with _connect(db_path) as conn: - _ensure_schema(conn) - for worker_id, payload_json in conn.execute( - "SELECT worker_id, payload_json FROM backend_pending " - "WHERE host_id = ? AND observation_state = 'open'", - (host_id,), - ).fetchall(): - try: - payload = json.loads(payload_json) - except (TypeError, ValueError): - continue - if isinstance(payload, Mapping): - out[str(worker_id)] = sanitize_public_mapping(payload) - return sanitize_public_mapping(out) def _backend_pending_health_from_rows( @@ -20353,9 +17099,10 @@ def _backend_pending_claim_context_conn( SELECT private_fingerprint, worker_fingerprint, turn_target_value FROM worker_bindings WHERE host_id = ? AND worker_id = ? AND worker_fingerprint = ? - AND backend = 'herdr' AND turn_target_kind = 'pane_id' + AND ((backend = 'herdr' AND turn_target_kind = 'pane_id') + OR (backend = 'acp' AND turn_target_kind = 'acp_session_id')) AND private_fingerprint = ? AND turn_target_value = ? - AND sendable = 1 AND expires_at > ? + AND sendable = 1 AND (expires_at IS NULL OR expires_at > ?) """, ( str(host_id), @@ -20697,13 +17444,12 @@ def claim_backend_pending_decision( observed_at=current_time, ) decision = _decision_from_pending_row(row[0], row[1], row[2]) - if ( - context is None - or decision is None - or decision.get("decision_ref") != str(decision_ref) - ): + if decision is None or decision.get("decision_ref") != str(decision_ref): conn.rollback() return BackendPendingDecisionClaim("decision_not_pending") + if context is None: + conn.rollback() + return BackendPendingDecisionClaim("acp_authority_unavailable") if int(decision["question_count"]) > 1: conn.rollback() return BackendPendingDecisionClaim("unsupported_decision") @@ -21161,78 +17907,6 @@ def effect(conn: sqlite3.Connection) -> None: return effect -def prune_backend_pending( - db_path: Path | str, - host_id: str, - live_binding_private_fingerprints: Iterable[str], - *, - deadline_monotonic: float | None = None, - cancelled: Callable[[], bool] | None = None, - observed_at: str | None = None, -) -> int: - """Delete state whose exact authoritative pane binding disappeared.""" - if not _sqlite_store_exists(db_path): - return 0 - live = { - str(private_fingerprint) - for private_fingerprint in live_binding_private_fingerprints - if str(private_fingerprint) - } - current_time, _ = _pending_observed_time(observed_at) - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - if not _begin_turn_refresh_transaction( - conn, - deadline_monotonic=deadline_monotonic, - cancelled=cancelled, - ): - return 0 - try: - if _turn_refresh_is_cancelled( - deadline_monotonic=deadline_monotonic, - cancelled=cancelled, - ): - conn.rollback() - return 0 - stored = [ - (str(row[0]), str(row[1]), str(row[2])) - for row in conn.execute( - """ - SELECT worker_id, binding_private_fingerprint, - observed_turn_target_value - FROM backend_pending - WHERE host_id = ? - """, - (str(host_id),), - ).fetchall() - ] - stale = [ - (worker_id, private_fingerprint, turn_target_value) - for worker_id, private_fingerprint, turn_target_value in stored - if private_fingerprint not in live - ] - for worker_id, private_fingerprint, turn_target_value in stale: - _apply_backend_pending_observation_conn( - conn, - str(host_id), - worker_id, - PendingObservation("worker_authoritatively_absent"), - observed_at=current_time, - stale_grace_seconds=DEFAULT_PENDING_STALE_GRACE_SECONDS, - binding_private_fingerprint=private_fingerprint, - observed_turn_target_value=turn_target_value, - ) - if _turn_refresh_is_cancelled( - deadline_monotonic=deadline_monotonic, - cancelled=cancelled, - ): - conn.rollback() - return 0 - conn.commit() - return len(stale) - except Exception: - conn.rollback() - raise def _decode_turn_content_rows( @@ -21476,7 +18150,6 @@ def _raw_owned_source_turn_candidates( raw_value: Any, *, owner_key: str, - source: str, kind: str, ) -> tuple[str, ...]: """Derive exact-source lookup tokens without invoking the public sanitizer. @@ -21507,15 +18180,7 @@ def _raw_owned_source_turn_candidates( }, } ) - legacy_token = "turnsrc-" + stable_fingerprint( - { - "seed": raw, - "public": {"source": str(source), "kind": str(kind)}, - } - ) - if owner_token == legacy_token: - return (owner_token,) - return owner_token, legacy_token + return (owner_token,) def _canonical_reobservation_text_matches( @@ -21542,7 +18207,6 @@ def _unchanged_owned_turn_reobservation_conn( content: Mapping[str, Any], *, observed_at: str, - turn_model: str, ) -> _TurnContentMergeResult | None: """Return a no-op merge result without decoding or sanitizing turn payloads. @@ -21602,7 +18266,6 @@ def _unchanged_owned_turn_reobservation_conn( source_candidates = _raw_owned_source_turn_candidates( content.get("source_turn_id"), owner_key=owner_key, - source=str(payload.get("source") or "snapshot"), kind=str(payload.get("kind") or "unknown"), ) if str(payload.get("source_turn_id") or "") in source_candidates: @@ -22061,22 +18724,13 @@ def _merge_observed_turn_content_conn( revision_repaired = False changed = metadata_changed or revision_changed or revision_repaired - owner_key, owner_key_version = _turn_submission_owner_identity( - current_worker_payload - ) + owner_identity = _turn_submission_owner_identity(current_worker_payload) submission_link = ( - (owner_key, instruction_fingerprint(incoming_user)) - if owner_key_version == 1 - else None - ) - candidate_owner_key, candidate_owner_key_version = ( - _turn_link_candidate_owner_identity(current_worker_payload) - ) - submission_link_rearm = ( - (candidate_owner_key, instruction_fingerprint(incoming_user)) - if candidate_owner_key_version == 1 + (owner_identity[0], instruction_fingerprint(incoming_user)) + if owner_identity is not None and owner_identity[1] == 1 else None ) + submission_link_rearm = submission_link current_revision = conn.execute( """ SELECT content_revision @@ -22108,7 +18762,6 @@ def _merge_turn_content_conn( content: Mapping[str, Any], *, observed_at: str, - turn_model: str, ) -> _TurnContentMergeResult: if not any(key in content for key in _TURN_CONTENT_FIELDS): return _TurnContentMergeResult(0) @@ -22130,7 +18783,6 @@ def _merge_turn_content_conn( current_worker_payload, content, observed_at=observed_at, - turn_model=turn_model, ) if unchanged is not None: return unchanged @@ -22176,12 +18828,54 @@ def _merge_turn_content_conn( ) -def _turn_refresh_binding_matches_conn( +def _turn_refresh_binding_matches_conn( + conn: sqlite3.Connection, + host_id: str, + worker_id: str, + expected: WorkerBinding, +) -> bool: + if ( + str(expected.host_id) != str(host_id) + or str(expected.worker_id) != str(worker_id) + ): + return False + row = conn.execute( + """ + SELECT + worker_id, + worker_fingerprint, + backend, + turn_target_kind, + turn_target_value + FROM worker_bindings + WHERE host_id = ? + AND backend = ? + AND private_fingerprint = ? + AND expires_at > ? + """, + ( + str(host_id), + str(expected.backend), + str(expected.private_fingerprint), + utc_timestamp(), + ), + ).fetchone() + return row is not None and tuple(row) == ( + str(expected.worker_id), + str(expected.worker_fingerprint), + str(expected.backend), + expected.turn_target_kind, + expected.turn_target_value, + ) + + +def _agent_event_binding_matches_conn( conn: sqlite3.Connection, host_id: str, worker_id: str, expected: WorkerBinding, ) -> bool: + """Match the complete routing generation used by an agent event stream.""" if ( str(expected.host_id) != str(host_id) or str(expected.worker_id) != str(worker_id) @@ -22193,6 +18887,8 @@ def _turn_refresh_binding_matches_conn( worker_id, worker_fingerprint, backend, + target_kind, + target_value, turn_target_kind, turn_target_value FROM worker_bindings @@ -22212,11 +18908,51 @@ def _turn_refresh_binding_matches_conn( str(expected.worker_id), str(expected.worker_fingerprint), str(expected.backend), + str(expected.target_kind), + str(expected.target_value), expected.turn_target_kind, expected.turn_target_value, ) +def _agent_event_authority_observed_at_conn( + conn: sqlite3.Connection, + host_id: str, + event_id: str, +) -> str | None: + """Return the immutable authority time for a retained event.""" + row = conn.execute( + "SELECT observed_at FROM agent_events WHERE host_id = ? AND event_id = ?", + (str(host_id), str(event_id)), + ).fetchone() + if row is None or row[0] is None: + return None + observed_at = _strict_utc_timestamp(row[0]) + if observed_at is None: + raise StoreSchemaError("invalid_agent_event_authority_time") + return observed_at + + +def _owned_turn_projection_exists_conn( + conn: sqlite3.Connection, + host_id: str, + worker_id: str, + source_turn_id: str, +) -> bool: + """Fail closed when any live or superseded owned projection still exists.""" + rows = conn.execute( + "SELECT payload_json FROM turns WHERE host_id = ? AND worker_id = ?", + (str(host_id), str(worker_id)), + ).fetchall() + for row in rows: + payload = _json_object(row[0]) + if _owned_source_turn_matches(payload, source_turn_id) or _source_turn_matches( + payload, source_turn_id + ): + return True + return False + + def _turn_refresh_is_cancelled( *, deadline_monotonic: float | None, @@ -22259,34 +18995,156 @@ def _begin_turn_refresh_transaction( raise +def append_agent_event_and_apply_turn_for_binding( + db_path: Path | str, + host_id: str, + event: AgentEvent, + *, + expected_binding: WorkerBinding, + content: Mapping[str, Any] | None = None, + _fault_inject: Callable[[str], None] | None = None, +) -> AppendProjectedAgentEventResult: + """Atomically journal an agent event and apply its text-only turn projection. + + A replay may repair a provably absent projection exactly once, using the + immutable observation time retained with the original event. It never + re-merges caller data into a live or superseded owned projection. + """ + + normalized_host = normalize_agent_event_identifier( + host_id, "host_id", required=True + ) + _canonical_agent_event_for_append(event) + if not isinstance(expected_binding, WorkerBinding): + raise ValueError("expected_binding must be a WorkerBinding") + if content is not None: + if not isinstance(content, Mapping): + raise ValueError("content must be a mapping or None") + if not str(content.get("source_turn_id") or "").strip(): + raise ValueError("projected agent content requires source_turn_id") + if _fault_inject is not None and not callable(_fault_inject): + raise TypeError("_fault_inject must be callable or None") + rearm_key: tuple[str, str] | None = None + + def fault(boundary: str) -> None: + if _fault_inject is not None: + _fault_inject(boundary) + + with _connect(db_path, prepare=True, isolation_level=None) as conn: + _ensure_schema(conn) + conn.execute("BEGIN IMMEDIATE") + try: + if not _agent_event_binding_matches_conn( + conn, + normalized_host or "", + event.worker_id, + expected_binding, + ): + conn.rollback() + return AppendProjectedAgentEventResult( + event=AppendBoundAgentEventResult( + status="binding_changed", + event_id=event.event_id, + ) + ) + fault("after_binding_check") + appended = _append_agent_event_conn( + conn, + normalized_host or "", + event, + ) + fault("after_event_append") + turn: TurnRefreshApplyResult | None = None + authority_time = _agent_event_authority_observed_at_conn( + conn, + normalized_host or "", + event.event_id, + ) + projection_allowed = content is not None + if not appended.inserted and content is not None: + durable_source_turn = str(event.source_turn_id or "").strip() + caller_source_turn = str(content.get("source_turn_id") or "").strip() + projection_allowed = bool( + authority_time is not None + and durable_source_turn + and caller_source_turn == durable_source_turn + and not _owned_turn_projection_exists_conn( + conn, + normalized_host or "", + event.worker_id, + durable_source_turn, + ) + ) + if projection_allowed and content is not None: + if authority_time is None: + raise StoreSchemaError("agent_event_authority_time_missing") + worker_exists = conn.execute( + "SELECT 1 FROM workers WHERE host_id = ? AND worker_id = ?", + (normalized_host or "", event.worker_id), + ).fetchone() + if worker_exists is None: + raise StoreSchemaError("agent_event_projection_worker_missing") + merge_result = _merge_turn_content_conn( + conn, + normalized_host or "", + event.worker_id, + content, + observed_at=authority_time, + ) + rearm_key = ( + merge_result.submission_link_rearm + or merge_result.submission_link + ) + if merge_result.submission_link is not None: + owner_key, fingerprint = merge_result.submission_link + _settle_submission_links_conn( + conn, + normalized_host or "", + owner_key, + fingerprint, + now=authority_time, + ) + turn = TurnRefreshApplyResult(merge_result.updated, False) + fault("after_turn_projection") + fault("before_commit") + conn.commit() + except Exception: + conn.rollback() + raise + if rearm_key is not None: + _rearm_submission_link_component( + db_path, + normalized_host or "", + rearm_key[0], + rearm_key[1], + ) + return AppendProjectedAgentEventResult( + event=AppendBoundAgentEventResult( + status="inserted" if appended.inserted else "replayed", + event_id=appended.event_id, + sequence=appended.sequence, + ), + turn=turn, + ) + + def apply_turn_refresh( db_path: Path | str, host_id: str, worker_id: str, content: Mapping[str, Any], *, - backend_pending: Mapping[str, Any] | None | object = _UNSET, backend_pending_observation: PendingObservation | object = _UNSET, expected_binding: WorkerBinding | None = None, deadline_monotonic: float | None = None, cancelled: Callable[[], bool] | None = None, observed_at: str | None = None, pending_stale_grace_seconds: float = DEFAULT_PENDING_STALE_GRACE_SECONDS, - turn_model: str = DEFAULT_TURN_MODEL, ) -> TurnRefreshApplyResult: """Atomically apply one binding's turn observation and optional pending state.""" if not _sqlite_store_exists(db_path): return TurnRefreshApplyResult(0, False) - normalized_turn_model = str(turn_model or "").strip().lower() - if normalized_turn_model not in TURN_MODELS: - allowed = ", ".join(sorted(TURN_MODELS)) - raise ValueError(f"turn_model must be one of: {allowed}") current_time, _ = _pending_observed_time(observed_at) - if ( - backend_pending is not _UNSET - and backend_pending_observation is not _UNSET - ): - raise ValueError("provide only one backend pending observation") with _connect(db_path, isolation_level=None) as conn: _ensure_schema(conn) # Transaction acquisition must precede every merge-base read. Use @@ -22319,7 +19177,6 @@ def apply_turn_refresh( str(worker_id), content, observed_at=current_time, - turn_model=normalized_turn_model, ) updated = merge_result.updated rearm_key = ( @@ -22338,7 +19195,7 @@ def apply_turn_refresh( owner_key, fingerprint = merge_result.submission_link conn.execute("SAVEPOINT settle_submission_links") try: - settle_submission_links_conn( + _settle_submission_links_conn( conn, str(host_id), owner_key, @@ -22387,14 +19244,6 @@ def apply_turn_refresh( # pane binding for the same stable worker. binding_authoritative=expected_binding is not None, ) - elif backend_pending is not _UNSET: - pending_changed = _merge_backend_pending_conn( - conn, - str(host_id), - str(worker_id), - backend_pending, - observed_at=current_time, - ) else: pending_changed = False if _turn_refresh_is_cancelled( @@ -22410,24 +19259,6 @@ def apply_turn_refresh( raise -def merge_turn_content( - db_path: Path | str, - host_id: str, - worker_id: str, - content: Mapping[str, Any], - *, - observed_at: str | None = None, - turn_model: str = DEFAULT_TURN_MODEL, -) -> int: - """Compatibility wrapper for the transactional authoritative turn merge.""" - return apply_turn_refresh( - db_path, - host_id, - worker_id, - content, - observed_at=observed_at, - turn_model=turn_model, - ).updated def _update_turn_row( @@ -22502,7 +19333,8 @@ def _update_turn_row( CASE WHEN revisions.final_state != 'absent' AND NOT (revisions.final_state = 'complete' AND revisions.final_char_length BETWEEN 1 AND :text_max) - THEN substr(revisions.assistant_final_text, 1, :preview_max) END + THEN substr(revisions.assistant_final_text, 1, :preview_max) END, + linked_submission.submission_id, linked_submission.state """ @@ -22515,7 +19347,8 @@ def _turn_delta_projection( revision, user_state, user_char_length, user_byte_length, user_page_count, user_inline, user_preview, final_state, final_char_length, final_byte_length, final_page_count, final_inline, final_preview, - ) = row[:18] + submission_id, submission_state, + ) = row[:20] try: loaded = json.loads(str(payload_json or "{}")) except (TypeError, json.JSONDecodeError): @@ -22548,16 +19381,9 @@ def _turn_delta_projection( final_byte_length=int(final_byte_length), final_page_count=int(final_page_count), final_inline=final_inline, final_preview=final_preview, )) - else: - legacy_user, legacy_user_state = _legacy_canonical_field(serialized.get("user_text")) - legacy_final, legacy_final_state = _legacy_canonical_field( - serialized.get("assistant_final_text") - ) - if legacy_user_state != "absent" or legacy_final_state != "absent": - item.update(project_turn_content( - str(turn_id), legacy_user, legacy_final, - user_state=legacy_user_state, final_state=legacy_final_state, - )) + if submission_id is not None and submission_state == "linked": + item["submission_id"] = str(submission_id) + item["submission_state"] = "linked" item["schema_version"] = TURN_DELTA_PROJECTION_SCHEMA_VERSION return item, dict(loaded) @@ -22611,6 +19437,8 @@ def _turn_delta_descriptor_only(projected: Mapping[str, Any]) -> dict[str, Any]: "superseded_by_turn_id", "superseded_at", "fingerprint", + "submission_id", + "submission_state", ) bounded = {key: projected[key] for key in descriptor_keys if key in projected} raw_content = projected.get("content") @@ -22655,7 +19483,6 @@ def _turn_delta_payload_from_store( batch_sequence_ceiling: int = TURN_DELTA_MAX_BATCH_SEQUENCES, bootstrap_max_rows: int = TURN_DELTA_BOOTSTRAP_MAX_ROWS, bootstrap_max_pages: int = TURN_DELTA_BOOTSTRAP_MAX_PAGES, - turn_model: str = DEFAULT_TURN_MODEL, ) -> dict[str, Any]: """Return one atomic, frozen, byte-bounded public turn-delta page.""" started = time.perf_counter() @@ -22702,28 +19529,27 @@ def _turn_delta_payload_from_store( ), } - if _submission_linking_enabled(turn_model): - sweep_key, sweep_due = _reserve_lazy_submission_link_sweep( - db_path, - host, - purpose="submission_links", - current_clock=clock, - refresh_interval_seconds=DEFAULT_TURN_REFRESH_INTERVAL_SECONDS, - ) - if sweep_due: - try: - sweep_submission_links( - Path(db_path), - host_id=host, - now=datetime.fromtimestamp(clock, tz=timezone.utc).isoformat(), - ) - except Exception: - _release_failed_submission_link_sweep( - sweep_key, - current_clock=clock, - ) - # Delta remains available if opportunistic linkage is contended. - pass + sweep_key, sweep_due = _reserve_lazy_submission_link_sweep( + db_path, + host, + purpose="submission_links", + current_clock=clock, + refresh_interval_seconds=DEFAULT_SUBMISSION_LINK_SWEEP_INTERVAL_SECONDS, + ) + if sweep_due: + try: + _sweep_submission_links( + Path(db_path), + host_id=host, + now=datetime.fromtimestamp(clock, tz=timezone.utc).isoformat(), + ) + except Exception: + _release_failed_submission_link_sweep( + sweep_key, + current_clock=clock, + ) + # Delta remains available if opportunistic linkage is contended. + pass with _connect(db_path, isolation_level=None) as conn: _ensure_schema(conn) @@ -22824,6 +19650,10 @@ def _turn_delta_payload_from_store( ON revisions.host_id = turns.host_id AND revisions.turn_id = turns.turn_id AND revisions.is_current = 1 + LEFT JOIN turn_submissions AS linked_submission + ON linked_submission.host_id = turns.host_id + AND linked_submission.linked_turn_id = turns.turn_id + AND linked_submission.state = 'linked' WHERE turns.host_id = :host_id AND turns.list_sequence <= :insertion_high AND COALESCE(json_extract(turns.payload_json, '$.superseded_at'), '') = '' @@ -22874,13 +19704,17 @@ def _turn_delta_payload_from_store( ON revisions.host_id = turns.host_id AND revisions.turn_id = turns.turn_id AND revisions.is_current = 1 + LEFT JOIN turn_submissions AS linked_submission + ON linked_submission.host_id = turns.host_id + AND linked_submission.linked_turn_id = turns.turn_id + AND linked_submission.state = 'linked' ORDER BY positioned.seq, positioned.turn_id """, params).fetchall() if work_counters is not None: if mode == "changes": work_counters.journal_queries += 1 work_counters.journal_rows_scanned += ( - int(rows[0][21]) if rows else 0 + int(rows[0][23]) if rows else 0 ) work_counters.projection_queries += 1 work_counters.projection_rows_read += len(rows) @@ -22902,9 +19736,9 @@ def _turn_delta_payload_from_store( worker, sequence, turn_id = str(row[3]), int(row[4]), str(row[2]) changed_at = str(row[1] or utc_timestamp()) else: - worker, sequence, turn_id = "", int(row[18]), str(row[19]) - changed_at = str(row[20] or utc_timestamp()) - projected, raw_payload = _turn_delta_projection(tuple(row[:18])) + worker, sequence, turn_id = "", int(row[20]), str(row[21]) + changed_at = str(row[22] or utc_timestamp()) + projected, raw_payload = _turn_delta_projection(tuple(row[:20])) tombstoned = bool(str(raw_payload.get("superseded_at") or "").strip()) if mode == "changes" and ( row[2] is None or tombstoned or projected is None @@ -22959,7 +19793,7 @@ def _turn_delta_payload_from_store( "checkpoint": checkpoint, "aggregate": { "journal_rows_scanned": ( - int(rows[0][21]) if mode == "changes" and rows else 0 + int(rows[0][23]) if mode == "changes" and rows else 0 ), "projection_rows_read": len(rows), "changes_returned": len(changes), @@ -22999,7 +19833,6 @@ def turn_delta_payload_from_store( batch_sequence_ceiling: int = TURN_DELTA_MAX_BATCH_SEQUENCES, bootstrap_max_rows: int = TURN_DELTA_BOOTSTRAP_MAX_ROWS, bootstrap_max_pages: int = TURN_DELTA_BOOTSTRAP_MAX_PAGES, - turn_model: str = DEFAULT_TURN_MODEL, ) -> dict[str, Any]: """Fail closed to the documented public outcome for unavailable stores.""" try: @@ -23014,7 +19847,6 @@ def turn_delta_payload_from_store( batch_sequence_ceiling=batch_sequence_ceiling, bootstrap_max_rows=bootstrap_max_rows, bootstrap_max_pages=bootstrap_max_pages, - turn_model=turn_model, ) except (sqlite3.Error, StoreSchemaError, LocalStateError, OSError): return { @@ -23038,7 +19870,6 @@ def turns_payload_from_store( now: float | int | None = None, work_counters: TurnContentWorkCounters | None = None, turn_refresh_interval_seconds: float = 2.0, - turn_model: str = DEFAULT_TURN_MODEL, ) -> dict[str, Any]: """Return one insertion-stable, byte-bounded turn-list page.""" requested_schema = int(schema_version) @@ -23071,31 +19902,30 @@ def turns_payload_from_store( } current_clock = time.time() if now is None else float(now) refresh_interval = float(turn_refresh_interval_seconds) - if _submission_linking_enabled(turn_model): - submission_sweep_key, submission_sweep_due = _reserve_lazy_submission_link_sweep( - db_path, - str(host_id), - purpose="submission_links", - current_clock=current_clock, - refresh_interval_seconds=refresh_interval, - ) - if submission_sweep_due: - try: - sweep_submission_links( - Path(db_path), - host_id=str(host_id), - now=datetime.fromtimestamp( - current_clock, - tz=timezone.utc, - ).isoformat(), - ) - except Exception: - _release_failed_submission_link_sweep( - submission_sweep_key, - current_clock=current_clock, - ) - # Listing remains available if opportunistic maintenance is contended. - pass + submission_sweep_key, submission_sweep_due = _reserve_lazy_submission_link_sweep( + db_path, + str(host_id), + purpose="submission_links", + current_clock=current_clock, + refresh_interval_seconds=refresh_interval, + ) + if submission_sweep_due: + try: + _sweep_submission_links( + Path(db_path), + host_id=str(host_id), + now=datetime.fromtimestamp( + current_clock, + tz=timezone.utc, + ).isoformat(), + ) + except Exception: + _release_failed_submission_link_sweep( + submission_sweep_key, + current_clock=current_clock, + ) + # Listing remains available if opportunistic maintenance is contended. + pass try: decoded_cursor = ( decode_turn_list_cursor( @@ -23410,34 +20240,6 @@ def turns_payload_from_store( for descriptor in fields.values() ) item.update(projection) - else: - legacy_user, legacy_user_state = _legacy_canonical_field( - serialized.get("user_text") - ) - legacy_final, legacy_final_state = _legacy_canonical_field( - serialized.get("assistant_final_text") - ) - if requested_schema == TURN_LIST_SCHEMA_VERSION and ( - legacy_user_state != "absent" or legacy_final_state != "absent" - ): - item.update( - project_turn_content( - str(turn_id), - legacy_user, - legacy_final, - user_state=legacy_user_state, - final_state=legacy_final_state, - ) - ) - elif requested_schema == 1: - incompatible_v1 = incompatible_v1 or ( - legacy_user_state == "known_incomplete" - or legacy_final_state == "known_incomplete" - ) - if legacy_user_state == "complete": - item["user_text"] = legacy_user - if legacy_final_state == "complete": - item["assistant_final_text"] = legacy_final if requested_schema == 1: item["schema_version"] = 1 item.pop("content", None) @@ -23679,7 +20481,7 @@ def _ensure_turn_content_page_boundaries_conn( def _backfill_missing_turn_content_page_boundaries_conn( conn: sqlite3.Connection, ) -> int: - """Stream complete legacy fields once and persist exact non-content boundaries.""" + """Repair missing page boundaries for complete current revisions.""" repaired_fields = 0 cursor = conn.execute( """ @@ -23797,7 +20599,6 @@ def get_turn_content( field: str, cursor: str | None = None, schema_version: int = 1, - turn_model: str = DEFAULT_TURN_MODEL, work_counters: TurnContentWorkCounters | None = None, ) -> dict[str, Any]: """Read one bounded page directly from the canonical SQLite value.""" @@ -24198,15 +20999,7 @@ def _snapshot_projection_refresh_required( _snapshot_projection_freshness(payload_data) ): return True - retained_created_at = str(latest[2]) - retained_at = _strict_utc_timestamp(retained_created_at) - return retained_at is None or ( - retained_at == _LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE - and not _legacy_snapshot_created_at_is_authoritative( - retained_created_at, - latest[3], - ) - ) + return _strict_utc_timestamp(latest[2]) is None def save_snapshot( @@ -24218,13 +21011,8 @@ def save_snapshot( binding_backend: str | None = None, binding_observation_authoritative: bool = False, binding_workers_present: bool = True, - turn_model: str = DEFAULT_TURN_MODEL, ) -> bool: """Persist a canonical snapshot; return whether it became the host projection.""" - normalized_turn_model = str(turn_model or "").strip().lower() - if normalized_turn_model not in TURN_MODELS: - allowed = ", ".join(sorted(TURN_MODELS)) - raise ValueError(f"turn_model must be one of: {allowed}") context = observation or SnapshotObservationContext() binding_list = ( None @@ -24323,17 +21111,7 @@ def save_snapshot( if latest is not None else None ) - retained_is_unknown = ( - latest is None - or retained_at is None - or ( - retained_at == _LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE - and not _legacy_snapshot_created_at_is_authoritative( - retained_created_at, - latest[3], - ) - ) - ) + retained_is_unknown = latest is None or retained_at is None refresh_current = retained_is_unknown exact_replay = False if retained_at is not None and not retained_is_unknown: @@ -24590,23 +21368,6 @@ def _attention_item_from_row(row: Any) -> dict[str, Any]: ) -def list_attention_items( - db_path: Path, - host_id: str, - *, - include_resolved: bool = False, -) -> list[dict[str, Any]]: - """Return public-safe persisted attention items for a host.""" - if not _sqlite_store_exists(db_path): - return [] - with _connect(db_path) as conn: - _ensure_schema(conn) - rows = _attention_rows_conn( - conn, - host_id, - include_resolved=include_resolved, - ) - return sanitize_public_value([_attention_item_from_row(row) for row in rows]) def attention_payload_from_store( @@ -24705,16 +21466,6 @@ def attention_payload_from_store( return sanitize_public_value(payload) -def list_hosts(db_path: Path) -> list[str]: - """Return distinct host_ids seen in the store.""" - if not _sqlite_store_exists(db_path): - return [] - with _connect(db_path) as conn: - _ensure_schema(conn) - rows = conn.execute( - "SELECT DISTINCT host_id FROM snapshots ORDER BY host_id" - ).fetchall() - return sanitize_public_value([row[0] for row in rows]) def upsert_worker_bindings(db_path: Path, bindings: Iterable[WorkerBinding]) -> int: @@ -24783,56 +21534,6 @@ def list_worker_bindings( return [_worker_binding_from_row(row) for row in rows] -def resolve_worker_binding( - db_path: Path, - host_id: str, - worker_id: str, - *, - worker_fingerprint: str | None = None, - backend: str | None = None, - now: str | None = None, -) -> WorkerBinding | None: - """Resolve a single current, sendable private binding for a public worker.""" - if not _sqlite_store_exists(db_path): - return None - current_time = now or utc_timestamp() - clauses = ["host_id = ?", "worker_id = ?", "sendable = 1", "expires_at > ?"] - params: list[Any] = [str(host_id), str(worker_id), current_time] - if worker_fingerprint: - clauses.append("worker_fingerprint = ?") - params.append(str(worker_fingerprint)) - if backend is not None: - clauses.append("backend = ?") - params.append(str(backend)) - where = " AND ".join(clauses) - with _connect(db_path) as conn: - _ensure_schema(conn) - rows = conn.execute( - f""" - SELECT - host_id, - worker_id, - worker_fingerprint, - backend, - target_kind, - target_value, - turn_target_kind, - turn_target_value, - sendable, - reason, - observed_at, - expires_at, - private_fingerprint - FROM worker_bindings - WHERE {where} - ORDER BY observed_at DESC, id DESC - LIMIT 2 - """, - params, - ).fetchall() - if len(rows) != 1: - return None - return _worker_binding_from_row(rows[0]) def expire_worker_bindings( @@ -24969,15 +21670,9 @@ def _canonical_request_matches( canonical_fingerprint: str, canonical_request_json: str, public_worker_id: str, - legacy_raw_payload_fingerprint: str | None, ) -> bool: if str(row[3]) != str(action): return False - if int(row[4]) == 0: - return ( - legacy_raw_payload_fingerprint is not None - and str(row[5]) == str(legacy_raw_payload_fingerprint) - ) return ( int(row[4]) == int(canonical_version) and str(row[5]) == str(canonical_fingerprint) @@ -25012,7 +21707,6 @@ def reserve_command_request( public_worker_id: str, pending_result_json: str, selector_proof: str = "", - legacy_raw_payload_fingerprint: str | None = None, owner_lease_seconds: float = COMMAND_RECEIPT_OWNER_LEASE_SECONDS, now: str | None = None, ) -> dict[str, Any]: @@ -25057,9 +21751,6 @@ def reserve_command_request( conn.execute("BEGIN IMMEDIATE") row = _command_request_row(conn, values["host_id"], values["request_id"]) if row is not None: - if bool(row[19]): - conn.commit() - return _command_request_response("terminal", row) if not _canonical_request_matches( row, action=values["action"], @@ -25067,7 +21758,6 @@ def reserve_command_request( canonical_fingerprint=values["canonical_fingerprint"], canonical_request_json=str(canonical_request_json), public_worker_id=str(public_worker_id), - legacy_raw_payload_fingerprint=legacy_raw_payload_fingerprint, ): conn.commit() return _command_request_response("request_id_conflict", row) @@ -25121,11 +21811,10 @@ def reserve_command_request( public_worker_id, state, status, result_json, owner_token_hash, owner_expires_at, binding_fingerprint, created_at, reserved_at, send_started_at, terminal_at, - updated_at, legacy_collision, legacy_collision_count, - selector_proof + updated_at, selector_proof ) VALUES ( ?, ?, ?, ?, ?, ?, ?, 'reserved', 'pending', ?, ?, ?, NULL, - ?, ?, NULL, NULL, ?, 0, 0, ? + ?, ?, NULL, NULL, ?, ? ) """, ( @@ -25218,7 +21907,6 @@ def reserve_terminal_command_replay( status: str, result_json: str, selector_proof: str = "", - legacy_raw_payload_fingerprint: str | None = None, event_payload: Mapping[str, Any] | None = None, now: str | None = None, ) -> dict[str, Any]: @@ -25265,9 +21953,6 @@ def reserve_terminal_command_replay( conn.execute("BEGIN IMMEDIATE") row = _command_request_row(conn, values["host_id"], values["request_id"]) if row is not None: - if bool(row[19]): - conn.commit() - return _command_request_response("terminal", row) if not _canonical_request_matches( row, action=values["action"], @@ -25275,7 +21960,6 @@ def reserve_terminal_command_replay( canonical_fingerprint=values["canonical_fingerprint"], canonical_request_json=str(canonical_request_json), public_worker_id=str(public_worker_id), - legacy_raw_payload_fingerprint=legacy_raw_payload_fingerprint, ): conn.commit() return _command_request_response("request_id_conflict", row) @@ -25293,11 +21977,10 @@ def reserve_terminal_command_replay( public_worker_id, state, status, result_json, owner_token_hash, owner_expires_at, binding_fingerprint, created_at, reserved_at, send_started_at, terminal_at, - updated_at, legacy_collision, legacy_collision_count, - selector_proof + updated_at, selector_proof ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', NULL, NULL, - ?, ?, NULL, ?, ?, 0, 0, ? + ?, ?, NULL, ?, ?, ? ) """, ( @@ -25338,37 +22021,9 @@ def reserve_terminal_command_replay( conn.close() -def sweep_expired_turn_submissions( - db_path: Path, - *, - host_id: str | None = None, - now: str | None = None, -) -> int: - """Expire unlinked shadow submissions past their precomputed hard TTL. - - This Stage-2 store hook is intentionally caller-driven until a later - lifecycle stage wires submission maintenance into the daemon scheduler. - """ - if not _sqlite_store_exists(db_path): - return 0 - current = _command_request_now(now) - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - expired = _expire_turn_submissions_conn( - conn, - current=current, - host_id=None if host_id is None else str(host_id), - ) - conn.commit() - return expired - except Exception: - conn.rollback() - raise -def sweep_submission_links( +def _sweep_submission_links( db_path: Path, *, host_id: str | None = None, @@ -25427,7 +22082,7 @@ def settle_submission_link_for_request( conn.commit() return None owner_key, fingerprint = map(str, component) - changed = settle_submission_links_conn( + changed = _settle_submission_links_conn( conn, str(host_id), owner_key, @@ -25507,50 +22162,6 @@ def linked_turn_for_submission( return payload -def cancel_turn_submission( - db_path: Path, - *, - host_id: str, - request_id: str, - now: str | None = None, -) -> bool: - """Apply the shadow-ledger side of an authoritative request cancellation. - - This Stage-2 store hook is intentionally caller-driven until a later stage - introduces an authoritative production cancellation workflow. - """ - if not _sqlite_store_exists(db_path): - return False - current = _command_request_now(now) - source_states = _turn_submission_transition_sources("cancelled") - if not source_states: - return False - state_placeholders = ", ".join("?" for _ in source_states) - with _connect(db_path, isolation_level=None) as conn: - _ensure_schema(conn) - conn.execute("BEGIN IMMEDIATE") - try: - updated = conn.execute( - f""" - UPDATE turn_submissions - SET state = 'cancelled', terminal_at = ?, updated_at = ? - WHERE host_id = ? AND request_id = ? - AND linked_turn_id IS NULL - AND state IN ({state_placeholders}) - """, - ( - current, - current, - str(host_id), - str(request_id), - *source_states, - ), - ) - conn.commit() - return int(updated.rowcount or 0) == 1 - except Exception: - conn.rollback() - raise def mark_command_send_started( @@ -26334,7 +22945,7 @@ def cleanup_command_request_retention( public_worker_id, state, status, result_json, owner_token_hash, owner_expires_at, binding_fingerprint, created_at, reserved_at, send_started_at, terminal_at, - updated_at, legacy_collision, legacy_collision_count + updated_at, selector_proof FROM command_receipts WHERE id = ? """, diff --git a/tests/fixtures/acp_fake_agent.py b/tests/fixtures/acp_fake_agent.py new file mode 100644 index 0000000..5b5dfc6 --- /dev/null +++ b/tests/fixtures/acp_fake_agent.py @@ -0,0 +1,314 @@ +"""Deterministic newline-delimited JSON-RPC peer used by ACP client tests.""" + +from __future__ import annotations + +import json +import signal +import sys +import time + + +MODE = sys.argv[1] if len(sys.argv) > 1 else "normal" + + +def send(value: object) -> None: + sys.stdout.write(json.dumps(value, separators=(",", ":")) + "\n") + sys.stdout.flush() + + +def response(request_id: object, result: object) -> None: + send({"jsonrpc": "2.0", "id": request_id, "result": result}) + + +def update(session_id: str, kind: str, **values: object) -> None: + send( + { + "jsonrpc": "2.0", + "method": "session/update", + "params": { + "sessionId": session_id, + "update": {"sessionUpdate": kind, **values}, + }, + } + ) + + +pending_prompt_id: object | None = None +pending_prompt_session = "" +pending_permission_ids: set[object] = set() + +if MODE == "no_read": + time.sleep(60) + +if MODE == "stubborn": + signal.signal(signal.SIGTERM, signal.SIG_IGN) + +for line in sys.stdin: + message = json.loads(line) + method = message.get("method") + request_id = message.get("id") + params = message.get("params", {}) + + if MODE == "initialize_only" and method != "initialize": + send( + { + "jsonrpc": "2.0", + "id": request_id, + "error": {"code": -32601, "message": "Method not found"}, + } + ) + continue + + if method == "initialize": + if MODE == "malformed": + sys.stdout.write("not-json\n") + sys.stdout.flush() + continue + if MODE == "partial_eof": + sys.stdout.write('{"jsonrpc":"2.0","id":') + sys.stdout.flush() + raise SystemExit(0) + if MODE == "oversize": + response(request_id, {"protocolVersion": 1, "padding": "x" * 10000}) + continue + response( + request_id, + { + "protocolVersion": True if MODE == "bool_version" else 1, + "agentCapabilities": ( + {} + if MODE in {"baseline", "initialize_only"} + else { + "loadSession": True, + "sessionCapabilities": { + "list": {}, + "delete": {}, + "resume": {}, + "close": {}, + "additionalDirectories": {}, + }, + "promptCapabilities": { + "image": True, + "audio": True, + "embeddedContext": True, + }, + "mcpCapabilities": {"http": True, "sse": True}, + "_meta": {"vendor.example": {"level": 2}}, + } + ), + "agentInfo": {"name": "fake", "version": "1.0"}, + **( + {"_meta": {"steering": {"supported": True}}} + if MODE == "steering" + else {} + ), + **( + { + "authMethods": [ + {}, + {"id": "missing-name"}, + {"id": "valid", "name": "Valid agent login"}, + {"type": "future", "id": "x", "name": "Future"}, + ] + } + if MODE == "auth_shapes" + else {} + ), + }, + ) + if MODE == "extensions": + send( + { + "jsonrpc": "2.0", + "method": "_vendor.example/future_notification", + "params": {"opaque": {"revision": 9}}, + } + ) + if MODE == "extension_flood": + for index in range(32): + send( + { + "jsonrpc": "2.0", + "method": "_vendor.example/progress", + "params": {"sequence": index}, + } + ) + if MODE in {"unknown_request", "supported_request"}: + send( + { + "jsonrpc": "2.0", + "id": 777 if MODE == "unknown_request" else 778, + "method": "_vendor.example/request", + "params": {"opaque": True}, + } + ) + if MODE == "null_response": + send( + { + "jsonrpc": "2.0", + "id": None, + "error": {"code": -32600, "message": "unrelated invalid request"}, + } + ) + if MODE == "stderr_tail": + sys.stderr.write("prefix-" + "x" * 500 + "-TAIL") + sys.stderr.flush() + if MODE == "exit_after_init": + time.sleep(0.05) + raise SystemExit(0) + if MODE == "flood": + time.sleep(0.05) + for index in range(4): + update( + "s-flood", + "agent_message_chunk", + content={"type": "text", "text": str(index)}, + ) + elif method == "session/new": + update("s-new", "agent_message_chunk", content={"type": "text", "text": "hi"}) + response( + request_id, + { + "sessionId": "s-new", + "modes": { + "currentModeId": "default", + "availableModes": [{"id": "default", "name": "Default"}], + }, + }, + ) + elif method == "session/load" or method == "session/resume": + if MODE == "load_replay" and method == "session/load": + for index in range(64): + update( + params["sessionId"], + "user_message_chunk" if index % 2 == 0 else "agent_message_chunk", + messageId=f"replay-{index}", + content={"type": "text", "text": str(index)}, + ) + response( + request_id, + { + "configOptions": [ + { + "id": "model", + "name": "Model", + "type": "select", + "currentValue": "x", + "options": [{"value": "x", "name": "Model X"}], + } + ] + }, + ) + elif method == "session/close" or method == "session/delete": + response(request_id, {"_meta": {"vendor.example": {"receipt": method}}}) + elif method == "session/list": + if MODE == "slow": + time.sleep(2) + continue + cursor = params.get("cursor") + response( + request_id, + { + "sessions": [ + { + "sessionId": "s2" if cursor else "s1", + "cwd": "/tmp/project", + "title": "second" if cursor else "first", + } + ], + **({} if cursor else {"nextCursor": "page-2"}), + }, + ) + elif method == "session/prompt": + if MODE == "echo_prompt": + response(request_id, {"stopReason": "end_turn"}) + continue + pending_prompt_id = request_id + pending_prompt_session = params["sessionId"] + update( + pending_prompt_session, + "agent_thought_chunk", + content={"type": "text", "text": "reasoning summary"}, + ) + send( + { + "jsonrpc": "2.0", + "id": 900, + "method": "session/request_permission", + "params": { + "sessionId": pending_prompt_session, + "toolCall": {"toolCallId": "tool-1", "status": "pending"}, + "options": [ + { + "optionId": "allow", + "name": "Allow once", + "kind": "allow_once", + }, + { + "optionId": "reject", + "name": "Reject once", + "kind": "reject_once", + }, + ], + }, + } + ) + pending_permission_ids.add(900) + elif method == "_session/steering": + update( + params["sessionId"], + "user_message_chunk", + content=params["prompt"][0], + ) + response(request_id, {"outcome": "injected"}) + elif method == "session/cancel": + if MODE == "cancel_race" and pending_prompt_id is not None: + send( + { + "jsonrpc": "2.0", + "id": 901, + "method": "session/request_permission", + "params": { + "sessionId": pending_prompt_session, + "toolCall": {"toolCallId": "tool-race", "status": "pending"}, + "options": [ + { + "optionId": "allow-race", + "name": "Allow once", + "kind": "allow_once", + } + ], + }, + } + ) + pending_permission_ids.add(901) + elif request_id in {777, 778}: + error = message.get("error", {}) + update( + "s-extension", + "agent_message_chunk", + content={ + "type": "text", + "text": "method-not-found" if error.get("code") == -32601 else "unexpected", + }, + ) + elif request_id in pending_permission_ids and pending_prompt_id is not None: + outcome = message["result"]["outcome"]["outcome"] + pending_permission_ids.remove(request_id) + if not pending_permission_ids: + update( + pending_prompt_session, + "plan", + entries=[ + {"content": "done", "priority": "medium", "status": "completed"} + ], + ) + response( + pending_prompt_id, + {"stopReason": "cancelled" if outcome == "cancelled" else "end_turn"}, + ) + pending_prompt_id = None + +if MODE == "stubborn": + while True: + time.sleep(60) diff --git a/tests/fixtures/herdr_acp_contract_v1.json b/tests/fixtures/herdr_acp_contract_v1.json new file mode 100644 index 0000000..37c3a73 --- /dev/null +++ b/tests/fixtures/herdr_acp_contract_v1.json @@ -0,0 +1,41 @@ +{ + "request": { + "id": "tw:acp:endpoint", + "method": "agent.acp_endpoint", + "params": {"target": "term_abc"} + }, + "result": { + "type": "agent_acp_endpoint", + "endpoint": { + "transport": "stdio", + "command": "herdr", + "args": [ + "agent", + "acp-attach", + "term_abc", + "--generation", + "42", + "--ticket", + "1234567890123456789012345678901234567890123" + ], + "protocol_version": 1 + }, + "console": { + "generation": 42, + "lease": "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQ" + }, + "worker": { + "terminal_id": "term_abc", + "workspace_id": "w1", + "tab_id": "w1:t1", + "pane_id": "w1:p1", + "name": "reviewer", + "agent": "codex", + "generation": 42 + }, + "adapter": {"name": "codex-acp", "version": "0.13.3"}, + "session": {"mode": "new"}, + "cwd": "/work", + "lifecycle": "acp_owned_ready" + } +} diff --git a/tests/store_helpers.py b/tests/store_helpers.py new file mode 100644 index 0000000..b2ca539 --- /dev/null +++ b/tests/store_helpers.py @@ -0,0 +1,128 @@ +"""Test adapters onto supported store entry points.""" + +from __future__ import annotations + +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +from tendwire.core.agent_events import AgentEvent, AppendAgentEventResult +from tendwire.core.models import sanitize_public_mapping, stable_fingerprint +from tendwire.core.turns import PendingObservation, PendingObservedChoice +from tendwire.store.sqlite import ( + apply_turn_refresh, + attention_payload_from_store, + list_agent_events, + record_agent_event, +) + + +def apply_test_turn_refresh( + db_path: Path | str, + host_id: str, + worker_id: str, + content: Mapping[str, Any], + *, + observed_at: str | None = None, +) -> int: + return apply_turn_refresh( + db_path, + host_id, + worker_id, + content, + observed_at=observed_at, + ).updated + + +def apply_test_backend_pending( + db_path: Path | str, + host_id: str, + worker_id: str, + pending: Mapping[str, Any] | None, +) -> bool: + if pending is None: + observation = PendingObservation("read_succeeded_no_prompt") + else: + clean = sanitize_public_mapping(pending) + choices = tuple( + PendingObservedChoice( + choice_id=( + "choice-" + + stable_fingerprint( + { + "domain": "test.pending-choice.v1", + "ordinal": ordinal, + "choice": choice, + }, + length=24, + ) + ), + label=str(choice.get("label") or "Option"), + picker_ordinal=ordinal, + ) + for ordinal, choice in enumerate(clean.get("choices", ()), 1) + if isinstance(choice, Mapping) + ) + observation = PendingObservation( + "open_prompt", + question=str(clean.get("question") or "Pending action"), + pending_kind=str(clean.get("kind") or "question"), + choices=choices, + revision_digest=stable_fingerprint( + {"domain": "test.pending-observation.v1", "payload": clean} + ), + ) + return apply_turn_refresh( + db_path, + host_id, + worker_id, + {}, + backend_pending_observation=observation, + ).pending_changed + + +def read_test_attention_items( + db_path: Path, + host_id: str, + *, + include_resolved: bool = False, +) -> list[dict[str, Any]]: + payload = attention_payload_from_store( + db_path, + host_id, + include_resolved=include_resolved, + ) + return [] if payload is None else list(payload["attention"]) + + +def record_test_agent_event( + db_path: Path | str, + host_id: str, + event: AgentEvent, +) -> AppendAgentEventResult: + return record_agent_event( + db_path, + host_id, + kind=event.kind, + source=event.source, + worker_id=event.worker_id, + payload=event.payload, + source_session_id=event.source_session_id, + source_turn_id=event.source_turn_id, + source_item_id=event.source_item_id, + source_message_id=event.source_message_id, + source_event_id=event.source_event_id, + source_sequence=event.source_sequence, + visibility=event.visibility, + observed_at=event.observed_at, + ) + + +def read_public_test_agent_events( + db_path: Path | str, + host_id: str, +) -> tuple[dict[str, Any], ...]: + return tuple( + item.public_dict() + for item in list_agent_events(db_path, host_id, visibility="public") + ) diff --git a/tests/test_acp_atomic_ingestion.py b/tests/test_acp_atomic_ingestion.py new file mode 100644 index 0000000..cd7eb4e --- /dev/null +++ b/tests/test_acp_atomic_ingestion.py @@ -0,0 +1,439 @@ +"""Atomicity and recovery tests for ACP journal-to-turn ingestion.""" + +from __future__ import annotations + +import sqlite3 +from dataclasses import replace +from pathlib import Path + +import pytest + +from tendwire.backends.acp_ingestion import AcpSessionIngestor +from tendwire.config import Config +from tendwire.core.agent_events import AgentEvent, agent_event +from tendwire.core.models import WorkerBinding +from tendwire.core.projector import project_from_raw +from tendwire.store import sqlite as store_sqlite +from tendwire.store.sqlite import ( + append_agent_event_and_apply_turn_for_binding, + cleanup_agent_event_retention, + init_store, + list_agent_events, + save_snapshot, + turns_payload_from_store, + upsert_worker_bindings, +) + + +def _store( + tmp_path: Path, + *, + stable_owner: bool = False, +) -> tuple[Config, WorkerBinding]: + db_path = tmp_path / "events.db" + config = Config(host_id="host-a", db_path=db_path) + snapshot = project_from_raw( + config, + workers=[ + { + "id": "worker-a", + "name": "Worker A", + "meta": ( + { + "stable_key": "wsk1_" + ("a" * 64), + "stable_key_version": 1, + } + if stable_owner + else {} + ), + } + ], + ) + init_store(db_path) + save_snapshot(db_path, snapshot) + worker = snapshot.workers[0] + binding = WorkerBinding( + host_id=config.host_id, + worker_id=worker.id, + worker_fingerprint=worker.fingerprint, + backend="herdr", + target_kind="pane_id", + target_value="pane-a", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + sendable=True, + private_fingerprint="private-a", + ) + upsert_worker_bindings(db_path, [binding]) + return config, binding + + +def _event( + binding: WorkerBinding, + *, + observed_at: str = "2026-07-31T00:00:00+00:00", + source_turn_id: str = "turn-a", + message_id: str = "message-a", + source_event_id: str = "event-a", + text: str = "answer", +) -> AgentEvent: + return agent_event( + kind="agent_message", + source="acp", + worker_id=binding.worker_id, + payload={"schema_version": 1, "message_id": message_id, "text": text}, + source_session_id="session-a", + source_turn_id=source_turn_id, + source_message_id=message_id, + source_event_id=source_event_id, + observed_at=observed_at, + ) + + +def _content( + *, + complete: bool = False, + source_turn_id: str = "turn-a", + text: str = "answer", +) -> dict[str, object]: + return { + "source_turn_id": source_turn_id, + "user_text": "", + "assistant_stream_text": "" if complete else text, + "assistant_final_text": text if complete else "", + "complete": complete, + "has_open_turn": not complete, + } + + +def _counts(db_path: Path) -> tuple[int, int]: + with sqlite3.connect(db_path) as conn: + events = int(conn.execute("SELECT COUNT(*) FROM agent_events").fetchone()[0]) + turns = int(conn.execute("SELECT COUNT(*) FROM turns").fetchone()[0]) + return events, turns + + +@pytest.mark.parametrize( + "boundary", + ( + "after_binding_check", + "after_event_append", + "after_turn_projection", + "before_commit", + ), +) +def test_every_atomic_boundary_rolls_back_and_retry_succeeds( + tmp_path: Path, + boundary: str, +) -> None: + config, binding = _store(tmp_path) + event = _event(binding) + + def fail(current: str) -> None: + if current == boundary: + raise RuntimeError(boundary) + + with pytest.raises(RuntimeError, match=boundary): + append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(), + _fault_inject=fail, + ) + + assert _counts(config.db_path) == (0, 0) + retried = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(), + ) + assert retried.event.status == "inserted" + assert retried.turn is not None + assert _counts(config.db_path) == (1, 1) + + +def test_stale_binding_writes_neither_journal_nor_projection(tmp_path: Path) -> None: + config, binding = _store(tmp_path) + stale = replace(binding, target_value="old-pane") + result = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + _event(binding), + expected_binding=stale, + content=_content(), + ) + + assert result.event.status == "binding_changed" + assert result.turn is None + assert _counts(config.db_path) == (0, 0) + + + + +def test_replay_cannot_supersede_newer_turn_or_requeue_connector( + tmp_path: Path, +) -> None: + config, binding = _store(tmp_path, stable_owner=True) + old = _event( + binding, + observed_at="2020-01-01T00:00:00+00:00", + source_turn_id="turn-old", + message_id="message-old", + source_event_id="event-old", + text="old", + ) + first = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + old, + expected_binding=binding, + content=_content(complete=True, source_turn_id="turn-old", text="old"), + ) + assert first.event.status == "inserted" + new = _event( + binding, + observed_at="2026-01-01T00:00:00+00:00", + source_turn_id="turn-new", + message_id="message-new", + source_event_id="event-new", + text="new", + ) + second = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + new, + expected_binding=binding, + content=_content(complete=True, source_turn_id="turn-new", text="new"), + ) + assert second.event.status == "inserted" + turns_before = turns_payload_from_store(config.db_path, config.host_id)["turns"] + assert turns_before[0]["assistant_final_text"] == "new" + with sqlite3.connect(config.db_path) as conn: + outbox_before = conn.execute( + "SELECT turn_id, status FROM connector_outbox ORDER BY id" + ).fetchall() + + replayed = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + replace(old, observed_at="2030-01-01T00:00:00+00:00"), + expected_binding=binding, + content=_content(complete=True, source_turn_id="turn-old", text="old"), + ) + + assert replayed.event.status == "replayed" + assert replayed.turn is None + turns = turns_payload_from_store(config.db_path, config.host_id)["turns"] + assert turns == turns_before + with sqlite3.connect(config.db_path) as conn: + assert conn.execute( + "SELECT turn_id, status FROM connector_outbox ORDER BY id" + ).fetchall() == outbox_before + + +def test_replay_repairs_only_absent_projection_with_original_authority_time( + tmp_path: Path, +) -> None: + config, binding = _store(tmp_path, stable_owner=True) + event = _event(binding, observed_at="2020-01-01T00:00:00+00:00") + inserted = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + ) + assert inserted.event.status == "inserted" + repaired = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + replace(event, observed_at="2030-01-01T00:00:00+00:00"), + expected_binding=binding, + content=_content(complete=True), + ) + assert repaired.event.status == "replayed" + assert repaired.turn is not None and repaired.turn.updated == 1 + with sqlite3.connect(config.db_path) as conn: + turn_before = conn.execute( + "SELECT payload_json, observed_at FROM turns" + ).fetchone() + outbox_before = conn.execute( + "SELECT turn_id, status FROM connector_outbox ORDER BY id" + ).fetchall() + assert turn_before is not None + assert turn_before[1] == "2020-01-01T00:00:00+00:00" + + ignored = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + replace(event, observed_at="2050-01-01T00:00:00+00:00"), + expected_binding=binding, + content=_content(complete=True, text="caller rewrite"), + ) + assert ignored.event.status == "replayed" + assert ignored.turn is None + with sqlite3.connect(config.db_path) as conn: + assert conn.execute( + "SELECT payload_json, observed_at FROM turns" + ).fetchone() == turn_before + assert conn.execute( + "SELECT turn_id, status FROM connector_outbox ORDER BY id" + ).fetchall() == outbox_before + + + + +def test_replay_repair_respects_binding_fence_and_rolls_back(tmp_path: Path) -> None: + config, binding = _store(tmp_path) + event = _event(binding) + append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + ) + stale = replace(binding, target_value="old-pane") + fenced = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=stale, + content=_content(), + ) + assert fenced.event.status == "binding_changed" + assert fenced.turn is None + assert _counts(config.db_path) == (1, 0) + + def fail(boundary: str) -> None: + if boundary == "before_commit": + raise RuntimeError("repair rollback") + + with pytest.raises(RuntimeError, match="repair rollback"): + append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(), + _fault_inject=fail, + ) + assert _counts(config.db_path) == (1, 0) + repaired = append_agent_event_and_apply_turn_for_binding( + config.db_path, + config.host_id, + event, + expected_binding=binding, + content=_content(), + ) + assert repaired.event.status == "replayed" + assert repaired.turn is not None + assert _counts(config.db_path) == (1, 1) + + +def _update(text: str) -> dict[str, object]: + return { + "method": "session/update", + "params": { + "sessionId": "session-a", + "update": { + "sessionUpdate": "agent_message_chunk", + "messageId": "message-a", + "content": {"type": "text", "text": text}, + }, + }, + } + + +def test_process_reconstruction_replays_text_then_completes_once( + tmp_path: Path, +) -> None: + config, binding = _store(tmp_path) + first = AcpSessionIngestor( + config, + session_id="session-a", + stream_generation="generation-a", + binding=binding, + ) + first.start_turn(producer_turn_id="producer-a") + first.ingest_update(_update("answer"), source_event_id="message-event-a") + + reconstructed = AcpSessionIngestor( + config, + session_id="session-a", + stream_generation="generation-b", + binding=binding, + ) + reconstructed.start_turn(producer_turn_id="producer-a") + replayed = reconstructed.ingest_update( + _update("answer"), source_event_id="message-event-a", replay=True + ) + completed = reconstructed.mark_prompt_complete() + + assert replayed.event is not None and replayed.event.status == "replayed" + assert completed.event is not None and completed.event.status == "inserted" + turns = turns_payload_from_store(config.db_path, config.host_id)["turns"] + assert len(turns) == 1 + assert turns[0]["assistant_final_text"] == "answer" + with sqlite3.connect(config.db_path) as conn: + before = int( + conn.execute("SELECT COUNT(*) FROM connector_outbox").fetchone()[0] + ) + + second_reconstruction = AcpSessionIngestor( + config, + session_id="session-a", + stream_generation="generation-c", + binding=binding, + ) + second_reconstruction.start_turn(producer_turn_id="producer-a") + second_reconstruction.ingest_update( + _update("answer"), source_event_id="message-event-a", replay=True + ) + completion_replay = second_reconstruction.mark_prompt_complete() + assert completion_replay.event is not None + assert completion_replay.event.status == "replayed" + with sqlite3.connect(config.db_path) as conn: + after = int(conn.execute("SELECT COUNT(*) FROM connector_outbox").fetchone()[0]) + assert after == before + + +def test_completion_failure_rolls_back_marker_and_final_then_retries( + tmp_path: Path, +) -> None: + config, binding = _store(tmp_path) + fail_completion = True + + def persist(*args, **kwargs): + def fault(boundary: str) -> None: + if ( + fail_completion + and args[2].kind == "extension" + and boundary == "before_commit" + ): + raise RuntimeError("completion fault") + + kwargs["_fault_inject"] = fault + return append_agent_event_and_apply_turn_for_binding(*args, **kwargs) + + ingestor = AcpSessionIngestor( + config, + session_id="session-a", + stream_generation="generation-a", + binding=binding, + persist_event=persist, + ) + ingestor.start_turn(producer_turn_id="producer-a") + ingestor.ingest_update(_update("answer"), source_event_id="message-event-a") + with pytest.raises(RuntimeError, match="completion fault"): + ingestor.mark_prompt_complete() + assert len(list_agent_events(config.db_path, config.host_id)) == 1 + + fail_completion = False + completed = ingestor.mark_prompt_complete() + assert completed.event is not None and completed.event.status == "inserted" + assert len(list_agent_events(config.db_path, config.host_id)) == 2 + turn = turns_payload_from_store(config.db_path, config.host_id)["turns"][0] + assert turn["assistant_final_text"] == "answer" diff --git a/tests/test_acp_client.py b/tests/test_acp_client.py new file mode 100644 index 0000000..1e02b95 --- /dev/null +++ b/tests/test_acp_client.py @@ -0,0 +1,767 @@ +from __future__ import annotations + +import sys +import threading +import time +from pathlib import Path + +import pytest + +from tendwire.backends.acp_client import ( + AcpCapabilityError, + BoundedAcpConnection as AcpClient, + AcpEventQueueFullError, + AcpRequestTimeoutError, + AcpTransportError, + ClientState, +) +from tendwire.backends.acp_protocol import ( + AcpProtocolError, + PermissionRequest, + SessionUpdate, + SessionUpdateKind, + StopReason, + SteeringOutcome, +) + + +FAKE_AGENT = Path(__file__).parent / "fixtures" / "acp_fake_agent.py" + + +def client(mode: str = "normal", **kwargs: object) -> AcpClient: + return AcpClient([sys.executable, "-u", str(FAKE_AGENT), mode], **kwargs) + + +def _typed_update(index: int) -> SessionUpdate: + update = { + "sessionUpdate": "agent_message_chunk", + "messageId": f"message-{index}", + "content": {"type": "text", "text": str(index)}, + } + raw = {"sessionId": "s", "update": update} + return SessionUpdate("s", SessionUpdateKind.AGENT_MESSAGE_CHUNK, update, None, raw) + + +def _typed_permission(index: int) -> PermissionRequest: + raw = { + "sessionId": "s", + "toolCall": {"toolCallId": f"tool-{index}"}, + "options": [], + } + return PermissionRequest( + index, + "s", + raw["toolCall"], + (), + None, + raw, + ) + + +def _install_permission(acp: AcpClient, request: PermissionRequest) -> None: + with acp._permission_lock: + acp._pending_permissions[request.request_id] = request + acp._put_session_event(request) + + +def _waiter_has_released_condition(acp: AcpClient, ready: threading.Event) -> None: + assert ready.wait(timeout=1) + # The routing callback signals just before Condition.wait(). Acquiring the + # condition proves that the consumer has actually released it to sleep. + with acp._session_event_condition: + pass + + +def _wait_for_condition_waiters(acp: AcpClient, count: int) -> None: + deadline = time.monotonic() + 1 + while time.monotonic() < deadline: + with acp._session_event_condition: + if len(acp._session_event_condition._waiters) >= count: + return + time.sleep(0.001) + raise AssertionError("session event consumers did not start waiting") + + +def _capture_failure(method, failures: list[BaseException]) -> None: + try: + method() + except BaseException as exc: + failures.append(exc) + + +def test_initialize_capabilities_and_session_lifecycle() -> None: + with client() as acp: + initialized = acp.initialize() + assert initialized.protocol_version == 1 + assert initialized.agent_info == {"name": "fake", "version": "1.0"} + assert initialized.capabilities.load_session + assert initialized.capabilities.session_list + assert initialized.capabilities.session_resume + assert initialized.capabilities.session_close + assert initialized.capabilities.session_delete + assert initialized.capabilities.raw["_meta"] == { + "vendor.example": {"level": 2} + } + assert acp.state is ClientState.INITIALIZED + + created = acp.new_session( + "/tmp/project", + additional_directories=["/tmp/other"], + ) + assert created.session_id == "s-new" + assert created.modes == { + "currentModeId": "default", + "availableModes": [{"id": "default", "name": "Default"}], + } + streamed = acp.next_update(timeout=1) + assert streamed.update_kind is SessionUpdateKind.AGENT_MESSAGE_CHUNK + + loaded = acp.load_session("s1", "/tmp/project") + resumed = acp.resume_session("s1", "/tmp/project") + assert loaded.session_id == "s1" + assert resumed.config_options[0]["id"] == "model" + + first = acp.list_sessions(cwd="/tmp/project") + assert first.sessions[0].session_id == "s1" + assert first.next_cursor == "page-2" + second = acp.list_sessions(cursor=first.next_cursor) + assert second.sessions[0].title == "second" + assert second.next_cursor is None + + assert acp.close_session("s1")["_meta"]["vendor.example"]["receipt"] == "session/close" + assert acp.delete_session("s2")["_meta"]["vendor.example"]["receipt"] == "session/delete" + + assert acp.state is ClientState.CLOSED + assert acp.exit is not None + assert acp.exit.returncode == 0 + + +def test_load_session_accepts_standard_null_result(monkeypatch: pytest.MonkeyPatch) -> None: + """ACP v1 completes load replay with a JSON-RPC null result.""" + + with client() as acp: + acp.initialize() + monkeypatch.setattr(acp, "request", lambda *_args, **_kwargs: None) + loaded = acp.load_session("s1", "/tmp/project") + + assert loaded.session_id == "s1" + assert loaded.modes is None + assert loaded.config_options == () + assert loaded.raw == {} + + +def test_prompt_stream_and_permission_response_can_run_concurrently() -> None: + with client() as acp: + acp.initialize() + acp.new_session("/tmp/project") + # Drain the new-session message update. + acp.next_update(timeout=1) + + outcome: list[object] = [] + failure: list[BaseException] = [] + + def run_prompt() -> None: + try: + outcome.append(acp.prompt("s-new", "please inspect")) + except BaseException as exc: # pragma: no cover - diagnostic path + failure.append(exc) + + thread = threading.Thread(target=run_prompt) + thread.start() + thought = acp.next_update(timeout=1) + assert thought.update_kind is SessionUpdateKind.AGENT_THOUGHT_CHUNK + permission = acp.next_permission_request(timeout=1) + assert permission.options[0].option_id == "allow" + acp.respond_permission(permission.request_id, option_id="allow") + plan = acp.next_update(timeout=1) + assert plan.update_kind is SessionUpdateKind.PLAN + thread.join(timeout=2) + + assert not thread.is_alive() + assert not failure + assert outcome[0].stop_reason is StopReason.END_TURN + + +def test_advertised_steering_extension_is_capability_gated() -> None: + with client("steering") as acp: + acp.initialize() + acp.new_session("/tmp/project") + acp.next_update(timeout=1) + assert acp.steering_supported + callbacks: list[str] = [] + result = acp.steer_session( + "s-new", + "live input", + on_send_start=lambda: callbacks.append("started"), + on_submitted=lambda: callbacks.append("submitted"), + ) + assert result.outcome is SteeringOutcome.INJECTED + assert callbacks == ["started", "submitted"] + echoed = acp.next_update(timeout=1) + assert echoed.update_kind is SessionUpdateKind.USER_MESSAGE_CHUNK + + with client() as acp: + acp.initialize() + assert not acp.steering_supported + with pytest.raises(AcpCapabilityError, match="steering"): + acp.steer_session("s1", "no capability") + + +def test_ordered_session_event_api_preserves_cross_kind_reader_order() -> None: + with client() as acp: + acp.initialize() + outcome: list[object] = [] + thread = threading.Thread( + target=lambda: outcome.append(acp.prompt("s1", "inspect")) + ) + thread.start() + first = acp.next_session_event(timeout=1) + second = acp.next_session_event(timeout=1) + assert first.update_kind is SessionUpdateKind.AGENT_THOUGHT_CHUNK + assert second.options[0].option_id == "allow" + acp.respond_permission(second.request_id, option_id="allow") + third = acp.next_session_event(timeout=1) + assert third.update_kind is SessionUpdateKind.PLAN + thread.join(timeout=2) + assert not thread.is_alive() + assert outcome[0].stop_reason is StopReason.END_TURN + + +@pytest.mark.parametrize("first_kind", ("permission", "update")) +def test_concurrent_typed_consumers_route_either_first_kind_without_deadlock( + first_kind: str, +) -> None: + acp = client() + update = _typed_update(1) + permission = _typed_permission(1) + waiting = threading.Event() + original_match = acp._matching_session_event_index + waiting_type = SessionUpdate if first_kind == "permission" else PermissionRequest + + def observe_wait(expected: object) -> int | None: + result = original_match(expected) + if expected is waiting_type and result is None: + waiting.set() + return result + + acp._matching_session_event_index = observe_wait # type: ignore[method-assign] + if first_kind == "permission": + _install_permission(acp, permission) + else: + acp._put_session_event(update) + + updates: list[SessionUpdate] = [] + permissions: list[PermissionRequest] = [] + update_thread = threading.Thread(target=lambda: updates.append(acp.next_update())) + permission_thread = threading.Thread( + target=lambda: permissions.append(acp.next_permission_request()) + ) + blocked_thread = update_thread if first_kind == "permission" else permission_thread + matching_thread = permission_thread if first_kind == "permission" else update_thread + blocked_thread.start() + _waiter_has_released_condition(acp, waiting) + matching_thread.start() + matching_thread.join(timeout=1) + assert not matching_thread.is_alive() + + if first_kind == "permission": + acp._put_session_event(update) + else: + _install_permission(acp, permission) + blocked_thread.join(timeout=1) + assert not blocked_thread.is_alive() + assert updates == [update] + assert permissions == [permission] + assert list(acp._session_events) == [] + + +def test_ordered_consumer_remains_exact_with_mixed_session_events() -> None: + acp = client() + expected: list[SessionUpdate | PermissionRequest] = [] + for index in range(20): + event: SessionUpdate | PermissionRequest + if index % 2: + event = _typed_permission(index) + _install_permission(acp, event) + else: + event = _typed_update(index) + acp._put_session_event(event) + expected.append(event) + + assert [acp.next_session_event() for _ in expected] == expected + assert list(acp._session_events) == [] + + +def test_close_and_failure_wake_all_session_event_waiters() -> None: + acp = client() + failures: list[BaseException] = [] + threads = [ + threading.Thread( + target=lambda method=method: _capture_failure(method, failures) + ) + for method in (acp.next_update, acp.next_permission_request) + ] + for thread in threads: + thread.start() + _wait_for_condition_waiters(acp, 2) + acp.close() + for thread in threads: + thread.join(timeout=1) + assert not thread.is_alive() + assert len(failures) == 2 + + failed = client() + failure: list[BaseException] = [] + thread = threading.Thread( + target=lambda: _capture_failure(failed.next_session_event, failure) + ) + thread.start() + _wait_for_condition_waiters(failed, 1) + failed._set_failed(AcpTransportError("boom")) + thread.join(timeout=1) + assert not thread.is_alive() + assert len(failure) == 1 + assert isinstance(failure[0], AcpTransportError) + + +def test_mixed_typed_consumer_stress_is_bounded_and_exactly_once() -> None: + acp = client(max_pending_events=8) + count = 200 + updates: list[SessionUpdate] = [] + permissions: list[PermissionRequest] = [] + update_thread = threading.Thread( + target=lambda: updates.extend(acp.next_update() for _ in range(count)) + ) + permission_thread = threading.Thread( + target=lambda: permissions.extend( + acp.next_permission_request() for _ in range(count) + ) + ) + update_thread.start() + permission_thread.start() + maximum_depth = 0 + for index in range(count): + update = _typed_update(index) + permission = _typed_permission(index) + if index % 2: + _install_permission(acp, permission) + acp._put_session_event(update) + else: + acp._put_session_event(update) + _install_permission(acp, permission) + with acp._session_event_condition: + maximum_depth = max(maximum_depth, len(acp._session_events)) + update_thread.join(timeout=3) + permission_thread.join(timeout=3) + + assert not update_thread.is_alive() + assert not permission_thread.is_alive() + assert maximum_depth <= acp.max_pending_events + assert [item.update["messageId"] for item in updates] == [ + f"message-{index}" for index in range(count) + ] + assert [item.request_id for item in permissions] == list(range(count)) + assert list(acp._session_events) == [] + + +def test_cancel_resolves_pending_permissions_as_cancelled() -> None: + with client() as acp: + acp.initialize() + result: list[object] = [] + thread = threading.Thread(target=lambda: result.append(acp.prompt("s1", "wait"))) + thread.start() + acp.next_update(timeout=1) + acp.next_permission_request(timeout=1) + acp.cancel("s1") + acp.next_update(timeout=1) + thread.join(timeout=2) + assert not thread.is_alive() + assert result[0].stop_reason is StopReason.CANCELLED + + +def test_cancel_also_resolves_permission_that_races_after_notification() -> None: + with client("cancel_race") as acp: + acp.initialize() + result: list[object] = [] + thread = threading.Thread(target=lambda: result.append(acp.prompt("s1", "wait"))) + thread.start() + acp.next_update(timeout=1) + acp.next_permission_request(timeout=1) + acp.cancel("s1") + acp.next_update(timeout=1) + thread.join(timeout=2) + assert not thread.is_alive() + assert result[0].stop_reason is StopReason.CANCELLED + with pytest.raises(AcpRequestTimeoutError): + acp.next_permission_request(timeout=0.05) + + +def test_optional_methods_require_advertised_capabilities() -> None: + with client("baseline") as acp: + acp.initialize() + with pytest.raises(AcpCapabilityError): + acp.list_sessions() + with pytest.raises(AcpCapabilityError): + acp.load_session("s1", "/tmp") + with pytest.raises(AcpCapabilityError): + acp.close_session("s1") + with pytest.raises(AcpCapabilityError): + acp.delete_session("s1") + + +def test_request_timeout_does_not_poison_transport() -> None: + with client("slow", request_timeout=0.05) as acp: + acp.initialize(timeout=1) + with pytest.raises(AcpRequestTimeoutError): + acp.list_sessions() + assert acp.state is ClientState.INITIALIZED + + +@pytest.mark.parametrize("mode", ["malformed", "oversize", "partial_eof"]) +def test_malformed_or_oversized_stdout_fails_connection(mode: str) -> None: + with client(mode, max_frame_bytes=1024) as acp: + with pytest.raises(AcpProtocolError): + acp.initialize(timeout=1) + assert acp.state is ClientState.FAILED + + +def test_absolute_session_paths_are_enforced_before_write() -> None: + with client() as acp: + acp.initialize() + with pytest.raises(ValueError, match="absolute"): + acp.new_session("relative/path") + + +def test_prewrite_callback_failure_emits_no_acp_frame( + monkeypatch: pytest.MonkeyPatch, +) -> None: + class ReceiptUnavailable(RuntimeError): + pass + + with client() as acp: + acp.initialize() + writes: list[object] = [] + + def forbidden_write(*args: object, **kwargs: object) -> int: + writes.append((args, kwargs)) + return 0 + + monkeypatch.setattr(acp, "_write_chunk", forbidden_write) + with pytest.raises(ReceiptUnavailable): + acp.request( + "session/list", + {}, + on_writing=lambda: (_ for _ in ()).throw(ReceiptUnavailable()), + ) + + assert writes == [] + assert acp._pending == {} + + +def test_write_lock_timeout_does_not_cross_prewrite_boundary() -> None: + with client() as acp: + acp.initialize() + callbacks: list[str] = [] + assert acp._write_lock.acquire(timeout=0.1) + try: + with pytest.raises(AcpRequestTimeoutError, match="waiting to write"): + acp.request( + "session/list", + {}, + timeout=0.01, + on_writing=lambda: callbacks.append("started"), + ) + finally: + acp._write_lock.release() + + assert callbacks == [] + assert acp._pending == {} + + +def test_unwritable_transport_does_not_cross_prewrite_boundary( + monkeypatch: pytest.MonkeyPatch, +) -> None: + with client() as acp: + acp.initialize() + callbacks: list[str] = [] + monkeypatch.setattr(acp, "_wait_writable", lambda _fd, _timeout: False) + with pytest.raises(AcpRequestTimeoutError, match="writing ACP frame"): + acp.request( + "session/list", + {}, + timeout=0.01, + on_writing=lambda: callbacks.append("started"), + ) + + assert callbacks == [] + assert acp._pending == {} + + +def test_prompt_content_is_validated_and_gated_by_negotiated_capabilities() -> None: + with client("baseline") as acp: + acp.initialize() + with pytest.raises(AcpCapabilityError, match="image"): + acp.prompt( + "s1", + [{"type": "image", "data": "AA==", "mimeType": "image/png"}], + ) + with pytest.raises(ValueError, match="text"): + acp.prompt("s1", [{"type": "text"}]) + with pytest.raises(ValueError, match="outside"): + acp.prompt("s1", [{"type": "text", "text": "ok", "vendor": True}]) + + +def test_prompt_content_matches_stable_v1_generated_shapes() -> None: + blocks = [ + {"type": "text", "text": "hello"}, + {"type": "image", "data": "AA==", "mimeType": "image/png"}, + {"type": "audio", "data": "AA==", "mimeType": "audio/wav"}, + {"type": "resource_link", "name": "main.py", "uri": "file:///tmp/main.py"}, + { + "type": "resource", + "resource": { + "uri": "file:///tmp/main.py", + "mimeType": "text/x-python", + "text": "print('ok')", + }, + }, + ] + with client("echo_prompt") as acp: + acp.initialize() + result = acp.prompt("s1", blocks) + assert result.stop_reason is StopReason.END_TURN + + +def test_mcp_servers_match_stable_v1_generated_shapes_and_capabilities() -> None: + with client() as acp: + acp.initialize() + created = acp.new_session( + "/tmp/project", + mcp_servers=[ + { + "name": "stdio-tools", + "command": "/usr/bin/tools", + "args": ["--stdio"], + "env": [{"name": "MODE", "value": "test"}], + }, + { + "type": "http", + "name": "http-tools", + "url": "https://example.invalid/mcp", + "headers": [{"name": "Authorization", "value": "opaque"}], + }, + { + "type": "sse", + "name": "legacy-tools", + "url": "https://example.invalid/sse", + "headers": [], + }, + ], + ) + assert created.session_id == "s-new" + + with client("baseline") as acp: + acp.initialize() + with pytest.raises(AcpCapabilityError, match="HTTP"): + acp.new_session( + "/tmp/project", + mcp_servers=[ + { + "type": "http", + "name": "http-tools", + "url": "https://example.invalid/mcp", + "headers": [], + } + ], + ) + with pytest.raises(ValueError, match="absolute"): + acp.new_session( + "/tmp/project", + mcp_servers=[ + {"name": "stdio-tools", "command": "tools", "args": [], "env": []} + ], + ) + with pytest.raises(ValueError, match="stable ACP v1"): + acp.new_session( + "/tmp/project", + mcp_servers=[ + { + "type": "stdio", + "name": "stdio-tools", + "command": "/usr/bin/tools", + "args": [], + "env": [], + } + ], + ) + + +def test_concurrent_initialize_is_exactly_once_and_returns_same_result() -> None: + with client() as acp: + results: list[object] = [] + failures: list[BaseException] = [] + + def initialize() -> None: + try: + results.append(acp.initialize()) + except BaseException as exc: # pragma: no cover - diagnostic path + failures.append(exc) + + threads = [threading.Thread(target=initialize) for _ in range(6)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=2) + assert not failures + assert len(results) == 6 + assert all(result is results[0] for result in results) + + +def test_backpressure_failure_remains_visible_after_full_queue_drains() -> None: + with client("flood", max_pending_events=1) as acp: + acp.initialize() + deadline = time.monotonic() + 1 + while acp.state is not ClientState.FAILED and time.monotonic() < deadline: + time.sleep(0.01) + assert acp.state is ClientState.FAILED + assert isinstance(acp.failure, AcpEventQueueFullError) + first = acp.next_update(timeout=1) + assert first.update_kind is SessionUpdateKind.AGENT_MESSAGE_CHUNK + with pytest.raises(AcpTransportError): + acp.next_update(timeout=1) + + +def test_blocked_or_partial_stdin_write_is_bounded_and_terminal( + monkeypatch: pytest.MonkeyPatch, +) -> None: + real_write = AcpClient._write_chunk + select_calls = 0 + + def partial_write(fd: int, data: memoryview) -> int: + return real_write(fd, data[:8]) + + def writable_once(fd: int, timeout: float) -> bool: + nonlocal select_calls + select_calls += 1 + return select_calls == 1 + + monkeypatch.setattr(AcpClient, "_write_chunk", staticmethod(partial_write)) + monkeypatch.setattr(AcpClient, "_wait_writable", staticmethod(writable_once)) + acp = client("no_read", request_timeout=0.2, close_timeout=0.05) + try: + started = time.monotonic() + with pytest.raises(AcpTransportError, match="partial write"): + acp.initialize( + client_capabilities={"vendor/padding": "small"}, + timeout=0.2, + ) + assert time.monotonic() - started < 1 + assert acp.state is ClientState.FAILED + finally: + acp.close() + + +def test_close_escalates_to_kill_for_stubborn_adapter() -> None: + acp = client("stubborn", close_timeout=0.05) + acp.initialize() + acp.close() + assert acp.state is ClientState.CLOSED + assert acp.exit is not None + assert acp.exit.returncode != 0 + + +def test_stderr_tail_is_bounded_and_keeps_suffix() -> None: + acp = client("stderr_tail", stderr_limit_bytes=64) + acp.initialize() + acp.close() + tail = acp.stderr_tail() + assert len(tail.encode()) <= 64 + assert tail.endswith("-TAIL") + + +def test_unknown_adapter_extensions_remain_observable() -> None: + method = "_vendor.example/future_notification" + with client("extensions", supported_extension_notifications=(method,)) as acp: + initialized = acp.initialize() + assert initialized.capabilities.raw["_meta"] == { + "vendor.example": {"level": 2} + } + notification = acp.next_notification(timeout=1) + assert notification.method == method + assert notification.params["opaque"] == {"revision": 9} + + +def test_unrecognized_extension_notifications_are_ignored_without_backpressure() -> None: + with client("extension_flood", max_pending_events=1) as acp: + acp.initialize() + time.sleep(0.1) + assert acp.state is ClientState.INITIALIZED + with pytest.raises(AcpRequestTimeoutError): + acp.next_notification(timeout=0.05) + + +def test_unsupported_inbound_request_gets_automatic_method_not_found() -> None: + with client("unknown_request") as acp: + acp.initialize() + confirmation = acp.next_update(timeout=1) + assert confirmation.session_id == "s-extension" + assert confirmation.update["content"]["text"] == "method-not-found" + + +def test_explicitly_supported_extension_request_remains_observable() -> None: + method = "_vendor.example/request" + with client("supported_request", supported_extension_requests=(method,)) as acp: + acp.initialize() + request = acp.next_inbound_request(timeout=1) + assert request.method == method + acp.reject_inbound_request(request.request_id) + confirmation = acp.next_update(timeout=1) + assert confirmation.update["content"]["text"] == "method-not-found" + + +def test_uncorrelated_null_error_response_does_not_poison_transport() -> None: + with client("null_response") as acp: + acp.initialize() + time.sleep(0.05) + assert acp.state is ClientState.INITIALIZED + + +def test_unexpected_clean_stdout_eof_is_transport_failure() -> None: + with client("exit_after_init") as acp: + acp.initialize() + deadline = time.monotonic() + 1 + while acp.state is not ClientState.FAILED and time.monotonic() < deadline: + time.sleep(0.01) + assert acp.state is ClientState.FAILED + assert isinstance(acp.failure, AcpTransportError) + + +def test_boolean_protocol_version_is_not_accepted_as_integer_one() -> None: + with client("bool_version") as acp: + with pytest.raises(AcpProtocolError, match="protocol version"): + acp.initialize() + assert acp.state is ClientState.FAILED + + +def test_unsupported_protocol_version_is_reaped_during_initialize() -> None: + acp = client("bool_version", close_timeout=0.1) + with pytest.raises(AcpProtocolError, match="protocol version"): + acp.initialize() + assert acp.process is not None + assert acp.process.poll() is not None + + +@pytest.mark.parametrize( + "kwargs", + [ + {"cwd": "relative"}, + {"cwd": "/tmp/bad\x00path"}, + {"env": {"BAD=NAME": "value"}}, + {"env": {"NAME": "bad\x00value"}}, + ], +) +def test_process_paths_and_environment_are_validated( + kwargs: dict[str, object], +) -> None: + with pytest.raises(ValueError): + client(**kwargs) diff --git a/tests/test_acp_coordinator.py b/tests/test_acp_coordinator.py new file mode 100644 index 0000000..d7de916 --- /dev/null +++ b/tests/test_acp_coordinator.py @@ -0,0 +1,2079 @@ +"""Production ACP coordinator and command-path contract tests.""" + +from __future__ import annotations + +import json +import threading +import time +from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager +from dataclasses import replace +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest + +from tendwire.backends.acp_coordinator import ( + AcpConsoleInputGap, + AcpCoordinatorError, + AcpSupervisor as AcpRuntimeCoordinator, + HerdrAcpConsoleEndpoint, + _SessionSlot as _RuntimeSlot, + _CONSOLE_BRIDGE_INTERVAL_SECONDS, + _derived_binding, + _console_event_output, + _bounded_console_output, + _console_output_wire_bytes, + _fit_console_output_batch, + _console_permission_selection, + _load_console_event_cursor, + _load_console_input_cursor, + _parse_console_exchange, + _parse_endpoint, + _parse_status, + _prepare_console_event_cursor, + _record_console_submission_outcome, + production_acp_supervisor_factory as production_acp_runtime_factory, +) +from tendwire.backends.acp_runtime import RuntimeState, SessionOpenMode +from tendwire.backends.herdr_protocol import HerdrErrorResponse +from tendwire.command_submission import submit_acp_command, submit_command +from tendwire.config import Config +from tendwire.core.models import ( + BackendHealth, + Snapshot, + Worker, + WorkerBinding, +) +from tendwire.daemon import DaemonHooks, TendwireDaemon +from tendwire.store.sqlite import ( + get_command_request, + init_store, + list_agent_events, + list_worker_bindings, + record_agent_event, + save_snapshot, + upsert_worker_bindings, +) + + +def _config(tmp_path: Path) -> Config: + return Config( + host_id="acp-host", + data_dir=tmp_path, + db_path=tmp_path / "tendwire.db", + herdr_bin="herdr", + ) + + +def _binding() -> WorkerBinding: + return WorkerBinding( + host_id="acp-host", + worker_id="worker-1", + worker_fingerprint="worker-fingerprint", + backend="herdr", + target_kind="pane_id", + target_value="pane-private", + turn_target_kind="pane_id", + turn_target_value="pane-private", + sendable=True, + private_fingerprint="herdr-private-binding", + ) + + +def _endpoint(*, generation: int = 42, lifecycle: str = "acp_owned_ready") -> dict[str, Any]: + return { + "type": "agent_acp_endpoint", + "endpoint": { + "transport": "stdio", + "command": "herdr", + "args": [ + "agent", + "acp-attach", + "pane-private", + "--generation", + str(generation), + "--ticket", + "one-shot-private-ticket", + ], + "protocol_version": 1, + }, + "console": { + "generation": generation, + "lease": "console-coordinator-private-lease", + }, + "worker": { + "terminal_id": "pane-private", + "workspace_id": "workspace-private", + "tab_id": "tab-private", + "pane_id": "pane-private", + "name": "agent-name", + "agent": "codex", + "generation": generation, + }, + "adapter": {"name": "codex-acp", "version": "1.2.3"}, + "session": {"mode": "new"}, + "cwd": "/tmp/project", + "lifecycle": lifecycle, + } + + +def _status(*, generation: int = 42, lifecycle: str = "acp_owned_attached") -> dict[str, Any]: + endpoint = _endpoint(generation=generation) + return { + "type": "agent_acp_status", + "worker": endpoint["worker"], + "adapter": endpoint["adapter"], + "session": endpoint["session"], + "cwd": endpoint["cwd"], + "lifecycle": lifecycle, + "console_lifecycle": "attached", + } + + +def test_endpoint_requires_explicit_acp_ownership_and_strict_attach_shape(tmp_path: Path) -> None: + config = _config(tmp_path) + parsed = _parse_endpoint(config, _binding(), _endpoint()) + assert parsed.generation == "42" + assert parsed.command[0:3] == ("herdr", "agent", "acp-attach") + + with pytest.raises(AcpCoordinatorError, match="not ready"): + _parse_endpoint(config, _binding(), _endpoint(lifecycle="ready")) + injected = _endpoint() + injected["endpoint"]["command"] = "/tmp/evil" + with pytest.raises(AcpCoordinatorError, match="configured Herdr"): + _parse_endpoint(config, _binding(), injected) + replayed = _endpoint(generation=43) + replayed["endpoint"]["args"][4] = "42" + with pytest.raises(AcpCoordinatorError, match="inconsistent"): + _parse_endpoint(config, _binding(), replayed) + wrong_terminal = _endpoint() + wrong_terminal["worker"]["terminal_id"] = "other-terminal" + terminal_binding = replace( + _binding(), + target_kind="terminal_id", + target_value="pane-private", + ) + with pytest.raises(AcpCoordinatorError, match="target changed"): + _parse_endpoint(config, terminal_binding, wrong_terminal) + + wrong_status = _status() + wrong_status["worker"]["terminal_id"] = "other-terminal" + with pytest.raises(AcpCoordinatorError, match="target changed"): + _parse_status(terminal_binding, wrong_status) + + +def test_console_exchange_requires_floor_and_next_sequence_contract() -> None: + result = { + "type": "agent_acp_console_exchange", + "inputs": [{"sequence": 3, "text": "continue"}], + "outputs": [], + "input_floor_sequence": 3, + "output_floor_sequence": 1, + "next_input_sequence": 4, + "next_output_sequence": 1, + } + assert _parse_console_exchange(result, 2) == ((3, "continue"),) + missing_next = dict(result) + missing_next.pop("next_input_sequence") + with pytest.raises(AcpCoordinatorError, match="shape"): + _parse_console_exchange(missing_next, 2) + lost = dict(result, input_floor_sequence=4) + with pytest.raises(AcpConsoleInputGap, match="gap") as raised: + _parse_console_exchange(lost, 2) + assert raised.value.recovery_after_sequence == 3 + + incomplete = dict(result, inputs=[], next_input_sequence=4) + with pytest.raises(AcpCoordinatorError, match="incomplete"): + _parse_console_exchange(incomplete, 2) + + output = dict( + result, + outputs=[ + { + "sequence": 7, + "event_id": "event-7", + "stream": "assistant", + "text": "done", + } + ], + output_floor_sequence=7, + next_output_sequence=8, + ) + assert _parse_console_exchange(output, 2) == ((3, "continue"),) + malformed_output = dict(output, outputs=[{"sequence": 7, "text": "done"}]) + with pytest.raises(AcpCoordinatorError, match="output shape"): + _parse_console_exchange(malformed_output, 2) + wrong_output_next = dict(output, next_output_sequence=9) + with pytest.raises(AcpCoordinatorError, match="next output"): + _parse_console_exchange(wrong_output_next, 2) + + +def test_live_only_console_policy_skips_lost_backlog_to_current_tail( + tmp_path: Path, +) -> None: + config = replace( + _config(tmp_path), + acp_console_input_policy="live_only", + ) + assert config.db_path is not None + init_store(config.db_path) + exchanges: list[int] = [] + + class EndpointClient: + def agent_acp_console_exchange( + self, + _target: Any, + *, + generation: int, + lease: str, + after_input_sequence: int, + output: Any, + timeout: float, + ) -> dict[str, Any]: + exchanges.append(after_input_sequence) + return { + "type": "agent_acp_console_exchange", + "inputs": ( + [ + {"sequence": 3, "text": "historical one"}, + {"sequence": 4, "text": "historical two"}, + ] + if after_input_sequence == 0 + else [] + ), + "outputs": [], + "input_floor_sequence": 3, + "output_floor_sequence": 1, + "next_input_sequence": 5, + "next_output_sequence": 1, + } + + def close(self) -> None: + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + reconcile_interval=60.0, + ) + binding = replace( + _binding(), + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + private_fingerprint="", + ) + slot = _RuntimeSlot( + continuity=_binding(), + generation="42", + runtime=SimpleNamespace(_binding=binding), + console=HerdrAcpConsoleEndpoint(42, "coordinator-lease"), + console_cursor_loaded=True, + ) + + coordinator._bridge_console_slot(slot) + + assert exchanges == [0, 4] + assert slot.console_input_sequence == 4 + assert _load_console_input_cursor( + config.db_path, + config.host_id, + "worker-1", + "session-a", + 42, + ) == 4 + + +def test_console_event_projection_covers_messages_thought_tools_and_plan() -> None: + assert _console_event_output("agent_message", {"text_delta": "done"}) == ( + "assistant", + "done", + ) + assert _console_event_output("thought", {"text_delta": "reason"}) == ( + "thought", + "reason", + ) + assert _console_event_output( + "tool_call_update", {"snapshot": {"title": "pytest", "status": "completed"}} + ) == ("tool", "pytest [completed]") + assert _console_event_output( + "tool_call_update", + { + "snapshot": { + "title": "Edit source", + "status": "completed", + "rawInput": {"secret": "not rendered"}, + "rawOutput": "not rendered", + "content": [ + {"type": "content", "content": {"type": "text", "text": "done"}}, + { + "type": "diff", + "path": "/workspace/source.py", + "oldText": "old", + "newText": "new", + }, + ], + } + }, + ) == ( + "tool", + "Edit source [completed]\ndone\ndiff /workspace/source.py\n- old\n+ new", + ) + assert _console_event_output( + "plan", {"entries": [{"content": "verify", "status": "in_progress"}]} + ) == ("plan", "[in_progress] verify") + chunks = [ + _console_event_output("agent_message", {"text_delta": "hello"}), + _console_event_output("agent_message", {"text_delta": " world"}), + ] + assert "".join(item[1] for item in chunks if item is not None) == "hello world" + + +def test_console_permission_selection_is_explicit_and_fail_closed() -> None: + options = (("1", "Allow once (allow_once)"), ("2", "Reject (reject_once)")) + assert _console_permission_selection("1", options) == "1" + assert _console_permission_selection("allow", options) == "1" + assert _console_permission_selection("deny", options) == "2" + assert _console_permission_selection("do something else", options) is None + + +def test_console_cursors_survive_restart_crash_boundaries(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + record_agent_event( + config.db_path, + config.host_id, + kind="extension", + source="tendwire-console", + worker_id="worker-1", + payload={"extension": "tendwire.acp.console_cursor", "sequence": 41}, + source_session_id="session-a", + source_event_id="cursor:41", + visibility="private", + ) + record_agent_event( + config.db_path, + config.host_id, + kind="extension", + source="tendwire-console", + worker_id="worker-1", + payload={ + "extension": "tendwire.acp.console_input_cursor", + "generation": 42, + "input_sequence": 7, + }, + source_session_id="session-a", + source_event_id="input:42:7", + visibility="private", + ) + assert _load_console_event_cursor( + config.db_path, config.host_id, "worker-1", "session-a" + ) == 41 + assert _load_console_input_cursor( + config.db_path, config.host_id, "worker-1", "session-a", 42 + ) == 7 + # The visible Herdr input queue survives an ACP adapter remint. Its cursor + # is owned by worker generation 42, so a replacement session must recover + # the already acknowledged input rather than reporting a false gap. + assert _load_console_input_cursor( + config.db_path, config.host_id, "worker-1", "session-b", 42 + ) == 7 + assert _load_console_input_cursor( + config.db_path, config.host_id, "worker-1", "session-a", 43 + ) == 0 + + +def test_missing_checkpoint_replays_from_zero_including_new_session_start_updates( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + record_agent_event( + config.db_path, + config.host_id, + kind="agent_message", + source="acp", + worker_id="worker-1", + payload={"text_delta": "stored before coordinator checkpoint"}, + source_session_id="session-resume", + source_event_id="agent-before-crash", + visibility="private", + ) + assert _prepare_console_event_cursor( + config.db_path, + config.host_id, + "worker-1", + "session-resume", + session_mode=SessionOpenMode.RESUME, + generation=42, + ) == 0 + assert _load_console_event_cursor( + config.db_path, config.host_id, "worker-1", "session-resume" + ) == 0 + + record_agent_event( + config.db_path, + config.host_id, + kind="agent_message", + source="acp", + worker_id="worker-1", + payload={"text_delta": "setup replay"}, + source_session_id="session-new", + source_event_id="agent-during-setup", + visibility="private", + ) + baseline = _prepare_console_event_cursor( + config.db_path, + config.host_id, + "worker-1", + "session-new", + session_mode=SessionOpenMode.NEW, + generation=43, + ) + assert baseline == 0 + assert _load_console_event_cursor( + config.db_path, config.host_id, "worker-1", "session-new" + ) == baseline + + +def test_console_failure_outcome_is_durable_before_input_ack(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + _record_console_submission_outcome( + config.db_path, + config.host_id, + "worker-1", + "session-a", + generation=42, + input_sequence=7, + outcome="error", + ) + # Simulate a crash before the following input-cursor write. The error is + # replayable even though Herdr input 7 was not acknowledged yet. + assert _load_console_input_cursor( + config.db_path, config.host_id, "worker-1", "session-a", 42 + ) == 0 + stored = list_agent_events( + config.db_path, + config.host_id, + worker_id="worker-1", + source="acp", + session_id="session-a", + ) + assert len(stored) == 1 + assert _console_event_output(stored[0].event.kind, stored[0].event.payload) == ( + "error", + "instruction failed", + ) + + +def test_console_output_batch_is_utf8_bounded_and_replay_deterministic() -> None: + first = _bounded_console_output("event-a", "tool", "😀" * 100_000) + second = _bounded_console_output("event-b", "assistant", "β" * 100_000) + assert first["text"].encode("utf-8").decode("utf-8") == first["text"] + assert "[console output truncated]" in first["text"] + budget = _console_output_wire_bytes([first]) + assert _fit_console_output_batch([first, second], budget=budget) == [first] + assert _fit_console_output_batch([first, second], budget=budget) == [first] + assert _console_output_wire_bytes([first]) <= budget + + +def test_console_bridge_byte_batches_and_advances_only_published_prefix( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + for index in range(10): + record_agent_event( + config.db_path, + config.host_id, + kind="agent_message", + source="acp", + worker_id="worker-1", + payload={"text_delta": "😀" * 40_000}, + source_session_id="session-a", + source_event_id=f"large-agent-chunk-{index}", + visibility="private", + ) + + class EndpointClient: + retained: list[dict[str, Any]] = [] + next_output = 1 + + def connect(self) -> None: + return None + + def close(self) -> None: + return None + + def agent_acp_console_exchange( + self, _target: str, **params: Any + ) -> dict[str, Any]: + for item in params["output"]: + EndpointClient.retained.append( + {"sequence": EndpointClient.next_output, **dict(item)} + ) + EndpointClient.next_output += 1 + floor = ( + EndpointClient.retained[0]["sequence"] + if EndpointClient.retained + else EndpointClient.next_output + ) + return { + "type": "agent_acp_console_exchange", + "inputs": [], + "outputs": list(EndpointClient.retained), + "input_floor_sequence": 1, + "output_floor_sequence": floor, + "next_input_sequence": 1, + "next_output_sequence": EndpointClient.next_output, + } + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + reconcile_interval=60.0, + ) + runtime = SimpleNamespace( + _binding=replace( + _binding(), + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + ) + ) + executor = ThreadPoolExecutor(max_workers=1) + slot = _RuntimeSlot( + continuity=_binding(), + generation="42", + runtime=runtime, + console=HerdrAcpConsoleEndpoint(42, "coordinator-lease"), + console_cursor_loaded=True, + console_executor=executor, + console_submissions={}, + console_local_turns=set(), + ) + try: + coordinator._bridge_console_slot(slot) + first_cursor = slot.console_event_sequence + assert 0 < first_cursor + assert len(EndpointClient.retained) < 10 + assert _console_output_wire_bytes(EndpointClient.retained) <= 512 * 1024 + # A full retained queue cannot make Tendwire acknowledge the remainder. + coordinator._bridge_console_slot(slot) + assert slot.console_event_sequence == first_cursor + # Once the pane drains, the exact unacknowledged suffix is published. + EndpointClient.retained.clear() + coordinator._bridge_console_slot(slot) + assert slot.console_event_sequence > first_cursor + finally: + executor.shutdown(wait=True, cancel_futures=True) + + +def test_console_bridge_polls_independently_of_slow_reconcile_interval( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), threading.Event(), reconcile_interval=60.0 + ) + ticks: list[float] = [] + + def tick() -> None: + ticks.append(time.monotonic()) + if len(ticks) == 3: + coordinator._stop.set() + + coordinator._bridge_console_slots = tick # type: ignore[method-assign] + started = time.monotonic() + coordinator._run_console_bridge() + assert len(ticks) == 3 + assert 0.25 <= _CONSOLE_BRIDGE_INTERVAL_SECONDS <= 0.5 + assert ticks[-1] - started < 3.5 * _CONSOLE_BRIDGE_INTERVAL_SECONDS + + +def test_console_bridge_dispatches_slow_workers_independently(tmp_path: Path) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), threading.Event(), reconcile_interval=60.0 + ) + slow_entered = threading.Event() + fast_entered = threading.Event() + release = threading.Event() + + slow_slot = _RuntimeSlot( + _binding(), + "42", + SimpleNamespace(), + console=HerdrAcpConsoleEndpoint(42, "slow-lease"), + ) + fast_slot = _RuntimeSlot( + replace(_binding(), worker_id="worker-2"), + "43", + SimpleNamespace(), + console=HerdrAcpConsoleEndpoint(43, "fast-lease"), + ) + coordinator._slots = {"worker-1": slow_slot, "worker-2": fast_slot} + + def bridge(slot: _RuntimeSlot) -> None: + if slot is slow_slot: + slow_entered.set() + assert release.wait(2.0) + else: + fast_entered.set() + + coordinator._bridge_console_slot = bridge # type: ignore[method-assign] + coordinator._bridge_console_slots() + try: + assert slow_entered.wait(1.0) + assert fast_entered.wait(1.0) + finally: + release.set() + for slot in (slow_slot, fast_slot): + thread = slot.console_bridge_thread + if thread is not None: + thread.join(timeout=1.0) + + +def test_console_failure_remains_degraded_after_slot_disappears(tmp_path: Path) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + slot = _RuntimeSlot( + _binding(), + "42", + SimpleNamespace(), + console=HerdrAcpConsoleEndpoint(42, "console-lease"), + ) + + def fail(_slot: _RuntimeSlot) -> None: + raise OSError("console unavailable") + + coordinator._bridge_console_slot = fail # type: ignore[method-assign] + coordinator._bridge_console_slot_supervised(slot) + assert coordinator.status()["healthy"] is False + coordinator._slots.clear() + coordinator._bridge_console_slots() + status = coordinator.status() + assert status["healthy"] is False + assert status["failure_type"] == "OSError" + + +def test_first_console_failure_immediately_fences_prompt_route_until_success( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + runtime = SimpleNamespace( + status=lambda: SimpleNamespace(healthy=True, failure_type=None), + _binding=_binding(), + ) + slot = _RuntimeSlot( + _binding(), + "42", + runtime, + console=HerdrAcpConsoleEndpoint(42, "console-lease"), + ) + coordinator._slots["worker-1"] = slot + worker = Worker( + id="worker-1", + name="worker", + status="idle", + fingerprint="worker-fingerprint", + ) + issued_before_loss = coordinator.prompt_route(worker) + assert issued_before_loss is not None + reconcile_calls = 0 + + def unexpected_reconcile(_worker_id: str, *, strict: bool) -> None: + nonlocal reconcile_calls + reconcile_calls += 1 + + coordinator._reconcile_worker = unexpected_reconcile # type: ignore[method-assign] + + failure_entered = threading.Event() + release_failure = threading.Event() + + def fail(_slot: _RuntimeSlot) -> None: + failure_entered.set() + assert release_failure.wait(2.0) + raise OSError("console unavailable") + + coordinator._bridge_console_slot = fail # type: ignore[method-assign] + failed_pass = threading.Thread( + target=coordinator._bridge_console_slot_supervised, + args=(slot,), + ) + failed_pass.start() + assert failure_entered.wait(1.0) + release_failure.set() + failed_pass.join(timeout=1.0) + assert not failed_pass.is_alive() + + assert coordinator.prompt_route(worker) is None + assert reconcile_calls == 0 + with pytest.raises(AcpCoordinatorError, match="visible console"): + _ = issued_before_loss.binding_fingerprint + assert coordinator.status()["healthy"] is False + + success_entered = threading.Event() + release_success = threading.Event() + + def succeed(_slot: _RuntimeSlot) -> None: + success_entered.set() + assert release_success.wait(2.0) + + coordinator._bridge_console_slot = succeed # type: ignore[method-assign] + successful_pass = threading.Thread( + target=coordinator._bridge_console_slot_supervised, + args=(slot,), + ) + successful_pass.start() + assert success_entered.wait(1.0) + # Starting a recovery bridge is not recovery; it must complete one + # visible-console exchange before either command ingress can route again. + assert coordinator.prompt_route(worker) is None + release_success.set() + successful_pass.join(timeout=1.0) + assert not successful_pass.is_alive() + + assert coordinator.prompt_route(worker) is not None + assert reconcile_calls == 0 + assert coordinator.status()["healthy"] is True + + +def test_superseded_console_success_cannot_clear_replacement_fence( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + runtime = SimpleNamespace( + status=lambda: SimpleNamespace(healthy=True, failure_type=None), + _binding=_binding(), + ) + old = _RuntimeSlot( + _binding(), + "42", + runtime, + console=HerdrAcpConsoleEndpoint(42, "old-lease"), + ) + replacement = _RuntimeSlot( + _binding(), + "43", + runtime, + console=HerdrAcpConsoleEndpoint(43, "replacement-lease"), + ) + coordinator._slots["worker-1"] = replacement + coordinator._console_failed_workers.add("worker-1") + coordinator._console_degraded = True + coordinator._bridge_console_slot = lambda _slot: None # type: ignore[method-assign] + + coordinator._bridge_console_slot_supervised(old) + assert "worker-1" in coordinator._console_failed_workers + assert coordinator.status()["healthy"] is False + + coordinator._bridge_console_slot_supervised(replacement) + assert "worker-1" not in coordinator._console_failed_workers + assert coordinator.status()["healthy"] is True + + +def test_console_submission_rejects_a_retired_generation_before_store_access( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), threading.Event(), reconcile_interval=60.0 + ) + coordinator._state = RuntimeState.RUNNING + stale = _RuntimeSlot(_binding(), "42", SimpleNamespace()) + replacement = _RuntimeSlot(_binding(), "43", SimpleNamespace()) + coordinator._slots["worker-1"] = replacement + with pytest.raises(AcpCoordinatorError, match="generation is stale"): + coordinator._submit_console_input(stale, 1, "must not cross sessions") + + +def test_console_local_turn_is_suppressed_before_acp_submission_emits( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + worker = Worker( + id="worker-1", + name="Agent", + status="active", + meta={"stable_key": "wsk1_" + "a" * 64, "stable_key_version": 1}, + ) + snapshot = Snapshot( + host_id=config.host_id, + updated_at="2026-01-01T00:00:00+00:00", + workers=[worker], + ) + monkeypatch.setattr( + "tendwire.backends.acp_coordinator.latest_snapshot", + lambda _path, _host: snapshot, + ) + observed: list[set[str]] = [] + + def submit_while_observing(*_args: Any, **_kwargs: Any) -> Any: + observed.append(set(slot.console_local_turns or ())) + return SimpleNamespace(status="accepted") + + monkeypatch.setattr( + "tendwire.command_submission.submit_command", submit_while_observing + ) + continuity = replace(_binding(), worker_fingerprint=worker.fingerprint) + runtime = SimpleNamespace(_session_id="session-a") + slot = _RuntimeSlot( + continuity, + "42", + runtime, + console_local_turns=set(), + ) + coordinator = AcpRuntimeCoordinator( + config, threading.Event(), reconcile_interval=60.0 + ) + + assert coordinator._submit_console_input_fenced(slot, 1, "hello") == "instruction" + assert len(observed) == 1 + assert len(observed[0]) == 1 + assert slot.console_local_turns == observed[0] + + +def test_stop_reports_failed_while_console_submission_thread_is_still_running( + tmp_path: Path, +) -> None: + coordinator = AcpRuntimeCoordinator( + _config(tmp_path), threading.Event(), reconcile_interval=60.0 + ) + coordinator._state = RuntimeState.RUNNING + entered = threading.Event() + release = threading.Event() + executor = ThreadPoolExecutor(max_workers=1) + + def block() -> None: + entered.set() + assert release.wait(2.0) + + future = executor.submit(block) + assert entered.wait(1.0) + runtime = SimpleNamespace(stop=lambda *, timeout: None) + slot = _RuntimeSlot( + _binding(), + "42", + runtime, + console_executor=executor, + console_submissions={1: future}, + ) + coordinator._slots["worker-1"] = slot + try: + coordinator.stop(timeout=0.05) + status = coordinator.status() + assert status["state"] == "failed" + assert status["failure_type"] == "AcpRuntimeStopTimeout" + finally: + release.set() + future.result(timeout=1.0) + + +def test_console_input_submission_worker_does_not_block_event_exchange( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + entered = threading.Event() + release = threading.Event() + exchanges: list[int] = [] + + class EndpointClient: + def connect(self) -> None: + return None + + def close(self) -> None: + return None + + def agent_acp_console_exchange(self, _target: str, **params: Any) -> Any: + exchanges.append(int(params["after_input_sequence"])) + return { + "type": "agent_acp_console_exchange", + "inputs": [{"sequence": 1, "text": "stream while I run"}], + "outputs": [], + "input_floor_sequence": 1, + "output_floor_sequence": 1, + "next_input_sequence": 2, + "next_output_sequence": 1, + } + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + reconcile_interval=60.0, + ) + + def blocking_submit(_slot: Any, _sequence: int, _text: str) -> str: + entered.set() + assert release.wait(2.0) + return "instruction" + + coordinator._submit_console_input = blocking_submit # type: ignore[method-assign] + runtime = SimpleNamespace( + _binding=replace( + _binding(), + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + ) + ) + executor = ThreadPoolExecutor(max_workers=1) + slot = _RuntimeSlot( + continuity=_binding(), + generation="42", + runtime=runtime, + console=HerdrAcpConsoleEndpoint(42, "coordinator-lease"), + console_cursor_loaded=True, + console_executor=executor, + console_submissions={}, + console_local_turns=set(), + ) + try: + coordinator._bridge_console_slot(slot) + assert entered.wait(1.0) + started = time.monotonic() + coordinator._bridge_console_slot(slot) + assert time.monotonic() - started < 0.5 + assert exchanges == [0, 0] + assert slot.console_input_sequence == 0 + finally: + release.set() + executor.shutdown(wait=True, cancel_futures=True) + + +def test_canonical_herdr_acp_contract_fixture_executes_configured_binary( + tmp_path: Path, +) -> None: + fixture = json.loads( + (Path(__file__).parent / "fixtures" / "herdr_acp_contract_v1.json").read_text( + encoding="utf-8" + ) + ) + assert fixture["request"] == { + "id": "tw:acp:endpoint", + "method": "agent.acp_endpoint", + "params": {"target": "term_abc"}, + } + config = replace(_config(tmp_path), herdr_bin="/opt/herdr/bin/herdr") + continuity = replace( + _binding(), + target_kind="terminal_id", + target_value="term_abc", + turn_target_kind="terminal_id", + turn_target_value="term_abc", + private_fingerprint="fixture-binding", + ) + parsed = _parse_endpoint(config, continuity, fixture["result"]) + assert parsed.command[0] == "/opt/herdr/bin/herdr" + assert parsed.command[1:] == tuple(fixture["result"]["endpoint"]["args"]) + assert parsed.generation == "42" + assert parsed.console == HerdrAcpConsoleEndpoint( + 42, "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQ" + ) + + +class _Route: + binding_fingerprint = "acp-private-binding" + + def __init__(self, failure: BaseException | None = None) -> None: + self.failure = failure + self.calls: list[tuple[str, str, float]] = [] + + @property + def supports_steering(self) -> bool: + return False + + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Any = None, + ) -> None: + self.calls.append((text, producer_turn_id, timeout)) + if callable(on_send_start): + on_send_start() + if self.failure is not None: + raise self.failure + + +class _PreparationFailureRoute(_Route): + @contextmanager + def prepare(self): + raise AcpCoordinatorError("transient generation status failure") + yield self + + +class _BeforeTransportFailureRoute(_Route): + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Any = None, + ) -> None: + del on_send_start + self.calls.append((text, producer_turn_id, timeout)) + raise AcpCoordinatorError("visible console route changed before write") + + +class _SteeringRoute(_Route): + def __init__(self, outcome: str = "injected") -> None: + super().__init__() + self.outcome = outcome + self.steering_calls: list[tuple[str, str, float]] = [] + + @property + def supports_steering(self) -> bool: + return True + + def steer( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Any = None, + ) -> object: + self.steering_calls.append((text, producer_turn_id, timeout)) + if callable(on_send_start): + on_send_start() + return SimpleNamespace(outcome=self.outcome) + + +def _seed(config: Config) -> Worker: + assert config.db_path is not None + init_store(config.db_path) + worker = Worker( + id="worker-1", + name="worker", + status="idle", + fingerprint="worker-fingerprint", + meta={ + "stable_key": "wsk1_" + ("a" * 64), + "stable_key_version": 1, + }, + ) + save_snapshot( + config.db_path, + Snapshot( + host_id=config.host_id, + updated_at="2026-07-31T00:00:00+00:00", + workers=[worker], + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty", + ) + ], + ), + ) + upsert_worker_bindings(config.db_path, [_binding()]) + return worker + + +def _request(request_id: str = "request-1") -> dict[str, Any]: + return { + "schema_version": 1, + "action": "send_instruction", + "request_id": request_id, + "dry_run": False, + "target": {"worker_id": "worker-1"}, + "instruction": {"text": "do the work"}, + } + + +def test_acp_command_uses_durable_receipt_and_duplicate_does_not_resend(tmp_path: Path) -> None: + config = _config(tmp_path) + worker = _seed(config) + route = _Route() + first = submit_command( + config, + _request(), + acp_prompt_router=lambda routed: route if routed == worker else None, + ) + second = submit_command( + config, + _request(), + acp_prompt_router=lambda _worker: route, + ) + assert first.status == "accepted" + assert second.status == "accepted" + assert len(route.calls) == 1 + receipt = get_command_request(config.db_path, config.host_id, "request-1") + assert receipt is not None and receipt["state"] == "accepted" + assert "acp-private-binding" not in json.dumps(first.to_dict()) + + +@pytest.mark.parametrize("observed_status", ["idle", "active"]) +def test_live_acp_route_uses_advertised_steering_despite_observer_lag( + tmp_path: Path, + observed_status: str, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + assert config.db_path is not None + observed = replace(worker, status=observed_status) + save_snapshot( + config.db_path, + Snapshot( + host_id=config.host_id, + updated_at="2026-07-31T00:00:01+00:00", + workers=[observed], + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty", + ) + ], + ), + ) + upsert_worker_bindings(config.db_path, [_binding()]) + route = _SteeringRoute() + + envelope = submit_command( + config, + _request("request-active-steer"), + acp_prompt_router=lambda routed: route if routed.id == observed.id else None, + ) + + assert envelope.status == "accepted" + assert envelope.disposition == "terminal_accepted" + assert route.calls == [] + assert len(route.steering_calls) == 1 + receipt = get_command_request( + config.db_path, + config.host_id, + "request-active-steer", + ) + assert receipt is not None and receipt["state"] == "accepted" + + +def test_definite_acp_steering_failure_is_rejected_not_uncertain( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + assert config.db_path is not None + active = replace(worker, status="active") + save_snapshot( + config.db_path, + Snapshot( + host_id=config.host_id, + updated_at="2026-07-31T00:00:01+00:00", + workers=[active], + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty", + ) + ], + ), + ) + upsert_worker_bindings(config.db_path, [_binding()]) + route = _SteeringRoute("failed") + + envelope = submit_command( + config, + _request("request-active-steer-failed"), + acp_prompt_router=lambda _routed: route, + ) + + assert envelope.status == "rejected" + assert envelope.disposition == "terminal_rejected" + receipt = get_command_request( + config.db_path, + config.host_id, + "request-active-steer-failed", + ) + assert receipt is not None and receipt["state"] == "rejected" + + +def test_acp_failure_after_send_started_is_uncertain_and_never_falls_back(tmp_path: Path) -> None: + config = _config(tmp_path) + _seed(config) + route = _Route(RuntimeError("private ticket and argv")) + legacy_calls: list[str] = [] + envelope = submit_command( + config, + _request("request-uncertain"), + acp_prompt_router=lambda _worker: route, + ) + assert envelope.status == "request_state_uncertain" + assert envelope.disposition == "terminal_uncertain" + assert len(route.calls) == 1 + assert "private ticket" not in json.dumps(envelope.to_dict()) + receipt = get_command_request( + config.db_path, + config.host_id, + "request-uncertain", + ) + assert receipt is not None and receipt["state"] == "uncertain" + + +def test_acp_generation_preflight_failure_is_retryable_before_receipt( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + route = _PreparationFailureRoute() + + envelope = submit_command( + config, + _request("request-preflight-retry"), + acp_prompt_router=lambda routed: route if routed == worker else None, + ) + + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert route.calls == [] + assert get_command_request( + config.db_path, + config.host_id, + "request-preflight-retry", + ) is None + + +def test_production_route_checks_generation_before_reserving_receipt( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + reconcile_interval=60.0, + ) + coordinator._state = RuntimeState.RUNNING + runtime = SimpleNamespace( + status=lambda: SimpleNamespace(healthy=True, failure_type=None), + _binding=_binding(), + ) + coordinator._slots[worker.id] = _RuntimeSlot( + _binding(), + "42", + runtime, + ) + coordinator._require_attached_generation = ( # type: ignore[method-assign] + lambda _slot: (_ for _ in ()).throw( + AcpCoordinatorError("transient Herdr status timeout") + ) + ) + + envelope = submit_command( + config, + _request("request-production-preflight"), + acp_prompt_router=coordinator.prompt_route, + ) + + assert envelope.status == "backend_unavailable" + assert envelope.disposition == "no_receipt" + assert get_command_request( + config.db_path, + config.host_id, + "request-production-preflight", + ) is None + + +def test_acp_failure_before_transport_boundary_is_immediately_retryable( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + worker = _seed(config) + failed_route = _BeforeTransportFailureRoute() + + first = submit_command( + config, + _request("request-prewrite-retry"), + acp_prompt_router=lambda routed: failed_route if routed == worker else None, + ) + assert first.status == "backend_unavailable" + assert first.disposition == "no_receipt" + receipt = get_command_request( + config.db_path, + config.host_id, + "request-prewrite-retry", + ) + assert receipt is not None + assert receipt["state"] == "reserved" + assert receipt["send_started_at"] is None + + good_route = _Route() + second = submit_command( + config, + _request("request-prewrite-retry"), + acp_prompt_router=lambda routed: good_route if routed == worker else None, + ) + assert second.status == "accepted" + assert second.disposition == "terminal_accepted" + assert len(good_route.calls) == 1 + receipt = get_command_request( + config.db_path, + config.host_id, + "request-prewrite-retry", + ) + assert receipt is not None and receipt["state"] == "accepted" + + +def test_route_authority_failure_is_safe_before_receipt_reservation(tmp_path: Path) -> None: + config = _config(tmp_path) + _seed(config) + + class VanishedRoute: + @property + def binding_fingerprint(self) -> str: + raise AcpCoordinatorError("slot changed") + + def prompt(self, *_args: Any, **_kwargs: Any) -> None: + raise AssertionError("a route without authority must not send") + + required = submit_acp_command( + config, + _request("route-race-required"), + prompt_router=lambda _worker: VanishedRoute(), + ) + assert required is not None + assert required.status == "backend_unavailable" + assert get_command_request( + config.db_path, + config.host_id, + "route-race-required", + ) is None + + +def test_concurrent_duplicate_acp_command_has_one_external_send(tmp_path: Path) -> None: + config = _config(tmp_path) + _seed(config) + entered = threading.Event() + release = threading.Event() + + class BlockingRoute(_Route): + def prompt( + self, + text: str, + *, + producer_turn_id: str, + timeout: float, + on_send_start: Any = None, + ) -> None: + self.calls.append((text, producer_turn_id, timeout)) + if callable(on_send_start): + on_send_start() + entered.set() + assert release.wait(1.0) + + route = BlockingRoute() + with ThreadPoolExecutor(max_workers=2) as pool: + first = pool.submit( + submit_command, + config, + _request("request-concurrent"), + acp_prompt_router=lambda _worker: route, + ) + assert entered.wait(1.0) + second = pool.submit( + submit_command, + config, + _request("request-concurrent"), + acp_prompt_router=lambda _worker: route, + ) + second_result = second.result(timeout=1.0) + release.set() + first_result = first.result(timeout=1.0) + assert first_result.status == "accepted" + assert second_result.status == "pending" + assert len(route.calls) == 1 + + +def test_required_has_no_legacy_fallback_when_route_is_absent(tmp_path: Path) -> None: + config = _config(tmp_path) + _seed(config) + envelope = submit_command( + config, + _request("request-required"), + acp_prompt_router=lambda _worker: None, + ) + assert envelope.status == "backend_unavailable" + assert get_command_request( + config.db_path, + config.host_id, + "request-required", + ) is None + + +def test_new_and_resumed_endpoint_session_invariants(tmp_path: Path) -> None: + config = _config(tmp_path) + invalid_new = _endpoint() + invalid_new["session"] = {"mode": "new", "id": "already-bound"} + with pytest.raises(AcpCoordinatorError, match="already has"): + _parse_endpoint(config, _binding(), invalid_new) + resumed = _endpoint() + resumed["session"] = {"mode": "resume", "id": "session-private"} + assert _parse_endpoint(config, _binding(), resumed).session_id == "session-private" + missing = _endpoint() + missing["session"] = {"mode": "resume"} + with pytest.raises(AcpCoordinatorError, match="session id"): + _parse_endpoint(config, _binding(), missing) + + +def test_derived_acp_binding_outlives_observation_lease() -> None: + continuity = replace( + _binding(), + observed_at="2026-07-31T00:00:00+00:00", + expires_at="2026-07-31T00:00:05+00:00", + ) + derived = _derived_binding(continuity, "session-private") + assert derived.expires_at.startswith("9999-") + assert derived.observed_at > continuity.observed_at + assert derived.private_fingerprint != continuity.private_fingerprint + + +def test_reconnect_remints_endpoint_instead_of_replaying_attach_ticket(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + minted: list[str] = [] + generation = [42] + runtimes: list[Any] = [] + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + value = _endpoint(generation=generation[0]) + ticket = f"one-shot-private-ticket-{len(minted) + 1}" + value["endpoint"]["args"][6] = ticket + minted.append(ticket) + return value + + def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: + return _status(generation=generation[0]) + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + # The endpoint generation fences Herdr authority, but must not be + # reused as the ACP transport stream generation. AcpRuntime owns a + # fresh nonce for every adapter process. + assert "stream_generation" not in kwargs + self._binding = kwargs["binding"] + self.stopped = False + self.prompt_calls = 0 + runtimes.append(self) + + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + self.stopped = True + + def status(self) -> Any: + return SimpleNamespace( + healthy=not self.stopped, + failure_type=None, + updates_ingested=0, + permissions_ingested=0, + permissions_selected=0, + permissions_cancelled=0, + invalid_permission_selections=0, + prompts_started=0, + prompts_completed=0, + prompts_failed=0, + cancellation_requests=0, + ) + + def submit_prompt(self, *_args: Any, **_kwargs: Any) -> None: + self.prompt_calls += 1 + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, + reconcile_interval=60.0, + ).start() + try: + assert minted == ["one-shot-private-ticket-1"] + assert coordinator._published_acp_claims == { + "worker-1": "worker-fingerprint" + } + coordinator._reconcile_worker("worker-1", strict=True) + worker = Worker( + id="worker-1", + name="worker", + status="idle", + fingerprint="worker-fingerprint", + ) + route = coordinator.prompt_route(worker) + assert route is not None + route.prompt("hello", producer_turn_id="producer", timeout=1.0) + assert minted == ["one-shot-private-ticket-1"] + assert runtimes[0].prompt_calls == 1 + generation[0] = 43 + with pytest.raises(AcpCoordinatorError, match="generation lease"): + route.prompt("stale", producer_turn_id="producer-2", timeout=1.0) + assert runtimes[0].prompt_calls == 1 + coordinator._reconcile_worker("worker-1", strict=True) + assert minted == [ + "one-shot-private-ticket-1", + "one-shot-private-ticket-2", + ] + replaced_route = coordinator.prompt_route(worker) + assert replaced_route is not None + generation[0] = 44 + coordinator._reconcile_worker("worker-1", strict=True) + with pytest.raises(AcpCoordinatorError, match="stale"): + _ = replaced_route.binding_fingerprint + with pytest.raises(AcpCoordinatorError, match="stale"): + replaced_route.prompt( + "must-not-cross-generations", + producer_turn_id="producer-3", + timeout=1.0, + ) + assert runtimes[-1].prompt_calls == 0 + finally: + coordinator.stop() + + +def test_concurrent_reconcile_mints_only_one_endpoint(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + entered = threading.Event() + release = threading.Event() + minted = 0 + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + nonlocal minted + minted += 1 + entered.set() + assert release.wait(2.0) + return _endpoint() + + def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: + return _status() + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self.stopped = False + + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + self.stopped = True + + def status(self) -> Any: + return SimpleNamespace(healthy=not self.stopped, failure_type=None) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, + reconcile_interval=60.0, + ).start() + upsert_worker_bindings(config.db_path, [_binding()]) + try: + with ThreadPoolExecutor(max_workers=2) as pool: + first = pool.submit(coordinator._reconcile_worker, "worker-1", strict=True) + assert entered.wait(1.0) + second = pool.submit(coordinator._reconcile_worker, "worker-1", strict=True) + release.set() + first.result(timeout=2.0) + second.result(timeout=2.0) + assert minted == 1 + finally: + release.set() + coordinator.stop() + + +def test_stop_cannot_leave_inflight_reconcile_runtime_published(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + entered = threading.Event() + release = threading.Event() + stop_done = threading.Event() + runtimes: list[Any] = [] + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + entered.set() + assert release.wait(2.0) + return _endpoint() + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self.stopped = False + runtimes.append(self) + + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + self.stopped = True + + def status(self) -> Any: + return SimpleNamespace(healthy=not self.stopped, failure_type=None) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, + reconcile_interval=60.0, + ).start() + upsert_worker_bindings(config.db_path, [_binding()]) + with ThreadPoolExecutor(max_workers=2) as pool: + reconcile = pool.submit(coordinator._reconcile_worker, "worker-1", strict=True) + assert entered.wait(1.0) + + def stop() -> None: + coordinator.stop() + stop_done.set() + + stopping = pool.submit(stop) + assert not stop_done.wait(0.1) + release.set() + with pytest.raises(AcpCoordinatorError, match="stopping"): + reconcile.result(timeout=2.0) + stopping.result(timeout=2.0) + assert coordinator._slots == {} + assert runtimes and all(runtime.stopped for runtime in runtimes) + + +def test_stop_closes_permission_waiter_before_generation_fence_deadline( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + answer_entered = threading.Event() + broker_closed = threading.Event() + runtime_stopped = threading.Event() + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + return _endpoint() + + def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: + return _status() + + def close(self) -> None: + return None + + class Broker: + def owns(self, _decision: Any) -> bool: + return True + + def answer(self, _decision: Any, *, timeout: float) -> None: + del timeout + answer_entered.set() + assert broker_closed.wait(2.0) + raise AcpCoordinatorError("permission bridge closed") + + def close(self) -> None: + broker_closed.set() + + broker = Broker() + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self._binder = kwargs["session_binding_callback"] + + def start(self) -> None: + if self._binder is not None: + self._binding = self._binder("session-private", self._binding) + + def stop(self, *, timeout: float) -> None: + del timeout + runtime_stopped.set() + + def status(self) -> Any: + return SimpleNamespace( + healthy=not runtime_stopped.is_set(), + failure_type=None, + ) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, + durable_permission_bridge=True, + reconcile_interval=60.0, + ).start() + slot = coordinator._slots["worker-1"] + slot.permission_broker = broker # type: ignore[assignment] + decision = SimpleNamespace(worker_id="worker-1") + + with ThreadPoolExecutor(max_workers=2) as pool: + answer = pool.submit( + coordinator.answer_permission_decision, + decision, + timeout=30.0, + ) + assert answer_entered.wait(1.0) + started = time.monotonic() + coordinator.stop(timeout=0.5) + assert time.monotonic() - started < 0.5 + with pytest.raises(AcpCoordinatorError, match="closed"): + answer.result(timeout=1.0) + + assert broker_closed.is_set() + assert runtime_stopped.is_set() + + +def test_prompt_frame_acknowledgement_fences_generation_retirement(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + generation = [42] + endpoint_calls = 0 + prompt_entered = threading.Event() + prompt_release = threading.Event() + runtimes: list[Any] = [] + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + nonlocal endpoint_calls + endpoint_calls += 1 + return _endpoint(generation=generation[0]) + + def agent_acp_status(self, _target: Any, *, timeout: float) -> Any: + return _status(generation=generation[0]) + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + self._binding = kwargs["binding"] + self._binder = kwargs["session_binding_callback"] + self.stopped = False + runtimes.append(self) + + def start(self) -> None: + if self._binder is not None: + self._binding = self._binder( + f"session-private-{len(runtimes)}", self._binding + ) + + def submit_prompt(self, *_args: Any, **_kwargs: Any) -> None: + prompt_entered.set() + assert prompt_release.wait(2.0) + + def stop(self, *, timeout: float) -> None: + self.stopped = True + + def status(self) -> Any: + return SimpleNamespace(healthy=not self.stopped, failure_type=None) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, + reconcile_interval=60.0, + ).start() + worker = Worker( + id="worker-1", + name="worker", + status="working", + fingerprint="worker-fingerprint", + ) + route = coordinator.prompt_route(worker) + assert route is not None + try: + with ThreadPoolExecutor(max_workers=2) as pool: + prompt = pool.submit( + route.prompt, + "one frame", + producer_turn_id="producer-1", + timeout=1.0, + ) + assert prompt_entered.wait(1.0) + generation[0] = 43 + reconcile = pool.submit( + coordinator._reconcile_worker, "worker-1", strict=True + ) + # A new endpoint cannot be minted and the old transport cannot be + # stopped while its request frame is still being acknowledged. + assert not reconcile.done() + assert endpoint_calls == 1 + assert not runtimes[0].stopped + prompt_release.set() + prompt.result(timeout=2.0) + reconcile.result(timeout=2.0) + assert endpoint_calls == 2 + assert runtimes[0].stopped + assert len(runtimes) == 2 + finally: + prompt_release.set() + coordinator.stop() + + +def test_runtime_factory_failure_rolls_back_resumed_binding(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + endpoint = _endpoint() + endpoint["session"] = {"mode": "resume", "id": "session-private"} + return endpoint + + def close(self) -> None: + return None + + class Client: + def __init__(self) -> None: + self.closed = False + + def close(self) -> None: + self.closed = True + + client = Client() + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + connection_factory=lambda *_args, **_kwargs: client, + session_factory=lambda *_args, **_kwargs: (_ for _ in ()).throw( + RuntimeError("constructor failed with --ticket private") + ), + reconcile_interval=60.0, + ).start() + upsert_worker_bindings(config.db_path, [_binding()]) + try: + with pytest.raises(RuntimeError, match="constructor failed"): + coordinator._reconcile_worker("worker-1", strict=True) + assert client.closed + assert list_worker_bindings(config.db_path, config.host_id, backend="acp") == [] + finally: + coordinator.stop() + + +def test_coordinator_start_revokes_orphaned_process_binding(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + orphaned = _derived_binding(_binding(), "orphaned-session") + upsert_worker_bindings(config.db_path, [orphaned]) + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: object(), + connection_factory=lambda *_args, **_kwargs: object(), + reconcile_interval=60.0, + ).start() + try: + assert list_worker_bindings( + config.db_path, config.host_id, backend="acp" + ) == [] + finally: + coordinator.stop() + + +def test_production_coordinator_installs_durable_permission_bridge( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + class EmptyLifecycleClient: + def workspace_list(self, *, timeout: float) -> list[Any]: + return [] + + def pane_list(self, *, timeout: float) -> list[Any]: + return [] + + def agent_list(self, *, timeout: float) -> list[Any]: + return [] + + def close(self) -> None: + return None + + monkeypatch.setattr( + "tendwire.backends.acp_coordinator._default_endpoint_client_factory", + lambda _config: EmptyLifecycleClient(), + ) + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + coordinator = production_acp_runtime_factory(config, threading.Event()) + + coordinator.start() + try: + assert coordinator.status()["state"] == "running" + assert coordinator._durable_permission_bridge is True + finally: + coordinator.stop() + + +def test_coordinator_rejects_disabled_reconciliation(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="reconcile_interval must be finite and positive"): + AcpRuntimeCoordinator( + _config(tmp_path), + threading.Event(), + reconcile_interval=0, + ) + + +def test_coordinator_forwards_explicit_permission_bridge(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + seen: list[Any] = [] + + class EndpointClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + return _endpoint() + + def close(self) -> None: + return None + + class Runtime: + def __init__(self, _client: Any, **kwargs: Any) -> None: + seen.append(kwargs["permission_callback"]) + self._binding = kwargs["binding"] + + def start(self) -> None: + return None + + def stop(self, *, timeout: float) -> None: + return None + + def status(self) -> Any: + return SimpleNamespace(healthy=True, failure_type=None) + + def permission_bridge(_request: Any) -> str | None: + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: EndpointClient(), + connection_factory=lambda *_args, **_kwargs: object(), + session_factory=Runtime, + permission_callback=permission_bridge, + require_permission_bridge=True, + reconcile_interval=60.0, + ).start() + try: + assert seen == [permission_bridge] + finally: + coordinator.stop() + + +def test_required_zero_workers_is_idle_healthy_then_new_unowned_worker_degrades( + tmp_path: Path, +) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + + class RejectingClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + raise AcpCoordinatorError("ACP ownership required") + + def close(self) -> None: + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: RejectingClient(), + reconcile_interval=60.0, + ).start() + try: + assert coordinator.status()["healthy"] is True + upsert_worker_bindings(config.db_path, [_binding()]) + coordinator._reconcile(strict=False) + health = coordinator.status() + assert health["healthy"] is False + assert health["failure_type"] == "AcpCoordinatorError" + finally: + coordinator.stop() + + +def test_required_does_not_cache_non_acp_endpoint_failure(tmp_path: Path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + upsert_worker_bindings(config.db_path, [_binding()]) + endpoint_calls = 0 + + class NonAcpClient: + def agent_acp_endpoint(self, _target: Any, *, timeout: float) -> Any: + nonlocal endpoint_calls + endpoint_calls += 1 + raise HerdrErrorResponse( + {"code": "acp_ownership_required", "message": "PTY-owned"}, + "request-private", + ) + + def close(self) -> None: + return None + + coordinator = AcpRuntimeCoordinator( + config, + threading.Event(), + endpoint_client_factory=lambda _config: NonAcpClient(), + reconcile_interval=60.0, + ) + with pytest.raises(AcpCoordinatorError, match="failed to attach"): + coordinator.start() + assert endpoint_calls == 1 diff --git a/tests/test_acp_ingestion.py b/tests/test_acp_ingestion.py new file mode 100644 index 0000000..3d638a8 --- /dev/null +++ b/tests/test_acp_ingestion.py @@ -0,0 +1,821 @@ +"""Integration-boundary tests for durable ACP event ingestion.""" + +from __future__ import annotations + +from dataclasses import replace +from pathlib import Path + +import pytest + +from tendwire.backends.acp_ingestion import AcpSessionIngestor +from tendwire.config import Config +from tendwire.core.agent_events import AgentEvent, AppendBoundAgentEventResult +from tendwire.core.models import Snapshot, Worker, WorkerBinding +from tendwire.backends.acp_protocol import StopReason +from tendwire.store.sqlite import ( + AppendProjectedAgentEventResult, + TurnRefreshApplyResult, + list_agent_events, + save_snapshot, + upsert_worker_bindings, +) +from .store_helpers import read_public_test_agent_events + + +def _binding() -> WorkerBinding: + return WorkerBinding( + host_id="host-a", + worker_id="worker-a", + worker_fingerprint="worker-fingerprint", + backend="herdr", + target_kind="pane_id", + target_value="private-pane", + turn_target_kind="acp_session_id", + turn_target_value="session-a", + private_fingerprint="binding-fingerprint", + ) + + +def _update(kind: str, **fields: object) -> dict[str, object]: + return { + "method": "session/update", + "params": { + "sessionId": "session-a", + "update": {"sessionUpdate": kind, **fields}, + }, + } + + +def _appended(sequence: int, event: AgentEvent) -> AppendBoundAgentEventResult: + return AppendBoundAgentEventResult("inserted", event.event_id, sequence) + + +def _persist( + append, + apply=None, +): + def persist( + path: Path | str, + host_id: str, + event: AgentEvent, + *, + expected_binding: WorkerBinding, + content=None, + **_kwargs, + ) -> AppendProjectedAgentEventResult: + appended = append( + path, + host_id, + event, + expected_binding=expected_binding, + ) + turn = None + if content is not None and appended.status != "binding_changed": + turn = ( + apply(path, host_id, event.worker_id, content) + if apply is not None + else TurnRefreshApplyResult(0, False) + ) + return AppendProjectedAgentEventResult(appended, turn) + + return persist + + +def _config(db_path: Path, **kwargs: object) -> Config: + return Config( + host_id="host-a", + db_path=db_path, + **kwargs, + ) + + +def test_messages_are_journaled_privately_and_projected_without_thoughts( + tmp_path: Path, +) -> None: + events: list[AgentEvent] = [] + turns: list[dict[str, object]] = [] + + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + *, + expected_binding: WorkerBinding, + ) -> AppendBoundAgentEventResult: + assert expected_binding.worker_id == "worker-a" + assert expected_binding.private_fingerprint == "binding-fingerprint" + events.append(event) + return _appended(len(events), event) + + def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): + turns.append(dict(content)) + return TurnRefreshApplyResult(1, False) + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + persist_event=_persist(append, apply), + ) + turn_id = ingestor.start_turn(producer_turn_id="private-turn") + ingestor.ingest_update( + _update( + "user_message_chunk", + messageId="user-1", + content={"type": "text", "text": "question"}, + ) + ) + ingestor.ingest_update( + _update( + "agent_thought_chunk", + messageId="reasoning-1", + content={"type": "text", "text": "private reasoning"}, + _meta={"tendwire.dev/thought_kind": "summary"}, + ) + ) + ingestor.ingest_update( + _update( + "agent_message_chunk", + messageId="assistant-1", + content={"type": "text", "text": "answer"}, + ) + ) + ingestor.mark_prompt_complete() + + assert turn_id.startswith("acpt_") + assert [event.kind for event in events] == [ + "user_message", + "agent_message", + "extension", + ] + assert all(event.visibility == "private" for event in events) + assert turns[-1]["assistant_final_text"] == "answer" + assert turns[-1]["user_text"] == "question" + assert "private reasoning" not in repr(turns) + assert turns[-1]["source_turn_id"] == turn_id + + +def test_disabled_thought_policy_discards_before_persistence(tmp_path: Path) -> None: + def unexpected_append(*_args, **_kwargs): + raise AssertionError("disabled thoughts must not be persisted") + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db", acp_thought_policy="disabled"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + persist_event=_persist(unexpected_append), + ) + result = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "discard me"}, + ) + ) + + assert result.ignored_reason == "thought_policy_disabled" + + +def test_synthetic_event_identity_is_scoped_to_stream_generation(tmp_path: Path) -> None: + seen: list[str] = [] + + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: + assert event.source_event_id is not None + seen.append(event.source_event_id) + return _appended(1, event) + + for generation in ("generation-a", "generation-b"): + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation=generation, + binding=_binding(), + persist_event=_persist(append), + ) + ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "same chunk"}, + ) + ) + + assert seen == ["stream:generation-a:1", "stream:generation-b:1"] + + +def test_constructor_rejects_binding_for_another_acp_session(tmp_path: Path) -> None: + binding = _binding() + mismatched = WorkerBinding( + **{**binding.__dict__, "turn_target_value": "another-session"} + ) + + with pytest.raises(ValueError, match="does not match"): + AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=mismatched, + ) + + +def test_notification_session_mismatch_is_rejected_before_state_or_persistence( + tmp_path: Path, +) -> None: + def unexpected(*_args, **_kwargs): + raise AssertionError("mismatched session must not cross the authority boundary") + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + persist_event=_persist(unexpected, unexpected), + ) + notification = _update( + "agent_message_chunk", + content={"type": "text", "text": "wrong worker"}, + ) + params = notification["params"] + assert isinstance(params, dict) + params["sessionId"] = "session-b" + + result = ingestor.ingest_update(notification) + + assert result.ignored_reason == "session_mismatch" + assert ingestor.source_turn_id is None + assert ingestor.projector.session_snapshot("session-b") is None + + +def test_required_mode_fails_closed_when_durable_binding_is_stale( + tmp_path: Path, +) -> None: + db_path = tmp_path / "events.db" + binding = _binding() + save_snapshot( + db_path, + Snapshot( + host_id="host-a", + updated_at="2026-01-01T00:00:00+00:00", + workers=[ + Worker( + id=binding.worker_id, + name="Worker A", + fingerprint=binding.worker_fingerprint, + ) + ], + ), + ) + upsert_worker_bindings(db_path, [binding]) + replacement = replace( + binding, + worker_id="replacement-worker", + worker_fingerprint="replacement-fingerprint", + ) + upsert_worker_bindings(db_path, [replacement]) + + ingestor = AcpSessionIngestor( + _config(db_path), + session_id="session-a", + stream_generation="generation-a", + binding=binding, + ) + + result = ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "must not publish"}, + ) + ) + + assert result.ignored_reason == "stale_binding" + assert result.event is not None + assert result.event.status == "binding_changed" + assert result.event.sequence is None + assert result.turn is None + assert list_agent_events(db_path, "host-a") == () + assert ingestor.source_turn_id is None + assert ingestor.projector.session_snapshot("session-a") is None + + +def test_default_authority_check_accepts_the_current_durable_binding( + tmp_path: Path, +) -> None: + db_path = tmp_path / "events.db" + binding = _binding() + upsert_worker_bindings(db_path, [binding]) + ingestor = AcpSessionIngestor( + _config(db_path), + session_id="session-a", + stream_generation="generation-a", + binding=binding, + ) + + result = ingestor.ingest_update(_update("usage_update", used=1, size=100)) + + assert result.event is not None + assert result.event.inserted + assert result.ignored_reason is None + + + +@pytest.mark.parametrize( + ("stop_reason", "outcome", "notice"), + ( + (StopReason.END_TURN, "completed", None), + (StopReason.MAX_TOKENS, "truncated_max_tokens", "token limit"), + ( + StopReason.MAX_TURN_REQUESTS, + "truncated_max_turn_requests", + "request limit", + ), + (StopReason.REFUSAL, "refused", "refused"), + (StopReason.CANCELLED, "cancelled", "cancelled"), + ), +) +def test_outgoing_prompt_is_durable_before_no_echo_completion_stop_reason( + tmp_path: Path, + stop_reason: StopReason, + outcome: str, + notice: str | None, +) -> None: + events: list[AgentEvent] = [] + turns: list[dict[str, object]] = [] + + def append(_path, _host, event, **_kwargs): + events.append(event) + return _appended(len(events), event) + + def apply(_path, _host, _worker, content, **_kwargs): + turns.append(dict(content)) + return TurnRefreshApplyResult(len(turns), False) + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + persist_event=_persist(append, apply), + ) + begun = ingestor.begin_prompt( + ({"type": "text", "text": "question not echoed"},), + producer_turn_id="turn-a", + ) + completed = ingestor.mark_prompt_complete(stop_reason) + + assert begun.event is not None and begun.event.status == "inserted" + assert events[0].kind == "user_message" + assert events[0].payload["outgoing"] is True + assert turns[0]["user_text"] == "question not echoed" + assert completed.event is not None + assert events[-1].payload["stop_reason"] == stop_reason.value + assert events[-1].payload["outcome"] == outcome + final_text = str(turns[-1]["assistant_final_text"]) + assert (notice is not None and notice in final_text) or ( + notice is None and final_text == "" + ) + + +def test_live_prompt_echo_is_suppressed_but_load_replay_user_message_is_retained( + tmp_path: Path, +) -> None: + db_path = tmp_path / "events.db" + binding = _binding() + save_snapshot( + db_path, + Snapshot( + host_id="host-a", + updated_at="2026-01-01T00:00:00+00:00", + workers=[ + Worker( + id=binding.worker_id, + name="Worker A", + fingerprint=binding.worker_fingerprint, + ) + ], + ), + ) + upsert_worker_bindings(db_path, [binding]) + ingestor = AcpSessionIngestor( + _config(db_path), + session_id="session-a", + stream_generation="generation-a", + binding=binding, + ) + ingestor.begin_prompt( + ({"type": "text", "text": "one question"},), + producer_turn_id="producer-turn-a", + ) + + echo = ingestor.ingest_update( + _update( + "user_message_chunk", + messageId="adapter-echo", + content={"type": "text", "text": "one question"}, + ) + ) + historical = ingestor.ingest_update( + _update( + "user_message_chunk", + messageId="historical-user", + content={"type": "text", "text": "historical question"}, + ), + replay=True, + setup_replay=True, + ) + + assert echo.event is None and echo.ignored_reason == "prompt_echo" + assert historical.event is not None + events = list_agent_events(db_path, "host-a") + assert [event.event.kind for event in events] == ["user_message", "user_message"] + assert events[0].event.payload["assembled_text"] == "one question" + assert events[1].event.payload["assembled_text"] == "historical question" + assert read_public_test_agent_events(db_path, "host-a") == () + + +def test_steering_prompt_appends_to_active_turn_without_resetting_identity( + tmp_path: Path, +) -> None: + db_path = tmp_path / "events.db" + events: list[AgentEvent] = [] + turns: list[dict[str, object]] = [] + + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: + events.append(event) + return _appended(len(events), event) + + def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): + turns.append(dict(content)) + return TurnRefreshApplyResult(len(turns), False) + + binding = _binding() + ingestor = AcpSessionIngestor( + _config(db_path), + session_id="session-a", + stream_generation="generation-a", + binding=binding, + persist_event=_persist(append, apply), + ) + begun = ingestor.begin_prompt( + ({"type": "text", "text": "initial"},), + producer_turn_id="producer-initial", + ) + source_turn_id = ingestor.source_turn_id + steered = ingestor.append_prompt( + ({"type": "text", "text": "live follow-up"},), + producer_turn_id="producer-steer", + ) + + assert begun.event is not None + assert steered.event is not None + assert events[1].payload["steering"] is True + assert ingestor.source_turn_id == source_turn_id + content = ingestor.projector.project_turn_content("session-a") + assert content["user_text"] == "initial\n\nlive follow-up" + + ingestor.mark_prompt_complete() + assert not ingestor.can_append_prompt() + with pytest.raises(RuntimeError, match="active turn"): + ingestor.append_prompt( + ({"type": "text", "text": "too late"},), + producer_turn_id="producer-late", + ) + + +@pytest.mark.parametrize( + ("update_kind", "fields"), + [ + ("available_commands_update", {"availableCommands": [{"name": "review"}]}), + ("current_mode_update", {"currentModeId": "agent"}), + ( + "config_option_update", + {"configOptions": [{"id": "model", "currentValue": "safe"}]}, + ), + ], +) +def test_stable_control_updates_persist_privately_and_replay_idempotently( + tmp_path: Path, + update_kind: str, + fields: dict[str, object], +) -> None: + db_path = tmp_path / f"{update_kind}.db" + binding = _binding() + upsert_worker_bindings(db_path, [binding]) + notification = _update(update_kind, **fields) + + outcomes = [] + for generation in ("generation-a", "generation-b"): + ingestor = AcpSessionIngestor( + _config(db_path), + session_id="session-a", + stream_generation=generation, + binding=binding, + ) + outcomes.append( + ingestor.ingest_update( + notification, + replay=True, + setup_replay=True, + ) + ) + + assert outcomes[0].event is not None and outcomes[0].event.status == "inserted" + assert outcomes[1].event is not None and outcomes[1].event.status == "replayed" + events = list_agent_events(db_path, "host-a") + assert len(events) == 1 + assert events[0].event.kind == "extension" + assert events[0].event.visibility == "private" + assert events[0].event.public_payload == {} + assert events[0].event.payload["extension"] == ( + f"acp.session_update.{update_kind}" + ) + assert read_public_test_agent_events(db_path, "host-a") == () + + +def test_duplicate_durable_event_can_idempotently_repair_projection(tmp_path: Path) -> None: + projected = False + + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: + return AppendBoundAgentEventResult("replayed", event.event_id, 9) + + def apply(*_args, **_kwargs): + nonlocal projected + projected = True + return TurnRefreshApplyResult(1, False) + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + persist_event=_persist(append, apply), + ) + result = ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "replayed"}, + ), + source_event_id="event-1", + replay=True, + ) + + assert result.ignored_reason == "duplicate_event" + assert projected + assert ingestor.source_turn_id is not None + assert ingestor.projector.session_snapshot("session-a") is not None + + +def test_append_exception_rolls_back_turn_identity_sequence_and_message( + tmp_path: Path, +) -> None: + attempts = 0 + + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: + nonlocal attempts + attempts += 1 + if attempts == 1: + raise RuntimeError("durable append failed") + return _appended(1, event) + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + persist_event=_persist( + append, + lambda *_args, **_kwargs: TurnRefreshApplyResult(1, False), + ), + ) + notification = _update( + "agent_message_chunk", + content={"type": "text", "text": "exactly once"}, + ) + + with pytest.raises(RuntimeError, match="durable append failed"): + ingestor.ingest_update(notification) + assert ingestor.source_turn_id is None + assert ingestor.projector.session_snapshot("session-a") is None + + accepted = ingestor.ingest_update(notification) + assert accepted.event is not None and accepted.event.status == "inserted" + snapshot = ingestor.projector.session_snapshot("session-a") + assert snapshot is not None and snapshot["sequence"] == 1 + assert ingestor.projector.project_turn_content("session-a")[ + "assistant_stream_text" + ] == "exactly once" + + +def test_oversized_first_chunk_does_not_leave_an_implicit_turn(tmp_path: Path) -> None: + from tendwire.backends.acp_projection import AcpEventProjector, AcpProjectionError + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + projector=AcpEventProjector(max_event_bytes=128), + ) + + with pytest.raises(AcpProjectionError, match="size limit"): + ingestor.ingest_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "x" * 300}, + ) + ) + assert ingestor.source_turn_id is None + assert ingestor.projector.session_snapshot("session-a") is None + + +def test_atomic_durable_replay_can_repair_projection( + tmp_path: Path, +) -> None: + db_path = tmp_path / "events.db" + binding = _binding() + upsert_worker_bindings(db_path, [binding]) + turns: list[dict[str, object]] = [] + + def apply(_path: Path | str, _host: str, _worker: str, content, **_kwargs): + turns.append(dict(content)) + return TurnRefreshApplyResult(len(turns), False) + + def ingestor() -> AcpSessionIngestor: + return AcpSessionIngestor( + _config(db_path), + session_id="session-a", + stream_generation="generation-a", + binding=binding, + persist_event=_persist( + lambda _path, _host, event, **_kwargs: ( + AppendBoundAgentEventResult( + "inserted" if not turns else "replayed", + event.event_id, + 1, + ) + ), + apply, + ), + ) + + notification = _update( + "agent_message_chunk", + messageId="assistant-1", + content={"type": "text", "text": "durable once"}, + ) + inserted = ingestor().ingest_update( + notification, + source_event_id="event-1", + ) + replayed = ingestor().ingest_update( + notification, + source_event_id="event-1", + replay=True, + ) + + assert inserted.event is not None + assert inserted.event.status == "inserted" + assert inserted.turn is not None + assert replayed.event is not None + assert replayed.event.status == "replayed" + assert replayed.ignored_reason == "duplicate_event" + assert replayed.turn is not None + assert len(turns) == 2 + + +def test_producer_turn_identity_survives_transport_recreation(tmp_path: Path) -> None: + identities: list[str] = [] + for generation in ("generation-a", "generation-b"): + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db"), + session_id="session-a", + stream_generation=generation, + binding=_binding(), + ) + identities.append(ingestor.start_turn(producer_turn_id="producer-turn-7")) + + assert identities[0] == identities[1] + + +def test_private_summary_requires_exact_trusted_marker_and_rejects_conflicts( + tmp_path: Path, +) -> None: + events: list[AgentEvent] = [] + + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: + events.append(event) + return _appended(len(events), event) + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db", acp_thought_policy="private_summary"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + persist_event=_persist(append), + ) + unclassified = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "adapter display summary"}, + ) + ) + raw = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "raw secret"}, + _meta={"tendwire": {"thought_kind": "raw"}}, + ) + ) + unknown = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "unknown secret"}, + _meta={"tendwire.dev/thought_kind": "SUMMARY"}, + ) + ) + conflicting = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={ + "type": "text", + "text": "conflicting secret", + "_meta": {"reasoning_kind": "raw"}, + }, + _meta={"tendwire.dev/thought_kind": "summary"}, + ) + ) + summary = ingestor.ingest_update( + _update( + "agent_thought_chunk", + messageId="summary-1", + content={"type": "text", "text": "readable summary"}, + _meta={"tendwire.dev/thought_kind": "summary"}, + ) + ) + + assert unclassified.ignored_reason == "thought_policy_requires_summary" + assert raw.ignored_reason == "thought_policy_requires_summary" + assert unknown.ignored_reason == "thought_policy_requires_summary" + assert conflicting.ignored_reason == "thought_policy_requires_summary" + assert summary.event is not None + assert len(events) == 1 + assert all(event.visibility == "private" for event in events) + assert all(event.public_payload == {} for event in events) + assert "raw secret" not in repr(events) + + +def test_private_all_policy_retains_marked_raw_thought_privately(tmp_path: Path) -> None: + events: list[AgentEvent] = [] + + def append( + _path: Path | str, + _host: str, + event: AgentEvent, + **_kwargs, + ) -> AppendBoundAgentEventResult: + events.append(event) + return _appended(1, event) + + ingestor = AcpSessionIngestor( + _config(tmp_path / "events.db", acp_thought_policy="private_all"), + session_id="session-a", + stream_generation="generation-a", + binding=_binding(), + persist_event=_persist(append), + ) + result = ingestor.ingest_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "raw local diagnostic"}, + _meta={"tendwire": {"thought_kind": "raw"}}, + ) + ) + + assert result.event is not None + assert len(events) == 1 + assert events[0].payload["text_delta"] == "raw local diagnostic" + assert events[0].public_payload == {} diff --git a/tests/test_acp_permissions.py b/tests/test_acp_permissions.py new file mode 100644 index 0000000..58e53ea --- /dev/null +++ b/tests/test_acp_permissions.py @@ -0,0 +1,449 @@ +"""End-to-end durable ACP permission decision bridge tests.""" + +from __future__ import annotations + +import json +import sqlite3 +import threading +import time +from concurrent.futures import ThreadPoolExecutor +from dataclasses import replace +from pathlib import Path +from typing import Any + +import pytest +import tendwire.store.sqlite as store_sqlite +import tendwire.command_submission as command_submission +from tendwire.backends.acp_permissions import AcpPermissionBroker +from tendwire.backends.acp_protocol import ( + PermissionOption, + PermissionOptionKind, + PermissionRequest, + SessionResult, +) +from tendwire.backends.acp_runtime import AcpWorkerSession, SessionOpenMode +from tendwire.command_submission import submit_command +from tendwire.config import Config +from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding +from tendwire.daemon import TendwireDaemon +from tendwire.store.sqlite import ( + expire_worker_bindings, + get_command_request, + init_store, + list_worker_bindings, + pending_payload_from_store, + save_snapshot, + upsert_worker_bindings, +) + +from tests.test_acp_runtime import FakeClient, FakeIngestor + + +def _config(tmp_path: Path) -> Config: + return Config( + host_id="cmd-host", + data_dir=tmp_path, + db_path=tmp_path / "commands.db", + ) + + +def _binding(worker: Worker) -> WorkerBinding: + return WorkerBinding( + host_id="cmd-host", + worker_id=worker.id, + worker_fingerprint=worker.fingerprint, + backend="herdr", + target_kind="agent_id", + target_value="agent-secret", + turn_target_kind=None, + turn_target_value=None, + sendable=True, + reason=None, + observed_at="2026-01-01T00:00:00+00:00", + private_fingerprint="private-secret", + ) + + +def _seed( + config: Config, + workers: list[Worker], + bindings: list[WorkerBinding], +) -> None: + assert config.db_path is not None + init_store(config.db_path) + save_snapshot( + config.db_path, + Snapshot( + host_id=config.host_id, + updated_at="2026-01-01T00:00:00+00:00", + workers=workers, + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="healthy_non_empty", + observed_at="2026-01-01T00:00:00+00:00", + counts={"workers": len(workers)}, + ) + ], + ), + ) + upsert_worker_bindings(config.db_path, bindings) + + +def _answer_request( + decision_ref: str, + *, + request_id: str = "decision-request-1", + worker_id: str = "w-1", + selection: dict[str, Any] | None = None, +) -> dict[str, Any]: + return { + "schema_version": 1, + "action": "answer_decision", + "request_id": request_id, + "dry_run": False, + "target": {"worker_id": worker_id}, + "params": { + "decision_ref": decision_ref, + "selection": selection or {"option_refs": ["2"]}, + }, + } + + +class _Router: + def __init__(self, broker: AcpPermissionBroker) -> None: + self.broker = broker + + def owns_permission_decision(self, decision: Any) -> bool: + return self.broker.owns(decision) + + def answer_permission_decision(self, decision: Any, *, timeout: float) -> None: + self.broker.answer(decision, timeout=timeout) + + +def _permission(session_id: str) -> PermissionRequest: + options = ( + PermissionOption( + "private-allow-id", + "Allow once", + PermissionOptionKind.ALLOW_ONCE, + {}, + ), + PermissionOption( + "private-reject-id", + "Reject", + PermissionOptionKind.REJECT_ONCE, + {}, + ), + ) + return PermissionRequest( + 71, + session_id, + { + "toolCallId": "private-tool-id", + "title": "Run tests", + "rawInput": "/secret", + }, + options, + {"private": "metadata"}, + {}, + ) + + +def _wait_pending(config: Any, worker_id: str) -> dict[str, Any]: + assert config.db_path is not None + deadline = time.monotonic() + 2 + while time.monotonic() < deadline: + payload = pending_payload_from_store(config.db_path, config.host_id) + rows = [ + row + for row in payload["pending_interactions"] + if row["worker_id"] == worker_id + ] + if rows: + return rows[0] + time.sleep(0.01) + raise AssertionError("permission overlay was not published") + + +def _setup(tmp_path: Path) -> tuple[Any, Worker, str, AcpPermissionBroker]: + config = _config(tmp_path) + worker = Worker(id="w-1", name="Alpha", status="active") + continuity = _binding(worker) + _seed(config, [worker], [continuity]) + session_id = "private-session-id" + acp_binding = replace( + continuity, + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="", + ) + assert config.db_path is not None + upsert_worker_bindings(config.db_path, [acp_binding]) + return ( + config, + worker, + session_id, + AcpPermissionBroker( + config, + worker_id=worker.id, + worker_fingerprint=worker.fingerprint, + generation="42", + timeout=2, + ), + ) + + +def test_permission_bridge_is_private_durable_and_frame_acknowledged( + tmp_path: Path, +) -> None: + config, worker, session_id, broker = _setup(tmp_path) + assert config.db_path is not None + acp_binding = list_worker_bindings( + config.db_path, config.host_id, backend="acp" + )[0] + client = FakeClient() + client.restored_session_result = SessionResult(session_id, None, (), {}) + runtime = AcpWorkerSession( + client, + config=config, + binding=acp_binding, + cwd=tmp_path, + session_mode=SessionOpenMode.LOAD, + session_id=session_id, + stream_generation="42", + permission_callback=broker, + ingestor=FakeIngestor(session_id), + poll_timeout=0.01, + stop_timeout=1, + ) + runtime.start() + try: + client.permissions.put(_permission(session_id)) + pending = _wait_pending(config, worker.id) + serialized = json.dumps(pending, sort_keys=True) + assert pending["question"] == "Run tests" + assert pending["meta"]["decision"]["options"] == [ + {"ref": "1", "label": "Allow once (allow_once)"}, + {"ref": "2", "label": "Reject (reject_once)"}, + ] + for secret in ( + "private-allow-id", + "private-reject-id", + "private-tool-id", + session_id, + "/secret", + ): + assert secret not in serialized + + result = submit_command( + config, + _answer_request( + pending["meta"]["decision"]["decision_ref"], + selection={"option_refs": ["1"]}, + ), + acp_permission_router=_Router(broker), + ) + assert result.ok is True + assert client.permission_responses == [(71, "private-allow-id", False)] + receipt = get_command_request( + config.db_path, config.host_id, "decision-request-1" + ) + assert receipt is not None and receipt["state"] == "accepted" + finally: + broker.close() + runtime.stop() + + +def test_acp_permission_never_falls_back_to_legacy_socket(tmp_path: Path) -> None: + config, worker, session_id, broker = _setup(tmp_path) + thread = threading.Thread(target=lambda: broker(_permission(session_id))) + thread.start() + pending = _wait_pending(config, worker.id) + assert config.db_path is not None + assert expire_worker_bindings( + config.db_path, + config.host_id, + backend="acp", + private_fingerprints=[ + row.private_fingerprint + for row in list_worker_bindings( + config.db_path, config.host_id, backend="acp" + ) + ], + reason="test_retired_before_answer", + ) == 1 + result = submit_command( + config, + _answer_request(pending["meta"]["decision"]["decision_ref"]), + acp_permission_router=None, + ) + assert result.ok is False + assert result.disposition == "no_receipt" + broker.close() + thread.join(timeout=2) + + +def test_concurrent_answers_write_exactly_one_permission_response( + tmp_path: Path, +) -> None: + config, worker, session_id, broker = _setup(tmp_path) + selections: list[Any] = [] + + def adapter_side() -> None: + selected = broker(_permission(session_id)) + selections.append(selected) + assert selected is not None + selected.response_written() + + adapter = threading.Thread(target=adapter_side) + adapter.start() + pending = _wait_pending(config, worker.id) + decision_ref = pending["meta"]["decision"]["decision_ref"] + router = _Router(broker) + requests = [ + _answer_request(decision_ref, request_id=f"concurrent-{index}") + for index in range(2) + ] + with ThreadPoolExecutor(max_workers=2) as pool: + results = list( + pool.map( + lambda request: submit_command( + config, + request, + acp_permission_router=router, + ), + requests, + ) + ) + adapter.join(timeout=2) + assert not adapter.is_alive() + assert sum(result.ok is True for result in results) == 1 + assert len(selections) == 1 + assert selections[0].option_id in { + "private-allow-id", + "private-reject-id", + } + + +def test_broker_stop_cancels_waiter_and_closes_public_overlay(tmp_path: Path) -> None: + config, worker, session_id, broker = _setup(tmp_path) + outcomes: list[Any] = [] + thread = threading.Thread( + target=lambda: outcomes.append(broker(_permission(session_id))) + ) + thread.start() + _wait_pending(config, worker.id) + broker.close() + thread.join(timeout=2) + assert not thread.is_alive() + assert outcomes == [None] + assert config.db_path is not None + payload = pending_payload_from_store(config.db_path, config.host_id) + assert all( + row["worker_id"] != worker.id + for row in payload["pending_interactions"] + ) + + +def test_late_permission_frame_completion_retires_uncertain_overlay( + tmp_path: Path, +) -> None: + config, worker, session_id, broker = _setup(tmp_path) + selections: list[Any] = [] + + def adapter_side() -> None: + selections.append(broker(_permission(session_id))) + + adapter = threading.Thread(target=adapter_side) + adapter.start() + pending = _wait_pending(config, worker.id) + router = _Router(broker) + result = submit_command( + replace(config, acp_request_timeout_seconds=0.01), + _answer_request(pending["meta"]["decision"]["decision_ref"]), + acp_permission_router=router, + ) + adapter.join(timeout=2) + assert not adapter.is_alive() + assert len(selections) == 1 and selections[0] is not None + assert result.status == "request_state_uncertain" + + # The JSON-RPC writer completes after the durable command deadline. The + # original receipt remains uncertain and cannot be replayed, while the + # now-resolved permission prompt is removed instead of becoming a forever + # visible, forever-unanswerable overlay. + selections[0].response_written() + assert config.db_path is not None + payload = pending_payload_from_store(config.db_path, config.host_id) + assert all( + row["worker_id"] != worker.id + for row in payload["pending_interactions"] + ) + receipt = get_command_request( + config.db_path, + config.host_id, + "decision-request-1", + ) + assert receipt is not None and receipt["state"] == "uncertain" + broker.close() + + +def test_failed_permission_frame_retires_overlay_without_retry( + tmp_path: Path, +) -> None: + config, worker, session_id, broker = _setup(tmp_path) + + def adapter_side() -> None: + selected = broker(_permission(session_id)) + assert selected is not None + selected.response_failed(OSError("partial private frame")) + + adapter = threading.Thread(target=adapter_side) + adapter.start() + pending = _wait_pending(config, worker.id) + result = submit_command( + config, + _answer_request(pending["meta"]["decision"]["decision_ref"]), + acp_permission_router=_Router(broker), + ) + adapter.join(timeout=2) + assert not adapter.is_alive() + assert result.status == "request_state_uncertain" + assert "private frame" not in json.dumps(result.to_dict()) + assert config.db_path is not None + payload = pending_payload_from_store(config.db_path, config.host_id) + assert all( + row["worker_id"] != worker.id + for row in payload["pending_interactions"] + ) + broker.close() + + + + +def test_daemon_routes_acp_permission_answers_without_a_prompt_route( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = _config(tmp_path) + + class Runtime: + def answer_permission_decision(self, _decision: Any, *, timeout: float) -> None: + del timeout + + runtime = Runtime() + daemon = TendwireDaemon(config) + daemon._acp_supervisor = runtime + captured: dict[str, Any] = {} + + def submit(_config: Any, _payload: Any, **kwargs: Any) -> str: + captured.update(kwargs) + return "routed" + + monkeypatch.setattr(command_submission, "submit_command", submit) + assert daemon.submit_command({"schema_version": 1, "action": "noop"}) == "routed" + assert captured["acp_permission_router"] is runtime + assert captured["acp_prompt_router"] is None diff --git a/tests/test_acp_probe.py b/tests/test_acp_probe.py new file mode 100644 index 0000000..5dfd4f4 --- /dev/null +++ b/tests/test_acp_probe.py @@ -0,0 +1,199 @@ +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +from tendwire.backends.acp_probe import ( + MAX_PROBE_CLOSE_TIMEOUT_SECONDS, + MAX_PROBE_TIMEOUT_SECONDS, + ProbeFailure, + _extension_capability_count, + main, + probe_adapter, +) + + +FAKE_AGENT = Path(__file__).parent / "fixtures" / "acp_fake_agent.py" + + +def adapter_argv(mode: str = "normal") -> list[str]: + return [sys.executable, "-u", str(FAKE_AGENT), mode] + + +def test_probe_negotiates_fresh_stable_v1_capabilities_and_reaps_process() -> None: + first = probe_adapter(adapter_argv()) + second = probe_adapter(adapter_argv()) + + for report in (first, second): + payload = report.to_payload() + assert payload["schema_version"] == 2 + assert payload["probe_scope"] == "initialize" + assert payload["initialization_compatible"] is True + assert payload["protocol_version"] == 1 + assert payload["process_reaped"] is True + assert payload["failure"] is None + assert payload["advertised_capabilities"]["session_load"] is True + assert payload["advertised_capabilities"]["session_close"] is True + assert payload["advertised_capabilities"]["session_delete"] is True + assert payload["extensions"] == { + "capability_count": 1, + "capability_count_capped": False, + } + + +def test_baseline_adapter_reports_only_baseline_capabilities() -> None: + payload = probe_adapter(adapter_argv("baseline")).to_payload() + assert payload["initialization_compatible"] is True + assert "session_prompt" not in payload["advertised_capabilities"] + assert "session_new" not in payload["advertised_capabilities"] + assert payload["advertised_capabilities"]["session_load"] is False + assert payload["advertised_capabilities"]["session_list"] is False + assert payload["extensions"]["capability_count"] == 0 + + +def test_initialize_only_agent_does_not_gain_untested_baseline_claims() -> None: + payload = probe_adapter(adapter_argv("initialize_only")).to_payload() + assert payload["initialization_compatible"] is True + assert payload["probe_scope"] == "initialize" + assert not { + "session_new", + "session_prompt", + "session_cancel", + "session_update", + }.intersection(payload["advertised_capabilities"]) + + +@pytest.mark.parametrize( + ("mode", "failure", "timeout"), + [ + ("malformed", ProbeFailure.PROTOCOL, 2.0), + ("partial_eof", ProbeFailure.PROTOCOL, 2.0), + ("bool_version", ProbeFailure.PROTOCOL_VERSION, 2.0), + ("no_read", ProbeFailure.TIMEOUT, 0.05), + ], +) +def test_probe_fails_closed_with_fixed_failure_categories( + mode: str, + failure: ProbeFailure, + timeout: float, +) -> None: + report = probe_adapter( + adapter_argv(mode), + timeout_seconds=timeout, + close_timeout_seconds=0.05, + ) + assert report.initialization_compatible is False + assert report.failure is failure + assert report.process_reaped is True + assert not any(report.advertised_capabilities.values()) + + +def test_missing_executable_does_not_expose_argv_or_exception_text() -> None: + secret = "TOP_SECRET_ADAPTER_ARGUMENT" + payload = probe_adapter(["/definitely/not/an/acp-adapter", secret]).to_payload() + encoded = json.dumps(payload) + assert payload["initialization_compatible"] is False + assert payload["failure"] == ProbeFailure.LAUNCH_FAILED.value + assert secret not in encoded + assert "/definitely/not" not in encoded + + +def test_report_never_exposes_agent_or_extension_controlled_text() -> None: + payload = probe_adapter(adapter_argv("extensions")).to_payload() + encoded = json.dumps(payload) + assert len(encoded.encode("utf-8")) < 2048 + assert "fake" not in encoded + assert "vendor.example" not in encoded + assert "_vendor.example/future_notification" not in encoded + assert "level" not in encoded + assert payload["extensions"]["capability_count"] == 1 + + +def test_timeout_configuration_is_strictly_bounded() -> None: + for timeout in (0, -1, float("inf"), MAX_PROBE_TIMEOUT_SECONDS + 1): + report = probe_adapter(adapter_argv(), timeout_seconds=timeout) + assert report.failure is ProbeFailure.INVALID_CONFIGURATION + report = probe_adapter( + adapter_argv(), + close_timeout_seconds=MAX_PROBE_CLOSE_TIMEOUT_SECONDS + 1, + ) + assert report.failure is ProbeFailure.INVALID_CONFIGURATION + + +def test_stubborn_adapter_is_reaped_without_becoming_source_dependency() -> None: + report = probe_adapter( + adapter_argv("stubborn"), + timeout_seconds=1, + close_timeout_seconds=0.05, + ) + assert report.initialization_compatible is True + assert report.process_reaped is True + + +def test_module_cli_outputs_one_bounded_json_object(capsys: pytest.CaptureFixture[str]) -> None: + exit_code = main(["--timeout", "1", "--", *adapter_argv("baseline")]) + captured = capsys.readouterr() + payload = json.loads(captured.out) + assert exit_code == 0 + assert captured.err == "" + assert payload["initialization_compatible"] is True + assert len(captured.out.encode("utf-8")) < 2048 + + +def test_module_runs_as_black_box_without_importing_adapter_source() -> None: + source_root = str(Path(__file__).parents[1] / "src") + env = dict(os.environ) + env["PYTHONPATH"] = source_root + completed = subprocess.run( + [ + sys.executable, + "-m", + "tendwire.backends.acp_probe", + "--timeout", + "1", + "--", + *adapter_argv("baseline"), + ], + cwd=Path(__file__).parents[1], + env=env, + capture_output=True, + text=True, + timeout=5, + check=False, + ) + payload = json.loads(completed.stdout) + assert completed.returncode == 0 + assert completed.stderr == "" + assert payload["initialization_compatible"] is True + assert payload["process_reaped"] is True + + +def test_extension_count_uses_only_spec_reserved_meta_locations() -> None: + assert ( + _extension_capability_count( + { + "_meta": {"vendor.one": {}}, + "promptCapabilities": {"_meta": {"vendor.two": {}}}, + "sessionCapabilities": { + "list": {"_meta": {"vendor.three": {}}} + }, + } + ) + == 3 + ) + assert _extension_capability_count({"forbiddenRootExtension": {}}) == 0 + + +def test_upstream_schema_rejects_invalid_authentication_items() -> None: + payload = probe_adapter(adapter_argv("auth_shapes")).to_payload() + assert payload["initialization_compatible"] is False + assert payload["authentication"] == { + "method_count": 0, + "method_count_capped": False, + } + assert payload["failure"] == "protocol_error" diff --git a/tests/test_acp_projection.py b/tests/test_acp_projection.py new file mode 100644 index 0000000..79fe025 --- /dev/null +++ b/tests/test_acp_projection.py @@ -0,0 +1,952 @@ +from __future__ import annotations + +import pytest + +from tendwire.backends.acp_projection import AcpEventProjector, AcpProjectionError + + +def _update(session_update: str, **fields: object) -> dict[str, object]: + return { + "jsonrpc": "2.0", + "method": "session/update", + "params": { + "sessionId": "session-1", + "update": {"sessionUpdate": session_update, **fields}, + }, + } + + +@pytest.mark.parametrize( + ("update_kind", "fields"), + [ + ( + "available_commands_update", + {"availableCommands": [{"name": "review", "description": "Review"}]}, + ), + ("current_mode_update", {"currentModeId": "agent"}), + ( + "config_option_update", + {"configOptions": [{"id": "model", "currentValue": "safe"}]}, + ), + ], +) +def test_stable_session_control_updates_are_private_canonical_extensions( + update_kind: str, + fields: dict[str, object], +) -> None: + projector = AcpEventProjector() + notification = _update(update_kind, **fields) + + event = projector.normalize_session_update( + notification, + source_event_id="stable-update-1", + ) + duplicate = projector.normalize_session_update( + notification, + source_event_id="stable-update-1", + replay=True, + ) + + assert event is not None + assert event["kind"] == "extension" + assert event["privacy"] == "private" + assert event["private_fields"] == ["payload"] + assert event["payload"] == { + "schema_version": 1, + "extension": f"acp.session_update.{update_kind}", + "update": fields, + } + assert duplicate is None + assert projector.project_turn_content("session-1")["has_open_turn"] is False + + +def test_message_chunks_are_assembled_by_session_kind_and_message_id() -> None: + projector = AcpEventProjector() + + first = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "hello "}, + ) + ) + second = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "world"}, + ) + ) + third = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-2", + content={"type": "text", "text": "follow-up"}, + ) + ) + + assert first is not None and first["payload"]["assembled_text"] == "hello " + assert second is not None and second["payload"]["assembled_text"] == "hello world" + assert third is not None and third["payload"]["message_index"] == 1 + assert [first["sequence"], second["sequence"], third["sequence"]] == [1, 2, 3] + assert projector.project_turn_content("session-1") == { + "user_text": "", + "assistant_stream_text": "hello world\n\nfollow-up", + "assistant_final_text": "", + "complete": False, + "has_open_turn": True, + } + + +def test_thoughts_are_private_and_never_enter_legacy_turn_content() -> None: + projector = AcpEventProjector() + thought = projector.normalize_session_update( + _update( + "agent_thought_chunk", + messageId="reasoning-1", + content={"type": "text", "text": "private reasoning"}, + ) + ) + projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "safe answer"}, + ) + ) + + assert thought is not None + assert thought["kind"] == "thought" + assert thought["privacy"] == "private" + assert thought["private_fields"] == ["payload"] + legacy = projector.mark_turn_complete("session-1") + assert legacy["assistant_final_text"] == "safe answer" + assert legacy["assistant_stream_text"] == "" + assert "reasoning" not in repr(legacy) + + +def test_non_text_content_is_preserved_without_becoming_turn_text() -> None: + projector = AcpEventProjector() + event = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="image-1", + content={"type": "image", "data": "sensitive-base64", "mimeType": "image/png"}, + ) + ) + + assert event is not None + assert event["payload"]["content"]["type"] == "image" + assert event["payload"]["text_delta"] == "" + assert projector.project_turn_content("session-1")["assistant_stream_text"] == "" + + +def test_assistant_only_text_stays_in_private_event_but_not_legacy_turn() -> None: + projector = AcpEventProjector() + private = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="private-1", + content={ + "type": "text", + "text": "assistant-only context", + "annotations": {"audience": ["assistant"]}, + }, + ) + ) + public = projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="public-1", + content={ + "type": "text", + "text": "safe answer", + "annotations": {"audience": ["user", "invalid-role"]}, + }, + ) + ) + + assert private is not None + assert private["payload"]["content"]["text"] == "assistant-only context" + assert private["payload"]["assembled_text"] == "" + assert public is not None + assert public["payload"]["content"]["annotations"]["audience"] == ["user"] + assert projector.project_turn_content("session-1")["assistant_stream_text"] == ( + "safe answer" + ) + + +def test_user_and_agent_text_remain_separate_and_reset_starts_new_turn() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "user_message_chunk", + messageId="prompt-1", + content={"type": "text", "text": "question"}, + ) + ) + projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "answer"}, + ) + ) + assert projector.mark_turn_complete("session-1") == { + "user_text": "question", + "assistant_stream_text": "", + "assistant_final_text": "answer", + "complete": True, + "has_open_turn": False, + } + + projector.reset_turn("session-1") + assert projector.project_turn_content("session-1")["user_text"] == "" + assert projector.session_snapshot("session-1")["sequence"] == 2 + + +def test_tool_lifecycle_merges_partial_updates_and_marks_raw_fields_private() -> None: + projector = AcpEventProjector() + started = projector.normalize_session_update( + _update( + "tool_call", + toolCallId="tool-1", + title="Read configuration", + kind="read", + status="pending", + rawInput={"path": "/private/file"}, + ) + ) + updated = projector.normalize_session_update( + _update( + "tool_call_update", + toolCallId="tool-1", + status="completed", + content=[{"type": "content", "content": {"type": "text", "text": "done"}}], + rawOutput={"secret": "value"}, + ) + ) + + assert started is not None and started["kind"] == "tool_call" + assert updated is not None and updated["kind"] == "tool_call_update" + snapshot = updated["payload"]["snapshot"] + assert snapshot["title"] == "Read configuration" + assert snapshot["status"] == "completed" + assert snapshot["rawInput"] == {"path": "/private/file"} + assert snapshot["rawOutput"] == {"secret": "value"} + assert updated["privacy"] == "mixed" + assert "payload.snapshot.rawInput" in updated["private_fields"] + assert "payload.snapshot.rawOutput" in updated["private_fields"] + assert projector.project_turn_content("session-1")["assistant_stream_text"] == "" + + +def test_permission_request_updates_tool_and_keeps_options() -> None: + projector = AcpEventProjector() + request = { + "jsonrpc": "2.0", + "id": 42, + "method": "session/request_permission", + "params": { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-9", "status": "pending"}, + "options": [ + {"optionId": "yes", "name": "Allow", "kind": "allow_once"}, + {"optionId": "no", "name": "Reject", "kind": "reject_once"}, + ], + }, + } + event = projector.normalize_permission_request(request, source_event_id="permission-42") + + assert event is not None + assert event["kind"] == "tool_call_update" + assert event["payload"]["permission"]["required"] is True + assert event["payload"]["permission"]["options"][1]["optionId"] == "no" + assert event["source_event_id"] == "permission-42" + assert event["event_id"] == "acp:session-1:permission-42" + assert "payload.permission" in event["private_fields"] + + assert ( + projector.normalize_permission_request( + request, source_event_id="permission-42" + ) + is None + ) + + +def test_plan_usage_and_session_info_are_full_or_merged_snapshots() -> None: + projector = AcpEventProjector() + plan = projector.normalize_session_update( + _update( + "plan", + entries=[ + {"content": "Implement", "priority": "high", "status": "in_progress"} + ], + ) + ) + usage = projector.normalize_session_update( + _update("usage_update", used=25, size=100, cost={"amount": 0.1, "currency": "USD"}) + ) + title = projector.normalize_session_update( + _update("session_info_update", title="ACP migration") + ) + cleared = projector.normalize_session_update( + _update("session_info_update", title=None, updatedAt="2026-07-31T12:00:00Z") + ) + + assert plan is not None and plan["payload"]["snapshot"] is True + assert usage is not None and usage["payload"]["used"] == 25 + assert title is not None and title["payload"]["title"] == "ACP migration" + assert cleared is not None and cleared["payload"]["title"] is None + assert cleared["payload"]["updatedAt"] == "2026-07-31T12:00:00Z" + + +@pytest.mark.parametrize( + "content", + [ + {"type": "text", "text": "hello", "_meta": {"traceparent": "00-abc"}}, + {"type": "image", "data": "aW1hZ2U=", "mimeType": "image/png"}, + {"type": "audio", "data": "YXVkaW8=", "mimeType": "audio/wav"}, + { + "type": "resource_link", + "name": "source.py", + "uri": "file:///workspace/source.py", + "mimeType": "text/x-python", + "size": 42, + }, + { + "type": "resource", + "resource": { + "uri": "file:///workspace/source.py", + "mimeType": "text/x-python", + "text": "print('ok')", + "_meta": {"messageCount": 1}, + }, + }, + ], +) +def test_official_v1_content_block_shapes_are_accepted(content: dict[str, object]) -> None: + projector = AcpEventProjector() + event = projector.normalize_session_update( + _update("agent_message_chunk", messageId="message-1", content=content) + ) + + assert event is not None + assert event["payload"]["content"] == content + + +def test_official_v1_tool_shapes_are_validated_and_preserved() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "tool_call", + toolCallId="tool-1", + title="Edit source", + kind="edit", + status="in_progress", + locations=[{"path": "/workspace/source.py", "line": 7}], + ) + ) + event = projector.normalize_session_update( + _update( + "tool_call_update", + toolCallId="tool-1", + status="completed", + content=[ + { + "type": "diff", + "path": "/workspace/source.py", + "oldText": "old", + "newText": "new", + "_meta": {"traceparent": "00-tool"}, + }, + {"type": "terminal", "terminalId": "terminal-1"}, + { + "type": "content", + "content": {"type": "text", "text": "done"}, + }, + ], + ) + ) + + assert event is not None + assert event["payload"]["snapshot"]["status"] == "completed" + assert event["payload"]["snapshot"]["locations"] == [ + {"path": "/workspace/source.py", "line": 7} + ] + + +@pytest.mark.parametrize( + "notification,match", + [ + (_update("agent_message_chunk", content={"type": "text"}), "string text"), + (_update("agent_message_chunk", content={"type": "future"}), "unsupported type"), + (_update("tool_call", toolCallId="tool-1"), "string title"), + (_update("usage_update", used=1), "usage size"), + (_update("usage_update", used=True, size=10), "usage used"), + ], +) +def test_malformed_supported_v1_updates_fail_without_allocating_state( + notification: dict[str, object], match: str +) -> None: + projector = AcpEventProjector() + with pytest.raises(AcpProjectionError, match=match): + projector.normalize_session_update(notification) + assert projector.session_snapshot("session-1") is None + + +def test_usage_update_is_a_complete_snapshot_and_omission_clears_cost() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "usage_update", + used=5, + size=100, + cost={"amount": 0.25, "currency": "USD"}, + ) + ) + event = projector.normalize_session_update( + _update("usage_update", used=7, size=100) + ) + + assert event is not None + assert event["payload"] == {"used": 7, "size": 100} + assert projector.session_snapshot("session-1")["usage"] == { + "used": 7, + "size": 100, + } + + +def test_optional_fields_default_and_invalid_collection_items_are_skipped() -> None: + projector = AcpEventProjector() + tool = projector.normalize_session_update( + _update( + "tool_call", + toolCallId="tool-1", + title="Forward compatible", + kind="future_kind", + status="cancelled", + content=[ + {"type": "future"}, + {"type": "content", "content": {"type": "text", "text": "ok"}}, + ], + locations=[{"path": "/valid"}, {"path": 7}], + _meta="invalid optional metadata", + ) + ) + plan = projector.normalize_session_update( + _update( + "plan", + entries=[ + {"content": "keep", "priority": "high", "status": "pending"}, + {"content": "skip", "priority": "future", "status": "pending"}, + ], + ) + ) + info = projector.normalize_session_update( + _update("session_info_update", title=7, updatedAt="valid") + ) + usage = projector.normalize_session_update( + _update( + "usage_update", + used=1, + size=2, + cost={"amount": 10**400, "currency": "USD"}, + ) + ) + + assert tool is not None + assert tool["payload"]["snapshot"]["kind"] == "other" + assert tool["payload"]["snapshot"]["status"] == "pending" + assert len(tool["payload"]["snapshot"]["content"]) == 1 + assert tool["payload"]["snapshot"]["locations"] == [{"path": "/valid"}] + assert plan is not None and plan["payload"]["entries"] == [ + {"content": "keep", "priority": "high", "status": "pending"} + ] + assert info is not None and info["payload"] == {"updatedAt": "valid"} + assert usage is not None and usage["payload"] == {"used": 1, "size": 2} + + +def test_implicit_v1_message_is_split_after_an_update_type_boundary() -> None: + projector = AcpEventProjector() + first = projector.normalize_session_update( + _update("agent_message_chunk", content={"type": "text", "text": "before"}) + ) + projector.normalize_session_update( + _update("tool_call", toolCallId="tool-1", title="Boundary") + ) + second = projector.normalize_session_update( + _update("agent_message_chunk", content={"type": "text", "text": "after"}) + ) + + assert first is not None and second is not None + assert first["payload"]["message_id"] == "implicit-agent_message-1" + assert second["payload"]["message_id"] == "implicit-agent_message-2" + assert projector.project_turn_content("session-1")["assistant_stream_text"] == ( + "before\n\nafter" + ) + + +def test_explicit_event_ids_dedupe_replay_but_identical_unidentified_chunks_do_not() -> None: + projector = AcpEventProjector() + notification = _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "ha"}, + ) + + first = projector.normalize_session_update( + notification, source_event_id="transport-sequence-7", replay=True + ) + duplicate = projector.normalize_session_update( + notification, source_event_id="transport-sequence-7", replay=True + ) + repeated_text = projector.normalize_session_update(notification) + + assert first is not None and first["replay"] is True + assert duplicate is None + assert repeated_text is not None + assert projector.project_turn_content("session-1")["assistant_stream_text"] == "haha" + assert len(first["dedupe_hint"]) == 64 + + +def test_meta_event_id_is_opaque_and_not_used_as_replay_key() -> None: + projector = AcpEventProjector() + notification = _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "once"}, + _meta={"eventId": "adapter-99"}, + ) + first = projector.normalize_session_update(notification) + second = projector.normalize_session_update(notification) + assert first is not None and second is not None + assert first["source_event_id"] is None + assert first["payload"]["extensions"]["update"] == {"eventId": "adapter-99"} + + +def test_unknown_update_is_forward_compatible_and_malformed_input_is_rejected() -> None: + projector = AcpEventProjector() + assert projector.normalize_session_update(_update("future_update", value=1)) is None + with pytest.raises(AcpProjectionError, match="sessionId"): + projector.normalize_session_update( + {"update": {"sessionUpdate": "agent_message_chunk", "content": {}}} + ) + with pytest.raises(AcpProjectionError, match="toolCallId"): + projector.normalize_permission_request( + {"sessionId": "session-1", "toolCall": {}, "options": []} + ) + with pytest.raises(AcpProjectionError, match="invalid kind"): + projector.normalize_permission_request( + { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-1"}, + "options": [ + {"optionId": "maybe", "name": "Maybe", "kind": "sometimes"} + ], + } + ) + + +def test_failed_normalization_does_not_consume_sequence_or_replay_id() -> None: + projector = AcpEventProjector() + malformed = _update( + "agent_message_chunk", messageId="answer-1", content="not-an-object" + ) + with pytest.raises(AcpProjectionError, match="missing object content"): + projector.normalize_session_update(malformed, source_event_id="event-1") + + valid = _update( + "agent_message_chunk", + messageId="answer-1", + content={"type": "text", "text": "recovered"}, + ) + event = projector.normalize_session_update(valid, source_event_id="event-1") + assert event is not None + assert event["sequence"] == 1 + assert event["dedupe_safe"] is True + + +def test_sessions_have_independent_ordering_and_defensive_snapshots() -> None: + projector = AcpEventProjector() + first = projector.normalize_session_update( + _update("usage_update", used=1, size=10) + ) + other = projector.normalize_session_update( + { + "sessionId": "session-2", + "update": {"sessionUpdate": "usage_update", "used": 2, "size": 20}, + } + ) + assert first is not None and first["sequence"] == 1 + assert other is not None and other["sequence"] == 1 + + snapshot = projector.session_snapshot("session-1") + assert snapshot is not None + snapshot["usage"]["used"] = 999 + assert projector.session_snapshot("session-1")["usage"]["used"] == 1 + + +def test_interleaved_explicit_messages_and_implicit_v1_chunks_do_not_alias() -> None: + projector = AcpEventProjector() + + for message_id, text in (("a", "A1"), ("b", "B")): + projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId=message_id, + content={"type": "text", "text": text}, + ) + ) + with pytest.raises(AcpProjectionError, match="reused after a message boundary"): + projector.normalize_session_update( + _update( + "agent_message_chunk", + messageId="a", + content={"type": "text", "text": "A2"}, + ) + ) + implicit = projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "stable-v1"}, + ) + ) + + assert implicit is not None + assert implicit["payload"]["message_id"] == "implicit-agent_message-1" + assert projector.project_turn_content("session-1")["assistant_stream_text"] == ( + "A1\n\nB\n\nstable-v1" + ) + + +def test_replay_identity_collision_is_rejected_and_sessions_are_isolated() -> None: + projector = AcpEventProjector() + first = _update( + "agent_message_chunk", + content={"type": "text", "text": "one"}, + ) + assert projector.normalize_session_update(first, source_event_id="event-7") + assert projector.normalize_session_update(first, source_event_id="event-7") is None + + with pytest.raises(AcpProjectionError, match="reused for different content"): + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "different"}, + ), + source_event_id="event-7", + ) + + other = { + "sessionId": "session-2", + "update": first["params"]["update"], + } + isolated = projector.normalize_session_update(other, source_event_id="event-7") + assert isolated is not None and isolated["sequence"] == 1 + + +def test_permission_request_ids_do_not_collide_with_notification_event_ids() -> None: + projector = AcpEventProjector() + notification = _update( + "tool_call", + toolCallId="tool-1", + title="Tracked tool", + status="pending", + _meta={"eventId": 42}, + ) + assert projector.normalize_session_update( + notification, source_event_id="notification-42" + ) is not None + permission = projector.normalize_permission_request( + { + "jsonrpc": "2.0", + "id": 42, + "method": "session/request_permission", + "params": { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-1"}, + "options": [ + {"optionId": "yes", "name": "Allow", "kind": "allow_once"} + ], + }, + } + ) + assert permission is not None + assert permission["source_event_id"] is None + + +def test_namespaced_extension_metadata_is_private_and_adapter_neutral() -> None: + projector = AcpEventProjector(max_sessions=1) + # Unknown variants neither allocate a session nor consume ordering state. + assert projector.normalize_session_update(_update("vendor/future", value=1)) is None + event = projector.normalize_session_update( + { + "params": { + "sessionId": "session-2", + "_meta": {"vendor.example/params": {"trace": "abc"}}, + "update": { + "sessionUpdate": "agent_message_chunk", + "content": { + "type": "text", + "text": "safe", + "_meta": { + "vendor.example/content": {"revision": 1}, + "unscoped": "discard", + }, + }, + "_meta": { + "vendor.example/update": {"opaque": True}, + "adapterInternal": "discard", + }, + }, + }, + } + ) + assert event is not None + assert event["sequence"] == 1 + assert event["payload"]["extensions"] == { + "params": {"vendor.example/params": {"trace": "abc"}}, + "update": { + "vendor.example/update": {"opaque": True}, + "adapterInternal": "discard", + }, + } + assert event["payload"]["content"]["_meta"] == { + "vendor.example/content": {"revision": 1}, + "unscoped": "discard", + } + assert "adapterInternal" in repr(event["payload"]) + assert "unscoped" in repr(event["payload"]) + + +def test_plan_is_a_validated_full_replacement() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "plan", + entries=[{"content": "old", "priority": "medium", "status": "pending"}], + ) + ) + replacement = projector.normalize_session_update( + _update( + "plan", + entries=[{"content": "new", "priority": "high", "status": "completed"}], + ) + ) + assert replacement is not None + assert replacement["payload"]["entries"] == [ + {"content": "new", "priority": "high", "status": "completed"} + ] + defaulted = projector.normalize_session_update(_update("plan", entries="bad")) + assert defaulted is not None and defaulted["payload"]["entries"] == [] + with pytest.raises(AcpProjectionError, match="entries must be an array"): + projector.normalize_session_update(_update("plan")) + assert projector.session_snapshot("session-1")["plan"] == [] + + +def test_completion_is_not_reopened_by_session_updates_and_requires_reset() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "final"}, + ) + ) + projector.mark_turn_complete("session-1") + projector.normalize_session_update(_update("usage_update", used=9, size=10)) + projector.normalize_session_update( + _update("session_info_update", title="still complete") + ) + assert projector.project_turn_content("session-1")["complete"] is True + assert projector.project_turn_content("session-1", complete=False)["complete"] is True + with pytest.raises(AcpProjectionError, match="reset_turn"): + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "late"}, + ) + ) + + projector.reset_turn("session-1") + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={"type": "text", "text": "next"}, + ) + ) + assert projector.project_turn_content("session-1")["assistant_stream_text"] == "next" + + +def test_bounded_state_fails_closed_and_drop_session_releases_capacity() -> None: + projector = AcpEventProjector( + max_sessions=1, + max_source_events_per_session=1, + max_messages_per_kind=1, + max_tool_calls_per_session=1, + max_state_fields=2, + max_plan_entries=1, + max_text_chars_per_message=3, + max_event_bytes=1024, + ) + assert projector.normalize_session_update( + _update("usage_update", used=1, size=10), source_event_id="one" + ) + with pytest.raises(AcpProjectionError, match="replay window"): + projector.normalize_session_update( + _update("usage_update", used=2, size=10), source_event_id="two" + ) + with pytest.raises(AcpProjectionError, match="session limit"): + projector.normalize_session_update( + { + "sessionId": "session-2", + "update": {"sessionUpdate": "usage_update", "used": 1, "size": 10}, + } + ) + assert projector.drop_session("session-1") is True + assert projector.drop_session("session-1") is False + assert projector.normalize_session_update( + { + "sessionId": "session-2", + "update": {"sessionUpdate": "usage_update", "used": 1, "size": 10}, + } + ) + + +def test_failed_or_oversized_input_does_not_allocate_or_mutate_session() -> None: + projector = AcpEventProjector(max_sessions=1, max_event_bytes=128) + with pytest.raises(AcpProjectionError, match="bounded JSON"): + projector.normalize_session_update( + _update("session_info_update", invalid={1, 2, 3}) + ) + assert projector.session_snapshot("session-1") is None + + with pytest.raises(AcpProjectionError, match="size limit"): + projector.normalize_session_update( + _update("session_info_update", value="x" * 200) + ) + assert projector.session_snapshot("session-1") is None + accepted = projector.normalize_session_update( + { + "sessionId": "session-2", + "update": {"sessionUpdate": "usage_update", "used": 1, "size": 10}, + } + ) + assert accepted is not None and accepted["sequence"] == 1 + + +def test_deep_json_is_rejected_as_a_projection_error_before_copying() -> None: + nested: object = "leaf" + for _ in range(130): + nested = {"next": nested} + projector = AcpEventProjector(max_json_depth=64) + + with pytest.raises(AcpProjectionError, match="nesting limit"): + projector.normalize_session_update( + _update( + "agent_message_chunk", + content={ + "type": "text", + "text": "safe", + "_meta": {"nested": nested}, + }, + ) + ) + assert projector.session_snapshot("session-1") is None + + +def test_replay_index_bytes_count_toward_retained_state_limit() -> None: + projector = AcpEventProjector( + max_session_state_bytes=100, + max_total_state_bytes=100, + ) + with pytest.raises(AcpProjectionError, match="retained session state"): + projector.normalize_session_update( + _update("session_info_update"), + source_event_id="x" * 50, + ) + assert projector.session_snapshot("session-1") is None + + +def test_aggregate_retained_session_state_has_a_hard_budget() -> None: + projector = AcpEventProjector(max_session_state_bytes=12) + with pytest.raises(AcpProjectionError, match="retained session state"): + projector.normalize_session_update( + _update("session_info_update", title="far too large") + ) + assert projector.session_snapshot("session-1") is None + + +def test_total_retained_state_is_bounded_across_isolated_sessions() -> None: + projector = AcpEventProjector( + max_session_state_bytes=100, + max_total_state_bytes=18, + ) + assert projector.normalize_session_update( + { + "sessionId": "session-a", + "update": {"sessionUpdate": "session_info_update", "title": "1234"}, + } + ) + with pytest.raises(AcpProjectionError, match="total retained state"): + projector.normalize_session_update( + { + "sessionId": "session-b", + "update": { + "sessionUpdate": "session_info_update", + "title": "1234", + }, + } + ) + assert projector.session_snapshot("session-b") is None + + +def test_all_non_message_events_remain_unreachable_from_legacy_turns() -> None: + projector = AcpEventProjector() + projector.normalize_session_update( + _update( + "agent_thought_chunk", + content={"type": "text", "text": "do not leak thought"}, + ) + ) + projector.normalize_session_update( + _update( + "tool_call", + toolCallId="tool-secret", + title="Private tool", + rawInput={"secret": "do not leak raw input"}, + ) + ) + projector.normalize_session_update( + _update( + "plan", + entries=[ + { + "content": "do not leak plan", + "priority": "low", + "status": "pending", + } + ], + ) + ) + projector.normalize_permission_request( + { + "jsonrpc": "2.0", + "id": "permission-secret", + "method": "session/request_permission", + "params": { + "sessionId": "session-1", + "toolCall": {"toolCallId": "tool-secret"}, + "options": [ + {"optionId": "secret", "name": "Private", "kind": "reject_once"} + ], + }, + } + ) + legacy = projector.project_turn_content("session-1") + assert legacy == { + "user_text": "", + "assistant_stream_text": "", + "assistant_final_text": "", + "complete": False, + "has_open_turn": False, + } diff --git a/tests/test_acp_protocol.py b/tests/test_acp_protocol.py new file mode 100644 index 0000000..c7caaa0 --- /dev/null +++ b/tests/test_acp_protocol.py @@ -0,0 +1,230 @@ +from __future__ import annotations + +import pytest + +from tendwire.backends.acp_protocol import ( + AgentCapabilities, + AcpEnvelopeError, + AcpFramingError, + AcpRemoteError, + JsonRpcNotification, + JsonRpcRequest, + JsonRpcResponse, + PermissionOptionKind, + SessionUpdateKind, + decode_json_line, + encode_message, + parse_permission_request, + parse_session_update, + request_envelope, + validate_envelope, +) + + +def test_strict_json_line_round_trip() -> None: + envelope = request_envelope(7, "session/new", {"cwd": "/tmp", "mcpServers": []}) + encoded = encode_message(envelope) + assert encoded.endswith(b"\n") + assert decode_json_line(encoded) == JsonRpcRequest( + 7, "session/new", {"cwd": "/tmp", "mcpServers": []} + ) + + +@pytest.mark.parametrize( + "line", + [ + b'{"jsonrpc":"2.0","method":"x"}', + b'{"jsonrpc":"2.0","method":"x"}\n{}\n', + b'{"jsonrpc":"2.0","method":"x","method":"y"}\n', + b'{"jsonrpc":"2.0","method":"x","params":[],"id":1}\n', + b'{"jsonrpc":"2.0","method":"x","params":{"n":NaN}}\n', + b"\xff\n", + ], +) +def test_rejects_invalid_or_ambiguous_frames(line: bytes) -> None: + with pytest.raises((AcpFramingError, AcpEnvelopeError)): + decode_json_line(line) + + +def test_rejects_oversized_inbound_and_outbound_frames() -> None: + with pytest.raises(AcpFramingError): + decode_json_line(b'{"jsonrpc":"2.0","method":"long"}\n', max_frame_bytes=10) + with pytest.raises(AcpFramingError): + encode_message( + {"jsonrpc": "2.0", "method": "long", "params": {}}, + max_frame_bytes=10, + ) + + +def test_response_error_is_typed() -> None: + response = decode_json_line( + b'{"jsonrpc":"2.0","id":"r1","error":{"code":-32000,"message":"nope","data":{"retry":false}}}\n' + ) + assert isinstance(response, JsonRpcResponse) + with pytest.raises(AcpRemoteError) as raised: + response.result_or_raise() + assert raised.value.request_id == "r1" + assert raised.value.code == -32000 + assert raised.value.data == {"retry": False} + + +def test_notification_and_session_update_are_typed_but_extension_safe() -> None: + message = decode_json_line( + b'{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"s1","update":{"sessionUpdate":"agent_thought_chunk","content":{"type":"text","text":"summary"}}}}\n' + ) + assert isinstance(message, JsonRpcNotification) + update = parse_session_update(message.params) + assert update.session_id == "s1" + assert update.update_kind is SessionUpdateKind.AGENT_THOUGHT_CHUNK + assert update.update["content"]["text"] == "summary" + + extension = parse_session_update( + {"sessionId": "s1", "update": {"sessionUpdate": "vendor_progress"}} + ) + assert extension.update_kind == "vendor_progress" + + +def test_permission_request_validation_and_typed_options() -> None: + request = JsonRpcRequest( + 42, + "session/request_permission", + { + "sessionId": "s1", + "toolCall": {"toolCallId": "tool-1", "status": "pending"}, + "options": [ + {"optionId": "yes", "name": "Allow", "kind": "allow_once"} + ], + }, + ) + parsed = parse_permission_request(request) + assert parsed.request_id == 42 + assert parsed.options[0].kind is PermissionOptionKind.ALLOW_ONCE + + +def test_capability_presence_uses_acp_object_semantics() -> None: + capabilities = AgentCapabilities.from_mapping( + { + "loadSession": True, + "sessionCapabilities": { + "list": {}, + "delete": {}, + "resume": {}, + "close": {}, + "additionalDirectories": {}, + }, + "promptCapabilities": { + "image": True, + "audio": True, + "embeddedContext": True, + }, + "mcpCapabilities": {"http": True, "sse": True}, + "auth": {"logout": {}}, + "_meta": {"vendor.example": {"revision": 3}}, + } + ) + assert capabilities.load_session + assert capabilities.session_list + assert capabilities.session_resume + assert capabilities.session_close + assert capabilities.session_delete + assert capabilities.additional_directories + assert capabilities.prompt_image + assert capabilities.prompt_audio + assert capabilities.prompt_embedded_context + assert capabilities.mcp_http + assert capabilities.mcp_sse + assert capabilities.auth_logout + assert capabilities.raw["_meta"] == {"vendor.example": {"revision": 3}} + + +def test_request_ids_follow_acp_json_rpc_domain() -> None: + assert isinstance( + validate_envelope({"jsonrpc": "2.0", "id": "", "method": "_vendor.example/x"}), + JsonRpcRequest, + ) + null_request = validate_envelope( + {"jsonrpc": "2.0", "id": None, "method": "_vendor.example/x"} + ) + assert isinstance(null_request, JsonRpcRequest) + assert null_request.request_id is None + assert isinstance( + validate_envelope( + {"jsonrpc": "2.0", "id": 2**63 - 1, "result": {}} + ), + JsonRpcResponse, + ) + for invalid_id in (True, 2**63, -(2**63) - 1, 1.5): + with pytest.raises(AcpEnvelopeError): + validate_envelope( + {"jsonrpc": "2.0", "id": invalid_id, "method": "_vendor.example/x"} + ) + + +def test_permission_option_ids_must_be_unambiguous() -> None: + request = JsonRpcRequest( + 42, + "session/request_permission", + { + "sessionId": "s1", + "toolCall": {"toolCallId": "tool-1"}, + "options": [ + {"optionId": "same", "name": "Allow", "kind": "allow_once"}, + {"optionId": "same", "name": "Reject", "kind": "reject_once"}, + ], + }, + ) + with pytest.raises(AcpEnvelopeError, match="unique"): + parse_permission_request(request) + + +def test_permission_request_matches_stable_v1_required_fields_and_enum() -> None: + empty_options = JsonRpcRequest( + 42, + "session/request_permission", + {"sessionId": "s1", "toolCall": {"toolCallId": "tool-1"}, "options": []}, + ) + assert parse_permission_request(empty_options).options == () + + missing_tool_call_id = JsonRpcRequest( + 43, + "session/request_permission", + { + "sessionId": "s1", + "toolCall": {}, + "options": [ + {"optionId": "allow", "name": "Allow", "kind": "allow_once"} + ], + }, + ) + with pytest.raises(AcpEnvelopeError, match="toolCallId"): + parse_permission_request(missing_tool_call_id) + + unknown_kind = JsonRpcRequest( + 44, + "session/request_permission", + { + "sessionId": "s1", + "toolCall": {"toolCallId": "tool-1"}, + "options": [ + {"optionId": "future", "name": "Future", "kind": "future_kind"} + ], + }, + ) + with pytest.raises(AcpEnvelopeError, match="ACP v1"): + parse_permission_request(unknown_kind) + + +def test_unknown_update_and_nested_extension_payload_are_preserved() -> None: + extension = parse_session_update( + { + "sessionId": "s1", + "update": { + "sessionUpdate": "vendor/future_progress", + "opaque": {"revision": 7, "items": [1, 2]}, + }, + "_meta": {"vendor.example/trace": "abc"}, + } + ) + assert extension.update_kind == "vendor/future_progress" + assert extension.update["opaque"] == {"revision": 7, "items": [1, 2]} + assert extension.meta == {"vendor.example/trace": "abc"} diff --git a/tests/test_acp_runtime.py b/tests/test_acp_runtime.py new file mode 100644 index 0000000..ea70c7c --- /dev/null +++ b/tests/test_acp_runtime.py @@ -0,0 +1,1831 @@ +from __future__ import annotations + +import queue +import sqlite3 +import sys +import threading +import time +from dataclasses import replace +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest + +from tendwire.backends.acp_client import ( + AcpRequestTimeoutError, + BoundedAcpConnection as AcpClient, +) +from tendwire.backends.acp_protocol import ( + PermissionOption, + PermissionOptionKind, + PermissionRequest, + PromptResult, + SessionResult, + SessionUpdate, + SessionUpdateKind, + StopReason, + SteeringOutcome, + SteeringResult, +) +from tendwire.backends.acp_runtime import ( + AcpWorkerSession as AcpRuntime, + AcpRuntimeBindingError, + AcpRuntimeProtocolError, + AcpRuntimeStateError, + AcpRuntimeStopTimeout, + RuntimeState, + SessionOpenMode, +) +from tendwire.config import Config +from tendwire.core.models import Snapshot, Worker, WorkerBinding, utc_timestamp +from tendwire.store.sqlite import ( + expire_stale_worker_bindings, + expire_worker_bindings, + list_agent_events, + list_worker_bindings, + save_snapshot, + upsert_worker_bindings, +) + + +_END = object() +FAKE_AGENT = Path(__file__).parent / "fixtures" / "acp_fake_agent.py" + + +class FakeClient: + def __init__(self) -> None: + self.events: queue.Queue[SessionUpdate | PermissionRequest | object] = queue.Queue() + # Existing tests enqueue through the typed names; both feed the one + # reader-ordered stream used by the runtime. + self.updates = self.events + self.permissions = self.events + self.calls: list[tuple[str, tuple[Any, ...], dict[str, Any]]] = [] + self.permission_responses: list[tuple[object, str | None, bool]] = [] + self.prompt_result: object = PromptResult(StopReason.END_TURN, {}) + self.prompt_failure: BaseException | None = None + self.initialize_failure: BaseException | None = None + self.new_session_result: SessionResult | None = None + self.restored_session_result: SessionResult | None = None + self.closed = False + self.close_calls = 0 + self.steering_supported = False + self.steering_result = SteeringResult( + SteeringOutcome.INJECTED, + {"outcome": "injected"}, + ) + + def initialize(self, **kwargs: Any) -> object: + self.calls.append(("initialize", (), kwargs)) + if self.initialize_failure is not None: + raise self.initialize_failure + return object() + + def new_session(self, cwd: Path, **kwargs: Any) -> SessionResult: + self.calls.append(("new", (cwd,), kwargs)) + return self.new_session_result or SessionResult( + "session-private", None, (), {} + ) + + def load_session( + self, session_id: str, cwd: Path, **kwargs: Any + ) -> SessionResult: + self.calls.append(("load", (session_id, cwd), kwargs)) + return self.restored_session_result or SessionResult( + session_id, None, (), {} + ) + + def resume_session( + self, session_id: str, cwd: Path, **kwargs: Any + ) -> SessionResult: + self.calls.append(("resume", (session_id, cwd), kwargs)) + return self.restored_session_result or SessionResult( + session_id, None, (), {} + ) + + def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: + self.calls.append(("prompt", (session_id, prompt), kwargs)) + if self.prompt_failure is not None: + raise self.prompt_failure + on_submitted = kwargs.get("on_submitted") + if callable(on_submitted): + on_submitted() + return self.prompt_result + + def prepare_prompt(self, prompt: object) -> tuple[dict[str, Any], ...]: + if isinstance(prompt, str): + return ({"type": "text", "text": prompt},) + return tuple(dict(block) for block in prompt) # type: ignore[arg-type] + + def steer_session(self, session_id: str, prompt: object, **kwargs: Any) -> object: + self.calls.append(("steer", (session_id, prompt), kwargs)) + on_send_start = kwargs.get("on_send_start") + if callable(on_send_start): + on_send_start() + on_submitted = kwargs.get("on_submitted") + if callable(on_submitted): + on_submitted() + return self.steering_result + + def cancel(self, session_id: str) -> None: + self.calls.append(("cancel", (session_id,), {})) + + def next_update(self, *, timeout: float) -> SessionUpdate: + try: + value = self.updates.get(timeout=timeout) + except queue.Empty as exc: + raise AcpRequestTimeoutError("idle") from exc + if value is _END: + raise EOFError("closed") + assert isinstance(value, SessionUpdate) + return value + + def next_permission_request(self, *, timeout: float) -> PermissionRequest: + try: + value = self.permissions.get(timeout=timeout) + except queue.Empty as exc: + raise AcpRequestTimeoutError("idle") from exc + if value is _END: + raise EOFError("closed") + assert isinstance(value, PermissionRequest) + return value + + def next_session_event( + self, + *, + timeout: float, + ) -> SessionUpdate | PermissionRequest: + try: + value = self.events.get(timeout=timeout) + except queue.Empty as exc: + raise AcpRequestTimeoutError("idle") from exc + if value is _END: + raise EOFError("closed") + assert isinstance(value, SessionUpdate | PermissionRequest) + return value + + def respond_permission( + self, + request_id: object, + *, + option_id: str | None = None, + cancelled: bool = False, + ) -> None: + self.permission_responses.append((request_id, option_id, cancelled)) + + def close(self) -> None: + self.close_calls += 1 + self.closed = True + self.events.put(_END) + + +class FakeIngestor: + def __init__(self, session_id: str = "session-private") -> None: + self.session_id = session_id + self.started: list[str | None] = [] + self.updates: list[object] = [] + self.permissions: list[tuple[object, str | None]] = [] + self.completions = 0 + self.completion_reasons: list[StopReason] = [] + self.load_resets = 0 + self.update_failure: BaseException | None = None + self.permission_failure: BaseException | None = None + self.completion_failure: BaseException | None = None + self.appended: list[tuple[object, str]] = [] + self.appendable = True + persisted = SimpleNamespace(status="inserted") + self.update_result: object = SimpleNamespace( + event=persisted, + turn=None, + ignored_reason=None, + ) + self.permission_result: object = SimpleNamespace( + event=persisted, + turn=None, + ignored_reason=None, + ) + self.completion_result: object = None + + def start_turn(self, *, producer_turn_id: str | None = None) -> str: + self.started.append(producer_turn_id) + return "opaque-turn" + + def ingest_update(self, raw: object, **_kwargs: Any) -> object: + if self.update_failure is not None: + raise self.update_failure + self.updates.append(raw) + return self.update_result + + def ingest_permission_request( + self, + raw: object, + *, + source_event_id: str | None = None, + **_kwargs: Any, + ) -> object: + if self.permission_failure is not None: + raise self.permission_failure + self.permissions.append((raw, source_event_id)) + return self.permission_result + + def begin_prompt( + self, + prompt: object, + *, + producer_turn_id: str | None = None, + ) -> object: + return self.start_turn(producer_turn_id=producer_turn_id) + + def can_append_prompt(self) -> bool: + return self.appendable + + def append_prompt(self, prompt: object, *, producer_turn_id: str) -> object: + self.appended.append((prompt, producer_turn_id)) + return self.update_result + + def reset_after_load(self) -> None: + self.load_resets += 1 + + def mark_prompt_complete( + self, + stop_reason: StopReason = StopReason.END_TURN, + ) -> object: + self.completions += 1 + self.completion_reasons.append(stop_reason) + if self.completion_failure is not None: + raise self.completion_failure + return self.completion_result + + +def binding(session_id: str = "session-private") -> WorkerBinding: + return WorkerBinding( + host_id="host-a", + worker_id="worker-public", + worker_fingerprint="worker-fingerprint", + backend="acp", + target_kind="pane_id", + target_value="pane-private-secret", + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="binding-private-secret", + ) + + +def continuity_binding() -> WorkerBinding: + return WorkerBinding( + host_id="host-a", + worker_id="worker-public", + worker_fingerprint="worker-fingerprint", + backend="herdr", + target_kind="pane_id", + target_value="pane-private-secret", + turn_target_kind="pane_id", + turn_target_value="pane-private-secret", + private_fingerprint="continuity-binding-private-secret", + ) + + +def binding_callback(db_path: Path): + def establish( + session_id: str, + continuity: WorkerBinding, + ) -> WorkerBinding: + bound = replace( + continuity, + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="", + ) + upsert_worker_bindings(db_path, [bound]) + return bound + + return establish + + +def runtime( + tmp_path: Path, + client: FakeClient, + ingestor: FakeIngestor | None = None, + **kwargs: Any, +) -> AcpRuntime: + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + return AcpRuntime( + client, # type: ignore[arg-type] + config=Config( + host_id="host-a", + db_path=db_path, + ), + binding=continuity, + cwd=tmp_path, + stream_generation="generation-private-secret", + session_binding_callback=binding_callback(db_path), + ingestor=ingestor or FakeIngestor(), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + **kwargs, + ) + + +def bound_runtime( + tmp_path: Path, + client: FakeClient, + current_binding: WorkerBinding, + **kwargs: Any, +) -> AcpRuntime: + db_path = tmp_path / "bound-events.db" + upsert_worker_bindings(db_path, [current_binding]) + return AcpRuntime( + client, # type: ignore[arg-type] + config=Config( + host_id="host-a", + db_path=db_path, + ), + binding=current_binding, + cwd=tmp_path, + session_mode=SessionOpenMode.LOAD, + session_id=current_binding.turn_target_value, + stream_generation="generation-private-secret", + poll_timeout=0.01, + stop_timeout=0.5, + **kwargs, + ) + + +def update(session_id: str = "session-private") -> SessionUpdate: + raw = { + "sessionId": session_id, + "update": { + "sessionUpdate": "agent_message_chunk", + "content": {"type": "text", "text": "answer"}, + }, + } + return SessionUpdate( + session_id, + SessionUpdateKind.AGENT_MESSAGE_CHUNK, + raw["update"], + None, + raw, + ) + + +def permission( + request_id: object = 7, session_id: str = "session-private" +) -> PermissionRequest: + options = ( + PermissionOption( + "allow-once", + "Allow once", + PermissionOptionKind.ALLOW_ONCE, + {"optionId": "allow-once", "name": "Allow once", "kind": "allow_once"}, + ), + PermissionOption( + "reject-once", + "Reject once", + PermissionOptionKind.REJECT_ONCE, + {"optionId": "reject-once", "name": "Reject once", "kind": "reject_once"}, + ), + ) + raw = { + "sessionId": session_id, + "toolCall": {"toolCallId": "tool-private"}, + "options": [dict(option.raw) for option in options], + } + return PermissionRequest( + request_id, + session_id, + raw["toolCall"], + options, + None, + raw, + ) + + +def wait_until(predicate, timeout: float = 1.0) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return + time.sleep(0.005) + raise AssertionError("condition did not become true") + + +def test_start_negotiates_opens_one_session_and_binds_factory(tmp_path: Path) -> None: + client = FakeClient() + captured: dict[str, object] = {} + ingestor = FakeIngestor() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def factory(config: Config, **kwargs: object) -> FakeIngestor: + captured.update(kwargs) + captured["config"] = config + return ingestor + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + stream_generation="generation-private-secret", + client_capabilities={"fs": {"readTextFile": True}}, + session_binding_callback=binding_callback(db_path), + ingestor_factory=factory, # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + try: + assert [call[0] for call in client.calls[:2]] == ["initialize", "new"] + assert client.calls[0][2]["client_capabilities"] == {} + assert captured["session_id"] == "session-private" + assert captured["binding"] is not None + assert captured["stream_generation"] == "generation-private-secret" + assert service.status().healthy + finally: + service.stop() + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + retired = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(retired) == 1 + assert retired[0].reason == "acp_runtime_stopped" + + +@pytest.mark.parametrize( + "claims", + [ + {"fs": {"readTextFile": True, "writeTextFile": True}}, + {"terminal": True}, + {"elicitation": {"form": {}, "url": {}}}, + {"session": {"configOptions": {"boolean": {}}}}, + {"_meta": {"example.test/capability": True}}, + ], +) +def test_runtime_strips_client_capabilities_it_cannot_serve( + tmp_path: Path, + claims: dict[str, object], +) -> None: + client = FakeClient() + service = runtime(tmp_path, client, client_capabilities=claims).start() + try: + assert client.calls[0][2]["client_capabilities"] == {} + finally: + service.stop() + + +def test_runtime_rejects_unknown_extension_capability_claims(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="unsupported client capabilities"): + runtime( + tmp_path, + FakeClient(), + client_capabilities={"example.test/custom": {}}, + ) + + +@pytest.mark.parametrize( + ("mode", "method"), + [(SessionOpenMode.LOAD, "load"), (SessionOpenMode.RESUME, "resume")], +) +def test_load_and_resume_use_requested_session( + tmp_path: Path, mode: SessionOpenMode, method: str +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + existing = binding("existing-private") + ingestor = FakeIngestor("existing-private") + upsert_worker_bindings(db_path, [existing]) + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config( + host_id="host-a", + db_path=db_path, + ), + binding=existing, + cwd=tmp_path, + session_mode=mode, + session_id="existing-private", + ingestor=ingestor, # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + try: + assert client.calls[1][0] == method + assert client.calls[1][1][0] == "existing-private" + assert ingestor.load_resets == (1 if mode is SessionOpenMode.LOAD else 0) + finally: + service.stop() + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + retired = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(retired) == 1 + assert retired[0].reason == "acp_runtime_stopped" + + +@pytest.mark.parametrize("mode", [SessionOpenMode.LOAD, SessionOpenMode.RESUME]) +def test_load_and_resume_reject_agent_session_mismatch_and_close( + tmp_path: Path, + mode: SessionOpenMode, +) -> None: + client = FakeClient() + client.restored_session_result = SessionResult( + "attacker-session-private", + None, + (), + {}, + ) + db_path = tmp_path / "events.db" + existing = binding("existing-private") + upsert_worker_bindings(db_path, [existing]) + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config( + host_id="host-a", + db_path=db_path, + ), + binding=existing, + cwd=tmp_path, + session_mode=mode, + session_id="existing-private", + poll_timeout=0.01, + stop_timeout=0.5, + ) + + with pytest.raises(AcpRuntimeProtocolError, match="bound session"): + service.start() + + assert client.closed + assert client.close_calls == 1 + assert service.status().state is RuntimeState.FAILED + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + retired = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(retired) == 1 + assert retired[0].reason == "acp_startup_rollback" + + +@pytest.mark.parametrize("mode", [SessionOpenMode.LOAD, SessionOpenMode.RESUME]) +def test_load_and_resume_reject_non_acp_binding_before_transport( + tmp_path: Path, + mode: SessionOpenMode, +) -> None: + client = FakeClient() + legacy = replace(binding("existing-private"), backend="herdr") + + with pytest.raises(ValueError, match="ACP backend binding"): + AcpRuntime( + client, # type: ignore[arg-type] + config=Config( + host_id="host-a", + db_path=tmp_path / "events.db", + ), + binding=legacy, + cwd=tmp_path, + session_mode=mode, + session_id="existing-private", + ) + + assert client.calls == [] + + +def test_new_accepts_unpredictable_agent_generated_session_id(tmp_path: Path) -> None: + client = FakeClient() + generated = "agent-generated-unpredictable-7f94" + client.new_session_result = SessionResult(generated, None, (), {}) + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + seen: list[tuple[str, WorkerBinding]] = [] + + def establish(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + seen.append((session_id, anchor)) + return binding_callback(db_path)(session_id, anchor) + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=establish, + ingestor=FakeIngestor(generated), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + try: + assert seen == [(generated, continuity)] + assert service.status().healthy + finally: + service.stop() + + +def test_new_acp_binding_survives_herdr_refresh_and_normal_stop( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=binding_callback(db_path), + ingestor=FakeIngestor(), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + + derived = list_worker_bindings(db_path, "host-a", backend="acp") + assert len(derived) == 1 + assert derived[0].turn_target_value == "session-private" + expire_stale_worker_bindings( + db_path, + "host-a", + backend="herdr", + current_private_fingerprints=[continuity.private_fingerprint], + ) + assert list_worker_bindings(db_path, "host-a", backend="acp") == derived + service.stop() + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + released = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(released) == 1 + assert released[0].reason == "acp_runtime_stopped" + + +def test_new_session_binding_accepts_concurrent_herdr_lease_refresh( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def bind_after_refresh(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + refreshed = replace( + anchor, + observed_at="2098-01-01T00:00:00+00:00", + expires_at="9999-12-31T23:59:59+00:00", + ) + upsert_worker_bindings(db_path, [refreshed]) + return binding_callback(db_path)(session_id, anchor) + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=bind_after_refresh, + ingestor=FakeIngestor(), # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + try: + assert service.status().healthy + current = list_worker_bindings(db_path, "host-a", backend="herdr") + assert current == [ + replace( + continuity, + observed_at="2098-01-01T00:00:00+00:00", + expires_at="9999-12-31T23:59:59+00:00", + ) + ] + finally: + service.stop() + + +@pytest.mark.parametrize("failure_mode", ("raise", "bad_return")) +def test_new_cleans_binding_persisted_by_failed_callback( + tmp_path: Path, + failure_mode: str, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def fail_after_persist(session_id: str, anchor: WorkerBinding): + bound = binding_callback(db_path)(session_id, anchor) + if failure_mode == "raise": + raise RuntimeError("after persist") + return object() + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=fail_after_persist, # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ) + with pytest.raises((RuntimeError, AcpRuntimeBindingError)): + service.start() + + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + retired = list_worker_bindings( + db_path, + "host-a", + backend="acp", + include_expired=True, + ) + assert len(retired) == 1 + assert retired[0].reason == "acp_startup_rollback" + assert list_worker_bindings(db_path, "host-a", backend="herdr") == [continuity] + + +def test_new_rolls_back_persisted_binding_when_ingestor_startup_fails( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def fail_factory(*_args: object, **_kwargs: object) -> FakeIngestor: + raise RuntimeError("factory failed") + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=binding_callback(db_path), + ingestor_factory=fail_factory, # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ) + with pytest.raises(RuntimeError, match="factory failed"): + service.start() + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + + +def test_new_callback_can_stop_runtime_without_lifecycle_deadlock( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + service: AcpRuntime + + def stop_during_bind(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + bound = binding_callback(db_path)(session_id, anchor) + service.stop(timeout=0.2) + return bound + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=stop_during_bind, + poll_timeout=0.01, + stop_timeout=0.5, + ) + started = time.monotonic() + with pytest.raises(AcpRuntimeStateError, match="stopped during"): + service.start() + assert time.monotonic() - started < 0.5 + assert list_worker_bindings(db_path, "host-a", backend="acp") == [] + + +def test_prompt_rechecks_binding_before_remote_send(tmp_path: Path) -> None: + client = FakeClient() + ingestor = FakeIngestor() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=binding_callback(db_path), + ingestor=ingestor, # type: ignore[arg-type] + poll_timeout=0.01, + stop_timeout=0.5, + ).start() + derived = list_worker_bindings(db_path, "host-a", backend="acp")[0] + expire_worker_bindings( + db_path, + derived.host_id, + backend="acp", + private_fingerprints=[derived.private_fingerprint], + reason="test_expiry", + ) + + with pytest.raises(AcpRuntimeBindingError): + service.prompt("must not send", producer_turn_id="producer-private") + assert [call[0] for call in client.calls].count("prompt") == 0 + assert ingestor.started == [] + assert ingestor.completions == 0 + + +def test_new_requires_explicit_session_binder_before_launch(tmp_path: Path) -> None: + client = FakeClient() + + with pytest.raises(ValueError, match="session_binding_callback is required"): + AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=tmp_path / "events.db"), + binding=continuity_binding(), + cwd=tmp_path, + ) + + assert client.calls == [] + assert client.close_calls == 0 + + +@pytest.mark.parametrize("mismatch", ["session", "worker"]) +def test_new_rejects_malicious_binder_return_and_closes_adapter( + tmp_path: Path, + mismatch: str, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def malicious(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + return replace( + anchor, + worker_id=( + "attacker-worker-private" + if mismatch == "worker" + else anchor.worker_id + ), + turn_target_kind="acp_session_id", + turn_target_value=( + "attacker-session-private" + if mismatch == "session" + else session_id + ), + private_fingerprint="", + ) + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=malicious, + poll_timeout=0.01, + stop_timeout=0.5, + ) + + with pytest.raises(AcpRuntimeBindingError): + service.start() + + assert client.closed + assert client.close_calls == 1 + assert service.status().state is RuntimeState.FAILED + assert service.status().failure_type == "AcpRuntimeBindingError" + assert list_worker_bindings(db_path, "host-a", backend="herdr") == [ + continuity + ] + + +def test_new_binder_exception_closes_adapter_and_fails_terminally( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + failure = RuntimeError("binder-private-failure") + + def fail(_session_id: str, _anchor: WorkerBinding) -> WorkerBinding: + raise failure + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=fail, + poll_timeout=0.01, + stop_timeout=0.5, + ) + + with pytest.raises(RuntimeError) as raised: + service.start() + + assert raised.value is failure + assert client.closed + assert client.close_calls == 1 + status = service.status() + assert status.state is RuntimeState.FAILED + assert status.failure_type == "RuntimeError" + assert "binder-private-failure" not in repr(status) + + +def test_new_rejects_valid_shaped_binding_that_was_not_persisted( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def dishonest(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + return replace( + anchor, + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="", + ) + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=dishonest, + poll_timeout=0.01, + stop_timeout=0.5, + ) + + with pytest.raises(AcpRuntimeBindingError, match="not current"): + service.start() + + assert client.closed + assert client.close_calls == 1 + assert service.status().state is RuntimeState.FAILED + + +def test_new_rejects_binder_that_overwrites_herdr_continuity( + tmp_path: Path, +) -> None: + client = FakeClient() + db_path = tmp_path / "events.db" + continuity = continuity_binding() + upsert_worker_bindings(db_path, [continuity]) + + def destructive(session_id: str, anchor: WorkerBinding) -> WorkerBinding: + upsert_worker_bindings( + db_path, + [ + replace( + anchor, + worker_id="replacement-worker-private", + worker_fingerprint="replacement-fingerprint-private", + observed_at=utc_timestamp(), + ) + ], + ) + bound = replace( + anchor, + backend="acp", + turn_target_kind="acp_session_id", + turn_target_value=session_id, + private_fingerprint="", + ) + upsert_worker_bindings(db_path, [bound]) + return bound + + service = AcpRuntime( + client, # type: ignore[arg-type] + config=Config(host_id="host-a", db_path=db_path), + binding=continuity, + cwd=tmp_path, + session_binding_callback=destructive, + poll_timeout=0.01, + stop_timeout=0.5, + ) + + with pytest.raises(AcpRuntimeBindingError, match="not current"): + service.start() + + assert client.closed + assert client.close_calls == 1 + assert service.status().state is RuntimeState.FAILED + + +def test_initialize_failure_is_cleaned_up_without_caller_stop(tmp_path: Path) -> None: + client = FakeClient() + failure = OSError("initialize failed") + client.initialize_failure = failure + service = runtime(tmp_path, client) + + with pytest.raises(OSError) as raised: + service.start() + + assert raised.value is failure + assert client.closed + assert client.close_calls == 1 + assert service.status().failure_type == "OSError" + + +def test_background_consumers_ingest_losslessly_and_permissions_fail_closed( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + client.updates.put(update()) + client.permissions.put(permission()) + wait_until(lambda: service.status().permissions_ingested == 1) + wait_until(lambda: service.status().updates_ingested == 1) + + assert len(ingestor.updates) == 1 + permission_event_id = ingestor.permissions[0][1] + assert permission_event_id is not None + assert permission_event_id.startswith("permission:") + assert permission_event_id != "permission:7" + assert client.permission_responses == [(7, None, True)] + assert service.status().permissions_cancelled == 1 + finally: + service.stop() + + +def test_callback_can_select_only_an_offered_permission(tmp_path: Path) -> None: + client = FakeClient() + decisions = iter(["not-offered", "allow-once"]) + service = runtime( + tmp_path, + client, + permission_callback=lambda _request: next(decisions), + ).start() + try: + client.permissions.put(permission(1)) + client.permissions.put(permission(2)) + wait_until(lambda: service.status().permissions_ingested == 2) + wait_until(lambda: len(client.permission_responses) == 2) + + assert client.permission_responses == [ + (1, None, True), + (2, "allow-once", False), + ] + status = service.status() + assert status.invalid_permission_selections == 1 + assert status.permissions_cancelled == 1 + assert status.permissions_selected == 1 + finally: + service.stop() + + +def test_callback_failure_cancels_permission_before_propagating(tmp_path: Path) -> None: + client = FakeClient() + callback_failure = LookupError("decision failed") + + def fail(_request: PermissionRequest) -> str: + raise callback_failure + + service = runtime(tmp_path, client, permission_callback=fail).start() + client.permissions.put(permission()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + assert client.permission_responses == [(7, None, True)] + assert service.status().permissions_cancelled == 1 + with pytest.raises(LookupError) as raised: + service.stop() + assert raised.value is callback_failure + + +def test_permission_ingestion_failure_cancels_before_runtime_fails( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + failure = OSError("journal unavailable") + ingestor.permission_failure = failure + service = runtime(tmp_path, client, ingestor).start() + client.permissions.put(permission()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + assert client.permission_responses == [(7, None, True)] + assert service.status().permissions_cancelled == 1 + assert service.status().permissions_ingested == 0 + with pytest.raises(OSError) as raised: + service.stop() + assert raised.value is failure + + +def test_replaced_binding_cancels_permission_before_callback_and_is_terminal( + tmp_path: Path, +) -> None: + client = FakeClient() + current = binding() + callback_called = threading.Event() + + def unsafe_allow(_request: PermissionRequest) -> str: + callback_called.set() + return "allow-once" + + service = bound_runtime( + tmp_path, + client, + current, + permission_callback=unsafe_allow, + ).start() + replacement = replace( + current, + worker_id="replacement-worker-private", + worker_fingerprint="replacement-fingerprint-private", + observed_at=utc_timestamp(), + ) + upsert_worker_bindings(tmp_path / "bound-events.db", [replacement]) + client.permissions.put(permission()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + status = service.status() + assert callback_called.is_set() is False + assert client.permission_responses == [(7, None, True)] + assert status.permissions_ingested == 0 + assert status.permissions_selected == 0 + assert status.permissions_cancelled == 1 + assert status.failure_type == "AcpRuntimeBindingError" + assert list_agent_events(tmp_path / "bound-events.db", "host-a") == () + rendered = repr(status) + assert "replacement-worker-private" not in rendered + assert "binding-private-secret" not in rendered + with pytest.raises(AcpRuntimeBindingError): + service.stop() + + +def test_binding_expiry_after_start_rejects_update_and_is_terminal( + tmp_path: Path, +) -> None: + client = FakeClient() + current = binding() + service = bound_runtime(tmp_path, client, current).start() + with sqlite3.connect(tmp_path / "bound-events.db") as conn: + conn.execute( + "UPDATE worker_bindings SET expires_at = ? " + "WHERE host_id = ? AND private_fingerprint = ?", + ( + "2000-01-01T00:00:00+00:00", + current.host_id, + current.private_fingerprint, + ), + ) + client.updates.put(update()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + status = service.status() + assert status.updates_ingested == 0 + assert status.failure_type == "AcpRuntimeBindingError" + assert list_agent_events(tmp_path / "bound-events.db", "host-a") == () + rendered = repr(status) + assert "session-private" not in rendered + assert "binding-private-secret" not in rendered + with pytest.raises(AcpRuntimeBindingError): + service.stop() + + +def test_permission_source_identity_distinguishes_jsonrpc_id_types( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + client.permissions.put(permission(1)) + client.permissions.put(permission("1")) + wait_until(lambda: service.status().permissions_ingested == 2) + + source_ids = [source_id for _raw, source_id in ingestor.permissions] + assert len(set(source_ids)) == 2 + assert all( + source_id is not None and source_id.startswith("permission:") + for source_id in source_ids + ) + finally: + service.stop() + + +def test_prompt_finalizes_only_after_valid_response_and_update_drain( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + client.updates.put(update()) + result = service.prompt("question", producer_turn_id="producer-private") + + assert result.stop_reason is StopReason.END_TURN + assert ingestor.started == ["producer-private"] + assert len(ingestor.updates) == 1 + assert ingestor.completions == 1 + status = service.status() + assert status.prompts_started == 1 + assert status.prompts_completed == 1 + assert status.prompts_failed == 0 + finally: + service.stop() + + +def test_prompt_requires_crash_stable_producer_identity_before_remote_send( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + with pytest.raises(ValueError, match="producer_turn_id"): + service.prompt("question") + with pytest.raises(ValueError, match="producer_turn_id"): + service.prompt("question", producer_turn_id=" ") + + assert [call[0] for call in client.calls].count("prompt") == 0 + assert ingestor.started == [] + assert service.status().prompts_started == 0 + assert service.status().state is RuntimeState.RUNNING + finally: + service.stop() + + +def test_submit_prompt_acknowledges_frame_before_end_of_turn(tmp_path: Path) -> None: + class BlockingPromptClient(FakeClient): + def __init__(self) -> None: + super().__init__() + self.release = threading.Event() + + def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: + self.calls.append(("prompt", (session_id, prompt), kwargs)) + on_submitted = kwargs.get("on_submitted") + assert callable(on_submitted) + on_submitted() + assert self.release.wait(1.0) + return self.prompt_result + + client = BlockingPromptClient() + service = runtime(tmp_path, client).start() + started = time.monotonic() + service.submit_prompt( + "long turn", + producer_turn_id="producer-turn-ack", + acknowledgement_timeout=0.25, + ) + assert time.monotonic() - started < 0.2 + assert service.status().prompts_completed == 0 + client.release.set() + wait_until(lambda: service.status().prompts_completed == 1) + service.stop() + + +def test_submit_steering_records_input_at_transport_boundary(tmp_path: Path) -> None: + client = FakeClient() + client.steering_supported = True + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + callbacks: list[str] = [] + result = service.submit_steering( + "live follow-up", + producer_turn_id="producer-steer", + acknowledgement_timeout=0.25, + on_send_start=lambda: callbacks.append("started"), + ) + assert result.outcome is SteeringOutcome.INJECTED + assert callbacks == ["started"] + assert len(ingestor.appended) == 1 + assert ingestor.appended[0][1] == "producer-steer" + assert [call[0] for call in client.calls].count("prompt") == 0 + assert [call[0] for call in client.calls].count("steer") == 1 + finally: + service.stop() + + +def test_submit_steering_retries_definite_non_application_once(tmp_path: Path) -> None: + class FailOnceSteeringClient(FakeClient): + def __init__(self) -> None: + super().__init__() + self.steering_supported = True + self.steering_attempts = 0 + + def steer_session( + self, session_id: str, prompt: object, **kwargs: Any + ) -> SteeringResult: + self.calls.append(("steer", (session_id, prompt), kwargs)) + self.steering_attempts += 1 + on_send_start = kwargs.get("on_send_start") + if callable(on_send_start): + on_send_start() + outcome = ( + SteeringOutcome.FAILED + if self.steering_attempts == 1 + else SteeringOutcome.INJECTED + ) + return SteeringResult(outcome, {"outcome": outcome.value}) + + client = FailOnceSteeringClient() + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + callbacks: list[str] = [] + result = service.submit_steering( + "live retry follow-up", + producer_turn_id="producer-steer-retry", + acknowledgement_timeout=0.25, + on_send_start=lambda: callbacks.append("started"), + ) + assert result.outcome is SteeringOutcome.INJECTED + assert callbacks == ["started"] + assert len(ingestor.appended) == 1 + assert ingestor.appended[0][1] == "producer-steer-retry" + assert [call[0] for call in client.calls].count("steer") == 2 + assert client.calls[-1][2].get("on_send_start") is None + assert 0 < client.calls[-1][2]["timeout"] <= 0.25 + finally: + service.stop() + + +def test_submit_steering_returns_second_definite_failure_without_third_attempt( + tmp_path: Path, +) -> None: + client = FakeClient() + client.steering_supported = True + client.steering_result = SteeringResult( + SteeringOutcome.FAILED, + {"outcome": SteeringOutcome.FAILED.value}, + ) + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + callbacks: list[str] = [] + result = service.submit_steering( + "live failed follow-up", + producer_turn_id="producer-steer-failed", + acknowledgement_timeout=0.25, + on_send_start=lambda: callbacks.append("started"), + ) + assert result.outcome is SteeringOutcome.FAILED + assert callbacks == ["started"] + assert len(ingestor.appended) == 1 + assert [call[0] for call in client.calls].count("steer") == 2 + finally: + service.stop() + + +def test_ignored_update_does_not_increment_persisted_counter(tmp_path: Path) -> None: + client = FakeClient() + ingestor = FakeIngestor() + ingestor.update_result = SimpleNamespace( + event=None, + turn=None, + ignored_reason="prompt_echo", + ) + service = runtime(tmp_path, client, ingestor).start() + try: + client.updates.put(update()) + wait_until(lambda: len(ingestor.updates) == 1) + assert service.status().updates_ingested == 0 + finally: + service.stop() + + +def test_stale_binding_completion_is_terminal_and_not_counted_complete( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + ingestor.completion_result = SimpleNamespace( + ignored_reason="stale_binding", + event=None, + turn=None, + ) + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRuntimeBindingError): + service.prompt("question", producer_turn_id="producer-private") + + status = service.status() + assert status.state is RuntimeState.FAILED + assert status.failure_type == "AcpRuntimeBindingError" + assert status.prompts_started == 1 + assert status.prompts_completed == 0 + assert status.prompts_failed == 1 + with pytest.raises(AcpRuntimeBindingError): + service.stop() + + +def test_prompt_finality_waits_for_permission_resolution(tmp_path: Path) -> None: + client = FakeClient() + ingestor = FakeIngestor() + callback_entered = threading.Event() + release_callback = threading.Event() + + def decide(_request: PermissionRequest) -> str: + callback_entered.set() + assert release_callback.wait(timeout=1) + return "allow-once" + + service = runtime( + tmp_path, + client, + ingestor, + permission_callback=decide, + ).start() + result: list[PromptResult] = [] + failure: list[BaseException] = [] + + def run_prompt() -> None: + try: + result.append( + service.prompt( + "question", + producer_turn_id="producer-private", + drain_timeout=0.5, + ) + ) + except BaseException as exc: + failure.append(exc) + + try: + client.permissions.put(permission()) + assert callback_entered.wait(timeout=1) + prompt_thread = threading.Thread(target=run_prompt) + prompt_thread.start() + time.sleep(0.04) + assert prompt_thread.is_alive() + assert ingestor.completions == 0 + + release_callback.set() + prompt_thread.join(timeout=1) + assert not prompt_thread.is_alive() + assert failure == [] + assert len(result) == 1 + assert ingestor.completions == 1 + assert client.permission_responses == [(7, "allow-once", False)] + finally: + release_callback.set() + service.stop() + + +def test_cross_kind_ingestion_cannot_overtake_an_active_update( + tmp_path: Path, +) -> None: + client = FakeClient() + update_entered = threading.Event() + release_update = threading.Event() + order: list[str] = [] + + class OrderedIngestor(FakeIngestor): + def ingest_update(self, raw: object, **kwargs: Any) -> object: + order.append("update-start") + update_entered.set() + assert release_update.wait(timeout=1) + result = super().ingest_update(raw, **kwargs) + order.append("update-end") + return result + + def ingest_permission_request( + self, + raw: object, + *, + source_event_id: str | None = None, + **kwargs: Any, + ) -> object: + order.append("permission") + return super().ingest_permission_request( + raw, + source_event_id=source_event_id, + **kwargs, + ) + + service = runtime(tmp_path, client, OrderedIngestor()).start() + try: + client.updates.put(update()) + assert update_entered.wait(timeout=1) + client.permissions.put(permission()) + time.sleep(0.04) + assert order == ["update-start"] + + release_update.set() + wait_until(lambda: service.status().permissions_ingested == 1) + assert order == ["update-start", "update-end", "permission"] + finally: + release_update.set() + service.stop() + + +def test_load_drains_replay_larger_than_client_queue_before_response( + tmp_path: Path, +) -> None: + db_path = tmp_path / "load.db" + current = binding("s-load") + upsert_worker_bindings(db_path, [current]) + client = AcpClient( + [sys.executable, "-u", str(FAKE_AGENT), "load_replay"], + max_pending_events=4, + ) + service = AcpRuntime( + client, + config=Config( + host_id="host-a", + db_path=db_path, + ), + binding=current, + cwd=tmp_path, + session_mode=SessionOpenMode.LOAD, + session_id="s-load", + poll_timeout=0.005, + stop_timeout=2, + ).start() + try: + assert service.status().updates_ingested == 64 + assert len(list_agent_events(db_path, "host-a")) == 64 + with sqlite3.connect(db_path) as conn: + assert conn.execute("SELECT COUNT(*) FROM connector_outbox").fetchone()[0] == 0 + finally: + service.stop(timeout=2) + + +@pytest.mark.parametrize("stop_reason", tuple(StopReason)) +def test_runtime_carries_every_prompt_stop_reason_to_completion( + tmp_path: Path, + stop_reason: StopReason, +) -> None: + client = FakeClient() + client.prompt_result = PromptResult(stop_reason, {"stopReason": stop_reason.value}) + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + try: + result = service.prompt("no echo", producer_turn_id="turn-a") + assert result.stop_reason is stop_reason + assert ingestor.started == ["turn-a"] + assert ingestor.completion_reasons == [stop_reason] + finally: + service.stop() + + +def test_prompt_transport_failure_cancels_and_makes_runtime_terminal( + tmp_path: Path, +) -> None: + client = FakeClient() + failure = AcpRequestTimeoutError("prompt timed out") + client.prompt_failure = failure + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRequestTimeoutError) as raised: + service.prompt("question", producer_turn_id="producer-private") + + assert raised.value is failure + assert ingestor.started == ["producer-private"] + assert ingestor.completions == 1 + assert ingestor.completion_reasons == [StopReason.CANCELLED] + assert ("cancel", ("session-private",), {}) in client.calls + assert service.status().cancellation_requests == 1 + assert service.status().state is RuntimeState.FAILED + with pytest.raises(AcpRequestTimeoutError): + service.prompt("unsafe retry", producer_turn_id="producer-private-2") + with pytest.raises(AcpRequestTimeoutError): + service.stop() + + +def test_prompt_failure_preserves_original_when_cancel_finalization_fails( + tmp_path: Path, +) -> None: + client = FakeClient() + original = AcpRequestTimeoutError("prompt timed out") + client.prompt_failure = original + ingestor = FakeIngestor() + ingestor.completion_failure = OSError("completion unavailable") + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRequestTimeoutError) as raised: + service.prompt("question", producer_turn_id="producer-private") + + assert raised.value is original + assert ingestor.completions == 1 + assert ingestor.completion_reasons == [StopReason.CANCELLED] + assert service.status().failure_type == "AcpRequestTimeoutError" + + +def test_prompt_failure_drains_queued_updates_before_cancelled_completion( + tmp_path: Path, +) -> None: + actions: list[str] = [] + + class UpdatingFailureClient(FakeClient): + def prompt(self, session_id: str, prompt: object, **kwargs: Any) -> object: + self.calls.append(("prompt", (session_id, prompt), kwargs)) + self.events.put(update()) + raise AcpRequestTimeoutError("prompt timed out") + + class OrderedIngestor(FakeIngestor): + def ingest_update(self, raw: object, **kwargs: Any) -> object: + outcome = super().ingest_update(raw, **kwargs) + actions.append("update") + return outcome + + def mark_prompt_complete( + self, + stop_reason: StopReason = StopReason.END_TURN, + ) -> object: + actions.append("complete") + return super().mark_prompt_complete(stop_reason) + + client = UpdatingFailureClient() + ingestor = OrderedIngestor() + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRequestTimeoutError): + service.prompt("question", producer_turn_id="producer-private") + + assert actions == ["update", "complete"] + assert ingestor.completion_reasons == [StopReason.CANCELLED] + + +def test_prompt_transport_failure_materializes_one_cancelled_final_projection( + tmp_path: Path, +) -> None: + client = FakeClient() + failure = AcpRequestTimeoutError("prompt timed out") + client.prompt_failure = failure + current = binding() + service = bound_runtime(tmp_path, client, current) + save_snapshot( + tmp_path / "bound-events.db", + Snapshot( + host_id="host-a", + updated_at=utc_timestamp(), + workers=[ + Worker( + id=current.worker_id, + name="worker-public", + status="active", + fingerprint=current.worker_fingerprint, + ) + ], + ), + ) + service.start() + + with pytest.raises(AcpRequestTimeoutError) as raised: + service.prompt("question", producer_turn_id="producer-private") + + assert raised.value is failure + with sqlite3.connect(tmp_path / "bound-events.db") as conn: + turns = conn.execute( + "SELECT payload_json FROM turns WHERE host_id = ?", + ("host-a",), + ).fetchall() + revisions = conn.execute( + "SELECT final_state, assistant_final_text " + "FROM turn_content_revisions WHERE host_id = ? AND is_current = 1", + ("host-a",), + ).fetchall() + assert len(turns) == 1 + assert revisions == [("complete", "[ACP prompt cancelled]")] + + +def test_invalid_prompt_response_cancels_and_finalizes_open_turn( + tmp_path: Path, +) -> None: + client = FakeClient() + client.prompt_result = {"stopReason": "end_turn"} + ingestor = FakeIngestor() + service = runtime(tmp_path, client, ingestor).start() + + with pytest.raises(AcpRuntimeProtocolError, match="invalid response"): + service.prompt("question", producer_turn_id="producer-private") + assert ("cancel", ("session-private",), {}) in client.calls + assert ingestor.completions == 1 + assert ingestor.completion_reasons == [StopReason.CANCELLED] + assert service.status().state is RuntimeState.FAILED + with pytest.raises(AcpRuntimeProtocolError): + service.stop() + + +def test_background_ingestion_failure_is_propagated_and_status_is_redacted( + tmp_path: Path, +) -> None: + client = FakeClient() + ingestor = FakeIngestor() + failure = OSError("session-private pane-private-secret") + ingestor.update_failure = failure + service = runtime(tmp_path, client, ingestor).start() + client.updates.put(update()) + wait_until(lambda: service.status().state is RuntimeState.FAILED) + + status = service.status() + assert not status.healthy + assert status.failure_type == "OSError" + rendered = repr(status) + assert "session-private" not in rendered + assert "pane-private-secret" not in rendered + assert "generation-private-secret" not in rendered + with pytest.raises(OSError) as raised: + service.raise_if_failed() + assert raised.value is failure + with pytest.raises(OSError): + service.stop() + + +def test_cancel_targets_bound_session_and_stop_joins_consumers(tmp_path: Path) -> None: + client = FakeClient() + service = runtime(tmp_path, client).start() + service.cancel() + assert ("cancel", ("session-private",), {}) in client.calls + assert service.status().cancellation_requests == 1 + + service.stop() + assert client.closed + assert service.join(timeout=0.1) + assert service.status().state is RuntimeState.STOPPED + + +def test_concurrent_stop_closes_adapter_exactly_once(tmp_path: Path) -> None: + client = FakeClient() + service = runtime(tmp_path, client).start() + failures: list[BaseException] = [] + + def stop() -> None: + try: + service.stop() + except BaseException as exc: + failures.append(exc) + + callers = [threading.Thread(target=stop) for _ in range(2)] + for caller in callers: + caller.start() + for caller in callers: + caller.join(timeout=1) + + assert failures == [] + assert all(not caller.is_alive() for caller in callers) + assert client.close_calls == 1 + assert service.status().state is RuntimeState.STOPPED + + +def test_stop_deadline_is_bounded_even_when_client_close_hangs(tmp_path: Path) -> None: + client = FakeClient() + release_close = threading.Event() + + def hanging_close() -> None: + release_close.wait(timeout=1) + + client.close = hanging_close # type: ignore[method-assign] + service = runtime(tmp_path, client).start() + started = time.monotonic() + try: + with pytest.raises(AcpRuntimeStopTimeout): + service.stop(timeout=0.05) + assert time.monotonic() - started < 0.25 + finally: + release_close.set() + + +def test_concurrent_stop_wait_for_lifecycle_is_also_deadline_bounded( + tmp_path: Path, +) -> None: + client = FakeClient() + close_entered = threading.Event() + release_close = threading.Event() + original_close = client.close + + def hanging_close() -> None: + close_entered.set() + release_close.wait(timeout=1) + original_close() + + client.close = hanging_close # type: ignore[method-assign] + service = runtime(tmp_path, client).start() + first_failures: list[BaseException] = [] + + def first_stop() -> None: + try: + service.stop(timeout=0.2) + except BaseException as exc: + first_failures.append(exc) + + caller = threading.Thread(target=first_stop) + caller.start() + assert close_entered.wait(timeout=1) + started = time.monotonic() + try: + with pytest.raises(AcpRuntimeStopTimeout, match="lifecycle"): + service.stop(timeout=0.03) + assert time.monotonic() - started < 0.15 + finally: + release_close.set() + caller.join(timeout=1) + + assert len(first_failures) == 1 + assert isinstance(first_failures[0], AcpRuntimeStopTimeout) + assert client.close_calls == 1 diff --git a/tests/test_actions.py b/tests/test_actions.py index c571acb..f95f4a9 100644 --- a/tests/test_actions.py +++ b/tests/test_actions.py @@ -5,15 +5,9 @@ import json from typing import Any -import pytest - from tendwire.config import Config from tendwire.core.actions import CommandContext, execute_command from tendwire.core.commands import ( - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DRY_RUN, - STATUS_INVALID_REQUEST, STATUS_NOT_FOUND, STATUS_REJECTED, STATUS_RESOLVED, @@ -44,52 +38,6 @@ def _workers(snapshot: Snapshot) -> list[Worker]: return list(snapshot.workers) -def _sendable_worker( - worker_id: str, - name: str, - *, - status: str = "active", - space_id: str | None = "s-1", - target_value: str | None = None, - sendable: bool = True, - reason: str | None = None, -) -> Worker: - return Worker( - id=worker_id, - name=name, - status=status, - space_id=space_id, - backend_target={ - "kind": "agent_id", - "value": target_value or f"agent-{worker_id}", - "sendable": sendable, - "reason": reason, - }, - ) - - -def _workers_with_backend_targets(snapshot: Snapshot) -> list[Worker]: - return [ - Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - fingerprint=worker.fingerprint, - backend_target={ - "kind": "agent_id", - "value": f"agent-{worker.id}", - "sendable": True, - "reason": None, - }, - ) - for worker in snapshot.workers - ] - - def test_noop_action_succeeds() -> None: request = CommandRequest(action="noop") context = CommandContext(host_id="host", workers=[]) @@ -170,224 +118,6 @@ def test_resolve_target_disallowed_status() -> None: assert envelope.status == STATUS_REJECTED -def test_send_instruction_dry_run_is_pure() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - dry_run=True, - ) - context = CommandContext( - host_id=snapshot.host_id, - workers=_workers_with_backend_targets(snapshot), - ) - envelope = execute_command(request, context) - assert envelope.ok is True - assert envelope.status == STATUS_DRY_RUN - assert envelope.result == { - "target": { - "worker_id": "w-1", - "name": "Alpha", - "space_id": "s-1", - "status": "active", - "worker_fingerprint": snapshot.workers[0].fingerprint, - }, - "instruction": {"text": "hello"}, - } - - -def test_send_instruction_non_dry_run_requires_request_id() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - envelope = execute_command(request, context) - assert envelope.ok is False - assert envelope.status == STATUS_INVALID_REQUEST - - -def test_send_instruction_non_dry_run_returns_backend_unsupported() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-1", - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - envelope = execute_command(request, context) - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - assert envelope.request_id == "req-1" - assert envelope.dry_run is False - - -def test_send_instruction_without_sendable_backend_target_is_unsupported() -> None: - request = CommandRequest( - action="send_instruction", - request_id="req-no-binding", - dry_run=False, - target={"worker_id": "w-no-binding"}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id="host", - workers=[ - _sendable_worker( - "w-no-binding", - "No Binding", - sendable=False, - reason="backend_unsupported", - ) - ], - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - - -def test_send_instruction_ambiguous_backend_target_is_rejected() -> None: - request = CommandRequest( - action="send_instruction", - request_id="req-ambiguous-binding", - dry_run=False, - target={"worker_id": "w-ambiguous"}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id="host", - workers=[ - _sendable_worker( - "w-ambiguous", - "Ambiguous", - sendable=False, - reason="duplicate_backend_target", - ) - ], - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_AMBIGUOUS_BACKEND_TARGET - - -def test_send_instruction_rejects_empty_target_before_resolution() -> None: - request = CommandRequest( - action="send_instruction", - request_id="req-empty", - dry_run=False, - target={}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id="host", - workers=[Worker(id="only-worker", name="Only", status="active")], - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_INVALID_REQUEST - - -def test_send_instruction_done_worker_still_requires_authoritative_submission() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-done", - dry_run=False, - target={"worker_id": "w-5"}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id=snapshot.host_id, - workers=_workers_with_backend_targets(snapshot), - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - assert envelope.error is not None - assert "authoritative command submission" in envelope.error["message"] - - -@pytest.mark.parametrize("worker_id", ["w-4", "w-6", "w-7", "w-8"]) -def test_send_instruction_rejects_closed_failed_unknown_statuses(worker_id: str) -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id=f"req-{worker_id}", - dry_run=False, - target={"worker_id": worker_id}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_REJECTED - - -def test_send_instruction_resolves_selector_but_does_not_mutate() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-1", - dry_run=False, - target={"name": "Beta"}, - instruction={"text": "hello"}, - ) - context = CommandContext( - host_id=snapshot.host_id, - workers=_workers_with_backend_targets(snapshot), - ) - - envelope = execute_command(request, context) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - -def test_send_instruction_respects_ambiguous_target_before_backend() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-1", - dry_run=False, - target={"name": "Alpha"}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - envelope = execute_command(request, context) - assert envelope.ok is False - assert envelope.status == "ambiguous_target" - - -def test_send_instruction_respects_rejected_status_before_backend() -> None: - snapshot = _snapshot() - request = CommandRequest( - action="send_instruction", - request_id="req-1", - dry_run=False, - target={"worker_id": "w-4"}, - instruction={"text": "hello"}, - ) - context = CommandContext(host_id=snapshot.host_id, workers=_workers(snapshot)) - envelope = execute_command(request, context) - assert envelope.ok is False - assert envelope.status == STATUS_REJECTED - - def test_public_result_contains_no_connector_fields() -> None: snapshot = _snapshot() request = CommandRequest(action="read_snapshot") diff --git a/tests/test_agent_events.py b/tests/test_agent_events.py new file mode 100644 index 0000000..64d6190 --- /dev/null +++ b/tests/test_agent_events.py @@ -0,0 +1,845 @@ +from __future__ import annotations + +import sqlite3 +import threading +import time +import tracemalloc +from concurrent.futures import ThreadPoolExecutor +from dataclasses import replace +from datetime import datetime, timezone +from pathlib import Path + +import pytest + +from tendwire.core.agent_events import ( + AGENT_EVENT_KINDS, + AGENT_EVENT_MAX_PAYLOAD_BYTES, + AGENT_EVENT_MAX_TOTAL_ITEMS, + AgentEventIdentityConflict, + agent_event, +) +from tendwire.core.models import WorkerBinding +from tendwire.store import sqlite as store_sqlite + +from .store_helpers import record_test_agent_event, read_public_test_agent_events + + +# Exact production table shapes from schema v22 and v23. These fixtures do +# not use the current target DDL, because doing so masks cross-version rebuild +# failures when historical public tool/plan rows are populated. +_HISTORICAL_V22_AGENT_EVENTS_DDL = """ +CREATE TABLE agent_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + host_id TEXT NOT NULL, + event_id TEXT NOT NULL, + kind TEXT NOT NULL CHECK ( + kind IN ( + 'user_message', 'agent_message', 'thought', 'tool_call', + 'tool_call_update', 'plan', 'usage', 'session_info' + ) + ), + source TEXT NOT NULL, + worker_id TEXT NOT NULL, + visibility TEXT NOT NULL CHECK (visibility IN ('private', 'public')), + source_session_id TEXT, + source_turn_id TEXT, + source_item_id TEXT, + source_message_id TEXT, + source_event_id TEXT, + source_sequence INTEGER CHECK (source_sequence >= 0), + observed_at TEXT NOT NULL, + payload_fingerprint TEXT NOT NULL, + private_payload_json TEXT NOT NULL, + public_payload_json TEXT NOT NULL, + UNIQUE (host_id, event_id), + CHECK (source_event_id IS NOT NULL OR source_sequence IS NOT NULL), + CHECK (kind != 'thought' OR visibility = 'private') +) +""" + +_HISTORICAL_V23_AGENT_EVENTS_DDL = """ +CREATE TABLE agent_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + host_id TEXT NOT NULL, + event_id TEXT NOT NULL, + kind TEXT NOT NULL CHECK ( + kind IN ( + 'user_message', 'agent_message', 'thought', 'tool_call', + 'tool_call_update', 'plan', 'usage', 'session_info', 'extension' + ) + ), + source TEXT NOT NULL, + worker_id TEXT NOT NULL, + visibility TEXT NOT NULL CHECK (visibility IN ('private', 'public')), + source_session_id TEXT, + source_turn_id TEXT, + source_item_id TEXT, + source_message_id TEXT, + source_event_id TEXT, + source_sequence INTEGER CHECK (source_sequence >= 0), + observed_at TEXT NOT NULL, + payload_fingerprint TEXT NOT NULL, + private_payload_json TEXT NOT NULL, + public_payload_json TEXT NOT NULL, + UNIQUE (host_id, event_id), + CHECK (length(host_id) BETWEEN 1 AND 2048), + CHECK (instr(host_id, char(0)) = 0), + CHECK (length(event_id) = 64), + CHECK (length(source) BETWEEN 1 AND 2048), + CHECK (instr(source, char(0)) = 0), + CHECK (length(worker_id) BETWEEN 1 AND 2048), + CHECK (instr(worker_id, char(0)) = 0), + CHECK ( + source_session_id IS NULL OR ( + length(source_session_id) BETWEEN 1 AND 2048 + AND instr(source_session_id, char(0)) = 0 + ) + ), + CHECK ( + source_turn_id IS NULL OR ( + length(source_turn_id) BETWEEN 1 AND 2048 + AND instr(source_turn_id, char(0)) = 0 + ) + ), + CHECK ( + source_item_id IS NULL OR ( + length(source_item_id) BETWEEN 1 AND 2048 + AND instr(source_item_id, char(0)) = 0 + ) + ), + CHECK ( + source_message_id IS NULL OR ( + length(source_message_id) BETWEEN 1 AND 2048 + AND instr(source_message_id, char(0)) = 0 + ) + ), + CHECK ( + source_event_id IS NULL OR ( + length(source_event_id) BETWEEN 1 AND 2048 + AND instr(source_event_id, char(0)) = 0 + ) + ), + CHECK (length(observed_at) BETWEEN 20 AND 40), + CHECK (length(payload_fingerprint) = 64), + CHECK ( + CASE WHEN json_valid(private_payload_json) + THEN json_type(private_payload_json) = 'object' ELSE 0 END + ), + CHECK (length(CAST(private_payload_json AS BLOB)) <= 65536), + CHECK ( + CASE WHEN json_valid(public_payload_json) + THEN json_type(public_payload_json) = 'object' ELSE 0 END + ), + CHECK (length(CAST(public_payload_json AS BLOB)) <= 65536), + CHECK (visibility != 'private' OR public_payload_json = '{}'), + CHECK (source_event_id IS NOT NULL OR source_sequence IS NOT NULL), + CHECK ( + source_event_id IS NOT NULL + OR (source_session_id IS NOT NULL AND source_sequence IS NOT NULL) + ), + CHECK (kind NOT IN ('thought', 'extension') OR visibility = 'private') +) +""" + + +def _message_event( + *, + sequence: int, + text: str = "hello", + visibility: str = "public", +): + return agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_turn_id="private-turn-1", + source_item_id="private-item-1", + source_message_id="private-message-1", + source_sequence=sequence, + visibility=visibility, + payload={"text": text}, + observed_at="2026-07-31T00:00:00+00:00", + ) + + +def test_agent_event_contract_covers_acp_primary_kinds() -> None: + assert AGENT_EVENT_KINDS == { + "user_message", + "agent_message", + "thought", + "tool_call", + "tool_call_update", + "plan", + "usage", + "session_info", + "extension", + } + + +def test_append_is_ordered_and_replay_is_idempotent(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + first = _message_event(sequence=10) + second = _message_event(sequence=11, text="world") + + inserted = record_test_agent_event(db_path, "host-1", first) + replayed = record_test_agent_event(db_path, "host-1", first) + later = record_test_agent_event(db_path, "host-1", second) + + assert inserted.inserted is True + assert replayed.inserted is False + assert replayed.sequence == inserted.sequence + assert later.sequence > inserted.sequence + events = store_sqlite.list_agent_events(db_path, "host-1") + assert [stored.event.payload["text"] for stored in events] == [ + "hello", + "world", + ] + + +def test_deterministic_identity_rejects_changed_replay(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + original = _message_event(sequence=4, text="original") + corrupt_replay = _message_event(sequence=4, text="changed") + assert original.event_id == corrupt_replay.event_id + record_test_agent_event(db_path, "host-1", original) + + with pytest.raises(AgentEventIdentityConflict): + record_test_agent_event(db_path, "host-1", corrupt_replay) + + stored = store_sqlite.list_agent_events(db_path, "host-1") + assert len(stored) == 1 + assert stored[0].event.payload == {"text": "original"} + + +def test_source_identity_reuse_cannot_evade_conflict_by_changing_kind_or_sequence( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + original = _message_event(sequence=4) + changed_kind = agent_event( + kind="plan", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_event_id="stable-source-id", + source_sequence=5, + payload={"entries": []}, + ) + original_with_id = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_event_id="stable-source-id", + source_sequence=4, + payload={"text": "hello"}, + ) + assert original_with_id.event_id == changed_kind.event_id + record_test_agent_event(db_path, "host-1", original_with_id) + with pytest.raises(AgentEventIdentityConflict): + record_test_agent_event(db_path, "host-1", changed_kind) + + changed_sequence_kind = agent_event( + kind="plan", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_sequence=original.source_sequence, + payload={"entries": []}, + ) + assert original.event_id == changed_sequence_kind.event_id + record_test_agent_event(db_path, "host-2", original) + with pytest.raises(AgentEventIdentityConflict): + record_test_agent_event(db_path, "host-2", changed_sequence_kind) + + +def test_private_ids_and_payload_never_enter_public_projection(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + event = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_session_id="session-secret", + source_item_id="item-secret", + source_message_id="message-secret", + source_event_id="event-secret", + visibility="public", + payload={ + "text": "safe status", + "session_id": "payload-session-secret", + "cwd": "/home/smith/private-repository", + }, + ) + record_test_agent_event(db_path, "host-1", event) + + private = store_sqlite.list_agent_events( + db_path, + "host-1", + session_id="session-secret", + ) + assert private[0].event.source_item_id == "item-secret" + assert private[0].event.payload["cwd"] == "/home/smith/private-repository" + public = read_public_test_agent_events(db_path, "host-1") + assert public[0]["payload"] == {"text": "safe status"} + encoded = repr(public[0]) + assert "session-secret" not in encoded + assert "item-secret" not in encoded + assert "message-secret" not in encoded + assert "event-secret" not in encoded + assert "/home/smith" not in encoded + + +def test_thought_events_are_private_and_not_publicly_listed(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="thought events must remain private"): + agent_event( + kind="thought", + source="acp", + worker_id="worker-1", + source_session_id="session-1", + source_sequence=1, + visibility="public", + payload={"text": "reasoning summary"}, + ) + + db_path = tmp_path / "store.db" + thought = agent_event( + kind="thought", + source="acp", + worker_id="worker-1", + source_session_id="session-1", + source_sequence=1, + payload={"text": "reasoning summary"}, + ) + record_test_agent_event(db_path, "host-1", thought) + assert read_public_test_agent_events(db_path, "host-1") == () + assert store_sqlite.list_agent_events(db_path, "host-1")[0].event.payload == { + "text": "reasoning summary" + } + + +@pytest.mark.parametrize("kind", ["tool_call", "tool_call_update", "plan"]) +def test_tool_and_plan_events_are_private_only(kind: str, tmp_path: Path) -> None: + with pytest.raises(ValueError, match=rf"{kind} events must remain private"): + agent_event( + kind=kind, + source="acp", + worker_id="worker-1", + source_session_id="session-1", + source_sequence=1, + visibility="public", + payload={"content": [{"type": "text", "text": "private tool data"}]}, + ) + + event = agent_event( + kind=kind, + source="acp", + worker_id="worker-1", + source_session_id="session-1", + source_sequence=1, + payload={"content": [{"type": "text", "text": "private tool data"}]}, + ) + db_path = tmp_path / f"{kind}.db" + record_test_agent_event(db_path, "host-1", event) + assert read_public_test_agent_events(db_path, "host-1") == () + assert "private tool data" in repr( + store_sqlite.list_agent_events(db_path, "host-1") + ) + with sqlite3.connect(db_path) as conn, pytest.raises(sqlite3.IntegrityError): + conn.execute( + "UPDATE agent_events SET visibility = 'public' WHERE event_id = ?", + (event.event_id,), + ) + + +def test_queries_filter_worker_session_turn_and_cursor(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + first = _message_event(sequence=1) + second = agent_event( + kind="plan", + source="acp", + worker_id="worker-2", + source_session_id="session-2", + source_turn_id="turn-2", + source_sequence=2, + payload={"entries": [{"content": "test", "status": "pending"}]}, + ) + first_result = record_test_agent_event(db_path, "host-1", first) + record_test_agent_event(db_path, "host-1", second) + + assert len( + store_sqlite.list_agent_events(db_path, "host-1", worker_id="worker-1") + ) == 1 + assert len( + store_sqlite.list_agent_events( + db_path, "host-1", session_id="session-2", turn_id="turn-2" + ) + ) == 1 + after = store_sqlite.list_agent_events( + db_path, "host-1", after_sequence=first_result.sequence + ) + assert [stored.event.kind for stored in after] == ["plan"] + + +def test_payload_is_bounded_and_json_safe() -> None: + with pytest.raises(ValueError, match="payload text is too long"): + _message_event(sequence=1, text="x" * (AGENT_EVENT_MAX_PAYLOAD_BYTES + 1)) + with pytest.raises(ValueError, match="JSON-safe"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={"opaque": object()}, + ) + for value in (float("nan"), float("inf"), float("-inf")): + with pytest.raises(ValueError, match="non-finite"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={"tokens": value}, + ) + with pytest.raises(ValueError, match="too many total items"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={ + str(index): [0] * 256 + for index in range((AGENT_EVENT_MAX_TOTAL_ITEMS // 256) + 1) + }, + ) + + +def test_payload_and_opaque_ids_preserve_valid_unicode_exactly() -> None: + circled = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_event_id="①", + visibility="public", + payload={"text": "① and fullwidth ｅ and NUL \x00 remain private-exact"}, + ) + ascii_event = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_event_id="1", + payload={"text": "1"}, + ) + assert circled.event_id != ascii_event.event_id + assert circled.source_event_id == "①" + assert circled.payload["text"] == "① and fullwidth ｅ and NUL \x00 remain private-exact" + assert "\x00" not in str(circled.public_payload.get("text", "")) + + with pytest.raises(ValueError, match="valid Unicode"): + _message_event(sequence=9, text="\ud800") + with pytest.raises(ValueError, match="must not contain NUL"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="bad\x00id", + payload={}, + ) + + +def test_journal_payload_is_adapter_neutral_and_preserves_namespaced_extensions( + tmp_path: Path, +) -> None: + payload = { + "text": "portable message", + "org.example.agent/experimental-v2": { + "futureField": [1, "two", {"enabled": True}], + "_meta": {"opaque": "retained privately"}, + }, + } + event = agent_event( + kind="extension", + source="org.example.agent/v2", + worker_id="worker-1", + source_event_id="extension-event", + payload=payload, + ) + record_test_agent_event(tmp_path / "store.db", "host-1", event) + stored = store_sqlite.list_agent_events(tmp_path / "store.db", "host-1") + assert stored[0].event.source == "org.example.agent/v2" + assert stored[0].event.payload == payload + with pytest.raises(ValueError, match="extension events must remain private"): + agent_event( + kind="extension", + source="org.example.agent/v2", + worker_id="worker-1", + source_event_id="unsafe-public-extension", + visibility="public", + payload=payload, + ) + + +def test_observed_at_is_strict_aware_and_canonical_utc() -> None: + event = agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={"at": datetime(2026, 7, 31, tzinfo=timezone.utc)}, + observed_at="2026-08-01T08:00:00+08:00", + ) + assert event.observed_at == "2026-08-01T00:00:00+00:00" + assert event.payload["at"] == "2026-07-31T00:00:00+00:00" + for invalid in ("not-a-time", "/home/private", "2026-07-31T00:00:00"): + with pytest.raises(ValueError, match="aware ISO-8601"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={}, + observed_at=invalid, + ) + with pytest.raises(ValueError, match="naive datetime"): + agent_event( + kind="usage", + source="acp", + worker_id="worker-1", + source_event_id="event-1", + payload={"at": datetime(2026, 7, 31)}, + ) + + +def test_store_fails_closed_when_public_projection_is_corrupted( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + record_test_agent_event(db_path, "host-1", _message_event(sequence=1)) + with sqlite3.connect(db_path) as conn: + conn.execute( + "UPDATE agent_events SET public_payload_json = ?", + ('{"cwd":"/home/private","text":"safe"}',), + ) + with pytest.raises(store_sqlite.StoreSchemaError, match="invalid_agent_event_row"): + read_public_test_agent_events(db_path, "host-1") + + +def test_host_scoping_and_concurrent_replay_are_isolated(tmp_path: Path) -> None: + db_path = tmp_path / "store.db" + event = _message_event(sequence=1) + store_sqlite.init_store(db_path) + + with ThreadPoolExecutor(max_workers=8) as executor: + results = list( + executor.map( + lambda _: record_test_agent_event(db_path, "host-1", event), + range(24), + ) + ) + assert sum(result.inserted for result in results) == 1 + assert len({result.sequence for result in results}) == 1 + + other = record_test_agent_event(db_path, "host-2", event) + assert other.inserted is True + assert len(store_sqlite.list_agent_events(db_path, "host-1")) == 1 + assert len(store_sqlite.list_agent_events(db_path, "host-2")) == 1 + assert store_sqlite.list_agent_events(db_path, "host-3") == () + with pytest.raises(ValueError, match="host_id must not be empty"): + store_sqlite.list_agent_events(db_path, " ") + + + + +def test_database_constraints_and_indexes_cover_public_and_source_identity( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + record_test_agent_event(db_path, "host-1", _message_event(sequence=1)) + with sqlite3.connect(db_path) as conn: + indexes = { + str(row[1]) for row in conn.execute("PRAGMA index_list(agent_events)") + } + assert { + "idx_agent_events_host_visibility_sequence", + "idx_agent_events_source_event_identity", + "idx_agent_events_source_sequence_identity", + } <= indexes + with pytest.raises(sqlite3.IntegrityError): + conn.execute( + "UPDATE agent_events SET kind = 'thought' WHERE host_id = 'host-1'" + ) + conn.rollback() + with pytest.raises(sqlite3.IntegrityError): + conn.execute( + "UPDATE agent_events SET public_payload_json = '[]' " + "WHERE host_id = 'host-1'" + ) + + +def test_retention_removes_private_payload_without_replay_state( + tmp_path: Path, +) -> None: + db_path = tmp_path / "store.db" + old = replace( + _message_event(sequence=1, text="private historical payload", visibility="private"), + observed_at="2026-06-01T00:00:00+00:00", + ) + recent = replace( + _message_event(sequence=2, text="recent", visibility="private"), + observed_at="2026-07-30T00:00:00+00:00", + ) + inserted = record_test_agent_event(db_path, "host-1", old) + record_test_agent_event(db_path, "host-1", recent) + + result = store_sqlite.cleanup_agent_event_retention( + db_path, + "host-1", + retention_days=7, + now="2026-07-31T00:00:00+00:00", + ) + + assert result["deleted"] == 1 + assert "tombstoned" not in result + assert [item.event.payload["text"] for item in store_sqlite.list_agent_events(db_path, "host-1")] == ["recent"] + with sqlite3.connect(db_path) as conn: + encoded = "\n".join(conn.iterdump()) + assert "agent_event_tombstones" not in encoded + assert "private historical payload" not in encoded + + replay = record_test_agent_event(db_path, "host-1", old) + assert replay.inserted is True + assert replay.sequence > inserted.sequence + with pytest.raises(AgentEventIdentityConflict): + record_test_agent_event( + db_path, + "host-1", + _message_event(sequence=1, text="changed", visibility="private"), + ) + + +def test_retention_streams_metadata_for_exact_16_mib_private_row( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "large-retention.db" + text_limit = 4 * 1024 * 1024 + empty_overhead = len( + store_sqlite._canonical_json( + {"part_a": "", "part_b": "", "part_c": "", "part_d": ""} + ).encode("utf-8") + ) + payload = { + "part_a": "a" * text_limit, + "part_b": "b" * text_limit, + "part_c": "c" * text_limit, + "part_d": "d" * (text_limit - empty_overhead), + } + assert len(store_sqlite._canonical_json(payload).encode("utf-8")) == ( + AGENT_EVENT_MAX_PAYLOAD_BYTES + ) + event = agent_event( + kind="agent_message", + source="acp", + worker_id="worker-1", + source_session_id="private-session-1", + source_sequence=91, + visibility="private", + payload=payload, + observed_at="2026-06-01T00:00:00+00:00", + ) + record_test_agent_event(db_path, "host-1", event) + assert "private_payload_json" not in ( + store_sqlite._AGENT_EVENT_RETENTION_SELECT.lower() + ) + + def forbidden_full_row(_row: object) -> object: + raise AssertionError("retention must not materialize a private event row") + + monkeypatch.setattr(store_sqlite, "_agent_event_from_row", forbidden_full_row) + tracemalloc.start() + result = store_sqlite.cleanup_agent_event_retention( + db_path, + "host-1", + retention_days=7, + now="2026-07-31T00:00:00+00:00", + ) + _current_bytes, peak_bytes = tracemalloc.get_traced_memory() + tracemalloc.stop() + + assert result["deleted"] == 1 + assert peak_bytes < 8 * 1024 * 1024 + with sqlite3.connect(db_path) as conn: + assert conn.execute( + "SELECT COUNT(*) FROM agent_events WHERE host_id = ?", + ("host-1",), + ).fetchone() == (0,) + + +def test_automatic_maintenance_retires_agent_events_only_when_due( + tmp_path: Path, +) -> None: + db_path = tmp_path / "automatic-agent-retention.db" + old = replace( + _message_event(sequence=1, text="old", visibility="private"), + observed_at="2026-01-01T00:00:00+00:00", + ) + recent = replace( + _message_event(sequence=2, text="recent", visibility="private"), + observed_at="2026-01-31T23:00:00+00:00", + ) + record_test_agent_event(db_path, "host-1", old) + record_test_agent_event(db_path, "host-1", recent) + policy = store_sqlite.SnapshotRetentionPolicy( + retention_days=30, + retention_count=100, + batch_size=10, + ) + + first = store_sqlite.maybe_run_automatic_store_maintenance( + db_path, + policy=policy, + agent_event_host_id="host-1", + agent_event_retention_days=7, + cadence_seconds=3600, + now="2026-02-01T00:00:00+00:00", + ) + late_old = replace( + _message_event(sequence=3, text="late old", visibility="private"), + observed_at="2026-01-02T00:00:00+00:00", + ) + record_test_agent_event(db_path, "host-1", late_old) + not_due = store_sqlite.maybe_run_automatic_store_maintenance( + db_path, + policy=policy, + agent_event_host_id="host-1", + agent_event_retention_days=7, + cadence_seconds=3600, + now="2026-02-01T00:10:00+00:00", + ) + second = store_sqlite.maybe_run_automatic_store_maintenance( + db_path, + policy=policy, + agent_event_host_id="host-1", + agent_event_retention_days=7, + cadence_seconds=3600, + now="2026-02-01T01:00:00+00:00", + ) + + assert first["agent_events"]["deleted"] == 1 + assert not_due["status"] == "not_due" + assert not_due["agent_events"]["deleted"] == 0 + assert second["agent_events"]["deleted"] == 1 + assert [ + item.event.payload["text"] + for item in store_sqlite.list_agent_events(db_path, "host-1") + ] == ["recent"] + replay = record_test_agent_event(db_path, "host-1", old) + assert replay.inserted is True + with pytest.raises(AgentEventIdentityConflict): + record_test_agent_event( + db_path, + "host-1", + _message_event(sequence=1, text="changed", visibility="private"), + ) + + +def test_automatic_agent_retention_failure_does_not_advance_cadence( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "automatic-agent-rollback.db" + old = replace( + _message_event(sequence=1, visibility="private"), + observed_at="2026-01-01T00:00:00+00:00", + ) + record_test_agent_event(db_path, "host-1", old) + original_cleanup = store_sqlite._cleanup_agent_event_retention_conn + + def fail_after_cleanup(*args: object, **kwargs: object) -> dict[str, object]: + original_cleanup(*args, **kwargs) + raise RuntimeError("controlled retention failure") + + monkeypatch.setattr( + store_sqlite, + "_cleanup_agent_event_retention_conn", + fail_after_cleanup, + ) + + with pytest.raises(RuntimeError, match="controlled retention failure"): + store_sqlite.maybe_run_automatic_store_maintenance( + db_path, + policy=store_sqlite.SnapshotRetentionPolicy( + retention_days=30, + retention_count=100, + batch_size=10, + ), + agent_event_host_id="host-1", + agent_event_retention_days=7, + now="2026-02-01T00:00:00+00:00", + ) + + with sqlite3.connect(db_path) as conn: + assert conn.execute( + "SELECT last_completed_at FROM store_maintenance_state " + "WHERE scope = 'automatic'" + ).fetchone() == (None,) + assert conn.execute("SELECT COUNT(*) FROM agent_events").fetchone() == (1,) + + +def test_retention_cleanup_serializes_concurrent_append_and_both_commit( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "retention-concurrency.db" + old = replace( + _message_event(sequence=1, visibility="private"), + observed_at="2026-01-01T00:00:00+00:00", + ) + record_test_agent_event(db_path, "host-1", old) + entered = threading.Event() + release = threading.Event() + original_cleanup = store_sqlite._cleanup_agent_event_retention_conn + + def blocking_cleanup(*args: object, **kwargs: object) -> dict[str, object]: + entered.set() + assert release.wait(timeout=5) + return original_cleanup(*args, **kwargs) + + monkeypatch.setattr( + store_sqlite, + "_cleanup_agent_event_retention_conn", + blocking_cleanup, + ) + with ThreadPoolExecutor(max_workers=2) as executor: + cleanup = executor.submit( + store_sqlite.cleanup_agent_event_retention, + db_path, + "host-1", + retention_days=7, + now="2026-02-01T00:00:00+00:00", + ) + assert entered.wait(timeout=5) + append = executor.submit( + record_test_agent_event, + db_path, + "host-1", + _message_event(sequence=2, text="new", visibility="private"), + ) + time.sleep(0.05) + assert append.done() is False + release.set() + assert cleanup.result(timeout=5)["deleted"] == 1 + assert append.result(timeout=5).inserted is True + + assert [ + item.event.payload["text"] + for item in store_sqlite.list_agent_events(db_path, "host-1") + ] == ["new"] + with sqlite3.connect(db_path) as conn: + assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] + + +def test_journal_accepts_acp_sized_private_text(tmp_path: Path) -> None: + event = _message_event(sequence=1, text="x" * (64 * 1024), visibility="private") + result = record_test_agent_event(tmp_path / "store.db", "host-1", event) + assert result.inserted is True diff --git a/tests/test_answer_decision.py b/tests/test_answer_decision.py deleted file mode 100644 index 249513d..0000000 --- a/tests/test_answer_decision.py +++ /dev/null @@ -1,888 +0,0 @@ -"""Semantic connector answers for current backend-owned Claude decisions.""" - -from __future__ import annotations - -from pathlib import Path -from typing import Any - -import pytest -import tendwire.command_submission as command_submission - -from tendwire.backends.herdr_decision import calibrate_decision_steps -from tendwire.backends.herdr_turns import ( - PENDING_DECISION_MAX_OPTIONS, - _pending_observation_from_turn, -) -from tendwire.command_submission import submit_command -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - STATUS_ACCEPTED, - STATUS_ANSWER_IN_PROGRESS, - STATUS_DECISION_NOT_PENDING, - STATUS_INVALID_SELECTION, - STATUS_PENDING, - STATUS_REQUEST_STATE_UNCERTAIN, - STATUS_UNKNOWN_WORKER, - STATUS_UNSUPPORTED_DECISION, - CommandRequest, - build_canonical_mutation, -) -from tendwire.core.models import Worker -from tendwire.store.sqlite import ( - abandon_backend_pending_choice_claim, - apply_backend_pending_observation, - claim_backend_pending_decision, - envelope_to_receipt_json, - get_command_request, - pending_payload_from_store, - reserve_command_request, -) - -from tests.test_command_submission import ( - _FakeSocketClient, - _binding, - _config, - _factory, - _seed, -) - - -def _decision_turn( - *, - prompt: str = "Choose a database", - kind: str = "AskUserQuestion", - multi_select: bool = False, - question_count: int = 1, -) -> dict[str, Any]: - return { - "pending_decision": { - "decision_id": "private-tool-use", - "kind": kind, - "question": prompt, - "options": [ - {"id": "postgres", "label": "Postgres"}, - {"id": "sqlite", "label": "SQLite"}, - {"id": "duckdb", "label": "DuckDB"}, - {"id": "mysql", "label": "MySQL"}, - ], - "multi_select": multi_select, - "question_count": question_count, - } - } - - -def _seed_pending_decision( - tmp_path: Path, - *, - turn: dict[str, Any] | None = None, - turn_model: str = "observed", -) -> tuple[Any, Worker, str]: - config = _config(tmp_path, turn_model=turn_model) - worker = Worker(id="w-1", name="Alpha", status="active") - binding = _binding( - worker, - private_fingerprint="decision-binding-private", - turn_target_value="decision-pane-private", - ) - _seed(config, [worker], [binding]) - assert config.db_path is not None - observation = _pending_observation_from_turn(turn or _decision_turn()) - assert apply_backend_pending_observation( - config.db_path, - config.host_id, - worker.id, - observation, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - payload = pending_payload_from_store(config.db_path, config.host_id) - row = next(item for item in payload["pending_interactions"] if item["worker_id"] == worker.id) - return config, worker, row["meta"]["decision"]["decision_ref"] - - -def _answer_request( - decision_ref: str, - *, - request_id: str = "decision-request-1", - worker_id: str = "w-1", - selection: dict[str, Any] | None = None, -) -> dict[str, Any]: - return { - "schema_version": 1, - "action": "answer_decision", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": worker_id}, - "params": { - "decision_ref": decision_ref, - "selection": selection or {"option_refs": ["2"]}, - }, - } - - -def test_pending_payload_carries_stable_structured_decision_and_rotates_ref( - tmp_path: Path, -) -> None: - config, worker, first_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - first = pending_payload_from_store(config.db_path, config.host_id) - first_row = next(item for item in first["pending_interactions"] if item["worker_id"] == worker.id) - assert first_row["meta"]["decision"] == { - "decision_ref": first_ref, - "kind": "single", - "prompt": "Choose a database", - "options": [ - {"ref": "1", "label": "Postgres"}, - {"ref": "2", "label": "SQLite"}, - {"ref": "3", "label": "DuckDB"}, - {"ref": "4", "label": "MySQL"}, - ], - "multi_select": False, - "question_count": 1, - } - - binding = _binding( - worker, - private_fingerprint="decision-binding-private", - turn_target_value="decision-pane-private", - ) - changed = _pending_observation_from_turn( - _decision_turn(prompt="Choose a durable database") - ) - assert apply_backend_pending_observation( - config.db_path, - config.host_id, - worker.id, - changed, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - second = pending_payload_from_store(config.db_path, config.host_id) - second_row = next(item for item in second["pending_interactions"] if item["worker_id"] == worker.id) - assert second_row["meta"]["decision"]["decision_ref"] != first_ref - - -def test_answer_decision_stale_ref_fails_before_pane_io(tmp_path: Path) -> None: - config, _worker, _decision_ref = _seed_pending_decision(tmp_path) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request("decision-stale"), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_DECISION_NOT_PENDING - assert calls == [] - - -def test_answer_decision_omitted_dry_run_is_preview_only(tmp_path: Path) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - calls: list[dict[str, Any]] = [] - request = _answer_request(decision_ref) - request.pop("dry_run") - - result = submit_command( - config, - request, - socket_client_factory=_factory(calls), - ) - - assert result.ok is True - assert result.status == "dry_run" - assert result.dry_run is True - assert calls == [] - - -def test_answer_decision_unknown_worker_fails_before_pane_io(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request("decision-any", worker_id="worker-missing"), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_UNKNOWN_WORKER - assert calls == [] - - -@pytest.mark.parametrize( - "selection", - [ - {"option_refs": ["9"]}, - {"option_refs": ["1", "2"]}, - {"option_refs": ["2", "2"]}, - {"text": ""}, - {"option_refs": ["1"], "text": "both"}, - ], -) -def test_answer_decision_invalid_selection_fails_before_pane_io( - tmp_path: Path, - selection: dict[str, Any], -) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request(decision_ref, selection=selection), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_INVALID_SELECTION - assert calls == [] - - -def test_answer_decision_refuses_multi_question_before_pane_io(tmp_path: Path) -> None: - config, _worker, decision_ref = _seed_pending_decision( - tmp_path, - turn=_decision_turn(question_count=2), - ) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request(decision_ref), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_UNSUPPORTED_DECISION - assert calls == [] - - -def test_answer_decision_rejects_plan_write_in_before_pane_io(tmp_path: Path) -> None: - config, _worker, decision_ref = _seed_pending_decision( - tmp_path, - turn=_decision_turn(kind="ExitPlanMode"), - ) - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request(decision_ref, selection={"text": "Revise this plan"}), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_INVALID_SELECTION - assert calls == [] - - -def test_decision_calibration_steps_cover_single_plan_write_in_and_multi() -> None: - single = calibrate_decision_steps( - kind="single", option_count=4, option_refs=("2",) - ) - assert [(item.operation, item.keys, item.text) for item in single] == [ - ("keys", ("2",), None) # digit alone selects AND submits (live-verified) - ] - - plan = calibrate_decision_steps( - kind="plan", option_count=2, option_refs=("1",) - ) - assert [(item.operation, item.keys, item.text) for item in plan] == [ - ("keys", ("1",), None) - ] - - write_in = calibrate_decision_steps( - kind="single", option_count=4, text="Use another database" - ) - assert [(item.operation, item.keys, item.text) for item in write_in] == [ - ("keys", ("Down", "Down", "Down", "Down"), None), # write-in row ignores digits - ("input", ("Enter",), "Use another database"), - ] - - multi = calibrate_decision_steps( - kind="multi", option_count=4, option_refs=("3", "1") - ) - assert [(item.operation, item.keys, item.text) for item in multi] == [ - ("keys", ("1",), None), # digits toggle rows absolutely - ("keys", ("3",), None), - ("keys", ("Right", "Enter"), None), # Submit tab, then submit - ] - - -def test_production_shaped_multi_decision_end_to_end(tmp_path: Path) -> None: - tool_input = { - "questions": [ - { - "question": "Which databases should we support?", - "header": "Database support", - "options": [ - {"label": "Postgres", "description": "Primary database"}, - {"label": "SQLite", "description": "Local database"}, - {"label": "DuckDB", "description": "Analytics database"}, - {"label": "MySQL", "description": "Compatibility database"}, - ], - "multiSelect": True, - } - ] - } - question = tool_input["questions"][0] - adapter_pending_decision = { - "decision_id": "toolu_multi_123", - "prompt": f'{question["header"]}: {question["question"]}', - "mode": "multi", - "multi_select": question["multiSelect"], - "options": [ - {"id": str(ordinal), "label": option["label"]} - for ordinal, option in enumerate(question["options"], 1) - ], - } - assert adapter_pending_decision == { - "decision_id": "toolu_multi_123", - "prompt": "Database support: Which databases should we support?", - "mode": "multi", - "multi_select": True, - "options": [ - {"id": "1", "label": "Postgres"}, - {"id": "2", "label": "SQLite"}, - {"id": "3", "label": "DuckDB"}, - {"id": "4", "label": "MySQL"}, - ], - } - - config, worker, decision_ref = _seed_pending_decision( - tmp_path, - turn={"pending_decision": adapter_pending_decision}, - ) - assert config.db_path is not None - payload = pending_payload_from_store(config.db_path, config.host_id) - pending = next( - row for row in payload["pending_interactions"] - if row["worker_id"] == worker.id - ) - assert pending["meta"]["decision"] == { - "decision_ref": decision_ref, - "kind": "multi", - "prompt": "Database support: Which databases should we support?", - "options": [ - {"ref": "1", "label": "Postgres"}, - {"ref": "2", "label": "SQLite"}, - {"ref": "3", "label": "DuckDB"}, - {"ref": "4", "label": "MySQL"}, - ], - "multi_select": True, - "question_count": 1, - } - - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request( - decision_ref, - request_id="multi-production-answer", - selection={"option_refs": ["3", "1"]}, - ), - socket_client_factory=_factory(calls), - ) - - assert result.ok is True - assert result.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "pane.send_keys", - "params": {"pane_id": "decision-pane-private", "keys": ["1"]}, - }, - { - "method": "pane.send_keys", - "params": {"pane_id": "decision-pane-private", "keys": ["3"]}, - }, - { - "method": "pane.send_keys", - "params": {"pane_id": "decision-pane-private", "keys": ["Right", "Enter"]}, - }, - ] - - -def _bounded_decision_turn(option_count: int, *, custom_last: bool = False) -> dict[str, Any]: - options = [ - {"id": str(ordinal), "label": f"Option {ordinal}"} - for ordinal in range(1, option_count + 1) - ] - if custom_last: - options.append({"id": "custom", "label": "Type something"}) - return { - "pending_decision": { - "decision_id": "toolu_bound", - "kind": "AskUserQuestion", - "prompt": "Choose one", - "multi_select": False, - "options": options, - } - } - - -def _unknown_decision_turn() -> dict[str, Any]: - turn = _bounded_decision_turn(2) - turn["pending_decision"]["kind"] = "FutureDecisionKind" - return turn - - -def test_decision_option_bound_accepts_exactly_nine(tmp_path: Path) -> None: - config, worker, _decision_ref = _seed_pending_decision( - tmp_path, - turn=_bounded_decision_turn(PENDING_DECISION_MAX_OPTIONS), - ) - assert config.db_path is not None - payload = pending_payload_from_store(config.db_path, config.host_id) - row = next( - item for item in payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert len(row["meta"]["decision"]["options"]) == PENDING_DECISION_MAX_OPTIONS - - -def test_decision_option_bound_accepts_nine_plus_trailing_write_in( - tmp_path: Path, -) -> None: - config, worker, _decision_ref = _seed_pending_decision( - tmp_path, - turn=_bounded_decision_turn( - PENDING_DECISION_MAX_OPTIONS, - custom_last=True, - ), - ) - assert config.db_path is not None - payload = pending_payload_from_store(config.db_path, config.host_id) - row = next( - item for item in payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert len(row["meta"]["decision"]["options"]) == PENDING_DECISION_MAX_OPTIONS - - -@pytest.mark.parametrize( - "turn", - [ - _bounded_decision_turn(PENDING_DECISION_MAX_OPTIONS + 1), - _bounded_decision_turn( - PENDING_DECISION_MAX_OPTIONS + 1, - custom_last=True, - ), - _unknown_decision_turn(), - ], - ids=[ - "ten-real-options", - "custom-row-after-ten-real-options", - "unknown-kind", - ], -) -def test_over_bound_decision_fails_closed_without_pane_io( - tmp_path: Path, - turn: dict[str, Any], -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - binding = _binding( - worker, - private_fingerprint="decision-binding-private", - turn_target_value="decision-pane-private", - ) - _seed(config, [worker], [binding]) - observation = _pending_observation_from_turn(turn) - assert observation.kind == "read_succeeded_unsupported_decision" - assert config.db_path is not None - assert apply_backend_pending_observation( - config.db_path, - config.host_id, - worker.id, - observation, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - payload = pending_payload_from_store(config.db_path, config.host_id) - assert not any( - item["worker_id"] == worker.id - for item in payload["pending_interactions"] - ) - - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - _answer_request("decision-unsupported-bound"), - socket_client_factory=_factory(calls), - ) - assert result.ok is False - assert result.status == STATUS_UNSUPPORTED_DECISION - assert calls == [] - - -def test_answer_decision_request_id_replay_does_not_resend_keys(tmp_path: Path) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - calls: list[dict[str, Any]] = [] - request = _answer_request(decision_ref) - - first = submit_command(config, request, socket_client_factory=_factory(calls)) - replay = submit_command(config, request, socket_client_factory=_factory(calls)) - - assert first.ok is True - assert first.status == STATUS_ACCEPTED - assert replay.to_dict() == first.to_dict() - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_answer_decision_observed_mode_completes_without_instruction_turn( - tmp_path: Path, -) -> None: - config, _worker, decision_ref = _seed_pending_decision( - tmp_path, - turn_model="observed", - ) - calls: list[dict[str, Any]] = [] - - result = submit_command( - config, - _answer_request(decision_ref, request_id="observed-answer-decision"), - socket_client_factory=_factory(calls), - ) - - assert result.ok is True - assert result.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_two_requests_race_one_decision_and_only_first_claimant_sends( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config, worker, decision_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - winner = _answer_request(decision_ref, request_id="decision-winner") - loser = _answer_request(decision_ref, request_id="decision-loser") - calls: list[dict[str, Any]] = [] - raced: dict[str, Any] = {} - real_mark = command_submission._mark_request_send_started - - def race_after_claim(*args: Any, **kwargs: Any) -> Any: - pending = pending_payload_from_store(config.db_path, config.host_id) - assert any( - item["worker_id"] == worker.id - and item["meta"]["decision"]["decision_ref"] == decision_ref - for item in pending["pending_interactions"] - ) - raced["loser"] = submit_command( - config, - loser, - socket_client_factory=_factory(calls), - ) - return real_mark(*args, **kwargs) - - monkeypatch.setattr( - command_submission, - "_mark_request_send_started", - race_after_claim, - ) - first = submit_command( - config, - winner, - socket_client_factory=_factory(calls), - ) - - assert first.ok is True - assert first.status == STATUS_ACCEPTED - assert raced["loser"].ok is False - assert raced["loser"].status == STATUS_ANSWER_IN_PROGRESS - assert raced["loser"].disposition == DISPOSITION_NO_RECEIPT - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_winner_safe_presend_failure_releases_claim_for_loser_retry( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - winner = _answer_request(decision_ref, request_id="decision-safe-failure") - loser = _answer_request(decision_ref, request_id="decision-safe-retry") - calls: list[dict[str, Any]] = [] - raced: dict[str, Any] = {} - real_mark = command_submission._mark_request_send_started - first_mark = True - - def fail_winner_before_send(*args: Any, **kwargs: Any) -> Any: - nonlocal first_mark - if first_mark: - first_mark = False - raced["loser"] = submit_command( - config, - loser, - socket_client_factory=_factory(calls), - ) - return command_submission._request_in_progress(args[1]) - return real_mark(*args, **kwargs) - - monkeypatch.setattr( - command_submission, - "_mark_request_send_started", - fail_winner_before_send, - ) - failed_winner = submit_command( - config, - winner, - socket_client_factory=_factory(calls), - ) - retried_loser = submit_command( - config, - loser, - socket_client_factory=_factory(calls), - ) - - assert failed_winner.status == STATUS_PENDING - assert raced["loser"].status == STATUS_ANSWER_IN_PROGRESS - assert raced["loser"].disposition == DISPOSITION_NO_RECEIPT - assert retried_loser.ok is True - assert retried_loser.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_claim_time_loser_releases_unsent_reservation_for_takeover( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config, worker, decision_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - request = _answer_request(decision_ref, request_id="claim-time-loser") - calls: list[dict[str, Any]] = [] - real_claim = command_submission._claim_pending_decision - injected_claim: dict[str, Any] = {} - - def lose_during_claim( - claim_config: Any, - claim_request: CommandRequest, - validated: Any, - ) -> Any: - if "value" not in injected_claim: - injected_claim["value"] = claim_backend_pending_decision( - config.db_path, - config.host_id, - worker.id, - decision_ref, - {"option_refs": ["2"]}, - claim=True, - ) - assert injected_claim["value"].status == "claimed" - return real_claim(claim_config, claim_request, validated) - - monkeypatch.setattr( - command_submission, - "_claim_pending_decision", - lose_during_claim, - ) - first = submit_command( - config, - request, - socket_client_factory=_factory(calls), - ) - receipt = get_command_request( - config.db_path, - config.host_id, - "claim-time-loser", - ) - - assert first.status == STATUS_ANSWER_IN_PROGRESS - assert first.disposition == DISPOSITION_IN_PROGRESS - assert receipt is not None - assert receipt["state"] == "reserved" - assert receipt["status"] == STATUS_PENDING - assert calls == [] - - assert abandon_backend_pending_choice_claim( - config.db_path, - config.host_id, - injected_claim["value"].claim_token, - ) - retry = submit_command( - config, - request, - socket_client_factory=_factory(calls), - ) - assert retry.ok is True - assert retry.status == STATUS_ACCEPTED - assert len(calls) == 1 - - -def test_abandoned_reservation_is_not_terminalized_by_live_claim( - tmp_path: Path, -) -> None: - config, worker, decision_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - payload = _answer_request(decision_ref, request_id="abandoned-loser") - request = CommandRequest.from_dict(payload) - canonical = build_canonical_mutation(request, public_worker_id=worker.id) - initial = reserve_command_request( - config.db_path, - host_id=config.host_id, - request_id=request.request_id or "", - action=canonical.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=envelope_to_receipt_json( - command_submission._request_in_progress(request) - ), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), - owner_lease_seconds=1, - now="2020-01-01T00:00:00+00:00", - ) - assert initial["status"] == "reserved" - competing = claim_backend_pending_decision( - config.db_path, - config.host_id, - worker.id, - decision_ref, - {"option_refs": ["1"]}, - claim=True, - ) - assert competing.status == "claimed" - - calls: list[dict[str, Any]] = [] - result = submit_command( - config, - payload, - socket_client_factory=_factory(calls), - ) - receipt = get_command_request( - config.db_path, - config.host_id, - request.request_id or "", - ) - - assert result.status == STATUS_ANSWER_IN_PROGRESS - assert result.disposition == DISPOSITION_IN_PROGRESS - assert receipt is not None - assert receipt["state"] == "reserved" - assert receipt["status"] == STATUS_PENDING - assert calls == [] - - -def test_process_loss_before_send_started_is_recovered_after_lease_expiry( - tmp_path: Path, -) -> None: - config, worker, decision_ref = _seed_pending_decision(tmp_path) - assert config.db_path is not None - payload = _answer_request(decision_ref, request_id="crashed-before-send") - request = CommandRequest.from_dict(payload) - canonical = build_canonical_mutation(request, public_worker_id=worker.id) - reservation = reserve_command_request( - config.db_path, - host_id=config.host_id, - request_id=request.request_id or "", - action=canonical.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=envelope_to_receipt_json( - command_submission._request_in_progress(request) - ), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), - owner_lease_seconds=1, - now="2020-01-01T00:00:00+00:00", - ) - assert reservation["status"] == "reserved" - abandoned_claim = claim_backend_pending_decision( - config.db_path, - config.host_id, - worker.id, - decision_ref, - {"option_refs": ["2"]}, - claim=True, - observed_at="2020-01-01T00:00:00+00:00", - claim_lease_seconds=1, - ) - assert abandoned_claim.status == "claimed" - - calls: list[dict[str, Any]] = [] - recovered = submit_command( - config, - payload, - socket_client_factory=_factory(calls), - ) - - assert recovered.ok is True - assert recovered.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] - - -def test_send_uncertainty_is_durable_and_never_resends( - tmp_path: Path, -) -> None: - config, _worker, decision_ref = _seed_pending_decision(tmp_path) - request = _answer_request(decision_ref, request_id="uncertain-decision") - calls: list[dict[str, Any]] = [] - - class FailingKeyClient(_FakeSocketClient): - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if method == "pane.send_keys": - raise RuntimeError("response lost after key send") - return {"accepted": True} - - first = submit_command( - config, - request, - socket_client_factory=lambda _config: FailingKeyClient(calls), - ) - replay = submit_command( - config, - request, - socket_client_factory=lambda _config: FailingKeyClient(calls), - ) - - assert first.ok is False - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert calls == [ - { - "method": "pane.send_keys", - "params": { - "pane_id": "decision-pane-private", - "keys": ["2"], - }, - } - ] diff --git a/tests/test_backend.py b/tests/test_backend.py deleted file mode 100644 index 918d691..0000000 --- a/tests/test_backend.py +++ /dev/null @@ -1,1789 +0,0 @@ -"""Tests for the Herdr CLI backend adapter contract.""" - -from __future__ import annotations - -import json -import os -import subprocess -from collections.abc import Sequence -from typing import Any - -import pytest - -from tendwire import cli as tendwire_cli -from tendwire.backends import herdr_cli, herdr_command -from tendwire.backends.herdr_cli import fetch_herdr_state -from tendwire.config import Config -from tendwire.core.commands import ( - STATUS_ACCEPTED, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_FAILED, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_REQUEST_STATE_UNCERTAIN, - CommandEnvelope, -) -from tendwire.core.models import Worker, WorkerBinding, worker_binding_private_fingerprint -from tendwire.core.projector import project_from_observations -from tendwire.store.sqlite import init_store, list_worker_bindings - - -_FORBIDDEN_FIELDS = { - "telegram", - "chat_id", - "topic_id", - "message_id", - "thread_id", - "token", - "bot_token", - "delivery", - "route", - "herdres_delivery", - "pane_id", - "terminal_id", - "backend_target", - "agent_session", - "session_id", - "herdr_state", - "herdres_state", - "target_kind", - "target_value", - "turn_target_kind", - "turn_target_value", - "private_fingerprint", - "argv", - "command", - "env", - "stderr", - "stdout", - "secret", - "secrets", - "shell", - "connector", - "connectors", -} -_FORBIDDEN_FIELDS_COMPACT = {field.replace("_", "") for field in _FORBIDDEN_FIELDS} - - -def _completed(stdout: str, returncode: int = 0) -> subprocess.CompletedProcess[str]: - return subprocess.CompletedProcess( - args=["herdr", "workspace", "list", "--json"], - returncode=returncode, - stdout=stdout, - stderr="", - ) - - -def _respond(args: Sequence[str], responses: dict[tuple[str, ...], Any]) -> subprocess.CompletedProcess[str] | None: - """Return a canned response for a herdr command tuple.""" - key = tuple(args) - if key not in responses: - return _completed("", returncode=1) - response = responses[key] - if isinstance(response, subprocess.CompletedProcess): - return response - if isinstance(response, str): - return _completed(response) - return _completed(json.dumps(response)) - - -def _assert_no_forbidden_fields(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - normalized = str(key).lower().replace("-", "_").replace(".", "_") - compact = normalized.replace("_", "") - segments = {part for part in normalized.split("_") if part} - assert ( - normalized not in _FORBIDDEN_FIELDS and compact not in _FORBIDDEN_FIELDS_COMPACT - and not (segments & _FORBIDDEN_FIELDS) - ), f"forbidden field {path}.{key}" - _assert_no_forbidden_fields(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_forbidden_fields(item, f"{path}[{index}]") - - -def _send_completed(returncode: int = 0) -> subprocess.CompletedProcess[str]: - return subprocess.CompletedProcess( - args=["herdr", "agent", "send", "worker-1", "hello"], - returncode=returncode, - stdout="", - stderr="", - ) - - -def test_send_instruction_uses_agent_send_argv(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[list[str], dict[str, Any]]] = [] - instruction_text = "line one\nline two\tindented" - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append((args, kwargs)) - return _send_completed() - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command.subprocess, "run", fake_run) - - envelope = herdr_command.send_instruction( - config, - { - "worker_id": "public-worker-1", - "backend_target": {"kind": "agent_id", "value": "agent-send-1", "sendable": True, "reason": None}, - "terminal_id": "ignored", - }, - {"text": instruction_text}, - ) - - assert envelope.ok is True - assert envelope.status == STATUS_ACCEPTED - assert envelope.result == {"target": {"worker_id": "public-worker-1"}} - assert not isinstance(envelope, CommandEnvelope) - assert set(envelope.to_dict()) == {"ok", "status", "result", "error"} - assert calls == [ - ( - ["herdr", "agent", "send", "agent-send-1", instruction_text], - { - "capture_output": True, - "text": True, - "check": False, - "timeout": config.herdr_timeout_seconds, - }, - ) - ] - assert "shell" not in calls[0][1] - - -def test_send_instruction_requires_private_backend_target(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[Any] = [] - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", lambda *args: calls.append(args)) - - envelope = herdr_command.send_instruction( - config, - {"worker_id": "public-worker-1", "agent_session": {"value": "sess-ignored"}}, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - assert calls == [] - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_maps_missing_binary_to_backend_unavailable(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="missing-herdr") - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: None) - monkeypatch.setattr( - herdr_command, - "_run_agent_send", - lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("should not run")), - ) - - envelope = herdr_command.send_instruction( - config, - {"worker_id": "worker-1", "backend_target": {"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}}, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_maps_nonzero_exit_to_backend_failed(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", lambda *args: _send_completed(returncode=2)) - - envelope = herdr_command.send_instruction( - config, - {"worker_id": "worker-1", "backend_target": {"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}}, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_FAILED - assert envelope.error["details"] == {"exit_code": 2} - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_maps_timeout_to_uncertain(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - - def raise_timeout(*args: Any) -> subprocess.CompletedProcess[str]: - raise subprocess.TimeoutExpired(cmd=["herdr", "agent", "send"], timeout=5.0) - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", raise_timeout) - - envelope = herdr_command.send_instruction( - config, - {"worker_id": "worker-1", "backend_target": {"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}}, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_REQUEST_STATE_UNCERTAIN - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_rejects_ambiguous_private_backend_target(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[Any] = [] - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", lambda *args: calls.append(args)) - - envelope = herdr_command.send_instruction( - config, - { - "worker_id": "public-worker-1", - "backend_target": { - "kind": "agent_id", - "value": "agent-1", - "sendable": False, - "reason": "duplicate_backend_target", - }, - }, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_AMBIGUOUS_BACKEND_TARGET - assert calls == [] - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_send_instruction_rejects_unsupported_private_backend_target_kind(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[Any] = [] - - monkeypatch.setattr(herdr_command.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_command, "_run_agent_send", lambda *args: calls.append(args)) - - envelope = herdr_command.send_instruction( - config, - { - "worker_id": "public-worker-1", - "backend_target": { - "kind": "session_id", - "value": "session-must-not-send", - "sendable": True, - "reason": None, - }, - }, - {"text": "hello"}, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_BACKEND_UNSUPPORTED - assert calls == [] - serialized = json.dumps(envelope.to_dict()) - assert "session-must-not-send" not in serialized - _assert_no_forbidden_fields(envelope.to_dict()) - - -def test_fetch_herdr_state_returns_empty_when_binary_missing() -> None: - config = Config(host_id="testhost", herdr_bin="definitely-not-a-real-herdr-binary") - spaces, workers = fetch_herdr_state(config) - assert spaces == [] - assert workers == [] - - -def test_fetch_herdr_state_returns_empty_on_cli_failure(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli, - "_run_herdr", - lambda args, cfg: _completed('{"workers":[{"id":"leaked"}]}', returncode=2), - ) - - spaces, workers = fetch_herdr_state(config) - - assert spaces == [] - assert workers == [] - - -def test_fetch_herdr_state_returns_empty_on_malformed_json(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _completed("not json")) - - spaces, workers = fetch_herdr_state(config) - - assert spaces == [] - assert workers == [] - - -def test_sample_herdr_projection_is_neutral_and_fingerprinted(monkeypatch) -> None: - config = Config(host_id="herdr-host", herdr_bin="herdr") - sample_payload = { - "spaces": [ - { - "id": "space-1", - "name": "Build", - "status": "running", - "status_line": "building package", - "telegram": "forbidden", - "chat_id": 111, - "safe": "space-meta", - } - ], - "workers": [ - { - "id": "worker-1", - "name": "Agent One", - "status": "panic", - "space_id": "space-1", - "summary": "crashed", - "topic_id": 222, - "message_id": 333, - "route": "telegram", - "delivery": {"chat_id": 444}, - "safe": "worker-meta", - } - ], - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli, - "_run_herdr", - lambda args, cfg: _completed(json.dumps(sample_payload)), - ) - - spaces, workers = fetch_herdr_state(config) - snapshot = project_from_observations(config, spaces=spaces, workers=workers) - payload = json.loads(snapshot.to_json()) - - assert payload["schema_version"] == 2 - assert len(payload["content_fingerprint"]) == 24 - assert payload["spaces"][0]["status"] == "active" - assert payload["spaces"][0]["fingerprint"] - assert payload["spaces"][0]["meta"]["safe"] == "space-meta" - assert payload["workers"][0]["status"] == "failed" - assert payload["workers"][0]["fingerprint"] - assert payload["workers"][0]["meta"]["raw_status"] == "panic" - assert payload["workers"][0]["meta"]["safe"] == "worker-meta" - assert payload["attention"][0]["source"] == "worker:worker-1" - assert payload["attention"][0]["fingerprint"] - _assert_no_forbidden_fields(payload) - - -def test_herdr_cli_strip_connector_fields_drops_dot_separated_aliases() -> None: - raw = { - "safe": "kept", - "backend.target": "sentinel-private-backend", - "message.id": "sentinel-private-message", - "bot.token": "sentinel-private-token", - "herdres.delivery": {"message.id": "sentinel-private-delivery"}, - "delivery.route": "sentinel-private-route", - "telegram.message.id": "sentinel-private-telegram", - "children": [ - { - "safe_child": "kept", - "topic.id": "sentinel-private-topic", - } - ], - } - - stripped = herdr_cli._strip_connector_fields(raw) - - assert stripped == {"safe": "kept", "children": [{"safe_child": "kept"}]} - assert "sentinel-private" not in json.dumps(stripped, sort_keys=True) - _assert_no_forbidden_fields(stripped) - assert herdr_cli._safe_text_sample("failed backend.target=sentinel-private-backend") is None - assert all( - herdr_cli._safe_text_sample(sample) is None - for sample in ( - "failed bot.token=sentinel-private-token", - "failed bot_token=sentinel-private-token", - "failed bot-token=sentinel-private-token", - "failed botToken=sentinel-private-token", - ) - ) - assert herdr_cli._safe_text_sample("plain diagnostic text") == "plain diagnostic text" - - -def test_herdr_agent_public_id_and_private_backend_target_are_separate(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-worker", - "agent_id": "send-agent", - "agent": "Coder", - "agent_session": {"value": "sess-must-not-leak"}, - "terminal_id": "term-must-not-leak", - "pane_id": "pane-must-not-leak", - "session_id": "session-must-not-leak", - "workspace_id": "ws-1", - } - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - snapshot = project_from_observations(config, spaces=spaces, workers=workers) - payload = json.loads(snapshot.to_json()) - - assert len(workers) == 1 - assert workers[0].id == "public-worker" - assert workers[0].backend_target is not None - assert workers[0].backend_target["kind"] == "agent_id" - assert workers[0].backend_target["value"] == "send-agent" - assert workers[0].backend_target["sendable"] is True - assert workers[0].backend_target["reason"] is None - assert payload["workers"][0]["id"] == "public-worker" - assert payload["workers"][0]["name"] == "Coder" - assert "sess-must-not-leak" not in json.dumps(payload) - assert "term-must-not-leak" not in json.dumps(payload) - assert "pane-must-not-leak" not in json.dumps(payload) - assert "session-must-not-leak" not in json.dumps(payload) - _assert_no_forbidden_fields(payload) - - -def test_herdr_bindings_reuse_worker_id_by_private_fingerprint_and_update_moved_target(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - first_responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-before", - "agent": "Coder", - "agent_session": {"value": "sess-stable"}, - "terminal_id": "term-before", - "pane_id": "pane-before", - "workspace_id": "ws-1", - } - ] - } - }, - } - second_responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-after", - "agent": "Coder", - "agent_session": {"value": "sess-stable"}, - "terminal_id": "term-after", - "pane_id": "pane-after", - "workspace_id": "ws-1", - } - ] - } - }, - } - responses = [first_responses, second_responses] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str] | None: - return _respond(args, responses[0]) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - _spaces, workers, bindings = fetch_herdr_state(config, include_bindings=True) - assert workers[0].id == "public-before" - assert bindings[0].worker_id == "public-before" - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == "term-before" - - responses.pop(0) - _spaces2, workers2, bindings2 = fetch_herdr_state( - config, - stored_bindings=bindings, - include_bindings=True, - ) - - assert workers2[0].id == "public-before" - assert bindings2[0].worker_id == "public-before" - assert bindings2[0].private_fingerprint == bindings[0].private_fingerprint - assert bindings2[0].target_kind == "terminal_id" - assert bindings2[0].target_value == "term-after" - payload = json.loads(project_from_observations(config, workers=workers2).to_json()) - assert "term-after" not in json.dumps(payload) - assert "pane-after" not in json.dumps(payload) - _assert_no_forbidden_fields(payload) - - -def test_herdr_reuses_worker_id_by_unique_backend_target_fallback(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - stored = [ - WorkerBinding( - host_id="testhost", - worker_id="stored-public", - worker_fingerprint="stored-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-send", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="2026-01-02T00:00:00+00:00", - private_fingerprint="old-private", - ) - ] - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "new-public", - "agent_id": "agent-send", - "agent": "Coder", - } - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _spaces, workers, bindings = fetch_herdr_state( - config, - stored_bindings=stored, - include_bindings=True, - ) - - assert workers[0].id == "stored-public" - assert bindings[0].worker_id == "stored-public" - assert bindings[0].target_value == "agent-send" - assert bindings[0].private_fingerprint != "old-private" - - -def test_herdr_backend_target_precedence_and_pane_fallback(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "id": "public-id", - "agent_id": "agent-send", - "agent": "agent-name", - "label": "agent-label", - "terminal_id": "term-fallback", - "pane_id": "pane-fallback", - }, - { - "slug": "pane-public", - "agent_session": {"value": "sess-not-sendable"}, - "terminal_id": "term-send", - "pane_id": "pane-send", - "state_labels": ["agent"], - }, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _, workers, bindings = fetch_herdr_state(config, include_bindings=True) - by_id = {worker.id: worker for worker in workers} - bindings_by_id = {binding.worker_id: binding for binding in bindings} - - assert by_id["public-id"].backend_target is not None - assert by_id["public-id"].backend_target["kind"] == "agent_id" - assert by_id["public-id"].backend_target["value"] == "agent-send" - assert by_id["public-id"].backend_target["sendable"] is True - assert by_id["pane-public"].backend_target is not None - assert by_id["pane-public"].backend_target["kind"] == "terminal_id" - assert by_id["pane-public"].backend_target["value"] == "term-send" - assert by_id["pane-public"].backend_target["sendable"] is True - assert bindings_by_id["public-id"].turn_target_kind == "pane_id" - assert bindings_by_id["public-id"].turn_target_value == "pane-fallback" - assert bindings_by_id["pane-public"].turn_target_kind == "pane_id" - assert bindings_by_id["pane-public"].turn_target_value == "pane-send" - assert all( - (worker.backend_target or {}).get("value") != "sess-not-sendable" - for worker in workers - ) - - -def test_duplicate_sendable_backend_targets_are_marked_not_sendable(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "public-a", "agent_id": "same-send", "agent": "A"}, - {"worker_id": "public-b", "agent_id": "same-send", "agent": "B"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _, workers = fetch_herdr_state(config) - - assert {worker.id for worker in workers} == {"public-a", "public-b"} - assert all((worker.backend_target or {}).get("kind") == "agent_id" for worker in workers) - assert all((worker.backend_target or {}).get("value") == "same-send" for worker in workers) - assert all((worker.backend_target or {}).get("sendable") is False for worker in workers) - assert all( - (worker.backend_target or {}).get("reason") == "duplicate_backend_target" - for worker in workers - ) - assert herdr_cli.assert_unique_sendable_backend_targets(workers) is True - - -def test_duplicate_backend_targets_mark_bindings_unsendable(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "public-a", "agent_id": "same-send", "agent": "A"}, - {"worker_id": "public-b", "agent_id": "same-send", "agent": "B"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _spaces, workers, bindings = fetch_herdr_state(config, include_bindings=True) - - assert {worker.id for worker in workers} == {"public-a", "public-b"} - assert len(bindings) == 2 - assert all(binding.target_kind == "agent_id" for binding in bindings) - assert all(binding.target_value == "same-send" for binding in bindings) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "duplicate_backend_target" for binding in bindings) - assert len({binding.private_fingerprint for binding in bindings}) == 2 - - -def test_bindings_from_workers_marks_duplicate_targets_unsendable() -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - workers = [ - Worker( - id="public-a", - name="A", - status="active", - backend_target={"kind": "agent_id", "value": "same-agent", "sendable": True, "reason": None}, - ), - Worker( - id="public-b", - name="B", - status="active", - backend_target={"kind": "agent_id", "value": "same-agent", "sendable": True, "reason": None}, - ), - ] - - bindings = herdr_cli.bindings_from_workers(config, workers, observed_at="2026-01-01T00:00:00+00:00") - - assert len(bindings) == 2 - assert {binding.worker_id for binding in bindings} == {"public-a", "public-b"} - assert {binding.sendable for binding in bindings} == {False} - assert {binding.reason for binding in bindings} == {"duplicate_backend_target"} - - -def test_duplicate_private_identity_bindings_stay_separate_across_reobserve(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-a", - "agent_id": "same-agent", - "agent": "A", - "workspace_id": "ws-1", - }, - { - "worker_id": "public-b", - "agent_id": "same-agent", - "agent": "B", - "workspace_id": "ws-1", - }, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _spaces, workers, bindings = fetch_herdr_state(config, include_bindings=True) - _spaces2, workers2, bindings2 = fetch_herdr_state( - config, - stored_bindings=bindings, - include_bindings=True, - ) - - assert {worker.id for worker in workers} == {"public-a", "public-b"} - assert {worker.id for worker in workers2} == {"public-a", "public-b"} - assert len(bindings) == 2 - assert len(bindings2) == 2 - assert {binding.sendable for binding in bindings} == {False} - assert {binding.reason for binding in bindings} == {"duplicate_backend_target"} - assert len({binding.private_fingerprint for binding in bindings}) == 2 - assert {binding.private_fingerprint for binding in bindings2} == { - binding.private_fingerprint for binding in bindings - } - - -def test_legacy_collapsed_private_identity_does_not_rewrite_duplicate_public_ids(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - original_private = worker_binding_private_fingerprint( - host_id="testhost", - backend="herdr", - identity_material={ - "agent_id": "same-agent", - "agent_session": None, - "session_id": None, - "space_id": "ws-1", - }, - ) - stored = [ - WorkerBinding( - host_id="testhost", - worker_id="collapsed-public", - worker_fingerprint="collapsed-fp", - backend="herdr", - target_kind="agent_id", - target_value="same-agent", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="2026-01-02T00:00:00+00:00", - private_fingerprint=original_private, - ) - ] - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-a", - "agent_id": "same-agent", - "agent": "A", - "workspace_id": "ws-1", - }, - { - "worker_id": "public-b", - "agent_id": "same-agent", - "agent": "B", - "workspace_id": "ws-1", - }, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _spaces, workers, bindings = fetch_herdr_state( - config, - stored_bindings=stored, - include_bindings=True, - ) - - assert {worker.id for worker in workers} == {"public-a", "public-b"} - assert "collapsed-public" not in {binding.worker_id for binding in bindings} - assert len({binding.private_fingerprint for binding in bindings}) == 2 - - -def test_duplicate_final_send_tokens_across_backend_kinds_are_not_sendable(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "agent-id-worker", "agent_id": "same-send", "agent": "A"}, - {"worker_id": "name-worker", "name": "same-send"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _, workers = fetch_herdr_state(config) - by_id = {worker.id: worker for worker in workers} - - assert by_id["agent-id-worker"].backend_target["kind"] == "agent_id" - assert by_id["name-worker"].backend_target["kind"] == "name" - assert all((worker.backend_target or {}).get("value") == "same-send" for worker in workers) - assert all((worker.backend_target or {}).get("sendable") is False for worker in workers) - assert all( - (worker.backend_target or {}).get("reason") == "duplicate_backend_target" - for worker in workers - ) - assert herdr_cli.assert_unique_sendable_backend_targets(workers) is True - - -def test_name_and_label_backend_fallbacks_are_sendable_only_when_unique(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "name-unique", "name": "NameUnique"}, - {"worker_id": "label-unique", "label": "LabelUnique"}, - {"worker_id": "name-dupe-a", "name": "DupText"}, - {"worker_id": "label-dupe-b", "label": "DupText"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - _, workers = fetch_herdr_state(config) - by_id = {worker.id: worker for worker in workers} - - assert by_id["name-unique"].backend_target == { - "kind": "name", - "value": "NameUnique", - "sendable": True, - "reason": None, - } - assert by_id["label-unique"].backend_target == { - "kind": "label", - "value": "LabelUnique", - "sendable": True, - "reason": None, - } - assert by_id["name-dupe-a"].backend_target["sendable"] is False - assert by_id["name-dupe-a"].backend_target["reason"] == "duplicate_backend_target" - assert by_id["label-dupe-b"].backend_target["sendable"] is False - assert by_id["label-dupe-b"].backend_target["reason"] == "duplicate_backend_target" - - -def test_fetch_herdr_command_observation_reports_healthy_empty(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): {"result": {"panes": []}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", lambda args, cfg: ("ok", _respond(args, responses).stdout and json.loads(_respond(args, responses).stdout))) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is True - assert observation.outcome == "empty_healthy" - assert observation.workers == [] - assert observation.backend_health[0].name == "herdr" - assert observation.backend_health[0].status == "healthy" - assert observation.backend_health[0].outcome == "empty_healthy" - assert observation.backend_health[0].counts == {"spaces": 0, "workers": 0} - - -def test_fetch_herdr_command_observation_degrades_unmatched_agent(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): { - "result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]} - }, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "w-1", "agent_id": "agent-1", "agent": "Coder"} - ] - } - }, - ("pane", "list"): {"result": {"panes": []}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli, - "_run_herdr", - lambda args, cfg: _respond(args, responses), - ) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is False - assert observation.status == "degraded" - assert observation.outcome == "continuity_unavailable" - assert observation.workers == [] - assert observation.bindings == [] - - -def test_fetch_herdr_snapshot_observation_reports_healthy_non_empty(monkeypatch, tmp_path) -> None: - config = Config(host_id="testhost", herdr_bin="herdr", data_dir=tmp_path / "state") - responses = { - ("workspace", "list"): {"result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]}}, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "w-1", - "agent_id": "agent-1", - "terminal_id": "terminal-1", - "agent": "Coder", - } - ] - } - }, - ("pane", "list"): { - "result": { - "panes": [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-1", - "agent": "Coder", - } - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - health = observation.backend_health[0] - - assert [space.id for space in observation.spaces] == ["wR9"] - assert [worker.id for worker in observation.workers] == ["w-1"] - assert observation.workers[0].meta["stable_key"].startswith("wsk1_") - assert health.to_dict() == { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": health.observed_at, - "message": "Herdr observation is healthy", - "counts": {"spaces": 1, "workers": 1}, - } - - -def test_fetch_herdr_snapshot_observation_degrades_unmatched_agent(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): { - "result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]} - }, - ("agent", "list"): { - "result": { - "agents": [ - {"worker_id": "w-1", "agent_id": "agent-1", "agent": "Coder"} - ] - } - }, - ("pane", "list"): {"result": {"panes": []}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli, - "_run_herdr", - lambda args, cfg: _respond(args, responses), - ) - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - - assert [space.id for space in observation.spaces] == ["wR9"] - assert observation.workers == [] - assert observation.bindings == [] - assert observation.backend_health[0].status == "degraded" - assert observation.backend_health[0].outcome == "continuity_unavailable" - - -def test_fetch_herdr_snapshot_observation_reports_healthy_empty(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): {"result": {"panes": []}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - health = observation.backend_health[0] - - assert observation.spaces == [] - assert observation.workers == [] - assert health.status == "healthy" - assert health.outcome == "empty_healthy" - assert health.counts == {"spaces": 0, "workers": 0} - - -def test_fetch_herdr_snapshot_observation_reports_missing_binary() -> None: - config = Config(host_id="testhost", herdr_bin="definitely-not-a-real-herdr-binary") - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - health = observation.backend_health[0] - - assert observation.spaces == [] - assert observation.workers == [] - assert health.status == "unavailable" - assert health.outcome == "missing_binary" - - -@pytest.mark.parametrize( - ("probe_outcome", "expected_status", "expected_outcome"), - [ - ("launch_error", "unavailable", "launch_error"), - ("timeout", "degraded", "timeout"), - ("deadline_exhausted", "degraded", "deadline_exhausted"), - ("malformed_json", "degraded", "malformed_json"), - ("nonzero", "degraded", "nonzero"), - ("unknown", "unknown", "unknown"), - ], -) -def test_fetch_herdr_snapshot_observation_maps_failure_outcomes( - monkeypatch, - probe_outcome: str, - expected_status: str, - expected_outcome: str, -) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - - def fake_probe(args: Sequence[str], cfg: Config) -> tuple[str, Any]: - return probe_outcome, None - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - observation = herdr_cli.fetch_herdr_snapshot_observation(config) - health = observation.backend_health[0] - - assert observation.spaces == [] - assert observation.workers == [] - assert health.status == expected_status - assert health.outcome == expected_outcome - assert health.message - - -def test_herdr_health_mapping_includes_socket_disconnect() -> None: - health = herdr_cli.herdr_backend_health("socket_disconnected") - - assert health.status == "unavailable" - assert health.outcome == "socket_disconnected" - - -def test_cli_snapshot_retains_authenticated_worker_while_pane_probe_recovers( - monkeypatch, - tmp_path, -) -> None: - config = Config( - host_id="pane-recovery", - herdr_bin="herdr", - data_dir=tmp_path / "state", - db_path=tmp_path / "pane-recovery.db", - ) - init_store(config.db_path) - responses = { - ("workspace", "list"): { - "result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]} - }, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-worker", - "agent_id": "private-agent", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "private-terminal", - "agent": "Coder", - } - ] - } - }, - ("pane", "list"): { - "result": { - "panes": [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "private-terminal", - "agent": "Coder", - } - ] - } - }, - } - pane_available = True - - def fake_probe(args: Sequence[str], cfg: Config, *unused: Any) -> tuple[str, Any]: - if tuple(args) == ("pane", "list") and not pane_available: - return "timeout", None - payload = responses.get(tuple(args)) - return ("ok", payload) if payload is not None else ("nonzero", None) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - first = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - first_worker = first.workers[0] - first_binding = list_worker_bindings(config.db_path, config.host_id, backend="herdr") - assert first.backend_health[0].status == "healthy" - assert first_worker.meta["stable_key"].startswith("wsk1_") - - pane_available = False - degraded = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert degraded.workers == first.workers - assert degraded.spaces == first.spaces - assert degraded.backend_health[0].status == "degraded" - assert degraded.backend_health[0].outcome == "timeout" - assert degraded.backend_health[0].counts == {"spaces": 1, "workers": 1} - assert list_worker_bindings(config.db_path, config.host_id, backend="herdr") == first_binding - - pane_available = True - recovered = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert recovered.backend_health[0].status == "healthy" - assert recovered.workers[0].meta["stable_key"] == first_worker.meta["stable_key"] - recovered_binding = list_worker_bindings( - config.db_path, - config.host_id, - backend="herdr", - ) - - healthy_panes = responses[("pane", "list")] - responses[("pane", "list")] = {"result": {"panes": []}} - unmatched = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert unmatched.workers == first.workers - assert unmatched.spaces == first.spaces - assert unmatched.backend_health[0].status == "degraded" - assert unmatched.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings(config.db_path, config.host_id, backend="herdr") - == recovered_binding - ) - - responses[("pane", "list")] = healthy_panes - recovered_again = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert recovered_again.backend_health[0].status == "healthy" - assert recovered_again.workers[0].meta["stable_key"] == first_worker.meta["stable_key"] - - -def test_cli_snapshot_retains_authenticated_worker_while_installation_key_recovers( - monkeypatch, - tmp_path, -) -> None: - config = Config( - host_id="key-recovery", - herdr_bin="herdr", - data_dir=tmp_path / "state", - db_path=tmp_path / "key-recovery.db", - ) - init_store(config.db_path) - responses = { - ("workspace", "list"): { - "result": {"workspaces": [{"workspace_id": "wR9", "label": "Build"}]} - }, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-worker", - "agent_id": "private-agent", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "private-terminal", - "agent": "Coder", - } - ] - } - }, - ("pane", "list"): { - "result": { - "panes": [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "private-terminal", - "agent": "Coder", - } - ] - } - }, - } - - def fake_probe(args: Sequence[str], cfg: Config, *unused: Any) -> tuple[str, Any]: - payload = responses.get(tuple(args)) - return ("ok", payload) if payload is not None else ("nonzero", None) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - first = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - first_worker = first.workers[0] - first_binding = list_worker_bindings(config.db_path, config.host_id, backend="herdr") - marker = config.installation_key_marker_path.read_bytes() - config.installation_key_marker_path.unlink() - - degraded = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert degraded.workers == first.workers - assert degraded.backend_health[0].status == "degraded" - assert degraded.backend_health[0].outcome == "continuity_unavailable" - assert degraded.backend_health[0].message == "Herdr continuity identity is unavailable" - assert degraded.backend_health[0].counts == {"spaces": 1, "workers": 1} - assert json.loads(degraded.to_json())["backend_health"][0]["outcome"] == "continuity_unavailable" - assert list_worker_bindings(config.db_path, config.host_id, backend="herdr") == first_binding - - degraded_again = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - assert degraded_again.workers == first.workers - assert degraded_again.spaces == first.spaces - assert degraded_again.backend_health[0].outcome == "continuity_unavailable" - assert degraded_again.backend_health[0].counts == {"spaces": 1, "workers": 1} - - def timeout_pane_probe( - args: Sequence[str], - cfg: Config, - *unused: Any, - ) -> tuple[str, Any]: - if tuple(args) == ("pane", "list"): - return "timeout", None - return fake_probe(args, cfg, *unused) - - monkeypatch.setattr(herdr_cli, "_probe_herdr", timeout_pane_probe) - alternate_failure = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - assert alternate_failure.workers == first.workers - assert alternate_failure.backend_health[0].outcome == "continuity_unavailable" - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - config.installation_key_marker_path.write_bytes(marker) - os.chmod(config.installation_key_marker_path, 0o600) - recovered = tendwire_cli.observe_public_snapshot(config, store_snapshot=True) - - assert recovered.backend_health[0].status == "healthy" - assert recovered.workers[0].meta["stable_key"] == first_worker.meta["stable_key"] - - -def test_probe_payload_variants_stops_after_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[str, ...]] = [] - - def fake_probe(args: Sequence[str], cfg: Config) -> tuple[str, Any]: - calls.append(tuple(args)) - return "timeout", None - - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - outcome, payload = herdr_cli._probe_payload_variants( - [["agent", "list"], ["agent", "list", "--json"]], - config, - ) - - assert outcome == "timeout" - assert payload is None - assert calls == [("agent", "list")] - - -def test_fetch_herdr_command_observation_stops_fallbacks_after_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[str, ...]] = [] - - def fake_probe(args: Sequence[str], cfg: Config) -> tuple[str, Any]: - calls.append(tuple(args)) - if tuple(args) == ("workspace", "list"): - return "ok", {"result": {"workspaces": []}} - return "timeout", None - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is False - assert observation.outcome == "timeout" - assert observation.workers == [] - assert calls == [("workspace", "list"), ("agent", "list")] - - -def test_fetch_herdr_state_short_circuits_after_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args)) - raise subprocess.TimeoutExpired(cmd=["herdr", *args], timeout=config.herdr_timeout_seconds) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert spaces == [] - assert workers == [] - assert calls == [("workspace", "list")] - - -def test_fetch_herdr_state_returns_spaces_and_skips_worker_fallback_after_agent_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - calls: list[tuple[str, ...]] = [] - - responses = { - ("workspace", "list"): {"result": {"workspaces": [{"workspace_id": "ws-1", "label": "Build"}]}}, - } - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args)) - if tuple(args) == ("workspace", "list"): - return _respond(args, responses) - raise subprocess.TimeoutExpired(cmd=["herdr", *args], timeout=config.herdr_timeout_seconds) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert [space.id for space in spaces] == ["ws-1"] - assert workers == [] - assert calls == [("workspace", "list"), ("agent", "list")] - - -def test_fetch_herdr_state_uses_aggregate_deadline_for_remaining_probe_timeout(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr", herdr_timeout_seconds=1.0) - current_time = [0.0] - calls: list[tuple[tuple[str, ...], float]] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append((tuple(args[1:]), float(kwargs["timeout"]))) - if len(calls) == 1: - current_time[0] += 4.75 - else: - current_time[0] += 0.30 - return subprocess.CompletedProcess(args=args, returncode=2, stdout="", stderr="") - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.time, "monotonic", lambda: current_time[0]) - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert spaces == [] - assert workers == [] - assert calls[0] == (("workspace", "list"), 1.0) - assert calls[1][0] == ("workspace", "list", "--json") - assert 0 < calls[1][1] <= 0.25 - assert len(calls) == 2 - - -def test_fetch_herdr_command_observation_stops_after_aggregate_deadline(monkeypatch) -> None: - config = Config(host_id="testhost", herdr_bin="herdr", herdr_timeout_seconds=1.0) - current_time = [0.0] - calls: list[tuple[str, ...]] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args[1:])) - current_time[0] += 5.1 - return subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps({"result": {"workspaces": []}}), - stderr="", - ) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.time, "monotonic", lambda: current_time[0]) - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is False - assert observation.status == "degraded" - assert observation.outcome == "deadline_exhausted" - assert calls == [("workspace", "list")] - - -@pytest.mark.parametrize("outcome", ["timeout", "malformed_json", "nonzero"]) -def test_fetch_herdr_command_observation_degraded_agent_probe(monkeypatch, outcome: str) -> None: - config = Config(host_id="testhost", herdr_bin="herdr") - - def fake_probe(args: Sequence[str], cfg: Config) -> tuple[str, Any]: - if tuple(args) == ("workspace", "list"): - return "ok", {"result": {"workspaces": []}} - if tuple(args) == ("workspace", "list", "--json"): - return "ok", {"result": {"workspaces": []}} - return outcome, None - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_probe_herdr", fake_probe) - - observation = herdr_cli.fetch_herdr_command_observation(config) - - assert observation.healthy is False - assert observation.status == "degraded" - assert observation.outcome == outcome - assert observation.workers == [] - - -def test_no_flag_workspace_and_agent_lists_preferred_without_json_calls(monkeypatch) -> None: - """Herdr 0.7.0 no-flag envelopes are used before compatibility --json fallbacks.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): { - "result": { - "workspaces": [ - { - "workspace_id": "ws-1", - "label": "Build", - "agent_status": "working", - "active_tab_id": "tab-1", - } - ] - } - }, - ("agent", "list"): { - "result": { - "agents": [ - { - "agent_session": {"value": "sess-1"}, - "agent": "Coder", - "workspace_id": "ws-1", - "agent_status": "done", - "cwd": "/home/dev", - } - ] - } - }, - ("pane", "list"): {"result": {"panes": []}}, - } - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str] | None: - calls.append(tuple(args)) - if "--json" in args: - raise AssertionError("--json fallback should not be called after valid no-flag output") - return _respond(args, responses) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert calls == [("workspace", "list"), ("agent", "list"), ("pane", "list")] - assert len(spaces) == 1 - assert spaces[0].id == "ws-1" - assert spaces[0].name == "Build" - assert spaces[0].status == "active" - assert spaces[0].meta.get("active_tab_id") == "tab-1" - assert spaces[0].meta.get("raw_status") == "working" - assert len(workers) == 1 - assert workers[0].id == "Coder" - assert workers[0].name == "Coder" - assert workers[0].status == "done" - assert workers[0].space_id == "ws-1" - assert workers[0].backend_target is not None - assert workers[0].backend_target["kind"] == "agent" - assert workers[0].backend_target["value"] == "Coder" - assert workers[0].backend_target["sendable"] is True - assert "cwd" not in workers[0].meta - assert "/home/dev" not in json.dumps(workers[0].to_dict()) - assert "raw_status" not in workers[0].meta - - -def test_json_workspace_list_fallback_when_no_flag_fails(monkeypatch) -> None: - """Compatibility --json workspace list is tried when no-flag output fails.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): _completed("usage", returncode=2), - ("workspace", "list", "--json"): { - "result": { - "workspaces": [{"workspace_id": "ws-json", "label": "Compat", "agent_status": "idle"}] - } - }, - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): {"result": {"panes": []}}, - } - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str] | None: - calls.append(tuple(args)) - return _respond(args, responses) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert calls[:2] == [("workspace", "list"), ("workspace", "list", "--json")] - assert len(spaces) == 1 - assert spaces[0].id == "ws-json" - assert spaces[0].name == "Compat" - assert workers == [] - - -def test_json_agent_list_fallback_when_no_flag_is_malformed(monkeypatch) -> None: - """Compatibility --json agent list is tried when no-flag output is malformed.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): _completed("not json"), - ("agent", "list", "--json"): { - "result": { - "agents": [ - { - "agent_session": {"value": "sess-json"}, - "agent": "CompatAgent", - "workspace_id": "ws-1", - } - ] - } - }, - ("pane", "list"): {"result": {"panes": []}}, - } - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str] | None: - calls.append(tuple(args)) - return _respond(args, responses) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("agent", "list", "--json"), - ("pane", "list"), - ] - assert spaces == [] - assert len(workers) == 1 - assert workers[0].id == "CompatAgent" - assert workers[0].name == "CompatAgent" - assert workers[0].space_id == "ws-1" - - -def test_result_envelopes_parse(monkeypatch) -> None: - """result.workspaces, result.agents, and result.panes envelopes parse.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): { - "result": { - "workspaces": [{"workspace_id": "ws-result", "label": "ResultSpace", "agent_status": "idle"}] - } - }, - ("agent", "list", "--json"): { - "result": {"agents": [{"agent_session": {"value": "sess-result"}, "agent": "Agent"}]} - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(spaces) == 1 - assert spaces[0].id == "ws-result" - assert len(workers) == 1 - assert workers[0].id == "Agent" - - -def test_pane_fallback_only_when_agent_list_yields_none(monkeypatch) -> None: - """Pane list fallback runs only when agents are empty and only for agent-bearing panes.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): _completed("", returncode=1), - ("workspace", "list"): _completed("", returncode=1), - ("agent", "list", "--json"): _completed("", returncode=1), - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): { - "result": { - "panes": [ - {"pane_id": "pane-agent", "agent": "Runner", "workspace_id": "ws-1"}, - {"pane_id": "pane-plain", "workspace_id": "ws-1"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(workers) == 1 - assert workers[0].id == "Runner" - assert workers[0].name == "Runner" - assert workers[0].backend_target is not None - assert workers[0].backend_target["kind"] == "pane_id" - assert workers[0].backend_target["value"] == "pane-agent" - assert workers[0].backend_target["sendable"] is True - - -def test_pane_list_enriches_without_adding_unmatched_panes_when_agents_present(monkeypatch) -> None: - """Pane list enriches matching agents without projecting unmatched fallback panes.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): _completed("", returncode=1), - ("workspace", "list"): _completed("", returncode=1), - ("agent", "list", "--json"): { - "result": {"agents": [{"agent_session": {"value": "sess-1"}, "agent": "Agent"}]} - }, - ("pane", "list"): {"result": {"panes": [{"pane_id": "sess-1", "agent": "Agent"}]}}, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(workers) == 1 - assert workers[0].id == "Agent" - - -def test_agent_and_pane_duplicates_emit_one_worker(monkeypatch) -> None: - """A worker described by both agent and pane payloads is deduplicated.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): _completed("", returncode=1), - ("workspace", "list"): _completed("", returncode=1), - ("agent", "list", "--json"): _completed("", returncode=1), - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): { - "result": { - "panes": [ - {"pane_id": "pane-1", "agent": "Runner", "workspace_id": "ws-1"}, - {"pane_id": "pane-1", "agent": "Runner", "workspace_id": "ws-1"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(workers) == 1 - assert workers[0].id == "Runner" - - -def test_repeated_agent_names_with_distinct_ids_remain_distinct(monkeypatch) -> None: - """Workers with the same display name but different session ids are kept.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): _completed("", returncode=1), - ("workspace", "list"): _completed("", returncode=1), - ("agent", "list", "--json"): { - "result": { - "agents": [ - {"agent_session": {"value": "sess-a"}, "agent": "Coder", "workspace_id": "ws-1"}, - {"agent_session": {"value": "sess-b"}, "agent": "Coder", "workspace_id": "ws-1"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert len(workers) == 2 - assert {w.id for w in workers} == {"Coder-1", "Coder-2"} - assert workers[0].id < workers[1].id - assert all((w.backend_target or {}).get("kind") == "agent" for w in workers) - assert all((w.backend_target or {}).get("value") == "Coder" for w in workers) - assert all((w.backend_target or {}).get("sendable") is False for w in workers) - assert all((w.backend_target or {}).get("reason") == "duplicate_backend_target" for w in workers) - - -def test_status_aliases_for_live_herdr(monkeypatch) -> None: - """Working maps to active, done maps to done, and raw_status is preserved.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): { - "result": { - "workspaces": [ - {"workspace_id": "ws-1", "label": "Space", "agent_status": "working"}, - {"workspace_id": "ws-2", "label": "Space2", "agent_status": "responding"}, - ] - } - }, - ("agent", "list", "--json"): { - "result": { - "agents": [ - {"agent_session": {"value": "sess-1"}, "agent": "Agent", "workspace_id": "ws-1", "agent_status": "done"}, - {"agent_session": {"value": "sess-2"}, "agent": "Agent", "workspace_id": "ws-2", "agent_status": "awaiting-input"}, - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - - assert spaces[0].status == "active" - assert spaces[0].meta.get("raw_status") == "working" - assert spaces[1].status == "waiting" - assert spaces[1].meta.get("raw_status") == "responding" - by_space = {w.space_id: w for w in workers} - assert by_space["ws-1"].status == "done" - assert "raw_status" not in by_space["ws-1"].meta - assert by_space["ws-2"].status == "waiting" - assert by_space["ws-2"].meta.get("raw_status") == "awaiting-input" - - -def test_forbidden_connector_fields_stripped_in_herdr_070(monkeypatch) -> None: - """Connector/delivery fields are stripped from live Herdr 0.7.0 payloads.""" - config = Config(host_id="testhost", herdr_bin="herdr") - responses = { - ("workspace", "list", "--json"): { - "result": { - "workspaces": [ - { - "workspace_id": "ws-1", - "label": "Space", - "agent_status": "idle", - "telegram": "leaked", - "chat_id": 123, - "bot.token": "leaked", - "message.id": "leaked", - "backend.target": "leaked", - } - ] - } - }, - ("agent", "list", "--json"): { - "result": { - "agents": [ - { - "agent_session": {"value": "sess-1"}, - "agent": "Agent", - "workspace_id": "ws-1", - "agent_status": "active", - "route": "telegram", - "delivery": {"topic_id": 456}, - "herdres_delivery": {"message_id": 789}, - "herdres.delivery": {"message.id": 789}, - "delivery.route": "telegram", - "telegram.message.id": "leaked", - } - ] - } - }, - } - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", lambda args, cfg: _respond(args, responses)) - - spaces, workers = fetch_herdr_state(config) - snapshot = project_from_observations(config, spaces=spaces, workers=workers) - payload = json.loads(snapshot.to_json()) - - _assert_no_forbidden_fields(payload) - assert "telegram" not in payload["spaces"][0]["meta"] - assert "route" not in payload["workers"][0]["meta"] diff --git a/tests/test_backend_pending.py b/tests/test_backend_pending.py deleted file mode 100644 index 537b0b7..0000000 --- a/tests/test_backend_pending.py +++ /dev/null @@ -1,1560 +0,0 @@ -"""Backend-provided pending prompts: a REAL pane prompt (question + choices, captured by the herdres -pending hook through the turn adapter) flows adapter -> herdr_turns -> backend_pending -> pending.list, -superseding the worker's synthetic attention-derived row.""" -from __future__ import annotations - -import json -import sqlite3 -import pytest -from pathlib import Path - -from tendwire.backends.herdr_turns import _backend_pending_from_turn, _pop_backend_pending -from tendwire.backends.herdr_turns import _pending_observation_from_turn -from tendwire.config import Config, DEFAULT_PENDING_STALE_GRACE_SECONDS, load_config -from tendwire.core.projector import project_from_raw -from tendwire.core.models import Snapshot, WorkerBinding -from tendwire.core.turns import PendingObservation, pending_payload_from_snapshot -from tendwire.daemon import TendwireDaemon -from tendwire.store.sqlite import ( - STORE_SCHEMA_VERSION, - abandon_backend_pending_choice_claim, - apply_backend_pending_observation, - backend_pending_health, - claim_backend_pending_choice, - finish_backend_pending_choice_send, - init_store, - list_backend_pending, - merge_backend_pending, - prune_backend_pending, - pending_payload_from_store, - save_snapshot, - start_backend_pending_choice_send, - upsert_worker_bindings, -) - - -def _decision_turn() -> dict: - return { - "available": True, - "complete": False, - "awaiting_input": True, - "user_text": "which db?", - "pending_decision": { - "decision_id": "toolu_123", - "prompt": "Which database should we use?", - "mode": "buttons", - "options": [ - {"id": "1", "label": "Postgres", "send_text": "Postgres"}, - {"id": "2", "label": "SQLite", "send_text": "SQLite"}, - {"id": "custom", "label": "Tell me differently", "send_text": ""}, - ], - }, - } - - -def test_extract_pending_decision(): - pending = _backend_pending_from_turn(_decision_turn()) - assert pending["question"] == "Which database should we use?" - assert pending["kind"] == "question" - assert [c["label"] for c in pending["choices"]] == ["Postgres", "SQLite", "Tell me differently"] - choice_ids = [c["choice_id"] for c in pending["choices"]] - assert all(choice_id.startswith("choice-") for choice_id in choice_ids) - assert choice_ids == [c["choice_id"] for c in _backend_pending_from_turn(_decision_turn())["choices"]] - assert not ({"1", "2", "custom"} & set(choice_ids)) - # The machine-send payload (send_text) is not published; choices carry only id + label. - assert all("value" not in c for c in pending["choices"]) - # decision_id (internal tool_use_id) is not published in public pending. - assert "decision_id" not in pending["meta"] - decision = pending["meta"]["decision"] - assert decision == { - "decision_ref": decision["decision_ref"], - "kind": "single", - "prompt": "Which database should we use?", - "options": [ - {"ref": "1", "label": "Postgres"}, - {"ref": "2", "label": "SQLite"}, - ], - "multi_select": False, - "question_count": 1, - } - assert decision["decision_ref"].startswith("decision-") - - -def test_extract_plan_approval_kind(): - turn = {"pending_decision": {"decision_id": "t", "prompt": "Approve this plan?", - "options": [{"id": "approve", "label": "Approve", "send_text": "1"}, - {"id": "revise", "label": "Revise", "send_text": ""}]}} - assert _backend_pending_from_turn(turn)["kind"] == "approval" - - -def test_extract_none_without_pending(): - assert _backend_pending_from_turn({"complete": True, "assistant_final_text": "done"}) is None - - -def test_pop_backend_pending_splits(): - content, pending = _pop_backend_pending({"user_text": "hi", "_backend_pending": {"question": "q"}}) - assert content == {"user_text": "hi"} and pending == {"question": "q"} - content, pending = _pop_backend_pending({"_backend_pending": {"question": "q"}}) - assert content is None and pending == {"question": "q"} - - -def test_merge_and_prune_backend_pending(tmp_path: Path): - db = tmp_path / "p.db" - init_store(db) - pending = {"question": "Q?", "kind": "question", "choices": [], "meta": {"source": "backend"}} - assert merge_backend_pending(db, "h", "w1", pending) is True - assert merge_backend_pending(db, "h", "w1", pending) is False # unchanged -> no write - assert list_backend_pending(db, "h") == {"w1": pending} - assert merge_backend_pending(db, "h", "w1", None) is True # answered -> pruned - assert list_backend_pending(db, "h") == {} - - -# --- Regression tests for the PR#3 review fixes --------------------------------------------- - -_GOOGLE_API_KEY_SENTINEL = "AI" + "zaSyD-ExampleKey1234567890abcdefghijk" -_OPENAI_KEY_SENTINEL = "sk-" + "live-SENTINELSECRET123ABC" - -_SENTINELS = [ - _OPENAI_KEY_SENTINEL, # sk- secret token - "/run/user/1000/herdr/sock-abcdef123456", # socket path - "w4V:p1", # pseudo pane id - "/home/example/.ssh/id_rsa", # absolute fs path - "toolu_SENTINELDECISION01", # internal tool_use_id (dropped, not published) - _GOOGLE_API_KEY_SENTINEL, # google api key - "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjMifQ.SflKxwRJSMeKKF2QT4fwpMeJf", # JWT - "alice.jones@internal-corp.example", # email / PII - "10.4.2.9:5432", # internal ip:port - "home/alice/.ssh/id_rsa", # home path without leading slash -] - - -def _leaky_decision_turn() -> dict: - return { - "pending_decision": { - "decision_id": "toolu_SENTINELDECISION01", - "prompt": ( - "Approve running against pane w4V:p1 at /home/example/.ssh/id_rsa via " - f"/run/user/1000/herdr/sock-abcdef123456? Also rotate {_GOOGLE_API_KEY_SENTINEL} " - "and eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjMifQ.SflKxwRJSMeKKF2QT4fwpMeJf; " - "notify alice.jones@internal-corp.example on db 10.4.2.9:5432 at home/alice/.ssh/id_rsa" - ), - "options": [ - { - "id": "approve", - "label": f"Use secret {_OPENAI_KEY_SENTINEL}", - "send_text": _OPENAI_KEY_SENTINEL, - }, - {"id": "postgres", "label": "Postgres", "send_text": "Postgres"}, - {"id": "run", "label": "Deploy to 10.4.2.9:5432", "send_text": "tmux send-keys -t w4V:p1 'rm -rf /home/example/.ssh'"}, - {"id": "shell", "label": "bash -lc 'echo untrusted option'", "send_text": "echo untrusted option"}, - ], - } - } - - -def test_ingestion_redacts_private_data_from_pending(): - """Blocker regression: no private path/pane-id/secret/tool-id survives ingestion.""" - pending = _backend_pending_from_turn(_leaky_decision_turn()) - blob = json.dumps(pending) - for sentinel in _SENTINELS: - assert sentinel not in blob, f"private sentinel leaked into ingested pending: {sentinel!r}" - # A benign label/value is preserved verbatim. - labels = [c["label"] for c in pending["choices"]] - assert "Postgres" in labels - assert "[redacted]" in pending["question"] - assert "bash -lc" not in blob - assert "echo untrusted option" not in blob - - -def test_get_pending_public_json_has_no_private_leak(tmp_path: Path): - """End-to-end: the sentinels must not reach the PUBLIC pending.list payload either.""" - db = tmp_path / "leak.db" - config = Config(host_id="h", db_path=db) - snapshot = project_from_raw(config, workers=[{"id": "worker-1", "name": "claude", "status": "blocked", "space_id": "s1"}]) - init_store(db) - save_snapshot(db, snapshot) - pending = _backend_pending_from_turn(_leaky_decision_turn()) - merge_backend_pending(db, "h", "worker-1", pending) - - public = json.dumps(TendwireDaemon(config).get_pending()) - for sentinel in _SENTINELS: - assert sentinel not in public, f"private sentinel leaked into public pending.list: {sentinel!r}" - # The benign choice survives; the raw-command choice value is dropped by _public_choice_value. - assert "Postgres" in public - - -def test_get_pending_recomputes_content_fingerprint_and_shows_choices(tmp_path: Path): - db = tmp_path / "fp.db" - config = Config(host_id="h", db_path=db) - snapshot = project_from_raw(config, workers=[{"id": "worker-1", "name": "claude", "status": "blocked", "space_id": "s1"}]) - init_store(db) - save_snapshot(db, snapshot) - baseline_fp = pending_payload_from_snapshot(snapshot)["content_fingerprint"] - - merge_backend_pending(db, "h", "worker-1", _backend_pending_from_turn(_decision_turn())) - payload = TendwireDaemon(config).get_pending() - - # The overlaid list moves the change-token (was left stale before the fix). - assert payload["content_fingerprint"] != baseline_fp - interactions = [p for p in payload["pending_interactions"] if p["worker_id"] == "worker-1"] - assert interactions and interactions[0]["question"] == "Which database should we use?" - assert [c["label"] for c in interactions[0]["choices"]] == ["Postgres", "SQLite", "Tell me differently"] - - -def test_prune_backend_pending_reaps_orphaned_workers(tmp_path: Path): - db = tmp_path / "orphan.db" - init_store(db) - live = PendingObservation( - "open_prompt", - question="Q?", - pending_kind="question", - revision_digest="live-revision", - ) - gone = PendingObservation( - "open_prompt", - question="Q?", - pending_kind="question", - revision_digest="gone-revision", - ) - apply_backend_pending_observation( - db, "h", "worker-live", live, binding_private_fingerprint="binding-live" - ) - apply_backend_pending_observation( - db, "h", "worker-gone", gone, binding_private_fingerprint="binding-gone" - ) - assert set(list_backend_pending(db, "h")) == {"worker-live", "worker-gone"} - - reaped = prune_backend_pending(db, "h", {"binding-live"}) - assert reaped == 1 - assert set(list_backend_pending(db, "h")) == {"worker-live"} - - -def _pending_fixture(tmp_path: Path) -> tuple[Path, Config, object]: - db = tmp_path / "pending-v10.db" - config = Config(host_id="h", db_path=db) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-1", - "name": "pane worker", - "status": "blocked", - "space_id": "space-1", - } - ], - ) - init_store(db) - save_snapshot(db, snapshot) - return db, config, snapshot - - -def test_pending_stale_grace_config_default_env_and_validation(monkeypatch) -> None: - monkeypatch.delenv("TENDWIRE_PENDING_STALE_GRACE_SECONDS", raising=False) - assert DEFAULT_PENDING_STALE_GRACE_SECONDS == 30.0 - assert load_config().pending_stale_grace_seconds == 30.0 - monkeypatch.setenv("TENDWIRE_PENDING_STALE_GRACE_SECONDS", "12.5") - assert load_config().pending_stale_grace_seconds == 12.5 - assert load_config(pending_stale_grace_seconds="4").pending_stale_grace_seconds == 4 - for invalid in (0, -1, "nan", "inf"): - try: - Config(pending_stale_grace_seconds=invalid) - except ValueError as exc: - assert "pending_stale_grace_seconds must be a finite positive number" in str(exc) - else: - raise AssertionError(f"accepted invalid grace: {invalid!r}") - - -def test_explicit_pending_transition_table_tombstone_stale_and_expiry( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - opened = _pending_observation_from_turn(_decision_turn()) - t0 = "2026-07-13T00:00:00+00:00" - t1 = "2026-07-13T00:00:01+00:00" - t2 = "2026-07-13T00:00:02+00:00" - t20 = "2026-07-13T00:00:20+00:00" - t32 = "2026-07-13T00:00:32+00:00" - - assert apply_backend_pending_observation(db, "h", "worker-1", opened, observed_at=t0) - fresh = pending_payload_from_store(db, "h") - backend_row = next( - row for row in fresh["pending_interactions"] if row["worker_id"] == "worker-1" - ) - assert backend_row["question"] == "Which database should we use?" - assert backend_row["meta"]["freshness"] == "fresh" - fresh_fingerprint = fresh["content_fingerprint"] - - assert not apply_backend_pending_observation( - db, "h", "worker-1", opened, observed_at=t1 - ) - assert pending_payload_from_store(db, "h")["content_fingerprint"] == fresh_fingerprint - with sqlite3.connect(db) as conn: - assert conn.execute( - "SELECT COUNT(*) FROM backend_pending WHERE host_id = 'h'" - ).fetchone()[0] == 1 - assert conn.execute( - "SELECT COUNT(*) FROM connector_outbox" - ).fetchone()[0] == 0 - - assert apply_backend_pending_observation( - db, "h", "worker-1", PendingObservation("read_failed"), observed_at=t2 - ) - stale = pending_payload_from_store(db, "h") - stale_row = next( - row for row in stale["pending_interactions"] if row["worker_id"] == "worker-1" - ) - assert stale_row["meta"]["freshness"] == "stale" - assert stale["pending_health"] == { - "status": "degraded", - "counts": {"fresh": 0, "stale": 1, "total": 1}, - } - assert stale["content_fingerprint"] != fresh_fingerprint - with sqlite3.connect(db) as conn: - deadline = conn.execute( - "SELECT grace_deadline FROM backend_pending WHERE worker_id = 'worker-1'" - ).fetchone()[0] - - assert not apply_backend_pending_observation( - db, "h", "worker-1", PendingObservation("read_failed"), observed_at=t20 - ) - with sqlite3.connect(db) as conn: - assert conn.execute( - "SELECT grace_deadline FROM backend_pending WHERE worker_id = 'worker-1'" - ).fetchone()[0] == deadline - - assert apply_backend_pending_observation( - db, "h", "worker-1", PendingObservation("read_failed"), observed_at=t32 - ) - expired = pending_payload_from_store(db, "h") - assert expired["pending_health"] == { - "status": "degraded", - "counts": {"fresh": 0, "stale": 1, "total": 1}, - } - assert expired["content_fingerprint"] != stale["content_fingerprint"] - assert any( - row["worker_id"] == "worker-1" - for row in expired["pending_interactions"] - ) - - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("read_succeeded_no_prompt"), - observed_at="2026-07-13T00:01:00+00:00", - ) - tombstoned = pending_payload_from_store(db, "h") - assert not any( - row["worker_id"] == "worker-1" - for row in tombstoned["pending_interactions"] - ) - assert tombstoned["pending_health"] == { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - } - assert _pending_observation_from_turn( - {"pending_decision": {"prompt": "Broken?", "options": "not-a-list"}} - ).kind == "read_succeeded_invalid_prompt" - with sqlite3.connect(db) as conn: - assert conn.execute( - "SELECT observation_state, freshness FROM backend_pending" - ).fetchone() == ("none", "fresh") - - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("worker_authoritatively_absent"), - observed_at="2026-07-13T00:01:01+00:00", - ) - assert backend_pending_health(db, "h")["counts"]["total"] == 0 - - -def test_malformed_source_prompt_exposes_snapshot_fallback_and_health( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - opened = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - opened, - observed_at="2026-07-13T00:00:00+00:00", - ) - malformed = _pending_observation_from_turn( - {"pending_decision": {"prompt": "Broken?", "options": "not-a-list"}} - ) - assert malformed.kind == "read_succeeded_invalid_prompt" - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - malformed, - observed_at="2026-07-13T00:00:01+00:00", - ) - fallback = pending_payload_from_store(db, "h") - assert any( - item["worker_id"] == "worker-1" - for item in fallback["pending_interactions"] - ) - assert all( - item["question"] != "Which database should we use?" - for item in fallback["pending_interactions"] - ) - assert fallback["pending_health"] == { - "status": "degraded", - "counts": {"fresh": 0, "stale": 1, "total": 1}, - } - fingerprint = fallback["content_fingerprint"] - assert not apply_backend_pending_observation( - db, - "h", - "worker-1", - malformed, - observed_at="2026-07-13T00:00:02+00:00", - ) - assert ( - pending_payload_from_store(db, "h")["content_fingerprint"] - == fingerprint - ) - with sqlite3.connect(db) as conn: - assert conn.execute( - "SELECT observation_state, freshness FROM backend_pending" - ).fetchone() == ("invalid", "stale") - - -def test_initial_read_failure_remains_degraded_until_success( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - failure = PendingObservation("read_failed") - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - failure, - observed_at="2026-07-13T00:00:00+00:00", - ) - failed = pending_payload_from_store(db, "h") - assert failed["pending_health"] == { - "status": "degraded", - "counts": {"fresh": 0, "stale": 1, "total": 1}, - } - assert any( - item["worker_id"] == "worker-1" - for item in failed["pending_interactions"] - ) - fingerprint = failed["content_fingerprint"] - assert not apply_backend_pending_observation( - db, - "h", - "worker-1", - failure, - observed_at="2026-07-13T00:00:01+00:00", - ) - assert ( - pending_payload_from_store(db, "h")["content_fingerprint"] - == fingerprint - ) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("read_succeeded_no_prompt"), - observed_at="2026-07-13T00:00:02+00:00", - ) - recovered = pending_payload_from_store(db, "h") - assert recovered["pending_health"] == { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - } - assert not any( - item["worker_id"] == "worker-1" - for item in recovered["pending_interactions"] - ) - - -def test_older_pending_observations_cannot_regress_newer_state( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - opened = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - opened, - observed_at="2026-07-13T00:00:10+00:00", - ) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("read_succeeded_no_prompt"), - observed_at="2026-07-13T00:00:20+00:00", - ) - assert not apply_backend_pending_observation( - db, - "h", - "worker-1", - opened, - observed_at="2026-07-13T00:00:15+00:00", - ) - assert not any( - item["worker_id"] == "worker-1" - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - ) - assert apply_backend_pending_observation( - db, - "h", - "worker-1", - opened, - observed_at="2026-07-13T00:00:30+00:00", - ) - assert not apply_backend_pending_observation( - db, - "h", - "worker-1", - PendingObservation("read_failed"), - observed_at="2026-07-13T00:00:25+00:00", - ) - current = pending_payload_from_store(db, "h") - row = next( - item - for item in current["pending_interactions"] - if item["worker_id"] == "worker-1" - ) - assert row["question"] == "Which database should we use?" - assert row["meta"]["freshness"] == "fresh" - - -def test_revision_bound_opaque_handles_and_two_phase_claim(tmp_path: Path) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="pane_id", - turn_target_value="pane-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private", - ) - upsert_worker_bindings(db, [binding]) - first = _pending_observation_from_turn(_decision_turn()) - changed_turn = _decision_turn() - changed_turn["pending_decision"]["decision_id"] = "toolu_changed_private" - second = _pending_observation_from_turn(changed_turn) - assert [choice.choice_id for choice in first.choices] != [ - choice.choice_id for choice in second.choices - ] - public_blob = json.dumps(_backend_pending_from_turn(changed_turn)) - assert "toolu_changed_private" not in public_blob - assert "send_text" not in public_blob - - assert apply_backend_pending_observation( - db, - "h", - worker.id, - first, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - projected = pending_payload_from_store(db, "h") - row = next(item for item in projected["pending_interactions"] if item["worker_id"] == worker.id) - choice_id = row["choices"][1]["choice_id"] - dry = claim_backend_pending_choice( - db, "h", row["id"], row["fingerprint"], choice_id, - claim=False, observed_at="2026-07-13T00:00:01+00:00", - ) - assert dry.status == "validated" - assert dry.claim_token is None - assert dry.picker_ordinal == 2 - claimed = claim_backend_pending_choice( - db, "h", row["id"], row["fingerprint"], choice_id, - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claimed.status == "claimed" - assert claimed.binding_private_fingerprint == "binding-private" - assert claimed.turn_target_value == "pane-private" - assert claimed.picker_ordinal == 2 - started = start_backend_pending_choice_send( - db, "h", claimed.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ) - assert started.status == "started" - assert started.turn_target_value == "pane-private" - assert started.picker_ordinal == 2 - assert abandon_backend_pending_choice_claim(db, "h", claimed.claim_token) is False - assert finish_backend_pending_choice_send( - db, "h", claimed.claim_token, accepted=False - ) is False - with sqlite3.connect(db) as conn: - assert conn.execute( - """ - SELECT state FROM backend_pending_claims - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ("send_started",) - assert conn.execute( - """ - SELECT observation_state FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ("open",) - assert finish_backend_pending_choice_send( - db, "h", claimed.claim_token, accepted=True - ) is True - assert list_backend_pending(db, "h") == {} - - -def test_accepted_answer_immediately_tombstones_snapshot_fallback( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - fallback = pending_payload_from_store(db, "h") - assert any( - item["worker_id"] == worker.id - for item in fallback["pending_interactions"] - ) - - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private-answer", - turn_target_kind="pane_id", - turn_target_value="pane-private-answer", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private-answer", - ) - upsert_worker_bindings(db, [binding]) - first = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - first, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - overlaid_payload = pending_payload_from_store(db, "h") - overlaid = next( - item - for item in overlaid_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert overlaid["question"] == "Which database should we use?" - assert binding.private_fingerprint not in json.dumps(overlaid_payload) - assert binding.turn_target_value not in json.dumps(overlaid_payload) - - claim = claim_backend_pending_choice( - db, - "h", - overlaid["id"], - overlaid["fingerprint"], - overlaid["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claim.status == "claimed" - assert start_backend_pending_choice_send( - db, - "h", - claim.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "started" - assert finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=True, - observed_at="2026-07-13T00:00:03Z", - ) - - answered = pending_payload_from_store(db, "h") - assert not any( - item["worker_id"] == worker.id - for item in answered["pending_interactions"] - ) - answered_blob = json.dumps(answered) - assert binding.private_fingerprint not in answered_blob - assert binding.turn_target_value not in answered_blob - with sqlite3.connect(db) as conn: - tombstone = conn.execute( - """ - SELECT payload_json, revision_digest, choice_routes_json, - binding_private_fingerprint, observed_turn_target_value, - observation_state, freshness, observed_at, last_success_at, - last_failure_at, grace_deadline, updated_at - FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() - assert tombstone == ( - "{}", - "", - "{}", - binding.private_fingerprint, - binding.turn_target_value, - "none", - "fresh", - "2026-07-13T00:00:03+00:00", - "2026-07-13T00:00:03+00:00", - None, - None, - "2026-07-13T00:00:03+00:00", - ) - assert conn.execute( - """ - SELECT COUNT(*) FROM backend_pending_claims - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == (0,) - - assert not finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=True, - observed_at="2026-07-13T00:00:04+00:00", - ) - with sqlite3.connect(db) as conn: - assert conn.execute( - """ - SELECT observation_state, freshness, observed_at - FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ( - "none", - "fresh", - "2026-07-13T00:00:03+00:00", - ) - - later_turn = _decision_turn() - later_turn["pending_decision"]["decision_id"] = "private-later-answer" - assert apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(later_turn), - observed_at="2026-07-13T00:00:05+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - reopened_payload = pending_payload_from_store(db, "h") - reopened = next( - item - for item in reopened_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert reopened["id"] != overlaid["id"] - assert reopened["fingerprint"] != overlaid["fingerprint"] - assert ( - reopened["choices"][0]["choice_id"] - != overlaid["choices"][0]["choice_id"] - ) - reopened_blob = json.dumps(reopened_payload) - assert binding.private_fingerprint not in reopened_blob - assert binding.turn_target_value not in reopened_blob - - -def test_accepted_finish_tombstones_exact_prompt_after_stale_expiry( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private-expiry", - turn_target_kind="pane_id", - turn_target_value="pane-private-expiry", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private-expiry", - ) - upsert_worker_bindings(db, [binding]) - observation = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - observation, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - pending = next( - item - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - if item["worker_id"] == worker.id - ) - claim = claim_backend_pending_choice( - db, - "h", - pending["id"], - pending["fingerprint"], - pending["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claim.status == "claimed" - assert start_backend_pending_choice_send( - db, - "h", - claim.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "started" - assert apply_backend_pending_observation( - db, - "h", - worker.id, - PendingObservation("read_failed"), - observed_at="2026-07-13T00:00:03+00:00", - stale_grace_seconds=1, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - PendingObservation("read_failed"), - observed_at="2026-07-13T00:00:04+00:00", - stale_grace_seconds=1, - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - with sqlite3.connect(db) as conn: - failed_row = conn.execute( - """ - SELECT observation_state, freshness, revision_digest - FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() - assert failed_row[:2] == ("failed", "stale") - assert failed_row[2] - assert conn.execute( - """ - SELECT state FROM backend_pending_claims - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ("send_started",) - - assert finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=True, - observed_at="2026-07-13T00:00:05+00:00", - ) - answered = pending_payload_from_store(db, "h") - assert not any( - item["worker_id"] == worker.id - for item in answered["pending_interactions"] - ) - with sqlite3.connect(db) as conn: - assert conn.execute( - """ - SELECT observation_state, freshness, - binding_private_fingerprint, observed_turn_target_value - FROM backend_pending - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == ( - "none", - "fresh", - binding.private_fingerprint, - binding.turn_target_value, - ) - assert conn.execute( - """ - SELECT COUNT(*) FROM backend_pending_claims - WHERE host_id = 'h' AND worker_id = ? - """, - (worker.id,), - ).fetchone() == (0,) - - -def test_newer_prompt_racing_accepted_finish_is_not_erased( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private-race", - turn_target_kind="pane_id", - turn_target_value="pane-private-race", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private-race", - ) - upsert_worker_bindings(db, [binding]) - first = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - first, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - initial = next( - item - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - if item["worker_id"] == worker.id - ) - claim = claim_backend_pending_choice( - db, - "h", - initial["id"], - initial["fingerprint"], - initial["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claim.status == "claimed" - assert start_backend_pending_choice_send( - db, - "h", - claim.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "started" - - newer_turn = _decision_turn() - newer_turn["pending_decision"]["decision_id"] = "private-racing-revision" - assert apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(newer_turn), - observed_at="2026-07-13T00:00:03+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - newer_before_finish = next( - item - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - if item["worker_id"] == worker.id - ) - assert newer_before_finish["id"] != initial["id"] - assert not finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=True, - observed_at="2026-07-13T00:00:04+00:00", - ) - newer_after_finish_payload = pending_payload_from_store(db, "h") - newer_after_finish = next( - item - for item in newer_after_finish_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert newer_after_finish == newer_before_finish - private_blob = json.dumps(newer_after_finish_payload) - assert binding.private_fingerprint not in private_blob - assert binding.turn_target_value not in private_blob - - -def test_identical_prompt_on_new_source_mints_new_public_handles( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="pane_id", - turn_target_value="pane-a-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="stable-binding-private", - ) - upsert_worker_bindings(db, [binding]) - observation = _pending_observation_from_turn(_decision_turn()) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - observation, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - before = next( - item - for item in pending_payload_from_store(db, "h")[ - "pending_interactions" - ] - if item["worker_id"] == worker.id - ) - with sqlite3.connect(db) as conn: - conn.execute( - """ - UPDATE worker_bindings - SET turn_target_value = 'pane-b-private' - WHERE private_fingerprint = ? - """, - (binding.private_fingerprint,), - ) - assert apply_backend_pending_observation( - db, - "h", - worker.id, - observation, - observed_at="2026-07-13T00:00:01+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value="pane-b-private", - ) - after_payload = pending_payload_from_store(db, "h") - after = next( - item - for item in after_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - assert after["id"] != before["id"] - assert after["fingerprint"] != before["fingerprint"] - assert after["choices"][0]["choice_id"] != before["choices"][0]["choice_id"] - assert "pane-a-private" not in json.dumps(after_payload) - assert "pane-b-private" not in json.dumps(after_payload) - assert claim_backend_pending_choice( - db, - "h", - before["id"], - before["fingerprint"], - before["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:02+00:00", - ).status == "not_found" - claimed = claim_backend_pending_choice( - db, - "h", - after["id"], - after["fingerprint"], - after["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:02+00:00", - ) - assert claimed.status == "claimed" - assert claimed.turn_target_value == "pane-b-private" - - -def test_claim_and_authoritative_prune_are_bound_to_exact_source_pane( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - source = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-source", - turn_target_kind="pane_id", - turn_target_value="pane-source-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="z-source-binding", - ) - decoy = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-decoy", - turn_target_kind="pane_id", - turn_target_value="pane-decoy-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="a-decoy-binding", - ) - upsert_worker_bindings(db, [decoy, source]) - apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(_decision_turn()), - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=source.private_fingerprint, - observed_turn_target_value=source.turn_target_value, - ) - payload = pending_payload_from_store(db, "h") - row = next( - item for item in payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - with sqlite3.connect(db) as conn: - conn.execute( - """ - UPDATE worker_bindings - SET turn_target_value = 'pane-moved-private' - WHERE private_fingerprint = ? - """, - (source.private_fingerprint,), - ) - assert claim_backend_pending_choice( - db, - "h", - row["id"], - row["fingerprint"], - row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:00.500000+00:00", - ).status == "not_found" - with sqlite3.connect(db) as conn: - conn.execute( - """ - UPDATE worker_bindings - SET turn_target_value = ? - WHERE private_fingerprint = ? - """, - (source.turn_target_value, source.private_fingerprint), - ) - claimed = claim_backend_pending_choice( - db, - "h", - row["id"], - row["fingerprint"], - row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claimed.status == "claimed" - assert claimed.binding_private_fingerprint == source.private_fingerprint - assert claimed.turn_target_value == source.turn_target_value - assert source.turn_target_value not in json.dumps(payload) - with sqlite3.connect(db) as conn: - conn.execute( - """ - UPDATE worker_bindings - SET turn_target_value = 'pane-moved-private' - WHERE private_fingerprint = ? - """, - (source.private_fingerprint,), - ) - assert start_backend_pending_choice_send( - db, - "h", - claimed.claim_token, - observed_at="2026-07-13T00:00:01.500000+00:00", - ).status == "binding_changed" - - assert prune_backend_pending(db, "h", {decoy.private_fingerprint}) == 1 - assert list_backend_pending(db, "h") == {} - assert start_backend_pending_choice_send( - db, - "h", - claimed.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "not_found" - - -@pytest.mark.parametrize("terminal_status", ["closed", "failed"]) -def test_start_send_rejects_latest_terminal_worker_status( - tmp_path: Path, - terminal_status: str, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="pane_id", - turn_target_value="pane-private", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="terminal-binding", - ) - upsert_worker_bindings(db, [binding]) - apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(_decision_turn()), - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - payload = pending_payload_from_store(db, "h") - row = next( - item for item in payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - claimed = claim_backend_pending_choice( - db, - "h", - row["id"], - row["fingerprint"], - row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claimed.status == "claimed" - terminal = project_from_raw( - config, - workers=[ - { - "id": worker.id, - "name": worker.name, - "status": terminal_status, - "space_id": worker.space_id, - } - ], - ) - terminal_payload = terminal.to_dict() - terminal_payload["workers"][0]["fingerprint"] = worker.fingerprint - terminal = Snapshot.from_dict(terminal_payload) - assert terminal.workers[0].fingerprint == worker.fingerprint - save_snapshot(db, terminal) - assert start_backend_pending_choice_send( - db, - "h", - claimed.claim_token, - observed_at="2026-07-13T00:00:02+00:00", - ).status == "binding_changed" - - -def test_presend_claim_lease_reclaims_only_unstarted_owner(tmp_path: Path) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - upsert_worker_bindings( - db, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent", - turn_target_kind="pane_id", - turn_target_value="pane", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="lease-binding", - ) - ], - ) - observation = _pending_observation_from_turn(_decision_turn()) - apply_backend_pending_observation( - db, - "h", - worker.id, - observation, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint="lease-binding", - observed_turn_target_value="pane", - ) - projected = pending_payload_from_store(db, "h") - row = next(item for item in projected["pending_interactions"] if item["worker_id"] == worker.id) - args = (db, "h", row["id"], row["fingerprint"], row["choices"][0]["choice_id"]) - old = claim_backend_pending_choice( - *args, - observed_at="2026-07-13T00:00:01+00:00", - claim_lease_seconds=30, - ) - assert old.status == "claimed" - assert claim_backend_pending_choice( - *args, - observed_at="2026-07-13T00:00:30+00:00", - claim_lease_seconds=30, - ).status == "already_claimed" - replacement = claim_backend_pending_choice( - *args, - observed_at="2026-07-13T00:00:31+00:00", - claim_lease_seconds=30, - ) - assert replacement.status == "claimed" - assert replacement.claim_token != old.claim_token - assert start_backend_pending_choice_send( - db, - "h", - old.claim_token, - observed_at="2026-07-13T00:00:31+00:00", - claim_lease_seconds=30, - ).status == "not_found" - assert start_backend_pending_choice_send( - db, - "h", - replacement.claim_token, - observed_at="2026-07-13T00:00:32+00:00", - claim_lease_seconds=30, - ).status == "started" - assert claim_backend_pending_choice( - *args, - observed_at="2026-07-14T00:00:00+00:00", - claim_lease_seconds=1, - ).status == "already_claimed" - - -def test_new_revision_retires_uncertain_claim_and_malformed_overlay_falls_back( - tmp_path: Path, -) -> None: - db, config, snapshot = _pending_fixture(tmp_path) - worker = snapshot.workers[0] - upsert_worker_bindings( - db, - [ - WorkerBinding( - host_id="h", - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent", - turn_target_kind="pane_id", - turn_target_value="pane", - sendable=True, - observed_at="2026-07-13T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding", - ) - ], - ) - first = _pending_observation_from_turn(_decision_turn()) - apply_backend_pending_observation( - db, - "h", - worker.id, - first, - observed_at="2026-07-13T00:00:00+00:00", - binding_private_fingerprint="binding", - observed_turn_target_value="pane", - ) - payload = pending_payload_from_store(db, "h") - row = next(item for item in payload["pending_interactions"] if item["worker_id"] == worker.id) - claim = claim_backend_pending_choice( - db, "h", row["id"], row["fingerprint"], row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:01+00:00", - ) - assert claim.status == "claimed" - assert start_backend_pending_choice_send( - db, - "h", - claim.claim_token, - observed_at="2026-07-13T00:00:01.500000+00:00", - ).status == "started" - assert not finish_backend_pending_choice_send( - db, - "h", - claim.claim_token, - accepted=False, - ) - changed = _decision_turn() - changed["pending_decision"]["decision_id"] = "private-new-revision" - apply_backend_pending_observation( - db, - "h", - worker.id, - _pending_observation_from_turn(changed), - observed_at="2026-07-13T00:00:02+00:00", - binding_private_fingerprint="binding", - observed_turn_target_value="pane", - ) - assert start_backend_pending_choice_send( - db, "h", claim.claim_token, - observed_at="2026-07-13T00:00:03+00:00", - ).status == "not_found" - changed_payload = pending_payload_from_store(db, "h") - changed_row = next( - item - for item in changed_payload["pending_interactions"] - if item["worker_id"] == worker.id - ) - replacement = claim_backend_pending_choice( - db, - "h", - changed_row["id"], - changed_row["fingerprint"], - changed_row["choices"][0]["choice_id"], - observed_at="2026-07-13T00:00:03.500000+00:00", - ) - assert replacement.status == "claimed" - - with sqlite3.connect(db) as conn: - conn.execute( - "UPDATE backend_pending SET payload_json = ? WHERE worker_id = ?", - ('{"question":"broken","kind":"question","choices":"bad"}', worker.id), - ) - fallback = pending_payload_from_store(db, "h") - assert any(item["worker_id"] == worker.id for item in fallback["pending_interactions"]) - assert all(item["question"] != "broken" for item in fallback["pending_interactions"]) - - -def test_cached_pending_projection_never_migrates_older_store( - tmp_path: Path, -) -> None: - db, _config, _snapshot = _pending_fixture(tmp_path) - with sqlite3.connect(db) as conn: - conn.execute("PRAGMA user_version = 9") - result = pending_payload_from_store(db, "h") - assert result["status"] == "store_unavailable" - assert result["ok"] is False - with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == 9 - - -def test_current_schema_creation_has_exact_binding_and_claim_state(tmp_path: Path) -> None: - db = tmp_path / "current-schema.db" - init_store(db) - with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION == 21 - columns = { - str(row[1]) - for row in conn.execute("PRAGMA table_info(backend_pending)").fetchall() - } - assert { - "revision_digest", - "choice_routes_json", - "binding_private_fingerprint", - "observed_turn_target_value", - "observation_state", - "freshness", - "last_success_at", - "last_failure_at", - "grace_deadline", - "updated_at", - } <= columns - claim_columns = { - str(row[1]) - for row in conn.execute( - "PRAGMA table_info(backend_pending_claims)" - ).fetchall() - } - assert { - "binding_private_fingerprint", - "turn_target_value", - "state", - "send_started_at", - } <= claim_columns - assert "backend_pending_claims" in { - str(row[0]) - for row in conn.execute( - "SELECT name FROM sqlite_master WHERE type = 'table'" - ).fetchall() - } - - -def test_v9_pending_migration_preserves_public_row_but_leaves_it_unrouted( - tmp_path: Path, -) -> None: - db = tmp_path / "legacy-v9.db" - init_store(db) - config = Config(host_id="h", db_path=db) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-legacy", - "name": "legacy worker", - "status": "blocked", - } - ], - ) - save_snapshot(db, snapshot) - legacy_payload = json.dumps( - { - "question": "Legacy approval?", - "kind": "approval", - "choices": [{"choice_id": "choice-0123456789abcdef01234567", "label": "Approve"}], - "meta": {"source": "backend"}, - }, - sort_keys=True, - separators=(",", ":"), - ) - with sqlite3.connect(db) as conn: - conn.execute("DROP TABLE backend_pending_claims") - conn.execute("ALTER TABLE backend_pending RENAME TO backend_pending_v10") - conn.execute( - """ - CREATE TABLE backend_pending ( - host_id TEXT NOT NULL, - worker_id TEXT NOT NULL, - payload_json TEXT NOT NULL, - observed_at TEXT NOT NULL, - PRIMARY KEY (host_id, worker_id) - ) - """ - ) - conn.execute( - "INSERT INTO backend_pending VALUES (?, ?, ?, ?)", - ("h", "worker-legacy", legacy_payload, "2026-07-13T00:00:00+00:00"), - ) - conn.execute("DROP TABLE backend_pending_v10") - conn.execute("PRAGMA user_version = 9") - init_store(db) - with sqlite3.connect(db) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == STORE_SCHEMA_VERSION - migrated = conn.execute( - """ - SELECT payload_json, observation_state, freshness, - choice_routes_json, binding_private_fingerprint, - observed_turn_target_value, last_success_at, updated_at - FROM backend_pending - """ - ).fetchone() - assert migrated[0] == legacy_payload - assert migrated[1:6] == ("open", "fresh", "{}", "", "") - assert migrated[6:] == ( - "2026-07-13T00:00:00+00:00", - "2026-07-13T00:00:00+00:00", - ) - projected = pending_payload_from_store(db, "h") - legacy = next( - row - for row in projected["pending_interactions"] - if row["worker_id"] == "worker-legacy" - ) - assert legacy["question"] == "Legacy approval?" - assert legacy["choices"][0]["label"] == "Approve" diff --git a/tests/test_cli.py b/tests/test_cli.py index 38827ab..06c1199 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,3216 +1,23 @@ -"""Tests for tendwire CLI snapshot JSON output and optional storage.""" - from __future__ import annotations -import io import json -import os -import sqlite3 -import subprocess -import sys -import threading -import time -from datetime import datetime -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest - -from tendwire.backends import herdr_cli -from tendwire.cli import _build_parser, main, observe_public_snapshot -from tendwire.config import Config -from tendwire.core.models import AttentionSignal, Snapshot, SuggestedAction, Worker, WorkerBinding -from tendwire.core.projector import project_from_raw -from tendwire.daemon_api import TendwireDaemonAPI, UnixSocketJSONServer -from tendwire.store.sqlite import ( - SnapshotObservationContext, - append_event, - get_turn_content, - init_store, - latest_snapshot, - list_worker_bindings, - merge_backend_pending, - pending_payload_from_store, - merge_turn_content, - save_snapshot, - turns_payload_from_store, -) - - -@pytest.fixture(autouse=True) -def _isolate_cli_paths(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - private_home = tmp_path / "home" - private_home.mkdir(mode=0o700) - monkeypatch.setenv("HOME", str(private_home)) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "tendwire-data")) - monkeypatch.delenv("TENDWIRE_DB_PATH", raising=False) - - -_PUBLIC_JSON_FORBIDDEN_KEYS = { - "tty", - "pty", - "pid", - "pids", - "process_id", - "process_ids", - "tmux", - "tmux_session", - "tmux_sessions", - "screen_session", - "screen_sessions", - "window_id", - "window_ids", - "tab_id", - "tab_ids", - "pane_id", - "pane_ids", - "terminal_id", - "terminal_ids", - "backend_target", - "backend_targets", - "session_id", - "private", - "private_binding", - "private_bindings", - "private_fingerprint", - "private_fingerprints", - "route", - "routes", - "delivery", - "deliveries", - "connector", - "connectors", - "command", - "command_args", - "command_argv", - "command_line", - "command_payload", - "command_text", - "raw_args", - "raw_argv", - "raw_command", - "raw_command_line", - "shell_command", - "chat_id", - "chat_ids", - "topic_id", - "topic_ids", - "message_id", - "message_ids", - "token", - "tokens", - "secret", - "secrets", - "password", - "passwords", - "credentials", - "cookie", - "auth_token", - "auth_tokens", -} -_PUBLIC_JSON_FORBIDDEN_COMPACT = { - key.replace("_", "") for key in _PUBLIC_JSON_FORBIDDEN_KEYS -} - - -def _assert_no_public_json_forbidden(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - normalized = str(key).lower().replace("-", "_") - assert ( - normalized not in _PUBLIC_JSON_FORBIDDEN_KEYS - and normalized.replace("_", "") not in _PUBLIC_JSON_FORBIDDEN_COMPACT - ), f"forbidden field {path}.{key}" - _assert_no_public_json_forbidden(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_public_json_forbidden(item, f"{path}[{index}]") - -def test_cli_snapshot_json_prints_contract_json_only(capsys) -> None: - code = main( - [ - "--host-id", - "cli-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--json", - ] - ) - captured = capsys.readouterr() +from tendwire.cli import main - assert code == 0 - assert captured.err == "" - payload = json.loads(captured.out) - assert payload["schema_version"] == 2 - assert payload["host_id"] == "cli-host" - assert len(payload["content_fingerprint"]) == 24 - assert {"updated_at", "spaces", "workers", "attention", "backend_health"} <= set(payload) - assert payload["backend_health"][0]["name"] == "herdr" - assert payload["backend_health"][0]["status"] == "unavailable" - assert payload["backend_health"][0]["outcome"] == "missing_binary" - -def test_cli_snapshot_no_herdr_works() -> None: - """Empty snapshot works even when herdr is not installed.""" - code = main(["--herdr-bin", "definitely-not-a-real-herdr-binary", "snapshot", "--json"]) - assert code == 0 - - -def test_cli_snapshot_post_send_timeout_never_observes_source( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - class TimeoutClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, _method: str, _params: dict[str, Any] | None = None) -> dict[str, Any]: - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("post-send timeout must not observe Herdr or mutate the store") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", TimeoutClient) - monkeypatch.setattr("tendwire.cli.observe_public_snapshot", forbidden) - - code = main( - [ - "--socket-path", - str(tmp_path / "daemon.sock"), - "snapshot", - "--json", - ] - ) +def test_snapshot_uses_daemon_and_never_reads_store(capsys, monkeypatch, tmp_path) -> None: + monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path)) + code = main(["snapshot", "--json"]) payload = json.loads(capsys.readouterr().out) - - assert code == 1 - assert payload == { - "schema_version": 2, - "ok": False, - "status": "daemon_timeout", - "error": { - "code": "daemon_timeout", - "message": "Tendwire daemon request timed out", - }, - } - - -def test_cli_socket_group_option_is_daemon_only_and_normalized(monkeypatch) -> None: - captured: list[Config] = [] - - def capture_daemon_config(config: Config) -> int: - captured.append(config) - return 0 - - monkeypatch.delenv("TENDWIRE_SOCKET_GROUP", raising=False) - monkeypatch.setattr("tendwire.cli.cmd_daemon", capture_daemon_config) - - snapshot_args = _build_parser().parse_args(["snapshot"]) - assert not hasattr(snapshot_args, "socket_group") - assert main(["daemon", "--socket-group", " daemon-clients "]) == 0 - assert captured[0].socket_group == "daemon-clients" - - -def test_cli_daemon_startup_conflict_is_clear_and_nonzero( - tmp_path: Path, - capsys, - monkeypatch: pytest.MonkeyPatch, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - def active_socket(_config: Config) -> int: - raise DaemonUnavailable( - "daemon socket is already active: holder is tendwire 0.1.0rc4 " - "(PID 4242); refusing to start tendwire 0.1.0rc5" - ) - - monkeypatch.setattr("tendwire.daemon.run_daemon", active_socket) - - code = main(["daemon", "--db-path", str(tmp_path / "daemon.db")]) - captured = capsys.readouterr() - - assert code == 1 - assert captured.out == "" - assert captured.err == ( - "tendwire daemon 0.1.0rc5: startup failed: " - "daemon socket is already active: holder is tendwire 0.1.0rc4 " - "(PID 4242); refusing to start tendwire 0.1.0rc5\n" - ) - - -def test_cli_turns_json_without_cached_store_is_publicly_unavailable(capsys) -> None: - code = main( - [ - "--host-id", - "turns-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload == { - "schema_version": 1, - "host_id": "turns-host", - "ok": False, - "status": "store_unavailable", - } - - -def test_cli_turns_schema_v2_daemon_request_requires_no_content_fetch( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - calls: list[tuple[str, dict[str, Any]]] = [] - - class FakeDaemonAPIClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - calls.append((method, dict(params or {}))) - return { - "ok": True, - "result": { - "schema_version": 2, - "host_id": "turns-host", - "turns": [ - { - "id": "turn-public", - "assistant_final_text": "short final", - "content": { - "schema_version": 1, - "content_revision": "twrev1.public", - "known_incomplete": False, - "fields": { - "assistant_final_text": { - "availability": "complete", - "inline": True, - "char_length": 11, - "byte_length": 11, - "page_count": 1, - "first_cursor": None, - } - }, - }, - } - ], - }, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - code = main( - [ - "--host-id", - "turns-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--schema-version", - "2", - "--json", - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 0 - assert payload["schema_version"] == 2 - assert payload["turns"][0]["assistant_final_text"] == "short final" - assert payload["turns"][0]["content"]["fields"]["assistant_final_text"]["inline"] is True - assert calls == [ - ( - "turn.list", - { - "schema_version": 2, - "limit": 100, - "cursor": None, - "since": None, - }, - ) - ] - - -def test_cli_turns_v1_upgrade_required_is_json_and_nonzero( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - class FakeDaemonAPIClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - assert method == "turn.list" - assert params == { - "schema_version": 1, - "limit": 100, - "cursor": None, - "since": None, - } - return { - "ok": True, - "result": { - "schema_version": 1, - "ok": False, - "status": "upgrade_required", - "required_turn_schema_version": 2, - "error": { - "code": "upgrade_required", - "message": "turn content requires schema version 2", - }, - }, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - code = main( - [ - "--host-id", - "turns-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["status"] == "upgrade_required" - assert payload["required_turn_schema_version"] == 2 - assert payload["error"]["code"] == "upgrade_required" - - -def test_cli_turns_parser_defaults_bounds_and_exclusive_positions() -> None: - parser = _build_parser() - - defaults = parser.parse_args(["turns"]) - assert defaults.limit == 100 - assert defaults.cursor is None - assert defaults.since is None - - bounded = parser.parse_args(["turns", "--limit", "250", "--cursor", "twlist1.page"]) - assert bounded.limit == 250 - assert bounded.cursor == "twlist1.page" - assert bounded.since is None - - for invalid_limit in ("0", "251", "1.5"): - with pytest.raises(SystemExit): - parser.parse_args(["turns", "--limit", invalid_limit]) - with pytest.raises(SystemExit): - parser.parse_args( - ["turns", "--cursor", "twlist1.page", "--since", "twsince1.new"] - ) - - -def test_cli_turns_definite_unavailable_refreshes_once_then_reads_exact_page( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - daemon_calls: list[tuple[str, dict[str, Any]]] = [] - refresh_calls: list[dict[str, Any]] = [] - store_calls: list[tuple[Any, ...]] = [] - - class UnavailableClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - daemon_calls.append((method, dict(params or {}))) - raise DaemonUnavailable("not listening", request_started=False) - - def refresh(_config: Config, **kwargs: Any) -> dict[str, Any]: - refresh_calls.append(kwargs) - return {"ok": True, "status": "ok", "updated": 1, "attempted": 1} - - def read_page(db_path: Path, host_id: str, **kwargs: Any) -> dict[str, Any]: - store_calls.append((db_path, host_id, kwargs)) - return { - "schema_version": 2, - "host_id": host_id, - "ok": True, - "status": "ok", - "turns": [{"id": "cached-turn"}], - "next_cursor": "twlist1.next", - "since": "twsince1.done", - } - - def forbidden_snapshot(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("turn fallback must not observe a snapshot") - - db_path = tmp_path / "fallback.db" - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", UnavailableClient) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", refresh) - monkeypatch.setattr("tendwire.cli.turns_payload_from_store", read_page) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden_snapshot) - - code = main( - [ - "--host-id", - "fallback-host", - "--herdr-timeout", - "0.75", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--schema-version", - "2", - "--limit", - "7", - "--db-path", - str(db_path), - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 0 - assert payload["turns"] == [{"id": "cached-turn"}] - assert daemon_calls == [ - ( - "turn.list", - {"schema_version": 2, "limit": 7, "cursor": None, "since": None}, - ) - ] - assert refresh_calls == [ - { - "adapter_timeout_seconds": 0.75, - "max_workers": 4, - "total_timeout_seconds": 1.75, - } - ] - assert store_calls == [ - ( - db_path, - "fallback-host", - { - "schema_version": 2, - "limit": 7, - "cursor": None, - "since": None, - "turn_refresh_interval_seconds": 2.0, - "turn_model": os.environ.get("TENDWIRE_TURN_MODEL", "observed"), - }, - ) - ] - - -@pytest.mark.parametrize( - ("position_flag", "position_value"), - [ - ("--cursor", "twlist1.page"), - ("--since", "twsince1.done"), - ], -) -def test_cli_turns_continuation_unavailable_reads_cache_without_refresh( - tmp_path: Path, - capsys, - monkeypatch, - position_flag: str, - position_value: str, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - store_calls: list[dict[str, Any]] = [] - - class UnavailableClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, _method: str, _params: dict[str, Any] | None = None) -> dict[str, Any]: - raise DaemonUnavailable("not listening", request_started=False) - - def forbidden_refresh(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("continuation must never refresh") - - def read_page(_db_path: Path, host_id: str, **kwargs: Any) -> dict[str, Any]: - store_calls.append(kwargs) - return { - "schema_version": 2, - "host_id": host_id, - "ok": True, - "status": "ok", - "turns": [], - "next_cursor": None, - "since": "twsince1.next", - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", UnavailableClient) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden_refresh) - monkeypatch.setattr("tendwire.cli.turns_payload_from_store", read_page) - - code = main( - [ - "--host-id", - "continuation-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--schema-version", - "2", - "--limit", - "9", - position_flag, - position_value, - "--db-path", - str(tmp_path / "cache.db"), - ] - ) - json.loads(capsys.readouterr().out) - - assert code == 0 - assert store_calls == [ - { - "schema_version": 2, - "limit": 9, - "cursor": position_value if position_flag == "--cursor" else None, - "since": position_value if position_flag == "--since" else None, - "turn_refresh_interval_seconds": 2.0, - "turn_model": os.environ.get("TENDWIRE_TURN_MODEL", "observed"), - } - ] - - -@pytest.mark.parametrize( - ("mode", "expected"), - [ - ( - "timeout", - { - "schema_version": 1, - "ok": False, - "status": "daemon_timeout", - "error": { - "code": "daemon_timeout", - "message": "Tendwire daemon request timed out", - }, - }, - ), - ( - "protocol", - { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", - }, - }, - ), - ( - "malformed", - { - "schema_version": 1, - "ok": False, - "status": "daemon_protocol_error", - "error": { - "code": "daemon_protocol_error", - "message": "Tendwire daemon returned an invalid response", - }, - }, - ), - ( - "daemon_error", - { - "schema_version": 1, - "ok": False, - "status": "error", - "result": None, - "error": { - "code": "invalid_params", - "message": "invalid parameters", - }, - }, - ), - ], -) -def test_cli_turns_reachable_or_ambiguous_failure_never_reads_sources( - tmp_path: Path, - capsys, - monkeypatch, - mode: str, - expected: dict[str, Any], -) -> None: - from tendwire.daemon_api import DaemonProtocolError, DaemonUnavailable - - calls = 0 - - class FailingClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "turn.list" - assert params == { - "schema_version": 1, - "limit": 100, - "cursor": None, - "since": None, - } - if mode == "timeout": - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) - if mode == "protocol": - raise DaemonProtocolError("invalid frame", request_started=True) - if mode == "malformed": - return {"ok": True, "result": ["not", "a", "mapping"]} - return expected - - def forbidden_read(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("ambiguous/reachable failures must not read any source") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FailingClient) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden_read) - monkeypatch.setattr("tendwire.cli.turns_payload_from_store", forbidden_read) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden_read) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden_read) - - code = main( - [ - "--host-id", - "failure-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--db-path", - str(tmp_path / "cache.db"), - ] - ) - captured = capsys.readouterr() - - assert code == 1 - assert captured.err == "" - assert json.loads(captured.out) == expected - assert calls == 1 - - -@pytest.mark.parametrize("status", ["invalid_cursor", "cursor_expired", "since_expired"]) -def test_cli_turns_reachable_invalid_or_expired_page_is_authoritative( - tmp_path: Path, - capsys, - monkeypatch, - status: str, -) -> None: - result = { - "schema_version": 2, - "ok": False, - "status": status, - } - - class AuthoritativeClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - assert method == "turn.list" - assert params == { - "schema_version": 2, - "limit": 11, - "cursor": "twlist1.requested", - "since": None, - } - return {"ok": True, "result": result} - - def forbidden_read(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("reachable page result must be authoritative") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", AuthoritativeClient) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden_read) - monkeypatch.setattr("tendwire.cli.turns_payload_from_store", forbidden_read) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden_read) - - code = main( - [ - "--host-id", - "authoritative-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turns", - "--schema-version", - "2", - "--limit", - "11", - "--cursor", - "twlist1.requested", - "--db-path", - str(tmp_path / "cache.db"), - ] - ) - assert code == 1 - assert json.loads(capsys.readouterr().out) == result - - -def test_cli_turns_traverses_over_one_mib_across_bounded_daemon_pages( - tmp_path: Path, - capsys, -) -> None: - socket_path = tmp_path / "paged-turns.sock" - snapshot = Snapshot(host_id="paged-host", updated_at="2026-01-01T00:00:00+00:00") - canonical_text = { - f"turn-{index:03d}": ( - f"\n# Turn {index:03d}\n" - + (f"exact-{index:03d}-αβγ\n" * 650) - + "終\n" - ) - for index in range(110) - } - ordered_ids = list(canonical_text) - seen_requests: list[dict[str, Any]] = [] - - def get_turns( - *, - schema_version: int, - limit: int, - cursor: str | None, - since: str | None, - ) -> dict[str, Any]: - seen_requests.append( - { - "schema_version": schema_version, - "limit": limit, - "cursor": cursor, - "since": since, - } - ) - start = 0 if cursor is None else 55 - page_ids = ordered_ids[start : start + 55] - return { - "schema_version": 2, - "host_id": "paged-host", - "ok": True, - "status": "ok", - "turns": [ - { - "id": turn_id, - "assistant_final_text": canonical_text[turn_id], - "content": { - "schema_version": 1, - "content_revision": f"twrev1.{turn_id}", - "known_incomplete": False, - "fields": { - "assistant_final_text": { - "availability": "complete", - "inline": True, - "char_length": len(canonical_text[turn_id]), - "byte_length": len(canonical_text[turn_id].encode("utf-8")), - "page_count": 1, - "first_cursor": None, - } - }, - }, - } - for turn_id in page_ids - ], - "next_cursor": "twlist1.second" if start == 0 else None, - "since": "twsince1.complete" if start else None, - } - - api = TendwireDaemonAPI( - get_snapshot=lambda: snapshot, - get_health=lambda: {"schema_version": 1, "status": "ok"}, - submit_command=lambda _params: {}, - get_turns=get_turns, - ) - server = UnixSocketJSONServer( - socket_path, - api.dispatch, - accept_timeout_seconds=0.05, - ) - thread = threading.Thread(target=server.serve_forever) - thread.start() - pages: list[dict[str, Any]] = [] - encoded_sizes: list[int] = [] - cursor: str | None = None - try: - deadline = time.monotonic() + 2 - while not server.listening and time.monotonic() < deadline: - time.sleep(0.01) - while True: - argv = [ - "--host-id", - "paged-host", - "--socket-path", - str(socket_path), - "turns", - "--schema-version", - "2", - "--limit", - "55", - ] - if cursor is not None: - argv += ["--cursor", cursor] - assert main(argv) == 0 - captured = capsys.readouterr() - assert captured.err == "" - encoded = captured.out.encode("utf-8") - assert len(encoded) < 1024 * 1024 - encoded_sizes.append(len(encoded)) - page = json.loads(captured.out) - pages.append(page) - cursor = page["next_cursor"] - if cursor is None: - break - finally: - server.close() - thread.join(timeout=2) - - listed = [turn for page in pages for turn in page["turns"]] - assert sum(encoded_sizes) > 1024 * 1024 - assert [turn["id"] for turn in listed] == ordered_ids - assert { - turn["id"]: turn["assistant_final_text"] for turn in listed - } == canonical_text - assert seen_requests == [ - { - "schema_version": 2, - "limit": 55, - "cursor": None, - "since": None, - }, - { - "schema_version": 2, - "limit": 55, - "cursor": "twlist1.second", - "since": None, - }, - ] - assert not thread.is_alive() - - -def test_cli_turn_content_get_preserves_exact_page_and_params( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - page_text = "\n " + ("界" * 20_000) + "\r\n " - calls: list[tuple[str, dict[str, Any]]] = [] - - class FakeDaemonAPIClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - calls.append((method, dict(params or {}))) - return { - "ok": True, - "result": { - "schema_version": 1, - "ok": True, - "status": "ok", - "turn_id": "turn-public", - "content_revision": "twrev1.public", - "field": "assistant_final_text", - "availability": "complete", - "segment_id": "twseg1.public", - "index": 1, - "count": 2, - "text": page_text, - "segment_char_length": len(page_text), - "segment_byte_length": len(page_text.encode("utf-8")), - "total_char_length": 40_000, - "total_byte_length": 120_000, - "next_cursor": None, - }, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - code = main( - [ - "--host-id", - "turns-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turn", - "content", - "get", - "--json", - "--turn-id", - "turn-public", - "--revision", - "twrev1.public", - "--field", - "assistant_final_text", - "--cursor", - "twcur1.public", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert captured.err == "" - assert payload["turn_id"] == "turn-public" - assert payload["segment_id"] == "twseg1.public" - assert payload["text"] == page_text - assert calls == [ - ( - "turn.content.get", - { - "schema_version": 1, - "turn_id": "turn-public", - "content_revision": "twrev1.public", - "field": "assistant_final_text", - "cursor": "twcur1.public", - }, - ) - ] - - -@pytest.mark.parametrize("with_db_path", [False, True]) -@pytest.mark.parametrize( - ("error_code", "details"), - [ - ("internal_error", {"type": "RuntimeError"}), - ("response_too_large", {"max_response_bytes": 1024 * 1024}), - ], -) -def test_cli_turn_content_preserves_reachable_daemon_errors_without_store_fallback( - tmp_path: Path, - capsys, - monkeypatch, - with_db_path: bool, - error_code: str, - details: dict[str, Any], -) -> None: - direct_calls: list[str] = [] - original_error = { - "code": error_code, - "message": f"daemon {error_code}", - "details": details, - } - - class FakeDaemonAPIClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - assert method == "turn.content.get" - return { - "schema_version": 1, - "ok": False, - "status": "error", - "result": None, - "error": original_error, - } - - def forbidden_store_call(*_args: Any, **_kwargs: Any) -> dict[str, Any]: - direct_calls.append("store") - raise AssertionError("reachable daemon errors must not fall back to the store") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - monkeypatch.setattr("tendwire.store.sqlite.init_store", forbidden_store_call) - monkeypatch.setattr("tendwire.store.sqlite.get_turn_content", forbidden_store_call) - argv = [ - "--host-id", - "turns-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "turn", - "content", - "get", - "--json", - "--turn-id", - "turn-public", - "--revision", - "twrev1.public", - "--field", - "assistant_final_text", - ] - if with_db_path: - argv += ["--db-path", str(tmp_path / "direct.db")] - - code = main(argv) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["status"] == "error" - assert payload["error"] == original_error - assert direct_calls == [] - - -def test_cli_long_content_pages_match_direct_store_and_daemon( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "long-content.db" - socket_path = tmp_path / "long-content.sock" - config = Config(host_id="long-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "Worker", "status": "active"}], - ) - canonical = ( - "# Exact heading\n\n" - + ("safe-value αβγ\n- nested-looking item\n```text\ncode\n```\n" * 30_000) - )[:1_100_000] + "終" - init_store(db_path) - save_snapshot(db_path, snapshot) - assert merge_turn_content( - db_path, - "long-host", - "worker-1", - { - "source_turn_id": "cli-long-content-source", - "user_text": "short prompt", - "assistant_final_text": canonical, - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:00+00:00", - ) == 1 - listed = turns_payload_from_store( - db_path, - "long-host", - snapshot=snapshot, - schema_version=2, - ) - turn = listed["turns"][0] - revision = turn["content"]["content_revision"] - descriptor = turn["content"]["fields"]["assistant_final_text"] - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", - lambda _config, **_kwargs: {"ok": True}, - ) + assert payload["status"] == "daemon_unavailable" + assert not (tmp_path / "tendwire.db").exists() - v1_code = main( - [ - "--host-id", - "long-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--db-path", - str(db_path), - "--json", - ] - ) - v1_payload = json.loads(capsys.readouterr().out) - v2_code = main( - [ - "--host-id", - "long-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--db-path", - str(db_path), - "--schema-version", - "2", - "--json", - ] - ) - v2_payload = json.loads(capsys.readouterr().out) - assert v1_code == 1 - assert v1_payload["status"] == "upgrade_required" - assert v1_payload["required_turn_schema_version"] == 2 - assert v2_code == 0 - assert v2_payload["schema_version"] == 2 - assert descriptor["inline"] is False - assert descriptor["char_length"] == len(canonical) - assert descriptor["byte_length"] == len(canonical.encode("utf-8")) - assert descriptor["page_count"] > 1 - - def fetch_pages(*, socket: Path | None, direct_db: Path | None) -> list[dict[str, Any]]: - pages: list[dict[str, Any]] = [] - cursor: str | None = None - while True: - argv = ["--host-id", "long-host"] - if socket is not None: - argv += ["--socket-path", str(socket)] - argv += [ - "turn", - "content", - "get", - "--json", - "--turn-id", - turn["id"], - "--revision", - revision, - "--field", - "assistant_final_text", - ] - if direct_db is not None: - argv += ["--db-path", str(direct_db)] - if cursor is not None: - argv += ["--cursor", cursor] - assert main(argv) == 0 - captured = capsys.readouterr() - assert captured.err == "" - page = json.loads(captured.out) - assert len(json.dumps(page, ensure_ascii=False).encode("utf-8")) < 1024 * 1024 - pages.append(page) - next_cursor = page["next_cursor"] - if next_cursor is None: - return pages - assert next_cursor not in {item.get("next_cursor") for item in pages[:-1]} - cursor = next_cursor - - direct_pages = fetch_pages(socket=None, direct_db=db_path) - bad_cursor_code = main( - [ - "--host-id", - "long-host", - "turn", - "content", - "get", - "--json", - "--turn-id", - turn["id"], - "--revision", - revision, - "--field", - "assistant_final_text", - "--cursor", - "twcur1.tampered", - "--db-path", - str(db_path), - ] - ) - bad_cursor_payload = json.loads(capsys.readouterr().out) - assert bad_cursor_code == 1 - assert bad_cursor_payload["status"] == "invalid_cursor" - - api = TendwireDaemonAPI( - get_snapshot=lambda: snapshot, - get_health=lambda: {"schema_version": 1, "status": "ok"}, - submit_command=lambda _params: {}, - get_turn_content=lambda params: get_turn_content( - db_path, - "long-host", - turn_id=params["turn_id"], - content_revision=params["content_revision"], - field=params["field"], - cursor=params.get("cursor"), - schema_version=params.get("schema_version", 1), - ), - ) - server = UnixSocketJSONServer( - socket_path, - api.dispatch, - accept_timeout_seconds=0.05, - ) - thread = threading.Thread(target=server.serve_forever) - thread.start() - try: - deadline = time.monotonic() + 2 - while not server.listening and time.monotonic() < deadline: - time.sleep(0.01) - daemon_pages = fetch_pages(socket=socket_path, direct_db=None) - daemon_bad_code = main( - [ - "--host-id", - "long-host", - "--socket-path", - str(socket_path), - "turn", - "content", - "get", - "--json", - "--turn-id", - turn["id"], - "--revision", - revision, - "--field", - "assistant_final_text", - "--cursor", - "twcur1.tampered", - ] - ) - daemon_bad_payload = json.loads(capsys.readouterr().out) - assert daemon_bad_code == 1 - assert daemon_bad_payload == bad_cursor_payload - finally: - server.close() - thread.join(timeout=2) - - assert daemon_pages == direct_pages - assert "".join(page["text"] for page in direct_pages) == canonical - assert [page["index"] for page in direct_pages] == list(range(len(direct_pages))) - assert all(page["count"] == len(direct_pages) for page in direct_pages) - assert not thread.is_alive() - - -def test_cli_short_v1_compatibility_then_known_incomplete_refusal( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "content-compatibility.db" - config = Config(host_id="compat-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "Worker", "status": "active"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - assert merge_turn_content( - db_path, - "compat-host", - "worker-1", - { - "source_turn_id": "cli-short-content-source", - "user_text": " short prompt\n", - "assistant_final_text": "\n short final ", - "complete": True, - }, - ) == 1 - monkeypatch.setattr( - "tendwire.cli.refresh_structured_turn_content", - lambda _config, **_kwargs: {"ok": True}, - ) - common = [ - "--host-id", - "compat-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--db-path", - str(db_path), - "--json", - ] - - short_v1_code = main(common) - short_v1 = json.loads(capsys.readouterr().out) - short_v2_code = main([*common, "--schema-version", "2"]) - short_v2 = json.loads(capsys.readouterr().out) - short_turn = short_v2["turns"][0] - - assert short_v1_code == 0 - assert short_v1["schema_version"] == 1 - assert short_v1["turns"][0]["assistant_final_text"] == "\n short final " - assert short_v1["turns"][0]["user_text"] == " short prompt\n" - assert "content" not in short_v1["turns"][0] - assert short_v2_code == 0 - assert short_turn["assistant_final_text"] == "\n short final " - assert short_turn["content"]["fields"]["assistant_final_text"]["inline"] is True - - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - UPDATE turn_content_revisions - SET final_state = 'known_incomplete' - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - """, - ("compat-host", short_turn["id"]), - ) - - incomplete_v1_code = main(common) - incomplete_v1 = json.loads(capsys.readouterr().out) - incomplete_v2_code = main([*common, "--schema-version", "2"]) - incomplete_v2 = json.loads(capsys.readouterr().out) - revision = incomplete_v2["turns"][0]["content"]["content_revision"] - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - UPDATE turn_content_revisions - SET content_revision = ? - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - """, - (revision, "compat-host", short_turn["id"]), - ) - content_code = main( - [ - "--host-id", - "compat-host", - "turn", - "content", - "get", - "--json", - "--turn-id", - short_turn["id"], - "--revision", - revision, - "--field", - "assistant_final_text", - "--db-path", - str(db_path), - ] - ) - content_error = json.loads(capsys.readouterr().out) - - assert incomplete_v1_code == 1 - assert incomplete_v1["status"] == "upgrade_required" - assert incomplete_v1["required_turn_schema_version"] == 2 - assert incomplete_v2_code == 0 - incomplete_field = incomplete_v2["turns"][0]["content"]["fields"]["assistant_final_text"] - assert incomplete_field["availability"] == "known_incomplete" - assert incomplete_field["inline"] is False - assert "assistant_final_text" not in incomplete_v2["turns"][0] - assert content_code == 1 - assert content_error["status"] == "content_known_incomplete" - - -def test_cli_pending_missing_store_is_fixed_and_never_observes_sources( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("pending fallback must not observe Herdr or source state") - - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden) - - code = main( - [ - "--host-id", - "pending-host", - "pending", - "--json", - "--db-path", - str(tmp_path / "missing.db"), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload == { - "schema_version": 1, - "host_id": "pending-host", - "ok": False, - "status": "store_unavailable", - "pending_interactions": [], - "backend_health": [], - "pending_health": { - "status": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - }, - } - _assert_no_public_json_forbidden(payload) - - -@pytest.mark.parametrize("mode", ["success", "error"]) -def test_cli_pending_structured_daemon_result_or_error_is_authoritative( - tmp_path: Path, - capsys, - monkeypatch, - mode: str, -) -> None: - success = { - "schema_version": 1, - "host_id": "authoritative-pending", - "pending_interactions": [], - "backend_health": [], - "pending_health": { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - }, - "content_fingerprint": "a" * 24, - } - error = { - "schema_version": 1, - "ok": False, - "status": "error", - "result": None, - "error": { - "code": "invalid_params", - "message": "invalid pending request", - }, - } - calls = 0 - - class AuthoritativeClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "pending.list" - assert params == {} - if mode == "success": - return {"ok": True, "result": success} - return error - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("authoritative daemon response must not read fallback state") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", AuthoritativeClient) - monkeypatch.setattr("tendwire.cli.pending_payload_from_store", forbidden) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden) - - code = main( - [ - "--host-id", - "authoritative-pending", - "--socket-path", - str(tmp_path / "daemon.sock"), - "pending", - "--json", - "--db-path", - str(tmp_path / "cache.db"), - ] - ) +def test_attention_uses_daemon_and_never_reads_store(capsys, monkeypatch, tmp_path) -> None: + monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path)) + code = main(["attention", "--json"]) payload = json.loads(capsys.readouterr().out) - - assert calls == 1 - assert payload == (success if mode == "success" else error) - assert code == (0 if mode == "success" else 1) - - -@pytest.mark.parametrize( - ("mode", "expected_status"), - [ - ("timeout", "daemon_timeout"), - ("protocol", "daemon_protocol_error"), - ("malformed", "daemon_protocol_error"), - ("post_send_unavailable", "daemon_protocol_error"), - ], -) -def test_cli_pending_post_send_failure_never_reads_or_retries( - tmp_path: Path, - capsys, - monkeypatch, - mode: str, - expected_status: str, -) -> None: - from tendwire.daemon_api import DaemonProtocolError, DaemonUnavailable - - calls = 0 - - class FailingClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "pending.list" - assert params == {} - if mode == "timeout": - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) - if mode == "protocol": - raise DaemonProtocolError("invalid frame", request_started=True) - if mode == "post_send_unavailable": - raise DaemonUnavailable("connection lost", request_started=True) - return {"ok": True, "result": ["not", "a", "mapping"]} - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("post-send failure must not read fallback state") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FailingClient) - monkeypatch.setattr("tendwire.cli.pending_payload_from_store", forbidden) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden) - - code = main( - [ - "--host-id", - "failed-pending", - "--socket-path", - str(tmp_path / "daemon.sock"), - "pending", - "--json", - "--db-path", - str(tmp_path / "cache.db"), - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 1 - assert calls == 1 - assert payload == { - "schema_version": 1, - "ok": False, - "status": expected_status, - "error": { - "code": expected_status, - "message": ( - "Tendwire daemon request timed out" - if expected_status == "daemon_timeout" - else "Tendwire daemon returned an invalid response" - ), - }, - } - - -@pytest.mark.parametrize( - ("mode", "expected_status"), - [ - ("timeout", "daemon_timeout"), - ("protocol", "daemon_protocol_error"), - ("post_send_unavailable", "daemon_protocol_error"), - ], -) -def test_cli_connector_post_send_failure_never_mutates_store_fallback( - tmp_path: Path, - capsys, - monkeypatch, - mode: str, - expected_status: str, -) -> None: - from tendwire.daemon_api import DaemonProtocolError, DaemonUnavailable - - calls = 0 - - class FailingClient: - def __init__( - self, - _socket_path: Any, - *, - timeout_seconds: float, - **_kwargs: Any, - ) -> None: - assert timeout_seconds == 30.0 - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "connector.poll" - assert params == { - "name": "turn-final", - "limit": 1, - "lease_seconds": 60, - } - if mode == "timeout": - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) - if mode == "protocol": - raise DaemonProtocolError( - "invalid frame", - request_started=True, - ) - raise DaemonUnavailable( - "connection lost", - request_started=True, - ) - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError( - "post-send connector failure must not execute store fallback" - ) - - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - FailingClient, - ) - monkeypatch.setattr("tendwire.store.sqlite.init_store", forbidden) - - code = main( - [ - "--host-id", - "connector-timeout-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "connector", - "poll", - "--db-path", - str(tmp_path / "cache.db"), - "--name", - "turn-final", - "--limit", - "1", - "--lease-seconds", - "60", - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 1 - assert calls == 1 - assert payload == { - "schema_version": 1, - "ok": False, - "status": expected_status, - "host_id": "connector-timeout-host", - "name": "turn-final", - "error": { - "code": expected_status, - "message": ( - "Tendwire daemon request timed out" - if expected_status == "daemon_timeout" - else "Tendwire daemon returned an invalid response" - ), - }, - } - - -def test_cli_connector_pre_send_unavailable_uses_store_fallback( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - db_path = tmp_path / "connector-fallback.db" - - class UnavailableClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - assert method == "connector.poll" - raise DaemonUnavailable( - "not listening", - request_started=False, - ) - - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - UnavailableClient, - ) - - code = main( - [ - "--host-id", - "connector-fallback-host", - "--socket-path", - str(tmp_path / "missing.sock"), - "connector", - "poll", - "--db-path", - str(db_path), - "--name", - "turn-final", - "--limit", - "1", - "--lease-seconds", - "60", - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 0 - assert payload["schema_version"] == 1 - assert payload["ok"] is True - assert payload["host_id"] == "connector-fallback-host" - assert payload["name"] == "turn-final" - assert payload["items"] == [] - - -def test_cli_store_hooks_print_json_only_and_support_dry_run(tmp_path: Path, capsys) -> None: - db_path = tmp_path / "store-cli.db" - init_store(db_path) - append_event( - db_path, - "store-cli", - "private.event", - {"pane_id": "sentinel-private-pane", "raw_payload": "sentinel-private-raw"}, - observed_at="2026-01-01T00:00:00+00:00", - ) - append_event( - db_path, - "store-cli", - "public.event", - {"safe": "kept"}, - observed_at="9999-01-09T00:00:00+00:00", - ) - - status_code = main(["--host-id", "store-cli", "store", "status", "--db-path", str(db_path)]) - status_captured = capsys.readouterr() - status_payload = json.loads(status_captured.out) - - tail_code = main( - [ - "--host-id", - "store-cli", - "store", - "events-tail", - "--db-path", - str(db_path), - "--limit", - "5", - ] - ) - tail_captured = capsys.readouterr() - tail_payload = json.loads(tail_captured.out) - - cleanup_code = main( - [ - "--host-id", - "store-cli", - "store", - "cleanup", - "--db-path", - str(db_path), - "--retention-days", - "7", - "--dry-run", - ] - ) - cleanup_captured = capsys.readouterr() - cleanup_payload = json.loads(cleanup_captured.out) - - missing_code = main( - [ - "--host-id", - "store-cli", - "store", - "status", - "--db-path", - str(tmp_path / "missing.db"), - ] - ) - missing_captured = capsys.readouterr() - missing_payload = json.loads(missing_captured.out) - - with sqlite3.connect(str(db_path)) as conn: - event_count = conn.execute("SELECT COUNT(*) FROM events WHERE host_id = ?", ("store-cli",)).fetchone()[0] - - assert status_code == 0 - assert tail_code == 0 - assert cleanup_code == 0 - assert missing_code == 1 - assert status_captured.err == tail_captured.err == cleanup_captured.err == missing_captured.err == "" - assert status_payload["counts"]["events"] == 2 - assert tail_payload["events"] - assert "sentinel-private" not in json.dumps(tail_payload) - assert "payload_json" not in json.dumps(tail_payload) - assert cleanup_payload["dry_run"] is True - assert cleanup_payload["retention"]["deleted"] == 1 - assert "last_examined_id" not in json.dumps(cleanup_payload) - assert event_count == 2 - assert missing_payload["status"] == "store_unavailable" - - -@pytest.mark.parametrize("timed_out", [False, True]) -def test_cli_pending_pre_send_unavailable_uses_durable_overlay_only( - tmp_path: Path, - capsys, - monkeypatch, - timed_out: bool, -) -> None: - from tendwire.daemon_api import DaemonUnavailable - - db_path = tmp_path / "pending-fallback.db" - snapshot = Snapshot( - host_id="projection-cli", - updated_at="2026-01-01T00:00:00+00:00", - workers=[ - Worker( - id="worker-1", - name="Worker One", - status="pending", - space_id="space-1", - summary="human approval required before continuing", - meta={ - "needs_human": True, - "pane_id": "sentinel-cli-pane", - }, - backend_target={ - "kind": "agent_id", - "value": "sentinel-cli-target", - "sendable": True, - }, - ) - ], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-01T00:00:00+00:00", - } - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - merge_backend_pending( - db_path, - snapshot.host_id, - "worker-1", - { - "question": "Which durable option?", - "kind": "choice", - "choices": [ - {"choice_id": "safe", "label": "Safe"}, - { - "choice_id": "private", - "label": "sentinel-cli-private", - "value": "sentinel-cli-command", - }, - ], - "meta": {"source": "backend", "pane_id": "sentinel-cli-pane"}, - }, - ) - expected = pending_payload_from_store(db_path, snapshot.host_id) - calls = 0 - - class UnavailableClient: - def __init__(self, _socket_path: Any, **_kwargs: Any) -> None: - pass - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - nonlocal calls - calls += 1 - assert method == "pending.list" - assert params == {} - raise DaemonUnavailable( - "not listening", - timed_out=timed_out, - request_started=False, - ) - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("durable pending fallback must not observe Herdr") - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", UnavailableClient) - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - monkeypatch.setattr("tendwire.cli.refresh_structured_turn_content", forbidden) - - code = main( - [ - "--host-id", - snapshot.host_id, - "--socket-path", - str(tmp_path / "missing.sock"), - "pending", - "--json", - "--db-path", - str(db_path), - ] - ) - payload = json.loads(capsys.readouterr().out) - - assert code == 0 - assert calls == 1 - assert payload == expected - assert payload["pending_interactions"][0]["question"] == "Which durable option?" - assert "sentinel-cli" not in json.dumps(payload, sort_keys=True) - _assert_no_public_json_forbidden(payload) - - -def test_cli_pending_durable_snapshot_strips_raw_command_action_material( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "raw-command-pending.db" - snapshot = Snapshot( - host_id="raw-command-cli", - updated_at="2026-01-01T00:00:00+00:00", - workers=[ - Worker( - id="worker-1", - name="Worker One", - status="waiting", - space_id="space-1", - summary="waiting for action", - ) - ], - attention=[ - AttentionSignal( - kind="worker_status", - severity="warning", - status="waiting", - reason="Choose next action", - source="worker:worker-1", - updated_at="2026-01-01T00:00:00+00:00", - suggested_actions=[ - SuggestedAction( - command="sentinel-cli-safe-looking-command-alias", - params={ - "safe_choice": "kept", - "commandLine": "sentinel-cli-command-line", - "terminal_id": "sentinel-cli-terminal", - "backendTarget": "sentinel-cli-backend", - "session-id": "sentinel-cli-session", - "token": "sentinel-cli-token", - "secret": "sentinel-cli-secret", - }, - ) - ], - meta={ - "worker_id": "worker-1", - "space_id": "space-1", - "needs_human": True, - }, - host_id="raw-command-cli", - ) - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - - def forbidden(*_args: Any, **_kwargs: Any) -> Any: - raise AssertionError("pending projection must not observe current state") - - monkeypatch.setattr("tendwire.cli._current_public_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", forbidden) - - code = main( - [ - "--host-id", - snapshot.host_id, - "pending", - "--json", - "--db-path", - str(db_path), - ] - ) - payload = json.loads(capsys.readouterr().out) - encoded = json.dumps(payload, sort_keys=True) - - assert code == 0 - assert payload["pending_interactions"][0]["choices"] == [ - { - "choice_id": payload["pending_interactions"][0]["choices"][0]["choice_id"], - "label": "Action", - } - ] - assert "sentinel-cli-" not in encoded - _assert_no_public_json_forbidden(payload) - - -def test_cli_snapshot_json_reports_healthy_empty_herdr(capsys, monkeypatch) -> None: - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): {"result": {"agents": []}}, - ("pane", "list"): {"result": {"panes": []}}, - } - - def _fake_run_herdr(args, cfg): - if tuple(args) in responses: - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps(responses[tuple(args)]), - stderr="", - ) - return subprocess.CompletedProcess(args=list(args), returncode=1, stdout="", stderr="") - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", _fake_run_herdr) - - code = main(["--host-id", "cli-empty", "--herdr-bin", "herdr", "snapshot", "--json"]) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert payload["spaces"] == [] - assert payload["workers"] == [] - assert payload["backend_health"][0]["status"] == "healthy" - assert payload["backend_health"][0]["outcome"] == "empty_healthy" - assert payload["backend_health"][0]["counts"] == {"spaces": 0, "workers": 0} - - -def test_cli_snapshot_store_persists_printed_snapshot(tmp_path: Path, capsys) -> None: - db_path = tmp_path / "cli.db" - code = main( - [ - "--host-id", - "cli-store", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--db-path", - str(db_path), - "--json", - "--store", - ] - ) - captured = capsys.readouterr() - - assert code == 0 - payload = json.loads(captured.out) - assert captured.err == "" - restored = latest_snapshot(db_path) - assert restored is not None - assert restored.host_id == "cli-store" - assert restored.content_fingerprint == payload["content_fingerprint"] - - -@pytest.mark.parametrize( - ("outcome", "has_workers", "expected_authority"), - [ - ("healthy_non_empty", True, "complete"), - ("empty_healthy", False, "complete"), - ("missing_binary", False, "none"), - ("timeout", False, "none"), - ("malformed_json", False, "none"), - ("continuity_unavailable", True, "none"), - ("unknown", False, "none"), - ], -) -def test_cli_snapshot_persistence_passes_explicit_observation_authority( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - outcome: str, - has_workers: bool, - expected_authority: str, -) -> None: - db_path = tmp_path / f"{outcome}.db" - init_store(db_path) - config = Config(host_id=f"cli-{outcome}", db_path=db_path) - observed_at = "2026-01-01T00:00:00+00:00" - workers = [Worker(id="worker-1", name="Worker One", status="active")] if has_workers else [] - health = herdr_cli.herdr_backend_health( - outcome, - observed_at=observed_at, - workers=workers, - ) - observation = SimpleNamespace( - spaces=[], - workers=workers, - bindings=[], - backend_health=[health], - ) - captured: list[SnapshotObservationContext] = [] - captured_atomic: list[tuple[list[WorkerBinding], str | None, bool, bool]] = [] - captured_turn_models: list[str] = [] - - monkeypatch.setattr( - "tendwire.cli.fetch_herdr_snapshot_observation", - lambda _config, *, stored_bindings: observation, - ) - - def _capture_save( - _db_path: Path, - _snapshot: Snapshot, - *, - turn_model: str, - observation: SnapshotObservationContext, - worker_bindings: list[WorkerBinding], - binding_backend: str | None, - binding_observation_authoritative: bool, - binding_workers_present: bool, - ) -> bool: - captured_turn_models.append(turn_model) - captured.append(observation) - captured_atomic.append( - ( - worker_bindings, - binding_backend, - binding_observation_authoritative, - binding_workers_present, - ) - ) - return True - - monkeypatch.setattr("tendwire.store.sqlite.save_snapshot", _capture_save) - - observe_public_snapshot(config, store_snapshot=True) - - assert len(captured) == 1 - assert captured[0].authority == expected_authority - assert captured[0].observed_at == observed_at - assert captured_atomic == [ - ([], "herdr", health.status == "healthy", bool(workers)) - ] - assert captured_turn_models == [config.turn_model] - - -def test_rejected_stale_snapshot_does_not_persist_stale_worker_bindings( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "rejected-stale-binding.db" - init_store(db_path) - config = Config(host_id="cli-stale-binding", db_path=db_path) - worker = Worker(id="worker-stale", name="Worker Stale", status="active") - health = herdr_cli.herdr_backend_health( - "healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - workers=[worker], - ) - observation = SimpleNamespace( - spaces=[], - workers=[worker], - bindings=[{"worker_id": worker.id, "turn_target_value": "stale-target"}], - backend_health=[health], - ) - monkeypatch.setattr( - "tendwire.cli.fetch_herdr_snapshot_observation", - lambda _config, *, stored_bindings: observation, - ) - monkeypatch.setattr( - "tendwire.store.sqlite.save_snapshot", - lambda *_args, **_kwargs: False, - ) - - def _forbidden_binding_persistence(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("rejected snapshot must not update private bindings") - - monkeypatch.setattr( - "tendwire.cli._persist_binding_observation", - _forbidden_binding_persistence, - ) - - observed = observe_public_snapshot(config, store_snapshot=True) - assert observed.host_id == config.host_id - - -def test_cli_atomic_snapshot_binding_write_serializes_delayed_older_observer( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - from tendwire.store import sqlite as store_sqlite - - db_path = tmp_path / "cli-atomic-binding-race.db" - config = Config(host_id="cli-atomic-binding", db_path=db_path) - init_store(db_path) - worker = Worker(id="worker-atomic", name="Worker Atomic", status="active") - - def binding(target: str, observed_at: str) -> WorkerBinding: - return WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value=target, - sendable=True, - observed_at=observed_at, - private_fingerprint="same-private-owner", - ) - - observations = { - "older-observer": SimpleNamespace( - spaces=[], - workers=[worker], - bindings=[binding("older-private-target", "2026-01-01T00:00:00+00:00")], - backend_health=[ - herdr_cli.herdr_backend_health( - "healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - workers=[worker], - ) - ], - ), - "newer-observer": SimpleNamespace( - spaces=[], - workers=[worker], - bindings=[binding("newer-private-target", "2026-01-01T00:00:01+00:00")], - backend_health=[ - herdr_cli.herdr_backend_health( - "healthy_non_empty", - observed_at="2026-01-01T00:00:01+00:00", - workers=[worker], - ) - ], - ), - } - monkeypatch.setattr( - "tendwire.cli.fetch_herdr_snapshot_observation", - lambda _config, *, stored_bindings: observations[threading.current_thread().name], - ) - original_upsert = store_sqlite._upsert_worker_bindings_conn - older_inside_transaction = threading.Event() - release_older = threading.Event() - - def delayed_upsert(conn: sqlite3.Connection, bindings: Any) -> int: - binding_list = list(bindings) - if binding_list and binding_list[0].target_value == "older-private-target": - older_inside_transaction.set() - assert release_older.wait(timeout=10) - return original_upsert(conn, binding_list) - - monkeypatch.setattr(store_sqlite, "_upsert_worker_bindings_conn", delayed_upsert) - errors: list[BaseException] = [] - - def observe() -> None: - try: - observe_public_snapshot(config, store_snapshot=True) - except BaseException as exc: - errors.append(exc) - - older = threading.Thread(target=observe, name="older-observer") - newer = threading.Thread(target=observe, name="newer-observer") - older.start() - assert older_inside_transaction.wait(timeout=10) - newer.start() - time.sleep(0.05) - release_older.set() - older.join(timeout=10) - newer.join(timeout=10) - - assert not errors - assert not older.is_alive() and not newer.is_alive() - stored = list_worker_bindings( - db_path, - config.host_id, - backend="herdr", - include_expired=True, - ) - assert len(stored) == 1 - assert stored[0].target_value == "newer-private-target" - - -def test_cli_legacy_observation_cannot_claim_complete_authority( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "legacy-observation.db" - init_store(db_path) - config = Config(host_id="cli-legacy", db_path=db_path) - worker = Worker(id="worker-1", name="Worker One", status="blocked") - captured: list[SnapshotObservationContext] = [] - captured_atomic: list[tuple[list[WorkerBinding], str | None]] = [] - captured_turn_models: list[str] = [] - - monkeypatch.setattr( - "tendwire.cli.fetch_herdr_state", - lambda _config, **_kwargs: ([], [worker]), - ) - - def _capture_save( - _db_path: Path, - _snapshot: Snapshot, - *, - turn_model: str, - observation: SnapshotObservationContext, - worker_bindings: list[WorkerBinding], - binding_backend: str | None, - binding_observation_authoritative: bool, - binding_workers_present: bool, - ) -> bool: - captured_turn_models.append(turn_model) - captured.append(observation) - captured_atomic.append((worker_bindings, binding_backend)) - return True - - monkeypatch.setattr("tendwire.store.sqlite.save_snapshot", _capture_save) - - observe_public_snapshot(config, store_snapshot=True) - - assert len(captured) == 1 - assert captured[0].authority == "none" - assert captured_atomic == [([], "herdr")] - assert captured_turn_models == [config.turn_model] - - -def test_cli_attention_json_reads_store_backed_lifecycle( - tmp_path: Path, - capsys, -) -> None: - db_path = tmp_path / "attention.db" - socket_path = tmp_path / "absent.sock" - config = Config(host_id="cli-attention", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-1", - "name": "Worker One", - "status": "blocked", - "meta": { - "safe": "kept", - "pane_id": "sentinel-private-pane", - "terminalId": "sentinel-private-terminal", - "backendTarget": "sentinel-private-backend", - "authToken": "sentinel-private-token", - }, - } - ], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-01T00:00:00+00:00", - "counts": {"workers": 1}, - } - ], - timestamp=datetime.fromisoformat("2026-01-01T00:00:00+00:00"), - ) - save_snapshot( - db_path, - snapshot, - observation=SnapshotObservationContext( - authority="complete", - observed_at="2026-01-01T00:00:00+00:00", - ), - ) - - code = main( - [ - "--host-id", - "cli-attention", - "--socket-path", - str(socket_path), - "attention", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert captured.err == "" - assert payload["host_id"] == "cli-attention" - assert payload["attention"][0]["lifecycle_status"] == "open" - assert payload["attention"][0]["first_seen_at"] == "2026-01-01T00:00:00+00:00" - assert payload["attention"][0]["signal_count"] == 1 - assert len(payload["attention"]) == 1 - assert not { - "family_key", - "generation", - "first_missing_at", - "missing_observation_count", - "last_accepted_at", - "last_observation_key", - "max_notified_severity_rank", - }.intersection(payload["attention"][0]) - assert "sentinel-private" not in json.dumps(payload, sort_keys=True) - _assert_no_public_json_forbidden(payload) - - -def test_cli_attention_json_falls_back_to_snapshot_when_store_is_unavailable( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "missing.db" - socket_path = tmp_path / "absent.sock" - - def _fake_herdr_state(config): - return [], [ - Worker( - id="worker-1", - name="Worker One", - status="blocked", - meta={"pane_id": "sentinel-private-pane"}, - ) - ] - - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - - code = main( - [ - "--host-id", - "cli-attention-fallback", - "--socket-path", - str(socket_path), - "attention", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert captured.err == "" - assert len(payload["attention"]) == 1 - assert payload["attention"][0]["status"] == "blocked" - assert "first_seen_at" not in payload["attention"][0] - assert "sentinel-private" not in json.dumps(payload, sort_keys=True) - _assert_no_public_json_forbidden(payload) - - -def test_cli_public_json_does_not_emit_connector_private_store_rows( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "connector-private.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "public-host", - "sentinel-connector-private", - "sentinel-delivery-key", - "queued", - '{"safe":"kept"}', - '{"chat_id":"sentinel-chat","route":"sentinel-route"}', - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - outbox_id = int(conn.execute("SELECT last_insert_rowid()").fetchone()[0]) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - outbox_id, - "public-host", - "sentinel-connector-private", - "sentinel-delivery-key", - 1, - "delivered", - '{"ok":true}', - '{"message_id":"sentinel-message","token":"sentinel-token"}', - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - - payloads: list[dict[str, Any]] = [] - - snapshot_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--json", - "--store", - "--db-path", - str(db_path), - ] - ) - snapshot_captured = capsys.readouterr() - payloads.append(json.loads(snapshot_captured.out)) - - turns_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "turns", - "--json", - ] - ) - turns_captured = capsys.readouterr() - payloads.append(json.loads(turns_captured.out)) - - pending_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "pending", - "--json", - "--db-path", - str(db_path), - ] - ) - pending_captured = capsys.readouterr() - payloads.append(json.loads(pending_captured.out)) - - doctor_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "doctor", - "--json", - ] - ) - doctor_captured = capsys.readouterr() - payloads.append(json.loads(doctor_captured.out)) - - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "read_snapshot"})), - ) - command_code = main( - [ - "--host-id", - "public-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - command_captured = capsys.readouterr() - payloads.append(json.loads(command_captured.out)) - - with sqlite3.connect(str(db_path)) as conn: - private_counts = ( - conn.execute("SELECT COUNT(*) FROM connector_outbox").fetchone()[0], - conn.execute("SELECT COUNT(*) FROM connector_deliveries").fetchone()[0], - ) - - encoded = json.dumps(payloads, sort_keys=True) - assert snapshot_code == 0 - assert turns_code == 1 - assert pending_code == 0 - assert doctor_code == 1 - assert command_code == 0 - assert private_counts == (1, 1) - assert "sentinel-" not in encoded - - -def test_cli_snapshot_store_persists_private_bindings_outside_snapshot_payload( - tmp_path: Path, - capsys, - monkeypatch, -) -> None: - db_path = tmp_path / "bindings.db" - responses = { - ("workspace", "list"): { - "result": { - "workspaces": [ - {"workspace_id": "wA", "label": "Bindings"} - ] - } - }, - ("agent", "list"): { - "result": { - "agents": [ - { - "worker_id": "public-worker", - "agent_id": "agent-secret", - "agent": "Worker", - "workspace_id": "wA", - "pane_id": "wA:p1", - } - ] - } - }, - ("pane", "list"): { - "result": { - "panes": [ - { - "workspace_id": "wA", - "pane_id": "wA:p1", - "terminal_id": "terminal-secret", - "agent": "Worker", - } - ] - } - }, - } - - def _fake_run_herdr(args, cfg): - if tuple(args) in responses: - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps(responses[tuple(args)]), - stderr="", - ) - return subprocess.CompletedProcess(args=list(args), returncode=1, stdout="", stderr="") - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", _fake_run_herdr) - - code = main( - [ - "--host-id", - "cli-bindings", - "--herdr-bin", - "herdr", - "snapshot", - "--db-path", - str(db_path), - "--json", - "--store", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - bindings = list_worker_bindings(db_path, "cli-bindings", backend="herdr") - - assert code == 0 - assert len(bindings) == 1 - assert bindings[0].worker_id == "public-worker" - assert bindings[0].target_kind == "agent_id" - assert bindings[0].target_value == "agent-secret" - encoded = json.dumps(payload) - assert "agent-secret" not in encoded - assert "wA:p1" not in encoded - assert "target_kind" not in encoded - - -def test_cli_module_invocation() -> None: - """python -m tendwire.cli snapshot --json works.""" - env = os.environ.copy() - env["PYTHONPATH"] = os.path.join(os.path.dirname(__file__), "..", "src") - result = subprocess.run( - [ - sys.executable, - "-m", - "tendwire.cli", - "--host-id", - "module-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--json", - ], - capture_output=True, - text=True, - check=False, - env=env, - ) - assert result.returncode == 0, result.stderr - turn_model = env.get("TENDWIRE_TURN_MODEL", "observed").strip().lower() - expected_stderr = ( - "" - if turn_model == "observed" - else f"turn_model={turn_model} is a compatibility alias and behaves as observed\n" - ) - assert result.stderr == expected_stderr - payload = json.loads(result.stdout) - assert payload["schema_version"] == 2 - assert payload["host_id"] == "module-host" - assert len(payload["content_fingerprint"]) == 24 - - -def test_cli_snapshot_with_live_shaped_herdr_fixtures(capsys, monkeypatch) -> None: - """CLI emits schema v2 JSON with non-empty spaces and workers from Herdr fixtures.""" - - def _fake_run_herdr(args, cfg): - if tuple(args) == ("workspace", "list", "--json"): - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps({ - "result": { - "workspaces": [ - { - "workspace_id": "wA", - "label": "CLI Space", - "agent_status": "working", - "focused": True, - } - ] - } - }), - stderr="", - ) - if tuple(args) == ("agent", "list", "--json"): - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps({ - "result": { - "agents": [ - { - "agent_session": {"value": "sess-cli"}, - "agent": "CLI Agent", - "workspace_id": "wA", - "pane_id": "wA:p1", - "agent_status": "executing", - "cwd": "/tmp", - } - ] - } - }), - stderr="", - ) - if tuple(args) == ("pane", "list"): - return subprocess.CompletedProcess( - args=list(args), - returncode=0, - stdout=json.dumps({ - "result": { - "panes": [ - { - "workspace_id": "wA", - "pane_id": "wA:p1", - "terminal_id": "terminal-cli", - "agent": "CLI Agent", - "agent_session": {"value": "sess-cli"}, - "agent_status": "executing", - } - ] - } - }), - stderr="", - ) - return subprocess.CompletedProcess(args=list(args), returncode=1, stdout="", stderr="") - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", _fake_run_herdr) - - code = main(["--host-id", "cli-live", "--herdr-bin", "herdr", "snapshot", "--json"]) - captured = capsys.readouterr() - - assert code == 0 - assert captured.err == "" - payload = json.loads(captured.out) - assert payload["schema_version"] == 2 - assert payload["host_id"] == "cli-live" - assert len(payload["spaces"]) == 1 - assert payload["spaces"][0]["id"] == "wA" - assert payload["spaces"][0]["status"] == "active" - assert len(payload["workers"]) == 1 - assert payload["workers"][0]["id"] == "CLI Agent" - assert payload["workers"][0]["status"] == "active" - assert payload["backend_health"][0]["name"] == "herdr" - assert payload["backend_health"][0]["status"] == "healthy" - assert payload["backend_health"][0]["outcome"] == "healthy_non_empty" - assert payload["backend_health"][0]["counts"] == {"spaces": 1, "workers": 1} - assert "agent_session" not in json.dumps(payload) - assert "sess-cli" not in json.dumps(payload) - - -def test_cli_store_compact_parser_requires_exactly_one_mode() -> None: - parser = _build_parser() - parsed = parser.parse_args( - [ - "store", - "compact", - "--db-path", - "private.db", - "--dry-run", - "--snapshot-retention-days", - "9", - "--snapshot-retention-count", - "17", - "--batch-size", - "3", - ] - ) - - assert parsed.store_action == "compact" - assert parsed.compact_dry_run is True - assert parsed.compact_execute is False - assert parsed.snapshot_retention_days == 9 - assert parsed.snapshot_retention_count == 17 - assert parsed.snapshot_batch_size == 3 - with pytest.raises(SystemExit): - parser.parse_args(["store", "compact"]) - with pytest.raises(SystemExit): - parser.parse_args( - ["store", "compact", "--dry-run", "--execute"] - ) - - -@pytest.mark.parametrize( - "extra", - [ - ["--execute"], - ["--execute", "--acknowledge-offline"], - ["--execute", "--backup-path", "private-backup.db"], - ["--dry-run", "--acknowledge-offline"], - ["--dry-run", "--backup-path", "private-backup.db"], - ["--dry-run", "--batch-size", "0"], - ], -) -def test_cli_store_compact_rejects_invalid_authority_as_one_json_object( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - extra: list[str], -) -> None: - db_path = tmp_path / "must-not-be-opened.db" - - code = main( - [ - "store", - "compact", - "--db-path", - str(db_path), - *extra, - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["status"] == "invalid_request" - assert payload["command"] == "store.compact" - assert not db_path.exists() - - -def test_cli_store_compact_dry_run_is_read_only_json_and_skips_generic_repair( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "compact-dry-run.db" - init_store(db_path) - before = tuple( - sorted( - ( - path.name, - path.stat().st_size, - path.stat().st_mtime_ns, - ) - for path in tmp_path.iterdir() - ) - ) - - def forbidden_repair(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("compact must not run generic permission repair") - - monkeypatch.setattr("tendwire.cli.repair_config_state", forbidden_repair) - code = main( - [ - "store", - "compact", - "--db-path", - str(db_path), - "--dry-run", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - after = tuple( - sorted( - ( - path.name, - path.stat().st_size, - path.stat().st_mtime_ns, - ) - for path in tmp_path.iterdir() - ) - ) - - assert code == 0 - assert captured.err == "" - assert payload["status"] == "dry_run" - assert payload["ok"] is True - assert payload["backup"]["created"] is False - assert after == before - - -def test_cli_store_compact_execute_success_is_public_safe_and_retains_backup( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], -) -> None: - db_path = tmp_path / "sentinel-private-store-name.db" - backup_path = tmp_path / "sentinel-private-backup-name.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - conn.execute("CREATE TABLE compact_private (value TEXT NOT NULL)") - conn.execute( - "INSERT INTO compact_private (value) VALUES (?)", - ("sentinel-private-payload",), - ) - - code = main( - [ - "store", - "compact", - "--db-path", - str(db_path), - "--execute", - "--acknowledge-offline", - "--backup-path", - str(backup_path), - "--snapshot-retention-days", - "7", - "--snapshot-retention-count", - "2", - "--batch-size", - "1", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - encoded = json.dumps(payload, sort_keys=True) - - assert code == 0 - assert captured.err == "" - assert payload["status"] == "completed" - assert payload["command"] == "store.compact" - assert payload["backup"] == { - "required": True, - "created": True, - "verified": True, - } - assert backup_path.is_file() - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - "SELECT value FROM compact_private" - ).fetchone()[0] == "sentinel-private-payload" - assert conn.execute("PRAGMA quick_check").fetchone()[0] == "ok" - for private_value in ( - str(db_path), - db_path.name, - str(backup_path), - backup_path.name, - "sentinel-private-payload", - ): - assert private_value not in encoded - - -def test_cli_store_cleanup_passes_snapshot_policy_overrides( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "cleanup-policy.db" - init_store(db_path) - captured_options: dict[str, Any] = {} - - def capture_maintenance( - _db_path: Path, - _host_id: str, - **kwargs: Any, - ) -> dict[str, Any]: - captured_options.update(kwargs) - return {"schema_version": 1, "ok": True, "status": "ok"} - - monkeypatch.setattr( - "tendwire.cli.run_store_maintenance", - capture_maintenance, - ) - code = main( - [ - "store", - "cleanup", - "--db-path", - str(db_path), - "--dry-run", - "--acknowledged-final-retention-days", - "17", - "--acknowledged-final-retention-count", - "29", - "--snapshot-retention-days", - "11", - "--snapshot-retention-count", - "23", - "--snapshot-batch-size", - "5", - ] - ) - captured = capsys.readouterr() - - assert code == 0 - assert captured.err == "" - assert json.loads(captured.out)["ok"] is True - assert captured_options["dry_run"] is True - assert captured_options["acknowledged_final_retention_days"] == 17 - assert captured_options["acknowledged_final_retention_count"] == 29 - assert captured_options["snapshot_retention_days"] == 11 - assert captured_options["snapshot_retention_count"] == 23 - assert captured_options["snapshot_batch_size"] == 5 - - -def test_cli_store_status_passes_configured_maintenance_policy( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "status-policy.db" - init_store(db_path) - monkeypatch.setenv("TENDWIRE_SNAPSHOT_RETENTION_DAYS", "19") - monkeypatch.setenv("TENDWIRE_SNAPSHOT_RETENTION_COUNT", "211") - monkeypatch.setenv("TENDWIRE_SNAPSHOT_MAINTENANCE_BATCH_SIZE", "37") - monkeypatch.setenv("TENDWIRE_STORE_MAINTENANCE_CADENCE_SECONDS", "7200") - monkeypatch.setenv("TENDWIRE_ACKNOWLEDGED_FINAL_RETENTION_DAYS", "31") - monkeypatch.setenv("TENDWIRE_ACKNOWLEDGED_FINAL_RETENTION_COUNT", "422") - received: dict[str, Any] = {} - - def capture_status( - _db_path: Path, - _host_id: str, - **kwargs: Any, - ) -> dict[str, Any]: - received.update(kwargs) - return {"schema_version": 1, "ok": True, "status": "ok"} - - monkeypatch.setattr("tendwire.cli.store_status", capture_status) - code = main( - [ - "store", - "status", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert code == 0 - assert captured.err == "" - assert json.loads(captured.out)["ok"] is True - assert received == { - "acknowledged_final_retention_days": 31, - "acknowledged_final_retention_count": 422, - "snapshot_retention_days": 19, - "snapshot_retention_count": 211, - "maintenance_batch_size": 37, - "maintenance_cadence_seconds": 7200, - "command_retry_horizon_seconds": 604800, - "command_receipt_retention_seconds": 2592000, - "command_receipt_retention_count": 4096, - } - - -def test_cli_doctor_with_absent_sqlite_sidecars_is_noncreating( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, -) -> None: - state_dir = tmp_path / "private-cli-doctor-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-cli-doctor-database" - init_store(db_path) - sidecars = tuple( - Path(f"{db_path}{suffix}") for suffix in ("-wal", "-shm", "-journal") - ) - assert all(not os.path.lexists(path) for path in sidecars) - before = ( - tuple(sorted(path.name for path in state_dir.iterdir())), - db_path.stat().st_ino, - db_path.stat().st_size, - db_path.stat().st_mtime_ns, - ) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(state_dir)) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(db_path)) - - def forbidden_repair(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("doctor must remain validation-only") - - monkeypatch.setattr("tendwire.cli.repair_config_state", forbidden_repair) - code = main( - [ - "--host-id", - "doctor-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "doctor", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - database_check = next( - check for check in payload["checks"] if check["name"] == "database_permissions" - ) - after = ( - tuple(sorted(path.name for path in state_dir.iterdir())), - db_path.stat().st_ino, - db_path.stat().st_size, - db_path.stat().st_mtime_ns, - ) - - assert code == 1 - assert captured.err == "" - assert database_check == { - "name": "database_permissions", - "ok": True, - "outcome": "compliant", - "remediation": "No action required.", - } - assert after == before - assert all(not os.path.lexists(path) for path in sidecars) - - -@pytest.mark.parametrize("hostile_member", ["main", "wal"]) -def test_cli_doctor_sqlite_failures_are_one_fixed_path_free_record( - tmp_path: Path, - capsys: pytest.CaptureFixture[str], - monkeypatch: pytest.MonkeyPatch, - hostile_member: str, -) -> None: - state_dir = tmp_path / "private-cli-hostile-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-cli-hostile-database" - target = state_dir / "private-cli-hostile-target" - private_contents = b"raw-OSError-private-cli-target" - target.write_bytes(private_contents) - os.chmod(target, 0o600) - if hostile_member == "main": - hostile_path = db_path - else: - init_store(db_path) - hostile_path = Path(f"{db_path}-wal") - hostile_path.symlink_to(target) - hostile_inode = str(os.lstat(hostile_path).st_ino) - target_before = target.read_bytes() - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(state_dir)) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(db_path)) - - def forbidden_repair(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("doctor must not repair hostile SQLite entries") - - monkeypatch.setattr("tendwire.cli.repair_config_state", forbidden_repair) - code = main( - [ - "--host-id", - "doctor-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "doctor", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - database_check = [ - check for check in payload["checks"] if check["name"] == "database_permissions" - ] - assert code == 1 - assert captured.err == "" - assert database_check == [ - { - "name": "database_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - ] - assert hostile_path.is_symlink() - assert target.read_bytes() == target_before - serialized = json.dumps(payload, sort_keys=True) - for forbidden in ( - str(state_dir), - str(db_path), - db_path.name, - str(hostile_path), - hostile_path.name, - str(target), - target.name, - private_contents.decode(), - hostile_inode, - "-wal", - "-shm", - "-journal", - "OSError", - "[Errno", - '"uid"', - '"gid"', - '"inode"', - ): - assert forbidden not in serialized + assert payload["status"] == "daemon_unavailable" + assert not (tmp_path / "tendwire.db").exists() diff --git a/tests/test_cli_command.py b/tests/test_cli_command.py index 86bff4e..1db0265 100644 --- a/tests/test_cli_command.py +++ b/tests/test_cli_command.py @@ -1,2656 +1,43 @@ -"""Tests for the `tendwire command --json` CLI orchestration.""" - from __future__ import annotations import io import json -import socket -import sqlite3 -from pathlib import Path -from typing import Any - -import pytest -from tendwire.backends.herdr_cli import HerdrCommandObservation from tendwire.cli import main -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - DISPOSITION_TERMINAL_ACCEPTED, - DISPOSITION_TERMINAL_REJECTED, - DISPOSITION_TERMINAL_UNCERTAIN, - STATUS_ACCEPTED, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_FAILED, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DRY_RUN, - STATUS_DUPLICATE_REQUEST, - STATUS_INVALID_REQUEST, - STATUS_NOT_FOUND, - STATUS_REQUEST_STATE_UNCERTAIN, - CommandEnvelope, - CommandRequest, - build_canonical_mutation, -) -from tendwire.core.models import Space, Worker, WorkerBinding -from tendwire.store.sqlite import ( - finish_command_request, - get_command_request, - init_store, - list_worker_bindings, - mark_command_send_started, - reserve_command_request, - upsert_worker_bindings, -) - - -@pytest.fixture(autouse=True) -def _isolate_cli_paths(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - private_home = tmp_path / "home" - private_home.mkdir(mode=0o700) - monkeypatch.setenv("HOME", str(private_home)) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "tendwire-data")) - monkeypatch.delenv("TENDWIRE_DB_PATH", raising=False) -def _fake_herdr_state(config: Any) -> tuple[list[Space], list[Worker]]: - workers = [ - Worker( - id="w-1", - name="Alpha", - status="active", - space_id="s-1", - backend_target={"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}, - ), - Worker( - id="w-2", - name="Beta", - status="idle", - space_id="s-1", - backend_target={"kind": "agent_id", "value": "agent-2", "sendable": True, "reason": None}, - ), - ] - return [], workers - - -def _fake_herdr_command_observation(config: Any) -> HerdrCommandObservation: - spaces, workers = _fake_herdr_state(config) - return HerdrCommandObservation( - spaces=spaces, - workers=workers, - status="healthy", - outcome="healthy_non_empty", - ) - - - - -def _seed_uncertain_request(db_path: Path, request: CommandRequest) -> None: - canonical = build_canonical_mutation(request, public_worker_id="w-1") - pending = CommandEnvelope.from_result( - request, - ok=False, - status=STATUS_REQUEST_STATE_UNCERTAIN, - disposition=DISPOSITION_TERMINAL_UNCERTAIN, - error={ - "code": STATUS_REQUEST_STATE_UNCERTAIN, - "message": "pending", - "details": {}, - }, - ) - reservation = reserve_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - action=request.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=pending.to_json(), - ) - owner_token = reservation["owner_token"] - assert isinstance(owner_token, str) - started = mark_command_send_started( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - binding_fingerprint="seed-binding", - ) - assert started["status"] == "send_started" - finished = finish_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - expected_state="send_started", - terminal_state="uncertain", - status=STATUS_REQUEST_STATE_UNCERTAIN, - result_json=pending.to_json(), - ) - assert finished["status"] == "uncertain" - - -def _seed_accepted_request( - db_path: Path, - request: CommandRequest, - *, - worker_id: str = "w-1", -) -> None: - init_store(db_path) - canonical = build_canonical_mutation(request, public_worker_id=worker_id) - pending = CommandEnvelope.from_result( - request, - ok=False, - status="pending", - disposition=DISPOSITION_IN_PROGRESS, - error={"code": "pending", "message": "pending"}, - ) - reservation = reserve_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - action=request.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=pending.to_json(), - ) - owner_token = reservation["owner_token"] - assert isinstance(owner_token, str) - started = mark_command_send_started( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - binding_fingerprint="seed-binding", - ) - assert started["status"] == "send_started" - accepted = CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result={"target": {"worker_id": worker_id}}, - ) - finished = finish_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - expected_state="send_started", - terminal_state="accepted", - status=STATUS_ACCEPTED, - result_json=accepted.to_json(), - ) - assert finished["status"] == "accepted" +def _run(monkeypatch, capsys, payload): + monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(payload))) + code = main(["command", "--json"]) + return code, json.loads(capsys.readouterr().out) -def test_cli_command_invalid_json(capsys, monkeypatch) -> None: - monkeypatch.setattr("sys.stdin", io.StringIO("not json")) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() +def test_invalid_command_is_rejected_locally(monkeypatch, capsys) -> None: + code, payload = _run(monkeypatch, capsys, {"schema_version": 1, "action": "unknown"}) assert code == 1 - payload = json.loads(captured.out) assert payload["ok"] is False - assert payload["status"] == STATUS_INVALID_REQUEST -def test_cli_command_noop_success(capsys, monkeypatch) -> None: - calls: list[str] = [] - - def guarded_fetch(config: Any) -> tuple[list[Space], list[Worker]]: - calls.append("fetch") - raise AssertionError("noop must not fetch Herdr state") - - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", guarded_fetch) - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "noop"})), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() +def test_noop_remains_pure(monkeypatch, capsys) -> None: + code, payload = _run(monkeypatch, capsys, {"schema_version": 1, "action": "noop"}) assert code == 0 - payload = json.loads(captured.out) - assert payload["ok"] is True assert payload["status"] == "noop" - assert payload["schema_version"] == 2 - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert captured.err == "" - assert calls == [] - - -def test_cli_command_unknown_action_rejected(capsys, monkeypatch) -> None: - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "explode"})), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - assert payload["ok"] is False - assert captured.err == "" - - -def test_cli_command_fingerprint_only_target_rejected(capsys, monkeypatch, tmp_path) -> None: - """A fingerprint-only send is invalid at the CLI, before any store or backend work.""" - db_path = tmp_path / "fingerprint-only.db" - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "fingerprint-only", - "dry_run": False, - "target": {"worker_fingerprint": "fingerprint-A"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "unavailable.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - - captured = capsys.readouterr() - payload = json.loads(captured.out) - assert code == 1 - assert payload["ok"] is False - assert payload["status"] == "invalid_request" - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert captured.err == "" - # A request this malformed never reaches durable state. - assert not db_path.exists() - - -def test_cli_command_read_snapshot_neutral_result(capsys, monkeypatch) -> None: - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "read_snapshot"})), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 0 - payload = json.loads(captured.out) - assert payload["ok"] is True - assert payload["status"] == "snapshot" - assert payload["result"]["snapshot"]["schema_version"] == 2 - assert payload["result"]["snapshot"]["backend_health"][0]["status"] == "unavailable" - assert payload["result"]["snapshot"]["backend_health"][0]["outcome"] == "missing_binary" - assert captured.err == "" - - -@pytest.mark.parametrize( - ("request_payload", "expected_result"), - [ - ( - { - "schema_version": 1, - "action": "send_instruction", - "target": {"name": "Alpha"}, - "instruction": {"text": "hello"}, - }, - { - "target": {"name": "Alpha"}, - "instruction": {"text": "hello"}, - }, - ), - ( - { - "schema_version": 1, - "action": "answer_pending", - "params": { - "pending_id": "pending-public", - "pending_fingerprint": "revision-public", - "choice_id": "choice-public", - }, - }, - { - "pending": { - "id": "pending-public", - "fingerprint": "revision-public", - }, - "choice": {"choice_id": "choice-public"}, - "delivery_state": "not_submitted", - }, - ), - ], -) -def test_cli_command_mutation_dry_run_is_pure_and_creates_no_receipt( - request_payload: dict[str, Any], - expected_result: dict[str, Any], - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[str] = [] - - def forbidden(*args: Any, **kwargs: Any) -> Any: - calls.append("io") - raise AssertionError("dry-run must not call daemon, backend, or command store") - - monkeypatch.setattr("tendwire.cli._try_daemon_attempt", forbidden) - monkeypatch.setattr("tendwire.command_submission.get_command_request", forbidden) - monkeypatch.setattr("tendwire.command_submission._current_snapshot", forbidden) - monkeypatch.setattr("tendwire.command_submission._validate_pending_choice", forbidden) - monkeypatch.setattr("tendwire.command_submission.reserve_command_request", forbidden) - - db_path = tmp_path / f"{request_payload['action']}.db" - init_store(db_path) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request_payload))) - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "unavailable.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert payload["ok"] is True - assert payload["status"] == "dry_run" - assert payload["dry_run"] is True - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert payload["result"] == expected_result - assert calls == [] - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 0 - assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 0 - - -@pytest.mark.parametrize("mode", ["cli_socket_path", "env_socket_path", "env_backend_socket"]) -def test_cli_command_socket_mode_mutation_unavailable_does_not_fallback( - mode: str, - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[str] = [] - - def guarded_fetch(*args: Any, **kwargs: Any) -> HerdrCommandObservation: - calls.append("fetch") - raise AssertionError("explicit daemon/socket mode must not fall back to Herdr observation") - - def guarded_send(*args: Any, **kwargs: Any) -> CommandEnvelope: - calls.append("send") - raise AssertionError("explicit daemon/socket mode must not send through Herdr CLI") - - monkeypatch.delenv("TENDWIRE_SOCKET_PATH", raising=False) - monkeypatch.delenv("TENDWIRE_HERDR_BACKEND", raising=False) - monkeypatch.delenv("TENDWIRE_DATA_DIR", raising=False) - - - - socket_path = tmp_path / f"{mode}.sock" - data_dir = tmp_path / "data" - args = [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - ] - forbidden_fragments = [str(socket_path)] - if mode == "cli_socket_path": - args.extend(["--socket-path", str(socket_path)]) - elif mode == "env_socket_path": - monkeypatch.setenv("TENDWIRE_SOCKET_PATH", str(socket_path)) - elif mode == "env_backend_socket": - monkeypatch.setenv("TENDWIRE_HERDR_BACKEND", "socket") - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(data_dir)) - forbidden_fragments.extend([str(data_dir), "tendwire.sock"]) - else: - raise AssertionError(f"unexpected mode {mode}") - - db_path = tmp_path / f"{mode}.db" - request_id = f"daemon-unavailable-{mode}" - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - *args, - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - serialized = json.dumps(payload) - - assert code == 1 - assert captured.err == "" - assert payload["ok"] is False - assert payload["status"] == STATUS_BACKEND_UNAVAILABLE - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert payload["request_id"] == request_id - assert calls == [] - for fragment in forbidden_fragments: - assert fragment not in serialized - _assert_no_command_public_forbidden_fields(payload) - - # A definite pre-start daemon failure never creates a local receipt. - assert get_command_request(db_path, "cmd-host", request_id) is None - - -def test_cli_daemon_client_uses_method_specific_timeouts( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[tuple[str, float]] = [] - - class FakeDaemonAPIClient: - def __init__(self, socket_path: Any, *, timeout_seconds: float, max_response_bytes: int = 1024 * 1024): - self.timeout_seconds = timeout_seconds - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - calls.append((method, self.timeout_seconds)) - if method == "snapshot.get": - return {"ok": True, "result": {"schema_version": 2, "spaces": [], "workers": []}} - return { - "ok": True, - "result": { - "schema_version": 2, - "action": "send_instruction", - "request_id": "daemon-timeout-method", - "ok": True, - "dry_run": False, - "status": STATUS_ACCEPTED, - "disposition": DISPOSITION_TERMINAL_ACCEPTED, - "result": {}, - "error": None, - "warnings": [], - }, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", FakeDaemonAPIClient) - socket_path = tmp_path / "daemon.sock" - - assert ( - main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(socket_path), - "snapshot", - "--json", - ] - ) - == 0 - ) - capsys.readouterr() - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "daemon-timeout-method", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - assert ( - main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(socket_path), - "command", - "--json", - "--db-path", - str(tmp_path / "cmd.db"), - ] - ) - == 0 - ) - - assert calls[0] == ("snapshot.get", 2.0) - assert calls[1][0] == "command.submit" - assert calls[1][1] > calls[0][1] - assert calls[1][1] >= 5.0 - - -def test_cli_command_socket_mode_daemon_timeout_is_uncertain( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[str] = [] - - def guarded_fetch(*args: Any, **kwargs: Any) -> HerdrCommandObservation: - calls.append("fetch") - raise AssertionError("explicit daemon/socket mode must not fall back to Herdr observation") - - def guarded_send(*args: Any, **kwargs: Any) -> CommandEnvelope: - calls.append("send") - raise AssertionError("explicit daemon/socket mode must not send through Herdr CLI") - - class TimeoutDaemonAPIClient: - def __init__(self, socket_path: Any, *, timeout_seconds: float, max_response_bytes: int = 1024 * 1024): - self.timeout_seconds = timeout_seconds - - def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str, Any]: - from tendwire.daemon_api import DaemonUnavailable - - try: - raise socket.timeout("timed out") - except socket.timeout as exc: - raise DaemonUnavailable( - "timed out", - timed_out=True, - request_started=True, - ) from exc - - - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", TimeoutDaemonAPIClient) - - db_path = tmp_path / "timeout.db" - request_id = "daemon-timeout-uncertain" - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - assert calls == [] - - # A lost daemon response with no authoritative receipt is not a command - # envelope and must never be labeled terminal uncertainty. - assert get_command_request(db_path, "cmd-host", request_id) is None - - -def test_cli_rejects_malformed_daemon_inner_envelope_without_fabricating_json( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - request = { - "schema_version": 1, - "action": "send_instruction", - "request_id": "malformed-daemon-result", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - - class MalformedResultClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - assert method == "command.submit" - assert params == request - return { - "schema_version": 1, - "ok": True, - "status": "ok", - "result": { - "schema_version": 2, - "action": request["action"], - "request_id": request["request_id"], - "ok": True, - "dry_run": False, - "status": STATUS_ACCEPTED, - "result": {}, - "error": None, - "warnings": [], - }, - "error": None, - } - - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - MalformedResultClient, - ) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request))) - db_path = tmp_path / "malformed-result.db" - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - assert get_command_request( - db_path, - "cmd-host", - "malformed-daemon-result", - ) is None - - -@pytest.mark.parametrize( - ("case", "response_request_id", "ok", "status", "disposition"), - [ - ( - "receipt-null-id", - None, - True, - STATUS_ACCEPTED, - DISPOSITION_TERMINAL_ACCEPTED, - ), - ( - "receipt-invalid-id", - "invalid id", - True, - STATUS_ACCEPTED, - DISPOSITION_TERMINAL_ACCEPTED, - ), - ( - "no-receipt-success", - "illegal-daemon-tuple", - True, - STATUS_BACKEND_UNAVAILABLE, - DISPOSITION_NO_RECEIPT, - ), - ( - "no-receipt-accepted", - "illegal-daemon-tuple", - False, - STATUS_ACCEPTED, - DISPOSITION_NO_RECEIPT, - ), - ( - "no-receipt-pending", - "illegal-daemon-tuple", - False, - "pending", - DISPOSITION_NO_RECEIPT, - ), - ( - "no-receipt-uncertain", - "illegal-daemon-tuple", - False, - STATUS_REQUEST_STATE_UNCERTAIN, - DISPOSITION_NO_RECEIPT, - ), - ], -) -def test_cli_strictly_rejects_illegal_daemon_disposition_tuples( - case: str, - response_request_id: Any, - ok: bool, - status: str, - disposition: str, - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - request = { - "schema_version": 1, - "action": "send_instruction", - "request_id": "illegal-daemon-tuple", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - inner = { - "schema_version": 2, - "action": request["action"], - "request_id": response_request_id, - "ok": ok, - "dry_run": False, - "status": status, - "disposition": disposition, - "result": {}, - "error": None if ok else {"code": status, "message": "invalid tuple"}, - "warnings": [], - } - - class IllegalTupleClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - assert method == "command.submit" - assert params == request - return { - "schema_version": 1, - "ok": True, - "status": "ok", - "result": inner, - "error": None, - } - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", IllegalTupleClient) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request))) - db_path = tmp_path / f"{case}.db" - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - assert get_command_request( - db_path, - "cmd-host", - request["request_id"], - ) is None - - -def test_cli_daemon_response_loss_recovers_accepted_receipt_exactly_once( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - from tendwire.daemon_api import DaemonProtocolError - - db_path = tmp_path / "accepted-loss.db" - init_store(db_path) - request_payload = { - "schema_version": 1, - "action": "send_instruction", - "request_id": "accepted-loss", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - request = CommandRequest.from_dict(request_payload) - canonical = build_canonical_mutation(request, public_worker_id="w-1") - effects: list[str] = [] - - class LostAcceptedResponseClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - assert method == "command.submit" - assert params == request_payload - pending = CommandEnvelope.from_result( - request, - ok=False, - status="pending", - disposition=DISPOSITION_IN_PROGRESS, - error={"code": "pending", "message": "pending"}, - ) - reservation = reserve_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - action=request.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=pending.to_json(), - ) - owner_token = reservation["owner_token"] - mark_command_send_started( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - binding_fingerprint="private-binding", - ) - effects.append("sent") - accepted = CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result={"target": {"worker_id": "w-1"}}, - ) - finish_command_request( - db_path, - host_id="cmd-host", - request_id=request.request_id or "", - canonical_fingerprint=canonical.fingerprint, - owner_token=owner_token, - expected_state="send_started", - terminal_state="accepted", - status=STATUS_ACCEPTED, - result_json=accepted.to_json(), - ) - raise DaemonProtocolError( - "accepted response lost", - request_started=True, - ) - - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - LostAcceptedResponseClient, - ) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request_payload))) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert captured.err == "" - assert payload["status"] == STATUS_ACCEPTED - assert payload["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - assert effects == ["sent"] - receipt = get_command_request(db_path, "cmd-host", "accepted-loss") - assert receipt is not None - assert receipt["state"] == "accepted" - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize( - ("case", "current_target", "expected_status"), - [ - ( - "changed-worker-id", - {"worker_id": "w-2"}, - STATUS_DUPLICATE_REQUEST, - ), - ("mutable-name", {"name": "Alpha"}, None), - ("mutable-space", {"space_id": "space-1"}, None), - ( - "worker-precondition", - {"worker_id": "w-1", "worker_fingerprint": "current-fingerprint"}, - STATUS_ACCEPTED, - ), - ( - "worker-plus-null-alias", - {"worker_id": "w-1", "name": None}, - None, - ), - ], -) -def test_cli_response_loss_reconciliation_only_uses_stored_explicit_worker_identity( - case: str, - current_target: dict[str, Any], - expected_status: str | None, - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - request_id = f"response-loss-{case}" - db_path = tmp_path / f"{case}.db" - stored_request = CommandRequest( - action="send_instruction", - request_id=request_id, - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - _seed_accepted_request(db_path, stored_request) - current_request = { - **stored_request.to_dict(), - "target": current_target, - } - daemon_calls: list[tuple[str, dict[str, Any]]] = [] - - def stored_rows() -> tuple[list[tuple[Any, ...]], list[tuple[Any, ...]]]: - with sqlite3.connect(str(db_path)) as conn: - return ( - conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall(), - conn.execute("SELECT * FROM events ORDER BY id").fetchall(), - ) - - rows_before = stored_rows() - - class LostResponseClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - from tendwire.daemon_api import DaemonProtocolError - - daemon_calls.append((method, dict(params or {}))) - raise DaemonProtocolError("response lost", request_started=True) - - def forbidden(*args: Any, **kwargs: Any) -> Any: - del args, kwargs - raise AssertionError( - "response-loss reconciliation must not submit or resolve mutable authority" - ) - - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", LostResponseClient) - monkeypatch.setattr("tendwire.command_submission._current_snapshot", forbidden) - monkeypatch.setattr("tendwire.cli.command_envelope_from_payload", forbidden) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(current_request))) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert daemon_calls == [("command.submit", current_request)] - assert stored_rows() == rows_before - if expected_status is None: - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - elif expected_status == STATUS_ACCEPTED: - assert code == 0 - assert captured.err == "" - payload = json.loads(captured.out) - assert payload["status"] == STATUS_ACCEPTED - assert payload["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - _assert_no_command_public_forbidden_fields(payload) - else: - assert code == 1 - assert captured.err == "" - payload = json.loads(captured.out) - assert payload["status"] == expected_status - assert payload["status"] != STATUS_ACCEPTED - assert payload["disposition"] == DISPOSITION_TERMINAL_REJECTED - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize( - ("action", "action_fields"), - [ - ( - "send_instruction", - { - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - }, - ), - ( - "answer_pending", - { - "params": { - "pending_id": "pending-" + ("a" * 24), - "pending_fingerprint": "b" * 24, - "choice_id": "choice-" + ("c" * 24), - }, - }, - ), - ], - ids=["send", "answer"], -) -@pytest.mark.parametrize( - "request_started", - [False, True], - ids=["pre-start", "may-have-started"], -) -def test_cli_mutations_never_fallback_or_write_receipt_on_daemon_edge_failure( - action: str, - action_fields: dict[str, Any], - request_started: bool, - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - calls: list[tuple[str, dict[str, Any]]] = [] - - class FailingDaemonAPIClient: - def __init__( - self, - socket_path: Any, - *, - timeout_seconds: float, - max_response_bytes: int = 1024 * 1024, - ) -> None: - del socket_path, timeout_seconds, max_response_bytes - - def request( - self, - method: str, - params: dict[str, Any] | None = None, - ) -> dict[str, Any]: - from tendwire.daemon_api import DaemonProtocolError - - calls.append((method, dict(params or {}))) - raise DaemonProtocolError( - "deterministic protocol failure", - request_started=request_started, - ) - - request_id = f"{action}-{request_started}" - request = { - "schema_version": 1, - "action": action, - "request_id": request_id, - "dry_run": False, - **action_fields, - } - db_path = tmp_path / f"{action}-{request_started}.db" - monkeypatch.setattr( - "tendwire.daemon_api.DaemonAPIClient", - FailingDaemonAPIClient, - ) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request))) - - code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(tmp_path / "daemon.sock"), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - - assert calls == [("command.submit", request)] - assert get_command_request(db_path, "cmd-host", request_id) is None - if request_started: - assert code == 2 - assert captured.out == "" - assert "unresolved" in captured.err - else: - payload = json.loads(captured.out) - assert code == 1 - assert captured.err == "" - assert payload["status"] == STATUS_BACKEND_UNAVAILABLE - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert payload["request_id"] == request_id - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize( - ("request_id", "include_request_id"), - [ - (None, False), - (None, True), - ("", True), - (" \t", True), - (" leading", True), - ("trailing ", True), - ("\twrapped\t", True), - ], -) -def test_cli_command_send_instruction_non_dry_run_requires_request_id( - capsys, - monkeypatch, - tmp_path: Path, - request_id: Any, - include_request_id: bool, -) -> None: - calls: list[str] = [] - - def guarded(*args: Any, **kwargs: Any) -> Any: - calls.append("called") - raise AssertionError("invalid request_id must stop before backend or store mutation") - - monkeypatch.setattr("tendwire.command_submission.reserve_command_request", guarded) - monkeypatch.setattr("tendwire.command_submission._default_socket_client_factory", guarded) - payload: dict[str, Any] = { - "schema_version": 1, - "action": "send_instruction", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - if include_request_id: - payload["request_id"] = request_id - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(payload))) - db_path = tmp_path / "invalid-request-id.db" - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - - captured = capsys.readouterr() - assert code == 1 - payload_out = json.loads(captured.out) - assert payload_out["ok"] is False - assert payload_out["status"] == STATUS_INVALID_REQUEST - assert calls == [] - assert not db_path.exists() - -def test_cli_command_send_instruction_requires_socket_backend_for_mutation( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "literal-false.db" - calls: list[tuple[Any, Any]] = [] - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "literal-false", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="literal-false") - assert payload["dry_run"] is False - assert calls == [] - - assert get_command_request(db_path, "cmd-host", "literal-false") is None - - -def test_cli_command_default_backend_does_not_rehydrate_private_stored_binding( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - db_path = tmp_path / "stored-binding.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="w-1", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-stored", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="stored-private", - ) - ], - ) - calls: list[tuple[Any, Any]] = [] - - def targetless_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[Worker(id="w-1", name="Alpha", status="active", space_id="s-1")], - status="healthy", - outcome="healthy_non_empty", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "stored-binding", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="stored-binding") - assert calls == [] - serialized = json.dumps(payload) - assert "agent-stored" not in serialized - assert "stored-private" not in serialized - _assert_no_command_public_forbidden_fields(payload) - - -def test_cli_command_does_not_send_through_expired_stored_binding( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - db_path = tmp_path / "expired-binding.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="w-1", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-expired", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="2026-01-02T00:00:00+00:00", - private_fingerprint="expired-private", - ) - ], - ) - calls: list[tuple[Any, Any]] = [] - - def targetless_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[Worker(id="w-1", name="Alpha", status="active", space_id="s-1")], - status="healthy", - outcome="healthy_non_empty", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "expired-binding", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="expired-binding") - assert calls == [] - assert "agent-expired" not in json.dumps(payload) - _assert_no_command_public_forbidden_fields(payload) - -def test_cli_command_duplicate_current_binding_is_ambiguous_and_not_expired( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - db_path = tmp_path / "current-duplicates.db" - calls: list[tuple[Any, Any]] = [] - def duplicate_observation(config: Any) -> HerdrCommandObservation: - worker_a = Worker( - id="dup-a", - name="Duplicate A", - status="active", - backend_target={"kind": "agent_id", "value": "same-agent", "sendable": True, "reason": None}, - ) - worker_b = Worker( - id="dup-b", - name="Duplicate B", - status="active", - backend_target={"kind": "agent_id", "value": "same-agent", "sendable": True, "reason": None}, - ) - return HerdrCommandObservation( - spaces=[], - workers=[worker_a, worker_b], - status="healthy", - outcome="healthy_non_empty", - bindings=[ - WorkerBinding( - host_id="cmd-host", - worker_id=worker_a.id, - worker_fingerprint=worker_a.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="same-agent", - sendable=False, - reason="duplicate_backend_target", - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="colliding-private", - ), - WorkerBinding( - host_id="cmd-host", - worker_id=worker_b.id, - worker_fingerprint=worker_b.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="same-agent", - sendable=False, - reason="duplicate_backend_target", - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="colliding-private", - ), - ], - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "current-duplicate", - "dry_run": False, - "target": {"worker_id": "dup-a"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - current = list_worker_bindings(db_path, "cmd-host", backend="herdr") - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="current-duplicate") - assert calls == [] - assert current == [] - assert "colliding-private" not in json.dumps(payload) - _assert_no_command_public_forbidden_fields(payload) - - -def test_cli_command_stored_duplicate_binding_is_ambiguous_and_skips_backend( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - db_path = tmp_path / "stored-duplicate.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="dup-stored", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="same-agent", - sendable=False, - reason="duplicate_backend_target", - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-duplicate-fingerprint", - ) - ], - ) - calls: list[tuple[Any, Any]] = [] - - def targetless_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[Worker(id="dup-stored", name="Duplicate", status="active", space_id="s-1")], - status="healthy", - outcome="healthy_non_empty", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "stored-duplicate", - "dry_run": False, - "target": {"worker_id": "dup-stored"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="stored-duplicate") - assert calls == [] - serialized = json.dumps(payload) - assert "same-agent" not in serialized - assert "private-duplicate-fingerprint" not in serialized - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize("value", ["false", "true", 0, 1, None, [], {}]) -def test_cli_command_rejects_non_boolean_dry_run_before_backend( - value: Any, capsys, monkeypatch -) -> None: - calls: list[str] = [] - - def guarded_observation(config: Any) -> HerdrCommandObservation: - calls.append("fetch") - raise AssertionError("invalid dry_run must not fetch") - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "bad-dry-run", - "dry_run": value, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert payload["status"] == STATUS_INVALID_REQUEST - assert calls == [] - _assert_no_command_public_forbidden_fields(payload) - - -@pytest.mark.parametrize("value", ["1", 1.0, True, False, None, [], {}, 2]) -def test_cli_command_rejects_malformed_schema_version_before_pipeline( - value: Any, capsys, monkeypatch -) -> None: - calls: list[str] = [] - - def guarded(*args: Any, **kwargs: Any) -> Any: - calls.append("called") - raise AssertionError("invalid schema_version must stop before pipeline work") - - monkeypatch.setattr("tendwire.command_submission.reserve_command_request", guarded) - monkeypatch.setattr("tendwire.command_submission._default_socket_client_factory", guarded) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": value, - "action": "send_instruction", - "request_id": "bad-schema", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert payload["status"] == STATUS_INVALID_REQUEST - assert calls == [] - _assert_no_command_public_forbidden_fields(payload) - - -def test_cli_command_send_instruction_empty_target_rejects_before_fetch(capsys, monkeypatch) -> None: - calls: list[str] = [] - - def guarded_fetch(config: Any) -> tuple[list[Space], list[Worker]]: - calls.append("fetch") - raise AssertionError("empty target must reject before Herdr fetch") - - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", guarded_fetch) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "empty-target", - "dry_run": False, - "target": {}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - assert code == 1 - assert payload["status"] == STATUS_INVALID_REQUEST - assert calls == [] - - -def test_cli_command_duplicate_request_id_same_payload_returns_cached(capsys, monkeypatch, tmp_path: Path) -> None: - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - - calls: list[tuple[Any, Any]] = [] - - db_path = tmp_path / "cmd.db" - payload = json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "dup-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - - monkeypatch.setattr("sys.stdin", io.StringIO(payload)) - code1 = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured1 = capsys.readouterr() - assert code1 == 1 - result1 = json.loads(captured1.out) - _assert_socket_backend_required_payload(result1, request_id="dup-1") - - assert get_command_request(db_path, "cmd-host", "dup-1") is None - - monkeypatch.setattr("sys.stdin", io.StringIO(payload)) - code2 = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured2 = capsys.readouterr() - assert code2 == 1 - result2 = json.loads(captured2.out) - assert result2["status"] == STATUS_BACKEND_UNAVAILABLE - assert result2 == result1 - assert calls == [] - - -def test_cli_command_duplicate_request_id_different_payload_rejects(capsys, monkeypatch, tmp_path: Path) -> None: - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - - calls: list[tuple[Any, Any]] = [] - - db_path = tmp_path / "cmd.db" - payload1 = json.dumps( +def test_live_mutation_never_falls_back_from_daemon(monkeypatch, capsys, tmp_path) -> None: + monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path)) + code, payload = _run( + monkeypatch, + capsys, { "schema_version": 1, "action": "send_instruction", - "request_id": "dup-2", + "request_id": "r1", "dry_run": False, - "target": {"worker_id": "w-1"}, + "target": {"worker_id": "worker"}, "instruction": {"text": "hello"}, - } - ) - payload2 = json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "dup-2", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "world"}, - } - ) - - monkeypatch.setattr("sys.stdin", io.StringIO(payload1)) - code1 = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - capsys.readouterr() - assert code1 == 1 - - monkeypatch.setattr("sys.stdin", io.StringIO(payload2)) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - assert code == 1 - result = json.loads(captured.out) - assert result["status"] == STATUS_BACKEND_UNAVAILABLE - assert calls == [] - - -def test_cli_command_pending_receipt_rejects_without_retry(capsys, monkeypatch, tmp_path: Path) -> None: - db_path = tmp_path / "cmd.db" - pending_request = CommandRequest( - action="send_instruction", - request_id="uncertain-1", - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - # Seed terminal uncertainty through the v12 CAS lifecycle. - init_store(db_path) - _seed_uncertain_request(db_path, pending_request) - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "uncertain-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - assert code == 1 - result = json.loads(captured.out) - assert result["status"] == STATUS_REQUEST_STATE_UNCERTAIN - assert result["disposition"] == DISPOSITION_TERMINAL_UNCERTAIN - - -def test_cli_command_uncertain_receipt_changed_payload_is_duplicate_without_retry( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "cmd.db" - - original_request = CommandRequest( - action="send_instruction", - request_id="uncertain-changed", - dry_run=False, - target={"worker_id": "w-1"}, - instruction={"text": "hello"}, - ) - init_store(db_path) - _seed_uncertain_request(db_path, original_request) - calls: list[str] = [] - - def guarded_backend(*args: Any, **kwargs: Any) -> Any: - calls.append("backend") - raise AssertionError("changed duplicate receipt must not reach the backend") - - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - guarded_backend, - ) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "uncertain-changed", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "world"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - result = json.loads(captured.out) - - assert code == 1 - assert result["status"] == STATUS_DUPLICATE_REQUEST - assert calls == [] - - -def test_cli_command_forbidden_field_rejected(capsys, monkeypatch) -> None: - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "noop", - "params": {"pane_id": "leaked"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - assert payload["status"] == STATUS_INVALID_REQUEST - - -def test_cli_command_rejects_control_sequence_instruction_as_json_only(capsys, monkeypatch) -> None: - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello\x1b[31mworld"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["ok"] is False - assert payload["status"] == STATUS_INVALID_REQUEST - assert payload["error"]["details"] == {"field": "instruction.text"} - _assert_no_command_public_forbidden_fields(payload) - - -def test_cli_command_legacy_backend_guard_and_receipt_are_sanitized( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "cmd.db" - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - - calls: list[tuple[Any, Any]] = [] - - def leaky_backend(config: Any, target: Any, instruction: Any) -> CommandEnvelope: - calls.append((target, instruction)) - return CommandEnvelope( - ok=False, - status=STATUS_BACKEND_FAILED, - action="send_instruction", - result={ - "target": target, - "pane_id": "p-1", - "nested": {"argv": ["herdr"], "safe": "kept"}, - }, - error={ - "code": STATUS_BACKEND_FAILED, - "message": "failed", - "details": {"terminal_id": "t-1", "safe": "kept"}, - }, - ) - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "sanitize-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - _assert_socket_backend_required_payload(payload, request_id="sanitize-1") - assert calls == [] - _assert_no_command_public_forbidden_fields(payload) - - assert get_command_request(db_path, "cmd-host", "sanitize-1") is None - assert captured.err == "" - - -_COMMAND_PUBLIC_FORBIDDEN_KEYS = { - "pane_id", - "terminal_id", - "pid", - "tty", - "pty", - "tmux", - "screen_session", - "window_id", - "tab_id", - "argv", - "shell", - "command", - "route", - "routes", - "delivery", - "deliveries", - "token", - "tokens", - "connector", - "connectors", - "backend_target", - "agent_session", - "session_id", - "herdr_state", - "herdres_state", - "target_kind", - "target_value", - "turn_target_kind", - "turn_target_value", - "private_fingerprint", -} - - -def _assert_no_command_public_forbidden_fields(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - assert key not in _COMMAND_PUBLIC_FORBIDDEN_KEYS, f"forbidden field {path}.{key}" - _assert_no_command_public_forbidden_fields(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_command_public_forbidden_fields(item, f"{path}[{index}]") - - -def _assert_socket_backend_required_payload(value: dict[str, Any], *, request_id: str) -> None: - assert value["ok"] is False - assert value["status"] == STATUS_BACKEND_UNAVAILABLE - assert value["request_id"] == request_id - assert value["error"]["code"] == STATUS_BACKEND_UNAVAILABLE - assert value["error"]["message"] == "Herdr socket backend is not enabled" - _assert_no_command_public_forbidden_fields(value) - - -def _fake_herdr_state_with_terminal(config: Any) -> tuple[list[Space], list[Worker]]: - return [], [ - Worker( - id="w-terminal", - name="Terminal", - status="active", - space_id="s-1", - meta={ - "pane_id": "p-1", - "terminal_id": "t-1", - "pid": 123, - "tty": "/dev/pts/0", - "pty": "pts", - "tmux": "sess", - "screen_session": "scr", - "window_id": "win-1", - "tab_id": "tab-1", - "argv": ["bash"], - "shell": "bash", - "command": "python app.py", - "route": "telegram", - "routes": ["r1"], - "delivery": {"id": 1}, - "deliveries": [{"id": 2}], - "token": "secret", - "tokens": ["t1"], - "connector": {"x": 1}, - "connectors": [{"y": 2}], - "backend_target": {"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}, - "agent_session": {"value": "sess-1"}, - "session_id": "session-1", - "safe": "kept", - }, - backend_target={"kind": "agent_id", "value": "agent-1", "sendable": True, "reason": None}, - ) - ] - - -def test_cli_command_read_snapshot_strips_command_public_terminal_fields( - capsys, monkeypatch -) -> None: - """Command-public read_snapshot strips terminal/connector identifiers while - leaving the ordinary snapshot --json output unchanged. - """ - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state_with_terminal) - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "read_snapshot"})), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 0 - payload = json.loads(captured.out) - assert payload["ok"] is True - assert payload["status"] == "snapshot" - assert payload["action"] == "read_snapshot" - assert "request_id" in payload - meta = payload["result"]["snapshot"]["workers"][0]["meta"] - for key in _COMMAND_PUBLIC_FORBIDDEN_KEYS: - assert key not in meta, key - assert meta["safe"] == "kept" - - # The standalone snapshot path is public too, so backend identifiers are absent there as well. - code2 = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "snapshot", - "--json", - ] - ) - captured2 = capsys.readouterr() - assert code2 == 0 - snapshot = json.loads(captured2.out) - assert snapshot["schema_version"] == 2 - snap_meta = snapshot["workers"][0]["meta"] - for key in ("pane_id", "terminal_id", "backend_target", "agent_session", "session_id"): - assert key not in snap_meta - assert snap_meta["safe"] == "kept" - - -def test_cli_command_does_not_auto_discover_default_daemon_socket( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - """A default socket file must not opt ordinary CLI commands into daemon mode.""" - data_dir = tmp_path / "data" - data_dir.mkdir() - (data_dir / "tendwire.sock").touch() - calls: list[str] = [] - - class GuardedDaemonAPIClient: - def __init__(self, *args: Any, **kwargs: Any) -> None: - calls.append("daemon") - raise AssertionError("implicit default socket should not be contacted") - - monkeypatch.delenv("TENDWIRE_SOCKET_PATH", raising=False) - monkeypatch.delenv("TENDWIRE_HERDR_BACKEND", raising=False) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(data_dir)) - monkeypatch.setattr("tendwire.daemon_api.DaemonAPIClient", GuardedDaemonAPIClient) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - monkeypatch.setattr( - "sys.stdin", - io.StringIO(json.dumps({"schema_version": 1, "action": "read_snapshot"})), - ) - - code = main(["--host-id", "cmd-host", "command", "--json"]) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 0 - assert calls == [] - assert payload["ok"] is True - assert payload["status"] == "snapshot" - assert payload["result"]["snapshot"]["workers"][0]["id"] == "w-1" - - -def test_cli_command_forbidden_field_rejects_before_backend_and_store( - capsys, monkeypatch -) -> None: - """A contract-invalid request must be rejected before any backend or store call.""" - calls: list[str] = [] - - def guarded_fetch(config: Any) -> tuple[list[Space], list[Worker]]: - calls.append("fetch") - raise AssertionError("fetch_herdr_state called before validation") - - def guarded_reserve_request(*args: Any, **kwargs: Any) -> Any: - calls.append("reserve_request") - raise AssertionError("reserve_command_request called before validation") - - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", guarded_fetch) - monkeypatch.setattr( - "tendwire.command_submission.reserve_command_request", - guarded_reserve_request, - ) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "rej-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - "params": {"pane_id": "leaked"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - assert payload["status"] == STATUS_INVALID_REQUEST - assert payload["request_id"] == "rej-1" - assert calls == [] - - -def test_cli_command_raw_top_level_forbidden_rejects_before_pipeline( - capsys, monkeypatch -) -> None: - """Raw top-level forbidden fields reject before store, projection, or backend work.""" - calls: list[str] = [] - - def guarded_reserve_request(*args: Any, **kwargs: Any) -> Any: - calls.append("reserve_request") - raise AssertionError("reserve_command_request called before raw validation") - - def guarded_fetch(config: Any) -> tuple[list[Space], list[Worker]]: - calls.append("fetch") - raise AssertionError("fetch_herdr_state called before raw validation") - - def guarded_project(*args: Any, **kwargs: Any) -> Any: - calls.append("project") - raise AssertionError("project_from_observations called before raw validation") - - def guarded_execute(*args: Any, **kwargs: Any) -> Any: - calls.append("execute") - raise AssertionError("execute_command called before raw validation") - - def guarded_send(*args: Any, **kwargs: Any) -> Any: - calls.append("backend") - raise AssertionError("backend sender called before raw validation") - - monkeypatch.setattr( - "tendwire.command_submission.reserve_command_request", - guarded_reserve_request, - ) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", guarded_fetch) - monkeypatch.setattr("tendwire.cli.project_from_observations", guarded_project) - monkeypatch.setattr("tendwire.cli.execute_command", guarded_execute) - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - guarded_send, - ) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "raw-rej", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - "pane_id": "leaked", - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - assert code == 1 - assert payload["status"] == STATUS_INVALID_REQUEST - assert payload["request_id"] is None - assert "pane_id" in str(payload["error"]["details"]) - assert calls == [] - - -def test_cli_command_backend_unavailable_preserves_request_id( - capsys, monkeypatch, tmp_path: Path -) -> None: - """A pre-start backend failure preserves request_id without store authority.""" - db_path = tmp_path / "req.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="still-live", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-still-live", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="still-live-private", - ) - ], - ) - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", _fake_herdr_state) - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "req-visible", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "definitely-not-a-real-herdr-binary", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - assert code == 1 - payload = json.loads(captured.out) - assert payload["status"] == STATUS_BACKEND_UNAVAILABLE - assert payload["disposition"] == DISPOSITION_NO_RECEIPT - assert payload["request_id"] == "req-visible" - - assert get_command_request(db_path, "cmd-host", "req-visible") is None - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT COUNT(*) - FROM commands - WHERE host_id = ? - AND request_id = ? - """, - ("cmd-host", "req-visible"), - ).fetchone()[0] == 0 - current = list_worker_bindings(db_path, "cmd-host", backend="herdr") - assert [binding.private_fingerprint for binding in current] == ["still-live-private"] - - -def test_cli_command_default_backend_blocks_degraded_observation_send( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "degraded.db" - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id="cmd-host", - worker_id="still-live", - worker_fingerprint="old-fp", - backend="herdr", - target_kind="agent_id", - target_value="agent-still-live", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="still-live-private", - ) - ], - ) - - def degraded_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[], - status="degraded", - outcome="malformed_json", - message="Herdr agent observation is not healthy", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "degraded-1", - "dry_run": False, - "target": {"worker_id": "missing"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="degraded-1") - assert get_command_request(db_path, "cmd-host", "degraded-1") is None - current = list_worker_bindings(db_path, "cmd-host", backend="herdr") - assert [binding.private_fingerprint for binding in current] == ["still-live-private"] - - -def test_cli_command_default_backend_blocks_healthy_empty_observation_send( - capsys, monkeypatch, tmp_path: Path -) -> None: - db_path = tmp_path / "empty.db" - - def empty_observation(config: Any) -> HerdrCommandObservation: - return HerdrCommandObservation( - spaces=[], - workers=[], - status="healthy", - outcome="empty_healthy", - ) - - - - monkeypatch.setattr( - "sys.stdin", - io.StringIO( - json.dumps( - { - "schema_version": 1, - "action": "send_instruction", - "request_id": "empty-1", - "dry_run": False, - "target": {"worker_id": "missing"}, - "instruction": {"text": "hello"}, - } - ) - ), - ) - - code = main( - [ - "--host-id", - "cmd-host", - "--herdr-bin", - "herdr", - "command", - "--json", - "--db-path", - str(db_path), - ] + }, ) - captured = capsys.readouterr() - payload = json.loads(captured.out) - assert code == 1 - _assert_socket_backend_required_payload(payload, request_id="empty-1") - assert get_command_request(db_path, "cmd-host", "empty-1") is None + assert payload["status"] == "backend_unavailable" + assert not (tmp_path / "tendwire.db").exists() diff --git a/tests/test_codex_session_reader.py b/tests/test_codex_session_reader.py deleted file mode 100644 index b5e8141..0000000 --- a/tests/test_codex_session_reader.py +++ /dev/null @@ -1,1226 +0,0 @@ -"""Deterministic contracts for the private Codex session resolver and reader.""" - -from __future__ import annotations -from dataclasses import replace - -import json -import os -from pathlib import Path -from uuid import UUID -from types import SimpleNamespace - -import pytest - -from tendwire.backends import herdr_turns - - -SESSION_A = "019f2307-092b-7810-8323-418d7c55bd26" -SESSION_B = "019f2307-092b-7810-8323-418d7c55bd27" -SESSION_C = "11111111-1111-4111-8111-111111111111" - - -def _event(kind: str, turn_id: str, **extra): - return { - "type": "event_msg", - "payload": {"type": kind, "turn_id": turn_id, **extra}, - } - - -def _message(turn_id: str, role: str, text: str, *, phase: str | None = None): - payload = { - "type": "message", - "role": role, - "content": [{"type": "output_text", "text": text}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - } - if phase is not None: - payload["phase"] = phase - return {"type": "response_item", "payload": payload} - - -def _jsonl(*records, terminate: bool = True) -> bytes: - body = b"\n".join( - json.dumps(record, ensure_ascii=False, separators=(",", ":")).encode("utf-8") - for record in records - ) - return body + (b"\n" if terminate and records else b"") - - -def _session_path(home: Path, session_id: str = SESSION_A, date: str = "2026-07-03") -> Path: - year, month, day = date.split("-") - return ( - home - / "sessions" - / year - / month - / day - / f"rollout-{date}T00-00-00-{session_id}.jsonl" - ) - - -def _write_session( - home: Path, - records: tuple[dict, ...] | list[dict], - *, - session_id: str = SESSION_A, - date: str = "2026-07-03", - terminate: bool = True, -) -> Path: - path = _session_path(home, session_id, date) - path.parent.mkdir(parents=True, exist_ok=True) - path.write_bytes(_jsonl(*records, terminate=terminate)) - return path - - -def _reset_codex() -> None: - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - herdr_turns._CODEX_SESSION_CACHE_LIVE_KEYS = None - herdr_turns._CODEX_SESSION_CACHE_BINDING_GENERATIONS = {} - herdr_turns._CODEX_SESSION_CACHE_BINDING_FINGERPRINTS = {} - - -@pytest.fixture(autouse=True) -def _isolated_codex_state(monkeypatch): - _reset_codex() - monkeypatch.setattr(herdr_turns, "_CODEX_INDEX_BUILD_OBSERVER", None) - monkeypatch.setattr(herdr_turns, "_CODEX_ISOLATED_READ_OBSERVER", None) - yield - _reset_codex() - - -def test_canonical_uuid_rejects_adversarial_spellings_before_filesystem_work( - tmp_path: Path, - monkeypatch, -) -> None: - invalid = [ - "", - "*", - "?", - "[abc]", - ".", - "..", - "/", - "\\", - f"../{SESSION_A}", - f"{SESSION_A}/x", - f"prefix-{SESSION_A}", - f"{SESSION_A}-suffix", - f" {SESSION_A}", - f"{SESSION_A} ", - SESSION_A.upper(), - SESSION_A.replace("-", ""), - "{" + SESSION_A + "}", - "urn:uuid:" + SESSION_A, - SESSION_A.replace("-", "‐"), - SESSION_A.replace("d", "ԁ"), - "00000000-0000-0000-0000-000000000000", - "x" * 10_000, - ] - monkeypatch.setenv("CODEX_HOME", str(tmp_path / "must-not-touch")) - monkeypatch.setattr( - herdr_turns, - "_build_codex_index", - lambda *_args: (_ for _ in ()).throw(AssertionError("invalid identity walked")), - ) - for value in invalid: - assert herdr_turns._canonical_codex_session_id(value) is None - assert herdr_turns._find_codex_session_file(value) is None - assert herdr_turns._canonical_codex_session_id(SESSION_A) == SESSION_A - assert herdr_turns._canonical_codex_session_id(SESSION_C) == SESSION_C - - -def test_invalid_identity_is_rejected_before_socket_or_process(monkeypatch) -> None: - monkeypatch.setattr( - herdr_turns.socket, - "socketpair", - lambda: (_ for _ in ()).throw(AssertionError("socket created")), - ) - assert ( - herdr_turns._read_file_turn_isolated( - "codex_session_id", - "*", - timeout_seconds=1, - ) - is None - ) - - -@pytest.mark.parametrize( - ("parts", "name", "expected"), - [ - (("2026", "07", "03"), f"rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", SESSION_A), - (("2024", "02", "29"), f"rollout-2024-02-29T23-59-59-{SESSION_C}.jsonl", SESSION_C), - (("2026", "07", "04"), f"rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", None), - (("2026", "02", "30"), f"rollout-2026-02-30T00-00-00-{SESSION_A}.jsonl", None), - (("2026", "07", "03"), f"rollout-2026-07-03T24-00-00-{SESSION_A}.jsonl", None), - (("2026", "07", "03"), f"rollout-2026-07-03T00-00-00-{SESSION_A}.JSONL", None), - (("2026", "07", "03"), f"rollout-2026-07-03T00-00-00.1-{SESSION_A}.jsonl", None), - (("2026", "07", "03"), f"copy-rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", None), - (("2026", "07", "03"), f"rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl.zst", None), - ], -) -def test_exact_rollout_filename_and_date_grammar(parts, name, expected) -> None: - assert herdr_turns._codex_rollout_identity(parts, name) == expected - - -def test_resolver_selects_only_exact_regular_and_rejects_decoys_and_symlinks( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - target = _write_session(home, [_event("task_started", "turn-a")]) - decoy = _write_session( - home, - [_event("task_started", "turn-b")], - session_id=SESSION_B, - ) - os.utime(decoy, (2_000_000_000, 2_000_000_000)) - target.parent.joinpath(f"prefix-{SESSION_A}.jsonl").write_bytes(b"decoy\n") - target.parent.joinpath( - f"rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl.zst" - ).write_bytes(b"compressed") - outside = tmp_path / "outside.jsonl" - outside.write_bytes(b"outside") - target.parent.joinpath( - f"rollout-2026-07-03T01-00-00-{SESSION_C}.jsonl" - ).symlink_to(outside) - monkeypatch.setenv("CODEX_HOME", str(home)) - - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert herdr_turns._find_codex_session_file(SESSION_B) == decoy.resolve() - assert herdr_turns._find_codex_session_file(SESSION_C) is None - - -def test_duplicate_exact_identity_is_ambiguous_independent_of_mtime( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - first = _write_session(home, [], date="2026-07-03") - second = _write_session(home, [], date="2026-07-04") - os.utime(first, (2_000_000_000, 2_000_000_000)) - os.utime(second, (1, 1)) - monkeypatch.setenv("CODEX_HOME", str(home)) - - resolution = herdr_turns._resolve_codex_session(SESSION_A) - assert resolution is not None - assert resolution.status == "ambiguous" - assert herdr_turns._find_codex_session_file(SESSION_A) is None - - -def test_complete_34k_index_build_is_bounded_and_deterministic(monkeypatch) -> None: - root = Path("/virtual/sessions") - - class Entry: - def __init__(self, path: str, name: str, kind: str): - self.path = path - self.name = name - self.kind = kind - - def is_dir(self, *, follow_symlinks: bool) -> bool: - assert follow_symlinks is False - return self.kind == "dir" - - def is_file(self, *, follow_symlinks: bool) -> bool: - assert follow_symlinks is False - return self.kind == "file" - - year = Entry(f"{root}/2026", "2026", "dir") - month = Entry(f"{year.path}/07", "07", "dir") - day = Entry(f"{month.path}/03", "03", "dir") - files = [] - for ordinal in range(1, 34_001): - session_id = str(UUID(int=ordinal)) - name = f"rollout-2026-07-03T00-00-00-{session_id}.jsonl" - files.append(Entry(f"{day.path}/{name}", name, "file")) - tree = { - str(root): [year], - year.path: [month], - month.path: [day], - day.path: files, - } - monkeypatch.setattr(herdr_turns.os, "scandir", lambda path: tree[os.fspath(path)]) - monkeypatch.setattr(herdr_turns, "_codex_root_signature", lambda _root: (1, 2, 3, 4)) - - generation = herdr_turns._build_codex_index(root) - assert generation.overflowed is False - assert len(generation.entries) == 34_000 - assert generation.entries[str(UUID(int=1))][0].endswith("000000000001.jsonl") - assert generation.entries[str(UUID(int=34_000))][0].endswith("0000000084d0.jsonl") - assert generation.visited == 34_003 - assert generation.retained_bytes <= herdr_turns._CODEX_INDEX_MAX_BYTES - - -def test_index_iterator_stops_at_first_overflow_sentinel(monkeypatch) -> None: - produced = 0 - closed = 0 - - class Entry: - def __init__(self, ordinal: int): - self.name = f"entry-{ordinal}" - self.path = f"/virtual/{self.name}" - - class UnboundedDirectory: - def __iter__(self): - return self - - def __next__(self): - nonlocal produced - produced += 1 - return Entry(produced) - - def close(self): - nonlocal closed - closed += 1 - - observed = [] - monkeypatch.setattr(herdr_turns, "_CODEX_INDEX_MAX_VISITS", 5) - monkeypatch.setattr( - herdr_turns, - "_codex_root_signature", - lambda _root: (1, 2, 3, 4), - ) - monkeypatch.setattr( - herdr_turns.os, - "scandir", - lambda _path: UnboundedDirectory(), - ) - monkeypatch.setattr( - herdr_turns, - "_CODEX_INDEX_BUILD_OBSERVER", - observed.append, - ) - - generation = herdr_turns._build_codex_index(Path("/virtual")) - - assert generation.overflowed is True - assert generation.entries == {} - assert generation.visited == 6 - assert produced == 6 - assert closed == 1 - assert observed == [6] - - -def test_warm_positive_and_negative_lookup_use_one_index_build( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - target = _write_session(home, []) - monkeypatch.setenv("CODEX_HOME", str(home)) - visits = [] - monkeypatch.setattr(herdr_turns, "_CODEX_INDEX_BUILD_OBSERVER", visits.append) - clock = [100.0] - monkeypatch.setattr(herdr_turns.time, "monotonic", lambda: clock[0]) - - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert herdr_turns._find_codex_session_file(SESSION_B) is None - assert herdr_turns._find_codex_session_file(SESSION_B) is None - assert len(visits) == 1 - clock[0] += herdr_turns._CODEX_NEGATIVE_TTL_SECONDS + 0.5 - assert herdr_turns._find_codex_session_file(SESSION_B) is None - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert len(visits) == 1 - clock[0] += herdr_turns._CODEX_POSITIVE_TTL_SECONDS - assert herdr_turns._find_codex_session_file(SESSION_A) == target.resolve() - assert len(visits) == 2 - - -def test_pure_interpreter_preserves_turn_id_precedence_and_stream_window() -> None: - work = herdr_turns._CodexWorkState( - resolver_generation=1, - root="/root", - root_file_id=(9, 9), - session_id=SESSION_A, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", - file_id=(1, 2), - observed_size=0, - mtime_ns=0, - ctime_ns=0, - ) - start = herdr_turns._codex_record_event(_event("task_started", "active")) - herdr_turns._apply_codex_event(work, start, herdr_turns._CodexRecordSpan(0, 1)) - for index, text in enumerate(("one", "two", "three", "four", "two", "five"), 1): - event = herdr_turns._codex_record_event( - _message("active", "assistant", text, phase="commentary") - ) - herdr_turns._apply_codex_event( - work, - event, - herdr_turns._CodexRecordSpan(index * 10, index * 10 + 5), - ) - assert [text for _span, text in work.stream_items] == ["three", "four", "two", "five"] - direct = { - "type": "response_item", - "payload": { - "type": "message", - "role": "user", - "turn_id": "direct", - "content": [{"text": "x"}], - "internal_chat_message_metadata_passthrough": {"turn_id": "metadata"}, - }, - } - assert herdr_turns._codex_record_event(direct).turn_id == "direct" - - -def test_partial_record_is_invisible_until_lf_and_commits_once( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "turn-partial" - base = [_event("task_started", turn_id), _message(turn_id, "user", "prompt")] - path = _write_session(home, base) - completion = _jsonl( - _event("task_complete", turn_id, last_agent_message="final once"), - terminate=False, - ) - split = len(completion) // 2 - with path.open("ab") as handle: - handle.write(completion[:split]) - monkeypatch.setenv("CODEX_HOME", str(home)) - observed = [] - monkeypatch.setattr(herdr_turns, "_CODEX_ISOLATED_READ_OBSERVER", observed.append) - - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["user_text"] == "prompt" - assert first["assistant_final_text"] is None - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - first_state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - committed = first_state.committed_offset - assert first_state.partial_record == completion[:split] - - with path.open("ab") as handle: - handle.write(completion[split:]) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - second_state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - assert second_state.committed_offset == committed - assert second_state.partial_record == completion - - with path.open("ab") as handle: - handle.write(b"\n") - final = herdr_turns._read_codex_session_turn(SESSION_A) - assert final["assistant_final_text"] == "final once" - assert final["complete"] is True - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - assert observed[-1] == 0 - - -def test_sparse_cold_read_and_warm_append_are_bounded( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - path = _session_path(home) - path.parent.mkdir(parents=True) - turn_id = "turn-sparse" - tail = _jsonl( - _event("task_started", turn_id), - _message(turn_id, "user", "sparse prompt"), - ) - with path.open("wb") as handle: - handle.seek(20 * 1024 * 1024) - handle.write(b"\n") - handle.write(tail) - monkeypatch.setenv("CODEX_HOME", str(home)) - observed = [] - monkeypatch.setattr(herdr_turns, "_CODEX_ISOLATED_READ_OBSERVER", observed.append) - - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["user_text"] == "sparse prompt" - assert observed[-1] <= herdr_turns._CODEX_RESYNC_INITIAL_BYTES - append = _jsonl(_message(turn_id, "assistant", "working", phase="commentary")) - with path.open("ab") as handle: - handle.write(append) - second = herdr_turns._read_codex_session_turn(SESSION_A) - assert second["assistant_stream_text"] == "working" - assert observed[-1] == len(append) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - assert observed[-1] == 0 - - -def test_delayed_incremental_poll_publishes_completed_turn_before_next_turn( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - first_turn = "turn-before-gap" - next_turn = "turn-after-gap" - path = _write_session( - home, - [ - _event("task_started", first_turn), - _message(first_turn, "user", "first prompt"), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - opened = herdr_turns._read_codex_session_turn(SESSION_A) - assert opened["source_turn_id"] == first_turn - assert opened["complete"] is False - - final_record = _message( - first_turn, - "assistant", - "first final", - phase="final_answer", - ) - first_batch = _jsonl( - _message(first_turn, "assistant", "first progress", phase="commentary"), - final_record, - ) - later_batch = _jsonl( - _event("task_started", next_turn), - _message(next_turn, "user", "next prompt"), - _message(next_turn, "assistant", "next progress", phase="commentary"), - ) - with path.open("ab") as handle: - handle.write(first_batch) - handle.write(later_batch) - - completed = herdr_turns._read_codex_session_turn(SESSION_A) - assert completed == { - # The coordinate-only checkpoint deliberately retains no canonical - # prompt body. Store merge preserves the previously persisted prompt. - "user_text": None, - "assistant_stream_text": None, - "assistant_final_text": "first final", - "complete": True, - "has_open_turn": False, - "source_turn_id": first_turn, - } - cache_key = (str((home / "sessions").resolve()), SESSION_A) - expected_offset = len( - _jsonl( - _event("task_started", first_turn), - _message(first_turn, "user", "first prompt"), - ) - ) + len(first_batch) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - completed_state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - assert completed_state.committed_offset == expected_offset - assert completed_state.observed_size == expected_offset - assert completed_state.partial_record == b"" - - following = herdr_turns._read_codex_session_turn(SESSION_A) - assert following["source_turn_id"] == next_turn - assert following["user_text"] == "next prompt" - assert following["assistant_stream_text"] == "next progress" - assert following["assistant_final_text"] is None - assert following["complete"] is False - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - - -def test_malformed_and_oversized_records_block_without_checkpoint_advance( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - path = _write_session( - home, - [_event("task_started", "blocked"), _message("blocked", "user", "safe")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["user_text"] == "safe" - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - before = herdr_turns._serialize_codex_state( - herdr_turns._CODEX_SESSION_CACHE[cache_key] - ) - with path.open("ab") as handle: - handle.write(b"{not-json}\n") - with pytest.raises(ValueError, match="invalid Codex record"): - herdr_turns._read_codex_session_turn(SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert herdr_turns._serialize_codex_state( - herdr_turns._CODEX_SESSION_CACHE[cache_key] - ) == before - - path.write_bytes(_jsonl(_event("task_started", "oversize")) + b"x" * 65) - monkeypatch.setattr(herdr_turns, "_CODEX_RECORD_MAX_BYTES", 64) - _reset_codex() - with pytest.raises(ValueError, match="oversized Codex record"): - herdr_turns._read_codex_session_turn(SESSION_A) - - -def test_truncate_and_inode_replacement_recover_latest_exact_turn( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - old_prompt = "old prompt " + ("x" * 1024) - old = [_event("task_started", "old"), _message("old", "user", old_prompt)] - path = _write_session(home, old) - monkeypatch.setenv("CODEX_HOME", str(home)) - assert herdr_turns._read_codex_session_turn(SESSION_A)["source_turn_id"] == "old" - - new_prompt = "new prompt" - new = [_event("task_started", "new"), _message("new", "user", new_prompt)] - path.write_bytes(_jsonl(*new)) - os.utime(path, ns=(1_800_000_000_000_000_000, 1_800_000_000_000_000_000)) - truncated = herdr_turns._read_codex_session_turn(SESSION_A) - assert truncated["source_turn_id"] == "new" - assert truncated["user_text"] == new_prompt - - replacement = path.with_name("replacement.tmp") - newest = [ - _event("task_started", "replacement"), - _message("replacement", "user", "replacement prompt"), - ] - replacement.write_bytes(_jsonl(*newest)) - replacement.replace(path) - replaced = herdr_turns._read_codex_session_turn(SESSION_A) - assert replaced["source_turn_id"] == "replacement" - assert replaced["user_text"] == "replacement prompt" - - -def test_huge_admitted_prompt_and_final_are_exact_and_not_cached( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - prompt = "π" * 300_000 - final = "終" * 300_000 - turn_id = "huge" - _write_session( - home, - [ - _event("task_started", turn_id), - _message(turn_id, "user", prompt), - _event("task_complete", turn_id, last_agent_message=final), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - content = herdr_turns._read_codex_session_turn(SESSION_A) - assert content["user_text"] == prompt - assert content["assistant_final_text"] == final - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - serialized = herdr_turns._serialize_codex_state( - herdr_turns._CODEX_SESSION_CACHE[cache_key] - ) - private_json = json.dumps(serialized, separators=(",", ":")) - assert prompt[:100] not in private_json - assert final[:100] not in private_json - assert serialized["stream_spans"] == [] - - -def test_parser_lru_enforces_count_and_weight_with_mru_touch(monkeypatch) -> None: - monkeypatch.setattr(herdr_turns, "_CODEX_SESSION_CACHE_CAPACITY", 3) - monkeypatch.setattr(herdr_turns, "_CODEX_SESSION_CACHE_MAX_BYTES", 64 * 1024) - - def state(session_id: str, ordinal: int): - return herdr_turns._CodexSessionState( - resolver_generation=1, - root="/root", - root_file_id=(9, 9), - session_id=session_id, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{session_id}.jsonl", - file_id=(1, ordinal), - observed_size=ordinal, - mtime_ns=ordinal, - ctime_ns=ordinal, - committed_offset=ordinal, - partial_record=b"", - active_turn_id="", - last_content_turn_id="", - turn_open=False, - final_seen=False, - complete=False, - stream_spans=(), - ) - - ids = [str(UUID(int=index)) for index in range(1, 5)] - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - for index, session_id in enumerate(ids[:3], 1): - key = ("/root", session_id) - assert herdr_turns._codex_cache_store_locked(key, state(session_id, index)) - herdr_turns._codex_cache_get_locked(("/root", ids[0])) - herdr_turns._codex_cache_store_locked(("/root", ids[3]), state(ids[3], 4)) - assert [key[1] for key in herdr_turns._CODEX_SESSION_CACHE] == [ - ids[2], - ids[0], - ids[3], - ] - assert herdr_turns._codex_cache_weight_locked() <= 64 * 1024 - - -def test_state_deserializer_rejects_bodies_overlap_and_wrong_rollout_path() -> None: - state = herdr_turns._CodexSessionState( - resolver_generation=1, - root="/root", - root_file_id=(9, 9), - session_id=SESSION_A, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", - file_id=(1, 2), - observed_size=100, - mtime_ns=1, - ctime_ns=1, - committed_offset=100, - partial_record=b"", - active_turn_id="turn", - last_content_turn_id="turn", - turn_open=True, - final_seen=False, - complete=False, - stream_spans=(herdr_turns._CodexRecordSpan(10, 20),), - ) - serialized = herdr_turns._serialize_codex_state(state) - assert herdr_turns._deserialize_codex_state(serialized) == state - body = dict(serialized) - body["user_text"] = "forbidden" - with pytest.raises(ValueError, match="invalid Codex parser state"): - herdr_turns._deserialize_codex_state(body) - overlap = dict(serialized) - overlap["stream_spans"] = [[10, 20], [20, 30]] - with pytest.raises(ValueError, match="overlapping"): - herdr_turns._deserialize_codex_state(overlap) - wrong = dict(serialized) - wrong["canonical_path"] = f"/root/2026/07/04/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl" - with pytest.raises(ValueError, match="invalid Codex rollout path"): - herdr_turns._deserialize_codex_state(wrong) - - -def test_partial_completion_is_transactional_at_every_byte_split( - tmp_path: Path, - monkeypatch, -) -> None: - turn_id = "all-splits" - completion = _jsonl( - _event("task_complete", turn_id, last_agent_message="split final"), - terminate=False, - ) - for split in range(len(completion) + 1): - _reset_codex() - home = tmp_path / f"split-{split}" - path = _write_session( - home, - [_event("task_started", turn_id), _message(turn_id, "user", "split prompt")], - ) - base_size = path.stat().st_size - with path.open("ab") as handle: - handle.write(completion[:split]) - monkeypatch.setenv("CODEX_HOME", str(home)) - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["assistant_final_text"] is None - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - assert state.committed_offset == base_size - assert state.partial_record == completion[:split] - if split < len(completion): - with path.open("ab") as handle: - handle.write(completion[split:]) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - with path.open("ab") as handle: - handle.write(b"\n") - completed = herdr_turns._read_codex_session_turn(SESSION_A) - assert completed["assistant_final_text"] == "split final" - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - - -def test_path_result_lru_has_deterministic_mru_and_capacity( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - session_ids = [str(UUID(int=index)) for index in range(101, 105)] - for session_id in session_ids: - _write_session(home, [], session_id=session_id) - monkeypatch.setenv("CODEX_HOME", str(home)) - monkeypatch.setattr(herdr_turns, "_CODEX_PATH_CACHE_CAPACITY", 3) - for session_id in session_ids[:3]: - assert herdr_turns._resolve_codex_session(session_id).status == "found" - assert herdr_turns._resolve_codex_session(session_ids[0]).status == "found" - assert herdr_turns._resolve_codex_session(session_ids[3]).status == "found" - with herdr_turns._CODEX_PATH_CACHE_LOCK: - assert [key[1] for key in herdr_turns._CODEX_PATH_CACHE] == [ - session_ids[2], - session_ids[0], - session_ids[3], - ] - assert ( - herdr_turns._codex_path_cache_weight_locked() - <= herdr_turns._CODEX_PATH_CACHE_MAX_BYTES - ) - - -def test_rotation_duplicate_and_parser_cold_start_are_fail_closed( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - path = _write_session( - home, - [_event("task_started", "rotation"), _message("rotation", "user", "rotating")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - assert herdr_turns._read_codex_session_turn(SESSION_A)["user_text"] == "rotating" - - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - cold = herdr_turns._read_codex_session_turn(SESSION_A) - assert cold["source_turn_id"] == "rotation" - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - fully_cold = herdr_turns._read_codex_session_turn(SESSION_A) - assert fully_cold["user_text"] == "rotating" - - duplicate = _session_path(home, SESSION_A, "2026-07-04") - duplicate.parent.mkdir(parents=True) - duplicate.write_bytes(path.read_bytes()) - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - assert herdr_turns._find_codex_session_file(SESSION_A) is None - assert herdr_turns._resolve_codex_session(SESSION_A).status == "ambiguous" - - -def test_long_warm_turn_advances_beyond_cold_resync_horizon( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "long-warm" - path = _write_session( - home, - [_event("task_started", turn_id), _message(turn_id, "user", "long prompt")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - monkeypatch.setattr(herdr_turns, "_CODEX_RESYNC_MAX_BYTES", 1024) - observed = [] - monkeypatch.setattr(herdr_turns, "_CODEX_ISOLATED_READ_OBSERVER", observed.append) - first = herdr_turns._read_codex_session_turn(SESSION_A) - assert first["user_text"] == "long prompt" - - newest = [] - for index in range(12): - text = f"progress-{index}-" + ("x" * 220) - newest.append(text) - append = _jsonl(_message(turn_id, "assistant", text, phase="commentary")) - with path.open("ab") as handle: - handle.write(append) - current = herdr_turns._read_codex_session_turn(SESSION_A) - assert current["source_turn_id"] == turn_id - assert current["complete"] is False - assert current["assistant_stream_text"].split("\n\n") == newest[-4:] - assert observed[-1] <= len(append) + 4 * herdr_turns._CODEX_RECORD_MAX_BYTES - assert path.stat().st_size > herdr_turns._CODEX_RESYNC_MAX_BYTES - - final_record = _jsonl( - _event("task_complete", turn_id, last_agent_message="long exact final") - ) - with path.open("ab") as handle: - handle.write(final_record) - final = herdr_turns._read_codex_session_turn(SESSION_A) - assert final["assistant_final_text"] == "long exact final" - assert final["assistant_stream_text"] is None - assert final["complete"] is True - - -def test_codex_cache_cas_is_monotone_and_does_not_resurrect_eviction() -> None: - cache_key = ("/root", SESSION_A) - prior = herdr_turns._CodexSessionState( - resolver_generation=1, - root="/root", - root_file_id=(9, 9), - session_id=SESSION_A, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", - file_id=(1, 2), - observed_size=10, - mtime_ns=1, - ctime_ns=1, - committed_offset=10, - partial_record=b"", - active_turn_id="turn", - last_content_turn_id="turn", - turn_open=True, - final_seen=False, - complete=False, - stream_spans=(), - ) - newer = replace( - prior, - observed_size=30, - mtime_ns=3, - ctime_ns=3, - committed_offset=30, - ) - older = replace( - prior, - observed_size=20, - mtime_ns=2, - ctime_ns=2, - committed_offset=20, - ) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE[cache_key] = prior - prior_value = herdr_turns._serialize_codex_state(prior) - assert herdr_turns._publish_codex_cache_state( - cache_key, - prior_value, - newer, - {"source_turn_id": "turn"}, - None, - ) == {"source_turn_id": "turn"} - assert herdr_turns._publish_codex_cache_state( - cache_key, - prior_value, - older, - {"source_turn_id": "turn"}, - None, - ) is None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert herdr_turns._CODEX_SESSION_CACHE[cache_key].committed_offset == 30 - herdr_turns._CODEX_SESSION_CACHE.clear() - assert herdr_turns._publish_codex_cache_state( - cache_key, - prior_value, - newer, - {"source_turn_id": "turn"}, - None, - ) is None - - -def test_symlinked_sessions_root_is_rejected( - tmp_path: Path, - monkeypatch, -) -> None: - outside_home = tmp_path / "outside" - target = _write_session(outside_home, [_event("task_started", "outside")]) - configured = tmp_path / "configured" - configured.mkdir() - (configured / "sessions").symlink_to(outside_home / "sessions", target_is_directory=True) - monkeypatch.setenv("CODEX_HOME", str(configured)) - - assert target.is_file() - assert herdr_turns._find_codex_session_file(SESSION_A) is None - - -def test_internal_turn_remains_suppressed_when_later_commentary_arrives( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "internal-turn" - path = _write_session( - home, - [ - _event("task_started", turn_id), - _message( - turn_id, - "user", - "Acme job\n\nTemplate: security-review\nTemplate instructions:", - ), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - cache_key = (str((home / "sessions").resolve()), SESSION_A) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert herdr_turns._CODEX_SESSION_CACHE[cache_key].internal_turn is True - - with path.open("ab") as handle: - handle.write( - _jsonl( - _message( - turn_id, - "assistant", - "ordinary-looking internal progress", - phase="commentary", - ) - ) - ) - assert herdr_turns._read_codex_session_turn(SESSION_A) is None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - state = herdr_turns._CODEX_SESSION_CACHE[cache_key] - assert state.internal_turn is True - assert state.stream_spans == () - - -def test_real_user_after_environment_context_makes_turn_public( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "context-then-user" - _write_session( - home, - [ - _event("task_started", turn_id), - _message( - turn_id, - "user", - "\n /private/path\n", - ), - _message(turn_id, "user", "fix the Telegram delivery"), - _message( - turn_id, - "assistant", - "Tracing the delivery path.", - phase="commentary", - ), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - content = herdr_turns._read_codex_session_turn(SESSION_A) - - assert content == { - "user_text": "fix the Telegram delivery", - "assistant_stream_text": "Tracing the delivery path.", - "assistant_final_text": None, - "complete": False, - "has_open_turn": True, - "source_turn_id": turn_id, - } - assert "/private/path" not in json.dumps(content) - - -def test_internal_context_after_real_user_does_not_hide_turn( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "user-then-context" - _write_session( - home, - [ - _event("task_started", turn_id), - _message(turn_id, "user", "keep this prompt"), - _message( - turn_id, - "user", - "private runtime metadata", - ), - _message( - turn_id, - "assistant", - "Visible progress.", - phase="commentary", - ), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - content = herdr_turns._read_codex_session_turn(SESSION_A) - - assert content["user_text"] == "keep this prompt" - assert content["assistant_stream_text"] == "Visible progress." - assert "private runtime metadata" not in json.dumps(content) - - -def test_empty_task_complete_preserves_existing_open_turn_semantics( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - turn_id = "empty-complete" - _write_session( - home, - [ - _event("task_started", turn_id), - _message(turn_id, "user", "still open by existing contract"), - _event("task_complete", turn_id), - ], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - - content = herdr_turns._read_codex_session_turn(SESSION_A) - assert content["user_text"] == "still open by existing contract" - assert content["assistant_final_text"] is None - assert content["complete"] is False - assert content["has_open_turn"] is True - - -def test_descriptor_relative_open_rejects_ancestor_symlink_swap( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - path = _write_session( - home, - [_event("task_started", "inside"), _message("inside", "user", "inside")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - resolution = herdr_turns._resolve_codex_session(SESSION_A) - assert resolution is not None and resolution.status == "found" - - outside_day = tmp_path / "outside" / "2026" / "07" / "03" - outside_day.mkdir(parents=True) - outside_file = outside_day / path.name - outside_file.write_bytes( - _jsonl( - _event("task_started", "outside"), - _message("outside", "user", "outside sentinel"), - ) - ) - original_open = herdr_turns.os.open - swapped = False - - def racing_open(path_value, flags, mode=0o777, *, dir_fd=None): - nonlocal swapped - if path_value == path.name and dir_fd is not None and not swapped: - swapped = True - saved = path.parent.with_name("03-saved") - path.parent.rename(saved) - path.parent.symlink_to(outside_day, target_is_directory=True) - return original_open(path_value, flags, mode, dir_fd=dir_fd) - - monkeypatch.setattr(herdr_turns.os, "open", racing_open) - with pytest.raises(herdr_turns._TurnReadFailed): - herdr_turns._open_verified_codex_file(resolution) - assert swapped is True - assert outside_file.read_bytes().endswith(b"\n") - - -def test_sessions_root_swap_during_resolution_is_rejected( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "configured" - _write_session(home, [_event("task_started", "inside")]) - outside = tmp_path / "outside" - _write_session(outside, [_event("task_started", "outside")]) - monkeypatch.setenv("CODEX_HOME", str(home)) - lexical_root = home / "sessions" - original_resolve = Path.resolve - swapped = False - - def racing_resolve(path_value, *args, **kwargs): - nonlocal swapped - if path_value == lexical_root and not swapped: - swapped = True - lexical_root.rename(home / "sessions-saved") - lexical_root.symlink_to(outside / "sessions", target_is_directory=True) - return original_resolve(path_value, *args, **kwargs) - - monkeypatch.setattr(Path, "resolve", racing_resolve) - assert herdr_turns._find_codex_session_file(SESSION_A) is None - assert swapped is True - - -def test_incremental_poll_limit_rejects_before_reading() -> None: - prior = herdr_turns._CodexSessionState( - resolver_generation=1, - root="/root", - root_file_id=(1, 1), - session_id=SESSION_A, - canonical_path=f"/root/2026/07/03/rollout-2026-07-03T00-00-00-{SESSION_A}.jsonl", - file_id=(2, 2), - observed_size=10, - mtime_ns=1, - ctime_ns=1, - committed_offset=10, - partial_record=b"", - active_turn_id="turn", - last_content_turn_id="turn", - turn_open=True, - final_seen=False, - complete=False, - stream_spans=(), - ) - opened = SimpleNamespace( - st_dev=2, - st_ino=2, - st_size=10 + herdr_turns._CODEX_POLL_MAX_BYTES + 1, - st_mtime_ns=2, - st_ctime_ns=2, - ) - with pytest.raises(ValueError, match="poll byte limit"): - herdr_turns._read_codex_incremental(-1, prior, opened) - - -def test_ipc_frame_bound_covers_maximum_compact_state_and_visible_turn() -> None: - assert herdr_turns._CODEX_IPC_FRAME_MAX_BYTES >= ( - herdr_turns._CODEX_STATE_IPC_MAX_BYTES - + (1 + herdr_turns._MAX_CODEX_STREAM_MESSAGES) - * herdr_turns._CODEX_RECORD_MAX_BYTES - ) - - -def test_positive_snapshot_discovers_duplicate_by_named_refresh_bound( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - original = _write_session(home, [], date="2026-07-03") - monkeypatch.setenv("CODEX_HOME", str(home)) - clock = [500.0] - monkeypatch.setattr(herdr_turns.time, "monotonic", lambda: clock[0]) - visits = [] - monkeypatch.setattr(herdr_turns, "_CODEX_INDEX_BUILD_OBSERVER", visits.append) - - assert herdr_turns._find_codex_session_file(SESSION_A) == original.resolve() - duplicate = _write_session(home, [], date="2026-07-04") - assert duplicate.is_file() - assert herdr_turns._find_codex_session_file(SESSION_A) == original.resolve() - assert len(visits) == 1 - - clock[0] += herdr_turns._CODEX_POSITIVE_TTL_SECONDS + 0.001 - assert herdr_turns._find_codex_session_file(SESSION_A) is None - assert herdr_turns._resolve_codex_session(SESSION_A).status == "ambiguous" - assert len(visits) == 2 - - -def test_root_replacement_invalidates_found_and_nonfound_even_with_same_rollout_inode( - tmp_path: Path, - monkeypatch, -) -> None: - home = tmp_path / "codex" - original_path = _write_session( - home, - [_event("task_started", "root-one")], - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - builds = [] - monkeypatch.setattr( - herdr_turns, - "_CODEX_INDEX_BUILD_OBSERVER", - builds.append, - ) - - first = herdr_turns._resolve_codex_session(SESSION_A) - assert first is not None and first.status == "found" - assert herdr_turns._resolve_codex_session(SESSION_B).status == "missing" - assert len(builds) == 1 - original_file_id = first.file_id - original_root_id = first.root_file_id - - old_root = home / "sessions-old" - (home / "sessions").rename(old_root) - replacement_path = _session_path(home) - replacement_path.parent.mkdir(parents=True) - os.link( - old_root / replacement_path.relative_to(home / "sessions"), - replacement_path, - ) - assert ( - replacement_path.stat().st_dev, - replacement_path.stat().st_ino, - ) == original_file_id - - refreshed_found = herdr_turns._resolve_codex_session(SESSION_A) - assert refreshed_found is not None - assert refreshed_found.status == "found" - assert refreshed_found.file_id == original_file_id - assert refreshed_found.root_file_id != original_root_id - assert refreshed_found.generation != first.generation - assert len(builds) == 2 - refreshed_missing = herdr_turns._resolve_codex_session(SESSION_B) - assert refreshed_missing is not None - assert refreshed_missing.status == "missing" - assert refreshed_missing.root_file_id == refreshed_found.root_file_id - assert len(builds) == 2 diff --git a/tests/test_command_presend_retryability.py b/tests/test_command_presend_retryability.py deleted file mode 100644 index 9a01d97..0000000 --- a/tests/test_command_presend_retryability.py +++ /dev/null @@ -1,1076 +0,0 @@ -"""Transient pre-send failures stay retryable; permanent ones stay terminal. - -Goal 11B made an existing receipt authoritative for its retry. Goal 11C closes -the gap that stress testing exposed: a transient *local* failure before any -backend send -- the binding store or receipt store raising, a socket that will -not connect, a pane read that times out -- was reserved and written as a durable -``terminal_rejected`` receipt, permanently dropping a command that was never -sent. - -The corrected rule classifies a pre-send failure by the last irreversible stage -it reached. A failed local or backend *operation* proves nothing durable and -stays ``no_receipt`` / retryable under the same request ID. Only an authoritative -observation of proven target unsuitability -- a disallowed worker status, an -unavailable backend, or a missing/stale/ambiguous private binding -- may -terminalize. Anything after a send may have started stays terminal uncertainty. -""" - -from __future__ import annotations - -import sqlite3 -import threading -from pathlib import Path -from typing import Any - -import pytest - -import tendwire.command_submission as command_submission - -from tendwire.backends.herdr_socket import HerdrSocketTimeoutError -from tendwire.command_submission import submit_command -from tendwire.config import Config -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - DISPOSITION_TERMINAL_ACCEPTED, - DISPOSITION_TERMINAL_REJECTED, - STATUS_ACCEPTED, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_PENDING, - STATUS_REJECTED, - STATUS_STALE_TARGET, -) -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.daemon_api import TendwireDaemonAPI -from tendwire.local_state import LocalStateError, LocalStateErrorCode -from tendwire.store.sqlite import ( - get_command_request, - init_store, - save_snapshot, - upsert_worker_bindings, -) - - -HOST_ID = "cmd-host" - - -def _config(tmp_path: Path) -> Config: - return Config( - host_id=HOST_ID, - data_dir=tmp_path, - db_path=tmp_path / "commands.db", - herdr_backend="socket", - herdr_timeout_seconds=5.0, - ) - - -def _worker(*, worker_id: str = "w-1", status: str = "active") -> Worker: - return Worker(id=worker_id, name="Alpha", status=status, space_id="space-1") - - -def _binding( - worker: Worker, - *, - sendable: bool = True, - fingerprint: str | None = None, -) -> WorkerBinding: - return WorkerBinding( - host_id=HOST_ID, - worker_id=worker.id, - worker_fingerprint=fingerprint or worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value=f"agent-{worker.id}", - turn_target_kind="pane_id", - turn_target_value=f"pane-{worker.id}", - sendable=sendable, - reason=None if sendable else "not_sendable", - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint=f"private-{worker.id}", - ) - - -def _health(status: str = "healthy") -> BackendHealth: - return BackendHealth( - name="herdr", - status=status, - outcome="healthy_non_empty" if status == "healthy" else "timeout", - observed_at="2026-01-01T00:00:00+00:00", - counts={"workers": 1}, - ) - - -def _seed( - config: Config, - workers: list[Worker], - bindings: list[WorkerBinding], - *, - health: str = "healthy", -) -> None: - assert config.db_path is not None - init_store(config.db_path) - save_snapshot( - config.db_path, - Snapshot( - host_id=HOST_ID, - updated_at="2026-01-01T00:00:00+00:00", - workers=workers, - backend_health=[_health(health)], - ), - ) - if bindings: - upsert_worker_bindings(config.db_path, bindings) - - -def _request( - *, - request_id: str, - worker_id: str = "w-1", - text: str = "hello", -) -> dict[str, Any]: - return { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": worker_id}, - "instruction": {"text": text}, - } - - -class _FakeSocketClient: - def __init__( - self, - calls: list[dict[str, Any]], - *, - agent_get_raises: BaseException | None = None, - agent_get_response: dict[str, Any] | None = None, - ) -> None: - self.calls = calls - self.agent_get_raises = agent_get_raises - self.agent_get_response = agent_get_response - - def connect(self) -> "_FakeSocketClient": - return self - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if method == "agent.get": - if self.agent_get_raises is not None: - raise self.agent_get_raises - if self.agent_get_response is not None: - return self.agent_get_response - return {"result": {"agent": {"pane_id": "pane-w-1"}}} - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": "Completed previous turn.\n── status: idle ──"}, - } - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": "pane-w-1"}, - "delivery": "submitted", - } - return {"accepted": True} - - def close(self) -> None: - return None - - -def _factory(calls: list[dict[str, Any]], **kwargs: Any): - def make_client(config: Config) -> _FakeSocketClient: - return _FakeSocketClient(calls, **kwargs) - - return make_client - - -def _forbidden_factory(config: Config) -> Any: - pytest.fail("a receipt replay must not create a socket client") - - -def _sent_texts(calls: list[dict[str, Any]]) -> list[str]: - return [ - str(call["params"].get("text")) - for call in calls - if call["method"] == "agent.prompt" - ] - - -def _receipt(config: Config, request_id: str) -> dict[str, Any] | None: - assert config.db_path is not None - return get_command_request(config.db_path, config.host_id, request_id) - - -def _receipt_count(config: Config) -> int: - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - return conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] - - -# --------------------------------------------------------------------------- -# Deterministic injected transients: each must stay retryable -# --------------------------------------------------------------------------- - - -class _TransientInjection: - """One armed pre-send transient that clears after the first attempt.""" - - def __init__(self, kind: str) -> None: - self.kind = kind - self.armed = True - - def install( - self, - monkeypatch: pytest.MonkeyPatch, - calls: list[dict[str, Any]], - ) -> Any: - real_bindings = command_submission.list_worker_bindings - real_latest_snapshot = command_submission.latest_snapshot - real_reserve = command_submission.reserve_command_request - - if self.kind == "snapshot_store_local_state": - def latest_snapshot(*a: Any, **k: Any) -> Any: - if self.armed: - raise LocalStateError( - LocalStateErrorCode.ENTRY_CHANGED, - LocalStateErrorCode.ENTRY_CHANGED, - "local-state entry changed during validation", - ) - return real_latest_snapshot(*a, **k) - - monkeypatch.setattr(command_submission, "latest_snapshot", latest_snapshot) - return _factory(calls) - - if self.kind == "binding_store_sqlite": - def bindings(*a: Any, **k: Any) -> Any: - if self.armed: - raise sqlite3.OperationalError("database is locked") - return real_bindings(*a, **k) - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings) - return _factory(calls) - - if self.kind == "binding_store_local_state": - def bindings(*a: Any, **k: Any) -> Any: - if self.armed: - raise LocalStateError( - LocalStateErrorCode.ENTRY_CHANGED, - "local-state entry changed during validation", - ) - return real_bindings(*a, **k) - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings) - return _factory(calls) - - if self.kind == "receipt_store_open": - def reserve(*a: Any, **k: Any) -> Any: - if self.armed: - raise sqlite3.OperationalError("database is locked") - return real_reserve(*a, **k) - - monkeypatch.setattr(command_submission, "reserve_command_request", reserve) - return _factory(calls) - - if self.kind == "socket_connect": - def make_client(config: Config) -> Any: - if self.armed: - raise ConnectionRefusedError("socket refused") - return _FakeSocketClient(calls) - - return make_client - - if self.kind == "pane_resolution": - def make_client(config: Config) -> _FakeSocketClient: - if self.armed: - return _FakeSocketClient( - calls, - agent_get_raises=HerdrSocketTimeoutError("pane read timeout"), - ) - return _FakeSocketClient(calls) - - return make_client - - raise AssertionError(f"unknown injection {self.kind!r}") - - -TRANSIENT_KINDS = [ - "snapshot_store_local_state", - "binding_store_sqlite", - "binding_store_local_state", - "receipt_store_open", - "socket_connect", - "pane_resolution", -] - - -@pytest.mark.parametrize("kind", TRANSIENT_KINDS) -def test_pre_send_transient_stays_retryable_then_succeeds_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - kind: str, -) -> None: - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - injection = _TransientInjection(kind) - factory = injection.install(monkeypatch, calls) - request_id = f"transient-{kind}" - - first = submit_command(config, _request(request_id=request_id), socket_client_factory=factory) - - # No external mutation began and no durable authority was written. - assert first.ok is False - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_NO_RECEIPT - assert _sent_texts(calls) == [] - assert _receipt(config, request_id) is None - assert _receipt_count(config) == 0 - - # The transient clears; the same request ID succeeds exactly once. - injection.armed = False - recovered = submit_command( - config, _request(request_id=request_id), socket_client_factory=factory - ) - assert recovered.status == STATUS_ACCEPTED - assert recovered.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert _sent_texts(calls) == ["hello"] - receipt = _receipt(config, request_id) - assert receipt is not None - assert receipt["state"] == "accepted" - - # A later replay returns the stored accepted result without another send. - replay = submit_command( - config, _request(request_id=request_id), socket_client_factory=_forbidden_factory - ) - assert replay.to_dict() == recovered.to_dict() - assert _sent_texts(calls) == ["hello"] - - -@pytest.mark.parametrize("kind", TRANSIENT_KINDS) -def test_pre_send_transient_never_requires_a_new_request_id( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - kind: str, -) -> None: - """Retrying is legal under the SAME id; a different id would double-send.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - injection = _TransientInjection(kind) - factory = injection.install(monkeypatch, calls) - - first = submit_command( - config, _request(request_id="same-id"), socket_client_factory=factory - ) - assert first.disposition == DISPOSITION_NO_RECEIPT - - # Two more attempts under the same id while still transient: still no receipt, - # still no send. Nothing accumulates. - injection.armed = True - again = submit_command( - config, _request(request_id="same-id"), socket_client_factory=factory - ) - assert again.disposition == DISPOSITION_NO_RECEIPT - assert _receipt_count(config) == 0 - assert _sent_texts(calls) == [] - - injection.armed = False - done = submit_command( - config, _request(request_id="same-id"), socket_client_factory=factory - ) - assert done.status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - # Exactly one receipt for the one request id. - assert _receipt_count(config) == 1 - - -def test_receipt_store_transient_closes_the_prepared_client( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A receipt-store transient after a successful prepare must not leak the socket.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - closed: list[bool] = [] - - class _TrackingClient(_FakeSocketClient): - def close(self) -> None: - closed.append(True) - - def factory(_config: Config) -> _TrackingClient: - return _TrackingClient([]) - - def reserve_raises(*a: Any, **k: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "reserve_command_request", reserve_raises) - - envelope = submit_command( - config, _request(request_id="store-transient"), socket_client_factory=factory - ) - - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert _receipt(config, "store-transient") is None - # The prepared socket was opened (pane resolved) but never sent, and closed. - assert closed == [True] - - -# --------------------------------------------------------------------------- -# Permanent pre-send failures: terminal and non-sending -# --------------------------------------------------------------------------- - - -def test_disallowed_worker_status_is_terminal( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = _worker(status="closed") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, _request(request_id="disallowed"), socket_client_factory=_factory(calls) - ) - replay = submit_command( - config, _request(request_id="disallowed"), socket_client_factory=_forbidden_factory - ) - - assert first.status == STATUS_REJECTED - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, "disallowed") - assert receipt is not None - assert receipt["state"] == "rejected" - - -def test_missing_private_binding_is_terminal( - tmp_path: Path, -) -> None: - """A worker with no sendable backend binding is an unsupported target.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], []) # worker resolves, but no herdr binding exists - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, _request(request_id="no-binding"), socket_client_factory=_factory(calls) - ) - replay = submit_command( - config, _request(request_id="no-binding"), socket_client_factory=_forbidden_factory - ) - - assert first.status == STATUS_BACKEND_UNSUPPORTED - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, "no-binding") - assert receipt is not None - assert receipt["state"] == "rejected" - - -def test_stale_private_binding_is_terminal( - tmp_path: Path, -) -> None: - """A binding whose fingerprint no longer matches the worker is a proven stale target.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker, fingerprint="stale-observation")]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, _request(request_id="stale-binding"), socket_client_factory=_factory(calls) - ) - replay = submit_command( - config, _request(request_id="stale-binding"), socket_client_factory=_forbidden_factory - ) - - assert first.status == STATUS_STALE_TARGET - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, "stale-binding") - assert receipt is not None - assert receipt["state"] == "rejected" - - -def test_authoritative_degraded_backend_is_terminal( - tmp_path: Path, -) -> None: - """A resolved worker plus a degraded backend is an authoritative rejection.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)], health="degraded") - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, _request(request_id="degraded"), socket_client_factory=_factory(calls) - ) - replay = submit_command( - config, _request(request_id="degraded"), socket_client_factory=_forbidden_factory - ) - - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, "degraded") - assert receipt is not None - assert receipt["state"] == "rejected" - - -def _pane_resolution_factory(kind: str, calls: list[dict[str, Any]]): - """A socket factory whose pane resolution fails in a specific way.""" - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - def make_client(config: Config) -> _FakeSocketClient: - if kind == "no_pane": - # Herdr answered, but the agent has no resolvable pane. - return _FakeSocketClient(calls, agent_get_response={"result": {"agent": {}}}) - if kind == "herdr_error_response": - # Herdr returned an authoritative error response. - return _FakeSocketClient( - calls, - agent_get_raises=HerdrErrorResponse({"message": "no such agent"}, "rid"), - ) - if kind == "unsupported": - # The resolution response was malformed / unsupported. - return _FakeSocketClient(calls, agent_get_raises=ValueError("bad agent info")) - raise AssertionError(f"unknown pane failure {kind!r}") - - return make_client - - -@pytest.mark.parametrize("kind", ["no_pane", "herdr_error_response", "unsupported"]) -def test_authoritative_pane_resolution_failure_is_terminal( - tmp_path: Path, - kind: str, -) -> None: - """A definite backend answer during pane resolution is a proven target failure. - - These are not transport read failures: Herdr answered (or the response was - unusable), and a same-ID retry would get the same answer. They must stay - terminal so the connector advances instead of retrying to its horizon. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - factory = _pane_resolution_factory(kind, calls) - request_id = f"pane-{kind}" - - first = submit_command(config, _request(request_id=request_id), socket_client_factory=factory) - replay = submit_command( - config, _request(request_id=request_id), socket_client_factory=_forbidden_factory - ) - - # A durable terminal rejection, replayed on retry, with no send. - assert first.ok is False - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert first.status in {"backend_failed", "backend_unavailable"} - assert replay.to_dict() == first.to_dict() - assert _sent_texts(calls) == [] - receipt = _receipt(config, request_id) - assert receipt is not None - assert receipt["state"] == "rejected" - # The pane read was attempted exactly once and never repeated by the replay. - assert [call["method"] for call in calls].count("agent.get") == 1 - - -def test_transient_pane_read_failure_stays_retryable( - tmp_path: Path, -) -> None: - """A pane-read timeout is a transport failure, not an authoritative answer.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - connect_ok = {"value": False} - - def flaky_factory(config: Config) -> _FakeSocketClient: - if not connect_ok["value"]: - return _FakeSocketClient( - calls, - agent_get_raises=HerdrSocketTimeoutError("pane read timeout"), - ) - return _FakeSocketClient(calls) - - first = submit_command( - config, _request(request_id="pane-timeout"), socket_client_factory=flaky_factory - ) - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_NO_RECEIPT - assert _receipt(config, "pane-timeout") is None - assert _sent_texts(calls) == [] - - connect_ok["value"] = True - recovered = submit_command( - config, _request(request_id="pane-timeout"), socket_client_factory=flaky_factory - ) - assert recovered.status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - - -def test_transient_binding_store_never_masks_a_permanent_rejection( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Once the store recovers and the target is unsuitable, the truth is terminal.""" - config = _config(tmp_path) - worker = _worker() - # No binding: the target is permanently unsupported, but the binding store - # is transiently unavailable on the first attempt. - _seed(config, [worker], []) - calls: list[dict[str, Any]] = [] - injection = _TransientInjection("binding_store_sqlite") - factory = injection.install(monkeypatch, calls) - - transient = submit_command( - config, _request(request_id="mixed"), socket_client_factory=factory - ) - assert transient.disposition == DISPOSITION_NO_RECEIPT - assert _receipt(config, "mixed") is None - - injection.armed = False - terminal = submit_command( - config, _request(request_id="mixed"), socket_client_factory=factory - ) - assert terminal.status == STATUS_BACKEND_UNSUPPORTED - assert terminal.disposition == DISPOSITION_TERMINAL_REJECTED - assert _sent_texts(calls) == [] - - -# --------------------------------------------------------------------------- -# Concurrency (barriers, not sleeps) -# --------------------------------------------------------------------------- - - -def test_concurrent_callers_with_one_transient_send_at_most_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Three same-id callers, a subset transient: one send, one terminal, no rejection.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - calls_lock = threading.Lock() - real_bindings = command_submission.list_worker_bindings - fail_budget = {"n": 2} # two of three callers hit a transient binding-store error - budget_lock = threading.Lock() - - def flaky_bindings(*a: Any, **k: Any) -> Any: - with budget_lock: - if fail_budget["n"] > 0: - fail_budget["n"] -= 1 - raise sqlite3.OperationalError("database is locked") - return real_bindings(*a, **k) - - monkeypatch.setattr(command_submission, "list_worker_bindings", flaky_bindings) - - class _SerializedClient(_FakeSocketClient): - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None): - with calls_lock: - return super().request(method, params, timeout=timeout) - - def factory(_config: Config) -> _SerializedClient: - return _SerializedClient(calls) - - barrier = threading.Barrier(3) - results: list[Any] = [] - results_lock = threading.Lock() - - def attempt() -> None: - barrier.wait(timeout=30) - envelope = submit_command( - config, _request(request_id="race"), socket_client_factory=factory - ) - with results_lock: - results.append(envelope) - - threads = [threading.Thread(target=attempt) for _ in range(3)] - for thread in threads: - thread.start() - for thread in threads: - thread.join(timeout=30) - - assert not any(thread.is_alive() for thread in threads) - assert len(results) == 3 - # At most one send, at most one terminal receipt, and no caller was durably - # rejected because of the transient binding-store failure. - assert _sent_texts(calls).count("hello") <= 1 - assert _receipt_count(config) <= 1 - for envelope in results: - assert envelope.disposition != DISPOSITION_TERMINAL_REJECTED - assert envelope.status in { - STATUS_ACCEPTED, - STATUS_PENDING, - STATUS_BACKEND_UNAVAILABLE, - } - # The transient callers can retry the same id and converge on the result. - settled = submit_command( - config, _request(request_id="race"), socket_client_factory=factory - ) - assert settled.status == STATUS_ACCEPTED - assert _sent_texts(calls).count("hello") == 1 - - -def test_transient_racing_reservation_creation_sends_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A transient caller cannot corrupt a concurrent caller's reservation+send.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - calls_lock = threading.Lock() - real_reserve = command_submission.reserve_command_request - at_reservation = threading.Event() - let_reservation_proceed = threading.Event() - - def gated_reserve(*a: Any, **k: Any) -> Any: - at_reservation.set() - assert let_reservation_proceed.wait(timeout=30) - return real_reserve(*a, **k) - - class _SerializedClient(_FakeSocketClient): - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None): - with calls_lock: - return super().request(method, params, timeout=timeout) - - def factory(_config: Config) -> _SerializedClient: - return _SerializedClient(calls) - - monkeypatch.setattr(command_submission, "reserve_command_request", gated_reserve) - - sender_result: list[Any] = [] - - def sender() -> None: - sender_result.append( - submit_command(config, _request(request_id="rr"), socket_client_factory=factory) - ) - - sender_thread = threading.Thread(target=sender, name="sender") - sender_thread.start() - assert at_reservation.wait(timeout=30), "sender never reached reservation" - - # While the sender is parked at reservation, a second caller hits a transient - # binding-store failure. It must not reserve, send, or reject durably. - real_bindings = command_submission.list_worker_bindings - - def bindings_raise(*a: Any, **k: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings_raise) - transient = submit_command( - config, _request(request_id="rr"), socket_client_factory=factory - ) - monkeypatch.setattr(command_submission, "list_worker_bindings", real_bindings) - - let_reservation_proceed.set() - sender_thread.join(timeout=30) - - assert not sender_thread.is_alive() - assert transient.disposition == DISPOSITION_NO_RECEIPT - assert transient.status == STATUS_BACKEND_UNAVAILABLE - assert sender_result[0].status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert _receipt_count(config) == 1 - - -def test_transient_racing_send_start_cas_sends_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A transient retry arriving during the sender's send-start replays in-progress.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - calls_lock = threading.Lock() - real_mark = command_submission.mark_command_send_started - at_send_start = threading.Event() - let_send_start_proceed = threading.Event() - - def gated_mark(*a: Any, **k: Any) -> Any: - at_send_start.set() - assert let_send_start_proceed.wait(timeout=30) - return real_mark(*a, **k) - - monkeypatch.setattr(command_submission, "mark_command_send_started", gated_mark) - - class _SerializedClient(_FakeSocketClient): - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None): - with calls_lock: - return super().request(method, params, timeout=timeout) - - def factory(_config: Config) -> _SerializedClient: - return _SerializedClient(calls) - - sender_result: list[Any] = [] - - def sender() -> None: - sender_result.append( - submit_command(config, _request(request_id="ss"), socket_client_factory=factory) - ) - - sender_thread = threading.Thread(target=sender, name="sender") - sender_thread.start() - assert at_send_start.wait(timeout=30), "sender never reached send-start" - - # The reservation now exists (state reserved). A retry that then hits a - # transient binding-store failure must read in-progress from that receipt, - # never a no-receipt failure and never a second send. - real_bindings = command_submission.list_worker_bindings - - def bindings_raise(*a: Any, **k: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings_raise) - retry = submit_command(config, _request(request_id="ss"), socket_client_factory=factory) - monkeypatch.setattr(command_submission, "list_worker_bindings", real_bindings) - - let_send_start_proceed.set() - sender_thread.join(timeout=30) - - assert not sender_thread.is_alive() - assert retry.status == STATUS_PENDING - assert retry.disposition == DISPOSITION_IN_PROGRESS - assert sender_result[0].status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert _receipt_count(config) == 1 - - -def test_retry_after_abandoned_reservation_with_transient_reports_in_progress( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """An abandoned reservation plus a transient retry stays in-progress, not rejected.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - # Reserve then crash before send by losing the send-start response. - real_mark = command_submission.mark_command_send_started - - def lose_send_start(*a: Any, **k: Any) -> Any: - raise HerdrSocketTimeoutError("send-start response lost") - - monkeypatch.setattr(command_submission, "mark_command_send_started", lose_send_start) - reserved = submit_command( - config, _request(request_id="abandoned"), socket_client_factory=_factory(calls) - ) - assert reserved.status == STATUS_PENDING - monkeypatch.setattr(command_submission, "mark_command_send_started", real_mark) - - # Expire the crashed owner's lease, then retry while a transient binding-store - # failure is active. The abandoned reservation must not be terminalized. - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET owner_expires_at = ? WHERE request_id = ?", - ("2020-01-01T00:00:00+00:00", "abandoned"), - ) - - def bindings_raise(*a: Any, **k: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "list_worker_bindings", bindings_raise) - retry = submit_command( - config, _request(request_id="abandoned"), socket_client_factory=_factory(calls) - ) - - assert retry.status == STATUS_PENDING - assert retry.disposition == DISPOSITION_IN_PROGRESS - assert _sent_texts(calls) == [] - receipt = _receipt(config, "abandoned") - assert receipt is not None - assert receipt["state"] == "reserved" - - -def test_process_response_loss_after_send_replays_without_second_send( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Once a send committed, a lost finish response is recovered, never re-sent.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - real_finish = command_submission.finish_command_request - finished_state: dict[str, Any] = {} - - def finish_then_lose(*a: Any, **k: Any) -> Any: - result = real_finish(*a, **k) - finished_state["result"] = result - raise HerdrSocketTimeoutError("finish response lost") - - monkeypatch.setattr(command_submission, "finish_command_request", finish_then_lose) - first = submit_command( - config, _request(request_id="resp-loss"), socket_client_factory=_factory(calls) - ) - monkeypatch.setattr(command_submission, "finish_command_request", real_finish) - - # The send happened and the receipt committed accepted; the caller only lost - # the response. A same-id retry replays it without a second send. - assert _sent_texts(calls) == ["hello"] - replay = submit_command( - config, _request(request_id="resp-loss"), socket_client_factory=_forbidden_factory - ) - assert replay.status == STATUS_ACCEPTED - assert replay.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert _receipt_count(config) == 1 - - -# --------------------------------------------------------------------------- -# Paired connector boundary: the exact tuple Herdres retries -# --------------------------------------------------------------------------- - - -@pytest.mark.parametrize("kind", TRANSIENT_KINDS) -def test_daemon_emits_the_retryable_tuple_herdres_expects( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - kind: str, -) -> None: - """The daemon JSON for a transient is the ``backend_unavailable/no_receipt`` - tuple that the Herdres client validates and the reduce loop retries under the - same request ID (see the Herdres client's ``_RETRY_DISPOSITIONS``). A durable - ``terminal_rejected`` would instead advance the offset and drop the command. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - injection = _TransientInjection(kind) - factory = injection.install(monkeypatch, calls) - payload = _request(request_id=f"paired-{kind}") - - api = TendwireDaemonAPI( - get_snapshot=lambda: Snapshot( - host_id=HOST_ID, updated_at="2026-01-01T00:00:00+00:00", workers=[worker] - ), - get_health=lambda: {"ok": True}, - submit_command=lambda params: submit_command( - config, params, socket_client_factory=factory - ), - ) - response = api.dispatch({"method": "command.submit", "params": payload, "id": "1"}) - result = response["result"] - - # The exact tuple asserted by the paired Herdres test - # ``test_backend_unavailable_authority_comes_only_from_disposition``. - assert result["ok"] is False - assert result["status"] == STATUS_BACKEND_UNAVAILABLE - assert result["disposition"] == DISPOSITION_NO_RECEIPT - assert result["request_id"] == payload["request_id"] - assert result["dry_run"] is False - assert _receipt(config, payload["request_id"]) is None - assert _sent_texts(calls) == [] - - # The same request ID, once the transient clears, produces the accepted tuple - # Herdres marks terminal -- exactly once. - injection.armed = False - accepted = api.dispatch({"method": "command.submit", "params": payload, "id": "2"}) - assert accepted["result"]["status"] == STATUS_ACCEPTED - assert accepted["result"]["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - assert _sent_texts(calls) == ["hello"] - - -# --------------------------------------------------------------------------- -# Bounded real SQLite/local-state stress fixture -# --------------------------------------------------------------------------- - - -def test_real_store_contention_never_durably_rejects_an_unsent_command( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Reproduce the original contention shape against a real store, bounded. - - Many concurrent same-id submits run against a real SQLite store while a real - SQLite ``OperationalError`` is injected into the binding-store read on a - bounded fraction of attempts -- the exact failure the stress run surfaced. - The invariant: at most one send, at most one terminal receipt, and no receipt - is ever a rejection produced solely by a pre-send transient. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - calls_lock = threading.Lock() - real_bindings = command_submission.list_worker_bindings - attempt_counter = {"n": 0} - counter_lock = threading.Lock() - - def contended_bindings(*a: Any, **k: Any) -> Any: - with counter_lock: - attempt_counter["n"] += 1 - # Fail the binding-store read on odd attempts with a real SQLite error. - fail = attempt_counter["n"] % 2 == 1 - if fail: - raise sqlite3.OperationalError("database is locked") - return real_bindings(*a, **k) - - monkeypatch.setattr(command_submission, "list_worker_bindings", contended_bindings) - - class _SerializedClient(_FakeSocketClient): - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None): - with calls_lock: - return super().request(method, params, timeout=timeout) - - def factory(_config: Config) -> _SerializedClient: - return _SerializedClient(calls) - - barrier = threading.Barrier(8) - results: list[Any] = [] - results_lock = threading.Lock() - - def attempt() -> None: - barrier.wait(timeout=30) - envelope = submit_command( - config, _request(request_id="contended"), socket_client_factory=factory - ) - with results_lock: - results.append(envelope) - - threads = [threading.Thread(target=attempt) for _ in range(8)] - for thread in threads: - thread.start() - for thread in threads: - thread.join(timeout=30) - - assert not any(thread.is_alive() for thread in threads) - assert len(results) == 8 - assert _sent_texts(calls).count("hello") <= 1 - assert _receipt_count(config) <= 1 - # No durable rejection was ever manufactured from the transient store errors. - receipt = _receipt(config, "contended") - if receipt is not None: - assert receipt["state"] in {"reserved", "send_started", "accepted"} - for envelope in results: - if envelope.disposition == DISPOSITION_TERMINAL_REJECTED: - pytest.fail("a pre-send transient produced a durable rejection") - - # After contention clears, exactly one accepted result stands. - monkeypatch.setattr(command_submission, "list_worker_bindings", real_bindings) - settled = submit_command( - config, _request(request_id="contended"), socket_client_factory=factory - ) - assert settled.status == STATUS_ACCEPTED - assert _sent_texts(calls).count("hello") == 1 diff --git a/tests/test_command_replay_authority.py b/tests/test_command_replay_authority.py deleted file mode 100644 index 0a3d31d..0000000 --- a/tests/test_command_replay_authority.py +++ /dev/null @@ -1,1441 +0,0 @@ -"""Receipt authority over mutable worker resolution for retried commands. - -Goal 11 made ``request_id`` the sole idempotency key. This module proves the -follow-up property: an existing receipt decides its own retry from stored -evidence, before any mutable worker snapshot is consulted. A worker that -vanishes, is renamed, or is recycled must never turn a live receipt into a -no-receipt failure, and must never let one request mutate the backend twice. -""" - -from __future__ import annotations - -import json -import sqlite3 -import threading -from pathlib import Path -from typing import Any - -import pytest - -import tendwire.command_submission as command_submission -import tendwire.store.sqlite as store_sqlite - -from tendwire.backends.herdr_socket import HerdrSocketTimeoutError -from tendwire.command_submission import replay_command_receipt, submit_command -from tendwire.config import Config -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - DISPOSITION_TERMINAL_ACCEPTED, - DISPOSITION_TERMINAL_REJECTED, - DISPOSITION_TERMINAL_UNCERTAIN, - STATUS_ACCEPTED, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DUPLICATE_REQUEST, - STATUS_INVALID_REQUEST, - STATUS_PENDING, - STATUS_REQUEST_STATE_UNCERTAIN, - CommandRequest, - build_selector_proof, - is_selector_proof, -) -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.daemon_api import TendwireDaemonAPI -from tendwire.store.sqlite import ( - get_command_request, - init_store, - run_store_maintenance, - save_snapshot, - store_status, - upsert_worker_bindings, -) - - -HOST_ID = "cmd-host" - -RECEIPT_STATES = ("reserved", "send_started", "accepted", "rejected", "uncertain") -SELECTOR_KINDS = ( - "worker_id", - "worker_id_fingerprint", - "name", - "space_id", - "name_and_space", -) - -_EXPECTED_REPLAY = { - "reserved": (STATUS_PENDING, DISPOSITION_IN_PROGRESS), - "send_started": (STATUS_PENDING, DISPOSITION_IN_PROGRESS), - "accepted": (STATUS_ACCEPTED, DISPOSITION_TERMINAL_ACCEPTED), - "rejected": (STATUS_BACKEND_UNSUPPORTED, DISPOSITION_TERMINAL_REJECTED), - "uncertain": (STATUS_REQUEST_STATE_UNCERTAIN, DISPOSITION_TERMINAL_UNCERTAIN), -} - -# What reaching each state costs at the backend. Only a send that was actually -# attempted puts text on a pane; the rest fail earlier. -_SENDS_TO_REACH = { - "reserved": [], - "send_started": [], - "accepted": ["hello"], - "rejected": [], - "uncertain": ["hello"], -} - - -def _config(tmp_path: Path) -> Config: - return Config( - host_id=HOST_ID, - data_dir=tmp_path, - db_path=tmp_path / "commands.db", - herdr_backend="socket", - herdr_timeout_seconds=5.0, - ) - - -def _worker( - *, - worker_id: str = "w-1", - name: str = "Alpha", - space_id: str | None = "space-1", - status: str = "active", -) -> Worker: - return Worker(id=worker_id, name=name, status=status, space_id=space_id) - - -def _binding(worker: Worker, *, sendable: bool = True) -> WorkerBinding: - # Each worker owns a distinct private route. Sharing one would make every - # worker an ambiguous backend target and mask what these tests measure. - return WorkerBinding( - host_id=HOST_ID, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value=f"agent-{worker.id}", - turn_target_kind="pane_id", - turn_target_value=f"pane-{worker.id}", - sendable=sendable, - reason=None if sendable else "not_sendable", - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint=f"private-{worker.id}", - ) - - -def _health(status: str = "healthy") -> BackendHealth: - return BackendHealth( - name="herdr", - status=status, - outcome="healthy_non_empty" if status == "healthy" else "timeout", - observed_at="2026-01-01T00:00:00+00:00", - counts={"workers": 1}, - ) - - -def _snapshot( - workers: list[Worker], - *, - health: str = "healthy", - updated_at: str = "2026-01-01T00:00:00+00:00", -) -> Snapshot: - return Snapshot( - host_id=HOST_ID, - updated_at=updated_at, - workers=workers, - backend_health=[_health(health)], - ) - - -def _seed( - config: Config, - workers: list[Worker], - bindings: list[WorkerBinding], -) -> None: - assert config.db_path is not None - init_store(config.db_path) - save_snapshot(config.db_path, _snapshot(workers)) - if bindings: - upsert_worker_bindings(config.db_path, bindings) - - -def _remove_workers(config: Config, *, health: str = "healthy") -> None: - """Publish a newer authoritative snapshot in which the worker is gone.""" - assert config.db_path is not None - save_snapshot( - config.db_path, - _snapshot([], health=health, updated_at="2026-01-01T00:05:00+00:00"), - ) - - -def _selector(kind: str, worker: Worker) -> dict[str, Any]: - return { - "worker_id": {"worker_id": worker.id}, - "worker_id_fingerprint": { - "worker_id": worker.id, - "worker_fingerprint": worker.fingerprint, - }, - "name": {"name": worker.name}, - "space_id": {"space_id": worker.space_id}, - "name_and_space": {"name": worker.name, "space_id": worker.space_id}, - }[kind] - - -def _request( - *, - request_id: str, - target: dict[str, Any], - text: str = "hello", -) -> dict[str, Any]: - return { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": dict(target), - "instruction": {"text": text}, - } - - -class _FakeSocketClient: - def __init__( - self, - calls: list[dict[str, Any]], - *, - raises: BaseException | None = None, - ) -> None: - self.calls = calls - self.raises = raises - - def connect(self) -> "_FakeSocketClient": - return self - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if self.raises is not None and method == "agent.prompt": - raise self.raises - if method == "agent.get": - return {"result": {"agent": {"pane_id": "pane-secret"}}} - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": "Completed previous turn.\n── status: idle ──"}, - } - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": "pane-secret"}, - "delivery": "submitted", - } - return {"accepted": True} - - def close(self) -> None: - return None - - -def _factory(calls: list[dict[str, Any]], *, raises: BaseException | None = None): - def make_client(config: Config) -> _FakeSocketClient: - return _FakeSocketClient(calls, raises=raises) - - return make_client - - -def _forbidden_factory(config: Config) -> Any: - pytest.fail("a receipt replay must not create a socket client") - - -def _receipt(config: Config, request_id: str) -> dict[str, Any]: - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, request_id) - assert receipt is not None - return receipt - - -def _receipt_rows(config: Config) -> list[tuple[Any, ...]]: - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - return conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - - -def _sent_texts(calls: list[dict[str, Any]]) -> list[str]: - return [ - str(call["params"].get("text")) - for call in calls - if call["method"] == "agent.prompt" - ] - - -def _drive_to_state( - config: Config, - payload: dict[str, Any], - state: str, - calls: list[dict[str, Any]], - monkeypatch: pytest.MonkeyPatch, -) -> Any: - """Submit one request and leave its receipt in the requested state. - - ``rejected`` is reached through an unsendable private binding, which the - submission path terminalizes before any send; the caller seeds that binding. - """ - if state == "accepted": - return submit_command(config, payload, socket_client_factory=_factory(calls)) - if state == "rejected": - return submit_command(config, payload, socket_client_factory=_factory(calls)) - if state == "uncertain": - return submit_command( - config, - payload, - socket_client_factory=_factory( - calls, - raises=HerdrSocketTimeoutError("send response lost"), - ), - ) - - real_mark = command_submission.mark_command_send_started - - def lose_send_start(*args: Any, **kwargs: Any) -> Any: - if state == "send_started": - result = real_mark(*args, **kwargs) - assert result["status"] == "send_started" - raise HerdrSocketTimeoutError("send-start response lost") - - monkeypatch.setattr(command_submission, "mark_command_send_started", lose_send_start) - envelope = submit_command(config, payload, socket_client_factory=_factory(calls)) - monkeypatch.setattr(command_submission, "mark_command_send_started", real_mark) - return envelope - - -@pytest.mark.parametrize("selector_kind", SELECTOR_KINDS) -@pytest.mark.parametrize("receipt_state", RECEIPT_STATES) -def test_exact_retry_replays_receipt_after_healthy_worker_removal( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - receipt_state: str, - selector_kind: str, -) -> None: - """Every receipt state replays for every selector shape once the worker is gone.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker, sendable=receipt_state != "rejected")]) - request_id = f"replay-{receipt_state}-{selector_kind}" - payload = _request(request_id=request_id, target=_selector(selector_kind, worker)) - calls: list[dict[str, Any]] = [] - - first = _drive_to_state(config, payload, receipt_state, calls, monkeypatch) - stored = _receipt(config, request_id) - assert stored["state"] == receipt_state - - _remove_workers(config) - rows_before = _receipt_rows(config) - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert (retry.status, retry.disposition) == _EXPECTED_REPLAY[receipt_state] - if receipt_state in {"accepted", "rejected"}: - assert retry.to_dict() == first.to_dict() - # The retry itself sent nothing and left the authoritative receipt intact. - assert _sent_texts(calls) == _SENDS_TO_REACH[receipt_state] - assert _receipt_rows(config) == rows_before - - -@pytest.mark.parametrize("selector_kind", SELECTOR_KINDS) -def test_exact_retry_replays_accepted_receipt_while_backend_is_degraded( - tmp_path: Path, - selector_kind: str, -) -> None: - """A degraded observation cannot override truth the receipt already holds.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - request_id = f"degraded-{selector_kind}" - payload = _request(request_id=request_id, target=_selector(selector_kind, worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - _remove_workers(config, health="degraded") - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.to_dict() == accepted.to_dict() - assert _sent_texts(calls) == ["hello"] - - -def test_fingerprint_only_targets_cannot_claim_another_workers_receipt( - tmp_path: Path, -) -> None: - """The collision that blocked 2edc6cc: two fingerprints, one request ID. - - A fingerprint is a mutable precondition, not identity, so the selector proof - excludes it. If a fingerprint could stand alone as the whole target, every - such target would share one proof -- and reusing the request ID with a - fingerprint naming a different worker would replay the first worker's - accepted result. Rejecting the shape outright is what closes that hole. - """ - config = _config(tmp_path) - first_worker = _worker() - second_worker = _worker(worker_id="w-2", name="Beta", space_id="space-2") - _seed( - config, - [first_worker, second_worker], - [_binding(first_worker), _binding(second_worker)], - ) - assert first_worker.fingerprint != second_worker.fingerprint - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request( - request_id="fingerprint-collision", - target={"worker_fingerprint": first_worker.fingerprint}, - ), - socket_client_factory=_factory(calls), - ) - changed = submit_command( - config, - _request( - request_id="fingerprint-collision", - target={"worker_fingerprint": second_worker.fingerprint}, - ), - socket_client_factory=_factory(calls), - ) - - # Under 2edc6cc the first was accepted and the second replayed its stored - # terminal_accepted body: one worker's result claimed for another. - for envelope in (first, changed): - assert envelope.ok is False - assert envelope.status == STATUS_INVALID_REQUEST - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert _sent_texts(calls) == [] - assert _receipt_rows(config) == [] - - -def test_fingerprint_only_target_does_no_store_source_or_backend_work( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Rejection happens in the parser, before anything durable or observable.""" - config = _config(tmp_path) - touched: list[str] = [] - - def forbidden(*args: Any, **kwargs: Any) -> Any: - touched.append("io") - raise AssertionError("an invalid target must not reach store or source work") - - monkeypatch.setattr(command_submission, "get_command_request", forbidden) - monkeypatch.setattr(command_submission, "_current_snapshot", forbidden) - monkeypatch.setattr(command_submission, "latest_snapshot", forbidden) - monkeypatch.setattr(command_submission, "reserve_command_request", forbidden) - monkeypatch.setattr( - "tendwire.command_submission.project_from_observations", - forbidden, - ) - - envelope = submit_command( - config, - _request( - request_id="fingerprint-only", - target={"worker_fingerprint": "fingerprint-A"}, - ), - socket_client_factory=forbidden, - ) - - assert envelope.ok is False - assert envelope.status == STATUS_INVALID_REQUEST - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert touched == [] - assert config.db_path is not None - assert not config.db_path.exists() - - -def test_fingerprint_only_target_is_rejected_by_daemon_and_read_only_replay( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request( - request_id="fingerprint-only", - target={"worker_fingerprint": worker.fingerprint}, - ) - api = TendwireDaemonAPI( - get_snapshot=lambda: _snapshot([worker]), - get_health=lambda: {"ok": True}, - submit_command=lambda params: submit_command( - config, - params, - socket_client_factory=_forbidden_factory, - ), - ) - - response = api.dispatch({"method": "command.submit", "params": payload, "id": "1"}) - - assert response["result"]["ok"] is False - assert response["result"]["status"] == STATUS_INVALID_REQUEST - assert response["result"]["disposition"] == DISPOSITION_NO_RECEIPT - # The rejection names the rule it broke without echoing the observation the - # caller sent, and it leaves no durable trace of the request. - rendered = json.dumps(response) - assert worker.fingerprint not in rendered - assert "worker_fingerprint" in response["result"]["error"]["message"] - assert response["result"]["error"]["details"]["allowed"] == [ - "name", - "space_id", - "worker_id", - ] - # The read-only response-loss path cannot resolve it either, and must never - # invent a receipt for a request that could not have created one. - assert replay_command_receipt(config, payload) is None - assert _receipt_rows(config) == [] - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - command_events = conn.execute( - "SELECT COUNT(*) FROM events WHERE aggregate_type = 'command_request'" - ).fetchone()[0] - assert command_events == 0 - - -@pytest.mark.parametrize("selector_kind", ["name", "space_id", "name_and_space"]) -def test_refreshed_fingerprint_beside_a_stable_selector_replays_stored_result( - tmp_path: Path, - selector_kind: str, -) -> None: - """A refreshed fingerprint is a precondition, not a different command. - - This is the alias path, so equivalence is proven by the stored selector - proof -- which is exactly the evidence that must ignore the fingerprint. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - stable = _selector(selector_kind, worker) - payload = _request( - request_id=f"fingerprint-beside-{selector_kind}", - target={**stable, "worker_fingerprint": worker.fingerprint}, - ) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - - # The worker is re-observed with fresh data, so its fingerprint moves. - refreshed_worker = _worker(status="waiting") - assert refreshed_worker.fingerprint != worker.fingerprint - assert config.db_path is not None - save_snapshot( - config.db_path, - _snapshot([refreshed_worker], updated_at="2026-01-01T00:06:00+00:00"), - ) - refreshed = _request( - request_id=f"fingerprint-beside-{selector_kind}", - target={**stable, "worker_fingerprint": refreshed_worker.fingerprint}, - ) - - retry = submit_command(config, refreshed, socket_client_factory=_forbidden_factory) - - assert retry.to_dict() == accepted.to_dict() - assert _sent_texts(calls) == ["hello"] - - -def test_refreshed_worker_fingerprint_stays_noncanonical_on_retry( - tmp_path: Path, -) -> None: - """A worker ID that survives with new observation data is the same target.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request( - request_id="fingerprint-churn", - target={"worker_id": worker.id, "worker_fingerprint": worker.fingerprint}, - ) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - - # The worker keeps its public ID but every observed attribute changes. - recycled = _worker(name="Renamed", space_id="space-9", status="waiting") - assert config.db_path is not None - save_snapshot( - config.db_path, - _snapshot([recycled], updated_at="2026-01-01T00:06:00+00:00"), - ) - refreshed = _request( - request_id="fingerprint-churn", - target={"worker_id": worker.id, "worker_fingerprint": recycled.fingerprint}, - ) - - retry = submit_command(config, refreshed, socket_client_factory=_forbidden_factory) - - assert retry.to_dict() == accepted.to_dict() - assert _sent_texts(calls) == ["hello"] - - -@pytest.mark.parametrize("receipt_state", RECEIPT_STATES) -@pytest.mark.parametrize( - ("collision", "changed"), - [ - ("instruction", {"instruction": {"text": "different"}}), - ("worker_id", {"target": {"worker_id": "w-2"}}), - ("name", {"target": {"name": "Beta"}}), - ("space_id", {"target": {"space_id": "space-2"}}), - ("action", {"action": "answer_pending"}), - ], -) -def test_changed_request_cannot_claim_a_stored_receipt( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - receipt_state: str, - collision: str, - changed: dict[str, Any], -) -> None: - """Reusing a request ID with any changed canonical field mutates nothing.""" - config = _config(tmp_path) - worker = _worker() - other = _worker(worker_id="w-2", name="Beta", space_id="space-2") - _seed( - config, - [worker, other], - [ - _binding(worker, sendable=receipt_state != "rejected"), - _binding(other, sendable=receipt_state != "rejected"), - ], - ) - request_id = f"collision-{receipt_state}-{collision}" - payload = _request(request_id=request_id, target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, receipt_state, calls, monkeypatch) - assert _receipt(config, request_id)["state"] == receipt_state - sent_before = list(_sent_texts(calls)) - rows_before = _receipt_rows(config) - - reused = _request(request_id=request_id, target=_selector("name", worker)) - reused.update(changed) - if collision == "action": - reused.pop("target", None) - reused.pop("instruction", None) - reused["params"] = { - "pending_id": "pending-public", - "pending_fingerprint": "revision-public", - "choice_id": "choice-public", - } - - conflict = submit_command( - config, - reused, - socket_client_factory=lambda _config: pytest.fail( - "a changed request must not create a socket client" - ), - ) - - assert conflict.status == STATUS_DUPLICATE_REQUEST - assert conflict.disposition == DISPOSITION_TERMINAL_REJECTED - assert _sent_texts(calls) == sent_before - assert _receipt_rows(config) == rows_before - - -@pytest.mark.parametrize("receipt_state", RECEIPT_STATES) -def test_unsupported_stored_selector_proof_fails_closed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - receipt_state: str, -) -> None: - """Evidence this version cannot read decides nothing rather than deciding wrong.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker, sendable=receipt_state != "rejected")]) - request_id = f"proof-{receipt_state}" - payload = _request(request_id=request_id, target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, receipt_state, calls, monkeypatch) - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET selector_proof = ? WHERE request_id = ?", - ("v9:not-a-supported-proof", request_id), - ) - _remove_workers(config) - sent_before = list(_sent_texts(calls)) - rows_before = _receipt_rows(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert _sent_texts(calls) == sent_before - assert _receipt_rows(config) == rows_before - - -def test_legacy_receipt_without_selector_proof_fails_closed_on_alias_retry( - tmp_path: Path, -) -> None: - """A migrated v12 alias receipt is never guessed into a replay.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="legacy-alias", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - # Reproduce what the v12 migration leaves behind: a canonical receipt whose - # original selector spelling was never recorded. - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute("UPDATE command_receipts SET selector_proof = ''") - _remove_workers(config) - rows_before = _receipt_rows(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert _sent_texts(calls) == ["hello"] - assert _receipt_rows(config) == rows_before - - -def test_legacy_alias_receipt_fails_closed_when_current_authority_store_raises( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A transient authority read cannot erase a receipt or permit a resend.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="legacy-store-race", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute("UPDATE command_receipts SET selector_proof = ''") - - def contended_snapshot(*args: Any, **kwargs: Any) -> Any: - raise sqlite3.OperationalError("database is locked") - - monkeypatch.setattr(command_submission, "latest_snapshot", contended_snapshot) - rows_before = _receipt_rows(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert _sent_texts(calls) == ["hello"] - assert _receipt_rows(config) == rows_before - - -def test_legacy_receipt_without_selector_proof_replays_explicit_worker_id( - tmp_path: Path, -) -> None: - """Explicit worker IDs still replay from a legacy receipt: the ID is the proof.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="legacy-explicit", target={"worker_id": worker.id}) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute("UPDATE command_receipts SET selector_proof = ''") - _remove_workers(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.to_dict() == accepted.to_dict() - assert _sent_texts(calls) == ["hello"] - - -def test_abandoned_reservation_is_redriven_not_replayed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """An expired reservation lease is the existing crash-recovery path, not a replay.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="abandoned", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, "reserved", calls, monkeypatch) - assert _receipt(config, "abandoned")["state"] == "reserved" - assert _sent_texts(calls) == [] - - # Expire the owner lease the crashed sender left behind. - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET owner_expires_at = ? WHERE request_id = ?", - ("2020-01-01T00:00:00+00:00", "abandoned"), - ) - - recovered = submit_command(config, payload, socket_client_factory=_factory(calls)) - - assert recovered.status == STATUS_ACCEPTED - assert recovered.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert _receipt(config, "abandoned")["state"] == "accepted" - - -def test_abandoned_reservation_reports_in_progress_when_worker_is_gone( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A vanished worker never restates a stored reservation as a no-receipt failure.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="abandoned-gone", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, "reserved", calls, monkeypatch) - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET owner_expires_at = ? WHERE request_id = ?", - ("2020-01-01T00:00:00+00:00", "abandoned-gone"), - ) - _remove_workers(config) - rows_before = _receipt_rows(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - assert retry.status == STATUS_PENDING - assert retry.disposition == DISPOSITION_IN_PROGRESS - assert _sent_texts(calls) == [] - assert _receipt_rows(config) == rows_before - - -def test_abandoned_reservation_conflicts_when_selector_now_names_another_worker( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Re-driving an abandoned send never redirects it to a different worker.""" - config = _config(tmp_path) - worker = _worker() - other = _worker(worker_id="w-2", name="Beta", space_id="space-2") - _seed(config, [worker, other], [_binding(worker), _binding(other)]) - payload = _request(request_id="abandoned-drift", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - _drive_to_state(config, payload, "reserved", calls, monkeypatch) - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET owner_expires_at = ?, selector_proof = ''", - ("2020-01-01T00:00:00+00:00",), - ) - # "Alpha" now names the other worker, so the abandoned reservation's target - # can no longer be reached by the spelling that created it. - save_snapshot( - config.db_path, - _snapshot( - [_worker(worker_id="w-2", name="Alpha", space_id="space-2")], - updated_at="2026-01-01T00:07:00+00:00", - ), - ) - rows_before = _receipt_rows(config) - - conflict = submit_command( - config, - payload, - socket_client_factory=lambda _config: pytest.fail( - "a redirected takeover must not create a socket client" - ), - ) - - assert conflict.status == STATUS_DUPLICATE_REQUEST - assert _sent_texts(calls) == [] - assert _receipt_rows(config) == rows_before - - -@pytest.mark.parametrize("selector_kind", SELECTOR_KINDS) -@pytest.mark.parametrize("stage", ["reservation", "send_start", "completion"]) -def test_exact_retry_racing_a_live_mutation_never_sends_twice( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - stage: str, - selector_kind: str, -) -> None: - """A retry landing mid-mutation replays the receipt, even as the worker vanishes. - - The sender is held at one stage of the state machine while an exact retry - runs against a snapshot that no longer contains the worker. The retry must - read in-progress from the receipt, open no socket, and leave the in-flight - mutation to finish exactly once. - """ - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - request_id = f"race-{stage}-{selector_kind}" - payload = _request(request_id=request_id, target=_selector(selector_kind, worker)) - calls: list[dict[str, Any]] = [] - - reached = threading.Event() - release = threading.Event() - - def pause() -> None: - reached.set() - assert release.wait(timeout=30), "the racing retry never released the sender" - - real_reserve = command_submission.reserve_command_request - real_mark = command_submission.mark_command_send_started - real_finish = command_submission.finish_command_request - - def reserve(*args: Any, **kwargs: Any) -> Any: - reserved = real_reserve(*args, **kwargs) - if stage == "reservation": - pause() - return reserved - - def mark(*args: Any, **kwargs: Any) -> Any: - started = real_mark(*args, **kwargs) - if stage == "send_start": - pause() - return started - - def finish(*args: Any, **kwargs: Any) -> Any: - if stage == "completion": - pause() - return real_finish(*args, **kwargs) - - monkeypatch.setattr(command_submission, "reserve_command_request", reserve) - monkeypatch.setattr(command_submission, "mark_command_send_started", mark) - monkeypatch.setattr(command_submission, "finish_command_request", finish) - - sent: list[Any] = [] - - def send() -> None: - sent.append( - submit_command(config, payload, socket_client_factory=_factory(calls)) - ) - - sender = threading.Thread(target=send, name=f"sender-{stage}") - sender.start() - assert reached.wait(timeout=30), "the sender never reached the raced stage" - - # Healthy authority loses the worker while the mutation is still in flight. - _remove_workers(config) - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - release.set() - sender.join(timeout=30) - - assert not sender.is_alive() - assert len(sent) == 1 - assert retry.status == STATUS_PENDING - assert retry.disposition == DISPOSITION_IN_PROGRESS - assert sent[0].status == STATUS_ACCEPTED - assert _sent_texts(calls) == ["hello"] - assert len(_receipt_rows(config)) == 1 - assert _receipt(config, request_id)["state"] == "accepted" - - # Once the race settles, the retry converges on the one stored result. - settled = submit_command(config, payload, socket_client_factory=_forbidden_factory) - assert settled.to_dict() == sent[0].to_dict() - assert _sent_texts(calls) == ["hello"] - - -def test_retry_racing_receipt_retention_never_sends_twice( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Retention deleting a receipt mid-replay must not reopen the mutation.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="retention-race", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - - real_reserve = command_submission.reserve_terminal_command_replay - - def delete_then_reserve(*args: Any, **kwargs: Any) -> Any: - # Retention wins the race between reading the receipt and replaying it. - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute("DELETE FROM command_receipts") - return real_reserve(*args, **kwargs) - - monkeypatch.setattr( - command_submission, - "reserve_terminal_command_replay", - delete_then_reserve, - ) - _remove_workers(config) - - retry = submit_command(config, payload, socket_client_factory=_forbidden_factory) - - # The accepted body is gone, so the only honest answer is terminal - # uncertainty. It must never become a fresh send. - assert retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert _sent_texts(calls) == ["hello"] - restored = _receipt(config, "retention-race") - assert restored["state"] == "uncertain" - # The rebuilt row keeps the original spelling's evidence. - assert is_selector_proof(restored["selector_proof"]) - assert restored["selector_proof"] == build_selector_proof( - CommandRequest.from_dict(payload) - ) - - -def test_read_only_replay_honors_selector_proof_without_observing( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """The CLI's response-loss path replays an exact alias without any observation.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="read-only", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - assert accepted.status == STATUS_ACCEPTED - monkeypatch.setattr( - command_submission, - "_current_snapshot", - lambda _config: pytest.fail("read-only replay must not consult authority"), - ) - rows_before = _receipt_rows(config) - - exact = replay_command_receipt(config, payload) - changed = _request(request_id="read-only", target={"name": "Beta"}) - unprovable = replay_command_receipt(config, changed) - - assert exact is not None - assert exact.to_dict() == accepted.to_dict() - # Proving a different spelling would need an observation this path may not - # make, so it stays unresolved rather than guessing. - assert unprovable is None - assert _receipt_rows(config) == rows_before - - -# --------------------------------------------------------------------------- -# Migration -# --------------------------------------------------------------------------- - -# A literal, frozen copy of the schema-v12 receipt table. It must not be rebuilt -# from the live DDL: the point is to migrate what v12 actually wrote. -_V12_COMMAND_RECEIPTS_TABLE = """ -CREATE TABLE command_receipts ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - request_id TEXT NOT NULL, - action TEXT NOT NULL, - canonical_version INTEGER NOT NULL CHECK (canonical_version >= 0), - canonical_fingerprint TEXT NOT NULL, - canonical_request_json TEXT NOT NULL, - public_worker_id TEXT NOT NULL, - state TEXT NOT NULL CHECK ( - state IN ('reserved', 'send_started', 'accepted', 'rejected', 'uncertain') - ), - status TEXT NOT NULL, - result_json TEXT NOT NULL, - owner_token_hash TEXT NOT NULL DEFAULT '', - owner_expires_at TEXT, - binding_fingerprint TEXT, - created_at TEXT NOT NULL, - reserved_at TEXT NOT NULL, - send_started_at TEXT, - terminal_at TEXT, - updated_at TEXT NOT NULL, - legacy_collision INTEGER NOT NULL DEFAULT 0 CHECK (legacy_collision IN (0, 1)), - legacy_collision_count INTEGER NOT NULL DEFAULT 0 CHECK ( - legacy_collision_count >= 0 - ), - CHECK ( - ( - state IN ('reserved', 'send_started') - AND terminal_at IS NULL - AND owner_token_hash <> '' - ) - OR ( - state IN ('accepted', 'rejected', 'uncertain') - AND terminal_at IS NOT NULL - AND owner_token_hash = '' - AND owner_expires_at IS NULL - ) - ), - CHECK (state NOT IN ('reserved', 'send_started') OR status = 'pending'), - CHECK ( - state != 'accepted' - OR (status = 'accepted' AND send_started_at IS NOT NULL) - ), - CHECK (state != 'uncertain' OR status = 'request_state_uncertain'), - CHECK ( - state != 'rejected' - OR status NOT IN ('pending', 'accepted', 'request_state_uncertain') - ), - CHECK ( - legacy_collision = 0 - OR (state = 'uncertain' AND legacy_collision_count >= 2) - ) -); -""" - -_V12_ACTIVE = ( - "host-a", - "v12-active", - "send_instruction", - 1, - "fingerprint-active", - '{"action":"send_instruction"}', - "w-1", - "reserved", - "pending", - '{"ok":false,"status":"pending"}', - "owner-hash", - "2026-01-01T00:00:30+00:00", - None, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - None, - None, - "2026-01-01T00:00:00+00:00", -) -_V12_TERMINAL = ( - "host-a", - "v12-terminal", - "send_instruction", - 1, - "fingerprint-terminal", - '{"action":"send_instruction"}', - "w-2", - "accepted", - "accepted", - '{"ok":true,"status":"accepted"}', - "", - None, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - "2026-01-01T00:00:02+00:00", - "2026-01-01T00:00:02+00:00", -) - - -def _write_v12_store(db_path: Path) -> None: - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - _V12_COMMAND_RECEIPTS_TABLE + store_sqlite.CREATE_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - _V12_ACTIVE, - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - created_at, reserved_at, send_started_at, terminal_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - _V12_TERMINAL, - ) - conn.execute("PRAGMA user_version = 12") - - -def test_v12_receipts_migrate_to_empty_selector_proof(tmp_path: Path) -> None: - """Active and terminal v12 receipts survive with no invented selector evidence.""" - db_path = tmp_path / "v12.db" - _write_v12_store(db_path) - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert ( - int(conn.execute("PRAGMA user_version").fetchone()[0]) - == store_sqlite.STORE_SCHEMA_VERSION - ) - proofs = conn.execute( - "SELECT request_id, selector_proof FROM command_receipts ORDER BY request_id" - ).fetchall() - # A v12 row records the worker a request resolved to, never how it was - # spelled. Deriving a proof from that would let a changed target replay it. - assert proofs == [("v12-active", ""), ("v12-terminal", "")] - - active = get_command_request(db_path, "host-a", "v12-active") - terminal = get_command_request(db_path, "host-a", "v12-terminal") - assert active is not None and terminal is not None - assert (active["state"], active["status"]) == ("reserved", "pending") - assert (terminal["state"], terminal["status"]) == ("accepted", "accepted") - assert terminal["result_json"] == '{"ok":true,"status":"accepted"}' - - -def test_v12_migration_is_idempotent_across_reruns(tmp_path: Path) -> None: - db_path = tmp_path / "v12-idempotent.db" - _write_v12_store(db_path) - - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - first = conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - init_store(db_path) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - second = conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - version = int(conn.execute("PRAGMA user_version").fetchone()[0]) - - assert first == second - assert version == store_sqlite.STORE_SCHEMA_VERSION - - -def test_v12_migration_rolls_back_without_partial_schema( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A failing v13 transition leaves the v12 store exactly as it was.""" - db_path = tmp_path / "v12-rollback.db" - _write_v12_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - before = conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - - real_migrate = store_sqlite._migrate_v12_to_v13_conn - - def failing_migrate(conn: sqlite3.Connection) -> None: - real_migrate(conn) - raise RuntimeError("v13 migration failed after adding the column") - - monkeypatch.setattr( - store_sqlite, - "MIGRATIONS", - tuple( - store_sqlite.Migration(item.from_version, item.to_version, failing_migrate) - if item.to_version == 13 - else item - for item in store_sqlite.MIGRATIONS - ), - ) - - with pytest.raises(RuntimeError): - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert int(conn.execute("PRAGMA user_version").fetchone()[0]) == 12 - columns = { - str(row[1]) - for row in conn.execute("PRAGMA table_info(command_receipts)").fetchall() - } - assert "selector_proof" not in columns - assert ( - conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - == before - ) - - -def _downgrade_store_to_v12(db_path: Path) -> None: - """Rebuild a live store as schema v12, dropping every selector proof.""" - with sqlite3.connect(str(db_path)) as conn: - conn.execute("ALTER TABLE command_receipts RENAME TO command_receipts_v13") - conn.executescript(_V12_COMMAND_RECEIPTS_TABLE) - conn.execute( - """ - INSERT INTO command_receipts ( - id, host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at, legacy_collision, legacy_collision_count - ) - SELECT - id, host_id, request_id, action, canonical_version, - canonical_fingerprint, canonical_request_json, public_worker_id, - state, status, result_json, owner_token_hash, owner_expires_at, - binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at, legacy_collision, legacy_collision_count - FROM command_receipts_v13 - """ - ) - conn.execute("DROP TABLE command_receipts_v13") - for statement in store_sqlite.CREATE_COMMAND_RECEIPT_INDEXES: - conn.execute(statement) - conn.execute("PRAGMA user_version = 12") - - -def test_migrated_v12_store_proves_new_requests_but_never_old_ones( - tmp_path: Path, -) -> None: - """Migration is conservative: it earns proofs forward, it does not backfill them.""" - config = _config(tmp_path) - assert config.db_path is not None - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - legacy_payload = _request( - request_id="pre-migration", - target=_selector("name", worker), - ) - legacy_accepted = submit_command( - config, - legacy_payload, - socket_client_factory=_factory(calls), - ) - assert legacy_accepted.status == STATUS_ACCEPTED - - _downgrade_store_to_v12(config.db_path) - init_store(config.db_path) - - with sqlite3.connect(str(config.db_path)) as conn: - assert ( - int(conn.execute("PRAGMA user_version").fetchone()[0]) - == store_sqlite.STORE_SCHEMA_VERSION - ) - assert _receipt(config, "pre-migration")["selector_proof"] == "" - - fresh_payload = _request( - request_id="post-migration", - target=_selector("name", worker), - ) - fresh_accepted = submit_command( - config, - fresh_payload, - socket_client_factory=_factory(calls), - ) - assert fresh_accepted.status == STATUS_ACCEPTED - assert is_selector_proof(_receipt(config, "post-migration")["selector_proof"]) - - _remove_workers(config) - legacy_retry = submit_command( - config, - legacy_payload, - socket_client_factory=_forbidden_factory, - ) - fresh_retry = submit_command( - config, - fresh_payload, - socket_client_factory=_forbidden_factory, - ) - - # The pre-migration alias has no evidence of how it was spelled, so it fails - # closed. The post-migration one carries its own proof and replays. - assert legacy_retry.status == STATUS_REQUEST_STATE_UNCERTAIN - assert legacy_retry.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert fresh_retry.to_dict() == fresh_accepted.to_dict() - assert _sent_texts(calls) == ["hello", "hello"] - - -# --------------------------------------------------------------------------- -# Public boundary -# --------------------------------------------------------------------------- - - -def _assert_no_selector_proof(value: Any, proof: str, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - assert "selector" not in str(key).lower().replace("-", "_"), ( - f"selector evidence leaked at {path}.{key}" - ) - _assert_no_selector_proof(item, proof, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_selector_proof(item, proof, f"{path}[{index}]") - elif isinstance(value, str): - assert proof not in value, f"selector proof leaked at {path}" - - -def test_selector_proof_never_reaches_a_public_surface(tmp_path: Path) -> None: - """The proof stays private evidence: not in envelopes, events, or the audit row.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - payload = _request(request_id="boundary", target=_selector("name", worker)) - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, payload, socket_client_factory=_factory(calls)) - _remove_workers(config) - replay = submit_command(config, payload, socket_client_factory=_forbidden_factory) - proof = _receipt(config, "boundary")["selector_proof"] - assert is_selector_proof(proof) - - for envelope in (accepted, replay): - _assert_no_selector_proof(envelope.to_dict(), proof) - _assert_no_selector_proof(json.loads(envelope.to_json()), proof) - - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - events = conn.execute("SELECT payload_json FROM events ORDER BY id").fetchall() - audit = conn.execute("SELECT * FROM commands ORDER BY id").fetchall() - audit_columns = { - str(row[1]) for row in conn.execute("PRAGMA table_info(commands)").fetchall() - } - assert events - for (payload_json,) in events: - _assert_no_selector_proof(json.loads(payload_json), proof) - assert proof not in payload_json - # The non-authoritative audit projection must not carry the proof at all. - assert "selector_proof" not in audit_columns - for row in audit: - assert proof not in json.dumps(row, default=str) - - # Daemon JSON, and the health and maintenance surfaces that summarize - # command receipts, are built from the same receipts and must stay clean. - health = store_status(config.db_path, config.host_id) - api = TendwireDaemonAPI( - get_snapshot=lambda: _snapshot([]), - get_health=lambda: health, - submit_command=lambda params: submit_command( - config, - params, - socket_client_factory=_forbidden_factory, - ), - ) - daemon_response = api.dispatch( - {"method": "command.submit", "params": payload, "id": "1"} - ) - assert daemon_response["result"]["status"] == STATUS_ACCEPTED - _assert_no_selector_proof(daemon_response, proof) - _assert_no_selector_proof(api.dispatch({"method": "health.get", "id": "2"}), proof) - _assert_no_selector_proof(health, proof) - _assert_no_selector_proof( - run_store_maintenance( - config.db_path, - config.host_id, - retention_days=14, - max_outbox_attempts=5, - ), - proof, - ) - - -def test_selector_proof_field_is_rejected_from_a_request(tmp_path: Path) -> None: - """No caller can inject or forge selector evidence through the public request.""" - config = _config(tmp_path) - worker = _worker() - _seed(config, [worker], [_binding(worker)]) - - for injected in ( - {"target": {"worker_id": "w-1", "selector_proof": "v1:" + "a" * 64}}, - {"instruction": {"text": "hello", "selector_proof": "v1:" + "a" * 64}}, - {"selector_proof": "v1:" + "a" * 64}, - ): - payload = _request(request_id="injected", target={"worker_id": worker.id}) - payload.update(injected) - envelope = submit_command( - config, - payload, - socket_client_factory=lambda _config: pytest.fail( - "an invalid request must not create a socket client" - ), - ) - assert envelope.ok is False - assert envelope.status == "invalid_request" - - assert _receipt_rows(config) == [] diff --git a/tests/test_command_submission.py b/tests/test_command_submission.py deleted file mode 100644 index 60b6584..0000000 --- a/tests/test_command_submission.py +++ /dev/null @@ -1,4533 +0,0 @@ -"""Tests for the authoritative daemon command submission path.""" - -from __future__ import annotations - -from concurrent.futures import ThreadPoolExecutor -from datetime import datetime, timedelta, timezone -from threading import Event - -import json -import sqlite3 -from pathlib import Path -from typing import Any - -import pytest -import tendwire.command_submission as command_submission -import tendwire.store.sqlite as store_sqlite - -from tendwire.backends.herdr_protocol import HerdrErrorResponse, HerdrProtocolError -from tendwire.backends.herdr_socket import ( - HerdrSocketDisconnectedError, - HerdrSocketTimeoutError, -) -from tendwire.command_submission import replay_command_receipt, submit_command -from tendwire.config import TURN_MODELS, Config -from tendwire.core.commands import ( - DISPOSITION_IN_PROGRESS, - DISPOSITION_NO_RECEIPT, - DISPOSITION_TERMINAL_ACCEPTED, - DISPOSITION_TERMINAL_REJECTED, - DISPOSITION_TERMINAL_UNCERTAIN, - STATUS_ACCEPTED, - STATUS_AMBIGUOUS_BACKEND_TARGET, - STATUS_BACKEND_UNAVAILABLE, - STATUS_BACKEND_UNSUPPORTED, - STATUS_DUPLICATE_REQUEST, - STATUS_INVALID_REQUEST, - STATUS_NOT_FOUND, - STATUS_PENDING, - STATUS_REJECTED, - STATUS_REQUEST_STATE_UNCERTAIN, - STATUS_STALE_TARGET, - CommandEnvelope, - CommandRequest, - build_canonical_mutation, - instruction_fingerprint, - turn_submission_id, -) -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.core.turns import PendingObservation, PendingObservedChoice, Turn -from tendwire.store.sqlite import ( - apply_backend_pending_observation, - cleanup_command_request_retention, - get_command_request, - init_store, - merge_turn_content, - pending_payload_from_store, - save_snapshot, - turns_payload_from_store, - upsert_worker_bindings, - reserve_command_request, -) - - -def _receipt_for_action( - db_path: Path, - host_id: str, - request_id: str, - action: str, -) -> dict[str, Any] | None: - receipt = get_command_request(db_path, host_id, request_id) - if receipt is None or receipt["action"] != action: - return None - return {**receipt, "uncertain": receipt["state"] in {"send_started", "uncertain"}} - - -_FORBIDDEN_PUBLIC_KEYS = { - "pane_id", - "terminal_id", - "backend_target", - "agent_session", - "argv", - "shell", - "target_kind", - "target_value", - "private_fingerprint", -} - - -def _assert_no_private_json(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - assert key not in _FORBIDDEN_PUBLIC_KEYS, f"forbidden field {path}.{key}" - _assert_no_private_json(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_private_json(item, f"{path}[{index}]") - - -def _config( - tmp_path: Path, - *, - backend: str = "socket", - timeout: float = 5.0, - turn_model: str = "observed", - submission_link_window_seconds: int = 60, - submission_hard_ttl_seconds: int = 86_400, -) -> Config: - return Config( - host_id="cmd-host", - data_dir=tmp_path, - db_path=tmp_path / "commands.db", - herdr_backend=backend, - herdr_timeout_seconds=timeout, - turn_model=turn_model, - submission_link_window_seconds=submission_link_window_seconds, - submission_hard_ttl_seconds=submission_hard_ttl_seconds, - ) - - -def _request( - *, - request_id: str = "req-1", - worker_id: str = "w-1", - text: str = "hello", - worker_fingerprint: str | None = None, - response_schema_version: int | None = None, -) -> dict[str, Any]: - target: dict[str, Any] = {"worker_id": worker_id} - if worker_fingerprint is not None: - target["worker_fingerprint"] = worker_fingerprint - payload = { - "schema_version": 1, - "action": "send_instruction", - "request_id": request_id, - "dry_run": False, - "target": target, - "instruction": {"text": text}, - } - if response_schema_version is not None: - payload["response_schema_version"] = response_schema_version - return payload - - -def _healthy_backend() -> BackendHealth: - return BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - counts={"workers": 1}, - ) - - -def _degraded_backend() -> BackendHealth: - return BackendHealth( - name="herdr", - status="degraded", - outcome="timeout", - observed_at="2026-01-01T00:01:00+00:00", - ) - - -def _seed( - config: Config, - workers: list[Worker], - bindings: list[WorkerBinding] | None = None, - *, - health: BackendHealth | None = None, -) -> None: - assert config.db_path is not None - init_store(config.db_path) - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:00:00+00:00", - workers=workers, - backend_health=[health or _healthy_backend()], - ), - turn_model=config.turn_model, - ) - if bindings: - upsert_worker_bindings(config.db_path, bindings) - - -def _binding( - worker: Worker, - *, - target_kind: str = "agent_id", - value: str = "agent-secret", - sendable: bool = True, - reason: str | None = None, - fingerprint: str | None = None, - private_fingerprint: str = "private-secret", - turn_target_kind: str | None = "pane_id", - turn_target_value: str | None = "pane-secret", -) -> WorkerBinding: - return WorkerBinding( - host_id="cmd-host", - worker_id=worker.id, - worker_fingerprint=fingerprint or worker.fingerprint, - backend="herdr", - target_kind=target_kind, - target_value=value, - turn_target_kind=turn_target_kind, - turn_target_value=turn_target_value, - sendable=sendable, - reason=reason, - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint=private_fingerprint, - ) - - -_REALISTIC_VISIBLE_PANE = """\ -╭─ Claude Code ─────────────────────────────────────────────────────────────╮ -│ Completed the previous task. │ -╰───────────────────────────────────────────────────────────────────────────╯ -───────────────────────────────────────────────────────────────────────────── - ⏵⏵ accept edits on · esc to interrupt -""" - - -class _FakeSocketClient: - def __init__( - self, - calls: list[dict[str, Any]], - *, - raises: BaseException | None = None, - pane_id: str = "pane-secret", - ) -> None: - self.calls = calls - self.raises = raises - self.pane_id = pane_id - self.close_count = 0 - - def connect(self) -> "_FakeSocketClient": - return self - - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if self.raises is not None and method in {"pane.send_input", "agent.prompt"}: - raise self.raises - if method == "agent.get": - return {"result": {"agent": {"pane_id": self.pane_id}}} - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": _REALISTIC_VISIBLE_PANE}, - } - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": self.pane_id}, - "delivery": "submitted", - } - return {"accepted": True} - - def close(self) -> None: - self.close_count += 1 - - -class _PromptVerdictClient(_FakeSocketClient): - def __init__( - self, - calls: list[dict[str, Any]], - *, - delivery: str = "submitted", - error_code: str | None = None, - pane_reads: list[str] | None = None, - ) -> None: - super().__init__(calls) - self.delivery = delivery - self.error_code = error_code - self.pane_reads = list(pane_reads or [_REALISTIC_VISIBLE_PANE]) - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - if method == "pane.read": - self.calls.append({"method": method, "params": dict(params)}) - text = self.pane_reads.pop(0) if self.pane_reads else "" - return {"type": "pane_read", "read": {"text": text}} - if method == "agent.prompt": - self.calls.append({"method": method, "params": dict(params)}) - if self.error_code is not None: - raise HerdrErrorResponse( - { - "code": self.error_code, - "message": self.error_code, - }, - "test-request", - ) - return { - "type": "agent_prompted", - "agent": {"pane_id": self.pane_id}, - "delivery": self.delivery, - } - return super().request(method, params, timeout=timeout) - - -def _factory(calls: list[dict[str, Any]], *, raises: BaseException | None = None, pane_id: str = "pane-secret"): - def make_client(config: Config) -> _FakeSocketClient: - return _FakeSocketClient(calls, raises=raises, pane_id=pane_id) - - return make_client - - -def _expected_submit_calls( - target: str = "agent-secret", - *, - text: str = "hello", - timeout_ms: int = 5000, -) -> list[dict[str, Any]]: - return [ - {"method": "agent.get", "params": {"target": target}}, - { - "method": "agent.prompt", - "params": { - "target": target, - "text": text, - "wait": {"until": ["working"], "timeout_ms": timeout_ms}, - }, - }, - ] - - -def _expected_private_clear_calls(pane_id: str = "pane-secret") -> list[dict[str, Any]]: - return [ - {"method": "pane.send_keys", "params": {"pane_id": pane_id, "keys": ["ctrl+u"]}}, - {"method": "pane.send_keys", "params": {"pane_id": pane_id, "keys": ["ctrl+a", "ctrl+k"]}}, - {"method": "pane.send_keys", "params": {"pane_id": pane_id, "keys": ["ctrl+a", "backspace"]}}, - ] - -@pytest.mark.parametrize( - "action", ["send_instruction", "answer_pending", "answer_decision"] -) -@pytest.mark.parametrize( - ("request_id", "include_request_id"), - [ - pytest.param(None, False, id="missing"), - pytest.param(None, True, id="null"), - pytest.param(123, True, id="non-string"), - pytest.param("", True, id="empty"), - pytest.param("x" * 129, True, id="max-plus-one"), - pytest.param("x" * ((1024 * 1024) - 256), True, id="near-frame-size"), - pytest.param(" leading", True, id="leading-space"), - pytest.param("trailing ", True, id="trailing-space"), - pytest.param("interior space", True, id="interior-space"), - pytest.param("\t", True, id="tab"), - pytest.param("\n", True, id="newline"), - pytest.param("\0", True, id="nul"), - pytest.param("é", True, id="unicode-nfc"), - pytest.param("e\u0301", True, id="unicode-nfd"), - pytest.param("Ａ", True, id="unicode-fullwidth"), - ], -) -def test_submit_command_rejects_invalid_request_id_before_mutation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - action: str, - request_id: Any, - include_request_id: bool, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - init_store(config.db_path) - calls: list[str] = [] - - def guarded_store(*args: Any, **kwargs: Any) -> Any: - calls.append("store") - raise AssertionError("invalid request_id must not access the store") - - def guarded_observation(config: Config) -> Snapshot: - calls.append("observe") - raise AssertionError("invalid request_id must not observe") - - def guarded_socket_factory(config: Config) -> _FakeSocketClient: - calls.append("socket") - raise AssertionError("invalid request_id must not construct a socket client") - - monkeypatch.setattr("tendwire.command_submission.project_from_observations", guarded_observation) - monkeypatch.setattr(command_submission, "get_command_request", guarded_store) - monkeypatch.setattr(command_submission, "latest_snapshot", guarded_store) - payload = _request() - if action == "answer_pending": - payload = { - "schema_version": 1, - "action": "answer_pending", - "request_id": request_id, - "dry_run": False, - "params": { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - }, - } - elif action == "answer_decision": - payload = { - "schema_version": 1, - "action": "answer_decision", - "request_id": request_id, - "dry_run": False, - "target": {"worker_id": "w-1"}, - "params": { - "decision_ref": "decision-public", - "selection": {"option_refs": ["1"]}, - }, - } - if include_request_id: - payload["request_id"] = request_id - else: - del payload["request_id"] - - envelope = submit_command(config, payload, socket_client_factory=guarded_socket_factory) - - assert envelope.status == STATUS_INVALID_REQUEST - assert envelope.error is not None - assert envelope.error["code"] == STATUS_INVALID_REQUEST - assert calls == [] - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 0 - assert conn.execute("SELECT COUNT(*) FROM events").fetchone()[0] == 0 - - -@pytest.mark.parametrize( - ("label", "factory_exception", "connect_exception"), - [ - ("factory", RuntimeError("raw setup failure with private detail"), None), - ("path", ValueError("bad socket path /private/herdr.sock"), None), - ("missing", None, FileNotFoundError("missing /private/herdr.sock")), - ("refused", None, ConnectionRefusedError("refused /private/herdr.sock")), - ("permission", None, PermissionError("denied /private/herdr.sock")), - ], -) -def test_submit_command_socket_setup_failures_are_backend_unavailable( - tmp_path: Path, - label: str, - factory_exception: BaseException | None, - connect_exception: BaseException | None, -) -> None: - config = _config(tmp_path / label) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - class SetupFailsClient: - def connect(self) -> "SetupFailsClient": - assert connect_exception is not None - raise connect_exception - - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None) -> dict[str, Any]: - calls.append({"method": method, "params": dict(params)}) - raise AssertionError("private send request must not run before setup succeeds") - - def close(self) -> None: - return None - - def make_client(config: Config) -> SetupFailsClient: - if factory_exception is not None: - raise factory_exception - return SetupFailsClient() - - envelope = submit_command( - config, - _request(request_id=f"setup-{label}"), - socket_client_factory=make_client, - ) - - # Socket setup failed before any transmission. That is a safe pre-send - # transient: no send began, so the request ID stays retryable and no durable - # rejection receipt is written. - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert envelope.error is not None - assert envelope.error["code"] == STATUS_BACKEND_UNAVAILABLE - assert "private" not in json.dumps(envelope.to_dict()) - assert calls == [] - assert envelope.status != STATUS_NOT_FOUND - assert envelope.status != STATUS_REQUEST_STATE_UNCERTAIN - assert config.db_path is not None - assert get_command_request(config.db_path, "cmd-host", f"setup-{label}") is None - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 0 - command_events = conn.execute( - "SELECT COUNT(*) FROM events WHERE aggregate_type = 'command_request'" - ).fetchone()[0] - assert command_events == 0 - - # Once the transient clears, the same request ID succeeds exactly once. - recovery_calls: list[dict[str, Any]] = [] - recovered = submit_command( - config, - _request(request_id=f"setup-{label}"), - socket_client_factory=_factory(recovery_calls), - ) - assert recovered.status == STATUS_ACCEPTED - assert recovered.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert [call["method"] for call in recovery_calls].count("agent.prompt") == 1 - receipt = get_command_request(config.db_path, "cmd-host", f"setup-{label}") - assert receipt is not None - assert receipt["state"] == "accepted" - - -@pytest.mark.parametrize( - "exc", - [ - HerdrSocketDisconnectedError("disconnected"), - HerdrProtocolError("malformed response"), - OSError("transport failed after write"), - ], -) -def test_submit_command_post_send_transport_failures_are_uncertain( - tmp_path: Path, - exc: BaseException, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _request(request_id=f"uncertain-{type(exc).__name__}"), - socket_client_factory=_factory(calls, raises=exc), - ) - - assert envelope.status == STATUS_REQUEST_STATE_UNCERTAIN - assert envelope.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert envelope.status != STATUS_BACKEND_UNAVAILABLE - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, "cmd-host", f"uncertain-{type(exc).__name__}", "send_instruction") - assert receipt is not None - assert receipt["uncertain"] is True - with sqlite3.connect(str(config.db_path)) as conn: - events = [row[0] for row in conn.execute("SELECT event_type FROM events ORDER BY id").fetchall()] - ledger = conn.execute( - """ - SELECT state, terminal_at, submitted_at - FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, f"uncertain-{type(exc).__name__}"), - ).fetchone() - assert ledger is not None - assert ledger[0] == "uncertain" - assert ledger[1] is not None - assert ledger[2] is None - assert "command.request.send_started" in events - assert "command.request.uncertain" in events - - -def test_submit_command_uses_socket_pane_input_once_and_caches_result(tmp_path: Path) -> None: - config = _config(tmp_path) - stable_key = "wsk1_" + ("9" * 64) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={"stable_key": stable_key, "stable_key_version": 1}, - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command(config, _request(), socket_client_factory=_factory(calls)) - second = submit_command(config, _request(), socket_client_factory=_factory(calls)) - duplicate = submit_command( - config, - _request(text="changed"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_ACCEPTED - assert first.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert first.result["turn_id"] is None - assert first.result == { - "target": {"worker_id": "w-1"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "target_state_at_send": "active", - "observed_turn_state": "pending_observation", - "turn_id": first.result["turn_id"], - "submission_verdict": "submitted", - } - assert second.to_dict() == first.to_dict() - assert duplicate.status == STATUS_DUPLICATE_REQUEST - assert duplicate.disposition == DISPOSITION_TERMINAL_REJECTED - assert calls == _expected_submit_calls() - - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, "cmd-host", "req-1", "send_instruction") - assert receipt is not None - assert receipt["status"] == STATUS_ACCEPTED - assert receipt["uncertain"] is False - stored_result = json.loads(receipt["result_json"]) - assert stored_result["schema_version"] == 2 - assert stored_result["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - assert set(stored_result) == { - "schema_version", - "action", - "request_id", - "ok", - "dry_run", - "status", - "disposition", - "result", - "error", - "warnings", - } - turns_payload = turns_payload_from_store(config.db_path, "cmd-host") - assert turns_payload["turns"] == [] - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[worker], - backend_health=[_healthy_backend()], - ), - ) - turns_after_snapshot = turns_payload_from_store(config.db_path, "cmd-host") - assert turns_after_snapshot["turns"] == [] - - with sqlite3.connect(str(config.db_path)) as conn: - event_rows = conn.execute("SELECT event_type, payload_json FROM events ORDER BY id").fetchall() - command_row = conn.execute( - "SELECT request_json, result_json FROM commands WHERE request_id = 'req-1'" - ).fetchone() - submission_rows = conn.execute( - """ - SELECT submission_id, request_id, owner_key, owner_key_version, - instruction_fingerprint, state, linked_turn_id, - link_not_before, link_expires_at, hard_expires_at, - terminal_at, submitted_at, send_started_at, updated_at - FROM turn_submissions - """ - ).fetchall() - supersession_count = conn.execute( - "SELECT COUNT(*) FROM turn_supersessions" - ).fetchone()[0] - assert len(submission_rows) == 1 - submission = submission_rows[0] - assert submission[:7] == ( - turn_submission_id("cmd-host", "req-1"), - "req-1", - stable_key, - 1, - instruction_fingerprint("hello"), - "submitted", - None, - ) - link_not_before = datetime.fromisoformat(submission[7]) - link_expires_at = datetime.fromisoformat(submission[8]) - hard_expires_at = datetime.fromisoformat(submission[9]) - send_started_at = datetime.fromisoformat(submission[12]) - assert (send_started_at - link_not_before).total_seconds() == 60 - assert (link_expires_at - send_started_at).total_seconds() == 60 - assert (hard_expires_at - send_started_at).total_seconds() == 86_400 - assert submission[10] == submission[11] == submission[13] - assert datetime.fromisoformat(submission[11]) >= send_started_at - assert supersession_count == 0 - assert [row[0] for row in event_rows] == [ - "snapshot.saved", - "command.request.reserved", - "command.request.send_started", - "command.request.accepted", - ] - command_events = [json.loads(row[1]) for row in event_rows[1:]] - assert [ - (event["state"], event["status"]) - for event in command_events - ] == [ - ("reserved", STATUS_PENDING), - ("send_started", STATUS_PENDING), - ("accepted", STATUS_ACCEPTED), - ] - assert "detail" not in command_events[0] - assert command_events[1]["detail"]["target"] == {"worker_id": "w-1"} - assert command_events[2]["detail"]["envelope"] == first.to_dict() - assert command_row is not None - assert json.loads(command_row[0])["target"] == {"worker_id": "w-1"} - assert "request_id" not in json.loads(command_row[0]) - - public_surfaces = [ - first.to_dict(), - duplicate.to_dict(), - turns_payload, - turns_after_snapshot, - json.loads(receipt["result_json"]), - json.loads(command_row[1]), - *[json.loads(row[1]) for row in event_rows], - ] - encoded = json.dumps(public_surfaces) - assert "agent-secret" not in encoded - assert "private-secret" not in encoded - for surface in public_surfaces: - _assert_no_private_json(surface) - - - -def test_request_id_can_be_resubmitted_after_receipt_retention_purge( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="purged-request"), - socket_client_factory=_factory(calls), - ) - survivor = submit_command( - config, - _request(request_id="retention-survivor"), - socket_client_factory=_factory(calls), - ) - assert first.status == survivor.status == STATUS_ACCEPTED - assert config.db_path is not None - - cleanup = cleanup_command_request_retention( - config.db_path, - retry_horizon_seconds=604_800, - retention_seconds=691_200, - retention_count=1, - host_id=config.host_id, - now="2099-01-01T00:00:00+00:00", - ) - assert cleanup["deleted"] == 1 - assert get_command_request( - config.db_path, - config.host_id, - "purged-request", - ) is None - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - """ - SELECT COUNT(*) FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, "purged-request"), - ).fetchone() == (0,) - - resubmitted = submit_command( - config, - _request(request_id="purged-request"), - socket_client_factory=_factory(calls), - ) - - assert resubmitted.status == STATUS_ACCEPTED - assert resubmitted.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert [call["method"] for call in calls].count("agent.prompt") == 3 - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - """ - SELECT state FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, "purged-request"), - ).fetchone() == ("submitted",) - - -def test_submission_envelope_v3_requires_explicit_negotiation(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - default = submit_command( - config, - _request(request_id="default-v2"), - socket_client_factory=_factory(calls), - ) - opted_in = submit_command( - config, - _request(request_id="opted-v3", response_schema_version=3), - socket_client_factory=_factory(calls), - ) - replayed = submit_command( - config, - _request(request_id="opted-v3", response_schema_version=3), - socket_client_factory=_factory(calls), - ) - - assert default.schema_version == 2 - assert "submission_id" not in default.result - assert set(default.to_dict()) == { - "schema_version", "action", "request_id", "ok", "dry_run", - "status", "disposition", "result", "error", "warnings", - } - assert opted_in.schema_version == 3 - assert opted_in.result["submission_id"] == turn_submission_id( - config.host_id, - "opted-v3", - ) - assert opted_in.result["turn_id"] is None - assert replayed.to_dict() == opted_in.to_dict() - assert [call["method"] for call in calls].count("agent.prompt") == 2 - - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, "opted-v3") - assert receipt is not None - stored = json.loads(receipt["result_json"]) - assert stored["schema_version"] == 2 - assert "submission_id" not in stored["result"] - - -def test_submission_fingerprint_is_owner_isolated(tmp_path: Path) -> None: - config = _config(tmp_path) - first_key = "wsk1_" + ("1" * 64) - second_key = "wsk1_" + ("2" * 64) - first_worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={"stable_key": first_key, "stable_key_version": 1}, - ) - second_worker = Worker( - id="w-2", - name="Beta", - status="active", - meta={"stable_key": second_key, "stable_key_version": 1}, - ) - _seed( - config, - [first_worker, second_worker], - [ - _binding( - first_worker, - value="agent-1", - private_fingerprint="private-1", - turn_target_value="pane-1", - ), - _binding( - second_worker, - value="agent-2", - private_fingerprint="private-2", - turn_target_value="pane-2", - ), - ], - ) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="owner-1", worker_id="w-1", text="hello world"), - socket_client_factory=_factory(calls, pane_id="pane-1"), - ) - second = submit_command( - config, - _request(request_id="owner-2", worker_id="w-2", text=" hello world "), - socket_client_factory=_factory(calls, pane_id="pane-2"), - ) - - assert first.status == second.status == STATUS_ACCEPTED - assert config.db_path is not None - with sqlite3.connect(str(config.db_path)) as conn: - rows = conn.execute( - """ - SELECT owner_key, owner_key_version, instruction_fingerprint - FROM turn_submissions ORDER BY request_id - """ - ).fetchall() - assert rows == [ - (first_key, 1, instruction_fingerprint("hello world")), - (second_key, 1, instruction_fingerprint("hello world")), - ] - - - - - -def test_submission_first_keeps_observation_authoritative_during_send( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={ - "stable_key": "wsk1_" + ("c" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - class ObservingClient(_FakeSocketClient): - def request(self, method, params, *, timeout=None): - result = super().request(method, params, timeout=timeout) - if method == "agent.prompt": - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "submission-first-source", - "user_text": "hello", - "assistant_final_text": "observed during send", - "complete": True, - "has_open_turn": False, - }, - observed_at=datetime.now(timezone.utc).isoformat(), - ) == 1 - return result - - accepted = submit_command( - config, - _request(request_id="submission-first-request"), - socket_client_factory=lambda _config: ObservingClient(calls), - ) - - assert accepted.status == STATUS_ACCEPTED - assert accepted.result["observed_turn_state"] == "complete" - assert isinstance(accepted.result["turn_id"], str) - turns = turns_payload_from_store( - config.db_path, - config.host_id, - schema_version=2, - )["turns"] - matching = [turn for turn in turns if turn.get("user_text") == "hello"] - assert len(matching) == 1 - assert accepted.result["turn_id"] == matching[0]["id"] - assert matching[0]["assistant_final_text"] == "observed during send" - - - -def test_submit_command_sends_identical_100_character_instructions_without_turn_rows( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={ - "stable_key": "wsk1_" + ("d" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - text = "x" * 100 - - first = submit_command( - config, - _request(request_id="long-1", text=text), - socket_client_factory=_factory(calls), - ) - second = submit_command( - config, - _request(request_id="long-2", text=text), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_ACCEPTED - assert second.status == STATUS_ACCEPTED - assert first.result["turn_id"] is None - assert second.result["turn_id"] is None - expected_send = _expected_submit_calls(text=text) - assert calls == [*expected_send, *expected_send] - - assert config.db_path is not None - first_receipt = get_command_request(config.db_path, "cmd-host", "long-1") - second_receipt = get_command_request(config.db_path, "cmd-host", "long-2") - assert first_receipt is not None - assert first_receipt["state"] == "accepted" - assert second_receipt is not None - assert second_receipt["state"] == "accepted" - with sqlite3.connect(str(config.db_path)) as conn: - turn_count = conn.execute("SELECT COUNT(*) FROM turns").fetchone()[0] - events = [ - row[0] - for row in conn.execute( - "SELECT event_type FROM events WHERE aggregate_type = 'command_request' ORDER BY id" - ).fetchall() - ] - assert turn_count == 0 - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "ambiguous-identical-source", - "user_text": text, - "assistant_final_text": "independent observation", - "complete": True, - "has_open_turn": False, - }, - observed_at="2099-07-19T10:00:00+00:00", - ) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - stored = [ - (str(turn_id), json.loads(payload_json)) - for turn_id, payload_json in conn.execute( - "SELECT turn_id, payload_json FROM turns WHERE host_id = ?", - (config.host_id,), - ).fetchall() - ] - relevant = [ - (turn_id, payload) - for turn_id, payload in stored - if payload.get("source_turn_id") - ] - assert len(relevant) == 1 - observed = [ - (turn_id, payload) - for turn_id, payload in relevant - if payload.get("source_turn_id") - ] - assert len(observed) == 1 - assert events == [ - "command.request.reserved", - "command.request.send_started", - "command.request.accepted", - "command.request.reserved", - "command.request.send_started", - "command.request.accepted", - ] - public_json = json.dumps( - [ - first.to_dict(), - second.to_dict(), - json.loads(first_receipt["result_json"]), - json.loads(second_receipt["result_json"]), - ] - ) - assert text not in public_json - assert "agent-secret" not in public_json - assert "private-secret" not in public_json - - -def test_submit_command_allows_same_instruction_after_worker_fingerprint_changes( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - old_worker = Worker(id="w-1", name="Alpha", status="active", fingerprint="old-fp") - new_worker = Worker(id="w-1", name="Alpha", status="active", fingerprint="new-fp") - _seed( - config, - [old_worker], - [_binding(old_worker, value="old-agent-secret", private_fingerprint="old-private-secret")], - ) - calls: list[dict[str, Any]] = [] - text = "When this exact long Telegram instruction appears for a new binding, it should send." - - first = submit_command( - config, - _request(request_id="fingerprint-1", text=text, worker_fingerprint="old-fp"), - socket_client_factory=_factory(calls), - ) - assert first.status == STATUS_ACCEPTED - - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[new_worker], - backend_health=[_healthy_backend()], - ), - ) - upsert_worker_bindings( - config.db_path, - [_binding(new_worker, value="new-agent-secret", private_fingerprint="new-private-secret")], - ) - - second = submit_command( - config, - _request(request_id="fingerprint-2", text=text, worker_fingerprint="new-fp"), - socket_client_factory=_factory(calls), - ) - - assert second.status == STATUS_ACCEPTED - assert calls == [ - *_expected_submit_calls("old-agent-secret", text=text), - *_expected_submit_calls("new-agent-secret", text=text), - ] - - -def test_submit_command_terminal_worker_id_and_fingerprint_replays_after_healthy_worker_churn( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - fingerprint="worker-fingerprint-1", - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request = _request( - request_id="terminal-worker-churn", - worker_fingerprint=worker.fingerprint, - ) - - accepted = submit_command( - config, - request, - socket_client_factory=_factory(calls), - ) - assert accepted.status == STATUS_ACCEPTED - assert accepted.disposition == DISPOSITION_TERMINAL_ACCEPTED - - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[], - backend_health=[_healthy_backend()], - ), - ) - no_backend = lambda _config: pytest.fail( - "terminal replay after worker churn must not create a socket client" - ) - - exact_replay = submit_command( - config, - request, - socket_client_factory=no_backend, - ) - refreshed_fingerprint = submit_command( - config, - _request( - request_id="terminal-worker-churn", - worker_fingerprint="worker-fingerprint-2", - ), - socket_client_factory=no_backend, - ) - changed_worker = submit_command( - config, - _request( - request_id="terminal-worker-churn", - worker_id="w-2", - worker_fingerprint=worker.fingerprint, - ), - socket_client_factory=no_backend, - ) - changed_instruction = submit_command( - config, - _request( - request_id="terminal-worker-churn", - text="changed", - worker_fingerprint=worker.fingerprint, - ), - socket_client_factory=no_backend, - ) - - assert exact_replay.to_dict() == accepted.to_dict() - assert refreshed_fingerprint.to_dict() == accepted.to_dict() - assert changed_worker.status == STATUS_DUPLICATE_REQUEST - assert changed_worker.disposition == DISPOSITION_TERMINAL_REJECTED - assert changed_instruction.status == STATUS_DUPLICATE_REQUEST - assert changed_instruction.disposition == DISPOSITION_TERMINAL_REJECTED - assert calls == _expected_submit_calls() - - -def test_submit_command_uses_verified_agent_prompt(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls)) - - assert envelope.status == STATUS_ACCEPTED - assert calls == _expected_submit_calls() - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - assert not any(call["method"] == "pane.read" for call in calls) - assert not any(call["method"] == "pane.send_text" for call in calls) - assert not any(call["method"] == "pane.send_keys" for call in calls) - - -def test_written_to_pty_prior_running_turn_observed_later_stays_queued( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker( - id="w-1", - name="Alpha", - status="working", - meta={ - "stable_key": "wsk1_" + ("7" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "queued-written-to-pty" - - queued = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - delivery="written_to_pty", - ), - ) - replay = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "queued replay must not issue a second prompt" - ), - ) - - assert queued.status == replay.status == STATUS_PENDING - assert queued.disposition == replay.disposition == DISPOSITION_IN_PROGRESS - assert queued.result["submission_verdict"] == "written_to_pty" - assert queued.result["delivery_state"] == "queued" - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - UPDATE turn_submissions - SET submitted_at = ?, send_started_at = ?, link_not_before = ? - WHERE host_id = ? AND request_id = ? - """, - ( - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - config.host_id, - request_id, - ), - ) - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "queued-observed-source", - "user_text": "hello", - "assistant_final_text": None, - "complete": False, - "has_open_turn": True, - }, - observed_at="2026-01-01T00:00:01+00:00", - ) == 1 - - still_queued = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "queued replay with an observed prior turn must not issue a second prompt" - ), - ) - - assert still_queued.status == STATUS_PENDING - assert still_queued.disposition == DISPOSITION_IN_PROGRESS - assert still_queued.result["submission_verdict"] == "written_to_pty" - assert still_queued.result["transport_state"] == "queued" - assert still_queued.result["turn_id"] is None - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - """ - SELECT linked_turn_id FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, request_id), - ).fetchone() == (None,) - - -def test_written_to_pty_expired_verification_becomes_terminal_uncertain( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker(id="w-1", name="Alpha", status="working") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "queued-verification-expired" - request = _request(request_id=request_id) - - queued = submit_command( - config, - request, - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - delivery="written_to_pty", - ), - ) - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - UPDATE turn_submissions - SET link_expires_at = ?, hard_expires_at = ? - WHERE host_id = ? AND request_id = ? - """, - ( - "2000-01-01T00:00:00+00:00", - "2000-01-01T00:00:00+00:00", - config.host_id, - request_id, - ), - ) - - uncertain = submit_command( - config, - request, - socket_client_factory=lambda _config: pytest.fail( - "expired queued replay must not issue another prompt" - ), - ) - replay = submit_command( - config, - request, - socket_client_factory=lambda _config: pytest.fail( - "terminal-uncertain replay must not issue another prompt" - ), - ) - - assert queued.status == STATUS_PENDING - assert uncertain.to_dict() == replay.to_dict() - assert uncertain.status == STATUS_REQUEST_STATE_UNCERTAIN - assert uncertain.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert uncertain.result["submission_verdict"] == "written_to_pty" - assert uncertain.result["delivery_state"] == "unknown" - assert uncertain.error is not None - assert "verification expired" in uncertain.error["message"] - assert "queued" not in uncertain.error["message"].lower() - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - """ - SELECT state, status - FROM command_receipts - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, request_id), - ).fetchone() == ("uncertain", STATUS_REQUEST_STATE_UNCERTAIN) - assert conn.execute( - """ - SELECT state - FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, request_id), - ).fetchone() == ("expired",) - - -def test_written_to_pty_replay_settles_only_its_component( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker(id="w-1", name="Alpha", status="working") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "queued-one-of-forty" - request = _request(request_id=request_id) - - queued = submit_command( - config, - request, - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - delivery="written_to_pty", - ), - ) - assert queued.status == STATUS_PENDING - with sqlite3.connect(str(config.db_path)) as conn: - for index in range(1, 40): - conn.execute( - """ - INSERT INTO turn_submissions ( - host_id, submission_id, request_id, owner_key, - owner_key_version, instruction_fingerprint, state, - linked_turn_id, link_not_before, link_expires_at, - hard_expires_at, linked_at, terminal_at, submitted_at, - send_started_at, updated_at - ) - SELECT host_id, ?, ?, ?, owner_key_version, ?, state, - linked_turn_id, link_not_before, link_expires_at, - hard_expires_at, linked_at, terminal_at, submitted_at, - send_started_at, updated_at - FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - ( - f"submission-extra-{index}", - f"request-extra-{index}", - f"owner-extra-{index}", - f"fingerprint-extra-{index}", - config.host_id, - request_id, - ), - ) - assert conn.execute( - """ - SELECT COUNT(DISTINCT owner_key || ':' || instruction_fingerprint) - FROM turn_submissions - WHERE host_id = ? - """, - (config.host_id,), - ).fetchone() == (40,) - own_component = conn.execute( - """ - SELECT owner_key, instruction_fingerprint - FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, request_id), - ).fetchone() - assert own_component is not None - - original_settle = store_sqlite.settle_submission_links_conn - settled_components: list[tuple[str, str]] = [] - - def counted_settle( - conn: sqlite3.Connection, - host_id: str, - owner_key: str, - instruction_fingerprint_value: str, - **kwargs: Any, - ) -> int: - settled_components.append((owner_key, instruction_fingerprint_value)) - return original_settle( - conn, - host_id, - owner_key, - instruction_fingerprint_value, - **kwargs, - ) - - monkeypatch.setattr( - store_sqlite, - "settle_submission_links_conn", - counted_settle, - ) - replay = submit_command( - config, - request, - socket_client_factory=lambda _config: pytest.fail( - "queued replay must not issue another prompt" - ), - ) - - assert replay.status == STATUS_PENDING - assert settled_components == [own_component] - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -@pytest.mark.parametrize( - "verdict", - [ - "agent_not_ready", - "agent_target_ambiguous", - "agent_prompt_not_received", - "agent_prompt_unsubmitted", - "agent_input_pending", - ], -) -def test_positive_non_delivery_verdict_is_terminal_rejected_without_retry( - tmp_path: Path, - verdict: str, -) -> None: - config = _config(tmp_path / verdict) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = f"not-delivered-{verdict}" - - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - error_code=verdict, - ), - ) - second = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "terminal non-delivery replay must not issue another prompt" - ), - ) - - assert first.to_dict() == second.to_dict() - assert first.status == STATUS_REJECTED - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert first.result["submission_verdict"] == verdict - assert first.result["delivery_state"] == "not_delivered" - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_agent_prompt_stalled_never_infers_composer_from_stale_scrollback( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "stalled-stale-scrollback" - - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - error_code="agent_prompt_stalled", - pane_reads=[ - f"{_REALISTIC_VISIBLE_PANE}\n" - "User: hello\n" - "Assistant: an answer from an old completed turn" - ], - ), - ) - second = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "stalled replay must not issue another prompt" - ), - ) - - assert first.to_dict() == second.to_dict() - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert first.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert first.result["submission_verdict"] == "agent_prompt_stalled" - assert "composer_state" not in first.result - assert not any(call["method"] == "pane.read" for call in calls) - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_agent_not_found_remains_unknown_without_retry(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "agent-not-found-unknown" - - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - error_code="agent_not_found", - ), - ) - second = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "unknown agent_not_found replay must not issue another prompt" - ), - ) - - assert first.to_dict() == second.to_dict() - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert first.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert first.result["submission_verdict"] == "unknown" - assert first.result["delivery_state"] == "unknown" - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_dirty_composer_non_delivery_is_surfaced_without_clear_or_resend( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "dirty-composer" - - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient( - calls, - error_code="agent_prompt_unsubmitted", - ), - ) - second = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "dirty-composer non-delivery replay must not resend" - ), - ) - - assert first.to_dict() == second.to_dict() - assert first.status == STATUS_REJECTED - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert first.result["submission_verdict"] == "agent_prompt_unsubmitted" - assert first.result["delivery_state"] == "not_delivered" - assert calls == _expected_submit_calls() - assert not any(call["method"] == "pane.send_keys" for call in calls) - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_crash_after_prompt_write_recovers_unknown_without_resend( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = "write-before-verdict-record" - real_finish = command_submission.finish_command_request - - def crash_before_record(*args: Any, **kwargs: Any) -> Any: - raise RuntimeError("simulated process loss after prompt write") - - monkeypatch.setattr( - command_submission, - "finish_command_request", - crash_before_record, - ) - first = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: _PromptVerdictClient(calls), - ) - assert first.status == STATUS_PENDING - assert first.disposition == DISPOSITION_IN_PROGRESS - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - monkeypatch.setattr( - command_submission, - "finish_command_request", - real_finish, - ) - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - UPDATE command_receipts - SET owner_expires_at = ? - WHERE host_id = ? AND request_id = ? - """, - ("2000-01-01T00:00:00+00:00", config.host_id, request_id), - ) - - recovered = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "unknown recovery must never resend" - ), - ) - - assert recovered.status == STATUS_REQUEST_STATE_UNCERTAIN - assert recovered.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert recovered.result["submission_verdict"] == "unknown" - assert recovered.result["delivery_state"] == "unknown" - assert sum(call["method"] == "agent.prompt" for call in calls) == 1 - - -def test_submit_command_reports_submitted_transport_and_worker_state(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="working") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls)) - - assert envelope.status == STATUS_ACCEPTED - assert envelope.result["turn_id"] is None - assert envelope.result == { - "target": {"worker_id": "w-1"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "target_state_at_send": "active", - "observed_turn_state": "pending_observation", - "turn_id": envelope.result["turn_id"], - "submission_verdict": "submitted", - } - - -def test_submit_command_marks_idle_worker_delivery_as_submitted(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="idle") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls)) - - assert envelope.status == STATUS_ACCEPTED - assert envelope.result["turn_id"] is None - assert envelope.result == { - "target": {"worker_id": "w-1"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "target_state_at_send": "idle", - "observed_turn_state": "pending_observation", - "turn_id": envelope.result["turn_id"], - "submission_verdict": "submitted", - } - - -def test_submit_command_terminal_binding_resolves_pane_and_submits_input(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker, target_kind="terminal_id", value="term-secret")]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls, pane_id="pane-private")) - - assert envelope.status == STATUS_ACCEPTED - assert calls == _expected_submit_calls("term-secret") - public_json = json.dumps(envelope.to_dict()) - assert "term-secret" not in public_json - assert "pane-private" not in public_json - _assert_no_private_json(envelope.to_dict()) - - -def test_submit_command_pane_binding_submits_without_public_pane_leak(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker, target_kind="pane_id", value="pane-private")]) - calls: list[dict[str, Any]] = [] - - envelope = submit_command(config, _request(), socket_client_factory=_factory(calls)) - - assert envelope.status == STATUS_ACCEPTED - assert calls == [ - { - "method": "agent.prompt", - "params": { - "target": "pane-private", - "text": "hello", - "wait": {"until": ["working"], "timeout_ms": 5000}, - }, - }, - ] - public_json = json.dumps(envelope.to_dict()) - assert "pane-private" not in public_json - _assert_no_private_json(envelope.to_dict()) - - -def test_submit_command_backend_unavailable_prevents_not_found_and_send(tmp_path: Path) -> None: - config = _config(tmp_path) - health = BackendHealth( - name="herdr", - status="degraded", - outcome="protocol_error", - observed_at="2026-01-01T00:00:00+00:00", - ) - _seed(config, [], [], health=health) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _request(worker_id="missing", request_id="degraded-1"), - socket_client_factory=_factory(calls), - ) - - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert calls == [] - assert envelope.status != STATUS_NOT_FOUND - assert config.db_path is not None - assert get_command_request(config.db_path, config.host_id, "degraded-1") is None - - -def test_submit_command_missing_worker_can_return_not_found_when_backend_healthy(tmp_path: Path) -> None: - config = _config(tmp_path) - _seed(config, [], []) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _request(worker_id="missing", request_id="missing-1"), - socket_client_factory=_factory(calls), - ) - - assert envelope.status == STATUS_NOT_FOUND - assert calls == [] -def test_submit_command_resolved_health_failure_is_terminal_and_replayed( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)], health=_degraded_backend()) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="resolved-degraded"), - socket_client_factory=_factory(calls), - ) - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:02:00+00:00", - workers=[worker], - backend_health=[_healthy_backend()], - ), - ) - replay = submit_command( - config, - _request(request_id="resolved-degraded"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_TERMINAL_REJECTED - assert replay.to_dict() == first.to_dict() - assert calls == [] - receipt = get_command_request( - config.db_path, - config.host_id, - "resolved-degraded", - ) - assert receipt is not None - assert receipt["state"] == "rejected" - assert receipt["status"] == STATUS_BACKEND_UNAVAILABLE - stored_result = json.loads(receipt["result_json"]) - assert stored_result["schema_version"] == 2 - assert stored_result["disposition"] == DISPOSITION_TERMINAL_REJECTED - - -def test_backend_unavailable_disposition_depends_on_receipt_authority( - tmp_path: Path, -) -> None: - no_authority_config = _config(tmp_path / "no-authority") - _seed(no_authority_config, [], [], health=_degraded_backend()) - no_authority = submit_command( - no_authority_config, - _request(request_id="unavailable-no-authority", worker_id="missing"), - socket_client_factory=lambda _config: pytest.fail( - "unresolved authority must not create a socket client" - ), - ) - - rejected_config = _config(tmp_path / "terminal-rejection") - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - rejected_config, - [worker], - [_binding(worker)], - health=_degraded_backend(), - ) - terminal_rejection = submit_command( - rejected_config, - _request(request_id="unavailable-terminal-rejection"), - socket_client_factory=lambda _config: pytest.fail( - "failed health must not create a socket client" - ), - ) - - assert no_authority.status == terminal_rejection.status == STATUS_BACKEND_UNAVAILABLE - assert no_authority.disposition == DISPOSITION_NO_RECEIPT - assert terminal_rejection.disposition == DISPOSITION_TERMINAL_REJECTED - assert no_authority_config.db_path is not None - assert rejected_config.db_path is not None - assert get_command_request( - no_authority_config.db_path, - no_authority_config.host_id, - "unavailable-no-authority", - ) is None - receipt = get_command_request( - rejected_config.db_path, - rejected_config.host_id, - "unavailable-terminal-rejection", - ) - assert receipt is not None - assert receipt["state"] == "rejected" - - -def test_submit_command_disallowed_worker_rejection_is_terminal_and_replayed( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - closed = Worker(id="w-1", name="Alpha", status="closed") - _seed(config, [closed], [_binding(closed)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="closed-worker"), - socket_client_factory=_factory(calls), - ) - active = Worker(id="w-1", name="Alpha", status="active") - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:02:00+00:00", - workers=[active], - backend_health=[_healthy_backend()], - ), - ) - replay = submit_command( - config, - _request(request_id="closed-worker"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_REJECTED - assert replay.to_dict() == first.to_dict() - assert calls == [] - receipt = get_command_request(config.db_path, config.host_id, "closed-worker") - assert receipt is not None - assert receipt["state"] == "rejected" - assert receipt["status"] == STATUS_REJECTED - - -def test_submit_command_rejects_stale_worker_fingerprint_and_binding(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker, fingerprint="old-fingerprint")]) - calls: list[dict[str, Any]] = [] - - stale_request = submit_command( - config, - _request(request_id="stale-request", worker_fingerprint="old-fingerprint"), - socket_client_factory=_factory(calls), - ) - stale_binding = submit_command( - config, - _request(request_id="stale-binding"), - socket_client_factory=_factory(calls), - ) - - assert stale_request.status == STATUS_STALE_TARGET - assert stale_binding.status == STATUS_STALE_TARGET - assert calls == [] - - -def test_submit_command_rejects_duplicate_missing_and_unsendable_bindings(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - - _seed(config, [worker], []) - calls: list[dict[str, Any]] = [] - missing = submit_command( - config, - _request(request_id="missing-binding"), - socket_client_factory=_factory(calls), - ) - assert missing.status == STATUS_BACKEND_UNSUPPORTED - - config = _config(tmp_path / "unsendable") - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker, sendable=False, reason="disabled")]) - unsendable = submit_command( - config, - _request(request_id="unsendable-binding"), - socket_client_factory=_factory(calls), - ) - assert unsendable.status == STATUS_BACKEND_UNSUPPORTED - - config = _config(tmp_path / "duplicate") - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [ - _binding(worker, value="agent-a", private_fingerprint="private-a"), - _binding(worker, value="agent-b", private_fingerprint="private-b"), - ], - ) - duplicate = submit_command( - config, - _request(request_id="duplicate-binding"), - socket_client_factory=_factory(calls), - ) - assert duplicate.status == STATUS_AMBIGUOUS_BACKEND_TARGET - assert calls == [] - - -def test_submit_command_timeout_after_send_start_is_uncertain_and_not_retried(tmp_path: Path) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="timeout-1"), - socket_client_factory=_factory(calls, raises=HerdrSocketTimeoutError("timeout")), - ) - second = submit_command( - config, - _request(request_id="timeout-1"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert first.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert second.status == STATUS_REQUEST_STATE_UNCERTAIN - assert second.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == [ - *_expected_submit_calls(), - ] - - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, "cmd-host", "timeout-1", "send_instruction") - assert receipt is not None - assert receipt["uncertain"] is True - with sqlite3.connect(str(config.db_path)) as conn: - event_rows = conn.execute( - "SELECT event_type, payload_json FROM events " - "WHERE aggregate_id = ? ORDER BY id", - ("timeout-1",), - ).fetchall() - assert [row[0] for row in event_rows] == [ - "command.request.reserved", - "command.request.send_started", - "command.request.uncertain", - ] - payloads = [json.loads(row[1]) for row in event_rows] - assert [(item["state"], item["status"]) for item in payloads] == [ - ("reserved", STATUS_PENDING), - ("send_started", STATUS_PENDING), - ("uncertain", STATUS_REQUEST_STATE_UNCERTAIN), - ] - - -def test_submit_command_unprovable_selector_spelling_fails_closed( - tmp_path: Path, -) -> None: - """A spelling the receipt cannot vouch for is never resolved by a degraded snapshot.""" - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _request(request_id="selector-unavailable"), - socket_client_factory=_factory(calls), - ) - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[], - backend_health=[_degraded_backend()], - ), - ) - # The receipt was issued for an explicit worker ID, so it holds no proof of - # either alias. Only a healthy observation could show they mean the same - # worker, and a degraded one may not stand in for it. - by_name = _request(request_id="selector-unavailable") - by_name["target"] = {"name": "Alpha"} - by_space = _request(request_id="selector-unavailable") - by_space["target"] = {"space_id": "space-1"} - - name_replay = submit_command(config, by_name, socket_client_factory=_factory(calls)) - space_replay = submit_command(config, by_space, socket_client_factory=_factory(calls)) - - assert first.status == STATUS_ACCEPTED - assert name_replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert name_replay.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert space_replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert space_replay.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == _expected_submit_calls() - receipt = get_command_request(config.db_path, config.host_id, "selector-unavailable") - assert receipt is not None - assert (receipt["state"], receipt["status"]) == ("accepted", STATUS_ACCEPTED) - - -def test_submit_command_equivalent_resolved_selectors_replay_once( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - by_name = _request(request_id="selector-equivalent") - by_name["target"] = {"name": "Alpha"} - by_space_with_origin = _request(request_id="selector-equivalent") - by_space_with_origin["target"] = {"space_id": "space-1"} - by_space_with_origin["params"] = {"origin": "connector-observation"} - - first = submit_command( - config, - _request(request_id="selector-equivalent"), - socket_client_factory=_factory(calls), - ) - second = submit_command(config, by_name, socket_client_factory=_factory(calls)) - third = submit_command( - config, - by_space_with_origin, - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_ACCEPTED - assert second.to_dict() == first.to_dict() - assert third.to_dict() == first.to_dict() - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, "selector-equivalent") - assert receipt is not None - assert receipt["public_worker_id"] == worker.id - canonical = json.loads(receipt["canonical_request_json"]) - assert canonical["target"] == {"worker_id": worker.id} - assert canonical["options"] == {} - assert "origin" not in receipt["canonical_request_json"] - - -@pytest.mark.parametrize("selector_kind", ["name", "space_id"]) -def test_submit_command_exact_selector_retry_survives_selector_reuse( - tmp_path: Path, - selector_kind: str, -) -> None: - """A reused name or space is worker churn, not a changed request.""" - config = _config(tmp_path / selector_kind) - first_worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - second_worker = Worker( - id="w-2", - name="Beta", - status="active", - space_id="space-2", - ) - _seed(config, [first_worker, second_worker], [_binding(first_worker)]) - selector = {"name": "Alpha"} if selector_kind == "name" else {"space_id": "space-1"} - request = _request(request_id=f"selector-reused-{selector_kind}") - request["target"] = selector - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, request, socket_client_factory=_factory(calls)) - # The selector the caller spelled now names a different public worker. - if selector_kind == "name": - changed_workers = [ - Worker(id="w-1", name="Former", status="active", space_id="space-1"), - Worker(id="w-2", name="Alpha", status="active", space_id="space-2"), - ] - else: - changed_workers = [ - Worker(id="w-1", name="Alpha", status="active", space_id="former-space"), - Worker(id="w-2", name="Beta", status="active", space_id="space-1"), - ] - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:02:00+00:00", - workers=changed_workers, - backend_health=[_healthy_backend()], - ), - ) - replay = submit_command( - config, - request, - socket_client_factory=lambda _config: pytest.fail( - "an exact retry must not create a socket client" - ), - ) - - assert accepted.status == STATUS_ACCEPTED - # Re-resolving the selector would deliver the instruction a second time, to - # a worker the caller never addressed. The receipt outranks the snapshot. - assert replay.to_dict() == accepted.to_dict() - assert replay.result is not None - assert replay.result["target"] == {"worker_id": "w-1"} - assert calls == _expected_submit_calls() - - -@pytest.mark.parametrize( - ("selector_kind", "changed_selector"), - [ - ("name", {"name": "Beta"}), - ("space_id", {"space_id": "space-2"}), - ("name_and_space", {"name": "Beta", "space_id": "space-2"}), - ], -) -def test_submit_command_changed_selector_conflicts_without_backend( - tmp_path: Path, - selector_kind: str, - changed_selector: dict[str, Any], -) -> None: - """Reusing a request ID with a different selector cannot claim its result.""" - config = _config(tmp_path / selector_kind) - first_worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - second_worker = Worker( - id="w-2", - name="Beta", - status="active", - space_id="space-2", - ) - _seed(config, [first_worker, second_worker], [_binding(first_worker)]) - request = _request(request_id=f"selector-changed-{selector_kind}") - request["target"] = {"name": "Alpha"} - calls: list[dict[str, Any]] = [] - - accepted = submit_command(config, request, socket_client_factory=_factory(calls)) - changed = _request(request_id=f"selector-changed-{selector_kind}") - changed["target"] = changed_selector - conflict = submit_command( - config, - changed, - socket_client_factory=lambda _config: pytest.fail( - "a changed selector must not create a socket client" - ), - ) - - assert accepted.status == STATUS_ACCEPTED - assert conflict.status == STATUS_DUPLICATE_REQUEST - assert conflict.disposition == DISPOSITION_TERMINAL_REJECTED - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - f"selector-changed-{selector_kind}", - ) - assert receipt is not None - assert (receipt["state"], receipt["status"]) == ("accepted", STATUS_ACCEPTED) - assert json.loads(receipt["result_json"]) == accepted.to_dict() - - -def test_submit_command_migrated_v11_exact_raw_request_replays( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - request_payload = _request(request_id="legacy-exact") - request = CommandRequest.from_dict(request_payload) - accepted = CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result={ - "target": {"worker_id": "w-1"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "target_state_at_send": "active", - "observed_turn_state": "pending_observation", - }, - ) - legacy_result = accepted.to_dict() - legacy_result.pop("disposition") - legacy_result["schema_version"] = 1 - legacy_fingerprint = request.payload_fingerprint() - result_json = json.dumps( - legacy_result, - sort_keys=True, - separators=(",", ":"), - ) - request_json = json.dumps( - request.to_dict(), - sort_keys=True, - separators=(",", ":"), - ) - with sqlite3.connect(str(config.db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0) - """, - ( - config.host_id, - request.request_id, - request.action, - legacy_fingerprint, - STATUS_ACCEPTED, - result_json, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES (?, ?, ?, ?, ?, 0, 0, ?, ?, ?, ?, ?, ?) - """, - ( - config.host_id, - request.request_id, - request.action, - legacy_fingerprint, - STATUS_ACCEPTED, - request_json, - result_json, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute("PRAGMA user_version = 11") - - init_store(config.db_path) - replay = submit_command( - config, - request_payload, - socket_client_factory=lambda _config: pytest.fail( - "legacy terminal replay must not create a socket client" - ), - ) - changed = submit_command( - config, - _request(request_id="legacy-exact", text="changed"), - socket_client_factory=lambda _config: pytest.fail( - "legacy request collision must not create a socket client" - ), - ) - - assert replay.to_dict() == accepted.to_dict() - assert replay.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert changed.status == STATUS_DUPLICATE_REQUEST - assert changed.disposition == DISPOSITION_TERMINAL_REJECTED - - -def test_submit_command_same_id_text_target_and_action_collide_without_backend( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - first_worker = Worker(id="w-1", name="Alpha", status="active") - second_worker = Worker(id="w-2", name="Beta", status="active") - _seed( - config, - [first_worker, second_worker], - [ - _binding(first_worker), - _binding( - second_worker, - value="other-agent-secret", - private_fingerprint="other-private-secret", - ), - ], - ) - calls: list[dict[str, Any]] = [] - request_id = "collision-1" - - accepted = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=_factory(calls), - ) - changed_text = submit_command( - config, - _request(request_id=request_id, text="changed"), - socket_client_factory=lambda _config: pytest.fail( - "text collision must not create a socket client" - ), - ) - changed_target = submit_command( - config, - _request(request_id=request_id, worker_id=second_worker.id), - socket_client_factory=lambda _config: pytest.fail( - "target collision must not create a socket client" - ), - ) - changed_action = submit_command( - config, - _answer_request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "action collision must not create a socket client" - ), - ) - unknown_options = _request(request_id=request_id) - unknown_options["options"] = {"mode": "invented"} - invalid_options = submit_command( - config, - unknown_options, - socket_client_factory=lambda _config: pytest.fail( - "invalid options must not create a socket client" - ), - ) - fresh_unknown_options = _request(request_id="options-invalid-fresh") - fresh_unknown_options["options"] = {"mode": "invented"} - fresh_invalid_options = submit_command( - config, - fresh_unknown_options, - socket_client_factory=lambda _config: pytest.fail( - "invalid options must not create a socket client" - ), - ) - - assert accepted.status == STATUS_ACCEPTED - assert accepted.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert changed_text.status == STATUS_DUPLICATE_REQUEST - assert changed_text.disposition == DISPOSITION_TERMINAL_REJECTED - assert changed_target.status == STATUS_DUPLICATE_REQUEST - assert changed_target.disposition == DISPOSITION_TERMINAL_REJECTED - assert changed_action.status == STATUS_DUPLICATE_REQUEST - assert changed_action.disposition == DISPOSITION_TERMINAL_REJECTED - assert invalid_options.status == STATUS_INVALID_REQUEST - assert fresh_invalid_options.status == STATUS_INVALID_REQUEST - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, request_id) - assert receipt is not None - assert receipt["state"] == "accepted" - assert receipt["public_worker_id"] == first_worker.id - assert ( - get_command_request(config.db_path, config.host_id, "options-invalid-fresh") - is None - ) - - -def test_submit_command_private_preparation_over_30_second_budget_precedes_reservation_and_sends_once( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path, timeout=31) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - pane_lookup_started = Event() - release_pane_lookup = Event() - first_calls: list[dict[str, Any]] = [] - second_calls: list[dict[str, Any]] = [] - clients: list[_FakeSocketClient] = [] - - class BlockingClient(_FakeSocketClient): - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - if method == "agent.get": - assert timeout == 31 - pane_lookup_started.set() - assert release_pane_lookup.wait(timeout=5) - return super().request(method, params, timeout=timeout) - - def first_factory(config: Config) -> BlockingClient: - client = BlockingClient(first_calls) - clients.append(client) - return client - - def second_factory(config: Config) -> _FakeSocketClient: - client = _FakeSocketClient(second_calls) - clients.append(client) - return client - - with ThreadPoolExecutor(max_workers=2) as executor: - first_future = executor.submit( - submit_command, - config, - _request(request_id="concurrent-1"), - socket_client_factory=first_factory, - ) - assert pane_lookup_started.wait(timeout=5) - assert config.db_path is not None - assert get_command_request(config.db_path, config.host_id, "concurrent-1") is None - second = submit_command( - config, - _request(request_id="concurrent-1"), - socket_client_factory=second_factory, - ) - release_pane_lookup.set() - first = first_future.result(timeout=5) - - assert first.status == STATUS_ACCEPTED - assert second.status == STATUS_ACCEPTED - assert first_calls == [{"method": "agent.get", "params": {"target": "agent-secret"}}] - assert second_calls == _expected_submit_calls(timeout_ms=31_000) - assert [client.close_count for client in clients] == [1, 1] - receipt = get_command_request(config.db_path, config.host_id, "concurrent-1") - assert receipt is not None - assert receipt["state"] == "accepted" - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - "SELECT COUNT(*) FROM command_receipts " - "WHERE host_id = ? AND request_id = ? AND state = 'accepted'", - (config.host_id, "concurrent-1"), - ).fetchone()[0] == 1 - assert sum(call["method"] == "agent.prompt" for call in first_calls + second_calls) == 1 - - -@pytest.mark.parametrize( - ("after_commit", "expected_status", "expected_state"), - [ - (False, STATUS_PENDING, "reserved"), - (True, STATUS_PENDING, "send_started"), - ], -) -def test_submit_command_send_start_exception_recovers_durable_state_and_closes_prepared_clients( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - after_commit: bool, - expected_status: str, - expected_state: str, -) -> None: - config = _config(tmp_path / expected_state) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - real_mark = command_submission.mark_command_send_started - calls: list[dict[str, Any]] = [] - clients: list[_FakeSocketClient] = [] - - def lose_send_start_response(*args: Any, **kwargs: Any) -> Any: - if after_commit: - result = real_mark(*args, **kwargs) - assert result["status"] == "send_started" - raise HerdrSocketTimeoutError("send-start response lost") - - def factory(config: Config) -> _FakeSocketClient: - client = _FakeSocketClient(calls) - clients.append(client) - return client - - monkeypatch.setattr( - command_submission, - "mark_command_send_started", - lose_send_start_response, - ) - first = submit_command( - config, - _request(request_id="send-start-loss"), - socket_client_factory=factory, - ) - - assert first.status == expected_status - assert first.disposition == DISPOSITION_IN_PROGRESS - assert calls == _expected_submit_calls()[:-1] - assert clients[0].close_count == 1 - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, "send-start-loss") - assert receipt is not None - assert receipt["state"] == expected_state - - if after_commit: - replay = submit_command( - config, - _request(request_id="send-start-loss"), - socket_client_factory=lambda _config: pytest.fail( - "send-started replay must not prepare another client" - ), - ) - assert replay.status == STATUS_PENDING - assert replay.disposition == DISPOSITION_IN_PROGRESS - assert len(clients) == 1 - else: - # The reservation is still owned, so both retries are decided by the - # receipt alone: no second client, no second private observation. - replay = submit_command( - config, - _request(request_id="send-start-loss"), - socket_client_factory=factory, - ) - assert replay.status == STATUS_PENDING - assert replay.disposition == DISPOSITION_IN_PROGRESS - conflict = submit_command( - config, - _request(request_id="send-start-loss", text="different"), - socket_client_factory=factory, - ) - assert conflict.status == STATUS_DUPLICATE_REQUEST - assert [client.close_count for client in clients] == [1] - assert calls == _expected_submit_calls()[:-1] - assert not any(call["method"] == "pane.send_input" for call in calls) - - -def test_submit_command_accepted_terminal_replay_delete_fences_prepared_takeover( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - initial_calls: list[dict[str, Any]] = [] - first = submit_command( - config, - _request(request_id="terminal-delete-race"), - socket_client_factory=_factory(initial_calls), - ) - assert first.status == STATUS_ACCEPTED - assert first.result["submission_verdict"] == "submitted" - assert sum(call["method"] == "agent.prompt" for call in initial_calls) == 1 - assert not any(call["method"] == "pane.send_input" for call in initial_calls) - assert config.db_path is not None - - active_worker = Worker(id="w-1", name="Alpha", status="active") - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:02:00+00:00", - workers=[active_worker], - backend_health=[_healthy_backend()], - ), - ) - upsert_worker_bindings(config.db_path, [_binding(active_worker)]) - - real_atomic_replay = command_submission.reserve_terminal_command_replay - deleted = Event() - contender_prepared = Event() - release_contender = Event() - atomic_calls = 0 - - def delete_then_insert_terminal(*args: Any, **kwargs: Any) -> Any: - nonlocal atomic_calls - atomic_calls += 1 - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "DELETE FROM command_receipts WHERE host_id = ? AND request_id = ?", - (config.host_id, "terminal-delete-race"), - ) - conn.commit() - deleted.set() - assert contender_prepared.wait(timeout=5) - try: - return real_atomic_replay(*args, **kwargs) - finally: - release_contender.set() - - monkeypatch.setattr( - command_submission, - "reserve_terminal_command_replay", - delete_then_insert_terminal, - ) - contender_calls: list[dict[str, Any]] = [] - - class PreparedContenderClient(_FakeSocketClient): - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - if method == "agent.get": - contender_prepared.set() - assert release_contender.wait(timeout=5) - return super().request(method, params, timeout=timeout) - - contender_client = PreparedContenderClient(contender_calls) - with ThreadPoolExecutor(max_workers=2) as executor: - replay_future = executor.submit( - submit_command, - config, - _request(request_id="terminal-delete-race"), - socket_client_factory=lambda _config: pytest.fail( - "terminal replay must not prepare another client" - ), - ) - assert deleted.wait(timeout=5) - contender_future = executor.submit( - submit_command, - config, - _request(request_id="terminal-delete-race"), - socket_client_factory=lambda _config: contender_client, - ) - replay = replay_future.result(timeout=5) - contender = contender_future.result(timeout=5) - - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert contender.to_dict() == replay.to_dict() - assert atomic_calls == 1 - assert contender_client.close_count == 1 - assert contender_calls == [ - {"method": "agent.get", "params": {"target": "agent-secret"}} - ] - receipt = get_command_request(config.db_path, config.host_id, "terminal-delete-race") - assert receipt is not None - assert receipt["state"] == "uncertain" - assert receipt["state"] != "reserved" - assert sum(call["method"] == "agent.prompt" for call in initial_calls) == 1 - - -def test_submit_command_timeout_before_send_start_stays_retryable( - tmp_path: Path, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - - class ConnectTimeoutClient: - def connect(self) -> None: - raise HerdrSocketTimeoutError("connect timeout") - - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - raise AssertionError("pane operations must not run") - - def close(self) -> None: - return None - - first = submit_command( - config, - _request(request_id="before-timeout"), - socket_client_factory=lambda _config: ConnectTimeoutClient(), - ) - - # A connect timeout occurs before any request transmission, so no send began. - # It must stay retryable rather than durably reject the unsent command. - assert first.status == STATUS_BACKEND_UNAVAILABLE - assert first.disposition == DISPOSITION_NO_RECEIPT - assert config.db_path is not None - assert get_command_request(config.db_path, config.host_id, "before-timeout") is None - - # A retry under the same request ID re-attempts and, once the socket - # responds, sends exactly once. - recovery_calls: list[dict[str, Any]] = [] - recovered = submit_command( - config, - _request(request_id="before-timeout"), - socket_client_factory=_factory(recovery_calls), - ) - assert recovered.status == STATUS_ACCEPTED - assert [call["method"] for call in recovery_calls].count("agent.prompt") == 1 - receipt = get_command_request(config.db_path, config.host_id, "before-timeout") - assert receipt is not None - assert receipt["state"] == "accepted" - assert receipt["send_started_at"] is not None - - -def test_submit_command_finalization_timeout_after_send_is_uncertain_and_not_retried( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - - def timeout_finish(*args: Any, **kwargs: Any) -> Any: - raise HerdrSocketTimeoutError("receipt finalization timeout") - - monkeypatch.setattr(command_submission, "finish_command_request", timeout_finish) - first = submit_command( - config, - _request(request_id="after-timeout"), - socket_client_factory=_factory(calls), - ) - replay = submit_command( - config, - _request(request_id="after-timeout"), - socket_client_factory=lambda _config: pytest.fail( - "send-started replay must not create a socket client" - ), - ) - - assert first.status == STATUS_PENDING - assert replay.status == STATUS_PENDING - assert first.disposition == DISPOSITION_IN_PROGRESS - assert replay.disposition == DISPOSITION_IN_PROGRESS - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request(config.db_path, config.host_id, "after-timeout") - assert receipt is not None - assert receipt["state"] == "send_started" - maintenance = cleanup_command_request_retention( - config.db_path, - retry_horizon_seconds=60, - retention_seconds=691_200, - retention_count=4096, - host_id=config.host_id, - now="2099-01-01T00:00:00+00:00", - ) - terminal = replay_command_receipt( - config, - _request(request_id="after-timeout"), - ) - assert maintenance["stale_active"] == 1 - assert terminal is not None - assert terminal.status == STATUS_REQUEST_STATE_UNCERTAIN - assert terminal.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == _expected_submit_calls() - - -def test_submit_command_accepted_finalization_response_loss_replays_accepted( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - real_finish = command_submission.finish_command_request - - def finish_then_lose_response(*args: Any, **kwargs: Any) -> Any: - result = real_finish(*args, **kwargs) - assert result["status"] == "accepted" - raise HerdrSocketTimeoutError("accepted response lost") - - monkeypatch.setattr( - command_submission, - "finish_command_request", - finish_then_lose_response, - ) - first = submit_command( - config, - _request(request_id="accepted-response-loss"), - socket_client_factory=_factory(calls), - ) - replay = submit_command( - config, - _request(request_id="accepted-response-loss"), - socket_client_factory=lambda _config: pytest.fail( - "accepted replay must not create a socket client" - ), - ) - - assert first.status == STATUS_ACCEPTED - assert first.disposition == DISPOSITION_TERMINAL_ACCEPTED - assert replay.to_dict() == first.to_dict() - assert calls == _expected_submit_calls() - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - "accepted-response-loss", - ) - assert receipt is not None - assert receipt["state"] == "accepted" - turns = turns_payload_from_store(config.db_path, config.host_id)["turns"] - assert turns == [] - - -@pytest.mark.parametrize( - ("schema_version", "legacy_v1"), - [ - pytest.param(True, True, id="v1-bool-alias"), - pytest.param(1.0, True, id="v1-float-alias"), - pytest.param(2.0, False, id="v2-float-alias"), - ], -) -def test_replay_command_receipt_rejects_non_exact_stored_schema_versions( - tmp_path: Path, - schema_version: Any, - legacy_v1: bool, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_payload = _request(request_id=f"schema-alias-{type(schema_version).__name__}") - accepted = submit_command( - config, - request_payload, - socket_client_factory=_factory(calls), - ) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - request_payload["request_id"], - ) - assert receipt is not None - stored = json.loads(receipt["result_json"]) - if legacy_v1: - stored.pop("disposition") - stored["schema_version"] = schema_version - malformed_json = json.dumps(stored, sort_keys=True, separators=(",", ":")) - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET result_json = ? " - "WHERE host_id = ? AND request_id = ?", - (malformed_json, config.host_id, request_payload["request_id"]), - ) - conn.commit() - rows_before = conn.execute( - "SELECT * FROM command_receipts ORDER BY id" - ).fetchall() - - replay = replay_command_receipt(config, request_payload) - - assert replay is not None - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert replay.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == _expected_submit_calls() - with sqlite3.connect(str(config.db_path)) as conn: - assert ( - conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall() - == rows_before - ) - - -def test_response_loss_replay_uses_only_current_exact_worker_selector( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - space_id="space-1", - ) - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_payload = _request(request_id="response-loss-target") - accepted = submit_command( - config, - request_payload, - socket_client_factory=_factory(calls), - ) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - - def stored_rows() -> tuple[list[tuple[Any, ...]], list[tuple[Any, ...]]]: - with sqlite3.connect(str(config.db_path)) as conn: - return ( - conn.execute("SELECT * FROM command_receipts ORDER BY id").fetchall(), - conn.execute("SELECT * FROM events ORDER BY id").fetchall(), - ) - - rows_before = stored_rows() - exact = replay_command_receipt(config, request_payload) - changed_request = _request(request_id="response-loss-target", worker_id="w-2") - changed = replay_command_receipt(config, changed_request) - - monkeypatch.setattr( - command_submission, - "_current_snapshot", - lambda _config: pytest.fail( - "read-only response-loss reconciliation must not consult current authority" - ), - ) - mutable_request = _request(request_id="response-loss-target") - mutable_request["target"] = {"name": "Alpha"} - mutable = replay_command_receipt(config, mutable_request) - - assert exact is not None - assert exact.to_dict() == accepted.to_dict() - assert changed is not None - assert changed.status == STATUS_DUPLICATE_REQUEST - assert changed.disposition == DISPOSITION_TERMINAL_REJECTED - assert mutable is None - assert calls == _expected_submit_calls() - assert stored_rows() == rows_before - - -@pytest.mark.parametrize("damage", ["malformed_result", "illegal_state"]) -def test_submit_command_illegal_or_malformed_terminal_receipt_fails_closed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - damage: str, -) -> None: - config = _config(tmp_path / damage) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed(config, [worker], [_binding(worker)]) - calls: list[dict[str, Any]] = [] - request_id = f"damaged-{damage}" - accepted = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=_factory(calls), - ) - assert accepted.status == STATUS_ACCEPTED - assert config.db_path is not None - - if damage == "malformed_result": - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - "UPDATE command_receipts SET result_json = '{' " - "WHERE host_id = ? AND request_id = ?", - (config.host_id, request_id), - ) - conn.commit() - else: - real_reserve = command_submission.reserve_terminal_command_replay - - def illegal_reserve(*args: Any, **kwargs: Any) -> dict[str, Any]: - result = real_reserve(*args, **kwargs) - receipt = dict(result["receipt"]) - receipt["state"] = "illegal" - return {**result, "receipt": receipt} - - monkeypatch.setattr( - command_submission, - "reserve_terminal_command_replay", - illegal_reserve, - ) - - replay = submit_command( - config, - _request(request_id=request_id), - socket_client_factory=lambda _config: pytest.fail( - "damaged receipt replay must not create a socket client" - ), - ) - - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert replay.disposition == DISPOSITION_TERMINAL_UNCERTAIN - assert calls == _expected_submit_calls() - - -def _answer_request( - *, - request_id: str = "answer-1", - dry_run: bool = False, - pending_id: str = "pending-public", - pending_fingerprint: str = "revision-public", - choice_id: str = "choice-public", -) -> dict[str, Any]: - return { - "schema_version": 1, - "action": "answer_pending", - "request_id": request_id, - "dry_run": dry_run, - "params": { - "pending_id": pending_id, - "pending_fingerprint": pending_fingerprint, - "choice_id": choice_id, - }, - } - - -@pytest.mark.parametrize("terminal_status", [STATUS_ACCEPTED, STATUS_REJECTED]) -def test_answer_pending_migrated_v11_terminal_replays_without_current_authority( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - terminal_status: str, -) -> None: - config = _config(tmp_path) - assert config.db_path is not None - request_payload = _answer_request(request_id=f"legacy-answer-{terminal_status}") - request = CommandRequest.from_dict(request_payload) - if terminal_status == STATUS_ACCEPTED: - terminal = CommandEnvelope.from_result( - request, - ok=True, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - status=terminal_status, - result={ - "target": {"worker_id": "legacy-worker"}, - "pending": { - "id": "pending-public", - "fingerprint": "revision-public", - }, - "choice": {"choice_id": "choice-public"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - }, - ) - else: - terminal = CommandEnvelope.from_result( - request, - ok=False, - disposition=DISPOSITION_TERMINAL_REJECTED, - status=terminal_status, - error={ - "code": terminal_status, - "message": "legacy pending answer was rejected before send", - }, - ) - raw_fingerprint = request.payload_fingerprint() - expected_terminal = terminal.to_dict() - legacy_terminal = dict(expected_terminal) - legacy_terminal.pop("disposition") - legacy_terminal["schema_version"] = 1 - result_json = json.dumps( - legacy_terminal, - sort_keys=True, - separators=(",", ":"), - ) - request_json = json.dumps( - request.to_dict(), - sort_keys=True, - separators=(",", ":"), - ) - with sqlite3.connect(str(config.db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0) - """, - ( - config.host_id, - request.request_id, - request.action, - raw_fingerprint, - terminal_status, - result_json, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES (?, ?, ?, ?, ?, 0, 0, ?, ?, ?, ?, ?, ?) - """, - ( - config.host_id, - request.request_id, - request.action, - raw_fingerprint, - terminal_status, - request_json, - result_json, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute("PRAGMA user_version = 11") - - init_store(config.db_path) - migrated = get_command_request( - config.db_path, - config.host_id, - request.request_id or "", - ) - assert migrated is not None - assert migrated["canonical_version"] == 0 - assert migrated["canonical_fingerprint"] == raw_fingerprint - assert migrated["public_worker_id"] == "" - assert migrated["state"] == ( - "accepted" if terminal_status == STATUS_ACCEPTED else "rejected" - ) - assert migrated["legacy_collision"] is False - - def command_rows() -> tuple[list[tuple[Any, ...]], list[tuple[Any, ...]]]: - with sqlite3.connect(str(config.db_path)) as conn: - return ( - conn.execute( - "SELECT * FROM command_receipts ORDER BY id" - ).fetchall(), - conn.execute("SELECT * FROM commands ORDER BY id").fetchall(), - ) - - rows_before = command_rows() - - def forbidden(*args: Any, **kwargs: Any) -> Any: - pytest.fail("legacy terminal replay must not consult or mutate current authority") - - monkeypatch.setattr(command_submission, "_validate_pending_choice", forbidden) - monkeypatch.setattr(command_submission, "_answer_pending", forbidden) - monkeypatch.setattr(command_submission, "reserve_command_request", forbidden) - - read_only_replay = replay_command_receipt(config, request_payload) - replay = submit_command( - config, - request_payload, - socket_client_factory=forbidden, - ) - changed_choice = submit_command( - config, - _answer_request( - request_id=request.request_id or "", - choice_id="changed-choice", - ), - socket_client_factory=forbidden, - ) - - assert read_only_replay is not None - assert read_only_replay.to_dict() == expected_terminal - assert replay.to_dict() == expected_terminal - assert replay.disposition == ( - DISPOSITION_TERMINAL_ACCEPTED - if terminal_status == STATUS_ACCEPTED - else DISPOSITION_TERMINAL_REJECTED - ) - assert changed_choice.status == STATUS_DUPLICATE_REQUEST - assert changed_choice.disposition == DISPOSITION_TERMINAL_REJECTED - assert command_rows() == rows_before - - -class _PendingClaim: - def __init__( - self, - status: str, - worker: Worker, - *, - claim_token: str | None, - private_fingerprint: str = "binding-private", - turn_target_value: str = "pane-secret", - picker_ordinal: int = 2, - ) -> None: - self.status = status - self.claim_token = claim_token - self.worker_id = worker.id if status in {"claimed", "validated"} else None - self.worker_fingerprint = worker.fingerprint if status in {"claimed", "validated"} else None - self.binding_private_fingerprint = ( - private_fingerprint if status in {"claimed", "validated"} else None - ) - self.turn_target_value = ( - turn_target_value if status in {"claimed", "validated"} else None - ) - self.picker_ordinal = picker_ordinal if status in {"claimed", "validated"} else None - - -class _PendingSend: - def __init__( - self, - status: str, - worker: Worker, - *, - private_fingerprint: str = "binding-private", - turn_target_value: str = "pane-secret", - picker_ordinal: int = 2, - ) -> None: - self.status = status - self.worker_id = worker.id if status == "started" else None - self.worker_fingerprint = worker.fingerprint if status == "started" else None - self.binding_private_fingerprint = private_fingerprint if status == "started" else None - self.turn_target_value = turn_target_value if status == "started" else None - self.picker_ordinal = picker_ordinal if status == "started" else None - - -def _patch_pending_store_flow( - monkeypatch: pytest.MonkeyPatch, - worker: Worker, - *, - claim_status: str = "claimed", - private_fingerprint: str = "binding-private", - picker_ordinal: int = 2, - turn_target_value: str = "pane-secret", - finish_result: bool = True, -) -> list[tuple[Any, ...]]: - transitions: list[tuple[Any, ...]] = [] - - def claim( - db_path: Path, - host_id: str, - pending_id: str, - pending_fingerprint: str, - choice_id: str, - *, - claim: bool = True, - observed_at: str | None = None, - ) -> _PendingClaim: - transitions.append( - ("claim", claim, pending_id, pending_fingerprint, choice_id, observed_at) - ) - status = claim_status - token: str | None = "claim-private" if status == "claimed" else None - if not claim and status == "claimed": - status = "validated" - return _PendingClaim( - status, - worker, - claim_token=token, - private_fingerprint=private_fingerprint, - turn_target_value=turn_target_value, - picker_ordinal=picker_ordinal, - ) - - def start( - db_path: Path, - host_id: str, - claim_token: str, - *, - observed_at: str | None = None, - ) -> _PendingSend: - transitions.append(("start", claim_token, observed_at)) - return _PendingSend( - "started", - worker, - private_fingerprint=private_fingerprint, - turn_target_value=turn_target_value, - picker_ordinal=picker_ordinal, - ) - - def finish_effect( - *, - host_id: str, - claim_token: str, - accepted: bool, - ) -> Any: - def apply(conn: sqlite3.Connection) -> None: - transitions.append(("finish", claim_token, accepted)) - if not finish_result: - raise RuntimeError("pending finish failed") - - return apply - - def abandon(db_path: Path, host_id: str, claim_token: str) -> bool: - transitions.append(("abandon", claim_token)) - return True - - monkeypatch.setattr(command_submission, "claim_backend_pending_choice", claim) - monkeypatch.setattr(command_submission, "start_backend_pending_choice_send", start) - monkeypatch.setattr( - command_submission, - "backend_pending_choice_terminal_effect", - finish_effect, - ) - monkeypatch.setattr(command_submission, "abandon_backend_pending_choice_claim", abandon) - return transitions - - -def _expected_answer_calls( - *, - pane_id: str = "pane-secret", - ordinal: int = 2, -) -> list[dict[str, Any]]: - return [ - *_expected_private_clear_calls(pane_id), - {"method": "pane.send_input", "params": {"pane_id": pane_id, "text": str(ordinal), "keys": ["Enter"]}}, - ] - - -@pytest.mark.parametrize( - ("payload", "expected_result"), - [ - ( - { - "schema_version": 1, - "action": "send_instruction", - "dry_run": True, - "target": {"name": "Alpha", "space_id": "space-1"}, - "instruction": {"text": "hello"}, - }, - { - "target": {"name": "Alpha", "space_id": "space-1"}, - "instruction": {"text": "hello"}, - }, - ), - ( - _answer_request(request_id="", dry_run=True), - { - "pending": { - "id": "pending-public", - "fingerprint": "revision-public", - }, - "choice": {"choice_id": "choice-public"}, - "delivery_state": "not_submitted", - }, - ), - ], -) -def test_mutation_dry_run_is_pure_without_store_snapshot_or_socket( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - payload: dict[str, Any], - expected_result: dict[str, Any], -) -> None: - config = _config(tmp_path, backend="cli") - calls: list[str] = [] - - def forbidden(*args: Any, **kwargs: Any) -> Any: - calls.append("io") - raise AssertionError("dry-run must not consult mutable command authority") - - monkeypatch.setattr(command_submission, "get_command_request", forbidden) - monkeypatch.setattr(command_submission, "_current_snapshot", forbidden) - monkeypatch.setattr(command_submission, "_validate_pending_choice", forbidden) - monkeypatch.setattr(command_submission, "reserve_command_request", forbidden) - - envelope = submit_command( - config, - payload, - socket_client_factory=forbidden, - ) - - assert envelope.ok is True - assert envelope.status == "dry_run" - assert envelope.result == expected_result - assert calls == [] - assert config.db_path is not None - assert not config.db_path.exists() -def test_answer_pending_reacquired_reservation_worker_drift_finishes_rejected( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - current_worker = Worker(id="w-2", name="Beta", status="active") - _seed(config, [current_worker], [_binding(current_worker)]) - payload = _answer_request(request_id="answer-reacquired") - request = CommandRequest.from_dict(payload) - canonical = build_canonical_mutation(request, public_worker_id="w-1") - pending = command_submission._request_in_progress(request) - assert config.db_path is not None - initial = reserve_command_request( - config.db_path, - host_id=config.host_id, - request_id=request.request_id or "", - action=canonical.action, - canonical_version=canonical.canonical_version, - canonical_fingerprint=canonical.fingerprint, - canonical_request_json=canonical.canonical_json, - public_worker_id=canonical.public_worker_id, - pending_result_json=store_sqlite.envelope_to_receipt_json(pending), - legacy_raw_payload_fingerprint=request.payload_fingerprint(), - owner_lease_seconds=1, - now="2020-01-01T00:00:00+00:00", - ) - assert initial["status"] == "reserved" - transitions = _patch_pending_store_flow(monkeypatch, current_worker) - calls: list[dict[str, Any]] = [] - - drift = submit_command( - config, - payload, - socket_client_factory=_factory(calls), - ) - replay = submit_command( - config, - payload, - socket_client_factory=_factory(calls), - ) - - assert drift.status == STATUS_DUPLICATE_REQUEST - assert replay.to_dict() == drift.to_dict() - assert calls == [] - assert transitions == [ - ( - "claim", - False, - "pending-public", - "revision-public", - "choice-public", - None, - ) - ] - receipt = get_command_request( - config.db_path, - config.host_id, - request.request_id or "", - ) - assert receipt is not None - assert receipt["state"] == "rejected" - assert receipt["status"] == STATUS_DUPLICATE_REQUEST - - -def test_answer_pending_claims_sends_only_ordinal_and_replays_receipt( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - monkeypatch.setattr( - command_submission, - "list_worker_bindings", - lambda *args, **kwargs: pytest.fail( - "answer_pending must use the binding authenticated by the claim API" - ), - ) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _answer_request(), - socket_client_factory=_factory(calls), - ) - assert config.db_path is not None - save_snapshot( - config.db_path, - Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:01:00+00:00", - workers=[worker], - backend_health=[_degraded_backend()], - ), - ) - replay = submit_command( - config, - _answer_request(), - socket_client_factory=_factory(calls), - ) - changed_choice = submit_command( - config, - _answer_request(choice_id="different-choice"), - socket_client_factory=lambda _config: pytest.fail( - "choice collision must not create a socket client" - ), - ) - changed_pending_revision = submit_command( - config, - _answer_request(pending_fingerprint="different-revision"), - socket_client_factory=lambda _config: pytest.fail( - "pending collision must not create a socket client" - ), - ) - - assert first.ok is True - assert first.status == STATUS_ACCEPTED - assert first.result == { - "target": {"worker_id": "w-1"}, - "pending": {"id": "pending-public", "fingerprint": "revision-public"}, - "choice": {"choice_id": "choice-public"}, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - } - assert replay.to_dict() == first.to_dict() - assert changed_choice.status == STATUS_DUPLICATE_REQUEST - assert changed_pending_revision.status == STATUS_DUPLICATE_REQUEST - assert calls == _expected_answer_calls() - assert transitions == [ - ( - "claim", - False, - "pending-public", - "revision-public", - "choice-public", - None, - ), - ( - "claim", - True, - "pending-public", - "revision-public", - "choice-public", - None, - ), - ("start", "claim-private", None), - ("finish", "claim-private", True), - ] - - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, "cmd-host", "answer-1", "answer_pending") - assert receipt is not None - assert receipt["status"] == STATUS_ACCEPTED - turns = turns_payload_from_store(config.db_path, config.host_id)["turns"] - assert not any(turn.get("origin_command_id") == "answer-1" for turn in turns) - - -def test_answer_pending_observed_mode_completes_without_instruction_turn( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path, turn_model="observed") - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - calls: list[dict[str, Any]] = [] - - result = submit_command( - config, - _answer_request(request_id="observed-answer-pending"), - socket_client_factory=_factory(calls), - ) - - assert result.ok is True - assert result.status == STATUS_ACCEPTED - assert calls == _expected_answer_calls() - assert transitions[-2:] == [ - ("start", "claim-private", None), - ("finish", "claim-private", True), - ] - - -@pytest.mark.parametrize( - "claim_status", - ["not_found", "stale", "changed", "unknown_choice", "already_claimed"], -) -def test_answer_pending_changed_disappeared_stale_or_unknown_fails_before_socket( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - claim_status: str, -) -> None: - config = _config(tmp_path / claim_status) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow( - monkeypatch, - worker, - claim_status=claim_status, - ) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _answer_request(request_id=f"answer-{claim_status}"), - socket_client_factory=_factory(calls), - ) - - assert envelope.ok is False - assert envelope.status == STATUS_STALE_TARGET - assert envelope.error == { - "code": STATUS_STALE_TARGET, - "message": "pending interaction changed or is no longer answerable", - "details": {}, - } - assert calls == [] - assert len(transitions) == 1 - assert transitions[0][0] == "claim" - - - - -def test_answer_pending_claim_race_closes_prepared_loser_without_second_send( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - original_claim = command_submission.claim_backend_pending_choice - mutating_claim_count = 0 - - def racing_claim(*args: Any, **kwargs: Any) -> _PendingClaim: - nonlocal mutating_claim_count - if kwargs.get("claim", True): - mutating_claim_count += 1 - if mutating_claim_count == 2: - transitions.append(("claim_race_lost",)) - return _PendingClaim("already_claimed", worker, claim_token=None) - return original_claim(*args, **kwargs) - - monkeypatch.setattr(command_submission, "claim_backend_pending_choice", racing_claim) - calls: list[dict[str, Any]] = [] - nested_result: list[Any] = [] - clients: list[_FakeSocketClient] = [] - - def nested_factory(config: Config) -> _FakeSocketClient: - client = _FakeSocketClient(calls) - clients.append(client) - return client - - class _RacingClient(_FakeSocketClient): - def connect(self) -> "_FakeSocketClient": - nested_result.append( - submit_command( - config, - _answer_request(request_id="answer-race-nested"), - socket_client_factory=nested_factory, - ) - ) - return self - - outer_client = _RacingClient(calls) - clients.append(outer_client) - outer = submit_command( - config, - _answer_request(request_id="answer-race-outer"), - socket_client_factory=lambda _config: outer_client, - ) - - assert outer.status == STATUS_STALE_TARGET - assert len(nested_result) == 1 - assert nested_result[0].status == STATUS_ACCEPTED - assert calls == _expected_answer_calls() - assert [client.close_count for client in clients] == [1, 1] - - -def test_answer_pending_post_send_failure_is_uncertain_and_not_retried( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - calls: list[dict[str, Any]] = [] - - first = submit_command( - config, - _answer_request(request_id="answer-uncertain"), - socket_client_factory=_factory( - calls, - raises=HerdrSocketDisconnectedError("disconnected"), - ), - ) - replay = submit_command( - config, - _answer_request(request_id="answer-uncertain"), - socket_client_factory=_factory(calls), - ) - - assert first.status == STATUS_REQUEST_STATE_UNCERTAIN - assert replay.status == STATUS_REQUEST_STATE_UNCERTAIN - assert transitions[-1] == ("finish", "claim-private", False) - assert calls == [ - *_expected_private_clear_calls(), - {"method": "pane.send_input", "params": {"pane_id": "pane-secret", "text": "2", "keys": ["Enter"]}}, - ] - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, - "cmd-host", - "answer-uncertain", - "answer_pending",) - assert receipt is not None - assert receipt["uncertain"] is True - - -def test_answer_pending_public_surfaces_recursively_exclude_private_route_values( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - private_binding = "raw-binding-sentinel" - private_target = "raw-target-sentinel" - private_pane = "raw-pane-sentinel" - _seed( - config, - [worker], - [ - _binding( - worker, - value=private_target, - private_fingerprint=private_binding, - turn_target_kind="pane_id", - turn_target_value=private_pane, - ) - ], - ) - _patch_pending_store_flow( - monkeypatch, - worker, - private_fingerprint=private_binding, - turn_target_value=private_pane, - picker_ordinal=3, - ) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _answer_request(request_id="answer-private"), - socket_client_factory=_factory(calls, pane_id=private_pane), - ) - - assert envelope.status == STATUS_ACCEPTED - assert calls == _expected_answer_calls( - pane_id=private_pane, - ordinal=3, - ) - assert config.db_path is not None - receipt = _receipt_for_action(config.db_path, - "cmd-host", - "answer-private", - "answer_pending",) - assert receipt is not None - with sqlite3.connect(str(config.db_path)) as conn: - event_payloads = [ - json.loads(row[0]) - for row in conn.execute( - "SELECT payload_json FROM events WHERE aggregate_id = ? ORDER BY id", - ("answer-private",), - ).fetchall() - ] - public_surfaces = [ - envelope.to_dict(), - json.loads(receipt["result_json"]), - *event_payloads, - ] - encoded = json.dumps(public_surfaces) - assert private_binding not in encoded - assert private_target not in encoded - assert private_pane not in encoded - for surface in public_surfaces: - _assert_no_private_json(surface) - - -def test_answer_pending_integrates_with_durable_two_phase_claim( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - binding = _binding( - worker, - private_fingerprint="binding-private", - turn_target_kind="pane_id", - turn_target_value="pane-secret", - ) - _seed(config, [worker], [binding]) - assert config.db_path is not None - assert apply_backend_pending_observation( - config.db_path, - config.host_id, - worker.id, - PendingObservation( - kind="open_prompt", - question="Choose a safe option", - pending_kind="choice", - choices=( - PendingObservedChoice( - choice_id="choice-aaaaaaaaaaaaaaaaaaaaaaaa", - label="First", - picker_ordinal=1, - ), - PendingObservedChoice( - choice_id="choice-bbbbbbbbbbbbbbbbbbbbbbbb", - label="Second", - picker_ordinal=2, - ), - ), - revision_digest="private-revision-digest", - ), - binding_private_fingerprint=binding.private_fingerprint, - observed_turn_target_value=binding.turn_target_value, - ) - pending_before = pending_payload_from_store(config.db_path, config.host_id) - assert len(pending_before["pending_interactions"]) == 1 - interaction = pending_before["pending_interactions"][0] - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _answer_request( - request_id="answer-real-claim", - pending_id=interaction["id"], - pending_fingerprint=interaction["fingerprint"], - choice_id=interaction["choices"][1]["choice_id"], - ), - socket_client_factory=_factory(calls), - ) - - assert envelope.status == STATUS_ACCEPTED - assert envelope.result == { - "target": {"worker_id": worker.id}, - "pending": { - "id": interaction["id"], - "fingerprint": interaction["fingerprint"], - }, - "choice": {"choice_id": interaction["choices"][1]["choice_id"]}, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - } - assert calls == _expected_answer_calls(ordinal=2) - pending_after = pending_payload_from_store(config.db_path, config.host_id) - assert pending_after["pending_interactions"] == [] - - -@pytest.mark.parametrize("start_status", ["changed", "stale", "binding_changed"]) -def test_answer_pending_post_receipt_send_start_cas_change_is_uncertain_without_pane_mutation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - start_status: str, -) -> None: - config = _config(tmp_path / start_status) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - - def changed_start( - db_path: Path, - host_id: str, - claim_token: str, - *, - observed_at: str | None = None, - ) -> _PendingSend: - transitions.append(("start", claim_token, start_status)) - return _PendingSend(start_status, worker) - - monkeypatch.setattr( - command_submission, - "start_backend_pending_choice_send", - changed_start, - ) - calls: list[dict[str, Any]] = [] - - envelope = submit_command( - config, - _answer_request(request_id=f"answer-start-{start_status}"), - socket_client_factory=_factory(calls), - ) - - assert envelope.status == STATUS_REQUEST_STATE_UNCERTAIN - assert envelope.error == { - "code": STATUS_REQUEST_STATE_UNCERTAIN, - "message": "previous request state is uncertain; not retrying mutation", - "details": {}, - } - assert calls == [] - assert transitions[-2:] == [ - ("abandon", "claim-private"), - ("finish", "claim-private", False), - ] - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - f"answer-start-{start_status}", - ) - assert receipt is not None - assert receipt["state"] == "uncertain" - - - -@pytest.mark.parametrize( - ("claim_released", "expected_status", "expected_state"), - [ - (True, STATUS_PENDING, "reserved"), - (False, STATUS_REQUEST_STATE_UNCERTAIN, "uncertain"), - ], -) -def test_answer_pending_send_start_exception_is_retryable_only_after_claim_release( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - claim_released: bool, - expected_status: str, - expected_state: str, -) -> None: - config = _config(tmp_path / expected_state) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="binding-private")], - ) - transitions = _patch_pending_store_flow(monkeypatch, worker) - - if not claim_released: - def fail_abandon(db_path: Path, host_id: str, claim_token: str) -> bool: - transitions.append(("abandon_failed", claim_token)) - return False - - monkeypatch.setattr( - command_submission, - "abandon_backend_pending_choice_claim", - fail_abandon, - ) - - def fail_before_send_start(*args: Any, **kwargs: Any) -> Any: - raise HerdrSocketTimeoutError("send-start unavailable before commit") - - monkeypatch.setattr( - command_submission, - "mark_command_send_started", - fail_before_send_start, - ) - client = _FakeSocketClient([]) - envelope = submit_command( - config, - _answer_request(request_id="answer-mark-failure"), - socket_client_factory=lambda _config: client, - ) - - assert envelope.status == expected_status - assert client.close_count == 1 - assert client.calls == [] - assert transitions[-1] == ( - ("abandon", "claim-private") - if claim_released - else ("abandon_failed", "claim-private") - ) - assert config.db_path is not None - receipt = get_command_request( - config.db_path, - config.host_id, - "answer-mark-failure", - ) - assert receipt is not None - assert receipt["state"] == expected_state - -def test_answer_pending_socket_setup_failure_precedes_claim_and_stays_retryable( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [_binding(worker, private_fingerprint="replacement-private")], - ) - transitions = _patch_pending_store_flow( - monkeypatch, - worker, - private_fingerprint="claimed-private", - ) - - connect_ok = {"value": False} - - def flaky_factory(config: Config) -> Any: - if not connect_ok["value"]: - raise OSError("socket unavailable") - return _FakeSocketClient([]) - - envelope = submit_command( - config, - _answer_request(request_id="answer-setup-failed"), - socket_client_factory=flaky_factory, - ) - - # The socket could not be reached, before any pending choice was claimed and - # before any transmission. That is a safe pre-send transient. - assert envelope.status == STATUS_BACKEND_UNAVAILABLE - assert envelope.disposition == DISPOSITION_NO_RECEIPT - assert envelope.error == { - "code": STATUS_BACKEND_UNAVAILABLE, - "message": "Herdr socket could not be reached", - "details": {}, - } - # Only the read-only validation ran; nothing was claimed or sent. - assert transitions == [ - ( - "claim", - False, - "pending-public", - "revision-public", - "choice-public", - None, - ) - ] - assert config.db_path is not None - assert _receipt_for_action( - config.db_path, - config.host_id, - "answer-setup-failed", - "answer_pending", - ) is None - - # The same request ID answers exactly once after the socket recovers. - connect_ok["value"] = True - recovered = submit_command( - config, - _answer_request(request_id="answer-setup-failed"), - socket_client_factory=flaky_factory, - ) - assert recovered.status == STATUS_ACCEPTED - assert recovered.disposition == DISPOSITION_TERMINAL_ACCEPTED - receipt = _receipt_for_action( - config.db_path, - config.host_id, - "answer-setup-failed", - "answer_pending", - ) - assert receipt is not None - assert receipt["state"] == "accepted" - assert ("finish", "claim-private", True) in transitions - - -@pytest.mark.parametrize("turn_model", sorted(TURN_MODELS)) -def test_submit_under_any_turn_model_never_creates_a_turn_row( - tmp_path: Path, - turn_model: str, -) -> None: - config = _config(tmp_path, turn_model=turn_model) - assert config.db_path is not None - stable_key = "wsk1_" + ("e" * 64) - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={"stable_key": stable_key, "stable_key_version": 1}, - ) - _seed(config, [worker], [_binding(worker)]) - - accepted = submit_command( - config, - _request(request_id=f"{turn_model}-submit", response_schema_version=3), - socket_client_factory=_factory([]), - ) - - assert accepted.status == STATUS_ACCEPTED - assert accepted.schema_version == 3 - assert accepted.result["turn_id"] is None - assert accepted.result["submission_id"] == turn_submission_id( - config.host_id, - f"{turn_model}-submit", - ) - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM turns").fetchone() == (0,) - assert conn.execute( - """ - SELECT state, linked_turn_id FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, f"{turn_model}-submit"), - ).fetchone() == ("submitted", None) - - -def test_observed_turn_identity_and_link_are_order_independent( - tmp_path: Path, -) -> None: - def exercise(order: str) -> tuple[str, dict[str, Any]]: - case_path = tmp_path / order - config = _config( - case_path, - turn_model="observed", - submission_link_window_seconds=30, - ) - assert config.db_path is not None - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={ - "stable_key": "wsk1_" + ("f" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - request = _request( - request_id=f"observed-{order}", - response_schema_version=3, - ) - - if order == "observation-first": - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "shared-source-turn", - "user_text": "hello", - "assistant_final_text": "done", - "complete": True, - "has_open_turn": False, - }, - turn_model="observed", - ) == 1 - accepted = submit_command( - config, - request, - socket_client_factory=_factory([]), - ) - if order == "submission-first": - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "shared-source-turn", - "user_text": "hello", - "assistant_final_text": "done", - "complete": True, - "has_open_turn": False, - }, - turn_model="observed", - ) == 1 - assert accepted.result["turn_id"] is None - with sqlite3.connect(str(config.db_path)) as conn: - expires_at = str( - conn.execute( - """ - SELECT link_expires_at FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, f"observed-{order}"), - ).fetchone()[0] - ) - store_sqlite.sweep_submission_links( - config.db_path, - host_id=config.host_id, - now=(datetime.fromisoformat(expires_at) + timedelta(seconds=1)).isoformat(), - ) - replayed = submit_command( - config, - request, - socket_client_factory=_factory([]), - ) - assert replayed.schema_version == 3 - assert isinstance(replayed.result["turn_id"], str) - with sqlite3.connect(str(config.db_path)) as conn: - rows = conn.execute( - "SELECT turn_id, payload_json FROM turns ORDER BY turn_id" - ).fetchall() - link = conn.execute( - """ - SELECT state, linked_turn_id FROM turn_submissions - WHERE host_id = ? AND request_id = ? - """, - (config.host_id, f"observed-{order}"), - ).fetchone() - assert len(rows) == 1 - turn_id, payload_json = rows[0] - payload = json.loads(payload_json) - assert payload.get("origin_command_id") is None - assert Turn.from_dict(payload).id == turn_id - assert link == ("linked", turn_id) - return str(turn_id), replayed.result - - submission_first = exercise("submission-first") - observation_first = exercise("observation-first") - assert submission_first[0] == observation_first[0] - assert submission_first[1]["turn_id"] == observation_first[1]["turn_id"] - - -def test_observed_identical_submissions_fail_closed_and_unobserved_expires( - tmp_path: Path, -) -> None: - config = _config( - tmp_path, - turn_model="observed", - submission_link_window_seconds=5, - submission_hard_ttl_seconds=30, - ) - assert config.db_path is not None - worker = Worker( - id="w-1", - name="Alpha", - status="active", - meta={ - "stable_key": "wsk1_" + ("1" * 64), - "stable_key_version": 1, - }, - ) - _seed(config, [worker], [_binding(worker)]) - for request_id in ("identical-a", "identical-b"): - assert submit_command( - config, - _request(request_id=request_id, response_schema_version=3), - socket_client_factory=_factory([]), - ).status == STATUS_ACCEPTED - assert merge_turn_content( - config.db_path, - config.host_id, - worker.id, - { - "source_turn_id": "identical-source", - "user_text": "hello", - "assistant_final_text": "one observation", - "complete": True, - "has_open_turn": False, - }, - turn_model="observed", - ) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - latest_expiry = max( - datetime.fromisoformat(str(row[0])) - for row in conn.execute( - "SELECT link_expires_at FROM turn_submissions" - ).fetchall() - ) - store_sqlite.sweep_submission_links( - config.db_path, - host_id=config.host_id, - now=(latest_expiry + timedelta(seconds=1)).isoformat(), - ) - with sqlite3.connect(str(config.db_path)) as conn: - assert conn.execute( - "SELECT state, linked_turn_id FROM turn_submissions ORDER BY request_id" - ).fetchall() == [("ambiguous", None), ("ambiguous", None)] - - expiry_config = _config( - tmp_path / "expiry", - turn_model="observed", - submission_link_window_seconds=1, - submission_hard_ttl_seconds=1, - ) - assert expiry_config.db_path is not None - _seed(expiry_config, [worker], [_binding(worker)]) - assert submit_command( - expiry_config, - _request(request_id="never-observed", response_schema_version=3), - socket_client_factory=_factory([]), - ).status == STATUS_ACCEPTED - with sqlite3.connect(str(expiry_config.db_path)) as conn: - hard_expiry = datetime.fromisoformat( - str( - conn.execute( - "SELECT hard_expires_at FROM turn_submissions" - ).fetchone()[0] - ) - ) - store_sqlite.sweep_submission_links( - expiry_config.db_path, - host_id=expiry_config.host_id, - now=(hard_expiry + timedelta(seconds=1)).isoformat(), - ) - with sqlite3.connect(str(expiry_config.db_path)) as conn: - assert conn.execute("SELECT state FROM turn_submissions").fetchone() == ( - "expired", - ) - assert conn.execute("SELECT COUNT(*) FROM turns").fetchone() == (0,) - assert conn.execute( - "SELECT COUNT(*) FROM turn_change_journal" - ).fetchone() == (0,) - - -def test_terminal_id_binding_falls_back_to_agent_list_when_agent_get_refuses(monkeypatch) -> None: - # Herdr 0.7.5 regression: agent.get no longer resolves terminal-id targets; - # agent.list still publishes terminal_id -> pane_id. A definite error from - # agent.get must fall back to the listing instead of terminalizing. - from tendwire import command_submission as cs - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - calls = [] - - def fake_socket_request(client, method, params, *, timeout): - calls.append(method) - if method == "agent.get": - raise HerdrErrorResponse({"code": "agent_not_found", "message": "agent target term_new not found"}, "req-1") - if method == "agent.list": - return { - "agents": [ - {"terminal_id": "term_other", "pane_id": "w1:p1"}, - {"terminal_id": "term_new", "pane_id": "w1:p9"}, - ] - } - raise AssertionError(f"unexpected method {method}") - - monkeypatch.setattr(cs, "_socket_request", fake_socket_request) - - class _Binding: - target_kind = "terminal_id" - target_value = "term_new" - - pane_id = cs._private_pane_id_for_binding(object(), _Binding(), timeout=5.0) - assert pane_id == "w1:p9" - assert calls == ["agent.get", "agent.list"] - - # A non-terminal binding kind must NOT consult the listing: the original - # error propagates. - class _SessionBinding: - target_kind = "agent_session" - target_value = "sess-1" - - calls.clear() - try: - cs._private_pane_id_for_binding(object(), _SessionBinding(), timeout=5.0) - raise AssertionError("expected HerdrErrorResponse to propagate") - except HerdrErrorResponse: - pass - assert calls == ["agent.get"] - - -def test_terminal_id_agent_list_fallback_rejects_distinct_conflicting_matches( - monkeypatch, -) -> None: - from tendwire import command_submission as cs - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - original_error = HerdrErrorResponse( - {"code": "agent_not_found", "message": "agent target term_dup not found"}, - "req-1", - ) - - def fake_socket_request(client, method, params, *, timeout): - if method == "agent.get": - raise original_error - if method == "agent.list": - return { - "agents": [ - {"terminal_id": "term_dup", "pane_id": "w1:p1"}, - {"terminal_id": "term_dup", "pane_id": "w1:p2"}, - ] - } - raise AssertionError(f"unexpected method {method}") - - monkeypatch.setattr(cs, "_socket_request", fake_socket_request) - - class _Binding: - target_kind = "terminal_id" - target_value = "term_dup" - - with pytest.raises(ValueError, match="ambiguous agent.list terminal_id match"): - cs._private_pane_id_for_binding(object(), _Binding(), timeout=5.0) - - -def test_terminal_id_agent_list_zero_matches_reraise_original_error( - monkeypatch, -) -> None: - from tendwire import command_submission as cs - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - original_error = HerdrErrorResponse( - {"code": "agent_not_found", "message": "agent target term_gone not found"}, - "req-1", - ) - - def fake_socket_request(client, method, params, *, timeout): - if method == "agent.get": - raise original_error - if method == "agent.list": - return { - "agents": [ - {"terminal_id": "term_other", "pane_id": "w1:p9"}, - ] - } - raise AssertionError(f"unexpected method {method}") - - monkeypatch.setattr(cs, "_socket_request", fake_socket_request) - - class _Binding: - target_kind = "terminal_id" - target_value = "term_gone" - - with pytest.raises(HerdrErrorResponse) as excinfo: - cs._private_pane_id_for_binding(object(), _Binding(), timeout=5.0) - assert excinfo.value is original_error - - -def test_terminal_id_agent_list_identical_duplicates_converge_and_send_succeeds( - tmp_path: Path, -) -> None: - from tendwire.backends.herdr_protocol import HerdrErrorResponse - - config = _config(tmp_path) - worker = Worker(id="w-1", name="Alpha", status="active") - _seed( - config, - [worker], - [ - _binding( - worker, - target_kind="terminal_id", - value="term-dup", - ) - ], - ) - calls: list[dict[str, Any]] = [] - - class DuplicateListingClient(_FakeSocketClient): - def request( - self, - method: str, - params: dict[str, Any], - *, - timeout: float | None = None, - ) -> dict[str, Any]: - self.calls.append({"method": method, "params": dict(params)}) - if method == "agent.get": - raise HerdrErrorResponse( - {"code": "agent_not_found", "message": "target not found"}, - "req-1", - ) - if method == "agent.list": - return { - "agents": [ - {"terminal_id": "term-dup", "pane_id": "w1:p1"}, - {"terminal_id": "term-dup", "pane_id": "w1:p1"}, - ] - } - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": _REALISTIC_VISIBLE_PANE}, - } - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": "w1:p1"}, - "delivery": "submitted", - } - return {"accepted": True} - - result = submit_command( - config, - _request(request_id="identical-terminal-rows"), - socket_client_factory=lambda _config: DuplicateListingClient(calls), - ) - - assert result.status == STATUS_ACCEPTED - assert calls == [ - {"method": "agent.get", "params": {"target": "term-dup"}}, - {"method": "agent.list", "params": {}}, - { - "method": "agent.prompt", - "params": { - "target": "term-dup", - "text": "hello", - "wait": {"until": ["working"], "timeout_ms": 5000}, - }, - }, - ] - - -@pytest.mark.parametrize( - "listing", - [ - None, - {}, - {"agents": {}}, - {"agents": ["not-an-agent"]}, - {"agents": [{"terminal_id": "term-shared"}]}, - {"agents": [{"terminal_id": "", "pane_id": "w1:p1"}]}, - {"agents": [{"terminal_id": "term-shared", "pane_id": ""}]}, - {"agents": [{"terminal_id": "term-shared", "pane_id": 7}]}, - ], -) -def test_terminal_id_agent_list_fallback_rejects_malformed_listing(listing) -> None: - from tendwire import command_submission as cs - - with pytest.raises(ValueError, match="invalid agent.list"): - cs._pane_id_from_terminal_listing(listing, "term-shared") diff --git a/tests/test_commands.py b/tests/test_commands.py index 7ce2299..d81ea1b 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -193,7 +193,6 @@ def test_allowed_actions_frozen() -> None: "read_snapshot", "resolve_target", "send_instruction", - "answer_pending", "answer_decision", } @@ -405,52 +404,9 @@ def test_canonical_send_instruction_fingerprint_changes_for_semantics( assert changed.fingerprint != baseline.fingerprint -def test_build_canonical_answer_pending_has_hard_coded_v1_identity() -> None: - request = _answer_pending_request(request_id="not-canonical") - mutation = build_canonical_mutation(request, public_worker_id="worker-public-7") - - assert mutation.canonical_version == 1 - assert mutation.action == "answer_pending" - assert mutation.public_worker_id == "worker-public-7" - assert mutation.canonical_json == ( - '{"action":"answer_pending","canonical_version":1,"options":{},' - '"pending":{"choice_id":"choice-public",' - '"pending_fingerprint":"pending-revision","pending_id":"pending-public"},' - '"target":{"worker_id":"worker-public-7"}}' - ) - assert mutation.fingerprint == "1a88307fbc8afd0a1205eaca" -@pytest.mark.parametrize( - ("field", "value"), - [ - ("pending_id", "pending-public-changed"), - ("pending_fingerprint", "pending-revision-changed"), - ("choice_id", " choice-public "), - ], -) -def test_canonical_answer_pending_fingerprint_changes_for_exact_semantics( - field: str, - value: str, -) -> None: - baseline = build_canonical_mutation( - _answer_pending_request(request_id="baseline"), - public_worker_id="worker-public-7", - ) - params = { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - } - params[field] = value - changed = build_canonical_mutation( - _answer_pending_request(request_id="changed", params=params), - public_worker_id="worker-public-7", - ) - - assert changed.fingerprint != baseline.fingerprint - def test_command_envelope_shape_matches_contract() -> None: request = CommandRequest(action="noop") @@ -646,7 +602,7 @@ def test_command_envelope_rejects_inconsistent_receipt_tuples( @pytest.mark.parametrize( - "action", ["send_instruction", "answer_pending", "answer_decision"] + "action", ["send_instruction", "answer_decision"] ) @pytest.mark.parametrize("request_id", [None, "", "not canonical"]) def test_command_envelope_live_mutations_require_canonical_request_ids( @@ -657,23 +613,15 @@ def test_command_envelope_live_mutations_require_canonical_request_ids( action=action, request_id=request_id, dry_run=False, - target={"worker_id": "w-1"} if action != "answer_pending" else None, + target={"worker_id": "w-1"}, instruction={"text": "hello"} if action == "send_instruction" else None, params=( { - "pending_id": "pending-1", - "pending_fingerprint": "revision-1", - "choice_id": "choice-1", + "decision_ref": "decision-1", + "selection": {"option_refs": ["1"]}, } - if action == "answer_pending" - else ( - { - "decision_ref": "decision-1", - "selection": {"option_refs": ["1"]}, - } - if action == "answer_decision" - else None - ) + if action == "answer_decision" + else None ), ) @@ -802,7 +750,7 @@ def test_answer_in_progress_allows_only_retryable_dispositions() -> None: @pytest.mark.parametrize( - "action", ["send_instruction", "answer_pending", "answer_decision"] + "action", ["send_instruction", "answer_decision"] ) @pytest.mark.parametrize("ok", [False, True]) @pytest.mark.parametrize("status", sorted(VALID_STATUSES)) @@ -833,7 +781,7 @@ def test_command_envelope_from_dict_enforces_terminal_rejected_matrix( @pytest.mark.parametrize( - "action", ["send_instruction", "answer_pending", "answer_decision"] + "action", ["send_instruction", "answer_decision"] ) @pytest.mark.parametrize("dry_run", [False, True], ids=["live", "dry-run"]) @pytest.mark.parametrize("ok", [False, True]) @@ -1108,7 +1056,12 @@ def test_validate_rejects_a_fingerprint_only_target(action: str) -> None: assert error is not None assert error["code"] == STATUS_INVALID_REQUEST assert "worker_fingerprint" in error["message"] - assert error["details"]["allowed"] == ["name", "space_id", "worker_id"] + assert error["details"]["allowed"] == [ + "name", + "space_id", + "stable_key", + "worker_id", + ] @pytest.mark.parametrize( @@ -1134,6 +1087,117 @@ def test_validate_accepts_a_fingerprint_beside_a_stable_selector( assert validate_request(request) is None +def test_stable_worker_key_is_a_strict_unique_target_selector() -> None: + stable_key = "wsk1_" + ("a" * 64) + workers = [ + Worker( + id="old-public-id", + name="Coda", + status="idle", + meta={"stable_key": stable_key, "stable_key_version": 1}, + ), + Worker( + id="other-worker", + name="Coda", + status="idle", + meta={"stable_key": "wsk1_" + ("b" * 64), "stable_key_version": 1}, + ), + ] + + resolved, candidates, status = resolve_target( + {"stable_key": stable_key, "stable_key_version": 1}, + workers, + ) + + assert status == STATUS_RESOLVED + assert resolved is not None and resolved["worker_id"] == "old-public-id" + assert [candidate["worker_id"] for candidate in candidates] == ["old-public-id"] + + +def test_stable_worker_key_selector_fails_closed_on_duplicate_live_owner() -> None: + stable_key = "wsk1_" + ("c" * 64) + workers = [ + Worker( + id=f"worker-{index}", + name=f"Coda {index}", + status="idle", + meta={"stable_key": stable_key, "stable_key_version": 1}, + ) + for index in (1, 2) + ] + resolved, candidates, status = resolve_target( + {"stable_key": stable_key, "stable_key_version": 1}, workers + ) + assert resolved is None + assert status == STATUS_AMBIGUOUS_TARGET + assert {item["worker_id"] for item in candidates} == {"worker-1", "worker-2"} + + +def test_stable_worker_key_and_worker_id_must_select_the_same_live_worker() -> None: + stable_key = "wsk1_" + ("d" * 64) + workers = [ + Worker( + id="worker-a", + name="A", + status="idle", + meta={"stable_key": stable_key, "stable_key_version": 1}, + ), + Worker(id="worker-b", name="B", status="idle"), + ] + resolved, candidates, status = resolve_target( + { + "worker_id": "worker-b", + "stable_key": stable_key, + "stable_key_version": 1, + }, + workers, + ) + assert (resolved, candidates, status) == (None, [], STATUS_NOT_FOUND) + + +def test_stable_worker_key_selector_never_trusts_non_meta_observation_fields() -> None: + stable_key = "wsk1_" + ("e" * 64) + worker = Worker(id=stable_key, name=stable_key, status="idle", meta={}) + resolved, candidates, status = resolve_target( + {"stable_key": stable_key, "stable_key_version": 1}, [worker] + ) + assert (resolved, candidates, status) == (None, [], STATUS_NOT_FOUND) + + +@pytest.mark.parametrize( + "target", + [ + {"stable_key": "wsk1_" + ("a" * 64)}, + {"stable_key_version": 1}, + {"stable_key": "wsk1_short", "stable_key_version": 1}, + {"stable_key": "wsk1_" + ("a" * 64), "stable_key_version": 2}, + {"stable_key": "wsk1_" + ("a" * 64), "stable_key_version": True}, + ], +) +def test_stable_worker_key_target_rejects_partial_or_unsupported_identity( + target: dict[str, Any], +) -> None: + request = CommandRequest(action="resolve_target", target=target) + error = validate_request(request) + assert error is not None + assert error["code"] == STATUS_INVALID_REQUEST + + +def test_selector_proof_fences_stable_worker_key_version() -> None: + def proof(key: str) -> str: + return build_selector_proof( + CommandRequest( + action="send_instruction", + request_id="stable-key-proof", + dry_run=False, + target={"stable_key": key, "stable_key_version": 1}, + instruction={"text": "hello"}, + ) + ) + + assert proof("wsk1_" + ("a" * 64)) != proof("wsk1_" + ("b" * 64)) + + @pytest.mark.parametrize( "stable", [ @@ -1233,7 +1297,6 @@ def test_mutation_request_id_accepts_exact_ascii_tokens_and_roundtrips( target={"worker_id": "w-1"}, instruction={"text": "ok"}, ), - _answer_pending_request(request_id=request_id), ] assert is_valid_request_id(request_id) @@ -1301,7 +1364,6 @@ def test_mutation_request_id_rejects_everything_outside_exact_ascii_grammar( target={"worker_id": "w-1"}, instruction={"text": "ok"}, ), - _answer_pending_request(request_id=request_id), ] assert not is_valid_request_id(request_id) @@ -1321,130 +1383,14 @@ def test_mutation_request_id_rejects_everything_outside_exact_ascii_grammar( assert parsed_error["code"] == STATUS_INVALID_REQUEST -def _answer_pending_request( - *, - request_id: str | None = "answer-1", - dry_run: bool = False, - params: Any = None, -) -> CommandRequest: - return CommandRequest( - action="answer_pending", - request_id=request_id, - dry_run=dry_run, - params=params - if params is not None - else { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - }, - ) - -def test_parse_answer_pending_accepts_exact_opaque_params() -> None: - payload = { - "schema_version": 1, - "action": "answer_pending", - "request_id": "answer-1", - "dry_run": False, - "params": { - "pending_id": " opaque pending ", - "pending_fingerprint": " opaque revision ", - "choice_id": " opaque choice ", - }, - } - - request, parse_error = parse_command_request(json.dumps(payload)) - - assert parse_error is None - assert request is not None - assert validate_request(request) is None - assert request.params == payload["params"] - - -@pytest.mark.parametrize( - "changes,field", - [ - ({"target": {"worker_id": "w-1"}}, "target"), - ({"instruction": {"text": "private"}}, "instruction"), - ({"params": None}, "params"), - ({"params": {}}, "params"), - ( - { - "params": { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - "extra": "no", - } - }, - "params", - ), - ( - { - "params": { - "pending_id": "", - "pending_fingerprint": "pending-revision", - "choice_id": "choice-public", - } - }, - "params.pending_id", - ), - ( - { - "params": { - "pending_id": "pending-public", - "pending_fingerprint": " \t", - "choice_id": "choice-public", - } - }, - "params.pending_fingerprint", - ), - ( - { - "params": { - "pending_id": "pending-public", - "pending_fingerprint": "pending-revision", - "choice_id": 1, - } - }, - "params.choice_id", - ), - ], -) -def test_validate_answer_pending_rejects_non_exact_shape( - changes: dict[str, Any], - field: str, -) -> None: - request = _answer_pending_request() - data = request.to_dict() - data.update(changes) - error = validate_request(CommandRequest.from_dict(data)) - assert error is not None - assert error["code"] == STATUS_INVALID_REQUEST - assert field in str(error) -@pytest.mark.parametrize( - "request_id", - [None, "", " \t", " leading", "trailing ", "\twrapped\t"], -) -def test_validate_answer_pending_non_dry_run_requires_canonical_request_id( - request_id: str | None, -) -> None: - error = validate_request(_answer_pending_request(request_id=request_id)) - assert error is not None - assert error["code"] == STATUS_INVALID_REQUEST - assert "request_id" in error["message"] -def test_validate_answer_pending_dry_run_does_not_require_request_id() -> None: - assert validate_request( - _answer_pending_request(request_id=None, dry_run=True) - ) is None @pytest.mark.parametrize( diff --git a/tests/test_config.py b/tests/test_config.py index bcf414d..d6ce246 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -9,16 +9,18 @@ import pytest from tendwire.config import ( + DEFAULT_ACP_MAX_FRAME_BYTES, + DEFAULT_ACP_CONSOLE_INPUT_POLICY, + DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS, + DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS, + DEFAULT_ACP_THOUGHT_POLICY, DEFAULT_COMMAND_RECEIPT_RETENTION_COUNT, DEFAULT_COMMAND_RECEIPT_RETENTION_SECONDS, DEFAULT_COMMAND_RETRY_HORIZON_SECONDS, DEFAULT_SUBMISSION_HARD_TTL_SECONDS, DEFAULT_SUBMISSION_LINK_WINDOW_SECONDS, - DEFAULT_TURN_MODEL, MAX_COMMAND_RETRY_HORIZON_SECONDS, MIN_COMMAND_RECEIPT_RETENTION_SECONDS, - DEFAULT_TURN_REFRESH_INTERVAL_SECONDS, - DEFAULT_TURN_REFRESH_WORKERS, MAX_MAINTENANCE_CADENCE_SECONDS, MAX_RETENTION_DAYS, MAX_SQLITE_INTEGER, @@ -27,24 +29,94 @@ ) -def test_turn_model_defaults_to_observed_and_accepts_compatibility_aliases( +def test_acp_defaults_are_required_runtime_settings_with_thoughts_disabled( monkeypatch, - caplog, ) -> None: - monkeypatch.delenv("TENDWIRE_TURN_MODEL", raising=False) - assert DEFAULT_TURN_MODEL == "observed" - assert load_config().turn_model == "observed" + for name in ( + "TENDWIRE_AGENT_EVENT_SOURCE", + "TENDWIRE_ACP_THOUGHT_POLICY", + "TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS", + "TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS", + "TENDWIRE_ACP_MAX_FRAME_BYTES", + "TENDWIRE_ACP_CONSOLE_INPUT_POLICY", + ): + monkeypatch.delenv(name, raising=False) - monkeypatch.setenv("TENDWIRE_TURN_MODEL", "shadow") - assert load_config().turn_model == "shadow" - assert load_config(turn_model="dual").turn_model == "dual" - assert "behaves as observed" in caplog.text + config = load_config() + + assert not hasattr(config, "agent_event_source") + assert config.acp_thought_policy == DEFAULT_ACP_THOUGHT_POLICY == "disabled" + assert config.acp_request_timeout_seconds == DEFAULT_ACP_REQUEST_TIMEOUT_SECONDS == 30.0 + assert config.acp_shutdown_timeout_seconds == DEFAULT_ACP_SHUTDOWN_TIMEOUT_SECONDS == 5.0 + assert config.acp_max_frame_bytes == DEFAULT_ACP_MAX_FRAME_BYTES == 8 * 1024 * 1024 + assert config.acp_console_input_policy == DEFAULT_ACP_CONSOLE_INPUT_POLICY == "preserve" + + +def test_acp_configuration_uses_explicit_before_environment(monkeypatch) -> None: + monkeypatch.setenv("TENDWIRE_AGENT_EVENT_SOURCE", "acp_shadow") + monkeypatch.setenv("TENDWIRE_ACP_THOUGHT_POLICY", "private_all") + monkeypatch.setenv("TENDWIRE_ACP_REQUEST_TIMEOUT_SECONDS", "11") + monkeypatch.setenv("TENDWIRE_ACP_SHUTDOWN_TIMEOUT_SECONDS", "3") + monkeypatch.setenv("TENDWIRE_ACP_MAX_FRAME_BYTES", "4096") + monkeypatch.setenv("TENDWIRE_ACP_CONSOLE_INPUT_POLICY", "live_only") + + environment = load_config() + explicit = load_config( + acp_thought_policy="disabled", + acp_request_timeout_seconds="7.5", + acp_shutdown_timeout_seconds="2.5", + acp_max_frame_bytes="8192", + acp_console_input_policy="preserve", + ) + + assert not hasattr(environment, "agent_event_source") + assert environment.acp_thought_policy == "private_all" + assert environment.acp_request_timeout_seconds == 11.0 + assert environment.acp_shutdown_timeout_seconds == 3.0 + assert environment.acp_max_frame_bytes == 4096 + assert environment.acp_console_input_policy == "live_only" + assert explicit.acp_thought_policy == "disabled" + assert explicit.acp_request_timeout_seconds == 7.5 + assert explicit.acp_shutdown_timeout_seconds == 2.5 + assert explicit.acp_max_frame_bytes == 8192 + assert explicit.acp_console_input_policy == "preserve" + + +@pytest.mark.parametrize("value", ["", "drop", "future"]) +def test_acp_console_input_policy_rejects_unknown_values(value: str) -> None: + with pytest.raises(ValueError, match="acp_console_input_policy must be one of"): + Config(acp_console_input_policy=value) + + +@pytest.mark.parametrize("value", ["", "public", "summary", "future"]) +def test_acp_thought_policy_rejects_unknown_values(value: str) -> None: + with pytest.raises(ValueError, match="acp_thought_policy must be one of"): + Config(acp_thought_policy=value) + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("acp_request_timeout_seconds", 0), + ("acp_request_timeout_seconds", float("inf")), + ("acp_shutdown_timeout_seconds", -1), + ("acp_shutdown_timeout_seconds", "invalid"), + ("acp_max_frame_bytes", 0), + ("acp_max_frame_bytes", True), + ("acp_max_frame_bytes", 64 * 1024 * 1024 + 1), + ], +) +def test_acp_bounds_reject_invalid_values(field: str, value: object) -> None: + with pytest.raises(ValueError): + Config(**{field: value}) -@pytest.mark.parametrize("value", ["", "future", "legacy,dual"]) -def test_turn_model_rejects_unknown_values(value: str) -> None: - with pytest.raises(ValueError, match="turn_model must be one of"): - Config(turn_model=value) +def test_runtime_turn_model_modes_are_removed(monkeypatch) -> None: + monkeypatch.delenv("TENDWIRE_TURN_MODEL", raising=False) + assert not hasattr(load_config(), "turn_model") + + monkeypatch.setenv("TENDWIRE_TURN_MODEL", "shadow") + assert not hasattr(load_config(), "turn_model") def test_submission_windows_have_defaults_and_explicit_precedence(monkeypatch) -> None: @@ -101,10 +173,8 @@ def test_submission_hard_ttl_must_cover_link_window() -> None: def test_pr16_runtime_knobs_have_documented_defaults(monkeypatch) -> None: for name in ( - "TENDWIRE_EVENT_DEBOUNCE_SECONDS", "TENDWIRE_RECONCILE_INTERVAL_SECONDS", "TENDWIRE_EVENT_RETENTION_DAYS", - "TENDWIRE_OUTPUT_EXCERPT_CHARS", "TENDWIRE_MAX_WORKERS", "TENDWIRE_MAX_OUTBOX_ATTEMPTS", "TENDWIRE_CONNECTOR_CLAIM_TTL_SECONDS", @@ -118,10 +188,8 @@ def test_pr16_runtime_knobs_have_documented_defaults(monkeypatch) -> None: config = load_config() - assert config.event_debounce_seconds == 0.05 - assert config.reconcile_interval_seconds == 300.0 + assert config.reconcile_interval_seconds == 15.0 assert config.event_retention_days == 7 - assert config.output_excerpt_chars == 200 assert config.max_workers == 512 assert config.max_outbox_attempts == 10 assert config.connector_claim_ttl_seconds == 60 @@ -133,10 +201,8 @@ def test_pr16_runtime_knobs_have_documented_defaults(monkeypatch) -> None: def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_EVENT_DEBOUNCE_SECONDS", "0.25") - monkeypatch.setenv("TENDWIRE_RECONCILE_INTERVAL_SECONDS", "0") + monkeypatch.setenv("TENDWIRE_RECONCILE_INTERVAL_SECONDS", "2") monkeypatch.setenv("TENDWIRE_EVENT_RETENTION_DAYS", "14") - monkeypatch.setenv("TENDWIRE_OUTPUT_EXCERPT_CHARS", "123") monkeypatch.setenv("TENDWIRE_MAX_WORKERS", "64") monkeypatch.setenv("TENDWIRE_MAX_OUTBOX_ATTEMPTS", "3") monkeypatch.setenv("TENDWIRE_CONNECTOR_CLAIM_TTL_SECONDS", "45") @@ -148,10 +214,8 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: env_config = load_config() explicit = load_config( - event_debounce_seconds="0.1", reconcile_interval_seconds="5", event_retention_days="2", - output_excerpt_chars="50", max_workers="9", max_outbox_attempts="4", connector_claim_ttl_seconds="15", @@ -161,10 +225,8 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: command_receipt_retention_seconds="691200", command_receipt_retention_count="12", ) - assert env_config.event_debounce_seconds == 0.25 - assert env_config.reconcile_interval_seconds == 0 + assert env_config.reconcile_interval_seconds == 2 assert env_config.event_retention_days == 14 - assert env_config.output_excerpt_chars == 123 assert env_config.max_workers == 64 assert env_config.max_outbox_attempts == 3 assert env_config.connector_claim_ttl_seconds == 45 @@ -173,10 +235,8 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: assert env_config.command_retry_horizon_seconds == 120 assert env_config.command_receipt_retention_seconds == 691_200 assert env_config.command_receipt_retention_count == 99 - assert explicit.event_debounce_seconds == 0.1 assert explicit.reconcile_interval_seconds == 5 assert explicit.event_retention_days == 2 - assert explicit.output_excerpt_chars == 50 assert explicit.max_workers == 9 assert explicit.max_outbox_attempts == 4 assert explicit.connector_claim_ttl_seconds == 15 @@ -190,10 +250,17 @@ def test_pr16_runtime_knobs_accept_constructor_and_env(monkeypatch) -> None: @pytest.mark.parametrize( ("field", "value", "message"), [ - ("event_debounce_seconds", -0.1, "event_debounce_seconds must be non-negative"), - ("reconcile_interval_seconds", -1, "reconcile_interval_seconds must be non-negative"), + ( + "reconcile_interval_seconds", + 0, + "reconcile_interval_seconds must be a finite positive number", + ), + ( + "reconcile_interval_seconds", + -1, + "reconcile_interval_seconds must be a finite positive number", + ), ("event_retention_days", 0, "event_retention_days must be >= 1"), - ("output_excerpt_chars", 0, "output_excerpt_chars must be >= 1"), ("max_workers", 0, "max_workers must be >= 1"), ("max_outbox_attempts", 0, "max_outbox_attempts must be >= 1"), ("connector_claim_ttl_seconds", 0, "connector_claim_ttl_seconds must be >= 1"), @@ -210,6 +277,15 @@ def test_pr16_runtime_knobs_reject_invalid_values(field: str, value: object, mes Config(**{field: value}) +def test_reconcile_interval_rejects_disabled_environment(monkeypatch) -> None: + monkeypatch.setenv("TENDWIRE_RECONCILE_INTERVAL_SECONDS", "0") + with pytest.raises( + ValueError, + match="reconcile_interval_seconds must be a finite positive number", + ): + load_config() + + ACKNOWLEDGED_FINAL_RETENTION_ENV_NAMES = ( "TENDWIRE_ACKNOWLEDGED_FINAL_RETENTION_DAYS", "TENDWIRE_ACKNOWLEDGED_FINAL_RETENTION_COUNT", @@ -399,83 +475,6 @@ def test_command_receipt_retention_must_strictly_exceed_retry_horizon( ) -TURN_REFRESH_ENV_NAMES = ( - "TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS", - "TENDWIRE_TURN_REFRESH_WORKERS", -) - - -def test_turn_refresh_knobs_have_documented_defaults(monkeypatch) -> None: - for name in TURN_REFRESH_ENV_NAMES: - monkeypatch.delenv(name, raising=False) - - config = load_config() - - assert DEFAULT_TURN_REFRESH_INTERVAL_SECONDS == 2.0 - assert DEFAULT_TURN_REFRESH_WORKERS == 4 - assert config.turn_refresh_interval_seconds == 2.0 - assert config.turn_refresh_workers == 4 - - -def test_turn_refresh_knobs_use_explicit_before_environment(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_TURN_REFRESH_INTERVAL_SECONDS", "3.5") - monkeypatch.setenv("TENDWIRE_TURN_REFRESH_WORKERS", "8") - - env_config = load_config(max_workers=16) - explicit = load_config( - max_workers=16, - turn_refresh_interval_seconds="0.25", - turn_refresh_workers="6", - ) - - assert env_config.turn_refresh_interval_seconds == 3.5 - assert env_config.turn_refresh_workers == 8 - assert explicit.turn_refresh_interval_seconds == 0.25 - assert explicit.turn_refresh_workers == 6 - - -@pytest.mark.parametrize("value", [0, -0.01, "nan", "inf", "-inf"]) -def test_turn_refresh_interval_rejects_nonpositive_or_nonfinite(value: object) -> None: - with pytest.raises( - ValueError, - match="turn_refresh_interval_seconds must be a finite positive number", - ): - Config(turn_refresh_interval_seconds=value) - - -@pytest.mark.parametrize( - ("value", "message"), - [ - (True, "turn_refresh_workers must be an integer >= 1"), - (0, "turn_refresh_workers must be >= 1"), - (-1, "turn_refresh_workers must be >= 1"), - (33, "turn_refresh_workers must be <= 32"), - (1.5, "turn_refresh_workers must be an integer >= 1"), - ], -) -def test_turn_refresh_workers_reject_invalid_bounds( - value: object, - message: str, -) -> None: - with pytest.raises(ValueError, match=message): - Config(turn_refresh_workers=value) - - -def test_turn_refresh_workers_cannot_exceed_observed_worker_max(monkeypatch) -> None: - with pytest.raises( - ValueError, - match="turn_refresh_workers must be <= max_workers", - ): - Config(max_workers=3, turn_refresh_workers=4) - - monkeypatch.setenv("TENDWIRE_TURN_REFRESH_WORKERS", "5") - with pytest.raises( - ValueError, - match="turn_refresh_workers must be <= max_workers", - ): - load_config(max_workers=4) - - SNAPSHOT_MAINTENANCE_ENV_NAMES = ( "TENDWIRE_SNAPSHOT_RETENTION_DAYS", "TENDWIRE_SNAPSHOT_RETENTION_COUNT", @@ -577,43 +576,7 @@ def test_socket_group_defaults_private_and_normalizes_without_lookup(monkeypatch assert load_config(socket_group=" ").socket_group is None -def test_herdr_backend_defaults_to_cli(monkeypatch) -> None: - monkeypatch.delenv("TENDWIRE_HERDR_BACKEND", raising=False) - - assert Config().herdr_backend == "cli" - assert load_config().herdr_backend == "cli" - - -@pytest.mark.parametrize( - ("raw", "expected"), - [ - ("cli", "cli"), - ("socket", "socket"), - (" CLI ", "cli"), - ("SOCKET", "socket"), - ], -) -def test_herdr_backend_accepts_explicit_values(monkeypatch, raw: str, expected: str) -> None: - monkeypatch.delenv("TENDWIRE_HERDR_BACKEND", raising=False) - - assert Config(herdr_backend=raw).herdr_backend == expected - assert load_config(herdr_backend=raw).herdr_backend == expected - - -def test_herdr_backend_reads_environment(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_HERDR_BACKEND", "socket") - - assert load_config().herdr_backend == "socket" - - -def test_herdr_backend_invalid_value_fails_clearly(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_HERDR_BACKEND", "events") - - with pytest.raises(ValueError, match="herdr_backend must be one of: cli, socket"): - load_config() - - -def test_cli_default_import_does_not_load_socket_event_backend() -> None: +def test_cli_default_import_does_not_load_socket_transport() -> None: code = """ import sys before = set(sys.modules) @@ -621,7 +584,6 @@ def test_cli_default_import_does_not_load_socket_event_backend() -> None: loaded = set(sys.modules) - before for name in sorted(loaded): if name in { - "tendwire.backends.herdr_events", "tendwire.backends.herdr_socket", "tendwire.backends.herdr_protocol", }: diff --git a/tests/test_connector_daemon_cli.py b/tests/test_connector_daemon_cli.py index 96b1c65..84c8dfa 100644 --- a/tests/test_connector_daemon_cli.py +++ b/tests/test_connector_daemon_cli.py @@ -2,10 +2,13 @@ from __future__ import annotations -from contextlib import closing +from collections.abc import Iterator +from contextlib import closing, contextmanager +from datetime import datetime, timedelta, timezone import io import json import sqlite3 +import threading from pathlib import Path from typing import Any @@ -16,7 +19,11 @@ from tendwire.config import Config from tendwire.core.models import Snapshot from tendwire.daemon import TendwireDaemon -from tendwire.daemon_api import TendwireDaemonAPI +from tendwire.daemon_api import ( + DaemonAPIClient, + TendwireDaemonAPI, + UnixSocketJSONServer, +) from tendwire.store import sqlite as store_sqlite from tendwire.store.sqlite import init_store @@ -64,6 +71,72 @@ def _enqueue(db_path: Path, *, host_id: str = "host-a", key: str = "job-1") -> N ) +def _canonical_final_turn(db_path: Path, *, host_id: str) -> tuple[str, str]: + init_store(db_path) + turn_id = "turn-worker-contract-source-contract" + final_text = "abcdefgh" + revision = store_sqlite.content_revision( + turn_id, + None, + final_text, + "absent", + "complete", + ) + created_at = "2026-01-01T00:00:00+00:00" + with closing(sqlite3.connect(str(db_path))) as conn, conn: + conn.execute( + """ + INSERT INTO turns ( + host_id, turn_id, worker_id, status, kind, updated_at, + fingerprint, snapshot_content_fingerprint, observed_at, + payload_json, list_sequence + ) VALUES (?, ?, ?, 'complete', 'turn', ?, ?, ?, ?, ?, 1) + """, + ( + host_id, + turn_id, + "worker-contract", + created_at, + "fingerprint-contract", + "snapshot-contract", + created_at, + json.dumps( + { + "source_turn_id": "source-contract", + "complete": True, + "meta": { + "stable_key": "wsk1_" + ("c" * 64), + "stable_key_version": 1, + }, + } + ), + ), + ) + conn.execute( + """ + INSERT INTO turn_content_revisions ( + host_id, turn_id, content_revision, user_text, + assistant_final_text, user_state, final_state, + user_char_length, user_byte_length, + final_char_length, final_byte_length, + user_page_count, final_page_count, + is_current, created_at, superseded_at + ) VALUES (?, ?, ?, NULL, ?, 'absent', 'complete', 0, 0, 8, 8, + 0, 1, 1, ?, NULL) + """, + (host_id, turn_id, revision, final_text, created_at), + ) + source_id = store_sqlite._ensure_final_ready_anchor_conn( + conn, + host_id=host_id, + turn_id=turn_id, + content_revision_value=revision, + now=created_at, + ) + assert source_id is not None + return turn_id, revision + + def _assert_json_only_and_safe(payload: dict[str, Any]) -> None: encoded = json.dumps(payload, sort_keys=True).lower() for forbidden in ( @@ -86,6 +159,489 @@ def _assert_json_only_and_safe(payload: dict[str, Any]) -> None: assert forbidden not in encoded +@contextmanager +def _socket_client( + tmp_path: Path, + api: TendwireDaemonAPI, +) -> Iterator[DaemonAPIClient]: + socket_path = tmp_path / "s" + server = UnixSocketJSONServer(socket_path, api.dispatch) + server.start() + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + yield DaemonAPIClient(socket_path, timeout_seconds=2) + finally: + server.close() + thread.join(timeout=2) + assert not thread.is_alive() + + +@pytest.mark.skipif( + not hasattr(__import__("socket"), "AF_UNIX"), + reason="Unix sockets required", +) +def test_socket_ack_lost_repoll_preserves_durable_identity_and_payload( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "ack-lost.db" + binding_db = tmp_path / "provider-bindings.db" + turn_id, revision = _canonical_final_turn(db_path, host_id="daemon-host") + current = datetime(2026, 1, 1, tzinfo=timezone.utc) + monkeypatch.setattr( + store_sqlite, + "utc_timestamp", + lambda: current.isoformat(timespec="seconds"), + ) + first_outbox = ConnectorOutboxAPI(db_path, "daemon-host") + first_api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=first_outbox.dispatch, + ) + + with closing(sqlite3.connect(str(binding_db))) as conn, conn: + conn.execute( + """ + CREATE TABLE provider_message_bindings ( + delivery_key TEXT PRIMARY KEY, + provider_message_id TEXT NOT NULL, + payload_json TEXT NOT NULL + ) + """ + ) + + with _socket_client(tmp_path, first_api) as client: + source_outer = client.request( + "connector.poll", + {"name": "turn-final", "lease_seconds": 5}, + ) + source = source_outer["result"]["items"][0] + assert source["payload"]["operation"] == "materialize" + assert source["payload"]["content_revision"] == revision + begun = client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "begin", + "name": "turn-final", + "turn_id": turn_id, + "content_revision": revision, + "presentation_version": "contract-v2", + "part_count": 2, + "source_ref": source["ref"], + }, + ) + plan_token = begun["result"]["plan_token"] + for ordinal, start in enumerate((0, 4)): + staged = client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "part", + "name": "turn-final", + "plan_token": plan_token, + "ordinal": ordinal, + "spans": [ + { + "field": "assistant_final_text", + "start_char": start, + "end_char": start + 4, + } + ], + }, + ) + assert staged["ok"] is True + assert staged["result"]["ok"] is True + committed = client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "commit", + "name": "turn-final", + "plan_token": plan_token, + "source_ref": source["ref"], + }, + ) + assert begun["ok"] is True + assert begun["result"]["ok"] is True + assert committed["ok"] is True + assert committed["result"]["ok"] is True + assert committed["result"]["job_count"] == 2 + with closing(sqlite3.connect(str(db_path))) as conn: + source_state = conn.execute( + """ + SELECT outbox.status, attempts.status + FROM connector_outbox AS outbox + JOIN connector_deliveries AS attempts + ON attempts.outbox_id = outbox.id + WHERE outbox.delivery_key = ? + """, + (source["key"],), + ).fetchone() + assert source_state == ("awaiting_ack", "awaiting_ack") + + first_outer = client.request( + "connector.poll", + {"name": "turn-final", "lease_seconds": 5}, + ) + first = first_outer["result"]["items"][0] + + # The provider accepted the message and Herdres persisted the binding, + # but the connector ACK was lost before Tendwire received it. + with closing(sqlite3.connect(str(binding_db))) as conn, conn: + conn.execute( + """ + INSERT INTO provider_message_bindings ( + delivery_key, provider_message_id, payload_json + ) VALUES (?, ?, ?) + """, + ( + first["key"], + "provider-message-17", + json.dumps( + first["payload"], + sort_keys=True, + separators=(",", ":"), + ), + ), + ) + + # The connector and its socket server have stopped. Advance beyond the lease, + # then construct fresh connector/server objects from the durable databases. + current += timedelta(seconds=6) + restarted_outbox = ConnectorOutboxAPI(db_path, "daemon-host") + restarted_api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=restarted_outbox.dispatch, + ) + + with _socket_client(tmp_path, restarted_api) as client: + reclaimed = client.request("connector.reclaim", {"name": "turn-final"}) + second_outer = client.request( + "connector.poll", + {"name": "turn-final", "lease_seconds": 5}, + ) + second = second_outer["result"]["items"][0] + + assert first_outer["ok"] is True + assert first_outer["result"]["ok"] is True + assert reclaimed["ok"] is True + assert reclaimed["result"]["reclaimed"] == 1 + assert second["key"] == first["key"] + assert second["payload"] == first["payload"] + assert json.dumps( + second["payload"], sort_keys=True, separators=(",", ":") + ) == json.dumps(first["payload"], sort_keys=True, separators=(",", ":")) + assert second["ref"] != first["ref"] + assert second["attempt"] == first["attempt"] + 1 + stale_ack = client.request( + "connector.ack", + {"name": "turn-final", "ref": first["ref"]}, + ) + assert stale_ack["ok"] is True + assert stale_ack["result"]["ok"] is False + assert stale_ack["result"]["status"] in { + "expired_ref", + "invalid_ref", + "stale_ref", + } + + # A thin connector recognizes the durable key, does not resend, and + # acknowledges the current attempt rather than the stale first ref. + with closing(sqlite3.connect(str(binding_db))) as conn, conn: + durable_binding = conn.execute( + """ + SELECT provider_message_id, payload_json + FROM provider_message_bindings + WHERE delivery_key = ? + """, + (second["key"],), + ).fetchone() + mutation_count_before = conn.total_changes + if durable_binding is None: + conn.execute( + """ + INSERT INTO provider_message_bindings ( + delivery_key, provider_message_id, payload_json + ) VALUES (?, ?, ?) + """, + (second["key"], "unexpected-second-send", "{}"), + ) + mutation_count_after = conn.total_changes + provider_row_count = conn.execute( + "SELECT COUNT(*) FROM provider_message_bindings" + ).fetchone()[0] + assert durable_binding == ( + "provider-message-17", + json.dumps(first["payload"], sort_keys=True, separators=(",", ":")), + ) + assert mutation_count_after == mutation_count_before + assert provider_row_count == 1 + acknowledged = client.request( + "connector.ack", + { + "name": "turn-final", + "ref": second["ref"], + "response": {"status": "deduplicated"}, + }, + ) + assert acknowledged["ok"] is True + assert acknowledged["result"]["ok"] is True + assert acknowledged["result"]["status"] == "acknowledged" + next_item = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert next_item["key"] != second["key"] + assert next_item["payload"]["plan_token"] == plan_token + assert next_item["payload"]["sequence_index"] == 1 + assert client.request( + "connector.ack", + {"name": "turn-final", "ref": next_item["ref"]}, + )["result"]["ok"] is True + assert client.request("connector.poll", {"name": "turn-final"})["result"][ + "items" + ] == [] + with closing(sqlite3.connect(str(db_path))) as conn: + completed_source_state = conn.execute( + """ + SELECT outbox.status, attempts.status + FROM connector_outbox AS outbox + JOIN connector_deliveries AS attempts + ON attempts.outbox_id = outbox.id + WHERE outbox.delivery_key = ? + """, + (source["key"],), + ).fetchone() + assert completed_source_state == ("delivered", "delivered") + + +@pytest.mark.skipif( + not hasattr(__import__("socket"), "AF_UNIX"), + reason="Unix sockets required", +) +def test_socket_keeps_transport_and_connector_errors_in_separate_envelopes( + tmp_path: Path, +) -> None: + db_path = tmp_path / "error-envelopes.db" + init_store(db_path) + outbox = ConnectorOutboxAPI(db_path, "daemon-host") + api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=outbox.dispatch, + ) + + with _socket_client(tmp_path, api) as client: + connector_error = client.request( + "connector.ack", + {"name": "attention", "ref": "not-a-live-ref"}, + ) + protocol_error = client.request("turn_final_ack", {}) + + assert connector_error["ok"] is True + assert connector_error["status"] == "ok" + assert connector_error["error"] is None + assert connector_error["result"]["ok"] is False + assert connector_error["result"]["status"] == "invalid_ref" + assert connector_error["result"]["error"]["code"] == "invalid_ref" + assert protocol_error["ok"] is False + assert protocol_error["status"] == "error" + assert protocol_error["result"] is None + assert protocol_error["error"]["code"] == "unknown_method" + + +@pytest.mark.skipif( + not hasattr(__import__("socket"), "AF_UNIX"), + reason="Unix sockets required", +) +def test_socket_explicit_retry_resets_generation_attempt_and_bounds_history( + tmp_path: Path, +) -> None: + db_path = tmp_path / "retry-generation.db" + _canonical_final_turn(db_path, host_id="daemon-host") + outbox = ConnectorOutboxAPI(db_path, "daemon-host", max_attempts=2) + api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=outbox.dispatch, + ) + + with _socket_client(tmp_path, api) as client: + first = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert first["attempt"] == 1 + assert client.request( + "connector.fail", + {"name": "turn-final", "ref": first["ref"], "delay_seconds": 0}, + )["result"]["status"] == "retry_scheduled" + second = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert second["key"] == first["key"] + assert second["attempt"] == 2 + assert client.request( + "connector.fail", + {"name": "turn-final", "ref": second["ref"], "delay_seconds": 0}, + )["result"]["status"] == "attempts_exhausted" + inspected = client.request( + "connector.inspect", + { + "schema_version": 1, + "name": "turn-final", + "status": "dead_letter", + "limit": 10, + }, + )["result"] + assert inspected["items"][0]["attempt_count"] == 2 + retried = client.request( + "connector.retry", + { + "schema_version": 1, + "name": "turn-final", + "key": first["key"], + }, + )["result"] + assert retried["status"] == "requeued" + assert retried["prior_attempt_count"] == 2 + + with closing(sqlite3.connect(str(db_path))) as conn: + compacted = conn.execute( + """ + SELECT COUNT(attempts.id), outbox.private_state_json + FROM connector_outbox AS outbox + LEFT JOIN connector_deliveries AS attempts + ON attempts.outbox_id = outbox.id + WHERE outbox.delivery_key = ? + GROUP BY outbox.id + """, + (first["key"],), + ).fetchone() + assert compacted is not None + assert compacted[0] == 0 + assert json.loads(compacted[1])["prior_attempt_count"] == 2 + fresh = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert fresh["key"] == first["key"] + assert fresh["attempt"] == 1 + + +@pytest.mark.skipif( + not hasattr(__import__("socket"), "AF_UNIX"), + reason="Unix sockets required", +) +def test_socket_preserves_opaque_connector_tokens_byte_for_byte( + tmp_path: Path, +) -> None: + db_path = tmp_path / "token-preservation.db" + turn_id, revision = _canonical_final_turn(db_path, host_id="daemon-host") + with closing(sqlite3.connect(str(db_path))) as conn: + key, raw_source_payload = conn.execute( + """ + SELECT delivery_key, payload_json + FROM connector_outbox + WHERE delivery_kind = 'final_ready' + """ + ).fetchone() + expected_source = json.loads(raw_source_payload) + outbox = ConnectorOutboxAPI(db_path, "daemon-host") + api = TendwireDaemonAPI( + get_snapshot=lambda: Snapshot(host_id="daemon-host"), + get_health=lambda: {}, + submit_command=lambda _params: {}, + connector_call=outbox.dispatch, + ) + + with _socket_client(tmp_path, api) as client: + source = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + assert source["key"].encode() == key.encode() + assert source["payload"]["final_identity"].encode() == expected_source[ + "final_identity" + ].encode() + assert source["payload"]["content_revision"].encode() == revision.encode() + assert source["payload"]["content"]["content_revision"].encode() == ( + expected_source["content"]["content_revision"].encode() + ) + + begun = client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "begin", + "name": "turn-final", + "turn_id": turn_id, + "content_revision": revision, + "presentation_version": "presentation-v2", + "part_count": 1, + "source_ref": source["ref"], + }, + )["result"] + assert begun["ok"] is True, begun + plan_token = begun["plan_token"] + assert client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "part", + "name": "turn-final", + "plan_token": plan_token, + "ordinal": 0, + "spans": [ + { + "field": "assistant_final_text", + "start_char": 0, + "end_char": 8, + } + ], + }, + )["result"]["ok"] is True + assert client.request( + "connector.prepare", + { + "schema_version": 1, + "action": "commit", + "name": "turn-final", + "plan_token": plan_token, + "source_ref": source["ref"], + }, + )["result"]["ok"] is True + part = client.request("connector.poll", {"name": "turn-final"})[ + "result" + ]["items"][0] + + with closing(sqlite3.connect(str(db_path))) as conn: + stored_part = json.loads( + conn.execute( + "SELECT payload_json FROM connector_outbox WHERE delivery_key = ?", + (part["key"],), + ).fetchone()[0] + ) + for path in ( + ("plan_token",), + ("content_revision",), + ("turn", "final_identity"), + ("turn", "content_revision"), + ("turn", "content", "content_revision"), + ): + actual: Any = part["payload"] + expected: Any = stored_part + for field in path: + actual = actual[field] + expected = expected[field] + assert actual.encode() == expected.encode() + assert part["payload"]["plan_token"].encode() == plan_token.encode() + + def test_daemon_api_routes_connector_methods_safely(tmp_path: Path) -> None: db_path = tmp_path / "daemon-connector.db" _enqueue(db_path, host_id="daemon-host") @@ -215,122 +771,6 @@ def reject_write(*_args: Any, **_kwargs: Any) -> None: daemon._connector_periodic_tick() -def test_cli_connector_poll_and_ack_print_json_only(tmp_path: Path, capsys) -> None: - db_path = tmp_path / "cli-connector.db" - _enqueue(db_path) - poll_code = main( - [ - "--host-id", - "host-a", - "connector", - "poll", - "--name", - "attention", - "--db-path", - str(db_path), - "--lease-seconds", - "60", - ] - ) - poll_captured = capsys.readouterr() - poll_payload = json.loads(poll_captured.out) - ref = poll_payload["items"][0]["ref"] - - ack_code = main( - [ - "--host-id", - "host-a", - "connector", - "ack", - "--name", - "attention", - "--ref", - ref, - "--response-json", - json.dumps({"safe": "kept", "chat_id": "must-strip", "provider": "telegram"}), - "--db-path", - str(db_path), - ] - ) - ack_captured = capsys.readouterr() - ack_payload = json.loads(ack_captured.out) - - assert poll_code == 0 - assert ack_code == 0 - assert poll_captured.err == "" - assert ack_captured.err == "" - assert poll_payload["items"][0]["payload"]["safe"] == "kept" - assert ack_payload["status"] == "acknowledged" - _assert_json_only_and_safe(poll_payload) - _assert_json_only_and_safe(ack_payload) - - -def test_cli_connector_renew_and_release_forward_live_ref_options( - tmp_path: Path, - capsys, -) -> None: - db_path = tmp_path / "cli-renew-release.db" - _enqueue(db_path, key="cli-renew-release") - poll_code = main( - [ - "--host-id", - "host-a", - "connector", - "poll", - "--name", - "attention", - "--db-path", - str(db_path), - ] - ) - poll_payload = json.loads(capsys.readouterr().out) - ref = poll_payload["items"][0]["ref"] - - renew_code = main( - [ - "--host-id", - "host-a", - "connector", - "renew", - "--name", - "attention", - "--ref", - ref, - "--lease-seconds", - "120", - "--db-path", - str(db_path), - ] - ) - renew_capture = capsys.readouterr() - renew_payload = json.loads(renew_capture.out) - release_code = main( - [ - "--host-id", - "host-a", - "connector", - "release", - "--name", - "attention", - "--ref", - ref, - "--db-path", - str(db_path), - ] - ) - release_capture = capsys.readouterr() - release_payload = json.loads(release_capture.out) - - assert poll_code == renew_code == release_code == 0 - assert renew_capture.err == release_capture.err == "" - assert renew_payload["status"] == "renewed" - assert renew_payload["leased_until"] - assert release_payload["status"] == "released" - assert release_payload["available_at"] - _assert_json_only_and_safe(renew_payload) - _assert_json_only_and_safe(release_payload) - - def test_cli_connector_prepare_reads_bounded_action_from_stdin( tmp_path: Path, capsys, @@ -415,6 +855,8 @@ def test_cli_daemon_connector_result_is_sanitized_before_printing( capsys, monkeypatch, ) -> None: + revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" + class FakeDaemonAPIClient: def __init__(self, socket_path: Any, *, timeout_seconds: float, max_response_bytes: int = 1024 * 1024): pass @@ -437,6 +879,11 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str "safe": "kept", "turn_id": "turn-public-final", "plan_token": "twplan1.publicPlan", + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + }, "chat_id": "sentinel-private-chat", "raw_payload": "sentinel-private-raw", }, @@ -470,6 +917,11 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str "safe": "kept", "turn_id": "turn-public-final", "plan_token": "twplan1.publicPlan", + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + }, } assert "sentinel-private" not in encoded assert "raw_payload" not in encoded @@ -477,6 +929,7 @@ def request(self, method: str, params: dict[str, Any] | None = None) -> dict[str def test_daemon_connector_preserves_public_turn_id_for_final_ready() -> None: + revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" api = TendwireDaemonAPI( get_snapshot=lambda: Snapshot(host_id="host-a"), get_health=lambda: {}, @@ -493,6 +946,13 @@ def test_daemon_connector_preserves_public_turn_id_for_final_ready() -> None: "schema_version": 2, "operation": "final_ready", "turn_id": "turn-public-final", + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + "known_incomplete": False, + "fields": {}, + }, "pane_id": "sentinel-private-pane", "session_id": "sentinel-private-session", "terminal_id": "sentinel-private-terminal", @@ -512,6 +972,13 @@ def test_daemon_connector_preserves_public_turn_id_for_final_ready() -> None: "schema_version": 2, "operation": "final_ready", "turn_id": "turn-public-final", + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + "known_incomplete": False, + "fields": {}, + }, } _assert_json_only_and_safe(response) diff --git a/tests/test_connector_outbox.py b/tests/test_connector_outbox.py index 8a75330..90f1079 100644 --- a/tests/test_connector_outbox.py +++ b/tests/test_connector_outbox.py @@ -176,6 +176,55 @@ def test_poll_leases_sanitized_item_and_skips_duplicate_live_lease(tmp_path: Pat _assert_no_forbidden(first) +def test_final_ready_poll_exposes_durable_source_age(tmp_path: Path) -> None: + db_path = tmp_path / "final-created-at.db" + key = _enqueue_final_root( + db_path, + key_suffix="created_at", + ordering_key="wsk1_created_at", + ) + api = ConnectorOutboxAPI(db_path, "host-a") + + item = api.poll({"name": "turn-final", "limit": 1})["items"][0] + + assert item["key"] == key + assert item["created_at"] == "2026-01-01T00:00:00+00:00" + + +def test_poll_preserves_strict_content_revision_tokens(tmp_path: Path) -> None: + db_path = tmp_path / "revision-token.db" + revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" + _enqueue_final_root( + db_path, + key_suffix="public", + ordering_key="wsk1_public", + ) + with sqlite3.connect(str(db_path)) as conn: + conn.execute( + "UPDATE connector_outbox SET payload_json = ?", + ( + json.dumps( + { + "schema_version": 2, + "content_revision": revision, + "content": { + "schema_version": 1, + "content_revision": revision, + }, + } + ), + ), + ) + + item = ConnectorOutboxAPI(db_path, "host-a").poll( + {"name": "turn-final", "limit": 1} + )["items"][0] + + assert item["payload"]["content_revision"] == revision + assert item["payload"]["content"]["content_revision"] == revision + _assert_no_forbidden(item) + + def test_poll_uses_configured_default_lease_and_explicit_lease_wins(tmp_path: Path) -> None: db_path = tmp_path / "lease-default.db" _enqueue(db_path, key="default-lease") @@ -1232,165 +1281,6 @@ def test_lifecycle_delivery_key_survives_fail_defer_reclaim_and_ack(tmp_path: Pa assert [row[2] for row in delivery_rows] == ["failed", "deferred", "expired", "delivered"] -def test_migration_terminalizes_noncanonical_live_leases_through_public_helpers( - tmp_path: Path, -) -> None: - db_path = tmp_path / "leased-duplicate-migration.db" - host_id = "host-migration-leases" - observed_at = "2026-01-01T00:00:00+00:00" - snapshot = Snapshot( - host_id=host_id, - updated_at=observed_at, - attention=[ - AttentionSignal( - kind="worker_status", - severity="warning", - status="waiting", - reason="Review the worker", - source="worker:worker-1", - updated_at=observed_at, - host_id=host_id, - ) - ], - ) - save_snapshot( - db_path, - snapshot, - observation=SnapshotObservationContext( - authority="complete", - observed_at=observed_at, - ), - ) - - with sqlite3.connect(str(db_path)) as conn: - for suffix in range(1, 4): - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) - SELECT - host_id, connector, ?, 'queued', payload_json, - '{}', created_at, updated_at, NULL - FROM connector_outbox - WHERE host_id = ? AND connector = 'attention' - ORDER BY id - LIMIT 1 - """, - (f"legacy-duplicate-{suffix}", host_id), - ) - - canonical = poll_connector_outbox( - db_path, - host_id, - "attention", - lease_seconds=100, - now="2026-01-01T00:00:01+00:00", - )["items"][0] - failed_lease = poll_connector_outbox( - db_path, - host_id, - "attention", - lease_seconds=100, - now="2026-01-01T00:00:02+00:00", - )["items"][0] - deferred_lease = poll_connector_outbox( - db_path, - host_id, - "attention", - lease_seconds=100, - now="2026-01-01T00:00:03+00:00", - )["items"][0] - expiring_lease = poll_connector_outbox( - db_path, - host_id, - "attention", - lease_seconds=5, - now="2026-01-01T00:00:04+00:00", - )["items"][0] - - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA user_version = 4") - init_store(db_path) - - failed = fail_connector_delivery( - db_path, - host_id=host_id, - name="attention", - ref=failed_lease["ref"], - delay_seconds=0, - now="2026-01-01T00:00:05+00:00", - ) - deferred = defer_connector_delivery( - db_path, - host_id=host_id, - name="attention", - ref=deferred_lease["ref"], - delay_seconds=0, - now="2026-01-01T00:00:06+00:00", - ) - reclaimed = reclaim_expired_connector_leases( - db_path, - host_id, - "attention", - now="2026-01-01T00:00:09+00:00", - ) - acknowledged = ack_connector_delivery( - db_path, - host_id=host_id, - name="attention", - ref=canonical["ref"], - now="2026-01-01T00:00:10+00:00", - ) - after = poll_connector_outbox( - db_path, - host_id, - "attention", - now="2026-01-01T00:00:11+00:00", - ) - - assert failed["status"] == "superseded" - assert failed["key"] == failed_lease["key"] - assert deferred["status"] == "superseded" - assert deferred["key"] == deferred_lease["key"] - assert reclaimed["reclaimed"] == 1 - assert acknowledged["status"] == "acknowledged" - assert acknowledged["key"] == canonical["key"] - assert after["items"] == [] - - with sqlite3.connect(str(db_path)) as conn: - outbox_rows = conn.execute( - """ - SELECT delivery_key, status - FROM connector_outbox - WHERE host_id = ? AND connector = 'attention' - ORDER BY id - """, - (host_id,), - ).fetchall() - delivery_rows = conn.execute( - """ - SELECT delivery_key, status - FROM connector_deliveries - WHERE host_id = ? AND connector = 'attention' - ORDER BY id - """, - (host_id,), - ).fetchall() - - assert outbox_rows == [ - (canonical["key"], "delivered"), - (failed_lease["key"], "superseded"), - (deferred_lease["key"], "superseded"), - (expiring_lease["key"], "superseded"), - ] - assert delivery_rows == [ - (canonical["key"], "delivered"), - (failed_lease["key"], "failed"), - (deferred_lease["key"], "deferred"), - (expiring_lease["key"], "expired"), - ] def _canonical_turn( @@ -1716,83 +1606,6 @@ def _downgrade_presentation_schema_to_v6(db_path: Path) -> None: ) -def test_v6_to_current_plan_migration_is_bounded_atomic_and_preserves_jobs( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "presentation-v6.db" - turn_id, revision = _canonical_turn(db_path, final_text="abcdefgh") - api = ConnectorOutboxAPI(db_path, "host-a") - plan = _stage_final_plan( - api, - turn_id=turn_id, - revision=revision, - ranges=[(0, 4), (4, 8)], - ) - _downgrade_presentation_schema_to_v6(db_path) - - init_store(db_path) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - version = conn.execute("PRAGMA user_version").fetchone()[0] - plan_row = conn.execute( - """ - SELECT plan_token, generation, recovers_plan_token, state - FROM turn_presentation_plans - """ - ).fetchone() - job_count = conn.execute( - "SELECT COUNT(*) FROM turn_presentation_jobs" - ).fetchone()[0] - outbox_count = conn.execute( - "SELECT COUNT(*) FROM connector_outbox WHERE connector = 'turn-final'" - ).fetchone()[0] - audit_columns = { - row[1] - for row in conn.execute( - "PRAGMA table_info(turn_presentation_recoveries)" - ).fetchall() - } - foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert version == store_sqlite.STORE_SCHEMA_VERSION == 21 - assert plan_row == (plan["plan_token"], 1, None, "active") - assert job_count == 2 - assert outbox_count == 3 - assert { - "request_id", - "failed_plan_id", - "recovered_plan_id", - "generation", - "delivered_prefix_count", - "fresh_job_count", - "retained_failed_job_count", - "prior_attempt_count", - } <= audit_columns - assert foreign_keys == [] - - _downgrade_presentation_schema_to_v6(db_path) - - def fail_rebuild(_conn: sqlite3.Connection) -> None: - raise RuntimeError("controlled v7 migration failure") - - monkeypatch.setattr( - store_sqlite, - "_rebuild_v6_presentation_plans_conn", - fail_rebuild, - ) - with pytest.raises(RuntimeError, match="controlled v7 migration failure"): - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == 6 - assert "generation" not in { - row[1] - for row in conn.execute( - "PRAGMA table_info(turn_presentation_plans)" - ).fetchall() - } - assert conn.execute( - "SELECT COUNT(*) FROM turn_presentation_jobs" - ).fetchone()[0] == 2 def test_prepare_stages_idempotently_and_rejects_conflicts_or_incomplete_coverage( @@ -1926,6 +1739,92 @@ def fail_second(*args: Any, **kwargs: Any) -> int: assert outbox_count == linked_count == 0 +def test_exhausted_source_fails_zero_job_preparing_plan_idempotently( + tmp_path: Path, +) -> None: + db_path = tmp_path / "prepare-source-exhausted.db" + turn_id, revision = _canonical_turn(db_path, final_text="abcdefgh") + with sqlite3.connect(str(db_path)) as conn: + conn.execute( + """ + INSERT INTO connector_outbox ( + host_id, connector, delivery_key, delivery_kind, + turn_id, content_revision, ordering_key, status, + payload_json, private_state_json, created_at, updated_at + ) VALUES ( + 'host-a', 'turn-final', ?, 'final_ready', ?, ?, + 'wsk1_source_exhausted', 'queued', '{}', '{}', ?, ? + ) + """, + ( + "turn-final:revision:twfinal1." + ("s" * 64), + turn_id, + revision, + "2026-01-01T00:00:00+00:00", + "2026-01-01T00:00:00+00:00", + ), + ) + api = ConnectorOutboxAPI(db_path, "host-a", max_attempts=1) + source = api.poll({"name": "turn-final", "limit": 1})["items"][0] + begun = api.prepare( + { + "schema_version": 1, + "action": "begin", + "name": "turn-final", + "turn_id": turn_id, + "content_revision": revision, + "presentation_version": "turn-present-v27", + "part_count": 1, + "source_ref": source["ref"], + } + ) + token = begun["plan_token"] + assert _put_final_part( + api, + plan_token=token, + ordinal=0, + start=0, + end=8, + )["ok"] is True + + exhausted = api.fail( + { + "name": "turn-final", + "ref": source["ref"], + "delay_seconds": 0, + } + ) + repeated_commit = api.prepare( + { + "schema_version": 1, + "action": "commit", + "name": "turn-final", + "plan_token": token, + "source_ref": source["ref"], + } + ) + + assert exhausted["status"] == "attempts_exhausted" + assert repeated_commit["ok"] is True + assert repeated_commit["state"] == "failed" + assert repeated_commit["job_count"] == 0 + with sqlite3.connect(str(db_path)) as conn: + assert conn.execute( + "SELECT state FROM turn_presentation_plans WHERE plan_token = ?", + (token,), + ).fetchone()[0] == "failed" + assert conn.execute( + """ + SELECT COUNT(*) + FROM turn_presentation_jobs + WHERE plan_id = ( + SELECT id FROM turn_presentation_plans WHERE plan_token = ? + ) AND outbox_id IS NOT NULL + """, + (token,), + ).fetchone()[0] == 0 + + def test_prepare_commit_rechecks_current_revision_and_creates_no_jobs_on_conflict( tmp_path: Path, ) -> None: @@ -2335,8 +2234,10 @@ def test_unrelated_plans_poll_concurrently_but_never_colease_siblings( assert {item["payload"]["sequence_index"] for item in items} == {0} -def test_replacement_waits_for_old_lease_then_activates_without_requeue( +@pytest.mark.parametrize("terminal_action", ["release", "fail", "defer"]) +def test_replacement_terminalizes_old_lease_then_activates_without_requeue( tmp_path: Path, + terminal_action: str, ) -> None: db_path = tmp_path / "prepare-replacement-barrier.db" turn_id, revision = _canonical_turn(db_path, final_text="abcdefgh") @@ -2365,10 +2266,15 @@ def test_replacement_waits_for_old_lease_then_activates_without_requeue( assert new["state"] == "waiting_predecessor" assert api.poll({"name": "turn-final", "limit": 10})["items"] == [] - terminalized = api.fail( - {"name": "turn-final", "ref": leased_old["ref"], "delay_seconds": 0} - ) + terminal_params: dict[str, Any] = { + "name": "turn-final", + "ref": leased_old["ref"], + } + if terminal_action in {"fail", "defer"}: + terminal_params["delay_seconds"] = 0 + terminalized = getattr(api, terminal_action)(terminal_params) assert terminalized["status"] == "superseded" + assert "available_at" not in terminalized activated = api.poll({"name": "turn-final", "limit": 10})["items"] assert len(activated) == 1 assert activated[0]["payload"]["plan_token"] == new["plan_token"] @@ -3504,162 +3410,3 @@ def test_awaiting_ack_without_plan_becomes_terminal_failed(tmp_path: Path) -> No "classification": "plan_unrecoverable", "terminalized_at": "2026-01-01T00:00:01+00:00", } - - -def test_v16_migration_backfills_ordering_and_awaiting_ack_deadlines( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "connector-v16-migration.db" - stable_key = "wsk1_" + ("c" * 64) - enqueue_stable_key = "wsk1_" + ("d" * 64) - tombstoned_turn, _ = _canonical_turn( - db_path, - worker_id="worker-tombstoned", - source_turn_id="source-tombstoned", - stable_key=stable_key, - final_text="done", - ) - fallback_turn, _ = _canonical_turn( - db_path, - worker_id="worker-fallback", - source_turn_id="source-fallback", - stable_key="invalid", - final_text="done", - ) - deleted_turns = [ - _canonical_turn( - db_path, - worker_id=f"worker-deleted-{suffix}", - source_turn_id=f"source-deleted-{suffix}", - final_text="done", - )[0] - for suffix in ("a", "b") - ] - with sqlite3.connect(str(db_path)) as conn: - assert store_sqlite._migrate_tombstone_command_turn_conn( - conn, - "host-a", - tombstoned_turn, - superseded_by_turn_id=None, - superseded_at="2026-01-02T00:00:00+00:00", - ) - for turn_id, status in ( - (tombstoned_turn, "awaiting_ack"), - (fallback_turn, "dead_letter"), - (fallback_turn, "superseded"), - ): - payload = ( - { - "stable_key": enqueue_stable_key, - "stable_key_version": 1, - } - if status == "dead_letter" - else {} - ) - cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, delivery_kind, turn_id, - ordering_key, status, payload_json, private_state_json, - created_at, updated_at - ) VALUES (?, 'turn-final', ?, 'final_ready', ?, '', ?, ?, '{}', ?, ?) - """, - ( - "host-a", - f"migration-{turn_id}-{status}", - turn_id, - status, - json.dumps(payload), - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - if status == "awaiting_ack": - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, - status, response_json, private_state_json, created_at - ) VALUES (?, 'host-a', 'turn-final', 'migration-awaiting', 1, - 'awaiting_ack', '{}', '{}', ?) - """, - (cursor.lastrowid, "2026-01-01T00:00:00+00:00"), - ) - for deleted_turn in deleted_turns: - conn.execute( - "DELETE FROM turn_content_revisions WHERE host_id = ? AND turn_id = ?", - ("host-a", deleted_turn), - ) - conn.execute( - "DELETE FROM turns WHERE host_id = ? AND turn_id = ?", - ("host-a", deleted_turn), - ) - orphan_ids: list[int] = [] - for suffix, deleted_turn in zip(("a", "b"), deleted_turns, strict=True): - cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, delivery_kind, turn_id, - ordering_key, status, payload_json, private_state_json, - created_at, updated_at - ) VALUES ( - 'host-a', 'turn-final', ?, 'final_ready', ?, '', - 'dead_letter', '{}', '{}', ?, ? - ) - """, - ( - f"migration-deleted-{suffix}", - deleted_turn, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - orphan_ids.append(int(cursor.lastrowid)) - conn.execute("DROP INDEX IF EXISTS idx_connector_outbox_final_ordering") - conn.execute("ALTER TABLE connector_outbox DROP COLUMN ordering_key") - conn.execute("PRAGMA user_version = 15") - migration_now = "2026-01-03T00:00:00+00:00" - monkeypatch.setenv("TENDWIRE_CONNECTOR_ACK_TTL_SECONDS", "999") - monkeypatch.setattr(store_sqlite, "utc_timestamp", lambda: migration_now) - init_store(db_path, connector_ack_ttl_seconds=123) - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - """ - SELECT delivery_key, turn_id, status, ordering_key, private_state_json - FROM connector_outbox - WHERE delivery_key LIKE 'migration-%' - ORDER BY id - """ - ).fetchall() - delivery_private = conn.execute( - "SELECT private_state_json FROM connector_deliveries WHERE status = 'awaiting_ack'" - ).fetchone()[0] - by_key = {row[0]: row[1:] for row in rows} - tombstoned_key = f"migration-{tombstoned_turn}-awaiting_ack" - dead_letter_key = f"migration-{fallback_turn}-dead_letter" - superseded_key = f"migration-{fallback_turn}-superseded" - assert by_key[tombstoned_key][0:3] == ( - tombstoned_turn, - "awaiting_ack", - stable_key, - ) - ack_deadline = json.loads(by_key[tombstoned_key][3])["ack_deadline_at"] - assert ( - datetime.fromisoformat(ack_deadline) - - datetime.fromisoformat(migration_now) - ).total_seconds() == 123 - assert by_key[dead_letter_key][0:3] == ( - fallback_turn, - "dead_letter", - enqueue_stable_key, - ) - assert by_key[superseded_key][0:3] == ( - fallback_turn, - "superseded", - "worker-fallback", - ) - assert by_key["migration-deleted-a"][2] == f"orphan:{orphan_ids[0]}" - assert by_key["migration-deleted-b"][2] == f"orphan:{orphan_ids[1]}" - assert by_key["migration-deleted-a"][2] != by_key["migration-deleted-b"][2] - assert json.loads(delivery_private)["ack_deadline_at"] == ack_deadline diff --git a/tests/test_daemon.py b/tests/test_daemon.py index 5028fca..e755f29 100644 --- a/tests/test_daemon.py +++ b/tests/test_daemon.py @@ -22,7 +22,6 @@ import pytest from tendwire import __version__ -from tendwire.backends.herdr_turns import TurnIngestionScheduler, TurnRefreshResult from tendwire.cli import main from tendwire.config import Config from tendwire.core.commands import ( @@ -69,12 +68,11 @@ get_command_request, init_store, latest_snapshot, - merge_backend_pending, - merge_turn_content, pending_payload_from_store, save_snapshot, upsert_worker_bindings, ) +from .store_helpers import apply_test_backend_pending, apply_test_turn_refresh _PUBLIC_JSON_FORBIDDEN_KEYS = { @@ -105,6 +103,69 @@ _PUBLIC_JSON_FORBIDDEN_COMPACT = {key.replace("_", "") for key in _PUBLIC_JSON_FORBIDDEN_KEYS} +@pytest.fixture(autouse=True) +def _required_acp_supervisor_for_daemon_unit_tests( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Keep non-ACP daemon tests focused on their own boundary.""" + + class Supervisor: + def __init__(self, config: Config) -> None: + self.config = config + + def start(self) -> None: + assert self.config.db_path is not None + init_store(self.config.db_path) + if latest_snapshot(self.config.db_path, self.config.host_id) is None: + save_snapshot( + self.config.db_path, + Snapshot( + host_id=self.config.host_id, + updated_at="2026-08-04T00:00:00+00:00", + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="empty_healthy", + ) + ], + ), + ) + return None + + def stop(self, *, timeout: float) -> None: + del timeout + + def join(self, *, timeout: float) -> bool: + del timeout + return True + + def status(self) -> dict[str, Any]: + return {"state": "running", "healthy": True} + + def prompt_route(self, _worker: Worker) -> None: + return None + + original_init = TendwireDaemon.__init__ + + def init_with_required_acp(self: TendwireDaemon, *args: Any, **kwargs: Any) -> None: + original_init(self, *args, **kwargs) + hooks = self.hooks + if ( + hooks.acp_supervisor_factory is not None + and getattr(hooks.acp_supervisor_factory, "__name__", "") + == "_default_acp_supervisor_factory" + ): + object.__setattr__( + hooks, + "acp_supervisor_factory", + lambda config, _stop: Supervisor(config), + ) + self._acp_supervisor = Supervisor(self.config) + + monkeypatch.setattr(TendwireDaemon, "__init__", init_with_required_acp) + + def _assert_no_public_json_forbidden(value: Any, path: str = "$") -> None: if isinstance(value, dict): for key, item in value.items(): @@ -243,74 +304,6 @@ def test_daemon_api_required_methods_are_public_safe() -> None: _assert_no_public_json_forbidden(command_response) -def test_daemon_answer_pending_response_is_recursively_public_safe() -> None: - snapshot = _public_snapshot() - request = CommandRequest( - action="answer_pending", - request_id="answer-public", - dry_run=False, - params={ - "pending_id": "pending-" + ("a" * 24), - "pending_fingerprint": "b" * 24, - "choice_id": "choice-" + ("c" * 24), - }, - ) - api = TendwireDaemonAPI( - get_snapshot=lambda: snapshot, - get_health=lambda: {"schema_version": 1, "status": "ok"}, - submit_command=lambda _params: CommandEnvelope.from_result( - request, - ok=True, - status=STATUS_ACCEPTED, - disposition=DISPOSITION_TERMINAL_ACCEPTED, - result={ - "target": { - "worker_id": "worker-public", - "pane_id": "sentinel-private-pane", - "private_binding": "sentinel-private-binding", - }, - "pending": { - "id": "pending-" + ("a" * 24), - "fingerprint": "b" * 24, - "decision_id": "sentinel-private-decision", - }, - "choice": { - "choice_id": "choice-" + ("c" * 24), - "tool_id": "sentinel-private-tool", - "raw_payload": "sentinel-private-option", - }, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - }, - ), - ) - - response = api.dispatch( - { - "method": "command.submit", - "params": request.to_dict(), - } - ) - result = response["result"]["result"] - - assert response["schema_version"] == 1 - assert response["ok"] is True - assert response["result"]["schema_version"] == 2 - assert response["result"]["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - assert result == { - "target": {"worker_id": "worker-public"}, - "pending": { - "id": "pending-" + ("a" * 24), - "fingerprint": "b" * 24, - }, - "choice": {"choice_id": "choice-" + ("c" * 24)}, - "delivery_state": "submitted", - "transport_state": "submitted", - "observed_pending_state": "pending_observation", - } - assert "sentinel-private" not in json.dumps(response, sort_keys=True) - _assert_no_public_json_forbidden(response) @pytest.mark.parametrize( @@ -475,7 +468,7 @@ def test_daemon_pending_matches_shared_durable_projection_and_fingerprint( recompute_pending_content_fingerprint(degraded) != baseline["content_fingerprint"] ) - merge_backend_pending( + apply_test_backend_pending( db_path, snapshot.host_id, "worker-1", @@ -618,7 +611,9 @@ def test_malformed_durable_snapshot_is_fixed_unavailable_for_daemon_and_cli( } assert cli_code == 1 - assert daemon_payload == cli_payload == expected + assert daemon_payload == expected + assert cli_payload["ok"] is False + assert cli_payload["status"] == "daemon_unavailable" assert "sentinel-private" not in json.dumps( {"daemon": daemon_payload, "cli": cli_payload}, sort_keys=True, @@ -661,7 +656,7 @@ def test_pending_store_projection_reads_snapshot_and_overlay_atomically( ) init_store(db_path) save_snapshot(db_path, snapshot_a) - merge_backend_pending( + apply_test_backend_pending( db_path, config.host_id, "worker-1", @@ -693,7 +688,7 @@ def publish_new_view() -> None: try: assert allow_writer.wait(timeout=5) save_snapshot(db_path, snapshot_b) - merge_backend_pending( + apply_test_backend_pending( db_path, config.host_id, "worker-1", @@ -735,6 +730,7 @@ def test_daemon_api_versions_turn_list_and_preserves_exact_content_page() -> Non turn_calls: list[dict[str, Any]] = [] page_calls: list[dict[str, Any]] = [] page_text = "\n " + ("α" * 20_000) + " \r\n" + revision = "twrev1.ueLJtVatFOQxa1UePvWId8C01qdrb05FpW_ipSSPHMM" def get_turns(**params: Any) -> dict[str, Any]: turn_calls.append(dict(params)) @@ -790,7 +786,7 @@ def get_turns(**params: Any) -> dict[str, Any]: "ok": True, "status": "ok", "turn_id": "turn-public", - "content_revision": "twrev1.public", + "content_revision": revision, "field": "assistant_final_text", "availability": "complete", "segment_id": "twseg1.public", @@ -821,7 +817,7 @@ def get_turns(**params: Any) -> dict[str, Any]: "params": { "schema_version": 1, "turn_id": "turn-public", - "content_revision": "twrev1.public", + "content_revision": revision, "field": "assistant_final_text", }, } @@ -861,11 +857,12 @@ def get_turns(**params: Any) -> dict[str, Any]: "since": None, } assert page["result"]["text"] == page_text + assert page["result"]["content_revision"] == revision assert page_calls == [ { "schema_version": 1, "turn_id": "turn-public", - "content_revision": "twrev1.public", + "content_revision": revision, "field": "assistant_final_text", } ] @@ -896,14 +893,8 @@ def test_daemon_turn_list_is_store_projection_only( updated_at="2026-01-01T00:00:00+00:00", ) save_snapshot(db_path, snapshot) - source_calls = 0 projection_calls: list[dict[str, Any]] = [] - def forbidden_source_refresh(*_args: Any, **_kwargs: Any) -> None: - nonlocal source_calls - source_calls += 1 - raise AssertionError("turn source read reached a cached daemon handler") - def project( path: Path, host_id: str, @@ -918,10 +909,6 @@ def project( "turns": [], } - monkeypatch.setattr( - "tendwire.backends.herdr_turns.refresh_structured_turn_content", - forbidden_source_refresh, - ) monkeypatch.setattr("tendwire.store.sqlite.turns_payload_from_store", project) daemon = TendwireDaemon(config) @@ -934,7 +921,6 @@ def project( ) assert result["status"] == "ok" - assert source_calls == 0 assert len(projection_calls) == 3 assert all( call == { @@ -945,8 +931,6 @@ def project( "limit": 17, "cursor": "twlist1.public", "since": None, - "turn_refresh_interval_seconds": 2.0, - "turn_model": config.turn_model, } for call in projection_calls ) @@ -1286,127 +1270,13 @@ def test_attention_recurrence_after_two_complete_misses_re_notifies(tmp_path: Pa assert generation == 2 -def test_socket_daemon_synthesized_fallback_has_no_lifecycle_authority( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "socket-fallback.db" - config = Config(host_id="socket-fallback-host", db_path=db_path, herdr_backend="socket") - base = datetime(2026, 1, 1, tzinfo=timezone.utc) - save_snapshot( - db_path, - project_from_raw( - config, - workers=_blocked_worker("blocked"), - backend_health=_HEALTHY_BACKEND, - timestamp=base, - ), - observation=_complete_observation(base), - ) - - class _HealthyState: - def to_backend_health(self) -> BackendHealth: - return BackendHealth( - name="herdr", - status="healthy", - outcome="empty_healthy", - observed_at=(base + timedelta(seconds=300)).isoformat(), - ) - - class _Backend: - health = _HealthyState() - - def start(self, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - - def stop(self) -> None: - pass - - backend = _Backend() - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(event_backend_factory=lambda _config, _stop_event: backend), - ) - monkeypatch.setattr("tendwire.store.sqlite.latest_snapshot", lambda _path, _host_id: None) - - fallback = daemon._start_socket_event_backend() - - assert fallback.attention == [] - assert len(attention_payload_from_store(db_path, config.host_id)["attention"]) == 1 - assert _attention_outbox_count(db_path, config.host_id) == 1 - with sqlite3.connect(str(db_path)) as conn: - missing_count = conn.execute( - "SELECT missing_observation_count FROM attention_lifecycles WHERE host_id = ?", - (config.host_id,), - ).fetchone()[0] - assert missing_count == 0 - - -@pytest.mark.parametrize( - "observed_at", - ["not-a-timestamp", "2026-01-01T00:00:00"], -) -def test_socket_daemon_fallback_drops_unordered_health_timestamp( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - observed_at: str, -) -> None: - config = Config( - host_id="socket-fallback-invalid-time", - db_path=tmp_path / "socket-fallback-invalid-time.db", - herdr_backend="socket", - ) - captured: list[SnapshotObservationContext] = [] - - class _HealthState: - def to_backend_health(self) -> BackendHealth: - return BackendHealth( - name="herdr", - status="healthy", - outcome="empty_healthy", - observed_at=observed_at, - ) - - class _Backend: - health = _HealthState() - - def start(self, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - - def _capture_save( - _db_path: Path, - _snapshot: Snapshot, - *, - turn_model: str, - observation: SnapshotObservationContext, - ) -> None: - assert turn_model == config.turn_model - captured.append(observation) - - backend = _Backend() - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(event_backend_factory=lambda _config, _stop_event: backend), - ) - monkeypatch.setattr("tendwire.store.sqlite.latest_snapshot", lambda _path, _host_id: None) - monkeypatch.setattr("tendwire.store.sqlite.save_snapshot", _capture_save) - - daemon._start_socket_event_backend() - - assert len(captured) == 1 - assert captured[0].authority == "none" - assert captured[0].observed_at is None - - def test_daemon_health_exposes_public_operational_status_without_private_values(tmp_path: Path) -> None: db_path = tmp_path / "health.db" config = Config( host_id="health-host", db_path=db_path, - event_debounce_seconds=0.2, - reconcile_interval_seconds=0, + reconcile_interval_seconds=2, event_retention_days=3, - output_excerpt_chars=80, max_workers=8, max_outbox_attempts=4, connector_claim_ttl_seconds=33, @@ -1422,7 +1292,6 @@ def test_daemon_health_exposes_public_operational_status_without_private_values( command_retry_horizon_seconds=120, command_receipt_retention_seconds=691_200, command_receipt_retention_count=77, - turn_model="shadow", ) snapshot = project_from_raw( config, @@ -1464,35 +1333,13 @@ def test_daemon_health_exposes_public_operational_status_without_private_values( ), ) - class PrivateSchedulerStatus: - def operational_status(self) -> dict[str, Any]: - return { - "status": "healthy", - "queue_depth": 2, - "active": 1, - "refreshed": 7, - "failed": 3, - "timed_out": 2, - "coalesced": 11, - "queue_full": 5, - "last_success": "2026-01-02T00:00:00+00:00", - "last_duration_ms": 12.5, - "stale_age_seconds": 0.25, - "max_workers": 999, - "queue_capacity": 64, - "refresh_interval_seconds": 999, - "adapter_timeout_seconds": 999, - "private_fingerprint": "sentinel-private-fingerprint", - "error": f"sentinel-private failure at {tmp_path}", - } - daemon = TendwireDaemon(config) - daemon._turn_scheduler = PrivateSchedulerStatus() health = daemon.get_health() encoded = json.dumps(health) assert health["status"] == "ok" - assert health["turn_model"] == "shadow" + assert health["acp"]["required"] is True + assert health["acp"]["healthy"] is True assert health["daemon"]["started_at"] assert health["store"]["counts"]["snapshots"] == 1 assert health["store"]["outbox"]["pending"] == 1 @@ -1537,10 +1384,8 @@ def operational_status(self) -> dict[str, Any]: "backlog": False, } assert health["limits"] == { - "event_debounce_seconds": 0.2, - "reconcile_interval_seconds": 0, + "reconcile_interval_seconds": 2, "event_retention_days": 3, - "output_excerpt_chars": 80, "max_workers": 8, "max_outbox_attempts": 4, "outbox_claim_ttl_seconds": 33, @@ -1556,30 +1401,12 @@ def operational_status(self) -> dict[str, Any]: "snapshot_maintenance_batch_size": 6, "store_maintenance_cadence_seconds": 44, } - assert health["turn_ingestion"] == { - "status": "healthy", - "queue": 2, - "active": 1, - "refreshed": 7, - "failed": 3, - "timed_out": 2, - "coalesced": 11, - "queue_full": 5, - "last_success": "2026-01-02T00:00:00+00:00", - "last_duration_ms": 12.5, - "stale_age": 0.25, - "bounds": { - "refresh_interval_seconds": 2.0, - "max_workers": 4, - "queue_capacity": 64, - "adapter_timeout_seconds": 5.0, - }, - } assert health["pending_ingestion"] == { "status": "healthy", "counts": {"fresh": 0, "stale": 0, "total": 0}, "bounds": {"stale_grace_seconds": 31.0}, } + assert health["backend"]["reconcile_enabled"] is True assert "health.db" not in encoded assert str(tmp_path) not in encoded assert "sentinel-private" not in encoded @@ -2159,22 +1986,6 @@ def capture_signal(signum: int, handler: Any) -> Any: ] -@_UNIX_SOCKET_TEST -def test_service_group_sigterm_exits_closes_socket_and_reaps_child() -> None: - root = Path(__file__).resolve().parents[1] - result = subprocess.run( - ["bash", str(root / "scripts/tendwired_lifecycle_smoke.sh")], - check=False, - capture_output=True, - text=True, - timeout=15, - env={**os.environ, "PYTHON": sys.executable}, - ) - - assert result.returncode == 0, result.stderr - assert result.stdout == "tendwired lifecycle smoke: ok\n" - - def _socket_mode(path: Path) -> int: return stat.S_IMODE(os.lstat(path).st_mode) @@ -2213,6 +2024,64 @@ def _assert_private_daemon_failure( assert value not in rendered +def test_snapshot_maintenance_wires_agent_event_retention_without_socket( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "maintenance-wiring.db" + config = Config( + host_id="daemon-host", + data_dir=tmp_path, + db_path=db_path, + event_retention_days=9, + snapshot_maintenance_batch_size=13, + ) + init_store(db_path) + captured: dict[str, Any] = {} + + def maintenance(path: Path, **kwargs: Any) -> dict[str, Any]: + captured.update({"path": path, **kwargs}) + return { + "schema_version": 1, + "ok": True, + "status": "ok", + "due": False, + "snapshot": { + "examined": 0, + "deleted": 0, + "remaining_candidates": False, + }, + "agent_events": { + "examined": 5, + "deleted": 4, + "remaining_candidates": True, + }, + } + + monkeypatch.setattr( + "tendwire.store.sqlite.maybe_run_automatic_store_maintenance", + maintenance, + ) + daemon = TendwireDaemon(config) + daemon._after_snapshot_saved() + + assert captured["path"] == db_path + assert captured["agent_event_host_id"] == "daemon-host" + assert captured["agent_event_retention_days"] == 9 + assert captured["policy"].batch_size == 13 + assert daemon._automatic_maintenance_status == { + "ok": True, + "status": "ok", + "due": False, + "examined": 0, + "deleted": 0, + "remaining_candidates": False, + "agent_events_examined": 5, + "agent_events_deleted": 4, + "agent_events_remaining_candidates": True, + } + + @_UNIX_SOCKET_TEST def test_cli_snapshot_barrier_checks_maintenance_once_and_reads_do_not( tmp_path: Path, @@ -2234,18 +2103,21 @@ def test_cli_snapshot_barrier_checks_maintenance_once_and_reads_do_not( command_receipt_retention_seconds=691_200, command_receipt_retention_count=77, ) - calls: list[tuple[Path, Any, int, int, int, int, int, int]] = [] + calls: list[ + tuple[Path, Any, str | None, int | None, int, int, int, int, int, int] + ] = [] - def observe(_config: Config) -> Snapshot: + def initialize(path: Path) -> None: + init_store(path) snapshot = _public_snapshot() save_snapshot(db_path, snapshot) - return snapshot def maintenance( path: Path, *, policy: Any, - turn_model: str = "legacy", + agent_event_host_id: str | None = None, + agent_event_retention_days: int | None = None, acknowledged_final_retention_days: int = 30, acknowledged_final_retention_count: int = 4096, command_retry_horizon_seconds: int = 604_800, @@ -2255,11 +2127,12 @@ def maintenance( now: str | None = None, ) -> dict[str, Any]: assert now is None - assert turn_model == config.turn_model calls.append( ( path, policy, + agent_event_host_id, + agent_event_retention_days, acknowledged_final_retention_days, acknowledged_final_retention_count, command_retry_horizon_seconds, @@ -2280,6 +2153,11 @@ def maintenance( "deleted": 0, "remaining_candidates": False, }, + "agent_events": { + "examined": 2, + "deleted": 1, + "remaining_candidates": True, + }, "batch_size": policy.batch_size, } @@ -2289,7 +2167,7 @@ def maintenance( ) daemon = TendwireDaemon( config, - hooks=DaemonHooks(init_store=init_store, observe_initial_snapshot=observe), + hooks=DaemonHooks(init_store=initialize), ) try: daemon.start() @@ -2302,19 +2180,44 @@ def maintenance( daemon.stop() assert len(calls) == 1 - path, policy, final_days, final_count, retry_horizon, retention_seconds, retention_count, cadence = calls[0] + ( + path, + policy, + agent_host, + agent_days, + final_days, + final_count, + retry_horizon, + retention_seconds, + retention_count, + cadence, + ) = calls[0] assert path == db_path assert ( policy.retention_days, policy.retention_count, policy.batch_size, + agent_host, + agent_days, final_days, final_count, retry_horizon, retention_seconds, retention_count, cadence, - ) == (21, 123, 17, 33, 456, 120, 691_200, 77, 91) + ) == ( + 21, + 123, + 17, + "daemon-host", + config.event_retention_days, + 33, + 456, + 120, + 691_200, + 77, + 91, + ) assert health["store"]["maintenance"]["last_check"] == { "ok": True, "status": "not_due", @@ -2322,6 +2225,9 @@ def maintenance( "examined": 0, "deleted": 0, "remaining_candidates": False, + "agent_events_examined": 2, + "agent_events_deleted": 1, + "agent_events_remaining_candidates": True, } @@ -2335,10 +2241,10 @@ def test_cli_snapshot_persists_when_automatic_maintenance_fails( config = Config(host_id="daemon-host", data_dir=data_dir, db_path=db_path) calls = 0 - def observe(_config: Config) -> Snapshot: + def initialize(path: Path) -> None: + init_store(path) snapshot = _public_snapshot() save_snapshot(db_path, snapshot) - return snapshot def maintenance_failure(*_args: Any, **_kwargs: Any) -> dict[str, Any]: nonlocal calls @@ -2351,7 +2257,7 @@ def maintenance_failure(*_args: Any, **_kwargs: Any) -> dict[str, Any]: ) daemon = TendwireDaemon( config, - hooks=DaemonHooks(init_store=init_store, observe_initial_snapshot=observe), + hooks=DaemonHooks(init_store=initialize), ) try: daemon.start() @@ -2373,6 +2279,9 @@ def maintenance_failure(*_args: Any, **_kwargs: Any) -> dict[str, Any]: "examined": 0, "deleted": 0, "remaining_candidates": False, + "agent_events_examined": 0, + "agent_events_deleted": 0, + "agent_events_remaining_candidates": False, } assert str(tmp_path) not in encoded assert "secret.db" not in encoded @@ -2402,10 +2311,7 @@ def test_daemon_default_socket_parent_and_endpoint_are_private_under_umask_zero( ) daemon = TendwireDaemon( config, - hooks=DaemonHooks( - init_store=lambda _path: None, - observe_initial_snapshot=lambda _config: _public_snapshot(), - ), + hooks=DaemonHooks(init_store=lambda _path: None), ) try: @@ -2433,7 +2339,7 @@ def test_daemon_startup_repairs_all_existing_state_before_empty_observation( data_dir.mkdir() os.chmod(data_dir, 0o755) db_path = data_dir / "daemon.db" - db_path.write_bytes(b"existing-database") + init_store(db_path) os.chmod(db_path, 0o644) config = Config(host_id="daemon-host", data_dir=data_dir, db_path=db_path) identity_paths = ( @@ -2444,29 +2350,16 @@ def test_daemon_startup_repairs_all_existing_state_before_empty_observation( for path in identity_paths: path.write_bytes(b"existing-identity") os.chmod(path, 0o644) - observations: list[Snapshot] = [] - def initialize_store(path: Path) -> None: assert path == db_path assert _socket_mode(data_dir) == 0o700 assert _socket_mode(db_path) == 0o600 assert all(_socket_mode(identity_path) == 0o600 for identity_path in identity_paths) - def observe(_config: Config) -> Snapshot: - snapshot = Snapshot( - host_id="daemon-host", - updated_at="2026-01-01T00:00:00+00:00", - ) - observations.append(snapshot) - return snapshot - for _attempt in range(2): daemon = TendwireDaemon( config, - hooks=DaemonHooks( - init_store=initialize_store, - observe_initial_snapshot=observe, - ), + hooks=DaemonHooks(init_store=initialize_store), ) try: daemon.start() @@ -2481,7 +2374,6 @@ def observe(_config: Config) -> Snapshot: finally: daemon.stop() - assert len(observations) == 2 assert not os.path.lexists(data_dir / "tendwire.sock") @@ -2507,16 +2399,9 @@ def initialize_store(_path: Path) -> None: hook_calls.append("init_store") raise AssertionError("store hook must not run") - def observe(_config: Config) -> Snapshot: - hook_calls.append("observe") - raise AssertionError("observation hook must not run") - daemon = TendwireDaemon( Config(host_id="daemon-host", data_dir=data_dir, db_path=db_path), - hooks=DaemonHooks( - init_store=initialize_store, - observe_initial_snapshot=observe, - ), + hooks=DaemonHooks(init_store=initialize_store), ) with pytest.raises(LocalStateError) as caught: @@ -2607,10 +2492,7 @@ def test_daemon_group_socket_and_client_use_exact_shared_mode( ) daemon = TendwireDaemon( config, - hooks=DaemonHooks( - init_store=lambda _path: None, - observe_initial_snapshot=lambda _config: _public_snapshot(), - ), + hooks=DaemonHooks(init_store=lambda _path: None), ) thread: threading.Thread | None = None @@ -2885,10 +2767,7 @@ def test_daemon_rejects_group_sharing_on_implicit_private_parent_before_mutation db_path=tmp_path / "daemon.db", socket_group=group_name, ), - hooks=DaemonHooks( - init_store=lambda _path: None, - observe_initial_snapshot=lambda _config: _public_snapshot(), - ), + hooks=DaemonHooks(init_store=lambda _path: None), ) with pytest.raises(DaemonUnavailable) as caught: @@ -3291,10 +3170,6 @@ def test_daemon_active_socket_fails_before_store_or_backend_work(tmp_path: Path) def forbidden_store(_path: Path) -> None: calls.append("init_store") - def forbidden_observe(_config: Config) -> Snapshot: - calls.append("observe") - raise AssertionError("live-socket guard must precede backend work") - daemon = TendwireDaemon( Config( host_id="fail-fast-host", @@ -3302,10 +3177,7 @@ def forbidden_observe(_config: Config) -> Snapshot: db_path=tmp_path / "fail-fast.db", socket_path=socket_path, ), - hooks=DaemonHooks( - init_store=forbidden_store, - observe_initial_snapshot=forbidden_observe, - ), + hooks=DaemonHooks(init_store=forbidden_store), ) try: with pytest.raises(DaemonUnavailable) as caught: @@ -3564,405 +3436,76 @@ def test_unix_socket_server_close_preserves_substituted_socket(tmp_path: Path) - @_UNIX_SOCKET_TEST -def test_daemon_binds_socket_after_store_observation_before_scheduler_io( - tmp_path: Path, -) -> None: - socket_path = tmp_path / "ordered-cli.sock" - calls: list[str] = [] +def test_daemon_store_startup_failure_never_publishes_socket(tmp_path: Path) -> None: + socket_path = tmp_path / "init-store.sock" - def record_unpublished(stage: str) -> None: + def assert_unpublished() -> None: assert not os.path.lexists(socket_path) - calls.append(stage) def initialize_store(path: Path) -> None: - record_unpublished("init_store") - init_store(path) - - def observe(_config: Config) -> Snapshot: - record_unpublished("observe") - snapshot = _public_snapshot() - save_snapshot(tmp_path / "ordered-cli.db", snapshot) - return snapshot - - class RecordingScheduler: - parent_fd: int | None = None - - def start(self) -> None: - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - import fcntl - - self.parent_fd = os.open( - tmp_path, - os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC, - ) - fcntl.flock(self.parent_fd, fcntl.LOCK_SH) - calls.append("scheduler_start") - - def request_refresh(self) -> None: - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - calls.append("scheduler_request") - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - if self.parent_fd is not None: - os.close(self.parent_fd) - self.parent_fd = None - calls.append(f"scheduler_stop:{flush_timeout_seconds}") - - def scheduler_factory(_config: Config) -> RecordingScheduler: - record_unpublished("scheduler_factory") - return RecordingScheduler() + assert_unpublished() + del path + raise RuntimeError("sentinel startup failure") config = Config( host_id="daemon-host", data_dir=tmp_path, - db_path=tmp_path / "ordered-cli.db", + db_path=tmp_path / "init-store.db", socket_path=socket_path, ) daemon = TendwireDaemon( config, - hooks=DaemonHooks( - init_store=initialize_store, - observe_initial_snapshot=observe, - turn_scheduler_factory=scheduler_factory, - ), + hooks=DaemonHooks(init_store=initialize_store), ) try: - daemon.start() - assert calls == [ - "init_store", - "observe", - "scheduler_factory", - "scheduler_start", - "scheduler_request", - ] - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - _assert_unix_socket_connects(socket_path) + with pytest.raises(RuntimeError, match="sentinel startup failure") as caught: + daemon.start() + + _assert_private_daemon_failure(caught.value, socket_path) + assert not os.path.lexists(socket_path) + with pytest.raises(RuntimeError, match="cannot start after shutdown"): + daemon.start() + assert not os.path.lexists(socket_path) finally: daemon.stop() - assert calls[-1] == "scheduler_stop:6.0" - assert not os.path.lexists(socket_path) - @_UNIX_SOCKET_TEST -def test_daemon_event_callback_is_attached_after_reconcile_before_ingestion( - tmp_path: Path, -) -> None: - socket_path = tmp_path / "ordered-events.sock" - db_path = tmp_path / "ordered-events.db" - calls: list[str] = [] - - def record_unpublished(stage: str) -> None: - assert not os.path.lexists(socket_path) - calls.append(stage) +def test_daemon_starts_persists_serves_and_removes_socket(tmp_path: Path) -> None: + db_path = tmp_path / "daemon.db" + socket_path = tmp_path / "daemon.sock" + config = Config(host_id="daemon-host", data_dir=tmp_path, db_path=db_path, socket_path=socket_path) - def initialize_store(path: Path) -> None: - record_unpublished("init_store") + def initialize(path: Path) -> None: init_store(path) + snapshot = project_from_raw( + config, + workers=[{"id": "worker-1", "name": "Worker One", "status": "active"}], + backend_health=[ + { + "name": "herdr", + "status": "healthy", + "outcome": "healthy_non_empty", + "observed_at": "2026-01-01T00:00:00+00:00", + "counts": {"workers": 1}, + } + ], + ) + save_snapshot(db_path, snapshot) - class RecordingEventBackend: - def __init__(self) -> None: - self.callback: Any | None = None - self.stopped = False - - def start(self, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - record_unpublished("backend_start") - save_snapshot(db_path, _public_snapshot()) - - def set_turn_refresh_callback(self, callback: Any | None) -> None: - if callback is not None: - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - self.callback = callback - calls.append("callback_attached" if callback is not None else "callback_detached") - - def flush(self) -> None: - calls.append("backend_flush") - if self.callback is not None: - self.callback() - - def stop(self) -> None: - calls.append("backend_stop") - self.stopped = True - - class RecordingScheduler: - def start(self) -> None: - assert stat.S_ISSOCK(os.lstat(socket_path).st_mode) - calls.append("scheduler_start") - - def request_refresh(self) -> None: - calls.append("scheduler_request") - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - calls.append(f"scheduler_stop:{flush_timeout_seconds}") - - backend = RecordingEventBackend() - - def event_backend_factory(_config: Config, _stop_event: threading.Event) -> Any: - record_unpublished("backend_factory") - return backend - - def scheduler_factory(_config: Config) -> RecordingScheduler: - record_unpublished("scheduler_factory") - return RecordingScheduler() - - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - init_store=initialize_store, - event_backend_factory=event_backend_factory, - turn_scheduler_factory=scheduler_factory, - ), - ) - - daemon.start() - assert calls == [ - "init_store", - "backend_factory", - "backend_start", - "scheduler_factory", - "callback_attached", - "scheduler_start", - "scheduler_request", - ] - assert backend.callback is not None - backend.callback() - assert calls[-1] == "scheduler_request" - daemon.stop() - daemon.stop() - - assert calls[-5:] == [ - "backend_flush", - "scheduler_request", - "callback_detached", - "scheduler_stop:6.0", - "backend_stop", - ] - assert backend.callback is None - assert backend.stopped is True - assert not os.path.lexists(socket_path) - - -@_UNIX_SOCKET_TEST -@pytest.mark.parametrize("failure_stage", ["init_store", "observe"]) -def test_daemon_startup_failure_never_publishes_socket( - tmp_path: Path, - failure_stage: str, -) -> None: - socket_path = tmp_path / f"{failure_stage}.sock" - - def assert_unpublished() -> None: - assert not os.path.lexists(socket_path) - - def initialize_store(path: Path) -> None: - assert_unpublished() - if failure_stage == "init_store": - raise RuntimeError("sentinel startup failure") - init_store(path) - - def observe(_config: Config) -> Snapshot: - assert_unpublished() - if failure_stage == "observe": - raise RuntimeError("sentinel startup failure") - return _public_snapshot() - - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=tmp_path / f"{failure_stage}.db", - socket_path=socket_path, - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - init_store=initialize_store, - observe_initial_snapshot=observe, - ), - ) - - try: - with pytest.raises(RuntimeError, match="sentinel startup failure") as caught: - daemon.start() - - _assert_private_daemon_failure(caught.value, socket_path) - assert not os.path.lexists(socket_path) - with pytest.raises(RuntimeError, match="cannot start after shutdown"): - daemon.start() - assert not os.path.lexists(socket_path) - finally: - daemon.stop() - - -@_UNIX_SOCKET_TEST -def test_daemon_backend_start_failure_stops_backend_without_publishing_socket( - tmp_path: Path, -) -> None: - socket_path = tmp_path / "backend-failure.sock" - - def assert_unpublished() -> None: - assert not os.path.lexists(socket_path) - - class FailingEventBackend: - def __init__(self) -> None: - self.started = False - self.stopped = False - - def start(self, *, wait_for_reconcile: bool) -> None: - assert wait_for_reconcile is True - assert_unpublished() - self.started = True - raise RuntimeError("sentinel backend startup failure") - - def stop(self) -> None: - self.stopped = True - - backend = FailingEventBackend() - - def initialize_store(path: Path) -> None: - assert_unpublished() - init_store(path) - - def event_backend_factory(_config: Config, _stop_event: threading.Event) -> Any: - assert_unpublished() - return backend - - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=tmp_path / "backend-failure.db", - socket_path=socket_path, - herdr_backend="socket", - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - init_store=initialize_store, - event_backend_factory=event_backend_factory, - ), - ) - - try: - with pytest.raises(RuntimeError, match="sentinel backend startup failure") as caught: - daemon.start() - - assert backend.started is True - assert backend.stopped is True - _assert_private_daemon_failure(caught.value, socket_path) - assert daemon.server is None - assert not os.path.lexists(socket_path) - finally: - daemon.stop() - - -@_UNIX_SOCKET_TEST -def test_daemon_scheduler_start_failure_detaches_callback_and_cleans_components( - tmp_path: Path, -) -> None: - socket_path = tmp_path / "scheduler-failure.sock" - db_path = tmp_path / "scheduler-failure.db" - calls: list[str] = [] - - class Backend: - callback: Any | None = None - - def start(self, *, wait_for_reconcile: bool) -> None: - calls.append("backend_start") - save_snapshot(db_path, _public_snapshot()) - - def set_turn_refresh_callback(self, callback: Any | None) -> None: - self.callback = callback - calls.append("callback_set" if callback is not None else "callback_clear") - - def stop(self) -> None: - calls.append("backend_stop") - - class FailingScheduler: - def request_refresh(self) -> None: - calls.append("scheduler_request") - - def start(self) -> None: - calls.append("scheduler_start") - raise RuntimeError("sentinel scheduler startup failure") - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - calls.append(f"scheduler_stop:{flush_timeout_seconds}") - - backend = Backend() - scheduler = FailingScheduler() - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - event_backend_factory=lambda _config, _stop_event: backend, - turn_scheduler_factory=lambda _config: scheduler, - ), - ) - - with pytest.raises(RuntimeError, match="sentinel scheduler startup failure"): - daemon.start() - - assert calls == [ - "backend_start", - "callback_set", - "scheduler_start", - "callback_clear", - "scheduler_stop:6.0", - "backend_stop", - ] - assert backend.callback is None - assert daemon.server is None - assert daemon._turn_scheduler is None - assert not os.path.lexists(socket_path) - - -def test_daemon_starts_observes_persists_serves_and_removes_socket(tmp_path: Path) -> None: - db_path = tmp_path / "daemon.db" - socket_path = tmp_path / "daemon.sock" - config = Config(host_id="daemon-host", data_dir=tmp_path, db_path=db_path, socket_path=socket_path) - - def observe(config: Config) -> Snapshot: - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "Worker One", "status": "active"}], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-01T00:00:00+00:00", - "counts": {"workers": 1}, - } - ], - ) - save_snapshot(db_path, snapshot) - return snapshot - - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(observe_initial_snapshot=observe), - ) - daemon.start() - thread = threading.Thread(target=daemon.serve_forever) - thread.start() - try: - assert latest_snapshot(db_path, "daemon-host") is not None - ping = DaemonAPIClient(socket_path).request("ping") - snapshot_response = DaemonAPIClient(socket_path).request("snapshot.get") - health_response = DaemonAPIClient(socket_path).request("health.get") + daemon = TendwireDaemon( + config, + hooks=DaemonHooks(init_store=initialize), + ) + daemon.start() + thread = threading.Thread(target=daemon.serve_forever) + thread.start() + try: + assert latest_snapshot(db_path, "daemon-host") is not None + ping = DaemonAPIClient(socket_path).request("ping") + snapshot_response = DaemonAPIClient(socket_path).request("snapshot.get") + health_response = DaemonAPIClient(socket_path).request("health.get") assert ping["ok"] is True assert ping["result"]["pong"] is True @@ -3982,271 +3525,6 @@ def observe(config: Config) -> Snapshot: @_UNIX_SOCKET_TEST -def test_blocked_turn_ingestion_does_not_delay_cached_real_socket_handlers( - tmp_path: Path, -) -> None: - db_path = tmp_path / "blocked-ingestion.db" - socket_path = tmp_path / "blocked-ingestion.sock" - entered = threading.Event() - release = threading.Event() - source_calls: list[str] = [] - worker = Worker(id="worker-1", name="Worker One", status="active") - binding = WorkerBinding( - host_id="daemon-host", - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="sentinel-private-agent", - turn_target_kind="pane_id", - turn_target_value="sentinel-private-pane", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="sentinel-private-binding", - ) - config = Config( - host_id="daemon-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_timeout_seconds=1, - turn_refresh_interval_seconds=3600, - turn_refresh_workers=1, - ) - - def observe(_config: Config) -> Snapshot: - snapshot = Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:00:00+00:00", - workers=[worker], - backend_health=[ - BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - ) - ], - ) - save_snapshot(db_path, snapshot) - upsert_worker_bindings(db_path, [binding]) - return snapshot - - def blocked_reader( - _config: Config, - current: WorkerBinding, - *, - adapter_timeout_seconds: float, - ) -> TurnRefreshResult: - source_calls.append(current.private_fingerprint) - entered.set() - assert release.wait(timeout=10) - return TurnRefreshResult("unchanged", 0) - - scheduler: TurnIngestionScheduler | None = None - - def scheduler_factory(current: Config) -> TurnIngestionScheduler: - nonlocal scheduler - scheduler = TurnIngestionScheduler( - current, - refresh_interval_seconds=3600, - max_workers=1, - reader=blocked_reader, - ) - return scheduler - - command_calls: list[str] = [] - - def submit_command(_config: Config, payload: str) -> CommandEnvelope: - command_calls.append(payload) - return CommandEnvelope( - ok=True, - status="accepted", - action="noop", - result={"accepted": True}, - ) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - observe_initial_snapshot=observe, - submit_command=submit_command, - turn_scheduler_factory=scheduler_factory, - ), - ) - server_thread: threading.Thread | None = None - try: - daemon.start() - server_thread = threading.Thread(target=daemon.serve_forever) - server_thread.start() - assert entered.wait(timeout=10) - client = DaemonAPIClient(socket_path, timeout_seconds=1) - - for _ in range(3): - listed = client.request( - "turn.list", - {"schema_version": 2, "limit": 10, "cursor": None, "since": None}, - ) - health = client.request("health.get") - snapshot = client.request("snapshot.get") - pending = client.request("pending.list") - assert listed["ok"] is True - assert health["result"]["turn_ingestion"]["active"] == 1 - assert snapshot["result"]["host_id"] == config.host_id - assert pending["ok"] is True - - command = client.request( - "command.submit", - {"schema_version": 1, "action": "noop", "dry_run": True}, - ) - assert command["ok"] is True - assert command["result"]["status"] == "accepted" - assert source_calls == ["sentinel-private-binding"] - assert len(command_calls) == 1 - finally: - release.set() - daemon.stop() - if server_thread is not None: - server_thread.join(timeout=2) - - assert scheduler is not None - assert server_thread is not None and not server_thread.is_alive() - assert not os.path.lexists(socket_path) - - -@_UNIX_SOCKET_TEST -def test_daemon_restart_scans_durable_bindings_without_touching_final_or_outbox( - tmp_path: Path, -) -> None: - db_path = tmp_path / "restart.db" - socket_path = tmp_path / "restart.sock" - config = Config( - host_id="restart-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - ) - init_store(db_path) - worker = Worker(id="worker-1", name="Worker One", status="idle") - snapshot = Snapshot( - host_id=config.host_id, - updated_at="2026-01-01T00:00:00+00:00", - workers=[worker], - ) - save_snapshot(db_path, snapshot) - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="sentinel-private-agent", - turn_target_kind="pane_id", - turn_target_value="sentinel-private-pane", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="sentinel-private-binding", - ) - ], - ) - assert merge_turn_content( - db_path, - config.host_id, - worker.id, - { - "source_turn_id": "source-turn-1", - "assistant_final_text": "durable final", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:01:00+00:00", - ) == 1 - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - config.host_id, - "attention", - "durable-delivery", - "queued", - '{"safe":"kept"}', - '{"opaque":"kept"}', - "2026-01-01T00:02:00+00:00", - "2026-01-01T00:02:00+00:00", - ), - ) - before_turns = conn.execute( - "SELECT * FROM turns WHERE host_id = ? ORDER BY turn_id", - (config.host_id,), - ).fetchall() - before_outbox = conn.execute( - "SELECT * FROM connector_outbox WHERE host_id = ? ORDER BY id", - (config.host_id,), - ).fetchall() - - scheduler_calls: list[tuple[str, int]] = [] - - class DurableScanScheduler: - def start(self) -> None: - scheduler_calls.append(("start", 0)) - - def request_refresh(self) -> None: - with sqlite3.connect(str(db_path)) as conn: - count = conn.execute( - "SELECT COUNT(*) FROM worker_bindings WHERE host_id = ?", - (config.host_id,), - ).fetchone()[0] - scheduler_calls.append(("request", int(count))) - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - scheduler_calls.append(("stop", int(flush_timeout_seconds or 0))) - - for _ in range(2): - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - observe_initial_snapshot=lambda _config: latest_snapshot( - db_path, - config.host_id, - ), - turn_scheduler_factory=lambda _config: DurableScanScheduler(), - ), - ) - daemon.start() - daemon.stop() - - with sqlite3.connect(str(db_path)) as conn: - after_turns = conn.execute( - "SELECT * FROM turns WHERE host_id = ? ORDER BY turn_id", - (config.host_id,), - ).fetchall() - after_outbox = conn.execute( - "SELECT * FROM connector_outbox WHERE host_id = ? ORDER BY id", - (config.host_id,), - ).fetchall() - - assert scheduler_calls == [ - ("start", 0), - ("request", 1), - ("stop", 6), - ("start", 0), - ("request", 1), - ("stop", 6), - ] - assert after_turns == before_turns - assert after_outbox == before_outbox - assert not os.path.lexists(socket_path) - - def test_daemon_server_survives_client_disconnect_during_response(tmp_path: Path) -> None: socket_path = tmp_path / "daemon.sock" request_seen = threading.Event() @@ -4669,226 +3947,6 @@ def request_into(target: list[BaseException | dict[str, Any]]) -> None: assert remaining == set() -def test_daemon_concurrent_same_request_id_sends_once_and_replays_accepted( - tmp_path: Path, - monkeypatch, - capsys, -) -> None: - db_path = tmp_path / "commands.db" - socket_path = tmp_path / "commands.sock" - config = Config( - host_id="cmd-host", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - init_store(db_path) - calls: list[dict[str, Any]] = [] - worker = Worker(id="w-1", name="Alpha", status="active") - binding = WorkerBinding( - host_id="cmd-host", - worker_id="w-1", - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - sendable=True, - reason=None, - observed_at="2026-01-01T00:00:00+00:00", - private_fingerprint="private-binding", - ) - - class FakeHealth: - def to_backend_health(self) -> BackendHealth: - return BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - observed_at="2026-01-01T00:00:00+00:00", - counts={"workers": 1}, - ) - - class FakeEventBackend: - health = FakeHealth() - - def __init__(self, config: Config, stop_event: threading.Event) -> None: - self.config = config - - def start(self, *, wait_for_reconcile: bool = True) -> None: - snapshot = Snapshot( - host_id="cmd-host", - updated_at="2026-01-01T00:00:00+00:00", - workers=[worker], - backend_health=[self.health.to_backend_health()], - ) - save_snapshot(db_path, snapshot) - upsert_worker_bindings(db_path, [binding]) - - def stop(self) -> None: - return None - - class FakeHerdrSocketClient: - def connect(self) -> "FakeHerdrSocketClient": - return self - - def request(self, method: str, params: dict[str, Any], *, timeout: float | None = None) -> dict[str, Any]: - calls.append({"method": method, "params": dict(params)}) - if method == "agent.get": - return {"result": {"agent": {"pane_id": "pane-private"}}} - if method == "pane.read": - return { - "type": "pane_read", - "read": {"text": "Completed previous turn.\n── status: idle ──"}, - } - if method == "agent.prompt": - return { - "type": "agent_prompted", - "agent": {"pane_id": "pane-private"}, - "delivery": "submitted", - } - return {"accepted": True} - - def close(self) -> None: - return None - - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - lambda config: FakeHerdrSocketClient(), - ) - from tendwire import command_submission - - real_reserve = command_submission.reserve_command_request - reservation_barrier = threading.Barrier(2, timeout=5) - - def synchronized_reserve(*args: Any, **kwargs: Any) -> dict[str, Any]: - result = real_reserve(*args, **kwargs) - reservation_barrier.wait() - return result - - monkeypatch.setattr( - command_submission, - "reserve_command_request", - synchronized_reserve, - ) - - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(event_backend_factory=lambda config, stop_event: FakeEventBackend(config, stop_event)), - ) - daemon.start() - thread = threading.Thread(target=daemon.serve_forever) - thread.start() - try: - request = { - "schema_version": 1, - "action": "send_instruction", - "request_id": "req-1", - "dry_run": False, - "target": {"worker_id": "w-1"}, - "instruction": {"text": "hello"}, - } - start_barrier = threading.Barrier(3, timeout=5) - results: list[dict[str, Any] | None] = [None, None] - errors: list[BaseException] = [] - - def submit(index: int) -> None: - try: - start_barrier.wait() - results[index] = DaemonAPIClient( - socket_path, - timeout_seconds=5, - ).request("command.submit", request) - except BaseException as exc: # noqa: BLE001 - errors.append(exc) - - clients = [ - threading.Thread(target=submit, args=(index,)) - for index in range(2) - ] - for client_thread in clients: - client_thread.start() - start_barrier.wait() - for client_thread in clients: - client_thread.join(timeout=5) - - assert errors == [] - assert all(not client_thread.is_alive() for client_thread in clients) - responses = [result for result in results if result is not None] - assert len(responses) == 2 - assert all(response["ok"] is True for response in responses) - assert sorted(response["result"]["status"] for response in responses) == [ - STATUS_ACCEPTED, - STATUS_PENDING, - ] - assert sorted( - response["result"]["disposition"] for response in responses - ) == [ - DISPOSITION_IN_PROGRESS, - DISPOSITION_TERMINAL_ACCEPTED, - ] - assert all(response["schema_version"] == 1 for response in responses) - assert all(response["result"]["schema_version"] == 2 for response in responses) - assert calls == [ - {"method": "agent.get", "params": {"target": "agent-private"}}, - {"method": "agent.get", "params": {"target": "agent-private"}}, - { - "method": "agent.prompt", - "params": { - "target": "agent-private", - "text": "hello", - "wait": {"until": ["working"], "timeout_ms": 5000}, - }, - }, - ] - assert not any(call["method"] == "pane.send_keys" for call in calls) - receipt = get_command_request(db_path, "cmd-host", "req-1") - assert receipt is not None - assert receipt["state"] == "accepted" - assert receipt["status"] == STATUS_ACCEPTED - assert receipt["terminal_at"] is not None - - monkeypatch.setattr( - command_submission, - "reserve_command_request", - real_reserve, - ) - replay = DaemonAPIClient(socket_path).request("command.submit", request) - assert replay["ok"] is True - assert replay["result"]["status"] == STATUS_ACCEPTED - assert replay["result"]["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "cli-state")) - monkeypatch.setattr("sys.stdin", io.StringIO(json.dumps(request))) - cli_code = main( - [ - "--host-id", - "cmd-host", - "--socket-path", - str(socket_path), - "command", - "--json", - "--db-path", - str(db_path), - ] - ) - captured = capsys.readouterr() - cli_result = json.loads(captured.out) - assert cli_code == 0 - assert captured.err == "" - assert cli_result == replay["result"] - assert cli_result["disposition"] == DISPOSITION_TERMINAL_ACCEPTED - _assert_no_public_json_forbidden(cli_result) - assert len([call for call in calls if call["method"] == "agent.prompt"]) == 1 - for response in [*responses, replay]: - encoded = json.dumps(response) - assert "agent-private" not in encoded - assert "pane-private" not in encoded - _assert_no_public_json_forbidden(response) - finally: - daemon.stop() - thread.join(timeout=2) - - def test_daemon_command_submit_rejects_blank_request_id_before_mutation( tmp_path: Path, monkeypatch, @@ -4898,19 +3956,10 @@ def test_daemon_command_submit_rejects_blank_request_id_before_mutation( host_id="cmd-host", data_dir=tmp_path, db_path=db_path, - herdr_backend="socket", ) init_store(db_path) calls: list[str] = [] - def guarded_socket_factory(config: Config) -> Any: - calls.append("socket") - raise AssertionError("invalid request_id must not construct Herdr socket client") - - monkeypatch.setattr( - "tendwire.command_submission._default_socket_client_factory", - guarded_socket_factory, - ) daemon = TendwireDaemon(config) request = { "schema_version": 1, @@ -4938,7 +3987,7 @@ def guarded_socket_factory(config: Config) -> Any: assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 0 assert conn.execute("SELECT COUNT(*) FROM events").fetchone()[0] == 0 -def test_cli_snapshot_falls_back_when_configured_socket_is_absent( +def test_cli_snapshot_requires_daemon_when_configured_socket_is_absent( tmp_path: Path, capsys, monkeypatch, @@ -4949,11 +3998,6 @@ def test_cli_snapshot_falls_back_when_configured_socket_is_absent( monkeypatch.setenv("TENDWIRE_DATA_DIR", os.fspath(data_dir)) monkeypatch.delenv("TENDWIRE_DB_PATH", raising=False) - def fake_state(config: Config) -> tuple[list[Any], list[Worker]]: - return [], [Worker(id="fallback-worker", name="Fallback", status="active")] - - monkeypatch.setattr("tendwire.cli.fetch_herdr_state", fake_state) - code = main( [ "--host-id", @@ -4967,9 +4011,11 @@ def fake_state(config: Config) -> tuple[list[Any], list[Worker]]: captured = capsys.readouterr() payload = json.loads(captured.out) - assert code == 0 + assert code == 1 assert captured.err == "" - assert payload["workers"][0]["id"] == "fallback-worker" + assert payload["ok"] is False + assert payload["status"] == "daemon_unavailable" + assert not (data_dir / "tendwire.db").exists() def test_cli_command_falls_back_when_configured_socket_is_stale( @@ -5526,7 +4572,6 @@ def test_isolated_daemon_survives_deterministic_real_wal_retirement_without_reso data_dir=tmp_path, db_path=db_path, socket_path=socket_path, - turn_refresh_interval_seconds=3600, acknowledged_final_retention_days=36500, ) worker = Worker(id="worker-race", name="Worker Race", status="active") @@ -5564,7 +4609,7 @@ def test_isolated_daemon_survives_deterministic_real_wal_retirement_without_reso ) ], ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, config.host_id, worker.id, @@ -5655,27 +4700,6 @@ def churn_wal() -> None: except threading.BrokenBarrierError: pass - scheduler_calls: list[str] = [] - - class NoopScheduler: - def start(self) -> None: - scheduler_calls.append("start") - - def request_refresh(self) -> None: - scheduler_calls.append("request") - - def stop(self, *, flush_timeout_seconds: float | None = None) -> None: - del flush_timeout_seconds - scheduler_calls.append("stop") - - def operational_status(self) -> dict[str, Any]: - return { - "status": "healthy", - "queue_depth": 0, - "active": 0, - "queue_capacity": 1, - } - def direct_child_processes() -> set[int]: children: set[int] = set() for task in (Path("/proc/self/task")).iterdir(): @@ -5710,16 +4734,7 @@ def fd_targets() -> dict[str, tuple[str, int, int, int]]: baseline_threads = {id(thread) for thread in threading.enumerate()} baseline_children = direct_child_processes() main_identity = (db_path.stat().st_dev, db_path.stat().st_ino) - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - observe_initial_snapshot=lambda _config: latest_snapshot( - db_path, - config.host_id, - ), - turn_scheduler_factory=lambda _config: NoopScheduler(), - ), - ) + daemon = TendwireDaemon(config) server_thread: threading.Thread | None = None writer_thread = threading.Thread(target=churn_wal) writer_started = False @@ -5797,7 +4812,6 @@ def fd_targets() -> dict[str, tuple[str, int, int, int]]: assert phase_calls == [ ("captured", LocalStateKind.DATABASE_WAL) ] * cycle_count - assert scheduler_calls == ["start", "request", "stop"] assert len(responses) == cycle_count * 3 assert (db_path.stat().st_dev, db_path.stat().st_ino) == main_identity assert not os.path.lexists(socket_path) diff --git a/tests/test_daemon_acp.py b/tests/test_daemon_acp.py new file mode 100644 index 0000000..89b1a4e --- /dev/null +++ b/tests/test_daemon_acp.py @@ -0,0 +1,202 @@ +"""Required ACP lifecycle contract tests for the Tendwire daemon.""" + +from __future__ import annotations + +import json +import threading +from pathlib import Path +from typing import Any + +import pytest + +from tendwire.config import Config +from tendwire.core.models import BackendHealth, Snapshot +from tendwire.daemon import DaemonHooks, TendwireDaemon +from tendwire.daemon_api import UnixSocketJSONServer +from tendwire.store.sqlite import init_store, save_snapshot + + +def _snapshot() -> Snapshot: + return Snapshot( + host_id="daemon-host", + updated_at="2026-08-04T00:00:00+00:00", + backend_health=[ + BackendHealth( + name="herdr", + status="healthy", + outcome="empty_healthy", + ) + ], + ) + + +class _Supervisor: + def __init__( + self, + calls: list[str], + *, + healthy: bool = True, + start_failure: BaseException | None = None, + ) -> None: + self.calls = calls + self.healthy = healthy + self.start_failure = start_failure + + def start(self) -> None: + self.calls.append("acp_start") + if self.start_failure is not None: + raise self.start_failure + + def status(self) -> dict[str, Any]: + return { + "state": "running" if self.healthy else "failed", + "healthy": self.healthy, + "updates_ingested": 7, + "permissions_ingested": 2, + "prompts_completed": 3, + "failure_type": None if self.healthy else "AcpTransportError", + "argv": ["sentinel-private-command"], + "session_id": "sentinel-private-session", + } + + def stop(self, *, timeout: float) -> None: + self.calls.append(f"acp_stop:{timeout}") + + def join(self, *, timeout: float) -> bool: + self.calls.append(f"acp_join:{timeout}") + return True + + +def _config(tmp_path: Path) -> Config: + tmp_path.chmod(0o700) + return Config( + host_id="daemon-host", + data_dir=tmp_path, + db_path=tmp_path / "daemon.db", + socket_path=tmp_path / "daemon.sock", + acp_shutdown_timeout_seconds=1.25, + ) + + +def _hooks( + config: Config, + supervisor_factory: Any, + calls: list[str], +) -> DaemonHooks: + assert config.db_path is not None + + def initialize(path: Path) -> None: + calls.append("init_store") + init_store(path) + snapshot = _snapshot() + save_snapshot(config.db_path, snapshot) + + return DaemonHooks( + init_store=initialize, + acp_supervisor_factory=supervisor_factory, + ) + + +def test_daemon_binds_then_removes_socket_when_acp_supervisor_is_missing( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = _config(tmp_path) + calls: list[str] = [] + original_start = UnixSocketJSONServer.start + + def record_bind(server: Any) -> None: + calls.append("socket_bind") + original_start(server) + + monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", record_bind) + daemon = TendwireDaemon(config, hooks=_hooks(config, None, calls)) + + with pytest.raises(RuntimeError, match="ACP supervisor is required"): + daemon.start() + + assert not config.socket_path.exists() + assert calls == ["init_store", "socket_bind"] + + +def test_daemon_starts_required_acp_and_exposes_only_public_health( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + config = _config(tmp_path) + calls: list[str] = [] + + def record_bind(_server: Any) -> None: + calls.append("socket_bind") + + monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", record_bind) + supervisor = _Supervisor(calls) + daemon = TendwireDaemon( + config, + hooks=_hooks(config, lambda _config, _stop: supervisor, calls), + ) + + daemon.start() + try: + health = daemon.get_health() + assert health["status"] == "ok" + assert health["acp"]["required"] is True + assert health["acp"]["healthy"] is True + assert health["acp"]["state"] == "running" + assert health["acp"]["counters"]["updates_ingested"] == 7 + assert "sentinel-private" not in json.dumps(health) + assert calls[:3] == ["init_store", "socket_bind", "acp_start"] + finally: + daemon.stop() + + assert calls[-2:] == ["acp_stop:1.25", "acp_join:1.25"] + + +@pytest.mark.parametrize( + "supervisor", + [ + pytest.param(_Supervisor([], healthy=False), id="unhealthy"), + pytest.param( + _Supervisor([], start_failure=OSError("private transport detail")), + id="start-failure", + ), + ], +) +def test_daemon_fails_closed_when_required_acp_cannot_start( + tmp_path: Path, + supervisor: _Supervisor, +) -> None: + config = _config(tmp_path) + calls = supervisor.calls + daemon = TendwireDaemon( + config, + hooks=_hooks(config, lambda _config, _stop: supervisor, calls), + ) + + with pytest.raises(RuntimeError, match="ACP supervisor is required"): + daemon.start() + + assert not config.socket_path.exists() + assert any(call.startswith("acp_stop:") for call in calls) + assert "private transport detail" not in repr(daemon._acp_startup_failure_type) + + +def test_supervisor_receives_daemon_stop_event( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr("tendwire.daemon.UnixSocketJSONServer.start", lambda _self: None) + config = _config(tmp_path) + calls: list[str] = [] + seen: list[threading.Event] = [] + + def factory(_config: Config, stop_event: threading.Event) -> _Supervisor: + seen.append(stop_event) + return _Supervisor(calls) + + daemon = TendwireDaemon(config, hooks=_hooks(config, factory, calls)) + daemon.start() + try: + assert seen == [daemon.stop_event] + finally: + daemon.stop() diff --git a/tests/test_delivery_retention.py b/tests/test_delivery_retention.py index a39babc..75b7423 100644 --- a/tests/test_delivery_retention.py +++ b/tests/test_delivery_retention.py @@ -19,12 +19,12 @@ from tendwire.store.sqlite import ( cleanup_acknowledged_final_retention, init_store, - merge_turn_content, reclaim_expired_connector_leases, save_snapshot, store_status, turns_payload_from_store, ) +from .store_helpers import apply_test_turn_refresh HOST_ID = "retention-host" @@ -91,7 +91,7 @@ def _merge_final( observed_at: str, host_id: str = HOST_ID, ) -> dict[str, Any]: - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, WORKER_ID, @@ -369,7 +369,7 @@ def test_dead_letter_final_blocks_no_later_finals(tmp_path: Path) -> None: final_text="first owner later final", observed_at="2026-01-01T00:01:00+00:00", ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, second_worker_id, @@ -696,7 +696,7 @@ def test_new_authoritative_revision_supersedes_stale_lease_without_double_send( _finish_source(api, current_source, version="retention-current-revision") assert _anchor_state(db_path, current_source["key"])[0] == "delivered" - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -708,7 +708,7 @@ def test_new_authoritative_revision_supersedes_stale_lease_without_double_send( }, observed_at="2026-01-03T00:00:00+00:00", ) == 0 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -803,7 +803,7 @@ def test_concurrent_final_merges_create_unique_anchors_drained_in_durable_order( def merge_one(index: int) -> int: barrier.wait(timeout=10) - return merge_turn_content( + return apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, diff --git a/tests/test_delivery_retention_hardening.py b/tests/test_delivery_retention_hardening.py index fff2239..d6b8dea 100644 --- a/tests/test_delivery_retention_hardening.py +++ b/tests/test_delivery_retention_hardening.py @@ -18,13 +18,13 @@ cleanup_acknowledged_final_retention, init_store, inspect_connector_outbox, - merge_turn_content, maybe_run_automatic_store_maintenance, SnapshotRetentionPolicy, reclaim_expired_connector_leases, save_snapshot, store_status, ) +from .store_helpers import apply_test_turn_refresh FINAL_NAME = "turn-final" @@ -303,7 +303,7 @@ def _deliver_final( text: str, observed_at: str, ) -> str: - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -534,7 +534,7 @@ def test_turn_final_fail_and_defer_keep_public_reason_contract_private( db_path = tmp_path / "turn-final-reason-codes.db" host_id = "reason-code-host" _snapshot, api = _new_delivery_store(db_path, host_id) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -1065,7 +1065,7 @@ def owner_snapshot(stable_key: str, worker_id: str, space_id: str, second: int) worker_b = owner_snapshot(STABLE_KEY, "worker-tombstone-b", "space-tombstone-b", 2) assert worker_b.workers[0].fingerprint != worker_a.workers[0].fingerprint assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-tombstone-b", @@ -1105,7 +1105,7 @@ def owner_snapshot(stable_key: str, worker_id: str, space_id: str, second: int) init_store(db_path) assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-tombstone-b", @@ -1149,7 +1149,7 @@ def owner_snapshot(stable_key: str, worker_id: str, space_id: str, second: int) 4, ) assert save_snapshot(db_path, owner_k2) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-tombstone-b", diff --git a/tests/test_delivery_retention_migration.py b/tests/test_delivery_retention_migration.py deleted file mode 100644 index 6f1e468..0000000 --- a/tests/test_delivery_retention_migration.py +++ /dev/null @@ -1,1792 +0,0 @@ -"""Behavioral coverage for the schema-v11 final-delivery migration.""" - -from __future__ import annotations - -import json -import sqlite3 -from datetime import datetime, timezone -from pathlib import Path -from typing import Any - -import pytest - -from tendwire.config import Config -from tendwire.core.projector import project_from_raw -from tendwire.connectors import ConnectorOutboxAPI -from tendwire.store import sqlite as store_sqlite -from tendwire.store.sqlite import init_store, merge_turn_content, save_snapshot - - -_HOST_ID = "host-migration" -_CREATED_AT = "2026-01-01T00:00:00+00:00" -_RAW_USER_MARKER = "private-user-chat_id-telegram" -_RAW_FINAL_MARKER = "private-final-bot_token-herdres" -_PRIVATE_ROUTE_MARKER = "private-route-topic_id" -_V11_OUTBOX_COLUMNS = {"delivery_kind", "turn_id", "content_revision"} -_STABLE_KEY = "wsk1_" + ("c" * 64) -_STABLE_KEY_2 = "wsk1_" + ("d" * 64) -_HISTORICAL_RAW_SOURCE = "legacy-migration-source" -_HISTORICAL_SOURCE_TOKEN = "turnsrc-251c9c4adc3e3ad33ceb344b" -_HISTORICAL_TURN_ID = "turn-8f770eac5cd028d511a2b0f1" -_HISTORICAL_REVISION = ( - "twrev1.kiyTUuBR3pSWUjD5Vke4eAao7SB9y2r3P6P9eRzsPO8" -) -_HISTORICAL_FINAL_IDENTITY = ( - "twfinal1.j3LZtRHojHQKn8Fmae_OMdQ1TCunF51DpO_EYzxmDP4" -) -_HISTORICAL_FINAL_KEY = ( - "turn-final:revision:twfinal1.j3LZtRHojHQKn8Fmae_OMdQ1TCunF51DpO_EYzxmDP4" -) -_HISTORICAL_USER_TEXT = "migration continuity prompt" -_HISTORICAL_FINAL_TEXT = "migration continuity final" -_HISTORICAL_PART_OUTBOX_ID = 4101 -_HISTORICAL_ROOT_OUTBOX_ID = 4102 -_HISTORICAL_PLAN_ID = 5101 -_HISTORICAL_JOB_ID = 6101 -_HISTORICAL_DELIVERY_ID = 7101 -_HISTORICAL_LIST_SEQUENCE = 41 - - -def _columns(conn: sqlite3.Connection, table: str) -> set[str]: - return {str(row[1]) for row in conn.execute(f"PRAGMA table_info({table})")} - - -def _assert_v10_shape(conn: sqlite3.Connection) -> None: - assert int(conn.execute("PRAGMA user_version").fetchone()[0]) == 10 - assert _V11_OUTBOX_COLUMNS.isdisjoint(_columns(conn, "connector_outbox")) - assert "source_outbox_id" not in _columns(conn, "turn_presentation_plans") - - -def _create_v10_store(db_path: Path) -> None: - """Create the historical schema through migrations, then restore v10 table shapes. - - The migration registry creates every earlier schema programmatically. Current - CREATE constants contain additive v11 columns, so the two affected empty - tables are put back into their exact pre-v11 shape before fixture rows are - inserted. - """ - - with store_sqlite._connect( - db_path, - prepare=True, - isolation_level=None, - ) as conn: - store_sqlite._run_migrations(conn, target_version=10) - - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys = OFF") - conn.execute("DROP TABLE turn_presentation_jobs") - conn.execute("DROP TABLE turn_presentation_plans") - for column in sorted(_V11_OUTBOX_COLUMNS): - conn.execute(f"ALTER TABLE connector_outbox DROP COLUMN {column}") - conn.executescript( - """ - CREATE TABLE turn_presentation_plans ( - id INTEGER PRIMARY KEY, - host_id TEXT NOT NULL, - name TEXT NOT NULL, - plan_token TEXT NOT NULL, - turn_id TEXT NOT NULL, - content_revision TEXT NOT NULL, - presentation_version TEXT NOT NULL, - generation INTEGER NOT NULL DEFAULT 1 CHECK (generation >= 1), - part_count INTEGER NOT NULL CHECK (part_count > 0), - state TEXT NOT NULL CHECK (state IN ( - 'preparing', - 'waiting_predecessor', - 'active', - 'completed', - 'superseded', - 'failed' - )), - replaces_plan_token TEXT, - recovers_plan_token TEXT, - created_at TEXT NOT NULL, - activated_at TEXT, - completed_at TEXT, - UNIQUE (host_id, name, plan_token), - UNIQUE ( - host_id, - name, - turn_id, - content_revision, - presentation_version, - generation - ), - FOREIGN KEY (host_id, turn_id, content_revision) - REFERENCES turn_content_revisions( - host_id, - turn_id, - content_revision - ) ON DELETE RESTRICT - ); - CREATE TABLE turn_presentation_jobs ( - id INTEGER PRIMARY KEY, - plan_id INTEGER NOT NULL, - sequence_index INTEGER NOT NULL CHECK (sequence_index >= 0), - operation TEXT NOT NULL CHECK (operation IN ('upsert', 'retire')), - part_ordinal INTEGER NOT NULL CHECK (part_ordinal >= 0), - spans_json TEXT NOT NULL, - outbox_id INTEGER UNIQUE, - created_at TEXT NOT NULL, - UNIQUE (plan_id, sequence_index), - UNIQUE (plan_id, operation, part_ordinal), - FOREIGN KEY (plan_id) - REFERENCES turn_presentation_plans(id) ON DELETE CASCADE, - FOREIGN KEY (outbox_id) - REFERENCES connector_outbox(id) ON DELETE RESTRICT - ); - CREATE INDEX idx_turn_presentation_jobs_plan_sequence - ON turn_presentation_jobs(plan_id, sequence_index); - CREATE INDEX idx_turn_presentation_jobs_outbox - ON turn_presentation_jobs(outbox_id); - PRAGMA user_version = 10; - """ - ) - conn.execute("PRAGMA foreign_keys = ON") - _assert_v10_shape(conn) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - -def _insert_current_final( - conn: sqlite3.Connection, - *, - turn_id: str, - user_text: str, - final_text: str, -) -> str: - revision = store_sqlite.content_revision( - turn_id, - user_text, - final_text, - "complete", - "complete", - ) - user_segments = store_sqlite.segment_canonical_text(user_text) - final_segments = store_sqlite.segment_canonical_text(final_text) - conn.execute( - """ - INSERT INTO turns ( - host_id, - turn_id, - worker_id, - worker_fingerprint, - space_id, - status, - kind, - updated_at, - fingerprint, - snapshot_content_fingerprint, - observed_at, - payload_json, - list_sequence - ) VALUES ( - ?, ?, ?, NULL, NULL, 'complete', 'turn', ?, ?, ?, ?, ?, - (SELECT COALESCE(MAX(list_sequence), 0) + 1 FROM turns WHERE host_id = ?) - ) - """, - ( - _HOST_ID, - turn_id, - f"worker-{turn_id}", - _CREATED_AT, - f"fingerprint-{turn_id}", - f"snapshot-{turn_id}", - _CREATED_AT, - json.dumps( - { - "source_turn_id": f"source-{turn_id}", - "complete": True, - "meta": { - "stable_key": _STABLE_KEY, - "stable_key_version": 1, - }, - "chat_id": _PRIVATE_ROUTE_MARKER, - }, - sort_keys=True, - ), - _HOST_ID, - ), - ) - conn.execute( - """ - INSERT INTO turn_content_revisions ( - host_id, - turn_id, - content_revision, - user_text, - assistant_final_text, - user_state, - final_state, - user_char_length, - user_byte_length, - final_char_length, - final_byte_length, - user_page_count, - final_page_count, - is_current, - created_at, - superseded_at - ) VALUES (?, ?, ?, ?, ?, 'complete', 'complete', ?, ?, ?, ?, ?, ?, 1, ?, NULL) - """, - ( - _HOST_ID, - turn_id, - revision, - user_text, - final_text, - len(user_text), - len(user_text.encode("utf-8")), - len(final_text), - len(final_text.encode("utf-8")), - len(user_segments), - len(final_segments), - _CREATED_AT, - ), - ) - for field, segments in ( - ("user_text", user_segments), - ("assistant_final_text", final_segments), - ): - conn.executemany( - """ - INSERT INTO turn_content_page_boundaries ( - host_id, - turn_id, - content_revision, - field, - page_index, - start_char, - start_byte - ) VALUES (?, ?, ?, ?, ?, ?, ?) - """, - ( - ( - _HOST_ID, - turn_id, - revision, - field, - int(segment.index), - int(segment.start_char), - int(segment.start_byte), - ) - for segment in segments - ), - ) - return revision - - -def _seed_v10_finals(db_path: Path) -> dict[str, tuple[str, str]]: - finals: dict[str, tuple[str, str]] = {} - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys = ON") - for ordinal, label in enumerate(("delivered", "hold-a", "hold-b"), start=1): - turn_id = f"turn-{ordinal:02d}-{label}" - revision = _insert_current_final( - conn, - turn_id=turn_id, - user_text=f"{_RAW_USER_MARKER}-{label}", - final_text=f"{_RAW_FINAL_MARKER}-{label}", - ) - finals[label] = (turn_id, revision) - - delivered_turn, delivered_revision = finals["delivered"] - part_payload = { - "schema_version": 1, - "operation": "upsert", - "sequence_index": 0, - "spans": [ - { - "field": "user_text", - "start_char": 0, - "end_char": len(f"{_RAW_USER_MARKER}-delivered"), - }, - { - "field": "assistant_final_text", - "start_char": 0, - "end_char": len(f"{_RAW_FINAL_MARKER}-delivered"), - }, - ], - } - part_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, - connector, - delivery_key, - status, - payload_json, - private_state_json, - created_at, - updated_at, - next_attempt_at - ) VALUES (?, 'turn-final', ?, 'delivered', ?, ?, ?, ?, NULL) - """, - ( - _HOST_ID, - "turn-final:legacy-completed:000000", - json.dumps(part_payload, sort_keys=True), - json.dumps({"route": _PRIVATE_ROUTE_MARKER}, sort_keys=True), - _CREATED_AT, - _CREATED_AT, - ), - ) - part_outbox_id = int(part_cursor.lastrowid) - plan_cursor = conn.execute( - """ - INSERT INTO turn_presentation_plans ( - host_id, - name, - plan_token, - turn_id, - content_revision, - presentation_version, - generation, - part_count, - state, - replaces_plan_token, - recovers_plan_token, - created_at, - activated_at, - completed_at - ) VALUES ( - ?, 'turn-final', 'twplan1.legacy-completed', ?, ?, - 'turn-present-v10', 1, 1, 'completed', NULL, NULL, ?, ?, ? - ) - """, - ( - _HOST_ID, - delivered_turn, - delivered_revision, - _CREATED_AT, - _CREATED_AT, - _CREATED_AT, - ), - ) - plan_id = int(plan_cursor.lastrowid) - conn.execute( - """ - INSERT INTO turn_presentation_jobs ( - plan_id, - sequence_index, - operation, - part_ordinal, - spans_json, - outbox_id, - created_at - ) VALUES (?, 0, 'upsert', 0, ?, ?, ?) - """, - ( - plan_id, - json.dumps(part_payload["spans"], sort_keys=True), - part_outbox_id, - _CREATED_AT, - ), - ) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, - host_id, - connector, - delivery_key, - attempt, - status, - response_json, - private_state_json, - created_at, - delivered_at - ) VALUES (?, ?, 'turn-final', ?, 1, 'delivered', '{}', '{}', ?, ?) - """, - ( - part_outbox_id, - _HOST_ID, - "turn-final:legacy-completed:000000", - _CREATED_AT, - _CREATED_AT, - ), - ) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - return finals - - -def _seed_literal_v10_continuity_final( - db_path: Path, - *, - delivered_proof: bool, -) -> None: - """Insert one frozen pre-owner-continuity graph without constructing a Turn.""" - - attempt_status = "delivered" if delivered_proof else "failed" - delivered_at = _CREATED_AT if delivered_proof else None - turn_payload = { - "schema_version": 1, - "id": _HISTORICAL_TURN_ID, - "host_id": _HOST_ID, - "worker_id": "worker-a", - "worker_fingerprint": "worker-a-fingerprint", - "space_id": "space-a", - "status": "done", - "kind": "task", - "title": "Historical migration worker", - "summary": "Historical migration continuity", - "source": "snapshot", - "updated_at": _CREATED_AT, - "complete": True, - "has_open_turn": False, - "source_turn_id": _HISTORICAL_SOURCE_TOKEN, - "fingerprint": "historical-turn-fingerprint", - "meta": { - "stable_key": _STABLE_KEY, - "stable_key_version": 1, - }, - } - part_payload = { - "schema_version": 1, - "operation": "upsert", - "sequence_index": 0, - "spans": [ - { - "field": "user_text", - "start_char": 0, - "end_char": len(_HISTORICAL_USER_TEXT), - }, - { - "field": "assistant_final_text", - "start_char": 0, - "end_char": len(_HISTORICAL_FINAL_TEXT), - }, - ], - } - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys = ON") - conn.execute( - """ - INSERT INTO turn_list_hosts ( - host_id, next_sequence, traversal_generation - ) VALUES (?, ?, 1) - """, - (_HOST_ID, _HISTORICAL_LIST_SEQUENCE + 1), - ) - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, worker_fingerprint, space_id, - status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, - list_sequence - ) VALUES (?, ?, 'worker-a', 'worker-a-fingerprint', 'space-a', - 'done', 'task', ?, 'historical-turn-fingerprint', - 'historical-snapshot-fingerprint', ?, ?, ?) - """, - ( - _HOST_ID, - _HISTORICAL_TURN_ID, - _CREATED_AT, - _CREATED_AT, - json.dumps(turn_payload, sort_keys=True), - _HISTORICAL_LIST_SEQUENCE, - ), - ) - conn.execute( - """ - INSERT INTO turn_content_revisions ( - host_id, turn_id, content_revision, user_text, - assistant_final_text, user_state, final_state, - user_char_length, user_byte_length, final_char_length, - final_byte_length, user_page_count, final_page_count, - is_current, created_at, superseded_at - ) VALUES (?, ?, ?, ?, ?, 'complete', 'complete', ?, ?, ?, ?, - 1, 1, 1, ?, NULL) - """, - ( - _HOST_ID, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - _HISTORICAL_USER_TEXT, - _HISTORICAL_FINAL_TEXT, - len(_HISTORICAL_USER_TEXT), - len(_HISTORICAL_USER_TEXT.encode("utf-8")), - len(_HISTORICAL_FINAL_TEXT), - len(_HISTORICAL_FINAL_TEXT.encode("utf-8")), - _CREATED_AT, - ), - ) - conn.executemany( - """ - INSERT INTO turn_content_page_boundaries ( - host_id, turn_id, content_revision, field, page_index, - start_char, start_byte - ) VALUES (?, ?, ?, ?, 0, 0, 0) - """, - ( - ( - _HOST_ID, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - "user_text", - ), - ( - _HOST_ID, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - "assistant_final_text", - ), - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - id, host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES (?, ?, 'turn-final', - 'turn-final:legacy-continuity:000000', 'delivered', - ?, '{"historical_part":true}', ?, ?, NULL) - """, - ( - _HISTORICAL_PART_OUTBOX_ID, - _HOST_ID, - json.dumps(part_payload, sort_keys=True), - _CREATED_AT, - _CREATED_AT, - ), - ) - conn.execute( - """ - INSERT INTO turn_presentation_plans ( - id, host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - replaces_plan_token, recovers_plan_token, created_at, - activated_at, completed_at - ) VALUES (?, ?, 'turn-final', 'twplan1.literal-continuity', - ?, ?, 'turn-present-literal-v10', 1, 1, 'completed', - NULL, NULL, ?, ?, ?) - """, - ( - _HISTORICAL_PLAN_ID, - _HOST_ID, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - _CREATED_AT, - _CREATED_AT, - _CREATED_AT, - ), - ) - conn.execute( - """ - INSERT INTO turn_presentation_jobs ( - id, plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - ) VALUES (?, ?, 0, 'upsert', 0, ?, ?, ?) - """, - ( - _HISTORICAL_JOB_ID, - _HISTORICAL_PLAN_ID, - json.dumps(part_payload["spans"], sort_keys=True), - _HISTORICAL_PART_OUTBOX_ID, - _CREATED_AT, - ), - ) - conn.execute( - """ - INSERT INTO connector_deliveries ( - id, outbox_id, host_id, connector, delivery_key, attempt, - status, response_json, private_state_json, created_at, - delivered_at - ) VALUES (?, ?, ?, 'turn-final', - 'turn-final:legacy-continuity:000000', 3, ?, - '{"literal_response":true}', - '{"literal_attempt_state":true}', ?, ?) - """, - ( - _HISTORICAL_DELIVERY_ID, - _HISTORICAL_PART_OUTBOX_ID, - _HOST_ID, - attempt_status, - _CREATED_AT, - delivered_at, - ), - ) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - -def _continuity_snapshot( - db_path: Path, - *, - worker_id: str, - space_id: str, - stable_key: str, - second: int, -): - return project_from_raw( - Config(host_id=_HOST_ID, db_path=db_path), - workers=[ - { - "id": worker_id, - "name": f"Continuity worker {worker_id}", - "status": "done", - "space_id": space_id, - "meta": { - "stable_key": stable_key, - "stable_key_version": 1, - }, - } - ], - timestamp=datetime(2026, 1, 1, 0, 0, second, tzinfo=timezone.utc), - ) - - -def _observe_literal_continuity_final( - db_path: Path, - *, - worker_id: str, - space_id: str, - stable_key: str, - second: int, -) -> int: - snapshot = _continuity_snapshot( - db_path, - worker_id=worker_id, - space_id=space_id, - stable_key=stable_key, - second=second, - ) - assert save_snapshot(db_path, snapshot) is True - return merge_turn_content( - db_path, - _HOST_ID, - worker_id, - { - "source_turn_id": _HISTORICAL_RAW_SOURCE, - "user_text": _HISTORICAL_USER_TEXT, - "assistant_final_text": _HISTORICAL_FINAL_TEXT, - "complete": True, - "has_open_turn": False, - }, - observed_at=f"2026-01-01T00:00:{second + 1:02d}+00:00", - ) - - -def _historical_graph(db_path: Path) -> dict[str, list[tuple[Any, ...]]]: - with sqlite3.connect(str(db_path)) as conn: - turn_row = conn.execute( - """ - SELECT turn_id, list_sequence, payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchone() - assert turn_row is not None - payload = json.loads(str(turn_row[2])) - return { - "turn_identity": [ - ( - str(turn_row[0]), - int(turn_row[1]), - str(payload.get("id")), - str(payload.get("source_turn_id")), - json.dumps(payload.get("meta"), sort_keys=True), - ) - ], - "revisions": conn.execute( - """ - SELECT turn_id, content_revision, user_text, - assistant_final_text, user_state, final_state, - is_current, created_at, superseded_at - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? - ORDER BY content_revision - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "boundaries": conn.execute( - """ - SELECT turn_id, content_revision, field, page_index, - start_char, start_byte - FROM turn_content_page_boundaries - WHERE host_id = ? AND turn_id = ? - ORDER BY field, page_index - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "plans": conn.execute( - """ - SELECT id, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - replaces_plan_token, recovers_plan_token, - source_outbox_id, created_at, activated_at, completed_at - FROM turn_presentation_plans - WHERE host_id = ? AND turn_id = ? - ORDER BY id - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "jobs": conn.execute( - """ - SELECT jobs.id, jobs.plan_id, jobs.sequence_index, - jobs.operation, jobs.part_ordinal, jobs.spans_json, - jobs.outbox_id, jobs.created_at - FROM turn_presentation_jobs AS jobs - JOIN turn_presentation_plans AS plans ON plans.id = jobs.plan_id - WHERE plans.host_id = ? AND plans.turn_id = ? - ORDER BY jobs.id - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "recoveries": conn.execute( - """ - SELECT recoveries.* - FROM turn_presentation_recoveries AS recoveries - JOIN turn_presentation_plans AS plans - ON plans.id = recoveries.failed_plan_id - WHERE plans.host_id = ? AND plans.turn_id = ? - ORDER BY recoveries.id - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchall(), - "outbox": conn.execute( - """ - SELECT id, delivery_key, delivery_kind, turn_id, - content_revision, status, payload_json, - private_state_json, created_at, updated_at, - next_attempt_at - FROM connector_outbox - WHERE id IN (?, ?) - ORDER BY id - """, - (_HISTORICAL_PART_OUTBOX_ID, _HISTORICAL_ROOT_OUTBOX_ID), - ).fetchall(), - "attempts": conn.execute( - """ - SELECT id, outbox_id, delivery_key, attempt, status, - response_json, private_state_json, created_at, - delivered_at - FROM connector_deliveries - WHERE id = ? - """, - (_HISTORICAL_DELIVERY_ID,), - ).fetchall(), - } - - -def _historical_root(db_path: Path) -> tuple[Any, ...]: - with sqlite3.connect(str(db_path)) as conn: - row = conn.execute( - """ - SELECT id, delivery_key, delivery_kind, turn_id, - content_revision, status, private_state_json, - next_attempt_at - FROM connector_outbox - WHERE id = ? - """, - (_HISTORICAL_ROOT_OUTBOX_ID,), - ).fetchone() - assert row is not None - return tuple(row) - - -def _assert_historical_route( - db_path: Path, - *, - worker_id: str, - worker_fingerprint: str, - space_id: str, -) -> None: - with sqlite3.connect(str(db_path)) as conn: - row = conn.execute( - """ - SELECT worker_id, worker_fingerprint, space_id, payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchone() - assert row is not None - payload = json.loads(str(row[3])) - assert tuple(row[:3]) == (worker_id, worker_fingerprint, space_id) - assert payload["id"] == _HISTORICAL_TURN_ID - assert payload["source_turn_id"] == _HISTORICAL_SOURCE_TOKEN - assert payload["worker_id"] == worker_id - assert payload["worker_fingerprint"] == worker_fingerprint - assert payload["space_id"] == space_id - - -def _final_key(turn_id: str, revision: str) -> str: - identity = store_sqlite.turn_final_delivery_identity( - _HOST_ID, - turn_id, - revision, - ) - return f"turn-final:revision:{identity}" - - -def _assert_no_private_or_raw(value: Any) -> None: - encoded = json.dumps(value, sort_keys=True).lower() - for forbidden in ( - _RAW_USER_MARKER, - _RAW_FINAL_MARKER, - _PRIVATE_ROUTE_MARKER, - "chat_id", - "topic_id", - "bot_token", - "telegram", - "herdres", - "private_state_json", - ): - assert forbidden.lower() not in encoded - - -def _anchor_rows(db_path: Path) -> list[tuple[Any, ...]]: - with sqlite3.connect(str(db_path)) as conn: - return conn.execute( - """ - SELECT - id, - delivery_key, - delivery_kind, - turn_id, - content_revision, - status, - payload_json, - private_state_json, - created_at, - updated_at, - next_attempt_at - FROM connector_outbox - WHERE connector = 'turn-final' - AND delivery_kind IN ('final_ready', 'final_migration_hold') - ORDER BY id - """ - ).fetchall() - - -def _database_dump(db_path: Path) -> tuple[int, str]: - with sqlite3.connect(str(db_path)) as conn: - version = int(conn.execute("PRAGMA user_version").fetchone()[0]) - return version, "\n".join(conn.iterdump()) - - -def test_v10_to_v12_migration_retains_finals_without_reposting_or_leaking( - tmp_path: Path, -) -> None: - db_path = tmp_path / "retention-v10.db" - _create_v10_store(db_path) - finals = _seed_v10_finals(db_path) - - init_store(db_path) - assert store_sqlite.STORE_SCHEMA_VERSION == 21 - - delivered_key = _final_key(*finals["delivered"]) - hold_keys = {_final_key(*finals[label]) for label in ("hold-a", "hold-b")} - rows_after_first_migration = _anchor_rows(db_path) - by_key = {str(row[1]): row for row in rows_after_first_migration} - assert set(by_key) == {delivered_key, *hold_keys} - assert by_key[delivered_key][2:6] == ( - "final_ready", - finals["delivered"][0], - finals["delivered"][1], - "delivered", - ) - for label in ("hold-a", "hold-b"): - key = _final_key(*finals[label]) - assert by_key[key][2:6] == ( - "final_migration_hold", - finals[label][0], - finals[label][1], - "dead_letter", - ) - for row in rows_after_first_migration: - assert row[1].startswith("turn-final:revision:twfinal1.") - root_payload = json.loads(str(row[6])) - assert root_payload["schema_version"] == 2 - assert root_payload["stable_key"] == _STABLE_KEY - assert root_payload["stable_key_version"] == 1 - assert "worker_fingerprint" not in root_payload - _assert_no_private_or_raw(root_payload) - - with sqlite3.connect(str(db_path)) as conn: - delivered_source_id = int(by_key[delivered_key][0]) - assert conn.execute( - """ - SELECT state, source_outbox_id - FROM turn_presentation_plans - WHERE plan_token = 'twplan1.legacy-completed' - """ - ).fetchone() == ("completed", delivered_source_id) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - init_store(db_path) - assert _anchor_rows(db_path) == rows_after_first_migration - - api = ConnectorOutboxAPI(db_path, _HOST_ID) - assert api.poll({"name": "turn-final", "limit": 100})["items"] == [] - inspected = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert inspected["ok"] is True - assert inspected["total"] == 2 - assert {item["key"] for item in inspected["items"]} == hold_keys - _assert_no_private_or_raw(inspected) - - with sqlite3.connect(str(db_path)) as conn: - new_turn = "turn-04-new-ready" - new_revision = _insert_current_final( - conn, - turn_id=new_turn, - user_text=f"{_RAW_USER_MARKER}-new", - final_text=f"{_RAW_FINAL_MARKER}-new", - ) - new_anchor_id = store_sqlite._ensure_final_ready_anchor_conn( - conn, - host_id=_HOST_ID, - turn_id=new_turn, - content_revision_value=new_revision, - now="2026-01-01T00:01:00+00:00", - ) - assert new_anchor_id is not None - new_key = _final_key(new_turn, new_revision) - - new_work = api.poll({"name": "turn-final", "limit": 100}) - assert [item["key"] for item in new_work["items"]] == [new_key] - _assert_no_private_or_raw(new_work) - assert delivered_key not in {item["key"] for item in new_work["items"]} - - selected_key = _final_key(*finals["hold-a"]) - untouched_key = _final_key(*finals["hold-b"]) - retried = api.retry( - { - "schema_version": 1, - "name": "turn-final", - "key": selected_key, - } - ) - assert retried["ok"] is True - assert retried["status"] == "requeued" - assert retried["key"] == selected_key - _assert_no_private_or_raw(retried) - - with sqlite3.connect(str(db_path)) as conn: - retry_states = dict( - conn.execute( - """ - SELECT delivery_key, delivery_kind || ':' || status - FROM connector_outbox - WHERE delivery_key IN (?, ?) - """, - (selected_key, untouched_key), - ).fetchall() - ) - assert retry_states == { - selected_key: "final_ready:queued", - untouched_key: "final_migration_hold:dead_letter", - } - - after_retry_inspect = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert after_retry_inspect["total"] == 1 - assert [item["key"] for item in after_retry_inspect["items"]] == [untouched_key] - _assert_no_private_or_raw(after_retry_inspect) - - released = api.poll({"name": "turn-final", "limit": 100}) - assert [item["key"] for item in released["items"]] == [selected_key] - _assert_no_private_or_raw(released) - assert delivered_key not in {item["key"] for item in released["items"]} - - -@pytest.mark.parametrize( - "invalid_meta", - [ - {}, - {"stable_key": _STABLE_KEY}, - {"stable_key": "wsk1_invalid", "stable_key_version": 1}, - {"stable_key": _STABLE_KEY, "stable_key_version": True}, - {"stable_key": _STABLE_KEY, "stable_key_version": 2}, - ], -) -def test_v10_to_v12_missing_stable_key_pair_becomes_nonroutable_hold( - tmp_path: Path, - invalid_meta: dict[str, object], -) -> None: - db_path = tmp_path / "retention-v10-stable-key-hold.db" - _create_v10_store(db_path) - finals = _seed_v10_finals(db_path) - turn_id, revision = finals["delivered"] - with sqlite3.connect(str(db_path)) as conn: - raw_payload = conn.execute( - """ - SELECT payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (_HOST_ID, turn_id), - ).fetchone()[0] - turn_payload = json.loads(str(raw_payload)) - turn_payload["meta"] = invalid_meta - conn.execute( - """ - UPDATE turns - SET payload_json = ? - WHERE host_id = ? AND turn_id = ? - """, - (json.dumps(turn_payload, sort_keys=True), _HOST_ID, turn_id), - ) - - init_store(db_path) - key = _final_key(turn_id, revision) - row = { - str(anchor[1]): anchor - for anchor in _anchor_rows(db_path) - }[key] - payload = json.loads(str(row[6])) - - assert row[2:6] == ( - "final_migration_hold", - turn_id, - revision, - "dead_letter", - ) - assert payload["schema_version"] == 1 - assert "stable_key" not in payload - assert "stable_key_version" not in payload - assert "worker_fingerprint" not in payload - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT source_outbox_id - FROM turn_presentation_plans - WHERE plan_token = 'twplan1.legacy-completed' - """ - ).fetchone() == (None,) - api = ConnectorOutboxAPI(db_path, _HOST_ID) - assert api.poll({"name": "turn-final", "limit": 100})["items"] == [] - inspected = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert key in {item.get("key") for item in inspected["items"]} - _assert_no_private_or_raw(inspected) - - -@pytest.mark.parametrize( - "proof_gap", - [ - "declared_part_missing", - "delivered_attempt_missing", - "delivered_attempt_contradicted", - "ack_time_missing", - "foreign_host_part", - ], -) -def test_v10_to_v12_migration_requires_complete_host_bound_ack_proof( - tmp_path: Path, - proof_gap: str, -) -> None: - db_path = tmp_path / f"retention-v10-{proof_gap}.db" - _create_v10_store(db_path) - finals = _seed_v10_finals(db_path) - with sqlite3.connect(str(db_path)) as conn: - if proof_gap == "declared_part_missing": - conn.execute( - """ - UPDATE turn_presentation_plans - SET part_count = 2 - WHERE plan_token = 'twplan1.legacy-completed' - """ - ) - elif proof_gap == "delivered_attempt_missing": - conn.execute( - """ - DELETE FROM connector_deliveries - WHERE delivery_key = 'turn-final:legacy-completed:000000' - """ - ) - elif proof_gap == "delivered_attempt_contradicted": - conn.execute( - """ - UPDATE connector_deliveries - SET status = 'failed', delivered_at = NULL - WHERE delivery_key = 'turn-final:legacy-completed:000000' - """ - ) - elif proof_gap == "ack_time_missing": - conn.execute( - """ - UPDATE turn_presentation_plans - SET completed_at = NULL - WHERE plan_token = 'twplan1.legacy-completed' - """ - ) - else: - conn.execute( - """ - UPDATE connector_outbox - SET host_id = 'foreign-host' - WHERE delivery_key = 'turn-final:legacy-completed:000000' - """ - ) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - init_store(db_path) - delivered_key = _final_key(*finals["delivered"]) - migrated = { - str(row[1]): row - for row in _anchor_rows(db_path) - }[delivered_key] - assert migrated[2:6] == ( - "final_migration_hold", - finals["delivered"][0], - finals["delivered"][1], - "dead_letter", - ) - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT source_outbox_id - FROM turn_presentation_plans - WHERE plan_token = 'twplan1.legacy-completed' - """ - ).fetchone() == (None,) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - api = ConnectorOutboxAPI(db_path, _HOST_ID) - assert api.poll({"name": "turn-final", "limit": 100})["items"] == [] - inspected = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert delivered_key in {item.get("key") for item in inspected["items"]} - _assert_no_private_or_raw(inspected) - -@pytest.mark.parametrize("owner_matches", [True, False]) -def test_v10_failed_plan_links_only_with_exact_immutable_job_route( - tmp_path: Path, - owner_matches: bool, -) -> None: - db_path = tmp_path / f"retention-v10-owner-{owner_matches}.db" - _create_v10_store(db_path) - turn_id = "turn-legacy-owner" - with sqlite3.connect(str(db_path)) as conn: - revision = _insert_current_final( - conn, - turn_id=turn_id, - user_text="legacy owner prompt", - final_text="legacy owner final", - ) - final_identity = store_sqlite.turn_final_delivery_identity( - _HOST_ID, - turn_id, - revision, - ) - route = { - "schema_version": 2, - "turn_id": turn_id, - "content_revision": revision, - "final_identity": final_identity, - "stable_key": _STABLE_KEY if owner_matches else "wsk1_" + ("b" * 64), - "stable_key_version": 1, - } - outbox_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES (?, 'turn-final', 'turn-final:legacy-owner:000000', - 'dead_letter', ?, '{}', ?, ?, NULL) - """, - ( - _HOST_ID, - json.dumps({"turn": route}, sort_keys=True), - _CREATED_AT, - _CREATED_AT, - ), - ) - plan_cursor = conn.execute( - """ - INSERT INTO turn_presentation_plans ( - host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - created_at, activated_at - ) VALUES (?, 'turn-final', 'twplan1.legacy-owner', ?, ?, - 'legacy-owner-v1', 1, 1, 'failed', ?, ?) - """, - (_HOST_ID, turn_id, revision, _CREATED_AT, _CREATED_AT), - ) - conn.execute( - """ - INSERT INTO turn_presentation_jobs ( - plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - ) VALUES (?, 0, 'upsert', 0, '[]', ?, ?) - """, - (int(plan_cursor.lastrowid), int(outbox_cursor.lastrowid), _CREATED_AT), - ) - - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - root = conn.execute( - """ - SELECT id, delivery_kind, status - FROM connector_outbox - WHERE delivery_kind IN ('final_ready', 'final_migration_hold') - """ - ).fetchone() - source_outbox_id = conn.execute( - """ - SELECT source_outbox_id - FROM turn_presentation_plans - WHERE plan_token = 'twplan1.legacy-owner' - """ - ).fetchone()[0] - - assert root is not None - if owner_matches: - assert root[1:3] == ("final_ready", "awaiting_ack") - assert source_outbox_id == root[0] - else: - assert root[1:3] == ("final_migration_hold", "dead_letter") - assert source_outbox_id is None - - - -def test_v10_to_v12_migration_failure_rolls_back_the_entire_transition( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "retention-v10-rollback.db" - _create_v10_store(db_path) - _seed_v10_finals(db_path) - before = _database_dump(db_path) - - real_payload = store_sqlite._final_ready_payload_conn - calls = 0 - - def fail_after_first_anchor(*args: Any, **kwargs: Any) -> dict[str, Any] | None: - nonlocal calls - calls += 1 - if calls == 2: - raise RuntimeError("controlled v11 migration failure") - return real_payload(*args, **kwargs) - - monkeypatch.setattr( - store_sqlite, - "_final_ready_payload_conn", - fail_after_first_anchor, - ) - - with pytest.raises(RuntimeError, match="controlled v11 migration failure"): - init_store(db_path) - - assert calls == 2 - assert _database_dump(db_path) == before - with sqlite3.connect(str(db_path)) as conn: - _assert_v10_shape(conn) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - assert conn.execute( - "SELECT COUNT(*) FROM connector_outbox WHERE connector = 'turn-final'" - ).fetchone()[0] == 1 - - -@pytest.mark.parametrize( - ( - "root_state", - "delivered_proof", - "expected_kind", - "expected_status", - "expected_attempt_status", - "expected_source_link", - ), - [ - ("queued", False, "final_ready", "queued", "failed", None), - ( - "delivered", - True, - "final_ready", - "delivered", - "delivered", - _HISTORICAL_ROOT_OUTBOX_ID, - ), - ( - "migration-hold", - False, - "final_migration_hold", - "dead_letter", - "failed", - None, - ), - ], -) -def test_v10_to_v12_history_stays_immutable_when_observed_turn_arrives( - tmp_path: Path, - root_state: str, - delivered_proof: bool, - expected_kind: str, - expected_status: str, - expected_attempt_status: str, - expected_source_link: int | None, -) -> None: - db_path = tmp_path / f"literal-v10-continuity-{root_state}.db" - _create_v10_store(db_path) - _seed_literal_v10_continuity_final( - db_path, - delivered_proof=delivered_proof, - ) - - init_store(db_path) - assert _historical_root(db_path)[:6] == ( - _HISTORICAL_ROOT_OUTBOX_ID, - _HISTORICAL_FINAL_KEY, - ( - "final_ready" - if delivered_proof - else "final_migration_hold" - ), - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - "delivered" if delivered_proof else "dead_letter", - ) - with sqlite3.connect(str(db_path)) as conn: - root_payload = json.loads( - str( - conn.execute( - "SELECT payload_json FROM connector_outbox WHERE id = ?", - (_HISTORICAL_ROOT_OUTBOX_ID,), - ).fetchone()[0] - ) - ) - assert root_payload["final_identity"] == _HISTORICAL_FINAL_IDENTITY - assert root_payload["turn_id"] == _HISTORICAL_TURN_ID - assert root_payload["content_revision"] == _HISTORICAL_REVISION - assert root_payload["stable_key"] == _STABLE_KEY - assert root_payload["stable_key_version"] == 1 - assert conn.execute( - """ - SELECT source_outbox_id - FROM turn_presentation_plans - WHERE id = ? - """, - (_HISTORICAL_PLAN_ID,), - ).fetchone() == (expected_source_link,) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - api = ConnectorOutboxAPI(db_path, _HOST_ID) - if root_state == "queued": - retried = api.retry( - { - "schema_version": 1, - "name": "turn-final", - "key": _HISTORICAL_FINAL_KEY, - } - ) - assert retried["ok"] is True - assert retried["status"] == "requeued" - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = ?, - next_attempt_at = ? - WHERE id = ? - """, - ( - json.dumps( - {"literal_root_state": root_state}, - sort_keys=True, - separators=(",", ":"), - ), - _CREATED_AT if root_state == "queued" else None, - _HISTORICAL_ROOT_OUTBOX_ID, - ), - ) - - expected_root = _historical_root(db_path) - assert expected_root[2:6] == ( - expected_kind, - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - expected_status, - ) - assert expected_root[7] == ( - _CREATED_AT if root_state == "queued" else None - ) - expected_graph = _historical_graph(db_path) - assert expected_graph["turn_identity"] == [ - ( - _HISTORICAL_TURN_ID, - _HISTORICAL_LIST_SEQUENCE, - _HISTORICAL_TURN_ID, - _HISTORICAL_SOURCE_TOKEN, - json.dumps( - { - "stable_key": _STABLE_KEY, - "stable_key_version": 1, - }, - sort_keys=True, - ), - ) - ] - assert expected_graph["revisions"][0][1] == _HISTORICAL_REVISION - assert expected_graph["plans"][0][0] == _HISTORICAL_PLAN_ID - assert expected_graph["plans"][0][1] == "twplan1.literal-continuity" - assert expected_graph["plans"][0][10] == expected_source_link - assert expected_graph["jobs"][0][0] == _HISTORICAL_JOB_ID - assert expected_graph["jobs"][0][6] == _HISTORICAL_PART_OUTBOX_ID - assert expected_graph["attempts"][0][:5] == ( - _HISTORICAL_DELIVERY_ID, - _HISTORICAL_PART_OUTBOX_ID, - "turn-final:legacy-continuity:000000", - 3, - expected_attempt_status, - ) - assert expected_graph["recoveries"] == [] - - worker_b = _continuity_snapshot( - db_path, - worker_id="worker-b", - space_id="space-b", - stable_key=_STABLE_KEY, - second=2, - ) - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-b", - space_id="space-b", - stable_key=_STABLE_KEY, - second=2, - ) == 1 - _assert_historical_route( - db_path, - worker_id="worker-a", - worker_fingerprint="worker-a-fingerprint", - space_id="space-a", - ) - assert _historical_root(db_path) == expected_root - assert _historical_graph(db_path) == expected_graph - with sqlite3.connect(str(db_path)) as conn: - observed = conn.execute( - """ - SELECT turn_id, worker_id, worker_fingerprint, space_id, - json_extract(payload_json, '$.source_turn_id') - FROM turns - WHERE host_id = ? AND turn_id != ? - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchone() - assert observed is not None - observed_turn_id = str(observed[0]) - assert tuple(observed[1:4]) == ( - "worker-b", - worker_b.workers[0].fingerprint, - "space-b", - ) - assert str(observed[4]) != _HISTORICAL_SOURCE_TOKEN - observed_root = conn.execute( - """ - SELECT delivery_key, delivery_kind, status - FROM connector_outbox - WHERE turn_id = ? - """, - (observed_turn_id,), - ).fetchone() - assert observed_root is not None - observed_key = str(observed_root[0]) - assert tuple(observed_root[1:]) == ("final_ready", "queued") - - init_store(db_path) - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-b", - space_id="space-b", - stable_key=_STABLE_KEY, - second=4, - ) == 0 - assert _historical_root(db_path) == expected_root - assert _historical_graph(db_path) == expected_graph - - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT COUNT(*) - FROM connector_outbox - WHERE delivery_kind IN ('final_ready', 'final_migration_hold') - AND turn_id = ? - AND content_revision = ? - """, - (_HISTORICAL_TURN_ID, _HISTORICAL_REVISION), - ).fetchone() == (1,) - assert conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - """, - (_HOST_ID, _HISTORICAL_TURN_ID), - ).fetchone() == (1,) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - polled = api.poll({"name": "turn-final", "limit": 100}) - assert [item["key"] for item in polled["items"]] == ( - [_HISTORICAL_FINAL_KEY] - if root_state == "queued" - else [observed_key] - ) - if root_state == "migration-hold": - inspected = api.inspect( - { - "schema_version": 1, - "name": "turn-final", - "status": "dead_letter", - "limit": 100, - } - ) - assert [item["key"] for item in inspected["items"]] == [ - _HISTORICAL_FINAL_KEY - ] - - -def test_v10_migration_hold_stays_immutable_under_observed_k2( - tmp_path: Path, -) -> None: - db_path = tmp_path / "literal-v10-continuity-k2.db" - _create_v10_store(db_path) - _seed_literal_v10_continuity_final(db_path, delivered_proof=False) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - UPDATE connector_outbox - SET private_state_json = '{"literal_root_state":"migration-hold"}' - WHERE id = ? - """, - (_HISTORICAL_ROOT_OUTBOX_ID,), - ) - - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-b", - space_id="space-b", - stable_key=_STABLE_KEY, - second=2, - ) == 1 - historical_graph = _historical_graph(db_path) - historical_root = _historical_root(db_path) - assert historical_root[2:6] == ( - "final_migration_hold", - _HISTORICAL_TURN_ID, - _HISTORICAL_REVISION, - "dead_letter", - ) - - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-c", - space_id="space-c", - stable_key=_STABLE_KEY_2, - second=4, - ) == 1 - assert _historical_graph(db_path) == historical_graph - assert _historical_root(db_path) == historical_root - - with sqlite3.connect(str(db_path)) as conn: - roots = [ - ( - int(root_id), - str(delivery_key), - str(delivery_kind), - str(turn_id), - str(status), - json.loads(str(payload_json)), - ) - for ( - root_id, - delivery_key, - delivery_kind, - turn_id, - status, - payload_json, - ) in conn.execute( - """ - SELECT id, delivery_key, delivery_kind, turn_id, status, - payload_json - FROM connector_outbox - WHERE delivery_kind IN ('final_ready', 'final_migration_hold') - ORDER BY id - """ - ).fetchall() - ] - source_turns = [ - (str(turn_id), json.loads(str(payload_json))) - for turn_id, payload_json in conn.execute( - """ - SELECT turn_id, payload_json - FROM turns - WHERE host_id = ? - ORDER BY turn_id - """, - (_HOST_ID,), - ).fetchall() - if json.loads(str(payload_json)).get("source_turn_id") - ] - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - assert len(roots) == 3 - historical = next(root for root in roots if root[0] == _HISTORICAL_ROOT_OUTBOX_ID) - assert historical[:5] == ( - _HISTORICAL_ROOT_OUTBOX_ID, - _HISTORICAL_FINAL_KEY, - "final_migration_hold", - _HISTORICAL_TURN_ID, - "dead_letter", - ) - observed_roots = [root for root in roots if root[0] != _HISTORICAL_ROOT_OUTBOX_ID] - by_owner = {root[5]["stable_key"]: root for root in observed_roots} - assert set(by_owner) == {_STABLE_KEY, _STABLE_KEY_2} - for owner_key in (_STABLE_KEY, _STABLE_KEY_2): - assert by_owner[owner_key][2] == "final_ready" - assert by_owner[owner_key][3] != _HISTORICAL_TURN_ID - assert by_owner[owner_key][4] == "queued" - assert len(source_turns) == 3 - historical_source = next( - row for row in source_turns if row[0] == _HISTORICAL_TURN_ID - ) - assert historical_source[1]["source_turn_id"] == _HISTORICAL_SOURCE_TOKEN - observed_source_by_owner = { - turn_payload["meta"]["stable_key"]: (turn_id, turn_payload) - for turn_id, turn_payload in source_turns - if turn_id != _HISTORICAL_TURN_ID - } - assert set(observed_source_by_owner) == {_STABLE_KEY, _STABLE_KEY_2} - assert all( - turn_payload["source_turn_id"] != _HISTORICAL_SOURCE_TOKEN - for _turn_id, turn_payload in observed_source_by_owner.values() - ) - same_owner_key = str(by_owner[_STABLE_KEY][1]) - new_key = str(by_owner[_STABLE_KEY_2][1]) - same_owner_turn_id = str(observed_source_by_owner[_STABLE_KEY][0]) - new_turn_id = str(observed_source_by_owner[_STABLE_KEY_2][0]) - source_identities = sorted( - ( - turn_id, - str(turn_payload["source_turn_id"]), - str(turn_payload["meta"]["stable_key"]), - ) - for turn_id, turn_payload in source_turns - ) - expected_source_revisions = { - (_HISTORICAL_TURN_ID, _HISTORICAL_REVISION), - ( - same_owner_turn_id, - str(by_owner[_STABLE_KEY][5]["content_revision"]), - ), - ( - new_turn_id, - str(by_owner[_STABLE_KEY_2][5]["content_revision"]), - ), - } - - init_store(db_path) - assert _observe_literal_continuity_final( - db_path, - worker_id="worker-c", - space_id="space-c", - stable_key=_STABLE_KEY_2, - second=6, - ) in {0, 1} - assert _historical_graph(db_path) == historical_graph - assert _historical_root(db_path) == historical_root - with sqlite3.connect(str(db_path)) as conn: - roots_after_restart = [ - ( - int(root_id), - str(delivery_key), - str(delivery_kind), - str(turn_id), - str(status), - json.loads(str(payload_json)), - ) - for ( - root_id, - delivery_key, - delivery_kind, - turn_id, - status, - payload_json, - ) in conn.execute( - """ - SELECT id, delivery_key, delivery_kind, turn_id, status, - payload_json - FROM connector_outbox - WHERE delivery_kind IN ('final_ready', 'final_migration_hold') - ORDER BY id - """ - ).fetchall() - ] - source_identities_after_restart = sorted( - ( - str(turn_id), - str(source_turn_id), - str(stable_key), - ) - for turn_id, source_turn_id, stable_key in conn.execute( - """ - SELECT - turn_id, - json_extract(payload_json, '$.source_turn_id'), - json_extract(payload_json, '$.meta.stable_key') - FROM turns - WHERE host_id = ? - AND json_extract(payload_json, '$.source_turn_id') IS NOT NULL - ORDER BY turn_id - """, - (_HOST_ID,), - ).fetchall() - ) - source_revisions_after_restart = { - (str(turn_id), str(content_revision)) - for turn_id, content_revision in conn.execute( - """ - SELECT turns.turn_id, revisions.content_revision - FROM turns - JOIN turn_content_revisions AS revisions - ON revisions.host_id = turns.host_id - AND revisions.turn_id = turns.turn_id - AND revisions.is_current = 1 - WHERE turns.host_id = ? - AND json_extract( - turns.payload_json, - '$.source_turn_id' - ) IS NOT NULL - """, - (_HOST_ID,), - ).fetchall() - } - assert roots_after_restart == roots - assert source_identities_after_restart == source_identities - assert source_revisions_after_restart == expected_source_revisions - api = ConnectorOutboxAPI(db_path, _HOST_ID) - polled = api.poll({"name": "turn-final", "limit": 100}) - assert [item["key"] for item in polled["items"]] == [same_owner_key, new_key] - assert _HISTORICAL_FINAL_KEY not in { - item["key"] for item in polled["items"] - } - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] diff --git a/tests/test_delivery_retention_projection.py b/tests/test_delivery_retention_projection.py index a941aec..9450dab 100644 --- a/tests/test_delivery_retention_projection.py +++ b/tests/test_delivery_retention_projection.py @@ -19,11 +19,11 @@ init_store, latest_snapshot, list_worker_bindings, - merge_turn_content, save_snapshot, turns_payload_from_store, upsert_worker_bindings, ) +from .store_helpers import apply_test_turn_refresh HOST_ID = "snapshot-retention-host" @@ -147,7 +147,7 @@ def _assert_continuity_integrity(db_path: Path) -> None: with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA user_version").fetchone() == ( store_sqlite.STORE_SCHEMA_VERSION, - ) == (21,) + ) assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] current_counts = conn.execute( """ @@ -170,7 +170,7 @@ def _merge_continuity_final( worker_id: str, observed_at: str, ) -> dict[str, Any]: - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, worker_id, @@ -325,7 +325,7 @@ def _empty_snapshot(db_path: Path, *, second: int) -> Snapshot: def _seed_complete_final(db_path: Path, snapshot: Snapshot) -> tuple[str, str]: init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -489,7 +489,7 @@ def test_missing_identity_merge_persists_nonpollable_hold_on_current_placeholder init_store(db_path) assert save_snapshot(db_path, snapshot) is True - changed = merge_turn_content( + changed = apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -546,7 +546,7 @@ def test_worker_id_reuse_binds_each_root_to_immutable_stable_key( ) init_store(db_path) save_snapshot(db_path, first) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -577,7 +577,7 @@ def test_worker_id_reuse_binds_each_root_to_immutable_stable_key( ) assert second.workers[0].fingerprint != first.workers[0].fingerprint save_snapshot(db_path, second) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -660,7 +660,7 @@ def owner_snapshot(stable_key: str, second: int) -> Snapshot: "complete": True, "has_open_turn": False, } - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -689,7 +689,7 @@ def owner_snapshot(stable_key: str, second: int) -> Snapshot: second = owner_snapshot(STABLE_KEY_B, 2) assert save_snapshot(db_path, second) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -698,7 +698,7 @@ def owner_snapshot(stable_key: str, second: int) -> Snapshot: ) == 1 init_store(db_path) assert save_snapshot(db_path, second) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -757,7 +757,7 @@ def test_missing_owner_replay_holds_without_overwriting_existing_owner_root( "complete": True, "has_open_turn": False, } - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -777,7 +777,7 @@ def test_missing_owner_replay_holds_without_overwriting_existing_owner_root( timestamp=datetime(2026, 1, 1, 0, 0, 2, tzinfo=timezone.utc), ) assert save_snapshot(db_path, missing) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -786,7 +786,7 @@ def test_missing_owner_replay_holds_without_overwriting_existing_owner_root( ) == 1 init_store(db_path) assert save_snapshot(db_path, missing) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -934,7 +934,7 @@ def test_snapshot_omission_preserves_observed_source_provenance( original = _snapshot(db_path) init_store(db_path) save_snapshot(db_path, original) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, WORKER_ID, @@ -1094,7 +1094,7 @@ def test_same_owner_exact_source_survives_worker_fingerprint_space_and_source_ch assert first.workers[0].fingerprint != second.workers[0].fingerprint assert save_snapshot(db_path, second) is True upsert_worker_bindings(db_path, [_private_binding(second, PRIVATE_ROUTE_B)]) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-b", @@ -1142,7 +1142,7 @@ def test_same_owner_exact_source_survives_worker_fingerprint_space_and_source_ch init_store(db_path) assert save_snapshot(db_path, second) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-b", diff --git a/tests/test_delivery_retention_recovery.py b/tests/test_delivery_retention_recovery.py index 2497924..7961f92 100644 --- a/tests/test_delivery_retention_recovery.py +++ b/tests/test_delivery_retention_recovery.py @@ -17,17 +17,15 @@ from tendwire.store.sqlite import ( cleanup_acknowledged_final_retention, init_store, - merge_turn_content, save_snapshot, ) +from .store_helpers import apply_test_turn_refresh HOST_ID = "recovery-host" FINAL_NAME = "turn-final" CREATED_AT = "2026-01-01T00:00:00+00:00" STABLE_KEY = "wsk1_" + ("d" * 64) -RECOVERY_RAW_SOURCE = "legacy-recovery-backend-source" -RECOVERY_LEGACY_SOURCE_TOKEN = "turnsrc-422ef48fec1cfb0720da05bd" PRIVATE_ROUTE_SENTINEL = "PRIVATE-RECOVERY-ROUTE-SENTINEL" @@ -768,375 +766,10 @@ def test_repeated_recovery_history_is_hard_bounded_and_cumulative( assert source_status == "delivered" -def _seed_v10_recovered_lineage(db_path: Path) -> tuple[str, str, str]: - turn_id = "turn-v10-recovered" - revision = _insert_revision(db_path, turn_id=turn_id, final_text="abcdefghijkl") - failed_token = "twplan1.legacy-failed" - recovered_token = "twplan1.legacy-recovered" - authoritative_route = { - "schema_version": 2, - "turn_id": turn_id, - "content_revision": revision, - "final_identity": store_sqlite.turn_final_delivery_identity( - HOST_ID, - turn_id, - revision, - ), - "stable_key": STABLE_KEY, - "stable_key_version": 1, - } - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys = ON") - conn.execute( - """ - UPDATE turns - SET worker_id = 'worker-recovery-a', - worker_fingerprint = 'fingerprint-recovery-a', - space_id = 'space-recovery-a', - payload_json = ? - WHERE host_id = ? AND turn_id = ? - """, - ( - json.dumps( - { - "schema_version": 1, - "id": turn_id, - "host_id": HOST_ID, - "worker_id": "worker-recovery-a", - "worker_fingerprint": "fingerprint-recovery-a", - "space_id": "space-recovery-a", - "status": "complete", - "kind": "turn", - "source": "herdr-a", - "source_turn_id": RECOVERY_LEGACY_SOURCE_TOKEN, - "complete": True, - "has_open_turn": False, - "updated_at": CREATED_AT, - "meta": { - "stable_key": STABLE_KEY, - "stable_key_version": 1, - }, - "chat_id": PRIVATE_ROUTE_SENTINEL, - }, - sort_keys=True, - ), - HOST_ID, - turn_id, - ), - ) - failed_cursor = conn.execute( - """ - INSERT INTO turn_presentation_plans ( - host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - created_at, activated_at - ) VALUES (?, ?, ?, ?, ?, 'legacy-recovery-v10', 1, 3, 'failed', ?, ?) - """, - (HOST_ID, FINAL_NAME, failed_token, turn_id, revision, CREATED_AT, CREATED_AT), - ) - failed_plan_id = int(failed_cursor.lastrowid) - recovered_cursor = conn.execute( - """ - INSERT INTO turn_presentation_plans ( - host_id, name, plan_token, turn_id, content_revision, - presentation_version, generation, part_count, state, - replaces_plan_token, recovers_plan_token, - created_at, activated_at, completed_at - ) VALUES ( - ?, ?, ?, ?, ?, 'legacy-recovery-v10', 2, 3, 'completed', - ?, ?, ?, ?, ? - ) - """, - ( - HOST_ID, - FINAL_NAME, - recovered_token, - turn_id, - revision, - failed_token, - failed_token, - CREATED_AT, - CREATED_AT, - CREATED_AT, - ), - ) - recovered_plan_id = int(recovered_cursor.lastrowid) - - def add_job( - plan_id: int, - sequence: int, - status: str, - key: str, - ) -> None: - outbox_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES (?, ?, ?, ?, ?, '{}', ?, ?, NULL) - """, - ( - HOST_ID, - FINAL_NAME, - key, - status, - json.dumps({"turn": authoritative_route}, sort_keys=True), - CREATED_AT, - CREATED_AT, - ), - ) - outbox_id = int(outbox_cursor.lastrowid) - spans_json = json.dumps( - [ - { - "field": "assistant_final_text", - "start_char": sequence * 4, - "end_char": (sequence + 1) * 4, - } - ], - sort_keys=True, - ) - conn.execute( - """ - INSERT INTO turn_presentation_jobs ( - plan_id, sequence_index, operation, part_ordinal, - spans_json, outbox_id, created_at - ) VALUES (?, ?, 'upsert', ?, ?, ?, ?) - """, - ( - plan_id, - sequence, - sequence, - spans_json, - outbox_id, - CREATED_AT, - ), - ) - if status in {"delivered", "dead_letter"}: - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, - status, response_json, private_state_json, - created_at, delivered_at - ) VALUES (?, ?, ?, ?, 1, ?, '{}', '{}', ?, ?) - """, - ( - outbox_id, - HOST_ID, - FINAL_NAME, - key, - "delivered" if status == "delivered" else "failed", - CREATED_AT, - CREATED_AT, - ), - ) - - add_job(failed_plan_id, 0, "delivered", "turn-final:legacy-root:000000") - add_job(failed_plan_id, 1, "dead_letter", "turn-final:legacy-root:000001") - add_job(failed_plan_id, 2, "queued", "turn-final:legacy-root:000002") - add_job(recovered_plan_id, 1, "delivered", "turn-final:legacy-recovered:000001") - add_job(recovered_plan_id, 2, "delivered", "turn-final:legacy-recovered:000002") - conn.execute( - """ - INSERT INTO turn_presentation_recoveries ( - host_id, name, request_id, failed_plan_id, recovered_plan_id, - failed_plan_token, recovered_plan_token, generation, - source_job_count, delivered_prefix_count, fresh_job_count, - retained_failed_job_count, prior_attempt_count, outcome, created_at - ) VALUES (?, ?, 'legacy-request', ?, ?, ?, ?, 2, 3, 1, 2, 1, 2, - 'recovered', ?) - """, - ( - HOST_ID, - FINAL_NAME, - failed_plan_id, - recovered_plan_id, - failed_token, - recovered_token, - CREATED_AT, - ), - ) - conn.execute("PRAGMA user_version = 10") - return turn_id, revision, failed_token -def test_v12_migration_uses_effective_recovery_lineage_without_repost_or_hold( - tmp_path: Path, -) -> None: - db_path = tmp_path / "v12-recovered-lineage.db" - turn_id, revision, failed_token = _seed_v10_recovered_lineage(db_path) - init_store(db_path) - key = _final_key(turn_id, revision) - api = ConnectorOutboxAPI(db_path, HOST_ID) - assert api.poll({"name": FINAL_NAME, "limit": 100})["items"] == [] - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == store_sqlite.STORE_SCHEMA_VERSION == 21 - anchor = conn.execute( - """ - SELECT delivery_kind, status - FROM connector_outbox - WHERE delivery_key = ? - """, - (key,), - ).fetchone() - failed_state = conn.execute( - "SELECT state FROM turn_presentation_plans WHERE plan_token = ?", - (failed_token,), - ).fetchone()[0] - linked_proof = conn.execute( - """ - SELECT plans.state - FROM turn_presentation_plans AS plans - JOIN connector_outbox AS source ON source.id = plans.source_outbox_id - WHERE source.delivery_key = ? - """, - (key,), - ).fetchone()[0] - assert anchor == ("final_ready", "delivered") - assert failed_state == "superseded" - assert linked_proof == "completed" - - cleanup = cleanup_acknowledged_final_retention( - db_path, - HOST_ID, - acknowledged_final_retention_days=1, - acknowledged_final_retention_count=1, - batch_size=100, - now="2099-01-01T00:00:00+00:00", - ) - assert cleanup["deleted"] == 1 - - -def test_v12_recovery_history_stays_immutable_when_observed_turn_arrives( - tmp_path: Path, -) -> None: - db_path = tmp_path / "v12-recovered-lineage-owner-churn.db" - turn_id, revision, failed_token = _seed_v10_recovered_lineage(db_path) - init_store(db_path) - - original_key = _final_key(turn_id, revision) - before = _turn_graph_snapshot(db_path, turn_id) - assert before["turn_identity"] == ( - turn_id, - turn_id, - RECOVERY_LEGACY_SOURCE_TOKEN, - 1, - ) - assert len(before["attempts"]) == 3 - assert [row[4] for row in before["attempts"]] == [ - "delivered", - "delivered", - "delivered", - ] - recovered_plan = next(row for row in before["plans"] if row[6] == "completed") - root = next(row for row in before["outbox"] if row[1] == original_key) - assert recovered_plan[9] == root[0] - assert before["recoveries"][0][4:6] == ( - failed_token, - recovered_plan[1], - ) - - worker_b = _owner_snapshot( - db_path, - worker_id="worker-recovery-b", - worker_name="Recovery Worker B", - space_id="space-recovery-b", - second=2, - ) - assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( - db_path, - HOST_ID, - "worker-recovery-b", - { - "source_turn_id": RECOVERY_RAW_SOURCE, - "assistant_final_text": "abcdefghijkl", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:03+00:00", - ) == 1 - - after_churn = _turn_graph_snapshot(db_path, turn_id) - assert after_churn == before - api = ConnectorOutboxAPI(db_path, HOST_ID) - observed_items = api.poll({"name": FINAL_NAME, "limit": 100})["items"] - assert len(observed_items) == 1 - assert observed_items[0]["key"] != original_key - with sqlite3.connect(str(db_path)) as conn: - current = conn.execute( - """ - SELECT worker_id, worker_fingerprint, space_id, payload_json - FROM turns - WHERE host_id = ? AND turn_id = ? - """, - (HOST_ID, turn_id), - ).fetchone() - assert current is not None - current_payload = json.loads(str(current[3])) - assert current[:3] == ( - "worker-recovery-a", - "fingerprint-recovery-a", - "space-recovery-a", - ) - assert current_payload["id"] == turn_id - assert current_payload["source_turn_id"] == RECOVERY_LEGACY_SOURCE_TOKEN - observed = conn.execute( - """ - SELECT turn_id, worker_id, worker_fingerprint, space_id, payload_json - FROM turns - WHERE host_id = ? AND turn_id != ? - """, - (HOST_ID, turn_id), - ).fetchone() - assert observed is not None - observed_payload = json.loads(str(observed[4])) - assert tuple(observed[1:4]) == ( - "worker-recovery-b", - worker_b.workers[0].fingerprint, - "space-recovery-b", - ) - assert observed_payload["id"] == str(observed[0]) - assert observed_payload["source_turn_id"] != RECOVERY_LEGACY_SOURCE_TOKEN - public_payloads = [ - str(row[0]) - for row in conn.execute( - """ - SELECT payload_json FROM turns - WHERE host_id = ? AND turn_id != ? - UNION ALL - SELECT payload_json FROM connector_outbox - WHERE host_id = ? AND turn_id != ? - """, - (HOST_ID, turn_id, HOST_ID, turn_id), - ).fetchall() - ] - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - encoded_public = "\n".join(public_payloads) - assert PRIVATE_ROUTE_SENTINEL not in encoded_public - assert RECOVERY_RAW_SOURCE not in encoded_public - - init_store(db_path) - assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( - db_path, - HOST_ID, - "worker-recovery-b", - { - "source_turn_id": RECOVERY_RAW_SOURCE, - "assistant_final_text": "abcdefghijkl", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:03+00:00", - ) == 0 - assert ConnectorOutboxAPI(db_path, HOST_ID).poll( - {"name": FINAL_NAME, "limit": 100} - )["items"] == [] - assert _turn_graph_snapshot(db_path, turn_id) == before def test_known_incomplete_final_is_hold_then_complete_revision_drains( @@ -1778,7 +1411,7 @@ def test_source_less_recovery_ids_survive_same_owner_worker_churn_and_ack_loss( ) init_store(db_path) assert save_snapshot(db_path, worker_a) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-source-less-a", @@ -1843,7 +1476,7 @@ def test_source_less_recovery_ids_survive_same_owner_worker_churn_and_ack_loss( ) assert worker_b.workers[0].fingerprint != worker_a.workers[0].fingerprint assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-source-less-b", @@ -1936,7 +1569,7 @@ def test_source_less_recovery_ids_survive_same_owner_worker_churn_and_ack_loss( assert replayed_recovery["failed_plan_token"] == failed_plan["plan_token"] assert replayed_recovery["plan_token"] == recovered["plan_token"] assert save_snapshot(db_path, worker_b) is True - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, HOST_ID, "worker-source-less-b", diff --git a/tests/test_diagnostics.py b/tests/test_diagnostics.py deleted file mode 100644 index dc3e034..0000000 --- a/tests/test_diagnostics.py +++ /dev/null @@ -1,1394 +0,0 @@ -"""Tests for the read-only Herdr doctor diagnostics.""" - -from __future__ import annotations - -import json -import os -import subprocess -import socket -import sqlite3 -import stat -from collections.abc import Sequence -from pathlib import Path -from typing import Any - -import pytest - -from tendwire.backends import herdr_cli -from tendwire.backends.herdr_cli import diagnose_herdr, fetch_herdr_state -from tendwire.cli import main -from tendwire.config import Config, load_config -from tendwire.local_state import ( - ConfigStateReport, - LocalStateErrorCode, - LocalStateIssue, - LocalStateKind, - PermissionResult, - PermissionState, -) -from tendwire.store.sqlite import init_store - - -FIXTURES = Path(__file__).parent / "fixtures" / "herdr" - -_FORBIDDEN_TEXT = ( - "telegram", - "chat_id", - "topic_id", - "message_id", - "thread_id", - "argv", - "token", - "bot_token", - "delivery", - "route", - "backend_target", -) - - -def _fixture(name: str) -> str: - return (FIXTURES / name).read_text(encoding="utf-8") - - -def _completed(args: Sequence[str], stdout: str = "", stderr: str = "", returncode: int = 0) -> subprocess.CompletedProcess[str]: - return subprocess.CompletedProcess( - args=["herdr", *args], - returncode=returncode, - stdout=stdout, - stderr=stderr, - ) - - -def _doctor_outcomes(payload: dict[str, Any]) -> dict[str, str]: - return {str(check["name"]): str(check["outcome"]) for check in payload["checks"]} - -_HERDR_CHECK_NAMES = frozenset( - { - "workspace_list", - "workspace_list_json", - "agent_list", - "agent_list_json", - "pane_list", - "pane_list_json", - } -) -_LOCAL_STATE_CHECK_NAMES = ( - "state_directory_permissions", - "database_permissions", - "identity_permissions", - "daemon_socket_permissions", -) - - -def _herdr_checks(payload: dict[str, Any]) -> list[dict[str, Any]]: - return [ - check - for check in payload["checks"] - if str(check["name"]) in _HERDR_CHECK_NAMES - ] - - -def _local_state_checks(payload: dict[str, Any]) -> dict[str, dict[str, Any]]: - return { - str(check["name"]): check - for check in payload["checks"] - if str(check["name"]) in _LOCAL_STATE_CHECK_NAMES - } - - -def _maintenance_check(payload: dict[str, Any]) -> dict[str, Any]: - matches = [ - check - for check in payload["checks"] - if check.get("name") == "store_maintenance" - ] - assert len(matches) == 1 - return matches[0] -def _pending_check(payload: dict[str, Any]) -> dict[str, Any]: - matches = [ - check - for check in payload["checks"] - if check.get("name") == "pending_ingestion" - ] - assert len(matches) == 1 - return matches[0] - - - - -def _patch_healthy_herdr(monkeypatch, calls: list[tuple[str, ...]] | None = None) -> None: - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - if calls is not None: - calls.append(tuple(args[1:])) - return _completed(args[1:], stdout='{"items": []}') - - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - -def _write_mode(path: Path, mode: int) -> None: - path.write_bytes(b"local-state") - os.chmod(path, mode) - - -def test_doctor_reports_missing_herdr_binary(monkeypatch, tmp_path: Path) -> None: - config = Config( - host_id="testhost", - herdr_bin="missing-herdr", - data_dir=tmp_path / "state", - ) - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: None) - - payload = diagnose_herdr(config) - - assert payload["status"] == "unavailable" - assert _doctor_outcomes(payload)["workspace_list"] == "missing_binary" - assert all(check["ok"] is False for check in _herdr_checks(payload)) - - -def test_doctor_reports_timeout_and_skips_remaining_checks(monkeypatch, tmp_path: Path) -> None: - config = Config( - host_id="testhost", - herdr_bin="herdr", - herdr_timeout_seconds=0.25, - data_dir=tmp_path / "state", - ) - calls: list[tuple[str, ...]] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args[1:])) - assert kwargs["timeout"] == 0.25 - raise subprocess.TimeoutExpired(cmd=args, timeout=kwargs["timeout"]) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - payload = diagnose_herdr(config) - outcomes = _doctor_outcomes(payload) - - assert payload["status"] == "timeout" - assert payload["aggregate_deadline_seconds"] == 1.5 - assert calls == [("workspace", "list")] - assert outcomes["workspace_list"] == "timeout" - assert outcomes["agent_list"] == "skipped_after_timeout" - - -def test_doctor_distinguishes_nonzero_malformed_empty_and_nonempty(monkeypatch, tmp_path: Path) -> None: - config = Config( - host_id="testhost", - herdr_bin="herdr", - data_dir=tmp_path / "state", - ) - responses = { - ("workspace", "list"): _completed( - ["workspace", "list"], - stderr=_fixture("nonzero_stderr.txt"), - returncode=2, - ), - ("workspace", "list", "--json"): _completed( - ["workspace", "list", "--json"], - stdout=_fixture("workspace_list_json_empty.json"), - ), - ("agent", "list"): _completed( - ["agent", "list"], - stdout=_fixture("malformed.txt"), - ), - ("agent", "list", "--json"): _completed( - ["agent", "list", "--json"], - stdout=_fixture("agent_list_no_flag_nonempty.json"), - ), - ("pane", "list"): _completed( - ["pane", "list"], - stdout=_fixture("pane_list_empty.json"), - ), - } - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - return responses[tuple(args[1:])] - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - - payload = diagnose_herdr(config) - outcomes = _doctor_outcomes(payload) - - assert payload["status"] == "degraded" - assert outcomes["workspace_list"] == "nonzero" - assert outcomes["workspace_list_json"] == "empty_healthy" - assert outcomes["agent_list"] == "malformed_json" - assert outcomes["agent_list_json"] == "healthy_non_empty" - assert outcomes["pane_list"] == "empty_healthy" - assert "herdr fixture error" in json.dumps(payload) - - -def test_doctor_cli_outputs_json_only_and_sanitizes_samples(capsys, monkeypatch, tmp_path: Path) -> None: - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - return _completed(args[1:], stdout="not json token=must-not-leak", returncode=0) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "state")) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(tmp_path / "state" / "tendwire.db")) - - code = main(["--host-id", "doctor-host", "--herdr-bin", "herdr", "doctor", "--json"]) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert payload["schema_version"] == 1 - assert payload["command"] == "doctor" - serialized = json.dumps(payload).lower() - assert not any(forbidden in serialized for forbidden in _FORBIDDEN_TEXT) - - -def test_cli_herdr_timeout_knob_is_used_by_doctor(capsys, monkeypatch, tmp_path: Path) -> None: - timeouts: list[float] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - timeouts.append(float(kwargs["timeout"])) - return _completed(args[1:], stdout=_fixture("pane_list_empty.json")) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(tmp_path / "state")) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(tmp_path / "state" / "tendwire.db")) - - code = main(["--herdr-timeout", "0.75", "--herdr-bin", "herdr", "doctor", "--json"]) - captured = capsys.readouterr() - payload = json.loads(captured.out) - - assert code == 1 - assert captured.err == "" - assert timeouts == [0.75, 0.75, 0.75] - assert payload["timeout_seconds"] == 0.75 - assert payload["aggregate_deadline_seconds"] == 4.5 - assert all("aggregate_deadline_seconds" in check for check in _herdr_checks(payload)) - assert _pending_check(payload)["outcome"] == "store_unavailable" - - -def test_doctor_appends_fixed_compliant_local_state_checks( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "state" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "tendwire.db", - socket_path=state_dir / "tendwire.sock", - ) - assert config.db_path is not None - init_store(config.db_path) - for identity_path in ( - config.installation_key_path, - config.installation_key_marker_path, - config.installation_key_sentinel_path, - ): - _write_mode(identity_path, 0o600) - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(config.socket_path)) - assert config.socket_path is not None - os.chmod(config.socket_path, 0o600) - calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, calls) - try: - payload = diagnose_herdr(config) - finally: - listener.close() - config.socket_path.unlink(missing_ok=True) - - assert set(payload) == { - "schema_version", - "command", - "herdr_bin", - "timeout_seconds", - "aggregate_deadline_seconds", - "status", - "checks", - } - assert payload["schema_version"] == 1 - assert payload["command"] == "doctor" - assert payload["status"] == "degraded" - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - assert len(_herdr_checks(payload)) == 6 - local_checks = _local_state_checks(payload) - assert tuple(local_checks) == _LOCAL_STATE_CHECK_NAMES - assert all( - check == { - "name": name, - "ok": True, - "outcome": "compliant", - "remediation": "No action required.", - } - for name, check in local_checks.items() - ) - assert _maintenance_check(payload)["outcome"] == "overdue" - - -def test_doctor_treats_uninitialized_state_and_stopped_socket_as_neutral( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "never-created-state" - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "never-created.db", - ) - assert config.socket_path is None - calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, calls) - - payload = diagnose_herdr(config) - - assert payload["status"] == "degraded" - assert not state_dir.exists() - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - local_checks = _local_state_checks(payload) - assert { - name: (check["ok"], check["outcome"], check["remediation"]) - for name, check in local_checks.items() - } == { - "state_directory_permissions": ( - True, - "not_initialized", - "No action required while local state is uninitialized.", - ), - "database_permissions": ( - True, - "not_initialized", - "No action required while local state is uninitialized.", - ), - "identity_permissions": ( - True, - "not_initialized", - "No action required while local state is uninitialized.", - ), - "daemon_socket_permissions": ( - True, - "not_running", - "No action required while the daemon is stopped.", - ), - } - assert _maintenance_check(payload) == { - "name": "store_maintenance", - "ok": True, - "outcome": "not_initialized", - "remediation": "No action required while the store is uninitialized.", - "snapshot_retention_days": 14, - "snapshot_retention_count": 4096, - "maintenance_batch_size": 100, - "maintenance_cadence_seconds": 3600, - "snapshot_count": 0, - "last_completed_at": None, - } - assert _pending_check(payload) == { - "name": "pending_ingestion", - "ok": False, - "outcome": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - "stale_grace_seconds": 30.0, - } - - -def test_doctor_inspects_broad_default_socket_without_mutating_or_disclosing_path( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "s" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - default_socket = state_dir / "tendwire.sock" - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(default_socket)) - os.chmod(default_socket, 0o666) - before = ( - os.lstat(default_socket).st_ino, - stat.S_IMODE(os.lstat(default_socket).st_mode), - ) - config = Config(host_id="doctor-host", herdr_bin="herdr", data_dir=state_dir) - assert config.socket_path is None - calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, calls) - try: - payload = diagnose_herdr(config) - after = ( - os.lstat(default_socket).st_ino, - stat.S_IMODE(os.lstat(default_socket).st_mode), - ) - finally: - listener.close() - default_socket.unlink(missing_ok=True) - - assert payload["status"] == "degraded" - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - assert after == before - assert _local_state_checks(payload)["daemon_socket_permissions"] == { - "name": "daemon_socket_permissions", - "ok": False, - "outcome": "repair_required", - "remediation": "Restart Tendwire to repair local state permissions.", - } - serialized = json.dumps(payload) - assert str(state_dir) not in serialized - assert str(default_socket) not in serialized - assert "tendwire.sock" not in serialized - - -def test_doctor_rejects_wrong_type_default_socket_without_mutating_or_disclosing_path( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "wrong-type-default-socket-state-value" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - default_socket = state_dir / "tendwire.sock" - private_contents = b"wrong-default-socket-content-value" - default_socket.write_bytes(private_contents) - os.chmod(default_socket, 0o600) - before = ( - os.lstat(default_socket).st_ino, - stat.S_IMODE(os.lstat(default_socket).st_mode), - ) - config = Config(host_id="doctor-host", herdr_bin="herdr", data_dir=state_dir) - assert config.socket_path is None - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - after = ( - os.lstat(default_socket).st_ino, - stat.S_IMODE(os.lstat(default_socket).st_mode), - ) - assert payload["status"] == "degraded" - assert after == before - assert default_socket.read_bytes() == private_contents - assert _local_state_checks(payload)["daemon_socket_permissions"] == { - "name": "daemon_socket_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - serialized = json.dumps(payload) - assert str(state_dir) not in serialized - assert str(default_socket) not in serialized - assert private_contents.decode() not in serialized - - -def test_doctor_rejects_symlink_default_socket_without_following_or_disclosing_path( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "symlink-default-socket-state-value" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - target = state_dir / "default-socket-target-value" - private_contents = b"default-socket-target-content-value" - target.write_bytes(private_contents) - os.chmod(target, 0o600) - default_socket = state_dir / "tendwire.sock" - default_socket.symlink_to(target) - before_target = ( - os.lstat(target).st_ino, - stat.S_IMODE(os.lstat(target).st_mode), - ) - config = Config(host_id="doctor-host", herdr_bin="herdr", data_dir=state_dir) - assert config.socket_path is None - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - after_target = ( - os.lstat(target).st_ino, - stat.S_IMODE(os.lstat(target).st_mode), - ) - assert payload["status"] == "degraded" - assert default_socket.is_symlink() - assert after_target == before_target - assert target.read_bytes() == private_contents - assert _local_state_checks(payload)["daemon_socket_permissions"] == { - "name": "daemon_socket_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - serialized = json.dumps(payload) - for forbidden in ( - str(state_dir), - str(default_socket), - str(target), - private_contents.decode(), - ): - assert forbidden not in serialized - - -def test_doctor_reports_broad_modes_without_mutating_and_cli_exits_degraded( - capsys, - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "state" - state_dir.mkdir() - os.chmod(state_dir, 0o755) - db_path = state_dir / "tendwire.db" - identity_paths = ( - state_dir / "installation.key", - state_dir / "installation.key.sha256", - state_dir / "installation.key.initialized", - ) - for path in (db_path, *identity_paths): - _write_mode(path, 0o644) - socket_path = state_dir / "tendwire.sock" - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(socket_path)) - os.chmod(socket_path, 0o666) - observed_paths = (state_dir, db_path, *identity_paths, socket_path) - before = { - str(path): (os.lstat(path).st_ino, stat.S_IMODE(os.lstat(path).st_mode)) - for path in observed_paths - } - calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, calls) - monkeypatch.setenv("TENDWIRE_DATA_DIR", str(state_dir)) - monkeypatch.setenv("TENDWIRE_DB_PATH", str(db_path)) - try: - code = main( - [ - "--herdr-bin", - "herdr", - "--socket-path", - str(socket_path), - "doctor", - "--json", - ] - ) - captured = capsys.readouterr() - finally: - listener.close() - socket_path.unlink(missing_ok=True) - - payload = json.loads(captured.out) - after = { - str(path): (os.lstat(path).st_ino, stat.S_IMODE(os.lstat(path).st_mode)) - for path in observed_paths - if path != socket_path - } - assert code == 1 - assert captured.err == "" - assert payload["status"] == "degraded" - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - assert after == {key: value for key, value in before.items() if key != str(socket_path)} - assert all( - check == { - "name": name, - "ok": False, - "outcome": "repair_required", - "remediation": "Restart Tendwire to repair local state permissions.", - } - for name, check in _local_state_checks(payload).items() - ) - assert _maintenance_check(payload)["outcome"] == "unsafe" - assert _maintenance_check(payload)["ok"] is False - - -def test_doctor_degrades_for_symlinks_and_wrong_entry_types_without_following( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "state-value-that-must-not-leak" - state_dir.mkdir() - os.chmod(state_dir, 0o700) - target = state_dir / "symlink-target-that-must-not-leak" - _write_mode(target, 0o600) - db_path = state_dir / "database-value-that-must-not-leak" - db_path.symlink_to(target) - identity_path = state_dir / "installation.key" - identity_path.mkdir() - socket_path = state_dir / "socket-value-that-must-not-leak" - socket_path.write_bytes(b"wrong socket type") - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - socket_path=socket_path, - socket_group="group-value-that-must-not-leak", - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - local_checks = _local_state_checks(payload) - assert payload["status"] == "degraded" - assert local_checks["state_directory_permissions"]["outcome"] == "compliant" - assert local_checks["database_permissions"]["outcome"] == "unsafe" - assert local_checks["identity_permissions"]["outcome"] == "unsafe" - assert local_checks["daemon_socket_permissions"]["outcome"] == "unsafe" - assert db_path.is_symlink() - assert target.read_bytes() == b"local-state" - assert _maintenance_check(payload)["outcome"] == "unsafe" - assert _maintenance_check(payload)["ok"] is False - serialized = json.dumps(payload) - for forbidden in ( - str(state_dir), - str(db_path), - str(socket_path), - str(target), - "group-value-that-must-not-leak", - "wrong socket type", - ): - assert forbidden not in serialized - - -def test_store_maintenance_rejects_wrong_database_type_without_mutation( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "wrong-type-private-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "wrong-type-private-database" - db_path.mkdir(mode=0o700) - sentinel = db_path / "private-sentinel" - sentinel.write_text("wrong-type-private-content", encoding="utf-8") - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - before = (stat.S_IMODE(db_path.stat().st_mode), sentinel.read_bytes()) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - assert payload["status"] == "degraded" - assert _maintenance_check(payload)["outcome"] == "unsafe" - assert (stat.S_IMODE(db_path.stat().st_mode), sentinel.read_bytes()) == before - serialized = json.dumps(payload) - for private in ( - str(state_dir), - str(db_path), - str(sentinel), - "wrong-type-private-content", - ): - assert private not in serialized - - -def test_store_maintenance_refuses_outdated_schema_without_migration( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "outdated-private-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "outdated-private-store.db" - private_table = "private_schema_sentinel" - with sqlite3.connect(str(db_path)) as conn: - conn.execute(f"CREATE TABLE {private_table} (private_value TEXT)") - conn.execute( - f"INSERT INTO {private_table} (private_value) VALUES (?)", - ("outdated-private-content",), - ) - conn.execute("PRAGMA user_version = 1") - os.chmod(db_path, 0o600) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - before = db_path.read_bytes() - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - assert payload["status"] == "degraded" - assert _maintenance_check(payload)["outcome"] == "unavailable" - assert db_path.read_bytes() == before - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone()[0] == 1 - assert conn.execute( - f"SELECT private_value FROM {private_table}" - ).fetchone()[0] == "outdated-private-content" - serialized = json.dumps(payload) - for private in ( - str(state_dir), - str(db_path), - private_table, - "outdated-private-content", - ): - assert private not in serialized - - -def _seed_maintenance_store( - config: Config, - *, - last_completed_at: str, - snapshot_count: int, -) -> tuple[bytes, str]: - assert config.db_path is not None - init_store(config.db_path) - private = "maintenance-private-payload-sentinel" - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - UPDATE store_maintenance_state - SET last_started_at = ?, - last_completed_at = ?, - last_status = 'ok' - WHERE scope = 'automatic' - """, - (last_completed_at, last_completed_at), - ) - for index in range(snapshot_count): - conn.execute( - """ - INSERT INTO snapshots ( - host_id, created_at, content_fingerprint, payload - ) VALUES (?, ?, ?, ?) - """, - ( - config.host_id, - f"2026-01-01T00:00:0{index}+00:00", - f"private-fingerprint-{index}", - json.dumps({"private": private, "index": index}), - ), - ) - return config.db_path.read_bytes(), private - - -def test_store_maintenance_reports_current_fixed_aggregate_without_mutation( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "current-private-state" - state_dir.mkdir(mode=0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "current-private-store.db", - snapshot_retention_days=21, - snapshot_retention_count=10, - snapshot_maintenance_batch_size=7, - store_maintenance_cadence_seconds=3600, - ) - before, private = _seed_maintenance_store( - config, - last_completed_at="2026-01-01T00:00:00+00:00", - snapshot_count=1, - ) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-01T00:30:00+00:00", - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - assert payload["status"] == "ok" - assert _maintenance_check(payload) == { - "name": "store_maintenance", - "ok": True, - "outcome": "ok", - "remediation": "No action required.", - "snapshot_retention_days": 21, - "snapshot_retention_count": 10, - "maintenance_batch_size": 7, - "maintenance_cadence_seconds": 3600, - "snapshot_count": 1, - "last_completed_at": "2026-01-01T00:00:00+00:00", - } - assert _pending_check(payload) == { - "name": "pending_ingestion", - "ok": True, - "outcome": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - "stale_grace_seconds": 30.0, - } - assert config.db_path is not None - assert config.db_path.read_bytes() == before - assert private not in json.dumps(payload) - - -def test_doctor_pending_ingestion_is_fixed_nonmutating_and_public_safe( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, -) -> None: - from tendwire.store import sqlite as store_sqlite - - state_dir = tmp_path / "pending-health-private-state" - state_dir.mkdir(mode=0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "pending-health-private.db", - pending_stale_grace_seconds=17, - ) - before, private = _seed_maintenance_store( - config, - last_completed_at="2026-01-01T00:00:00+00:00", - snapshot_count=1, - ) - herdr_calls: list[tuple[str, ...]] = [] - _patch_healthy_herdr(monkeypatch, herdr_calls) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-01T00:30:00+00:00", - ) - durable_calls: list[tuple[Path, str]] = [] - - def durable_health(db_path: Path, host_id: str) -> dict[str, Any]: - durable_calls.append((db_path, host_id)) - return { - "status": "degraded", - "counts": {"fresh": 2, "stale": 1, "total": 3}, - "pane_id": "sentinel-private-pane", - "source_path": str(tmp_path / "sentinel-private-source"), - "tool_id": "sentinel-private-tool", - "error": "sentinel-private-error", - } - - monkeypatch.setattr(store_sqlite, "backend_pending_health", durable_health) - - payload = diagnose_herdr(config) - - assert payload["status"] == "degraded" - assert durable_calls == [(config.db_path, config.host_id)] - assert herdr_calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), - ] - assert _pending_check(payload) == { - "name": "pending_ingestion", - "ok": False, - "outcome": "degraded", - "counts": {"fresh": 2, "stale": 1, "total": 3}, - "stale_grace_seconds": 17.0, - } - assert config.db_path.read_bytes() == before - serialized = json.dumps(payload, sort_keys=True) - assert private not in serialized - assert "sentinel-private" not in serialized - monkeypatch.setattr( - store_sqlite, - "backend_pending_health", - lambda *_args: { - "status": "healthy", - "counts": {"fresh": 1, "stale": 1, "total": 2}, - }, - ) - fail_closed = diagnose_herdr(config) - assert _pending_check(fail_closed) == { - "name": "pending_ingestion", - "ok": False, - "outcome": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - "stale_grace_seconds": 17.0, - } - monkeypatch.setattr( - store_sqlite, - "backend_pending_health", - lambda *_args: { - "status": "healthy", - "counts": {"fresh": 3, "stale": 0, "total": 3}, - }, - ) - recovered = diagnose_herdr(config) - assert recovered["status"] == "ok" - assert _pending_check(recovered) == { - "name": "pending_ingestion", - "ok": True, - "outcome": "healthy", - "counts": {"fresh": 3, "stale": 0, "total": 3}, - "stale_grace_seconds": 17.0, - } - - -def test_store_maintenance_reports_overdue_without_mutation( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "overdue-private-state" - state_dir.mkdir(mode=0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "overdue-private-store.db", - snapshot_retention_count=10, - store_maintenance_cadence_seconds=3600, - ) - before, private = _seed_maintenance_store( - config, - last_completed_at="2026-01-01T00:00:00+00:00", - snapshot_count=1, - ) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-01T02:00:00+00:00", - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - check = _maintenance_check(payload) - assert payload["status"] == "degraded" - assert check["outcome"] == "overdue" - assert check["ok"] is False - assert check["snapshot_count"] == 1 - assert check["last_completed_at"] == "2026-01-01T00:00:00+00:00" - assert config.db_path is not None - assert config.db_path.read_bytes() == before - assert private not in json.dumps(payload) - - -def test_store_maintenance_reports_backlog_before_cadence_without_mutation( - monkeypatch, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "backlog-private-state" - state_dir.mkdir(mode=0o700) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=state_dir / "backlog-private-store.db", - snapshot_retention_days=36500, - snapshot_retention_count=1, - snapshot_maintenance_batch_size=1, - store_maintenance_cadence_seconds=3600, - ) - before, private = _seed_maintenance_store( - config, - last_completed_at="2026-01-01T00:00:00+00:00", - snapshot_count=2, - ) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-01T00:30:00+00:00", - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - check = _maintenance_check(payload) - assert payload["status"] == "degraded" - assert check["outcome"] == "backlog" - assert check["ok"] is False - assert check["snapshot_count"] == 2 - assert check["last_completed_at"] == "2026-01-01T00:00:00+00:00" - assert config.db_path is not None - assert config.db_path.read_bytes() == before - assert private not in json.dumps(payload) - - -def test_doctor_maps_owner_and_group_failures_to_fixed_unsafe_records( - monkeypatch, - tmp_path: Path, -) -> None: - private_remediation = "private-remediation-value-that-must-not-leak" - report = ConfigStateReport( - ok=False, - entries=( - PermissionResult( - LocalStateKind.STATE_DIRECTORY, - PermissionState.PRIVATE, - 0o700, - ), - ), - issues=( - LocalStateIssue( - LocalStateKind.PRIVATE_FILE, - LocalStateErrorCode.WRONG_OWNER, - private_remediation, - ), - LocalStateIssue( - LocalStateKind.SOCKET_GROUP, - LocalStateErrorCode.WRONG_GROUP, - private_remediation, - ), - ), - ) - monkeypatch.setattr(herdr_cli, "inspect_config_state", lambda *args, **kwargs: report) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr( - Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=tmp_path / "private-state", - ) - ) - - local_checks = _local_state_checks(payload) - assert payload["status"] == "degraded" - assert local_checks["identity_permissions"] == { - "name": "identity_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - assert local_checks["daemon_socket_permissions"] == { - "name": "daemon_socket_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - assert private_remediation not in json.dumps(payload) - - -def test_doctor_recursively_redacts_configured_and_subprocess_private_values( - monkeypatch, - tmp_path: Path, -) -> None: - private_bin = str(tmp_path / "configured-bin-value" / "herdr") - private_state = tmp_path / "configured-state-value" - private_db = private_state / "configured-database-value" - private_socket = private_state / "configured-socket-value" - private_group = "configured-group-value-that-must-not-leak" - private_sample = "subprocess-sample-value-that-must-not-leak" - which_values: list[str] = [] - subprocess_values: list[str] = [] - - def fake_which(value: str) -> str: - which_values.append(value) - return "/usr/bin/herdr" - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - subprocess_values.append(args[0]) - return _completed( - args[1:], - stdout=f"stdout: {private_sample} /home/alice/private-output", - stderr=f"stderr: {private_sample} /run/user/1000/private.sock", - returncode=2, - ) - - monkeypatch.setattr(herdr_cli.shutil, "which", fake_which) - monkeypatch.setattr(herdr_cli.subprocess, "run", fake_run) - payload = diagnose_herdr( - Config( - host_id="doctor-host", - herdr_bin=private_bin, - data_dir=private_state, - db_path=private_db, - socket_path=private_socket, - socket_group=private_group, - ) - ) - - assert which_values == [private_bin] - assert subprocess_values == [private_bin] * 6 - assert isinstance(payload["herdr_bin"], str) - assert payload["herdr_bin"] != private_bin - assert all( - "stdout_sample" in check and "stderr_sample" in check - for check in _herdr_checks(payload) - ) - serialized = json.dumps(payload) - for forbidden in ( - private_bin, - str(private_state), - str(private_db), - str(private_socket), - private_group, - private_sample, - "/home/alice/private-output", - "/run/user/1000/private.sock", - ): - assert forbidden not in serialized - - -def test_doctor_never_serializes_raw_inspection_or_launch_exceptions( - monkeypatch, - tmp_path: Path, -) -> None: - raw_error = "raw-exception-value-that-must-not-leak" - - def fail(*args: Any, **kwargs: Any) -> Any: - raise OSError(raw_error) - - monkeypatch.setattr(herdr_cli.shutil, "which", fail) - monkeypatch.setattr(herdr_cli, "inspect_config_state", fail) - - payload = diagnose_herdr( - Config( - host_id="doctor-host", - herdr_bin=str(tmp_path / "private-bin"), - data_dir=tmp_path / "private-state", - ) - ) - - assert payload["status"] == "unavailable" - assert raw_error not in json.dumps(payload) - assert all( - check["outcome"] == "unsafe" and check["ok"] is False - for check in _local_state_checks(payload).values() - ) - - -def test_tilde_path_expansion_for_configured_paths(monkeypatch) -> None: - monkeypatch.setenv("TENDWIRE_DATA_DIR", "~/tendwire-data") - monkeypatch.setenv("TENDWIRE_DB_PATH", "~/tendwire-data/tendwire.db") - monkeypatch.setenv("TENDWIRE_HERDR_BIN", "~/bin/herdr") - monkeypatch.setenv("TENDWIRE_HERDR_TIMEOUT_SECONDS", "0.5") - - config = load_config() - - assert str(config.data_dir).startswith(str(Path.home())) - assert str(config.db_path).startswith(str(Path.home())) - assert config.herdr_bin.startswith(str(Path.home())) - assert config.herdr_timeout_seconds == 0.5 - - -def test_fixture_outputs_parse_through_snapshot_fail_soft_path(monkeypatch) -> None: - config = Config(host_id="fixture-host", herdr_bin="herdr") - responses = { - ("workspace", "list"): _completed( - ["workspace", "list"], - stdout=_fixture("workspace_list_no_flag_nonempty.json"), - ), - ("agent", "list"): _completed( - ["agent", "list"], - stdout=_fixture("agent_list_no_flag_nonempty.json"), - ), - } - calls: list[tuple[str, ...]] = [] - - def fake_run(args: Sequence[str], cfg: Config) -> subprocess.CompletedProcess[str]: - calls.append(tuple(args)) - return responses.get(tuple(args), _completed(args, stdout="", returncode=1)) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) - - spaces, workers = fetch_herdr_state(config) - - assert calls == [("workspace", "list"), ("agent", "list"), ("pane", "list")] - assert len(spaces) == 1 - assert spaces[0].id == "ws-fixture" - assert len(workers) == 1 - assert workers[0].id == "Fixture Agent" - - -def test_doctor_initialized_store_with_absent_sqlite_sidecars_is_validation_only( - monkeypatch: Any, - tmp_path: Path, -) -> None: - state_dir = tmp_path / "private-doctor-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-doctor-database" - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - init_store(db_path) - sidecars = tuple( - Path(f"{db_path}{suffix}") for suffix in ("-wal", "-shm", "-journal") - ) - assert all(not os.path.lexists(path) for path in sidecars) - before = ( - tuple(sorted(path.name for path in state_dir.iterdir())), - db_path.stat().st_ino, - db_path.stat().st_size, - db_path.stat().st_mtime_ns, - stat.S_IMODE(db_path.stat().st_mode), - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - after = ( - tuple(sorted(path.name for path in state_dir.iterdir())), - db_path.stat().st_ino, - db_path.stat().st_size, - db_path.stat().st_mtime_ns, - stat.S_IMODE(db_path.stat().st_mode), - ) - assert _local_state_checks(payload)["database_permissions"] == { - "name": "database_permissions", - "ok": True, - "outcome": "compliant", - "remediation": "No action required.", - } - assert after == before - assert all(not os.path.lexists(path) for path in sidecars) - - -@pytest.mark.parametrize("hostile_member", ["main", "wal"]) -def test_doctor_sqlite_failures_are_fixed_typed_and_path_free( - monkeypatch: Any, - tmp_path: Path, - hostile_member: str, -) -> None: - state_dir = tmp_path / "private-doctor-hostile-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-doctor-hostile-database" - target = state_dir / "private-doctor-hostile-target" - private_contents = b"raw-OSError-private-sidecar-target" - target.write_bytes(private_contents) - os.chmod(target, 0o600) - if hostile_member == "main": - hostile_path = db_path - else: - init_store(db_path) - hostile_path = Path(f"{db_path}-wal") - hostile_path.symlink_to(target) - target_before = ( - target.read_bytes(), - target.stat().st_ino, - stat.S_IMODE(target.stat().st_mode), - ) - hostile_inode = str(os.lstat(hostile_path).st_ino) - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - database_check = _local_state_checks(payload)["database_permissions"] - assert database_check == { - "name": "database_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - assert _maintenance_check(payload)["outcome"] == "unsafe" - assert hostile_path.is_symlink() - assert ( - target.read_bytes(), - target.stat().st_ino, - stat.S_IMODE(target.stat().st_mode), - ) == target_before - serialized = json.dumps(payload, sort_keys=True) - for forbidden in ( - str(state_dir), - str(db_path), - db_path.name, - str(hostile_path), - hostile_path.name, - str(target), - target.name, - private_contents.decode(), - hostile_inode, - "-wal", - "-shm", - "-journal", - "OSError", - "[Errno", - '"uid"', - '"gid"', - '"inode"', - ): - assert forbidden not in serialized - - -def test_doctor_selected_main_disappearance_is_typed_and_publicly_fixed( - monkeypatch: Any, - tmp_path: Path, -) -> None: - from tendwire import local_state as local_state_module - - state_dir = tmp_path / "private-doctor-main-race-state" - state_dir.mkdir(mode=0o700) - db_path = state_dir / "private-doctor-main-race-database" - config = Config( - host_id="doctor-host", - herdr_bin="herdr", - data_dir=state_dir, - db_path=db_path, - ) - init_store(db_path) - selected_inode = str(db_path.stat().st_ino) - removed = False - observed_codes: list[LocalStateErrorCode] = [] - original_inspect = herdr_cli.inspect_config_state - - def remove_selected_main(phase: str, kind: LocalStateKind) -> None: - nonlocal removed - if phase == "captured" and kind is LocalStateKind.DATABASE and not removed: - removed = True - db_path.unlink() - - def capture_typed_failure(*args: Any, **kwargs: Any) -> ConfigStateReport: - report = original_inspect(*args, **kwargs) - observed_codes.extend( - issue.code - for issue in report.issues - if issue.kind is LocalStateKind.DATABASE - ) - return report - - monkeypatch.setattr( - local_state_module, - "_sqlite_family_test_phase", - remove_selected_main, - ) - monkeypatch.setattr( - herdr_cli, - "inspect_config_state", - capture_typed_failure, - ) - _patch_healthy_herdr(monkeypatch) - - payload = diagnose_herdr(config) - - assert removed - assert observed_codes == [LocalStateErrorCode.ENTRY_CHANGED] - assert not os.path.lexists(db_path) - assert _local_state_checks(payload)["database_permissions"] == { - "name": "database_permissions", - "ok": False, - "outcome": "unsafe", - "remediation": "Move unsafe local state aside and restore from a trusted backup.", - } - assert _maintenance_check(payload)["outcome"] == "unsafe" - serialized = json.dumps(payload, sort_keys=True) - for forbidden in ( - str(state_dir), - str(db_path), - db_path.name, - selected_inode, - "-wal", - "-shm", - "-journal", - "OSError", - "[Errno", - '"uid"', - '"gid"', - '"inode"', - ): - assert forbidden not in serialized diff --git a/tests/test_herdr_events.py b/tests/test_herdr_events.py deleted file mode 100644 index f1a89f3..0000000 --- a/tests/test_herdr_events.py +++ /dev/null @@ -1,6464 +0,0 @@ -"""Tests for the opt-in Herdr socket event backend.""" - -from __future__ import annotations - -import builtins -import importlib -import json -import os -import sqlite3 -import socket -import subprocess -import sys -import threading -import time -from collections.abc import Callable, Mapping -from contextlib import closing -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest - -from tendwire.backends import herdr_cli, herdr_events, herdr_turns -from tendwire.backends.herdr_events import ( - DEFAULT_SUBSCRIBE_METHOD, - HerdrEventBackend, - HerdrEventBackendError, - HerdrEventId, - HerdrProducerSequence, - normalize_event, -) -from tendwire.backends.herdr_cli import HerdrContinuityUnavailableError -from tendwire.backends.herdr_socket import ( - HerdrSocketClient, - HerdrSocketDisconnectedError, - HerdrSocketTimeoutError, -) -from tendwire.backends.herdr_protocol import ( - HERDR_EVENTS_SUBSCRIBE_METHOD, - HERDR_OFFICIAL_EVENT_NAMES, - HerdrEnvelopeError, - HerdrErrorResponse, - build_events_subscribe_params, -) -from tendwire.config import Config -from tendwire.core.models import BackendHealth, Snapshot, Worker, WorkerBinding -from tendwire.core.projector import project_from_observations -from tendwire.core.turns import PendingObservation -from tendwire.daemon import DaemonHooks, TendwireDaemon -from tendwire.store.sqlite import ( - SnapshotObservationContext, - SnapshotRetentionPolicy, - apply_backend_pending_observation, - get_herdr_turn_watermark, - init_store, - latest_snapshot, - list_attention_items, - list_backend_pending, - list_worker_bindings, - maybe_run_automatic_store_maintenance, - merge_turn_content, - pending_payload_from_store, - save_snapshot, - set_herdr_turn_watermark, - turns_payload_from_store, -) - - -_PUBLIC_JSON_FORBIDDEN_KEYS = { - "pane_id", - "terminal_id", - "backend_target", - "chat_id", - "topic_id", - "message_id", - "connector", - "argv", - "args", - "env", - "environment", - "stdin", - "stdout", - "stderr", - "token", - "tokens", - "secret", - "secrets", - "raw_payload", - "socket_path", - "target_kind", - "target_value", - "private_fingerprint", -} -_PUBLIC_JSON_FORBIDDEN_COMPACT = {key.replace("_", "") for key in _PUBLIC_JSON_FORBIDDEN_KEYS} - - -def _assert_no_public_json_forbidden(value: Any, path: str = "$") -> None: - if isinstance(value, dict): - for key, item in value.items(): - normalized = str(key).lower().replace("-", "_") - assert ( - normalized not in _PUBLIC_JSON_FORBIDDEN_KEYS - and normalized.replace("_", "") not in _PUBLIC_JSON_FORBIDDEN_COMPACT - ), f"forbidden field {path}.{key}" - _assert_no_public_json_forbidden(item, f"{path}.{key}") - elif isinstance(value, list): - for index, item in enumerate(value): - _assert_no_public_json_forbidden(item, f"{path}[{index}]") - - - -_NO_OP_TABLES = ( - "commands", - "command_receipts", - "turns", - "attention_items", - "connector_outbox", - "connector_deliveries", -) - -_PERSISTED_EVENT_EFFECT_TABLES = ( - "snapshots", - "events", - "workers", - "worker_bindings", - "attention_items", - "connector_outbox", -) - - -def _table_count(db_path: Path, host_id: str, table: str) -> int: - with closing(sqlite3.connect(str(db_path))) as conn, conn: - return int(conn.execute(f"SELECT COUNT(*) FROM {table} WHERE host_id = ?", (host_id,)).fetchone()[0]) - -def _persisted_event_effect_counts(backend: HerdrEventBackend) -> dict[str, int]: - return { - table: _table_count(backend.db_path, backend.config.host_id, table) - for table in _PERSISTED_EVENT_EFFECT_TABLES - } - - -def _attention_lifecycle_rows(backend: HerdrEventBackend) -> tuple[tuple[Any, ...], ...]: - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - return tuple( - conn.execute( - """ - SELECT - generation, - lifecycle_status, - current_attention_id, - first_seen_at, - last_positive_at, - first_missing_at, - missing_observation_count, - last_accepted_at, - last_observation_key, - max_notified_severity_rank - FROM attention_lifecycles - WHERE host_id = ? - ORDER BY family_key - """, - (backend.config.host_id,), - ).fetchall() - ) - - -def _attention_event_types(backend: HerdrEventBackend) -> list[str]: - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - rows = conn.execute( - """ - SELECT payload_json - FROM connector_outbox - WHERE host_id = ? AND connector = 'attention' - ORDER BY id - """, - (backend.config.host_id,), - ).fetchall() - return [str(json.loads(row[0])["event_type"]) for row in rows] - - -def _set_observation_time(monkeypatch: Any, value: str) -> None: - monkeypatch.setattr("tendwire.backends.herdr_cli.utc_timestamp", lambda: value) - monkeypatch.setattr("tendwire.backends.herdr_events.utc_timestamp", lambda: value) - -def _no_op_state(backend: HerdrEventBackend) -> tuple[str, tuple[tuple[Any, ...], ...], dict[str, int]]: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - bindings = tuple( - sorted( - ( - binding.worker_id, - binding.private_fingerprint, - binding.target_kind, - binding.target_value, - binding.sendable, - binding.reason, - binding.expires_at, - ) - for binding in list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - include_expired=True, - ) - ) - ) - counts = {table: _table_count(backend.db_path, backend.config.host_id, table) for table in _NO_OP_TABLES} - return snapshot.to_json(), bindings, counts - -class _SocketConnection: - def __init__(self, conn: socket.socket, requests: list[dict[str, Any]]) -> None: - self.conn = conn - self.requests = requests - self._buffer = bytearray() - - def read_request(self) -> dict[str, Any]: - while b"\n" not in self._buffer: - chunk = self.conn.recv(4096) - if not chunk: - raise ConnectionError("client disconnected before request") - self._buffer.extend(chunk) - index = self._buffer.index(b"\n") - line = bytes(self._buffer[: index + 1]) - del self._buffer[: index + 1] - request = json.loads(line.decode("utf-8")) - self.requests.append(request) - return request - - def send_json(self, payload: Mapping[str, Any]) -> None: - self.conn.sendall(json.dumps(dict(payload), separators=(",", ":")).encode("utf-8") + b"\n") - - -class _FakeHerdrSocketServer: - def __init__( - self, - tmp_path: Path, - handler: Callable[[_SocketConnection], None], - *, - connections: int = 1, - ) -> None: - self.path = tmp_path / f"herdr-events-{time.monotonic_ns()}.sock" - self.handler = handler - self.connections = connections - self.requests: list[dict[str, Any]] = [] - self.errors: list[BaseException] = [] - self._ready = threading.Event() - self._listener: socket.socket | None = None - self._thread: threading.Thread | None = None - - def __enter__(self) -> "_FakeHerdrSocketServer": - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(self.path)) - listener.listen(self.connections) - listener.settimeout(0.2) - self._listener = listener - self._thread = threading.Thread(target=self._run, daemon=True) - self._thread.start() - assert self._ready.wait(1) - return self - - def __exit__(self, exc_type: object, exc: object, tb: object) -> None: - if self._listener is not None: - self._listener.close() - if self._thread is not None: - self._thread.join(timeout=1) - try: - self.path.unlink() - except FileNotFoundError: - pass - if exc_type is None and self.errors: - raise AssertionError(f"fake Herdr socket failed: {self.errors!r}") - - def _run(self) -> None: - self._ready.set() - try: - assert self._listener is not None - for _index in range(self.connections): - conn, _addr = self._listener.accept() - with conn: - self.handler(_SocketConnection(conn, self.requests)) - except OSError: - pass - except BaseException as exc: - self.errors.append(exc) - - -class _StaticClient: - def __init__( - self, - *, - workspaces: Any | None = None, - tabs: Any | None = None, - panes: Any | None = None, - agents: Any | None = None, - ) -> None: - self.workspaces = {"workspaces": list(workspaces or [])} - self.tabs = {"tabs": list(tabs or [])} - self.panes = {"panes": list(panes or [])} - self.agents = {"agents": list(agents or [])} - self.calls: list[str] = [] - - def workspace_list(self) -> Any: - self.calls.append("workspace.list") - return self.workspaces - - def tab_list(self) -> Any: - self.calls.append("tab.list") - return self.tabs - - def pane_list(self) -> Any: - self.calls.append("pane.list") - return self.panes - - def agent_list(self) -> Any: - self.calls.append("agent.list") - return self.agents - - -def _config(tmp_path: Path, host_id: str = "events-host") -> Config: - return Config( - host_id=host_id, - data_dir=tmp_path, - db_path=tmp_path / f"{host_id}.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - ) - - -def _backend(tmp_path: Path, host_id: str = "events-host", *, debounce_seconds: float = 0) -> HerdrEventBackend: - config = _config(tmp_path, host_id) - init_store(Path(config.db_path)) - return HerdrEventBackend( - config, - debounce_seconds=debounce_seconds, - reconnect_delay_seconds=0, - ) - - -def _initial_pane_client() -> _StaticClient: - return _StaticClient( - workspaces=[{"id": "space-1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - } - ], - agents=[], - ) - -def _status_event(status: str) -> dict[str, Any]: - """Return the confirmed idless Herdr EventEnvelope shape.""" - return { - "event": "pane_agent_status_changed", - "data": { - "pane_id": "pane-1", - "agent": "Agent One", - "status": status, - }, - } - - -def _write_decision_adapter(tmp_path: Path) -> Path: - adapter = tmp_path / "fake-herdr-turn-adapter" - adapter.write_text( - "#!/usr/bin/env python3\n" - "import json\n" - "print(json.dumps({'result': {'turn': {" - "'available': True, 'complete': False, 'awaiting_input': True, " - "'user_text': 'Choose a rollout.', 'source_turn_id': 'producer-decision-turn', " - "'pending_decision': {'prompt': 'Choose a rollout.', 'mode': 'buttons', " - "'options': [{'id': 'alpha', 'label': 'Alpha', 'send_text': 'Alpha'}, " - "{'id': 'beta', 'label': 'Beta', 'send_text': 'Beta'}]}}}}))\n", - encoding="utf-8", - ) - adapter.chmod(0o700) - return adapter - - -def _decision_backend( - tmp_path: Path, - turn_model: str, -) -> tuple[HerdrEventBackend, WorkerBinding]: - adapter = _write_decision_adapter(tmp_path) - config = Config( - host_id=f"decision-{turn_model}", - data_dir=tmp_path, - db_path=tmp_path / f"decision-{turn_model}.db", - herdr_backend="socket", - herdr_bin=str(adapter), - herdr_timeout_seconds=1, - turn_model=turn_model, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) - snapshot = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[ - { - "pane_id": "w123456789abcde:pA", - "terminal_id": "terminal-decision", - "agent": "claude", - "workspace_id": "w123456789abcde", - "agent_status": "working", - } - ], - agents=[], - ) - ) - binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - # Reproduce the production wedge: the stable worker still has a durable - # no-prompt row owned by an expired pane binding. - assert apply_backend_pending_observation( - backend.db_path, - backend.config.host_id, - snapshot.workers[0].id, - PendingObservation("read_succeeded_no_prompt"), - binding_private_fingerprint="expired-pane-binding", - observed_turn_target_value="old-pane", - ) - return backend, binding - - -def _assert_decision_persisted(backend: HerdrEventBackend) -> None: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - worker = snapshot.workers[0] - backend_pending = list_backend_pending(backend.db_path, backend.config.host_id) - assert list(backend_pending) == [worker.id] - assert [choice["label"] for choice in backend_pending[worker.id]["choices"]] == [ - "Alpha", - "Beta", - ] - pending = pending_payload_from_store(backend.db_path, backend.config.host_id) - interaction = next( - item for item in pending["pending_interactions"] if item["worker_id"] == worker.id - ) - assert [choice["label"] for choice in interaction["choices"]] == ["Alpha", "Beta"] - turns = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=snapshot, - )["turns"] - turn = next(item for item in turns if item.get("awaiting_input") is True) - assert turn["complete"] is False - assert turn["pending_decision"] == { - "prompt": "Choose a rollout.", - "mode": "buttons", - "options": [ - {"id": "1", "label": "Alpha"}, - {"id": "2", "label": "Beta"}, - ], - "multi_select": False, - "question_count": 1, - } - - -def test_startup_reconcile_uses_socket_client_persists_projection_and_private_bindings(tmp_path: Path) -> None: - def handler(conn: _SocketConnection) -> None: - results = { - "workspace.list": { - "workspaces": [ - { - "id": "space-1", - "name": "Build", - "status": "active", - "pane_id": "private-pane", - } - ] - }, - "tab.list": {"tabs": [{"id": "tab-private", "workspace_id": "space-1"}]}, - "pane.list": { - "panes": [ - { - "pane_id": "pane-1", - "terminal_id": "terminal-private", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - } - ] - }, - "agent.list": { - "agents": [ - { - "agent_id": "agent-private", - "name": "Agent One", - "workspace_id": "space-1", - "status": "waiting", - "pane_id": "pane-1", - } - ] - }, - } - request = conn.read_request() - conn.send_json({"id": request["id"], "result": results[request["method"]]}) - - config = _config(tmp_path, "socket-reconcile") - init_store(Path(config.db_path)) - with _FakeHerdrSocketServer(tmp_path, handler, connections=4) as server: - backend = HerdrEventBackend(config, debounce_seconds=0) - client = HerdrSocketClient(str(server.path), timeout=1) - snapshot = backend.reconcile_once(client=client) - client.close() - - assert [request["method"] for request in server.requests] == [ - "workspace.list", - "tab.list", - "pane.list", - "agent.list", - ] - assert snapshot.backend_health[0].status == "healthy" - assert "agent-private" not in {worker.id for worker in snapshot.workers} - assert all("private" not in worker.id.lower() for worker in snapshot.workers) - bindings = list_worker_bindings(Path(config.db_path), config.host_id, backend="herdr") - assert bindings - assert bindings[0].target_value == "agent-private" - encoded = snapshot.to_json() - assert "agent-private" not in encoded - assert "private-pane" not in encoded - assert "terminal-private" not in encoded - _assert_no_public_json_forbidden(json.loads(encoded)) - - -@pytest.mark.parametrize("event_name", HERDR_OFFICIAL_EVENT_NAMES) -def test_normalize_event_accepts_each_official_event_name(event_name: str) -> None: - event = normalize_event({"event": event_name, "data": {}}) - - assert event is not None - assert event.name == event_name - assert event.producer_identity is None - - -@pytest.mark.parametrize( - ("raw_name", "canonical_name"), - [ - ("agent.status_changed", "pane.agent_status_changed"), - ("agent_status_changed", "pane.agent_status_changed"), - ("agent.detected", "pane.agent_detected"), - ("pane_output_changed", "pane.updated"), - ("pane.observed", "pane.created"), - ("workspace.observed", "workspace.updated"), - ("worktree.updated", "worktree.opened"), - ("worktree.closed", "worktree.removed"), - ], -) -def test_normalize_event_tolerates_legacy_inbound_aliases_only_after_receive( - raw_name: str, - canonical_name: str, -) -> None: - event = normalize_event({"event": raw_name, "data": {}}) - - assert event is not None - assert event.name == canonical_name - - -def test_normalize_event_accepts_confirmed_live_idless_event_data_shape() -> None: - event = normalize_event( - { - "event": "pane_agent_status_changed", - "data": {"agent": "Agent One", "status": "blocked"}, - } - ) - - assert event is not None - assert event.name == "pane.agent_status_changed" - assert event.payload == {"agent": "Agent One", "status": "blocked"} - assert event.producer_identity is None - - -def test_normalize_event_prefers_confirmed_data_over_legacy_payload() -> None: - event = normalize_event( - { - "event": "pane_agent_status_changed", - "data": {"status": "working"}, - "payload": {"status": "idle"}, - } - ) - - assert event is not None - assert event.payload == {"status": "working"} - assert event.producer_identity is None - - -def test_normalize_event_keeps_receive_only_legacy_payload_compatibility() -> None: - event = normalize_event( - { - "event": "pane_agent_status_changed", - "payload": {"status": "idle"}, - } - ) - - assert event is not None - assert event.payload == {"status": "idle"} - assert event.producer_identity is None - - -def test_normalize_event_exposes_forward_compatible_producer_identity_types() -> None: - by_id = normalize_event( - {"event": "pane_agent_status_changed", "data": {}, "event_id": "event-1"} - ) - by_sequence = normalize_event( - { - "event": "pane_agent_status_changed", - "data": {}, - "server_id": "server-1", - "sequence": 7, - } - ) - - assert by_id is not None - assert by_id.producer_identity == HerdrEventId("event-1") - assert by_sequence is not None - assert by_sequence.producer_identity == HerdrProducerSequence("server-1", "7") - - -def test_normalize_event_never_uses_entity_data_as_producer_identity() -> None: - event = normalize_event( - { - "event": "pane_agent_status_changed", - "data": { - "event_id": "entity-event", - "server_id": "entity-server", - "sequence": 9, - "revision": 10, - }, - } - ) - - assert event is not None - assert event.producer_identity is None - - -@pytest.mark.parametrize( - "metadata", - [ - {"event_id": True}, - {"event_id": 1}, - {"event_id": 1.5}, - {"event_id": "event id"}, - {"event_id": {"id": "nested-event"}}, - {"event_id": ["nested-event"]}, - {"server_id": True, "sequence": 1}, - {"server_id": "server id", "sequence": 1}, - {"server_id": {"id": "nested-server"}, "sequence": 1}, - {"server_id": "server-1", "sequence": True}, - {"server_id": "server-1", "sequence": 1.5}, - {"server_id": "server-1", "sequence": "1"}, - {"server_id": "server-1", "sequence": {"value": 1}}, - {"server_id": "server-1"}, - {"sequence": 1}, - {"event_id": True, "server_id": "server-1", "sequence": 1}, - ], -) -def test_malformed_producer_metadata_is_idless_and_preserves_transitions( - tmp_path: Path, - metadata: dict[str, Any], -) -> None: - backend = _backend(tmp_path, f"malformed-producer-{len(str(metadata))}") - backend.reconcile_once(client=_initial_pane_client()) - accepted: list[bool] = [] - - for status in ("working", "idle", "working"): - envelope = {**_status_event(status), **metadata} - normalized = normalize_event(envelope) - assert normalized is not None - assert normalized.producer_identity is None - accepted.append(backend.queue_event_envelope(envelope)) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert accepted == [True, True, True] - assert snapshot.workers[0].status == "active" - - -def test_backend_default_subscription_uses_official_shape_without_legacy_defaults(tmp_path: Path) -> None: - config = _config(tmp_path, "default-subscribe") - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) - - class SubscribeClient(_StaticClient): - def __init__(self) -> None: - super().__init__() - self.subscriptions: list[tuple[str, dict[str, Any]]] = [] - - def connect(self) -> None: - return None - - def close(self) -> None: - return None - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - self.subscriptions.append((method, dict(params))) - backend.stop_event.set() - return SimpleNamespace(subscription_id="sub-default") - - client = SubscribeClient() - backend.client_factory = lambda _config: client - - backend.run_forever() - - expected_params = { - "subscriptions": [ - {"type": name} - for name in HERDR_OFFICIAL_EVENT_NAMES - if name not in {"pane.agent_status_changed", "pane.output_matched"} - ] - } - assert DEFAULT_SUBSCRIBE_METHOD == HERDR_EVENTS_SUBSCRIBE_METHOD - assert client.subscriptions == [(HERDR_EVENTS_SUBSCRIBE_METHOD, expected_params)] - subscribed_names = {subscription["type"] for subscription in expected_params["subscriptions"]} - assert { - "pane.observed", - "workspace.observed", - "agent.status_changed", - "worktree.updated", - }.isdisjoint(subscribed_names) - - -def test_backend_falls_back_to_herdr_074_pane_scoped_event_subscriptions(tmp_path: Path) -> None: - config = _config(tmp_path, "pane-scoped-subscribe") - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) - - class PaneScopedClient(_StaticClient): - def __init__(self) -> None: - super().__init__( - workspaces=[{"id": "space-1", "name": "Build"}], - panes=[ - { - "pane_id": "pane-private", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - } - ], - ) - self.mixed_attempts = 0 - self.subscriptions: list[tuple[str, dict[str, Any]]] = [] - self.closed = 0 - self.connected = 0 - - def connect(self) -> None: - self.connected += 1 - - def close(self) -> None: - self.closed += 1 - - def events_subscribe( - self, - event_names: Any, - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - raise AssertionError("0.7.4 fallback must retain the pane-scoped request") - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - self.subscriptions.append((method, dict(params))) - if any( - item.get("type") == "pane.updated" - for item in params.get("subscriptions", []) - ): - self.mixed_attempts += 1 - raise HerdrErrorResponse( - { - "code": "invalid_request", - "message": "invalid request: unknown variant pane.updated", - }, - "subscribe-1", - uncorrelated=True, - ) - backend.stop_event.set() - return SimpleNamespace(subscription_id="pane-scoped-sub") - - client = PaneScopedClient() - backend.client_factory = lambda _config: client - - backend.run_forever() - - assert client.mixed_attempts == 1 - assert client.closed >= 1 - assert client.connected >= 1 - assert len(client.subscriptions) == 2 - method, params = client.subscriptions[1] - assert method == HERDR_EVENTS_SUBSCRIBE_METHOD - subscriptions = params["subscriptions"] - fallback_names = set(HERDR_OFFICIAL_EVENT_NAMES) - { - "workspace.focused", - "pane.updated", - "pane.focused", - "pane.agent_detected", - "pane.output_matched", - } - assert len(subscriptions) == len(fallback_names) - assert {item["type"] for item in subscriptions} == fallback_names - assert {item["pane_id"] for item in subscriptions} == {"pane-private"} - - -def test_backend_empty_installation_falls_back_to_herdr_074_global_subscription( - tmp_path: Path, -) -> None: - config = _config(tmp_path, "empty-global-fallback") - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0, reconnect_delay_seconds=0) - - class EmptyInstallationClient(_StaticClient): - def __init__(self) -> None: - super().__init__() - self.subscriptions: list[tuple[str, dict[str, Any]]] = [] - self.closed = 0 - self.connected = 0 - - def connect(self) -> None: - self.connected += 1 - - def close(self) -> None: - self.closed += 1 - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - copied = (method, {"subscriptions": [dict(item) for item in params["subscriptions"]]}) - self.subscriptions.append(copied) - if any(item.get("type") == "pane.updated" for item in params["subscriptions"]): - raise HerdrErrorResponse( - { - "code": "invalid_request", - "message": "invalid request: unknown variant pane.updated", - }, - "subscribe-1", - uncorrelated=True, - ) - backend.stop_event.set() - return SimpleNamespace(subscription_id="empty-global-sub") - - client = EmptyInstallationClient() - backend.client_factory = lambda _config: client - - backend.run_forever() - - assert client.closed >= 1 - assert client.connected >= 1 - assert len(client.subscriptions) == 2 - method, params = client.subscriptions[1] - assert method == HERDR_EVENTS_SUBSCRIBE_METHOD - subscriptions = params["subscriptions"] - assert subscriptions - assert all(set(item) == {"type"} for item in subscriptions) - assert {item["type"] for item in subscriptions} == set(HERDR_OFFICIAL_EVENT_NAMES) - { - "pane.updated" - } - - -@pytest.mark.parametrize( - "failure", - [ - HerdrErrorResponse( - {"code": "permission_denied", "message": "subscription denied"}, - "subscribe-1", - ), - HerdrErrorResponse( - {"code": "invalid_request", "message": "invalid request: correlated"}, - "subscribe-1", - ), - HerdrEnvelopeError("malformed subscription response"), - ], -) -def test_backend_reconnects_instead_of_downgrading_unrelated_subscription_failures( - tmp_path: Path, - failure: Exception, -) -> None: - backend = _backend(tmp_path, "no-unrelated-subscription-downgrade") - - class RejectingClient: - def __init__(self) -> None: - self.calls = 0 - self.closed = 0 - self.connected = 0 - - def subscribe(self, method, params, **kwargs): - self.calls += 1 - raise failure - - def close(self) -> None: - self.closed += 1 - - def connect(self) -> None: - self.connected += 1 - - client = RejectingClient() - with pytest.raises(type(failure)): - backend._subscribe_event_stream(client) - assert client.calls == 1 - assert client.closed == 0 - assert client.connected == 0 - - -def test_herdr_075_observation_paths_preserve_474_identity_inputs_byte_for_byte( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Feed one pane through every event path without changing its identity bytes.""" - fixed_installation_key = b"tendwire-identity-regression-key" - assert len(fixed_installation_key) == 32 - derivation_inputs: list[bytes] = [] - real_stable_worker_key = herdr_cli.stable_worker_key - - def capture_stable_worker_key( - installation_key: bytes, - *, - backend: str, - host_id: str, - workspace_id: str, - pane_id: str, - ) -> str: - derivation_inputs.append( - installation_key - + b"\0" - + json.dumps( - { - "backend": backend, - "host_id": host_id, - "pane_id": pane_id, - "workspace_id": workspace_id, - }, - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") - ) - return real_stable_worker_key( - installation_key, - backend=backend, - host_id=host_id, - workspace_id=workspace_id, - pane_id=pane_id, - ) - - monkeypatch.setattr( - herdr_cli, - "load_or_create_installation_key", - lambda _data_dir: fixed_installation_key, - ) - monkeypatch.setattr(herdr_cli, "stable_worker_key", capture_stable_worker_key) - - pane = { - "pane_id": "w123456789abcde:pA", - "terminal_id": "terminal-identity", - "agent": "claude", - "workspace_id": "w123456789abcde", - "agent_status": "working", - "label": "identity-pane", - } - workspaces = [{"id": "w123456789abcde", "name": "Build"}] - - def reconciled_backend(name: str) -> HerdrEventBackend: - data_dir = tmp_path / name - config = Config( - host_id="identity-host", - data_dir=data_dir, - db_path=data_dir / "tendwire.db", - herdr_backend="socket", - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - backend.reconcile_once( - client=_StaticClient( - workspaces=workspaces, - panes=[dict(pane)], - agents=[], - ) - ) - return backend - - def stable_key_bytes(backend: HerdrEventBackend) -> bytes: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - return snapshot.workers[0].meta["stable_key"].encode("ascii") - - def worker_meta_bytes(backend: HerdrEventBackend) -> bytes: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - return json.dumps( - snapshot.workers[0].meta, - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") - - reference_backend = reconciled_backend("474-reference-reconcile") - assert len(derivation_inputs) == 1 - reference_input = derivation_inputs[0] - reference_key = stable_key_bytes(reference_backend) - reference_meta = worker_meta_bytes(reference_backend) - - legacy_backend = reconciled_backend("herdr-074-status-path") - legacy_path_start = len(derivation_inputs) - 1 - assert legacy_backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": {**pane, "agent_status": "blocked"}, - } - ) - # Scalar status payloads are not PaneInfo and must not re-derive identity. - assert derivation_inputs[legacy_path_start:] == [reference_input] - assert stable_key_bytes(legacy_backend) == reference_key - assert worker_meta_bytes(legacy_backend) == reference_meta - - new_backend = reconciled_backend("herdr-075-pane-updated-path") - new_path_start = len(derivation_inputs) - 1 - before_new_meta = worker_meta_bytes(new_backend) - refreshes: list[None] = [] - new_backend.set_turn_refresh_callback(lambda: refreshes.append(None)) - - # Herdr 0.7.5's pane.updated is the scalar PaneOutputChanged event. It may - # trigger a turn refresh but must not rebuild worker identity. - assert new_backend.queue_event_envelope( - { - "event": "pane.updated", - "data": { - "type": "pane_updated", - "workspace_id": pane["workspace_id"], - "pane_id": pane["pane_id"], - "revision": 2, - }, - } - ) - assert refreshes == [None] - assert derivation_inputs[new_path_start:] == [reference_input] - assert stable_key_bytes(new_backend) == reference_key - assert before_new_meta == worker_meta_bytes(new_backend) == reference_meta - - class Herdr074FallbackClient: - def __init__(self) -> None: - self.params: list[dict[str, Any]] = [] - self.closed = 0 - self.connected = 0 - - def subscribe(self, _method: str, params: Mapping[str, Any], **_kwargs: Any) -> Any: - copied = json.loads(json.dumps(params)) - self.params.append(copied) - if any( - item.get("type") == "pane.updated" - for item in copied["subscriptions"] - ): - raise HerdrErrorResponse( - { - "code": "invalid_request", - "message": "invalid request: unknown variant pane.updated", - }, - "subscribe-1", - uncorrelated=True, - ) - return SimpleNamespace(subscription_id="herdr-074-compatible") - - def close(self) -> None: - self.closed += 1 - - def connect(self) -> None: - self.connected += 1 - - fallback = Herdr074FallbackClient() - before_fallback_inputs = list(derivation_inputs) - before_fallback = latest_snapshot(new_backend.db_path, new_backend.config.host_id) - stream = new_backend._subscribe_event_stream(fallback) - after_fallback = latest_snapshot(new_backend.db_path, new_backend.config.host_id) - assert stream.subscription_id == "herdr-074-compatible" - assert fallback.closed == fallback.connected == 1 - assert len(fallback.params) == 2 - assert all( - item["type"] != "pane.updated" - for item in fallback.params[1]["subscriptions"] - ) - assert {item["pane_id"] for item in fallback.params[1]["subscriptions"]} == { - pane["pane_id"] - } - assert derivation_inputs == before_fallback_inputs - assert before_fallback == after_fallback - - fallback_event_start = len(derivation_inputs) - assert new_backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": {**pane, "agent_status": "blocked"}, - } - ) - assert derivation_inputs[fallback_event_start:] == [] - assert stable_key_bytes(new_backend) == reference_key - assert worker_meta_bytes(new_backend) == reference_meta - - # This models the observation-layer failure: a scalar refresh arrives with - # a second identity-looking tuple. It must neither call the derivation - # function nor replace any persisted worker metadata. - before_dangerous_update_inputs = list(derivation_inputs) - before_dangerous_update_meta = worker_meta_bytes(new_backend) - assert new_backend.queue_event_envelope( - { - "event": "pane.updated", - "data": { - "type": "pane_updated", - "workspace_id": "wD2", - "pane_id": "wD2:p7", - "revision": 3, - }, - } - ) - assert refreshes == [None, None, None] - assert derivation_inputs == before_dangerous_update_inputs - assert stable_key_bytes(new_backend) == reference_key - assert worker_meta_bytes(new_backend) == before_dangerous_update_meta - - -def test_cross_path_and_cross_representation_observations_keep_one_stable_key( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - adapter = tmp_path / "identity-turn-adapter" - adapter.write_text( - "#!/usr/bin/env python3\n" - "import json\n" - "print(json.dumps({'result': {'turn': {" - "'available': True, 'complete': True, 'user_text': 'identity read', " - "'assistant_final_text': 'stable', 'source_turn_id': 'identity-turn', " - "'workspace_id': 7, 'pane_id': 41}}}))\n", - encoding="utf-8", - ) - adapter.chmod(0o700) - config = Config( - host_id="cross-path-identity", - data_dir=tmp_path, - db_path=tmp_path / "cross-path.db", - herdr_backend="socket", - herdr_bin=str(adapter), - herdr_timeout_seconds=1, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - record_observations: list[ - tuple[str, bool, str | None, str | None, str | None, str | None] - ] = [] - real_worker_record_from_item = herdr_cli._worker_record_from_item - - def capture_worker_record( - item: Mapping[str, Any], - record_config: Config | None = None, - *, - pane_info_observed: bool = False, - identity_source: str = "unknown", - ) -> Any: - record = real_worker_record_from_item( - item, - record_config, - pane_info_observed=pane_info_observed, - identity_source=identity_source, - ) - record_observations.append( - ( - record.identity_source, - record.pane_info_observed, - record.observed_workspace_id, - record.observed_pane_id, - record.workspace_id, - record.pane_id, - ) - ) - return record - - monkeypatch.setattr(herdr_cli, "_worker_record_from_item", capture_worker_record) - monkeypatch.setattr(herdr_events, "_worker_record_from_item", capture_worker_record) - pane = { - "workspace_id": "w65383a2e877513", - "pane_id": "w65383a2e877513:pA", - "terminal_id": "terminal-cross-path", - "agent": "claude", - "agent_status": "working", - "label": "cross-path", - } - client = _StaticClient( - workspaces=[{"id": pane["workspace_id"], "name": "Build"}], - panes=[pane], - ) - - stable_keys: list[str] = [] - - def capture_key() -> None: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert len(snapshot.workers) == 1 - stable_keys.append(str(snapshot.workers[0].meta["stable_key"])) - - backend.reconcile_once(client=client) - capture_key() - assert ( - "pane.list", - True, - pane["workspace_id"], - pane["pane_id"], - pane["workspace_id"], - pane["pane_id"], - ) in record_observations - records = backend._records_from_reconcile_payloads( - {"agents": []}, - {"panes": [pane]}, - ) - assert len(records) == 1 - assert records[0].identity_source == "pane.list" - assert (records[0].workspace_id, records[0].pane_id) == ( - pane["workspace_id"], - pane["pane_id"], - ) - - # Full PaneInfo events may use aliases, but record construction still - # stores the exact canonical public pair used by pane.list. - assert backend.queue_event_envelope( - { - "event": "pane.created", - "data": { - "type": "pane_created", - "pane": { - "workspaceId": pane["workspace_id"], - "paneId": pane["pane_id"], - "terminalId": pane["terminal_id"], - "agent": "claude", - "agentStatus": "idle", - "label": "cross-path", - }, - }, - } - ) - capture_key() - assert ( - "event:pane.created", - True, - pane["workspace_id"], - pane["pane_id"], - pane["workspace_id"], - pane["pane_id"], - ) in record_observations - - # pane.updated is a scalar refresh notification in Herdr 0.7.5. Even a - # different identity-looking representation cannot enter worker records. - assert backend.queue_event_envelope( - { - "event": "pane.updated", - "data": { - "type": "pane_output_changed", - "workspace_id": 7, - "pane_id": 41, - "revision": 2, - }, - } - ) - capture_key() - assert all(source != "event:pane.updated" for source, *_rest in record_observations) - - # Scalar events may carry raw runtime representations. They can update a - # matched worker but cannot become PaneInfo or feed stable-key derivation. - assert backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": { - "workspace_id": 7, - "pane_id": 41, - "terminal_id": pane["terminal_id"], - "agent": "claude", - "agent_status": "working", - }, - } - ) - capture_key() - assert ( - "event:pane.agent_status_changed", - False, - "7", - "41", - None, - None, - ) in record_observations - - binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - assert binding.turn_target_kind == "pane_id" - assert herdr_turns.refresh_turn_binding(config, binding).status in { - "updated", - "unchanged", - } - capture_key() - - backend.reconcile_once(client=client) - capture_key() - assert len(set(stable_keys)) == 1 - - -def test_one_hundred_interleaved_identity_observations_never_drift( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - real_stable_worker_key = herdr_cli.stable_worker_key - derived_keys: set[str] = set() - - def capture_stable_worker_key(*args: Any, **kwargs: Any) -> str: - stable_key = real_stable_worker_key(*args, **kwargs) - derived_keys.add(stable_key) - return stable_key - - monkeypatch.setattr(herdr_cli, "stable_worker_key", capture_stable_worker_key) - backend = _backend(tmp_path, "interleaved-identity") - pane = { - "workspace_id": "w65383a2e877513", - "pane_id": "w65383a2e877513:pA", - "terminal_id": "terminal-interleaved", - "agent": "claude", - "agent_status": "working", - } - client = _StaticClient( - workspaces=[{"id": pane["workspace_id"], "name": "Build"}], - panes=[pane], - ) - backend.reconcile_once(client=client) - binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - monkeypatch.setattr( - herdr_turns, - "_read_turn_for_binding", - lambda *_args, **_kwargs: { - "complete": True, - "user_text": "identity read", - "assistant_final_text": "stable", - "source_turn_id": "identity-turn", - "workspace_id": 7, - "pane_id": 41, - }, - ) - - observed_keys: set[str] = set() - - def remember_key() -> None: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - observed_keys.add(str(snapshot.workers[0].meta["stable_key"])) - - remember_key() - for index in range(100): - path = index % 5 - if path == 0: - assert backend.queue_event_envelope( - { - "event": "pane.created", - "data": {"pane": {**pane, "agent_status": "idle"}}, - } - ) - elif path == 1: - assert backend.queue_event_envelope( - { - "event": "pane.updated", - "data": { - "workspace_id": 7, - "pane_id": 41, - "revision": index, - }, - } - ) - elif path == 2: - assert backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": { - "workspace_id": 7, - "pane_id": 41, - "terminal_id": pane["terminal_id"], - "agent": "claude", - "agent_status": "working", - }, - } - ) - elif path == 3: - backend.reconcile_once(client=client) - binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - else: - assert herdr_turns.refresh_turn_binding( - backend.config, - binding, - ).status in {"updated", "unchanged"} - remember_key() - - assert len(observed_keys) == 1 - assert derived_keys == observed_keys - - -@pytest.mark.parametrize("turn_model", ["legacy", "observed"]) -@pytest.mark.parametrize( - "event_envelope", - [ - pytest.param( - { - "event": "pane.agent_status_changed", - "data": { - "pane_id": "w123456789abcde:pA", - "workspace_id": "w123456789abcde", - "agent": "claude", - "agent_status": "blocked", - }, - }, - id="herdr-074-status-event", - ), - pytest.param( - { - "event": "pane_updated", - "data": { - "type": "pane_updated", - "pane": { - "pane_id": "w123456789abcde:pA", - "workspace_id": "w123456789abcde", - "terminal_id": "terminal-decision", - "agent": "claude", - "agent_status": "blocked", - }, - }, - }, - id="herdr-075-pane-updated-event", - ), - pytest.param( - { - "event": "pane_agent_status_changed", - "data": { - "type": "pane_agent_status_changed", - "pane_id": "w123456789abcde:pA", - "workspace_id": "w123456789abcde", - "agent": "claude", - "display_agent": "Claude", - "agent_status": "blocked", - "state_labels": {}, - "title": None, - }, - }, - id="herdr-075-agent-status-event", - ), - ], -) -def test_idless_blocked_event_persists_and_lists_decision( - tmp_path: Path, - turn_model: str, - event_envelope: dict[str, Any], -) -> None: - backend, _binding = _decision_backend(tmp_path, turn_model) - refreshes: list[Any] = [] - - def refresh_current() -> None: - binding = next(iter(backend._bindings.values())) - refreshes.append( - herdr_turns.refresh_turn_binding( - backend.config, binding, adapter_timeout_seconds=1 - ) - ) - - backend.set_turn_refresh_callback(refresh_current) - - def handler(conn: _SocketConnection) -> None: - request = conn.read_request() - subscriptions = request["params"]["subscriptions"] - assert {"type": "pane.updated"} in subscriptions - assert { - "type": "pane.agent_status_changed", - "pane_id": "w123456789abcde:pA", - } in subscriptions - conn.send_json( - {"id": request["id"], "result": {"type": "subscription_started"}} - ) - conn.send_json(event_envelope) - - with _FakeHerdrSocketServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = backend._subscribe_event_stream(client) - envelope = client.read_event(stream.subscription_id, timeout=1) - assert envelope.get("id") is None - assert backend.queue_event_envelope(envelope) is True - client.close() - - assert refreshes == [herdr_turns.TurnRefreshResult("updated", 1, True)] - _assert_decision_persisted(backend) - - -@pytest.mark.parametrize("turn_model", ["legacy", "observed"]) -def test_reconcile_fallback_refreshes_and_lists_decision_without_status_event( - tmp_path: Path, - turn_model: str, -) -> None: - backend, _binding = _decision_backend(tmp_path, turn_model) - refreshes: list[Any] = [] - - def refresh_current() -> None: - binding = next(iter(backend._bindings.values())) - refreshes.append( - herdr_turns.refresh_turn_binding( - backend.config, binding, adapter_timeout_seconds=1 - ) - ) - - backend.set_turn_refresh_callback(refresh_current) - - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[ - { - "pane_id": "w123456789abcde:pA", - "terminal_id": "terminal-decision", - "agent": "claude", - "workspace_id": "w123456789abcde", - "agent_status": "blocked", - } - ], - agents=[], - ) - ) - - assert refreshes == [herdr_turns.TurnRefreshResult("updated", 1, True)] - _assert_decision_persisted(backend) - - -def test_backend_rejects_non_official_subscribe_method(tmp_path: Path) -> None: - config = _config(tmp_path, "custom-subscribe") - init_store(Path(config.db_path)) - - with pytest.raises(HerdrEventBackendError): - HerdrEventBackend(config, subscribe_method="custom.subscribe") - - -def test_pane_agent_detected_official_event_updates_worker_and_private_binding(tmp_path: Path) -> None: - backend = _backend(tmp_path, "agent-detected") - backend.reconcile_once(client=_StaticClient(workspaces=[{"id": "space-1", "name": "Build"}])) - - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent_id": "agent-2", - "name": "Agent Two", - "workspace_id": "space-1", - "pane_id": "pane-2", - "status": "running", - } - }} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") - assert snapshot is not None - assert {worker.id for worker in snapshot.workers} == {"agent-2"} - assert bindings[0].target_kind == "agent_id" - assert bindings[0].target_value == "agent-2" - _assert_no_public_json_forbidden(json.loads(snapshot.to_json())) - - -@pytest.mark.parametrize( - "event_name", - [ - "pane.created", - "pane.focused", - "pane.agent_detected", - "pane.agent_status_changed", - ], -) -@pytest.mark.parametrize("entity_name", ["agent", "worker"]) -def test_supported_nested_agent_or_worker_canonical_fields_cannot_mint_continuity( - tmp_path: Path, - event_name: str, - entity_name: str, -) -> None: - backend = _backend( - tmp_path, - f"nested-no-mint-{event_name}-{entity_name}", - ) - backend.reconcile_once( - client=_StaticClient(workspaces=[{"id": "wR9", "name": "Build"}]) - ) - entity = { - "worker_id": f"public-{entity_name}", - "agent_id": f"{entity_name}-target-secret", - "name": "codex", - "agent": "codex", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "status": "running", - "agent_session": { - "source": "compatibility-secret", - "agent": "codex", - "kind": "id", - "value": "compatibility-session-secret", - }, - } - - assert backend.queue_event_envelope( - {"event": event_name, "data": {entity_name: entity}} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert len(snapshot.workers) == 1 - worker = snapshot.workers[0] - assert "stable_key" not in worker.meta - assert "stable_key_version" not in worker.meta - assert not backend.config.installation_key_path.exists() - _assert_no_public_json_forbidden(json.loads(snapshot.to_json())) - - -def test_nested_compatibility_event_cannot_duplicate_authenticated_turn_owner( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "nested-turn-owner-conflict") - session = { - "source": "old-source-secret", - "agent": "codex", - "kind": "id", - "value": "shared-session-secret", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": session, - "status": "running", - } - ], - agents=[ - { - "worker_id": "public-old-owner", - "agent_id": "old-agent-target-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": session, - "status": "running", - } - ], - ) - ) - - assert backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "worker_id": "public-compatibility-claimant", - "agent_id": "new-agent-target-secret", - "agent": "codex", - "agent_session": { - "source": "new-source-secret", - "agent": "codex", - "kind": "id", - "value": "shared-session-secret", - }, - "status": "running", - } - }} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert len(snapshot.workers) == len(bindings) == 1 - assert "stable_key" not in snapshot.workers[0].meta - assert bindings[0].sendable is False - assert bindings[0].reason == "ambiguous_pane_match" - assert bindings[0].turn_target_kind is None - assert bindings[0].turn_target_value is None - - -@pytest.mark.parametrize("entity_source", ["top_level", "pane"]) -def test_official_pane_tuple_provenance_mints_continuity( - tmp_path: Path, - entity_source: str, -) -> None: - backend = _backend(tmp_path, f"official-pane-{entity_source}") - backend.reconcile_once( - client=_StaticClient(workspaces=[{"id": "wR9", "name": "Build"}]) - ) - pane = { - "agent": "codex", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "official-terminal-secret", - "status": "running", - "agent_session": { - "source": "official-source-secret", - "agent": "codex", - "kind": "id", - "value": "official-session-secret", - }, - } - payload = pane if entity_source == "top_level" else {"pane": pane} - - assert backend.queue_event_envelope( - {"event": "pane.created", "data": payload} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert len(snapshot.workers) == 1 - assert snapshot.workers[0].meta["stable_key"].startswith("wsk1_") - assert snapshot.workers[0].meta["stable_key_version"] == 1 - assert backend.config.installation_key_path.exists() - - -@pytest.mark.parametrize( - "event_name", - ["pane.agent_detected", "pane.agent_status_changed"], -) -@pytest.mark.parametrize("key_failure", [False, True]) -def test_official_idless_event_reuses_single_authenticated_pane_owner( - tmp_path: Path, - key_failure: bool, - event_name: str, -) -> None: - backend = _backend( - tmp_path, - f"event-pane-owner-id-churn-{key_failure}-{event_name}", - ) - old_session = { - "source": "old-source-secret", - "agent": "codex", - "kind": "id", - "value": "old-session-secret", - } - pane = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": old_session, - "status": "running", - } - agent = { - "worker_id": "public-old-owner", - "agent_id": "old-agent-target-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": old_session, - "status": "running", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane], - agents=[agent], - ) - ) - before = latest_snapshot(backend.db_path, backend.config.host_id) - before_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert before is not None - assert len(before.workers) == len(before_bindings) == 1 - worker_id = before.workers[0].id - stable_key = before.workers[0].meta["stable_key"] - if key_failure: - backend.config.installation_key_marker_path.unlink() - - event_payload = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": { - "worker_id": "public-new-owner", - "agent_id": "new-agent-target-secret", - "terminal_id": "new-terminal-secret", - "agent": "codex", - "agent_session": { - "source": "new-source-secret", - "agent": "codex", - "kind": "id", - "value": "new-session-secret", - }, - "status": "working", - }, - } - assert backend.queue_event_envelope( - {"event": event_name, "data": event_payload} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert after is not None - assert len(after.workers) == len(bindings) == 1 - assert after.workers[0].id == worker_id - assert after.workers[0].meta["stable_key"] == stable_key - assert after.backend_health[0].status == "healthy" - if key_failure: - assert not backend.config.installation_key_marker_path.exists() - assert bindings[0].worker_id == worker_id - assert bindings[0].private_fingerprint == before_bindings[0].private_fingerprint - assert (bindings[0].target_kind, bindings[0].target_value) == ( - before_bindings[0].target_kind, - before_bindings[0].target_value, - ) - assert (bindings[0].turn_target_kind, bindings[0].turn_target_value) == ( - before_bindings[0].turn_target_kind, - before_bindings[0].turn_target_value, - ) - _assert_no_public_json_forbidden(json.loads(after.to_json())) - - -@pytest.mark.parametrize("shared_owner", ["terminal_id", "agent_session"]) -@pytest.mark.parametrize("key_failure", [False, True]) -def test_incremental_shared_private_owner_fails_closed_across_canonical_panes( - tmp_path: Path, - shared_owner: str, - key_failure: bool, -) -> None: - backend = _backend( - tmp_path, - f"event-shared-{shared_owner}-owner-{key_failure}", - ) - backend.reconcile_once( - client=_StaticClient(workspaces=[{"id": "wR9", "name": "Build"}]) - ) - - for suffix, pane_id in (("a", "wR9:pA"), ("b", "wR9:pB")): - assert backend.queue_event_envelope( - {"event": "pane.created", "data": { - "workspace_id": "wR9", - "pane_id": pane_id, - "terminal_id": ( - "shared-terminal-secret" - if shared_owner == "terminal_id" - else f"terminal-{suffix}-secret" - ), - "agent_id": f"agent-{suffix}-secret", - "agent": "codex", - "agent_session": { - "source": f"source-{suffix}-secret", - "agent": "codex", - "kind": "id", - "value": ( - "shared-session-secret" - if shared_owner == "agent_session" - else f"session-{suffix}-secret" - ), - }, - "status": "running", - }} - ) - if suffix == "a" and key_failure: - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.created", "data": { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": ( - "shared-terminal-secret" - if shared_owner == "terminal_id" - else "terminal-b-secret" - ), - "agent_id": "agent-b-secret", - "agent": "codex", - "agent_session": { - "source": "source-b-secret", - "agent": "codex", - "kind": "id", - "value": ( - "shared-session-secret" - if shared_owner == "agent_session" - else "session-b-secret" - ), - }, - "status": "running", - }} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert len(snapshot.workers) == len(bindings) == 1 - if key_failure: - assert snapshot.workers[0].meta["stable_key"].startswith("wsk1_") - assert snapshot.workers[0].meta["stable_key_version"] == 1 - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "continuity_unavailable" - assert bindings[0].sendable is True - assert bindings[0].reason is None - assert bindings[0].turn_target_kind is not None - assert bindings[0].turn_target_value is not None - else: - assert "stable_key" not in snapshot.workers[0].meta - assert "stable_key_version" not in snapshot.workers[0].meta - assert bindings[0].sendable is False - assert bindings[0].reason == "ambiguous_pane_match" - assert bindings[0].turn_target_kind is None - assert bindings[0].turn_target_value is None - - -@pytest.mark.parametrize("complete_move", [False, True]) -def test_move_into_owned_pane_fails_closed_for_both_owners( - tmp_path: Path, - complete_move: bool, -) -> None: - backend = _backend( - tmp_path, - f"event-move-owned-destination-{complete_move}", - ) - panes = [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent A", - "status": "running", - }, - { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "agent": "Agent B", - "status": "running", - }, - ] - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=panes, - ) - ) - - payload: dict[str, Any] = { - "previous_pane_id": "wR9:pB", - "new_pane_id": "wR9:pA", - } - if complete_move: - payload["pane"] = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent B", - "status": "running", - } - assert backend.queue_event_envelope( - {"event": "pane.moved", "data": payload} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert len(snapshot.workers) == len(bindings) == 2 - assert all("stable_key" not in worker.meta for worker in snapshot.workers) - assert all("stable_key_version" not in worker.meta for worker in snapshot.workers) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert all(binding.turn_target_kind is None for binding in bindings) - assert all(binding.turn_target_value is None for binding in bindings) - - -def test_key_failure_precedes_move_conflict_mutation(tmp_path: Path) -> None: - backend = _backend(tmp_path, "event-move-conflict-key-failure") - panes = [ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent A", - "status": "running", - }, - { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "agent": "Agent B", - "status": "running", - }, - ] - before = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=panes, - ) - ) - before_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - backend.config.installation_key_marker_path.unlink() - - assert backend.queue_event_envelope( - {"event": "pane.moved", "data": { - "previous_pane_id": "wR9:pB", - "pane": { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent B", - "status": "running", - }, - }} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.workers == before.workers - assert after.spaces == before.spaces - assert after.backend_health[0].status == "degraded" - assert after.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == before_bindings - ) - - -@pytest.mark.parametrize("key_failure", [False, True]) -def test_authoritative_move_resolves_agent_targeted_source_by_previous_pane( - tmp_path: Path, - key_failure: bool, -) -> None: - backend = _backend( - tmp_path, - f"event-move-agent-targeted-source-{key_failure}", - ) - old_session = { - "source": "old-source-secret", - "agent": "codex", - "kind": "id", - "value": "old-session-secret", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[ - {"id": "wR9", "name": "Source"}, - {"id": "wD2", "name": "Destination"}, - ], - panes=[ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": old_session, - "status": "running", - } - ], - agents=[ - { - "worker_id": "public-source-owner", - "agent_id": "old-agent-target-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "old-terminal-secret", - "agent": "codex", - "agent_session": old_session, - "status": "running", - } - ], - ) - ) - before = latest_snapshot(backend.db_path, backend.config.host_id) - before_binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - assert before is not None - worker_id = before.workers[0].id - original_key = before.workers[0].meta["stable_key"] - if key_failure: - key_bytes = backend.config.installation_key_path.read_bytes() - backend.config.installation_key_path.write_bytes( - bytes(byte ^ 0xFF for byte in key_bytes) - ) - - assert backend.queue_event_envelope( - {"event": "pane.moved", "data": { - "previous_pane_id": "wR9:pA", - "pane": { - "workspace_id": "wD2", - "pane_id": "wD2:p7", - "terminal_id": "new-terminal-secret", - "agent": "codex", - "agent_session": { - "source": "new-source-secret", - "agent": "codex", - "kind": "id", - "value": "new-session-secret", - }, - "status": "running", - }, - }} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert after is not None - assert len(after.workers) == len(bindings) == 1 - assert after.workers[0].id == worker_id - if key_failure: - assert after.workers == before.workers - assert after.workers[0].meta["stable_key"] == original_key - assert bindings == [before_binding] - assert after.backend_health[0].status == "degraded" - assert after.backend_health[0].outcome == "continuity_unavailable" - else: - assert after.workers[0].meta["stable_key"] != original_key - assert after.workers[0].space_id == "wD2" - assert bindings[0].private_fingerprint == before_binding.private_fingerprint - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == "new-terminal-secret" - assert bindings[0].turn_target_kind == "codex_session_id" - assert bindings[0].turn_target_value == "new-session-secret" - - -def test_reconcile_retains_authenticated_snapshot_until_installation_key_recovers( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "reconcile-key-recovery") - client = _StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[ - { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-secret", - "agent": "codex", - "status": "running", - } - ], - agents=[ - { - "worker_id": "public-worker", - "agent_id": "agent-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-secret", - "agent": "codex", - "status": "running", - } - ], - ) - first = backend.reconcile_once(client=client) - first_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - stable_key = first.workers[0].meta["stable_key"] - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - - degraded = backend.reconcile_once(client=client) - - assert degraded.workers == first.workers - assert degraded.spaces == first.spaces - assert degraded.backend_health[0].status == "degraded" - assert degraded.backend_health[0].outcome == "continuity_unavailable" - assert degraded.backend_health[0].counts == {"spaces": 1, "workers": 1} - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == first_bindings - ) - - previous_max_workers = backend.max_workers - backend.max_workers = 1 - capped = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - agents=[ - {"worker_id": "agent-a", "agent": "Agent A"}, - {"worker_id": "agent-b", "agent": "Agent B"}, - ], - ) - ) - assert capped.backend_health[0].outcome == "continuity_unavailable" - backend.max_workers = previous_max_workers - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - recovered = backend.reconcile_once(client=client) - - assert recovered.backend_health[0].status == "healthy" - assert recovered.workers[0].meta["stable_key"] == stable_key - recovered_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - - unmatched = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[], - agents=[ - { - "worker_id": "public-worker", - "agent_id": "agent-secret", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-secret", - "agent": "codex", - "status": "running", - } - ], - ) - ) - - assert unmatched.workers == recovered.workers - assert unmatched.spaces == recovered.spaces - assert unmatched.backend_health[0].status == "degraded" - assert unmatched.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == recovered_bindings - ) - - recovered_again = backend.reconcile_once(client=client) - assert recovered_again.backend_health[0].status == "healthy" - assert recovered_again.workers[0].meta["stable_key"] == stable_key - - -def test_incomplete_pane_event_does_not_clear_continuity_failure( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "incomplete-pane-key-recovery") - pane = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-secret", - "agent": "codex", - "status": "running", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane], - ) - ) - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane": pane}} - ) - failed = latest_snapshot(backend.db_path, backend.config.host_id) - assert failed is not None - assert failed.backend_health[0].outcome == "continuity_unavailable" - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane": {"pane_id": "wR9:pA"}}} - ) - - incomplete = latest_snapshot(backend.db_path, backend.config.host_id) - assert incomplete is not None - assert incomplete.backend_health[0].status == "degraded" - assert incomplete.backend_health[0].outcome == "continuity_unavailable" - - -def test_over_cap_authenticated_retry_does_not_clear_continuity_failure( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "over-cap-key-recovery") - pane_a = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "agent": "Agent A", - "status": "running", - } - pane_b = { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "agent": "Agent B", - "status": "running", - } - first = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane_a], - ) - ) - first_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - backend.max_workers = 1 - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.created", "data": {"pane": pane_b}} - ) - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - assert backend.queue_event_envelope( - {"event": "pane.created", "data": {"pane": pane_b}} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.workers == first.workers - assert after.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == first_bindings - ) - - -def test_conflicting_close_does_not_revalidate_continuity( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "conflicting-close-key-recovery") - pane_a = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-a-secret", - "agent": "Agent A", - "status": "running", - } - pane_b = { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "terminal-b-secret", - "agent": "Agent B", - "status": "running", - } - first = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane_a, pane_b], - ) - ) - first_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane": pane_a}} - ) - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - conflicting_close = { - **pane_a, - "terminal_id": pane_b["terminal_id"], - } - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane": conflicting_close}} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.workers == first.workers - assert after.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == first_bindings - ) - - -def test_conflicting_upsert_preserves_latched_authenticated_state( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "conflicting-upsert-key-recovery") - pane_a = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-a-secret", - "agent": "Agent A", - "status": "running", - } - pane_b = { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "terminal-b-secret", - "agent": "Agent B", - "status": "running", - } - first = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build"}], - panes=[pane_a, pane_b], - ) - ) - first_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - marker = backend.config.installation_key_marker_path.read_bytes() - backend.config.installation_key_marker_path.unlink() - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane": pane_a}} - ) - - backend.config.installation_key_marker_path.write_bytes(marker) - os.chmod(backend.config.installation_key_marker_path, 0o600) - conflicting_pane = { - **pane_a, - "terminal_id": pane_b["terminal_id"], - "status": "blocked", - } - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane": conflicting_pane}} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.workers == first.workers - assert after.backend_health[0].status == "degraded" - assert after.backend_health[0].outcome == "continuity_unavailable" - assert ( - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - == first_bindings - ) - - -def test_official_pane_event_generic_id_remains_private_binding_only(tmp_path: Path) -> None: - backend = _backend(tmp_path, "pane-id-private") - backend.reconcile_once(client=_StaticClient(workspaces=[{"id": "space-1", "name": "Build"}])) - - assert ( - backend.queue_event_envelope( - {"event": "pane.created", "data": { - "id": "pane-secret", - "agent": "Agent Two", - "workspace_id": "space-1", - "status": "running", - }} - ) - is True - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") - assert snapshot is not None - public_json = snapshot.to_json() - assert "pane-secret" not in public_json - assert {worker.id for worker in snapshot.workers} == {"Agent Two"} - assert bindings[0].target_kind == "pane_id" - assert bindings[0].target_value == "pane-secret" - _assert_no_public_json_forbidden(json.loads(public_json)) - - -def test_unknown_and_malformed_known_events_do_not_mutate_any_public_or_private_state( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "event-noop") - backend.reconcile_once(client=_initial_pane_client()) - before = _no_op_state(backend) - - for envelope in ( - {"event": "unknown.future", "data": {"pane_id": "pane-secret", "stdout": "secret"}}, - {"event": "workspace.created", "data": {"pane_id": "pane-secret", "stdout": "secret"}}, - {"event": "workspace.renamed", "data": {"new_name": "secret"}}, - {"event": "worktree.created", "data": {"worktree_id": "worktree-secret", "stderr": "secret"}}, - {"event": "pane.created", "data": {"labels": ["agent"], "argv": ["secret"]}}, - {"event": "pane.agent_detected", "data": []}, - {"event": "pane.agent_status_changed", "data": {"status": "failed", "stderr": "secret"}}, - {"event": "pane.output_matched", "data": { - "pane_id": "pane-secret", - "terminal_id": "terminal-secret", - "stdout": "secret", - "stderr": "secret", - "token": "secret", - }}, - ): - backend.queue_event_envelope(envelope) - - after = _no_op_state(backend) - assert after == before - _assert_no_public_json_forbidden(json.loads(after[0])) - assert "pane-secret" not in after[0] - assert "terminal-secret" not in after[0] - assert "secret" not in after[0] - - -def test_worktree_events_only_update_existing_workspace_observations(tmp_path: Path) -> None: - backend = _backend(tmp_path, "worktree-adjacent") - backend.reconcile_once(client=_StaticClient(workspaces=[{"id": "space-1", "name": "Build"}])) - before = latest_snapshot(backend.db_path, backend.config.host_id) - assert before is not None - - assert ( - backend.queue_event_envelope( - {"event": "worktree.created", "data": {"workspace_id": "new-space", "name": "Should Not Appear"}} - ) - is True - ) - unchanged = latest_snapshot(backend.db_path, backend.config.host_id) - assert unchanged is not None - assert [space.id for space in unchanged.spaces] == ["space-1"] - assert unchanged.spaces[0].name == "Build" - - assert ( - backend.queue_event_envelope( - {"event": "worktree.opened", "data": { - "workspace_id": "space-1", - "name": "Build Worktree", - "status": "active", - }} - ) - is True - ) - updated = latest_snapshot(backend.db_path, backend.config.host_id) - assert updated is not None - assert [space.id for space in updated.spaces] == ["space-1"] - assert updated.spaces[0].name == "Build Worktree" - _assert_no_public_json_forbidden(json.loads(updated.to_json())) - -def test_run_forever_reconnect_accepts_identical_idless_event_again(tmp_path: Path) -> None: - config = _config(tmp_path, "reconnect-resubscribe") - init_store(Path(config.db_path)) - - class SequenceClient(_StaticClient): - def __init__(self, label: str, events: list[Any], *, pane_status: str) -> None: - super().__init__( - workspaces=[{"id": "space-1", "name": "Build"}], - panes=[ - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": pane_status, - } - ], - ) - self.label = label - self.events = list(events) - self.subscriptions: list[tuple[str, dict[str, Any]]] = [] - self.read_calls = 0 - self.closed = False - - def connect(self) -> None: - return None - - def close(self) -> None: - self.closed = True - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - self.subscriptions.append((method, dict(params))) - return SimpleNamespace(subscription_id=f"{self.label}-sub") - - def read_event(self, subscription_id: str, *, timeout: float | None = None) -> dict[str, Any]: - self.read_calls += 1 - if not self.events: - backend.stop_event.set() - raise HerdrSocketTimeoutError("idle") - event = self.events.pop(0) - if event == "disconnect": - raise HerdrSocketDisconnectedError("disconnect") - return dict(event) - - working = _status_event("working") - first = SequenceClient("first", [working, "disconnect"], pane_status="idle") - second = SequenceClient("second", [working], pane_status="idle") - clients = [first, second] - backend = HerdrEventBackend( - config, - client_factory=lambda _config: clients.pop(0), - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - - backend.run_forever() - - expected = { - "subscriptions": [ - {"type": name} - for name in HERDR_OFFICIAL_EVENT_NAMES - if name not in {"pane.agent_status_changed", "pane.output_matched"} - ] - + [{"type": "pane.agent_status_changed", "pane_id": "pane-1"}] - } - assert first.subscriptions == [(HERDR_EVENTS_SUBSCRIBE_METHOD, expected)] - assert second.subscriptions == [(HERDR_EVENTS_SUBSCRIBE_METHOD, expected)] - assert first.read_calls == 2 - assert second.read_calls == 2 - assert first.closed is True - assert second.closed is True - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "active" - - -def test_run_forever_retains_complete_reconcile_when_event_stream_disconnects( - tmp_path: Path, -) -> None: - config = _config(tmp_path, "reconcile-before-stream-disconnect") - init_store(Path(config.db_path)) - - class DisconnectingClient(_StaticClient): - def __init__(self) -> None: - super().__init__(workspaces=[{"id": "space-1", "name": "Build"}]) - - def connect(self) -> None: - return None - - def close(self) -> None: - return None - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - return SimpleNamespace(subscription_id="disconnect-sub") - - def read_event( - self, - subscription_id: str, - *, - timeout: float | None = None, - ) -> dict[str, Any]: - backend.stop_event.set() - raise HerdrSocketDisconnectedError("event stream closed") - - backend = HerdrEventBackend( - config, - client_factory=lambda _config: DisconnectingClient(), - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - - backend.run_forever() - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.backend_health[0].status == "healthy" - assert snapshot.backend_health[0].outcome == "healthy_non_empty" - - -def test_start_stop_are_idempotent_and_bounded_for_idle_subscription(tmp_path: Path) -> None: - config = Config( - host_id="bounded-stop", - data_dir=tmp_path, - db_path=tmp_path / "bounded-stop.db", - herdr_backend="socket", - herdr_timeout_seconds=0.05, - ) - init_store(Path(config.db_path)) - - class IdleClient(_StaticClient): - def __init__(self) -> None: - super().__init__() - self.subscriptions = 0 - - def connect(self) -> None: - return None - - def close(self) -> None: - return None - - def subscribe( - self, - method: str, - params: Mapping[str, Any], - *, - timeout: float | None = None, - event_timeout: float | None = None, - ) -> Any: - self.subscriptions += 1 - return SimpleNamespace(subscription_id="idle-sub") - - def read_event(self, subscription_id: str, *, timeout: float | None = None) -> dict[str, Any]: - raise HerdrSocketTimeoutError("idle") - - client = IdleClient() - backend = HerdrEventBackend( - config, - client_factory=lambda _config: client, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - - started = time.monotonic() - backend.start(wait_for_reconcile=True, timeout_seconds=0.2) - deadline = time.monotonic() + 0.5 - while client.subscriptions < 1 and time.monotonic() < deadline: - time.sleep(0.01) - assert backend.ready is True - assert client.subscriptions >= 1 - - backend.stop() - backend.stop() - - assert backend.running is False - assert time.monotonic() - started < 2.0 - - -@pytest.mark.parametrize("batched", [False, True], ids=["one-flush-per-event", "one-batch"]) -def test_real_idless_working_idle_working_preserves_every_transition( - tmp_path: Path, - monkeypatch: Any, - batched: bool, -) -> None: - backend = _backend( - tmp_path, - f"real-idless-transitions-{batched}", - debounce_seconds=60 if batched else 0, - ) - backend.reconcile_once(client=_initial_pane_client()) - bindings_before = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - applied: list[str] = [] - original_apply = backend._apply_event - - def recording_apply(event: Any) -> bool: - applied.append(str(event.payload["status"])) - return original_apply(event) - - monkeypatch.setattr(backend, "_apply_event", recording_apply) - observed_statuses: list[str] = [] - accepted = [] - for status in ("working", "idle", "working"): - accepted.append(backend.queue_event_envelope(_status_event(status), flush=not batched)) - if not batched: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - observed_statuses.append(snapshot.workers[0].status) - if batched: - backend.flush() - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings_after = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert accepted == [True, True, True] - assert applied == ["working", "idle", "working"] - if not batched: - assert observed_statuses == ["active", "idle", "active"] - assert snapshot.workers[0].status == "active" - assert len(snapshot.workers) == 1 - assert bindings_after == bindings_before - - -@pytest.mark.parametrize("batched", [False, True], ids=["separate-flushes", "one-batch"]) -def test_adjacent_idless_duplicates_do_not_duplicate_persisted_effects( - tmp_path: Path, - batched: bool, -) -> None: - backend = _backend( - tmp_path, - f"idless-repeat-effects-{batched}", - debounce_seconds=60 if batched else 0, - ) - backend.reconcile_once(client=_initial_pane_client()) - before = _persisted_event_effect_counts(backend) - blocked = _status_event("blocked") - - assert backend.queue_event_envelope(blocked, flush=not batched) is True - if batched: - assert backend.queue_event_envelope(blocked, flush=False) is True - backend.flush() - after = _persisted_event_effect_counts(backend) - else: - after_first = _persisted_event_effect_counts(backend) - assert backend.queue_event_envelope(blocked, flush=True) is True - after = _persisted_event_effect_counts(backend) - assert after == after_first - - assert after["snapshots"] == before["snapshots"] + 1 - assert after["events"] == before["events"] + 1 - assert after["workers"] == before["workers"] == 1 - assert after["worker_bindings"] == before["worker_bindings"] == 1 - assert after["attention_items"] == before["attention_items"] + 1 - assert after["connector_outbox"] == before["connector_outbox"] + 1 - - -def test_snapshot_observation_context_matches_each_herdr_persistence_barrier( - tmp_path: Path, - monkeypatch: Any, -) -> None: - calls: list[SnapshotObservationContext] = [] - binding_calls: list[tuple[Any, str | None, bool, bool]] = [] - original_save_snapshot = save_snapshot - - def recording_save_snapshot( - db_path: Path, - snapshot: Any, - *, - turn_model: str, - observation: SnapshotObservationContext | None = None, - worker_bindings: Any = None, - binding_backend: str | None = None, - binding_observation_authoritative: bool = False, - binding_workers_present: bool = True, - ) -> bool: - assert observation is not None - calls.append(observation) - binding_calls.append( - ( - worker_bindings, - binding_backend, - binding_observation_authoritative, - binding_workers_present, - ) - ) - return original_save_snapshot( - db_path, - snapshot, - turn_model=turn_model, - observation=observation, - worker_bindings=worker_bindings, - binding_backend=binding_backend, - binding_observation_authoritative=binding_observation_authoritative, - binding_workers_present=binding_workers_present, - ) - - monkeypatch.setattr( - "tendwire.backends.herdr_events.save_snapshot", - recording_save_snapshot, - ) - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, "herdr-observation-context") - - backend.reconcile_once(client=_initial_pane_client()) - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - backend.queue_event_envelope(_status_event("blocked")) - _set_observation_time(monkeypatch, "2026-01-01T00:00:20Z") - backend._mark_worker_cap_exceeded_locked(999) - _set_observation_time(monkeypatch, "2026-01-01T00:00:30Z") - backend._mark_unhealthy("socket_disconnected") - - assert [(call.authority, call.observed_at) for call in calls] == [ - ("complete", "2026-01-01T00:00:00Z"), - ("positive", "2026-01-01T00:00:10Z"), - ("none", "2026-01-01T00:00:20Z"), - ("none", "2026-01-01T00:00:30Z"), - ] - assert binding_calls[0][0] - assert binding_calls[0][1:] == ("herdr", True, True) - assert all(call[0] is None for call in binding_calls[1:]) - -def test_event_snapshot_expiry_serializes_delayed_older_observer( - tmp_path: Path, - monkeypatch: Any, -) -> None: - from tendwire.store import sqlite as store_sqlite - - backend = _backend(tmp_path, "herdr-atomic-expiry") - worker = Worker(id="worker-expiry", name="Worker Expiry", status="active") - - def binding(target: str, observed_at: str, fingerprint: str) -> WorkerBinding: - return WorkerBinding( - host_id=backend.config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value=target, - sendable=True, - observed_at=observed_at, - private_fingerprint=fingerprint, - ) - - initial = Snapshot( - host_id=backend.config.host_id, - updated_at="2026-01-01T00:00:00+00:00", - workers=[worker], - ) - init_store(backend.db_path) - assert save_snapshot( - backend.db_path, - initial, - worker_bindings=[ - binding( - "initial-private-target", - "2026-01-01T00:00:00+00:00", - "initial-private-owner", - ) - ], - binding_backend="herdr", - binding_observation_authoritative=True, - ) is True - older = Snapshot( - host_id=backend.config.host_id, - updated_at="2026-01-01T00:00:01+00:00", - workers=[], - ) - newer = Snapshot( - host_id=backend.config.host_id, - updated_at="2026-01-01T00:00:02+00:00", - workers=[worker], - ) - newer_binding = binding( - "newer-private-target", - "2026-01-01T00:00:02+00:00", - "newer-private-owner", - ) - original_expire = store_sqlite._expire_stale_worker_bindings_conn - older_inside_transaction = threading.Event() - release_older = threading.Event() - - def delayed_expire(conn: sqlite3.Connection, host_id: str, **kwargs: Any) -> int: - if kwargs.get("now") == "2026-01-01T00:00:01+00:00": - older_inside_transaction.set() - assert release_older.wait(timeout=10) - return original_expire(conn, host_id, **kwargs) - - monkeypatch.setattr( - store_sqlite, - "_expire_stale_worker_bindings_conn", - delayed_expire, - ) - errors: list[BaseException] = [] - - def save_older() -> None: - try: - backend._save_snapshot( - older, - observation=SnapshotObservationContext( - authority="complete", - observed_at=older.updated_at, - ), - worker_bindings=[], - binding_observation_authoritative=True, - binding_workers_present=False, - ) - except BaseException as exc: - errors.append(exc) - - def save_newer() -> None: - try: - backend._save_snapshot( - newer, - observation=SnapshotObservationContext( - authority="complete", - observed_at=newer.updated_at, - ), - worker_bindings=[newer_binding], - binding_observation_authoritative=True, - binding_workers_present=True, - ) - except BaseException as exc: - errors.append(exc) - - older_thread = threading.Thread(target=save_older) - newer_thread = threading.Thread(target=save_newer) - older_thread.start() - assert older_inside_transaction.wait(timeout=10) - newer_thread.start() - time.sleep(0.05) - release_older.set() - older_thread.join(timeout=10) - newer_thread.join(timeout=10) - - assert not errors - assert not older_thread.is_alive() and not newer_thread.is_alive() - active = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - now="2026-01-01T00:00:03+00:00", - ) - assert len(active) == 1 - assert active[0].target_value == "newer-private-target" - - -def test_same_fingerprint_observations_refresh_attention_and_run_bounded_cadence( - tmp_path: Path, - monkeypatch: Any, -) -> None: - config = Config( - host_id="same-fingerprint-cadence", - data_dir=tmp_path, - db_path=tmp_path / "same-fingerprint-cadence.db", - herdr_backend="socket", - snapshot_retention_days=14, - snapshot_retention_count=1, - snapshot_maintenance_batch_size=100, - store_maintenance_cadence_seconds=3600, - acknowledged_final_retention_days=33, - acknowledged_final_retention_count=456, - command_retry_horizon_seconds=120, - command_receipt_retention_seconds=691_200, - command_receipt_retention_count=77, - ) - init_store(Path(config.db_path)) - maintenance_times = iter( - ( - "2026-01-01T00:00:00Z", - "2026-01-01T00:00:10Z", - "2026-01-01T00:00:20Z", - "2026-01-01T01:00:10Z", - ) - ) - maintenance_calls: list[ - tuple[SnapshotRetentionPolicy, int, int, int, int, int, int, dict[str, Any]] - ] = [] - - def fixed_clock_maintenance( - db_path: Path, - *, - policy: SnapshotRetentionPolicy, - turn_model: str = "legacy", - acknowledged_final_retention_days: int = 30, - acknowledged_final_retention_count: int = 4096, - command_retry_horizon_seconds: int = 604_800, - command_receipt_retention_seconds: int = 2_592_000, - command_receipt_retention_count: int = 4096, - cadence_seconds: int = 3600, - now: str | None = None, - ) -> dict[str, Any]: - assert now is None - result = maybe_run_automatic_store_maintenance( - db_path, - policy=policy, - turn_model=turn_model, - cadence_seconds=cadence_seconds, - acknowledged_final_retention_days=acknowledged_final_retention_days, - acknowledged_final_retention_count=acknowledged_final_retention_count, - command_retry_horizon_seconds=command_retry_horizon_seconds, - command_receipt_retention_seconds=command_receipt_retention_seconds, - command_receipt_retention_count=command_receipt_retention_count, - now=next(maintenance_times), - ) - maintenance_calls.append( - ( - policy, - acknowledged_final_retention_days, - acknowledged_final_retention_count, - command_retry_horizon_seconds, - command_receipt_retention_seconds, - command_receipt_retention_count, - cadence_seconds, - result, - ) - ) - return result - - monkeypatch.setattr( - "tendwire.backends.herdr_events.maybe_run_automatic_store_maintenance", - fixed_clock_maintenance, - ) - saved_fingerprints: list[str] = [] - - def recording_save( - db_path: Path, - snapshot: Any, - *, - turn_model: str, - observation: SnapshotObservationContext | None = None, - worker_bindings: Any = None, - binding_backend: str | None = None, - binding_observation_authoritative: bool = False, - binding_workers_present: bool = True, - ) -> bool: - saved_fingerprints.append(snapshot.content_fingerprint) - return save_snapshot( - db_path, - snapshot, - turn_model=turn_model, - observation=observation, - worker_bindings=worker_bindings, - binding_backend=binding_backend, - binding_observation_authoritative=binding_observation_authoritative, - binding_workers_present=binding_workers_present, - ) - - monkeypatch.setattr("tendwire.backends.herdr_events.save_snapshot", recording_save) - backend = HerdrEventBackend(config, debounce_seconds=0) - - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend.reconcile_once(client=_initial_pane_client()) - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - assert backend.queue_event_envelope(_status_event("blocked")) is True - assert _attention_lifecycle_rows(backend)[0][4] == "2026-01-01T00:00:10+00:00" - - _set_observation_time(monkeypatch, "2026-01-01T00:00:20Z") - backend._persist_current_state(observed_at="2026-01-01T00:00:20Z") - assert _attention_lifecycle_rows(backend)[0][4] == "2026-01-01T00:00:20+00:00" - - _set_observation_time(monkeypatch, "2026-01-01T01:00:10Z") - backend._persist_current_state(observed_at="2026-01-01T01:00:10Z") - - assert len(maintenance_calls) == 4 - assert len(saved_fingerprints) == 4 - assert len(set(saved_fingerprints[1:])) == 1 - assert [result["status"] for *_, result in maintenance_calls] == [ - "ok", - "not_due", - "not_due", - "ok", - ] - assert sum(bool(result["due"]) for *_, result in maintenance_calls) == 2 - assert { - ( - policy.retention_days, - policy.retention_count, - policy.batch_size, - final_days, - final_count, - retry_horizon, - retention_seconds, - retention_count, - cadence, - ) - for ( - policy, - final_days, - final_count, - retry_horizon, - retention_seconds, - retention_count, - cadence, - _, - ) in maintenance_calls - } == {(14, 1, 100, 33, 456, 120, 691_200, 77, 3600)} - assert _table_count(backend.db_path, config.host_id, "snapshots") == 1 - assert _attention_lifecycle_rows(backend)[0][4] == "2026-01-01T01:00:10+00:00" - assert backend.operational_status["automatic_maintenance"] == { - "ok": True, - "status": "ok", - "due": True, - "examined": 1, - "deleted": 1, - "remaining_candidates": False, - } - - -def test_maintenance_failure_cannot_undo_committed_backend_snapshot( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "maintenance-failure-contained") - - def fail_maintenance(*_args: Any, **_kwargs: Any) -> dict[str, Any]: - raise RuntimeError(f"private maintenance failure at {tmp_path}/secret.db") - - monkeypatch.setattr( - "tendwire.backends.herdr_events.maybe_run_automatic_store_maintenance", - fail_maintenance, - ) - - snapshot = backend.reconcile_once(client=_initial_pane_client()) - persisted = latest_snapshot(backend.db_path, backend.config.host_id) - status = backend.operational_status - encoded = json.dumps(status, sort_keys=True) - - assert persisted is not None - assert persisted.content_fingerprint == snapshot.content_fingerprint - assert status["automatic_maintenance"] == { - "ok": False, - "status": "failed", - "due": False, - "examined": 0, - "deleted": 0, - "remaining_candidates": False, - } - assert str(tmp_path) not in encoded - assert "secret.db" not in encoded - _assert_no_public_json_forbidden(status) - - -def test_incremental_positive_lifecycle_and_complete_absence_are_separate( - tmp_path: Path, - monkeypatch: Any, -) -> None: - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, "incremental-lifecycle") - backend.reconcile_once(client=_initial_pane_client()) - assert _attention_lifecycle_rows(backend) == () - - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - backend.queue_event_envelope(_status_event("blocked")) - opened = _attention_lifecycle_rows(backend) - assert len(opened) == 1 - assert opened[0][0:2] == (1, "open") - assert opened[0][5:7] == (None, 0) - initial_rank = int(opened[0][9]) - assert _attention_event_types(backend) == ["attention_created"] - - _set_observation_time(monkeypatch, "2026-01-01T00:00:20Z") - backend.queue_event_envelope(_status_event("failed")) - escalated = _attention_lifecycle_rows(backend) - assert len(escalated) == 1 - assert escalated[0][0:2] == (1, "open") - assert int(escalated[0][9]) > initial_rank - assert _attention_event_types(backend) == [ - "attention_created", - "attention_escalated", - ] - current = list_attention_items(backend.db_path, backend.config.host_id) - assert len(current) == 1 - assert current[0]["severity"] == "critical" - - _set_observation_time(monkeypatch, "2026-01-01T00:00:30Z") - backend.reconcile_once(client=_initial_pane_client()) - first_missing = _attention_lifecycle_rows(backend) - assert first_missing[0][0:2] == (1, "open") - assert first_missing[0][5] is not None - assert first_missing[0][6] == 1 - - _set_observation_time(monkeypatch, "2026-01-01T00:00:40Z") - backend.queue_event_envelope(_status_event("blocked")) - cleared = _attention_lifecycle_rows(backend) - assert cleared[0][0:2] == (1, "open") - assert cleared[0][5:7] == (None, 0) - assert _attention_event_types(backend) == [ - "attention_created", - "attention_escalated", - ] - - _set_observation_time(monkeypatch, "2026-01-01T00:00:50Z") - backend.queue_event_envelope(_status_event("idle")) - assert _attention_lifecycle_rows(backend) == cleared - - _set_observation_time(monkeypatch, "2026-01-01T00:01:40Z") - backend.reconcile_once(client=_initial_pane_client()) - pending = _attention_lifecycle_rows(backend) - assert pending[0][0:2] == (1, "open") - assert pending[0][5] is not None - assert pending[0][6] == 1 - pending_since = pending[0][5] - - _set_observation_time(monkeypatch, "2026-01-01T00:03:40Z") - backend.reconcile_once(client=_initial_pane_client()) - resolved = _attention_lifecycle_rows(backend) - assert resolved[0][0:3] == (1, "resolved", None) - assert resolved[0][5:7] == (pending_since, 2) - assert list_attention_items(backend.db_path, backend.config.host_id) == [] - assert _attention_event_types(backend) == [ - "attention_created", - "attention_escalated", - ] - - -def test_non_authoritative_herdr_saves_do_not_advance_pending_absence( - tmp_path: Path, - monkeypatch: Any, -) -> None: - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, "non-authoritative-lifecycle") - backend.reconcile_once(client=_initial_pane_client()) - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - backend.queue_event_envelope(_status_event("blocked")) - _set_observation_time(monkeypatch, "2026-01-01T00:01:40Z") - backend.reconcile_once(client=_initial_pane_client()) - - pending = _attention_lifecycle_rows(backend) - outbox = _attention_event_types(backend) - assert pending[0][5] is not None - assert pending[0][6] == 1 - - for timestamp, outcome in ( - ("2026-01-01T00:03:40Z", "socket_disconnected"), - ("2026-01-01T00:04:40Z", "protocol_error"), - ("2026-01-01T00:05:40Z", "continuity_unavailable"), - ): - _set_observation_time(monkeypatch, timestamp) - backend._mark_unhealthy(outcome) - assert _attention_lifecycle_rows(backend) == pending - assert _attention_event_types(backend) == outbox - - failed_worker = backend._workers[next(iter(backend._workers))] - backend._workers[failed_worker.id] = Worker( - id=failed_worker.id, - name=failed_worker.name, - status="failed", - space_id=failed_worker.space_id, - meta=failed_worker.meta, - last_seen_at=failed_worker.last_seen_at, - summary=failed_worker.summary, - backend_target=failed_worker.backend_target, - ) - _set_observation_time(monkeypatch, "2026-01-01T00:06:40Z") - backend._persist_current_state(observed_at="2026-01-01T00:06:40Z") - assert _attention_lifecycle_rows(backend) == pending - assert _attention_event_types(backend) == outbox - - _set_observation_time(monkeypatch, "2026-01-01T00:07:40Z") - backend._mark_worker_cap_exceeded_locked(999) - assert _attention_lifecycle_rows(backend) == pending - assert _attention_event_types(backend) == outbox - - -def test_event_flush_and_direct_persistence_use_the_same_lifecycle_executor( - tmp_path: Path, - monkeypatch: Any, -) -> None: - def exercise(host_id: str, *, direct: bool) -> tuple[Any, ...]: - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, host_id) - backend.reconcile_once(client=_initial_pane_client()) - for timestamp, status in ( - ("2026-01-01T00:00:10Z", "blocked"), - ("2026-01-01T00:00:20Z", "failed"), - ): - _set_observation_time(monkeypatch, timestamp) - envelope = _status_event(status) - if direct: - event = normalize_event(envelope) - assert event is not None - assert backend._apply_event(event) is True - backend._persist_current_state(observed_at=timestamp) - else: - assert backend.queue_event_envelope(envelope) is True - lifecycle = _attention_lifecycle_rows(backend) - public = list_attention_items(backend.db_path, backend.config.host_id) - return ( - lifecycle[0][0], - lifecycle[0][1], - lifecycle[0][5], - lifecycle[0][6], - lifecycle[0][9], - _attention_event_types(backend), - public[0]["severity"], - public[0]["status"], - public[0]["lifecycle_status"], - ) - - assert exercise("event-lifecycle-path", direct=False) == exercise( - "direct-lifecycle-path", - direct=True, - ) - - -def test_restart_preserves_pending_absence_until_complete_confirmation( - tmp_path: Path, - monkeypatch: Any, -) -> None: - _set_observation_time(monkeypatch, "2026-01-01T00:00:00Z") - backend = _backend(tmp_path, "restart-pending-lifecycle") - backend.reconcile_once(client=_initial_pane_client()) - _set_observation_time(monkeypatch, "2026-01-01T00:00:10Z") - backend.queue_event_envelope(_status_event("blocked")) - _set_observation_time(monkeypatch, "2026-01-01T00:01:40Z") - backend.reconcile_once(client=_initial_pane_client()) - pending = _attention_lifecycle_rows(backend) - assert pending[0][0:2] == (1, "open") - assert pending[0][5] is not None - assert pending[0][6] == 1 - pending_since = pending[0][5] - - restarted = HerdrEventBackend( - backend.config, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - assert _attention_lifecycle_rows(restarted) == pending - - _set_observation_time(monkeypatch, "2026-01-01T00:03:40Z") - restarted.reconcile_once(client=_initial_pane_client()) - resolved = _attention_lifecycle_rows(restarted) - assert resolved[0][0:3] == (1, "resolved", None) - assert resolved[0][5:7] == (pending_since, 2) - assert _attention_event_types(restarted) == ["attention_created"] - - - -@pytest.mark.parametrize("identity_kind", ["event_id", "producer_sequence"]) -def test_forward_compatible_producer_identity_dedupes_retries_with_bounded_lru( - tmp_path: Path, - identity_kind: str, -) -> None: - config = _config(tmp_path, f"producer-dedupe-{identity_kind}") - init_store(Path(config.db_path)) - backend = HerdrEventBackend( - config, - debounce_seconds=0, - reconnect_delay_seconds=0, - dedupe_size=2, - ) - backend.reconcile_once(client=_initial_pane_client()) - - def identity(value: int) -> dict[str, Any]: - if identity_kind == "event_id": - return {"event_id": f"event-{value}"} - return {"server_id": "server-1", "sequence": value} - - first = {**_status_event("blocked"), **identity(1)} - retry = {**_status_event("failed"), **identity(1)} - assert backend.queue_event_envelope(first) is True - assert backend.queue_event_envelope(retry) is False - assert backend.queue_event_envelope({**_status_event("idle"), **identity(2)}) is True - assert backend.queue_event_envelope({**_status_event("working"), **identity(3)}) is True - assert len(backend._producer_dedupe) == 2 - assert backend.queue_event_envelope({**_status_event("waiting"), **identity(1)}) is True - assert len(backend._producer_dedupe) == 2 - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "waiting" - - -def test_pending_producer_identity_is_not_committed_before_successful_flush( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "pending-producer-dedupe", debounce_seconds=60) - backend.reconcile_once(client=_initial_pane_client()) - identity = HerdrEventId("pending-event") - first = {**_status_event("blocked"), "event_id": identity.value} - duplicate = {**_status_event("failed"), "event_id": identity.value} - - assert backend.queue_event_envelope(first, flush=False) is True - assert backend._producer_dedupe == {} - assert len(backend._pending_events) == 1 - assert backend.queue_event_envelope(duplicate, flush=False) is False - assert backend._producer_dedupe == {} - assert len(backend._pending_events) == 1 - - backend.flush() - - assert list(backend._producer_dedupe) == [identity] - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "blocked" - - -def test_continuity_failure_leaves_producer_identity_retryable( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "continuity-producer-retry") - backend.reconcile_once(client=_initial_pane_client()) - identity = HerdrEventId("continuity-retry") - envelope = {**_status_event("blocked"), "event_id": identity.value} - original_apply = backend._apply_event - fail_next = True - - def fail_once(event: Any) -> bool: - nonlocal fail_next - if fail_next: - fail_next = False - raise HerdrContinuityUnavailableError("continuity unavailable") - return original_apply(event) - - monkeypatch.setattr(backend, "_apply_event", fail_once) - - assert backend.queue_event_envelope(envelope) is True - assert identity not in backend._producer_dedupe - failed = latest_snapshot(backend.db_path, backend.config.host_id) - assert failed is not None - assert failed.workers[0].status == "active" - assert failed.backend_health[0].outcome == "continuity_unavailable" - - assert backend.queue_event_envelope(envelope) is True - recovered = latest_snapshot(backend.db_path, backend.config.host_id) - assert recovered is not None - assert recovered.workers[0].status == "blocked" - assert identity in backend._producer_dedupe - assert backend.queue_event_envelope(envelope) is False - - -def test_snapshot_failure_leaves_identity_retryable_and_retry_persists_dirty_state( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "snapshot-producer-retry") - backend.reconcile_once(client=_initial_pane_client()) - identity = HerdrEventId("snapshot-retry") - envelope = {**_status_event("blocked"), "event_id": identity.value} - original_persist = backend._persist_current_state - fail_next = True - - def fail_once(*, observed_at: str | None = None) -> Any: - nonlocal fail_next - if fail_next: - fail_next = False - raise RuntimeError("snapshot unavailable") - return original_persist(observed_at=observed_at) - - monkeypatch.setattr(backend, "_persist_current_state", fail_once) - - with pytest.raises(RuntimeError, match="snapshot unavailable"): - backend.queue_event_envelope(envelope) - assert identity not in backend._producer_dedupe - stale = latest_snapshot(backend.db_path, backend.config.host_id) - assert stale is not None - assert stale.workers[0].status == "active" - assert backend._workers[stale.workers[0].id].status == "blocked" - - assert backend.queue_event_envelope(envelope) is True - persisted = latest_snapshot(backend.db_path, backend.config.host_id) - assert persisted is not None - assert persisted.workers[0].status == "blocked" - assert identity in backend._producer_dedupe - assert backend.queue_event_envelope(envelope) is False - - -def test_concurrent_queueing_applies_events_in_backend_lock_order( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "concurrent-event-order", debounce_seconds=60) - backend.reconcile_once(client=_initial_pane_client()) - first_queued = threading.Event() - idle_queued = threading.Event() - accepted: dict[str, bool] = {} - applied: list[str] = [] - original_apply = backend._apply_event - - def recording_apply(event: Any) -> bool: - applied.append(str(event.payload["status"])) - return original_apply(event) - - monkeypatch.setattr(backend, "_apply_event", recording_apply) - - def queue_working_events() -> None: - accepted["first"] = backend.queue_event_envelope(_status_event("working"), flush=False) - first_queued.set() - if idle_queued.wait(1): - accepted["last"] = backend.queue_event_envelope(_status_event("working"), flush=False) - - def queue_idle_event() -> None: - if first_queued.wait(1): - accepted["middle"] = backend.queue_event_envelope(_status_event("idle"), flush=False) - idle_queued.set() - - working_thread = threading.Thread(target=queue_working_events) - idle_thread = threading.Thread(target=queue_idle_event) - working_thread.start() - idle_thread.start() - working_thread.join(timeout=1) - idle_thread.join(timeout=1) - - assert working_thread.is_alive() is False - assert idle_thread.is_alive() is False - assert accepted == {"first": True, "middle": True, "last": True} - backend.flush() - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert applied == ["working", "idle", "working"] - assert snapshot.workers[0].status == "active" - assert len(snapshot.workers) == 1 - assert len(list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")) == 1 - -def test_reconcile_turn_refresh_observes_durable_state_and_replaced_ownership_maps( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "reconcile-turn-refresh") - client = _StaticClient( - workspaces=[{"id": "space-1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "pane-1", - "terminal_id": "terminal-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - } - ], - agents=[], - ) - observed: list[tuple[str, str]] = [] - - def callback() -> None: - assert not backend._lock._is_owned() - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert snapshot is not None - assert bindings - assert backend._pane_terminals == {"pane-1": "terminal-1"} - observed.append((snapshot.to_json(), bindings[0].private_fingerprint)) - # The callback-state lock must also be released before invocation. - backend.set_turn_refresh_callback(None) - - backend.set_turn_refresh_callback(callback) - reconciled = backend.reconcile_once(client=client) - - assert observed == [ - ( - reconciled.to_json(), - list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0].private_fingerprint, - ) - ] - - -@pytest.mark.parametrize( - ("event_name", "data"), - [ - ( - "pane.created", - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "working", - }, - ), - ( - "pane.focused", - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "working", - }, - ), - ( - "pane.updated", - { - "pane": { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "working", - }, - }, - ), - ( - "pane.moved", - { - "old_pane_id": "pane-1", - "pane_id": "pane-2", - "agent": "Agent One", - "workspace_id": "space-1", - }, - ), - ("pane.closed", {"pane_id": "pane-1"}), - ("pane.exited", {"pane_id": "pane-1"}), - ( - "pane.agent_detected", - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "working", - }, - ), - ( - "pane.agent_status_changed", - { - "pane_id": "pane-1", - "agent": "Agent One", - "status": "blocked", - }, - ), - ("pane.output_matched", {"pane_id": "pane-1"}), - ], -) -def test_each_turn_relevant_event_notifies_once( - tmp_path: Path, - event_name: str, - data: dict[str, Any], -) -> None: - backend = _backend(tmp_path, f"turn-refresh-{event_name.replace('.', '-')}") - backend.reconcile_once(client=_initial_pane_client()) - calls: list[None] = [] - backend.set_turn_refresh_callback(lambda: calls.append(None)) - - assert backend.queue_event_envelope({"event": event_name, "data": data}) is True - - assert calls == [None] - - -def test_relevant_event_burst_notifies_once_after_batch_commit(tmp_path: Path) -> None: - backend = _backend(tmp_path, "turn-refresh-burst", debounce_seconds=60) - backend.reconcile_once(client=_initial_pane_client()) - calls: list[str] = [] - identity = HerdrEventId("turn-refresh-burst") - - def callback() -> None: - assert not backend._lock._is_owned() - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "blocked" - assert identity in backend._producer_dedupe - calls.append(snapshot.updated_at) - - backend.set_turn_refresh_callback(callback) - assert backend.queue_event_envelope( - {"event": "pane.focused", "data": {"pane_id": "pane-1", "agent": "Agent One"}}, - flush=False, - ) - assert backend.queue_event_envelope( - {**_status_event("blocked"), "event_id": identity.value}, - flush=False, - ) - assert backend.queue_event_envelope( - {"event": "pane.output_matched", "data": {"pane_id": "pane-1"}}, - flush=False, - ) - - backend.flush() - - assert len(calls) == 1 - - -def test_projection_neutral_output_match_notifies_without_rewriting_snapshot(tmp_path: Path) -> None: - backend = _backend(tmp_path, "turn-refresh-output") - backend.reconcile_once(client=_initial_pane_client()) - before = latest_snapshot(backend.db_path, backend.config.host_id) - assert before is not None - calls: list[None] = [] - backend.set_turn_refresh_callback(lambda: calls.append(None)) - - assert backend.queue_event_envelope( - {"event": "pane.output_matched", "data": {"pane_id": "pane-1"}} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - assert after is not None - assert after.to_json() == before.to_json() - assert calls == [None] - - -def test_irrelevant_duplicate_and_invalid_events_do_not_notify(tmp_path: Path) -> None: - backend = _backend(tmp_path, "turn-refresh-noop") - backend.reconcile_once(client=_initial_pane_client()) - calls: list[None] = [] - backend.set_turn_refresh_callback(lambda: calls.append(None)) - - assert backend.queue_event_envelope( - { - "event": "workspace.updated", - "data": {"workspace_id": "space-1", "name": "Renamed"}, - } - ) - assert backend.queue_event_envelope( - { - "event": "worktree.created", - "data": {"workspace_id": "missing-space", "name": "Ignored"}, - } - ) - assert backend.queue_event_envelope({"event": "not.official", "data": {}}) is False - assert backend.queue_event_envelope({"event": 7, "data": {}}) is False - assert calls == [] - - backend.set_turn_refresh_callback(None) - duplicate = { - "event": "pane.output_matched", - "event_id": "already-applied", - "data": {"pane_id": "pane-1"}, - } - assert backend.queue_event_envelope(duplicate) is True - backend.set_turn_refresh_callback(lambda: calls.append(None)) - assert backend.queue_event_envelope(duplicate) is False - assert calls == [] - - -def test_persistence_failure_does_not_notify_turn_refresh( - tmp_path: Path, - monkeypatch: Any, -) -> None: - backend = _backend(tmp_path, "turn-refresh-persist-failure") - backend.reconcile_once(client=_initial_pane_client()) - calls: list[None] = [] - backend.set_turn_refresh_callback(lambda: calls.append(None)) - - def fail_persistence(*, observed_at: str | None = None) -> Any: - raise RuntimeError(f"snapshot unavailable at {observed_at}") - - monkeypatch.setattr(backend, "_persist_current_state", fail_persistence) - - with pytest.raises(RuntimeError, match="snapshot unavailable"): - backend.queue_event_envelope(_status_event("blocked")) - assert calls == [] - - -def test_callback_failure_cannot_roll_back_durable_event_state(tmp_path: Path) -> None: - backend = _backend(tmp_path, "turn-refresh-callback-failure") - backend.reconcile_once(client=_initial_pane_client()) - identity = HerdrEventId("callback-failure") - - def fail_callback() -> None: - raise RuntimeError("scheduler unavailable") - - backend.set_turn_refresh_callback(fail_callback) - assert backend.queue_event_envelope( - {**_status_event("blocked"), "event_id": identity.value} - ) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "blocked" - assert identity in backend._producer_dedupe - assert backend.queue_event_envelope( - {**_status_event("blocked"), "event_id": identity.value} - ) is False - - -@pytest.mark.parametrize("event_name", ["pane.closed", "pane.exited"]) -def test_close_refresh_callback_observes_expired_binding( - tmp_path: Path, - event_name: str, -) -> None: - backend = _backend(tmp_path, f"turn-refresh-{event_name.replace('.', '-')}-binding") - backend.reconcile_once(client=_initial_pane_client()) - observed_reasons: list[str | None] = [] - - def callback() -> None: - assert list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) == [] - expired = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - include_expired=True, - ) - assert len(expired) == 1 - observed_reasons.append(expired[0].reason) - - backend.set_turn_refresh_callback(callback) - assert backend.queue_event_envelope( - {"event": event_name, "data": {"pane_id": "pane-1"}} - ) - - assert observed_reasons == [event_name.replace(".", "_")] - - - -def test_pane_moved_preserves_public_worker_and_updates_private_binding(tmp_path: Path) -> None: - backend = _backend(tmp_path, "pane-moved") - backend.reconcile_once(client=_initial_pane_client()) - before = latest_snapshot(backend.db_path, backend.config.host_id) - assert before is not None - worker_id = before.workers[0].id - binding = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - - backend.queue_event_envelope( - {"event": "pane.moved", "data": { - "old_pane_id": "pane-1", - "pane_id": "pane-2", - "agent": "Agent One", - "workspace_id": "space-1", - }} - ) - - after = latest_snapshot(backend.db_path, backend.config.host_id) - moved_binding = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - assert after is not None - assert after.workers[0].id == worker_id - assert moved_binding.private_fingerprint == binding.private_fingerprint - assert moved_binding.target_kind == "pane_id" - assert moved_binding.target_value == "pane-2" - - -def test_pane_closed_closes_worker_and_expires_matching_binding(tmp_path: Path) -> None: - backend = _backend(tmp_path, "pane-closed") - backend.reconcile_once(client=_initial_pane_client()) - - backend.queue_event_envelope({"event": "pane.closed", "data": {"pane_id": "pane-1"}}) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "closed" - assert list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") == [] - expired = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - include_expired=True, - ) - assert expired[0].sendable is False - assert expired[0].reason == "pane_closed" - -def test_pane_exited_closes_worker_and_expires_matching_binding(tmp_path: Path) -> None: - backend = _backend(tmp_path, "pane-exited") - backend.reconcile_once(client=_initial_pane_client()) - - backend.queue_event_envelope({"event": "pane.exited", "data": {"pane_id": "pane-1"}}) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "closed" - assert list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") == [] - expired = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - include_expired=True, - ) - assert expired[0].sendable is False - assert expired[0].reason == "pane_exited" - - -def test_disconnect_degraded_state_preserves_workers_and_bindings(tmp_path: Path) -> None: - backend = _backend(tmp_path, "degraded") - backend.reconcile_once(client=_initial_pane_client()) - binding_before = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - - backend._mark_unhealthy("socket_disconnected") - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - binding_after = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - assert snapshot is not None - assert snapshot.workers[0].status == "active" - assert snapshot.backend_health[0].status == "unavailable" - assert binding_after.private_fingerprint == binding_before.private_fingerprint - assert binding_after.sendable is True - - -def test_healthy_empty_reconnect_closes_missing_workers_and_expires_bindings(tmp_path: Path) -> None: - backend = _backend(tmp_path, "healthy-empty") - backend.reconcile_once(client=_initial_pane_client()) - - backend.reconcile_once(client=_StaticClient(workspaces=[], tabs=[], panes=[], agents=[])) - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.backend_health[0].status == "healthy" - assert snapshot.backend_health[0].outcome == "empty_healthy" - assert snapshot.workers[0].status == "closed" - assert list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") == [] - - -def test_worker_cap_exceeded_preserves_previous_authoritative_snapshot(tmp_path: Path) -> None: - config = Config( - host_id="worker-cap", - data_dir=tmp_path, - db_path=tmp_path / "worker-cap.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - max_workers=1, - turn_refresh_workers=1, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - backend.reconcile_once(client=_initial_pane_client()) - - capped = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "space-1", "name": "Build"}], - panes=[ - {"pane_id": "pane-1", "agent": "Agent One", "workspace_id": "space-1"}, - {"pane_id": "pane-2", "agent": "Agent Two", "workspace_id": "space-1"}, - ], - ) - ) - latest = latest_snapshot(backend.db_path, backend.config.host_id) - - assert latest is not None - assert capped.content_fingerprint == latest.content_fingerprint - assert [worker.name for worker in capped.workers] == ["Agent One"] - assert capped.backend_health[0].status == "degraded" - assert capped.backend_health[0].outcome == "worker_cap_exceeded" - assert list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") - _assert_no_public_json_forbidden(json.loads(capped.to_json())) - - -def test_output_excerpt_limit_bounds_public_worker_summary(tmp_path: Path) -> None: - config = Config( - host_id="output-excerpt", - data_dir=tmp_path, - db_path=tmp_path / "output-excerpt.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - output_excerpt_chars=12, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - long_summary = "x" * 40 - - snapshot = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "space-1", "name": "Build"}], - panes=[ - { - "pane_id": "pane-1", - "agent": "Agent One", - "workspace_id": "space-1", - "description": long_summary, - } - ], - ) - ) - binding = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr")[0] - - assert snapshot.workers[0].summary == "xxxxxxxxx..." - assert len(snapshot.workers[0].summary or "") == 12 - assert latest_snapshot(backend.db_path, backend.config.host_id).workers[0].summary == "xxxxxxxxx..." - assert binding.worker_fingerprint == snapshot.workers[0].fingerprint - assert long_summary not in snapshot.to_json() - - -def test_incremental_event_over_worker_cap_is_ignored_with_degraded_health(tmp_path: Path) -> None: - config = Config( - host_id="event-worker-cap", - data_dir=tmp_path, - db_path=tmp_path / "event-worker-cap.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - max_workers=1, - turn_refresh_workers=1, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - backend.reconcile_once(client=_initial_pane_client()) - - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent_id": "agent-2", - "name": "Agent Two", - "workspace_id": "space-1", - "pane_id": "pane-2", - } - }} - ) - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - - assert snapshot is not None - assert [worker.name for worker in snapshot.workers] == ["Agent One"] - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "worker_cap_exceeded" - - -def test_closed_worker_reactivation_over_worker_cap_is_ignored(tmp_path: Path) -> None: - config = Config( - host_id="event-worker-cap-reactivate", - data_dir=tmp_path, - db_path=tmp_path / "event-worker-cap-reactivate.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - max_workers=1, - turn_refresh_workers=1, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - backend.reconcile_once(client=_initial_pane_client()) - - backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane": {"pane_id": "pane-1", "agent": "Agent One", "workspace_id": "space-1"}}} - ) - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent_id": "agent-2", - "name": "Agent Two", - "workspace_id": "space-1", - "pane_id": "pane-2", - "status": "running", - } - }} - ) - - before_reactivation = latest_snapshot(backend.db_path, backend.config.host_id) - assert before_reactivation is not None - assert {worker.name: worker.status for worker in before_reactivation.workers} == { - "Agent One": "closed", - "Agent Two": "active", - } - - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent": "Agent One", - "workspace_id": "space-1", - "pane_id": "pane-3", - "status": "running", - } - }} - ) - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - - assert snapshot is not None - assert {worker.name: worker.status for worker in snapshot.workers} == { - "Agent One": "closed", - "Agent Two": "active", - } - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "worker_cap_exceeded" - - -def test_pane_moved_reactivation_over_worker_cap_is_ignored(tmp_path: Path) -> None: - config = Config( - host_id="event-worker-cap-moved-reactivate", - data_dir=tmp_path, - db_path=tmp_path / "event-worker-cap-moved-reactivate.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - max_workers=1, - turn_refresh_workers=1, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - first = backend.reconcile_once(client=_initial_pane_client()) - first_worker = first.workers[0] - closed_worker = Worker( - id=first_worker.id, - name=first_worker.name, - status="closed", - space_id=first_worker.space_id, - meta=first_worker.meta, - last_seen_at=first_worker.last_seen_at, - summary=first_worker.summary, - backend_target=first_worker.backend_target, - ) - backend._workers[closed_worker.id] = closed_worker - save_snapshot( - backend.db_path, - project_from_observations( - backend.config, - spaces=first.spaces, - workers=[closed_worker], - backend_health=first.backend_health, - ), - ) - - backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "agent_id": "agent-2", - "name": "Agent Two", - "workspace_id": "space-1", - "pane_id": "pane-2", - "status": "running", - } - }} - ) - before_move = latest_snapshot(backend.db_path, backend.config.host_id) - assert before_move is not None - assert {worker.name: worker.status for worker in before_move.workers} == { - "Agent One": "closed", - "Agent Two": "active", - } - - backend.queue_event_envelope( - {"event": "pane.moved", "data": { - "old_pane_id": "pane-1", - "pane_id": "pane-3", - "agent": "Agent One", - "workspace_id": "space-1", - "status": "running", - }} - ) - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - - assert snapshot is not None - assert {worker.name: worker.status for worker in snapshot.workers} == { - "Agent One": "closed", - "Agent Two": "active", - } - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "worker_cap_exceeded" - - -def test_periodic_reconcile_uses_config_and_zero_disables_it(tmp_path: Path) -> None: - disabled_config = Config( - host_id="periodic-disabled", - data_dir=tmp_path, - db_path=tmp_path / "periodic-disabled.db", - herdr_backend="socket", - reconcile_interval_seconds=0, - ) - init_store(Path(disabled_config.db_path)) - disabled = HerdrEventBackend(disabled_config, debounce_seconds=0) - disabled_client = _initial_pane_client() - disabled._next_reconcile_monotonic = time.monotonic() - 1 - disabled._run_periodic_reconcile_if_due(disabled_client) - - enabled_config = Config( - host_id="periodic-enabled", - data_dir=tmp_path, - db_path=tmp_path / "periodic-enabled.db", - herdr_backend="socket", - reconcile_interval_seconds=0.001, - ) - init_store(Path(enabled_config.db_path)) - enabled = HerdrEventBackend(enabled_config, debounce_seconds=0) - enabled_client = _initial_pane_client() - enabled._next_reconcile_monotonic = time.monotonic() - 1 - enabled._run_periodic_reconcile_if_due(enabled_client) - - assert disabled_client.calls == [] - assert enabled_client.calls == ["workspace.list", "tab.list", "pane.list", "agent.list"] - assert enabled.operational_status["last_reconcile_at"] is not None - - -def test_debounce_batches_until_flush_and_shutdown_flushes(tmp_path: Path) -> None: - backend = _backend(tmp_path, "debounce", debounce_seconds=60) - backend.reconcile_once(client=_initial_pane_client()) - - backend.queue_event_envelope( - {"event": "pane.agent_status_changed", "data": {"agent": "Agent One", "status": "blocked"}} - ) - pending = latest_snapshot(backend.db_path, backend.config.host_id) - assert pending is not None - assert pending.workers[0].status == "active" - - backend.stop() - - flushed = latest_snapshot(backend.db_path, backend.config.host_id) - assert flushed is not None - assert flushed.workers[0].status == "blocked" - - -def test_idle_event_timeout_keeps_polling_without_marking_backend_unhealthy(tmp_path: Path) -> None: - backend = _backend(tmp_path, "idle-timeout") - backend.reconcile_once(client=_initial_pane_client()) - - class IdleThenEventClient: - def __init__(self) -> None: - self.calls = 0 - - def read_event(self, subscription_id: str, *, timeout: float | None = None) -> dict[str, Any]: - self.calls += 1 - if self.calls == 1: - raise HerdrSocketTimeoutError("idle") - backend.stop_event.set() - return { - "id": subscription_id, - "event": "pane.agent_status_changed", - "data": {"agent": "Agent One", "status": "blocked"}, - } - - client = IdleThenEventClient() - backend._read_event_stream(client, "sub-1") - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "blocked" - assert snapshot.backend_health[0].status == "healthy" - assert client.calls == 2 - - -def test_mark_unhealthy_safe_sets_ready_even_when_persist_fails(tmp_path: Path, monkeypatch: Any) -> None: - backend = _backend(tmp_path, "ready-on-error") - - def boom(*_args: Any, **_kwargs: Any) -> None: - raise RuntimeError("store unavailable") - - monkeypatch.setattr("tendwire.backends.herdr_events.save_snapshot", boom) - - try: - backend._mark_unhealthy_safe("protocol_error") - except RuntimeError: - pass - - assert backend.ready is True - - -def test_protocol_error_health_is_degraded_and_specific(tmp_path: Path) -> None: - backend = _backend(tmp_path, "protocol-health") - - backend._mark_unhealthy("protocol_error") - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "protocol_error" - - -def test_daemon_starts_socket_backend_only_when_configured(tmp_path: Path) -> None: - db_path = tmp_path / "daemon-socket.db" - socket_path = tmp_path / "daemon.sock" - config = Config( - host_id="daemon-socket", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - calls: list[str] = [] - - class FakeBackend: - def __init__(self, config: Config, stop_event: threading.Event) -> None: - self.config = config - self.stop_event = stop_event - - def start(self, *, wait_for_reconcile: bool = True) -> None: - calls.append(f"start:{wait_for_reconcile}") - snapshot = project_from_observations( - self.config, - workers=[Worker(id="worker-1", name="Worker", status="active")], - backend_health=[ - BackendHealth( - name="herdr", - status="healthy", - outcome="healthy_non_empty", - counts={"workers": 1}, - ) - ], - ) - save_snapshot(Path(self.config.db_path), snapshot) - - def stop(self) -> None: - calls.append("stop") - - def observe_cli(_config: Config) -> Any: - raise AssertionError("CLI observation must not run in socket mode") - - daemon = TendwireDaemon( - config, - hooks=DaemonHooks( - observe_initial_snapshot=observe_cli, - event_backend_factory=lambda cfg, stop_event: FakeBackend(cfg, stop_event), - ), - ) - daemon.start() - try: - assert calls == ["start:True"] - assert daemon.get_snapshot().workers[0].id == "worker-1" - _assert_no_public_json_forbidden(daemon.get_health()) - finally: - daemon.stop() - - assert "stop" in calls - - -def test_daemon_socket_fallback_uses_backend_health_when_snapshot_missing(tmp_path: Path) -> None: - db_path = tmp_path / "daemon-fallback.db" - socket_path = tmp_path / "daemon-fallback.sock" - config = Config( - host_id="daemon-fallback", - data_dir=tmp_path, - db_path=db_path, - socket_path=socket_path, - herdr_backend="socket", - ) - - class FakeBackend: - def __init__(self, config: Config, stop_event: threading.Event) -> None: - self.config = config - self.stop_event = stop_event - self.health = HerdrEventBackend(config)._health_for("protocol_error") - - def start(self, *, wait_for_reconcile: bool = True) -> None: - return None - - def stop(self) -> None: - return None - - daemon = TendwireDaemon( - config, - hooks=DaemonHooks(event_backend_factory=lambda cfg, stop_event: FakeBackend(cfg, stop_event)), - ) - daemon.start() - try: - snapshot = daemon.get_snapshot() - assert snapshot.backend_health[0].status == "degraded" - assert snapshot.backend_health[0].outcome == "protocol_error" - finally: - daemon.stop() - - -def test_status_event_with_pane_id_only_updates_bound_worker_not_a_phantom(tmp_path: Path) -> None: - """Regression: status events that only carry a pane id must resolve through - the binding turn target instead of inserting a duplicate re-lettered worker - that freezes the real worker's status (the 'stuck working icon' bug).""" - backend = _backend(tmp_path, "phantom-host") - client = _StaticClient( - workspaces=[{"id": "space-1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "w1:p1", - "terminal_id": "term-1", - "agent": "claude", - "workspace_id": "space-1", - "agent_status": "working", - }, - { - "pane_id": "w1:p2", - "terminal_id": "term-2", - "agent": "claude", - "workspace_id": "space-1", - "agent_status": "working", - }, - ], - agents=[], - ) - backend.reconcile_once(client=client) - snapshot = latest_snapshot(Path(backend.db_path), backend.config.host_id) - assert snapshot is not None - ids_before = sorted(worker.id for worker in snapshot.workers) - assert ids_before == ["claude-1", "claude-2"] - - event = normalize_event( - {"event": "pane.agent_status_changed", "data": {"pane": {"pane_id": "w1:p2", "agent": "claude", "status": "idle"}}} - ) - assert event is not None - assert backend._apply_event(event) is True - backend._persist_projection_locked() if hasattr(backend, "_persist_projection_locked") else None - workers = backend._workers - assert sorted(workers) == ["claude-1", "claude-2"], f"phantom worker inserted: {sorted(workers)}" - by_target = {} - for binding in backend._bindings.values(): - by_target[binding.target_value] = binding.worker_id - idle_worker_id = by_target["term-2"] - assert workers[idle_worker_id].status in {"idle", "done"} - - -def test_pane_id_only_status_event_resolves_codex_binding_via_pane_terminal_map(tmp_path: Path) -> None: - """Regression: codex bindings' turn target is a session id, so pane-id-only - status events must resolve through the pane->terminal map remembered from - reconcile instead of inserting a phantom bare 'codex' worker.""" - backend = _backend(tmp_path, "codex-phantom-host") - client = _StaticClient( - workspaces=[{"id": "wX8", "name": "projectx", "status": "active"}], - panes=[ - { - "pane_id": "wX8:p1", - "terminal_id": "term-ctx", - "agent": "codex", - "agent_session": {"agent": "codex", "kind": "id", "value": "019f-session"}, - "workspace_id": "wX8", - "agent_status": "working", - } - ], - agents=[], - ) - backend.reconcile_once(client=client) - assert sorted(backend._workers) == ["codex"] - binding = next(iter(backend._bindings.values())) - assert binding.turn_target_kind == "codex_session_id" - assert backend._pane_terminals == {"wX8:p1": "term-ctx"} - - event = normalize_event( - {"event": "pane.agent_status_changed", "data": {"pane_id": "wX8:p1", "workspace_id": "wX8", "agent_status": "idle"}} - ) - assert event is not None - assert backend._apply_event(event) is True - assert sorted(backend._workers) == ["codex"], f"phantom inserted: {sorted(backend._workers)}" - assert backend._workers["codex"].status in {"idle", "done"} - updated_binding = next(iter(backend._bindings.values())) - assert updated_binding.private_fingerprint == binding.private_fingerprint - assert updated_binding.target_kind == binding.target_kind == "terminal_id" - assert updated_binding.target_value == binding.target_value == "term-ctx" - assert updated_binding.turn_target_kind == binding.turn_target_kind == "codex_session_id" - assert updated_binding.turn_target_value == binding.turn_target_value == "019f-session" - - -def test_reconcile_drops_unbound_missing_workers_but_keeps_bound_closed(tmp_path: Path) -> None: - backend = _backend(tmp_path, "phantom-aging-host") - worker_bound = Worker(id="codex-1", name="codex", status="active", space_id="wX8") - worker_phantom = Worker(id="codex", name="codex", status="working", space_id="wX8") - merged = backend._workers_with_closed_missing( - [worker_bound, worker_phantom], - [], - bound_worker_ids={"codex-1"}, - ) - ids = {worker.id: worker.status for worker in merged} - assert "codex" not in ids, "unbound phantom must be dropped, not carried as closed" - assert ids.get("codex-1") == "closed" - - -def test_nested_agent_pane_claim_cannot_poison_terminal_close_matching( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "pane-terminal-provenance") - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "W1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "P1", - "terminal_id": "T1", - "workspace_id": "W1", - "agent": "codex", - "agent_status": "working", - } - ], - ) - ) - worker = initial.workers[0] - binding = next(iter(backend._bindings.values())) - assert backend._pane_terminals == {"P1": "T1"} - - assert backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": { - "agent": { - "worker_id": "nested-agent-claimant", - "pane_id": "Pfake", - "terminal_id": "T1", - "agent": "codex", - "status": "working", - } - }} - ) - assert backend._pane_terminals == {"P1": "T1"} - assert backend._pane_owners == {"P1": {worker.id}} - - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane_id": "Pfake"}} - ) - - current = backend._workers[worker.id] - assert current.status == worker.status - assert current.status != "closed" - assert "Pfake" not in backend._pane_terminals - assert backend._pane_owners == {"P1": {worker.id}} - stored = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert len(stored) == 1 - assert stored[0].private_fingerprint == binding.private_fingerprint - assert stored[0].sendable is True - - -def test_reconcile_maps_only_accepted_pane_info_rows( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "reconcile-pane-map-provenance") - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "W1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "P1", - "terminal_id": "T1", - "workspace_id": "W1", - "agent": "codex", - "agent_status": "working", - }, - { - "pane_id": "Pfake", - "terminal_id": "T1", - "workspace_id": "W1", - "agent_status": "working", - }, - ], - ) - ) - worker = initial.workers[0] - binding = next(iter(backend._bindings.values())) - assert backend._pane_terminals == {"P1": "T1"} - assert backend._pane_owners == {"P1": {worker.id}} - - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane_id": "Pfake"}} - ) - - assert backend._workers[worker.id].status == worker.status - assert backend._workers[worker.id].status != "closed" - stored = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert len(stored) == 1 - assert stored[0].private_fingerprint == binding.private_fingerprint - - -@pytest.mark.parametrize( - ("source_field", "source_value"), - [ - ("previous_pane_id", "P1"), - ("previous_terminal_id", "T1"), - ], -) -def test_accepted_move_removes_source_pane_terminal_alias_before_stale_close( - tmp_path: Path, - source_field: str, - source_value: str, -) -> None: - backend = _backend(tmp_path, "moved-pane-map-provenance") - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "W1", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "P1", - "terminal_id": "T1", - "workspace_id": "W1", - "agent": "codex", - "agent_status": "working", - } - ], - ) - ) - worker = initial.workers[0] - - assert backend.queue_event_envelope( - {"event": "pane.moved", "data": { - source_field: source_value, - "pane": { - "pane_id": "P2", - "terminal_id": "T1", - "workspace_id": "W1", - "agent": "codex", - "agent_status": "working", - }, - }} - ) - assert backend._pane_terminals == {"P2": "T1"} - assert backend._pane_owners == {"P2": {worker.id}} - - assert backend.queue_event_envelope( - {"event": "pane.closed", "data": {"pane_id": "P1"}} - ) - - assert backend._workers[worker.id].status == worker.status - assert backend._workers[worker.id].status != "closed" - assert backend._pane_terminals == {"P2": "T1"} - - -def test_pane_only_status_preserves_unobserved_owner_aliases_for_later_conflict( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "status-owner-alias-provenance") - session = { - "source": "old-source-secret", - "agent": "codex", - "kind": "id", - "value": "old-session-secret", - } - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "wR9:pA", - "terminal_id": "shared-terminal-secret", - "workspace_id": "wR9", - "agent": "codex", - "agent_session": session, - "agent_status": "working", - } - ], - agents=[ - { - "worker_id": "public-owner", - "agent_id": "old-agent-target-secret", - "pane_id": "wR9:pA", - "terminal_id": "shared-terminal-secret", - "workspace_id": "wR9", - "agent": "codex", - "agent_session": session, - "agent_status": "working", - } - ], - ) - ) - worker = initial.workers[0] - binding = next(iter(backend._bindings.values())) - assert binding.target_kind == "agent_id" - assert binding.turn_target_kind == "codex_session_id" - - assert backend.queue_event_envelope( - {"event": "pane.agent_status_changed", "data": { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "status": "idle", - }} - ) - status_binding = next(iter(backend._bindings.values())) - assert status_binding.private_fingerprint == binding.private_fingerprint - assert status_binding.target_kind == binding.target_kind - assert status_binding.target_value == binding.target_value - assert status_binding.turn_target_kind == binding.turn_target_kind - assert status_binding.turn_target_value == binding.turn_target_value - assert backend._terminal_owners == { - "shared-terminal-secret": {worker.id}, - } - assert backend._session_owners == { - "old-session-secret": {worker.id}, - } - - assert backend.queue_event_envelope( - {"event": "pane.created", "data": { - "pane": { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "shared-terminal-secret", - "agent_id": "new-agent-target-secret", - "agent": "codex", - "agent_session": { - "source": "new-source-secret", - "agent": "codex", - "kind": "id", - "value": "new-session-secret", - }, - "status": "working", - } - }} - ) - - assert set(backend._workers) == {worker.id} - conflicted = next(iter(backend._bindings.values())) - assert conflicted.worker_id == worker.id - assert conflicted.sendable is False - assert conflicted.reason == "ambiguous_pane_match" - assert "wR9:pB" not in backend._pane_terminals - - -def test_nested_compatibility_replay_cannot_rehabilitate_ambiguous_binding( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "compatibility-ambiguity-provenance") - session = { - "source": "shared-source-secret", - "agent": "codex", - "kind": "id", - "value": "shared-session-secret", - } - original_agent = { - "worker_id": "public-owner-a", - "agent_id": "agent-target-a-secret", - "pane_id": "wR9:pA", - "terminal_id": "terminal-a-secret", - "workspace_id": "wR9", - "agent": "codex", - "agent_session": session, - "status": "working", - } - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "wR9", "name": "Build", "status": "active"}], - panes=[ - { - "pane_id": "wR9:pA", - "terminal_id": "terminal-a-secret", - "workspace_id": "wR9", - "agent": "codex", - "agent_session": session, - "status": "working", - } - ], - agents=[original_agent], - ) - ) - worker = initial.workers[0] - - assert backend.queue_event_envelope( - {"event": "pane.created", "data": { - "pane": { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "terminal-b-secret", - "agent_id": "agent-target-b-secret", - "agent": "codex", - "agent_session": session, - "status": "working", - } - }} - ) - ambiguous = next(iter(backend._bindings.values())) - assert ambiguous.worker_id == worker.id - assert ambiguous.target_value == "agent-target-a-secret" - assert ambiguous.sendable is False - assert ambiguous.reason == "ambiguous_pane_match" - assert ambiguous.turn_target_kind is None - assert ambiguous.turn_target_value is None - assert backend._workers[worker.id].backend_target == ambiguous.backend_target() - - assert backend.queue_event_envelope( - {"event": "pane.agent_detected", "data": {"agent": original_agent}} - ) - - replayed = next(iter(backend._bindings.values())) - assert replayed.private_fingerprint == ambiguous.private_fingerprint - assert replayed.target_kind == ambiguous.target_kind - assert replayed.target_value == ambiguous.target_value - assert replayed.sendable is False - assert replayed.reason == "ambiguous_pane_match" - assert replayed.turn_target_kind is None - assert replayed.turn_target_value is None - assert backend._workers[worker.id].backend_target == replayed.backend_target() - assert backend._pane_terminals == { - "wR9:pA": "terminal-a-secret", - } - - -def _run_authoritative_recovery_trace( - tmp_path: Path, - monkeypatch: Any, -) -> dict[str, Any]: - backend = _backend(tmp_path, "authoritative-turn-recovery") - session = { - "source": "codex", - "agent": "codex", - "kind": "id", - "value": "session-private-recovery", - } - agent = { - "worker_id": "public-recovery-worker", - "agent_id": "agent-private-recovery", - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-private-a", - "agent": "codex", - "agent_session": session, - "status": "working", - } - pane_a = { - "workspace_id": "wR9", - "pane_id": "wR9:pA", - "terminal_id": "terminal-private-a", - "agent": "codex", - "agent_session": session, - "status": "working", - } - workspace = [{"id": "wR9", "name": "Build", "status": "active"}] - - initial = backend.reconcile_once( - client=_StaticClient( - workspaces=workspace, - panes=[pane_a], - agents=[agent], - ) - ) - worker = initial.workers[0] - initial_binding = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - )[0] - assert initial_binding.worker_id == worker.id - assert initial_binding.turn_target_kind == "codex_session_id" - assert initial_binding.turn_target_value == "session-private-recovery" - assert merge_turn_content( - backend.db_path, - backend.config.host_id, - worker.id, - { - "user_text": "Recover the deterministic producer final.", - "assistant_stream_text": "Still working.", - "assistant_final_text": None, - "complete": False, - "has_open_turn": True, - "source_turn_id": "producer-turn-42", - }, - ) == 1 - seeded_payload = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=initial, - ) - seeded_turns = [ - turn - for turn in seeded_payload["turns"] - if turn.get("user_text") == "Recover the deterministic producer final." - ] - assert len(seeded_turns) == 1 - public_turn_id = seeded_turns[0]["id"] - public_source_turn_id = seeded_turns[0]["source_turn_id"] - assert public_source_turn_id != "producer-turn-42" - assert seeded_turns[0]["worker_id"] == worker.id - assert seeded_turns[0]["complete"] is False - - adapter_calls: list[tuple[str, str | None]] = [] - - def refresh_existing_final( - config: Config, - binding: Any, - *, - adapter_timeout_seconds: float | None = None, - ) -> Any: - del adapter_timeout_seconds - adapter_calls.append((binding.worker_id, binding.turn_target_value)) - applied = herdr_turns.apply_turn_refresh( - config.db_path, - config.host_id, - binding.worker_id, - { - "user_text": "Recover the deterministic producer final.", - "assistant_stream_text": None, - "assistant_final_text": "The durable producer final.", - "complete": True, - "has_open_turn": False, - "source_turn_id": "producer-turn-42", - }, - expected_binding=binding, - ) - assert applied.updated == 1 - return herdr_turns.TurnRefreshResult("updated", 1, applied.pending_changed) - - monkeypatch.setattr(herdr_turns, "refresh_turn_binding", refresh_existing_final) - pane_b = { - "workspace_id": "wR9", - "pane_id": "wR9:pB", - "terminal_id": "terminal-private-b", - "agent": "codex", - "agent_session": session, - "status": "working", - } - quarantined = backend.reconcile_once( - client=_StaticClient( - workspaces=workspace, - panes=[pane_a, pane_b], - agents=[agent], - ) - ) - quarantined_bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert len(quarantined_bindings) == 1 - quarantined_binding = quarantined_bindings[0] - assert quarantined_binding.worker_id == worker.id - assert quarantined_binding.sendable is False - assert quarantined_binding.reason == "ambiguous_pane_match" - assert quarantined_binding.turn_target_kind is None - assert quarantined_binding.turn_target_value is None - assert herdr_turns.refresh_structured_turn_content(backend.config) == { - "ok": True, - "status": "ok", - "updated": 0, - "attempted": 0, - } - assert adapter_calls == [] - quarantined_payload = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=quarantined, - ) - quarantined_turn = next( - turn for turn in quarantined_payload["turns"] if turn["id"] == public_turn_id - ) - assert quarantined_turn["source_turn_id"] == public_source_turn_id - assert quarantined_turn["assistant_final_text"] is None - assert quarantined_turn["assistant_stream_text"] == "Still working." - assert quarantined_turn["complete"] is False - - restarted = HerdrEventBackend( - backend.config, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - assert len(restarted._bindings) == 1 - restarted_binding = next(iter(restarted._bindings.values())) - assert restarted_binding.worker_id == worker.id - assert restarted_binding.reason == "ambiguous_pane_match" - assert restarted_binding.turn_target_value is None - - recovered = restarted.reconcile_once( - client=_StaticClient( - workspaces=workspace, - panes=[pane_a], - agents=[agent], - ) - ) - recovered_bindings = list_worker_bindings( - restarted.db_path, - restarted.config.host_id, - backend="herdr", - ) - assert len(recovered_bindings) == 1 - recovered_binding = recovered_bindings[0] - assert recovered.workers[0].id == worker.id - assert recovered_binding.worker_id == initial_binding.worker_id - assert recovered_binding.private_fingerprint == initial_binding.private_fingerprint - assert recovered_binding.sendable is True - assert recovered_binding.reason is None - assert recovered_binding.turn_target_kind == "codex_session_id" - assert recovered_binding.turn_target_value == "session-private-recovery" - - assert herdr_turns.refresh_turn_binding( - restarted.config, - recovered_binding, - ) == herdr_turns.TurnRefreshResult("updated", 1) - assert adapter_calls == [(worker.id, "session-private-recovery")] - - final_payload = turns_payload_from_store( - restarted.db_path, - restarted.config.host_id, - snapshot=recovered, - ) - final_turns = [ - turn - for turn in final_payload["turns"] - if turn.get("assistant_final_text") == "The durable producer final." - ] - assert len(final_turns) == 1 - assert final_turns[0]["id"] == public_turn_id - assert final_turns[0]["source_turn_id"] == public_source_turn_id - assert final_turns[0]["worker_id"] == worker.id - assert final_turns[0]["assistant_stream_text"] is None - assert final_turns[0]["complete"] is True - assert final_turns[0]["has_open_turn"] is False - public_json = json.dumps(final_payload, sort_keys=True) - for private_value in ( - "producer-turn-42", - "agent-private-recovery", - "session-private-recovery", - "terminal-private-a", - "terminal-private-b", - "wR9:pA", - "wR9:pB", - ): - assert private_value not in public_json - _assert_no_public_json_forbidden(json.loads(public_json)) - snapshot_payload = json.loads(recovered.to_json()) - snapshot_payload["ok"] = True - return { - "turns_payload": final_payload, - "snapshot_payload": snapshot_payload, - "public_turn": final_turns[0], - } - - -def test_authoritative_reconcile_quarantines_then_recovers_existing_final_once( - tmp_path: Path, - monkeypatch: Any, -) -> None: - _run_authoritative_recovery_trace(tmp_path, monkeypatch) - - -@pytest.mark.skipif( - not os.environ.get("HERDRES_SOURCE_DIR"), - reason="HERDRES_SOURCE_DIR is required for the cross-repository recovery contract", -) -def test_authoritative_recovery_payload_promotes_herdres_working_once( - tmp_path: Path, - monkeypatch: Any, -) -> None: - trace = _run_authoritative_recovery_trace(tmp_path, monkeypatch) - turns_payload = trace["turns_payload"] - snapshot_payload = trace["snapshot_payload"] - public_turn = trace["public_turn"] - assert turns_payload["schema_version"] == 1 - assert public_turn["id"] - - source_dir = Path(os.environ["HERDRES_SOURCE_DIR"]).expanduser().resolve() - package_roots = [ - candidate - for candidate in (source_dir, source_dir / "src") - if (candidate / "herdres_connector" / "source_sync.py").is_file() - ] - assert len(package_roots) == 1, ( - "HERDRES_SOURCE_DIR must contain herdres_connector/source_sync.py " - "either directly or under src/" - ) - package_root = package_roots[0] - monkeypatch.syspath_prepend(str(package_root)) - preloaded_herdres_modules = sorted( - name - for name in sys.modules - if name == "herdres_connector" or name.startswith("herdres_connector.") - ) - assert preloaded_herdres_modules == [] - direct_boundary_attempts: list[str] = [] - - def reject_direct_boundary(*_args: Any, **_kwargs: Any) -> Any: - direct_boundary_attempts.append("process_or_socket") - raise AssertionError("Herdres source sync must not access Herdr outside Tendwire") - - class RejectDirectSocket(socket.socket): - def __new__(cls, *_args: Any, **_kwargs: Any) -> Any: - direct_boundary_attempts.append("socket") - raise AssertionError("Herdres source sync must not open a direct socket") - - - real_import = builtins.__import__ - - def guarded_import( - name: str, - globals: Any = None, - locals: Any = None, - fromlist: Any = (), - level: int = 0, - ) -> Any: - private_roots = { - "herdr", - "herdr_turn_adapter", - "herdr_socket", - "herdr_cli", - "herdr_events", - } - if name.split(".", 1)[0] in private_roots or name.startswith( - "tendwire.backends" - ): - direct_boundary_attempts.append(f"import:{name[:80]}") - raise AssertionError("Herdres source sync must not import a direct Herdr client") - return real_import(name, globals, locals, fromlist, level) - - monkeypatch.setattr(builtins, "__import__", guarded_import) - monkeypatch.setattr(subprocess, "run", reject_direct_boundary) - monkeypatch.setattr(subprocess, "Popen", reject_direct_boundary) - monkeypatch.setattr(socket, "socket", RejectDirectSocket) - monkeypatch.setattr(socket, "create_connection", reject_direct_boundary) - monkeypatch.setenv("HERDRES_TENDWIRE_MODE", "source") - monkeypatch.setenv("HERDRES_PINNED_STATUS", "0") - source_sync = importlib.import_module("herdres_connector.source_sync") - herdres_state = importlib.import_module("herdres_connector.state") - source_sync_path = Path(source_sync.__file__).resolve() - loaded_herdres_files = { - name: Path(module.__file__).resolve() - for name, module in sys.modules.items() - if (name == "herdres_connector" or name.startswith("herdres_connector.")) - and getattr(module, "__file__", None) - } - assert source_sync_path == package_root / "herdres_connector" / "source_sync.py" - assert loaded_herdres_files - assert all(path.is_relative_to(package_root) for path in loaded_herdres_files.values()) - - worker_observation = next( - worker - for worker in snapshot_payload["workers"] - if worker["id"] == public_turn["worker_id"] - ) - space_observation = next( - space - for space in snapshot_payload["spaces"] - if space["id"] == public_turn["space_id"] - ) - store = { - "enabled": True, - "telegram": {"chat_id": "-100", "general_thread_id": "1"}, - "panes": {}, - "spaces": {}, - "tendwired_bootstrap_complete": True, - } - _worker_key, worker_entry, _created = herdres_state.upsert_worker_entry( - store, - worker_observation, - topic_id="77", - ) - herdres_state.upsert_space_entry( - store, - space_observation, - topic_id="77", - ) - worker_entry.update( - { - "last_stream_turn_id": public_turn["id"], - "last_stream_hash": "persisted-working-hash", - "last_stream_message_id": "555", - "last_stream_bot_kind": "manager", - } - ) - herdres_state.bind_message_to_worker( - store, - "555", - worker_entry, - topic_id="77", - kind="working", - turn_id=public_turn["id"], - bot_kind="manager", - ) - - class RecoveryTendwire: - def __init__(self) -> None: - self.calls: list[str] = [] - self.turn_payload_objects: list[dict[str, Any]] = [] - - def snapshot(self) -> dict[str, Any]: - self.calls.append("snapshot") - return snapshot_payload - - def turns(self) -> dict[str, Any]: - self.calls.append("turns") - self.turn_payload_objects.append(turns_payload) - return turns_payload - - def pending(self) -> dict[str, Any]: - self.calls.append("pending") - return {"ok": True, "pending_interactions": []} - - class RecoveryTelegram: - dry_run = False - - def __init__( - self, - token: str = "fake", - shared: dict[str, list[Any]] | None = None, - ) -> None: - self.token = token - self.shared = shared or { - "sent": [], - "edited": [], - "topics": [], - "deleted_topics": [], - "renamed_topics": [], - "pins": [], - "api_calls": [], - "icon_edits": [], - } - self.sent = self.shared["sent"] - self.edited = self.shared["edited"] - self.topics = self.shared["topics"] - self.deleted_topics = self.shared["deleted_topics"] - self.renamed_topics = self.shared["renamed_topics"] - self.pins = self.shared["pins"] - self.api_calls = self.shared["api_calls"] - self.icon_edits = self.shared["icon_edits"] - - def with_token(self, token: str) -> Any: - return RecoveryTelegram(token=token, shared=self.shared) - - def api(self, method: str, payload: dict[str, Any]) -> dict[str, Any]: - self.api_calls.append((method, dict(payload), self.token)) - if method == "editMessageText": - rich_payload = payload.get("rich_message") - rich = json.loads(rich_payload) if rich_payload else {} - html = str(rich.get("html") or payload.get("text") or "") - message_id = str(payload.get("message_id") or "") - self.edited.append((str(payload.get("chat_id") or ""), message_id, html)) - return {"ok": True, "result": {"message_id": message_id}} - if method == "sendRichMessage": - message_id = str(100 + len(self.sent)) - rich = json.loads(payload.get("rich_message") or "{}") - self.sent.append( - ( - str(payload.get("chat_id") or ""), - str(rich.get("html") or ""), - {"thread_id": str(payload.get("message_thread_id") or "")}, - message_id, - ) - ) - return {"ok": True, "result": {"message_id": message_id}} - return {"ok": True, "result": {"message_id": "0"}} - - def create_topic( - self, - _chat_id: str, - name: str, - icon_color: int | None = None, - ) -> dict[str, Any]: - self.topics.append((name, icon_color)) - return {"ok": True, "topic_id": str(76 + len(self.topics))} - - def rename_topic( - self, - chat_id: str, - thread_id: str, - name: str, - ) -> dict[str, Any]: - self.renamed_topics.append((chat_id, thread_id, name)) - return {"ok": True} - - def edit_topic_icon( - self, - chat_id: str, - thread_id: str, - emoji_id: str, - ) -> dict[str, Any]: - self.icon_edits.append((chat_id, thread_id, emoji_id)) - return {"ok": True} - - def delete_topic(self, _chat_id: str, thread_id: str) -> dict[str, Any]: - self.deleted_topics.append(thread_id) - return {"ok": True} - - def send_message( - self, - chat_id: str, - html: str, - **kwargs: Any, - ) -> dict[str, Any]: - message_id = str(100 + len(self.sent)) - self.sent.append((chat_id, html, dict(kwargs), message_id)) - return {"ok": True, "message_id": message_id} - - def edit_message( - self, - chat_id: str, - message_id: str, - html: str, - ) -> dict[str, Any]: - self.edited.append((chat_id, str(message_id), html)) - return {"ok": True, "message_id": str(message_id)} - - def pin_message(self, chat_id: str, message_id: str) -> dict[str, Any]: - self.pins.append((chat_id, str(message_id))) - return {"ok": True} - - tendwire = RecoveryTendwire() - telegram = RecoveryTelegram() - runtime = source_sync.SyncRuntime(tendwire, telegram, with_outbox=False) - - first = source_sync.sync_once(store, runtime) - ledger_after_first = json.loads( - json.dumps(herdres_state.delivered_turns(store), sort_keys=True) - ) - edits_after_first = list(telegram.edited) - sends_after_first = list(telegram.sent) - second = source_sync.sync_once(store, runtime) - ledger_after_second = json.loads( - json.dumps(herdres_state.delivered_turns(store), sort_keys=True) - ) - third = source_sync.sync_once(store, runtime) - - binding = herdres_state.find_message_binding(store, "555", topic_id="77") - ledger = herdres_state.delivered_turns(store) - assert tendwire.calls == ["snapshot", "turns", "pending"] * 3 - assert tendwire.turn_payload_objects == [turns_payload] * 3 - assert all(payload is turns_payload for payload in tendwire.turn_payload_objects) - assert first["feed_sent"] == first["sent"] == 1 - assert len(telegram.edited) == 1 - assert telegram.edited[0][1] == "555" - assert public_turn["assistant_final_text"] in telegram.edited[0][2] - assert telegram.sent == [] - assert second["feed_sent"] == second["sent"] == second["turn_updates"] == 0 - assert third["feed_sent"] == third["sent"] == third["turn_updates"] == 0 - assert telegram.edited == edits_after_first - assert telegram.sent == sends_after_first - assert ledger_after_second == ledger_after_first - assert ledger == ledger_after_first - assert len(ledger) == 1 - assert list(ledger.values())[0]["turn_id"] == public_turn["id"] - assert binding is not None - assert binding["kind"] == "final" - assert binding["turn_id"] == public_turn["id"] - assert binding["topic_id"] == "77" - assert worker_entry["topic_id"] == "77" - assert worker_entry["last_turn_id"] == public_turn["id"] - assert worker_entry["last_clean_message_id"] == "555" - assert worker_entry["last_clean_message_ids"] == ["555"] - assert "last_stream_turn_id" not in worker_entry - assert "last_stream_hash" not in worker_entry - assert "last_stream_message_id" not in worker_entry - assert "last_stream_bot_kind" not in worker_entry - assert direct_boundary_attempts == [] - - -def _turn_api_pane() -> dict[str, Any]: - return { - "pane_id": "w123456789abcde:pA", - "terminal_id": "terminal-turn-api", - "agent": "claude", - "workspace_id": "w123456789abcde", - "agent_status": "idle", - "last_completed_turn": { - "turn": 0, - "turn_epoch": 7, - "completed_unix_ms": 0, - }, - "turn": 0, - "turn_epoch": 7, - "outcome": "completed", - } - - -def _turn_record( - turn: int, - *, - epoch: int = 7, - outcome: str = "completed", -) -> dict[str, Any]: - return { - "turn": turn, - "turn_epoch": epoch, - "outcome": outcome, - "completed_unix_ms": 1_700_000_000_000 + turn, - "message": f"turn {turn}", - "message_truncated": False, - "agent_session_path": None, - } - - -def _turn_api_backend( - tmp_path: Path, - host_id: str, - processor: Callable[..., Any], -) -> HerdrEventBackend: - backend = _backend(tmp_path, host_id) - backend.turn_completion_processor = processor - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[_turn_api_pane()], - ) - ) - return backend - - -def test_turn_api_absent_preserves_old_subscription_negotiation(tmp_path: Path) -> None: - backend = _turn_api_backend( - tmp_path, - "turn-api-absent", - lambda *_args, **_kwargs: SimpleNamespace(status="updated"), - ) - - class OldClient: - subscriptions: list[dict[str, Any]] = [] - - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - raise HerdrErrorResponse( - {"code": "invalid_params", "message": "unknown method"}, - "probe", - ) - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - self.subscriptions.append(dict(params)) - return SimpleNamespace(subscription_id="old") - - client = OldClient() - backend._replay_turns_after_reconcile(client) - backend._subscribe_event_stream(client) - - assert backend._turn_api_supported is False - assert all( - item["type"] != "pane.turn_completed" - for item in client.subscriptions[0]["subscriptions"] - ) - assert get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - _turn_api_pane()["pane_id"], - ) is None - - -def test_turn_api_idless_unknown_variant_probe_is_inert( - tmp_path: Path, -) -> None: - backend = _turn_api_backend( - tmp_path, - "turn-api-idless-unknown-variant", - lambda *_args, **_kwargs: SimpleNamespace(status="updated"), - ) - # Exact envelope emitted by live stock Herdr 0.7.5 (reported upstream): - # the id is present but EMPTY, and the method name is backticked. - raw_probe_error = { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: unknown variant `pane.turns`, expected one of `pane.read`, `pane.list`, `agent.list`", - }, - } - - def handler(conn: _SocketConnection) -> None: - probe = conn.read_request() - assert probe["method"] == "pane.turns" - conn.send_json(raw_probe_error) - - ordinary = conn.read_request() - assert ordinary["method"] == "workspace.list" - conn.send_json({"id": ordinary["id"], "result": {"workspaces": []}}) - - subscription = conn.read_request() - assert subscription["method"] == HERDR_EVENTS_SUBSCRIBE_METHOD - assert all( - item["type"] != "pane.turn_completed" - for item in subscription["params"]["subscriptions"] - ) - conn.send_json( - { - "id": subscription["id"], - "result": {"type": "subscription_started"}, - } - ) - - with _FakeHerdrSocketServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - backend._replay_turns_after_reconcile(client) - assert client.workspace_list() == {"workspaces": []} - backend._subscribe_event_stream(client) - client.close() - - assert [request["method"] for request in server.requests] == [ - "pane.turns", - "workspace.list", - HERDR_EVENTS_SUBSCRIBE_METHOD, - ] - assert backend._turn_api_supported is False - assert ( - _table_count( - backend.db_path, - backend.config.host_id, - "herdr_turn_watermarks", - ) - == 0 - ) - assert get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - _turn_api_pane()["pane_id"], - ) is None - - -def test_turn_api_unknown_variant_classifier_requires_exact_code_and_prefix() -> None: - assert HerdrEventBackend._turn_api_method_unsupported( - HerdrErrorResponse( - { - "code": "invalid_request", - "message": "invalid request: unknown variant pane.turns", - }, - "probe", - ) - ) - assert not HerdrEventBackend._turn_api_method_unsupported( - HerdrErrorResponse( - { - "code": "invalid_params", - "message": "invalid request: unknown variant pane.turns", - }, - "probe", - ) - ) - assert not HerdrEventBackend._turn_api_method_unsupported( - HerdrErrorResponse( - { - "code": "invalid_request", - "message": "pane.turns failed: unknown variant", - }, - "probe", - ) - ) - - -def test_turn_api_first_connect_baselines_full_ring_without_processing( - tmp_path: Path, -) -> None: - processed: list[str] = [] - binding_hints: list[str | None] = [] - - def process(_config: Config, pane_id: str, **kwargs: Any) -> Any: - processed.append(pane_id) - binding_hints.append(kwargs.get("binding_private_fingerprint")) - return SimpleNamespace( - status="updated", - worker_id="claude", - refreshed_turn_id=f"public-turn-{len(processed)}", - ) - - backend = _turn_api_backend(tmp_path, "turn-api-happy", process) - - class NewClient: - subscriptions: list[dict[str, Any]] = [] - - def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: - assert params == { - "pane_id": _turn_api_pane()["pane_id"], - "since": 0, - } - return { - "type": "pane_turns", - "turns": { - "pane_id": _turn_api_pane()["pane_id"], - "turn_epoch": 7, - "records": [_turn_record(turn) for turn in range(1, 65)], - "truncated": False, - "oldest_available": 1, - }, - } - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - self.subscriptions.append(dict(params)) - return SimpleNamespace(subscription_id="new") - - client = NewClient() - backend._replay_turns_after_reconcile(client) - backend._subscribe_event_stream(client) - - assert processed == [] - assert binding_hints == [] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - _turn_api_pane()["pane_id"], - ) - assert watermark is not None - assert (watermark.turn_epoch, watermark.last_turn) == (7, 64) - assert watermark.completeness_break_count == 0 - assert { - "type": "pane.turn_completed", - "pane_id": _turn_api_pane()["pane_id"], - } in client.subscriptions[0]["subscriptions"] - with sqlite3.connect(str(backend.db_path)) as conn: - assert conn.execute( - """ - SELECT turn, outcome, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? - ORDER BY turn - """, - (backend.config.host_id,), - ).fetchall() == [] - - -def test_turn_completion_late_attaches_final_after_idle_event_wins_race( - tmp_path: Path, - monkeypatch: Any, -) -> None: - config = Config( - host_id="turn-api-content-race", - data_dir=tmp_path, - db_path=tmp_path / "turn-api-content-race.db", - herdr_backend="socket", - herdr_bin="turn-adapter", - herdr_timeout_seconds=0.5, - ) - init_store(Path(config.db_path)) - backend = HerdrEventBackend( - config, - debounce_seconds=0, - reconnect_delay_seconds=0, - ) - pane_id = _turn_api_pane()["pane_id"] - working_pane = { - **_turn_api_pane(), - "agent_status": "working", - } - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[working_pane], - ) - ) - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - - # Reproduce the production ordering: the status stream closes the public - # projection first, before any semantic adapter read has run. - assert backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": { - **working_pane, - "agent_status": "idle", - }, - } - ) - idle_snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert idle_snapshot is not None - idle_payload = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=idle_snapshot, - ) - assert idle_snapshot.workers[0].status == "idle" - assert not any( - turn.get("assistant_final_text") for turn in idle_payload["turns"] - ) - - adapter_calls: list[list[str]] = [] - final_payload = { - "result": { - "turn": { - "available": True, - "user_text": "reply with purple elephant 42", - "assistant_final_text": "purple elephant 42", - "complete": True, - "has_open_turn": False, - "source_turn_id": "event-close-race-turn", - } - } - } - - def fake_run(args: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]: - adapter_calls.append(args) - return subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps(final_payload), - stderr="", - ) - - monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) - backend._turn_api_supported = True - assert backend.queue_event_envelope( - { - "event": "pane.turn_completed", - "data": { - "pane": { - **working_pane, - "agent_status": "idle", - }, - **_turn_record(1), - }, - } - ) - - completed_snapshot = latest_snapshot( - backend.db_path, - backend.config.host_id, - ) - assert completed_snapshot is not None - captured_payload = turns_payload_from_store( - backend.db_path, - backend.config.host_id, - snapshot=completed_snapshot, - ) - captured = next( - turn - for turn in captured_payload["turns"] - if turn.get("assistant_final_text") == "purple elephant 42" - ) - assert captured["status"] == "idle" - assert captured["complete"] is True - assert adapter_calls == [ - [ - "turn-adapter", - "pane", - "turn", - pane_id, - "--last", - "--format", - "json", - ] - ] - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - completion = conn.execute( - """ - SELECT turn, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - """, - (backend.config.host_id, pane_id), - ).fetchone() - assert completion == (1, captured["id"]) - - -@pytest.mark.parametrize( - "refresh_status", - [ - "timeout", - "failed", - "stale_binding", - "binding_missing", - "binding_ambiguous", - "store_unavailable", - ], -) -def test_default_completion_processor_nonterminal_status_advances_with_diagnostic( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - refresh_status: str, -) -> None: - backend = _backend(tmp_path, f"default-composition-{refresh_status}") - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[_turn_api_pane()], - ) - ) - pane_id = _turn_api_pane()["pane_id"] - bindings = list_worker_bindings( - backend.db_path, - backend.config.host_id, - backend="herdr", - ) - assert len(bindings) == 1 - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=0, - ) - - if refresh_status in {"timeout", "failed", "stale_binding"}: - monkeypatch.setattr( - herdr_turns, - "_refresh_turn_binding", - lambda *_args, **_kwargs: herdr_turns.TurnRefreshResult( - refresh_status, - 0, - ), - ) - elif refresh_status == "binding_missing": - monkeypatch.setattr( - herdr_turns, - "list_worker_bindings", - lambda *_args, **_kwargs: [], - ) - elif refresh_status == "binding_ambiguous": - monkeypatch.setattr( - herdr_turns, - "list_worker_bindings", - lambda *_args, **_kwargs: [bindings[0], bindings[0]], - ) - else: - def unavailable(*_args: Any, **_kwargs: Any) -> Any: - raise RuntimeError("store unavailable") - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", unavailable) - - assert backend.queue_event_envelope( - { - "event": "pane.turn_completed", - "data": {"pane": _turn_api_pane(), **_turn_record(1)}, - } - ) - - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 1 - assert backend.operational_status["turn_completion_diagnostics"] == { - f"herdr_turn_completion_refresh_skipped:{refresh_status}": 1 - } - with closing(sqlite3.connect(str(backend.db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - """, - (backend.config.host_id, pane_id), - ).fetchall() == [(1, None)] - - -def test_replay_timeout_on_one_pane_does_not_block_other_panes_or_subscribe( - tmp_path: Path, -) -> None: - pane_a = _turn_api_pane() - pane_b = { - **_turn_api_pane(), - "pane_id": "w123456789abcde:pB", - "terminal_id": "terminal-turn-api-b", - "agent": "codex", - } - processed: list[str] = [] - - def process(_config: Config, pane_id: str, **_kwargs: Any) -> Any: - processed.append(pane_id) - return SimpleNamespace( - status="timeout" if pane_id == pane_a["pane_id"] else "updated", - worker_id=pane_id, - refreshed_turn_id=f"public-{pane_id}", - ) - - backend = _backend(tmp_path, "turn-api-pane-timeout") - backend.turn_completion_processor = process - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[pane_a, pane_b], - ) - ) - for pane in (pane_a, pane_b): - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane["pane_id"], - turn_epoch=7, - last_turn=0, - ) - - class Client: - subscriptions: list[dict[str, Any]] = [] - - def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: - pane_id = str(params["pane_id"]) - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": [{**_turn_record(1), "pane_id": pane_id}], - "truncated": False, - "oldest_available": 1, - } - } - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - self.subscriptions.append(dict(params)) - return SimpleNamespace(subscription_id="healthy") - - client = Client() - backend._replay_turns_after_reconcile(client) - backend._subscribe_event_stream(client) - - assert set(processed) == {pane_a["pane_id"], pane_b["pane_id"]} - assert len(client.subscriptions) == 1 - for pane in (pane_a, pane_b): - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane["pane_id"], - ) - assert watermark is not None and watermark.last_turn == 1 - - -def test_first_probe_pane_not_found_skips_only_that_pane(tmp_path: Path) -> None: - pane_a = _turn_api_pane() - pane_b = { - **_turn_api_pane(), - "pane_id": "w123456789abcde:pB", - "terminal_id": "terminal-turn-api-b", - "agent": "codex", - } - processed: list[str] = [] - backend = _backend(tmp_path, "turn-api-probe-pane-missing") - backend.turn_completion_processor = ( - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) or SimpleNamespace(status="unchanged") - ) - ) - backend.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[pane_a, pane_b], - ) - ) - for pane in (pane_a, pane_b): - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane["pane_id"], - turn_epoch=7, - last_turn=0, - ) - - class Client: - subscriptions: list[dict[str, Any]] = [] - - def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: - pane_id = str(params["pane_id"]) - if pane_id == pane_a["pane_id"]: - raise HerdrErrorResponse( - {"code": "pane_not_found", "message": "pane disappeared"}, - "probe", - ) - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": [{**_turn_record(1), "pane_id": pane_id}], - "truncated": False, - "oldest_available": 1, - } - } - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - self.subscriptions.append(dict(params)) - return SimpleNamespace(subscription_id="new") - - client = Client() - backend._replay_turns_after_reconcile(client) - backend._subscribe_event_stream(client) - - assert backend._turn_api_supported is True - assert processed == [pane_b["pane_id"]] - assert len(client.subscriptions) == 1 - assert { - "type": "pane.turn_completed", - "pane_id": pane_b["pane_id"], - } in client.subscriptions[0]["subscriptions"] - - -def test_malformed_completion_is_quarantined_without_poisoning_batch( - tmp_path: Path, -) -> None: - backend = _backend(tmp_path, "turn-api-malformed-batch", debounce_seconds=1) - backend.reconcile_once(client=_initial_pane_client()) - assert backend.queue_event_envelope( - { - "event": "pane.turn_completed", - "data": { - "pane_id": "pane-1", - "turn": "not-an-integer", - "turn_epoch": 7, - "outcome": "completed", - "completed_unix_ms": 1, - }, - }, - flush=False, - ) - assert backend.queue_event_envelope(_status_event("idle"), flush=False) - - backend.flush() - - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - assert snapshot is not None - assert snapshot.workers[0].status == "idle" - assert backend.health.outcome != "protocol_error" - assert backend.operational_status["turn_completion_diagnostics"] == { - "herdr_turn_completion_record_quarantined": 1 - } - - -def test_replay_gap_inside_records_records_break_without_processing( - tmp_path: Path, -) -> None: - processed: list[str] = [] - backend = _turn_api_backend( - tmp_path, - "turn-api-interior-gap", - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) or SimpleNamespace(status="updated") - ), - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=2, - ) - - class GapClient: - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": [_turn_record(3), _turn_record(5)], - "truncated": False, - "oldest_available": 3, - } - } - - backend._replay_turns_after_reconcile(GapClient()) - - assert processed == [] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None - assert watermark.last_turn == 5 - assert watermark.last_completeness_break_reason == "replay_gap" - - -def test_optional_signature_fallback_never_double_invokes_typeerror( - tmp_path: Path, -) -> None: - calls = 0 - - def broken_processor(*_args: Any, **_kwargs: Any) -> Any: - nonlocal calls - calls += 1 - raise TypeError("processor body failed") - - backend = _turn_api_backend( - tmp_path, - "turn-api-no-double-invoke", - broken_processor, - ) - with pytest.raises(TypeError, match="processor body failed"): - backend._completion_processor_result( - herdr_events._turn_completion_record( - {"pane_id": _turn_api_pane()["pane_id"], **_turn_record(1)} - ) - ) - assert calls == 1 - - pane_calls = 0 - - class BrokenClient: - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - nonlocal pane_calls - pane_calls += 1 - raise TypeError("pane.turns body failed") - - with pytest.raises(TypeError, match="pane.turns body failed"): - backend._call_pane_turns( - BrokenClient(), - _turn_api_pane()["pane_id"], - since=0, - expected_epoch=None, - ) - assert pane_calls == 1 - - -def test_turn_api_run_loop_closes_probe_to_subscribe_completion_race( - tmp_path: Path, -) -> None: - config = _config(tmp_path, "turn-api-subscribe-race") - init_store(Path(config.db_path)) - pane_id = _turn_api_pane()["pane_id"] - processed: list[str] = [] - operations: list[str] = [] - - class RaceClient(_StaticClient): - def __init__(self) -> None: - super().__init__( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[_turn_api_pane()], - ) - self.subscribed = False - - def connect(self) -> None: - return None - - def close(self) -> None: - return None - - def pane_turns( - self, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - operations.append("replay" if self.subscribed else "probe") - records = [_turn_record(1)] if self.subscribed else [] - if self.subscribed: - backend.stop_event.set() - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": records, - "truncated": False, - "oldest_available": 1 if records else None, - } - } - - def subscribe( - self, - _method: str, - params: Mapping[str, Any], - **_kwargs: Any, - ) -> Any: - operations.append("subscribe") - assert { - "type": "pane.turn_completed", - "pane_id": pane_id, - } in params["subscriptions"] - self.subscribed = True - return SimpleNamespace(subscription_id="race") - - client = RaceClient() - backend = HerdrEventBackend( - config, - client_factory=lambda _config: client, - debounce_seconds=0, - reconnect_delay_seconds=0, - turn_completion_processor=lambda _config, current_pane, **_kwargs: ( - processed.append(current_pane) - or SimpleNamespace(status="unchanged") - ), - ) - - backend.run_forever() - - assert operations == ["probe", "subscribe", "replay"] - assert processed == [pane_id] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None - assert (watermark.turn_epoch, watermark.last_turn) == (7, 1) - - -def test_turn_api_restart_replays_exactly_the_missed_turn(tmp_path: Path) -> None: - pane_id = _turn_api_pane()["pane_id"] - first = _turn_api_backend( - tmp_path, - "turn-api-restart", - lambda *_args, **_kwargs: SimpleNamespace(status="unchanged"), - ) - set_herdr_turn_watermark( - first.db_path, - first.config.host_id, - pane_id, - turn_epoch=7, - last_turn=2, - ) - processed: list[str] = [] - restarted = HerdrEventBackend( - first.config, - debounce_seconds=0, - reconnect_delay_seconds=0, - turn_completion_processor=lambda _config, current_pane, **_kwargs: ( - processed.append(current_pane) - or SimpleNamespace(status="unchanged") - ), - ) - restarted.reconcile_once( - client=_StaticClient( - workspaces=[{"id": "w123456789abcde", "name": "Build"}], - panes=[_turn_api_pane()], - ) - ) - - class ReplayClient: - params: dict[str, Any] | None = None - - def pane_turns(self, params: Mapping[str, Any], **_kwargs: Any) -> Any: - self.params = dict(params) - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": 7, - "records": [_turn_record(3)], - "truncated": False, - "oldest_available": 1, - } - } - - client = ReplayClient() - restarted._replay_turns_after_reconcile(client) - - assert client.params == { - "pane_id": pane_id, - "since": 2, - "expected_epoch": 7, - } - assert processed == [pane_id] - watermark = get_herdr_turn_watermark( - restarted.db_path, - restarted.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 3 - - -@pytest.mark.parametrize( - ("mode", "expected_epoch", "expected_turn", "expected_reason"), - [ - ("truncated", 7, 12, "replay_truncated"), - ("epoch", 8, 1, "turn_epoch_mismatch"), - ], -) -def test_turn_api_breaks_rebaseline_without_inferring_completion( - tmp_path: Path, - mode: str, - expected_epoch: int, - expected_turn: int, - expected_reason: str, -) -> None: - processed: list[str] = [] - backend = _turn_api_backend( - tmp_path, - f"turn-api-break-{mode}", - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) - or SimpleNamespace(status="updated") - ), - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=2, - ) - - class BreakClient: - calls = 0 - - def pane_turns(self, _params: Mapping[str, Any], **_kwargs: Any) -> Any: - self.calls += 1 - if mode == "epoch" and self.calls == 1: - raise HerdrErrorResponse( - { - "code": "turn_epoch_mismatch", - "message": "epoch changed", - }, - "epoch", - ) - records = ( - [_turn_record(1, epoch=8)] - if mode == "epoch" - else [_turn_record(11), _turn_record(12)] - ) - return { - "turns": { - "pane_id": pane_id, - "turn_epoch": expected_epoch, - "records": records, - "truncated": mode == "truncated", - "oldest_available": records[0]["turn"], - } - } - - backend._replay_turns_after_reconcile(BreakClient()) - - assert processed == [] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None - assert (watermark.turn_epoch, watermark.last_turn) == ( - expected_epoch, - expected_turn, - ) - assert watermark.completeness_break_count == 1 - assert watermark.last_completeness_break_reason == expected_reason - - -def test_live_aborted_completion_refreshes_then_advances_with_provenance( - tmp_path: Path, -) -> None: - processed: list[str] = [] - backend = _turn_api_backend( - tmp_path, - "turn-api-live-abort", - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) - or SimpleNamespace( - status="updated", - worker_id="claude", - refreshed_turn_id="public-aborted-turn", - ) - ), - ) - pane_id = _turn_api_pane()["pane_id"] - set_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - turn_epoch=7, - last_turn=1, - ) - backend._turn_api_supported = True - - assert backend.queue_event_envelope( - { - "event": "pane.turn_completed", - "data": { - "pane": _turn_api_pane(), - **_turn_record(2, outcome="aborted"), - }, - } - ) - - assert processed == [pane_id] - watermark = get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) - assert watermark is not None and watermark.last_turn == 2 - with sqlite3.connect(str(backend.db_path)) as conn: - assert conn.execute( - """ - SELECT outcome, refreshed_turn_id - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? AND turn = 2 - """, - (backend.config.host_id, pane_id), - ).fetchone() == ("aborted", "public-aborted-turn") - - -def test_status_turn_hints_never_advance_completion_watermarks( - tmp_path: Path, -) -> None: - processed: list[str] = [] - backend = _turn_api_backend( - tmp_path, - "turn-api-hints-only", - lambda _config, pane_id, **_kwargs: ( - processed.append(pane_id) - or SimpleNamespace(status="updated") - ), - ) - pane_id = _turn_api_pane()["pane_id"] - - assert backend.queue_event_envelope( - { - "event": "pane.agent_status_changed", - "data": { - "pane_id": pane_id, - "workspace_id": "w123456789abcde", - "agent": "claude", - "agent_status": "idle", - "turn": 19, - "turn_epoch": 7, - }, - } - ) - - assert processed == [] - assert get_herdr_turn_watermark( - backend.db_path, - backend.config.host_id, - pane_id, - ) is None diff --git a/tests/test_herdr_protocol.py b/tests/test_herdr_protocol.py index d425611..0b786cf 100644 --- a/tests/test_herdr_protocol.py +++ b/tests/test_herdr_protocol.py @@ -1,288 +1,38 @@ -"""Tests for the inactive Herdr socket protocol helpers.""" - from __future__ import annotations -import json -from pathlib import Path - import pytest from tendwire.backends.herdr_protocol import ( HerdrEnvelopeError, HerdrMalformedLineError, HerdrRequestIdMismatchError, - HerdrSocketPathError, - HERDR_EVENTS_SUBSCRIBE_METHOD, - HERDR_OFFICIAL_EVENT_NAME_SET, - HERDR_OFFICIAL_EVENT_NAMES, - build_events_subscribe_params, - build_events_subscribe_request, build_request, ensure_response_id, - error_payload, frame_request, - is_error_response, - is_event, - is_result_response, parse_json_line, resolve_socket_path, - result_payload, - validate_event, - validate_response, validate_server_envelope, ) -def test_resolve_socket_path_prefers_explicit_absolute_path(tmp_path: Path) -> None: - socket_path = tmp_path / "herdr.sock" - env = {"TENDWIRE_HERDR_SOCKET": str(tmp_path / "ignored.sock")} - - assert resolve_socket_path(socket_path, env=env) == str(socket_path) - - -def test_resolve_socket_path_expands_home_for_explicit_path(tmp_path: Path) -> None: - home = tmp_path / "home" - - assert resolve_socket_path("~/custom.sock", home=home) == str(home / "custom.sock") - - -def test_resolve_socket_path_rejects_relative_explicit_path() -> None: - with pytest.raises(HerdrSocketPathError): - resolve_socket_path("relative.sock") - - -def test_resolve_socket_path_env_order_and_home_expansion(tmp_path: Path) -> None: - home = tmp_path / "home" - env = { - "TENDWIRE_HERDR_SOCKET": "~/primary.sock", - "HERDR_SOCKET_PATH": str(tmp_path / "secondary.sock"), - "TENDWIRE_HERDR_SESSION": "session-a", - "HERDR_SESSION": "session-b", - } - - assert resolve_socket_path(env=env, home=home) == str(home / "primary.sock") - - -def test_resolve_socket_path_uses_herdr_socket_path_when_primary_empty(tmp_path: Path) -> None: - env = { - "TENDWIRE_HERDR_SOCKET": "", - "HERDR_SOCKET_PATH": str(tmp_path / "secondary.sock"), - "TENDWIRE_HERDR_SESSION": "ignored", - } - - assert resolve_socket_path(env=env, home=tmp_path) == str(tmp_path / "secondary.sock") - - -def test_resolve_socket_path_uses_tendwire_session_before_herdr_session(tmp_path: Path) -> None: - env = { - "TENDWIRE_HERDR_SESSION": "alpha", - "HERDR_SESSION": "beta", - } - - assert resolve_socket_path(env=env, home=tmp_path) == str( - tmp_path / ".config" / "herdr" / "sessions" / "alpha" / "herdr.sock" - ) - - -def test_resolve_socket_path_uses_herdr_session_when_tendwire_session_empty(tmp_path: Path) -> None: - env = { - "TENDWIRE_HERDR_SOCKET": "", - "HERDR_SOCKET_PATH": " ", - "TENDWIRE_HERDR_SESSION": "", - "HERDR_SESSION": "beta", - } - - assert resolve_socket_path(env=env, home=tmp_path) == str( - tmp_path / ".config" / "herdr" / "sessions" / "beta" / "herdr.sock" - ) - - -def test_resolve_socket_path_defaults_to_config_socket(tmp_path: Path) -> None: - assert resolve_socket_path(env={}, home=tmp_path) == str( - tmp_path / ".config" / "herdr" / "herdr.sock" - ) - - -def test_resolve_socket_path_rejects_relative_env_socket_path() -> None: - with pytest.raises(HerdrSocketPathError): - resolve_socket_path(env={"TENDWIRE_HERDR_SOCKET": "relative.sock"}) - - -def test_build_request_uses_unique_string_ids_and_newline_framing() -> None: - first = build_request("pane.read", {"pane_id": "p-1"}) - second = build_request("pane.read", {"pane_id": "p-1"}) - - assert isinstance(first["id"], str) - assert isinstance(second["id"], str) - assert first["id"] != second["id"] - assert first["method"] == "pane.read" - assert first["params"] == {"pane_id": "p-1"} - - line = frame_request(first) - assert line.endswith(b"\n") - assert json.loads(line.decode("utf-8")) == first - - - -def test_build_events_subscribe_request_uses_official_method_params_and_default_order() -> None: - params = build_events_subscribe_params() - - assert params == {"subscriptions": [{"type": name} for name in HERDR_OFFICIAL_EVENT_NAMES]} - assert build_events_subscribe_request(request_id="sub-1") == { - "id": "sub-1", - "method": HERDR_EVENTS_SUBSCRIBE_METHOD, - "params": params, - } - assert HERDR_OFFICIAL_EVENT_NAME_SET == frozenset(HERDR_OFFICIAL_EVENT_NAMES) - - -@pytest.mark.parametrize("event_name", HERDR_OFFICIAL_EVENT_NAMES) -def test_build_events_subscribe_params_accepts_each_official_event_name(event_name: str) -> None: - assert build_events_subscribe_params([event_name]) == {"subscriptions": [{"type": event_name}]} - - -def test_official_event_subscription_names_exclude_legacy_aliases() -> None: - legacy = { - "pane.observed", - "workspace.observed", - "agent.status_changed", - "agent.detected", - "worktree.updated", - } - - assert legacy.isdisjoint(HERDR_OFFICIAL_EVENT_NAME_SET) +def test_request_round_trip_is_one_correlated_json_line() -> None: + request = build_request("agent.acp_status", {"target": "worker"}, request_id="r1") + assert parse_json_line(frame_request(request)) == request + ensure_response_id({"id": "r1", "result": {}}, "r1") + with pytest.raises(HerdrRequestIdMismatchError): + ensure_response_id({"id": "other", "result": {}}, "r1") -@pytest.mark.parametrize( - "event_names", - [ - ["pane.observed"], - ["workspace.observed"], - ["agent.status_changed"], - ["worktree.updated"], - [""], - [" "], - [" pane.created "], - [123], - [None], - object(), - ], -) -def test_build_events_subscribe_params_rejects_unknown_non_string_and_empty_names( - event_names: object, -) -> None: +def test_protocol_rejects_events_and_malformed_lines() -> None: with pytest.raises(HerdrEnvelopeError): - build_events_subscribe_params(event_names) # type: ignore[arg-type] - -def test_parse_valid_result_error_and_event_envelopes() -> None: - result = validate_response( - parse_json_line( - b'{"id":"req-1","result":{"items":[{"id":"a"}]},"future":"ignored"}\n' - ) - ) - error = validate_response(parse_json_line(b'{"id":"req-2","error":{"message":"no"}}\n')) - event = validate_event( - parse_json_line( - b'{"id":"sub-1","event":"pane.output_matched","data":{"text":"hello"}}\n' - ) - ) - idless_event = validate_event( - parse_json_line( - b'{"event":"pane.agent_status_changed","data":{"status":"blocked"}}\n' - ) - ) - - assert is_result_response(result) is True - assert result_payload(result) == {"items": [{"id": "a"}]} - assert is_error_response(error) is True - assert error_payload(error) == {"message": "no"} - assert is_event(event) is True - assert event == { - "id": "sub-1", - "event": "pane.output_matched", - "data": {"text": "hello"}, - } - assert is_event(idless_event) is True - assert idless_event == { - "event": "pane.agent_status_changed", - "data": {"status": "blocked"}, - } - - -def test_parse_json_line_rejects_malformed_json() -> None: + validate_server_envelope({"event": "pane.created", "data": {}}) with pytest.raises(HerdrMalformedLineError): - parse_json_line(b"{not json}\n") - + parse_json_line(b"not-json\n") -def test_parse_json_line_rejects_malformed_utf8() -> None: - with pytest.raises(HerdrMalformedLineError): - parse_json_line(b"\xff\n") - -def test_validate_server_envelope_rejects_missing_id() -> None: - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope({"result": {"ok": True}}) - - -def test_validate_server_envelope_scopes_herdr_075_uncorrelated_error() -> None: - envelope = { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: missing field pane_id", - }, - } - - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope(envelope) - - assert validate_server_envelope( - envelope, - allow_uncorrelated_error=True, - ) == envelope - - for error in ( - {"code": "permission_denied", "message": "invalid request: denied"}, - {"code": "invalid_request", "message": "subscription unavailable"}, - ): - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope( - {"id": "", "error": error}, - allow_uncorrelated_error=True, - ) - - -def test_validate_server_envelope_rejects_missing_id_error() -> None: - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope({"error": {"message": "uncorrelated"}}) - - -@pytest.mark.parametrize( - "envelope", - [ - {"id": "req-1"}, - {"id": "req-1", "result": {}, "error": {}}, - {"id": "req-1", "event": "pane.output", "result": {}}, - ], -) -def test_validate_server_envelope_rejects_wrong_envelope_shape(envelope: dict[str, object]) -> None: - with pytest.raises(HerdrEnvelopeError): - validate_server_envelope(envelope) - - -def test_parse_json_line_rejects_non_object_envelope() -> None: - with pytest.raises(HerdrEnvelopeError): - parse_json_line(b'["not","an","object"]\n') - - -def test_unknown_fields_are_tolerated_without_changing_result_payload() -> None: - response = validate_response( - {"id": "req-1", "result": {"raw": {"unknown": [1, 2]}}, "extra": {"ignored": True}} +def test_socket_resolution_keeps_frozen_precedence(tmp_path) -> None: + explicit = tmp_path / "explicit.sock" + assert resolve_socket_path(explicit, env={"HERDR_SESSION": "ignored"}) == str(explicit) + assert resolve_socket_path(env={"TENDWIRE_HERDR_SESSION": "work"}, home=tmp_path) == str( + tmp_path / ".config" / "herdr" / "sessions" / "work" / "herdr.sock" ) - - assert result_payload(response) == {"raw": {"unknown": [1, 2]}} - - -def test_ensure_response_id_rejects_mismatch() -> None: - with pytest.raises(HerdrRequestIdMismatchError): - ensure_response_id({"id": "actual", "result": {}}, "expected") diff --git a/tests/test_herdr_smoke.py b/tests/test_herdr_smoke.py deleted file mode 100644 index f333523..0000000 --- a/tests/test_herdr_smoke.py +++ /dev/null @@ -1,678 +0,0 @@ -import importlib.util -import json -import types -from pathlib import Path - - -import pytest - - -PROJECT_ROOT = Path(__file__).resolve().parents[1] -SMOKE_SCRIPT = PROJECT_ROOT / "scripts" / "herdr_smoke.py" -FIXTURE_ROOT = PROJECT_ROOT / "tests" / "fixtures" / "herdr" / "live_smoke" -OK_FIXTURES = FIXTURE_ROOT / "ok" -NEGATIVE_FIXTURES = FIXTURE_ROOT / "negative_private" -RAW_STATUS_FIXTURE = PROJECT_ROOT / "tests" / "fixtures" / "herdr" / "event_replay" / "status_transitions.json" - -REQUIRED_CHECKS = { - "create_attach", - "observe", - "send_addressing", - "target_validation", - "event_subscription", - "status_agent_status_changed", - "pane_moved_binding_update", - "close_exited", - "degraded_backend_preserves_workers", - "public_safety", -} - -# The public schema deliberately contains scenario names such as -# target_validation and pane_moved_binding_update. The banned list therefore -# names concrete private surfaces rather than the neutral words "target" or -# "binding" by themselves. -FORBIDDEN_PUBLIC_TERMS = ( - "telegram", - "herdres", - "raw pane", - "pane_id", - "pane-id", - "terminal_id", - "terminal", - "socket", - "backend_target", - "target_kind", - "target_value", - "private_binding", - "private_fingerprint", - "connector", - "outbox", - "delivery", - "argv", - "env", - "stdout", - "stderr", - "token", - "secret", - "fingerprint", -) - -PRIVATE_MARKERS = ( - "explicit-smoke", - "caller-smoke", - "tendwire-smoke", - "do-not-leak-token", - "do-not-leak-secret", - "actual-private-session", - "socket:///tmp/forbidden.sock", - "private fingerprint abc123", - "pane-secret", - "agent-secret", - "pane-1", - "status_transitions.json", -) - - -def _load_smoke_module(): - spec = importlib.util.spec_from_file_location("tendwire_herdr_smoke_under_test", SMOKE_SCRIPT) - module = importlib.util.module_from_spec(spec) - assert spec.loader is not None - spec.loader.exec_module(module) - return module - - -@pytest.fixture() -def smoke_module(): - return _load_smoke_module() - - -def _patch_which(monkeypatch, module, result): - if not hasattr(module, "shutil"): - monkeypatch.setattr(module, "shutil", types.SimpleNamespace(), raising=False) - monkeypatch.setattr(module.shutil, "which", lambda _name: result, raising=False) - - -def _run_main(module, argv, capsys, *, env=None, runner=None): - try: - return_code = module.main(argv, env={} if env is None else env, runner=runner) - except SystemExit as exc: - return_code = exc.code - - captured = capsys.readouterr() - assert captured.err == "" - public_text = captured.out.strip() - assert public_text, "smoke harness must print one JSON summary" - data = json.loads(public_text) - assert isinstance(data, dict) - module.validate_public_summary(data) - return return_code, public_text, data - - -def _all_strings(value): - if isinstance(value, dict): - for key, child in value.items(): - yield str(key) - yield from _all_strings(child) - elif isinstance(value, list): - for child in value: - yield from _all_strings(child) - elif isinstance(value, (str, int, float, bool)) or value is None: - yield str(value) - - -def _summary_text(data): - return " ".join(_all_strings(data)).lower() - - -def _assert_public_json_safe(public_text, *extra_absent): - lowered = public_text.lower() - for term in FORBIDDEN_PUBLIC_TERMS: - assert term not in lowered - for marker in extra_absent: - assert marker.lower() not in lowered - - -def _is_skip(data): - text = _summary_text(data) - return "skip" in text or "skipped" in text - - -def _is_failure_or_skip(data): - text = _summary_text(data) - return "fail" in text or "failed" in text or "error" in text or _is_skip(data) - - -def _check_records(data): - checks = data.get("checks", []) - if isinstance(checks, dict): - records = [] - for name, record in checks.items(): - if isinstance(record, dict): - records.append({"name": name, **record}) - else: - records.append({"name": name, "status": record}) - return records - assert isinstance(checks, list), "checks must be a list or object" - return [record for record in checks if isinstance(record, dict)] - - -def _check_names(data): - return {record.get("name") for record in _check_records(data)} - - -def _check_by_name(data, name): - for record in _check_records(data): - if record.get("name") == name: - return record - raise AssertionError(f"missing check {name}") - - -class ExplodingRunner: - def __init__(self): - self.calls = [] - - def __call__(self, *args, **kwargs): - self.calls.append((args, kwargs)) - raise AssertionError("runner must not be called") - - -class RecordingRunner: - def __init__(self): - self.calls = [] - - def __call__(self, *args, **kwargs): - argv = args[0] if args else kwargs.get("args") or kwargs.get("argv") - assert isinstance(argv, list), "Herdr commands must be argv lists, not shell strings" - assert all(isinstance(part, str) for part in argv) - assert kwargs.get("shell") is not True - - child_env = kwargs.get("env") - if child_env is None: - for value in args[1:]: - if isinstance(value, dict): - child_env = value - break - assert isinstance(child_env, dict), "runner must receive an explicit child environment" - - self.calls.append({"argv": list(argv), "env": dict(child_env), "kwargs": dict(kwargs)}) - return types.SimpleNamespace(returncode=0, stdout=self._stdout_for(argv), stderr="") - - def _stdout_for(self, argv): - joined = " ".join(part.lower() for part in argv) - if "status" in joined and "server" in joined: - return "status: running\n" - if argv[3:5] == ["agent", "start"]: - return json.dumps( - { - "id": "cli:agent:start", - "result": { - "type": "agent_started", - "agent": { - "name": "tendwire-smoke-address-probe", - "pane_id": "private-pane-id", - }, - }, - } - ) - if argv[3:5] == ["pane", "move"]: - return json.dumps({"id": "cli:pane:move", "result": {"type": "ok"}}) - if argv[3:5] == ["pane", "close"]: - return json.dumps({"id": "cli:pane:close", "result": {"type": "ok"}}) - if "workspace" in joined and "list" in joined: - return json.dumps({"status": "ok", "items": [{"label": "smoke-space"}], "count": 1}) - if "agent" in joined and "list" in joined: - return json.dumps({"status": "ok", "items": [{"name": "smoke-worker"}], "count": 1}) - if "status" in joined or "event" in joined: - return (OK_FIXTURES / "status_agent_status_changed.json").read_text() - if "send" in joined or "address" in joined: - return (OK_FIXTURES / "send_addressing.json").read_text() - return json.dumps({"status": "ok", "items": [{"name": "generic-worker"}], "count": 1}) - - -class StoppedSessionRunner(RecordingRunner): - def _stdout_for(self, argv): - joined = " ".join(part.lower() for part in argv) - if "status" in joined and "server" in joined: - return "status: not running\nsocket: /private/path\n" - raise AssertionError("stopped smoke scope must fail before observe/send commands") - - -class StoppedUnderscoreSessionRunner(RecordingRunner): - def _stdout_for(self, argv): - joined = " ".join(part.lower() for part in argv) - if "status" in joined and "server" in joined: - return json.dumps({"status": "not_running"}) - raise AssertionError("stopped smoke scope must fail before observe/send commands") - - -class NonzeroSendRunner(RecordingRunner): - def __call__(self, *args, **kwargs): - result = super().__call__(*args, **kwargs) - argv = self.calls[-1]["argv"] - if argv[3:5] == ["agent", "send"]: - result.returncode = 1 - result.stdout = "" - result.stderr = "private send failure" - return result - - -class ZeroAcceptedSendRunner(RecordingRunner): - def __call__(self, *args, **kwargs): - result = super().__call__(*args, **kwargs) - argv = self.calls[-1]["argv"] - if argv[3:5] == ["agent", "send"]: - result.returncode = 0 - result.stdout = json.dumps({"status": "ok", "accepted_count": 0}) - result.stderr = "" - return result - - -def test_no_live_opt_in_skips_without_subprocess_calls(smoke_module, monkeypatch, capsys): - runner = ExplodingRunner() - _patch_which(monkeypatch, smoke_module, "/unused/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, [], capsys, env={}, runner=runner) - - assert return_code in (0, None) - assert runner.calls == [] - assert _is_skip(data) - assert data.get("mode") != "live" - assert REQUIRED_CHECKS <= _check_names(data) - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -@pytest.mark.parametrize( - ("argv", "env", "expected_session", "expected_default", "expected_explicit", "expect_send"), - [ - (["--live"], {}, "tendwire-smoke", True, False, True), - (["--live"], {"HERDR_SESSION": "tendwire-smoke"}, "tendwire-smoke", False, True, True), - (["--live", "--session", "explicit-smoke"], {}, "explicit-smoke", False, True, False), - (["--live"], {"HERDR_SESSION": "caller-smoke"}, "caller-smoke", False, True, False), - ], -) -def test_live_session_selection_and_argv_construction( - smoke_module, - monkeypatch, - capsys, - argv, - env, - expected_session, - expected_default, - expected_explicit, - expect_send, -): - runner = RecordingRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, argv, capsys, env=env, runner=runner) - - assert return_code in (0, None) - assert runner.calls, "live opt-in must execute Herdr checks through the injected runner" - for call in runner.calls: - assert "HERDR_SESSION" not in call["env"] - assert isinstance(call["argv"], list) - assert call["kwargs"].get("shell") is not True - assert call["argv"][1:3] == ["--session", expected_session] - send_calls = [call for call in runner.calls if call["argv"][3:5] == ["agent", "send"]] - assert bool(send_calls) is expect_send - start_calls = [call for call in runner.calls if call["argv"][3:5] == ["agent", "start"]] - move_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "move"]] - close_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "close"]] - assert bool(start_calls) is expect_send - assert bool(move_calls) is expect_send - assert bool(close_calls) is expect_send - assert data.get("default_isolated_session") is expected_default - assert data.get("explicit_session") is expected_explicit - assert REQUIRED_CHECKS <= _check_names(data) - assert _check_by_name(data, "observe")["observed"] is True - create = _check_by_name(data, "create_attach") - if expect_send: - assert create["status"] == "ok" - assert create["detail"] == "live_created" - assert "limitation" not in create - assert _check_by_name(data, "pane_moved_binding_update")["detail"] == "live_moved" - assert _check_by_name(data, "close_exited")["detail"] == "live_closed" - else: - assert create["limitation"] == "caller_override" - assert _check_by_name(data, "pane_moved_binding_update")["limitation"] == "live_skipped_unreliable" - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_environment_variable_opts_into_live_mode(smoke_module, monkeypatch, capsys): - runner = RecordingRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main( - smoke_module, - [], - capsys, - env={"TENDWIRE_HERDR_LIVE_SMOKE": "1"}, - runner=runner, - ) - - assert return_code in (0, None) - assert runner.calls - for call in runner.calls: - assert "HERDR_SESSION" not in call["env"] - assert call["argv"][1:3] == ["--session", "tendwire-smoke"] - assert data.get("mode") == "live" - assert _check_by_name(data, "create_attach")["detail"] == "live_created" - assert _check_by_name(data, "send_addressing")["send_attempts"] == 1 - assert _check_by_name(data, "pane_moved_binding_update")["detail"] == "live_moved" - assert _check_by_name(data, "close_exited")["detail"] == "live_closed" - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_live_stopped_selected_scope_fails_before_observe_or_send(smoke_module, monkeypatch, capsys): - runner = StoppedSessionRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}, runner=runner) - - assert return_code not in (0, None) - assert data.get("ok") is False - assert data.get("status") == "unavailable" - assert len(runner.calls) == 1 - assert runner.calls[0]["argv"][1:5] == ["--session", "tendwire-smoke", "status", "server"] - observe = _check_by_name(data, "observe") - assert observe["ok"] is False - assert observe["workspace_count"] == 0 - assert observe["worker_count"] == 0 - send = _check_by_name(data, "send_addressing") - assert send["status"] == "skipped" - assert send["send_attempts"] == 0 - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_live_not_running_machine_status_fails_before_observe_or_send(smoke_module, monkeypatch, capsys): - runner = StoppedUnderscoreSessionRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}, runner=runner) - - assert return_code not in (0, None) - assert data.get("ok") is False - assert data.get("status") == "unavailable" - assert len(runner.calls) == 1 - assert runner.calls[0]["argv"][1:5] == ["--session", "tendwire-smoke", "status", "server"] - observe = _check_by_name(data, "observe") - assert observe["ok"] is False - assert observe["status"] == "unavailable" - send = _check_by_name(data, "send_addressing") - assert send["status"] == "skipped" - assert send["send_attempts"] == 0 - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_live_nonzero_send_is_not_ok(smoke_module, monkeypatch, capsys): - runner = NonzeroSendRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}, runner=runner) - - assert return_code not in (0, None) - assert data.get("ok") is False - assert data.get("status") == "failed" - send = _check_by_name(data, "send_addressing") - assert send["ok"] is False - assert send["status"] == "nonzero" - assert send["exit_code"] == 1 - assert send["accepted_count"] == 0 - send_calls = [call for call in runner.calls if call["argv"][3:5] == ["agent", "send"]] - move_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "move"]] - close_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "close"]] - assert len(send_calls) == 1 - assert len(move_calls) == 1 - assert len(close_calls) == 1 - assert send_calls[0]["argv"][1:3] == ["--session", "tendwire-smoke"] - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_live_zero_accepted_send_is_not_ok(smoke_module, monkeypatch, capsys): - runner = ZeroAcceptedSendRunner() - _patch_which(monkeypatch, smoke_module, "/fake/bin/herdr") - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}, runner=runner) - - assert return_code not in (0, None) - assert data.get("ok") is False - assert data.get("status") == "failed" - send = _check_by_name(data, "send_addressing") - assert send["ok"] is False - assert send["status"] == "zero_accepted" - assert send["exit_code"] == 0 - assert send["json_status"] == "valid" - assert send["accepted_count"] == 0 - send_calls = [call for call in runner.calls if call["argv"][3:5] == ["agent", "send"]] - move_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "move"]] - close_calls = [call for call in runner.calls if call["argv"][3:5] == ["pane", "close"]] - assert len(send_calls) == 1 - assert len(move_calls) == 1 - assert len(close_calls) == 1 - assert send_calls[0]["argv"][1:3] == ["--session", "tendwire-smoke"] - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_send_ok_envelope_counts_as_one_accepted(smoke_module): - accepted_count, json_status = smoke_module._send_accepted_count( - json.dumps({"id": "cli:agent:send", "result": {"type": "ok"}}) - ) - - assert accepted_count == 1 - assert json_status == "valid" - - -def test_fixture_replay_is_deterministic_and_public_safe(smoke_module, monkeypatch, capsys): - runner = ExplodingRunner() - _patch_which(monkeypatch, smoke_module, None) - - return_code, public_text, data = _run_main( - smoke_module, - ["--fixture-dir", str(OK_FIXTURES)], - capsys, - env={}, - runner=runner, - ) - - assert return_code in (0, None) - assert runner.calls == [] - assert data.get("ok") is True - assert data.get("mode") == "fixture" - assert REQUIRED_CHECKS <= _check_names(data) - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - assert _check_by_name(data, "create_attach")["created_count"] == 1 - assert _check_by_name(data, "observe")["worker_count"] == 2 - assert _check_by_name(data, "send_addressing")["send_attempts"] == 1 - assert _check_by_name(data, "target_validation")["rejected_send_attempts"] == 0 - status_check = _check_by_name(data, "status_agent_status_changed") - assert status_check["changed_count"] == 1 - assert status_check["event_count"] == 4 - assert status_check["accepted_count"] == 4 - assert status_check["exact_shape"] is True - assert status_check["idless"] is True - assert status_check["order_preserved"] is True - assert status_check["final_source_status"] == "working" - assert status_check["final_status"] == "active" - assert status_check["status_buckets"] == ["active"] - assert status_check["persistence_unchanged"] is True - assert status_check["repeat_effect_count"] == 0 - assert status_check["repeat_snapshot_delta"] == 0 - assert status_check["repeat_event_delta"] == 0 - assert status_check["repeat_attention_delta"] == 0 - assert status_check["repeat_queue_delta"] == 0 - assert _check_by_name(data, "pane_moved_binding_update")["preserved"] is True - assert _check_by_name(data, "close_exited")["exited_count"] == 1 - assert _check_by_name(data, "degraded_backend_preserves_workers")["preserved"] is True - event_check = _check_by_name(data, "event_subscription") - assert event_check.get("method") == "events.subscribe" - assert event_check.get("official_event_count") == len(smoke_module.OFFICIAL_EVENT_TYPES) - assert event_check.get("params_shape_ok") is True - assert event_check.get("legacy_event_count") == 0 - assert "subscriptions" not in event_check - assert "subscriptions" not in public_text - for raw_name in ( - "workspace.created", - "pane.created", - "pane.observed", - "workspace.observed", - "agent.status_changed", - "worktree.updated", - ): - assert raw_name not in public_text - - -def test_raw_status_fixture_replays_exact_idless_envelopes_without_duplicate_effects(smoke_module): - aggregate = json.loads((OK_FIXTURES / "status_agent_status_changed.json").read_text()) - replay_path = (OK_FIXTURES / aggregate["replay_fixture"]).resolve() - assert replay_path == RAW_STATUS_FIXTURE.resolve() - - envelopes = json.loads(replay_path.read_text()) - assert len(envelopes) == 4 - assert all(list(envelope) == ["event", "data"] for envelope in envelopes) - assert all(envelope["event"] == "pane.agent_status_changed" for envelope in envelopes) - assert [envelope["data"]["status"] for envelope in envelopes] == [ - "working", - "idle", - "working", - "working", - ] - assert envelopes[-1] == envelopes[-2] - assert smoke_module._contains_synthetic_event_identity(envelopes) is False - - replay = smoke_module._replay_fixture_status_events(replay_path) - assert replay["exact_shape"] is True - assert replay["idless"] is True - assert replay["source_statuses"] == ("working", "idle", "working", "working") - assert replay["accepted"] == (True, True, True, True) - assert replay["canonical_statuses"] == ("active", "idle", "active", "active") - assert replay["order_preserved"] is True - assert replay["final_source_status"] == "working" - assert replay["final_status"] == "active" - assert replay["changed_count"] == 1 - assert replay["repeat_row_deltas"] == { - "snapshots": 0, - "events": 0, - "attention_items": 0, - "connector_outbox": 0, - } - - -def test_deterministic_target_validation_sends_only_valid_case(smoke_module): - calls = [] - - check = smoke_module._deterministic_target_validation_check(calls.append) - - assert check["ok"] is True - assert check["valid_cases"] == 1 - assert check["invalid_cases"] == 2 - assert check["ambiguous_cases"] == 1 - assert check["send_attempts"] == 1 - assert check["rejected_send_attempts"] == 0 - assert calls == ["valid"] - - -def test_deterministic_event_backend_covers_move_close_exited_and_degraded(smoke_module, monkeypatch): - smoke_module._ensure_src_on_path() - from tendwire.backends.herdr_events import HerdrEventBackend - - recorded_envelopes = [] - original_queue = HerdrEventBackend.queue_event_envelope - - def recording_queue(self, envelope, *, flush=None): - recorded_envelopes.append(dict(envelope)) - return original_queue(self, envelope, flush=flush) - - monkeypatch.setattr(HerdrEventBackend, "queue_event_envelope", recording_queue) - checks = smoke_module._deterministic_event_backend_checks() - - assert checks["status_agent_status_changed"]["changed_count"] == 1 - assert checks["pane_moved_binding_update"]["preserved"] is True - assert checks["pane_moved_binding_update"]["worker_count_before"] == checks["pane_moved_binding_update"]["worker_count_after"] - assert checks["close_exited"]["closed_count"] == 1 - assert checks["close_exited"]["exited_count"] == 1 - assert checks["degraded_backend_preserves_workers"]["preserved"] is True - assert checks["degraded_backend_preserves_workers"]["worker_count_before"] == checks["degraded_backend_preserves_workers"]["worker_count_after"] - - assert [envelope["event"] for envelope in recorded_envelopes] == [ - "pane.agent_status_changed", - "pane.moved", - "pane.exited", - ] - assert all(set(envelope) == {"event", "data"} for envelope in recorded_envelopes) - assert smoke_module._contains_synthetic_event_identity(recorded_envelopes) is False - - -def test_event_subscription_builder_rejects_unknown_and_legacy_names(smoke_module): - params = smoke_module._event_subscription_params() - assert list(params) == ["subscriptions"] - assert len(params["subscriptions"]) == len(smoke_module.OFFICIAL_EVENT_TYPES) - assert smoke_module._event_subscription_params_shape_ok(params) is True - - for bad_name in ( - "", - " workspace.created ", - "pane.observed", - "workspace.observed", - "agent.status_changed", - "worktree.updated", - 123, - ): - names = list(smoke_module.OFFICIAL_EVENT_TYPES) - names[0] = bad_name - with pytest.raises(ValueError): - smoke_module._event_subscription_params(names) - - -def test_negative_fixture_rejects_recursive_forbidden_data(smoke_module, monkeypatch, capsys): - runner = ExplodingRunner() - _patch_which(monkeypatch, smoke_module, None) - - return_code, public_text, data = _run_main( - smoke_module, - ["--fixture-dir", str(NEGATIVE_FIXTURES)], - capsys, - env={}, - runner=runner, - ) - - assert return_code not in (0, None) or data.get("ok") is False - assert runner.calls == [] - assert _is_failure_or_skip(data) - assert any(word in _summary_text(data) for word in ("forbidden", "unsafe", "rejected")) - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) - - -def test_public_safety_rejects_forbidden_keys_values_and_allows_neutral_record_names(smoke_module): - for value in ( - {"pane_id": "hidden"}, - {"nested": [{"backend_target": "hidden"}]}, - {"detail": "socket:///tmp/forbidden.sock"}, - {"detail": "actual target value"}, - {"detail": "private fingerprint abc123"}, - {"stdout": "hidden"}, - ): - with pytest.raises(smoke_module.PublicSafetyError): - smoke_module.validate_public_summary(value) - - smoke_module.validate_public_summary( - { - "checks": [ - {"name": "target_validation", "status": "ok", "required": True, "ok": True}, - {"name": "pane_moved_binding_update", "status": "ok", "required": True, "ok": True}, - ] - } - ) - - -def test_missing_herdr_binary_reports_clear_skip_without_runner(smoke_module, monkeypatch, capsys): - _patch_which(monkeypatch, smoke_module, None) - - return_code, public_text, data = _run_main(smoke_module, ["--live"], capsys, env={}) - - assert return_code not in (0, None) or data.get("ok") is False - assert _is_failure_or_skip(data) - assert any(word in _summary_text(data) for word in ("missing", "not found", "unavailable", "requires")) - assert REQUIRED_CHECKS <= _check_names(data) - assert _check_by_name(data, "observe")["ok"] is False - assert _check_by_name(data, "target_validation")["ok"] is True - _assert_public_json_safe(public_text, *PRIVATE_MARKERS) diff --git a/tests/test_herdr_socket.py b/tests/test_herdr_socket.py index 038c866..d25dd06 100644 --- a/tests/test_herdr_socket.py +++ b/tests/test_herdr_socket.py @@ -1,637 +1,132 @@ -"""Tests for the inactive Herdr Unix socket client.""" - from __future__ import annotations import json -import os import socket -import subprocess -import sys import threading import time -from collections.abc import Callable -from pathlib import Path -from typing import Any import pytest -from tendwire.backends.herdr_protocol import ( - HerdrEnvelopeError, - HerdrErrorResponse, - HerdrMalformedLineError, - HerdrRequestIdMismatchError, - HERDR_EVENTS_SUBSCRIBE_METHOD, - HERDR_OFFICIAL_EVENT_NAMES, - build_events_subscribe_params, -) +from tendwire.backends.herdr_protocol import HerdrFrameTooLargeError, HerdrRequestIdMismatchError from tendwire.backends.herdr_socket import ( HerdrSocketClient, - HerdrSocketDisconnectedError, + HerdrSocketConnectionError, HerdrSocketTimeoutError, + _MAX_FRAME_BYTES, ) -class _Connection: - def __init__(self, conn: socket.socket, requests: list[dict[str, Any]]) -> None: - self.conn = conn - self.requests = requests - self._buffer = bytearray() - - def read_request(self) -> dict[str, Any]: - while b"\n" not in self._buffer: - chunk = self.conn.recv(4096) - if not chunk: - raise ConnectionError("client disconnected before request") - self._buffer.extend(chunk) - index = self._buffer.index(b"\n") - line = bytes(self._buffer[: index + 1]) - del self._buffer[: index + 1] - request = json.loads(line.decode("utf-8")) - self.requests.append(request) - return request - - def send_json(self, payload: dict[str, Any]) -> None: - self.conn.sendall(json.dumps(payload, separators=(",", ":")).encode("utf-8") + b"\n") - - def send_bytes(self, payload: bytes) -> None: - self.conn.sendall(payload) - - -class _FakeHerdrServer: - def __init__(self, tmp_path: Path, handler: Callable[[_Connection], None]) -> None: - self.path = tmp_path / f"herdr-{time.monotonic_ns()}.sock" - self.handler = handler - self.requests: list[dict[str, Any]] = [] - self.errors: list[BaseException] = [] - self._ready = threading.Event() - self._done = threading.Event() - self._listener: socket.socket | None = None - self._thread: threading.Thread | None = None - - def __enter__(self) -> "_FakeHerdrServer": - try: - self.path.unlink() - except FileNotFoundError: - pass - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(self.path)) - listener.listen(1) - listener.settimeout(0.2) - self._listener = listener - self._thread = threading.Thread(target=self._run, daemon=True) - self._thread.start() - if not self._ready.wait(1): - raise AssertionError("fake Herdr server did not start") - return self +def _serve(path, responses, requests) -> threading.Thread: + ready = threading.Event() - def __exit__(self, exc_type: object, exc: object, tb: object) -> None: - if self._listener is not None: - self._listener.close() - if self._thread is not None: - self._thread.join(timeout=1) - try: - self.path.unlink() - except FileNotFoundError: - pass - if exc_type is None and self.errors: - raise AssertionError(f"fake Herdr server failed: {self.errors!r}") - - def _run(self) -> None: - self._ready.set() - try: - assert self._listener is not None - conn, _addr = self._listener.accept() - with conn: - self.handler(_Connection(conn, self.requests)) - except OSError: - pass - except BaseException as exc: - self.errors.append(exc) - finally: - self._done.set() - - -class _FakeOneShotHerdrServer: - def __init__(self, tmp_path: Path, handler: Callable[[_Connection], None], *, connections: int) -> None: - self.path = tmp_path / f"herdr-oneshot-{time.monotonic_ns()}.sock" - self.handler = handler - self.connections = connections - self.requests: list[dict[str, Any]] = [] - self.errors: list[BaseException] = [] - self._ready = threading.Event() - self._listener: socket.socket | None = None - self._thread: threading.Thread | None = None - - def __enter__(self) -> "_FakeOneShotHerdrServer": - listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - listener.bind(str(self.path)) - listener.listen(self.connections) - listener.settimeout(0.5) - self._listener = listener - self._thread = threading.Thread(target=self._run, daemon=True) - self._thread.start() - if not self._ready.wait(1): - raise AssertionError("fake one-shot Herdr server did not start") - return self - - def __exit__(self, exc_type: object, exc: object, tb: object) -> None: - if self._listener is not None: - self._listener.close() - if self._thread is not None: - self._thread.join(timeout=1) - try: - self.path.unlink() - except FileNotFoundError: - pass - if exc_type is None and self.errors: - raise AssertionError(f"fake one-shot Herdr server failed: {self.errors!r}") - - def _run(self) -> None: - self._ready.set() - try: - assert self._listener is not None - for _index in range(self.connections): - conn, _addr = self._listener.accept() + def run() -> None: + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as server: + server.bind(str(path)) + server.listen() + ready.set() + for result in responses: + conn, _ = server.accept() with conn: - self.handler(_Connection(conn, self.requests)) - except OSError: - pass - except BaseException as exc: - self.errors.append(exc) - - -def _responding_handler(result: Any) -> Callable[[_Connection], None]: - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": result}) - - return handler - - -def test_client_successful_request_matches_id_and_returns_raw_result(tmp_path: Path) -> None: - result = {"items": [{"id": "w-1", "future": {"kept": True}}]} - with _FakeHerdrServer(tmp_path, _responding_handler(result)) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert client.request("workspace.list", {"scope": "all"}) == result - client.close() - - assert server.requests[0]["method"] == "workspace.list" - assert server.requests[0]["params"] == {"scope": "all"} - - -def test_client_pane_turns_wrapper_uses_additive_method(tmp_path: Path) -> None: - result = { - "type": "pane_turns", - "turns": { - "pane_id": "w1:p1", - "turn_epoch": 3, - "records": [], - "truncated": False, - }, - } - with _FakeHerdrServer(tmp_path, _responding_handler(result)) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert client.pane_turns( - {"pane_id": "w1:p1", "since": 4, "expected_epoch": 3} - ) == result - client.close() - - assert server.requests[0]["method"] == "pane.turns" - assert server.requests[0]["params"] == { - "pane_id": "w1:p1", - "since": 4, - "expected_epoch": 3, - } - assert isinstance(server.requests[0]["id"], str) - - -def test_client_reconnects_after_one_shot_response_connection_closes(tmp_path: Path) -> None: - results = { - "workspace.list": {"workspaces": []}, - "agent.list": {"agents": []}, - } - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": results[request["method"]]}) - - with _FakeOneShotHerdrServer(tmp_path, handler, connections=2) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert client.request("workspace.list") == {"workspaces": []} - assert client.request("agent.list") == {"agents": []} - client.close() - - assert [request["method"] for request in server.requests] == ["workspace.list", "agent.list"] - - -def test_client_timeout_waiting_for_response(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - time.sleep(0.15) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=0.05) - with pytest.raises(HerdrSocketTimeoutError): - client.request("workspace.list") - client.close() - - -def test_client_malformed_response_raises_protocol_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_bytes(b"{not json}\n") - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrMalformedLineError): - client.request("workspace.list") - client.close() - - -def test_client_malformed_envelope_shape_raises_protocol_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "not_result": True}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.request("workspace.list") - client.close() - - -def test_ordinary_request_rejects_uncorrelated_empty_id_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json( - { - "id": "", - "error": { - "code": "invalid_request", - "message": "uncorrelated ordinary request error", - }, - } - ) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.request("workspace.list") - client.close() - - -def test_ordinary_request_rejects_idless_unknown_variant_error( - tmp_path: Path, -) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json( - { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: unknown variant `pane.turns`, expected one of `pane.read`, `pane.list`, `agent.list`", - }, - } - ) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.pane_turns({"pane_id": "w1:p1", "since": 0}) - client.close() - - -def test_client_non_utf8_response_raises_protocol_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_bytes(b"\xff\n") - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrMalformedLineError): - client.request("workspace.list") - client.close() - - -def test_client_disconnect_before_response_raises(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrSocketDisconnectedError): - client.request("workspace.list") - client.close() - - -def test_client_handles_partial_reads_split_across_recv_boundaries(tmp_path: Path) -> None: - result = {"text": "hello"} - - def handler(conn: _Connection) -> None: - request = conn.read_request() - response = json.dumps({"id": request["id"], "result": result}).encode("utf-8") + b"\n" - conn.send_bytes(response[:7]) - time.sleep(0.01) - conn.send_bytes(response[7:]) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - assert client.request("pane.read", {"pane_id": "p-1"}) == result - client.close() - - -def test_client_error_response_raises_with_raw_error_payload(tmp_path: Path) -> None: - error = {"code": "not_found", "message": "missing", "extra": {"kept": True}} - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "error": error, "future": "ignored"}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrErrorResponse) as excinfo: - client.request("pane.get", {"pane_id": "missing"}) - client.close() - - assert excinfo.value.error == error - - -def test_client_response_id_mismatch_raises(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json({"id": "wrong-id", "result": {"ok": True}}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrRequestIdMismatchError): - client.request("workspace.list") - client.close() - - -def test_subscription_ack_events_and_stream_termination(tmp_path: Path) -> None: - events = [ - {"event": "pane.output", "payload": {"text": "one"}, "future": {"kept": 1}}, - {"event": "pane.output", "payload": {"text": "two"}, "future": {"kept": 2}}, - ] - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": {"subscribed": True, "raw": [1]}}) - for event in events: - conn.send_json({"id": request["id"], **event}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.subscribe("pane.watch", {"pane_id": "p-1"}) - - assert stream.ack == {"subscribed": True, "raw": [1]} - assert list(stream) == [ - {"id": server.requests[0]["id"], **events[0]}, - {"id": server.requests[0]["id"], **events[1]}, - ] - client.close() - - -def test_subscription_accepts_uncorrelated_idless_event_data_frames(tmp_path: Path) -> None: - events = [ - {"event": "pane.agent_status_changed", "data": {"status": "blocked"}}, - {"event": "pane.closed", "data": {"pane_id": "p-1"}}, + request = json.loads(conn.makefile("rb").readline()) + requests.append(request) + response_id = result.pop("response_id", request["id"]) + conn.sendall(json.dumps({"id": response_id, **result}).encode() + b"\n") + + thread = threading.Thread(target=run, daemon=True) + thread.start() + assert ready.wait(2) + return thread + + +def test_discovery_lists_use_one_request_per_connection(tmp_path) -> None: + path = tmp_path / "herdr.sock" + requests: list[dict] = [] + thread = _serve( + path, + [ + {"result": {"workspaces": []}}, + {"result": {"panes": []}}, + {"result": {"agents": []}}, + ], + requests, + ) + client = HerdrSocketClient(str(path), timeout=1) + assert client.workspace_list() == {"workspaces": []} + assert client._socket is None + assert client.pane_list() == {"panes": []} + assert client._socket is None + assert client.agent_list() == {"agents": []} + assert client._socket is None + thread.join(2) + assert [item["method"] for item in requests] == [ + "workspace.list", + "pane.list", + "agent.list", ] - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": {"subscribed": True}}) - for event in events: - conn.send_json(event) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.subscribe("events.subscribe", {"subscriptions": []}) - - assert stream.ack == {"subscribed": True} - assert list(stream) == events - client.close() - - -def test_subscription_buffers_idless_event_before_correlated_ack(tmp_path: Path) -> None: - event = { - "event": "pane.output_matched", - "data": {"pane_id": "p-1"}, - } - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json(event) - conn.send_json({"id": request["id"], "result": {"subscribed": True}}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.events_subscribe(["pane.output_matched"]) - - assert stream.ack == {"subscribed": True} - assert list(stream) == [event] - client.close() - - -def test_subscription_surfaces_herdr_075_empty_id_schema_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json( - { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: missing field pane_id", - }, - } - ) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrErrorResponse, match="missing field pane_id") as raised: - client.events_subscribe(["pane.agent_status_changed"]) - assert raised.value.uncorrelated is True - client.close() - - -@pytest.mark.parametrize( - "error", - [ - {"code": "permission_denied", "message": "invalid request: denied"}, - {"code": "invalid_request", "message": "subscription unavailable"}, - ], -) -def test_subscription_rejects_other_empty_id_errors( - tmp_path: Path, - error: dict[str, str], -) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json({"id": "", "error": error}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.events_subscribe(["pane.agent_status_changed"]) - client.close() - - -def test_ordinary_request_rejects_herdr_075_empty_id_error(tmp_path: Path) -> None: - def handler(conn: _Connection) -> None: - conn.read_request() - conn.send_json( - { - "id": "", - "error": { - "code": "invalid_request", - "message": "invalid request: ordinary request error", - }, - } - ) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - with pytest.raises(HerdrEnvelopeError): - client.pane_list() - client.close() - - -def test_events_subscribe_wrapper_sends_official_method_and_params(tmp_path: Path) -> None: - event_names = ("workspace.created", "pane.agent_status_changed") - - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": {"subscribed": True}}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.events_subscribe(event_names) - client.close() - - assert stream.ack == {"subscribed": True} - assert server.requests[0]["method"] == HERDR_EVENTS_SUBSCRIBE_METHOD - assert server.requests[0]["params"] == build_events_subscribe_params(event_names) - - -@pytest.mark.parametrize("event_name", HERDR_OFFICIAL_EVENT_NAMES) -def test_events_subscribe_wrapper_accepts_each_official_event_name( - tmp_path: Path, - event_name: str, -) -> None: - def handler(conn: _Connection) -> None: - request = conn.read_request() - conn.send_json({"id": request["id"], "result": {"subscribed": event_name}}) - - with _FakeHerdrServer(tmp_path, handler) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - stream = client.events_subscribe([event_name]) - client.close() - - assert stream.ack == {"subscribed": event_name} - assert server.requests[0]["method"] == HERDR_EVENTS_SUBSCRIBE_METHOD - assert server.requests[0]["params"] == {"subscriptions": [{"type": event_name}]} - -def test_context_manager_and_close_are_idempotent(tmp_path: Path) -> None: - with _FakeHerdrServer(tmp_path, _responding_handler({"ok": True})) as server: - with HerdrSocketClient(str(server.path), timeout=1) as client: - assert client.request("agent.get", {"agent_id": "a-1"}) == {"ok": True} - client.close() - client.close() - - -@pytest.mark.parametrize( - ("wrapper_name", "method", "params", "result"), - [ - ("workspace_list", "workspace.list", {"all": True}, {"workspaces": [{"id": "w"}]}), - ("tab_list", "tab.list", {"workspace_id": "w"}, {"tabs": [{"id": "t"}]}), - ("pane_list", "pane.list", {"tab_id": "t"}, {"panes": [{"id": "p"}]}), - ("agent_list", "agent.list", {"workspace_id": "w"}, {"agents": [{"id": "a"}]}), - ("pane_get", "pane.get", {"pane_id": "p"}, {"id": "p", "raw": {"kept": True}}), - ("agent_get", "agent.get", {"agent_id": "a"}, {"id": "a", "raw": {"kept": True}}), - ("pane_read", "pane.read", {"pane_id": "p", "limit": 20}, {"text": "raw"}), - ], -) -def test_allowed_read_wrappers_send_exact_method_and_params( - tmp_path: Path, - wrapper_name: str, - method: str, - params: dict[str, Any], - result: dict[str, Any], -) -> None: - with _FakeHerdrServer(tmp_path, _responding_handler(result)) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert getattr(client, wrapper_name)(params) == result - client.close() - - assert server.requests[0]["method"] == method - assert server.requests[0]["params"] == params - - -def test_agent_send_is_the_only_exposed_mutate_wrapper_and_shape_is_exact(tmp_path: Path) -> None: - result = {"accepted": True, "opaque": {"server": "kept"}} - params = {"agent_id": "a-1", "text": "hello"} - with _FakeHerdrServer(tmp_path, _responding_handler(result)) as server: - client = HerdrSocketClient(str(server.path), timeout=1) - - assert client.agent_send(params) == result - client.close() - - assert server.requests[0]["method"] == "agent.send" - assert server.requests[0]["params"] == params - - excluded_public_api = { - "pane_send_text", - "pane_send_keys", - "pane_run", - "send_text", - "send_keys", - "run", - "shell", - "raw_terminal_control", - "source_mode", - "connector_polling", - "poll_connectors", - "event_backend_replacement", - } - for name in excluded_public_api: - assert not hasattr(client, name), name - - -def test_cli_import_does_not_load_socket_client_by_default() -> None: - code = """ -import sys -before = set(sys.modules) -import tendwire.cli -loaded = set(sys.modules) - before -for name in sorted(loaded): - if name in {"tendwire.backends.herdr_socket", "tendwire.backends.herdr_protocol"}: - print(name) -""" - env = os.environ.copy() - env["PYTHONPATH"] = os.path.join(os.path.dirname(__file__), "..", "src") - result = subprocess.run( - [sys.executable, "-c", code], - capture_output=True, - text=True, - check=False, - env=env, +def test_acp_methods_use_frozen_socket_shapes_without_connection_reuse(tmp_path) -> None: + path = tmp_path / "herdr.sock" + requests: list[dict] = [] + thread = _serve( + path, + [ + {"result": {"type": "agent_acp_status"}}, + {"result": {"type": "agent_acp_endpoint"}}, + ], + requests, ) - assert result.returncode == 0 - assert result.stdout == "" + client = HerdrSocketClient(str(path), timeout=1) + assert client.agent_acp_status("term") == {"type": "agent_acp_status"} + assert client._socket is None + assert client.agent_acp_endpoint("term") == {"type": "agent_acp_endpoint"} + assert client._socket is None + thread.join(2) + assert [(item["method"], item["params"]) for item in requests] == [ + ("agent.acp_status", {"target": "term"}), + ("agent.acp_endpoint", {"target": "term"}), + ] -def test_existing_production_backend_files_do_not_import_socket_client() -> None: - root = Path(__file__).resolve().parents[1] - for relative in ( - "src/tendwire/cli.py", - "src/tendwire/backends/herdr_cli.py", - "src/tendwire/backends/herdr_command.py", - ): - text = (root / relative).read_text(encoding="utf-8") - assert "herdr_socket" not in text +def test_response_id_mismatch_fails_closed(tmp_path) -> None: + path = tmp_path / "herdr.sock" + thread = _serve(path, [{"response_id": "wrong", "result": {}}], []) + with pytest.raises(HerdrRequestIdMismatchError): + HerdrSocketClient(str(path), timeout=1).agent_acp_status("term") + thread.join(2) + + +def test_connection_and_timeout_fail_closed(tmp_path) -> None: + with pytest.raises(HerdrSocketConnectionError): + HerdrSocketClient(str(tmp_path / "missing.sock"), timeout=0.01).pane_list() + + class TimedOutSocket: + def settimeout(self, _value): return None + def recv(self, _size): raise socket.timeout + def shutdown(self, _how): return None + def close(self): return None + + client = HerdrSocketClient(str(tmp_path / "unused.sock"), timeout=1) + client._socket = TimedOutSocket() + with pytest.raises(HerdrSocketTimeoutError): + client._read_line(deadline=time.monotonic() + 1) + + +def test_request_and_response_frames_are_bounded(tmp_path) -> None: + class UnusedSocket: + def settimeout(self, _value): return None + def sendall(self, _payload): raise AssertionError("oversize request must not be sent") + def recv(self, _size): return b"x" + def shutdown(self, _how): return None + def close(self): return None + + outbound = HerdrSocketClient(str(tmp_path / "unused.sock"), timeout=1) + outbound._socket = UnusedSocket() + with pytest.raises(HerdrFrameTooLargeError): + outbound.request("pane.list", {"padding": "x" * _MAX_FRAME_BYTES}) + + inbound = HerdrSocketClient(str(tmp_path / "unused.sock"), timeout=1) + inbound._socket = UnusedSocket() + inbound._buffer.extend(b"x" * _MAX_FRAME_BYTES) + with pytest.raises(HerdrFrameTooLargeError): + inbound._read_line(deadline=time.monotonic() + 1) diff --git a/tests/test_herdr_turns.py b/tests/test_herdr_turns.py deleted file mode 100644 index cbceb2f..0000000 --- a/tests/test_herdr_turns.py +++ /dev/null @@ -1,2320 +0,0 @@ -"""Tests for private Herdr turn ingestion into public Tendwire turns.""" - -from __future__ import annotations - -import json -import multiprocessing -import sqlite3 -import subprocess -import threading -import pytest -from pathlib import Path -from typing import Any - -from tendwire.backends import herdr_turns -from tendwire.backends.herdr_turns import refresh_structured_turn_content -from tendwire.config import Config -from tendwire.core.models import WorkerBinding, sanitize_canonical_turn_text -from tendwire.core.projector import project_from_raw -from tendwire.core.turns import ( - TURN_CONTENT_PAGE_MAX_UTF8_BYTES, - TURN_STREAM_TEXT_MAX_CHARS, - is_internal_automation_turn_payload, -) -from tendwire.store import sqlite as store_sqlite -from tendwire.store.sqlite import ( - init_store, - save_snapshot, - turns_payload_from_store, - upsert_worker_bindings, -) - - -def _read_test_ipc_request(channel): - return json.loads( - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._CODEX_STATE_IPC_MAX_BYTES, - ).decode("utf-8") - ) - - -def _send_test_ipc_response(channel, response) -> None: - herdr_turns._blocking_send_frame( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - ) - - -def _failed_isolated_turn_child(channel) -> None: - try: - request = _read_test_ipc_request(channel) - _send_test_ipc_response( - channel, - { - "protocol": 1, - "nonce": request["nonce"], - "disposition": "failed", - "content": None, - "parser_state": None, - "bytes_read": 0, - }, - ) - finally: - channel.close() - - -def _invalid_isolated_turn_child(channel) -> None: - try: - _read_test_ipc_request(channel) - _send_test_ipc_response( - channel, - { - "protocol": 1, - "nonce": "not-the-request-nonce", - "disposition": "ok", - "content": None, - "parser_state": None, - "bytes_read": 0, - }, - ) - finally: - channel.close() - - -def _blocked_isolated_turn_child(channel) -> None: - _read_test_ipc_request(channel) - threading.Event().wait(30) - - -def _growing_isolated_turn_child(channel) -> None: - try: - request = _read_test_ipc_request(channel) - large_final = "grew-during-read-" + ("g" * (2 * 1024 * 1024)) - with open(request["target_value"], "a", encoding="utf-8") as handle: - handle.write( - "\n" - + json.dumps( - { - "type": "message", - "id": "grown-final", - "message": { - "role": "assistant", - "stopReason": "stop", - "content": [{"type": "text", "text": large_final}], - }, - }, - separators=(",", ":"), - ) - ) - parser_state = request["parser_state"] - assert parser_state["source"] == "omp" - parsed = herdr_turns._read_omp_session_turn_with_state( - request["target_value"], - herdr_turns._deserialize_omp_state(parser_state["state"]), - ) - content, checkpoint, bytes_read = parsed - response = { - "protocol": 1, - "nonce": request["nonce"], - "disposition": "ok", - "content": content, - "parser_state": { - "source": "omp", - "state": herdr_turns._serialize_omp_state(checkpoint), - }, - "bytes_read": bytes_read, - } - herdr_turns._blocking_send_streamed_omp_response( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - request["nonce"], - ) - finally: - channel.close() - - -def test_refresh_structured_turn_content_uses_private_binding_without_public_leak( - tmp_path: Path, - monkeypatch, -) -> None: - db_path = tmp_path / "turns.db" - config = Config( - host_id="turn-host", - db_path=db_path, - herdr_bin="herdr_turn_adapter.py", - herdr_timeout_seconds=2, - ) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "codex", "status": "active", "space_id": "space-1"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="pane_id", - turn_target_value="pane-private", - sendable=True, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-binding", - ) - ], - ) - calls: list[tuple[list[str], dict[str, Any]]] = [] - - def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: - calls.append((args, kwargs)) - return subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps( - { - "result": { - "turn": { - "available": True, - "source_turn_id": "private-binding-source", - "user_text": "Why is Telegram showing lifecycle status?", - "assistant_final_text": "Use Tendwire turn text, not pane_id pane-private.", - "assistant_stream_text": "Checking source mode...", - "complete": True, - "has_open_turn": False, - } - } - } - ), - stderr="", - ) - - monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) - - result = refresh_structured_turn_content(config) - payload = turns_payload_from_store(db_path, config.host_id, snapshot=snapshot) - - assert result["attempted"] == 1 - assert result["updated"] == 1 - assert calls[0][0] == [ - "herdr_turn_adapter.py", - "pane", - "turn", - "pane-private", - "--last", - "--format", - "json", - ] - assert calls[0][1]["timeout"] == 2 - turn = payload["turns"][0] - assert turn["user_text"] == "Why is Telegram showing lifecycle status?" - assert "Use Tendwire turn text" in turn["assistant_final_text"] - public_json = json.dumps(payload) - assert "pane-private" not in public_json - assert "agent-private" not in public_json - assert turn["complete"] is True - assert turn["has_open_turn"] is False - - -def test_refresh_structured_turn_content_reads_codex_session_jsonl( - tmp_path: Path, - monkeypatch, -) -> None: - db_path = tmp_path / "turns.db" - codex_home = tmp_path / "codex-home" - session_id = "019f2307-092b-7810-8323-418d7c55bd26" - session_file = ( - codex_home - / "sessions" - / "2026" - / "07" - / "03" - / f"rollout-2026-07-03T00-00-00-{session_id}.jsonl" - ) - session_file.parent.mkdir(parents=True) - turn_id = "turn-live" - lines = [ - { - "timestamp": "2026-07-03T08:00:00Z", - "type": "event_msg", - "payload": {"type": "task_started", "turn_id": turn_id}, - }, - { - "timestamp": "2026-07-03T08:00:01Z", - "type": "response_item", - "payload": { - "type": "message", - "role": "user", - "content": [{"type": "input_text", "text": "Please fix the source feed"}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - }, - }, - { - "timestamp": "2026-07-03T08:00:02Z", - "type": "response_item", - "payload": { - "type": "message", - "role": "assistant", - "phase": "commentary", - "content": [{"type": "output_text", "text": "Checking source state."}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - }, - }, - { - "timestamp": "2026-07-03T08:00:03Z", - "type": "event_msg", - "payload": { - "type": "task_complete", - "turn_id": turn_id, - "last_agent_message": "Fixed the source feed.", - }, - }, - ] - session_file.write_text( - "\n".join(json.dumps(item) for item in lines) + "\n", - encoding="utf-8", - ) - monkeypatch.setenv("CODEX_HOME", str(codex_home)) - monkeypatch.setattr( - herdr_turns.subprocess, - "run", - lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("pane turn fallback should not run")), - ) - - config = Config(host_id="turn-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "codex", "status": "active", "space_id": "space-1"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value="term-private", - turn_target_kind="codex_session_id", - turn_target_value=session_id, - sendable=True, - observed_at="2026-07-03T08:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-binding", - ) - ], - ) - - result = refresh_structured_turn_content(config) - payload = turns_payload_from_store(db_path, config.host_id, snapshot=snapshot) - - assert result == {"ok": True, "status": "ok", "updated": 1, "attempted": 1} - turn = payload["turns"][0] - assert turn["user_text"] == "Please fix the source feed" - assert turn["assistant_final_text"] == "Fixed the source feed." - assert turn["assistant_stream_text"] is None - assert turn["complete"] is True - assert turn["has_open_turn"] is False - public_json = json.dumps(payload) - assert session_id not in public_json - assert "term-private" not in public_json - - -def test_refresh_structured_turn_content_skips_codex_automation_protocol_turn( - tmp_path: Path, - monkeypatch, -) -> None: - db_path = tmp_path / "turns.db" - codex_home = tmp_path / "codex-home" - session_id = "019f31a8-57cf-7353-b4f0-c25e523267af" - session_file = ( - codex_home - / "sessions" - / "2026" - / "07" - / "05" - / f"rollout-2026-07-05T00-00-00-{session_id}.jsonl" - ) - session_file.parent.mkdir(parents=True) - turn_id = "automation-turn" - lines = [ - { - "timestamp": "2026-07-05T08:00:00Z", - "type": "event_msg", - "payload": {"type": "task_started", "turn_id": turn_id}, - }, - { - "timestamp": "2026-07-05T08:00:01Z", - "type": "response_item", - "payload": { - "type": "message", - "role": "user", - "content": [ - { - "type": "input_text", - "text": "Acme job\n\nTemplate: review-lead\nTemplate instructions:", - } - ], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - }, - }, - { - "timestamp": "2026-07-05T08:00:02Z", - "type": "event_msg", - "payload": { - "type": "task_complete", - "turn_id": turn_id, - "last_agent_message": '{"acme_result":{"decision":"approved","summary":"internal job result"}}', - }, - }, - ] - session_file.write_text( - "\n".join(json.dumps(item) for item in lines) + "\n", - encoding="utf-8", - ) - monkeypatch.setenv("CODEX_HOME", str(codex_home)) - - config = Config(host_id="turn-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "codex", "status": "active", "space_id": "space-1"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value="term-private", - turn_target_kind="codex_session_id", - turn_target_value=session_id, - sendable=True, - observed_at="2026-07-05T08:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-binding", - ) - ], - ) - - result = refresh_structured_turn_content(config) - payload = turns_payload_from_store(db_path, config.host_id, snapshot=snapshot) - public_json = json.dumps(payload) - - assert result == {"ok": True, "status": "ok", "updated": 0, "attempted": 1} - assert "Acme job" not in public_json - assert "acme_result" not in public_json - - -def test_turns_payload_from_store_quarantines_existing_automation_protocol_rows( - tmp_path: Path, -) -> None: - db_path = tmp_path / "turns.db" - host_id = "turn-host" - init_store(db_path) - with sqlite3.connect(db_path) as conn: - rows = [ - ( - host_id, - "bad-turn", - "worker-1", - "active", - "task", - "2026-07-05T08:00:00+00:00", - "bad-fp", - "snap-fp", - "2026-07-05T08:00:00+00:00", - json.dumps( - { - "host_id": host_id, - "worker_id": "worker-1", - "status": "active", - "kind": "task", - "user_text": "Acme job\n\nTemplate: review-lead\nTemplate instructions:", - "assistant_final_text": '{"acme_result":{"decision":"approved","summary":"internal"}}', - } - ), - 1, - ), - ( - host_id, - "good-turn", - "worker-1", - "active", - "task", - "2026-07-05T08:01:00+00:00", - "good-fp", - "snap-fp", - "2026-07-05T08:01:00+00:00", - json.dumps( - { - "host_id": host_id, - "worker_id": "worker-1", - "status": "active", - "kind": "task", - "user_text": "Please review the issue", - "assistant_final_text": "Normal answer.", - } - ), - 2, - ), - ] - conn.executemany( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, list_sequence - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - rows, - ) - - payload = turns_payload_from_store(db_path, host_id) - public_json = json.dumps(payload) - - assert len(payload["turns"]) == 1 - assert payload["turns"][0]["assistant_final_text"] == "Normal answer." - assert "Acme job" not in public_json - assert "acme_result" not in public_json - - -def test_internal_user_text_detects_local_command_artifacts() -> None: - assert herdr_turns._is_internal_user_text("Caveat: ...") - assert herdr_turns._is_internal_user_text(" /model") - assert herdr_turns._is_internal_user_text("Set model") - assert herdr_turns._is_internal_user_text("context") - assert herdr_turns._is_internal_user_text("done") - assert not herdr_turns._is_internal_user_text("another test") - - -def test_internal_turn_filter_detects_automation_protocol_without_blocking_discussion() -> None: - assert herdr_turns._is_internal_user_text( - "Acme job\n\nTemplate: review-lead\nTemplate instructions:" - ) - assert herdr_turns._is_internal_user_text( - "Your previous response did not contain a valid acme_result JSON object.\n" - "Validation errors (fix every line):" - ) - assert is_internal_automation_turn_payload( - {"assistant_final_text": '{"acme_result":{"decision":"approved","summary":"internal job result"}}'} - ) - assert is_internal_automation_turn_payload( - {"assistant_final_text": '```json\n{"acme_result":{"decision":"blocked"}}\n```'} - ) - assert not herdr_turns._is_internal_user_text("Can you investigate why automation job responses leaked?") - assert not is_internal_automation_turn_payload( - {"assistant_final_text": "I found a leaked automation_result row in the Tendwire DB."} - ) - assert not is_internal_automation_turn_payload( - { - "user_text": "Please return a JSON status object.", - "assistant_final_text": '{"acme_result":{"status":"ok"}}', - } - ) - - -def test_read_private_turn_skips_local_command_turns(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "user_text": "Caveat: The messages below were generated by the user while running local commands.", - "has_open_turn": True, - "complete": False, - } - } - } - - def fake_run(args, **kwargs): - return subprocess.CompletedProcess(args=args, returncode=0, stdout=json.dumps(payload), stderr="") - - monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) - assert herdr_turns._read_private_turn(config, "pane-1") is None - - -def test_read_private_turn_skips_automation_protocol_turns(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "user_text": "Acme job\n\nTemplate: review-lead\nTemplate instructions:", - "assistant_final_text": '{"acme_result":{"decision":"approved"}}', - "has_open_turn": False, - "complete": True, - "source_turn_id": "automation-turn", - } - } - } - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - assert herdr_turns._read_private_turn(config, "pane-1") is None - - -def test_read_private_turn_skips_promptless_status_finals(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "assistant_final_text": "Initial state (review in progress; gate phase not yet reached). Waiting quietly for the gate verdict or merge. Standing by.", - "complete": True, - "has_open_turn": False, - "source_turn_id": "status-only-turn", - "model": "claude-opus-4-8", - } - } - } - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - assert herdr_turns._read_private_turn(config, "pane-1") is None - - -def test_read_private_turn_keeps_prompted_status_like_final(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "user_text": "What is the current state?", - "assistant_final_text": "Current state: the review is complete.", - "complete": True, - "has_open_turn": False, - "source_turn_id": "prompted-turn", - } - } - } - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - content = herdr_turns._read_private_turn(config, "pane-1") - assert content is not None - assert content["user_text"] == "What is the current state?" - assert content["assistant_final_text"] == "Current state: the review is complete." - - -def _run_returning(payload): - def fake_run(args, **kwargs): - return subprocess.CompletedProcess(args=args, returncode=0, stdout=json.dumps(payload), stderr="") - return fake_run - - -def test_read_private_turn_emits_open_turn_from_open_fields(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - # top level is the PREVIOUS completed turn - "complete": True, - "has_open_turn": True, - "user_text": "previous prompt", - "assistant_final_text": "previous answer", - "source_turn_id": "prompt-prev", - # the in-progress turn is carried in open_* fields - "open_turn_id": "prompt-open", - "open_user_text": "current prompt", - "assistant_stream_text": "thinking live...", - } - } - } - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - content = herdr_turns._read_private_turn(config, "pane-1") - assert content is not None - assert content["user_text"] == "current prompt" - assert content["assistant_stream_text"] == "thinking live..." - assert content["assistant_final_text"] is None - assert content["complete"] is False - assert content["has_open_turn"] is True - # keyed by the OPEN turn's stable prompt id, not the completed one - assert content["source_turn_id"] == "prompt-open" - - -def test_open_turn_and_its_completion_share_source_turn_id(monkeypatch) -> None: - """The open turn (prompt-open) and its later completion must share the id so - a working card edits into the final instead of duplicating.""" - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - open_payload = { - "result": { - "turn": { - "available": True, - "complete": True, - "has_open_turn": True, - "user_text": "older", - "assistant_final_text": "older answer", - "source_turn_id": "prompt-older", - "open_turn_id": "prompt-X", - "open_user_text": "the question", - "assistant_stream_text": "working...", - } - } - } - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(open_payload)) - open_content = herdr_turns._read_private_turn(config, "pane-1") - assert open_content["source_turn_id"] == "prompt-X" - assert open_content["complete"] is False - - # Now the same turn completes (no open fields; it is the last completed one). - done_payload = { - "result": { - "turn": { - "available": True, - "complete": True, - "has_open_turn": False, - "user_text": "the question", - "assistant_final_text": "the answer", - "source_turn_id": "prompt-X", - "turn_id": "assistant-uuid-differs", - } - } - } - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(done_payload)) - done_content = herdr_turns._read_private_turn(config, "pane-1") - assert done_content["source_turn_id"] == "prompt-X" # same id, not the assistant uuid - assert done_content["complete"] is True - assert done_content["assistant_final_text"] == "the answer" - - -def test_read_private_turn_prefers_source_turn_id_over_turn_id(monkeypatch) -> None: - config = Config(host_id="turn-host", herdr_bin="herdr", herdr_timeout_seconds=2) - payload = { - "result": { - "turn": { - "available": True, - "complete": True, - "has_open_turn": False, - "user_text": "q", - "assistant_final_text": "a", - "source_turn_id": "stable-prompt", - "turn_id": "assistant-uuid", - } - } - } - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - content = herdr_turns._read_private_turn(config, "pane-1") - assert content["source_turn_id"] == "stable-prompt" - - -def test_omp_agent_session_id_also_maps_to_omp_turn_target() -> None: - from tendwire.backends.herdr_cli import _turn_target_from_item - - item = { - "agent": "omp", - "pane_id": "wX:p1", - "agent_session": {"agent": "omp", "kind": "id", "value": "019f-omp-session"}, - } - assert _turn_target_from_item(item) == ("omp_session_path", "019f-omp-session") - - -def _write_omp_session(tmp_path, lines): - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - herdr_turns._OMP_SESSION_CACHE_BINDING_GENERATIONS.clear() - root = tmp_path / "omp-sessions" - session_dir = root / "-demoapp" - session_dir.mkdir(parents=True) - path = session_dir / "2026-07-05T00-00-00-000Z_session.jsonl" - path.write_text("\n".join(json.dumps(line) for line in lines), encoding="utf-8") - return root, path - - -def _write_valid_git_head(git_dir: Path) -> None: - git_dir.mkdir(parents=True) - (git_dir / "HEAD").write_text("ref: refs/heads/main\n", encoding="ascii") - - -def _mark_git_repository(path: Path) -> None: - _write_valid_git_head(path / ".git") - - -def _omp_msg(entry_id, role, text, stop=None, attribution=None): - message = {"role": role, "content": [{"type": "text", "text": text}]} - if stop: - message["stopReason"] = stop - if attribution: - message["attribution"] = attribution - return {"type": "message", "id": entry_id, "message": message} - - -def _omp_read_msg(entry_id, path_value): - return { - "type": "message", - "id": entry_id, - "message": { - "role": "assistant", - "stopReason": "toolUse", - "content": [ - { - "type": "toolCall", - "id": f"{entry_id}-tool", - "name": "read", - "arguments": {"path": path_value}, - } - ], - }, - } - - -def test_read_omp_session_open_then_complete_turn(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session( - tmp_path, - [ - {"type": "session", "id": "s1"}, - _omp_msg("u1", "user", "please fix the bug", attribution="user"), - _omp_msg("a1", "assistant", "looking at the code", stop="toolUse"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - content = herdr_turns._read_omp_session_turn(str(path)) - assert content["user_text"] == "please fix the bug" - assert content["assistant_stream_text"] == "looking at the code" - assert content["complete"] is False - assert content["has_open_turn"] is True - assert content["source_turn_id"] == "u1" - - # Same turn completes: same source id, final text, stream cleared. - path.write_text( - path.read_text(encoding="utf-8") - + "\n" - + json.dumps(_omp_msg("a2", "assistant", "fixed and pushed", stop="stop")), - encoding="utf-8", - ) - done = herdr_turns._read_omp_session_turn(str(path)) - assert done["source_turn_id"] == "u1" - assert done["assistant_final_text"] == "fixed and pushed" - assert done["complete"] is True - assert done["assistant_stream_text"] is None - - -def test_read_omp_session_rejects_paths_outside_root(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session(tmp_path, [_omp_msg("u1", "user", "hi", attribution="user")]) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(tmp_path / "elsewhere")) - assert herdr_turns._read_omp_session_turn(str(path)) is None - - -def test_omp_agent_session_path_maps_to_omp_turn_target() -> None: - from tendwire.backends.herdr_cli import _turn_target_from_item - - item = { - "agent": "omp", - "pane_id": "wX:p1", - "agent_session": {"agent": "omp", "kind": "path", "value": "/home/user/.omp/agent/sessions/-x/a.jsonl"}, - } - assert _turn_target_from_item(item) == ("omp_session_path", "/home/user/.omp/agent/sessions/-x/a.jsonl") - - -def test_omp_open_turn_streams_thinking_headlines(tmp_path, monkeypatch) -> None: - def thinking(entry_id, text): - return {"type": "message", "id": entry_id, "message": {"role": "assistant", "stopReason": "toolUse", "content": [{"type": "thinking", "thinking": text}, {"type": "toolCall", "id": "c1", "name": "bash"}]}} - - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("u1", "user", "add the feature", attribution="user"), - thinking("a1", "**Reading the goal doc**\n\nlong reasoning body..."), - thinking("a2", "checking the branch state first\nmore detail"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - content = herdr_turns._read_omp_session_turn(str(path)) - assert content["has_open_turn"] is True - assert content["assistant_stream_text"] == ( - "Reading the goal doc\n\n" - "step 1 · run command\n\n" - "checking the branch state first\n\n" - "step 2 · run command" - ) - - -def test_omp_cold_start_scans_back_until_current_user_prompt(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("u1", "user", "large turn please", attribution="user"), - _omp_msg("a1", "assistant", "x" * 512, stop="toolUse"), - _omp_msg("a2", "assistant", "finished large turn", stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr(herdr_turns, "_OMP_TAIL_BYTES", 80) - - content = herdr_turns._read_omp_session_turn(str(path)) - - assert content["source_turn_id"] == "u1" - assert content["user_text"] == "large turn please" - assert content["assistant_final_text"] == "finished large turn" - assert content["complete"] is True - - -def test_omp_cold_start_ignores_internal_user_lines_when_finding_prompt(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("u1", "user", "real prompt", attribution="user"), - _omp_msg("a1", "assistant", "x" * 512, stop="toolUse"), - _omp_msg("internal", "user", "\nignore me", attribution="user"), - _omp_msg("a2", "assistant", "still answering real prompt", stop="toolUse"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr(herdr_turns, "_OMP_TAIL_BYTES", 80) - - content = herdr_turns._read_omp_session_turn(str(path)) - - assert content["source_turn_id"] == "u1" - assert content["user_text"] == "real prompt" - assert "still answering real prompt" in content["assistant_stream_text"] - assert "" not in content["assistant_stream_text"] - - -def test_omp_incremental_cache_keeps_turn_state_when_prompt_leaves_tail(tmp_path, monkeypatch) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("u1", "user", "keep streaming this", attribution="user"), - _omp_msg("a1", "assistant", "started", stop="toolUse"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr(herdr_turns, "_OMP_TAIL_BYTES", 80) - - first = herdr_turns._read_omp_session_turn(str(path)) - assert first["source_turn_id"] == "u1" - assert first["assistant_stream_text"] == "started" - - def fail_cold_start(*_args): - raise AssertionError("incremental read should not cold-scan a cached growing file") - - monkeypatch.setattr(herdr_turns, "_read_omp_state_from_recent", fail_cold_start) - path.write_text( - path.read_text(encoding="utf-8") - + "\n" - + json.dumps( - { - "type": "message", - "id": "a2", - "message": { - "role": "assistant", - "stopReason": "toolUse", - "content": [ - {"type": "thinking", "thinking": "**Still working**\n\n" + ("x" * 512)}, - {"type": "toolCall", "id": "c1", "name": "bash", "arguments": {"command": "git status"}}, - ], - }, - } - ), - encoding="utf-8", - ) - - second = herdr_turns._read_omp_session_turn(str(path)) - - assert second["source_turn_id"] == "u1" - assert second["user_text"] == "keep streaming this" - assert second["complete"] is False - assert "Still working" in second["assistant_stream_text"] - assert "step 1 · git status" in second["assistant_stream_text"] - - -def test_isolated_omp_response_bound_tracks_valid_growth_during_child_read( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [_omp_msg("growing-user", "user", "wait for large final", attribution="user")], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr(herdr_turns, "_file_turn_child", _growing_isolated_turn_child) - - content = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=5, - ) - - assert content["source_turn_id"] == "growing-user" - assert content["assistant_final_text"].startswith("grew-during-read-") - assert len(content["assistant_final_text"]) > 2 * 1024 * 1024 - assert content["complete"] is True - - -def test_isolated_omp_authoritative_final_over_64mib_streams_losslessly( - tmp_path: Path, - monkeypatch, -) -> None: - large_final = "lossless-omp-" + ( - "z" * (herdr_turns._CODEX_POLL_MAX_BYTES + 257) - ) - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("over-limit-user", "user", "retain exact OMP final", attribution="user"), - _omp_msg("over-limit-final", "assistant", large_final, stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - before_children = {child.pid for child in multiprocessing.active_children()} - before_threads = { - thread.ident - for thread in threading.enumerate() - if thread.name.startswith("tendwire-turn") - } - - content = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=60, - ) - - assert content["assistant_final_text"] == large_final - assert content["complete"] is True - assert {child.pid for child in multiprocessing.active_children()} == before_children - assert { - thread.ident - for thread in threading.enumerate() - if thread.name.startswith("tendwire-turn") - } == before_threads - - -def test_isolated_omp_large_final_unchanged_fast_path_and_appended_turn( - tmp_path: Path, - monkeypatch, -) -> None: - large_final = "canonical-" + ("x" * (6 * 1024 * 1024)) - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("six-mib-user", "user", "first private prompt", attribution="user"), - _omp_msg("six-mib-final", "assistant", large_final, stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - byte_reads: list[int] = [] - monkeypatch.setattr(herdr_turns, "_OMP_ISOLATED_READ_OBSERVER", byte_reads.append) - - first = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=10, - ) - assert first["assistant_final_text"] == large_final - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._serialize_omp_state( - herdr_turns._OMP_SESSION_CACHE[cache_key] - ) - checkpoint_json = json.dumps(checkpoint, separators=(",", ":")) - assert len(checkpoint_json.encode("utf-8")) < 1024 - assert "canonical-" not in checkpoint_json - assert set(checkpoint) == { - "offset", - "observed_size", - "file_id", - "mtime_ns", - "ctime_ns", - "replay_offset", - "turn_open", - "project_root", - } - assert checkpoint["turn_open"] is False - - original_get_context = herdr_turns.multiprocessing.get_context - - def fail_if_spawned(*_args, **_kwargs): - raise AssertionError("unchanged OMP poll spawned a child") - - monkeypatch.setattr(herdr_turns.multiprocessing, "get_context", fail_if_spawned) - second = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=1, - ) - monkeypatch.setattr(herdr_turns.multiprocessing, "get_context", original_get_context) - assert second is herdr_turns._UNCHANGED_TURN - assert byte_reads[-1] == 0 - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert ( - herdr_turns._serialize_omp_state(herdr_turns._OMP_SESSION_CACHE[cache_key]) - == checkpoint - ) - - appended = "\n".join( - [ - "", - json.dumps( - _omp_msg("next-user", "user", "new appended prompt", attribution="user"), - separators=(",", ":"), - ), - json.dumps( - _omp_msg("next-final", "assistant", "new answer", stop="stop"), - separators=(",", ":"), - ), - ] - ) - with open(path, "a", encoding="utf-8") as handle: - handle.write(appended) - third = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=5, - ) - assert third["source_turn_id"] == "next-user" - assert third["user_text"] == "new appended prompt" - assert third["assistant_final_text"] == "new answer" - assert large_final not in json.dumps(third) - assert byte_reads[-1] == len(appended.encode("utf-8")) - - -def test_isolated_omp_cache_validates_identity_and_retains_good_state_on_failures( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [_omp_msg("original-user", "user", "original prompt", attribution="user")], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - original_child = herdr_turns._file_turn_child - - first = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - - def cached_state(): - with herdr_turns._OMP_SESSION_CACHE_LOCK: - return herdr_turns._serialize_omp_state(herdr_turns._OMP_SESSION_CACHE[cache_key]) - - first_state = cached_state() - assert first["source_turn_id"] == "original-user" - - for child, expected_error, timeout_seconds in ( - (_blocked_isolated_turn_child, herdr_turns._TurnReadTimeout, 0.1), - (_failed_isolated_turn_child, herdr_turns._TurnReadFailed, 5), - (_invalid_isolated_turn_child, herdr_turns._TurnReadFailed, 5), - ): - with open(path, "a", encoding="utf-8") as handle: - handle.write( - "\n" - + json.dumps( - {"type": "ignored", "failure_attempt": child.__name__}, - separators=(",", ":"), - ) - ) - monkeypatch.setattr(herdr_turns, "_file_turn_child", child) - with pytest.raises(expected_error): - herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=timeout_seconds, - ) - assert cached_state() == first_state - monkeypatch.setattr(herdr_turns, "_file_turn_child", original_child) - - replacement = path.with_name("replacement.jsonl") - replacement.write_text( - json.dumps(_omp_msg("replacement-user", "user", "replacement prompt", attribution="user")), - encoding="utf-8", - ) - replacement.replace(path) - replaced = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - replaced_state = cached_state() - assert replaced["source_turn_id"] == "replacement-user" - assert replaced_state["file_id"] != first_state["file_id"] - - replaced_inode = path.stat().st_ino - path.write_text( - json.dumps(_omp_msg("truncated-user", "user", "short", attribution="user")), - encoding="utf-8", - ) - assert path.stat().st_ino == replaced_inode - truncated = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - assert truncated["source_turn_id"] == "truncated-user" - - final_line = json.dumps(_omp_msg("final-answer", "assistant", "published", stop="stop")) - with open(path, "a", encoding="utf-8") as handle: - handle.write("\n" + final_line) - final = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - final_state = cached_state() - assert final["source_turn_id"] == "truncated-user" - assert final["assistant_final_text"] == "published" - assert final_state["offset"] == path.stat().st_size - - -def test_omp_same_inode_same_size_rewrite_forces_cold_rescan( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("user-old", "user", "prompt-old", attribution="user"), - _omp_msg("final-old", "assistant", "answer-old", stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - first = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - original_stat = path.stat() - original_size = original_stat.st_size - assert first["assistant_final_text"] == "answer-old" - - rewritten = "\n".join( - json.dumps(line) - for line in ( - _omp_msg("user-new", "user", "prompt-new", attribution="user"), - _omp_msg("final-new", "assistant", "answer-new", stop="stop"), - ) - ) - assert len(rewritten.encode("utf-8")) == original_size - path.write_text(rewritten, encoding="utf-8") - rewritten_stat = path.stat() - assert rewritten_stat.st_ino == original_stat.st_ino - assert rewritten_stat.st_size == original_size - assert ( - rewritten_stat.st_mtime_ns != original_stat.st_mtime_ns - or rewritten_stat.st_ctime_ns != original_stat.st_ctime_ns - ) - - second = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - - assert second["source_turn_id"] == "user-new" - assert second["user_text"] == "prompt-new" - assert second["assistant_final_text"] == "answer-new" - - -def test_omp_every_assistant_after_final_is_ignored_until_next_user( - tmp_path: Path, - monkeypatch, -) -> None: - irrelevant = { - "type": "message", - "id": "metadata-only", - "message": { - "role": "assistant", - "stopReason": "toolUse", - "content": [{"type": "metadata", "value": "private bookkeeping"}], - }, - } - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("stable-user", "user", "stable prompt", attribution="user"), - _omp_msg("stable-final", "assistant", "stable answer", stop="stop"), - irrelevant, - _omp_msg( - "renderable-progress", - "assistant", - "must not reopen the completed turn", - stop="toolUse", - ), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - - assert content["assistant_final_text"] == "stable answer" - assert content["complete"] is True - assert content["assistant_stream_text"] is None - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._OMP_SESSION_CACHE[cache_key] - assert checkpoint.turn_open is False - assert checkpoint.replay_offset == checkpoint.offset - - -def test_omp_appended_assistant_after_committed_final_advances_idle_checkpoint( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("done-user", "user", "finish once", attribution="user"), - _omp_msg("done-final", "assistant", "finished once", stop="stop"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - first = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - assert first["assistant_final_text"] == "finished once" - - appended = "\n" + json.dumps( - _omp_msg( - "late-progress", - "assistant", - "late renderable progress", - stop="toolUse", - ), - separators=(",", ":"), - ) - with open(path, "a", encoding="utf-8") as handle: - handle.write(appended) - second = herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=2, - ) - - assert second is None - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._OMP_SESSION_CACHE[cache_key] - assert checkpoint.turn_open is False - assert checkpoint.offset == path.stat().st_size - assert checkpoint.replay_offset == checkpoint.offset - assert ( - herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=1, - ) - is herdr_turns._UNCHANGED_TURN - ) - - -def test_omp_retries_atomic_replacement_that_occurs_during_read( - tmp_path: Path, - monkeypatch, -) -> None: - root, path = _write_omp_session( - tmp_path, - [ - _omp_msg("old-user", "user", "old prompt", attribution="user"), - _omp_msg("old-final", "assistant", "stale answer", stop="stop"), - ], - ) - replacement = path.with_name("during-read-replacement.jsonl") - replacement.write_text( - "\n".join( - json.dumps(line, separators=(",", ":")) - for line in ( - _omp_msg("new-user", "user", "new prompt", attribution="user"), - _omp_msg("new-final", "assistant", "current answer", stop="stop"), - ) - ), - encoding="utf-8", - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - original_open = open - replaced = False - - class ReplacingReader: - def __init__(self, handle): - self.handle = handle - - def __enter__(self): - self.handle.__enter__() - return self - - def __exit__(self, *args): - return self.handle.__exit__(*args) - - def fileno(self): - return self.handle.fileno() - - def seek(self, *args): - return self.handle.seek(*args) - - def read(self, *args): - nonlocal replaced - if not replaced: - replaced = True - replacement.replace(path) - return self.handle.read(*args) - - def racing_open(file, mode="r", *args, **kwargs): - handle = original_open(file, mode, *args, **kwargs) - if Path(file) == path and mode == "rb" and not replaced: - return ReplacingReader(handle) - return handle - - monkeypatch.setattr("builtins.open", racing_open) - content = herdr_turns._read_omp_session_turn(str(path)) - - assert replaced is True - assert content["source_turn_id"] == "new-user" - assert content["assistant_final_text"] == "current answer" - assert "stale answer" not in json.dumps(content) - - -def test_omp_concurrent_cache_loser_cannot_overwrite_accepted_checkpoint() -> None: - cache_key = "concurrent-publication" - prior = herdr_turns._OmpSessionState( - offset=100, - observed_size=100, - file_id=(7, 11), - ) - accepted = herdr_turns._OmpSessionState( - offset=300, - observed_size=300, - file_id=(7, 11), - ) - losing = herdr_turns._OmpSessionState( - offset=300, - observed_size=300, - file_id=(7, 11), - turn_open=True, - ) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - herdr_turns._OMP_SESSION_CACHE[cache_key] = accepted - - returned = herdr_turns._publish_omp_cache_state( - cache_key, - herdr_turns._serialize_omp_state(prior), - losing, - {"assistant_final_text": "losing duplicate"}, - ) - - assert returned is None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert herdr_turns._OMP_SESSION_CACHE[cache_key] is accepted - - -def test_omp_cache_lru_capacity_moves_hits_and_evicts_oldest(monkeypatch) -> None: - monkeypatch.setattr(herdr_turns, "_OMP_SESSION_CACHE_CAPACITY", 3) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - for index in range(3): - state = herdr_turns._OmpSessionState( - offset=index + 1, - observed_size=index + 1, - file_id=(1, index + 1), - ) - herdr_turns._publish_omp_cache_state( - f"key-{index}", - None, - state, - None, - ) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert herdr_turns._omp_cache_get_locked("key-0") is not None - - newest = herdr_turns._OmpSessionState( - offset=4, - observed_size=4, - file_id=(1, 4), - ) - herdr_turns._publish_omp_cache_state( - "key-3", - None, - newest, - None, - ) - - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert list(herdr_turns._OMP_SESSION_CACHE) == ["key-2", "key-0", "key-3"] - assert len(herdr_turns._OMP_SESSION_CACHE) == 3 - - -def test_omp_sixty_four_large_completed_sessions_keep_only_bounded_coordinates( - tmp_path: Path, - monkeypatch, -) -> None: - root = tmp_path / "omp-sessions" - session_dir = root / "-many" - session_dir.mkdir(parents=True) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - - for index in range(64): - large_final = f"large-final-{index}-" + ("z" * (128 * 1024)) - path = session_dir / f"{index:02d}.jsonl" - path.write_text( - "\n".join( - json.dumps(line, separators=(",", ":")) - for line in ( - _omp_msg(f"user-{index}", "user", f"prompt-{index}", attribution="user"), - _omp_msg(f"final-{index}", "assistant", large_final, stop="stop"), - ) - ), - encoding="utf-8", - ) - content = herdr_turns._read_omp_session_turn(str(path)) - assert content["assistant_final_text"] == large_final - - with herdr_turns._OMP_SESSION_CACHE_LOCK: - serialized = json.dumps( - { - key: herdr_turns._serialize_omp_state(state) - for key, state in herdr_turns._OMP_SESSION_CACHE.items() - }, - separators=(",", ":"), - ) - assert len(herdr_turns._OMP_SESSION_CACHE) == 64 - assert herdr_turns._omp_cache_weight_locked() <= herdr_turns._OMP_SESSION_CACHE_MAX_BYTES - assert all(not state.turn_open for state in herdr_turns._OMP_SESSION_CACHE.values()) - assert "large-final-" not in serialized - assert "z" * 1024 not in serialized - - -def test_omp_cache_enforces_serialized_byte_bound(monkeypatch) -> None: - monkeypatch.setattr(herdr_turns, "_OMP_SESSION_CACHE_CAPACITY", 64) - monkeypatch.setattr(herdr_turns, "_OMP_SESSION_CACHE_MAX_BYTES", 600) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - for index in range(20): - herdr_turns._omp_cache_store_locked( - f"long-cache-key-{index}-" + ("k" * 40), - herdr_turns._OmpSessionState( - offset=index, - observed_size=index, - file_id=(1, index), - ), - ) - assert len(herdr_turns._OMP_SESSION_CACHE) < 20 - assert herdr_turns._omp_cache_weight_locked() <= 600 - - -def test_omp_cache_prunes_disappeared_bindings_and_keeps_live_binding( - tmp_path: Path, - monkeypatch, -) -> None: - root, live_path = _write_omp_session( - tmp_path, - [_omp_msg("live", "user", "live", attribution="user")], - ) - stale_path = live_path.with_name("stale.jsonl") - stale_path.write_text( - json.dumps(_omp_msg("stale", "user", "stale", attribution="user")), - encoding="utf-8", - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - live_key = herdr_turns._omp_cache_key(str(live_path)) - stale_key = herdr_turns._omp_cache_key(str(stale_path)) - assert live_key is not None - assert stale_key is not None - live_state = herdr_turns._OmpSessionState( - offset=1, - observed_size=1, - file_id=(1, 1), - ) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._omp_cache_store_locked(live_key, live_state) - herdr_turns._omp_cache_store_locked( - stale_key, - herdr_turns._OmpSessionState(offset=1, file_id=(1, 2)), - ) - live_binding = WorkerBinding( - host_id="host", - worker_id="worker", - worker_fingerprint="worker-fingerprint", - backend="herdr", - target_kind="agent_id", - target_value="agent", - turn_target_kind="omp_session_path", - turn_target_value=str(live_path), - sendable=True, - observed_at="2026-07-12T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private", - ) - - herdr_turns._prune_omp_cache_for_bindings([live_binding]) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert list(herdr_turns._OMP_SESSION_CACHE) == [live_key] - retained_prior = herdr_turns._serialize_omp_state( - herdr_turns._OMP_SESSION_CACHE[live_key] - ) - retained_generation = herdr_turns._omp_cache_binding_generation_locked(live_key) - - live_path.unlink() - herdr_turns._prune_omp_cache_for_bindings([live_binding]) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert list(herdr_turns._OMP_SESSION_CACHE) == [live_key] - assert herdr_turns._omp_cache_binding_generation_locked(live_key) == retained_generation - - herdr_turns._prune_omp_cache_for_bindings([]) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert not herdr_turns._OMP_SESSION_CACHE - - herdr_turns._prune_omp_cache_for_bindings([live_binding]) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert ( - herdr_turns._omp_cache_binding_generation_locked(live_key) - != retained_generation - ) - - resurrected = herdr_turns._OmpSessionState( - offset=2, - observed_size=2, - file_id=(1, 1), - ) - returned = herdr_turns._publish_omp_cache_state( - live_key, - retained_prior, - resurrected, - None, - retained_generation, - ) - assert returned is None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert live_key not in herdr_turns._OMP_SESSION_CACHE - - -def test_omp_tool_progress_uses_only_allowlisted_structured_summaries(tmp_path, monkeypatch) -> None: - project = tmp_path / "project" - project.mkdir() - _mark_git_repository(project) - (project / "README.md").write_text("public", encoding="utf-8") - private_key_path = "/home/alice/.ssh/id_ed25519" - herdr_socket_path = "/run/user/1000/herdr/private.sock" - credential_url = "https://alice:password@internal.example/private" - provider_key = "sk-" + "proj-" + "PUBLICSAFETY1234567890" - raw_tool_id = "toolu_PUBLICSAFETYTOOL01" - tool_message = { - "type": "message", - "id": "a1", - "message": { - "role": "assistant", - "stopReason": "toolUse", - "content": [ - { - "type": "toolCall", - "id": raw_tool_id, - "name": "bash", - "arguments": { - "command": f"cat {private_key_path}; connect {herdr_socket_path} {credential_url}", - "env": {"TOKEN": provider_key}, - }, - }, - { - "type": "toolCall", - "id": "c2", - "toolName": "bash", - "input": {"command": "python -m pytest -q tests/test_turns.py"}, - }, - { - "type": "toolCall", - "id": "c3", - "tool": "read", - "args": {"path": "README.md", "stdout": private_key_path}, - }, - { - "type": "toolCall", - "id": "c4", - "name": raw_tool_id, - "arguments": { - "nested": [private_key_path, herdr_socket_path, provider_key], - "url": credential_url, - }, - }, - ], - }, - } - root, path = _write_omp_session( - tmp_path, - [ - {"type": "session", "id": "private-session", "cwd": str(project)}, - _omp_msg("u1", "user", "show safe tool progress", attribution="user"), - tool_message, - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(path)) - stream = content["assistant_stream_text"] - - assert stream.split("\n\n") == [ - "step 1 · run command", - "step 2 · test: pytest", - "step 3 · read: README.md", - "step 4 · tool", - ] - for private_value in ( - private_key_path, - herdr_socket_path, - credential_url, - provider_key, - raw_tool_id, - ): - assert private_value not in stream - - -def test_omp_shell_progress_uses_a_small_constant_allowlist() -> None: - cases = [ - ("git status --short", "step 1 · git status"), - ("pytest -q tests/test_turns.py", "step 1 · test: pytest"), - ("uv run pytest tests/test_turns.py", "step 1 · test: pytest"), - ("cargo test --workspace", "step 1 · test: cargo"), - ("npm run build", "step 1 · build: npm"), - ("make all", "step 1 · build: make"), - ("git checkout -b private-branch", "step 1 · run command"), - ("echo arbitrary private text", "step 1 · run command"), - ] - - for command, expected in cases: - item = {"name": "bash", "arguments": {"command": command}} - assert herdr_turns._omp_tool_snippet(item, 1) == expected - - -def test_omp_file_progress_requires_repository_root_proof(tmp_path: Path) -> None: - project = tmp_path / "repo" - docs = project / "docs" - docs.mkdir(parents=True) - _mark_git_repository(project) - readme = project / "README.md" - guide = docs / "guide.md" - outside = tmp_path / "outside.txt" - readme.write_text("readme", encoding="utf-8") - guide.write_text("guide", encoding="utf-8") - outside.write_text("private", encoding="utf-8") - escape = project / "escape" - escape.symlink_to(outside) - - def snippet(path_value: str, root: Path | None = project) -> str: - return herdr_turns._omp_tool_snippet( - {"name": "read", "arguments": {"path": path_value}}, - 1, - root, - ) - - assert snippet("README.md") == "step 1 · read: README.md" - assert snippet(str(guide)) == "step 1 · read: docs/guide.md" - assert snippet("README.md", None) == "step 1 · read file" - assert snippet(str(outside)) == "step 1 · read file" - assert snippet("../outside.txt") == "step 1 · read file" - assert snippet(str(escape)) == "step 1 · read file" - assert snippet("~/.ssh/id_ed25519") == "step 1 · read file" - assert snippet("docs/../README.md") == "step 1 · read file" - assert snippet(".env") == "step 1 · read file" - assert snippet(".git/config") == "step 1 · read file" - assert snippet("secrets/key.txt") == "step 1 · read file" - assert snippet("credentials.json") == "step 1 · read file" - - -def test_omp_file_progress_rejects_unproven_session_cwds(tmp_path: Path, monkeypatch) -> None: - home = tmp_path / "home" / "alice" - home.mkdir(parents=True) - (home / ".bashrc").write_text("operator shell settings", encoding="utf-8") - notes = tmp_path / "operator-work" - notes.mkdir() - (notes / "operator-notes.txt").write_text("private operator notes", encoding="utf-8") - - cases = ( - ("filesystem-root", Path("/"), "/etc/passwd"), - ("home", home, ".bashrc"), - ("notes", notes, "operator-notes.txt"), - ) - for label, cwd, path_value in cases: - root, session_path = _write_omp_session( - tmp_path / label, - [ - {"type": "session", "id": label, "cwd": str(cwd)}, - _omp_msg(f"{label}-user", "user", "inspect a file", attribution="user"), - _omp_read_msg(f"{label}-assistant", path_value), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(session_path)) - - assert content is not None - assert content["assistant_stream_text"] == "step 1 · read file" - - -def test_omp_file_progress_finds_repository_above_session_cwd( - tmp_path: Path, - monkeypatch, -) -> None: - project = tmp_path / "repo-with-subdir" - docs = project / "docs" - docs.mkdir(parents=True) - _mark_git_repository(project) - guide = docs / "guide.md" - guide.write_text("public", encoding="utf-8") - root, session_path = _write_omp_session( - tmp_path / "subdir-session", - [ - {"type": "session", "id": "subdir", "cwd": str(docs)}, - _omp_msg("subdir-user", "user", "inspect the guide", attribution="user"), - _omp_read_msg("subdir-assistant", str(guide)), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(session_path)) - - assert content is not None - assert content["assistant_stream_text"] == "step 1 · read: docs/guide.md" - - -def test_omp_file_progress_accepts_worktree_gitdir_file(tmp_path: Path, monkeypatch) -> None: - checkout = tmp_path / "checkout" - checkout.mkdir() - (checkout / "README.md").write_text("public", encoding="utf-8") - worktree_git_dir = tmp_path / "main" / ".git" / "worktrees" / "checkout" - _write_valid_git_head(worktree_git_dir) - (checkout / ".git").write_text( - f"gitdir: {worktree_git_dir}\n", - encoding="utf-8", - ) - root, session_path = _write_omp_session( - tmp_path / "worktree-session", - [ - {"type": "session", "id": "worktree", "cwd": str(checkout)}, - _omp_msg("worktree-user", "user", "inspect the readme", attribution="user"), - _omp_read_msg("worktree-assistant", "README.md"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(session_path)) - - assert content is not None - assert content["assistant_stream_text"] == "step 1 · read: README.md" - - -def test_omp_file_progress_rejects_invalid_worktree_gitdir_file( - tmp_path: Path, - monkeypatch, -) -> None: - checkout = tmp_path / "invalid-checkout" - checkout.mkdir() - (checkout / "README.md").write_text("public", encoding="utf-8") - (checkout / ".git").write_text("gitdir: ../missing-git-dir\n", encoding="utf-8") - root, session_path = _write_omp_session( - tmp_path / "invalid-worktree-session", - [ - {"type": "session", "id": "invalid-worktree", "cwd": str(checkout)}, - _omp_msg("invalid-user", "user", "inspect the readme", attribution="user"), - _omp_read_msg("invalid-assistant", "README.md"), - ], - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - - content = herdr_turns._read_omp_session_turn(str(session_path)) - - assert content is not None - assert content["assistant_stream_text"] == "step 1 · read file" - - -def _prepare_pane_turn_store( - tmp_path: Path, - *, - herdr_bin: str = "herdr", -) -> tuple[Config, Any]: - db_path = tmp_path / "turns.db" - config = Config( - host_id="turn-host", - db_path=db_path, - herdr_bin=herdr_bin, - herdr_timeout_seconds=10, - ) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "codex", "status": "active", "space_id": "space-1"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value="private-agent", - turn_target_kind="pane_id", - turn_target_value="private-pane", - sendable=True, - observed_at="2026-07-11T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="private-binding", - ) - ], - ) - return config, snapshot - - -def _pane_turn_payload( - *, - user_text: str | None, - final_text: str | None, - stream_text: str | None = None, - complete: bool = True, - source_turn_id: str = "source-turn-long", -) -> dict[str, Any]: - return { - "result": { - "turn": { - "available": True, - "user_text": user_text, - "assistant_final_text": final_text, - "assistant_stream_text": stream_text, - "complete": complete, - "has_open_turn": not complete, - "source_turn_id": source_turn_id, - } - } - } - - -def _current_content_turn( - config: Config, - snapshot: Any, -) -> tuple[dict[str, Any], str]: - payload = turns_payload_from_store( - config.db_path, - config.host_id, - snapshot=snapshot, - schema_version=2, - ) - turn = next( - item - for item in payload["turns"] - if (item.get("content") or {}).get("content_revision") - ) - return turn, str(turn["content"]["content_revision"]) - - -def _reconstruct_turn_field( - config: Config, - *, - turn_id: str, - revision: str, - field: str, -) -> str: - cursor: str | None = None - pages: list[dict[str, Any]] = [] - while True: - page = store_sqlite.get_turn_content( - config.db_path, - config.host_id, - turn_id=turn_id, - content_revision=revision, - field=field, - cursor=cursor, - schema_version=1, - ) - assert page["availability"] == "complete" - assert page["index"] == len(pages) - pages.append(page) - cursor = page["next_cursor"] - if cursor is None: - break - assert all(page["count"] == len(pages) for page in pages) - assert all( - len(str(page["text"]).encode("utf-8")) <= TURN_CONTENT_PAGE_MAX_UTF8_BYTES - for page in pages - ) - return "".join(str(page["text"]) for page in pages) - - -@pytest.mark.parametrize("content_size", [20_000, 1024 * 1024 + 257]) -def test_wrapper_adapter_round_trips_unbounded_authoritative_content( - tmp_path: Path, - monkeypatch, - content_size: int, -) -> None: - adapter = tmp_path / "long_turn_adapter.py" - adapter.write_text( - r"""#!/usr/bin/env python3 -import json -import os - -size = int(os.environ["TENDWIRE_TEST_TURN_SIZE"]) -prompt = " Prompt fi\n" + ("p" * size) + "\u200b\x00\n" -final = "\n# Final\n" + ("f" * size) + "\u200b\x00 " -print(json.dumps({"result": {"turn": { - "available": True, - "user_text": prompt, - "assistant_final_text": final, - "assistant_stream_text": None, - "complete": True, - "has_open_turn": False, - "source_turn_id": "source-turn-long", -}}})) -""", - encoding="utf-8", - ) - adapter.chmod(0o700) - monkeypatch.setenv("TENDWIRE_TEST_TURN_SIZE", str(content_size)) - config, snapshot = _prepare_pane_turn_store(tmp_path, herdr_bin=str(adapter)) - - result = refresh_structured_turn_content(config) - turn, revision = _current_content_turn(config, snapshot) - expected_prompt = sanitize_canonical_turn_text( - " Prompt fi\n" + ("p" * content_size) + "\u200b\x00\n" - ) - expected_final = sanitize_canonical_turn_text( - "\n# Final\n" + ("f" * content_size) + "\u200b\x00 " - ) - serialized_source = json.dumps( - _pane_turn_payload( - user_text=" Prompt fi\n" + ("p" * content_size) + "\u200b\x00\n", - final_text="\n# Final\n" + ("f" * content_size) + "\u200b\x00 ", - ) - ).encode("utf-8") - assert len(serialized_source) > content_size * 2 - - assert result == {"ok": True, "status": "ok", "updated": 1, "attempted": 1} - assert expected_prompt is not None - assert expected_final is not None - assert _reconstruct_turn_field( - config, - turn_id=turn["id"], - revision=revision, - field="user_text", - ) == expected_prompt - assert _reconstruct_turn_field( - config, - turn_id=turn["id"], - revision=revision, - field="assistant_final_text", - ) == expected_final - - -def test_refresh_keeps_only_a_rolling_bounded_stream( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot = _prepare_pane_turn_store(tmp_path) - stream = "".join(str(index % 10) for index in range(TURN_STREAM_TEXT_MAX_CHARS + 137)) - monkeypatch.setattr( - herdr_turns.subprocess, - "run", - _run_returning( - _pane_turn_payload( - user_text="stream this turn", - final_text=None, - stream_text=stream, - complete=False, - ) - ), - ) - - assert refresh_structured_turn_content(config)["updated"] == 1 - turn, _revision = _current_content_turn(config, snapshot) - - assert turn["assistant_stream_text"] == stream[-TURN_STREAM_TEXT_MAX_CHARS:] - assert len(turn["assistant_stream_text"]) == TURN_STREAM_TEXT_MAX_CHARS - - -def test_empty_later_observation_does_not_erase_authoritative_final( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot = _prepare_pane_turn_store(tmp_path) - final_text = "authoritative final" - observations = iter( - [ - _pane_turn_payload(user_text="prompt", final_text=final_text), - _pane_turn_payload(user_text="", final_text=""), - ] - ) - monkeypatch.setattr( - herdr_turns.subprocess, - "run", - lambda args, **kwargs: subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps(next(observations)), - stderr="", - ), - ) - - assert refresh_structured_turn_content(config)["updated"] == 1 - first_turn, first_revision = _current_content_turn(config, snapshot) - second_result = refresh_structured_turn_content(config) - second_turn, second_revision = _current_content_turn(config, snapshot) - - assert second_result["updated"] == 0 - assert second_turn["id"] == first_turn["id"] - assert second_revision == first_revision - assert _reconstruct_turn_field( - config, - turn_id=second_turn["id"], - revision=second_revision, - field="assistant_final_text", - ) == final_text - - -def test_identical_source_turn_observation_is_a_revision_noop( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot = _prepare_pane_turn_store(tmp_path) - payload = _pane_turn_payload(user_text="same prompt", final_text="same final") - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(payload)) - - assert refresh_structured_turn_content(config)["updated"] == 1 - first_turn, first_revision = _current_content_turn(config, snapshot) - with sqlite3.connect(config.db_path) as conn: - first_count = conn.execute( - "SELECT COUNT(*) FROM turn_content_revisions WHERE host_id = ? AND turn_id = ?", - (config.host_id, first_turn["id"]), - ).fetchone()[0] - - assert refresh_structured_turn_content(config)["updated"] == 0 - second_turn, second_revision = _current_content_turn(config, snapshot) - with sqlite3.connect(config.db_path) as conn: - second_count = conn.execute( - "SELECT COUNT(*) FROM turn_content_revisions WHERE host_id = ? AND turn_id = ?", - (config.host_id, second_turn["id"]), - ).fetchone()[0] - - assert second_turn["id"] == first_turn["id"] - assert second_revision == first_revision - assert second_count == first_count - - -def test_complete_reobservation_recovers_known_incomplete_source_turn( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot = _prepare_pane_turn_store(tmp_path) - fragment = ("legacy fragment " * 900)[:11_988] + "\n[truncated]" - complete_final = fragment.removesuffix("\n[truncated]") + " recovered authoritative suffix" - first_payload = _pane_turn_payload(user_text="recover this", final_text=fragment) - complete_payload = _pane_turn_payload(user_text="recover this", final_text=complete_final) - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(first_payload)) - - assert refresh_structured_turn_content(config)["updated"] == 1 - turn, incomplete_revision = _current_content_turn(config, snapshot) - with sqlite3.connect(config.db_path) as conn: - conn.execute( - """ - UPDATE turn_content_revisions - SET final_state = 'known_incomplete' - WHERE host_id = ? AND turn_id = ? AND content_revision = ? - """, - (config.host_id, turn["id"], incomplete_revision), - ) - conn.commit() - - monkeypatch.setattr(herdr_turns.subprocess, "run", _run_returning(complete_payload)) - assert refresh_structured_turn_content(config)["updated"] == 1 - recovered_turn, recovered_revision = _current_content_turn(config, snapshot) - with sqlite3.connect(config.db_path) as conn: - revisions = conn.execute( - """ - SELECT content_revision, final_state, is_current - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? - ORDER BY created_at, content_revision - """, - (config.host_id, turn["id"]), - ).fetchall() - - assert recovered_turn["id"] == turn["id"] - assert recovered_revision != incomplete_revision - assert (incomplete_revision, "known_incomplete", 0) in revisions - assert (recovered_revision, "complete", 1) in revisions - assert _reconstruct_turn_field( - config, - turn_id=recovered_turn["id"], - revision=recovered_revision, - field="assistant_final_text", - ) == complete_final - - -def test_completed_pane_refresh_uses_authoritative_binding_hint( - tmp_path: Path, - monkeypatch: Any, -) -> None: - config = Config(host_id="completion-route", db_path=tmp_path / "route.db") - binding = WorkerBinding( - host_id=config.host_id, - worker_id="worker-1", - worker_fingerprint="worker-fingerprint", - backend="herdr", - target_kind="agent_id", - target_value="agent-private", - turn_target_kind="codex_session_id", - turn_target_value="session-private", - sendable=True, - observed_at="2026-07-23T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-private", - ) - monkeypatch.setattr( - herdr_turns, - "list_worker_bindings", - lambda *_args, **_kwargs: [binding], - ) - refreshed: list[WorkerBinding] = [] - monkeypatch.setattr( - herdr_turns, - "_refresh_turn_binding", - lambda _config, current, **_kwargs: ( - refreshed.append((current, _kwargs.get("pane_target_override"))) - or herdr_turns.TurnRefreshResult("updated", 1) - ), - ) - monkeypatch.setattr( - herdr_turns, - "latest_turn_id_for_worker", - lambda *_args, **_kwargs: "public-turn-1", - ) - - result = herdr_turns.refresh_completed_pane_turn( - config, - "w123456789abcde:pA", - terminal_id="different-terminal", - binding_private_fingerprint="binding-private", - ) - - assert refreshed == [(binding, "w123456789abcde:pA")] - assert result == herdr_turns.CompletedPaneTurnRefreshResult( - "updated", - worker_id="worker-1", - refreshed_turn_id="public-turn-1", - ) - - -def test_completed_pane_refresh_ignores_stale_session_generation_for_content_read( - tmp_path: Path, - monkeypatch: Any, -) -> None: - db_path = tmp_path / "generation-mismatch.db" - config = Config( - host_id="generation-mismatch", - db_path=db_path, - herdr_bin="turn-adapter", - herdr_timeout_seconds=2, - ) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-generation-a", - "name": "claude", - "status": "idle", - "space_id": "space-1", - } - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - worker = snapshot.workers[0] - stale_binding = WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="terminal_id", - target_value="terminal-stable", - turn_target_kind="codex_session_id", - turn_target_value="session-generation-a", - sendable=True, - observed_at="2026-07-23T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="binding-generation-a", - ) - upsert_worker_bindings(db_path, [stale_binding]) - adapter_calls: list[list[str]] = [] - payload = _pane_turn_payload( - user_text="reply with purple elephant 42", - final_text="purple elephant 42", - source_turn_id="generation-b-turn", - ) - - def fake_run(args: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]: - adapter_calls.append(args) - return subprocess.CompletedProcess( - args=args, - returncode=0, - stdout=json.dumps(payload), - stderr="", - ) - - monkeypatch.setattr(herdr_turns.subprocess, "run", fake_run) - monkeypatch.setattr( - herdr_turns, - "_read_file_turn_isolated", - lambda *_args, **_kwargs: pytest.fail( - "completion must not read the stale session generation" - ), - ) - - result = herdr_turns.refresh_completed_pane_turn( - config, - "pane-generation-b", - terminal_id="terminal-stable", - binding_private_fingerprint="binding-generation-a", - ) - payload_from_store = turns_payload_from_store( - db_path, - config.host_id, - snapshot=snapshot, - ) - - assert result.status == "updated" - assert result.worker_id == worker.id - assert adapter_calls == [ - [ - "turn-adapter", - "pane", - "turn", - "pane-generation-b", - "--last", - "--format", - "json", - ] - ] - captured = next( - turn - for turn in payload_from_store["turns"] - if turn.get("assistant_final_text") - ) - assert captured["status"] == "idle" - assert captured["assistant_final_text"] == "purple elephant 42" diff --git a/tests/test_local_state_permissions.py b/tests/test_local_state_permissions.py index 5e4650a..661be4f 100644 --- a/tests/test_local_state_permissions.py +++ b/tests/test_local_state_permissions.py @@ -83,6 +83,29 @@ def _assert_path_free(value: object, *paths: Path) -> None: rendered = repr(value) for path in paths: assert str(path) not in rendered + + +def _open_fd_identities() -> dict[int, tuple[int, int, int, int]]: + identities: dict[int, tuple[int, int, int, int]] = {} + for raw_fd in os.listdir("/proc/self/fd"): + fd = int(raw_fd) + try: + current = os.fstat(fd) + except OSError as exc: + # Listing /proc/self/fd briefly exposes the directory descriptor + # used by listdir itself; it is closed before fstat can inspect it. + if exc.errno == errno.EBADF: + continue + raise + identities[fd] = ( + current.st_dev, + current.st_ino, + current.st_mode, + current.st_rdev, + ) + return identities + + def _sqlite_replacement_source( tmp_path: Path, *, mode: int = 0o600 ) -> tuple[Path, Path, int]: @@ -527,7 +550,7 @@ def remove_at_preflight(phase: str, selected_kind: LocalStateKind) -> None: assert results[result_index].mode is None assert not sidecar.exists() assert _mode(db_path) == 0o600 - assert set(os.listdir("/proc/self/fd")) == before_fds + assert set(os.listdir("/proc/self/fd")) <= before_fds finally: os.close(parent_fd) @@ -806,7 +829,7 @@ def test_sqlite_family_sidecar_replacement_fails_without_target_mutation( unexpected_uid = os.geteuid() + 100_000 replacement_active = False replacement_identity = None - before_fds = set(os.listdir("/proc/self/fd")) + before_fds = _open_fd_identities() preflight_calls = 0 def owner_aware_lstat_at(dir_fd: int, name: str): @@ -876,7 +899,11 @@ def replace_at_preflight(phase: str, selected_kind: LocalStateKind) -> None: else: assert sidecar.read_bytes() == b"hostile-replacement" assert _mode(sidecar) == 0o644 - assert set(os.listdir("/proc/self/fd")) == before_fds + # The full suite can finish background subprocess cleanup while this + # test runs, legitimately closing descriptors owned by pytest. Require + # every remaining descriptor to retain its original identity, which + # still catches both new descriptors and reuse of a closed fd number. + assert _open_fd_identities().items() <= before_fds.items() finally: os.close(parent_fd) diff --git a/tests/test_public_content_safety.py b/tests/test_public_content_safety.py index 2a6b4ae..1f13de7 100644 --- a/tests/test_public_content_safety.py +++ b/tests/test_public_content_safety.py @@ -39,11 +39,11 @@ attention_payload_from_store, init_store, get_turn_content, - merge_turn_content, save_snapshot, tail_event_metadata, turns_payload_from_store, ) +from .store_helpers import apply_test_turn_refresh def _sentinel_corpus() -> dict[str, str]: @@ -375,6 +375,9 @@ def test_public_submission_verdict_is_closed_vocabulary() -> None: assert sanitize_public_value({"submission_verdict": "agent_not_ready"}) == { "submission_verdict": "agent_not_ready" } + assert sanitize_public_value({"submission_verdict": "steering_failed"}) == { + "submission_verdict": "steering_failed" + } assert sanitize_public_value( {"submission_verdict": "agent_target_ambiguous"} ) == {"submission_verdict": "agent_target_ambiguous"} @@ -890,38 +893,22 @@ def save_complete(observed_at: str, attention: list[AttentionSignal]) -> None: {"name": "attention", "limit": 10} ) - # Copy the leased lifecycle jobs into a v4-shaped fixture. Migration must - # preserve live refs privately without putting its grouping markers into - # either the public attention projection or the connector payload. - migration_db_path = tmp_path / "migration-public-boundaries.db" - with ( - sqlite3.connect(str(db_path)) as source, - sqlite3.connect(str(migration_db_path)) as destination, - ): - source.backup(destination) - with sqlite3.connect(str(migration_db_path)) as conn: - conn.execute("DROP TABLE attention_lifecycles") - conn.execute("PRAGMA user_version = 4") - - init_store(migration_db_path) - migrated_feed = attention_payload_from_store(migration_db_path, host_id) - with sqlite3.connect(str(migration_db_path)) as conn: - migrated_attention_rows = [ + # Re-open the exact current schema and prove that every store-backed public + # edge re-sanitizes persisted data after initialization as well as in the + # original process. + init_store(db_path) + reloaded_feed = attention_payload_from_store(db_path, host_id) + with sqlite3.connect(str(db_path)) as conn: + reloaded_attention_rows = [ json.loads(row[0]) for row in conn.execute("SELECT payload_json FROM attention_items") ] - migrated_outbox_rows = [ + reloaded_outbox_rows = [ json.loads(row[0]) for row in conn.execute( "SELECT payload_json FROM connector_outbox ORDER BY id" ) ] - migration_private_states = [ - json.loads(row[0]) - for row in conn.execute( - "SELECT private_state_json FROM connector_outbox ORDER BY id" - ) - ] assert initial_feed is not None assert [item["severity"] for item in initial_feed["attention"]] == ["warning"] @@ -941,9 +928,9 @@ def save_complete(observed_at: str, attention: list[AttentionSignal]) -> None: "attention_created", ] - assert migrated_feed is not None - assert len(migrated_feed["attention"]) == 1 - assert migrated_feed["attention"][0]["reason"] == "Review the safe public result" + assert reloaded_feed is not None + assert len(reloaded_feed["attention"]) == 1 + assert reloaded_feed["attention"][0]["reason"] == "Review the safe public result" assert connector_payload["ok"] is True assert connector_payload["items"] assert all( @@ -953,9 +940,8 @@ def save_complete(observed_at: str, attention: list[AttentionSignal]) -> None: assert snapshot_rows assert event_rows assert attention_rows - assert migrated_attention_rows - assert migrated_outbox_rows - assert any("migration_group" in state for state in migration_private_states) + assert reloaded_attention_rows + assert reloaded_outbox_rows public_attention_surfaces = ( initial_feed, @@ -963,13 +949,13 @@ def save_complete(observed_at: str, attention: list[AttentionSignal]) -> None: pending_feed, resolved_feed, recurrence_feed, - migrated_feed, + reloaded_feed, attention_rows, - migrated_attention_rows, + reloaded_attention_rows, ) outbox_payload_surfaces = ( pre_migration_outbox_rows, - migrated_outbox_rows, + reloaded_outbox_rows, [item["payload"] for item in connector_payload["items"]], ) for surface in public_attention_surfaces + outbox_payload_surfaces: @@ -1099,7 +1085,7 @@ def test_boundary_secret_stays_redacted_through_store_pages_and_plan_outbox( ) init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, diff --git a/tests/test_release_readiness.py b/tests/test_release_readiness.py index 00443e7..e23073b 100644 --- a/tests/test_release_readiness.py +++ b/tests/test_release_readiness.py @@ -17,8 +17,6 @@ from pathlib import Path from typing import Any -from tendwire.backends import herdr_cli -from tendwire.backends.herdr_cli import diagnose_herdr from tendwire.daemon import TendwireDaemon from tendwire.config import Config @@ -244,24 +242,6 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( ) health = TendwireDaemon(config).get_health() - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _value: "/usr/bin/herdr") - monkeypatch.setattr( - herdr_cli.subprocess, - "run", - lambda args, **_kwargs: subprocess.CompletedProcess( - args=args, - returncode=0, - stdout='{"items":[]}', - stderr="", - ), - ) - monkeypatch.setattr( - herdr_cli, - "utc_timestamp", - lambda *_args, **_kwargs: "2026-01-10T00:30:00+00:00", - ) - doctor = diagnose_herdr(config) - before_compaction = db_path.read_bytes() compaction = compact_store( db_path, @@ -286,6 +266,15 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( "deleted": 0, "remaining_candidates": False, }, + "agent_events": { + "host_id": None, + "retention_days": 36500, + "cutoff_at": "1926-02-04T00:00:00+00:00", + "batch_size": 5, + "examined": 0, + "deleted": 0, + "remaining_candidates": False, + }, "final_retention": { "examined": 0, "deleted": 0, @@ -307,23 +296,6 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( "deleted": 0, "remaining_candidates": False, }, - "herdr_turns": { - "schema_version": 1, - "ok": True, - "status": "ok", - "scope": "database", - "host_id": None, - "dry_run": False, - "retention_days": 36500, - "retention_count": 100, - "cutoff_at": "1926-02-04T00:00:00+00:00", - "batch_size": 5, - "examined": 0, - "deleted": 0, - "deleted_completions": 0, - "deleted_watermarks": 0, - "remaining_candidates": False, - }, "batch_size": 5, } assert status["counts"]["snapshots"] == 1 @@ -387,18 +359,6 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( assert cleanup["snapshots"]["examined"] == 0 assert cleanup["outbox"]["updated"] == 0 assert cleanup["turn_content"]["examined"] == 0 - assert cleanup["herdr_turns"] == { - "dry_run": True, - "retention_days": 30, - "retention_count": 4096, - "cutoff_at": "2025-12-11T00:00:00+00:00", - "batch_size": 100, - "examined": 0, - "deleted": 0, - "deleted_completions": 0, - "deleted_watermarks": 0, - "remaining_candidates": False, - } assert cleanup["command_requests"] == { "ok": True, "status": "ok", @@ -438,23 +398,6 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( assert health["limits"]["acknowledged_final_retention_days"] == 36500 assert health["limits"]["acknowledged_final_retention_count"] == 100 assert health["store"]["final_retention"] == status["final_retention"] - maintenance_checks = [ - check for check in doctor["checks"] if check["name"] == "store_maintenance" - ] - assert maintenance_checks == [ - { - "name": "store_maintenance", - "ok": True, - "outcome": "ok", - "remediation": "No action required.", - "snapshot_retention_days": 36500, - "snapshot_retention_count": 100, - "maintenance_batch_size": 5, - "maintenance_cadence_seconds": 3600, - "snapshot_count": 1, - "last_completed_at": "2026-01-10T00:00:00+00:00", - } - ] assert compaction["command"] == "store.compact" assert compaction["scope"] == "database" assert compaction["dry_run"] is True @@ -463,7 +406,7 @@ def test_maintenance_release_surfaces_are_fixed_aggregate_and_private_clean( assert compaction["snapshots"]["deleted"] == 0 assert db_path.read_bytes() == before_compaction - public_surfaces = [status, automatic, cleanup, health, doctor, compaction] + public_surfaces = [status, automatic, cleanup, health, compaction] for surface in public_surfaces: _assert_public_clean(surface) serialized = json.dumps(public_surfaces, sort_keys=True) diff --git a/tests/test_snapshot_sanitize_performance.py b/tests/test_snapshot_sanitize_performance.py index 1eb26ca..46c7900 100644 --- a/tests/test_snapshot_sanitize_performance.py +++ b/tests/test_snapshot_sanitize_performance.py @@ -64,20 +64,15 @@ def _turn_content() -> dict[str, object]: } -@pytest.mark.parametrize("turn_model", ["legacy", "observed"]) -@pytest.mark.parametrize("compatibility_source_token", [False, True]) def test_unchanged_turn_reobservation_performs_no_forbidden_phrase_scans( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, - turn_model: str, - compatibility_source_token: bool, ) -> None: - db_path = tmp_path / f"unchanged-turn-{turn_model}.db" + db_path = tmp_path / "unchanged-turn.db" store_sqlite.init_store(db_path) store_sqlite.save_snapshot( db_path, _snapshot(1, [_worker(1)]), - turn_model=turn_model, ) content = _turn_content() first = store_sqlite.apply_turn_refresh( @@ -86,43 +81,9 @@ def test_unchanged_turn_reobservation_performs_no_forbidden_phrase_scans( "worker-1", content, observed_at="2026-07-20T00:00:02+00:00", - turn_model=turn_model, ) assert first.updated == 1 - if compatibility_source_token: - with sqlite3.connect(db_path) as conn: - row = conn.execute( - """ - SELECT turn_id, payload_json - FROM turns - WHERE host_id = ? - AND json_extract(payload_json, '$.source_turn_id') != '' - """, - (HOST_ID,), - ).fetchone() - assert row is not None - payload = store_sqlite._json_object(row[1]) - candidates = store_sqlite.turn_source_id_candidates( - content["source_turn_id"], - meta=payload["meta"], - source=payload["source"], - kind=payload["kind"], - ) - assert len(candidates) == 2 - payload["source_turn_id"] = candidates[1] - conn.execute( - """ - UPDATE turns SET payload_json = ? - WHERE host_id = ? AND turn_id = ? - """, - ( - store_sqlite._canonical_json(payload), - HOST_ID, - str(row[0]), - ), - ) - scans = 0 original_scan = models._is_forbidden_public_text_phrase @@ -142,7 +103,6 @@ def recording_scan(value: str) -> bool: "worker-1", content, observed_at="2026-07-20T00:00:03+00:00", - turn_model=turn_model, ) assert second.updated == 0 @@ -158,7 +118,6 @@ def test_unchanged_observed_turn_still_attempts_submission_settlement( store_sqlite.save_snapshot( db_path, _snapshot(1, [_worker(1)]), - turn_model="observed", ) content = _turn_content() store_sqlite.apply_turn_refresh( @@ -167,11 +126,10 @@ def test_unchanged_observed_turn_still_attempts_submission_settlement( "worker-1", content, observed_at="2026-07-20T00:00:02+00:00", - turn_model="observed", ) settle_calls = 0 - original_settle = store_sqlite.settle_submission_links_conn + original_settle = store_sqlite._settle_submission_links_conn def recording_settle(*args, **kwargs): nonlocal settle_calls @@ -180,7 +138,7 @@ def recording_settle(*args, **kwargs): monkeypatch.setattr( store_sqlite, - "settle_submission_links_conn", + "_settle_submission_links_conn", recording_settle, ) second = store_sqlite.apply_turn_refresh( @@ -189,7 +147,6 @@ def recording_settle(*args, **kwargs): "worker-1", content, observed_at="2026-07-20T00:00:03+00:00", - turn_model="observed", ) assert second.updated == 0 diff --git a/tests/test_store.py b/tests/test_store.py index 3286984..cd24708 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -6,6 +6,7 @@ import gc import hashlib import json +import logging import multiprocessing import os import sqlite3 @@ -41,7 +42,6 @@ CompactionOptions, ack_connector_delivery, backend_pending_choice_terminal_effect, - append_event, attention_payload_from_store, cleanup_event_retention, compact_store, @@ -55,23 +55,24 @@ get_command_request, init_store, latest_snapshot, - list_attention_items, - list_hosts, list_worker_bindings, reclaim_expired_connector_leases, poll_connector_outbox, mark_command_send_started, reserve_command_request, reserve_terminal_command_replay, - resolve_worker_binding, run_store_maintenance, save_snapshot, store_status, tail_event_metadata, - merge_turn_content, turns_payload_from_store, upsert_worker_bindings, ) +from .store_helpers import ( + read_test_attention_items, + apply_test_backend_pending, + apply_test_turn_refresh, +) _PR6_TABLES = { @@ -256,8 +257,6 @@ def _invoke_creation_capable_store_path(name: str, db_path: Path) -> None: ) if name == "init_store": init_store(db_path) - elif name == "append_event": - append_event(db_path, "host-a", "store.test", {"safe": True}) elif name == "save_snapshot": save_snapshot(db_path, snapshot) elif name == "upsert_worker_bindings": @@ -341,7 +340,7 @@ def broaden_then_validate(parent_fd: int, leaf: str) -> None: assert validated_live_family is True -def test_store_startup_repairs_broad_modes_idempotently_and_preserves_data( +def test_store_startup_repairs_broad_modes_before_explicit_schema_discard( tmp_path: Path, ) -> None: state_dir = tmp_path / "broad-state" @@ -358,20 +357,24 @@ def test_store_startup_repairs_broad_modes_idempotently_and_preserves_data( init_store(db_path) first_modes = (_mode(state_dir), _mode(db_path)) with closing(sqlite3.connect(str(db_path))) as conn, conn: - first_value = conn.execute("SELECT value FROM preserved").fetchone()[0] + first_preserved = conn.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'preserved'" + ).fetchone() first_version = _user_version(conn) conn.close() init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - second_value = conn.execute("SELECT value FROM preserved").fetchone()[0] + second_preserved = conn.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'preserved'" + ).fetchone() second_version = _user_version(conn) conn.close() assert first_modes == (0o700, 0o600) assert (_mode(state_dir), _mode(db_path)) == first_modes assert db_path.stat().st_ino == inode - assert (first_value, second_value) == ("kept", "kept") + assert (first_preserved, second_preserved) == (None, None) assert (first_version, second_version) == ( store_sqlite.STORE_SCHEMA_VERSION, store_sqlite.STORE_SCHEMA_VERSION, @@ -592,7 +595,6 @@ def reject_mutation(*_args: Any, **_kwargs: Any) -> Any: "entrypoint", [ "init_store", - "append_event", "save_snapshot", "upsert_worker_bindings", "expire_worker_bindings", @@ -1279,234 +1281,6 @@ def capture_connection(*args: Any, **kwargs: Any) -> sqlite3.Connection: assert {process.pid for process in multiprocessing.active_children()} == before_children -def test_store_initializes_v8_schema_with_companion_attention_lifecycle(tmp_path: Path) -> None: - db_path = tmp_path / "tendwire.db" - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _PR6_TABLES <= _table_names(conn) - columns = {row[1] for row in conn.execute("PRAGMA table_info(snapshots)")} - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - assert {"host_id", "created_at", "payload", "content_fingerprint"} <= columns - snapshot_indexes = { - str(row[1]) - for row in conn.execute("PRAGMA index_list(snapshots)").fetchall() - } - assert { - "idx_snapshots_host_newest", - "idx_snapshots_created_host_id", - } <= snapshot_indexes - assert { - "idx_snapshots_host_id", - "idx_snapshots_created_at", - "idx_snapshots_content_fingerprint", - "idx_snapshots_host_created_id", - }.isdisjoint(snapshot_indexes) - binding_columns = {row[1] for row in conn.execute("PRAGMA table_info(worker_bindings)")} - assert { - "host_id", - "worker_id", - "worker_fingerprint", - "backend", - "target_kind", - "target_value", - "turn_target_kind", - "turn_target_value", - "sendable", - "reason", - "observed_at", - "expires_at", - "private_fingerprint", - } <= binding_columns - binding_indexed = _indexed_columns(conn, "worker_bindings") - assert { - "worker_id", - "worker_fingerprint", - "private_fingerprint", - "target_kind", - "target_value", - "expires_at", - } <= binding_indexed - command_columns = { - row[1] for row in conn.execute("PRAGMA table_info(commands)") - } - assert { - "host_id", - "request_id", - "action", - "canonical_version", - "canonical_fingerprint", - "public_worker_id", - "state", - "status", - "request_json", - "result_json", - "reserved_at", - "send_started_at", - "terminal_at", - "updated_at", - "legacy_collision", - "legacy_collision_count", - } == command_columns - {"id", "created_at"} - receipt_columns = { - row[1] for row in conn.execute("PRAGMA table_info(command_receipts)") - } - assert { - "canonical_version", - "canonical_fingerprint", - "canonical_request_json", - "public_worker_id", - "state", - "owner_token_hash", - "owner_expires_at", - "binding_fingerprint", - "reserved_at", - "send_started_at", - "terminal_at", - "updated_at", - } <= receipt_columns - assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" - attention_columns = {row[1] for row in conn.execute("PRAGMA table_info(attention_items)")} - assert { - "attention_id", - "fingerprint", - "first_seen_at", - "last_seen_at", - "last_changed_at", - "resolved_at", - "lifecycle_status", - "resolved_reason", - "signal_count", - } <= attention_columns - attention_indexed = _indexed_columns(conn, "attention_items") - assert {"lifecycle_status", "last_seen_at", "fingerprint"} <= attention_indexed - lifecycle_columns = { - row[1] - for row in conn.execute("PRAGMA table_info(attention_lifecycles)") - } - assert lifecycle_columns == { - "host_id", - "family_key", - "generation", - "lifecycle_status", - "current_attention_id", - "first_seen_at", - "last_positive_at", - "first_missing_at", - "missing_observation_count", - "last_accepted_at", - "last_observation_key", - "max_notified_severity_rank", - } - assert {"lifecycle_status", "current_attention_id"} <= _indexed_columns( - conn, "attention_lifecycles" - ) - assert { - row[1] - for row in conn.execute("PRAGMA table_info(turn_content_revisions)") - } == { - "host_id", - "turn_id", - "content_revision", - "user_text", - "assistant_final_text", - "user_state", - "final_state", - "user_char_length", - "user_byte_length", - "final_char_length", - "final_byte_length", - "user_page_count", - "final_page_count", - "is_current", - "created_at", - "superseded_at", - } - revision_indexes = { - row[1] - for row in conn.execute("PRAGMA index_list(turn_content_revisions)") - } - assert {"ux_turn_content_current", "idx_turn_content_cleanup"} <= revision_indexes - assert { - row[1] - for row in conn.execute( - "PRAGMA table_info(turn_content_page_boundaries)" - ) - } == { - "host_id", - "turn_id", - "content_revision", - "field", - "page_index", - "start_char", - "start_byte", - } - assert { - row[1] - for row in conn.execute("PRAGMA table_info(turn_presentation_plans)") - } == { - "id", - "host_id", - "name", - "plan_token", - "turn_id", - "content_revision", - "presentation_version", - "generation", - "part_count", - "state", - "replaces_plan_token", - "recovers_plan_token", - "created_at", - "activated_at", - "completed_at", - "source_outbox_id", - } - assert { - row[1] - for row in conn.execute("PRAGMA table_info(turn_presentation_jobs)") - } == { - "id", - "plan_id", - "sequence_index", - "operation", - "part_ordinal", - "spans_json", - "outbox_id", - "created_at", - } - job_indexes = { - row[1] - for row in conn.execute("PRAGMA index_list(turn_presentation_jobs)") - } - assert { - "idx_turn_presentation_jobs_plan_sequence", - "idx_turn_presentation_jobs_outbox", - } <= job_indexes - assert { - row[1] - for row in conn.execute( - "PRAGMA table_info(turn_presentation_recoveries)" - ) - } == { - "id", - "host_id", - "name", - "request_id", - "failed_plan_id", - "recovered_plan_id", - "failed_plan_token", - "recovered_plan_token", - "generation", - "source_job_count", - "delivered_prefix_count", - "fresh_job_count", - "retained_failed_job_count", - "prior_attempt_count", - "outcome", - "created_at", - } def test_store_connections_apply_wal_busy_timeout_and_foreign_keys(tmp_path: Path) -> None: @@ -1553,29 +1327,23 @@ def test_store_command_receipts_have_unique_logical_key_index(tmp_path: Path) -> def test_store_status_tail_and_retention_cleanup_are_host_scoped_and_bounded(tmp_path: Path) -> None: db_path = tmp_path / "maintenance.db" config = Config(host_id="storehost", db_path=db_path) - snapshot = project_from_raw(config, workers=[{"id": "worker-1", "name": "Worker One"}]) - save_snapshot(db_path, snapshot) - append_event( - db_path, - "storehost", - "private.event", - {"pane_id": "sentinel-private-pane", "raw_payload": "sentinel-private-raw"}, - observed_at="2026-01-01T00:00:00+00:00", + old_snapshot = project_from_raw( + config, + workers=[{"id": "worker-1", "name": "Worker One"}], + timestamp=datetime.fromisoformat("2026-01-01T00:00:00+00:00"), ) - append_event( - db_path, - "storehost", - "public.event", - {"safe": "kept"}, - observed_at="2026-01-09T00:00:00+00:00", + new_snapshot = project_from_raw( + config, + workers=[{"id": "worker-2", "name": "Worker Two"}], + timestamp=datetime.fromisoformat("2026-01-09T00:00:00+00:00"), ) - append_event( - db_path, - "otherhost", - "other.event", - {"safe": "kept"}, - observed_at="2026-01-01T00:00:00+00:00", + other_snapshot = project_from_raw( + Config(host_id="otherhost", db_path=db_path), + workers=[{"id": "worker-3", "name": "Other Worker"}], + timestamp=datetime.fromisoformat("2026-01-01T00:00:00+00:00"), ) + for snapshot in (old_snapshot, new_snapshot, other_snapshot): + save_snapshot(db_path, snapshot) with sqlite3.connect(str(db_path)) as conn: conn.execute( """ @@ -1629,72 +1397,10 @@ def test_store_status_tail_and_retention_cleanup_are_host_scoped_and_bounded(tmp assert cleanup["deleted"] == 1 assert host_events == before["counts"]["events"] - 1 assert other_events == 1 - assert snapshots == 1 + assert snapshots == 2 assert outbox_rows == 1 -def test_store_operational_metadata_buckets_unsafe_labels(tmp_path: Path) -> None: - db_path = tmp_path / "unsafe-metadata.db" - init_store(db_path) - append_event( - db_path, - "storehost", - "telegram.delivery", - {"safe": "kept"}, - aggregate_type="raw_payload", - observed_at="2026-01-01T00:00:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "storehost", - "attention", - "job-unsafe", - "telegram_delivery", - '{"safe":"kept"}', - "{}", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "storehost", - "attention", - "job-queued", - "queued", - '{"safe":"kept"}', - "{}", - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - - status = store_status(db_path, "storehost") - tail = tail_event_metadata(db_path, "storehost", limit=10) - encoded = json.dumps({"status": status, "tail": tail}, sort_keys=True).lower() - - assert status["outbox"]["pending"] == 1 - assert status["outbox"]["by_status"]["queued"] == 1 - assert status["outbox"]["by_status"]["unknown"] == 1 - assert "telegram_delivery" not in status["outbox"]["by_status"] - assert tail["events"][0]["event_type"] == "unknown" - assert tail["events"][0]["aggregate_type"] == "unknown" - assert "telegram" not in encoded - assert "raw_payload" not in encoded - assert "delivery" not in encoded def test_attention_payload_from_store_buckets_unsafe_row_text(tmp_path: Path) -> None: @@ -1840,19 +1546,6 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( ) _save_observation(db_path, snapshot, "positive", snapshot.updated_at) - append_event( - db_path, - "public-host", - "private.adapter", - { - "note": unsafe_value, - "pane_id": "private-pane", - "markdown": safe_markdown, - }, - aggregate_type="worker", - aggregate_id="worker-public", - observed_at="2026-01-01T00:01:00+00:00", - ) binding = WorkerBinding( host_id="public-host", worker_id="worker-public", @@ -1903,7 +1596,7 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( now="2026-01-01T00:01:02+00:00", ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "public-host", "worker-public", @@ -1915,7 +1608,7 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( }, observed_at="2026-01-01T00:02:00+00:00", ) == 1 - assert store_sqlite.merge_backend_pending( + assert apply_test_backend_pending( db_path, "public-host", "worker-public", @@ -1982,8 +1675,10 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( turns = turns_payload_from_store(db_path, "public-host", snapshot=restored) attention = attention_payload_from_store(db_path, "public-host") assert attention is not None - attention_items = list_attention_items(db_path, "public-host") - backend_pending = store_sqlite.list_backend_pending(db_path, "public-host") + attention_items = read_test_attention_items(db_path, "public-host") + backend_pending = store_sqlite.pending_payload_from_store( + db_path, "public-host" + ) tail = tail_event_metadata(db_path, "public-host", limit=20) public_readers = { "snapshot": restored.to_dict(), @@ -2035,18 +1730,6 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( assert values, f"{table}.{column} should be populated" assert all(unsafe_value not in value for value in values) - event_payload = json.loads( - conn.execute( - """ - SELECT payload_json - FROM events - WHERE host_id = ? AND event_type = ? - ORDER BY id DESC - LIMIT 1 - """, - ("public-host", "private.adapter"), - ).fetchone()[0] - ) outbox_private = json.loads( conn.execute( """ @@ -2060,8 +1743,6 @@ def test_store_public_json_boundaries_share_recursive_sanitizer( ).fetchone()[0] ) - assert event_payload["note"] == unsafe_value - assert event_payload["pane_id"] == "private-pane" assert outbox_private["note"] == unsafe_value assert "lease_token" in outbox_private private_bindings = list_worker_bindings( @@ -2150,287 +1831,10 @@ def test_store_maintenance_dry_run_and_exhausted_outbox_status(tmp_path: Path) - assert json.loads(private_state) == {} -def test_store_maintenance_bounds_herdr_completions_and_drops_dead_watermarks( - tmp_path: Path, -) -> None: - db_path = tmp_path / "herdr-turn-maintenance.db" - init_store(db_path) - host_id = "herdr-retention-host" - dead_pane = "workspace:pDead" - active_pane = "workspace:pLive" - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - dead_pane, - turn_epoch=7, - last_turn=0, - observed_at="2026-01-01T00:00:00+00:00", - ) - for turn in range(1, 4): - store_sqlite.record_herdr_turn_completion( - db_path, - host_id, - dead_pane, - turn_epoch=7, - turn=turn, - outcome="completed", - completed_unix_ms=turn, - message=None, - message_truncated=False, - agent_session_path=None, - worker_id="dead-worker", - refreshed_turn_id=None, - observed_at=f"2026-01-0{turn}T00:00:00+00:00", - ) - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - active_pane, - turn_epoch=7, - last_turn=9, - observed_at="2026-01-01T00:00:00+00:00", - ) - store_sqlite.record_herdr_turn_completion( - db_path, - host_id, - active_pane, - turn_epoch=7, - turn=9, - outcome="completed", - completed_unix_ms=9, - message=None, - message_truncated=False, - agent_session_path=None, - worker_id="dead-worker", - refreshed_turn_id=None, - observed_at="2026-01-09T00:00:00+00:00", - ) - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=host_id, - # The same logical worker moved from dead_pane to active_pane. - # Only a current pane target keeps a watermark alive. - worker_id="dead-worker", - worker_fingerprint="live-fingerprint", - backend="herdr", - target_kind="terminal_id", - target_value="live-terminal", - turn_target_kind="pane_id", - turn_target_value=active_pane, - sendable=True, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="live-private-fingerprint", - ) - ], - ) - - result = run_store_maintenance( - db_path, - host_id, - retention_days=30, - max_outbox_attempts=3, - now="2026-03-15T00:00:00+00:00", - herdr_turn_retention_days=30, - herdr_turn_retention_count=1, - herdr_turn_batch_size=4, - ) - - assert result["ok"] is True - assert result["herdr_turns"]["deleted_completions"] == 3 - assert result["herdr_turns"]["deleted_watermarks"] == 1 - with closing(sqlite3.connect(str(db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - ORDER BY turn - """, - (host_id, dead_pane), - ).fetchall() == [] - assert conn.execute( - """ - SELECT pane_id, turn - FROM herdr_turn_completions - WHERE host_id = ? - ORDER BY pane_id, turn - """, - (host_id,), - ).fetchall() == [(active_pane, 9)] - assert conn.execute( - """ - SELECT pane_id - FROM herdr_turn_watermarks - WHERE host_id = ? - ORDER BY pane_id - """, - (host_id,), - ).fetchall() == [(active_pane,)] - - -def test_herdr_turn_maintenance_reports_watermark_deferred_by_full_batch( - tmp_path: Path, -) -> None: - db_path = tmp_path / "herdr-turn-deferred-watermark.db" - init_store(db_path) - host_id = "herdr-recent-retention-host" - active_pane = "workspace:pLive" - dead_pane = "workspace:pDead" - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - dead_pane, - turn_epoch=7, - last_turn=0, - observed_at="2026-01-01T00:00:00+00:00", - ) - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - active_pane, - turn_epoch=7, - last_turn=0, - observed_at="2026-01-01T00:00:00+00:00", - ) - for turn in range(1, 4): - store_sqlite.record_herdr_turn_completion( - db_path, - host_id, - active_pane, - turn_epoch=7, - turn=turn, - outcome="completed", - completed_unix_ms=turn, - message=None, - message_truncated=False, - agent_session_path=None, - worker_id="live-worker", - refreshed_turn_id=None, - observed_at=f"2026-01-0{turn}T00:00:00+00:00", - ) - upsert_worker_bindings( - db_path, - [ - WorkerBinding( - host_id=host_id, - worker_id="live-worker", - worker_fingerprint="live-fingerprint", - backend="herdr", - target_kind="terminal_id", - target_value="live-terminal", - turn_target_kind="pane_id", - turn_target_value=active_pane, - sendable=True, - observed_at="2026-01-01T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint="live-private-fingerprint", - ) - ], - ) - - first = store_sqlite.cleanup_herdr_turn_retention( - db_path, - host_id=host_id, - retention_days=30, - retention_count=1, - batch_size=2, - now="2026-03-15T00:00:00+00:00", - ) - - assert first["deleted_completions"] == 2 - assert first["deleted_watermarks"] == 0 - assert first["remaining_candidates"] is True - - second = store_sqlite.cleanup_herdr_turn_retention( - db_path, - host_id=host_id, - retention_days=30, - retention_count=1, - batch_size=2, - now="2026-03-15T00:00:00+00:00", - ) - - assert second["deleted_completions"] == 0 - assert second["deleted_watermarks"] == 1 - assert second["remaining_candidates"] is False - with closing(sqlite3.connect(str(db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - ORDER BY turn - """, - (host_id, active_pane), - ).fetchall() == [(3,)] - assert conn.execute( - """ - SELECT pane_id - FROM herdr_turn_watermarks - WHERE host_id = ? - ORDER BY pane_id - """, - (host_id,), - ).fetchall() == [(active_pane,)] -def test_herdr_turn_maintenance_low_count_preserves_fresh_rows( - tmp_path: Path, -) -> None: - db_path = tmp_path / "herdr-turn-fresh-count-floor.db" - init_store(db_path) - host_id = "herdr-fresh-retention-host" - pane_id = "workspace:pFresh" - store_sqlite.set_herdr_turn_watermark( - db_path, - host_id, - pane_id, - turn_epoch=7, - last_turn=0, - observed_at="2026-03-14T00:00:00+00:00", - ) - for turn in range(1, 4): - store_sqlite.record_herdr_turn_completion( - db_path, - host_id, - pane_id, - turn_epoch=7, - turn=turn, - outcome="completed", - completed_unix_ms=turn, - message=None, - message_truncated=False, - agent_session_path=None, - worker_id="fresh-worker", - refreshed_turn_id=None, - observed_at=f"2026-03-14T00:00:0{turn}+00:00", - ) - result = store_sqlite.cleanup_herdr_turn_retention( - db_path, - host_id=host_id, - retention_days=30, - retention_count=1, - batch_size=100, - now="2026-03-15T00:00:00+00:00", - ) - assert result["deleted_completions"] == 0 - assert result["deleted_watermarks"] == 0 - assert result["remaining_candidates"] is False - with closing(sqlite3.connect(str(db_path))) as conn, conn: - assert conn.execute( - """ - SELECT turn - FROM herdr_turn_completions - WHERE host_id = ? AND pane_id = ? - ORDER BY turn - """, - (host_id, pane_id), - ).fetchall() == [(1,), (2,), (3,)] def test_exhaust_connector_retries_reclaims_expired_leases_before_dead_letter(tmp_path: Path) -> None: @@ -2523,161 +1927,8 @@ def test_exhaust_connector_retries_reclaims_expired_leases_before_dead_letter(tm assert (attempt_count, max_attempt) == (2, 2) -def test_store_migrates_v1_schema_and_persists_content_fingerprint(tmp_path: Path) -> None: - db_path = tmp_path / "legacy.db" - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - """ - CREATE TABLE snapshots ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - created_at TEXT NOT NULL, - payload TEXT NOT NULL - ); - PRAGMA user_version = 1; - """ - ) - - init_store(db_path) - config = Config(host_id="storehost", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "Agent One", "status": "blocked"}], - ) - save_snapshot(db_path, snapshot) - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - row = conn.execute( - "SELECT host_id, content_fingerprint, payload FROM snapshots ORDER BY id DESC LIMIT 1" - ).fetchone() - assert row[0] == "storehost" - assert row[1] == snapshot.content_fingerprint - assert json.loads(row[2]) == json.loads(snapshot.to_json()) - restored = latest_snapshot(db_path) - assert restored is not None - assert restored.host_id == "storehost" - assert restored.content_fingerprint == snapshot.content_fingerprint - - -def test_store_migrates_partial_v3_db_with_legacy_data_idempotently(tmp_path: Path) -> None: - db_path = tmp_path / "partial-v3.db" - snapshot = project_empty(Config(host_id="legacy-host", db_path=db_path)) - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - """ - CREATE TABLE snapshots ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - created_at TEXT NOT NULL, - content_fingerprint TEXT NOT NULL DEFAULT '', - payload TEXT NOT NULL - ); - CREATE TABLE command_receipts ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - request_id TEXT NOT NULL, - action TEXT NOT NULL, - payload_fingerprint TEXT NOT NULL, - status TEXT NOT NULL, - result_json TEXT NOT NULL, - created_at TEXT NOT NULL, - completed_at TEXT, - uncertain INTEGER NOT NULL DEFAULT 0 - ); - CREATE TABLE worker_bindings ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - host_id TEXT NOT NULL, - worker_id TEXT NOT NULL, - worker_fingerprint TEXT NOT NULL, - backend TEXT NOT NULL, - target_kind TEXT NOT NULL, - target_value TEXT NOT NULL, - turn_target_kind TEXT, - turn_target_value TEXT, - sendable INTEGER NOT NULL DEFAULT 0, - reason TEXT, - observed_at TEXT NOT NULL, - expires_at TEXT NOT NULL, - private_fingerprint TEXT NOT NULL - ); - PRAGMA user_version = 3; - """ - ) - conn.execute( - """ - INSERT INTO snapshots (host_id, created_at, content_fingerprint, payload) - VALUES (?, ?, ?, ?) - """, - ( - snapshot.host_id, - snapshot.updated_at, - snapshot.content_fingerprint, - snapshot.to_json(), - ), - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "legacy-host", - "legacy-req", - "send_instruction", - "legacy-fp", - STATUS_ACCEPTED, - '{"status":"accepted"}', - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - 0, - ), - ) - conn.execute( - """ - INSERT INTO worker_bindings ( - host_id, worker_id, worker_fingerprint, backend, target_kind, - target_value, sendable, reason, observed_at, expires_at, - private_fingerprint - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - "legacy-host", - "worker-legacy", - "worker-fp", - "herdr", - "agent_id", - "agent-private", - 1, - None, - "2026-01-01T00:00:00+00:00", - "9999-12-31T23:59:59+00:00", - "legacy-private", - ), - ) - - init_store(db_path) - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _PR6_TABLES <= _table_names(conn) - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - assert conn.execute("SELECT COUNT(*) FROM snapshots").fetchone()[0] == 1 - assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 1 - assert conn.execute("SELECT COUNT(*) FROM worker_bindings").fetchone()[0] == 1 - command = conn.execute( - """ - SELECT status, canonical_fingerprint, result_json - FROM commands - WHERE host_id = 'legacy-host' - AND request_id = 'legacy-req' - """ - ).fetchone() - - assert command == (STATUS_ACCEPTED, "legacy-fp", '{"status":"accepted"}') def _snapshot_with_worker_status( @@ -2775,7 +2026,9 @@ def test_store_default_context_is_non_authoritative_and_snapshot_fallback_remain save_snapshot(db_path, snapshot) assert _lifecycle_rows(db_path) == [] - assert list_attention_items(db_path, "attention-host") == [] + assert read_test_attention_items(db_path, "attention-host")[0]["id"] == ( + snapshot.attention[0].id + ) payload = attention_payload_from_store(db_path, "attention-host") assert payload is not None assert payload["attention"][0]["id"] == snapshot.attention[0].id @@ -2845,8 +2098,8 @@ def test_store_resolution_requires_distinct_misses_and_120_seconds(tmp_path: Pat lifecycle = _lifecycle_rows(db_path)[0] assert lifecycle[3:5] == ("resolved", None) - assert list_attention_items(db_path, "attention-host") == [] - audit = list_attention_items( + assert read_test_attention_items(db_path, "attention-host") == [] + audit = read_test_attention_items( db_path, "attention-host", include_resolved=True ) assert audit[0]["resolved_reason"] == "gone" @@ -2894,7 +2147,7 @@ def test_store_none_authority_is_lifecycle_byte_equivalent( ) _save_observation(db_path, present, "complete", present.updated_at) before_lifecycle = _lifecycle_rows(db_path) - before_attention = list_attention_items( + before_attention = read_test_attention_items( db_path, "attention-host", include_resolved=True ) before_outbox = _connector_outbox_rows(db_path) @@ -2908,7 +2161,7 @@ def test_store_none_authority_is_lifecycle_byte_equivalent( _save_observation(db_path, degraded, "none", degraded.updated_at) assert _lifecycle_rows(db_path) == before_lifecycle - assert list_attention_items( + assert read_test_attention_items( db_path, "attention-host", include_resolved=True ) == before_attention assert _connector_outbox_rows(db_path) == before_outbox @@ -2991,10 +2244,10 @@ def test_store_escalation_downgrade_one_generation_one_current_pointer( _save_observation(db_path, snapshot, "positive", at) assert _lifecycle_rows(db_path)[0][2:4] == (1, "open") - current = list_attention_items(db_path, "attention-host") + current = read_test_attention_items(db_path, "attention-host") assert len(current) == 1 assert current[0]["status"] == "blocked" - audit = list_attention_items( + audit = read_test_attention_items( db_path, "attention-host", include_resolved=True ) assert len(audit) == 2 @@ -3149,7 +2402,7 @@ def test_store_same_family_variant_selection_is_order_independent(tmp_path: Path data["attention"] = list(reversed(variants)) if reverse else variants snapshot = store_sqlite.Snapshot.from_dict(data) _save_observation(db_path, snapshot, "complete", snapshot.updated_at) - current = list_attention_items(db_path, "attention-host") + current = read_test_attention_items(db_path, "attention-host") assert len(current) == 1 selected_ids.append(current[0]["id"]) assert selected_ids == ["critical-variant", "critical-variant"] @@ -3168,7 +2421,7 @@ def test_store_invalid_or_naive_lifecycle_time_is_noop( _save_observation(db_path, first, "complete", first.updated_at) before = ( _lifecycle_rows(db_path), - list_attention_items(db_path, "attention-host", include_resolved=True), + read_test_attention_items(db_path, "attention-host", include_resolved=True), _connector_outbox_rows(db_path), ) failed = _snapshot_with_worker_status( @@ -3177,7 +2430,7 @@ def test_store_invalid_or_naive_lifecycle_time_is_noop( _save_observation(db_path, failed, "positive", observed_at) after = ( _lifecycle_rows(db_path), - list_attention_items(db_path, "attention-host", include_resolved=True), + read_test_attention_items(db_path, "attention-host", include_resolved=True), _connector_outbox_rows(db_path), ) assert after == before @@ -3195,7 +2448,7 @@ def test_store_strict_order_preserves_subsecond_observations(tmp_path: Path) -> ) _save_observation(db_path, snapshot, "positive", at) lifecycle = _lifecycle_rows(db_path)[0] - current = list_attention_items(db_path, "attention-host")[0] + current = read_test_attention_items(db_path, "attention-host")[0] assert lifecycle[9] == "2026-01-01T00:00:00.000002+00:00" assert current["signal_count"] == 2 assert current["last_seen_at"] == lifecycle[9] @@ -3221,1358 +2474,164 @@ def test_store_deterministic_30_minute_flap_has_one_episode_and_initial( lifecycle = _lifecycle_rows(db_path)[0] assert lifecycle[2:4] == (1, "open") assert lifecycle[7:9] == (None, 0) - assert len(list_attention_items(db_path, "attention-host")) == 1 + assert len(read_test_attention_items(db_path, "attention-host")) == 1 assert len(_connector_outbox_rows(db_path)) == 1 -def _reset_store_to_v4(db_path: Path) -> None: - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - conn.execute("DROP TABLE attention_lifecycles") - conn.execute("PRAGMA user_version = 4") -def _insert_legacy_attention( - conn: sqlite3.Connection, - *, - attention_id: str, - source: str, - severity: str, - status: str, - lifecycle_status: str, - at: str, - first_seen_at: str | None = None, - signal_count: int = 1, - last_seen_at: str | None = None, - last_changed_at: str | None = None, -) -> None: - payload = { - "id": attention_id, - "source": source, - "kind": "worker_status", - "severity": severity, - "status": status, - "fingerprint": f"fp-{attention_id}", - } - # last_seen_at drives the migration's positive_at; last_changed_at drives its - # change/resolve progress. Allowing them to differ from `at` is what lets a - # test build a resolved episode whose resolution is newer than its last - # positive (the ordering the collapsed default hides). - seen_at = last_seen_at or at - changed_at = last_changed_at or at - resolved_at = (changed_at if lifecycle_status != "open" else None) - conn.execute( - """ - INSERT INTO attention_items ( - host_id, attention_id, source, kind, severity, status, updated_at, - fingerprint, snapshot_content_fingerprint, observed_at, - first_seen_at, last_seen_at, last_changed_at, resolved_at, - lifecycle_status, resolved_reason, signal_count, payload_json - ) VALUES ( - 'legacy-host', ?, ?, 'worker_status', ?, ?, ?, ?, 'snapshot-fp', ?, - ?, ?, ?, ?, ?, ?, ?, ? - ) - """, - ( - attention_id, - source, - severity, - status, - at, - f"fp-{attention_id}", - at, - first_seen_at or at, - seen_at, - changed_at, - resolved_at, - lifecycle_status, - "gone" if lifecycle_status != "open" else None, - signal_count, - json.dumps(payload, sort_keys=True), - ), - ) -def test_store_v4_collision_migration_is_deterministic_and_preserves_audit( - tmp_path: Path, -) -> None: - winners: list[tuple[Any, ...]] = [] - for reverse in (False, True): - db_path = tmp_path / f"collision-{reverse}.db" - _reset_store_to_v4(db_path) - candidates = [ - { - "attention_id": "attn-blocked", - "severity": "warning", - "status": "blocked", - "lifecycle_status": "open", - "at": "2026-01-01T00:01:00+00:00", - "first_seen_at": "2026-01-01T00:00:00+00:00", - "signal_count": 3, - }, - { - "attention_id": "attn-failed", - "severity": "critical", - "status": "failed", - "lifecycle_status": "open", - "at": "2026-01-01T00:02:00+00:00", - "first_seen_at": "2026-01-01T00:01:00+00:00", - "signal_count": 4, - }, - { - "attention_id": "attn-resolved", - "severity": "critical", - "status": "failed", - "lifecycle_status": "resolved", - "at": "2026-01-01T00:03:00+00:00", - "signal_count": 2, - }, - ] - with sqlite3.connect(str(db_path)) as conn: - for candidate in ( - reversed(candidates) if reverse else candidates - ): - _insert_legacy_attention( - conn, - source="worker:legacy", - **candidate, - ) - init_store(db_path) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - lifecycle = conn.execute( - """ - SELECT generation, lifecycle_status, current_attention_id, - first_seen_at, last_positive_at, - max_notified_severity_rank - FROM attention_lifecycles - """ - ).fetchone() - public_rows = conn.execute( - """ - SELECT attention_id, lifecycle_status, resolved_reason, - signal_count - FROM attention_items ORDER BY attention_id - """ - ).fetchall() - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - winners.append((lifecycle, public_rows)) - assert winners[0] == winners[1] - lifecycle, public_rows = winners[0] - assert lifecycle == ( - 1, - "resolved", - None, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:03:00+00:00", - 2, - ) - assert public_rows == [ - ("attn-blocked", "resolved", "superseded", 3), - ("attn-failed", "resolved", "superseded", 4), - ("attn-resolved", "resolved", "gone", 2), - ] +def _worker_binding( + *, + worker_id: str = "worker-1", + worker_fingerprint: str = "fp-1", + target_kind: str = "pane_id", + target_value: str = "pane-1", + private_fingerprint: str = "priv-1", + sendable: bool = True, + reason: str | None = None, + observed_at: str = "2026-01-01T00:00:00+00:00", + expires_at: str = "2026-01-02T00:00:00+00:00", +) -> WorkerBinding: + return WorkerBinding( + host_id="host-a", + worker_id=worker_id, + worker_fingerprint=worker_fingerprint, + backend="herdr", + target_kind=target_kind, + target_value=target_value, + turn_target_kind=None, + turn_target_value=None, + sendable=sendable, + reason=reason, + observed_at=observed_at, + expires_at=expires_at, + private_fingerprint=private_fingerprint, + ) -_MIG_T0 = "2026-01-01T00:00:00+00:00" -_MIG_T5 = "2026-01-01T00:05:00+00:00" -_MIG_T10 = "2026-01-01T00:10:00+00:00" -_MIG_T11 = "2026-01-01T00:11:00+00:00" +def test_store_worker_binding_upsert_list_resolve_and_expire(tmp_path: Path) -> None: + db_path = tmp_path / "bindings.db" + first = _worker_binding() + moved = _worker_binding( + target_value="pane-2", + observed_at="2026-01-01T00:10:00+00:00", + ) -def _migrate_resolved_skewed_episode(db_path: Path) -> None: - """Legacy resolved episode whose resolution (t10) is newer than its last - positive (t0), migrated to v5.""" - _reset_store_to_v4(db_path) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-legacy", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="resolved", - at=_MIG_T0, - first_seen_at=_MIG_T0, - last_seen_at=_MIG_T0, - last_changed_at=_MIG_T10, - signal_count=2, - ) init_store(db_path) + assert upsert_worker_bindings(db_path, [first]) == 1 + assert upsert_worker_bindings(db_path, [moved]) == 1 - -def _migrated_lifecycle_row(db_path: Path) -> tuple[Any, ...]: - with sqlite3.connect(str(db_path)) as conn: - return conn.execute( - """ - SELECT generation, lifecycle_status, current_attention_id, - last_positive_at, last_accepted_at - FROM attention_lifecycles - """ - ).fetchone() - - -def _attention_outbox_job_count(db_path: Path) -> int: - with sqlite3.connect(str(db_path)) as conn: - return int( - conn.execute( - "SELECT COUNT(*) FROM connector_outbox WHERE connector = 'attention'" - ).fetchone()[0] - ) - - -def _legacy_worker_snapshot(status: str, timestamp: str): - from datetime import datetime - - config = Config(host_id="legacy-host", db_path=Path("unused")) - return project_from_raw( - config, - workers=[{"id": "legacy", "name": "Legacy Worker", "status": status}], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": timestamp, - "counts": {"workers": 1}, - } - ], - timestamp=datetime.fromisoformat(timestamp), + current = list_worker_bindings( + db_path, + "host-a", + backend="herdr", + now="2026-01-01T00:30:00+00:00", ) - - -def test_migration_resolved_episode_seeds_accepted_watermark_from_resolution( - tmp_path: Path, -) -> None: - """The migrated watermark is the resolution progress (t10), not the last - positive (t0), and a delayed positive at t5 cannot reopen the lifecycle.""" - db_path = tmp_path / "skewed-resolved.db" - _migrate_resolved_skewed_episode(db_path) - - assert _migrated_lifecycle_row(db_path) == ( - 1, - "resolved", - None, - _MIG_T0, # last_positive_at = actual latest positive - _MIG_T10, # last_accepted_at = newest lifecycle progress (resolution) + assert len(current) == 1 + assert current[0].target_value == "pane-2" + assert current[0].worker_id == "worker-1" + expired_count = expire_worker_bindings( + db_path, + "host-a", + backend="herdr", + private_fingerprints=["priv-1"], + now="2026-01-01T00:45:00+00:00", + reason="stale_target", ) - assert _attention_outbox_job_count(db_path) == 0 - - # A delayed positive observation timestamped t5 (< the authoritative t10 - # resolution) must be ignored: no reopen, no generation 2, no job. - save_snapshot( + assert expired_count == 1 + assert list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:46:00+00:00") == [] + history = list_worker_bindings( db_path, - _legacy_worker_snapshot("blocked", _MIG_T5), - observation=SnapshotObservationContext(authority="positive", observed_at=_MIG_T5), + "host-a", + backend="herdr", + include_expired=True, + now="2026-01-01T00:46:00+00:00", ) - assert _migrated_lifecycle_row(db_path) == (1, "resolved", None, _MIG_T0, _MIG_T10) - assert _attention_outbox_job_count(db_path) == 0 - + assert len(history) == 1 + assert history[0].sendable is False + assert history[0].reason == "stale_target" -def test_migration_resolved_episode_genuine_later_positive_opens_one_generation( - tmp_path: Path, -) -> None: - """A genuine positive after the resolution watermark opens generation 2 and - enqueues exactly one notification.""" - db_path = tmp_path / "genuine-reopen.db" - _migrate_resolved_skewed_episode(db_path) - assert _attention_outbox_job_count(db_path) == 0 - save_snapshot( - db_path, - _legacy_worker_snapshot("blocked", _MIG_T11), - observation=SnapshotObservationContext(authority="positive", observed_at=_MIG_T11), +def test_store_worker_bindings_allow_duplicate_targets_and_expire_stale(tmp_path: Path) -> None: + db_path = tmp_path / "duplicate-bindings.db" + binding_a = _worker_binding( + worker_id="worker-a", + worker_fingerprint="fp-a", + private_fingerprint="priv-a", + target_value="same-pane", + sendable=False, + reason="duplicate_backend_target", ) - generation, status, current, _positive, accepted = _migrated_lifecycle_row(db_path) - assert (generation, status) == (2, "open") - assert current is not None - assert accepted == _MIG_T11 - assert _attention_outbox_job_count(db_path) == 1 - + binding_b = _worker_binding( + worker_id="worker-b", + worker_fingerprint="fp-b", + private_fingerprint="priv-b", + target_value="same-pane", + sendable=False, + reason="duplicate_backend_target", + ) + upsert_worker_bindings(db_path, [binding_a, binding_b]) -def test_migration_resolved_episode_delayed_complete_miss_is_inert( - tmp_path: Path, -) -> None: - """A delayed 'complete' (missing) observation before the resolution watermark - cannot mutate the migrated resolved lifecycle.""" - db_path = tmp_path / "delayed-complete.db" - _migrate_resolved_skewed_episode(db_path) + current = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:30:00+00:00") + assert len(current) == 2 + assert {binding.target_value for binding in current} == {"same-pane"} + assert {binding.reason for binding in current} == {"duplicate_backend_target"} - save_snapshot( + expired_count = expire_stale_worker_bindings( db_path, - _legacy_worker_snapshot("idle", _MIG_T5), - observation=SnapshotObservationContext(authority="complete", observed_at=_MIG_T5), + "host-a", + backend="herdr", + current_private_fingerprints=["priv-a"], + now="2026-01-01T00:40:00+00:00", + reason="stale_observation", ) - assert _migrated_lifecycle_row(db_path) == (1, "resolved", None, _MIG_T0, _MIG_T10) - assert _attention_outbox_job_count(db_path) == 0 + assert expired_count == 1 + remaining = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:41:00+00:00") + assert [binding.private_fingerprint for binding in remaining] == ["priv-a"] -def _legacy_attention_job_payload( - *, - event_type: str = "attention_created", - severity: str = "warning", - attention_id: str = "attn-legacy", - transition_at: str = "2026-01-01T00:00:00+00:00", -) -> str: - return json.dumps( - { - "schema_version": 1, - "event_type": event_type, - "host_id": "legacy-host", - "attention": { - "id": attention_id, - "source": "worker:legacy", - "kind": "worker_status", - "severity": severity, - "status": "blocked", - "fingerprint": "fp-attn-legacy", - }, - "transition_at": transition_at, - }, - sort_keys=True, +def test_store_upsert_separates_colliding_duplicate_private_fingerprints(tmp_path: Path) -> None: + db_path = tmp_path / "colliding-bindings.db" + binding_a = _worker_binding( + worker_id="worker-a", + worker_fingerprint="fp-a", + target_value="same-agent", + private_fingerprint="colliding-private", + ) + binding_b = _worker_binding( + worker_id="worker-b", + worker_fingerprint="fp-b", + target_value="same-agent", + private_fingerprint="colliding-private", ) + assert upsert_worker_bindings(db_path, [binding_a, binding_b]) == 2 -def test_store_v4_migration_consolidates_duplicate_jobs_and_preserves_terminal_audit( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-jobs.db" - _reset_store_to_v4(db_path) - created_payload = _legacy_attention_job_payload() - escalation_payload = _legacy_attention_job_payload( - event_type="attention_escalated", severity="critical" + current = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:30:00+00:00") + assert len(current) == 2 + assert {binding.worker_id for binding in current} == {"worker-a", "worker-b"} + assert {binding.sendable for binding in current} == {False} + assert {binding.reason for binding in current} == {"duplicate_backend_target"} + assert "colliding-private" not in {binding.private_fingerprint for binding in current} + assert len({binding.private_fingerprint for binding in current}) == 2 + + +def test_store_snapshot_payload_does_not_contain_private_worker_bindings(tmp_path: Path) -> None: + db_path = tmp_path / "payload-clean.db" + config = Config(host_id="host-a", db_path=db_path) + snapshot = project_from_raw( + config, + workers=[{"id": "worker-1", "name": "Worker", "status": "active"}], ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-legacy", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:00:00+00:00", - ) - outbox_ids: dict[str, int] = {} - for index, status in enumerate( - ("queued", "retry", "deferred", "delivered", "dead_letter") - ): - cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, ?, ?, '{}', ?, ?, NULL) - """, - ( - f"legacy-created-{index}", - status, - created_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - outbox_ids[status] = int(cursor.lastrowid) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'legacy-created-3', 1, - 'delivered', '{}', '{}', ?, ?) - """, - ( - outbox_ids["delivered"], - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - leased_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'legacy-created-leased', - 'leased', ?, '{}', ?, ?, NULL) - """, - ( - created_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'legacy-created-leased', 1, - 'leased', '{}', '{}', ?, NULL) - """, - ( - int(leased_cursor.lastrowid), - "2026-01-01T00:00:00+00:00", - ), - ) - for index in range(8): - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, 'queued', ?, '{}', ?, ?, NULL) - """, - ( - f"legacy-escalation-{index}", - escalation_payload, - "2026-01-01T00:01:00+00:00", - "2026-01-01T00:01:00+00:00", - ), - ) - - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - created_statuses = conn.execute( - """ - SELECT status, private_state_json FROM connector_outbox - WHERE delivery_key LIKE 'legacy-created-%' - ORDER BY id - """ - ).fetchall() - escalation_statuses = conn.execute( - """ - SELECT status, delivery_key FROM connector_outbox - WHERE json_extract(payload_json, '$.event_type') = 'attention_escalated' - ORDER BY id - """ - ).fetchall() - delivered_count = conn.execute( - "SELECT COUNT(*) FROM connector_deliveries WHERE status = 'delivered'" - ).fetchone()[0] - assert created_statuses == [ - ("superseded", "{}"), - ("superseded", "{}"), - ("superseded", "{}"), - ("delivered", "{}"), - ("dead_letter", "{}"), - ( - "leased", - next( - private - for status, private in created_statuses - if status == "leased" - ), - ), - ] - leased_state = json.loads(created_statuses[-1][1]) - assert leased_state["migration_canonical"] is False - assert leased_state["terminal_after_lease"] is True - assert sum(status == "queued" for status, _ in escalation_statuses) == 1 - assert sum(status == "superseded" for status, _ in escalation_statuses) == 8 - assert delivered_count == 1 - - -def test_store_v4_delivered_old_does_not_suppress_active_current_recurrence( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-delivered-old.db" - _reset_store_to_v4(db_path) - old_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:00:00+00:00", - ) - current_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - first_seen_at="2026-01-01T00:00:00+00:00", - ) - delivered_ids: list[int] = [] - for key in ("old-delivered-audit", "old-delivered-outbox-only"): - cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, 'delivered', ?, '{}', - ?, ?, NULL) - """, - ( - key, - old_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - delivered_ids.append(int(cursor.lastrowid)) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'old-delivered-audit', 1, - 'delivered', '{}', '{}', ?, ?) - """, - ( - delivered_ids[0], - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'current-recurrence', - 'queued', ?, '{}', ?, ?, NULL) - """, - ( - current_payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - """ - SELECT delivery_key, status, payload_json - FROM connector_outbox ORDER BY id - """ - ).fetchall() - assert [row[1] for row in rows] == [ - "delivered", - "delivered", - "superseded", - "queued", - ] - assert json.loads(rows[-1][2])["attention"]["id"] == "attn-current" - assert rows[-1][0].startswith("attention:attention_created:") - - -@pytest.mark.parametrize("with_delivery_audit", [False, True]) -def test_store_v4_delivered_current_episode_suppresses_duplicate( - tmp_path: Path, - with_delivery_audit: bool, -) -> None: - db_path = tmp_path / f"migration-delivered-current-{with_delivery_audit}.db" - _reset_store_to_v4(db_path) - payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - ) - delivered_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'current-delivered', - 'delivered', ?, '{}', ?, ?, NULL) - """, - ( - payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - if with_delivery_audit: - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, - status, response_json, private_state_json, created_at, - delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'current-delivered', - 1, 'delivered', '{}', '{}', ?, ?) - """, - ( - int(delivered_cursor.lastrowid), - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'current-duplicate', - 'queued', ?, '{}', ?, ?, NULL) - """, - ( - payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - statuses = conn.execute( - "SELECT status FROM connector_outbox ORDER BY id" - ).fetchall() - assert statuses == [("delivered",), ("superseded",)] - - -@pytest.mark.parametrize( - ("dead_at", "expected_statuses"), - [ - ( - "2026-01-01T00:00:00+00:00", - [("dead_letter",), ("superseded",), ("queued",)], - ), - ( - "2026-01-01T00:10:00+00:00", - [("dead_letter",), ("superseded",)], - ), - ], -) -def test_store_v4_dead_letter_suppresses_only_proven_current_episode( - tmp_path: Path, - dead_at: str, - expected_statuses: list[tuple[str]], -) -> None: - db_path = tmp_path / f"migration-dead-{dead_at[14:19].replace(':', '-')}.db" - _reset_store_to_v4(db_path) - dead_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at=dead_at, - ) - current_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - first_seen_at="2026-01-01T00:00:00+00:00", - ) - for key, status, payload in ( - ("current-dead", "dead_letter", dead_payload), - ("current-active", "queued", current_payload), - ): - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, ?, ?, '{}', ?, ?, NULL) - """, - ( - key, - status, - payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - statuses = conn.execute( - "SELECT status FROM connector_outbox ORDER BY id" - ).fetchall() - assert statuses == expected_statuses - - -def test_store_v4_resolved_lifecycle_terminalizes_all_active_jobs( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-resolved-active.db" - _reset_store_to_v4(db_path) - payload = _legacy_attention_job_payload( - attention_id="attn-resolved", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-resolved", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="resolved", - at="2026-01-01T00:10:00+00:00", - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'resolved-active', 'queued', - ?, '{}', ?, ?, NULL) - """, - ( - payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - lifecycle_status = conn.execute( - "SELECT lifecycle_status FROM attention_lifecycles" - ).fetchone()[0] - outbox_status = conn.execute( - "SELECT status FROM connector_outbox" - ).fetchone()[0] - assert (lifecycle_status, outbox_status) == ("resolved", "superseded") - - - - -@pytest.mark.parametrize("terminal_action", ["fail", "defer", "expiry"]) -def test_store_v4_current_pollable_outranks_stale_live_lease( - tmp_path: Path, - terminal_action: str, -) -> None: - db_path = tmp_path / f"migration-stale-lease-{terminal_action}.db" - init_store(db_path) - old_payload = _legacy_attention_job_payload( - attention_id="attn-old", - transition_at="2026-01-01T00:00:00+00:00", - ) - current_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'old-leased', 'queued', - ?, '{}', ?, ?, NULL) - """, - ( - old_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - old_item = poll_connector_outbox( - db_path, - "legacy-host", - "attention", - lease_seconds=30, - now="2026-01-01T00:00:00+00:00", - )["items"][0] - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'current-queued', 'queued', - ?, '{}', ?, ?, NULL) - """, - ( - current_payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - conn.execute("DROP TABLE attention_lifecycles") - conn.execute("PRAGMA user_version = 4") - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - """ - SELECT delivery_key, status, private_state_json - FROM connector_outbox ORDER BY id - """ - ).fetchall() - assert [row[1] for row in rows] == ["leased", "superseded", "queued"] - stale_state = json.loads(rows[0][2]) - assert stale_state["migration_canonical"] is False - assert stale_state["terminal_after_lease"] is True - if terminal_action == "fail": - result = fail_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=old_item["ref"], - now="2026-01-01T00:00:10+00:00", - ) - assert result["status"] == "superseded" - elif terminal_action == "defer": - result = defer_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=old_item["ref"], - now="2026-01-01T00:00:10+00:00", - ) - assert result["status"] == "superseded" - else: - result = reclaim_expired_connector_leases( - db_path, - "legacy-host", - "attention", - now="2026-01-01T00:00:31+00:00", - ) - assert result["reclaimed"] == 1 - with sqlite3.connect(str(db_path)) as conn: - statuses = conn.execute( - "SELECT status, COUNT(*) FROM connector_outbox GROUP BY status" - ).fetchall() - assert dict(statuses) == {"queued": 1, "superseded": 2} - - -def test_store_v4_generated_flap_damage_migrates_bounded_and_idempotent( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-generated-flap.db" - _reset_store_to_v4(db_path) - current_payload = _legacy_attention_job_payload( - attention_id="attn-current", - transition_at="2026-01-01T00:10:00+00:00", - ) - old_payload = _legacy_attention_job_payload( - attention_id="attn-old", - transition_at="2026-01-01T00:00:00+00:00", - ) - with sqlite3.connect(str(db_path)) as conn: - for index in range(509): - _insert_legacy_attention( - conn, - attention_id=f"attn-flap-{index:04d}", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:00:00+00:00", - signal_count=2, - ) - _insert_legacy_attention( - conn, - attention_id="attn-current", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:10:00+00:00", - signal_count=172, - ) - for index in range(600): - status = ("queued", "retry", "deferred")[index % 3] - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, ?, ?, '{}', ?, ?, NULL) - """, - ( - f"flap-active-{index:04d}", - status, - current_payload, - "2026-01-01T00:10:00+00:00", - "2026-01-01T00:10:00+00:00", - ), - ) - delivered_cursor = conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'flap-old-delivered', - 'delivered', ?, '{}', ?, ?, NULL) - """, - ( - old_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_deliveries ( - outbox_id, host_id, connector, delivery_key, attempt, status, - response_json, private_state_json, created_at, delivered_at - ) VALUES (?, 'legacy-host', 'attention', 'flap-old-delivered', 1, - 'delivered', '{}', '{}', ?, ?) - """, - ( - int(delivered_cursor.lastrowid), - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:01+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'flap-old-dead', - 'dead_letter', ?, '{}', ?, ?, NULL) - """, - ( - old_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - - def migration_evidence() -> tuple[Any, ...]: - with sqlite3.connect(str(db_path)) as conn: - lifecycle = conn.execute( - """ - SELECT COUNT(*), lifecycle_status, current_attention_id - FROM attention_lifecycles - """ - ).fetchone() - attention = conn.execute( - """ - SELECT COUNT(*), - MAX(CASE WHEN lifecycle_status = 'open' - THEN signal_count ELSE 0 END) - FROM attention_items - """ - ).fetchone() - outbox = dict( - conn.execute( - "SELECT status, COUNT(*) FROM connector_outbox GROUP BY status" - ).fetchall() - ) - delivered_audit = conn.execute( - """ - SELECT COUNT(*) FROM connector_deliveries - WHERE status = 'delivered' - """ - ).fetchone()[0] - canonical_payload = conn.execute( - """ - SELECT payload_json FROM connector_outbox - WHERE status = 'queued' - """ - ).fetchall() - return ( - lifecycle, - attention, - outbox, - delivered_audit, - canonical_payload, - _user_version(conn), - ) - - init_store(db_path) - first = migration_evidence() - init_store(db_path) - second = migration_evidence() - with sqlite3.connect(str(db_path)) as conn: - integrity = conn.execute("PRAGMA integrity_check").fetchone()[0] - - assert first == second - lifecycle, attention, outbox, delivered_audit, canonical_payload, version = first - assert lifecycle == (1, "open", "attn-current") - assert attention == (510, 1190) - assert outbox == { - "dead_letter": 1, - "delivered": 1, - "queued": 1, - "superseded": 600, - } - assert delivered_audit == 1 - assert len(canonical_payload) == 1 - assert json.loads(canonical_payload[0][0])["attention"]["id"] == "attn-current" - assert version == store_sqlite.STORE_SCHEMA_VERSION - assert integrity == "ok" - - -def test_store_v4_migration_retains_single_and_multiple_live_leases_safely( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migration-leases.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - _insert_legacy_attention( - conn, - attention_id="attn-legacy", - source="worker:legacy", - severity="warning", - status="blocked", - lifecycle_status="open", - at="2026-01-01T00:00:00+00:00", - ) - created_payload = _legacy_attention_job_payload() - escalation_payload = _legacy_attention_job_payload( - event_type="attention_escalated", severity="critical" - ) - for index in range(5): - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', ?, 'queued', ?, '{}', ?, ?, NULL) - """, - ( - f"leased-created-{index}", - created_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - conn.execute( - """ - INSERT INTO connector_outbox ( - host_id, connector, delivery_key, status, payload_json, - private_state_json, created_at, updated_at, next_attempt_at - ) VALUES ('legacy-host', 'attention', 'leased-escalation', 'queued', - ?, '{}', ?, ?, NULL) - """, - ( - escalation_payload, - "2026-01-01T00:00:00+00:00", - "2026-01-01T00:00:00+00:00", - ), - ) - leased = poll_connector_outbox( - db_path, - "legacy-host", - "attention", - limit=6, - lease_seconds=600, - now="2026-01-01T00:00:00+00:00", - )["items"] - assert len(leased) == 6 - with sqlite3.connect(str(db_path)) as conn: - conn.execute("DROP TABLE attention_lifecycles") - conn.execute("PRAGMA user_version = 4") - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - leased_rows = conn.execute( - """ - SELECT id, delivery_key, private_state_json - FROM connector_outbox WHERE status = 'leased' ORDER BY id - """ - ).fetchall() - pollable = conn.execute( - """ - SELECT COUNT(*) FROM connector_outbox - WHERE status IN ('queued', 'retry', 'deferred') - """ - ).fetchone()[0] - assert len(leased_rows) == 6 - assert pollable == 0 - created_rows = [row for row in leased_rows if "created" in row[1]] - created_states = [json.loads(row[2]) for row in created_rows] - assert sum(bool(state["migration_canonical"]) for state in created_states) == 1 - assert sum(bool(state.get("terminal_after_lease")) for state in created_states) == 4 - escalation_state = json.loads( - next(row[2] for row in leased_rows if row[1] == "leased-escalation") - ) - assert escalation_state["migration_canonical"] is True - assert "terminal_after_lease" not in escalation_state - - items_by_key = {item["key"]: item for item in leased} - canonical_created = next( - row for row in created_rows if json.loads(row[2])["migration_canonical"] - ) - duplicate_created = [ - row - for row in created_rows - if json.loads(row[2]).get("terminal_after_lease") - ] - canonical_ack = ack_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key[canonical_created[1]]["ref"], - now="2026-01-01T00:00:05+00:00", - ) - assert canonical_ack["status"] == "acknowledged" - with sqlite3.connect(str(db_path)) as conn: - sibling_states = [ - json.loads(row[0]) - for row in conn.execute( - """ - SELECT private_state_json FROM connector_outbox - WHERE status = 'leased' - AND delivery_key LIKE 'leased-created-%' - """ - ).fetchall() - ] - assert len(sibling_states) == 4 - assert all(state["terminal_after_lease"] for state in sibling_states) - - failed = fail_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key[duplicate_created[0][1]]["ref"], - now="2026-01-01T00:00:10+00:00", - ) - deferred = defer_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key[duplicate_created[1][1]]["ref"], - now="2026-01-01T00:00:20+00:00", - ) - assert failed["status"] == deferred["status"] == "superseded" - with sqlite3.connect(str(db_path)) as conn: - expiring = duplicate_created[2] - delivery = conn.execute( - """ - SELECT id, private_state_json FROM connector_deliveries - WHERE outbox_id = ? AND status = 'leased' - """, - (expiring[0],), - ).fetchone() - delivery_state = json.loads(delivery[1]) - delivery_state["lease_expires_at"] = "2026-01-01T00:00:30+00:00" - conn.execute( - "UPDATE connector_deliveries SET private_state_json = ? WHERE id = ?", - (json.dumps(delivery_state, sort_keys=True), delivery[0]), - ) - reclaimed = reclaim_expired_connector_leases( - db_path, - "legacy-host", - "attention", - now="2026-01-01T00:01:00+00:00", - ) - assert reclaimed["reclaimed"] == 1 - duplicate_ack = ack_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key[duplicate_created[3][1]]["ref"], - now="2026-01-01T00:01:30+00:00", - ) - assert duplicate_ack["status"] == "acknowledged" - single_ack = ack_connector_delivery( - db_path, - host_id="legacy-host", - name="attention", - ref=items_by_key["leased-escalation"]["ref"], - now="2026-01-01T00:02:00+00:00", - ) - assert single_ack["status"] == canonical_ack["status"] == "acknowledged" - with sqlite3.connect(str(db_path)) as conn: - statuses = conn.execute( - "SELECT status, COUNT(*) FROM connector_outbox GROUP BY status" - ).fetchall() - assert dict(statuses) == {"delivered": 3, "superseded": 3} - - -def _worker_binding( - *, - worker_id: str = "worker-1", - worker_fingerprint: str = "fp-1", - target_kind: str = "pane_id", - target_value: str = "pane-1", - private_fingerprint: str = "priv-1", - sendable: bool = True, - reason: str | None = None, - observed_at: str = "2026-01-01T00:00:00+00:00", - expires_at: str = "2026-01-02T00:00:00+00:00", -) -> WorkerBinding: - return WorkerBinding( - host_id="host-a", - worker_id=worker_id, - worker_fingerprint=worker_fingerprint, - backend="herdr", - target_kind=target_kind, - target_value=target_value, - turn_target_kind=None, - turn_target_value=None, - sendable=sendable, - reason=reason, - observed_at=observed_at, - expires_at=expires_at, - private_fingerprint=private_fingerprint, - ) - - -def test_store_worker_binding_upsert_list_resolve_and_expire(tmp_path: Path) -> None: - db_path = tmp_path / "bindings.db" - first = _worker_binding() - moved = _worker_binding( - target_value="pane-2", - observed_at="2026-01-01T00:10:00+00:00", - ) - - init_store(db_path) - assert upsert_worker_bindings(db_path, [first]) == 1 - assert upsert_worker_bindings(db_path, [moved]) == 1 - - current = list_worker_bindings( - db_path, - "host-a", - backend="herdr", - now="2026-01-01T00:30:00+00:00", - ) - assert len(current) == 1 - assert current[0].target_value == "pane-2" - assert current[0].worker_id == "worker-1" - resolved = resolve_worker_binding( - db_path, - "host-a", - "worker-1", - worker_fingerprint="fp-1", - backend="herdr", - now="2026-01-01T00:30:00+00:00", - ) - assert resolved is not None - assert resolved.target_value == "pane-2" - - expired_count = expire_worker_bindings( - db_path, - "host-a", - backend="herdr", - private_fingerprints=["priv-1"], - now="2026-01-01T00:45:00+00:00", - reason="stale_target", - ) - assert expired_count == 1 - assert list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:46:00+00:00") == [] - history = list_worker_bindings( - db_path, - "host-a", - backend="herdr", - include_expired=True, - now="2026-01-01T00:46:00+00:00", - ) - assert len(history) == 1 - assert history[0].sendable is False - assert history[0].reason == "stale_target" - assert resolve_worker_binding( - db_path, - "host-a", - "worker-1", - backend="herdr", - now="2026-01-01T00:46:00+00:00", - ) is None - - -def test_store_worker_bindings_allow_duplicate_targets_and_expire_stale(tmp_path: Path) -> None: - db_path = tmp_path / "duplicate-bindings.db" - binding_a = _worker_binding( - worker_id="worker-a", - worker_fingerprint="fp-a", - private_fingerprint="priv-a", - target_value="same-pane", - sendable=False, - reason="duplicate_backend_target", - ) - binding_b = _worker_binding( - worker_id="worker-b", - worker_fingerprint="fp-b", - private_fingerprint="priv-b", - target_value="same-pane", - sendable=False, - reason="duplicate_backend_target", - ) - upsert_worker_bindings(db_path, [binding_a, binding_b]) - - current = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:30:00+00:00") - assert len(current) == 2 - assert {binding.target_value for binding in current} == {"same-pane"} - assert {binding.reason for binding in current} == {"duplicate_backend_target"} - assert resolve_worker_binding( - db_path, - "host-a", - "worker-a", - backend="herdr", - now="2026-01-01T00:30:00+00:00", - ) is None - - expired_count = expire_stale_worker_bindings( - db_path, - "host-a", - backend="herdr", - current_private_fingerprints=["priv-a"], - now="2026-01-01T00:40:00+00:00", - reason="stale_observation", - ) - assert expired_count == 1 - remaining = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:41:00+00:00") - assert [binding.private_fingerprint for binding in remaining] == ["priv-a"] - - -def test_store_upsert_separates_colliding_duplicate_private_fingerprints(tmp_path: Path) -> None: - db_path = tmp_path / "colliding-bindings.db" - binding_a = _worker_binding( - worker_id="worker-a", - worker_fingerprint="fp-a", - target_value="same-agent", - private_fingerprint="colliding-private", - ) - binding_b = _worker_binding( - worker_id="worker-b", - worker_fingerprint="fp-b", - target_value="same-agent", - private_fingerprint="colliding-private", - ) - - assert upsert_worker_bindings(db_path, [binding_a, binding_b]) == 2 - - current = list_worker_bindings(db_path, "host-a", backend="herdr", now="2026-01-01T00:30:00+00:00") - assert len(current) == 2 - assert {binding.worker_id for binding in current} == {"worker-a", "worker-b"} - assert {binding.sendable for binding in current} == {False} - assert {binding.reason for binding in current} == {"duplicate_backend_target"} - assert "colliding-private" not in {binding.private_fingerprint for binding in current} - assert len({binding.private_fingerprint for binding in current}) == 2 - assert resolve_worker_binding( - db_path, - "host-a", - "worker-a", - backend="herdr", - now="2026-01-01T00:30:00+00:00", - ) is None - - -def test_store_snapshot_payload_does_not_contain_private_worker_bindings(tmp_path: Path) -> None: - db_path = tmp_path / "payload-clean.db" - config = Config(host_id="host-a", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "Worker", "status": "active"}], - ) - binding = _worker_binding(target_value="pane-secret", private_fingerprint="priv-secret") - - save_snapshot(db_path, snapshot) - upsert_worker_bindings(db_path, [binding]) - + binding = _worker_binding(target_value="pane-secret", private_fingerprint="priv-secret") + + save_snapshot(db_path, snapshot) + upsert_worker_bindings(db_path, [binding]) + with sqlite3.connect(str(db_path)) as conn: payload = conn.execute("SELECT payload FROM snapshots ORDER BY id DESC LIMIT 1").fetchone()[0] target_value = conn.execute("SELECT target_value FROM worker_bindings LIMIT 1").fetchone()[0] @@ -4701,7 +2760,7 @@ def test_store_merges_public_turn_content_without_private_labels(tmp_path: Path) init_store(db_path) save_snapshot(db_path, snapshot) - updated = merge_turn_content( + updated = apply_test_turn_refresh( db_path, "turn-host", "worker-1", @@ -4731,7 +2790,7 @@ def test_store_merges_public_turn_content_without_private_labels(tmp_path: Path) -def test_store_save_latest_host_scope_and_list_hosts(tmp_path: Path) -> None: +def test_store_save_and_latest_snapshot_are_host_scoped(tmp_path: Path) -> None: db_path = tmp_path / "tendwire.db" config_a = Config(host_id="host-a", db_path=db_path) config_b = Config(host_id="host-b", db_path=db_path) @@ -4774,7 +2833,6 @@ def test_store_save_latest_host_scope_and_list_hosts(tmp_path: Path) -> None: assert restored_b.workers[0].id == "worker-b" assert latest_snapshot(db_path, "missing-host") is None - assert list_hosts(db_path) == ["host-a", "host-b"] def _reserve_test_request( @@ -4853,7 +2911,7 @@ def test_store_host_wide_request_identity_conflicts_across_actions_and_tombstone collision = _reserve_test_request( db_path, request_id="same-id", - action="answer_pending", + action="answer_decision", fingerprint="answer-fingerprint", ) assert collision["status"] == "request_id_conflict" @@ -5038,7 +3096,7 @@ def test_backend_pending_choice_terminal_effect_is_atomic_with_acceptance( reservation = _reserve_test_request( db_path, request_id="choice-effect", - action="answer_pending", + action="answer_decision", fingerprint="choice-effect-fingerprint", ) started = mark_command_send_started( @@ -5112,7 +3170,7 @@ def test_backend_pending_choice_terminal_effect_is_atomic_with_acceptance( missing_reservation = _reserve_test_request( db_path, request_id="missing-choice", - action="answer_pending", + action="answer_decision", fingerprint="missing-choice-fingerprint", now="2026-01-01T00:01:00+00:00", ) @@ -5521,255 +3579,18 @@ def attempt() -> None: assert not any(thread.is_alive() for thread in threads) assert sorted(result["status"] for result in results) == [ "in_progress", - "reserved", - ] - assert sum(result["owner_token"] is not None for result in results) == 1 - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 1 - assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 1 - - -def test_store_v11_host_request_collision_migrates_to_uncertain_tombstone( - tmp_path: Path, -) -> None: - db_path = tmp_path / "legacy-collision.db" - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - for index, (action, fingerprint) in enumerate( - ( - ("send_instruction", "send-fingerprint"), - ("answer_pending", "answer-fingerprint"), - ) - ): - created = f"2026-01-01T00:00:0{index}+00:00" - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES (?, 'collision', ?, ?, 'accepted', ?, ?, ?, 0) - """, - ( - "host-a", - action, - fingerprint, - '{"ok":true,"private":"must-not-survive"}', - created, - created, - ), - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES (?, 'collision', ?, ?, 'accepted', 0, 0, ?, ?, ?, ?, ?, ?) - """, - ( - "host-a", - action, - fingerprint, - '{"target":{"worker_id":"public","worker_fingerprint":"private"}}', - '{"ok":true,"private":"must-not-survive"}', - created, - created, - created, - created, - ), - ) - conn.execute("PRAGMA user_version = 11") - - init_store(db_path) - receipt = get_command_request(db_path, "host-a", "collision") - assert receipt is not None - assert receipt["state"] == "uncertain" - assert receipt["status"] == "request_state_uncertain" - assert receipt["legacy_collision"] is True - assert receipt["legacy_collision_count"] == 4 - assert receipt["canonical_request_json"] == "{}" - assert "must-not-survive" not in receipt["result_json"] - assert _reserve_test_request( - db_path, - request_id="collision", - fingerprint="new-fingerprint", - )["status"] == "terminal" - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - "SELECT state, legacy_collision FROM commands" - ).fetchone() == ("uncertain", 1) - assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" - conn.execute(store_sqlite.CREATE_EVENTS_TABLE) - - _accept_test_request( - db_path, - request_id="newer-than-collision", - now="2026-02-02T00:00:00+00:00", - ) - cleanup = cleanup_command_request_retention( - db_path, - retry_horizon_seconds=604_800, - retention_seconds=2_592_000, - retention_count=1, - now="2026-03-05T00:00:00+00:00", - ) - assert cleanup["deleted"] == 1 - assert get_command_request(db_path, "host-a", "collision") is None - assert get_command_request( - db_path, "host-a", "newer-than-collision" - ) is not None - - - -def test_store_v11_noncollision_replays_only_exact_legacy_raw_fingerprint( - tmp_path: Path, -) -> None: - db_path = tmp_path / "legacy-noncollision.db" - legacy_fingerprint = "legacy-raw-payload-fingerprint" - canonical_json = '{"action":"send_instruction","worker_id":"worker-public"}' - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES ( - 'host-a', 'legacy-exact', 'send_instruction', ?, 'accepted', - '{"ok":true,"status":"accepted"}', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', 0 - ) - """, - (legacy_fingerprint,), - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES ( - 'host-a', 'legacy-exact', 'send_instruction', ?, 'accepted', - 0, 0, '{"target":{"worker_id":"worker-public"}}', - '{"ok":true,"status":"accepted"}', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', - '2026-01-01T00:00:01+00:00' - ) - """, - (legacy_fingerprint,), - ) - conn.execute("PRAGMA user_version = 11") - - init_store(db_path) - canonical_only = reserve_command_request( - db_path, - host_id="host-a", - request_id="legacy-exact", - action="send_instruction", - canonical_version=1, - canonical_fingerprint=legacy_fingerprint, - canonical_request_json=canonical_json, - public_worker_id="worker-public", - pending_result_json='{"ok":false,"status":"pending"}', - ) - assert canonical_only["status"] == "request_id_conflict" - - exact_replay = reserve_command_request( - db_path, - host_id="host-a", - request_id="legacy-exact", - action="send_instruction", - canonical_version=1, - canonical_fingerprint="new-canonical-fingerprint", - canonical_request_json=canonical_json, - public_worker_id="worker-public", - pending_result_json='{"ok":false,"status":"pending"}', - legacy_raw_payload_fingerprint=legacy_fingerprint, - ) - assert exact_replay["status"] == "terminal" - assert exact_replay["receipt"]["canonical_version"] == 0 - assert exact_replay["receipt"]["canonical_fingerprint"] == legacy_fingerprint - assert exact_replay["receipt"]["result_json"] == ( - '{"ok":true,"status":"accepted"}' - ) - - wrong_raw = reserve_command_request( - db_path, - host_id="host-a", - request_id="legacy-exact", - action="send_instruction", - canonical_version=1, - canonical_fingerprint="new-canonical-fingerprint", - canonical_request_json=canonical_json, - public_worker_id="worker-public", - pending_result_json='{"ok":false,"status":"pending"}', - legacy_raw_payload_fingerprint="different-legacy-raw-fingerprint", - ) - wrong_action = reserve_command_request( - db_path, - host_id="host-a", - request_id="legacy-exact", - action="answer_pending", - canonical_version=1, - canonical_fingerprint="new-canonical-fingerprint", - canonical_request_json=canonical_json, - public_worker_id="worker-public", - pending_result_json='{"ok":false,"status":"pending"}', - legacy_raw_payload_fingerprint=legacy_fingerprint, - ) - assert wrong_raw["status"] == "request_id_conflict" - assert wrong_action["status"] == "request_id_conflict" - -def test_store_v11_receipt_audit_disagreement_fails_closed(tmp_path: Path) -> None: - db_path = tmp_path / "legacy-disagreement.db" - with sqlite3.connect(str(db_path)) as conn: - conn.executescript( - store_sqlite.CREATE_LEGACY_COMMAND_RECEIPTS_TABLE - + store_sqlite.CREATE_LEGACY_COMMANDS_TABLE - ) - conn.execute( - """ - INSERT INTO command_receipts ( - host_id, request_id, action, payload_fingerprint, status, - result_json, created_at, completed_at, uncertain - ) VALUES ( - 'host-a', 'disagree', 'send_instruction', 'same-fingerprint', - 'accepted', '{"ok":true}', '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', 0 - ) - """ - ) - conn.execute( - """ - INSERT INTO commands ( - host_id, request_id, action, payload_fingerprint, status, - dry_run, uncertain, request_json, result_json, created_at, - reserved_at, completed_at, updated_at - ) VALUES ( - 'host-a', 'disagree', 'send_instruction', 'same-fingerprint', - 'backend_failed', 0, 0, '{}', '{"ok":false}', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', - '2026-01-01T00:00:01+00:00' - ) - """ - ) - conn.execute("PRAGMA user_version = 11") - init_store(db_path) - receipt = get_command_request(db_path, "host-a", "disagree") - assert receipt is not None - assert receipt["state"] == "uncertain" - assert receipt["legacy_collision"] is True + "reserved", + ] + assert sum(result["owner_token"] is not None for result in results) == 1 + with sqlite3.connect(str(db_path)) as conn: + assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone()[0] == 1 + assert conn.execute("SELECT COUNT(*) FROM commands").fetchone()[0] == 1 + + + + + + def test_store_command_retention_obeys_age_count_host_and_batch_floors( @@ -6328,7 +4149,7 @@ def test_distinct_source_turns_mint_distinct_public_turn_ids(tmp_path: Path) -> [("first question", "first answer"), ("second question", "second answer")], start=1, ): - merge_turn_content( + apply_test_turn_refresh( db_path, "turn-host", "worker-1", @@ -6352,7 +4173,7 @@ def test_distinct_source_turns_mint_distinct_public_turn_ids(tmp_path: Path) -> assert all(not t.get("assistant_final_text") and not t.get("user_text") for t in base_rows) # Same source turn observed again updates its row, keeping the id stable. - merge_turn_content( + apply_test_turn_refresh( db_path, "turn-host", "worker-1", @@ -6421,473 +4242,95 @@ def test_twenty_offline_source_finals_are_retained_as_unique_ready_anchors( init_store(db_path) save_snapshot(db_path, snapshot) for index in range(20): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "turn-host", "worker-1", { - "assistant_final_text": f"answer {index}", - "complete": True, - "source_turn_id": f"uuid-{index}", - }, - observed_at=f"2026-01-01T00:{index:02d}:00+00:00", - ) == 1 - - payload = turns_payload_from_store(db_path, "turn-host", snapshot=snapshot) - source_rows = [turn for turn in payload["turns"] if turn.get("source_turn_id")] - with sqlite3.connect(str(db_path)) as conn: - anchors = conn.execute( - """ - SELECT delivery_key, payload_json - FROM connector_outbox - WHERE host_id = ? - AND delivery_kind = 'final_ready' - AND status = 'queued' - ORDER BY id - """, - ("turn-host",), - ).fetchall() - - assert len(source_rows) == 20 - assert len(anchors) == len({row[0] for row in anchors}) == 20 - assert source_rows[0]["assistant_final_text"] == "answer 19" - assert all( - row[0].startswith("turn-final:revision:twfinal1.") - for row in anchors - ) - encoded_anchors = "\n".join(row[1] for row in anchors) - assert "answer 0" not in encoded_anchors - assert "answer 19" not in encoded_anchors - assert "source_turn_id" not in encoded_anchors - - - - - - - - - - - - - - - - - -def _reset_store_to_v5_with_legacy_turn( - db_path: Path, - *, - final_text: str, -) -> tuple[Any, str]: - config = Config(host_id="legacy-turn-host", db_path=db_path) - snapshot = project_from_raw( - config, - workers=[ - { - "id": "worker-1", - "name": "claude", - "status": "active", - "space_id": "space-1", - } - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - assert merge_turn_content( - db_path, - "legacy-turn-host", - "worker-1", - { - "source_turn_id": "v5-migration-source", - "user_text": "legacy prompt", - "assistant_final_text": final_text, - "complete": True, - "has_open_turn": False, - }, - ) == 1 - with sqlite3.connect(str(db_path)) as conn: - turn_id, payload_json = conn.execute( - """ - SELECT turn_id, payload_json - FROM turns - WHERE host_id = 'legacy-turn-host' - """ - ).fetchone() - payload = json.loads(payload_json) - payload["user_text"] = "legacy prompt" - payload["assistant_final_text"] = final_text - payload["complete"] = True - payload["has_open_turn"] = False - conn.execute( - "UPDATE turns SET payload_json = ? WHERE host_id = ? AND turn_id = ?", - ( - json.dumps(payload, sort_keys=True), - "legacy-turn-host", - str(turn_id), - ), - ) - conn.execute("DROP TABLE turn_presentation_recoveries") - conn.execute("DROP TABLE turn_presentation_jobs") - conn.execute("DROP TABLE turn_presentation_plans") - conn.execute("DROP TABLE turn_content_page_boundaries") - conn.execute("DROP TABLE turn_content_revisions") - conn.execute("CREATE TABLE preserved_v5 (value TEXT NOT NULL)") - conn.execute("INSERT INTO preserved_v5 (value) VALUES ('untouched')") - conn.execute("PRAGMA user_version = 5") - return snapshot, str(turn_id) - - -def _reconstruct_turn_content( - db_path: Path, - *, - host_id: str, - turn_id: str, - revision: str, - field: str, - work_counters: store_sqlite.TurnContentWorkCounters | None = None, -) -> tuple[str, list[dict[str, Any]]]: - cursor: str | None = None - pages: list[dict[str, Any]] = [] - while True: - page = store_sqlite.get_turn_content( - db_path, - host_id, - turn_id=turn_id, - content_revision=revision, - field=field, - cursor=cursor, - work_counters=work_counters, - ) - assert page.get("status") is None - pages.append(page) - cursor = page["next_cursor"] - if cursor is None: - break - return "".join(str(page["text"]) for page in pages), pages - - -def test_store_v5_to_v6_migration_is_atomic_idempotent_and_marks_incomplete( - tmp_path: Path, -) -> None: - db_path = tmp_path / "turn-v5.db" - fragment = ("x" * 11_988) + "\n[truncated]" - snapshot, turn_id = _reset_store_to_v5_with_legacy_turn( - db_path, - final_text=fragment, - ) - - init_store(db_path) - first_v2 = turns_payload_from_store( - db_path, - "legacy-turn-host", - snapshot=snapshot, - schema_version=2, - ) - init_store(db_path) - second_v2 = turns_payload_from_store( - db_path, - "legacy-turn-host", - snapshot=snapshot, - schema_version=2, - ) - - with sqlite3.connect(str(db_path)) as conn: - version = _user_version(conn) - tables = _table_names(conn) - revision_rows = conn.execute( - """ - SELECT - content_revision, user_text, assistant_final_text, - user_state, final_state, user_page_count, final_page_count, - is_current - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? - """, - ("legacy-turn-host", turn_id), - ).fetchall() - stored_payload = conn.execute( - "SELECT payload_json FROM turns WHERE host_id = ? AND turn_id = ?", - ("legacy-turn-host", turn_id), - ).fetchone()[0] - preserved = conn.execute("SELECT value FROM preserved_v5").fetchone()[0] - integrity = conn.execute("PRAGMA integrity_check").fetchone()[0] - foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - - assert version == store_sqlite.STORE_SCHEMA_VERSION - assert { - "turn_content_revisions", - "turn_presentation_plans", - "turn_presentation_jobs", - } <= tables - assert len(revision_rows) == 1 - revision = revision_rows[0] - assert revision[1:] == ( - "legacy prompt", - fragment, - "complete", - "known_incomplete", - 1, - 0, - 1, - ) - assert json.loads(stored_payload).get("assistant_final_text") is None - assert fragment not in stored_payload - assert preserved == "untouched" - assert integrity == "ok" - assert foreign_keys == [] - assert first_v2 == second_v2 - turn = next(item for item in first_v2["turns"] if item["id"] == turn_id) - assert turn["content"]["known_incomplete"] is True - assert turn["content"]["fields"]["assistant_final_text"] == { - "availability": "known_incomplete", - "inline": False, - "char_length": len(fragment), - "byte_length": len(fragment.encode("utf-8")), - "page_count": 0, - "first_cursor": None, - } - assert "assistant_final_text" not in turn - assert turn["assistant_final_preview"] == fragment[:1000] - assert turns_payload_from_store( - db_path, - "legacy-turn-host", - schema_version=1, - ) == { - "schema_version": 1, - "ok": False, - "status": "upgrade_required", - "required_turn_schema_version": 2, - } - assert store_sqlite.get_turn_content( - db_path, - "legacy-turn-host", - turn_id=turn_id, - content_revision=revision[0], - field="assistant_final_text", - ) == { - "schema_version": 1, - "ok": False, - "status": "content_known_incomplete", - } - -def test_store_v5_to_v6_migration_rolls_back_all_v6_ddl( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "turn-v5-rollback.db" - _reset_store_to_v5_with_legacy_turn(db_path, final_text="legacy answer") - - def fail_backfill(conn: sqlite3.Connection) -> None: - raise RuntimeError("controlled v6 migration failure") - - monkeypatch.setattr( - store_sqlite, - "_backfill_legacy_turn_content_conn", - fail_backfill, - ) - with pytest.raises(RuntimeError, match="controlled v6 migration failure"): - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == 5 - assert "turn_content_revisions" not in _table_names(conn) - assert "turn_content_page_boundaries" not in _table_names(conn) - assert "turn_presentation_plans" not in _table_names(conn) - assert "turn_presentation_jobs" not in _table_names(conn) - assert "turn_presentation_recoveries" not in _table_names(conn) - assert conn.execute("SELECT value FROM preserved_v5").fetchone()[0] == "untouched" - - -def test_v6_to_v7_repairs_mixed_turns_with_absent_content_descriptors( - tmp_path: Path, -) -> None: - db_path = tmp_path / "mixed-v6-turn-content.db" - host_id = "mixed-v6-host" - snapshot = project_from_raw( - Config(host_id=host_id, db_path=db_path), - workers=[ - {"id": "worker-complete", "name": "Complete", "status": "active"}, - {"id": "worker-working", "name": "Working", "status": "active"}, - ], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - assert merge_turn_content( - db_path, - host_id, - "worker-complete", - { - "source_turn_id": "complete-source", - "assistant_final_text": "complete final", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:00+00:00", - ) == 1 - assert merge_turn_content( - db_path, - host_id, - "worker-working", - { - "source_turn_id": "working-source", - "user_text": "observed working turn", - "complete": False, - "has_open_turn": True, - }, - observed_at="2026-01-01T00:01:00+00:00", - ) == 1 - - with sqlite3.connect(str(db_path)) as conn: - complete_turn_id = str( - conn.execute( - """ - SELECT turn_id - FROM turn_content_revisions - WHERE host_id = ? AND final_state = 'complete' AND is_current = 1 - """, - (host_id,), - ).fetchone()[0] - ) - missing_rows = conn.execute( - """ - SELECT turn_id, payload_json - FROM turns - WHERE host_id = ? AND turn_id != ? - ORDER BY turn_id - """, - (host_id, complete_turn_id), - ).fetchall() - assert len(missing_rows) == 1 - missing_turn_ids = [str(row[0]) for row in missing_rows] - for turn_id, payload_json in missing_rows: - payload = json.loads(payload_json) - payload["assistant_stream_text"] = "working progress" - conn.execute( - """ - UPDATE turns - SET payload_json = ? - WHERE host_id = ? AND turn_id = ? - """, - (json.dumps(payload, sort_keys=True), host_id, str(turn_id)), - ) - placeholders = ",".join("?" for _ in missing_turn_ids) - conn.execute( - f""" - DELETE FROM turn_content_page_boundaries - WHERE host_id = ? AND turn_id IN ({placeholders}) - """, - (host_id, *missing_turn_ids), - ) - conn.execute( - f""" - DELETE FROM turn_content_revisions - WHERE host_id = ? AND turn_id IN ({placeholders}) - """, - (host_id, *missing_turn_ids), - ) - conn.execute("DROP TABLE turn_content_page_boundaries") - conn.execute("PRAGMA user_version = 6") - - init_store(db_path) - first_v2 = turns_payload_from_store( - db_path, - host_id, - snapshot=snapshot, - schema_version=2, - ) - first_v1 = turns_payload_from_store( - db_path, - host_id, - snapshot=snapshot, - schema_version=1, - ) - with sqlite3.connect(str(db_path)) as conn: - first_rows = conn.execute( - """ - SELECT turn_id, content_revision, user_state, final_state, is_current - FROM turn_content_revisions - WHERE host_id = ? AND turn_id IN ( - SELECT turn_id - FROM turns - WHERE host_id = ? AND turn_id != ? - ) - ORDER BY turn_id, content_revision - """, - (host_id, host_id, complete_turn_id), - ).fetchall() - version = conn.execute("PRAGMA user_version").fetchone()[0] - init_store(db_path) - second_v2 = turns_payload_from_store( - db_path, - host_id, - snapshot=snapshot, - schema_version=2, - ) + "assistant_final_text": f"answer {index}", + "complete": True, + "source_turn_id": f"uuid-{index}", + }, + observed_at=f"2026-01-01T00:{index:02d}:00+00:00", + ) == 1 + + payload = turns_payload_from_store(db_path, "turn-host", snapshot=snapshot) + source_rows = [turn for turn in payload["turns"] if turn.get("source_turn_id")] with sqlite3.connect(str(db_path)) as conn: - second_rows = conn.execute( + anchors = conn.execute( """ - SELECT turn_id, content_revision, user_state, final_state, is_current - FROM turn_content_revisions - WHERE host_id = ? AND turn_id IN ( - SELECT turn_id - FROM turns - WHERE host_id = ? AND turn_id != ? - ) - ORDER BY turn_id, content_revision + SELECT delivery_key, payload_json + FROM connector_outbox + WHERE host_id = ? + AND delivery_kind = 'final_ready' + AND status = 'queued' + ORDER BY id """, - (host_id, host_id, complete_turn_id), + ("turn-host",), ).fetchall() - absent_field = { - "availability": "absent", - "inline": False, - "char_length": 0, - "byte_length": 0, - "page_count": 0, - "first_cursor": None, - } - assert version == store_sqlite.STORE_SCHEMA_VERSION - assert first_v2 == second_v2 - assert first_rows == second_rows - assert len(first_rows) == len(missing_turn_ids) + assert len(source_rows) == 20 + assert len(anchors) == len({row[0] for row in anchors}) == 20 + assert source_rows[0]["assistant_final_text"] == "answer 19" assert all( - row - == ( - row[0], - store_sqlite.content_revision( - str(row[0]), - None, - None, - "absent", - "absent", - ), - "absent", - "absent", - 1, - ) - for row in first_rows + row[0].startswith("turn-final:revision:twfinal1.") + for row in anchors ) - assert len(first_v2["turns"]) >= 2 - for turn in first_v2["turns"]: - assert turn["content"]["schema_version"] == 1 - if turn["id"] not in missing_turn_ids: - continue - assert turn["assistant_stream_text"] == "working progress" - assert turn["content"]["known_incomplete"] is False - assert turn["content"]["fields"] == { - "user_text": absent_field, - "assistant_final_text": absent_field, - } - assert "user_text" not in turn - assert "assistant_final_text" not in turn - assert first_v1["schema_version"] == 1 - assert all("content" not in turn for turn in first_v1["turns"]) - for turn in first_v1["turns"]: - if turn["id"] in missing_turn_ids: - assert turn["user_text"] is None - assert turn["assistant_final_text"] is None + encoded_anchors = "\n".join(row[1] for row in anchors) + assert "answer 0" not in encoded_anchors + assert "answer 19" not in encoded_anchors + assert "source_turn_id" not in encoded_anchors + + + + + + + + + + + + + + + + + + + +def _reconstruct_turn_content( + db_path: Path, + *, + host_id: str, + turn_id: str, + revision: str, + field: str, + work_counters: store_sqlite.TurnContentWorkCounters | None = None, +) -> tuple[str, list[dict[str, Any]]]: + cursor: str | None = None + pages: list[dict[str, Any]] = [] + while True: + page = store_sqlite.get_turn_content( + db_path, + host_id, + turn_id=turn_id, + content_revision=revision, + field=field, + cursor=cursor, + work_counters=work_counters, + ) + assert page.get("status") is None + pages.append(page) + cursor = page["next_cursor"] + if cursor is None: + break + return "".join(str(page["text"]) for page in pages), pages + + + + + def _exact_utf8_fixture(byte_length: int) -> str: @@ -6917,7 +4360,7 @@ def test_store_list_is_preview_bounded_and_sequential_pages_are_linear( save_snapshot(db_path, snapshot) final = _exact_utf8_fixture(target_byte_length) assert len(final.encode("utf-8")) == target_byte_length - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -6989,153 +4432,6 @@ def test_store_list_is_preview_bounded_and_sequential_pages_are_linear( ) -def test_migrated_v6_boundaries_make_first_long_read_page_bounded( - tmp_path: Path, -) -> None: - db_path = tmp_path / "migrated-v6-page-boundaries.db" - host_id = "legacy-boundary-host" - worker_id = "worker-boundary" - snapshot = project_from_raw( - Config(host_id=host_id, db_path=db_path), - workers=[{"id": worker_id, "name": "Boundary", "status": "active"}], - ) - init_store(db_path) - save_snapshot(db_path, snapshot) - final = _exact_utf8_fixture(1024 * 1024) - assert merge_turn_content( - db_path, - host_id, - worker_id, - { - "source_turn_id": "migrated-boundary-source", - "assistant_final_text": final, - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-01-01T00:00:00+00:00", - ) == 1 - listed = turns_payload_from_store( - db_path, - host_id, - snapshot=snapshot, - schema_version=2, - ) - turn = listed["turns"][0] - revision = turn["content"]["content_revision"] - page_count = turn["content"]["fields"]["assistant_final_text"]["page_count"] - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - DELETE FROM turn_content_page_boundaries - WHERE host_id = ? - AND turn_id = ? - AND content_revision = ? - AND field = 'assistant_final_text' - """, - (host_id, turn["id"], revision), - ) - conn.execute("PRAGMA user_version = 6") - - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - migrated_version = conn.execute("PRAGMA user_version").fetchone()[0] - migrated_boundaries = conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_page_boundaries - WHERE host_id = ? - AND turn_id = ? - AND content_revision = ? - AND field = 'assistant_final_text' - """, - (host_id, turn["id"], revision), - ).fetchone()[0] - first_counters = store_sqlite.TurnContentWorkCounters() - first_rebuilt, first_pages = _reconstruct_turn_content( - db_path, - host_id=host_id, - turn_id=turn["id"], - revision=revision, - field="assistant_final_text", - work_counters=first_counters, - ) - - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - DELETE FROM turn_content_page_boundaries - WHERE host_id = ? - AND turn_id = ? - AND content_revision = ? - AND field = 'assistant_final_text' - """, - (host_id, turn["id"], revision), - ) - init_store(db_path) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - current_boundaries = conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_page_boundaries - WHERE host_id = ? - AND turn_id = ? - AND content_revision = ? - AND field = 'assistant_final_text' - """, - (host_id, turn["id"], revision), - ).fetchone()[0] - incomplete_boundary_fields = conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_revisions AS revisions - WHERE ( - revisions.user_state = 'complete' - AND revisions.user_page_count != ( - SELECT COUNT(*) - FROM turn_content_page_boundaries AS boundaries - WHERE boundaries.host_id = revisions.host_id - AND boundaries.turn_id = revisions.turn_id - AND boundaries.content_revision = revisions.content_revision - AND boundaries.field = 'user_text' - ) - ) OR ( - revisions.final_state = 'complete' - AND revisions.final_page_count != ( - SELECT COUNT(*) - FROM turn_content_page_boundaries AS boundaries - WHERE boundaries.host_id = revisions.host_id - AND boundaries.turn_id = revisions.turn_id - AND boundaries.content_revision = revisions.content_revision - AND boundaries.field = 'assistant_final_text' - ) - ) - """ - ).fetchone()[0] - failed_page = store_sqlite.get_turn_content( - db_path, - host_id, - turn_id=turn["id"], - content_revision=revision, - field="assistant_final_text", - ) - - assert migrated_version == store_sqlite.STORE_SCHEMA_VERSION - assert migrated_boundaries == page_count - assert current_boundaries == 0 - assert incomplete_boundary_fields == 1 - assert first_rebuilt == final - assert len(first_pages) == page_count - assert first_counters.page_blob_reads == page_count - assert first_counters.page_chars_examined == len(final) - assert first_counters.page_bytes_examined <= ( - len(final.encode("utf-8")) + 3 * (page_count - 1) - ) - assert failed_page == { - "schema_version": 1, - "ok": False, - "status": "content_not_available", - } def test_many_long_turn_descriptors_do_one_bounded_list_query(tmp_path: Path) -> None: @@ -7152,7 +4448,7 @@ def test_many_long_turn_descriptors_do_one_bounded_list_query(tmp_path: Path) -> init_store(db_path) save_snapshot(db_path, snapshot) for index, worker_id in enumerate(worker_ids): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7214,7 +4510,7 @@ def test_store_canonical_pages_round_trip_long_content_without_duplicate_copy( final = "\n# Heading\n\n```\n" + ("🙂" * 270_000) + "\n```\n" assert len(final.encode("utf-8")) > 1024 * 1024 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "long-host", "worker-1", @@ -7361,7 +4657,7 @@ def test_store_canonical_pages_round_trip_long_content_without_duplicate_copy( assert final[:1000] not in payload_json assert revision_count == 1 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "long-host", "worker-1", @@ -7372,7 +4668,7 @@ def test_store_canonical_pages_round_trip_long_content_without_duplicate_copy( }, observed_at="2026-01-01T00:01:00+00:00", ) == 0 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "long-host", "worker-1", @@ -7404,7 +4700,7 @@ def test_store_merge_distinguishes_whitespace_content_from_empty_or_absent_field init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7416,14 +4712,14 @@ def test_store_merge_distinguishes_whitespace_content_from_empty_or_absent_field }, observed_at="2026-01-01T00:00:00+00:00", ) == 1 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, {"source_turn_id": "merge-precedence-source", "user_text": ""}, observed_at="2026-01-01T00:01:00+00:00", ) == 0 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7444,7 +4740,7 @@ def test_store_merge_distinguishes_whitespace_content_from_empty_or_absent_field assert preserved["assistant_final_text"] == "known final" whitespace = " \t\r\n " - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7463,7 +4759,7 @@ def test_store_merge_distinguishes_whitespace_content_from_empty_or_absent_field assert replaced["user_text"] == "known prompt" assert replaced["assistant_final_text"] == whitespace - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -7492,7 +4788,7 @@ def test_store_revision_replacement_rolls_back_projection_and_current_flip( ) init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "rollback-host", "worker-1", @@ -7520,7 +4816,7 @@ def fail_insert(*args: Any, **kwargs: Any) -> str: monkeypatch.setattr(store_sqlite, "_insert_turn_content_revision_conn", fail_insert) with pytest.raises(RuntimeError, match="controlled revision insert failure"): - merge_turn_content( + apply_test_turn_refresh( db_path, "rollback-host", "worker-1", @@ -7573,7 +4869,7 @@ def _seed_superseded_content_revision( ) init_store(db_path) save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -7584,7 +4880,7 @@ def _seed_superseded_content_revision( }, observed_at="2026-01-01T00:00:00+00:00", ) == 1 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -8351,7 +5647,7 @@ def test_acknowledged_revision_count_bounds_one_frequently_revised_turn( api = ConnectorOutboxAPI(db_path, host_id) revisions: list[str] = [] for index in range(4): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -8557,7 +5853,6 @@ def test_turn_content_maintenance_preserves_live_and_young_final_sources( attempt_at if attempt_status == "delivered" else None, ), ) - result = run_store_maintenance( db_path, host_id, @@ -8798,7 +6093,7 @@ def test_turn_content_maintenance_preserves_all_reference_classes( workers=[{"id": "worker-1", "name": "claude", "status": "active"}], ), ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -9067,7 +6362,7 @@ def test_source_turn_history_pruning_retains_referenced_old_turn( init_store(db_path) save_snapshot(db_path, snapshot) for index in range(6): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "protected-host", "worker-1", @@ -9148,7 +6443,7 @@ def test_source_turn_history_pruning_retains_referenced_old_turn( ) for index in range(6, 10): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "protected-host", "worker-1", @@ -9218,7 +6513,7 @@ def test_source_turn_history_pruning_retains_referenced_old_turn( max_outbox_attempts=99, now="2026-01-20T00:00:00+00:00", ) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "protected-host", "worker-1", @@ -9391,7 +6686,7 @@ def test_store_v8_maintenance_schema_singleton_and_ordered_indexes( assert created == ("created_at", "host_id", "id") -def test_current_v9_schema_gate_and_second_init_have_no_mutation_or_wal_setting( +def test_current_v29_schema_gate_and_second_init_have_no_mutation_or_wal_setting( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -9417,20 +6712,6 @@ def traced_pragmas( original_pragmas(conn, path) monkeypatch.setattr(store_sqlite, "_apply_connection_pragmas", traced_pragmas) - monkeypatch.setattr( - store_sqlite, - "MIGRATIONS", - tuple( - store_sqlite.Migration( - migration.from_version, - migration.to_version, - lambda _conn: (_ for _ in ()).throw( - AssertionError("current schema dispatched a migration") - ), - ) - for migration in store_sqlite.MIGRATIONS - ), - ) original_flock = fcntl.flock def reject_parent_ex(fd: int, operation: int) -> None: @@ -9452,6 +6733,200 @@ def reject_parent_ex(fd: int, operation: int) -> None: for statement in normalized ) + +def test_base_v28_schema_is_loudly_reset_without_removed_store_objects( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + db_path = tmp_path / "base-v28-reset.db" + with sqlite3.connect(str(db_path)) as conn: + conn.executescript( + """ + CREATE TABLE herdr_turn_watermarks ( + host_id TEXT NOT NULL, + pane_id TEXT NOT NULL, + last_turn INTEGER NOT NULL + ); + INSERT INTO herdr_turn_watermarks VALUES ('host-a', 'pane-private', 7); + CREATE TABLE agent_event_tombstones ( + host_id TEXT NOT NULL, + event_id TEXT NOT NULL + ); + INSERT INTO agent_event_tombstones VALUES ('host-a', 'event-private'); + CREATE TABLE backend_pending ( + sentinel TEXT NOT NULL, + route_kind TEXT NOT NULL DEFAULT 'legacy' + ); + INSERT INTO backend_pending VALUES ('pending-private', 'legacy'); + CREATE TABLE command_receipts ( + sentinel TEXT NOT NULL, + legacy_collision INTEGER NOT NULL DEFAULT 0, + legacy_collision_count INTEGER NOT NULL DEFAULT 0 + ); + INSERT INTO command_receipts VALUES ('receipt-private', 1, 2); + PRAGMA user_version = 28; + """ + ) + + with caplog.at_level(logging.WARNING, logger=store_sqlite.__name__): + init_store(db_path) + + with sqlite3.connect(str(db_path)) as conn: + assert _user_version(conn) == 29 + object_names = { + str(row[0]) + for row in conn.execute( + "SELECT name FROM sqlite_master WHERE name NOT LIKE 'sqlite_%'" + ).fetchall() + } + assert "herdr_turn_watermarks" not in object_names + assert "agent_event_tombstones" not in object_names + assert "route_kind" not in { + str(row[1]) for row in conn.execute("PRAGMA table_info(backend_pending)") + } + receipt_columns = { + str(row[1]) for row in conn.execute("PRAGMA table_info(command_receipts)") + } + assert "legacy_collision" not in receipt_columns + assert "legacy_collision_count" not in receipt_columns + assert conn.execute("SELECT COUNT(*) FROM backend_pending").fetchone() == (0,) + assert conn.execute("SELECT COUNT(*) FROM command_receipts").fetchone() == (0,) + + message = caplog.records[-1].getMessage() + assert "previous_version=28" in message + assert "target_version=29" in message + assert "table:herdr_turn_watermarks" in message + assert "table:agent_event_tombstones" in message + + +def _create_discarded_schema(db_path: Path, version: int) -> None: + with sqlite3.connect(str(db_path)) as conn: + conn.executescript( + """ + CREATE TABLE discarded_data (id INTEGER PRIMARY KEY, value TEXT); + CREATE INDEX discarded_index ON discarded_data(value); + CREATE TABLE discarded_audit (value TEXT); + CREATE TRIGGER discarded_trigger + AFTER INSERT ON discarded_data + BEGIN + INSERT INTO discarded_audit(value) VALUES (NEW.value); + END; + CREATE VIEW discarded_view AS + SELECT id, value FROM discarded_data; + INSERT INTO discarded_data(value) VALUES ('must be discarded'); + """ + ) + conn.execute(f"PRAGMA user_version = {int(version)}") + + +def _application_objects(db_path: Path) -> tuple[tuple[str, str], ...]: + with sqlite3.connect(str(db_path)) as conn: + return tuple( + (str(row[0]), str(row[1])) + for row in conn.execute( + """ + SELECT type, name FROM sqlite_master + WHERE name NOT LIKE 'sqlite_%' + ORDER BY type, name + """ + ).fetchall() + ) + + +def test_older_schema_is_loudly_discarded_and_recreated( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + db_path = tmp_path / "older.db" + _create_discarded_schema(db_path, store_sqlite.STORE_SCHEMA_VERSION - 1) + + with caplog.at_level(logging.WARNING, logger=store_sqlite.__name__): + init_store(db_path) + + with sqlite3.connect(str(db_path)) as conn: + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION + assert conn.execute( + "SELECT COUNT(*) FROM snapshots" + ).fetchone() == (0,) + names = {name for _kind, name in _application_objects(db_path)} + assert not any(name.startswith("discarded_") for name in names) + message = caplog.records[-1].getMessage() + for discarded in ( + "view:discarded_view", + "trigger:discarded_trigger", + "index:discarded_index", + "table:discarded_data", + "table:discarded_audit", + ): + assert discarded in message + + +def test_newer_schema_is_loudly_discarded_and_recreated( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + db_path = tmp_path / "newer.db" + _create_discarded_schema(db_path, store_sqlite.STORE_SCHEMA_VERSION + 1) + + with caplog.at_level(logging.WARNING, logger=store_sqlite.__name__): + init_store(db_path) + + with sqlite3.connect(str(db_path)) as conn: + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION + message = caplog.records[-1].getMessage() + assert f"previous_version={store_sqlite.STORE_SCHEMA_VERSION + 1}" in message + assert f"target_version={store_sqlite.STORE_SCHEMA_VERSION}" in message + assert "table:discarded_data" in message + + +def test_v0_with_application_objects_is_loudly_discarded_and_recreated( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + db_path = tmp_path / "v0-with-objects.db" + _create_discarded_schema(db_path, 0) + + with caplog.at_level(logging.WARNING, logger=store_sqlite.__name__): + init_store(db_path) + + with sqlite3.connect(str(db_path)) as conn: + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION + assert conn.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'discarded_data'" + ).fetchone() is None + assert "previous_version=0" in caplog.records[-1].getMessage() + + +def test_schema_rebuild_failure_rolls_back_every_discarded_object( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path = tmp_path / "rebuild-rollback.db" + _create_discarded_schema(db_path, store_sqlite.STORE_SCHEMA_VERSION - 1) + before = _application_objects(db_path) + + def fail_rebuild(_conn: sqlite3.Connection) -> None: + raise RuntimeError("injected rebuild failure") + + monkeypatch.setattr( + store_sqlite, + "_create_current_schema_objects_conn", + fail_rebuild, + ) + with store_sqlite._connect(db_path, prepare=True) as conn: + assert conn.execute("PRAGMA foreign_keys").fetchone() == (1,) + with pytest.raises(RuntimeError, match="injected rebuild failure"): + store_sqlite.ensure_schema(conn) + assert conn.execute("PRAGMA foreign_keys").fetchone() == (1,) + assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - 1 + assert conn.execute( + "SELECT value FROM discarded_data" + ).fetchone() == ("must be discarded",) + assert conn.execute( + "SELECT value FROM discarded_audit" + ).fetchone() == ("must be discarded",) + assert _application_objects(db_path) == before + @pytest.mark.parametrize("version", (0, 1)) @pytest.mark.parametrize( "factory", @@ -9564,6 +7039,10 @@ def close() -> None: try: store_sqlite.ensure_schema(conn) assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION + assert conn.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'legacy_sentinel'" + ).fetchone() is None + assert "snapshots" in _table_names(conn) assert len(close_errors) == 1 assert isinstance(close_errors[0], sqlite3.ProgrammingError) finally: @@ -9880,107 +7359,6 @@ def fail_first_shared_restore(fd: int, operation: int) -> None: assert set(os.listdir("/proc/self/fd")) - before_fds == set() -def test_noncurrent_schema_downgrade_external_ex_contention_fails_closed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "downgrade-ex-legacy.db" - with sqlite3.connect(str(db_path)) as legacy: - legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") - legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") - legacy.execute("PRAGMA user_version = 1") - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - before_threads = {id(thread) for thread in threading.enumerate()} - before_children = {process.pid for process in multiprocessing.active_children()} - context = multiprocessing.get_context("spawn") - acquired = context.Queue() - release = context.Event() - process = context.Process( - target=_cross_process_hold_parent_lock, - args=(str(db_path.parent), fcntl.LOCK_EX, acquired, release), - ) - connection: sqlite3.Connection | None = None - lock_fd = -1 - started = False - connection_id = -1 - try: - connection = store_sqlite._connect(db_path, prepare=True) - authority = store_sqlite._schema_connection_authority(connection) - assert authority.parent_fd is not None - parent_fd = authority.parent_fd - connection_id = id(connection) - original_run_migrations = store_sqlite._run_migrations - original_flock = fcntl.flock - failed_restores = 0 - - def start_external_ex_after_migration(*args: Any, **kwargs: Any) -> None: - nonlocal started - original_run_migrations(*args, **kwargs) - process.start() - started = True - - def lose_downgrade_to_external_ex(fd: int, operation: int) -> None: - nonlocal failed_restores - if fd == parent_fd and operation == (fcntl.LOCK_SH | fcntl.LOCK_NB): - failed_restores += 1 - original_flock(fd, fcntl.LOCK_UN) - if failed_restores == 1: - assert acquired.get(timeout=5) is None - raise BlockingIOError() - original_flock(fd, operation) - - monkeypatch.setattr( - store_sqlite, - "_run_migrations", - start_external_ex_after_migration, - ) - monkeypatch.setattr( - store_sqlite.fcntl, - "flock", - lose_downgrade_to_external_ex, - ) - with pytest.raises(LocalStateError) as caught: - store_sqlite.ensure_schema(connection) - assert caught.value.code is LocalStateErrorCode.OPERATION_FAILED - assert failed_restores == store_sqlite._SCHEMA_PARENT_SHARED_LOCK_RECOVERY_ATTEMPTS - with pytest.raises(sqlite3.ProgrammingError): - connection.execute("SELECT 1") - with pytest.raises(OSError): - os.fstat(parent_fd) - assert connection_id not in store_sqlite._SCHEMA_CONNECTION_AUTHORITIES - - release.set() - process.join(timeout=15) - if process.is_alive(): - pytest.fail("external parent EX holder did not terminate") - assert process.exitcode == 0 - process.close() - started = False - - lock_fd = os.open( - db_path.parent, - os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), - ) - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - fcntl.flock(lock_fd, fcntl.LOCK_UN) - finally: - release.set() - if started: - process.join(timeout=15) - if process.is_alive(): - process.terminate() - process.join(timeout=5) - process.close() - if connection is not None: - connection.close() - if lock_fd >= 0: - os.close(lock_fd) - acquired.close() - acquired.join_thread() - - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children def test_noncurrent_schema_live_shared_parent_fails_before_persistent_mutation( @@ -10042,103 +7420,6 @@ def traced_connect(*args: Any, **kwargs: Any) -> sqlite3.Connection: assert {process.pid for process in multiprocessing.active_children()} == before_children -@pytest.mark.parametrize("change", ("unlink", "replace")) -def test_noncurrent_schema_finalization_rejects_changed_pinned_main_without_creation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - change: str, -) -> None: - db_path = tmp_path / "unlinked-legacy.db" - replacement_bytes = b"replacement main must remain untouched" - with sqlite3.connect(str(db_path)) as legacy: - legacy.execute("CREATE TABLE legacy_sentinel (value TEXT NOT NULL)") - legacy.execute("INSERT INTO legacy_sentinel VALUES ('preserved')") - legacy.execute("PRAGMA user_version = 1") - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - - selected_identity = entry_identity(os.lstat(db_path)) - before_fds = set(os.listdir("/proc/self/fd")) - before_threads = {id(thread) for thread in threading.enumerate()} - before_children = {process.pid for process in multiprocessing.active_children()} - original_prepare = store_sqlite.prepare_sqlite_family_at - original_migrations = store_sqlite._run_migrations - prepare_calls = 0 - migrated = False - fired = False - connection: sqlite3.Connection | None = None - lock_fd = -1 - - def record_migration(*args: Any, **kwargs: Any) -> None: - nonlocal migrated - original_migrations(*args, **kwargs) - migrated = True - - def change_before_final_family_prepare( - parent_fd: int, - leaf: str, - **kwargs: Any, - ) -> tuple[PermissionResult, ...]: - nonlocal prepare_calls, fired - prepare_calls += 1 - if prepare_calls == 2: - assert migrated - assert kwargs["_parent_exclusive_lock_held"] is True - assert kwargs["_expected_main_identity"] == selected_identity - os.unlink(leaf, dir_fd=parent_fd) - if change == "replace": - replacement_fd = os.open( - leaf, - os.O_WRONLY | os.O_CREAT | os.O_EXCL, - 0o600, - dir_fd=parent_fd, - ) - try: - os.write(replacement_fd, replacement_bytes) - os.fchmod(replacement_fd, 0o600) - finally: - os.close(replacement_fd) - fired = True - return original_prepare(parent_fd, leaf, **kwargs) - - monkeypatch.setattr(store_sqlite, "_run_migrations", record_migration) - monkeypatch.setattr( - store_sqlite, - "prepare_sqlite_family_at", - change_before_final_family_prepare, - ) - try: - connection = store_sqlite._connect(db_path, prepare=True) - with pytest.raises(LocalStateError) as caught: - store_sqlite.ensure_schema(connection) - - assert fired - assert prepare_calls == 2 - assert caught.value.code is LocalStateErrorCode.ENTRY_CHANGED - if change == "replace": - assert db_path.read_bytes() == replacement_bytes - assert _mode(db_path) == 0o600 - else: - assert not db_path.exists() - lock_fd = os.open( - db_path.parent, - os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), - ) - with pytest.raises(BlockingIOError): - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - finally: - if connection is not None: - connection.close() - if lock_fd >= 0: - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - fcntl.flock(lock_fd, fcntl.LOCK_UN) - os.close(lock_fd) - - if change == "replace": - assert db_path.read_bytes() == replacement_bytes - assert set(os.listdir("/proc/self/fd")) - before_fds == set() - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children def test_noncurrent_schema_keeps_sidecars_private_and_restores_shared_parent_lock( @@ -10161,253 +7442,40 @@ def test_noncurrent_schema_keeps_sidecars_private_and_restores_shared_parent_loc connection = store_sqlite._connect(db_path, prepare=True) store_sqlite.ensure_schema(connection) - assert _user_version(connection) == store_sqlite.STORE_SCHEMA_VERSION - assert connection.execute( - "SELECT value FROM legacy_sentinel" - ).fetchone() == ("preserved",) - for suffix in ("", "-wal", "-shm"): - assert _mode(Path(f"{db_path}{suffix}")) == 0o600 - - lock_fd = os.open( - db_path.parent, - os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), - ) - with pytest.raises(BlockingIOError): - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - - connection.close() - connection = None - fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - fcntl.flock(lock_fd, fcntl.LOCK_UN) - finally: - if connection is not None: - connection.close() - if lock_fd >= 0: - os.close(lock_fd) - os.umask(previous_umask) - - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children - - -def test_direct_empty_creation_does_not_replay_migration_registry( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "direct-current.db" - monkeypatch.setattr( - store_sqlite, - "MIGRATIONS", - tuple( - store_sqlite.Migration( - migration.from_version, - migration.to_version, - lambda _conn: (_ for _ in ()).throw( - AssertionError("direct creation replayed history") - ), - ) - for migration in store_sqlite.MIGRATIONS - ), - ) - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - assert "turn_list_hosts" in _table_names(conn) - assert {"store_maintenance_state", "turn_list_state"} <= _table_names(conn) - assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - -def test_v13_migration_repairs_nonpositive_turn_sequences_and_blocks_recurrence( - tmp_path: Path, -) -> None: - db_path = tmp_path / "legacy-turn-sequence.db" - observed_at = "2026-07-15T00:00:00+00:00" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=13) - conn.execute("DROP TABLE turns") - conn.execute( - store_sqlite.CREATE_TURNS_TABLE.replace( - " CHECK (list_sequence > 0)", - "", - ) - ) - for statement in store_sqlite.CREATE_TURN_LIST_INDEXES: - conn.execute(statement) - for turn_id, sequence in (("turn-valid", 5), ("turn-invalid", 0)): - payload = { - "id": turn_id, - "worker_id": "worker-a", - "status": "complete", - "kind": "prompt", - "source": "snapshot", - "updated_at": observed_at, - } - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, worker_fingerprint, space_id, - status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, - list_sequence - ) VALUES (?, ?, 'worker-a', NULL, NULL, 'complete', 'prompt', - ?, '', '', ?, ?, ?) - """, - ( - "legacy-host", - turn_id, - observed_at, - observed_at, - json.dumps(payload), - sequence, - ), - ) - conn.execute( - """ - INSERT INTO turn_list_hosts ( - host_id, next_sequence, traversal_generation - ) VALUES ('legacy-host', 6, 7) - """ - ) - conn.commit() - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION == 21 - assert conn.execute( - """ - SELECT turn_id, list_sequence - FROM turns - WHERE host_id = 'legacy-host' - ORDER BY list_sequence - """ - ).fetchall() == [("turn-valid", 5), ("turn-invalid", 6)] - assert conn.execute( - """ - SELECT next_sequence, traversal_generation - FROM turn_list_hosts - WHERE host_id = 'legacy-host' - """ - ).fetchone() == (7, 8) - assert conn.execute( - """ - SELECT COUNT(*) - FROM sqlite_master - WHERE type = 'trigger' - AND name LIKE 'trg_turns_positive_list_sequence_%' - """ - ).fetchone() == (2,) - with pytest.raises(sqlite3.IntegrityError, match="invalid turn list sequence"): - conn.execute( - """ - UPDATE turns SET list_sequence = 0 - WHERE host_id = 'legacy-host' AND turn_id = 'turn-valid' - """ - ) - conn.rollback() - with pytest.raises(sqlite3.IntegrityError, match="invalid turn list sequence"): - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, status, kind, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, - list_sequence - ) VALUES ( - 'legacy-host', 'turn-recurrence', 'worker-a', 'complete', - 'prompt', '', '', ?, '{}', 0 - ) - """, - (observed_at,), - ) - conn.rollback() - - first = turns_payload_from_store( - db_path, - "legacy-host", - schema_version=2, - limit=1, - now=1_800_000_000, - ) - assert first["has_more"] is True - assert isinstance(first["next_cursor"], str) - second = turns_payload_from_store( - db_path, - "legacy-host", - schema_version=2, - limit=1, - cursor=first["next_cursor"], - now=1_800_000_001, - ) - assert second["has_more"] is False - assert [item["id"] for item in first["turns"] + second["turns"]] == [ - "turn-invalid", - "turn-valid", - ] - + assert _user_version(connection) == store_sqlite.STORE_SCHEMA_VERSION + assert connection.execute( + "SELECT 1 FROM sqlite_master WHERE name = 'legacy_sentinel'" + ).fetchone() is None + assert "snapshots" in _table_names(connection) + for suffix in ("", "-wal", "-shm"): + assert _mode(Path(f"{db_path}{suffix}")) == 0o600 + + lock_fd = os.open( + db_path.parent, + os.O_RDONLY | os.O_DIRECTORY | getattr(os, "O_CLOEXEC", 0), + ) + with pytest.raises(BlockingIOError): + fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + + connection.close() + connection = None + fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + fcntl.flock(lock_fd, fcntl.LOCK_UN) + finally: + if connection is not None: + connection.close() + if lock_fd >= 0: + os.close(lock_fd) + os.umask(previous_umask) + + assert {id(thread) for thread in threading.enumerate()} == before_threads + assert {process.pid for process in multiprocessing.active_children()} == before_children -@pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) -def test_migration_registry_transition_rolls_back_resumes_and_reruns( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - source_version: int, -) -> None: - db_path = tmp_path / f"registry-{source_version}.db" - original_registry = store_sqlite.MIGRATIONS - assert tuple( - (migration.from_version, migration.to_version) - for migration in original_registry - ) == tuple( - (version, version + 1) - for version in range(store_sqlite.STORE_SCHEMA_VERSION) - ) - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys=ON") - conn.execute("CREATE TABLE durable_sentinel (value TEXT NOT NULL)") - conn.execute("INSERT INTO durable_sentinel VALUES ('preserved')") - conn.commit() - store_sqlite._run_migrations(conn, target_version=source_version) - assert _user_version(conn) == source_version - transition = original_registry[source_version] - def apply_then_fail(current: sqlite3.Connection) -> None: - transition.apply(current) - raise RuntimeError("controlled migration interruption") - interrupted = list(original_registry) - interrupted[source_version] = store_sqlite.Migration( - transition.from_version, - transition.to_version, - apply_then_fail, - ) - monkeypatch.setattr(store_sqlite, "MIGRATIONS", tuple(interrupted)) - with pytest.raises(RuntimeError, match="controlled migration interruption"): - store_sqlite._run_migrations( - conn, - target_version=source_version + 1, - ) - assert _user_version(conn) == source_version - assert conn.execute("SELECT value FROM durable_sentinel").fetchone() == ( - "preserved", - ) - monkeypatch.setattr(store_sqlite, "MIGRATIONS", original_registry) - store_sqlite._run_migrations(conn, target_version=source_version + 1) - assert _user_version(conn) == source_version + 1 - conn.execute("BEGIN IMMEDIATE") - transition.apply(conn) - conn.commit() - assert conn.execute("SELECT value FROM durable_sentinel").fetchone() == ( - "preserved", - ) - assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] def test_ten_thousand_adjacent_identical_saves_keep_one_row_and_event( @@ -10617,448 +7685,39 @@ def test_snapshot_created_at_is_canonical_utc_and_invalid_input_fails_before_ope later = project_from_raw( config, timestamp=datetime.fromisoformat("2026-06-30T21:00:00-04:00"), - ) - assert first.content_fingerprint == later.content_fingerprint - - save_snapshot(db_path, first) - save_snapshot(db_path, later) - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - "SELECT created_at FROM snapshots" - ).fetchall() - assert rows == [("2026-07-01T01:00:00+00:00",)] - - invalid_path = tmp_path / "invalid" / "store.db" - - def forbidden_connect(*_args: Any, **_kwargs: Any) -> None: - raise AssertionError("invalid timestamp reached sqlite open") - - monkeypatch.setattr(store_sqlite, "_connect", forbidden_connect) - for invalid_timestamp in ( - "malformed-timestamp", - "0001-01-01T00:00:00+14:00", - "9999-12-31T23:59:59-14:00", - ): - invalid = project_empty( - Config(host_id="invalid-host", db_path=invalid_path) - ) - object.__setattr__(invalid, "updated_at", invalid_timestamp) - with pytest.raises(ValueError, match="invalid snapshot updated_at"): - save_snapshot(invalid_path, invalid) - assert not invalid_path.parent.exists() - - -def test_v7_timestamp_normalization_is_transactional_idempotent_and_age_safe( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "offset-migration.db" - init_store(db_path) - raw_rows = [ - ( - "2026-07-01T01:00:00+02:00", - "valid-old-positive-offset", - ), - ( - "2026-06-30T13:00:00-12:00", - "adversarial-newer-negative-offset", - ), - ("malformed-legacy-time", "malformed-quarantine"), - ( - "0001-01-01T00:00:00+14:00", - "underflow-quarantine", - ), - ( - "9999-12-31T23:59:59-14:00", - "overflow-quarantine", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-year-9999-quarantine", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-malformed-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-different-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-underflow-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-overflow-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legitimate-year-9999-observation", - ), - ( - "2026-07-01T00:30:00+02:00", - "old-but-latest", - ), - ] - with sqlite3.connect(str(db_path)) as conn: - conn.execute("DELETE FROM snapshots") - conn.executemany( - """ - INSERT INTO snapshots ( - host_id, created_at, content_fingerprint, payload - ) VALUES ('offset-host', ?, ?, '{}') - """, - raw_rows, - ) - conn.executemany( - """ - UPDATE snapshots - SET payload = ? - WHERE content_fingerprint = ? - """, - ( - ( - json.dumps({"updated_at": updated_at}, sort_keys=True), - fingerprint, - ) - for fingerprint, updated_at in ( - ("legacy-sentinel-malformed-payload", "not-a-time"), - ( - "legacy-sentinel-different-payload", - "2026-01-01T00:00:00+00:00", - ), - ( - "legacy-sentinel-underflow-payload", - "0001-01-01T00:00:00+14:00", - ), - ( - "legacy-sentinel-overflow-payload", - "9999-12-31T23:59:59-14:00", - ), - ( - "legitimate-year-9999-observation", - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - ), - ) - ), - ) - conn.execute("DROP TABLE store_maintenance_state") - conn.execute("DROP INDEX idx_snapshots_host_newest") - conn.execute("DROP INDEX idx_snapshots_created_host_id") - conn.execute( - "CREATE INDEX idx_snapshots_host_id ON snapshots(host_id)" - ) - conn.execute( - "CREATE INDEX idx_snapshots_created_at ON snapshots(created_at)" - ) - conn.execute( - """ - CREATE INDEX idx_snapshots_content_fingerprint - ON snapshots(content_fingerprint) - """ - ) - conn.execute("PRAGMA user_version = 7") - - original_registry = store_sqlite.MIGRATIONS - transition = original_registry[7] - - def normalize_then_fail(conn: sqlite3.Connection) -> None: - transition.apply(conn) - raise RuntimeError("controlled timestamp migration interruption") - - interrupted = list(original_registry) - interrupted[7] = store_sqlite.Migration(7, 8, normalize_then_fail) - monkeypatch.setattr(store_sqlite, "MIGRATIONS", tuple(interrupted)) - with pytest.raises( - RuntimeError, - match="controlled timestamp migration interruption", - ): - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == 7 - assert conn.execute( - """ - SELECT created_at, content_fingerprint - FROM snapshots - ORDER BY id - """ - ).fetchall() == raw_rows - assert "store_maintenance_state" not in _table_names(conn) - - monkeypatch.setattr(store_sqlite, "MIGRATIONS", original_registry) - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - normalized = conn.execute( - """ - SELECT created_at, content_fingerprint - FROM snapshots - ORDER BY id - """ - ).fetchall() - indexes = { - str(row[1]) - for row in conn.execute("PRAGMA index_list(snapshots)").fetchall() - } - conn.execute("BEGIN IMMEDIATE") - store_sqlite._migrate_v7_to_v8_conn(conn) - conn.commit() - rerun = conn.execute( - """ - SELECT created_at, content_fingerprint - FROM snapshots - ORDER BY id - """ - ).fetchall() - - assert normalized == [ - ( - "2026-06-30T23:00:00+00:00", - "valid-old-positive-offset", - ), - ( - "2026-07-01T01:00:00+00:00", - "adversarial-newer-negative-offset", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "malformed-quarantine", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "underflow-quarantine", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "overflow-quarantine", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-year-9999-quarantine", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-malformed-payload", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-different-payload", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-underflow-payload", - ), - ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE, - "legacy-sentinel-overflow-payload", - ), - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - "legitimate-year-9999-observation", - ), - ( - "2026-06-30T22:30:00+00:00", - "old-but-latest", - ), - ] - assert ( - store_sqlite._strict_utc_timestamp( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE - ) - is None - ) - assert ( - store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE - > store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE - ) - assert rerun == normalized - assert indexes == { - "idx_snapshots_host_newest", - "idx_snapshots_created_host_id", - } - - cleanup = store_sqlite.cleanup_snapshot_retention( - db_path, - retention_days=1, - retention_count=100, - batch_size=100, - now="2026-07-02T00:00:00Z", - ) - with sqlite3.connect(str(db_path)) as conn: - retained = conn.execute( - """ - SELECT content_fingerprint - FROM snapshots - ORDER BY id - """ - ).fetchall() - integrity = conn.execute("PRAGMA integrity_check").fetchone()[0] - foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - - assert cleanup["deleted"] == 1 - assert retained == [ - ("adversarial-newer-negative-offset",), - ("malformed-quarantine",), - ("underflow-quarantine",), - ("overflow-quarantine",), - ("legacy-year-9999-quarantine",), - ("legacy-sentinel-malformed-payload",), - ("legacy-sentinel-different-payload",), - ("legacy-sentinel-underflow-payload",), - ("legacy-sentinel-overflow-payload",), - ("legitimate-year-9999-observation",), - ("old-but-latest",), - ] - assert integrity == "ok" - assert foreign_keys == [] - - -def test_legacy_sentinel_mismatched_payload_recovers_then_remains_monotonic( - tmp_path: Path, -) -> None: - db_path = tmp_path / "quarantine-recovery.db" - config = Config(host_id="quarantine-host", db_path=db_path) - base_at = "2026-01-01T00:00:00+00:00" - fresh_at = "2027-01-01T00:00:00+00:00" - stale_at = "2026-06-01T00:00:00+00:00" - base = _snapshot_with_worker_status( - config, - status="blocked", - observed_at=base_at, - ) - fresh = _snapshot_with_worker_status( - config, - status="blocked", - observed_at=fresh_at, - ) - stale = _snapshot_with_worker_status( - config, - status="blocked", - observed_at=stale_at, - ) - assert { - base.content_fingerprint, - fresh.content_fingerprint, - stale.content_fingerprint, - } == {base.content_fingerprint} - _save_observation(db_path, base, "positive", base_at) - - with sqlite3.connect(str(db_path)) as conn: - row_id = conn.execute( - """ - SELECT id - FROM snapshots - WHERE host_id = ? - """, - ("quarantine-host",), - ).fetchone()[0] - conn.execute( - """ - UPDATE snapshots - SET created_at = ? - WHERE id = ? - """, - ( - store_sqlite._LEGACY_SNAPSHOT_CREATED_AT_QUARANTINE, - row_id, - ), - ) - conn.execute( - """ - UPDATE workers - SET observed_at = 'malformed-legacy-time' - WHERE host_id = ? - """, - ("quarantine-host",), - ) - conn.execute( - """ - UPDATE backend_health - SET observed_at = 'malformed-legacy-time' - WHERE host_id = ? - """, - ("quarantine-host",), - ) - conn.execute("DROP TABLE store_maintenance_state") - conn.execute("DROP INDEX idx_snapshots_host_newest") - conn.execute("DROP INDEX idx_snapshots_created_host_id") - conn.execute("PRAGMA user_version = 7") + ) + assert first.content_fingerprint == later.content_fingerprint - init_store(db_path) + save_snapshot(db_path, first) + save_snapshot(db_path, later) with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - "SELECT created_at FROM snapshots WHERE id = ?", - (row_id,), - ).fetchone() == (store_sqlite._SNAPSHOT_CREATED_AT_QUARANTINE,) + rows = conn.execute( + "SELECT created_at FROM snapshots" + ).fetchall() + assert rows == [("2026-07-01T01:00:00+00:00",)] + + invalid_path = tmp_path / "invalid" / "store.db" + + def forbidden_connect(*_args: Any, **_kwargs: Any) -> None: + raise AssertionError("invalid timestamp reached sqlite open") + + monkeypatch.setattr(store_sqlite, "_connect", forbidden_connect) + for invalid_timestamp in ( + "malformed-timestamp", + "0001-01-01T00:00:00+14:00", + "9999-12-31T23:59:59-14:00", + ): + invalid = project_empty( + Config(host_id="invalid-host", db_path=invalid_path) + ) + object.__setattr__(invalid, "updated_at", invalid_timestamp) + with pytest.raises(ValueError, match="invalid snapshot updated_at"): + save_snapshot(invalid_path, invalid) + assert not invalid_path.parent.exists() - _save_observation(db_path, fresh, "positive", fresh_at) - def current_state() -> tuple[Any, ...]: - with sqlite3.connect(str(db_path)) as conn: - retained = conn.execute( - """ - SELECT id, created_at, payload - FROM snapshots - WHERE host_id = ? - """, - ("quarantine-host",), - ).fetchone() - return ( - int(retained[0]), - str(retained[1]), - json.loads(retained[2])["updated_at"], - conn.execute( - """ - SELECT observed_at - FROM workers - WHERE host_id = ? AND worker_id = 'worker-1' - """, - ("quarantine-host",), - ).fetchone()[0], - conn.execute( - """ - SELECT observed_at - FROM backend_health - WHERE host_id = ? AND backend_name = 'herdr' - """, - ("quarantine-host",), - ).fetchone()[0], - conn.execute( - """ - SELECT last_seen_at, signal_count - FROM attention_items - WHERE host_id = ? - """, - ("quarantine-host",), - ).fetchone(), - conn.execute( - "SELECT COUNT(*) FROM snapshots WHERE host_id = ?", - ("quarantine-host",), - ).fetchone()[0], - conn.execute( - """ - SELECT COUNT(*) - FROM events - WHERE host_id = ? AND event_type = 'snapshot.saved' - """, - ("quarantine-host",), - ).fetchone()[0], - ) - recovered = current_state() - assert recovered == ( - row_id, - fresh_at, - fresh_at, - fresh_at, - fresh_at, - (fresh_at, 2), - 1, - 1, - ) - _save_observation(db_path, stale, "positive", stale_at) - assert current_state() == recovered @@ -12242,7 +8901,7 @@ def test_compact_store_dry_run_reports_low_headroom_without_mutating( assert _tree_metadata(tmp_path) == before -def test_compact_store_requires_current_v9_without_migrating( +def test_compact_store_requires_current_schema_without_rebuilding( tmp_path: Path, ) -> None: db_path, _private_payload = _seed_compaction_fixture(tmp_path) @@ -12465,552 +9124,110 @@ def interrupt_after_retention(phase: str) -> None: acknowledge_offline=True, backup_path=backup_path, snapshot_retention_days=14, - snapshot_retention_count=1, - batch_size=2, - ), - now="2026-02-01T00:00:00+00:00", - phase_hook=interrupt_after_retention, - ) - - assert result["status"] == "rollback_failed" - assert result["rollback"] == {"status": "failed"} - assert result["snapshots"]["deleted"] == 10 - assert restore_calls == 1 - assert checkpoint_calls == [None] - assert publish_calls == [] - assert backup_path.is_file() - current = db_path.stat() - assert (current.st_dev, current.st_ino) == ( - substitute_stat.st_dev, - substitute_stat.st_ino, - ) - assert hashlib.sha256(db_path.read_bytes()).hexdigest() == substitute_digest - verification = sqlite3.connect( - f"file:{db_path}?mode=ro&immutable=1", - uri=True, - ) - try: - assert verification.execute( - "SELECT value FROM substitute_sentinel" - ).fetchone() == ("must-not-be-overwritten",) - assert verification.execute( - "SELECT COUNT(*) FROM snapshots" - ).fetchone() == (0,) - finally: - verification.close() - assert displaced_source.is_file() - assert not any( - path.name.startswith(".tendwire-sqlite-") - for path in tmp_path.iterdir() - ) - assert set(os.listdir("/proc/self/fd")) == before_fds - assert {id(thread) for thread in threading.enumerate()} == before_threads - assert {process.pid for process in multiprocessing.active_children()} == before_children - - -def test_compact_store_cleans_vacuum_output_when_sqlite_raises_after_creation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path, _private_payload = _seed_compaction_fixture(tmp_path) - backup_path = tmp_path / "vacuum-failure-backup.db" - original_connect = store_sqlite._connect - - class RaiseAfterVacuum: - def __init__(self, connection: sqlite3.Connection) -> None: - self.connection = connection - - def __enter__(self) -> "RaiseAfterVacuum": - self.connection.__enter__() - return self - - def __exit__(self, *args: Any) -> Any: - return self.connection.__exit__(*args) - - def __getattr__(self, name: str) -> Any: - return getattr(self.connection, name) - - def execute( - self, - sql: str, - parameters: Any = (), - ) -> Any: - result = self.connection.execute(sql, parameters) - if sql.lstrip().upper().startswith("VACUUM INTO"): - raise sqlite3.OperationalError("private-vacuum-failure") - return result - - def intercept_connect(*args: Any, **kwargs: Any) -> RaiseAfterVacuum: - return RaiseAfterVacuum(original_connect(*args, **kwargs)) - - monkeypatch.setattr(store_sqlite, "_connect", intercept_connect) - result = compact_store( - db_path, - options=CompactionOptions( - dry_run=False, - acknowledge_offline=True, - backup_path=backup_path, - snapshot_retention_days=14, - snapshot_retention_count=8, - batch_size=2, - ), - now="2026-02-01T00:00:00+00:00", - ) - - assert result["status"] == "rollback_completed" - assert result["rollback"] == {"status": "completed"} - assert backup_path.is_file() - assert not any( - path.name.startswith(".tendwire-sqlite-") - for path in tmp_path.iterdir() - ) - _assert_compaction_logical_evidence(db_path) - - -def test_v7_to_current_conservatively_classifies_legacy_final_and_preserves_state( - tmp_path: Path, -) -> None: - db_path = tmp_path / "complete-preservation.db" - backup_path = tmp_path / "complete-preservation-backup.db" - init_store(db_path) - old_rows = [ - ( - host_id, - f"2025-12-{sequence + 1:02d}T00:00:00+00:00", - f"old-{host_id}-{sequence}", - json.dumps({"old": sequence, "host_id": host_id}, sort_keys=True), - ) - for host_id in ("host-a", "host-b") - for sequence in range(3) - ] - with sqlite3.connect(str(db_path)) as conn: - conn.executemany( - """ - INSERT INTO snapshots ( - host_id, created_at, content_fingerprint, payload - ) VALUES (?, ?, ?, ?) - """, - old_rows, - ) - - host_a_config = Config(host_id="host-a", db_path=db_path) - host_b_config = Config(host_id="host-b", db_path=db_path) - host_a_snapshot = project_from_raw( - host_a_config, - spaces=[{"id": "space-a", "name": "Space A", "status": "active"}], - workers=[ - { - "id": "worker-1", - "name": "Worker One", - "status": "pending", - "space_id": "space-a", - "summary": "human approval required before continuing", - } - ], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-31T00:00:00+00:00", - "counts": {"workers": 1}, - } - ], - timestamp=datetime.fromisoformat("2026-01-31T00:00:00+00:00"), - ) - host_b_snapshot = project_from_raw( - host_b_config, - spaces=[{"id": "space-b", "name": "Space B", "status": "active"}], - workers=[ - { - "id": "worker-b", - "name": "Worker B", - "status": "active", - "space_id": "space-b", - } - ], - backend_health=[ - { - "name": "herdr", - "status": "healthy", - "outcome": "healthy_non_empty", - "observed_at": "2026-01-31T00:01:00+00:00", - "counts": {"workers": 1}, - } - ], - timestamp=datetime.fromisoformat("2026-01-31T00:01:00+00:00"), - ) - _save_observation( - db_path, - host_a_snapshot, - "positive", - "2026-01-31T00:00:00+00:00", - ) - _save_observation( - db_path, - host_b_snapshot, - "positive", - "2026-01-31T00:01:00+00:00", - ) - assert merge_turn_content( - db_path, - "host-a", - "worker-1", - { - "user_text": "Preserve this prompt.", - "assistant_final_text": "Preserve this final.", - "complete": True, - "has_open_turn": False, - "source_turn_id": "complete-preservation-turn", - }, - observed_at="2026-01-31T00:02:00+00:00", - ) == 1 - assert upsert_worker_bindings( - db_path, - [ - _worker_binding( - observed_at="2026-01-31T00:00:00+00:00", - expires_at="2027-01-31T00:00:00+00:00", - ) - ], - ) == 1 - reserved = reserve_command_request( - db_path, - host_id="host-a", - request_id="preserved-request", - action="send_instruction", - canonical_version=1, - canonical_fingerprint="preserved-command-fingerprint", - canonical_request_json='{"action":"send_instruction"}', - public_worker_id="worker-1", - pending_result_json='{"status":"pending"}', - now="2026-01-31T00:02:01+00:00", - ) - assert reserved["status"] == "reserved" - started = mark_command_send_started( - db_path, - host_id="host-a", - request_id="preserved-request", - canonical_fingerprint="preserved-command-fingerprint", - owner_token=reserved["owner_token"], - binding_fingerprint="preserved-private-binding", - now="2026-01-31T00:02:02+00:00", - ) - finish_command_request( - db_path, - host_id="host-a", - request_id="preserved-request", - canonical_fingerprint="preserved-command-fingerprint", - owner_token=started["owner_token"], - expected_state="send_started", - terminal_state="accepted", - status=STATUS_ACCEPTED, - result_json='{"status":"accepted","result":"preserved"}', - now="2026-01-31T00:02:03+00:00", + snapshot_retention_count=1, + batch_size=2, + ), + now="2026-02-01T00:00:00+00:00", + phase_hook=interrupt_after_retention, ) - assert store_sqlite.merge_backend_pending( - db_path, - "host-a", - "worker-1", - {"kind": "approval", "safe": "preserved"}, + + assert result["status"] == "rollback_failed" + assert result["rollback"] == {"status": "failed"} + assert result["snapshots"]["deleted"] == 10 + assert restore_calls == 1 + assert checkpoint_calls == [None] + assert publish_calls == [] + assert backup_path.is_file() + current = db_path.stat() + assert (current.st_dev, current.st_ino) == ( + substitute_stat.st_dev, + substitute_stat.st_ino, ) - leased = poll_connector_outbox( - db_path, - "host-a", - "attention", - now="2026-01-31T00:03:00+00:00", + assert hashlib.sha256(db_path.read_bytes()).hexdigest() == substitute_digest + verification = sqlite3.connect( + f"file:{db_path}?mode=ro&immutable=1", + uri=True, + ) + try: + assert verification.execute( + "SELECT value FROM substitute_sentinel" + ).fetchone() == ("must-not-be-overwritten",) + assert verification.execute( + "SELECT COUNT(*) FROM snapshots" + ).fetchone() == (0,) + finally: + verification.close() + assert displaced_source.is_file() + assert not any( + path.name.startswith(".tendwire-sqlite-") + for path in tmp_path.iterdir() ) - assert len(leased["items"]) == 1 + assert set(os.listdir("/proc/self/fd")) == before_fds + assert {id(thread) for thread in threading.enumerate()} == before_threads + assert {process.pid for process in multiprocessing.active_children()} == before_children - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - INSERT OR REPLACE INTO pending_interactions ( - host_id, pending_id, worker_id, worker_fingerprint, space_id, - kind, status, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json - ) VALUES ( - 'host-a', 'durable-pending', 'worker-1', 'worker-fingerprint', - 'space-a', 'approval', 'pending', - '2026-01-31T00:00:00+00:00', 'pending-fingerprint', - ?, '2026-01-31T00:00:00+00:00', - '{"kind":"approval","safe":"preserved"}' - ) - """, - (host_a_snapshot.content_fingerprint,), - ) - conn.execute( - "CREATE TABLE unrelated_preservation_sentinel (value TEXT NOT NULL)" - ) - conn.execute( - "INSERT INTO unrelated_preservation_sentinel VALUES ('preserved')" - ) - conn.execute("DROP TABLE store_maintenance_state") - conn.execute("DROP INDEX idx_snapshots_host_newest") - conn.execute("DROP INDEX idx_snapshots_created_host_id") - conn.execute( - "CREATE INDEX idx_snapshots_host_id ON snapshots(host_id)" - ) - conn.execute( - "CREATE INDEX idx_snapshots_created_at ON snapshots(created_at)" - ) - conn.execute( - """ - CREATE INDEX idx_snapshots_content_fingerprint - ON snapshots(content_fingerprint) - """ - ) - conn.execute("PRAGMA user_version = 7") - preserved_tables = ( - "commands", - "command_receipts", - "worker_bindings", - "pending_interactions", - "backend_pending", - "attention_items", - "attention_lifecycles", - "spaces", - "workers", - "turns", - "turn_content_revisions", - "turn_content_page_boundaries", - "backend_health", - "connector_outbox", - "connector_deliveries", - "unrelated_preservation_sentinel", - ) - - def logical_evidence() -> dict[str, Any]: - with sqlite3.connect(str(db_path)) as conn: - table_rows = { - table: tuple( - sorted( - ( - tuple(row) - for row in conn.execute( - f"SELECT * FROM {table}" - ).fetchall() - ), - key=repr, - ) - ) - for table in preserved_tables - } - latest = tuple( - conn.execute( - """ - SELECT snapshot.host_id, snapshot.content_fingerprint, - snapshot.payload - FROM snapshots AS snapshot - WHERE snapshot.id = ( - SELECT MAX(newest.id) - FROM snapshots AS newest - WHERE newest.host_id = snapshot.host_id - ) - ORDER BY snapshot.host_id - """ - ).fetchall() - ) - snapshot_count = int( - conn.execute("SELECT COUNT(*) FROM snapshots").fetchone()[0] - ) - return { - "tables": table_rows, - "latest": latest, - "snapshot_count": snapshot_count, - "integrity": conn.execute( - "PRAGMA integrity_check" - ).fetchone()[0], - "foreign_keys": conn.execute( - "PRAGMA foreign_key_check" - ).fetchall(), - } +def test_compact_store_cleans_vacuum_output_when_sqlite_raises_after_creation( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + db_path, _private_payload = _seed_compaction_fixture(tmp_path) + backup_path = tmp_path / "vacuum-failure-backup.db" + original_connect = store_sqlite._connect - before_migration = logical_evidence() - assert all(before_migration["tables"][table] for table in preserved_tables) - assert before_migration["snapshot_count"] == 8 - assert before_migration["integrity"] == "ok" - assert before_migration["foreign_keys"] == [] + class RaiseAfterVacuum: + def __init__(self, connection: sqlite3.Connection) -> None: + self.connection = connection - init_store(db_path) - after_migration = logical_evidence() - for table in preserved_tables: - if table != "connector_outbox": - assert after_migration["tables"][table] == before_migration["tables"][table] - assert after_migration["latest"] == before_migration["latest"] - assert after_migration["snapshot_count"] == before_migration["snapshot_count"] - with sqlite3.connect(str(db_path)) as conn: - legacy_final = conn.execute( - """ - SELECT delivery_kind, status, payload_json - FROM connector_outbox - WHERE host_id = 'host-a' AND connector = 'turn-final' - """ - ).fetchone() - assert legacy_final is not None - assert legacy_final[:2] == ("final_migration_hold", "dead_letter") - assert json.loads(legacy_final[2])["operation"] == "materialize" - assert poll_connector_outbox( - db_path, - "host-a", - "turn-final", - now="2026-01-31T00:04:00+00:00", - )["items"] == [] - with sqlite3.connect(str(db_path)) as conn: - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - assert conn.execute( - "SELECT scope FROM store_maintenance_state" - ).fetchone() == ("automatic",) + def __enter__(self) -> "RaiseAfterVacuum": + self.connection.__enter__() + return self - while True: - retention = store_sqlite.cleanup_snapshot_retention( - db_path, - retention_days=14, - retention_count=1, - batch_size=2, - now="2026-02-01T00:00:00+00:00", - ) - if not retention["remaining_candidates"]: - break - after_retention = logical_evidence() - assert after_retention["tables"] == after_migration["tables"] - assert after_retention["latest"] == after_migration["latest"] - assert after_retention["snapshot_count"] == 2 - assert after_retention["integrity"] == "ok" - assert after_retention["foreign_keys"] == [] + def __exit__(self, *args: Any) -> Any: + return self.connection.__exit__(*args) + + def __getattr__(self, name: str) -> Any: + return getattr(self.connection, name) + + def execute( + self, + sql: str, + parameters: Any = (), + ) -> Any: + result = self.connection.execute(sql, parameters) + if sql.lstrip().upper().startswith("VACUUM INTO"): + raise sqlite3.OperationalError("private-vacuum-failure") + return result - compacted = compact_store( + def intercept_connect(*args: Any, **kwargs: Any) -> RaiseAfterVacuum: + return RaiseAfterVacuum(original_connect(*args, **kwargs)) + + monkeypatch.setattr(store_sqlite, "_connect", intercept_connect) + result = compact_store( db_path, options=CompactionOptions( dry_run=False, acknowledge_offline=True, backup_path=backup_path, snapshot_retention_days=14, - snapshot_retention_count=1, + snapshot_retention_count=8, batch_size=2, ), now="2026-02-01T00:00:00+00:00", ) - after_compaction = logical_evidence() - assert compacted["status"] == "completed" - assert compacted["ok"] is True - assert after_compaction == after_retention + + assert result["status"] == "rollback_completed" + assert result["rollback"] == {"status": "completed"} assert backup_path.is_file() + assert not any( + path.name.startswith(".tendwire-sqlite-") + for path in tmp_path.iterdir() + ) + _assert_compaction_logical_evidence(db_path) -def test_store_v8_to_v9_backfills_host_local_sequences_and_paging_indexes( - tmp_path: Path, -) -> None: - db_path = tmp_path / "turn-v8-to-v9.db" - with sqlite3.connect(str(db_path)) as conn: - conn.execute( - """ - CREATE TABLE turns ( - host_id TEXT NOT NULL, - turn_id TEXT NOT NULL, - worker_id TEXT NOT NULL, - worker_fingerprint TEXT, - space_id TEXT, - status TEXT NOT NULL, - kind TEXT NOT NULL, - updated_at TEXT, - fingerprint TEXT NOT NULL, - snapshot_content_fingerprint TEXT NOT NULL, - observed_at TEXT NOT NULL, - payload_json TEXT NOT NULL, - PRIMARY KEY (host_id, turn_id) - ) - """ - ) - conn.execute("PRAGMA user_version=8") - rows = ( - ("host-a", "turn-c", "worker-a", "2026-01-02T00:00:00+00:00"), - ("host-a", "turn-b", "worker-a", "2026-01-01T00:00:00+00:00"), - ("host-a", "turn-a", "worker-a", "2026-01-01T00:00:00+00:00"), - ("host-b", "turn-z", "worker-z", "2026-01-03T00:00:00+00:00"), - ) - for host_id, turn_id, worker_id, observed_at in rows: - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, worker_fingerprint, space_id, - status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json - ) VALUES (?, ?, ?, NULL, NULL, 'active', 'task', ?, '', '', ?, '{}') - """, - (host_id, turn_id, worker_id, observed_at, observed_at), - ) - conn.commit() - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - first = conn.execute( - """ - SELECT host_id, turn_id, list_sequence - FROM turns - ORDER BY host_id, list_sequence - """ - ).fetchall() - indexes = { - str(row[1]): tuple( - str(column[2]) - for column in conn.execute( - f"PRAGMA index_info({row[1]})" - ).fetchall() - ) - for row in conn.execute("PRAGMA index_list(turns)").fetchall() - } - epoch = conn.execute( - "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" - ).fetchone()[0] - host_states = conn.execute( - """ - SELECT host_id, next_sequence, traversal_generation - FROM turn_list_hosts - ORDER BY host_id - """ - ).fetchall() - assert _user_version(conn) == store_sqlite.STORE_SCHEMA_VERSION - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - second = conn.execute( - """ - SELECT host_id, turn_id, list_sequence - FROM turns - ORDER BY host_id, list_sequence - """ - ).fetchall() - second_epoch = conn.execute( - "SELECT store_epoch FROM turn_list_state WHERE scope = 'turn-list'" - ).fetchone()[0] - second_host_states = conn.execute( - """ - SELECT host_id, next_sequence, traversal_generation - FROM turn_list_hosts - ORDER BY host_id - """ - ).fetchall() - assert first == [ - ("host-a", "turn-a", 1), - ("host-a", "turn-b", 2), - ("host-a", "turn-c", 3), - ("host-b", "turn-z", 1), - ] - assert second == first - assert second_epoch == epoch - assert host_states == second_host_states == [ - ("host-a", 4, 1), - ("host-b", 2, 1), - ] - assert indexes["ux_turns_host_list_sequence"] == ("host_id", "list_sequence") - assert indexes["idx_turns_host_worker_list_sequence"] == ( - "host_id", - "worker_id", - "list_sequence", - "turn_id", - ) def test_all_api_turn_insertions_allocate_unique_immutable_sequences_concurrently( @@ -13030,7 +9247,7 @@ def test_all_api_turn_insertions_allocate_unique_immutable_sequences_concurrentl def insert_observation(index: int) -> None: try: barrier.wait(timeout=10) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, "worker-1", @@ -13097,7 +9314,7 @@ def test_turn_list_pagination_is_insert_stable_and_since_discovers_only_new_rows init_store(db_path) save_snapshot(db_path, snapshot) for index, worker in enumerate(snapshot.workers): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker.id, @@ -13123,7 +9340,7 @@ def test_turn_list_pagination_is_insert_stable_and_since_discovers_only_new_rows (host_id,), ).fetchall() } - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, snapshot.workers[0].id, @@ -13180,7 +9397,7 @@ def test_turn_list_tokens_distinguish_invalid_cursor_cursor_expiry_and_since_exp init_store(db_path) save_snapshot(db_path, snapshot) for index, worker in enumerate(snapshot.workers): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, "expiry-host", worker.id, @@ -13273,6 +9490,17 @@ def test_turn_list_pages_remain_below_frame_cap_for_over_one_mib_logical_list( index + 1, ), ) + store_sqlite._insert_turn_content_revision_conn( + conn, + host_id=host_id, + turn_id=str(item["id"]), + user_text=None, + assistant_final_text=str(item["assistant_final_text"]), + user_state="absent", + final_state="complete", + created_at=str(item["updated_at"]), + is_current=True, + ) conn.commit() ids: list[str] = [] @@ -13319,7 +9547,7 @@ def test_same_source_completion_is_observation_monotonic_and_never_reopens( init_store(db_path) save_snapshot(db_path, snapshot) source = "same-source" - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13349,7 +9577,7 @@ def test_same_source_completion_is_observation_monotonic_and_never_reopens( "2029-12-31T23:59:59+00:00", "2030-01-01T00:00:00+00:00", ): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13361,7 +9589,7 @@ def test_same_source_completion_is_observation_monotonic_and_never_reopens( }, observed_at=observed_at, ) == 0 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13399,7 +9627,7 @@ def test_same_source_completion_is_observation_monotonic_and_never_reopens( assert revisions[0][1] == 0 assert revisions[1][1] == 1 - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13443,23 +9671,27 @@ def test_apply_turn_refresh_rolls_back_turn_pending_and_rejects_stale_binding( turn_target_value="private-pane", ) upsert_worker_bindings(db_path, [binding]) - original_pending_apply = store_sqlite._merge_backend_pending_conn + original_pending_apply = store_sqlite._apply_backend_pending_observation_conn def fail_pending(*args: Any, **kwargs: Any) -> bool: raise RuntimeError("controlled pending failure") - monkeypatch.setattr(store_sqlite, "_merge_backend_pending_conn", fail_pending) + monkeypatch.setattr(store_sqlite, "_apply_backend_pending_observation_conn", fail_pending) with pytest.raises(RuntimeError, match="controlled pending failure"): store_sqlite.apply_turn_refresh( db_path, host_id, worker.id, {"assistant_final_text": "must roll back", "complete": True}, - backend_pending={"question": "must roll back"}, + backend_pending_observation=store_sqlite.PendingObservation( + "open_prompt", + question="must roll back", + revision_digest="must-roll-back", + ), expected_binding=binding, observed_at="2099-01-01T00:00:00+00:00", ) - assert not store_sqlite.list_backend_pending(db_path, host_id) + assert not store_sqlite.pending_payload_from_store(db_path, host_id)["pending_interactions"] assert all( turn.get("assistant_final_text") in (None, "") for turn in turns_payload_from_store(db_path, host_id)["turns"] @@ -13467,7 +9699,7 @@ def fail_pending(*args: Any, **kwargs: Any) -> bool: monkeypatch.setattr( store_sqlite, - "_merge_backend_pending_conn", + "_apply_backend_pending_observation_conn", original_pending_apply, ) with sqlite3.connect(str(db_path)) as conn: @@ -13484,12 +9716,16 @@ def fail_pending(*args: Any, **kwargs: Any) -> bool: host_id, worker.id, {"assistant_final_text": "stale result", "complete": True}, - backend_pending={"question": "stale pending"}, + backend_pending_observation=store_sqlite.PendingObservation( + "open_prompt", + question="stale pending", + revision_digest="stale-pending", + ), expected_binding=binding, observed_at="2099-01-01T00:00:01+00:00", ) assert stale == store_sqlite.TurnRefreshApplyResult(0, False, True) - assert not store_sqlite.list_backend_pending(db_path, host_id) + assert not store_sqlite.pending_payload_from_store(db_path, host_id)["pending_interactions"] assert all( turn.get("assistant_final_text") in (None, "") for turn in turns_payload_from_store(db_path, host_id)["turns"] @@ -13561,7 +9797,7 @@ def barrier_read( ) def write_final() -> None: - results["final"] = merge_turn_content( + results["final"] = apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13575,7 +9811,7 @@ def write_final() -> None: ) def write_late_working() -> None: - results["working"] = merge_turn_content( + results["working"] = apply_test_turn_refresh( db_path, host_id, worker_id, @@ -13643,7 +9879,11 @@ def test_apply_turn_refresh_deadline_while_writer_locked_never_commits_later( host_id, worker_id, {"assistant_final_text": "must never commit", "complete": True}, - backend_pending={"question": "must never persist"}, + backend_pending_observation=store_sqlite.PendingObservation( + "open_prompt", + question="must never persist", + revision_digest="must-never-persist", + ), deadline_monotonic=started + 0.15, observed_at="2099-01-01T00:00:00+00:00", ) @@ -13659,7 +9899,7 @@ def test_apply_turn_refresh_deadline_while_writer_locked_never_commits_later( blocker.rollback() blocker.close() - assert not store_sqlite.list_backend_pending(db_path, host_id) + assert not store_sqlite.pending_payload_from_store(db_path, host_id)["pending_interactions"] assert all( turn.get("assistant_final_text") in (None, "") for turn in turns_payload_from_store(db_path, host_id)["turns"] @@ -13689,73 +9929,23 @@ def cancel_apply_before_commit() -> bool: host_id, worker_id, {"assistant_final_text": "rollback at commit seam", "complete": True}, - backend_pending={"question": "rollback at commit seam"}, + backend_pending_observation=store_sqlite.PendingObservation( + "open_prompt", + question="rollback at commit seam", + revision_digest="rollback-at-commit-seam", + ), cancelled=cancel_apply_before_commit, observed_at="2099-01-01T00:00:02+00:00", ) assert precommit_cancelled.cancelled is True assert apply_checks == 3 - assert not store_sqlite.list_backend_pending(db_path, host_id) + assert not store_sqlite.pending_payload_from_store(db_path, host_id)["pending_interactions"] assert all( turn.get("assistant_final_text") in (None, "") for turn in turns_payload_from_store(db_path, host_id)["turns"] ) -def test_prune_backend_pending_deadline_while_writer_locked_is_non_mutating( - tmp_path: Path, -) -> None: - db_path = tmp_path / "pending-prune-deadline.db" - init_store(db_path) - assert store_sqlite.merge_backend_pending( - db_path, - "prune-host", - "orphan-worker", - {"question": "still present"}, - ) - blocker = store_sqlite._connect(db_path, isolation_level=None) - try: - blocker.execute("BEGIN IMMEDIATE") - started = time.monotonic() - assert store_sqlite.prune_backend_pending( - db_path, - "prune-host", - (), - deadline_monotonic=started + 0.15, - ) == 0 - assert time.monotonic() - started < 1.0 - finally: - blocker.rollback() - blocker.close() - - assert "orphan-worker" in store_sqlite.list_backend_pending( - db_path, - "prune-host", - ) - prune_checks = 0 - - def cancel_prune_before_commit() -> bool: - nonlocal prune_checks - prune_checks += 1 - return prune_checks == 3 - - assert store_sqlite.prune_backend_pending( - db_path, - "prune-host", - (), - cancelled=cancel_prune_before_commit, - ) == 0 - assert prune_checks == 3 - assert "orphan-worker" in store_sqlite.list_backend_pending( - db_path, - "prune-host", - ) - assert store_sqlite.prune_backend_pending( - db_path, - "prune-host", - (), - ) == 1 - assert not store_sqlite.list_backend_pending(db_path, "prune-host") def test_turn_list_filtered_rows_advance_bounded_cursor_without_hiding_public_rows( @@ -13813,6 +10003,17 @@ def test_turn_list_filtered_rows_advance_bounded_cursor_without_hiding_public_ro """, (host_id, public_payload), ) + store_sqlite._insert_turn_content_revision_conn( + conn, + host_id=host_id, + turn_id="public-turn", + user_text="Public prompt", + assistant_final_text="Public answer", + user_state="complete", + final_state="complete", + created_at="1970-01-01T00:00:00+00:00", + is_current=True, + ) first = turns_payload_from_store( db_path, @@ -13852,7 +10053,7 @@ def test_turn_sequence_high_water_never_reuses_deleted_max_and_since_finds_inser init_store(db_path) save_snapshot(db_path, snapshot) for index, worker in enumerate(snapshot.workers): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker.id, @@ -13883,7 +10084,7 @@ def test_turn_sequence_high_water_never_reuses_deleted_max_and_since_finds_inser str(highest_turn), ) conn.commit() - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, snapshot.workers[0].id, @@ -13950,7 +10151,7 @@ def test_turn_list_interior_deletion_expires_cursor_but_not_since_watermark( init_store(db_path) save_snapshot(db_path, snapshot) for index, worker in enumerate(snapshot.workers): - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, worker.id, @@ -14006,7 +10207,7 @@ def test_turn_list_interior_deletion_expires_cursor_but_not_since_watermark( assert insertion_poll["turns"] == [] -def test_source_reconciliation_isolates_stable_owners_and_legacy_no_owner( +def test_source_reconciliation_isolates_stable_owners( tmp_path: Path, ) -> None: db_path = tmp_path / "owner-source-isolation.db" @@ -14017,14 +10218,10 @@ def test_source_reconciliation_isolates_stable_owners_and_legacy_no_owner( def snapshot_for( worker_id: str, - stable_key: str | None, + stable_key: str, observed_at: str, ) -> Snapshot: - meta = ( - {"stable_key": stable_key, "stable_key_version": 1} - if stable_key is not None - else {} - ) + meta = {"stable_key": stable_key, "stable_key_version": 1} return Snapshot( host_id=host_id, updated_at=observed_at, @@ -14060,28 +10257,10 @@ def snapshot_for( "owner two final", "2026-07-13T03:01:01+00:00", ), - ( - snapshot_for( - "shared-worker", - None, - "2026-07-13T03:02:00+00:00", - ), - "legacy same-worker final", - "2026-07-13T03:02:01+00:00", - ), - ( - snapshot_for( - "legacy-worker-b", - None, - "2026-07-13T03:03:00+00:00", - ), - "legacy changed-worker final", - "2026-07-13T03:03:01+00:00", - ), ) for snapshot, final_text, observed_at in cases: save_snapshot(db_path, snapshot) - assert merge_turn_content( + assert apply_test_turn_refresh( db_path, host_id, snapshot.workers[0].id, @@ -14103,63 +10282,14 @@ def snapshot_for( assert set(source_turns) == { "owner one final", "owner two final", - "legacy same-worker final", - "legacy changed-worker final", } ids = {str(turn["id"]) for turn in source_turns.values()} tokens = {str(turn["source_turn_id"]) for turn in source_turns.values()} - assert len(ids) == len(tokens) == 4 + assert len(ids) == len(tokens) == 2 assert ( source_turns["owner one final"]["meta"]["stable_key"], source_turns["owner two final"]["meta"]["stable_key"], ) == (stable_key_1, stable_key_2) - assert source_turns["legacy same-worker final"]["meta"].get("stable_key") is None - assert source_turns["legacy changed-worker final"]["meta"].get("stable_key") is None - assert ( - source_turns["legacy same-worker final"]["source_turn_id"] - == "turnsrc-fdc56cfa0289296df514b264" - ) - assert source_turns["legacy same-worker final"]["worker_id"] == "shared-worker" - assert source_turns["legacy changed-worker final"]["worker_id"] == "legacy-worker-b" assert raw_source not in json.dumps(payload, sort_keys=True) with sqlite3.connect(str(db_path)) as conn: assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] - - -def test_v20_to_v21_adds_herdr_turn_watermark_and_provenance_tables( - tmp_path: Path, -) -> None: - db_path = tmp_path / "herdr-turn-v20.db" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=20) - assert conn.execute("PRAGMA user_version").fetchone() == (20,) - assert conn.execute( - """ - SELECT COUNT(*) FROM sqlite_master - WHERE type = 'table' - AND name IN ( - 'herdr_turn_watermarks', - 'herdr_turn_completions' - ) - """ - ).fetchone() == (0,) - os.chmod(tmp_path, 0o700) - os.chmod(db_path, 0o600) - - init_store(db_path) - - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute("PRAGMA user_version").fetchone() == (21,) - assert { - str(row[0]) - for row in conn.execute( - """ - SELECT name FROM sqlite_master - WHERE type = 'table' - AND name IN ( - 'herdr_turn_watermarks', - 'herdr_turn_completions' - ) - """ - ) - } == {"herdr_turn_watermarks", "herdr_turn_completions"} diff --git a/tests/test_turn_delta.py b/tests/test_turn_delta.py index 540a08d..e1a4f57 100644 --- a/tests/test_turn_delta.py +++ b/tests/test_turn_delta.py @@ -81,6 +81,7 @@ def _insert_turn( summary: str | None = None, updated_at: str = TS, extra: Mapping[str, Any] | None = None, + with_revision: bool = True, ) -> None: payload = _payload( turn_id, @@ -113,6 +114,84 @@ def _insert_turn( sequence, ), ) + if with_revision: + store_sqlite._insert_turn_content_revision_conn( + conn, + host_id=host_id, + turn_id=turn_id, + user_text=None, + assistant_final_text=None, + user_state="absent", + final_state="absent", + created_at=updated_at, + is_current=True, + ) + + +def _seed_current_store(db_path: Path, count: int) -> None: + init_store(db_path) + turns: list[tuple[object, ...]] = [] + revisions: list[tuple[object, ...]] = [] + for index in range(count): + turn_id = f"historical-{index:05d}" + worker_id = f"worker-{index % 8}" + status = "working" if index < 8 else "complete" + payload = _payload( + turn_id, + worker_id=worker_id, + status=status, + summary=f"retained public result {index}", + ) + turns.append( + ( + HOST, + turn_id, + worker_id, + status, + TS, + f"fingerprint-{index}", + f"snapshot-{index}", + TS, + stable_json_dumps(payload), + index + 1, + ) + ) + revisions.append( + ( + HOST, + turn_id, + f"empty-revision-{index}", + "absent", + "absent", + TS, + ) + ) + with sqlite3.connect(str(db_path)) as conn: + conn.executemany( + """ + INSERT INTO turns ( + host_id, turn_id, worker_id, status, kind, updated_at, + fingerprint, snapshot_content_fingerprint, observed_at, + payload_json, list_sequence + ) VALUES (?, ?, ?, ?, 'prompt', ?, ?, ?, ?, ?, ?) + """, + turns, + ) + conn.executemany( + """ + INSERT INTO turn_content_revisions ( + host_id, turn_id, content_revision, + user_text, assistant_final_text, user_state, final_state, + user_char_length, user_byte_length, + final_char_length, final_byte_length, + user_page_count, final_page_count, + is_current, created_at, superseded_at + ) VALUES (?, ?, ?, NULL, NULL, ?, ?, 0, 0, 0, 0, 0, 0, 1, ?, NULL) + """, + revisions, + ) + conn.execute("DELETE FROM turn_change_journal") + conn.commit() def _mutate_turn( @@ -160,47 +239,6 @@ def _tombstone_turn(db_path: Path, turn_id: str, replacement: str | None = None) conn.commit() -def _seed_pre_v18_store(db_path: Path, count: int) -> None: - with sqlite3.connect(str(db_path)) as conn: - conn.execute("PRAGMA foreign_keys=ON") - store_sqlite._run_migrations(conn, target_version=17) - rows = [] - for index in range(count): - turn_id = f"historical-{index:05d}" - worker_id = f"worker-{index % 8}" - status = "working" if index < 8 else "complete" - payload = _payload( - turn_id, - worker_id=worker_id, - status=status, - summary=f"retained public result {index}", - ) - rows.append( - ( - HOST, - turn_id, - worker_id, - status, - TS, - f"fingerprint-{index}", - f"snapshot-{index}", - TS, - stable_json_dumps(payload), - index + 1, - ) - ) - conn.executemany( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, status, kind, updated_at, - fingerprint, snapshot_content_fingerprint, observed_at, - payload_json, list_sequence - ) VALUES (?, ?, ?, ?, 'prompt', ?, ?, ?, ?, ?, ?) - """, - rows, - ) - conn.commit() - init_store(db_path) def _bootstrap_checkpoint(db_path: Path, *, limit: int = 100) -> str: @@ -259,7 +297,7 @@ def test_goal13_acceptance_1_to_3_ten_thousand_bootstrap_and_unchanged_polls( ) -> None: """10k bootstrap is stable/bounded; unchanged polls traverse no list/content.""" db_path = tmp_path / "ten-thousand.db" - _seed_pre_v18_store(db_path, 10_000) + _seed_current_store(db_path, 10_000) with sqlite3.connect(str(db_path)) as conn: assert conn.execute("SELECT COUNT(*) FROM turn_change_journal").fetchone() == (0,) @@ -331,7 +369,7 @@ def content_reader_spy(*args: Any, **kwargs: Any) -> dict[str, Any]: def test_bootstrap_size_gate_is_independent_of_client_limit_two(tmp_path: Path) -> None: db_path = tmp_path / "limit-two-bootstrap.db" - _seed_pre_v18_store(db_path, 5_000) + _seed_current_store(db_path, 5_000) first = turn_delta_payload_from_store( db_path, @@ -395,6 +433,13 @@ def test_goal13_acceptance_4_working_mutation_is_one_upsert_and_revision_only_ch """, (HOST, "working-turn"), ).fetchone()[0] + conn.execute( + """ + UPDATE turn_content_revisions SET is_current = 0, superseded_at = ? + WHERE host_id = ? AND turn_id = ? AND is_current = 1 + """, + (TS, HOST, "working-turn"), + ) conn.execute( """ UPDATE turn_content_revisions SET is_current = 1 @@ -421,7 +466,7 @@ def test_current_revision_insert_alone_emits_one_upsert(tmp_path: Path) -> None: db_path = tmp_path / "revision-insert.db" init_store(db_path) with sqlite3.connect(str(db_path)) as conn: - _insert_turn(conn, "revision-insert", 1) + _insert_turn(conn, "revision-insert", 1, with_revision=False) conn.commit() checkpoint = _bootstrap_checkpoint(db_path) @@ -489,6 +534,7 @@ def test_single_oversized_change_degrades_and_advances_checkpoint(tmp_path: Path 1, summary="large public descriptor", extra={"meta": {"items": ["x" * 12_000 for _ in range(100)]}}, + with_revision=False, ) store_sqlite._insert_turn_content_revision_conn( conn, @@ -708,7 +754,7 @@ def test_goal13_acceptance_9_token_outcomes_compaction_and_store_epoch_rebuild( batch_size=10, now="2030-07-18T12:00:00+00:00", ) - assert compacted["deleted"] == 2 + assert compacted["deleted"] == 5 assert turn_delta_payload_from_store( db_path, HOST, watermark=checkpoint )["status"] == "expired_watermark" @@ -745,42 +791,6 @@ def test_goal13_acceptance_9_token_outcomes_compaction_and_store_epoch_rebuild( )["status"] == "invalid_watermark" -@pytest.mark.parametrize("source_version", range(18)) -def test_goal13_acceptance_11_every_prior_migration_installs_empty_v18_journal( - tmp_path: Path, - source_version: int, -) -> None: - db_path = tmp_path / f"migration-{source_version}.db" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=source_version) - assert conn.execute("PRAGMA user_version").fetchone() == (source_version,) - store_sqlite._run_migrations(conn, target_version=18) - assert conn.execute("PRAGMA user_version").fetchone() == (18,) - assert conn.execute("SELECT COUNT(*) FROM turn_change_journal").fetchone() == (0,) - columns = tuple( - row[1] for row in conn.execute("PRAGMA table_info(turn_change_journal)") - ) - assert columns == ("seq", "host_id", "turn_id", "op", "changed_at") - epoch = conn.execute( - "SELECT store_epoch FROM turn_change_state WHERE scope = 'turn-delta'" - ).fetchone() - assert epoch is not None and len(str(epoch[0])) >= 32 - trigger_names = { - str(row[0]) - for row in conn.execute( - "SELECT name FROM sqlite_master WHERE type = 'trigger'" - ) - } - assert { - "trg_turn_change_after_insert", - "trg_turn_change_after_update", - "trg_turn_change_after_delete", - "trg_turn_change_revision_current", - "trg_turn_change_revision_insert_current", - "trg_turn_change_journal_no_update", - } <= trigger_names - assert conn.execute("PRAGMA integrity_check").fetchone() == ("ok",) - assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] def test_goal13_capture_is_trigger_backed_immutable_and_public_minimal(tmp_path: Path) -> None: @@ -821,7 +831,13 @@ def test_delta_page_bytes_do_not_change_when_submission_sweep_fires_mid_request( ) -> None: db_path = tmp_path / "sweep-byte-stability.db" init_store(db_path) - worker = {"id": "worker-0"} + worker = { + "id": "worker-0", + "meta": { + "stable_key": "wsk1_" + ("a" * 64), + "stable_key_version": 1, + }, + } with sqlite3.connect(str(db_path)) as conn: store_sqlite._insert_turn_submission_conn( conn, @@ -861,7 +877,6 @@ def reserve_sweep(*_args: Any, **_kwargs: Any) -> tuple[tuple[str, str, str], bo db_path, HOST, now=current, - turn_model="observed", ) ).encode("utf-8") during = stable_json_dumps( @@ -869,7 +884,6 @@ def reserve_sweep(*_args: Any, **_kwargs: Any) -> tuple[tuple[str, str, str], bo db_path, HOST, now=current, - turn_model="observed", ) ).encode("utf-8") @@ -922,7 +936,7 @@ def journal_since(sequence: int) -> list[tuple[str, str]]: ] before = journal_high() - assert store_sqlite.merge_turn_content( + assert store_sqlite.apply_turn_refresh( db_path, HOST, "worker-0", @@ -934,7 +948,7 @@ def journal_since(sequence: int) -> list[tuple[str, str]]: "has_open_turn": True, }, observed_at="2026-01-01T00:01:00+00:00", - ) == 1 + ).updated == 1 first_rows = journal_since(before) assert first_rows and all(op == "upsert" for _turn_id, op in first_rows) @@ -976,7 +990,7 @@ def test_turn_delta_rpc_advertises_feature_and_cannot_invoke_delivery(tmp_path: assert delivery_calls == [] -def test_turn_delta_cli_bootstrap_and_incremental_read(tmp_path: Path, capsys) -> None: +def test_turn_delta_cli_requires_daemon_and_does_not_read_store(tmp_path: Path, capsys) -> None: db_path = tmp_path / "cli.db" socket_path = tmp_path / "missing.sock" init_store(db_path) @@ -994,15 +1008,11 @@ def test_turn_delta_cli_bootstrap_and_incremental_read(tmp_path: Path, capsys) - "--db-path", str(db_path), ] - assert main(base_args) == 0 - bootstrap = json.loads(capsys.readouterr().out) - assert bootstrap["changes"][0]["turn"]["summary"] == "first CLI projection" - checkpoint = str(bootstrap["checkpoint"]) - - _mutate_turn(db_path, "cli-turn", summary="second CLI projection") - assert main([*base_args, "--watermark", checkpoint]) == 0 - changed = json.loads(capsys.readouterr().out) - assert changed["changes"][0]["turn"]["summary"] == "second CLI projection" + assert main(base_args) == 1 + payload = json.loads(capsys.readouterr().out) + assert payload["ok"] is False + assert payload["status"] == "daemon_unavailable" + assert "changes" not in payload def test_turn_change_retention_config_defaults_env_and_bounds( diff --git a/tests/test_turn_ingestion.py b/tests/test_turn_ingestion.py deleted file mode 100644 index 04213c1..0000000 --- a/tests/test_turn_ingestion.py +++ /dev/null @@ -1,2371 +0,0 @@ -"""Deterministic concurrency tests for daemon-owned turn ingestion.""" - -from __future__ import annotations - -import json -import multiprocessing -import os -import sqlite3 -import threading -import time -from pathlib import Path -from typing import Any - -from tendwire.backends import herdr_turns -from tendwire.backends.herdr_turns import ( - TurnIngestionScheduler, - TurnRefreshResult, -) -from tendwire.config import Config -from tendwire.core.models import Snapshot, Worker, WorkerBinding -from tendwire.core.turns import PendingObservation, PendingObservedChoice -from tendwire.core.projector import project_from_raw -from tendwire.store.sqlite import ( - apply_backend_pending_observation, - init_store, - merge_turn_content, - pending_payload_from_store, - save_snapshot, - turns_payload_from_store, - upsert_worker_bindings, -) - - -def _blocked_codex_child(channel) -> None: - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._CODEX_STATE_IPC_MAX_BYTES, - ) - threading.Event().wait(30) - - -def _wrong_source_codex_child(channel) -> None: - try: - request = json.loads( - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._CODEX_STATE_IPC_MAX_BYTES, - ).decode("utf-8") - ) - response = { - "protocol": 1, - "nonce": request["nonce"], - "disposition": "ok", - "content": { - "user_text": "must not publish", - "assistant_final_text": "must not publish", - "complete": True, - "has_open_turn": False, - "source_turn_id": "wrong-source", - }, - "parser_state": {"source": "omp", "state": None}, - "bytes_read": 0, - } - herdr_turns._blocking_send_frame( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - ) - finally: - channel.close() - - -def _oversized_direct_omp_child(channel) -> None: - try: - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._OMP_REQUEST_MAX_BYTES, - ) - herdr_turns._blocking_send_frame( - channel, - b"x" * (herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES + 1), - ) - finally: - channel.close() - - -def _large_direct_codex_child(channel) -> None: - try: - request = json.loads( - herdr_turns._blocking_recv_frame( - channel, - herdr_turns._CODEX_STATE_IPC_MAX_BYTES, - ).decode("utf-8") - ) - final = "codex-frame-" + ( - "c" * (herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES + 1024) - ) - response = { - "protocol": 1, - "nonce": request["nonce"], - "disposition": "ok", - "content": { - "assistant_final_text": final, - "complete": True, - "has_open_turn": False, - "source_turn_id": "codex-large-frame", - }, - "parser_state": request["parser_state"], - "bytes_read": 0, - } - herdr_turns._blocking_send_frame( - channel, - json.dumps(response, separators=(",", ":")).encode("utf-8"), - ) - finally: - channel.close() - - -def _wait_until(predicate, timeout: float = 2.0) -> None: - deadline = time.monotonic() + timeout - while not predicate(): - remaining = deadline - time.monotonic() - if remaining <= 0: - raise AssertionError("condition did not become true") - threading.Event().wait(min(0.01, remaining)) - - -def _binding(config: Config, worker: Any, ordinal: int, *, target: str | None = None) -> WorkerBinding: - return WorkerBinding( - host_id=config.host_id, - worker_id=worker.id, - worker_fingerprint=worker.fingerprint, - backend="herdr", - target_kind="agent_id", - target_value=f"agent-{ordinal}", - turn_target_kind="pane_id", - turn_target_value=target or f"pane-{ordinal}", - sendable=True, - observed_at="2026-07-12T00:00:00+00:00", - expires_at="9999-12-31T23:59:59+00:00", - private_fingerprint=f"private-{ordinal}", - ) - - -def _scheduler_store(tmp_path: Path, count: int) -> tuple[Config, Any, list[WorkerBinding]]: - config = Config( - host_id="ingestion-host", - db_path=tmp_path / "ingestion.db", - herdr_timeout_seconds=0.5, - turn_refresh_interval_seconds=100.0, - turn_refresh_workers=4, - ) - snapshot = project_from_raw( - config, - workers=[ - {"id": f"worker-{index}", "name": f"worker {index}", "status": "active"} - for index in range(count) - ], - ) - init_store(config.db_path) - save_snapshot(config.db_path, snapshot) - bindings = [_binding(config, worker, index) for index, worker in enumerate(snapshot.workers)] - upsert_worker_bindings(config.db_path, bindings) - return config, snapshot, bindings - - -def test_background_scheduler_discovers_and_clears_pending_without_turn_list( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 1) - current = { - "observation": PendingObservation( - "open_prompt", - question="Background choice?", - pending_kind="question", - choices=( - PendingObservedChoice( - "choice-0123456789abcdef01234567", - "Continue", - 1, - ), - ), - revision_digest="revision-background", - ) - } - utc_now = ["2026-07-13T00:00:00+00:00"] - - def read_pending(*_args, **_kwargs): - return {"_backend_pending_observation": current["observation"]} - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_pending) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - utc_clock=lambda: utc_now[0], - ) - scheduler.start() - try: - _wait_until( - lambda: any( - row["question"] == "Background choice?" - for row in pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - ) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute( - """ - SELECT observation_state, binding_private_fingerprint - FROM backend_pending - WHERE host_id = ? AND worker_id = ? - """, - (config.host_id, bindings[0].worker_id), - ).fetchone() == ("open", bindings[0].private_fingerprint) - current["observation"] = PendingObservation("read_succeeded_no_prompt") - utc_now[0] = "2026-07-13T00:00:01+00:00" - scheduler.request_refresh() - _wait_until( - lambda: not pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute( - """ - SELECT observation_state, freshness, binding_private_fingerprint - FROM backend_pending - WHERE host_id = ? AND worker_id = ? - """, - (config.host_id, bindings[0].worker_id), - ).fetchone() == ("none", "fresh", bindings[0].private_fingerprint) - finally: - scheduler.stop() - - -def test_background_authoritative_scan_reaps_removed_pane_pending( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot, bindings = _scheduler_store(tmp_path, 1) - observation = PendingObservation( - "open_prompt", - question="Will be removed?", - pending_kind="question", - revision_digest="revision-removal", - ) - def read_pending(_config, binding, **_kwargs): - selected = ( - observation - if binding.private_fingerprint == bindings[0].private_fingerprint - else PendingObservation("read_succeeded_no_prompt") - ) - return {"_backend_pending_observation": selected} - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_pending) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - utc_clock=lambda: "2026-07-13T00:00:00+00:00", - ) - scheduler.start() - try: - _wait_until( - lambda: bool( - pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - ) - decoy = _binding(config, snapshot.workers[0], 99) - upsert_worker_bindings(config.db_path, [decoy]) - with sqlite3.connect(config.db_path) as conn: - conn.execute( - """ - DELETE FROM worker_bindings - WHERE host_id = ? AND private_fingerprint = ? - """, - (config.host_id, bindings[0].private_fingerprint), - ) - scheduler.request_refresh() - _wait_until( - lambda: not pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute( - """ - SELECT private_fingerprint FROM worker_bindings - WHERE host_id = ? - """, - (config.host_id,), - ).fetchall() == [(decoy.private_fingerprint,)] - finally: - scheduler.stop() - - -def test_same_key_never_overlaps_and_burst_causes_one_rerun(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - first_entered = threading.Event() - release_first = threading.Event() - second_entered = threading.Event() - calls = 0 - active = 0 - maximum_active = 0 - lock = threading.Lock() - - def reader(_config, _binding, *, adapter_timeout_seconds): - nonlocal calls, active, maximum_active - with lock: - calls += 1 - call = calls - active += 1 - maximum_active = max(maximum_active, active) - if call == 1: - first_entered.set() - assert release_first.wait(2) - else: - second_entered.set() - with lock: - active -= 1 - return TurnRefreshResult("unchanged", 0) - - original_list = herdr_turns.list_worker_bindings - scan_after_burst = threading.Event() - list_calls = 0 - - def observed_list(*args, **kwargs): - nonlocal list_calls - result = original_list(*args, **kwargs) - list_calls += 1 - if first_entered.is_set() and list_calls >= 3: - scan_after_burst.set() - return result - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", observed_list) - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, reader=reader) - scheduler.start() - assert first_entered.wait(2) - for _ in range(20): - scheduler.request_refresh() - assert scan_after_burst.wait(2) - release_first.set() - assert second_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - scheduler.stop() - - assert calls == 2 - assert maximum_active == 1 - assert scheduler.operational_status()["coalesced"] >= 19 - - -def test_scan_dispatches_reader_before_blocked_orphan_prune_completes( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - prune_entered = threading.Event() - reader_entered = threading.Event() - original_prune = herdr_turns.prune_backend_pending - - def observed_prune(*args, **kwargs): - prune_entered.set() - return original_prune(*args, **kwargs) - - def reader(_config, _binding, *, adapter_timeout_seconds): - reader_entered.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "prune_backend_pending", observed_prune) - writer = sqlite3.connect(config.db_path, isolation_level=None, timeout=1) - writer.execute("BEGIN IMMEDIATE") - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - try: - assert prune_entered.wait(2) - assert reader_entered.wait(2) - finally: - writer.rollback() - writer.close() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - scheduler.stop() - - - -def test_transient_initial_binding_scan_retries_once_and_dispatches( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - original_list = herdr_turns.list_worker_bindings - reader_entered = threading.Event() - calls = 0 - lock = threading.Lock() - - def transient_list(*args, **kwargs): - nonlocal calls - with lock: - calls += 1 - call = calls - if call == 1: - raise sqlite3.OperationalError("transient initial read") - return original_list(*args, **kwargs) - - def reader(_config, _binding, *, adapter_timeout_seconds): - reader_entered.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", transient_list) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - assert reader_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - status = scheduler.operational_status() - assert status["failed"] == 1 - assert status["status"] == "healthy" - scheduler.stop() - - -def test_persistent_initial_binding_scan_failure_retries_once_without_spin( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 1) - second_call = threading.Event() - calls = 0 - lock = threading.Lock() - - prune_calls = 0 - - apply_backend_pending_observation( - config.db_path, - config.host_id, - bindings[0].worker_id, - PendingObservation( - "open_prompt", - question="Must survive failed scan?", - pending_kind="question", - revision_digest="failed-scan-revision", - ), - binding_private_fingerprint=bindings[0].private_fingerprint, - observed_turn_target_value=bindings[0].turn_target_value, - ) - def failed_list(*_args, **_kwargs): - nonlocal calls - with lock: - calls += 1 - if calls == 2: - second_call.set() - raise sqlite3.OperationalError("persistent read failure") - - def observed_prune(*_args, **_kwargs): - nonlocal prune_calls - prune_calls += 1 - return 0 - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", failed_list) - monkeypatch.setattr(herdr_turns, "prune_backend_pending", observed_prune) - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100) - scheduler.start() - assert second_call.wait(2) - _wait_until(lambda: scheduler.operational_status()["failed"] == 2) - with scheduler._condition: - assert calls == 2 - assert scheduler._scan_retry_remaining == 0 - assert scheduler._rescan_requested is False - assert scheduler.operational_status()["status"] == "degraded" - assert prune_calls == 0 - assert any( - row["question"] == "Must survive failed scan?" - for row in pending_payload_from_store( - config.db_path, - config.host_id, - )["pending_interactions"] - ) - scheduler.stop() - - -def test_transient_binding_revalidation_failure_retries_after_prune( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - original_list = herdr_turns.list_worker_bindings - original_prune = herdr_turns.prune_backend_pending - validation_failed = threading.Event() - prune_entered = threading.Event() - release_prune = threading.Event() - reader_entered = threading.Event() - validation_calls = 0 - lock = threading.Lock() - - def transient_validation(*args, **kwargs): - nonlocal validation_calls - if threading.current_thread().name.startswith("tendwire-turn-ingestion"): - with lock: - validation_calls += 1 - call = validation_calls - if call == 1: - validation_failed.set() - raise sqlite3.OperationalError("transient validation read") - return original_list(*args, **kwargs) - - def blocked_prune(*args, **kwargs): - prune_entered.set() - assert release_prune.wait(5) - return original_prune(*args, **kwargs) - - def reader(_config, _binding, *, adapter_timeout_seconds): - reader_entered.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", transient_validation) - monkeypatch.setattr(herdr_turns, "prune_backend_pending", blocked_prune) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - try: - assert validation_failed.wait(2) - assert prune_entered.wait(2) - assert not reader_entered.is_set() - release_prune.set() - assert reader_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - finally: - release_prune.set() - scheduler.stop() - status = scheduler.operational_status() - assert validation_calls == 2 - assert status["failed"] == 1 - - -def test_persistent_binding_revalidation_failure_is_bounded_and_skips_reader( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - original_list = herdr_turns.list_worker_bindings - second_failure = threading.Event() - reader_entered = threading.Event() - validation_calls = 0 - lock = threading.Lock() - - def failed_validation(*args, **kwargs): - nonlocal validation_calls - if threading.current_thread().name.startswith("tendwire-turn-ingestion"): - with lock: - validation_calls += 1 - if validation_calls == 2: - second_failure.set() - raise sqlite3.OperationalError("persistent validation read") - return original_list(*args, **kwargs) - - def reader(_config, _binding, *, adapter_timeout_seconds): - reader_entered.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", failed_validation) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - assert second_failure.wait(2) - _wait_until(lambda: scheduler.operational_status()["failed"] == 2) - with scheduler._condition: - key = next(iter(scheduler._binding_retry_remaining)) - assert scheduler._binding_retry_remaining[key] == 0 - assert key not in scheduler._binding_retry_due - assert validation_calls == 2 - assert not reader_entered.is_set() - assert scheduler.operational_status()["status"] == "degraded" - scheduler.stop() - - -def test_dirty_rerun_survives_full_queue_at_completion(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 3) - a_entered = threading.Event() - release_a = threading.Event() - a_finished = threading.Event() - b_entered = threading.Event() - release_b = threading.Event() - c_entered = threading.Event() - release_c = threading.Event() - third_prune_entered = threading.Event() - release_third_prune = threading.Event() - rerun_a_entered = threading.Event() - calls: list[str] = [] - first_target: str | None = None - b_target: str | None = None - target_calls: dict[str, int] = {} - lock = threading.Lock() - - def reader(_config, binding, *, adapter_timeout_seconds): - nonlocal first_target, b_target - target = str(binding.turn_target_value) - with lock: - calls.append(target) - if first_target is None: - first_target = target - target_calls[target] = target_calls.get(target, 0) + 1 - current_target_call = target_calls[target] - if target != first_target and b_target is None: - b_target = target - role = "a" if target == first_target else ("b" if target == b_target else "c") - if role == "a" and current_target_call == 1: - a_entered.set() - assert release_a.wait(5) - a_finished.set() - elif role == "a": - rerun_a_entered.set() - elif role == "b": - b_entered.set() - assert release_b.wait(5) - else: - c_entered.set() - assert release_c.wait(5) - return TurnRefreshResult("unchanged", 0) - - prune_calls = 0 - prune_lock = threading.Lock() - - def block_third_prune(*_args, **_kwargs): - nonlocal prune_calls - with prune_lock: - prune_calls += 1 - call = prune_calls - if call == 3: - third_prune_entered.set() - assert release_third_prune.wait(5) - return 0 - - monkeypatch.setattr(herdr_turns, "prune_backend_pending", block_third_prune) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=2, - queue_capacity=1, - reader=reader, - ) - scheduler.start() - try: - assert a_entered.wait(2) - scheduler.request_refresh() - assert b_entered.wait(2) - assert b_target is not None - with sqlite3.connect(config.db_path) as conn: - conn.execute( - "DELETE FROM worker_bindings WHERE host_id = ? AND turn_target_value = ?", - (config.host_id, b_target), - ) - scheduler.request_refresh() - assert third_prune_entered.wait(2) - assert not c_entered.is_set() - - release_a.set() - assert a_finished.wait(2) - _wait_until( - lambda: any( - item.turn_target_value == first_target and future.done() - for item, future, _started_at in scheduler._running.values() - ) - ) - release_third_prune.set() - assert c_entered.wait(2) - release_b.set() - release_c.set() - assert rerun_a_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - finally: - release_a.set() - release_b.set() - release_c.set() - release_third_prune.set() - scheduler.stop() - - assert first_target is not None - assert calls.count(first_target) == 2 - assert len(calls) == 4 - assert scheduler.operational_status()["queue_full"] >= 2 - - -def test_distinct_keys_use_four_workers_and_fifth_waits(tmp_path: Path) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 5) - release = threading.Event() - four_entered = threading.Event() - fifth_entered = threading.Event() - lock = threading.Lock() - active = 0 - maximum_active = 0 - calls = 0 - - def reader(_config, _binding, *, adapter_timeout_seconds): - nonlocal active, maximum_active, calls - with lock: - calls += 1 - active += 1 - maximum_active = max(maximum_active, active) - if calls == 4: - four_entered.set() - elif calls == 5: - fifth_entered.set() - assert release.wait(2) - with lock: - active -= 1 - return TurnRefreshResult("unchanged", 0) - - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=4, - reader=reader, - ) - scheduler.start() - assert four_entered.wait(2) - assert not fifth_entered.is_set() - status = scheduler.operational_status() - assert status["active"] == 4 - assert status["queue_depth"] == 1 - release.set() - assert fifth_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - scheduler.stop() - assert maximum_active == 4 - - -def test_queue_saturation_is_recovered_by_next_cadence_scan(tmp_path: Path) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 4) - - class Clock: - def __init__(self) -> None: - self.value = 0.0 - self.lock = threading.Lock() - - def __call__(self) -> float: - with self.lock: - return self.value - - def advance(self, seconds: float) -> None: - with self.lock: - self.value += seconds - - clock = Clock() - first_entered = threading.Event() - release = threading.Event() - all_seen = threading.Event() - seen: set[str] = set() - lock = threading.Lock() - - def reader(_config, binding, *, adapter_timeout_seconds): - with lock: - seen.add(binding.private_fingerprint) - if len(seen) == len(bindings): - all_seen.set() - if not first_entered.is_set(): - first_entered.set() - assert release.wait(2) - return TurnRefreshResult("unchanged", 0) - - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=2, - max_workers=1, - queue_capacity=2, - clock=clock, - reader=reader, - ) - scheduler.start() - assert first_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["queue_full"] >= 1) - release.set() - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - assert len(seen) == 2 - clock.advance(2.1) - with scheduler._condition: - scheduler._condition.notify_all() - assert all_seen.wait(2) - scheduler.stop() - assert scheduler.operational_status()["queue_full"] >= 1 - - -def test_target_change_discards_old_result_and_runs_latest_binding(tmp_path: Path, monkeypatch) -> None: - config, snapshot, bindings = _scheduler_store(tmp_path, 1) - old_binding = bindings[0] - old_entered = threading.Event() - release_old = threading.Event() - new_entered = threading.Event() - reads: list[str] = [] - - def read_binding(_config, binding, *, timeout_seconds, cancel_event=None): - target = str(binding.turn_target_value) - reads.append(target) - if target == "pane-0": - old_entered.set() - assert release_old.wait(2) - else: - new_entered.set() - return { - "source_turn_id": "source-stable", - "user_text": "question", - "assistant_final_text": target, - "complete": True, - "has_open_turn": False, - } - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_binding) - original_list = herdr_turns.list_worker_bindings - latest_scan = threading.Event() - - def observed_list(*args, **kwargs): - result = original_list(*args, **kwargs) - if old_entered.is_set() and any(item.turn_target_value == "pane-new" for item in result): - latest_scan.set() - return result - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", observed_list) - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, max_workers=1) - scheduler.start() - assert old_entered.wait(2) - replacement = WorkerBinding( - **{ - **old_binding.__dict__, - "turn_target_value": "pane-new", - "observed_at": "2026-07-12T00:00:01+00:00", - } - ) - upsert_worker_bindings(config.db_path, [replacement]) - scheduler.request_refresh() - assert latest_scan.wait(2) - release_old.set() - assert new_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - scheduler.stop() - - - turns = turns_payload_from_store(config.db_path, config.host_id, snapshot=snapshot)["turns"] - assert any(turn.get("assistant_final_text") == "pane-new" for turn in turns) - assert reads == ["pane-0", "pane-new"] - assert scheduler.operational_status()["failed"] == 1 - - -def test_atomic_binding_guard_closes_revalidation_commit_race(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 1) - original = bindings[0] - with sqlite3.connect(config.db_path) as conn: - baseline_revisions = conn.execute( - "SELECT COUNT(*) FROM turn_content_revisions" - ).fetchone()[0] - replacement = WorkerBinding( - **{ - **original.__dict__, - "turn_target_value": "pane-replaced-after-check", - "observed_at": "2026-07-12T00:00:02+00:00", - } - ) - monkeypatch.setattr( - herdr_turns, - "_read_turn_for_binding", - lambda _config, _binding, *, timeout_seconds, cancel_event=None: { - "source_turn_id": "must-not-commit", - "user_text": "stale question", - "assistant_final_text": "stale final", - "complete": True, - "has_open_turn": False, - }, - ) - - def race_after_revalidation(_config, _item): - upsert_worker_bindings(config.db_path, [replacement]) - return True - - monkeypatch.setattr(herdr_turns, "_binding_still_matches", race_after_revalidation) - result = herdr_turns.refresh_turn_binding(config, original) - - assert result == TurnRefreshResult("stale_binding", 0) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] == baseline_revisions - assert conn.execute( - "SELECT COUNT(*) FROM turns WHERE payload_json LIKE '%must-not-commit%'" - ).fetchone()[0] == 0 - - -def test_omp_compact_checkpoint_publishes_only_after_validated_durable_apply( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 1) - root = tmp_path / "omp-sessions" - session_dir = root / "-retry" - session_dir.mkdir(parents=True) - path = session_dir / "retry.jsonl" - path.write_text( - "\n".join( - json.dumps(line, separators=(",", ":")) - for line in ( - { - "type": "message", - "id": "retry-user", - "message": { - "role": "user", - "attribution": "user", - "content": [{"type": "text", "text": "retry prompt"}], - }, - }, - { - "type": "message", - "id": "retry-final", - "message": { - "role": "assistant", - "stopReason": "stop", - "content": [{"type": "text", "text": "durable final"}], - }, - }, - ) - ), - encoding="utf-8", - ) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - original = bindings[0] - omp_binding = WorkerBinding( - **{ - **original.__dict__, - "turn_target_kind": "omp_session_path", - "turn_target_value": str(path), - "private_fingerprint": "omp-retry-private", - } - ) - upsert_worker_bindings(config.db_path, [omp_binding]) - cache_key = herdr_turns._omp_cache_key(str(path)) - assert cache_key is not None - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._OMP_SESSION_CACHE_LIVE_KEYS = None - - original_matches = herdr_turns._binding_still_matches - monkeypatch.setattr(herdr_turns, "_binding_still_matches", lambda *_args: False) - stale = herdr_turns._refresh_turn_binding( - config, - omp_binding, - adapter_timeout_seconds=10, - ) - assert stale == TurnRefreshResult("stale_binding", 0) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._OMP_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "_binding_still_matches", original_matches) - original_apply = herdr_turns.apply_turn_refresh - - def fail_apply(*_args, **_kwargs): - raise sqlite3.OperationalError("injected apply failure") - - monkeypatch.setattr(herdr_turns, "apply_turn_refresh", fail_apply) - failed = herdr_turns._refresh_turn_binding( - config, - omp_binding, - adapter_timeout_seconds=10, - ) - assert failed == TurnRefreshResult("failed", 0) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._OMP_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "apply_turn_refresh", original_apply) - applied = herdr_turns._refresh_turn_binding( - config, - omp_binding, - adapter_timeout_seconds=10, - ) - assert applied.status == "updated" - with herdr_turns._OMP_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._serialize_omp_state( - herdr_turns._OMP_SESSION_CACHE[cache_key] - ) - assert checkpoint["turn_open"] is False - assert "retry prompt" not in json.dumps(checkpoint) - assert "durable final" not in json.dumps(checkpoint) - - unchanged = herdr_turns._refresh_turn_binding( - config, - omp_binding, - adapter_timeout_seconds=10, - ) - assert unchanged == TurnRefreshResult("unchanged", 0) - - -def test_codex_checkpoint_publication_requires_validated_durable_apply_and_cas( - tmp_path: Path, - monkeypatch, -) -> None: - config, snapshot, bindings = _scheduler_store(tmp_path, 1) - session_id = "019f5590-3333-7333-8333-333333333333" - home = tmp_path / "codex-publication" - path = ( - home - / "sessions" - / "2026" - / "07" - / "12" - / f"rollout-2026-07-12T00-00-00-{session_id}.jsonl" - ) - path.parent.mkdir(parents=True) - turn_id = "codex-durable-turn" - records = ( - { - "type": "event_msg", - "payload": {"type": "task_started", "turn_id": turn_id}, - }, - { - "type": "response_item", - "payload": { - "type": "message", - "role": "user", - "content": [{"type": "input_text", "text": "durable prompt"}], - "internal_chat_message_metadata_passthrough": {"turn_id": turn_id}, - }, - }, - { - "type": "event_msg", - "payload": { - "type": "task_complete", - "turn_id": turn_id, - "last_agent_message": "durable Codex final", - }, - }, - ) - path.write_text( - "\n".join(json.dumps(record, separators=(",", ":")) for record in records) - + "\n", - encoding="utf-8", - ) - monkeypatch.setenv("CODEX_HOME", str(home)) - original = bindings[0] - binding = WorkerBinding( - **{ - **original.__dict__, - "turn_target_kind": "codex_session_id", - "turn_target_value": session_id, - "private_fingerprint": "codex-publication-private", - } - ) - upsert_worker_bindings(config.db_path, [binding]) - cache_key = (str((home / "sessions").resolve()), session_id) - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - herdr_turns._CODEX_SESSION_CACHE.clear() - herdr_turns._CODEX_SESSION_CACHE_LIVE_KEYS = None - - original_matches = herdr_turns._binding_still_matches - monkeypatch.setattr(herdr_turns, "_binding_still_matches", lambda *_args: False) - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) == TurnRefreshResult("stale_binding", 0) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "_binding_still_matches", original_matches) - original_apply = herdr_turns.apply_turn_refresh - monkeypatch.setattr( - herdr_turns, - "apply_turn_refresh", - lambda *_args, **_kwargs: (_ for _ in ()).throw( - sqlite3.OperationalError("injected Codex apply failure") - ), - ) - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) == TurnRefreshResult("failed", 0) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "apply_turn_refresh", original_apply) - real_child = herdr_turns._file_turn_child - monkeypatch.setattr(herdr_turns, "_file_turn_child", _wrong_source_codex_child) - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) == TurnRefreshResult("failed", 0) - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "_file_turn_child", _blocked_codex_child) - before_children = {child.pid for child in multiprocessing.active_children()} - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=0.05, - ) == TurnRefreshResult("timeout", 0) - assert {child.pid for child in multiprocessing.active_children()} == before_children - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - - monkeypatch.setattr(herdr_turns, "_file_turn_child", real_child) - real_commit = herdr_turns._file_publication_commit - monkeypatch.setattr( - herdr_turns, - "_file_publication_commit", - lambda _publication, content: content, - ) - applied_without_checkpoint = herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) - assert applied_without_checkpoint.status == "updated" - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - assert cache_key not in herdr_turns._CODEX_SESSION_CACHE - payload = turns_payload_from_store( - config.db_path, - config.host_id, - snapshot=snapshot, - ) - assert payload["turns"][0]["assistant_final_text"] == "durable Codex final" - - monkeypatch.setattr(herdr_turns, "_file_publication_commit", real_commit) - retry = herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) - assert retry.status == "unchanged" - with herdr_turns._CODEX_SESSION_CACHE_LOCK: - checkpoint = herdr_turns._serialize_codex_state( - herdr_turns._CODEX_SESSION_CACHE[cache_key] - ) - assert "durable prompt" not in json.dumps(checkpoint) - assert "durable Codex final" not in json.dumps(checkpoint) - assert herdr_turns._refresh_turn_binding( - config, - binding, - adapter_timeout_seconds=10, - ) == TurnRefreshResult("unchanged", 0) - - -def _codex_lifecycle_file(tmp_path: Path, monkeypatch, session_id: str) -> None: - home = tmp_path / "codex-lifecycle" - path = ( - home - / "sessions" - / "2026" - / "07" - / "12" - / f"rollout-2026-07-12T00-00-00-{session_id}.jsonl" - ) - path.parent.mkdir(parents=True, exist_ok=True) - path.write_bytes(b"") - monkeypatch.setenv("CODEX_HOME", str(home)) - with herdr_turns._CODEX_PATH_CACHE_LOCK: - herdr_turns._CODEX_PATH_CACHE.clear() - herdr_turns._CODEX_INDEX_GENERATION = None -def test_direct_omp_first_frame_over_chunk_bound_is_rejected_and_reaped( - tmp_path: Path, - monkeypatch, -) -> None: - root = tmp_path / "omp-direct-frame" - path = root / "-session" / "session.jsonl" - path.parent.mkdir(parents=True) - path.write_bytes(b"") - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - monkeypatch.setattr( - herdr_turns, - "_file_turn_child", - _oversized_direct_omp_child, - ) - before_children = {child.pid for child in multiprocessing.active_children()} - - try: - herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(path), - timeout_seconds=5, - ) - except herdr_turns._TurnReadFailed: - pass - else: - raise AssertionError("oversized direct OMP frame was accepted") - assert {child.pid for child in multiprocessing.active_children()} == before_children - - -def test_codex_direct_frame_can_exceed_omp_chunk_bound( - tmp_path: Path, - monkeypatch, -) -> None: - session_id = "019f5590-4444-7444-8444-444444444444" - _codex_lifecycle_file(tmp_path, monkeypatch, session_id) - monkeypatch.setattr( - herdr_turns, - "_file_turn_child", - _large_direct_codex_child, - ) - - observed = herdr_turns._read_file_turn_isolated( - "codex_session_id", - session_id, - timeout_seconds=5, - defer_cache=True, - ) - - assert isinstance(observed, herdr_turns._ObservedFileTurn) - final = observed.content["assistant_final_text"] - assert len(final) > herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES - assert len(final) < herdr_turns._CODEX_IPC_FRAME_MAX_BYTES - - - - -def test_oversized_isolated_request_is_rejected_without_process_or_helper( - monkeypatch, -) -> None: - before_children = {child.pid for child in multiprocessing.active_children()} - before_threads = {thread.ident for thread in threading.enumerate()} - started = time.monotonic() - try: - herdr_turns._read_file_turn_isolated( - "codex_session_id", - "x" * (8 * 1024 * 1024), - timeout_seconds=0.05, - ) - except herdr_turns._TurnReadFailed: - pass - else: - raise AssertionError("oversized private request was accepted") - elapsed = time.monotonic() - started - - assert elapsed < 0.2 - assert {child.pid for child in multiprocessing.active_children()} == before_children - assert {thread.ident for thread in threading.enumerate()} == before_threads - - -def test_isolated_process_construction_failure_closes_both_socket_fds( - tmp_path: Path, - monkeypatch, -) -> None: - session_id = "019f5590-1111-7111-8111-111111111111" - _codex_lifecycle_file(tmp_path, monkeypatch, session_id) - real_socketpair = herdr_turns.socket.socketpair - opened = [] - - def tracked_socketpair(): - pair = real_socketpair() - opened.extend(pair) - return pair - - real_context = multiprocessing.get_context("spawn") - - class FailingContext: - def Process(self, **_kwargs): - raise RuntimeError("injected process construction failure") - - monkeypatch.setattr(herdr_turns.socket, "socketpair", tracked_socketpair) - monkeypatch.setattr( - herdr_turns.multiprocessing, - "get_context", - lambda _method: FailingContext(), - ) - try: - herdr_turns._read_file_turn_isolated( - "codex_session_id", - session_id, - timeout_seconds=0.1, - ) - except RuntimeError: - pass - else: - raise AssertionError("process construction failure was hidden") - - assert len(opened) == 2 - assert all(channel.fileno() == -1 for channel in opened) - assert not real_context.active_children() - - -def test_isolated_start_delay_times_out_then_reaps_with_bounded_grace( - tmp_path: Path, - monkeypatch, -) -> None: - session_id = "019f5590-2222-7222-8222-222222222222" - _codex_lifecycle_file(tmp_path, monkeypatch, session_id) - context = multiprocessing.get_context("spawn") - process_type = type(context.Process()) - original_start = process_type.start - original_deadline_check = herdr_turns._check_ipc_deadline - start_entered = False - - def delayed_start(process): - nonlocal start_entered - start_entered = True - time.sleep(0.08) - return original_start(process) - - def deadline_after_start(deadline, cancel_event): - if start_entered: - original_deadline_check(deadline, cancel_event) - - monkeypatch.setattr(process_type, "start", delayed_start) - monkeypatch.setattr( - herdr_turns, - "_check_ipc_deadline", - deadline_after_start, - ) - before_children = {child.pid for child in multiprocessing.active_children()} - started = time.monotonic() - try: - herdr_turns._read_file_turn_isolated( - "codex_session_id", - session_id, - timeout_seconds=0.02, - ) - except herdr_turns._TurnReadTimeout: - pass - else: - raise AssertionError("delayed process start ignored the request deadline") - elapsed = time.monotonic() - started - - assert elapsed >= 0.08 - assert elapsed < 0.08 + herdr_turns._OMP_TEARDOWN_GRACE_SECONDS + 0.25 - assert {child.pid for child in multiprocessing.active_children()} == before_children - assert not any( - thread.name == "tendwire-turn-ipc" - for thread in threading.enumerate() - ) - - -def test_pathological_reap_never_uses_unbounded_join() -> None: - class NeverReaped: - pid = 123 - - def __init__(self): - self.joins = [] - self.kills = 0 - self.terminates = 0 - - def is_alive(self): - return True - - def join(self, timeout=None): - self.joins.append(timeout) - - def terminate(self): - self.terminates += 1 - - def kill(self): - self.kills += 1 - - process = NeverReaped() - started = time.monotonic() - herdr_turns._terminate_and_reap(process) - elapsed = time.monotonic() - started - - assert process.terminates == 1 - assert process.kills == 2 - assert process.joins - assert all(timeout is not None and timeout >= 0 for timeout in process.joins) - assert elapsed < herdr_turns._OMP_TEARDOWN_GRACE_SECONDS - - - - -def test_back_to_back_frames_are_received_without_trailing_byte_loss() -> None: - sender, receiver = herdr_turns.socket.socketpair() - try: - payloads = (b"first-frame", b"second-frame") - sender.sendall( - b"".join( - herdr_turns._OMP_FRAME_HEADER.pack(len(payload)) + payload - for payload in payloads - ) - ) - receiver.setblocking(False) - deadline = time.monotonic() + 1 - assert herdr_turns._recv_frame_until(receiver, deadline, None, 1024) == payloads[0] - assert herdr_turns._recv_frame_until(receiver, deadline, None, 1024) == payloads[1] - finally: - sender.close() - receiver.close() - - -def test_streamed_omp_response_reassembles_small_coalesced_chunks_without_leaks() -> None: - sender, receiver = herdr_turns.socket.socketpair() - receiver.setblocking(False) - payload = (b"chunked-private-response-" * 257) + b"end" - nonce = "stream-nonce" - failures = [] - - def send() -> None: - try: - herdr_turns._blocking_send_streamed_omp_response( - sender, - payload, - nonce, - chunk_bytes=7, - ) - except BaseException as exc: - failures.append(exc) - finally: - sender.close() - - thread = threading.Thread(target=send, name="test-omp-stream-sender") - thread.start() - try: - deadline = time.monotonic() + 5 - first = herdr_turns._recv_frame_until(receiver, deadline, None, 1024) - assembled = herdr_turns._recv_streamed_omp_response_until( - receiver, - first, - nonce, - "omp_session_path", - deadline, - None, - ) - finally: - receiver.close() - thread.join(5) - assert assembled == payload - assert failures == [] - assert thread.is_alive() is False - -def test_stream_manifest_rejects_chunk_bound_above_one_mib() -> None: - sender, receiver = herdr_turns.socket.socketpair() - receiver.setblocking(False) - nonce = "oversized-chunk-manifest" - chunk_bytes = herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES + 1 - try: - herdr_turns._blocking_send_streamed_omp_response( - sender, - b"x", - nonce, - chunk_bytes=chunk_bytes, - ) - except ValueError: - pass - else: - raise AssertionError("oversized OMP sender chunk bound was accepted") - manifest = json.dumps( - { - "protocol": 1, - "nonce": nonce, - "stream": "omp_response", - "chunks": 1, - "total_bytes": 1, - "chunk_bytes": chunk_bytes, - }, - separators=(",", ":"), - ).encode("utf-8") - herdr_turns._blocking_send_frame(sender, manifest) - sender.close() - try: - deadline = time.monotonic() + 1 - first = herdr_turns._recv_frame_until( - receiver, - deadline, - None, - herdr_turns._OMP_IPC_RESPONSE_CHUNK_BYTES, - ) - try: - herdr_turns._recv_streamed_omp_response_until( - receiver, - first, - nonce, - "omp_session_path", - deadline, - None, - ) - except herdr_turns._TurnReadFailed: - pass - else: - raise AssertionError("oversized OMP chunk manifest was accepted") - finally: - receiver.close() - - -def test_streamed_omp_response_rejects_extra_frame_after_terminator() -> None: - sender, receiver = herdr_turns.socket.socketpair() - receiver.setblocking(False) - nonce = "extra-frame-nonce" - manifest = json.dumps( - { - "protocol": 1, - "nonce": nonce, - "stream": "omp_response", - "chunks": 1, - "total_bytes": 3, - "chunk_bytes": 3, - }, - separators=(",", ":"), - ).encode("utf-8") - end = json.dumps( - {"protocol": 1, "nonce": nonce, "stream": "omp_response_end"}, - separators=(",", ":"), - ).encode("utf-8") - frames = (manifest, b"abc", end, b"unexpected") - sender.sendall( - b"".join( - herdr_turns._OMP_FRAME_HEADER.pack(len(payload)) + payload - for payload in frames - ) - ) - sender.close() - try: - deadline = time.monotonic() + 1 - first = herdr_turns._recv_frame_until(receiver, deadline, None, 1024) - try: - herdr_turns._recv_streamed_omp_response_until( - receiver, - first, - nonce, - "omp_session_path", - deadline, - None, - ) - except herdr_turns._TurnReadFailed: - pass - else: - raise AssertionError("extra streamed IPC frame was accepted") - finally: - receiver.close() - - - -def test_file_adapter_timeout_kills_reaps_and_leaves_no_ipc_threads(tmp_path: Path, monkeypatch) -> None: - root = tmp_path / "sessions" - root.mkdir() - fifo = root / "blocked.jsonl" - os.mkfifo(fifo) - monkeypatch.setenv("OMP_SESSIONS_DIR", str(root)) - before_children = {child.pid for child in multiprocessing.active_children()} - before_threads = {thread.ident for thread in threading.enumerate() if thread.name == "tendwire-turn-ipc"} - started = time.monotonic() - - for _ in range(3): - try: - herdr_turns._read_file_turn_isolated( - "omp_session_path", - str(fifo), - timeout_seconds=0.1, - ) - except herdr_turns._TurnReadTimeout: - pass - else: - raise AssertionError("blocked file reader did not time out") - elapsed = time.monotonic() - started - - assert {child.pid for child in multiprocessing.active_children()} == before_children - assert {thread.ident for thread in threading.enumerate() if thread.name == "tendwire-turn-ipc"} == before_threads - assert elapsed < 2.0 - - -def test_stop_terminates_and_reaps_active_pane_adapter(tmp_path: Path, monkeypatch) -> None: - base_config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - pid_file = tmp_path / "adapter.pid" - adapter = tmp_path / "blocked_adapter.py" - adapter.write_text( - "#!/usr/bin/env python3\n" - "import os, time\n" - "with open(os.environ['TENDWIRE_TEST_ADAPTER_PID'], 'w') as handle:\n" - " handle.write(str(os.getpid()))\n" - "time.sleep(30)\n", - encoding="utf-8", - ) - adapter.chmod(0o700) - monkeypatch.setenv("TENDWIRE_TEST_ADAPTER_PID", str(pid_file)) - config = Config( - host_id=base_config.host_id, - db_path=base_config.db_path, - herdr_bin=str(adapter), - herdr_timeout_seconds=10, - turn_refresh_interval_seconds=100, - turn_refresh_workers=1, - ) - scheduler = TurnIngestionScheduler(config) - scheduler.start() - _wait_until( - lambda: pid_file.exists() - and bool(pid_file.read_text(encoding="utf-8").strip()) - ) - pid = int(pid_file.read_text(encoding="utf-8")) - started = time.monotonic() - scheduler.stop(flush_timeout_seconds=1) - elapsed = time.monotonic() - started - - assert elapsed < 1 - assert scheduler.operational_status()["active"] == 0 - assert not any( - thread.name.startswith("tendwire-turn-ingestion") - for thread in threading.enumerate() - ) - try: - os.kill(pid, 0) - except ProcessLookupError: - pass - else: - raise AssertionError("pane adapter child was not reaped") - - -def test_stop_rejects_new_refresh_and_boundedly_drains_started_work(tmp_path: Path) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - entered = threading.Event() - release = threading.Event() - finished = threading.Event() - calls = 0 - - def reader(_config, _binding, *, adapter_timeout_seconds): - nonlocal calls - calls += 1 - entered.set() - assert release.wait(2) - finished.set() - return TurnRefreshResult("unchanged", 0) - - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, reader=reader) - scheduler.start() - assert entered.wait(2) - stopper = threading.Thread( - target=lambda: scheduler.stop(flush_timeout_seconds=0.5), - name="test-scheduler-stopper", - ) - started = time.monotonic() - stopper.start() - _wait_until(lambda: scheduler.operational_status()["status"] == "stopping") - scheduler.request_refresh() - release.set() - stopper.join(1) - assert not stopper.is_alive() - assert time.monotonic() - started < 1 - assert finished.wait(1) - assert calls == 1 - assert scheduler.operational_status()["queue_depth"] == 0 - - -def test_direct_fallback_actively_feeds_only_worker_bound(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, bindings = _scheduler_store(tmp_path, 5) - release = threading.Event() - two_entered = threading.Event() - lock = threading.Lock() - calls: list[str] = [] - active = 0 - maximum_active = 0 - - def reader( - _config, - binding, - *, - adapter_timeout_seconds, - cancel_event=None, - apply_deadline_monotonic=None, - ): - nonlocal active, maximum_active - with lock: - calls.append(binding.private_fingerprint) - active += 1 - maximum_active = max(maximum_active, active) - if len(calls) == 2: - two_entered.set() - assert release.wait(2) - with lock: - active -= 1 - return TurnRefreshResult("updated", 1) - - monkeypatch.setattr(herdr_turns, "_refresh_turn_binding", reader) - returned: list[dict[str, Any]] = [] - fallback = threading.Thread( - target=lambda: returned.append( - herdr_turns.refresh_structured_turn_content( - config, - max_workers=2, - total_timeout_seconds=2, - ) - ), - name="test-turn-fallback", - ) - fallback.start() - assert two_entered.wait(2) - assert len(calls) == 2 - release.set() - fallback.join(2) - assert not fallback.is_alive() - assert returned == [{"ok": True, "status": "ok", "updated": 5, "attempted": 5}] - assert len(calls) == len(bindings) - assert len(set(calls)) == len(bindings) - assert maximum_active == 2 - - -def test_direct_fallback_total_deadline_stops_feeding_new_work(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 5) - calls = 0 - lock = threading.Lock() - - def reader( - _config, - _binding, - *, - adapter_timeout_seconds, - cancel_event=None, - apply_deadline_monotonic=None, - ): - nonlocal calls - with lock: - calls += 1 - threading.Event().wait(adapter_timeout_seconds) - return TurnRefreshResult("timeout", 0) - - monkeypatch.setattr(herdr_turns, "_refresh_turn_binding", reader) - started = time.monotonic() - result = herdr_turns.refresh_structured_turn_content( - config, - max_workers=2, - total_timeout_seconds=1, - ) - assert result == { - "ok": False, - "status": "deadline_exceeded", - "updated": 0, - "attempted": 2, - } - assert calls == 2 - assert time.monotonic() - started < 1 - _wait_until( - lambda: not any( - thread.name.startswith("tendwire-turn-fallback") - for thread in threading.enumerate() - ) - ) - - -def test_direct_fallback_deadline_reaps_real_pane_child(tmp_path: Path, monkeypatch) -> None: - base_config, _snapshot, _bindings = _scheduler_store(tmp_path, 3) - pid_file = tmp_path / "fallback-adapter.pid" - adapter = tmp_path / "fallback_blocked_adapter.py" - adapter.write_text( - "#!/usr/bin/env python3\n" - "import os, time\n" - "with open(os.environ['TENDWIRE_TEST_FALLBACK_PID'], 'w') as handle:\n" - " handle.write(str(os.getpid()))\n" - "time.sleep(30)\n", - encoding="utf-8", - ) - adapter.chmod(0o700) - monkeypatch.setenv("TENDWIRE_TEST_FALLBACK_PID", str(pid_file)) - config = Config( - host_id=base_config.host_id, - db_path=base_config.db_path, - herdr_bin=str(adapter), - herdr_timeout_seconds=10, - turn_refresh_interval_seconds=100, - turn_refresh_workers=1, - ) - - started = time.monotonic() - result = herdr_turns.refresh_structured_turn_content( - config, - max_workers=1, - total_timeout_seconds=1, - ) - elapsed = time.monotonic() - started - - assert result == { - "ok": False, - "status": "deadline_exceeded", - "updated": 0, - "attempted": 1, - } - assert elapsed < 1 - assert pid_file.exists() - pid = int(pid_file.read_text(encoding="utf-8")) - _wait_until( - lambda: not any( - thread.name.startswith("tendwire-turn-fallback") - for thread in threading.enumerate() - ) - ) - try: - os.kill(pid, 0) - except ProcessLookupError: - pass - else: - raise AssertionError("fallback pane child was not reaped") - - -def test_fallback_deadline_cancels_blocked_store_apply_without_late_commit( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 3) - content = { - "source_turn_id": "blocked-fallback-source", - "user_text": "must not commit", - "assistant_final_text": "must not commit", - "complete": True, - "has_open_turn": False, - } - read_done = threading.Event() - - def read_now(_config, _binding, *, timeout_seconds, cancel_event=None): - read_done.set() - return content - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_now) - with sqlite3.connect(config.db_path) as conn: - baseline = conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - blocker = sqlite3.connect(config.db_path, isolation_level=None) - blocker.execute("BEGIN IMMEDIATE") - returned: list[dict[str, Any]] = [] - fallback = threading.Thread( - target=lambda: returned.append( - herdr_turns.refresh_structured_turn_content( - config, - max_workers=1, - total_timeout_seconds=1, - ) - ), - name="test-locked-fallback", - ) - started = time.monotonic() - fallback.start() - assert read_done.wait(1) - fallback.join(1.1) - elapsed = time.monotonic() - started - assert not fallback.is_alive() - assert elapsed < 1.1 - assert returned[0]["status"] == "deadline_exceeded" - blocker.rollback() - blocker.close() - _wait_until( - lambda: not any( - thread.name.startswith("tendwire-turn-fallback") - for thread in threading.enumerate() - ) - ) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] == baseline - assert conn.execute( - "SELECT COUNT(*) FROM turns WHERE payload_json LIKE '%blocked-fallback-source%'" - ).fetchone()[0] == 0 - - - - -def test_fallback_prune_obeys_total_deadline_without_late_delete( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - with sqlite3.connect(config.db_path) as conn: - conn.execute( - """ - INSERT INTO backend_pending (host_id, worker_id, payload_json, observed_at) - VALUES (?, ?, ?, ?) - """, - ( - config.host_id, - "orphan-worker", - '{"prompt":"must survive"}', - "2026-07-12T00:00:00+00:00", - ), - ) - job_finished = threading.Event() - - def reader( - _config, - _binding, - *, - adapter_timeout_seconds, - cancel_event=None, - apply_deadline_monotonic=None, - ): - job_finished.set() - return TurnRefreshResult("unchanged", 0) - - monkeypatch.setattr(herdr_turns, "_refresh_turn_binding", reader) - blocker = sqlite3.connect(config.db_path, isolation_level=None) - blocker.execute("BEGIN IMMEDIATE") - started = time.monotonic() - result = herdr_turns.refresh_structured_turn_content( - config, - max_workers=1, - total_timeout_seconds=0.4, - ) - elapsed = time.monotonic() - started - - assert job_finished.is_set() - assert result == { - "ok": False, - "status": "deadline_exceeded", - "updated": 0, - "attempted": 1, - } - assert elapsed < 0.7 - blocker.rollback() - blocker.close() - threading.Event().wait(0.2) - with sqlite3.connect(config.db_path) as conn: - assert conn.execute( - """ - SELECT COUNT(*) FROM backend_pending - WHERE host_id = ? AND worker_id = ? - """, - (config.host_id, "orphan-worker"), - ).fetchone()[0] == 1 - _wait_until( - lambda: not any( - thread.name.startswith("tendwire-turn-fallback") - for thread in threading.enumerate() - ) - ) - - -def test_scheduler_stop_cancels_blocked_store_apply_without_late_commit( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - read_done = threading.Event() - - def read_now(_config, _binding, *, timeout_seconds, cancel_event=None): - read_done.set() - return { - "source_turn_id": "blocked-scheduler-source", - "user_text": "must not commit", - "assistant_final_text": "must not commit", - "complete": True, - "has_open_turn": False, - } - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_now) - monkeypatch.setattr(herdr_turns, "prune_backend_pending", lambda *args, **kwargs: 0) - with sqlite3.connect(config.db_path) as conn: - baseline = conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - blocker = sqlite3.connect(config.db_path, isolation_level=None) - blocker.execute("BEGIN IMMEDIATE") - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, max_workers=1) - scheduler.start() - assert read_done.wait(1) - started = time.monotonic() - scheduler.stop(flush_timeout_seconds=0.5) - elapsed = time.monotonic() - started - assert elapsed < 0.5 - assert scheduler.operational_status()["active"] == 0 - blocker.rollback() - blocker.close() - with sqlite3.connect(config.db_path) as conn: - assert conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] == baseline - assert conn.execute( - "SELECT COUNT(*) FROM turns WHERE payload_json LIKE '%blocked-scheduler-source%'" - ).fetchone()[0] == 0 - - -def test_scheduler_restart_with_identical_final_is_revision_noop(tmp_path: Path, monkeypatch) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - content = { - "source_turn_id": "restart-source", - "user_text": "same prompt", - "assistant_final_text": "same final", - "complete": True, - "has_open_turn": False, - } - monkeypatch.setattr( - herdr_turns, - "_read_turn_for_binding", - lambda _config, _binding, *, timeout_seconds, cancel_event=None: content, - ) - - def run_once() -> None: - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100, max_workers=1) - scheduler.start() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - scheduler.stop() - - run_once() - with sqlite3.connect(config.db_path) as conn: - first = conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - run_once() - with sqlite3.connect(config.db_path) as conn: - second = conn.execute("SELECT COUNT(*) FROM turn_content_revisions").fetchone()[0] - assert second == first - - -def test_operational_status_recovers_after_successful_empty_binding_scan( - tmp_path: Path, - monkeypatch, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 0) - stale_cache_key = "disappeared-omp-binding" - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._omp_cache_store_locked( - stale_cache_key, - herdr_turns._OmpSessionState(offset=1, file_id=(1, 1)), - ) - original_list = herdr_turns.list_worker_bindings - allow_success = threading.Event() - successful_scan = threading.Event() - - def flaky_list(*args, **kwargs): - if not allow_success.is_set(): - raise sqlite3.OperationalError("deterministic scan failure") - bindings = original_list(*args, **kwargs) - successful_scan.set() - return bindings - - monkeypatch.setattr(herdr_turns, "list_worker_bindings", flaky_list) - scheduler = TurnIngestionScheduler(config, refresh_interval_seconds=100) - scheduler.start() - _wait_until(lambda: scheduler.operational_status()["failed"] == 1) - assert scheduler.operational_status()["status"] == "degraded" - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert stale_cache_key in herdr_turns._OMP_SESSION_CACHE - - allow_success.set() - scheduler.request_refresh() - assert successful_scan.wait(2) - _wait_until(lambda: scheduler.operational_status()["status"] == "stale") - recovered = scheduler.operational_status() - assert recovered["failed"] == 1 - assert recovered["refreshed"] == 0 - assert recovered["timed_out"] == 0 - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert stale_cache_key not in herdr_turns._OMP_SESSION_CACHE - scheduler.stop() - - -def test_fallback_successful_empty_scan_prunes_disappeared_omp_cache( - tmp_path: Path, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 0) - with herdr_turns._OMP_SESSION_CACHE_LOCK: - herdr_turns._OMP_SESSION_CACHE.clear() - herdr_turns._omp_cache_store_locked( - "fallback-disappeared-omp", - herdr_turns._OmpSessionState(offset=1, file_id=(1, 1)), - ) - - result = herdr_turns.refresh_structured_turn_content(config) - - assert result == {"ok": True, "status": "ok", "updated": 0, "attempted": 0} - with herdr_turns._OMP_SESSION_CACHE_LOCK: - assert not herdr_turns._OMP_SESSION_CACHE - - -def test_operational_status_recovers_from_completed_failures_and_stale_churn( - tmp_path: Path, -) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 1) - outcomes = [ - "failed", - "unchanged", - "stale_binding", - "updated", - "timeout", - "unchanged", - "blocked", - ] - blocked_entered = threading.Event() - release_blocked = threading.Event() - calls = 0 - lock = threading.Lock() - - def reader(_config, _binding, *, adapter_timeout_seconds): - nonlocal calls - with lock: - outcome = outcomes[calls] - calls += 1 - if outcome == "blocked": - blocked_entered.set() - assert release_blocked.wait(2) - outcome = "unchanged" - return TurnRefreshResult(outcome, 1 if outcome == "updated" else 0) - - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - reader=reader, - ) - scheduler.start() - _wait_until(lambda: scheduler.operational_status()["failed"] == 1) - failed_now = scheduler.operational_status() - assert failed_now["status"] == "degraded" - assert failed_now["refreshed"] == 0 - assert failed_now["timed_out"] == 0 - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 1) - recovered = scheduler.operational_status() - assert recovered["status"] == "healthy" - assert recovered["failed"] == 1 - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["failed"] == 2) - stale_churn = scheduler.operational_status() - assert stale_churn["status"] == "healthy" - assert stale_churn["refreshed"] == 1 - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 2) - assert scheduler.operational_status()["status"] == "healthy" - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["timed_out"] == 1) - current_timeout = scheduler.operational_status() - assert current_timeout["status"] == "degraded" - assert current_timeout["failed"] == 2 - assert current_timeout["refreshed"] == 2 - - scheduler.request_refresh() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 3) - assert scheduler.operational_status()["status"] == "healthy" - - scheduler.request_refresh() - assert blocked_entered.wait(2) - active_fresh = scheduler.operational_status() - assert active_fresh["active"] == 1 - assert active_fresh["status"] == "healthy" - release_blocked.set() - _wait_until(lambda: scheduler.operational_status()["refreshed"] == 4) - final = scheduler.operational_status() - assert final["status"] == "healthy" - assert final["failed"] == 2 - assert final["timed_out"] == 1 - assert final["refreshed"] == 4 - scheduler.stop() - - -def test_operational_status_has_only_fixed_aggregate_fields(tmp_path: Path) -> None: - config, _snapshot, _bindings = _scheduler_store(tmp_path, 0) - scheduler = TurnIngestionScheduler(config) - assert set(scheduler.operational_status()) == { - "status", - "queue_depth", - "active", - "refreshed", - "failed", - "timed_out", - "coalesced", - "queue_full", - "last_success", - "last_duration_ms", - "stale_age_seconds", - "max_workers", - "queue_capacity", - "refresh_interval_seconds", - "adapter_timeout_seconds", - } - - -def test_structured_refresh_rebinds_same_owner_after_stale_a_rejection_and_current_b_retry( - tmp_path: Path, - monkeypatch, -) -> None: - config = Config( - host_id="ingestion-owner-host", - db_path=tmp_path / "ingestion-owner.db", - herdr_timeout_seconds=0.5, - turn_refresh_interval_seconds=100.0, - turn_refresh_workers=1, - ) - assert config.db_path is not None - stable_key = "wsk1_" + ("7" * 64) - worker_a = Worker( - id="structured-worker-a", - name="Structured Worker A", - status="active", - space_id="structured-space-a", - fingerprint="structured-fingerprint-a", - meta={"stable_key": stable_key, "stable_key_version": 1}, - ) - worker_b = Worker( - id="structured-worker-b", - name="Structured Worker B", - status="waiting", - space_id="structured-space-b", - fingerprint="structured-fingerprint-b", - meta={"stable_key": stable_key, "stable_key_version": 1}, - ) - snapshot_a = Snapshot( - host_id=config.host_id, - updated_at="2026-07-13T05:00:00+00:00", - workers=[worker_a], - ) - snapshot_b = Snapshot( - host_id=config.host_id, - updated_at="2026-07-13T05:01:00+00:00", - workers=[worker_b], - ) - binding_a = _binding(config, worker_a, 70, target="structured-pane-a-private") - binding_b = _binding(config, worker_b, 71, target="structured-pane-b-private") - raw_source = "019f5590-4444-7444-8444-444444444444" - - init_store(config.db_path) - save_snapshot(config.db_path, snapshot_a) - upsert_worker_bindings(config.db_path, [binding_a]) - assert merge_turn_content( - config.db_path, - config.host_id, - worker_a.id, - { - "source_turn_id": raw_source, - "user_text": "stable owner prompt", - "assistant_final_text": "initial A final", - "complete": True, - "has_open_turn": False, - }, - observed_at="2026-07-13T05:00:01+00:00", - ) == 1 - with sqlite3.connect(str(config.db_path)) as conn: - source_before = conn.execute( - """ - SELECT turn_id, list_sequence, - json_extract(payload_json, '$.source_turn_id') - FROM turns - WHERE host_id = ? - AND json_extract(payload_json, '$.source_turn_id') IS NOT NULL - """, - (config.host_id,), - ).fetchone() - list_state_before = conn.execute( - """ - SELECT next_sequence, traversal_generation - FROM turn_list_hosts - WHERE host_id = ? - """, - (config.host_id,), - ).fetchone() - assert source_before is not None - - old_entered = threading.Event() - release_old = threading.Event() - current_entered = threading.Event() - reads: list[tuple[str, str]] = [] - - def read_binding(_config, binding, *, timeout_seconds, cancel_event=None): - reads.append((str(binding.worker_id), str(binding.turn_target_value))) - if binding.worker_id == worker_a.id: - old_entered.set() - assert release_old.wait(2) - final_text = "stale A final must not commit" - else: - current_entered.set() - final_text = "current B final" - return { - "source_turn_id": raw_source, - "user_text": "stable owner prompt", - "assistant_final_text": final_text, - "complete": True, - "has_open_turn": False, - } - - monkeypatch.setattr(herdr_turns, "_read_turn_for_binding", read_binding) - scheduler = TurnIngestionScheduler( - config, - refresh_interval_seconds=100, - max_workers=1, - ) - scheduler.start() - try: - assert old_entered.wait(2) - save_snapshot(config.db_path, snapshot_b) - upsert_worker_bindings(config.db_path, [binding_b]) - with sqlite3.connect(str(config.db_path)) as conn: - conn.execute( - """ - DELETE FROM worker_bindings - WHERE host_id = ? AND private_fingerprint = ? - """, - (config.host_id, binding_a.private_fingerprint), - ) - scheduler.request_refresh() - release_old.set() - assert current_entered.wait(2) - _wait_until(lambda: scheduler.operational_status()["active"] == 0) - finally: - release_old.set() - scheduler.stop() - - public_payload = turns_payload_from_store( - config.db_path, - config.host_id, - snapshot=snapshot_b, - schema_version=2, - ) - source_turns = [ - turn for turn in public_payload["turns"] if turn.get("source_turn_id") - ] - assert len(source_turns) == 1 - source_turn = source_turns[0] - assert source_turn["id"] == source_before[0] - assert source_turn["source_turn_id"] == source_before[2] - assert source_turn["source_turn_id"] != raw_source - assert source_turn["worker_id"] == worker_b.id - assert source_turn["worker_fingerprint"] == worker_b.fingerprint - assert source_turn["space_id"] == worker_b.space_id - assert source_turn["assistant_final_text"] == "current B final" - assert source_turn["complete"] is True - assert source_turn["has_open_turn"] is False - assert reads == [ - (worker_a.id, "structured-pane-a-private"), - (worker_b.id, "structured-pane-b-private"), - ] - assert scheduler.operational_status()["failed"] == 1 - with sqlite3.connect(str(config.db_path)) as conn: - persisted_source = conn.execute( - """ - SELECT turn_id, list_sequence, - json_extract(payload_json, '$.source_turn_id') - FROM turns - WHERE host_id = ? - AND json_extract(payload_json, '$.source_turn_id') IS NOT NULL - """, - (config.host_id,), - ).fetchone() - list_state_after = conn.execute( - """ - SELECT next_sequence, traversal_generation - FROM turn_list_hosts - WHERE host_id = ? - """, - (config.host_id,), - ).fetchone() - current_revisions = conn.execute( - """ - SELECT COUNT(*) - FROM turn_content_revisions - WHERE host_id = ? AND turn_id = ? AND is_current = 1 - """, - (config.host_id, source_turn["id"]), - ).fetchone()[0] - foreign_keys = conn.execute("PRAGMA foreign_key_check").fetchall() - assert persisted_source == source_before - assert persisted_source[1] == source_before[1] - assert list_state_after == list_state_before - assert current_revisions == 1 - assert foreign_keys == [] - encoded = json.dumps( - [public_payload, scheduler.operational_status()], - sort_keys=True, - ) - for private_value in ( - raw_source, - "structured-pane-a-private", - "structured-pane-b-private", - binding_a.private_fingerprint, - binding_b.private_fingerprint, - "stale A final must not commit", - ): - assert private_value not in encoded diff --git a/tests/test_turn_ingestion_benchmark.py b/tests/test_turn_ingestion_benchmark.py deleted file mode 100644 index e143e4f..0000000 --- a/tests/test_turn_ingestion_benchmark.py +++ /dev/null @@ -1,168 +0,0 @@ -from __future__ import annotations - -import importlib.util -import json -import os -import subprocess -import sys -from pathlib import Path -from types import ModuleType - -import pytest - - -ROOT = Path(__file__).resolve().parents[1] -SCRIPT = ROOT / "scripts" / "turn_ingestion_benchmark.py" - - -def _load_driver() -> ModuleType: - spec = importlib.util.spec_from_file_location("turn_ingestion_benchmark", SCRIPT) - assert spec is not None and spec.loader is not None - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -def _invoke(*arguments: str) -> subprocess.CompletedProcess[str]: - environment = os.environ.copy() - environment["PYTHONPATH"] = str(ROOT / "src") - return subprocess.run( - [sys.executable, str(SCRIPT), *arguments], - cwd=ROOT, - stdin=subprocess.DEVNULL, - capture_output=True, - text=True, - env=environment, - check=False, - timeout=30, - ) - - -def _single_object(completed: subprocess.CompletedProcess[str]) -> dict[str, object]: - assert completed.stderr == "" - lines = completed.stdout.splitlines() - assert len(lines) == 1 - payload = json.loads(lines[0]) - assert isinstance(payload, dict) - assert lines[0] == json.dumps( - payload, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - ) - return payload - - -def test_pending_validator_requires_fixed_durable_health_shape() -> None: - driver = _load_driver() - result = { - "schema_version": 1, - "host_id": "benchmark-validator-host", - "pending_interactions": [], - "backend_health": [], - "pending_health": { - "status": "healthy", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - }, - } - result["content_fingerprint"] = ( - driver.recompute_pending_content_fingerprint(result) - ) - valid = {"ok": True, "result": result} - - driver._validate_pending(valid, 2) - for invalid in ( - {**valid, "ok": False}, - { - "ok": True, - "result": { - **valid["result"], - "pending_health": { - "status": "store_unavailable", - "counts": {"fresh": 0, "stale": 0, "total": 0}, - }, - }, - }, - { - "ok": True, - "result": { - **valid["result"], - "pending_health": { - "status": "healthy", - "counts": {"fresh": 1, "stale": 0, "total": 0}, - }, - }, - }, - { - "ok": True, - "result": { - **valid["result"], - "content_fingerprint": "0" * 24, - }, - }, - ): - with pytest.raises(RuntimeError, match="pending_list_contract_failed"): - driver._validate_pending(invalid, 2) - - -@pytest.mark.skipif( - not sys.platform.startswith("linux") or not Path("/dev/shm").is_dir(), - reason="benchmark contract requires Linux tmpfs and Unix sockets", -) -def test_tiny_run_measures_production_pending_without_source_or_turn_reads() -> None: - completed = _invoke( - "--workers", - "2", - "--blocked-workers", - "2", - "--blocked-seconds", - "0.1", - "--warmups", - "0", - "--samples", - "1", - "--json", - ) - - assert completed.returncode == 0 - report = _single_object(completed) - assert report["ok"] is True - assert report["status"] == "completed" - assert report["latency_ns"]["pending_list"]["samples"] == 1 - assert ( - report["latency_ns"]["pending_list"]["documented_host_budget_ns"] - == 350_000_000 - ) - assert report["latency_ns"]["pending_list"]["documented_host_budget_met"] is True - assert report["checks"]["production_pending_handler_measured"] is True - assert report["checks"]["production_event_callback_bound"] is True - assert report["checks"]["pending_list_started_no_turn_reads"] is True - assert report["checks"]["pending_list_started_no_source_reads"] is True - assert report["checks"]["pending_list_store_rows_unchanged"] is True - assert report["checks"]["cached_requests_started_no_source_reads"] is True - assert report["checks"]["independent_pending_discovered"] is True - assert report["checks"]["independent_pending_cleared"] is True - assert report["checks"]["independent_pending_zero_turn_calls"] is True - assert ( - report["checks"]["independent_pending_discovery_fingerprint_changed"] - is True - ) - assert ( - report["checks"]["independent_pending_unchanged_fingerprint_stable"] - is True - ) - assert report["checks"]["independent_pending_clear_fingerprint_changed"] is True - assert report["checks"]["independent_pending_clear_restored_baseline"] is True - assert report["checks"]["no_duplicate_pending_rows"] is True - assert report["checks"]["independent_pending_health_coherent"] is True - assert ( - report["checks"]["independent_pending_rows_coherent_after_clear"] is True - ) - assert report["ingestion"]["turn_list_calls_during_pending_measurement"] == 0 - assert report["ingestion"]["independent_turn_list_calls"] == 0 - assert report["ingestion"]["independent_prompt_count"] == 2 - assert report["ingestion"]["independent_clear_count"] == 0 - assert ( - report["transport"]["method_dispatches"]["pending.list"] - == 1 + report["ingestion"]["independent_pending_polls"] - ) diff --git a/tests/test_turn_submissions.py b/tests/test_turn_submissions.py index 1aa634f..cd3bef6 100644 --- a/tests/test_turn_submissions.py +++ b/tests/test_turn_submissions.py @@ -22,10 +22,8 @@ from tendwire.store import sqlite as store_sqlite from tendwire.store.sqlite import ( TURN_SUBMISSION_STATE_TRANSITIONS, - cancel_turn_submission, init_store, is_valid_turn_submission_state_transition, - sweep_expired_turn_submissions, turn_delta_payload_from_store, ) @@ -151,42 +149,8 @@ def test_fresh_v20_store_creates_empty_turn_ledgers_and_all_indexes( _assert_empty_v20_ledgers(conn) -def test_v18_to_v20_migration_matches_fresh_schema(tmp_path: Path) -> None: - fresh_path = tmp_path / "fresh.db" - upgrade_path = tmp_path / "upgrade.db" - init_store(fresh_path) - with sqlite3.connect(str(fresh_path)) as fresh: - fresh_schema = _ledger_schema(fresh) - with sqlite3.connect(str(upgrade_path)) as upgrade: - store_sqlite._run_migrations(upgrade, target_version=18) - assert not set(_LEDGER_TABLES) & { - str(row[0]) - for row in upgrade.execute( - "SELECT name FROM sqlite_master WHERE type = 'table'" - ).fetchall() - } - store_sqlite._run_migrations(upgrade, target_version=20) - _assert_empty_v20_ledgers(upgrade, expected_version=20) - assert _ledger_schema(upgrade) == fresh_schema - - -@pytest.mark.parametrize("source_version", range(store_sqlite.STORE_SCHEMA_VERSION)) -def test_every_prior_schema_upgrades_to_identical_empty_v20_ledgers( - tmp_path: Path, - source_version: int, -) -> None: - fresh_path = tmp_path / f"fresh-{source_version}.db" - upgrade_path = tmp_path / f"upgrade-{source_version}.db" - init_store(fresh_path) - with sqlite3.connect(str(fresh_path)) as fresh: - fresh_schema = _ledger_schema(fresh) - with sqlite3.connect(str(upgrade_path)) as upgrade: - store_sqlite._run_migrations(upgrade, target_version=source_version) - store_sqlite._run_migrations(upgrade) - _assert_empty_v20_ledgers(upgrade) - assert _ledger_schema(upgrade) == fresh_schema def _insert_historical_send_receipt( @@ -218,12 +182,11 @@ def _insert_historical_send_receipt( canonical_fingerprint, canonical_request_json, public_worker_id, state, status, result_json, owner_token_hash, owner_expires_at, binding_fingerprint, created_at, reserved_at, send_started_at, - terminal_at, updated_at, legacy_collision, - legacy_collision_count + terminal_at, updated_at ) VALUES ( 'host-a', ?, 'send_instruction', 1, ?, ?, 'worker-a', ?, ?, '{}', ?, ?, NULL, '2026-01-01T00:00:00+00:00', - '2026-01-01T00:00:01+00:00', ?, ?, ?, 0, 0 + '2026-01-01T00:00:01+00:00', ?, ?, ? ) """, ( @@ -245,196 +208,10 @@ def _insert_historical_send_receipt( ) -def test_v18_to_v20_backfills_historical_submission_receipt( - tmp_path: Path, -) -> None: - db_path = tmp_path / "submission-backfill.db" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=18) - _insert_historical_send_receipt( - conn, - request_id="historical-submit", - state="accepted", - status="accepted", - instruction_text=" historical prompt ", - ) - conn.commit() - store_sqlite._run_migrations(conn) - - assert conn.execute( - """ - SELECT owner_key, owner_key_version, instruction_fingerprint, - state, linked_turn_id - FROM turn_submissions - WHERE host_id = 'host-a' AND request_id = 'historical-submit' - """ - ).fetchone() == ( - "legacy-worker:worker-a", - 0, - instruction_fingerprint("historical prompt"), - "submitted", - None, - ) - - -def test_v19_to_v20_repairs_legacy_schema_without_phase2_ledgers( - tmp_path: Path, -) -> None: - db_path = tmp_path / "legacy-v19-without-phase2-ledgers.db" - with sqlite3.connect(str(db_path)) as conn: - store_sqlite._run_migrations(conn, target_version=18) - _insert_historical_send_receipt( - conn, - request_id="legacy-v19-submit", - state="accepted", - status="accepted", - instruction_text="legacy v19 prompt", - ) - # The deployed pre-Phase-2 lineage used version 19 without creating - # the two Phase-2 ledgers. - conn.execute("PRAGMA user_version = 19") - conn.commit() - - assert not store_sqlite._table_columns(conn, "turn_submissions") - assert not store_sqlite._table_columns(conn, "turn_supersessions") - - store_sqlite._run_migrations(conn) - - assert conn.execute("PRAGMA user_version").fetchone() == ( - store_sqlite.STORE_SCHEMA_VERSION, - ) - assert conn.execute( - """ - SELECT owner_key, state - FROM turn_submissions - WHERE host_id = 'host-a' AND request_id = 'legacy-v19-submit' - """ - ).fetchone() == ("legacy-worker:worker-a", "submitted") - assert store_sqlite._table_columns(conn, "turn_supersessions") - - -def test_v19_to_v20_backfills_legacy_tombstone_alias(tmp_path: Path) -> None: - db_path = tmp_path / "supersession-backfill.db" - _seed_link_worker(db_path) - canonical_turn_id = _observe_link_turn( - db_path, - source_turn_id="canonical-source", - ) - legacy_turn_id = "turn-" + ("2" * 24) - observed_at = "2026-02-01T12:00:01+00:00" - legacy_payload = { - "id": legacy_turn_id, - "host_id": "host-a", - "worker_id": "worker-a", - "status": "done", - "kind": "task", - "source": "command", - "origin_command_id": "historical-submit", - "complete": True, - "has_open_turn": False, - "updated_at": observed_at, - "superseded_at": observed_at, - "superseded_by_turn_id": canonical_turn_id, - } - with sqlite3.connect(str(db_path)) as conn: - next_sequence = int( - conn.execute( - "SELECT COALESCE(MAX(list_sequence), 0) + 1 FROM turns" - ).fetchone()[0] - ) - conn.execute( - """ - INSERT INTO turns ( - host_id, turn_id, worker_id, worker_fingerprint, space_id, - status, kind, updated_at, fingerprint, - snapshot_content_fingerprint, observed_at, payload_json, - list_sequence - ) VALUES ( - 'host-a', ?, 'worker-a', NULL, NULL, 'done', 'task', ?, '', - '', ?, ?, ? - ) - """, - ( - legacy_turn_id, - observed_at, - observed_at, - json.dumps(legacy_payload, sort_keys=True, separators=(",", ":")), - next_sequence, - ), - ) - conn.execute("DELETE FROM turn_supersessions") - conn.execute("PRAGMA user_version = 19") - conn.commit() - store_sqlite._run_migrations(conn) - - assert conn.execute( - """ - SELECT canonical_turn_id, reason - FROM turn_supersessions - WHERE host_id = 'host-a' AND superseded_turn_id = ? - """, - (legacy_turn_id,), - ).fetchone() == (canonical_turn_id, "phase1_migration") -@pytest.mark.parametrize( - ("receipt_state", "receipt_status", "expected"), - ( - (None, None, None), - ({"state": "accepted"}, [], None), - ("unknown", "accepted", None), - ("reserved", "pending", None), - ("send_started", "pending", "send_started"), - ("accepted", "accepted", "submitted"), - ("uncertain", "request_state_uncertain", "uncertain"), - ("rejected", "cancelled", "cancelled"), - ("accepted", "purged", None), - ), -) -def test_backfill_submission_state_fails_closed_for_malformed_receipt_values( - receipt_state: object, - receipt_status: object, - expected: str | None, -) -> None: - assert ( - store_sqlite._backfill_submission_state(receipt_state, receipt_status) - == expected - ) -@pytest.mark.parametrize( - "canonical_request_json", - ( - None, - "not-json", - json.dumps([]), - json.dumps({"action": "observe", "instruction": {"text": "hello"}}), - json.dumps({"action": "send_instruction"}), - json.dumps({"action": "send_instruction", "instruction": []}), - json.dumps({"action": "send_instruction", "instruction": {}}), - json.dumps( - {"action": "send_instruction", "instruction": {"text": 42}} - ), - ), -) -def test_receipt_instruction_text_rejects_malformed_receipt_shapes( - canonical_request_json: object, -) -> None: - assert store_sqlite._receipt_instruction_text(canonical_request_json) is None - - -def test_receipt_instruction_text_accepts_valid_send_instruction() -> None: - canonical_request_json = json.dumps( - { - "action": "send_instruction", - "instruction": {"text": "ship the release"}, - } - ) - - assert ( - store_sqlite._receipt_instruction_text(canonical_request_json) - == "ship the release" - ) def test_turn_submission_state_transition_table() -> None: @@ -538,75 +315,8 @@ def _insert_submission( ) -def test_submission_expiry_sweeper_expires_only_old_unlinked_rows( - tmp_path: Path, -) -> None: - db_path = tmp_path / "expiry.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - _insert_submission( - conn, - request_id="old", - state="submitted", - hard_expires_at="2026-01-02T00:00:00+00:00", - ) - _insert_submission( - conn, - request_id="future", - state="uncertain", - hard_expires_at="2026-03-01T00:00:00+00:00", - ) - - assert sweep_expired_turn_submissions( - db_path, - host_id="host-a", - now="2026-02-01T00:00:00+00:00", - ) == 1 - - with sqlite3.connect(str(db_path)) as conn: - rows = conn.execute( - """ - SELECT request_id, state, terminal_at - FROM turn_submissions ORDER BY request_id - """ - ).fetchall() - assert rows == [ - ("future", "uncertain", None), - ("old", "expired", "2026-02-01T00:00:00+00:00"), - ] -def test_submission_cancellation_is_terminal_and_idempotent(tmp_path: Path) -> None: - db_path = tmp_path / "cancel.db" - init_store(db_path) - with sqlite3.connect(str(db_path)) as conn: - _insert_submission( - conn, - request_id="cancel-me", - state="send_started", - hard_expires_at="2026-03-01T00:00:00+00:00", - ) - - assert cancel_turn_submission( - db_path, - host_id="host-a", - request_id="cancel-me", - now="2026-02-01T00:00:00+00:00", - ) - assert not cancel_turn_submission( - db_path, - host_id="host-a", - request_id="cancel-me", - now="2026-02-01T00:00:01+00:00", - ) - - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT state, terminal_at FROM turn_submissions - WHERE host_id = 'host-a' AND request_id = 'cancel-me' - """ - ).fetchone() == ("cancelled", "2026-02-01T00:00:00+00:00") def _seed_link_worker( @@ -679,30 +389,6 @@ def _insert_link_submission( ) -def _set_link_worker_prod_shape( - db_path: Path, - *, - worker_id: str = "worker-a", - host_id: str = "host-a", - explicit_null: bool = False, -) -> None: - version_update = ( - "json_set(payload_json, '$.meta.stable_key_version', NULL)" - if explicit_null - else "json_remove(payload_json, '$.meta.stable_key_version')" - ) - with sqlite3.connect(str(db_path)) as conn: - updated = conn.execute( - f""" - UPDATE workers - SET payload_json = {version_update} - WHERE host_id = ? AND worker_id = ? - """, - (host_id, worker_id), - ) - assert updated.rowcount == 1 - - def _observe_link_turn( db_path: Path, *, @@ -711,7 +397,6 @@ def _observe_link_turn( host_id: str = "host-a", instruction_text: str = "hello", observed_at: str = "2026-02-01T12:00:00+00:00", - turn_model: str = "dual", ) -> str: result = store_sqlite.apply_turn_refresh( db_path, @@ -725,7 +410,6 @@ def _observe_link_turn( "has_open_turn": False, }, observed_at=observed_at, - turn_model=turn_model, ) assert result.updated in {0, 1} with sqlite3.connect(str(db_path)) as conn: @@ -769,7 +453,7 @@ def _submission_rows(db_path: Path) -> list[tuple[str, str, str | None]]: ] -def test_shadow_linker_handles_both_race_directions_without_changing_turn_id( +def test_observed_submission_linker_handles_both_race_directions_without_changing_turn_id( tmp_path: Path, ) -> None: submission_first = tmp_path / "submission-first.db" @@ -815,198 +499,13 @@ def test_shadow_linker_handles_both_race_directions_without_changing_turn_id( ] -@pytest.mark.parametrize( - ("order", "explicit_null"), - (("submission-first", True), ("observation-first", False)), -) -def test_observed_linker_accepts_prod_shape_turn_owner_version( - tmp_path: Path, - order: str, - explicit_null: bool, -) -> None: - db_path = tmp_path / f"prod-shape-{order}.db" - owner_key = _seed_link_worker(db_path) - assert store_sqlite._turn_submission_owner_identity( - { - "id": "worker-a", - "meta": {"stable_key": owner_key, "stable_key_version": 1}, - } - ) == (owner_key, 1) - _set_link_worker_prod_shape(db_path, explicit_null=explicit_null) - - if order == "submission-first": - _insert_link_submission( - db_path, - request_id=order, - owner_key=owner_key, - ) - observed_turn_id = _observe_link_turn( - db_path, - source_turn_id=f"prod-shape-{order}-source", - turn_model="observed", - ) - if order == "observation-first": - _insert_link_submission( - db_path, - request_id=order, - owner_key=owner_key, - ) - - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - - assert _submission_rows(db_path) == [(order, "linked", observed_turn_id)] - with sqlite3.connect(str(db_path)) as conn: - turn_shape = conn.execute( - """ - SELECT json_extract(turns.payload_json, '$.meta.stable_key'), - json_extract(turns.payload_json, '$.meta.stable_key_version'), - json_extract(turns.payload_json, '$.stable_key_version'), - json_extract(turns.payload_json, '$.user_text'), - revisions.user_text - FROM turns - JOIN turn_content_revisions AS revisions - ON revisions.host_id = turns.host_id - AND revisions.turn_id = turns.turn_id - AND revisions.is_current = 1 - WHERE turns.host_id = 'host-a' AND turns.turn_id = ? - """, - (observed_turn_id,), - ).fetchone() - turn_payload = store_sqlite._json_object( - conn.execute( - """ - SELECT payload_json FROM turns - WHERE host_id = 'host-a' AND turn_id = ? - """, - (observed_turn_id,), - ).fetchone()[0] - ) - assert turn_shape == (owner_key, None, None, None, "hello") - turn_worker = {"id": "worker-a", "meta": turn_payload.get("meta")} - assert store_sqlite._turn_submission_owner_identity(turn_worker) == ( - "legacy-worker:worker-a", - 0, - ) - assert store_sqlite._turn_link_candidate_owner_identity(turn_worker) == ( - owner_key, - 1, - ) -@pytest.mark.parametrize( - ("submission_count", "observation_count"), - ((2, 1), (1, 2), (2, 2)), -) -def test_observed_linker_prod_shape_turns_still_fail_closed_on_ambiguity( - tmp_path: Path, - submission_count: int, - observation_count: int, -) -> None: - db_path = tmp_path / ( - f"prod-shape-ambiguous-{submission_count}-{observation_count}.db" - ) - owner_key = _seed_link_worker(db_path) - _set_link_worker_prod_shape(db_path) - for index in range(submission_count): - _insert_link_submission( - db_path, - request_id=f"prod-shape-{index}", - owner_key=owner_key, - ) - for index in range(observation_count): - _observe_link_turn( - db_path, - source_turn_id=f"prod-shape-ambiguous-source-{index}", - turn_model="observed", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - rows = _submission_rows(db_path) - assert [state for _request, state, _turn in rows] == [ - "ambiguous" - ] * submission_count - assert all(linked_turn_id is None for _request, _state, linked_turn_id in rows) -def test_observed_linker_prod_shape_turns_keep_owner_hash_isolated( - tmp_path: Path, -) -> None: - db_path = tmp_path / "prod-shape-owner-isolation.db" - first_owner = _seed_link_worker(db_path) - second_owner = "wsk1_" + ("b" * 64) - snapshot = project_from_raw( - Config(host_id="host-a", db_path=db_path), - workers=[ - { - "id": "worker-a", - "name": "worker-a", - "status": "active", - "meta": { - "stable_key": first_owner, - "stable_key_version": 1, - }, - }, - { - "id": "worker-b", - "name": "worker-b", - "status": "active", - "meta": { - "stable_key": second_owner, - "stable_key_version": 1, - }, - }, - ], - ) - store_sqlite.save_snapshot(db_path, snapshot) - _set_link_worker_prod_shape(db_path, worker_id="worker-a") - _set_link_worker_prod_shape(db_path, worker_id="worker-b", explicit_null=True) - _insert_link_submission( - db_path, - request_id="first-owner", - owner_key=first_owner, - link_expires_at="2026-02-01T12:01:00+00:00", - ) - - _observe_link_turn( - db_path, - worker_id="worker-b", - source_turn_id="wrong-owner-source", - turn_model="observed", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - assert _submission_rows(db_path) == [("first-owner", "submitted", None)] - - matching_turn_id = _observe_link_turn( - db_path, - worker_id="worker-a", - source_turn_id="first-owner-source", - observed_at="2026-02-01T12:00:02+00:00", - turn_model="observed", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:03+00:00", - ) - assert _submission_rows(db_path) == [ - ("first-owner", "linked", matching_turn_id) - ] - - -def test_shadow_linker_failure_keeps_observation_and_rolls_back_link_attempt( +def test_observed_submission_linker_failure_keeps_observation_and_rolls_back_link_attempt( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -1034,7 +533,7 @@ def fail_after_partial_settlement( monkeypatch.setattr( store_sqlite, - "settle_submission_links_conn", + "_settle_submission_links_conn", fail_after_partial_settlement, ) turn_id = _observe_link_turn( @@ -1067,7 +566,7 @@ def fail_after_partial_settlement( ) -def test_shadow_linker_never_uses_unverified_send_started_as_turn_evidence( +def test_observed_submission_linker_never_uses_unverified_send_started_as_turn_evidence( tmp_path: Path, ) -> None: linked_path = tmp_path / "send-started-linked.db" @@ -1105,7 +604,7 @@ def test_shadow_linker_never_uses_unverified_send_started_as_turn_evidence( ] -def test_shadow_linker_waits_for_window_close_before_failing_closed( +def test_observed_submission_linker_waits_for_window_close_before_failing_closed( tmp_path: Path, ) -> None: db_path = tmp_path / "delayed.db" @@ -1132,91 +631,103 @@ def test_shadow_linker_waits_for_window_close_before_failing_closed( assert _submission_rows(db_path) == [("delayed", "ambiguous", None)] -@pytest.mark.parametrize("turn_model", ("dual", "observed")) -def test_single_open_submission_links_on_first_sweep_after_observation( - tmp_path: Path, - turn_model: str, -) -> None: - db_path = tmp_path / f"instant-single-{turn_model}.db" - owner_key = _seed_link_worker(db_path) - _set_link_worker_prod_shape(db_path) - _insert_link_submission( - db_path, - request_id=f"instant-single-{turn_model}", - owner_key=owner_key, - link_expires_at="2026-02-01T12:01:00+00:00", - ) - observed_at = "2026-02-01T12:00:03+00:00" - turn_id = _observe_link_turn( - db_path, - source_turn_id=f"instant-single-{turn_model}-source", - observed_at=observed_at, - turn_model=turn_model, - ) - assert _submission_rows(db_path) == [ - (f"instant-single-{turn_model}", "submitted", None) - ] - swept_at = "2026-02-01T12:00:05+00:00" - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now=swept_at, - ) - assert _submission_rows(db_path) == [ - (f"instant-single-{turn_model}", "linked", turn_id) - ] - with sqlite3.connect(str(db_path)) as conn: - linked_at = conn.execute( - """ - SELECT linked_at FROM turn_submissions - WHERE host_id = 'host-a' AND request_id = ? - """, - (f"instant-single-{turn_model}",), - ).fetchone()[0] - assert linked_at == swept_at - assert ( - datetime.fromisoformat(linked_at) - - datetime.fromisoformat(observed_at) - ).total_seconds() == 2 - assert linked_at < "2026-02-01T12:01:00+00:00" -def test_two_open_same_fingerprint_submissions_do_not_instant_link( + + +@pytest.mark.parametrize( + ( + "scenario", + "initial_state", + "submission_count", + "observe_candidate", + "terminal_state", + ), + ( + ("no-candidate", "submitted", 1, False, "expired"), + ("stale-send-started", "send_started", 1, True, "expired"), + ("two-by-one", "submitted", 2, True, "ambiguous"), + ), +) +def test_link_window_close_settles_current_submission_components( tmp_path: Path, + scenario: str, + initial_state: str, + submission_count: int, + observe_candidate: bool, + terminal_state: str, ) -> None: - db_path = tmp_path / "two-open-no-instant.db" + db_path = tmp_path / f"window-close-{scenario}.db" owner_key = _seed_link_worker(db_path) - for index in range(2): + for index in range(submission_count): _insert_link_submission( db_path, - request_id=f"two-open-{index}", + request_id=f"{scenario}-{index}", owner_key=owner_key, + state=initial_state, link_expires_at="2026-02-01T12:01:00+00:00", + hard_expires_at="2026-02-02T12:00:00+00:00", + ) + if observe_candidate: + observed_at = "2026-02-01T12:00:03+00:00" + if initial_state == "send_started": + observed_at = ( + datetime.fromisoformat("2026-02-01T12:00:00+00:00") + + timedelta( + seconds=store_sqlite.SUBMISSION_SEND_ACK_TIMEOUT_SECONDS + 1 + ) + ).isoformat() + _observe_link_turn( + db_path, + source_turn_id=f"{scenario}-source", + observed_at=observed_at, ) - _observe_link_turn( + before_close = turn_delta_payload_from_store( db_path, - source_turn_id="two-open-source", - observed_at="2026-02-01T12:00:03+00:00", + "host-a", + now=datetime.fromisoformat( + "2026-02-01T12:00:30+00:00" + ).timestamp(), ) - store_sqlite.sweep_submission_links( + assert before_close["host_id"] == "host-a" + assert _submission_rows(db_path) == [ + (f"{scenario}-{index}", initial_state, None) + for index in range(submission_count) + ] + + at_close = turn_delta_payload_from_store( db_path, - host_id="host-a", - now="2026-02-01T12:00:05+00:00", + "host-a", + now=datetime.fromisoformat( + "2026-02-01T12:01:00+00:00" + ).timestamp(), ) - + assert at_close["host_id"] == "host-a" assert _submission_rows(db_path) == [ - ("two-open-0", "submitted", None), - ("two-open-1", "submitted", None), + (f"{scenario}-{index}", terminal_state, None) + for index in range(submission_count) ] + with sqlite3.connect(str(db_path)) as conn: + stamps = conn.execute( + """ + SELECT terminal_at, hard_expires_at + FROM turn_submissions + WHERE host_id = 'host-a' + ORDER BY request_id + """ + ).fetchall() + assert stamps == [ + ("2026-02-01T12:01:00+00:00", "2026-02-02T12:00:00+00:00") + ] * submission_count -def test_disconnected_singleton_component_still_links_immediately( +def test_disconnected_submission_components_settle_independently( tmp_path: Path, ) -> None: - db_path = tmp_path / "disconnected-singleton-instant.db" + db_path = tmp_path / "disconnected-components.db" owner_key = _seed_link_worker(db_path) _insert_link_submission( db_path, @@ -1237,16 +748,31 @@ def test_disconnected_singleton_component_still_links_immediately( observed_at="2026-02-01T12:00:03+00:00", ) - store_sqlite.sweep_submission_links( + payload = turn_delta_payload_from_store( db_path, - host_id="host-a", - now="2026-02-01T12:00:05+00:00", + "host-a", + now=datetime.fromisoformat( + "2026-02-01T12:00:05+00:00" + ).timestamp(), ) + assert payload["host_id"] == "host-a" assert _submission_rows(db_path) == [ ("live-singleton", "linked", turn_id), ("old-disconnected", "expired", None), ] + with sqlite3.connect(str(db_path)) as conn: + assert conn.execute( + """ + SELECT request_id, terminal_at, linked_at + FROM turn_submissions + WHERE host_id = 'host-a' + ORDER BY request_id + """ + ).fetchall() == [ + ("live-singleton", None, "2026-02-01T12:00:03+00:00"), + ("old-disconnected", "2026-02-01T12:00:03+00:00", None), + ] def test_manual_same_text_turn_links_single_open_submission( @@ -1274,169 +800,12 @@ def test_manual_same_text_turn_links_single_open_submission( ] -def test_stale_send_started_submission_uses_windowed_settlement( - tmp_path: Path, -) -> None: - db_path = tmp_path / "stale-send-started.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="stale-send-started", - owner_key=owner_key, - state="send_started", - link_expires_at="2026-02-01T12:01:00+00:00", - ) - observed_at = ( - datetime.fromisoformat("2026-02-01T12:00:00+00:00") - + timedelta( - seconds=store_sqlite.SUBMISSION_SEND_ACK_TIMEOUT_SECONDS + 1 - ) - ).isoformat() - _observe_link_turn( - db_path, - source_turn_id="stale-send-started-source", - observed_at=observed_at, - ) - assert _submission_rows(db_path) == [ - ("stale-send-started", "send_started", None) - ] - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:30+00:00", - ) - assert _submission_rows(db_path) == [ - ("stale-send-started", "send_started", None) - ] - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:01:00+00:00", - ) - assert _submission_rows(db_path) == [ - ("stale-send-started", "expired", None) - ] -def test_ambiguous_component_is_stamped_at_window_close_not_hard_ttl( - tmp_path: Path, -) -> None: - db_path = tmp_path / "ambiguous-at-window-close.db" - owner_key = _seed_link_worker(db_path) - for index in range(2): - _insert_link_submission( - db_path, - request_id=f"ambiguous-at-close-{index}", - owner_key=owner_key, - link_expires_at="2026-02-01T12:01:00+00:00", - hard_expires_at="2026-02-02T12:00:00+00:00", - ) - _observe_link_turn( - db_path, - source_turn_id="ambiguous-at-close-source", - observed_at="2026-02-01T12:00:03+00:00", - ) - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:01:00+00:00", - ) - assert _submission_rows(db_path) == [ - ("ambiguous-at-close-0", "ambiguous", None), - ("ambiguous-at-close-1", "ambiguous", None), - ] - with sqlite3.connect(str(db_path)) as conn: - stamps = conn.execute( - """ - SELECT terminal_at, hard_expires_at - FROM turn_submissions - WHERE host_id = 'host-a' - ORDER BY request_id - """ - ).fetchall() - assert stamps == [ - ("2026-02-01T12:01:00+00:00", "2026-02-02T12:00:00+00:00"), - ("2026-02-01T12:01:00+00:00", "2026-02-02T12:00:00+00:00"), - ] - - -def test_lone_submission_without_candidate_expires_at_window_close( - tmp_path: Path, -) -> None: - db_path = tmp_path / "no-candidate-at-window-close.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="no-candidate-at-window-close", - owner_key=owner_key, - link_expires_at="2026-02-01T12:01:00+00:00", - hard_expires_at="2026-02-02T12:00:00+00:00", - ) - - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:30+00:00", - ) - assert _submission_rows(db_path) == [ - ("no-candidate-at-window-close", "submitted", None) - ] - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:01:00+00:00", - ) - - assert _submission_rows(db_path) == [ - ("no-candidate-at-window-close", "expired", None) - ] - with sqlite3.connect(str(db_path)) as conn: - assert conn.execute( - """ - SELECT terminal_at, hard_expires_at - FROM turn_submissions - WHERE host_id = 'host-a' - AND request_id = 'no-candidate-at-window-close' - """ - ).fetchone() == ( - "2026-02-01T12:01:00+00:00", - "2026-02-02T12:00:00+00:00", - ) - - -def test_each_no_candidate_component_expires_at_its_window_close( - tmp_path: Path, -) -> None: - db_path = tmp_path / "multiple-no-candidate-components.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="closed-no-candidate", - owner_key=owner_key, - link_expires_at="2026-02-01T12:00:10+00:00", - ) - _insert_link_submission( - db_path, - request_id="open-no-candidate", - owner_key=owner_key, - link_not_before="2026-02-01T12:01:00+00:00", - link_expires_at="2026-02-01T12:02:00+00:00", - ) - - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:01:30+00:00", - ) - - assert _submission_rows(db_path) == [ - ("closed-no-candidate", "expired", None), - ("open-no-candidate", "submitted", None), - ] def test_turn_observed_outside_link_window_never_links( @@ -1474,7 +843,7 @@ def test_turn_observed_outside_link_window_never_links( ("submission_count", "observation_count"), ((2, 1), (1, 2), (2, 2)), ) -def test_shadow_linker_marks_larger_identical_components_ambiguous( +def test_observed_submission_linker_marks_larger_identical_components_ambiguous( tmp_path: Path, submission_count: int, observation_count: int, @@ -1504,7 +873,7 @@ def test_shadow_linker_marks_larger_identical_components_ambiguous( assert all(linked_turn_id is None for _request, _state, linked_turn_id in rows) -def test_submission_linker_isolates_owners_and_legacy_alias_links( +def test_submission_linker_isolates_stable_owners( tmp_path: Path, ) -> None: first_path = tmp_path / "owners.db" @@ -1544,17 +913,6 @@ def test_submission_linker_isolates_owners_and_legacy_alias_links( ("owner-b", "linked", second_turn_id), ] - legacy_path = tmp_path / "legacy.db" - owner_key = _seed_link_worker(legacy_path) - _insert_link_submission(legacy_path, request_id="legacy", owner_key=owner_key) - legacy_turn_id = _observe_link_turn( - legacy_path, - source_turn_id="legacy-source", - turn_model="legacy", - ) - assert _submission_rows(legacy_path) == [("legacy", "linked", legacy_turn_id)] - - def test_goal13_delta_is_unperturbed_when_observed_turn_links_later( tmp_path: Path, ) -> None: @@ -1647,7 +1005,6 @@ def test_idle_observation_first_submission_links_from_lazy_turn_read( now=datetime.fromisoformat( "2026-02-01T12:00:01+00:00" ).timestamp(), - turn_model="observed", ) assert payload["turns"] @@ -1656,120 +1013,12 @@ def test_idle_observation_first_submission_links_from_lazy_turn_read( ] -def test_observed_prod_shape_sweep_never_runs_public_sanitizers( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "prod-shape-no-public-sanitize.db" - owner_key = _seed_link_worker(db_path) - _set_link_worker_prod_shape(db_path) - observed_turn_id = _observe_link_turn( - db_path, - source_turn_id="prod-shape-no-public-sanitize-source", - turn_model="observed", - ) - _insert_link_submission( - db_path, - request_id="prod-shape-no-public-sanitize", - owner_key=owner_key, - ) - contains_calls = 0 - sanitize_calls = 0 - original_contains = core_turns._contains_forbidden_public_text - original_sanitize = core_turns.sanitize_public_text - - def record_contains(value: str) -> bool: - nonlocal contains_calls - contains_calls += 1 - return original_contains(value) - - def record_sanitize(value: object, **kwargs: object) -> str: - nonlocal sanitize_calls - sanitize_calls += 1 - return original_sanitize(value, **kwargs) - - monkeypatch.setattr( - core_turns, - "_contains_forbidden_public_text", - record_contains, - ) - monkeypatch.setattr(core_turns, "sanitize_public_text", record_sanitize) - - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - - assert contains_calls == 0 - assert sanitize_calls == 0 - assert _submission_rows(db_path) == [ - ("prod-shape-no-public-sanitize", "linked", observed_turn_id) - ] - - -def test_submission_link_sweep_backs_off_until_matching_observation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - db_path = tmp_path / "submission-link-backoff.db" - owner_key = _seed_link_worker(db_path) - _insert_link_submission( - db_path, - request_id="submission-link-backoff", - owner_key=owner_key, - ) - candidate_calls = 0 - original_candidates = store_sqlite._submission_link_candidate_turns_conn - - def record_candidates(*args: object, **kwargs: object): - nonlocal candidate_calls - candidate_calls += 1 - return original_candidates(*args, **kwargs) - - monkeypatch.setattr( - store_sqlite, - "_submission_link_candidate_turns_conn", - record_candidates, - ) - - for _ in range(2): - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T11:59:30+00:00", - ) - assert candidate_calls == 1 - - # Production observations can persist the authenticated stable owner key - # without its version marker. That shape cannot use the observation-time - # direct settlement path, so this specifically proves that the observation - # re-arms the component for the next sweep. - _set_link_worker_prod_shape(db_path) - observed_turn_id = _observe_link_turn( - db_path, - source_turn_id="submission-link-backoff-source", - observed_at="2026-02-01T12:00:00+00:00", - turn_model="observed", - ) - assert candidate_calls == 1 - store_sqlite.sweep_submission_links( - db_path, - host_id="host-a", - now="2026-02-01T12:00:01+00:00", - ) - assert candidate_calls == 2 - assert _submission_rows(db_path) == [ - ("submission-link-backoff", "linked", observed_turn_id) - ] -@pytest.mark.parametrize("turn_model", sorted(store_sqlite.TURN_MODELS)) -def test_turn_alias_resolves_public_content_and_final_root_under_every_model( +def test_turn_alias_resolves_public_content_and_final_root( tmp_path: Path, - turn_model: str, ) -> None: db_path = tmp_path / "alias-lookup.db" _seed_link_worker(db_path) @@ -1808,7 +1057,6 @@ def test_turn_alias_resolves_public_content_and_final_root_under_every_model( turn_id=legacy_turn_id, content_revision=revision, field="assistant_final_text", - turn_model=turn_model, ) assert page["turn_id"] == canonical_turn_id assert page["text"] == "answer for alias-source" @@ -1828,7 +1076,6 @@ def test_turn_alias_resolves_public_content_and_final_root_under_every_model( presentation_version="alias-aware-v1", part_count=1, source_ref=leased["ref"], - turn_model=turn_model, now="2026-02-01T12:00:03+00:00", ) assert begun["ok"] is True @@ -1842,130 +1089,124 @@ def test_turn_alias_resolves_public_content_and_final_root_under_every_model( ).fetchone() == (canonical_turn_id,) -def test_link_candidate_owner_identity_normalizes_only_missing_version() -> None: - # Guard-regression for the prod-shape normalization: ONLY a syntactically - # valid stable key whose version is exactly None is treated as v1; every - # other shape falls through to the strict submission-side identity. - valid_key = "wsk1_" + ("a" * 64) - normalized = store_sqlite._turn_link_candidate_owner_identity( - {"id": "worker-x", "meta": {"stable_key": valid_key, "stable_key_version": None}} - ) - assert normalized == (valid_key, 1) - - fallthrough_cases = [ - {"stable_key": valid_key, "stable_key_version": 2}, - {"stable_key": valid_key, "stable_key_version": 0}, - {"stable_key": valid_key, "stable_key_version": "1"}, - {"stable_key": valid_key, "stable_key_version": True}, - {"stable_key": "wsk1_short", "stable_key_version": None}, - {"stable_key": "not-a-key", "stable_key_version": None}, - {"stable_key": "", "stable_key_version": None}, - {"stable_key": None, "stable_key_version": None}, - ] - for meta in fallthrough_cases: - result = store_sqlite._turn_link_candidate_owner_identity( - {"id": "worker-x", "meta": meta} - ) - assert result == ("legacy-worker:worker-x", 0), meta -def test_observed_lazy_delta_sweep_links_first_poll_after_observation( +def test_stable_key_delta_lazy_sweep_rearms_component_backoff( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - """Mirror the idle-pane production canary timeline through turn.delta.""" - db_path = tmp_path / "observed-live-timeline.db" + db_path = tmp_path / "stable-key-delta-rearm.db" owner_key = _seed_link_worker(db_path) _insert_link_submission( db_path, - request_id="observed-live-timeline", + request_id="stable-key-delta-rearm", owner_key=owner_key, link_not_before="2026-07-22T12:01:23+00:00", link_expires_at="2026-07-22T12:03:23+00:00", hard_expires_at="2026-07-23T12:02:23+00:00", ) - _set_link_worker_prod_shape(db_path) - - # Herdres polls turn.delta every ~5s. The early empty sweeps exercise the - # component backoff before the idle pane produces its first observation. - for second in range(23, 36, 5): - turn_delta_payload_from_store( - db_path, - "host-a", - now=datetime.fromisoformat( - f"2026-07-22T12:01:{second:02d}+00:00" - ).timestamp(), - turn_model="observed", - ) - for second in range(38, 60, 5): - turn_delta_payload_from_store( - db_path, - "host-a", - now=datetime.fromisoformat( - f"2026-07-22T12:01:{second:02d}+00:00" - ).timestamp(), - turn_model="observed", - ) - for second in range(3, 34, 5): - turn_delta_payload_from_store( - db_path, - "host-a", - now=datetime.fromisoformat( - f"2026-07-22T12:02:{second:02d}+00:00" - ).timestamp(), - turn_model="observed", - ) - candidate_calls = 0 + rearmed_keys: list[tuple[str, str]] = [] original_candidates = store_sqlite._submission_link_candidate_turns_conn + original_rearm = store_sqlite._rearm_submission_link_component + original_settle = store_sqlite._settle_submission_links_conn + fail_direct_settlement = False def record_candidates(*args: object, **kwargs: object): nonlocal candidate_calls candidate_calls += 1 return original_candidates(*args, **kwargs) + def record_rearm( + db: Path | str, + host: str, + owner: str, + fingerprint: str, + ) -> None: + rearmed_keys.append((owner, fingerprint)) + original_rearm(db, host, owner, fingerprint) + + def fail_one_direct_settlement(*args: object, **kwargs: object) -> int: + if fail_direct_settlement: + raise RuntimeError("controlled direct settlement failure") + return original_settle(*args, **kwargs) + monkeypatch.setattr( store_sqlite, "_submission_link_candidate_turns_conn", record_candidates, ) - # The live observed prompt retained Herdr's trailing U+0001 framing byte. - # Submission input rejects that byte, so matching must ignore it only at - # the observation edge and re-arm the original component key. + monkeypatch.setattr( + store_sqlite, + "_rearm_submission_link_component", + record_rearm, + ) + monkeypatch.setattr( + store_sqlite, + "_settle_submission_links_conn", + fail_one_direct_settlement, + ) + + for observed_at in ( + "2026-07-22T12:02:33+00:00", + "2026-07-22T12:02:38+00:00", + ): + payload = turn_delta_payload_from_store( + db_path, + "host-a", + now=datetime.fromisoformat(observed_at).timestamp(), + ) + assert payload["host_id"] == "host-a" + assert candidate_calls == 1 + assert _submission_rows(db_path) == [ + ("stable-key-delta-rearm", "submitted", None) + ] + + fail_direct_settlement = True observed_turn_id = _observe_link_turn( db_path, - source_turn_id="turn-803b8be4224ccec08a20c794", + source_turn_id="stable-key-delta-rearm-source", instruction_text="hello\x01", - observed_at="2026-07-22T12:02:36+00:00", - turn_model="observed", + observed_at="2026-07-22T12:02:40+00:00", ) + fail_direct_settlement = False + assert (owner_key, instruction_fingerprint("hello")) in rearmed_keys + assert all(owner == owner_key for owner, _fingerprint in rearmed_keys) + assert candidate_calls == 1 assert _submission_rows(db_path) == [ - ("observed-live-timeline", "submitted", None) + ("stable-key-delta-rearm", "submitted", None) ] - candidate_calls_after_observation = candidate_calls - turn_delta_payload_from_store( + linked_page = turn_delta_payload_from_store( db_path, "host-a", now=datetime.fromisoformat( - "2026-07-22T12:02:38+00:00" + "2026-07-22T12:02:42+00:00" ).timestamp(), - turn_model="observed", ) - assert candidate_calls == candidate_calls_after_observation + 1 + assert linked_page["host_id"] == "host-a" + assert candidate_calls == 2 assert _submission_rows(db_path) == [ - ("observed-live-timeline", "linked", observed_turn_id) + ("stable-key-delta-rearm", "linked", observed_turn_id) ] + linked_turn = next( + change["turn"] + for change in linked_page["changes"] + if change.get("op") == "upsert" + and change.get("turn_id") == observed_turn_id + ) + assert linked_turn["submission_id"] == turn_submission_id( + "host-a", "stable-key-delta-rearm" + ) + assert linked_turn["submission_state"] == "linked" with sqlite3.connect(str(db_path)) as conn: - linked_at = conn.execute( + assert conn.execute( """ SELECT linked_at FROM turn_submissions WHERE host_id = 'host-a' - AND request_id = 'observed-live-timeline' + AND request_id = 'stable-key-delta-rearm' """ - ).fetchone()[0] - assert linked_at == "2026-07-22T12:02:38+00:00" - assert linked_at < "2026-07-22T12:03:23+00:00" + ).fetchone() == ("2026-07-22T12:02:42+00:00",) def test_observed_link_rearm_uses_stable_owner_across_worker_renumber( @@ -1988,7 +1229,6 @@ def test_observed_link_rearm_uses_stable_owner_across_worker_renumber( now=datetime.fromisoformat( "2026-07-22T12:02:33+00:00" ).timestamp(), - turn_model="observed", ) renumbered = project_from_raw( @@ -2009,7 +1249,6 @@ def test_observed_link_rearm_uses_stable_owner_across_worker_renumber( assert store_sqlite.save_snapshot( db_path, renumbered, - turn_model="observed", ) rearmed_keys: list[tuple[str, str]] = [] @@ -2035,10 +1274,9 @@ def record_rearm( source_turn_id="renumbered-source", instruction_text="hello\x01", observed_at="2026-07-22T12:02:36+00:00", - turn_model="observed", ) assert (owner_key, instruction_fingerprint("hello")) in rearmed_keys - assert all(not owner.startswith("legacy-worker:") for owner, _ in rearmed_keys) + assert all(owner == owner_key for owner, _ in rearmed_keys) turn_delta_payload_from_store( db_path, @@ -2046,7 +1284,6 @@ def record_rearm( now=datetime.fromisoformat( "2026-07-22T12:03:23+00:00" ).timestamp(), - turn_model="observed", ) assert _submission_rows(db_path) == [ ("observed-renumber", "linked", observed_turn_id) @@ -2071,7 +1308,6 @@ def test_observed_busy_pane_completion_links_immediately( "has_open_turn": True, }, observed_at="2026-07-22T12:01:30+00:00", - turn_model="observed", ) assert started.updated == 1 _insert_link_submission( @@ -2086,7 +1322,6 @@ def test_observed_busy_pane_completion_links_immediately( db_path, source_turn_id="busy-pane-source", observed_at="2026-07-22T12:02:36+00:00", - turn_model="observed", ) assert _submission_rows(db_path) == [ ("observed-busy-pane", "linked", observed_turn_id) diff --git a/tests/test_worker_label_and_model.py b/tests/test_worker_label_and_model.py index 1a8cc76..f513f4c 100644 --- a/tests/test_worker_label_and_model.py +++ b/tests/test_worker_label_and_model.py @@ -1,196 +1,34 @@ -"""Pane label in public worker meta + model on turns (consumed by herdres for topic names and the -pinned status board).""" from __future__ import annotations -from pathlib import Path - -from tendwire.backends.herdr_cli import _worker_from_item, _workers_and_bindings_from_records -from tendwire.backends.herdr_events import HerdrEventBackend -from tendwire.backends.herdr_turns import _TURN_CONTENT_KEYS +from tendwire.backends.acp_coordinator import _discovered_workers from tendwire.config import Config from tendwire.core.turns import Turn -from tendwire.core.projector import project_from_raw -from tendwire.store.sqlite import init_store, merge_turn_content, save_snapshot, turns_payload_from_store - - -def _config(tmp_path: Path) -> Config: - return Config( - host_id="label-host", - data_dir=tmp_path, - db_path=tmp_path / "label-host.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, - ) - - -def _pane_item(label: str = "review-pane") -> dict: - return { - "pane_id": "ws-1:p2Q", - "terminal_id": "term-1", - "workspace_id": "ws-1", - "agent": "claude", - "agent_session": {"kind": "id", "value": "sess-1"}, - "label": label, - "cwd": "/root/temp", - "agent_status": "idle", - } - - -def _agent_item() -> dict: - return { - "agent_id": "agent-private", - "name": "claude", - "agent": "claude", - "workspace_id": "ws-1", - "status": "waiting", - "pane_id": "ws-1:p2Q", - "terminal_id": "term-1", - "agent_session": {"kind": "id", "value": "sess-1"}, - } -def test_pane_label_lands_in_public_worker_meta() -> None: - worker = _worker_from_item(_pane_item()) - assert worker is not None - assert worker.meta.get("label") == "review-pane" - # name resolution unchanged: agent-first - assert worker.name == "claude" - - -def test_reconcile_merges_pane_label_into_agent_record_without_replacing_it(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - records = backend._records_from_reconcile_payloads( - {"agents": [_agent_item()]}, - {"panes": [_pane_item()]}, +def test_pane_label_is_public_but_cwd_and_target_are_private(tmp_path) -> None: + workers, bindings = _discovered_workers( + Config(host_id="h", data_dir=tmp_path, db_path=tmp_path / "db"), + {"panes": [{ + "workspace_id": "wR9", + "pane_id": "wR9:pA", + "terminal_id": "term-private", + "agent": "claude", + "label": "Review pane", + "cwd": "/private/path", + }]}, + {"agents": []}, + "2026-01-01T00:00:00+00:00", ) - workers, bindings = _workers_and_bindings_from_records(config, records) - - assert len(records) == 1 - assert len(workers) == 1 - assert len(bindings) == 1 - assert records[0].worker.meta.get("label") == "review-pane" - assert "cwd" not in records[0].worker.meta - assert records[0].worker.status == "waiting" - assert bindings[0].target_kind == "agent_id" - assert bindings[0].target_value == "agent-private" - - -def test_reconcile_drops_agent_and_pane_cwd_from_public_worker(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - agent = {**_agent_item(), "cwd": "/root/agent-cwd"} - pane = {**_pane_item(), "cwd": "/root/pane-cwd"} - records = backend._records_from_reconcile_payloads({"agents": [agent]}, {"panes": [pane]}) - - assert len(records) == 1 - assert records[0].worker.meta.get("label") == "review-pane" - assert "cwd" not in records[0].worker.meta - assert "/root/agent-cwd" not in str(records[0].worker.to_dict()) - assert "/root/pane-cwd" not in str(records[0].worker.to_dict()) - - -def test_reconcile_keeps_agent_turn_target_when_present(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - agent = {**_agent_item(), "agent": "codex", "name": "codex"} - records = backend._records_from_reconcile_payloads({"agents": [agent]}, {"panes": [_pane_item()]}) + assert workers[0].meta["label"] == "Review pane" + assert "private" not in str(workers[0].to_dict()) + assert bindings[0].target_value == "term-private" - assert len(records) == 1 - assert records[0].turn_target_kind == "codex_session_id" - assert records[0].turn_target_value == "sess-1" - assert records[0].worker.meta.get("label") == "review-pane" - -def test_reconcile_uses_matched_pane_turn_target_when_agent_lacks_one(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - agent = _agent_item() - agent.pop("pane_id") - records = backend._records_from_reconcile_payloads({"agents": [agent]}, {"panes": [_pane_item()]}) - workers, bindings = _workers_and_bindings_from_records(config, records) - - assert len(records) == 1 - assert records[0].turn_target_kind == "pane_id" - assert records[0].turn_target_value == "ws-1:p2Q" - assert len(bindings) == 1 - assert bindings[0].turn_target_kind == "pane_id" - assert bindings[0].turn_target_value == "ws-1:p2Q" - assert workers[0].meta.get("label") == "review-pane" - - -def test_reconcile_only_fills_missing_agent_backend_target_from_pane(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - agent = { - "workspace_id": "ws-1", - "status": "waiting", - "agent_session": {"kind": "id", "value": "sess-1"}, - } - records = backend._records_from_reconcile_payloads({"agents": [agent]}, {"panes": [_pane_item()]}) - workers, bindings = _workers_and_bindings_from_records(config, records) - - assert len(records) == 1 - assert workers[0].backend_target is not None - assert workers[0].backend_target["kind"] == "terminal_id" - assert workers[0].backend_target["value"] == "term-1" - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == "term-1" - - -def test_reconcile_preserves_pane_only_worker_when_no_agent_record(tmp_path: Path) -> None: - config = _config(tmp_path) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - records = backend._records_from_reconcile_payloads({"agents": []}, {"panes": [_pane_item()]}) - workers, bindings = _workers_and_bindings_from_records(config, records) - - assert len(workers) == 1 - assert workers[0].meta.get("label") == "review-pane" - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == "term-1" - - -def test_turn_model_round_trip_and_id_stability() -> None: - base = {"host_id": "h", "worker_id": "w1", "kind": "turn", "source": "herdr", - "user_text": "hi", "assistant_final_text": "done", "complete": True} +def test_llm_model_round_trip_and_turn_id_stability() -> None: + base = {"host_id": "h", "worker_id": "w", "kind": "turn", "source": "acp", "complete": True} plain = Turn.from_dict(base) - with_model = Turn.from_dict({**base, "model": "claude-fable-5[1m]"}) - assert with_model.model == "claude-fable-5[1m]" - assert with_model.to_dict()["model"] == "claude-fable-5[1m]" - assert plain.id == with_model.id # model is content, NOT identity (no id re-mint) - assert plain.fingerprint != with_model.fingerprint # but the content fingerprint reflects it - - -def test_turn_content_keys_include_model() -> None: - assert "model" in _TURN_CONTENT_KEYS - - -def test_merge_turn_content_persists_model(tmp_path: Path) -> None: - db = tmp_path / "turns.db" - config = Config(host_id="turn-host", db_path=db) - snapshot = project_from_raw( - config, - workers=[{"id": "worker-1", "name": "claude", "status": "active", "space_id": "space-1"}], - ) - init_store(db) - save_snapshot(db, snapshot) - updated = merge_turn_content( - db, "turn-host", "worker-1", - { - "source_turn_id": "model-source", - "user_text": "hi", - "assistant_final_text": "done", - "complete": True, - "model": "claude-fable-5", - }, - observed_at="2026-01-01T00:00:00+00:00", - ) - payload = turns_payload_from_store(db, "turn-host", snapshot=snapshot) - assert updated == 1 - assert payload["turns"][0].get("model") == "claude-fable-5" + modeled = Turn.from_dict({**base, "model": "claude-fable-5"}) + assert modeled.model == "claude-fable-5" + assert modeled.to_dict()["model"] == "claude-fable-5" + assert plain.id == modeled.id + assert plain.fingerprint != modeled.fingerprint diff --git a/tests/test_worker_stable_key.py b/tests/test_worker_stable_key.py index 3819fd4..743b81d 100644 --- a/tests/test_worker_stable_key.py +++ b/tests/test_worker_stable_key.py @@ -1,2352 +1,257 @@ -"""Stable worker continuity from Herdr's persisted workspace/public-pane identity.""" - from __future__ import annotations -from concurrent.futures import ThreadPoolExecutor -import hashlib -import hmac import json -import os import re -import stat -import subprocess -from copy import deepcopy -from pathlib import Path -from typing import Any +import threading +from dataclasses import replace import pytest -from tendwire import worker_identity -from tendwire.backends import herdr_cli -from tendwire.backends.herdr_cli import ( - _private_identity_material_from_item, - _worker_record_from_item, - _workers_and_bindings_from_records, +from tendwire.backends.acp_coordinator import ( + AcpSupervisor, + _discovered_spaces, + _discovered_workers, ) -from tendwire.backends.herdr_events import HerdrEventBackend from tendwire.config import Config -from tendwire.core.models import Worker, worker_binding_private_fingerprint -from tendwire.store.sqlite import init_store, latest_snapshot, list_worker_bindings -from tendwire.worker_identity import ( - InstallationKeyError, - load_or_create_installation_key, - reset_installation_key, +from tendwire.core.models import Snapshot, WorkerBinding +from tendwire.store.sqlite import ( + init_store, + latest_snapshot, + list_worker_bindings, + save_snapshot, + upsert_worker_bindings, ) -_STABLE_KEY = re.compile(r"^wsk1_[0-9a-f]{64}$") -_FIXTURE_PATH = Path(__file__).parent / "fixtures" / "herdr" / "worker_identity_restore.json" - -@pytest.fixture(autouse=True) -def _isolated_home(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - home = tmp_path / "unused-home" - home.mkdir() - monkeypatch.setenv("HOME", str(home)) +OBSERVED_AT = "2026-01-01T00:00:00+00:00" -def _fixture() -> dict[str, Any]: - return json.loads(_FIXTURE_PATH.read_text(encoding="utf-8")) +def _config(tmp_path) -> Config: + return Config(host_id="host", data_dir=tmp_path, db_path=tmp_path / "db.sqlite") -def _config(data_dir: Path, *, host_id: str = "stable-host") -> Config: - return Config( - host_id=host_id, - data_dir=data_dir, - db_path=data_dir / "stable-host.db", - herdr_backend="socket", - herdr_timeout_seconds=0.5, +def _discover(tmp_path, panes, agents=(), *, prior_bindings=()): + return _discovered_workers( + _config(tmp_path), + {"panes": panes}, + {"agents": list(agents)}, + OBSERVED_AT, + prior_bindings=prior_bindings, ) -def _project( - config: Config, - agents: list[dict[str, Any]], - panes: list[dict[str, Any]] | None = None, -) -> tuple[HerdrEventBackend, list[Any], list[Any], list[Any]]: - config.data_dir.mkdir(parents=True, mode=0o700, exist_ok=True) - init_store(Path(config.db_path)) - backend = HerdrEventBackend(config, debounce_seconds=0) - records = backend._records_from_reconcile_payloads( - {"agents": deepcopy(agents)}, - {"panes": deepcopy(panes or [])}, - ) - workers, bindings = _workers_and_bindings_from_records(config, records) - return backend, workers, bindings, records - - -def _single_worker(config: Config, item: dict[str, Any]) -> Any: - _backend, workers, _bindings, _records = _project(config, [], [item]) - assert len(workers) == 1 - return workers[0] - - -def _stable(worker: Any) -> str: - value = worker.meta.get("stable_key") - assert isinstance(value, str) - return value - - -def _mode(path: Path) -> int: - return stat.S_IMODE(os.lstat(path).st_mode) - - -def _tree_snapshot(root: Path) -> dict[str, tuple[int, int, bytes | None]]: - snapshot: dict[str, tuple[int, int, bytes | None]] = {} - for entry in [root, *sorted(root.rglob("*"))]: - current = os.lstat(entry) - content = entry.read_bytes() if stat.S_ISREG(current.st_mode) else None - snapshot[str(entry.relative_to(root))] = ( - current.st_ino, - stat.S_IMODE(current.st_mode), - content, - ) - return snapshot - - -def _reserved_meta_keys(value: Any, *, include_root: bool = True) -> list[str]: - found: list[str] = [] - if isinstance(value, dict): - for key, child in value.items(): - compact = str(key).lower().replace("_", "").replace("-", "").replace(".", "") - if include_root and compact.startswith("stablekey"): - found.append(str(key)) - found.extend(_reserved_meta_keys(child)) - elif isinstance(value, list): - for child in value: - found.extend(_reserved_meta_keys(child)) - return found - - -def test_restore_fixture_matches_verified_herdr_contract() -> None: - """Fixture follows authoritative Herdr commit 46174563489273199a17c982356c6e4674ef00d4.""" - fixture = _fixture() - session = fixture["session_snapshot"] - workspace = session["workspaces"][0] - tab = workspace["tabs"][0] - before = fixture["pre_restore"] - after = fixture["post_restore"] - - assert session["version"] == 3 - assert workspace["id"] == "wR9" - assert workspace["public_pane_numbers"] == {"41": 10, "42": 11} - assert tab["layout"]["Split"]["first"] == {"Pane": 41} - assert tab["layout"]["Split"]["second"] == {"Pane": 42} - assert set(tab["panes"]) == {"41", "42"} - assert before["pane_info"]["pane_id"] == after["pane_info"]["pane_id"] == "wR9:pA" - assert before["sibling_pane_info"]["pane_id"] == after["sibling_pane_info"]["pane_id"] == "wR9:pB" - for field in ("raw_pane_id", "runtime_id", "worker_id", "agent_id"): - assert before[field] != after[field] - for field in ("terminal_id", "agent"): - assert before["pane_info"][field] != after["pane_info"][field] - assert before["pane_info"]["agent_session"] != after["pane_info"]["agent_session"] - split = fixture["split_creation"] - assert split["event"] == split["data"]["type"] == "pane_created" - assert split["data"]["pane"]["pane_id"] == before["sibling_pane_info"]["pane_id"] - assert split["data"]["pane"]["workspace_id"] == workspace["id"] - for move in (fixture["same_workspace_move"], fixture["cross_workspace_move"]): - assert move["event"] == "pane_moved" - assert move["data"]["type"] == "pane_moved" - assert "pane" in move["data"] - - -def test_turn_observation_fields_are_byte_identical_identity_exclusions( - tmp_path: Path, -) -> None: - config = _config(tmp_path / "identity-turn-exclusion") - pane = deepcopy(_fixture()["pre_restore"]["pane_info"]) - pane["meta"] = {"provider": {"label": "stable"}} - _backend, baseline_workers, baseline_bindings, _records = _project( - config, - [], - [pane], - ) - observed = deepcopy(pane) - observed.update( - { - "turn": 41, - "turn_epoch": 99, - "last_completed_turn": { - "turn": 41, - "turn_epoch": 99, - "completed_unix_ms": 1_700_000_000_000, - }, - "outcome": "aborted", - "state_change_seq": 100, - } - ) - observed["meta"]["provider"].update( - { - "turn": 41, - "turn_epoch": 99, - "last_completed_turn": {"turn": 41}, - "outcome": "aborted", - "state_change_seq": 100, - } - ) - _backend, turn_workers, turn_bindings, _records = _project( - config, - [], - [observed], - ) - - assert len(baseline_workers) == len(turn_workers) == 1 - assert json.dumps( - baseline_workers[0].to_dict(), - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") == json.dumps( - turn_workers[0].to_dict(), - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") - assert baseline_workers[0].fingerprint == turn_workers[0].fingerprint - assert baseline_bindings[0].private_fingerprint == ( - turn_bindings[0].private_fingerprint - ) - - -def test_exact_format_version_and_domain_separated_hmac( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - fixture = _fixture() - pane = fixture["pre_restore"]["pane_info"] - config = _config(tmp_path / "state") - key = bytes(range(32)) - config.data_dir.mkdir(mode=0o700) - config.installation_key_path.write_bytes(key) - os.chmod(config.installation_key_path, 0o600) - captured_messages: list[bytes] = [] - original_hmac_new = hmac.new - - def capture_hmac( - hmac_key: bytes, - message: bytes, - digestmod: Any, - ) -> hmac.HMAC: - captured_messages.append(message) - return original_hmac_new(hmac_key, message, digestmod) - - monkeypatch.setattr(worker_identity.hmac, "new", capture_hmac) - worker = _single_worker(config, pane) - message = ( - b'{"backend":"herdr","domain":"tendwire.worker-stable-key",' - b'"host_id":"stable-host","pane_id":"wR9:pA","version":1,' - b'"workspace_id":"wR9"}' - ) - expected = "wsk1_" + original_hmac_new(key, message, hashlib.sha256).hexdigest() - - assert _stable(worker) == expected - assert captured_messages == [message] - assert _STABLE_KEY.fullmatch(expected) - assert type(worker.meta["stable_key_version"]) is int - assert worker.meta["stable_key_version"] == 1 - public_meta = worker.to_dict()["meta"] - assert public_meta["stable_key"] == expected - assert type(public_meta["stable_key_version"]) is int - assert public_meta["stable_key_version"] == 1 - assert config.installation_key_marker_path.read_bytes() == hashlib.sha256(key).hexdigest().encode("ascii") - - -@pytest.mark.parametrize( - ("workspace_id", "pane_id", "expected"), - [ - ("w0", "w0:p0", ("w0", "w0:p0")), - ("w1", "w1:p1", ("w1", "w1:p1")), - ("wZ", "wZ:pZ", ("wZ", "wZ:pZ")), - ( - "w65383a2e877513", - "w65383a2e877513:p4", - ("w65383a2e877513", "w65383a2e877513:p4"), - ), - ( - "w653e50b41be581", - "w653e50b41be581:pC", - ("w653e50b41be581", "w653e50b41be581:pC"), - ), - ( - "wABCDEFGHJKMNPQRSTVWXYZ0123456789", - "wABCDEFGHJKMNPQRSTVWXYZ0123456789:" - "p9876543210ZYXWVTSRQPNMKJHGFEDCBA", - ( - "wABCDEFGHJKMNPQRSTVWXYZ0123456789", - "wABCDEFGHJKMNPQRSTVWXYZ0123456789:" - "p9876543210ZYXWVTSRQPNMKJHGFEDCBA", - ), - ), - (None, "wA:pA", None), - ("wA", None, None), - ("", ":pA", None), - ("w", "w:pA", None), - ("W1", "W1:p1", None), - ("wwA", "wwA:pA", None), - ("wa", "wa:pA", None), - ("w65383a2e87751", "w65383a2e87751:pA", None), - ("w65383a2e8775133", "w65383a2e8775133:pA", None), - ("w65383A2e877513", "w65383A2e877513:pA", None), - ("w65383g2e877513", "w65383g2e877513:pA", None), - ("wAa", "wAa:pA", None), - ("wA-B", "wA-B:pA", None), - ("wA_B", "wA_B:pA", None), - ("wI", "wI:pA", None), - ("wL", "wL:pA", None), - ("wO", "wO:pA", None), - ("wU", "wU:pA", None), - ("wΑ", "wΑ:pA", None), - ("wＡ", "wＡ:pA", None), - ("wA", "wA:p", None), - ("wA", "wA:PA", None), - ("wA", "wA:pa", None), - ("wA", "wA:pAa", None), - ("wA", "wA:pA-B", None), - ("wA", "wA:pA_B", None), - ("wA", "wA:pI", None), - ("wA", "wA:pL", None), - ("wA", "wA:pO", None), - ("wA", "wA:pU", None), - ("wA", "wA:pΑ", None), - ("wA", "wA:pＡ", None), - ("wA", "wA:pA:pB", None), - ("wA", " wA:pA", None), - ("wA", "wA:pA ", None), - ("wA", "wB:pA", None), - ("wA", "wAA:pA", None), - ], - ids=[ - "zero-boundary", - "one-boundary", - "uppercase-boundary", - "current-hex-workspace-numbered-pane", - "current-hex-workspace-uppercase-pane", - "full-authoritative-alphabet", - "missing-workspace", - "missing-pane", - "empty-workspace", - "empty-workspace-suffix", - "uppercase-structural-prefix", - "extra-workspace-prefix", - "lowercase-workspace-suffix", - "short-current-hex-workspace", - "long-current-hex-workspace", - "mixed-current-hex-workspace", - "nonhex-current-workspace", - "mixed-case-workspace-suffix", - "hyphenated-workspace-suffix", - "underscored-workspace-suffix", - "workspace-ascii-I-confusable", - "workspace-ascii-L-confusable", - "workspace-ascii-O-confusable", - "workspace-ascii-U", - "workspace-greek-alpha-confusable", - "workspace-fullwidth-alpha-confusable", - "empty-pane-suffix", - "uppercase-pane-structural-prefix", - "lowercase-pane-suffix", - "mixed-case-pane-suffix", - "hyphenated-pane-suffix", - "underscored-pane-suffix", - "pane-ascii-I-confusable", - "pane-ascii-L-confusable", - "pane-ascii-O-confusable", - "pane-ascii-U", - "pane-greek-alpha-confusable", - "pane-fullwidth-alpha-confusable", - "injected-pane-structure", - "leading-pane-whitespace", - "trailing-pane-whitespace", - "cross-workspace", - "cross-workspace-prefix", - ], -) -def test_canonical_herdr_pane_identity_uses_exact_authoritative_grammar( - workspace_id: str | None, - pane_id: str | None, - expected: tuple[str, str] | None, -) -> None: - assert worker_identity.canonical_herdr_pane_identity(workspace_id, pane_id) == expected - - -def test_worker_record_separates_canonical_identity_from_raw_observation() -> None: - canonical = _worker_record_from_item( - { - "workspaceId": "w65383a2e877513", - "paneId": "w65383a2e877513:pA", - "agent": "codex", - }, - pane_info_observed=True, - identity_source="event:pane.updated", - ) - assert canonical.workspace_id == "w65383a2e877513" - assert canonical.pane_id == "w65383a2e877513:pA" - assert canonical.observed_workspace_id == canonical.workspace_id - assert canonical.observed_pane_id == canonical.pane_id - assert canonical.identity_source == "event:pane.updated" - assert canonical.pane_info_observed is True - - raw_runtime_identity = _worker_record_from_item( - { - "workspace_id": 7, - "pane_id": 41, - "agent": "codex", - }, - pane_info_observed=True, - identity_source="event:pane.agent_status_changed", - ) - assert raw_runtime_identity.workspace_id is None - assert raw_runtime_identity.pane_id is None - assert raw_runtime_identity.observed_workspace_id == "7" - assert raw_runtime_identity.observed_pane_id == "41" - assert raw_runtime_identity.identity_source == "event:pane.agent_status_changed" - assert raw_runtime_identity.pane_info_observed is True - assert herdr_cli._stable_pane_identity(raw_runtime_identity) is None - - -@pytest.mark.parametrize( - ("workspace_id", "pane_id"), - [ - ("w0", "w0:p0"), - ("w1", "w1:p1"), - ("wZ", "wZ:pZ"), - ("w65383a2e877513", "w65383a2e877513:p4"), - ("w653e50b41be581", "w653e50b41be581:pC"), - ( - "wABCDEFGHJKMNPQRSTVWXYZ0123456789", - "wABCDEFGHJKMNPQRSTVWXYZ0123456789:" - "p9876543210ZYXWVTSRQPNMKJHGFEDCBA", - ), - ], -) -def test_every_supported_identity_form_is_restart_stable_and_private( - tmp_path: Path, - workspace_id: str, - pane_id: str, -) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item["workspace_id"] = workspace_id - item["pane_id"] = pane_id - data_dir = tmp_path / "state" - before = _single_worker(_config(data_dir), item) - - restarted_item = deepcopy(item) - restarted_item["terminal_id"] = "runtime-terminal-after-restart" - restarted_item["agent"] = "runtime-agent-after-restart" - restarted_item["agent_session"] = { - "source": "runtime-source-after-restart", - "agent": "runtime-agent-after-restart", - "kind": "id", - "value": "runtime-session-after-restart", - } - after = _single_worker(_config(data_dir), restarted_item) - - assert _stable(after) == _stable(before) - public = json.dumps(after.to_dict(), sort_keys=True) - assert pane_id not in public - assert restarted_item["terminal_id"] not in public - assert restarted_item["agent_session"]["source"] not in public - assert restarted_item["agent_session"]["value"] not in public - - -def test_restore_continuity_ignores_changed_runtime_terminal_agent_and_session(tmp_path: Path) -> None: - fixture = _fixture() - config = _config(tmp_path / "state") - - before = _single_worker(config, fixture["pre_restore"]["pane_info"]) - after = _single_worker(config, fixture["post_restore"]["pane_info"]) - - assert before.id != after.id - assert _stable(before) == _stable(after) - - -def test_split_panes_have_distinct_keys_and_survive_sibling_close_or_reorder(tmp_path: Path) -> None: - fixture = _fixture() - primary = fixture["post_restore"]["pane_info"] - sibling = fixture["post_restore"]["sibling_pane_info"] - config = _config(tmp_path / "state") - - _backend, first, _bindings, _records = _project(config, [], [primary, sibling]) - by_name = {worker.name: _stable(worker) for worker in first} - assert len(set(by_name.values())) == 2 - - _backend, reordered, _bindings, _records = _project(config, [], [sibling, primary]) - assert {worker.name: _stable(worker) for worker in reordered} == by_name - - primary_after_close = _single_worker(config, primary) - assert _stable(primary_after_close) == by_name[primary["agent"]] - - -def test_split_creation_adds_a_distinct_restart_stable_identity(tmp_path: Path) -> None: - fixture = _fixture() - primary = deepcopy(fixture["post_restore"]["pane_info"]) - created = deepcopy(fixture["split_creation"]["data"]["pane"]) - config = _config(tmp_path / "state") - backend, workers, bindings, records = _project(config, [], [primary]) - original = workers[0] - original_key = _stable(original) - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {primary["pane_id"]: primary["terminal_id"]} - backend._replace_ownership_maps(records, bindings) - - assert backend.queue_event_envelope(fixture["split_creation"], flush=True) - - assert len(backend._workers) == 2 - keys = {_stable(worker) for worker in backend._workers.values()} - assert original_key in keys - assert len(keys) == 2 - restarted_created = _single_worker(_config(config.data_dir), created) - assert _stable(restarted_created) in keys - public = json.dumps( - [worker.to_dict() for worker in backend._workers.values()], - sort_keys=True, - ) - assert created["pane_id"] not in public - assert created["terminal_id"] not in public - - -def test_same_pane_suffix_in_distinct_workspaces_has_distinct_keys(tmp_path: Path) -> None: - first_item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - first_item["workspace_id"] = "wA" - first_item["pane_id"] = "wA:p7" - second_item = deepcopy(first_item) - second_item["workspace_id"] = "wB" - second_item["pane_id"] = "wB:p7" - config = _config(tmp_path / "state") - - first = _single_worker(config, first_item) - second = _single_worker(config, second_item) - - assert _stable(first) != _stable(second) - assert "wA:p7" not in json.dumps(first.to_dict(), sort_keys=True) - assert "wB:p7" not in json.dumps(second.to_dict(), sort_keys=True) - - -def test_session_targeted_agent_adopts_matched_pane_identity_privately(tmp_path: Path) -> None: - fixture = _fixture() - pane = deepcopy(fixture["post_restore"]["pane_info"]) - agent = deepcopy(pane) - agent["agent"] = "codex" - agent.pop("workspace_id") - agent.pop("pane_id") - config = _config(tmp_path / "state") - - _backend, workers, _bindings, records = _project(config, [agent], [pane]) - - assert len(records) == len(workers) == 1 - assert records[0].workspace_id == "wR9" - assert records[0].pane_id == "wR9:pA" - assert records[0].turn_target_kind == "codex_session_id" - assert _STABLE_KEY.fullmatch(_stable(workers[0])) - public = json.dumps(workers[0].to_dict(), sort_keys=True) - assert "wR9:pA" not in public - assert pane["terminal_id"] not in public - assert pane["agent_session"]["value"] not in public - - -def test_matched_pane_overrides_conflicting_agent_continuity_and_workspace( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" - agent = { - "worker_id": "public-conflicting-agent", - "agent_id": "agent-send-secret", - "terminal_id": pane["terminal_id"], - "agent": "codex", - "agent_status": "working", - "agent_session": { - "source": "conflicting-agent-source-secret", - "agent": "codex", - "kind": "id", - "value": "conflicting-agent-session-secret", - }, - "workspace_id": "wD2", - "pane_id": "wD2:pA", - } - config = _config(tmp_path / "state") - - _backend, pane_workers, _bindings, pane_records = _project(config, [], [pane]) - _backend, agent_workers, _bindings, agent_records = _project(config, [agent]) - _backend, merged_workers, merged_bindings, merged_records = _project(config, [agent], [pane]) - - assert len(pane_workers) == len(agent_workers) == len(merged_workers) == 1 - assert len(pane_records) == len(agent_records) == len(merged_records) == 1 - assert _stable(merged_workers[0]) == _stable(pane_workers[0]) - assert "stable_key" not in agent_workers[0].meta - assert "stable_key_version" not in agent_workers[0].meta - assert merged_records[0].workspace_id == pane_records[0].workspace_id == "wR9" - assert merged_records[0].pane_id == pane_records[0].pane_id == "wR9:pA" - assert merged_records[0].workspace_id != agent_records[0].workspace_id - assert merged_records[0].pane_id != agent_records[0].pane_id - assert merged_workers[0].space_id == pane_workers[0].space_id == "wR9" - assert merged_workers[0].space_id != agent_workers[0].space_id - assert merged_records[0].turn_target_kind == "codex_session_id" - assert merged_records[0].turn_target_value == pane["agent_session"]["value"] - assert merged_workers[0].backend_target == { - "kind": "agent_id", - "value": "agent-send-secret", - "sendable": True, - "reason": None, - } - assert len(merged_bindings) == 1 - assert merged_bindings[0].target_kind == "agent_id" - assert merged_bindings[0].target_value == "agent-send-secret" - assert merged_bindings[0].turn_target_kind == "codex_session_id" - assert merged_bindings[0].turn_target_value == pane["agent_session"]["value"] - - public = json.dumps(merged_workers[0].to_dict(), sort_keys=True) - for private_value in ( - pane["pane_id"], - pane["terminal_id"], - pane["agent_session"]["source"], - pane["agent_session"]["value"], - agent["agent_id"], - agent["pane_id"], - agent["workspace_id"], - agent["agent_session"]["source"], - agent["agent_session"]["value"], - ): - assert private_value not in public - - -@pytest.mark.parametrize( - ("pane_workspace_id", "pane_id"), - [ - (None, "wR9:pA"), - ("wR9", None), - ("wR9", "wR9:pI"), - ], -) -def test_matched_incomplete_or_invalid_pane_suppresses_agent_identity_derivation( - tmp_path: Path, - pane_workspace_id: str | None, - pane_id: str | None, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" - if pane_workspace_id is None: - pane.pop("workspace_id") - else: - pane["workspace_id"] = pane_workspace_id - if pane_id is None: - pane.pop("pane_id") - else: - pane["pane_id"] = pane_id - agent = { - "worker_id": "public-conflicting-agent", - "agent_id": "agent-send-secret", - "terminal_id": pane["terminal_id"], +def _pane(**updates): + pane = { + "workspace_id": "wR9", + "pane_id": "wR9:pA", + "terminal_id": "term-one", "agent": "codex", - "agent_session": deepcopy(pane["agent_session"]), - "workspace_id": "wD2", - "pane_id": "wD2:pA", + "label": "Review", } - config = _config(tmp_path / f"state-{pane_workspace_id}-{pane_id}") - - _backend, pane_workers, _bindings, pane_records = _project(config, [], [pane]) - _backend, agent_workers, _bindings, _agent_records = _project(config, [agent]) - _backend, merged_workers, _bindings, merged_records = _project(config, [agent], [pane]) - - assert "stable_key" not in agent_workers[0].meta - assert "stable_key_version" not in agent_workers[0].meta - assert "stable_key" not in pane_workers[0].meta - assert "stable_key" not in merged_workers[0].meta - assert "stable_key_version" not in merged_workers[0].meta - assert merged_records[0].workspace_id == pane_records[0].workspace_id - assert merged_records[0].pane_id == pane_records[0].pane_id - assert merged_workers[0].space_id == pane_workers[0].space_id - assert merged_records[0].turn_target_kind == "codex_session_id" - assert merged_records[0].turn_target_value == pane["agent_session"]["value"] - assert merged_workers[0].backend_target is not None - assert merged_workers[0].backend_target["kind"] == "agent_id" - assert merged_workers[0].backend_target["value"] == "agent-send-secret" - - -def test_unmatched_agent_list_identity_never_authorizes_continuity( - tmp_path: Path, -) -> None: - agent = deepcopy(_fixture()["post_restore"]["pane_info"]) - agent["worker_id"] = "public-unmatched-agent" - agent["agent_id"] = "unmatched-agent-target-secret" - agent["agent"] = "codex" - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project(config, [agent]) + pane.update(updates) + return pane + + +def test_camel_snake_nested_envelopes_preserve_workspace_pane_agent_association(tmp_path) -> None: + spaces = _discovered_spaces( + {"result": {"data": {"workspaces": [{"workspaceId": "wR9", "title": "Repo"}]}}} + ) + workers, bindings = _discovered_workers( + _config(tmp_path), + {"result": {"payload": {"panes": [{ + "workspaceId": "wR9", + "paneId": "wR9:pA", + "terminalId": "term-one", + "agent": "pane fallback", + }]}}}, + {"data": {"agents": [{ + "agentId": "agent-private", + "paneId": "wR9:pA", + "name": "codex", + "agentStatus": "waiting", + }]}}, + OBSERVED_AT, + ) + assert [(space.id, space.name) for space in spaces] == [("wR9", "Repo")] + assert [(worker.name, worker.space_id, worker.status) for worker in workers] == [ + ("codex", "wR9", "waiting") + ] + assert bindings[0].target_kind == "agent_id" + assert bindings[0].target_value == "agent-private" - assert len(records) == len(workers) == len(bindings) == 1 - assert records[0].pane_info_observed is False - assert "stable_key" not in workers[0].meta - assert "stable_key_version" not in workers[0].meta - assert not config.installation_key_path.exists() - assert workers[0].backend_target == { - "kind": "agent_id", - "value": "unmatched-agent-target-secret", - "sendable": True, - "reason": None, - } - assert bindings[0].turn_target_kind == "codex_session_id" - assert bindings[0].turn_target_value == agent["agent_session"]["value"] - public = json.dumps(workers[0].to_dict(), sort_keys=True) - for private_value in ( - agent["pane_id"], - agent["terminal_id"], - agent["agent_id"], - agent["agent_session"]["source"], - agent["agent_session"]["value"], - ): - assert private_value not in public +def test_discovery_derives_opaque_restart_stable_key_across_rename_and_reorder(tmp_path) -> None: + pane = _pane() + workers, bindings = _discover(tmp_path, [pane]) + first_key = workers[0].meta["stable_key"] + renamed = {**pane, "agent": "renamed", "label": "Renamed"} + reordered_workers, _ = _discover(tmp_path, [_pane(pane_id="wR9:pB"), renamed]) + renamed_worker = next(worker for worker in reordered_workers if worker.meta["stable_key"] == first_key) + assert renamed_worker.name == "renamed" + assert re.fullmatch(r"wsk1_[0-9a-f]{64}", first_key) + assert bindings[0].target_value == "term-one" + assert "term-one" not in json.dumps(workers[0].to_dict()) -def test_conflicting_match_keys_across_two_panes_fail_closed( - tmp_path: Path, -) -> None: - fixture = _fixture()["post_restore"] - first = deepcopy(fixture["pane_info"]) - second = deepcopy(fixture["sibling_pane_info"]) - second["agent"] = "codex" - second["agent_session"] = { - "source": "second-pane-source-secret", - "agent": "codex", - "kind": "id", - "value": "second-pane-session-secret", - } - agent = { - "worker_id": "public-ambiguous-agent", - "agent_id": "ambiguous-agent-target-secret", - "workspace_id": first["workspace_id"], - "pane_id": first["pane_id"], - "terminal_id": second["terminal_id"], - "agent": "codex", - "agent_session": deepcopy(second["agent_session"]), - } - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project( - config, - [agent], - [first, second], +def test_discovery_reuses_prior_public_worker_id_by_private_binding(tmp_path) -> None: + workers, bindings = _discover(tmp_path, [_pane(agent="old-name")]) + prior = replace( + bindings[0], + worker_id="durable-public-id", + worker_fingerprint=workers[0].fingerprint, ) - - assert len(records) == len(workers) == len(bindings) == 1 - assert records[0].pane_info_observed is False - assert records[0].turn_target_kind is None - assert records[0].turn_target_value is None - assert "stable_key" not in workers[0].meta - assert "stable_key_version" not in workers[0].meta - assert workers[0].backend_target == { - "kind": "agent_id", - "value": "ambiguous-agent-target-secret", - "sendable": False, - "reason": "ambiguous_pane_match", - } - assert bindings[0].sendable is False - assert bindings[0].reason == "ambiguous_pane_match" - assert bindings[0].turn_target_kind is None - assert bindings[0].turn_target_value is None - - public = json.dumps(workers[0].to_dict(), sort_keys=True) - for private_value in ( - first["pane_id"], - first["terminal_id"], - first["agent_session"]["value"], - second["pane_id"], - second["terminal_id"], - second["agent_session"]["source"], - second["agent_session"]["value"], - agent["agent_id"], - ): - assert private_value not in public + current, current_bindings = _discover( + tmp_path, + [_pane(agent="new-name")], + prior_bindings=[prior], + ) + assert current[0].id == "durable-public-id" + assert current_bindings[0].worker_id == "durable-public-id" -def test_two_agents_claiming_one_pane_fail_closed_independent_of_order( - tmp_path: Path, +@pytest.mark.parametrize("match_mode", ["multi_key", "same_pane"]) +@pytest.mark.parametrize("reverse", [False, True]) +def test_conflicting_agent_matches_are_non_sendable_in_any_row_order( + tmp_path, match_mode, reverse ) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" + second_match = ( + {"terminal_id": "term-one"} + if match_mode == "multi_key" + else {"pane_id": "wR9:pA"} + ) agents = [ - { - "worker_id": f"public-agent-{suffix}", - "agent_id": f"agent-target-{suffix}-secret", - "workspace_id": pane["workspace_id"], - "pane_id": pane["pane_id"], - "terminal_id": pane["terminal_id"], - "agent": "codex", - "agent_session": deepcopy(pane["agent_session"]), - } - for suffix in ("a", "b") + {"agent_id": "by-pane", "pane_id": "wR9:pA", "name": "one"}, + {"agent_id": "second", "name": "two", **second_match}, ] - projections: list[tuple[tuple[Any, ...], ...]] = [] - - for index, ordered_agents in enumerate((agents, list(reversed(agents)))): - config = _config(tmp_path / f"state-{index}") - _backend, workers, bindings, records = _project( - config, - ordered_agents, - [pane], - ) + if reverse: + agents.reverse() + workers, bindings = _discover(tmp_path, [_pane()], agents) + assert len(workers) == len(bindings) == 1 + assert workers[0].backend_target["sendable"] is False + assert workers[0].backend_target["reason"] == "ambiguous_pane_match" + assert bindings[0].sendable is False + assert bindings[0].reason == "ambiguous_pane_match" - assert len(records) == len(workers) == len(bindings) == 2 - assert all(record.pane_info_observed is False for record in records) - assert all(record.turn_target_kind is None for record in records) - assert all(record.turn_target_value is None for record in records) - assert all("stable_key" not in worker.meta for worker in workers) - assert all("stable_key_version" not in worker.meta for worker in workers) - assert all( - worker.backend_target is not None - and worker.backend_target["sendable"] is False - and worker.backend_target["reason"] == "ambiguous_pane_match" - for worker in workers - ) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert all(binding.turn_target_kind is None for binding in bindings) - assert all(binding.turn_target_value is None for binding in bindings) - assert not config.installation_key_path.exists() - projections.append( - tuple( - sorted( - ( - worker.id, - worker.name, - worker.space_id, - tuple(sorted((worker.backend_target or {}).items())), - ) - for worker in workers - ) - ) - ) - assert projections[0] == projections[1] +def test_duplicate_private_identity_fences_public_and_private_targets(tmp_path) -> None: + workers, bindings = _discover(tmp_path, [_pane(), _pane()]) + assert len({worker.id for worker in workers}) == 2 + assert all(worker.backend_target["sendable"] is False for worker in workers) + assert all(worker.backend_target["reason"] == "duplicate_backend_target" for worker in workers) + assert all(binding.sendable is False for binding in bindings) + assert all(binding.worker_fingerprint == worker.fingerprint for worker, binding in zip(workers, bindings)) -@pytest.mark.parametrize( - "shared_agent_owner", - ["backend_target", "turn_target", "send_token"], -) -def test_distinct_panes_with_shared_agent_owner_fail_closed_in_any_order( - tmp_path: Path, - shared_agent_owner: str, -) -> None: - fixture = _fixture()["post_restore"] +def test_duplicate_send_token_is_fenced_even_when_private_identities_differ(tmp_path) -> None: panes = [ - deepcopy(fixture["pane_info"]), - deepcopy(fixture["sibling_pane_info"]), + _pane(pane_id="wR9:pA", terminal_id="same", agent="A"), + _pane(pane_id="wR9:pB", terminal_id="same", agent="B"), ] - for pane in panes: - pane["agent"] = "codex" - pane.pop("agent_session", None) - if shared_agent_owner == "send_token": - panes[1]["terminal_id"] = "shared-agent-target-secret" - agents = [] - for index, pane in enumerate(panes): - agents.append( - { - "worker_id": f"public-owner-{index}", - "agent_id": ( - "shared-agent-target-secret" - if shared_agent_owner == "backend_target" - or ( - shared_agent_owner == "send_token" - and index == 0 - ) - else ( - None - if shared_agent_owner == "send_token" - else f"agent-target-{index}-secret" - ) - ), - "workspace_id": pane["workspace_id"], - "pane_id": pane["pane_id"], - "terminal_id": pane["terminal_id"], - "agent": "codex", - "agent_session": { - "source": f"source-{index}-secret", - "agent": "codex", - "kind": "id", - "value": ( - "shared-session-secret" - if shared_agent_owner == "turn_target" - else f"session-{index}-secret" - ), - }, - } - ) - - projections: list[tuple[tuple[Any, ...], ...]] = [] - for order, (agent_rows, pane_rows) in enumerate( - ( - (agents, panes), - (list(reversed(agents)), list(reversed(panes))), - ) - ): - config = _config(tmp_path / f"{shared_agent_owner}-{order}") - _backend, workers, bindings, records = _project( - config, - agent_rows, - pane_rows, - ) - - assert len(records) == len(workers) == len(bindings) == 2 - assert all(record.pane_info_observed is False for record in records) - assert all(record.turn_target_kind is None for record in records) - assert all(record.turn_target_value is None for record in records) - assert all("stable_key" not in worker.meta for worker in workers) - assert all( - worker.backend_target is not None - and worker.backend_target["sendable"] is False - and worker.backend_target["reason"] == "ambiguous_pane_match" - for worker in workers - ) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert all(binding.turn_target_kind is None for binding in bindings) - assert all(binding.turn_target_value is None for binding in bindings) - projections.append( - tuple( - sorted( - ( - worker.id, - tuple(sorted((worker.backend_target or {}).items())), - ) - for worker in workers - ) - ) - ) - - assert projections[0] == projections[1] - - -@pytest.mark.parametrize( - "shared_owner_key", - ["terminal_id", "agent_session", "private_fingerprint"], -) -def test_conflicting_pane_owner_key_fails_closed_independent_of_row_order( - tmp_path: Path, - shared_owner_key: str, -) -> None: - fixture = _fixture()["post_restore"] - first = deepcopy(fixture["pane_info"]) - second = deepcopy(fixture["sibling_pane_info"]) - first["agent"] = "codex" - second["agent"] = "omp" - first["agent_session"] = { - "source": "first-source-secret", - "agent": "codex", - "kind": "id", - "value": "first-session-secret", - } - second["agent_session"] = { - "source": "second-source-secret", - "agent": "omp", - "kind": "id", - "value": "second-session-secret", - } - if shared_owner_key == "terminal_id": - first["terminal_id"] = second["terminal_id"] = "shared-terminal-secret" - elif shared_owner_key == "agent_session": - second["agent_session"]["value"] = first["agent_session"]["value"] - else: - first.pop("agent_session") - second.pop("agent_session") - first["agent_id"] = second["agent_id"] = "shared-agent-target-secret" - - projections: list[tuple[tuple[Any, ...], ...]] = [] - for index, panes in enumerate(([first, second], [second, first])): - config = _config(tmp_path / f"{shared_owner_key}-{index}") - _backend, workers, bindings, records = _project(config, [], panes) - - assert len(records) == len(workers) == len(bindings) == 2 - assert all(record.pane_info_observed is False for record in records) - assert all(record.turn_target_kind is None for record in records) - assert all(record.turn_target_value is None for record in records) - assert all("stable_key" not in worker.meta for worker in workers) - assert all("stable_key_version" not in worker.meta for worker in workers) - assert all( - worker.backend_target is not None - and worker.backend_target["sendable"] is False - and worker.backend_target["reason"] == "ambiguous_pane_match" - for worker in workers - ) - assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert all(binding.turn_target_kind is None for binding in bindings) - assert all(binding.turn_target_value is None for binding in bindings) - assert not config.installation_key_path.exists() - projections.append( - tuple( - sorted( - ( - worker.id, - worker.name, - worker.space_id, - tuple(sorted((worker.backend_target or {}).items())), - ) - for worker in workers - ) - ) - ) - - assert projections[0] == projections[1] - - -def test_unmatched_agent_send_token_colliding_with_pane_fails_closed( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["terminal_id"] = "shared-send-token-secret" - agent = { - "worker_id": "public-unmatched-agent", - "agent_id": "shared-send-token-secret", - "workspace_id": "wD2", - "agent": "other-agent", - } - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project( - config, - [agent], - [pane], - ) - - assert len(records) == len(workers) == len(bindings) == 2 - assert all(record.pane_info_observed is False for record in records) - assert all(record.turn_target_kind is None for record in records) - assert all(record.turn_target_value is None for record in records) - assert all( - worker.backend_target is not None - and worker.backend_target["sendable"] is False - and worker.backend_target["reason"] == "ambiguous_pane_match" - for worker in workers - ) + workers, bindings = _discover(tmp_path, panes) + assert all(worker.backend_target["sendable"] is False for worker in workers) assert all(binding.sendable is False for binding in bindings) - assert all(binding.reason == "ambiguous_pane_match" for binding in bindings) - assert not config.installation_key_path.exists() - - -def test_exact_duplicate_pane_rows_collapse_without_losing_continuity( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project(config, [], [pane, deepcopy(pane)]) - - assert len(records) == len(workers) == len(bindings) == 1 - assert records[0].pane_info_observed is True - assert _STABLE_KEY.fullmatch(_stable(workers[0])) - assert bindings[0].sendable is True - - -def test_matched_pane_replaces_conflicting_pane_scoped_targets( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" - agent = { - "worker_id": "public-pane-target-conflict", - "workspace_id": "wD2", - "pane_id": "wD2:pA", - "agent": "other-agent", - "agent_session": deepcopy(pane["agent_session"]), - } - config = _config(tmp_path / "state") - - _backend, workers, bindings, records = _project(config, [agent], [pane]) - assert len(records) == len(workers) == len(bindings) == 1 - assert records[0].pane_info_observed is True - assert _STABLE_KEY.fullmatch(_stable(workers[0])) - assert records[0].workspace_id == pane["workspace_id"] - assert records[0].pane_id == pane["pane_id"] - assert records[0].terminal_id == pane["terminal_id"] - assert workers[0].space_id == pane["workspace_id"] - assert workers[0].backend_target == { - "kind": "terminal_id", - "value": pane["terminal_id"], - "sendable": True, - "reason": None, - } - assert bindings[0].target_kind == "terminal_id" - assert bindings[0].target_value == pane["terminal_id"] - assert bindings[0].turn_target_kind == "codex_session_id" - assert bindings[0].turn_target_value == pane["agent_session"]["value"] - assert agent["pane_id"] not in { - bindings[0].target_value, - bindings[0].turn_target_value, - } - - public = json.dumps(workers[0].to_dict(), sort_keys=True) - for private_value in ( - pane["pane_id"], - pane["terminal_id"], - pane["agent_session"]["source"], - pane["agent_session"]["value"], - agent["pane_id"], - ): - assert private_value not in public - - -def test_cli_agent_success_always_enriches_identity_from_matching_pane( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - pane = deepcopy(_fixture()["post_restore"]["pane_info"]) - pane["agent"] = "codex" - agent = { - "terminal_id": pane["terminal_id"], - "agent": "codex", - "agent_status": "working", - "agent_session": deepcopy(pane["agent_session"]), - } - responses = { - ("workspace", "list"): {"result": {"workspaces": []}}, - ("agent", "list"): {"result": {"agents": [agent]}}, - ("pane", "list"): {"result": {"panes": [pane]}}, - } - calls: list[tuple[str, ...]] = [] - def fake_run(args: Any, config: Config) -> subprocess.CompletedProcess[str]: - del config - calls.append(tuple(args)) - response = responses.get(tuple(args)) - return subprocess.CompletedProcess( - args=list(args), - returncode=0 if response is not None else 1, - stdout=json.dumps(response) if response is not None else "", - stderr="", - ) - - monkeypatch.setattr(herdr_cli.shutil, "which", lambda _binary: "/usr/bin/herdr") - monkeypatch.setattr(herdr_cli, "_run_herdr", fake_run) +def test_missing_and_malformed_pane_identity_fail_closed(tmp_path) -> None: + workers, bindings = _discover( + tmp_path, + [ + {"workspace_id": "bad", "pane_id": "pane", "terminal_id": "term", "agent": "bad"}, + {"workspace_id": "wR9", "agent": "missing"}, + ], + ) + assert len(workers) == len(bindings) == 1 + assert bindings[0].reason == "invalid_pane_identity" + assert workers[0].backend_target["sendable"] is False - _spaces, workers = herdr_cli.fetch_herdr_state(_config(tmp_path / "state")) - assert calls == [ - ("workspace", "list"), - ("agent", "list"), - ("pane", "list"), +def test_closed_and_unknown_statuses_are_projected_canonically(tmp_path) -> None: + panes = [ + _pane(pane_id="wR9:pA", terminal_id="one", agent="A", status="closed"), + _pane(pane_id="wR9:pB", terminal_id="two", agent="B", status="mystery"), ] - assert len(workers) == 1 - assert _STABLE_KEY.fullmatch(_stable(workers[0])) - - -@pytest.mark.parametrize( - ("workspace_id", "pane_id"), - [ - (None, "wR9:pA"), - ("wR9", None), - ("wR9", "wOther:pA"), - ("wR9", "wR9:pI"), - ("wR9", "wR9:pa"), - ("wR9", "wR9:p"), - ("wR9", "wR9:A"), - ("not-herdr-id", "not-herdr-id:pA"), - ("wI", "wI:pA"), - ], -) -def test_identity_requires_both_canonical_membership_and_herdr_alphabet( - tmp_path: Path, - workspace_id: str | None, - pane_id: str | None, -) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item["terminal_id"] = "durable-terminal-is-not-a-fallback" - if workspace_id is None: - item.pop("workspace_id", None) - else: - item["workspace_id"] = workspace_id - if pane_id is None: - item.pop("pane_id", None) - else: - item["pane_id"] = pane_id - - worker = _single_worker(_config(tmp_path / "state"), item) - - assert "stable_key" not in worker.meta - assert "stable_key_version" not in worker.meta - - -def test_reconcile_and_identical_events_share_one_canonical_worker_fingerprint( - tmp_path: Path, -) -> None: - pane = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state", host_id="fingerprint-stability") - config.data_dir.mkdir(parents=True, mode=0o700) - init_store(Path(config.db_path)) - - class PaneClient: - def workspace_list(self, **_kwargs: Any) -> dict[str, Any]: - return {"workspaces": [{"id": pane["workspace_id"], "name": "Continuity"}]} - - def tab_list(self, **_kwargs: Any) -> dict[str, Any]: - return {"tabs": []} - - def pane_list(self, **_kwargs: Any) -> dict[str, Any]: - return {"panes": [deepcopy(pane)]} + workers, _ = _discover(tmp_path, panes) + assert {worker.name: worker.status for worker in workers} == {"A": "closed", "B": "unknown"} - def agent_list(self, **_kwargs: Any) -> dict[str, Any]: - return {"agents": []} - backend = HerdrEventBackend(config, debounce_seconds=0) - client = PaneClient() - phases: list[tuple[Any, Any]] = [] - - def capture() -> None: - snapshot = latest_snapshot(backend.db_path, backend.config.host_id) - bindings = list_worker_bindings(backend.db_path, backend.config.host_id, backend="herdr") - assert snapshot is not None - assert len(snapshot.workers) == len(bindings) == 1 - phases.append((snapshot.workers[0], bindings[0])) - - backend.reconcile_once(client=client) - capture() - envelope = {"event": "pane.focused", "data": {"pane": deepcopy(pane)}} - assert backend.queue_event_envelope(envelope) is True - capture() - backend.reconcile_once(client=client) - capture() - assert backend.queue_event_envelope(envelope) is True - capture() - - workers = [worker for worker, _binding in phases] - bindings = [binding for _worker, binding in phases] - assert len({worker.id for worker in workers}) == 1 - assert len({_stable(worker) for worker in workers}) == 1 - assert len({worker.fingerprint for worker in workers}) == 1 - assert len({binding.worker_fingerprint for binding in bindings}) == 1 - assert all(binding.worker_id == worker.id for worker, binding in phases) - assert all(binding.worker_fingerprint == worker.fingerprint for worker, binding in phases) - assert len( - { - ( - binding.target_kind, - binding.target_value, - binding.turn_target_kind, - binding.turn_target_value, - binding.private_fingerprint, - binding.sendable, - binding.reason, - ) - for binding in bindings - } - ) == 1 - - worker = workers[0] - canonical = Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=worker.meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, - ) - pre_key_meta = { - key: value - for key, value in worker.meta.items() - if key not in {"stable_key", "stable_key_version"} - } - pre_key = Worker( - id=worker.id, - name=worker.name, - status=worker.status, - space_id=worker.space_id, - meta=pre_key_meta, - last_seen_at=worker.last_seen_at, - summary=worker.summary, +def test_authoritative_empty_discovery_expires_prior_binding(tmp_path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + workers, bindings = _discover(tmp_path, [_pane()]) + save_snapshot( + config.db_path, + Snapshot(host_id=config.host_id, updated_at=OBSERVED_AT, workers=workers), ) - assert worker.fingerprint == canonical.fingerprint - assert worker.fingerprint != pre_key.fingerprint + upsert_worker_bindings(config.db_path, bindings) + class EmptyLifecycleClient: + def workspace_list(self, *, timeout): return [] + def pane_list(self, *, timeout): return [] + def agent_list(self, *, timeout): return [] + def close(self): return None -def test_same_workspace_move_preserves_and_cross_workspace_move_changes_key(tmp_path: Path) -> None: - fixture = _fixture() - config = _config(tmp_path / "state") - initial = fixture["pre_restore"]["pane_info"] - backend, workers, bindings, _records = _project(config, [initial], [initial]) - original = workers[0] - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {initial["pane_id"]: initial["terminal_id"]} - - assert backend.queue_event_envelope(fixture["same_workspace_move"], flush=True) - same_workspace = backend._workers[original.id] - assert _stable(same_workspace) == _stable(original) - - assert backend.queue_event_envelope(fixture["cross_workspace_move"], flush=True) - cross_workspace = backend._workers[original.id] - assert _stable(cross_workspace) != _stable(original) - expected = _single_worker( + supervisor = AcpSupervisor( config, - fixture["cross_workspace_move"]["data"]["pane"], - ) - assert _stable(cross_workspace) == _stable(expected) - assert cross_workspace.space_id == "wD2" - - -def test_compatibility_only_partial_move_preserves_authenticated_local_key( - tmp_path: Path, -) -> None: - initial = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state") - backend, workers, bindings, _records = _project(config, [initial], [initial]) - original = workers[0] - original_key = _stable(original) - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {initial["pane_id"]: initial["terminal_id"]} - - assert backend.queue_event_envelope( - { - "event": "pane.moved", - "payload": { - "previous_pane_id": initial["pane_id"], - "new_pane_id": "wR9:pB", - }, - }, - flush=True, - ) - - moved = backend._workers[original.id] - moved_binding = next(iter(backend._bindings.values())) - assert _stable(moved) == original_key - assert moved_binding.target_kind == "pane_id" - assert moved_binding.target_value == "wR9:pB" - - -def test_complete_authoritative_move_retains_state_when_rederivation_fails( - tmp_path: Path, -) -> None: - initial = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state") - backend, workers, bindings, _records = _project(config, [initial], [initial]) - original = workers[0] - original_binding = bindings[0] - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {initial["pane_id"]: initial["terminal_id"]} - replacement = bytes( - byte ^ 0xFF for byte in config.installation_key_path.read_bytes() - ) - config.installation_key_path.write_bytes(replacement) - os.chmod(config.installation_key_path, 0o600) - moved_pane = deepcopy(initial) - moved_pane["workspace_id"] = "wD2" - moved_pane["pane_id"] = "wD2:p7" - moved_pane["terminal_id"] = "terminal-moved-secret" - - assert backend.queue_event_envelope( - { - "event": "pane.moved", - "data": { - "previous_pane_id": initial["pane_id"], - "pane": moved_pane, - }, - }, - flush=True, - ) - - assert backend._workers[original.id] == original - assert next(iter(backend._bindings.values())) == original_binding - assert backend.health.status == "degraded" - assert backend.health.outcome == "continuity_unavailable" - diagnostic = json.dumps(backend.health.to_backend_health().to_dict(), sort_keys=True) - for private_value in ( - initial["pane_id"], - initial["terminal_id"], - moved_pane["pane_id"], - moved_pane["terminal_id"], - moved_pane["agent_session"]["source"], - moved_pane["agent_session"]["value"], - ): - assert private_value not in diagnostic - - -def test_partial_event_preserves_local_key_and_authoritative_failure_retains_state( - tmp_path: Path, -) -> None: - initial = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state") - backend, workers, bindings, _records = _project(config, [initial], [initial]) - original = workers[0] - original_key = _stable(original) - backend._workers = {original.id: original} - backend._bindings = {binding.private_fingerprint: binding for binding in bindings} - backend._pane_terminals = {initial["pane_id"]: initial["terminal_id"]} - - replacement = bytes(byte ^ 0xFF for byte in config.installation_key_path.read_bytes()) - config.installation_key_path.write_bytes(replacement) - os.chmod(config.installation_key_path, 0o600) - - assert backend.queue_event_envelope( - { - "event": "pane_agent_status_changed", - "data": {"agent": initial["agent"], "status": "blocked"}, - }, - flush=True, - ) - partial = backend._workers[original.id] - assert _stable(partial) == original_key - - assert backend.queue_event_envelope( - { - "event": "pane_agent_status_changed", - "data": {"pane_id": initial["pane_id"], "status": "working"}, - }, - flush=True, - ) - pane_only = backend._workers[original.id] - binding_before_failure = next(iter(backend._bindings.values())) - assert _stable(pane_only) == original_key - - full = deepcopy(initial) - full["agent_status"] = "idle" - assert backend.queue_event_envelope( - { - "event": "pane_agent_status_changed", - "data": {"pane": full}, - }, - flush=True, + threading.Event(), + endpoint_client_factory=lambda _config: object(), + discovery_client_factory=lambda _config: EmptyLifecycleClient(), + connection_factory=lambda *_args, **_kwargs: object(), ) + supervisor._discover_continuity() + snapshot = latest_snapshot(config.db_path, config.host_id) + assert snapshot is not None and snapshot.workers == [] + assert list_worker_bindings(config.db_path, config.host_id, backend="herdr") == [] - assert backend._workers[original.id] == pane_only - assert next(iter(backend._bindings.values())) == binding_before_failure - assert backend.health.status == "degraded" - assert backend.health.outcome == "continuity_unavailable" - - -@pytest.mark.parametrize( - ( - "observed_workspace_id", - "observed_pane_id", - ), - [ - ("wR9", "wR9:pA"), - ("wR9", "wR9:pB"), - ("wD2", "wR9:pA"), - ], -) -def test_key_loss_rejects_every_authoritative_pane_update( - tmp_path: Path, - observed_workspace_id: str, - observed_pane_id: str, -) -> None: - initial = deepcopy(_fixture()["pre_restore"]["pane_info"]) - config = _config(tmp_path / "state") - backend, workers, bindings, records = _project(config, [initial], [initial]) - original = workers[0] - original_binding = bindings[0] - backend._workers = {original.id: original} - backend._bindings = { - binding.private_fingerprint: binding - for binding in bindings - } - backend._pane_terminals = { - initial["pane_id"]: initial["terminal_id"], - } - backend._replace_ownership_maps(records, bindings) - config.installation_key_marker_path.unlink() - - observed = deepcopy(initial) - observed["pane_id"] = observed_pane_id - observed["workspace_id"] = observed_workspace_id - observed["agent"] = "codex-runtime-after-key-loss" - assert backend.queue_event_envelope( - { - "event": "pane.created", - "data": {"pane": observed}, - }, - flush=True, - ) - - assert set(backend._workers) == {original.id} - current = backend._workers[original.id] - current_binding = next(iter(backend._bindings.values())) - assert current == original - assert current_binding == original_binding - assert _stable(current) == _stable(original) - assert backend.health.status == "degraded" - assert backend.health.outcome == "continuity_unavailable" - assert backend._pane_terminals == { - initial["pane_id"]: initial["terminal_id"], - } - assert backend._pane_owners == { - initial["pane_id"]: {original.id}, - } - - -def test_installations_are_unlinkable_and_host_is_part_of_message(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - first = _single_worker(_config(tmp_path / "one"), pane) - second = _single_worker(_config(tmp_path / "two"), pane) - other_host = _single_worker(_config(tmp_path / "one", host_id="other-host"), pane) - - assert _stable(first) != _stable(second) - assert _stable(first) != _stable(other_host) - - -def test_first_bootstrap_publishes_key_marker_and_initialization_sentinel( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "state" - candidate = bytes(range(32)) - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return candidate - assert load_or_create_installation_key(data_dir, random_bytes=generated) == candidate - assert generated_sizes == [32] - assert (data_dir / "installation.key").read_bytes() == candidate - assert (data_dir / "installation.key.sha256").read_bytes() == hashlib.sha256( - candidate, - ).hexdigest().encode("ascii") - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" +def test_supervisor_reconcile_reuses_exact_binding_then_rebinds_target_churn(tmp_path) -> None: + config = _config(tmp_path) + assert config.db_path is not None + init_store(config.db_path) + class MutableLifecycleClient: + pane = _pane(agent="original", terminal_id="term-one") -def test_valid_pre_sentinel_pair_upgrades_only_after_validation(tmp_path: Path) -> None: - data_dir = tmp_path / "state" - data_dir.mkdir(mode=0o700) - key = b"p" * 32 - (data_dir / "installation.key").write_bytes(key) - (data_dir / "installation.key.sha256").write_bytes( - hashlib.sha256(key).hexdigest().encode("ascii"), - ) - os.chmod(data_dir / "installation.key", 0o600) - os.chmod(data_dir / "installation.key.sha256", 0o600) - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"n" * size - - assert load_or_create_installation_key(data_dir, random_bytes=generated) == key - assert generated_sizes == [] - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_complete_initialized_pair_loss_fails_without_replacement(tmp_path: Path) -> None: - data_dir = tmp_path / "state" - original = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"a" * size, - ) - (data_dir / "installation.key").unlink() - (data_dir / "installation.key.sha256").unlink() - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"b" * size - - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key(data_dir, random_bytes=generated) - - assert original == b"a" * 32 - assert generated_sizes == [] - assert not (data_dir / "installation.key").exists() - assert not (data_dir / "installation.key.sha256").exists() - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_explicit_acknowledged_reset_allows_offline_key_rotation(tmp_path: Path) -> None: - data_dir = tmp_path / "state" - original = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"a" * size, - ) - - with pytest.raises(InstallationKeyError, match="reset was not acknowledged"): - reset_installation_key(data_dir, acknowledge_continuity_break=False) - assert (data_dir / "installation.key").read_bytes() == original - - reset_installation_key(data_dir, acknowledge_continuity_break=True) - assert not (data_dir / "installation.key").exists() - assert not (data_dir / "installation.key.sha256").exists() - assert not (data_dir / "installation.key.initialized").exists() - - rotated = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"b" * size, - ) - assert rotated == b"b" * 32 - assert rotated != original - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_initialized_digest_loss_fails_without_rewriting_or_randomness(tmp_path: Path) -> None: - data_dir = tmp_path / "state" - key = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"k" * size, - ) - marker_path = data_dir / "installation.key.sha256" - marker_path.unlink() - key_path = data_dir / "installation.key" - sentinel_path = data_dir / "installation.key.initialized" - key_stat = os.lstat(key_path) - sentinel_stat = os.lstat(sentinel_path) - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"n" * size - - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key(data_dir, random_bytes=generated) - - assert generated_sizes == [] - assert key_path.read_bytes() == key - assert (os.lstat(key_path).st_ino, os.lstat(key_path).st_mtime_ns) == ( - key_stat.st_ino, - key_stat.st_mtime_ns, - ) - assert not marker_path.exists() - assert sentinel_path.read_bytes() == b"1" - assert (os.lstat(sentinel_path).st_ino, os.lstat(sentinel_path).st_mtime_ns) == ( - sentinel_stat.st_ino, - sentinel_stat.st_mtime_ns, - ) + def workspace_list(self, *, timeout): return [{"id": "wR9"}] + def pane_list(self, *, timeout): return [dict(self.pane)] + def agent_list(self, *, timeout): return [] + def close(self): return None - -def test_initialized_replaced_key_and_digest_loss_fails_without_rewriting_or_randomness( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "state" - original = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"a" * size, - ) - key_path = data_dir / "installation.key" - marker_path = data_dir / "installation.key.sha256" - sentinel_path = data_dir / "installation.key.initialized" - replacement = bytes(byte ^ 0xFF for byte in original) - key_path.write_bytes(replacement) - marker_path.unlink() - key_stat = os.lstat(key_path) - sentinel_stat = os.lstat(sentinel_path) - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"n" * size - - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key(data_dir, random_bytes=generated) - - assert generated_sizes == [] - assert key_path.read_bytes() == replacement - assert (os.lstat(key_path).st_ino, os.lstat(key_path).st_mtime_ns) == ( - key_stat.st_ino, - key_stat.st_mtime_ns, - ) - assert not marker_path.exists() - assert sentinel_path.read_bytes() == b"1" - assert (os.lstat(sentinel_path).st_ino, os.lstat(sentinel_path).st_mtime_ns) == ( - sentinel_stat.st_ino, - sentinel_stat.st_mtime_ns, - ) - - -def test_missing_key_with_marker_fails_closed_without_source_fallback(tmp_path: Path) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item["meta"] = {"stableKey": "source-fallback", "stable-key-version": 999} - config = _config(tmp_path / "state") - first = _single_worker(config, item) - assert _stable(first) - config.installation_key_path.unlink() - - after_loss = _single_worker(config, item) - - assert "stable_key" not in after_loss.meta - assert "stable_key_version" not in after_loss.meta - assert "source-fallback" not in json.dumps(after_loss.to_dict()) - - -def test_replaced_key_or_marker_mismatch_fails_closed(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - config = _config(tmp_path / "state") - assert _stable(_single_worker(config, pane)) - original_key = config.installation_key_path.read_bytes() - replacement = bytes(byte ^ 0xFF for byte in original_key) - config.installation_key_path.write_bytes(replacement) - os.chmod(config.installation_key_path, 0o600) - - rotated = _single_worker(config, pane) - assert "stable_key" not in rotated.meta - assert "stable_key_version" not in rotated.meta - - config.installation_key_path.write_bytes(original_key) - config.installation_key_marker_path.write_bytes(b"0" * 64) - mismatched_marker = _single_worker(config, pane) - assert "stable_key" not in mismatched_marker.meta - - -def test_exact_key_and_marker_content_is_required(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - - short_config = _config(tmp_path / "short") - short_config.data_dir.mkdir(mode=0o700) - short_config.installation_key_path.write_bytes(b"x" * 31) - os.chmod(short_config.installation_key_path, 0o600) - assert "stable_key" not in _single_worker(short_config, pane).meta - assert not short_config.installation_key_marker_path.exists() - - marker_config = _config(tmp_path / "marker") - marker_config.data_dir.mkdir(mode=0o700) - key = b"k" * 32 - marker_config.installation_key_path.write_bytes(key) - marker_config.installation_key_marker_path.write_bytes(hashlib.sha256(key).hexdigest().encode("ascii") + b"\n") - os.chmod(marker_config.installation_key_path, 0o600) - os.chmod(marker_config.installation_key_marker_path, 0o600) - assert "stable_key" not in _single_worker(marker_config, pane).meta - assert not marker_config.installation_key_sentinel_path.exists() - - -def test_permissive_umask_still_creates_private_modes(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - config = _config(tmp_path / "state") - previous_umask = os.umask(0) - try: - assert _stable(_single_worker(config, pane)) - finally: - os.umask(previous_umask) - - assert _mode(config.data_dir) == 0o700 - assert _mode(config.installation_key_path) == 0o600 - assert _mode(config.installation_key_marker_path) == 0o600 - assert _mode(config.installation_key_sentinel_path) == 0o600 - - -def test_existing_broad_identity_modes_are_narrowed_in_place_idempotently( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "state" - data_dir.mkdir(mode=0o700) - key = b"m" * 32 - paths = { - "installation.key": data_dir / "installation.key", - "installation.key.sha256": data_dir / "installation.key.sha256", - "installation.key.initialized": data_dir / "installation.key.initialized", - } - paths["installation.key"].write_bytes(key) - paths["installation.key.sha256"].write_bytes( - hashlib.sha256(key).hexdigest().encode("ascii"), - ) - paths["installation.key.initialized"].write_bytes(b"1") - os.chmod(data_dir, 0o755) - for path in paths.values(): - os.chmod(path, 0o644) - - data_dir_inode = os.lstat(data_dir).st_ino - before = { - name: (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for name, path in paths.items() - } - generated_sizes: list[int] = [] - - def generated(size: int) -> bytes: - generated_sizes.append(size) - return b"n" * size - - assert load_or_create_installation_key(data_dir, random_bytes=generated) == key - assert generated_sizes == [] - assert (os.lstat(data_dir).st_ino, _mode(data_dir)) == (data_dir_inode, 0o700) - assert { - name: (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for name, path in paths.items() - } == before - assert {_mode(path) for path in paths.values()} == {0o600} - - repaired = { - name: (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for name, path in paths.items() - } - assert load_or_create_installation_key(data_dir, random_bytes=generated) == key - assert generated_sizes == [] - assert (os.lstat(data_dir).st_ino, _mode(data_dir)) == (data_dir_inode, 0o700) - assert { - name: (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for name, path in paths.items() - } == repaired - assert {_mode(path) for path in paths.values()} == {0o600} - - -def test_equal_or_stricter_private_identity_modes_are_accepted_without_widening( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "strict" - data_dir.mkdir(mode=0o700) - key = b"m" * 32 - marker = hashlib.sha256(key).hexdigest().encode("ascii") - (data_dir / "installation.key").write_bytes(key) - (data_dir / "installation.key.sha256").write_bytes(marker) - (data_dir / "installation.key.initialized").write_bytes(b"1") - os.chmod(data_dir / "installation.key", 0o400) - os.chmod(data_dir / "installation.key.sha256", 0o400) - os.chmod(data_dir / "installation.key.initialized", 0o400) - os.chmod(data_dir, 0o500) - generated_sizes: list[int] = [] - - assert load_or_create_installation_key( - data_dir, - random_bytes=lambda size: generated_sizes.append(size) or b"n" * size, - ) == key - assert generated_sizes == [] - assert _mode(data_dir) == 0o500 - assert _mode(data_dir / "installation.key") == 0o400 - assert _mode(data_dir / "installation.key.sha256") == 0o400 - assert _mode(data_dir / "installation.key.initialized") == 0o400 - - -def test_symlink_identity_files_and_data_dir_fail_closed(tmp_path: Path) -> None: - pane = _fixture()["pre_restore"]["pane_info"] - - key_link_config = _config(tmp_path / "key-link") - key_link_config.data_dir.mkdir(mode=0o700) - outside_key = tmp_path / "outside.key" - outside_key.write_bytes(b"z" * 32) - key_link_config.installation_key_path.symlink_to(outside_key) - assert "stable_key" not in _single_worker(key_link_config, pane).meta - assert not key_link_config.installation_key_marker_path.exists() - - sentinel_link_config = _config(tmp_path / "sentinel-link") - sentinel_link_config.data_dir.mkdir(mode=0o700) - key = b"s" * 32 - sentinel_link_config.installation_key_path.write_bytes(key) - sentinel_link_config.installation_key_marker_path.write_bytes( - hashlib.sha256(key).hexdigest().encode("ascii"), - ) - outside_sentinel = tmp_path / "outside.initialized" - outside_sentinel.write_bytes(b"1") - sentinel_link_config.installation_key_sentinel_path.symlink_to(outside_sentinel) - assert "stable_key" not in _single_worker(sentinel_link_config, pane).meta - - - real_dir = tmp_path / "real-dir" - real_dir.mkdir(mode=0o700) - linked_dir = tmp_path / "linked-dir" - linked_dir.symlink_to(real_dir, target_is_directory=True) - with pytest.raises(InstallationKeyError) as raised: - load_or_create_installation_key(linked_dir) - assert str(raised.value) == "installation identity is unavailable" - - -@pytest.mark.parametrize("operation", ["load-or-create", "acknowledged-reset"]) -def test_identity_lifecycle_stays_on_pinned_data_directory_after_leaf_replacement( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - operation: str, -) -> None: - configured_parent = tmp_path / "configured" - configured_parent.mkdir() - data_dir = configured_parent / "state" - if operation == "acknowledged-reset": - assert load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"o" * size, - ) == b"o" * 32 - - replacement_target = tmp_path / "replacement-target" - replacement_target.mkdir(mode=0o750) - guard = replacement_target / "target.sentinel" - guard.write_bytes(b"must remain unchanged") - os.chmod(guard, 0o640) - target_before = _tree_snapshot(replacement_target) - detached_data_dir = configured_parent / "detached-state" - original_prepare = ( - worker_identity.local_state.prepare_and_open_private_directory - ) - prepare_calls = 0 - - def prepare_then_replace( - path: str | os.PathLike[str], - ) -> tuple[int, Any]: - nonlocal prepare_calls - prepare_calls += 1 - fd, result = original_prepare(path) - Path(path).rename(detached_data_dir) - Path(path).symlink_to(replacement_target, target_is_directory=True) - return fd, result - - monkeypatch.setattr( - worker_identity.local_state, - "prepare_and_open_private_directory", - prepare_then_replace, - ) - - if operation == "load-or-create": - candidate = b"c" * 32 - assert load_or_create_installation_key( - data_dir, - random_bytes=lambda size: candidate[:size], - ) == candidate - assert (detached_data_dir / "installation.key").read_bytes() == candidate - assert ( - detached_data_dir / "installation.key.sha256" - ).read_bytes() == hashlib.sha256(candidate).hexdigest().encode("ascii") - assert ( - detached_data_dir / "installation.key.initialized" - ).read_bytes() == b"1" - else: - reset_installation_key( - data_dir, - acknowledge_continuity_break=True, - ) - assert list(detached_data_dir.iterdir()) == [] - - assert prepare_calls == 1 - assert data_dir.is_symlink() - assert _tree_snapshot(replacement_target) == target_before - - -@pytest.mark.parametrize( - "relative", - [ - pytest.param(False, id="absolute"), - pytest.param(True, id="relative"), - ], -) -def test_intermediate_symlink_above_identity_data_dir_blocks_load_or_create( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - relative: bool, -) -> None: - protected_target = tmp_path / "protected-target" - protected_target.mkdir(mode=0o750) - guard = protected_target / "target.sentinel" - guard.write_bytes(b"must remain unchanged") - os.chmod(guard, 0o640) - - configured_root = tmp_path / "configured" - configured_root.mkdir() - (configured_root / "redirect").symlink_to( - protected_target, - target_is_directory=True, - ) - absolute_data_dir = configured_root / "redirect" / "one" / "two" / "state" - data_dir = ( - Path("configured") / "redirect" / "one" / "two" / "state" - if relative - else absolute_data_dir - ) - if relative: - monkeypatch.chdir(tmp_path) - - before = _tree_snapshot(protected_target) - generated_sizes: list[int] = [] - with pytest.raises(InstallationKeyError) as raised: - load_or_create_installation_key( - data_dir, - random_bytes=lambda size: generated_sizes.append(size) or b"x" * size, - ) - - assert str(raised.value) == "installation identity is unavailable" - assert "configured" not in str(raised.value) - assert "protected-target" not in str(raised.value) - assert generated_sizes == [] - assert _tree_snapshot(protected_target) == before - - -@pytest.mark.parametrize( - "relative", - [ - pytest.param(False, id="absolute"), - pytest.param(True, id="relative"), - ], -) -def test_intermediate_symlink_above_identity_data_dir_blocks_acknowledged_reset( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - relative: bool, -) -> None: - protected_target = tmp_path / "protected-target" - protected_target.mkdir(mode=0o750) - guard = protected_target / "target.sentinel" - guard.write_bytes(b"must remain unchanged") - os.chmod(guard, 0o640) - target_data_dir = protected_target / "one" / "two" / "state" - key = load_or_create_installation_key( - target_data_dir, - random_bytes=lambda size: b"r" * size, - ) - assert key == b"r" * 32 - preserved_temp = target_data_dir / ".tendwire-preserved.tmp" - preserved_temp.write_bytes(b"must not be removed") - os.chmod(preserved_temp, 0o640) - os.chmod(target_data_dir, 0o755) - for name in ( - "installation.key", - "installation.key.sha256", - "installation.key.initialized", - ): - os.chmod(target_data_dir / name, 0o644) - - configured_root = tmp_path / "configured" - configured_root.mkdir() - (configured_root / "redirect").symlink_to( - protected_target, - target_is_directory=True, - ) - absolute_data_dir = configured_root / "redirect" / "one" / "two" / "state" - data_dir = ( - Path("configured") / "redirect" / "one" / "two" / "state" - if relative - else absolute_data_dir - ) - if relative: - monkeypatch.chdir(tmp_path) - - before = _tree_snapshot(protected_target) - with pytest.raises(InstallationKeyError) as raised: - reset_installation_key( - data_dir, - acknowledge_continuity_break=True, - ) - - assert str(raised.value) == "installation identity is unavailable" - assert "configured" not in str(raised.value) - assert "protected-target" not in str(raised.value) - assert _tree_snapshot(protected_target) == before - - -@pytest.mark.parametrize( - "relative", - [ - pytest.param(False, id="absolute"), - pytest.param(True, id="relative"), - ], -) -def test_multi_level_missing_parent_identity_bootstrap_uses_resolved_directories( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - relative: bool, -) -> None: - absolute_data_dir = tmp_path / "bootstrap" / "one" / "two" / "state" - data_dir = ( - Path("bootstrap") / "one" / "two" / "state" - if relative - else absolute_data_dir - ) - if relative: - monkeypatch.chdir(tmp_path) - candidate = bytes(range(32)) - - assert load_or_create_installation_key( - data_dir, - random_bytes=lambda size: candidate[:size], - ) == candidate - - for directory in ( - tmp_path / "bootstrap", - tmp_path / "bootstrap" / "one", - tmp_path / "bootstrap" / "one" / "two", - absolute_data_dir, - ): - assert directory.is_dir() - assert _mode(directory) == 0o700 - assert (absolute_data_dir / "installation.key").read_bytes() == candidate - assert (absolute_data_dir / "installation.key.sha256").read_bytes() == ( - hashlib.sha256(candidate).hexdigest().encode("ascii") - ) - assert (absolute_data_dir / "installation.key.initialized").read_bytes() == b"1" - assert { - _mode(absolute_data_dir / name) - for name in ( - "installation.key", - "installation.key.sha256", - "installation.key.initialized", - ) - } == {0o600} - assert list(absolute_data_dir.glob(".tendwire-*.tmp")) == [] - - -@pytest.mark.parametrize( - "target_name", - [ - pytest.param(None, id="state-directory"), - pytest.param("installation.key", id="key"), - pytest.param("installation.key.sha256", id="digest"), - pytest.param("installation.key.initialized", id="sentinel"), - ], -) -def test_nonregular_identity_entries_fail_closed_without_randomness( - tmp_path: Path, - target_name: str | None, -) -> None: - data_dir = tmp_path / "state" - target = data_dir - if target_name is None: - data_dir.write_bytes(b"not a directory") - else: - data_dir.mkdir(mode=0o700) - key = b"r" * 32 - if target_name != "installation.key": - (data_dir / "installation.key").write_bytes(key) - os.chmod(data_dir / "installation.key", 0o600) - if target_name == "installation.key.initialized": - (data_dir / "installation.key.sha256").write_bytes( - hashlib.sha256(key).hexdigest().encode("ascii"), - ) - os.chmod(data_dir / "installation.key.sha256", 0o600) - target = data_dir / target_name - target.mkdir() - generated_sizes: list[int] = [] - - with pytest.raises(InstallationKeyError) as raised: - load_or_create_installation_key( - data_dir, - random_bytes=lambda size: generated_sizes.append(size) or b"n" * size, - ) - - assert str(raised.value) == "installation identity is unavailable" - assert generated_sizes == [] - if target_name is None: - assert target.read_bytes() == b"not a directory" - else: - assert target.is_dir() - - -def test_wrong_owner_identity_directory_fails_closed_without_changes( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - data_dir = tmp_path / "state" - key = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"o" * size, - ) - paths = ( - data_dir / "installation.key", - data_dir / "installation.key.sha256", - data_dir / "installation.key.initialized", - ) - before = tuple( - (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for path in paths - ) - actual_uid = os.geteuid() - monkeypatch.setattr( - worker_identity.local_state.os, - "geteuid", - lambda: actual_uid + 1, - ) - generated_sizes: list[int] = [] - - with pytest.raises(InstallationKeyError) as raised: - load_or_create_installation_key( - data_dir, - random_bytes=lambda size: generated_sizes.append(size) or b"n" * size, - ) - - assert str(raised.value) == "installation identity is unavailable" - assert generated_sizes == [] - assert tuple( - (path.read_bytes(), os.lstat(path).st_ino, os.lstat(path).st_mtime_ns) - for path in paths - ) == before - assert paths[0].read_bytes() == key - - -def test_source_stable_key_family_is_recursively_stripped_then_replaced(tmp_path: Path) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item["Stable.Key.Future"] = "outer-injection" - item["meta"] = { - "stable_key": "snake-injection", - "StableKeyVersion": 999, - "stable-key-rotation": "kebab-injection", - "sTaBlEkEyFuture": "camel-injection", - "safe": {"stable.key.next": "nested-injection", "kept": "yes"}, - "items": [{"STABLE_KEY_NEXT": "list-injection", "kept": 1}], - } - - worker = _single_worker(_config(tmp_path / "state"), item) - - assert _STABLE_KEY.fullmatch(_stable(worker)) - assert worker.meta["stable_key_version"] == 1 - assert worker.meta["safe"] == {"kept": "yes"} - assert worker.meta["items"] == [{"kept": 1}] - assert set(_reserved_meta_keys(worker.meta)) == {"stable_key", "stable_key_version"} - public = json.dumps(worker.to_dict()) - for sentinel in ( - "outer-injection", - "snake-injection", - "kebab-injection", - "camel-injection", - "nested-injection", - "list-injection", - ): - assert sentinel not in public - - -def test_source_stable_key_injection_without_identity_is_never_preserved(tmp_path: Path) -> None: - item = deepcopy(_fixture()["pre_restore"]["pane_info"]) - item.pop("workspace_id") - item["stableKey"] = "top-source" - item["meta"] = { - "STABLE-KEY-VERSION": 22, - "nested": {"stable.key.future": "nested-source", "safe": True}, - } - - worker = _single_worker(_config(tmp_path / "state"), item) - - assert _reserved_meta_keys(worker.meta) == [] - assert worker.meta["nested"] == {"safe": True} - assert "source" not in json.dumps(worker.to_dict()) - - -def test_key_load_occurs_once_for_a_worker_batch(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - fixture = _fixture() - calls = 0 - original = herdr_cli.load_or_create_installation_key - - def counted(data_dir: Path) -> bytes: - nonlocal calls - calls += 1 - return original(data_dir) - - monkeypatch.setattr(herdr_cli, "load_or_create_installation_key", counted) - config = _config(tmp_path / "state") - _backend, workers, _bindings, _records = _project( + client = MutableLifecycleClient() + supervisor = AcpSupervisor( config, - [], - [fixture["pre_restore"]["pane_info"], fixture["pre_restore"]["sibling_pane_info"]], - ) - - assert len(workers) == 2 - assert calls == 1 - - -def test_atomic_publication_never_exposes_partial_final_file( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - data_dir = tmp_path / "state" - original_write_all = worker_identity.local_state._write_all - - def interrupted_write(fd: int, content: bytes) -> None: - del fd, content - raise OSError("simulated interrupted write") - - monkeypatch.setattr(worker_identity.local_state, "_write_all", interrupted_write) - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key(data_dir) - - assert not (data_dir / "installation.key").exists() - assert not (data_dir / "installation.key.sha256").exists() - assert not (data_dir / "installation.key.initialized").exists() - assert list(data_dir.glob(".tendwire-*.tmp")) == [] - - monkeypatch.setattr(worker_identity.local_state, "_write_all", original_write_all) - assert len(load_or_create_installation_key(data_dir)) == 32 - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_interrupted_sentinel_publication_recovers_without_key_rotation( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - data_dir = tmp_path / "state" - candidate = b"c" * 32 - original_write_all = worker_identity.local_state._write_all - - def interrupt_sentinel(fd: int, content: bytes) -> None: - if content == b"1": - raise OSError("simulated interrupted sentinel write") - original_write_all(fd, content) - - monkeypatch.setattr(worker_identity.local_state, "_write_all", interrupt_sentinel) - with pytest.raises(InstallationKeyError, match="installation identity is unavailable"): - load_or_create_installation_key( - data_dir, - random_bytes=lambda size: candidate, - ) - - assert (data_dir / "installation.key").read_bytes() == candidate - assert (data_dir / "installation.key.sha256").read_bytes() == hashlib.sha256( - candidate, - ).hexdigest().encode("ascii") - assert not (data_dir / "installation.key.initialized").exists() - assert list(data_dir.glob(".tendwire-*.tmp")) == [] - - monkeypatch.setattr(worker_identity.local_state, "_write_all", original_write_all) - recovered = load_or_create_installation_key( - data_dir, - random_bytes=lambda size: b"replacement-that-must-not-be-used"[:size], - ) - assert recovered == candidate - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - - -def test_concurrent_creators_publish_one_complete_key_marker_and_sentinel( - tmp_path: Path, -) -> None: - data_dir = tmp_path / "nested" / "state" - with ThreadPoolExecutor(max_workers=8) as executor: - keys = list(executor.map(lambda _index: load_or_create_installation_key(data_dir), range(16))) - - assert len(set(keys)) == 1 - key = keys[0] - assert (data_dir / "installation.key").read_bytes() == key - assert (data_dir / "installation.key.sha256").read_bytes() == hashlib.sha256(key).hexdigest().encode("ascii") - assert (data_dir / "installation.key.initialized").read_bytes() == b"1" - assert list(data_dir.glob(".tendwire-*.tmp")) == [] - - -def test_public_output_excludes_private_identity_and_binding_fingerprint(tmp_path: Path) -> None: - fixture = _fixture() - before = fixture["pre_restore"] - item = before["pane_info"] - config = _config(tmp_path / "state") - installation_key = b"0123456789abcdef0123456789abcdef" - config.data_dir.mkdir(mode=0o700) - config.installation_key_path.write_bytes(installation_key) - os.chmod(config.installation_key_path, 0o600) - backend, workers, _bindings, records = _project(config, [], [item]) - del backend - worker = workers[0] - record = records[0] - expected_private = worker_binding_private_fingerprint( - host_id=config.host_id, - backend="herdr", - identity_material=_private_identity_material_from_item(item), - ) - public = json.dumps(worker.to_dict(), sort_keys=True) - - assert record.private_fingerprint == expected_private - assert record.private_fingerprint != _stable(worker) - assert record.private_fingerprint not in public - assert installation_key.decode("ascii") not in public - assert installation_key.hex() not in public - assert item["pane_id"] not in public - assert item["terminal_id"] not in public - assert item["agent_session"]["source"] not in public - assert item["agent_session"]["value"] not in public - for field in ("runtime_id", "worker_id", "agent_id"): - assert str(before[field]) not in public - - -def test_stable_derivation_does_not_use_binding_or_public_fingerprint_helpers( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - from tendwire.core.models import Worker - - record = herdr_cli._WorkerRecord( - worker=Worker(id="public", name="Public", status="working", space_id="wR9"), - private_fingerprint="existing-private-fingerprint", - workspace_id="wR9", - pane_id="wR9:pA", - pane_info_observed=True, - ) - - def forbidden(*args: Any, **kwargs: Any) -> str: - raise AssertionError("unrelated fingerprint helper was called") - - monkeypatch.setattr(herdr_cli, "worker_binding_private_fingerprint", forbidden) - monkeypatch.setattr(herdr_cli, "stable_fingerprint", forbidden) - workers, bindings = _workers_and_bindings_from_records(_config(tmp_path / "state"), [record]) - - assert bindings == [] - assert _STABLE_KEY.fullmatch(_stable(workers[0])) + threading.Event(), + endpoint_client_factory=lambda _config: object(), + discovery_client_factory=lambda _config: client, + connection_factory=lambda *_args, **_kwargs: object(), + ) + supervisor._discover_continuity() + first = latest_snapshot(config.db_path, config.host_id) + assert first is not None and first.workers[0].id == "original" + + first_stable_key = first.workers[0].meta["stable_key"] + client.pane = _pane(agent="renamed", terminal_id="term-one") + supervisor._discover_continuity() + second = latest_snapshot(config.db_path, config.host_id) + assert second is not None and second.workers[0].id == "original" + + client.pane = _pane(agent="renamed-again", terminal_id="term-two") + supervisor._discover_continuity() + third = latest_snapshot(config.db_path, config.host_id) + bindings = list_worker_bindings(config.db_path, config.host_id, backend="herdr") + assert third is not None + assert third.workers[0].meta["stable_key"] == first_stable_key + assert bindings[0].worker_id == third.workers[0].id + assert bindings[0].target_value == "term-two" + + +def test_public_projection_contains_no_raw_private_identifiers(tmp_path) -> None: + workers, _ = _discover( + tmp_path, + [_pane(cwd="/secret/private", terminal_id="term-private")], + [{"agent_id": "agent-private", "pane_id": "wR9:pA", "name": "codex"}], + ) + encoded = json.dumps(workers[0].to_dict(), sort_keys=True) + assert "term-private" not in encoded + assert "agent-private" not in encoded + assert "/secret/private" not in encoded