Add transactional approval and durable execution recovery

[alexeygrigorev]

Add transactional approval and durable execution recovery

Status: pending
Tags: enhancement, assistant, backend, infra, testing, data, P1
Depends on: #123
Blocks: M5 conversational todo and M6 Typefully saved-draft children of #122
Architecture: _docs/CONVERSATIONAL_AGENT_PLUGIN_ARCHITECTURE.md at 4e5402c

Product decisions

Keep this slice channel- and domain-independent. It proves the safety boundary with deterministic fake renderers, authorization checks, executors, and reconcilers only. It does not call Telegram, z.ai, task mutation APIs, or Typefully.

Use the conversational state table and repositories introduced by #123. Add one DynamoDB Stream-triggered execution worker and one scheduled recovery invocation of that same worker. Do not add SQS or a general workflow engine in this slice.

Use one immutable proposal version, one or more channel presentations, and one deterministic execution-attempt ID per proposal version. Approval creates no new semantic content: the worker can execute only the stored ProposalSpec.

Use conservative crash handling. Once a worker durably records that provider dispatch is starting, a lost response is potentially an external success. Recovery may retry only with the same provider idempotency key, or may perform a read-only correlation lookup. Otherwise the attempt becomes outcome_unknown and requires an explicit admin resolution. It is better to surface uncertainty than create a duplicate effect.

Scope

Deterministic proposal and presentation service

Add a core service that:

• validates a normalized ProposalSpec from Define conversational state, proposal records, retention, and recovery #123 and canonicalizes it with stable key ordering and explicit normalization rules;
• stores a SHA-256 canonical payload hash;
• renders a deterministic preview through an injected renderer and stores a SHA-256 rendered-view hash;
• rejects nondeterministic output, omitted semantic fields, unknown permissions, unavailable executor builds, or hash mismatch;
• creates a ProposalPresentation with a cryptographically random opaque token of at least 256 bits, stores only its SHA-256 hash, binds it to the exact actor identity, conversation, channel binding, proposal version, rendered view, and expiry, and defaults expiry to 30 minutes;
• revokes older active presentations when a revision creates a new proposal version;
• never accepts proposal content, target data, permissions, or provider parameters from callback/action payloads.

Test time, token generation, renderer, identity/authorization lookup, canonical-target revision lookup, executor registry, and reconciler are injected dependencies.

Atomic approval claim

Approval receives only the opaque presentation token plus trusted request provenance. It hashes the token, loads the referenced server-side records, and performs all read checks before attempting a transaction.

Immediately before the transaction, recheck:

• the actor is authenticated and maps to the active identity bound to the presentation;
• the actor still has the proposal's core permission;
• the conversation and channel binding match the trusted request provenance;
• the presentation is active, unexpired, and bound to the current rendered-view hash;
• the proposal is presented, not superseded/canceled/expired/conflicted/claimed, and its plugin build, schema, policy, payload, and rendered-view digests match;
• the capability-scoped executor build referenced by the proposal is still available;
• the canonical target still has the proposal's stored base revision.

If the base revision changed, conditionally mark the unclaimed proposal conflicted, revoke its active presentations, create no execution attempt, and return a conflict that requires a new proposal. Authorization, actor, channel, expiry, hash, or build mismatch fails closed and creates no attempt.

One TransactWriteItems call must then:

1. condition-check the still-active identity/permission, conversation/channel binding, and canonical target revision records;
2. change exactly one matching presentation from active to consumed;
3. change exactly one matching proposal from presented to claimed;
4. insert one queued ExecutionAttempt if absent; and
5. append a minimal approval audit event if Define conversational state, proposal records, retention, and recovery #123 stores audit events in the same transactional table.

Every item uses condition expressions for the values checked above. The attempt ID and idempotency key are deterministic from the proposal-version identity and immutable payload hash, not from a callback delivery. A retried or simultaneous approval that loses the transaction reads and returns the already stored attempt/result when it belongs to the same proposal; it never queues a second attempt.

The approval path returns a normalized execution_pending interaction after a successful/already-successful claim. It never invokes an executor inline.

Capability-scoped execution and authorization recheck

Define a narrow executor contract keyed by capability and exact build digest. The worker supplies the stored proposal spec, attempt ID, stable idempotency key, and an abort/deadline signal. It does not supply arbitrary database/model/channel clients.

Before any effect, the worker must recheck active identity binding, current core permission, proposal/attempt relationship, exact executor build availability, immutable hashes, and the canonical target base revision. A revoked identity/permission, missing build, corrupt spec/hash, or changed target is a proven no-write failure: record failed_safe with a bounded redacted reason and do not call the executor. A target change after approval does not rewrite or silently refresh the proposal.

The fake executor records calls by idempotency key and can simulate:

• success with a safe effect receipt;
• a proved no-write error;
• timeout/lost response before or after applying the fake effect;
• provider-idempotent replay;
• correlation lookup reporting applied, not_applied, or unknown;
• operator-reconciliation-only behavior.

Result receipts contain only safe IDs, hashes, timestamps, and bounded redacted metadata. Secrets, action tokens, proposal bodies, and provider payloads are prohibited.

Leasing, dispatch boundary, and recovery

An ExecutionAttempt follows only these transitions:

queued -> executing -> succeeded
                    -> failed_safe
                    -> outcome_unknown
outcome_unknown -> succeeded          (authoritative read-only reconciliation)
                -> failed_safe        (authoritative read-only reconciliation)
                -> manually_resolved  (admin records applied/no-effect; no executor call)

succeeded, failed_safe, and manually_resolved are terminal. A terminal attempt is never executed again.

The worker:

• claims an eligible queued attempt with one conditional update, recording a unique lease owner, lease expiry, incremented lease generation, bounded attempt history, and current executor build;
• permits only the current unexpired lease generation to record dispatch or a result;
• records a durable dispatch_started_at immediately before the external/fake executor call;
• records success or safe failure with a conditional update and emits a normalized status/result event;
• treats a stale worker's late result as non-authoritative and routes it through reconciliation rather than overwriting newer state.

The recovery invocation queries the #123 due-work index by nextActionAt; it must not scan the table. It handles:

• queued attempts whose Stream delivery was missed by making them eligible for normal leasing;
• expired executing leases with no dispatch marker by returning them to queued;
• expired dispatched attempts in provider_idempotency mode by leasing and retrying with the same key;
• expired dispatched attempts in correlation_lookup mode by running read-only reconciliation first, then recording succeeded, failed_safe, or outcome_unknown from the authoritative result;
• expired dispatched attempts in operator_reconciliation_only mode by recording outcome_unknown without another executor call.

Infrastructure/transient failures before dispatch use bounded exponential backoff and a configured maximum pre-dispatch lease count. Exceeding it becomes failed_safe only when the system can prove no effect was attempted. outcome_unknown is never blindly requeued.

Status and reconciliation API

Add authenticated backend routes consistent with existing browser/portal authentication:

• GET /api/conversational/execution-attempts/{id} returns a redacted status/result only to the owning authorized actor or an admin;
• POST /api/conversational/execution-attempts/{id}/reconcile is admin-only, accepts no provider result from the caller, and invokes only the attempt's registered read-only reconciler;
• POST /api/conversational/execution-attempts/{id}/resolve is admin-only and valid only for outcome_unknown; it records effect_applied or no_effect, a required bounded reason, actor, timestamp, and safe receipt, transitions to manually_resolved, and never invokes the executor.

All mutation routes require the current attempt revision and use conditional writes. Duplicate reconciliation/resolution requests return the stored state. Non-admin, wrong-owner, stale-revision, malformed, secret-bearing, or non-unknown resolution requests fail closed.

Atomic invariants

• A proposal version has at most one execution attempt and one stable idempotency key.
• A presentation can be consumed at most once; consuming one presentation globally claims the proposal, so every sibling presentation becomes unusable.
• There is no reachable state where a presentation is consumed or a proposal is claimed without the corresponding queued attempt.
• There is no reachable state where an attempt exists for an unclaimed proposal.
• The executor receives exactly the stored spec whose payload and rendered view were approved.
• Only one unexpired lease generation may dispatch or finalize an attempt.
• No executor call occurs from the approval request.
• No automatic executor call occurs after an operator-only uncertain outcome.
• Authorization is checked both at approval and immediately before execution.
• A duplicate callback, Stream record, scheduled recovery event, Lambda retry, or stale worker result cannot create a second effect or overwrite terminal truth.

Acceptance criteria

• Deterministic canonicalization/rendering binds every semantically relevant ProposalSpec field to stored payload and rendered-view hashes; any semantic edit requires a new version/presentation.
• Presentations use short opaque tokens, store only token hashes, bind actor/conversation/channel/version/view, expire after the configurable 30-minute default, and revoke on supersession.
• Approval rechecks identity, permission, channel, state, expiry, exact hashes/build/policy/schema, and base revision using trusted server-side state.
• One DynamoDB transaction consumes the presentation, claims the proposal, and inserts exactly one queued attempt; transaction conditions make partial approval states impossible.
• Duplicate/simultaneous approval returns the same stored attempt/result and never creates a second attempt.
• Approval returns execution_pending without calling an executor.
• The worker uses capability-scoped fake executors, conditional leases, stable idempotency keys, exact stored specs, deadlines, redacted receipts, and a pre-effect authorization/base-revision recheck.
• Stream retries, concurrent workers, expired leases, and stale results preserve single ownership and terminal state.
• Recovery queries due work through an index and applies the declared delivery mode; it never table-scans or blindly retries outcome_unknown.
• Provider-idempotent recovery reuses the same key; correlation recovery performs lookup before any retry; operator-only uncertainty requires manual resolution.
• Authenticated status and admin-only reconciliation/resolution routes enforce owner/role/revision rules and never accept an executable/provider result from the caller.
• Approval, lease, dispatch, result, reconciliation, and manual resolution produce bounded redacted audit/status events without proposal content, tokens, secrets, or raw provider data.
• SAM declares the Stream event source, worker, scheduled recovery trigger, failure destination, environment configuration, and least-privilege IAM; production code creates none of these resources on cold start.
• Existing APIs and assistant-job approval behavior remain unchanged; this new approval contract is not retrofitted onto legacy jobs in this issue.

Test scenarios

Exact preview binding

Given semantically identical objects with different key order, and objects differing in destination, content, permission, policy/schema/build digest, source revision, base revision, or expiry
When the service canonicalizes and renders them
Then key order produces the same hash, every semantic change produces a different hash, the preview is deterministic, and execution receives only the stored approved spec.

Concurrent approval

Given one active presentation and 25 simultaneous approval requests plus duplicate delivery retries
When all requests run against DynamoDB Local
Then one transaction consumes/claims/queues, one attempt ID exists, all successful/idempotent responses refer to it, and the fake executor is not called by the request path.

Wrong or stale action

Given wrong actor/channel, disabled identity, revoked permission, expired/revoked/consumed token, superseded proposal, changed policy/schema/build/hash, and changed base revision
When approval is attempted
Then no attempt is queued; target drift marks the proposal conflicted/revokes controls; every other case fails closed without exposing record existence.

Worker race and stale lease

Given duplicate Stream records, two workers, an expired lease, and a late result from the old worker
When they race to lease and finalize the same attempt
Then only one lease generation may dispatch/finalize, the late result cannot overwrite state, and the fake effect ledger contains at most one effect.

Crash boundaries

Given deterministic crashes (a) after approval transaction, (b) after lease before dispatch marker, (c) after dispatch marker before fake call, (d) after fake effect before response, and (e) after response before result write
When Stream retries and scheduled recovery run
Then (a) is eventually leased, (b) safely returns to queued, and (c-e) follow the declared provider-idempotency/correlation/operator-only policy without blind duplicate execution.

Delivery modes

Given identical crash cases for all three delivery modes
When recovery runs
Then provider-idempotent mode reuses one key, correlation mode performs a read-only lookup first, and operator-only mode becomes outcome_unknown with zero automatic retries.

Authorization change after approval

Given a claimed queued attempt whose actor is disabled, permission is revoked, executor build disappears, or base revision changes before dispatch
When the worker rechecks authorization
Then the fake executor call count remains zero and the attempt becomes a redacted failed_safe.

Reconciliation and manual resolution

Given outcome_unknown attempts and admin/operator/wrong-owner callers
When status, reconcile, and resolve routes are called concurrently or repeatedly
Then owner/admin status access is redacted, only admin may reconcile/resolve, the reconciler is read-only, one conditional resolution wins, repeat calls return stored truth, and no route calls the executor.

Infrastructure and failure delivery

Given the SAM template
When template assertions, sam validate, and sam build run
Then the correct table Stream invokes only the execution worker, scheduled recovery invokes the same worker in recovery mode, failed Stream batches reach the declared failure destination, worker/recovery IAM is limited to required table/index/stream/log resources, and the API function cannot assume worker-only permissions.

Regression and redaction

Given existing backend APIs and errors containing secret-like values
When the full backend suite runs
Then existing behavior passes unchanged and responses, logs, audit events, attempt receipts, and failure-destination payloads expose no secrets, opaque tokens, or proposal bodies.

Required verification

Software Engineer and Tester must run:

npm --prefix backend test
npm --prefix backend run typecheck
npm --prefix backend run build
make sam-validate
make sam-build

Also run focused DynamoDB Local tests for the transaction/race/lease/crash scenarios above, focused route authorization tests, and static SAM assertions for resources, event filters, failure handling, environment variables, and IAM. Use injected clocks and deterministic fake executors/reconcilers; automated tests make no external network calls or writes.

No portal UI changes are in scope, so Playwright and screenshots are not required unless implementation unexpectedly changes a web surface; Tester must state that rationale.

Infrastructure details

• Reuse the Define conversational state, proposal records, retention, and recovery #123 conversational state table, keys, due-work index, encryption, PITR, retention policies, and export/restore boundary.
• Enable DynamoDB Streams with the smallest view needed by the worker and filter the event source to execution-attempt records that become queued.
• Add a separate execution-worker Lambda rather than granting external execution responsibilities to the public backend function.
• Add bounded batch/retry settings, partial batch response, and an encrypted retained SQS failure queue for exhausted Stream delivery. This queue is diagnostic dead-letter storage only, not a work queue or orchestration layer.
• Invoke the same worker on a short EventBridge recovery schedule with an explicit recovery event; do not add a second recovery implementation.
• Grant the worker only required read/conditional-write/query/transaction permissions for conversational execution records and its due-work index. Grant Stream read permissions only through the event source mapping. Fake executors require no provider secrets or unrelated tables.
• Grant the backend only the conversational transaction/status operations it needs. The public API must not receive worker-only provider permissions.
• Configuration covers presentation TTL default, lease duration, pre-dispatch retry/backoff bounds, due-work bucket/index names, and fake executor enablement. Use safe defaults and validate bounds at startup.
• Production resources are SAM/CloudFormation-owned; local table/event setup remains test/local-only.

Dependencies

#123 must first provide the persistent records and conditional repository primitives for proposals, presentations, attempts, audit events, actor/channel bindings, TTL/retention, deterministic lookup by token hash, one unique attempt per proposal version, and a queryable due-work key/index. If #123 chooses different physical tables, it must still support the single approval transaction and indexed recovery invariants above.

This issue intentionally does not depend on M2. The executor/renderer/authorization interfaces are injected, capability-scoped contracts exercised with fakes. M2 and later domain plugins will register real build-digested implementations against these contracts.

Out of scope

• Telegram callbacks/keyboards, voice notes, images, channel rendering, or group behavior.
• z.ai, Groq, Typefully, task, GitHub, or any other real provider/domain mutation.
• Todo or social-post schemas and plugin registration.
• Model calls, skill_load, skill_invoke, plugin selection, or general agent behavior.
• Web proposal UI, cross-channel approval, notification delivery, or cancellation of an executing external action.
• Multiple effects, batches, partial success, compensating transactions, arbitrary workflow graphs, SQS orchestration, Step Functions, or a general flow DSL.
• Retrofitting/replacing legacy AssistantJob approval routes.
• Production reconciliation runbooks or real-provider smoke tests; those belong to the domain plugin and rollout issues.
• Modifying ../dtc-operations, ../datatasks, or ../podcast-assistant.

[alexeygrigorev]

GROOMED

This issue is ready for implementation after #123 closes.

Key product decisions:

• Keep M4 channel/domain independent and prove it with fake renderers, executors, authorization, and reconcilers only.
• Reuse the Define conversational state, proposal records, retention, and recovery #123 conversational state table; add one Stream-triggered worker and invoke that same worker on a recovery schedule. No general workflow engine or SQS work queue.
• Approval is one DynamoDB transaction with authorization/base-revision condition checks plus presentation consume, proposal claim, deterministic attempt creation, and transactional audit when supported by Define conversational state, proposal records, retention, and recovery #123.
• Recheck authorization and target revision immediately before dispatch.
• Mark dispatch durably before the call. After that boundary, retry only with the same provider idempotency key or perform read-only correlation lookup; otherwise surface outcome_unknown.
• Keep manual resolution admin-only, revision-checked, audited, and incapable of invoking the executor.
• Require deterministic DynamoDB Local concurrency/crash tests and SAM/IAM/event-source assertions in addition to the full backend verification.

No real Telegram, z.ai, task, Typefully, or other external mutation is in this slice.