Epic: Deliver a Telegram-first conversational agent MVP

[alexeygrigorev]

Epic: Deliver a Telegram-first conversational agent MVP

Status: pending (planning epic; implement through seven dependency-ordered child issues)
Tags: enhancement, assistant, backend, infra, testing, data, P1
Depends on: None
Blocks: None
Architecture: _docs/CONVERSATIONAL_AGENT_PLUGIN_ARCHITECTURE.md at 4e5402c

Product decision

This remains an epic. Do not implement it as one branch.

The smallest complete MVP supports verified DataOps users in Telegram private chats using text, voice notes, and photos. It proves exactly one internal mutation—creating one todo through preview and approval—then one external mutation—creating one unscheduled, unpublished Typefully draft through the same approval/recovery system.

The MVP deliberately excludes /todo, Telegram group conversations, generic uploads, the web adapter, cross-channel continuation, SOP/podcast mutation, workflows, recurring work, history-wide retrieval, and durable personal memory.

Operator journeys

Text, voice, and photo input

• A verified user may type, send one Telegram OGG voice note, or send one Telegram photo/JPEG with an optional caption.
• Voice is downloaded through the authenticated Telegram file API, bounded to 20 MB and five minutes, and transcribed by Groq whisper-large-v3.
• A photo is downloaded through the same bounded path, limited to 10 MB/20 megapixels, and described/OCRed by z.ai glm-4.6v through its native multimodal endpoint.
• The bot shows the derived transcript/description before it becomes conversational input so the operator can correct or discard it.
• Conversion failure creates no guessed intent and asks the operator to send text.
• Temporary media bytes go only to the dedicated media processor, never DynamoDB, model context, logs, audit payloads, or exports. A finally path deletes them after success or failure, and bounded orphan cleanup covers crashes.

Conversational todo

1. A verified user asks for one todo using text or confirmed derived media text.
2. The runtime selects the todo plugin through skill_load; a separate bounded model turn receives only the todo instructions/schema and calls validated skill_invoke.
3. The bot asks only required clarifications and presents the exact todo.
4. Approve todo atomically consumes the current presentation and queues durable execution. Request changes revises the draft. Cancel proposal revokes approval controls but retains the draft. Discard draft abandons working state.
5. The bot reports execution_pending, then a status update/result. One task is created at most once.

There is no direct /todo shortcut in the MVP. Every todo mutation uses the proposal boundary.

Typefully saved draft

1. After the todo path proves approval and crash recovery, a verified user asks for social copy from public source material and selects Alexey or DataTalksClub plus X and/or LinkedIn.
2. The z.ai conversational model uses the Anthropic-compatible Messages endpoint to create structured copy. Nothing is written to Typefully.
3. The bot previews all current copy. It does not ask for a preferred publish time because this action cannot schedule.
4. Approve and add to Typefully queues the exact stored proposal for execution.
5. The executor creates one saved draft only, with no publish_at, scheduling, publication, update, or other side effect.
6. The bot reports that the result remains unscheduled/unpublished and returns the private edit link only to the authorized operator.

Organization, user-private, and restricted source documents are blocked from Typefully. A saved draft is third-party data egress even when unpublished.

Core contracts and ownership

• Authentication Session records remain separate from conversations.
• Channel adapters normalize message, voice_note, photo, button_action, and session_command; plugins never call Telegram.
• Core outgoing interactions include clarification, proposal_preview, approval_request, execution_pending, status_update, result, and error.
• The core owns select_option, submit_input, approve, request_changes, cancel_proposal, discard_draft, and conversation transitions. Plugins cannot define approval, cancellation, policy, or next-state rules.
• Plugins are trusted TypeScript objects in a build-time registry. No runtime plugin installation, arbitrary packages, flow DSL, or multi-plugin mutation plan.
• skill_load ends the current model turn. skill_invoke occurs in a separate turn bound to conversation/revision, plugin build digest, and one-time load nonce.
• Model/provider text is untrusted and can prepare a proposal only. It cannot grant permissions or execute.
• Inactivity never silently pauses, replaces, or resumes a conversation. Session changes are explicit. Proposal presentations expire independently after a configurable 30-minute default.

Persistent state and execution model

Use separate logical records rather than one overloaded proposal status:

• IdentityBinding: DataOps user ↔ immutable Telegram numeric user ID, status, provisioner, timestamps, revision.
• Conversation, ChannelBinding, ordered append-only ConversationEvent, and replaceable SummaryCheckpoint with source sequence receipt.
• PluginDraft: collecting | ready | abandoned.
• immutable ProposalVersion: presented | superseded | expired | canceled | claimed | conflicted plus normalized ProposalSpec, plugin build/schema/policy digests, base revision, canonical payload hash, and rendered-view hash.
• ProposalPresentation: active | consumed | revoked | expired, channel/actor binding, opaque token hash, and expiry.
• ExecutionAttempt: queued | executing | succeeded | failed_safe | outcome_unknown | manually_resolved, lease, idempotency/correlation data, result receipt, and attempt history.
• bounded AuditEvent metadata/hashes without conversation bodies or media-derived text.

Approval uses one DynamoDB transaction to consume the presentation, claim the exact proposal, and insert a queued ExecutionAttempt. The webhook returns execution_pending; it does not perform the mutation inline.

A DynamoDB Stream-triggered worker leases attempts and invokes only a capability-scoped executor. A scheduled recovery scan redelivers unleased/interrupted queued work and recovers expired leases. Duplicate callbacks and delivery retries return stored state and never create a second effect.

Every external executor declares provider_idempotency, correlation_lookup, or operator_reconciliation_only. outcome_unknown forbids automatic retry. Typefully cannot be enabled until its provider guarantee, reliable lookup, or explicit manual reconciliation procedure is accepted.

Retention, privacy, and knowledge boundary

• Raw messages, derived voice/photo text, summaries, proposal payloads, and redacted provider results expire after 30 days.
• Minimal audit metadata/effect receipts are retained for one year.
• Expiry/deletion removes related derived context and never creates automatic durable memory.
• Portable export/validation and restore drills preserve relationships and terminal state but redact/omit secrets, action-token material, temporary bytes, and replayable approvals.
• Production tables are owned by SAM/CloudFormation with encryption, PITR, retained resources, stack-scoped names/tags, and least-privilege IAM. Production code creates no tables on cold start.
• Public DataTalksClub/dataops contains runtime code, schemas, sanitized fixtures, infrastructure, and public-safe planning only.
• Operational documents, process-bearing plugin instructions, generated SOP/podcast content, and private artifacts belong in DataTalksClub/dataops-knowledge or private managed storage.
• A future SOP approval opens a branch/PR in the private knowledge repository; it never merges or writes directly to canonical main.
• Existing public content/ is migration debt, not an agent write target.

Provider and credential requirements

Automated tests use injected/fake clients and never external network writes.

Production needs managed, least-privilege secret/config wiring for:

• z.ai conversational model through https://api.z.ai/api/anthropic/v1/messages (default current model glm-5.2);
• z.ai native multimodal glm-4.6v photo processing;
• Groq whisper-large-v3 transcription;
• Telegram bot token/webhook secret and allowlist;
• Typefully token and configured account/social-set identifiers.

Secrets must be pre-created out of band in AWS Secrets Manager and referenced by SAM parameters/environment names. GitHub Actions passes only secret references through the existing OIDC deployment. No API key belongs in source, CloudFormation plaintext parameters, issue text, logs, events, exports, screenshots, model context, or public fixtures.

A valid development z.ai key has already passed a minimal operator smoke, but the deployed Lambda still requires dedicated z.ai secret/SAM/IAM/deploy wiring. Real provider/Telegram checks remain [HUMAN].

Seven child issues

The orchestrator should create/link these issues in this order. Reuse and regroom #121 for M3 rather than filing a duplicate. Each child independently follows PM → Software Engineer → Tester → PM acceptance → commit/merge/push → On-Call.

M1 — Define conversational state, proposal records, retention, and recovery

Owns normalized channel-neutral event/interaction types; IdentityBinding, conversation/event/checkpoint, PluginDraft, ProposalVersion, ProposalPresentation, ExecutionAttempt, and audit schemas; DynamoDB repositories/conditional primitives; 30-day/one-year retention; table/IAM/SAM/local setup; portable export/validation; deletion and restore-drill contracts.

It defines state and storage primitives only. It does not call Telegram, models, plugins, task writers, or Typefully.

M2 — Build the static plugin runtime and z.ai conversational adapter

Owns build-time registry validation, core-owned actions/transitions, compact catalog, enable/role/channel policy, two-turn skill_load/skill_invoke, plugin build/schema digests, stale-turn rejection, token-budgeted context receipts, strict model output validation, redaction/limits, and the z.ai Anthropic-compatible conversational client plus production secret/config wiring.

Uses fake plugins/executors only; no production mutation.

M3 — Consolidate private Telegram identity, text, voice, and photo input (#121)

Owns audited admin-managed identity binding, private-chat-only enforcement, Telegram message/callback/file normalization, update deduplication, ordered conversation events, explicit /new, /sessions, /continue, /cancel, /discard, /help, inline keyboard/status rendering, and group fail-closed behavior.

Also owns bounded Telegram media download, Groq whisper-large-v3 voice processing, z.ai glm-4.6v photo description/OCR, derived-text review/correction/discard, dedicated temporary-byte lifecycle/orphan cleanup, media provider secrets/config, and fake media clients. Generic documents/images sent as files remain unsupported.

M4 — Add transactional approval and durable execution recovery

Owns deterministic proposal rendering/hashes, opaque presentations, 30-minute expiry, actor/channel authorization, base-revision conflict, supersession/revocation, one-transaction consume+claim+queue, execution_pending/status interactions, Stream worker, leases, scheduled recovery, capability-scoped executors, idempotency receipts, failed_safe, outcome_unknown, and reconciliation/manual-resolution APIs.

Uses fake internal/external executors; it does not mutate tasks or Typefully.

M5 — Ship approved conversational todo creation

Registers create-only todo, connects verified Telegram text/confirmed media-derived text to clarification, one-todo preview, revision/cancel/discard, transactional approval, durable task execution, result status, conflict/duplicate/crash recovery, and existing task/audit relationships.

No batches, update/delete, partial success, or /todo shortcut.

M6 — Ship approved Typefully saved-draft creation

Registers create-only typefully, removes the current premature Typefully write, reuses the safe request builder and sanitized public style examples, enforces public-source-only egress, previews all X/LinkedIn copy with no preferred-time field, queues exact approval, records safe draft proof, and handles provider idempotency/reconciliation/timeouts.

No update, schedule, publish, next-free-slot, analytics, media upload, or account management. /social is retired or reduced to a compatibility message only after conversational parity.

M7 — Roll out and cut over the Telegram conversational MVP

Owns feature/plugin kill switches, production secret readiness, compatibility cutover for legacy Telegram routes, redacted metrics/alarms, retention/reaper/recovery operations, real-provider smoke gates, operator-facing help/error copy, and rollback evidence. Legacy paths remain available or fail safely until replacement parity is verified.

Dependency graph

M1 state/storage
 ├──> M2 plugin runtime + conversational z.ai
 └──> M4 approval + durable worker

M1 + M2 ──> M3 private Telegram + voice/photo
M2 + M3 + M4 ──> M5 conversational todo
M5 ──> M6 Typefully create-only
M5 + M6 ──> M7 rollout/cutover

M2 and M4 may run in parallel after M1. M3 follows M2 because it consumes the shared model/config boundary. Typefully follows M5 rather than running in parallel because the accepted architecture requires todo to prove approval and crash recovery first.

Epic acceptance criteria

MVP behavior and safety

• Seven linked child issues are independently groomed and delivered in the dependency order above; Consolidate private Telegram identity, text, voice, and photo input #121 is reused as M3.
• A verified private-chat operator can use text, one bounded voice note, or one bounded photo; derived text is shown before conversational use and can be corrected/discarded.
• Voice uses Groq whisper-large-v3; photo understanding/OCR uses z.ai glm-4.6v; conversation uses z.ai's Anthropic-compatible Messages API. Provider/model IDs are validated configuration.
• Temporary media bytes are isolated, never persisted/logged, deleted on success/failure, and covered by crash-orphan cleanup; derived text is private/untrusted and expires after 30 days.
• A verified operator can create exactly one todo through complete preview and exact approval; no direct /todo exists in the MVP.
• After the todo path proves recovery, a verified operator can create exactly one Typefully saved draft from public sources after approval.
• Typefully receives no request before approval and never receives publish_at; scheduling, publishing, updating, preferred-time metadata, and private/restricted sources are absent.
• Unlinked/disabled users, group messages, stale/wrong-actor callbacks, detached “yes”, expired/revoked presentations, plugin-build mismatch, and changed base revisions execute nothing.
• Core—not plugins—owns approval, request changes, cancel proposal, discard draft, and session transitions.
• PluginDraft, immutable ProposalVersion, channel-specific ProposalPresentation, and durable ExecutionAttempt remain distinct records with explicit states.
• Approval transactionally consumes one presentation, claims one proposal, and queues one attempt; a leased worker executes outside the webhook and recovery handles interrupted delivery/leases.
• Duplicate updates/callbacks/worker delivery create at most one effect; outcome_unknown is never automatically retried.
• Inactivity never changes conversation identity or revives an approval; session lifecycle is explicit and presentations expire after 30 minutes by default.
• Authentication sessions remain separate; private payload retention is 30 days and minimal audit/effect receipts are one year.
• Production tables/secrets/IAM/deploy wiring are stack-owned, encrypted, recoverable, least privilege, and portable without exposing secrets or replayable approvals.
• Generic uploads, groups, web, SOP/podcast mutation, workflows, recurring work, history search, memory, and /todo remain out of the MVP.
• Kill switches disable the runtime or a plugin without breaking unrelated DataOps APIs.

HUMAN rollout gates

• [HUMAN] Provision/confirm production z.ai secret/IAM/SAM/OIDC wiring and run one redacted conversational glm-5.2 smoke.
• [HUMAN] Provision/confirm Groq transcription configuration and send one real multilingual Telegram voice note; verify the shown transcript, correction/discard path, and deletion behavior.
• [HUMAN] Send one real Telegram photo through z.ai glm-4.6v; verify useful description/OCR, caption handling, correction/discard path, and no retained raw image.
• [HUMAN] In a real linked private chat, revise and approve one todo; duplicate taps create one task, while group/unlinked attempts fail closed.
• [HUMAN] Approve one public-source Typefully proposal and confirm one draft appears in the intended account as saved, unscheduled, and unpublished.
• [HUMAN] Exercise kill switches and inspect production alarms/logs for redacted, actionable webhook, media, model, worker, reconciliation, and retention failures.

Automated test scenarios

Identity, channel, and concurrency

Given two linked users/chats, an unlinked/disabled user, group updates, duplicate/out-of-order update_id values, and concurrent turns
When the adapter processes messages/actions
Then private state never crosses users; groups/unlinked users invoke no model or mutation; events are ordered/idempotent; stale model work cannot commit over a newer conversation revision.

Voice and photo boundaries

Given fake Telegram download, Groq, and z.ai vision clients plus valid/oversized/wrong-format/corrupt/time-out inputs
When voice/photo events are handled
Then only verified private users reach bounded processing; exactly the declared provider receives temporary bytes; derived text is shown before downstream use and may be corrected/discarded; failures ask for text; bytes are deleted in finally; orphan cleanup removes crash leftovers; no bytes/derived text appear in logs/audit/export/model calls before confirmation.

Plugin loading and context

Given fake todo and typefully registrations
When a capability is selected
Then skill_load ends turn one; turn two is bound to revision/build digest/nonce and includes only that plugin schema; wrong plugin/version/nonce, extra fields, stale turn, invalid action, or plugin-defined transition is rejected before proposal/execution.

Exact proposal and core actions

Given presented v1, revised v2, cancel/discard/end actions, expired controls, and a changed canonical base revision
When controls are used
Then v1 is superseded, cancel revokes presentations but retains the draft, discard abandons working state, ending closes the conversation, expired/wrong controls execute nothing, and a changed target becomes conflicted without overwrite.

Transactional claim, duplicate tap, and worker recovery

Given simultaneous valid callbacks, duplicate Stream delivery, a worker crash before/after provider call, and an expired lease
When approval/execution/recovery runs
Then one transaction creates one queued attempt; the webhook returns execution_pending; one worker owns a lease; recovery safely redelivers only eligible work; receipts are idempotent; a possible external success becomes outcome_unknown, not an automatic retry.

Conversational todo

Given mocked z.ai output and task executor
When the user clarifies, revises, cancels, discards, and finally approves one todo
Then no task exists before approval, exactly the displayed todo is created once after approval, /todo is not routed as a mutation shortcut, and completion/error statuses are understandable.

Typefully create-only and classification

Given public, organization, user-private, and restricted source references plus fake z.ai/Typefully clients
When social copy is proposed, revised, canceled, expired, or approved
Then Typefully call count is zero before current approval; only public sources may proceed; the approved request creates one draft with no preferred time or publish_at; update/schedule/publish attempts fail validation; proof contains safe IDs only.

Provider contract and redaction

Given injected conversational, transcription, vision, Telegram, and Typefully clients
When providers succeed, time out, rate limit, return malformed output, exceed limits, or emit secret-like errors
Then endpoints/models/config are correct, calls/time/turns/tokens are bounded, failures are redacted and recoverable, no invalid proposal/write occurs, and automated tests use no live credential/network.

Retention, export, restore, and replay resistance

Given sanitized conversations, derived media text, proposal/presentation/attempt states, audit receipts, and records older than policy
When retention, export validation, and a non-production restore drill run
Then private payloads expire at 30 days; minimal receipts remain one year; relationships and terminal outcomes restore; secrets/tokens/media are omitted; a restored consumed presentation cannot execute again.

Infrastructure and rollback

Given the full SAM template and OIDC deploy workflow
When infrastructure tests, SAM validation/build, and feature-flag tests run
Then tables have encryption/PITR/retention, worker/recovery resources and IAM are least privilege, all provider secret references reach only required components, no plaintext secret exists, and disabling runtime/media/plugin paths preserves unrelated APIs.

Required verification for affected children

Run the full relevant workflow:

npm --prefix backend test
npm --prefix backend run typecheck
npm --prefix backend run build
make sam-validate
make sam-build

Also:

• Add focused unit/API tests for record transitions, DynamoDB transactions/races, leases/recovery, auth, event normalization, media cleanup, model/plugin contracts, egress classification, and executors.
• Use fake Telegram, Groq, z.ai, Typefully, task, and GitHub clients in normal automated tests; no external writes/network.
• Run portable export validation and a non-destructive restore drill in M1 and whenever persistent/export contracts change; evidence stays under .tmp/exports/.
• Run npm --prefix backend run test:e2e and capture/read screenshots only when a child changes the portal/web UI. Pure core/Telegram children may omit it with Tester rationale.
• Assistant Engineer reviews M2, M3 media/provider boundaries, M5, and M6 before Tester acceptance.
• On-Call monitors the existing OIDC deployment after every shipped child. Secret provisioning and real provider/Telegram delivery remain [HUMAN].

Out of scope for the MVP

• Direct /todo or any mutation command bypass.
• Telegram groups/topics/shared context; a group mention may return only a static private-chat invitation.
• Generic file/image-as-document uploads, PDF/office/archive/link processing, or broad attachment classification.
• Web conversation UI, cross-channel continuation, and web approval.
• SOP/podcast mutation, GitHub writes, workflows, recurring work, calendar, newsletter, sponsor, bookkeeping, or multi-effect plugins.
• Typefully update, schedule, publish, preferred timing, next-free-slot, analytics, media upload, or account management.
• History-wide ambiguous-reference search or automatic durable memory.
• Dynamic plugin discovery, arbitrary package loading, semantic catalog search, general flow DSL, or multi-plugin mutation plans.
• Direct model access to task/docs/GitHub/Typefully mutation APIs or treating assistant-job approval as execution authorization.
• Raw operational prompts/documents, private links, credentials, generated operational artifacts, or unsanitized media/style fixtures in this public repository.
• Modifying ../dtc-operations, ../datatasks, or ../podcast-assistant.

Post-MVP sequence

After M7 is stable, file separate epics/issues for: private-repository SOP PR creation; podcast document PR creation; web adapter/cross-channel presentation; authorized history lookup; workflow/recurring plugins; then optional explicit memory operations after privacy decisions.

[alexeygrigorev]

PM GROOMED: Reframed this as a dependency-ordered epic rather than one implementation branch. The MVP is intentionally limited to private Telegram, admin-provisioned identity links, persistent bounded conversations, a static two-turn plugin runtime, immutable proposal approval, conversational todo plus deterministic /todo, and Typefully saved-draft creation only after exact approval. The body records seven MVP children (reusing #121 for the Telegram adapter), their dependency graph, the current z.ai credential/deployment gap, complete automated race/security/provider/retention tests, and four [HUMAN] rollout gates. SOP, podcast, history/memory, group chats, workflow plugins, uploads, and the web conversation UI are explicitly post-MVP. No child issues were created during this grooming pass; the orchestrator can file/link them from the specified breakdown before implementation.