[Architecture Review][P0] Unified state control-plane stores

[qqeasonchen]

Execution meta — see plan: #5296

• Phase: 2 — unified state control plane
• Severity: P0
• Depends on: [Bug] Make RocksDB and Meta offset writes atomically monotonic #5289, [Bug] Do not retire deliveries when durable offset persistence fails #5290, [Bug] Prevent stale ACKs from matching reused delivery IDs #5291, [Bug] Make DLQ transition durable before retiring source delivery #5292
• Blocks: [Bug] Preserve unacknowledged broadcast deliveries during cursor recovery #5294, [Bug] Gate RocketMQ 5 POP broker ACK on distribution completion #5295, [Architecture Review][P1] A2A must not form a parallel Runtime #5302

Problem

State persistence is inconsistent across subsystems. Offsets have a RocksDB path, but subscription registration, push buffers, in-flight deliveries, and A2A tasks remain largely in-memory. A2A documentation also states that InMemoryA2AMessageTransport and the in-memory TaskRegistry are part of the current implementation.

Risks

• A Runtime crash, restart, migration, or network partition produces uncontrollable redelivery or state loss.
• A2A can bypass the Runtime's reliability, DLQ, audit, and authorization paths.
• "At-least-once" cannot be formed into an end-to-end, verifiable contract.

Proposed direction

Introduce a unified state control-plane interface set:

OffsetStore
SubscriptionStore
DeliveryStateStore
SessionStore
DeadLetterStore
TaskStore

In production mode, require:

• subscription and delivery state are recoverable;
• ACK carries deliveryId, deliveryEpoch, and an idempotency key;
• offsets advance monotonically only;
• disconnected push events return to the unified dispatcher;
• A2A tasks reuse the Runtime transport and persisted state.

Related concrete bugs

The following at-least-once reliability defects in the current develop code are the concrete manifestations of this missing unified state control plane. Landing this issue should fix them together:

• [Bug] Make RocksDB and Meta offset writes atomically monotonic #5289 — RocksDB/Meta offset writes are not atomically monotonic
• [Bug] Do not retire deliveries when durable offset persistence fails #5290 — deliveries are retired when durable offset persistence fails
• [Bug] Prevent stale ACKs from matching reused delivery IDs #5291 — stale ACKs match reused delivery IDs
• [Bug] Make DLQ transition durable before retiring source delivery #5292 — DLQ transition is not durable
• [Bug] Preserve unacknowledged broadcast deliveries during cursor recovery #5294 — unacknowledged broadcast deliveries are lost during cursor recovery
• [Bug] Gate RocketMQ 5 POP broker ACK on distribution completion #5295 — RocketMQ 5 POP broker ACK is not gated on distribution completion

Acceptance criteria

• Each store has an interface + at least one durable implementation (RocksDB or Meta-backed).
• Delivery state is recoverable after a hard restart (no orphaned in-flight deliveries).
• ACK validation binds to delivery ownership (id + epoch + idempotency key).
• A2A tasks persist through TaskStore and route through the Runtime dispatcher.

---

Part of the Architecture Review.