Skip to content

Paradigm completion: fault tolerance and recovery on the Flink runtime #6

Description

@Ugbot

The bar: state as a materialized view over the runtime own ordered log plus checkpoints — a separate durable event log is not required.

Two runtimes clear it. agentic-pekko uses real Pekko Persistence (EventSourcedBehavior + journal) and has a test proving the transcript survives entity restart while brain invocations stay flat. agentic-clj uses real Datomic with working as-of time travel.

The flagship Flink runtime does not. enableCheckpointing, CheckpointingMode, setStateBackend and CheckpointedFunction appear zero times in the framework — only in a build-excluded demo, and there with the non-durable HashMapStateBackend.

Supporting gaps:

  • PollingSource splits are explicitly stateless (serialize() returns new byte[0]), so every non-Kafka channel is at-most-once across a restart.
  • No sink implements SupportsCommitter/TwoPhaseCommittingSink.
  • Savepoint restore of an A2A bridge stream NPEs (snapshot restored with a null serializer arg).
  • Both Flink-state vector memories call getMapState() from non-keyed ProcessFunctions, so they throw in open() and are unreachable through the shipped corpus/retrieval API.

Deliverable: checkpointing configured (or explicitly documented as a caller responsibility), a durable state backend option, sources that actually snapshot offsets, and a test that restarts from a checkpoint and asserts the conversation view is intact.


Roadmap epic. Detailed findings, file:line citations and sub-tasks live in the AI-grind tracker (project AGS), which is the source of truth for this work. Local dev — build, test, tracking, profiling — is managed with AI-grind.


Tracked as AGS-6 in the devtools-mcp tracker.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions