Skip to content

Fix SQL sink final flush and handler errors when the stream ends - #1011

Merged
maoueh merged 1 commit into
streamingfast:developfrom
aggris2:fix/sql-sink-stop-and-handler-errors
Oct 29, 2026
Merged

maoueh merged 1 commit into
streamingfast:developfrom
aggris2:fix/sql-sink-stop-and-handler-errors

Conversation

@aggris2

@aggris2 aggris2 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Builds on #1007. Until that one is merged, this PR also shows its commit.

A few problems with how the sink finishes, found while testing restarts and failovers:

The final flush prunes undo history. When the stream ends (stopped, or at the stop block), the completion flush passes the stream's latest block as the final block. That deletes the history rows of blocks that are still reversible. If one of them is reorged after the sink restarts, the undo has nothing to revert and the orphaned rows stay. With --undo-buffer-size it also saves the stream's cursor, which is ahead of the blocks still in the buffer, so those get skipped on restart.

A handler error wrapping io.EOF ends the run as if it completed. The stream loop checks errors.Is(err, io.EOF) first, so a lost database connection (failed to begin db transaction: EOF) looks like the stop block was reached. A live sink then exits with status 0 and isn't restarted on failure.

Handler errors are swallowed while stopping. Once the context is cancelled, any error is ignored and the run ends as completed, so the completion handler can flush a block the handler didn't finish.

Also, Revert's deferred rollback never ran: it checked a different err.

Fix:

  • The SQL sinker keeps the cursor of the last block it handled and the stream's final block height, and the completion flush uses those. Nothing is flushed if a block failed to be handled.
  • The sink library marks handler errors:
    • they never count as end of stream;
    • while stopping, they skip the completion handler, and are reported unless the stop caused them.
  • The completion handler gets the cursor of the last block the handler accepted; Sinker.LastReceivedCursor gives the stream's.
  • The CLI waits up to 25s for the final flush after a stop, and returns a failure that happens while finishing.

Added TestSinker_CompletionFlush, TestUndoAfterRestart_Postgres (needs PG_DSN), and sink/sinker_stream_test.go, which runs the sinker against a fake stream: completion with an undo buffer, a handler failing while stopping, a handler returning io.EOF. All fail on the base and pass with this change. The existing Postgres integration suites pass.

@sduchesneau
sduchesneau requested review from maoueh and removed request for UlysseCorbeil October 23, 2026 19:23
@aggris2
aggris2 force-pushed the fix/sql-sink-stop-and-handler-errors branch from c433edb to 072ed60 Compare October 23, 2026 20:46
@maoueh
maoueh force-pushed the fix/sql-sink-stop-and-handler-errors branch 3 times, most recently from e6d6e76 to 3c45683 Compare October 26, 2026 17:36
…server

Sink library: Sinker.AwaitRun(ctx) waits for Run to return and gives its
first failure. A stop terminates the sinker right away while Run still
finishes its completion handler, so its failure cannot be the termination
error anymore. The SQL changes sinker now relies on it instead of tracking
its own run, and the CLI waits up to 45s for the final flush on a stop.

A handler failure while stopping is taken as caused by the stop: a
cancelled database call does not always wrap context.Canceled. A stop
that comes before the sinkers wired their termination callbacks now still
cancels the stream.

The fake Substreams stream server and its response builders move to a
new sink/sinktest package, used by the sink library tests and the SQL
integration suite. The restart/undo test moves to that suite, against a
Postgres testcontainer, with a case for the undo buffer cursor.
@maoueh
maoueh force-pushed the fix/sql-sink-stop-and-handler-errors branch from 3c45683 to 6393b7f Compare October 29, 2026 13:31
@maoueh
maoueh merged commit 033dcc4 into streamingfast:develop Oct 29, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants