[aggris2]

Builds on #1007, #1011 and #1012. Until those are merged, this PR also shows their commits.

Today only one sink can safely write to a database, so running the sink on more than one machine needs something outside it to make sure only one runs (start/stop scripts tied to the database failover, for example). This adds a standby mode to the SQL sink (PostgreSQL): several sinks run against one database, one writes, and the others take over by themselves.

--standby-lock: before streaming, the sink takes a session advisory lock keyed on the cursors table (one writer per cursors table, whatever the module). It reads its cursor and does all its writes through that same session, so losing the session loses the lock and the ability to write together. Other sinks wait as standbys, without opening a Substreams stream, checking every --standby-lock-poll-interval (default 2s). The first to see the lock released takes over from the stored cursor.

• A sink whose locked session is gone stops instead of retrying its flush.
• A connection whose lock state is uncertain (an error while locking or unlocking) is closed rather than returned to the pool.
• Standbys serve the --prometheus-addr endpoint while they wait; substreams_sink_postgres_writer_lock_held is 1 on the active sink.

--stall-timeout: stops the sink when the stream delivers no block or undo for that long once live, so a stuck sink can't keep the lock. Blocks held by an undo buffer count (new Sinker.LastBlockMessageAt). On PostgreSQL a flush now honours its context, so the stop cancels a database call in progress and its transaction rolls back. ClickHouse flushes still run to the end, since its writes aren't transactional.

The cursor check from #1012 keeps any other writer from corrupting data. The new docs/references/sql/standby-sinks.md covers the setup: server TCP keepalives so a dead client's session (and lock) is released, session connections (not a transaction-mode pooler), failover relying on the old primary being fenced, stalls and network cuts.

Tests, with real Postgres (PG_DSN):

• TestWriterLock_Postgres: a standby waits, takes over when the active sink's session is terminated, the old writer can't write anymore, a sink of another module waits too, a standby stops when told to, closing releases the lock;
• TestDiscardEndsLockedSession_Postgres and TestIsCancellation;
• TestSinker_StallTimeout, and a cursor conflict while stopping is still reported.

End to end, over 20k mainnet blocks with a flush per block:

• two supervised sinks with --standby-lock, killing the active one, cutting its database session, then killing the standby;
• a primary and an async replica behind a proxy, failing over to the replica.

Final tables were identical to a clean run in every case, and the standby took over within about a second.

[sduchesneau]

You should probably ping the connection every standby-lock-poll-interval so that, when processing large segments (initial loading) or on long stretches that yield no result (slow chain, sparse content...), the active connection doesn't get killed by a proxy or idle_session_timeout. -- this would cause some failover pingpong between instances.

it's the opposite of the idea of stall-timeout, it's when you want to support sparse data and prevent "spurious failovers".

[aggris2]

@sduchesneau Good call, thanks. Done in c2aaffa: the active sink pings its locked session every --standby-lock-poll-interval, so idle stretches don't end it. c0dad19 fixes a race the ping introduced with the startup cursor read.

[sduchesneau]

@aggris2 all good, please resolve conflicts and push, I will approve/merge it then

[aggris2]

@sduchesneau conflicts resolved

[sduchesneau]

LGTM