Repository navigation
[release] fix(lib-pairing): close pairing sessions that stop sending heartbeats (1.2.0) - #154
Merged
Merged
Conversation
… (1.2.0) A pairing connection dropped without a close frame (e.g. a network or load balancer drop) leaves a half-open session registered on the server. Writes to it still succeed locally, so it keeps taking messages from the shared outbound queue and they are silently lost until TCP gives up, which can take minutes. Callers see "Unable to retrieve a value for key: pairing-inbound-queue" and retry against the same dead session. Track the last heartbeat of each session and, every 10s, unregister and close (GOING_AWAY) the sessions silent for longer than PairingConfig#getHeartbeatTimeout (default 120s, clients beat every 50s). Sessions that never sent a heartbeat are not affected. The new config method has a default, so existing PairingConfig implementations compile unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 of 11 tasks
jordeu
approved these changes
Oct 15, 2026
…ession sweep Drop the assertion on a session that never enters the heartbeat map (it could not fail) and test that a session whose heartbeat is refreshed after the sweep picked it up is not closed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…meout from a property A Groovy 4 interface default method compiles as a trait (abstract method), so Java PairingConfig implementations would no longer compile. Read the timeout from the optional wave.pairing.heartbeat-timeout property (default 120s) instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A pairing connection that drops without a WebSocket close frame (a network or load balancer drop) leaves a half-open session registered on the server. Writes to it still succeed locally (they go into the kernel send buffer), so the session keeps taking messages from the endpoint's shared outbound queue and they are silently lost until TCP gives up, which can take minutes. The caller's inbound future times out (
Unable to retrieve a value for key: pairing-inbound-queue/v1:<msgId>) andTowerConnectorretries, sometimes against the same dead session.Neither side checks that heartbeats come back: Platform sends
PairingHeartbeatevery 50s but only logs the reply, and the server answers but never closes a session that stopped sending them. So detection depends on TCP or the server-wideidle-timeout(910s in Wave).Evidence (Wave prod, 14 Oct, Seqera Cloud)
WavePairingClientPairingWebSocketclose1006 Abnormal Closure10061006Platform reconnected within ~5s each time, while the dead session stayed registered and took work: 456 failed lookups to
api.cloud.seqera.iothat hour.Change
@Scheduledsweep every 10s handles sessions silent for longer than thewave.pairing.heartbeat-timeoutproperty (default 120s, about 2.4 missed beats). It unregisters each from the channel right away, so it stops taking messages even if@OnCloseis slow for a half-open socket, then closes it withGOING_AWAYso the client reconnects.@OnCloseremoves the entry.PairingConfigis unchanged, so existing implementations (Groovy or Java) keep compiling. Zero or negative disables the check.Limits
This bounds the worst case: a dead session is removed about 70–130s after the drop, instead of whenever TCP or the 910s idle timeout notices. It doesn't shorten windows TCP already catches within a minute or two (as on 14 Oct). For that, Platform would need a shorter heartbeat interval, or the server would need to requeue a message whose reply times out.
Tests
should close sessions with no heartbeat within the heartbeat timeout: a session that sent a heartbeat is untouched before the timeout, unregistered and closed once after it, and a session that never sent one is never closed.:lib-pairing:compileTestGroovy); tests run in CI.Bumps
lib-pairingto1.2.0(VERSION, changelog, README).🤖 Generated with Claude Code