Skip to content

[release] fix(lib-pairing): close pairing sessions that stop sending heartbeats (1.2.0) - #154

Merged
pditommaso merged 3 commits into
masterfrom
fix/pairing-heartbeat-deadline
Oct 15, 2026
Merged

pditommaso merged 3 commits into
masterfrom
fix/pairing-heartbeat-deadline

Conversation

@pditommaso

@pditommaso pditommaso commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A pairing connection that drops without a WebSocket close frame (a network or load balancer drop) leaves a half-open session registered on the server. Writes to it still succeed locally (they go into the kernel send buffer), so the session keeps taking messages from the endpoint's shared outbound queue and they are silently lost until TCP gives up, which can take minutes. The caller's inbound future times out (Unable to retrieve a value for key: pairing-inbound-queue/v1:<msgId>) and TowerConnector retries, sometimes against the same dead session.

Neither side checks that heartbeats come back: Platform sends PairingHeartbeat every 50s but only logs the reply, and the server answers but never closes a session that stopped sending them. So detection depends on TCP or the server-wide idle-timeout (910s in Wave).

Evidence (Wave prod, 14 Oct, Seqera Cloud)

Platform WavePairingClient Wave PairingWebSocket close Window as a zombie
11:37:38 1006 Abnormal Closure 11:39:53 2m15s
11:44:27 1006 11:46:41 2m14s
11:50:09 1006 11:51:17 1m08s

Platform reconnected within ~5s each time, while the dead session stayed registered and took work: 456 failed lookups to api.cloud.seqera.io that hour.

Change

  • Record the last heartbeat time of each session (tracked from its first heartbeat).
  • A @Scheduled sweep every 10s handles sessions silent for longer than the wave.pairing.heartbeat-timeout property (default 120s, about 2.4 missed beats). It unregisters each from the channel right away, so it stops taking messages even if @OnClose is slow for a half-open socket, then closes it with GOING_AWAY so the client reconnects.
  • If a newer heartbeat arrives during the sweep, that session is left alone. @OnClose removes the entry.
  • Sessions that never sent a heartbeat are not affected, so older clients without heartbeats keep working.
  • PairingConfig is unchanged, so existing implementations (Groovy or Java) keep compiling. Zero or negative disables the check.

Limits

This bounds the worst case: a dead session is removed about 70–130s after the drop, instead of whenever TCP or the 910s idle timeout notices. It doesn't shorten windows TCP already catches within a minute or two (as on 14 Oct). For that, Platform would need a shorter heartbeat interval, or the server would need to requeue a message whose reply times out.

Tests

  • New should close sessions with no heartbeat within the heartbeat timeout: a session that sent a heartbeat is untouched before the timeout, unregistered and closed once after it, and a session that never sent one is never closed.
  • Compiled locally (:lib-pairing:compileTestGroovy); tests run in CI.

Bumps lib-pairing to 1.2.0 (VERSION, changelog, README).

🤖 Generated with Claude Code

… (1.2.0)

A pairing connection dropped without a close frame (e.g. a network or
load balancer drop) leaves a half-open session registered on the server.
Writes to it still succeed locally, so it keeps taking messages from the
shared outbound queue and they are silently lost until TCP gives up,
which can take minutes. Callers see "Unable to retrieve a value for key:
pairing-inbound-queue" and retry against the same dead session.

Track the last heartbeat of each session and, every 10s, unregister and
close (GOING_AWAY) the sessions silent for longer than
PairingConfig#getHeartbeatTimeout (default 120s, clients beat every 50s).
Sessions that never sent a heartbeat are not affected. The new config
method has a default, so existing PairingConfig implementations compile
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread lib-pairing/src/main/groovy/io/seqera/service/pairing/PairingConfig.groovy Outdated
pditommaso and others added 2 commits October 15, 2026 12:36
…ession sweep

Drop the assertion on a session that never enters the heartbeat map (it
could not fail) and test that a session whose heartbeat is refreshed
after the sweep picked it up is not closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…meout from a property

A Groovy 4 interface default method compiles as a trait (abstract method),
so Java PairingConfig implementations would no longer compile. Read the
timeout from the optional wave.pairing.heartbeat-timeout property
(default 120s) instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pditommaso
pditommaso merged commit 22fe794 into master Oct 15, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants