Skip to content

[release] fix(lib-jedis-pool): find the cluster primary by probing, not from CLUSTER SHARDS (1.3.1) - #150

Merged
jonmarti merged 2 commits into
masterfrom
fix/jedis-pool-probe-discovery
Oct 16, 2026
Merged

jonmarti merged 2 commits into
masterfrom
fix/jedis-pool-probe-discovery

Conversation

@pditommaso

Copy link
Copy Markdown
Contributor

Summary

An alternative to #149 for #148. Like #149, it keeps redis.mode=cluster-primary connected through a MemoryDB failover. Instead of teaching discovery to read MemoryDB's CLUSTER SHARDS reply, it stops reading the topology altogether.

How discovery works now: each address of the discovery endpoint (clustercfg.…) is probed with the key already used to validate pooled connections (jedis-pool:primary-probe):

  • the primary answers, and is the one used;
  • a replica answers MOVED to its primary, which is confirmed with the same probe;
  • a node that is unreachable or answers CLUSTERDOWN is skipped, and the next address is tried.

Why this instead of #149

Replaying the stage failover

JedisConnectorMemoryDbFailoverTest replays the stage timeline against two fake nodes (a minimal RESP server, no Docker): the old primary goes down and the promoted node's CLUSTER SHARDS lists it as an extra shard. Since the real entry is unknown, each plausible shape is a variant. The table is the result of running the same scenario against each version; "stuck" means no successful write until the reply is single-shard again.

Second entry in the promoted node's reply 1.3.0 #149 this PR
failed primary, no slots, fail stuck (found 2 shards) recovers recovers
failed primary, no slots, still online stuck recovers recovers
failed primary, still owning the slots stuck stuck recovers
promoted node answering CLUSTERDOWN to keyed commands stuck stuck stuck
  • On 1.3.0 the replay fails with the same found 2 shards error as stage, so it reproduces the incident.
  • No client can handle the last row: if MemoryDB's promoted node isn't serving yet, nothing is. The stage capture below rules it in or out.

Tests

  • ./gradlew :lib-jedis-pool:test: 39 of 39 pass, including the existing Testcontainers failover (Redis 7.2) and the hardened-user tests (NOPERM on the probe key is still reported).
  • Run locally, not committed: the same failover scenarios against real redis:7.2 and valkey:8.0 containers (graceful, paused + takeover, stopped + takeover, old primary rejoining, 20 concurrent threads). This PR passes all of them on both servers: 11–12 ms from promotion to the first write, about 1 s when the dead node hangs (one connect timeout), and no failed write while the old primary rejoins. 1.3.0 and [release] fix(lib-jedis-pool): stay connected through a MemoryDB failover that lists the failed primary as an extra shard (1.3.1) #149 also pass these, because neither server produces MemoryDB's extra shard.

To verify on stage

  • TLS: the port a replica reports in MOVED must be the one the client connects on. With TLS, Redis reports the TLS port to TLS clients, but this needs checking on MemoryDB.
  • The extra shard: repeat the forced failover with a capture of CLUSTER SHARDS and the probe on each node every 0.5 s, to see the real reply and confirm the promoted node serves keyed commands.

Trade-off

The check that refused a multi-shard cluster ran on every rediscovery; it is gone. A multi-shard cluster is now caught only by its symptoms: keys outside the primary's shard fail with MOVED. The README already documents single-shard as a constraint. If we want the guard back, it fits as a one-time startup check, outside the recovery path.

Fixes #148. Alternative to #149: merge one of the two.

🤖 Generated with Claude Code

…USTER SHARDS (1.3.1)

In cluster-primary mode, discovery rejected the promoted primary's
CLUSTER SHARDS reply after a MemoryDB failover, because it listed the
failed primary as an extra shard, so sched stayed disconnected for
about 4 minutes (#148).

Discovery no longer reads the cluster topology. It probes each address
the discovery endpoint resolves to with the probe key already used to
validate pooled connections: the primary answers, a replica redirects
with MOVED to its primary, which is confirmed with the same probe. A
node answering CLUSTERDOWN is skipped rather than ending discovery for
that address. The Redis user no longer needs @slow.

Alternative to #149.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

@jonmarti jonmarti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pditommaso, approving: this is the better fix for #148, and I'll close #149 in its favour. Not reading the topology removes the dependency on MemoryDB's CLUSTER SHARDS reply, which we never captured, and the replay that reproduces the stage failure on 1.3.0 is a stronger test than the unit tests in #149. Three non-blocking notes inline. On the table: the third row (both entries owning every slot) can't come from open-source Redis, where each slot has a single owner in a node's view, but we can't rule it out for MemoryDB, which is the case for not reading it at all.

Comment thread lib-jedis-pool/README.md Outdated
Comment thread lib-jedis-pool/src/main/java/io/seqera/jedis/JedisConnector.java Outdated
Comment thread lib-jedis-pool/src/main/java/io/seqera/jedis/JedisConnector.java Outdated
…inds no primary

The discovery error named only the endpoint, so the reason each address
was rejected (unreachable, CLUSTERDOWN, a MOVED to a node that is not
usable) showed only at DEBUG. List them in the message, e.g.
"10.0.0.1:6379: Failed to connect ...; 10.0.0.2:6379: CLUSTERDOWN Hash
slot not served". The cause is still the last exception, so a NOPERM on
the probe key is reported as before.

Document in the README that a multi-shard cluster is not detected and
how it shows up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jon Martí <jonathan.marti@seqera.io>
@jonmarti
jonmarti merged commit 61bae57 into master Oct 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lib-jedis-pool: cluster-primary mode stays disconnected for ~4 min after a MemoryDB failover ("found 2 shards")

2 participants