Repository navigation
[release] fix(lib-jedis-pool): find the cluster primary by probing, not from CLUSTER SHARDS (1.3.1) - #150
Merged
Conversation
…USTER SHARDS (1.3.1) In cluster-primary mode, discovery rejected the promoted primary's CLUSTER SHARDS reply after a MemoryDB failover, because it listed the failed primary as an extra shard, so sched stayed disconnected for about 4 minutes (#148). Discovery no longer reads the cluster topology. It probes each address the discovery endpoint resolves to with the probe key already used to validate pooled connections: the primary answers, a replica redirects with MOVED to its primary, which is confirmed with the same probe. A node answering CLUSTERDOWN is skipped rather than ending discovery for that address. The Redis user no longer needs @slow. Alternative to #149. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
jonmarti
approved these changes
Oct 16, 2026
jonmarti
left a comment
Contributor
There was a problem hiding this comment.
@pditommaso, approving: this is the better fix for #148, and I'll close #149 in its favour. Not reading the topology removes the dependency on MemoryDB's CLUSTER SHARDS reply, which we never captured, and the replay that reproduces the stage failure on 1.3.0 is a stronger test than the unit tests in #149. Three non-blocking notes inline. On the table: the third row (both entries owning every slot) can't come from open-source Redis, where each slot has a single owner in a node's view, but we can't rule it out for MemoryDB, which is the case for not reading it at all.
…inds no primary The discovery error named only the endpoint, so the reason each address was rejected (unreachable, CLUSTERDOWN, a MOVED to a node that is not usable) showed only at DEBUG. List them in the message, e.g. "10.0.0.1:6379: Failed to connect ...; 10.0.0.2:6379: CLUSTERDOWN Hash slot not served". The cause is still the last exception, so a NOPERM on the probe key is reported as before. Document in the README that a multi-shard cluster is not detected and how it shows up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jon Martí <jonathan.marti@seqera.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An alternative to #149 for #148. Like #149, it keeps
redis.mode=cluster-primaryconnected through a MemoryDB failover. Instead of teaching discovery to read MemoryDB'sCLUSTER SHARDSreply, it stops reading the topology altogether.How discovery works now: each address of the discovery endpoint (
clustercfg.…) is probed with the key already used to validate pooled connections (jedis-pool:primary-probe):MOVEDto its primary, which is confirmed with the same probe;CLUSTERDOWNis skipped, and the next address is tried.Why this instead of #149
masternodes in one shard, and [release] fix(lib-jedis-pool): stay connected through a MemoryDB failover that lists the failed primary as an extra shard (1.3.1) #149 adds slotless entries. Here there's nothing to special-case.CLUSTERDOWNno longer ends discovery for an address. On stage the returning old primary answeredCLUSTERDOWN(12:51:30). In 1.3.0 and [release] fix(lib-jedis-pool): stay connected through a MemoryDB failover that lists the failed primary as an extra shard (1.3.1) #149 that error stops probing the remaining candidates from that address.@slow, only read access to the probe key.primariesOfand the TLS-port and endpoint parsing are gone (−66/+40 inJedisConnector).Replaying the stage failover
JedisConnectorMemoryDbFailoverTestreplays the stage timeline against two fake nodes (a minimal RESP server, no Docker): the old primary goes down and the promoted node'sCLUSTER SHARDSlists it as an extra shard. Since the real entry is unknown, each plausible shape is a variant. The table is the result of running the same scenario against each version; "stuck" means no successful write until the reply is single-shard again.failfound 2 shards)onlineCLUSTERDOWNto keyed commandsfound 2 shardserror as stage, so it reproduces the incident.Tests
./gradlew :lib-jedis-pool:test: 39 of 39 pass, including the existing Testcontainers failover (Redis 7.2) and the hardened-user tests (NOPERMon the probe key is still reported).redis:7.2andvalkey:8.0containers (graceful, paused + takeover, stopped + takeover, old primary rejoining, 20 concurrent threads). This PR passes all of them on both servers: 11–12 ms from promotion to the first write, about 1 s when the dead node hangs (one connect timeout), and no failed write while the old primary rejoins. 1.3.0 and [release] fix(lib-jedis-pool): stay connected through a MemoryDB failover that lists the failed primary as an extra shard (1.3.1) #149 also pass these, because neither server produces MemoryDB's extra shard.To verify on stage
MOVEDmust be the one the client connects on. With TLS, Redis reports the TLS port to TLS clients, but this needs checking on MemoryDB.CLUSTER SHARDSand the probe on each node every 0.5 s, to see the real reply and confirm the promoted node serves keyed commands.Trade-off
The check that refused a multi-shard cluster ran on every rediscovery; it is gone. A multi-shard cluster is now caught only by its symptoms: keys outside the primary's shard fail with
MOVED. The README already documents single-shard as a constraint. If we want the guard back, it fits as a one-time startup check, outside the recovery path.Fixes #148. Alternative to #149: merge one of the two.
🤖 Generated with Claude Code