Repository navigation
[release] fix(lib-jedis-pool): stay connected through a MemoryDB failover that lists the failed primary as an extra shard (1.3.1) - #149
Closed
jonmarti wants to merge 1 commit into
Conversation
…lists the failed primary as an extra shard (1.3.1) In cluster-primary mode, primary discovery rejected any CLUSTER SHARDS reply that was not exactly one shard. After a MemoryDB failover the promoted primary lists the failed one as a shard of its own, with no slots, until the node is recovered, so sched stayed disconnected for about 4 minutes after MemoryDB had finished failing over. Count only the shards that own slots: more than one is still refused as a multi-shard cluster. The primaries of every entry remain candidates, those of the shard that owns the slots first, and the existing probe tells the real primary apart. Keeping the slotless entries matters: Redis 7.2 reports the shard with no slots right after a takeover, and the failover integration test fails if they are dropped. Discovery errors now carry a summary of the reply, and the discovery log line includes it when the reply lists more than one shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jon Martí <jonathan.marti@seqera.io>
Contributor
|
An alternative to this PR is open: #150. Instead of reading MemoryDB's
Both PRs bump to 1.3.1, so only one should be merged. The comparison table is in the #150 description. |
Contributor
Author
|
@pditommaso, closing in favour of #150: finding the primary by probing doesn't depend on MemoryDB's |
jonmarti
added a commit
that referenced
this pull request
Oct 16, 2026
…ot from CLUSTER SHARDS (1.3.1) (#150) * fix(lib-jedis-pool): find the cluster primary by probing, not from CLUSTER SHARDS (1.3.1) In cluster-primary mode, discovery rejected the promoted primary's CLUSTER SHARDS reply after a MemoryDB failover, because it listed the failed primary as an extra shard, so sched stayed disconnected for about 4 minutes (#148). Discovery no longer reads the cluster topology. It probes each address the discovery endpoint resolves to with the probe key already used to validate pooled connections: the primary answers, a replica redirects with MOVED to its primary, which is confirmed with the same probe. A node answering CLUSTERDOWN is skipped rather than ending discovery for that address. The Redis user no longer needs @slow. Alternative to #149. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * fix(lib-jedis-pool): list what each address answered when discovery finds no primary The discovery error named only the endpoint, so the reason each address was rejected (unreachable, CLUSTERDOWN, a MOVED to a node that is not usable) showed only at DEBUG. List them in the message, e.g. "10.0.0.1:6379: Failed to connect ...; 10.0.0.2:6379: CLUSTERDOWN Hash slot not served". The cause is still the last exception, so a NOPERM on the probe key is reported as before. Document in the README that a multi-shard cluster is not detected and how it shows up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jon Martí <jonathan.marti@seqera.io> --------- Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Signed-off-by: Jon Martí <jonathan.marti@seqera.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Jon Martí <jonathan.marti@seqera.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With
redis.mode=cluster-primary, a service stays connected to Redis through a MemoryDB failover. It reconnects as soon as MemoryDB has promoted the replica, instead of minutes later. Fixes #148, COMP-2823.In a forced failover of sched's stage MemoryDB cluster, MemoryDB had a new primary after 40 s, but sched stayed disconnected for about 4 more minutes. Primary discovery rejected the promoted primary's
CLUSTER SHARDSreply because it listed 2 shards: the failed primary appears as a shard of its own, with no slots, while MemoryDB recovers it. During that time every Redis-backed path in sched failed, and its API answered 401 to every authenticated call.Change
Why not simply drop the entries that own no slots: Redis 7.2 also reports the shard with no slots right after a takeover. Dropping them makes the existing failover integration test (
should follow the primary across a failover) fail.Tests
requires a single-shard cluster - found 2 shards. The Redis 7.2 test and the failover integration test fail if slotless entries are dropped../gradlew :lib-jedis-pool:test: 43 of 43 pass.Not included
🤖 Generated with Claude Code