Skip to content

[release] fix(lib-jedis-pool): stay connected through a MemoryDB failover that lists the failed primary as an extra shard (1.3.1) - #149

Closed
jonmarti wants to merge 1 commit into
masterfrom
fix/COMP-2823-jedis-pool-failover-extra-shard
Closed

jonmarti wants to merge 1 commit into
masterfrom
fix/COMP-2823-jedis-pool-failover-extra-shard

Conversation

@jonmarti

@jonmarti jonmarti commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

With redis.mode=cluster-primary, a service stays connected to Redis through a MemoryDB failover. It reconnects as soon as MemoryDB has promoted the replica, instead of minutes later. Fixes #148, COMP-2823.

In a forced failover of sched's stage MemoryDB cluster, MemoryDB had a new primary after 40 s, but sched stayed disconnected for about 4 more minutes. Primary discovery rejected the promoted primary's CLUSTER SHARDS reply because it listed 2 shards: the failed primary appears as a shard of its own, with no slots, while MemoryDB recovers it. During that time every Redis-backed path in sched failed, and its API answered 401 to every authenticated call.

Change

  • Multi-shard check: only shards that own slots count. A reply where two or more shards own slots is still refused.
  • Candidates: the online primaries of every entry, those of the shard that owns the slots first, each confirmed with the existing probe.
  • Diagnostics: discovery errors carry a one-line summary of the reply, and the "Redis cluster primary is …" log line includes it when the reply lists more than one shard.

Why not simply drop the entries that own no slots: Redis 7.2 also reports the shard with no slots right after a takeover. Dropping them makes the existing failover integration test (should follow the primary across a failover) fail.

Tests

  • New unit tests:
    • the MemoryDB reply, with the failed primary reported as failed and as still online;
    • the Redis 7.2 reply (one shard listed with no slots);
    • slots split across shards (still refused, with the reply in the error);
    • no online primary (the error includes the reply).
  • The MemoryDB test fails on the previous code with requires a single-shard cluster - found 2 shards. The Redis 7.2 test and the failover integration test fail if slotless entries are dropped.
  • ./gradlew :lib-jedis-pool:test: 43 of 43 pass.

Not included

  • No integration test for MemoryDB's exact reply. Open-source engines don't produce it: Redis 7.2, Valkey 7.2 and Valkey 8.0 all report one shard during a takeover. The unit test reproduces the shape inferred from sched's logs, which only recorded the count.
  • Verification on stage: after release, sched bumps lib-jedis-pool and the forced failover on stage is repeated. The new log line will show MemoryDB's actual reply.
  • The 401 sched returns while Redis is unreachable: a separate sched issue.

🤖 Generated with Claude Code

…lists the failed primary as an extra shard (1.3.1)

In cluster-primary mode, primary discovery rejected any CLUSTER SHARDS
reply that was not exactly one shard. After a MemoryDB failover the
promoted primary lists the failed one as a shard of its own, with no
slots, until the node is recovered, so sched stayed disconnected for
about 4 minutes after MemoryDB had finished failing over.

Count only the shards that own slots: more than one is still refused as
a multi-shard cluster. The primaries of every entry remain candidates,
those of the shard that owns the slots first, and the existing probe
tells the real primary apart. Keeping the slotless entries matters:
Redis 7.2 reports the shard with no slots right after a takeover, and
the failover integration test fails if they are dropped.

Discovery errors now carry a summary of the reply, and the discovery
log line includes it when the reply lists more than one shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jon Martí <jonathan.marti@seqera.io>
@pditommaso

Copy link
Copy Markdown
Contributor

An alternative to this PR is open: #150. Instead of reading MemoryDB's CLUSTER SHARDS reply, it finds the primary by probing each node of the discovery endpoint and following a replica's MOVED.

  • Works whatever the extra shard holds: in a replay of the stage failover, this PR stays stuck when the failed primary is still listed as owning the slots. #150 recovers in that case too.
  • CLUSTERDOWN is skipped: confirmPrimary here only catches JedisConnectionException. So when the old primary returned and answered CLUSTERDOWN (stage, 12:51:30), discovery stopped trying the rest of that node's candidates.

Both PRs bump to 1.3.1, so only one should be merged. The comparison table is in the #150 description.

@jonmarti

Copy link
Copy Markdown
Contributor Author

@pditommaso, closing in favour of #150: finding the primary by probing doesn't depend on MemoryDB's CLUSTER SHARDS reply, which we never captured.

@jonmarti jonmarti closed this Oct 16, 2026
jonmarti added a commit that referenced this pull request Oct 16, 2026
…ot from CLUSTER SHARDS (1.3.1) (#150)

* fix(lib-jedis-pool): find the cluster primary by probing, not from CLUSTER SHARDS (1.3.1)

In cluster-primary mode, discovery rejected the promoted primary's
CLUSTER SHARDS reply after a MemoryDB failover, because it listed the
failed primary as an extra shard, so sched stayed disconnected for
about 4 minutes (#148).

Discovery no longer reads the cluster topology. It probes each address
the discovery endpoint resolves to with the probe key already used to
validate pooled connections: the primary answers, a replica redirects
with MOVED to its primary, which is confirmed with the same probe. A
node answering CLUSTERDOWN is skipped rather than ending discovery for
that address. The Redis user no longer needs @slow.

Alternative to #149.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* fix(lib-jedis-pool): list what each address answered when discovery finds no primary

The discovery error named only the endpoint, so the reason each address
was rejected (unreachable, CLUSTERDOWN, a MOVED to a node that is not
usable) showed only at DEBUG. List them in the message, e.g.
"10.0.0.1:6379: Failed to connect ...; 10.0.0.2:6379: CLUSTERDOWN Hash
slot not served". The cause is still the last exception, so a NOPERM on
the probe key is reported as before.

Document in the README that a multi-shard cluster is not detected and
how it shows up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jon Martí <jonathan.marti@seqera.io>

---------

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Jon Martí <jonathan.marti@seqera.io>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Jon Martí <jonathan.marti@seqera.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lib-jedis-pool: cluster-primary mode stays disconnected for ~4 min after a MemoryDB failover ("found 2 shards")

2 participants