Skip to content

fix: Time out the ZooKeeper calls in the znode controller - #1109

Merged
siegfriedweber merged 7 commits into
mainfrom
fix/znode-connect-timeout
Nov 19, 2026
Merged

siegfriedweber merged 7 commits into
mainfrom
fix/znode-connect-timeout

Conversation

@siegfriedweber

@siegfriedweber siegfriedweber commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Description

Part of stackabletech/issues#917

The znode cleanup talks to ZooKeeper, and none of those calls had a time limit. If the ensemble was already gone, the cleanup did not fail — it just waited, and the zookeeper.stackable.tech/znode finalizer held up the whole Namespace while it did, with nothing in the log to say why.

There were three places it could hang:

Step Why it hangs For how long
Looking up the ensemble's address DNS lookups have no time limit of their own, so an unreachable CoreDNS stalls them seconds to minutes
Connecting Once the ensemble's Service is gone, the connection attempt is silently dropped instead of refused, so the OS keeps retrying over 2 minutes
Each request The ensemble's Pods disappear without closing the connection, so the reply never arrives and the OS only notices much later ~15 minutes

The third one is the worst: deleting a znode walks its subtree one request at a time, so the Pods can go away halfway through. One Namespace deletion was stuck like that for 957 seconds.

All three now give up after 10 seconds and the reconcile is retried. By the time it runs again the ZookeeperCluster is usually gone too, so the cleanup is skipped through the ZkDoesNotExist path that already exists. If the cluster is still there, the retry just picks up where it left off — deleting the subtree is resumable.

This applies to creating a znode as well, which had the same problem. So znode creation can now fail with a timeout error where it used to block. It is retried, and creating a znode that already exists is handled already, so retrying is safe.

Why request timeouts wait longer before retrying

Giving up on a request does not close the connection. Closing it properly needs an answer from the ensemble that just stopped answering, so the connection lingers — until the ensemble gets round to the close, or, if the ensemble is gone for good, until the OS gives up on it. With the default tcp_retries2 that last case should take something like 15 minutes, though that part has not been measured.

Retrying every 5 seconds could therefore stack these up on an ensemble that is already struggling, so a request timeout waits 60 seconds instead. How much this matters in practice is not clear-cut: an ensemble that is gone leaves only client-side sockets behind, which is cheap, while one that is alive but too slow to process the close might accumulate sessions against its per-client connection limit (maxClientCnxns, 60 by default). The 60 seconds is a cautious guess rather than a tuned value, and a different number would be just as defensible.

Connect and lookup timeouts keep the usual 5 seconds. They usually have nothing to leave behind, since a connect that times out has normally not got as far as opening a session, and slowing them down would add a minute or so to the Namespace deletion below — the opposite of the point.

Release notes

Platform fixes

Apache ZooKeeper

Deleting a ZookeeperZnode, or a whole Namespace containing one, is no longer held up for minutes when the ZooKeeper ensemble has become unreachable. The operator's ZooKeeper calls now time out after 10 seconds and are retried, rather than waiting for the operating system to notice that the connection is dead.

Definition of Done Checklist

Author

  • Changes are OpenShift compatible
  • Integration tests passed locally
    PASS: kuttl/harness/delete-rolegroup_zookeeper-3.9.6_openshift-false (123.35s)
    PASS: kuttl/harness/cluster-operation_zookeeper-latest-3.9.6_openshift-false (108.46s)
    PASS: kuttl/harness/logging_zookeeper-3.9.6_openshift-false (108.73s)
    PASS: kuttl/harness/smoke_zookeeper-3.9.6_use-server-tls-true_use-client-auth-tls-true_openshift-false (131.88s)
    PASS: kuttl/harness/znode_zookeeper-latest-3.9.6_openshift-false (41.49s)
    
  • Changes need to be "offline" compatible
  • Release note snippet added to this description

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added also passed on Openshift
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Proper release label has been added
  • Release note snippet added

@siegfriedweber siegfriedweber changed the title fix: Time out the ZooKeeper connect in the znode controller fix: Time out the ZooKeeper calls in the znode controller Nov 18, 2026
@siegfriedweber
siegfriedweber marked this pull request as ready for review November 18, 2026 13:59
@siegfriedweber siegfriedweber added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Nov 18, 2026
@siegfriedweber
siegfriedweber requested a review from a team November 18, 2026 13:59
@siegfriedweber siegfriedweber moved this to Development: Waiting for Review in Stackable Engineering Nov 18, 2026
@adwk67 adwk67 moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Nov 19, 2026
@adwk67
adwk67 requested review from adwk67 and removed request for a team November 19, 2026 08:05
adwk67
adwk67 previously approved these changes Nov 19, 2026

@adwk67 adwk67 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just two non-blocking comments.

Comment thread CHANGELOG.md Resolved
Techassi
Techassi previously approved these changes Nov 19, 2026
@siegfriedweber
siegfriedweber dismissed stale reviews from Techassi and adwk67 via a373d00 November 19, 2026 08:57
@siegfriedweber
siegfriedweber added this pull request to the merge queue Nov 19, 2026
@siegfriedweber siegfriedweber moved this from Development: In Review to Development: Done in Stackable Engineering Nov 19, 2026
Merged via the queue into main with commit 86a29a5 Nov 19, 2026
13 checks passed
@siegfriedweber
siegfriedweber deleted the fix/znode-connect-timeout branch November 19, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release/26.11.0 release-note Denotes a PR that will be considered when it comes time to generate release notes.

Projects

Development

Successfully merging this pull request may close these issues.

4 participants