Repository navigation
fix: Time out the ZooKeeper calls in the znode controller - #1109
Merged
Merged
Conversation
14 tasks done
siegfriedweber
marked this pull request as ready for review
November 18, 2026 13:59
adwk67
previously approved these changes
Nov 19, 2026
adwk67
left a comment
Member
There was a problem hiding this comment.
LGTM. Just two non-blocking comments.
Techassi
previously approved these changes
Nov 19, 2026
adwk67
approved these changes
Nov 19, 2026
siegfriedweber
enabled auto-merge
November 19, 2026 08:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of stackabletech/issues#917
The znode cleanup talks to ZooKeeper, and none of those calls had a time limit. If the ensemble was already gone, the cleanup did not fail — it just waited, and the
zookeeper.stackable.tech/znodefinalizer held up the whole Namespace while it did, with nothing in the log to say why.There were three places it could hang:
The third one is the worst: deleting a znode walks its subtree one request at a time, so the Pods can go away halfway through. One Namespace deletion was stuck like that for 957 seconds.
All three now give up after 10 seconds and the reconcile is retried. By the time it runs again the
ZookeeperClusteris usually gone too, so the cleanup is skipped through theZkDoesNotExistpath that already exists. If the cluster is still there, the retry just picks up where it left off — deleting the subtree is resumable.This applies to creating a znode as well, which had the same problem. So znode creation can now fail with a timeout error where it used to block. It is retried, and creating a znode that already exists is handled already, so retrying is safe.
Why request timeouts wait longer before retrying
Giving up on a request does not close the connection. Closing it properly needs an answer from the ensemble that just stopped answering, so the connection lingers — until the ensemble gets round to the close, or, if the ensemble is gone for good, until the OS gives up on it. With the default
tcp_retries2that last case should take something like 15 minutes, though that part has not been measured.Retrying every 5 seconds could therefore stack these up on an ensemble that is already struggling, so a request timeout waits 60 seconds instead. How much this matters in practice is not clear-cut: an ensemble that is gone leaves only client-side sockets behind, which is cheap, while one that is alive but too slow to process the close might accumulate sessions against its per-client connection limit (
maxClientCnxns, 60 by default). The 60 seconds is a cautious guess rather than a tuned value, and a different number would be just as defensible.Connect and lookup timeouts keep the usual 5 seconds. They usually have nothing to leave behind, since a connect that times out has normally not got as far as opening a session, and slowing them down would add a minute or so to the Namespace deletion below — the opposite of the point.
Release notes
Platform fixes
Apache ZooKeeper
Deleting a ZookeeperZnode, or a whole Namespace containing one, is no longer held up for minutes when the ZooKeeper ensemble has become unreachable. The operator's ZooKeeper calls now time out after 10 seconds and are retried, rather than waiting for the operating system to notice that the connection is dead.
Definition of Done Checklist
Author
Reviewer
addedalso passed on OpenshiftAcceptance