fix: KRaft controller pods are now dynamically added/removed from the voter list

.gitignore

@@ -20,3 +20,5 @@ tilt_options.json
 .envrc
 
 .DS_Store
+
+.worktrees/

CHANGELOG.md

@@ -4,6 +4,12 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+### Added
+
+- KRaft controller replicas can now be scaled up and down on a running cluster: a new
+  `quorum-manager` sidecar container on each controller pod admits itself into the KRaft
+  voter set on startup and removes itself before termination ([#NNNN]).
+
 ### Changed
 
 - Internal operator refactoring: introduce a build() step in the reconciler that
@@ -16,6 +22,9 @@ All notable changes to this project will be documented in this file.
 - All product containers now run with `securityContext.runAsNonRoot` set to `true` to improve security ([#998]).
 - The reconciler now applies resources and derives the cluster status in discrete
   apply and update_status steps ([#1000]).
+- BREAKING: KRaft mode now requires Kafka 3.9.0 or later; Kafka 3.7.x is no longer supported and its previous
+  special-casing has been removed entirely, rather than narrowed. Running KRaft mode on an unsupported Kafka
+  version is undefined behavior ([#NNNN]).
 
 ### Fixed
 
@@ -28,6 +37,7 @@ All notable changes to this project will be documented in this file.
 [#994]: https://github.com/stackabletech/kafka-operator/pull/994
 [#998]: https://github.com/stackabletech/kafka-operator/pull/998
 [#1000]: https://github.com/stackabletech/kafka-operator/pull/1000
+[#NNNN]: https://github.com/stackabletech/kafka-operator/pull/NNNN
 
 ## [26.7.0] - 2026-07-21
 

docs/modules/kafka/pages/usage-guide/kraft-controller.adoc

@@ -16,6 +16,11 @@ WARNING: The Stackable Operator for Apache Kafka currently does not support auto
 * Full Replacement: Kafka 4.0.0 (2025) removes ZooKeeper completely.
 * Migration: Tools exist to migrate from ZooKeeper to KRaft, but new deployments should start with KRaft.
 
+IMPORTANT: The Stackable Operator for Apache Kafka requires Kafka 3.9.0 or later for KRaft mode. Kafka 3.7.x is not
+supported: the operator relies on the `kafka-storage.sh format --initial-controllers` option and the KIP-853 dynamic
+quorum tooling, both of which require 3.9.0+. Running KRaft mode on an unsupported Kafka version is undefined
+behavior, up to and including `kafka-storage.sh` rejecting the operator-generated formatting command outright.
+
 == Configuration
 
 The Stackable Kafka operator introduces a new xref:concepts:roles-and-role-groups.adoc[role] in the KafkaCluster CRD called KRaft `Controller`.
@@ -85,13 +90,27 @@ KRaft mode requires major configuration changes compared to ZooKeeper:
 * `cluster-id`: This is set to the `metadata.name` of the KafkaCluster resource during initial formatting
 * `node.id`: This is a calculated integer, hashed from the `role` and `rolegroup` and added `replica` id.
 * `process.roles`: Will always only be `broker` or `controller`. Mixed `broker,controller` servers are not supported.
-* The operator configures a static voter list containing the controller pods. Controllers are not dynamically managed.
+* When Kerberos is not enabled, each controller pod runs an additional `quorum-manager` sidecar container (requires
+  Kafka 3.9.0 or later, see the minimum-version note above). On startup it admits the pod into the KRaft voter set
+  (`kafka-metadata-quorum.sh add-controller`), and on pod termination (`preStop`) it removes the pod from the voter
+  set again (`remove-controller`), but only if doing so would not remove the last remaining voter.
 
 == Known Issues
 
 * Automatic migration from Apache ZooKeeper to KRaft is not supported.
-* Scaling controller replicas might lead to unstable clusters.
 * Kerberos is currently not supported for KRaft in all versions.
+* Scaling controllers down to a single replica is not verified under the sidecar-based mechanism described above;
+  only scale-downs that keep an odd number of controllers greater than one have been tested.
+* If a `remove-controller` call fails or times out during pod termination (for example, no reachable leader within
+  the pod's grace period), the pod terminates anyway and can leave a stale voter entry in the quorum behind. This is
+  not fully automatic in every case and may require manual cleanup with `kafka-metadata-quorum.sh remove-controller`.
+* When a new controller is added on scale-up, it is formatted with `kafka-storage.sh format --initial-controllers`
+  listing the *full* desired voter set, including itself, rather than following Kafka's documented procedure for
+  joining an already-formed quorum (format with no initial controllers, then explicitly `add-controller`). In manual
+  live testing this has been observed to work correctly: the new pod starts as an `observer` and the sidecar's
+  `add-controller` call succeeds. However, this has not yet been confirmed end-to-end through the fully automated
+  sidecar path on a live cluster. This is a known, tracked risk to verify before or soon after this feature ships;
+  changing the format-step behavior itself would be a separate, larger design change.
 
 == Troubleshooting
 
@@ -108,10 +127,19 @@ Likely caused by controller resource starvation or unstable Kubernetes schedulin
 Ensure Kafka version 3.9.x and higher and follow the official migration documentation.
 The Stackable Kafka operator currently does not support the migration.
 
-=== Scaling issues
+=== Scaling controllers
+
+Controller replicas can be scaled up and down on a running cluster. A per-pod `quorum-manager` sidecar admits and
+removes the pod from the KRaft voter set as described under "Internal operator details" above. This requires Kafka
+3.9.0 or later (see the minimum-version note in "Overview"), which supports the
+https://developers.redhat.com/articles/2024/11/27/dynamic-kafka-controller-quorum[KIP-853 dynamic quorum tooling].
+
+Scaling more than one controller down at a time is processed one pod at a time (`OrderedReady` pod management), not
+in parallel, so that each pod's removal from the voter set can complete before the next one is terminated.
 
-The https://developers.redhat.com/articles/2024/11/27/dynamic-kafka-controller-quorum[Dynamic scaling] is only supported from Kafka version 3.9.0.
-If you are using older versions, automatic scaling may not work properly (e.g. adding or removing controller replicas).
+This has been validated on a live cluster and is covered by unit tests, but a full end-to-end scale-up/scale-down
+kuttl test suite run has not yet produced a clean pass; treat controller scaling as functional but not yet fully
+verified end-to-end.
 
 == Kraft migration guide