fix(sts): rotate TLS certificates

Cargo.toml

@@ -11,7 +11,7 @@ homepage = "https://rustfs.com"
 chrono = { version = "0.4", features = ["serde"] }
 const-str = "1.0.0"
 serde = { version = "1.0.228", features = ["derive"] }
-tokio = { version = "1.49.0", features = ["rt", "rt-multi-thread", "macros", "fs", "io-std", "io-util"] }
+tokio = { version = "1.49.0", features = ["rt", "rt-multi-thread", "macros", "fs", "io-std", "io-util", "sync"] }
 tokio-rustls = "0.26"
 tokio-stream = { version = "0.1", features = ["sync"] }
 tokio-util = { version = "0.7", features = ["io", "compat", "rt"] }
@@ -30,6 +30,8 @@ rustls = { version = "0.23", default-features = false, features = ["ring"] }
 rustls-pemfile = "2.2.0"
 webpki = { package = "rustls-webpki", version = "0.103" }
 rcgen = "0.13"
+x509-parser = "0.18.1"
+time = "0.3.44"
 sha2 = "0.10"
 hmac = "0.12"
 hex = "0.4"

deploy/k8s-dev/operator-deployment.yaml

@@ -81,5 +81,6 @@ spec:
               value: rustfs-operator-sts
             - name: OPERATOR_STS_TLS_ENABLED
               value: "true"
+            # Kind/E2E development explicitly opts in to Operator-generated certificates.
             - name: OPERATOR_STS_TLS_AUTO
               value: "true"

deploy/k8s-dev/operator-rbac.yaml

@@ -92,8 +92,8 @@ subjects:
     name: rustfs-operator
     namespace: rustfs-system
 ---
-# STS auto TLS needs to create one Secret and replace it only when a previously generated
-# Secret becomes invalid. Keep write access in the operator namespace.
+# STS auto TLS needs to create one Secret and replace it when a previously generated Secret
+# becomes invalid, uses the legacy policy, or approaches expiry. Keep write access namespaced.
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:

deploy/rustfs-operator/README.md

@@ -21,6 +21,11 @@ To install in a specific namespace:
 helm install rustfs-operator deploy/rustfs-operator/ --namespace rustfs-system --create-namespace
 ```
 
+Operator STS TLS is enabled while automatic certificate generation is disabled by default.
+Before installing, pre-create `sts-tls` in the release namespace with `tls.crt`, `tls.key`, and
+`ca.crt`. For development environments only, opt in to Operator-generated certificates with
+`--set sts.tls.auto=true`.
+
 ### OpenShift Installation
 
 Enable the OpenShift profile so the chart omits the fixed Pod and container
@@ -123,7 +128,7 @@ manifests remain consistent.
 | `sts.audience` | Kubernetes TokenReview audience expected by the operator STS endpoint | `sts.rustfs.com` |
 | `sts.port` | Operator container port for STS | `4223` |
 | `sts.tls.enabled` | Serve the operator STS endpoint over TLS | `true` |
-| `sts.tls.auto` | Create and replace an invalid Operator-managed STS TLS Secret with namespaced write RBAC | `true` |
+| `sts.tls.auto` | Create and rotate an invalid, legacy, or soon-to-expire Operator-managed STS TLS Secret with namespaced write RBAC | `false` |
 | `sts.service.type` | Kubernetes Service type for STS | `ClusterIP` |
 | `sts.service.port` | Kubernetes Service port for STS | `4223` |
 
@@ -135,7 +140,11 @@ POST /sts/{tenantNamespace}/{tenantName}
 
 This differs from MinIO Operator's namespace-only route. A `PolicyBinding` still lives in the Tenant namespace, but the workload must call STS with both the Tenant namespace and the Tenant name.
 
-The STS service is HTTPS by default. When `sts.tls.auto=true`, the operator creates the fixed `sts-tls` Secret in the operator namespace with `tls.crt`, `tls.key`, and `ca.crt`. With `rbac.create=true`, the chart creates a namespaced Role that can create Secrets and update only `sts-tls`; the ClusterRole keeps all Secret and ConfigMap access read-only. If `rbac.create=false`, you must provide an equivalent Role and RoleBinding for the operator ServiceAccount: namespaced Secret `create`, plus `get` and `update` restricted to the `sts-tls` resource name. Workloads must trust that CA. To use an externally issued certificate, pre-create `sts-tls` with a certificate signed by a CA already trusted by the workload and set `sts.tls.auto=false`; the chart then omits the namespaced Secret write Role.
+The STS service is HTTPS by default, and `sts.tls.auto=false` makes externally issued certificates the default ownership model. Pre-create the fixed `sts-tls` Secret in the operator namespace with `tls.crt`, `tls.key`, and `ca.crt`; startup fails with an actionable error when TLS is enabled and that Secret is absent. Update the Secret to rotate the certificate manually, and the operator hot-loads valid replacement material within five minutes while retaining the last valid configuration on refresh failures. The chart does not grant namespaced Secret write access in this mode.
+
+Set `sts.tls.auto=true` explicitly for development or other deployments that accept an Operator-managed self-signed CA. The operator then creates `sts-tls` when missing and rotates invalid, legacy, or soon-to-expire managed material. Server certificates are valid for one year and rotate 30 days before expiry while retaining the same ten-year CA. The Operator-managed `sts-tls` Secret stores `ca.key` so leaf renewal can reuse the CA without breaking existing client trust; operations must keep this sensitive key within the intended security boundary when replicating or backing up the Secret. The CA is replaced only during legacy-policy migration or when it enters its own 30-day renewal window. Existing legacy Operator-managed Secrets do not contain `ca.key` and are replaced once after upgrade, so refresh every STS client's trusted `ca.crt` as part of that upgrade. Use the CA-expiry metric to plan a coordinated trust update when the ten-year CA approaches expiry. With `rbac.create=true`, the chart creates a namespaced Role that can create Secrets and update only `sts-tls`; the ClusterRole keeps all Secret and ConfigMap access read-only. If `rbac.create=false`, provide an equivalent Role and RoleBinding for the operator ServiceAccount: namespaced Secret `create`, plus `get` and `update` restricted to the `sts-tls` resource name.
+
+Monitor `rustfs_operator_sts_tls_certificate_expiry_timestamp_seconds` and `rustfs_operator_sts_tls_ca_expiry_timestamp_seconds` and alert before either timestamp is reached.
 
 STS only issues credentials for TLS-enabled Tenants. For Tenant upstream calls, the operator selects the Tenant HTTPS service endpoint and trusts the CA recorded in `status.certificates.tls.caSecretRef`.
 

deploy/rustfs-operator/values.yaml

@@ -127,11 +127,11 @@ sts:
   tls:
     # Operator STS listens with TLS by default and uses this Secret for server certs.
     enabled: true
-    # When true, the chart grants namespaced create access plus update access to the fixed
-    # sts-tls Secret. The operator creates a missing Secret and replaces an invalid Secret only
-    # when it is marked as Operator-managed. If rbac.create=false, provide an equivalent Role and
-    # RoleBinding in the operator namespace.
-    auto: true
+    # By default, pre-create sts-tls with tls.crt, tls.key, and ca.crt. Set this to true only when
+    # the operator should generate and rotate its own CA and server certificate. When true, the
+    # chart grants namespaced Secret create/update access; if rbac.create=false, provide an
+    # equivalent Role and RoleBinding in the operator namespace.
+    auto: false
   service:
     type: ClusterIP
     port: 4223

docs/operator-user-guide.md

@@ -71,6 +71,11 @@ When a Tenant is applied, the operator creates and owns:
 
 ## 4. Install the Operator
 
+Operator STS TLS is enabled while automatic certificate generation is disabled by default.
+Before installing, pre-create `sts-tls` in the Operator namespace with `tls.crt`, `tls.key`, and
+`ca.crt`. For Kind and other development environments, explicitly opt in to generated
+certificates with `--set sts.tls.auto=true`.
+
 Install with the included Helm chart:
 
 ```bash
@@ -319,15 +324,17 @@ sts:
     timeoutSeconds: 30
   tls:
     enabled: true
-    auto: true
+    auto: false
 ```
 
 Notes:
 
 - `operator.leaderElect` can be unset. The chart enables leader election automatically when `operator.replicas > 1`.
 - Keep `console.jwtSecret` stable when running multiple Console replicas. If unset, the chart generates or reuses a Secret.
 - Keep `CONSOLE_COOKIE_SECURE` enabled for production HTTPS. Only disable it for local HTTP testing.
-- `sts.tls.auto=true` lets the operator create or repair `sts-tls`; with `rbac.create=true`, the chart isolates write access in a namespaced Role while keeping cluster-wide Secret and ConfigMap access read-only. With `rbac.create=false`, provide an equivalent Role and RoleBinding for the operator ServiceAccount: namespaced Secret `create`, plus `get` and `update` restricted to the `sts-tls` resource name.
+- `sts.tls.auto=false` is the default. Pre-create `sts-tls` with externally issued `tls.crt`, `tls.key`, and `ca.crt`; otherwise startup fails with an actionable error. Replace that Secret to rotate the certificate manually. A valid replacement is hot-loaded within five minutes while refresh errors retain the last valid configuration.
+- Set `sts.tls.auto=true` explicitly for Kind or other development environments that should use an Operator-managed self-signed CA. Server certificates are valid for one year and rotate 30 days before expiry while retaining the same ten-year CA. The Operator-managed `sts-tls` Secret stores `ca.key` so leaf renewal can reuse the CA without breaking existing client trust; operations must keep this sensitive key within the intended security boundary when replicating or backing up the Secret. The CA changes only during legacy-policy migration or in its own 30-day renewal window. A legacy Operator-managed Secret does not contain `ca.key` and is replaced once after upgrade, so refresh clients that trust its `ca.crt`. Use the CA-expiry metric to plan a coordinated trust update when the ten-year CA approaches expiry. With `rbac.create=true`, the chart isolates write access in a namespaced Role while keeping cluster-wide Secret and ConfigMap access read-only. With `rbac.create=false`, provide an equivalent Role and RoleBinding for the Operator ServiceAccount: namespaced Secret `create`, plus `get` and `update` restricted to the `sts-tls` resource name.
+- Monitor `rustfs_operator_sts_tls_certificate_expiry_timestamp_seconds` and `rustfs_operator_sts_tls_ca_expiry_timestamp_seconds`.
 
 ## 6. Create a Tenant
 

docs/operator-user-guide.zh-CN.md

@@ -73,6 +73,10 @@ Operator 提供以下能力：
 
 ## 4. 安装 Operator
 
+Operator STS 默认启用 TLS，但默认不自动生成证书。安装前，需要在 Operator namespace
+中预创建包含 `tls.crt`、`tls.key` 和 `ca.crt` 的 `sts-tls` Secret。Kind 等开发环境可通过
+`--set sts.tls.auto=true` 显式启用自动生成。
+
 使用仓库内 Helm Chart 安装：
 
 ```bash
@@ -296,15 +300,17 @@ sts:
     timeoutSeconds: 30
   tls:
     enabled: true
-    auto: true
+    auto: false
 ```
 
 配置说明：
 
 - `operator.leaderElect` 可以不配置；当 `operator.replicas > 1` 时 Chart 会自动启用 leader election。
 - 多副本 Console 部署需要保持 `console.jwtSecret` 稳定；不设置时 Chart 会生成或复用已有 Secret。
 - 生产环境应使用 HTTPS 并保持 `CONSOLE_COOKIE_SECURE` 启用。仅本地 HTTP 调试时才关闭。
-- `sts.tls.auto=true` 时，Operator 会创建或修复 `sts-tls`；当 `rbac.create=true` 时，Chart 通过命名空间级 Role 隔离写权限，全集群 Secret 和 ConfigMap 权限保持只读。当 `rbac.create=false` 时，必须为 Operator ServiceAccount 自行提供等效的 Role 和 RoleBinding：允许在 Operator namespace 内 `create` Secret，并将 `get`、`update` 限定到名为 `sts-tls` 的资源。
+- `sts.tls.auto=false` 是默认值。必须预创建包含外部签发的 `tls.crt`、`tls.key` 和 `ca.crt` 的 `sts-tls`，否则 Operator 会输出可执行的错误信息并启动失败。替换该 Secret 可手动轮换证书；有效的新证书会在五分钟内热加载，刷新失败时继续使用最后一个有效配置。
+- Kind 等开发环境需要显式设置 `sts.tls.auto=true`，由 Operator 管理自签 CA。服务端证书有效期为一年，并在到期前 30 天轮换，同时复用同一个十年期 CA。Operator 托管的 `sts-tls` Secret 保存 `ca.key`，使叶证书续期不会破坏现有客户端信任；复制或备份该 Secret 时，运维必须确保这个敏感私钥不会离开预期的安全边界。只有旧策略迁移或 CA 自身进入 30 天续期窗口时才会更换 CA。旧策略生成的 Operator 托管 Secret 不包含 `ca.key`，升级后会被替换一次，因此需要同步刷新客户端信任的 `ca.crt`。十年期 CA 临近到期时，应结合 CA 到期指标安排一次协调后的信任更新。当 `rbac.create=true` 时，Chart 通过命名空间级 Role 隔离写权限，全集群 Secret 和 ConfigMap 权限保持只读。当 `rbac.create=false` 时，必须为 Operator ServiceAccount 自行提供等效的 Role 和 RoleBinding：允许在 Operator namespace 内 `create` Secret，并将 `get`、`update` 限定到名为 `sts-tls` 的资源。
+- 请监控 `rustfs_operator_sts_tls_certificate_expiry_timestamp_seconds` 和 `rustfs_operator_sts_tls_ca_expiry_timestamp_seconds`。
 
 ## 6. 创建 Tenant