From 60d1ac568bc73590db70a18df927a515bd855867 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Wed, 7 Oct 2026 19:26:15 +0200 Subject: [PATCH 1/3] docs(operations): add encryption at rest guide for etcd with KMS v2 Add an Encryption at Rest section that explains what Cozystack stores where and how it is protected by default, and a guide to encrypt the management cluster etcd with KMS v2 and HashiCorp Vault Transit on Talos v1.14. The guide also encrypts Flux HelmReleases, which carry application values, explains what applies to aggregated API servers, and shows the local-key option for tenant Kubernetes clusters, where KMS is not supported yet. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- .../docs/next/operations/encryption/_index.md | 27 + .../next/operations/encryption/etcd-kms.md | 460 ++++++++++++++++++ 2 files changed, 487 insertions(+) create mode 100644 content/en/docs/next/operations/encryption/_index.md create mode 100644 content/en/docs/next/operations/encryption/etcd-kms.md diff --git a/content/en/docs/next/operations/encryption/_index.md b/content/en/docs/next/operations/encryption/_index.md new file mode 100644 index 00000000..223eb107 --- /dev/null +++ b/content/en/docs/next/operations/encryption/_index.md @@ -0,0 +1,27 @@ +--- +title: "Encryption at Rest" +linkTitle: "Encryption at Rest" +description: "Where Cozystack stores sensitive data and how to encrypt it with a key held outside the cluster" +weight: 38 +--- + +Cozystack keeps sensitive data in several places: Kubernetes Secrets and application values in the management cluster etcd, Secrets of tenant Kubernetes clusters in their own etcd, and user data of the platform identity provider in PostgreSQL. This section explains what is encrypted by default, where the keys live, and how to move the keys out of the cluster into a key management service (KMS). + +## What Is Encrypted by Default + +| Data | Stored in | Default protection | With KMS | +| --- | --- | --- | --- | +| Secrets of the management cluster | Management cluster etcd | Encrypted by Talos with `secretbox`; the key is in the machine configuration of every control-plane node | [KMS v2 with Vault Transit](/docs/next/operations/encryption/etcd-kms/) | +| Application values (`spec` of every Cozystack application, which may contain passwords) | Management cluster etcd, inside Flux `HelmRelease` objects | Not encrypted | [KMS v2 with Vault Transit](/docs/next/operations/encryption/etcd-kms/#choose-what-to-encrypt), once `helmreleases` is added to the encrypted resources | +| Secrets of tenant Kubernetes clusters | etcd of the tenant | Not encrypted | Not supported yet; a [local key](/docs/next/operations/encryption/etcd-kms/#tenant-kubernetes-clusters) can be used | +| Users of the platform Keycloak (usernames, emails, names, credentials) | PostgreSQL of Keycloak | Not encrypted | [keycloak-kms-proxy](/docs/next/operations/encryption/keycloak/) | + +The Cozystack API (`apps.cozystack.io`, `core.cozystack.io`) has no storage of its own: applications are stored as Flux `HelmRelease` objects and tenant secrets as regular Secrets, all in the management cluster etcd. Encrypting that etcd covers them. + +## Why Move the Key Out + +With the default `secretbox` encryption, the key sits on the same disks as the data it protects. A copy of a control-plane disk, or an etcd backup together with the machine configuration, is enough to read every Secret. + +With KMS v2, kube-apiserver encrypts each object with a data encryption key, and that key is in turn encrypted by a key encryption key that never leaves the KMS. Decrypting data requires a call to the KMS from the control-plane nodes, so stolen disks and backups are useless on their own. + +The guides in this section use [HashiCorp Vault](https://developer.hashicorp.com/vault/docs/secrets/transit) Transit as the KMS. diff --git a/content/en/docs/next/operations/encryption/etcd-kms.md b/content/en/docs/next/operations/encryption/etcd-kms.md new file mode 100644 index 00000000..d87ad02c --- /dev/null +++ b/content/en/docs/next/operations/encryption/etcd-kms.md @@ -0,0 +1,460 @@ +--- +title: "Encrypting etcd with KMS v2" +linkTitle: "etcd with KMS v2" +description: "Encrypt Secrets and application values in the management cluster etcd with a key held in HashiCorp Vault, using KMS v2 on Talos Linux" +weight: 10 +--- + +This guide moves the encryption key of the management cluster etcd out of the cluster. kube-apiserver encrypts each object with a data encryption key (DEK), and the DEK is encrypted by a key encryption key (KEK) that stays in [HashiCorp Vault](https://developer.hashicorp.com/vault/docs/secrets/transit) Transit. [vault-kubernetes-kms](https://github.com/FalcoSuessgott/vault-kubernetes-kms) is the [KMS v2](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/) plugin that connects kube-apiserver to Vault. + +After you finish, an etcd snapshot or a stolen disk no longer exposes Secrets or application values: decrypting them requires a call to Vault from one of the control-plane nodes. See [Encryption at Rest](/docs/next/operations/encryption/) for what is stored where. + +## How It Fits Together + +On every control-plane node: + +1. The KMS plugin runs as a Talos static pod on the host network and listens on a Unix socket in `/var/kms`. +2. kube-apiserver mounts `/var/kms` and reaches the plugin through that socket. +3. Talos renders the kube-apiserver `EncryptionConfiguration` from the `KubeEtcdEncryptionConfig` document in the machine configuration. +4. The plugin authenticates to Vault with AppRole and asks Transit to encrypt and decrypt DEKs with a key that cannot be exported. + +The plugin must be a static pod rather than a Deployment: kube-apiserver needs it to read Secrets, so it has to start before the API server does. + +## Prerequisites + +- Talos Linux v1.14 or later on all control-plane nodes. Talos v1.13 generates the encryption configuration itself and does not allow replacing it; the `KubeEtcdEncryptionConfig` document first appeared in [Talos v1.14](https://docs.siderolabs.com/talos/v1.14/reference/configuration/kubernetes/kubeetcdencryptionconfig). Clusters installed with an earlier Cozystack release may still run Talos v1.13; upgrade them first (see [step 1](#1-upgrade-talos)). +- A Kubernetes version supported by Talos v1.14: 1.33 to 1.37, per the [Talos support matrix](https://docs.siderolabs.com/talos/v1.14/getting-started/support-matrix). +- Talm v0.35.0 or later. Earlier releases cannot render configuration for Talos v1.14 nodes. Keep `talosVersion` in `Chart.yaml` at `v1.13` or lower, as the Talm documentation requires. +- A HashiCorp Vault server reachable from every control-plane node, with permission to enable a secrets engine and an auth method. The examples below use `https://vault.example.com:8200`. +- `talosctl` with the `os:admin` role, and `kubectl` with cluster-admin access to the management cluster. +- For verification: `etcd`, `etcdutl` and `etcdctl` binaries of the same minor version as the cluster etcd (Talos v1.14.1 runs etcd v3.7.1), and `jq`. + +{{% alert color="warning" %}} +Vault becomes a dependency of the Kubernetes API. If the Transit key is lost, every Secret encrypted with it is lost as well, and an etcd backup does not help: it holds only ciphertext. Back up Vault, and never delete or trim the Transit key while Secrets encrypted with it may still exist. +{{% /alert %}} + +## 1. Upgrade Talos + +Skip this step if the control-plane nodes already run Talos v1.14 or later (`talosctl --nodes version`). + +Talos v1.14 no longer loads kernel modules on demand, and DRBD loads its network transport that way. Without the module loaded explicitly, DRBD resources on an upgraded node stay in `Connecting`. The module is already present on Talos v1.13, so add it to the `values.yaml` of your Talm project and apply it to every node before the upgrade: + +```yaml +extraKernelModules: + - name: drbd_transport_tcp +``` + +Then set the Talos image of this release in `values.yaml` and upgrade the nodes one at a time, waiting for each node to come back and for LINSTOR resources to be in sync before the next one: + +```yaml +image: "ghcr.io/cozystack/cozystack/talos:{{< version-pin "talos" >}}" +``` + +```bash +talm upgrade -f nodes/cp1.yaml +``` + +## 2. Prepare Vault + +Enable the Transit secrets engine and create a key for this cluster. Use a separate key per cluster. By default a Transit key is `aes256-gcm96`, cannot be exported and cannot be deleted. + +```bash +vault secrets enable transit +vault write -f transit/keys/mgmt-etcd +``` + +Create a policy that allows the plugin to use this key and nothing else. The plugin reads the key to report its current version to kube-apiserver, so `read` on the key is required: + +```hcl +# mgmt-etcd-kms.hcl +path "auth/token/lookup-self" { + capabilities = ["read"] +} +path "transit/encrypt/mgmt-etcd" { + capabilities = ["update"] +} +path "transit/decrypt/mgmt-etcd" { + capabilities = ["update"] +} +path "transit/keys/mgmt-etcd" { + capabilities = ["read"] +} +``` + +```bash +vault policy write mgmt-etcd-kms mgmt-etcd-kms.hcl +``` + +Enable AppRole and create a role bound to the control-plane node addresses. The binding matters: the plugin credentials end up in the machine configuration, and with `secret_id_bound_cidrs` and `token_bound_cidrs` a credential copied from a stolen disk is useless anywhere but on the control-plane nodes themselves. + +```bash +vault auth enable approle +vault write auth/approle/role/mgmt-etcd-kms \ + token_policies=mgmt-etcd-kms \ + secret_id_bound_cidrs=192.168.100.11/32,192.168.100.12/32,192.168.100.13/32 \ + token_bound_cidrs=192.168.100.11/32,192.168.100.12/32,192.168.100.13/32 \ + secret_id_ttl=0 \ + secret_id_num_uses=0 \ + token_ttl=1h \ + token_max_ttl=4h +vault read -field=role_id auth/approle/role/mgmt-etcd-kms/role-id +vault write -f -field=secret_id auth/approle/role/mgmt-etcd-kms/secret-id +``` + +Replace the addresses with the IPs your control-plane nodes use to reach Vault. The plugin logs in again with the same secret ID whenever its token expires, so the secret ID must not expire or run out of uses (`secret_id_ttl=0`, `secret_id_num_uses=0`). Keep the role ID and secret ID for the next step. + +## 3. Add the KMS Plugin to the Control-Plane Nodes + +This step starts the plugin and lets kube-apiserver read data encrypted with KMS, but new writes still use the existing `secretbox` key. Nothing changes for the data yet. The switch comes in the next step, once every control-plane node can decrypt with both providers: during a rolling change the API servers must always be able to read what the others have written, as the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#rotating-a-decryption-key) warns. + +### Find the Current secretbox Key + +Talos names the current key `key2`. Its value is `secrets.secretboxencryptionsecret` in the `secrets.yaml` of your Talm project (decrypt the project first with `talm init --decrypt` if it is encrypted): + +```bash +yq '.secrets.secretboxencryptionsecret' secrets.yaml +``` + +You will copy this value into the encryption configuration below, so that existing Secrets stay readable. + +### Choose What to Encrypt + +Talos encrypts only `secrets` by default. In Cozystack, the `spec` of every application (Postgres, ClickHouse, Kubernetes and so on) is stored inline in the `spec.values` of a Flux `HelmRelease`, and some applications keep passwords there. Add `helmreleases.helm.toolkit.fluxcd.io` to the encrypted resources to cover them. The examples below encrypt both. + +The Helm release history that Flux keeps for every application is stored in Secrets, so `secrets` already covers it. + +### Edit the Node Files + +Add the following to the body of every control-plane node file (`nodes/.yaml`), below the `# talm:` modeline. Talm applies the body as a patch on top of the rendered configuration, so these fields are sent on every `talm apply`. + +```yaml +machine: + pods: + - apiVersion: v1 + kind: Pod + metadata: + name: vault-kubernetes-kms + namespace: kube-system + spec: + hostNetwork: true + priorityClassName: system-node-critical + initContainers: + # The kubelet creates /var/kms owned by root; hand it to the + # kube-apiserver user so the plugin can create its socket there. + - name: socket-dir-owner + image: docker.io/library/busybox:1.37.0 + command: ["chown", "65534:65534", "/var/kms"] + securityContext: + runAsUser: 0 + volumeMounts: + - name: socket + mountPath: /var/kms + containers: + - name: vault-kubernetes-kms + image: ghcr.io/falcosuessgott/vault-kubernetes-kms:v1.4.0 + command: + - /vault-kubernetes-kms + - -vault-address=https://vault.example.com:8200 + - -auth-method=approle + - -approle-mount=approle + - -transit-mount=transit + - -transit-key=mgmt-etcd + - -socket=unix:///var/kms/vaultkms.socket + - -force-socket-overwrite=true + # Health and metrics endpoint; it listens on all host + # interfaces (default 8080), so pick a port that is free. + - -health-port=18080 + env: + - name: VAULT_KMS_APPROLE_ROLE_ID + value: "" + - name: VAULT_KMS_APPROLE_SECRET_ID + value: "" + securityContext: + # kube-apiserver runs as 65534 on Talos and can only connect + # to a socket owned by that user. + runAsUser: 65534 + runAsGroup: 65534 + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumeMounts: + - name: socket + mountPath: /var/kms + volumes: + - name: socket + hostPath: + path: /var/kms + type: DirectoryOrCreate +cluster: + # The secretbox key moves into KubeEtcdEncryptionConfig below; + # Talos rejects a configuration that sets both. + secretboxEncryptionSecret: + $patch: delete + apiServer: + extraVolumes: + - hostPath: /var/kms + mountPath: /var/kms + readonly: false +--- +apiVersion: v1alpha1 +kind: KubeEtcdEncryptionConfig +config: + resources: + - resources: + - secrets + - helmreleases.helm.toolkit.fluxcd.io + providers: + - secretbox: + keys: + - name: key2 + secret: "" + - kms: + apiVersion: v2 + name: vault + endpoint: unix:///var/kms/vaultkms.socket + timeout: 3s + - identity: {} +``` + +Replace the Vault address, the Transit key name, the role ID, the secret ID and the secretbox key with your values. If the node file already has `cluster.apiServer.extraVolumes` (for example, for OIDC), add the `/var/kms` entry to the existing list. + +A few things to know about this configuration: + +- `apiVersion: v2` is required. Without it the provider defaults to KMS v1, which is disabled since Kubernetes 1.29, and kube-apiserver refuses the configuration. +- Talos does not validate the body of `KubeEtcdEncryptionConfig` on apply. A typo is only reported later, when Talos renders the file for kube-apiserver. Double-check the field names. +- `machine.pods` and `cluster.apiServer.extraVolumes` are deprecated in Talos v1.14 but still honoured. Talos has no replacement for mounting a host directory into kube-apiserver yet. +- The plugin runs on the host network, because it has to work before the cluster network does: the CNI needs the API server, and the API server needs the plugin. +- Do not add a liveness probe on the plugin's `/live` endpoint. It calls Vault on every check, so a short Vault or network hiccup would make the kubelet restart the plugin. +- If Vault uses a private CA, put the CA certificate on the node and pass it with the `-vault-ca-cert` flag. + +{{% alert color="warning" %}} +The node files now contain the AppRole secret ID in plain text. Do not commit them to Git as is. If your Talm project lives in Git, move the pod definition into the project templates and the two IDs into [encrypted user values](/docs/next/install/kubernetes/talm/#24-encrypted-user-values-and-secret-redaction-talm-v032), which Talm decrypts in memory on apply. + +Also note that `talm template -I` rewrites node files from the templates and drops everything added by hand. Keep a copy of these blocks outside `nodes/`, and add them again before the next apply if you regenerate a node file. An apply without them would remove the `kms` provider, and kube-apiserver would no longer be able to read Secrets encrypted with KMS. +{{% /alert %}} + +### Apply, One Node at a Time + +Preview the change first. The dry run also catches the conflict between `secretboxEncryptionSecret` and `KubeEtcdEncryptionConfig`: + +```bash +talm apply -f nodes/cp1.yaml --dry-run +``` + +Then apply it and wait until the node is healthy before moving to the next one: + +```bash +talm apply -f nodes/cp1.yaml +kubectl --namespace kube-system get pods --selector k8s-app=kube-apiserver --output wide +kubectl get --raw '/readyz?verbose' | grep kms +``` + +The last command should print `[+]kms-providers ok`. Repeat for every control-plane node. + +## 4. Switch Encryption to KMS + +Once all control-plane nodes run the configuration from the previous step, swap the first two providers in every node file, so that KMS encrypts new writes and `secretbox` is kept only to read old data: + +```yaml + providers: + - kms: + apiVersion: v2 + name: vault + endpoint: unix:///var/kms/vaultkms.socket + timeout: 3s + - secretbox: + keys: + - name: key2 + secret: "" + - identity: {} +``` + +Apply the node files one at a time again, checking `[+]kms-providers ok` after each node. The first provider in the list encrypts; all providers are tried in order to decrypt. + +## 5. Re-encrypt Existing Data + +Objects written before the switch are still stored with the old provider: Secrets with `secretbox`, HelmReleases in plain text. Rewrite all of them so that kube-apiserver stores them again with the new first provider: + +```bash +kubectl get secrets --all-namespaces --output json | kubectl replace --filename - +kubectl get helmreleases.helm.toolkit.fluxcd.io --all-namespaces --output json | kubectl replace --filename - +``` + +Run it when the cluster is quiet. An object that changes while the command runs may fail with a conflict error; running the command again is safe. + +## 6. Verify + +### Check the Active Configuration + +Talos shows the encryption configuration it renders for kube-apiserver. The output contains key material, so reading it requires the `os:admin` role: + +```bash +talosctl --nodes get etcdencryptionconfigs --output yaml +``` + +The providers must be listed in the order `kms`, `secretbox`, `identity`. + +### Check the Data in etcd + +On Talos, etcd is a system service rather than a pod, and `talosctl` cannot read individual keys. Take a snapshot, restore it locally and read the Secret from the copy: + +```bash +kubectl --namespace default create secret generic kms-check --from-literal=probe=value + +talosctl --nodes etcd snapshot db.snapshot +etcdutl snapshot restore db.snapshot --data-dir ./etcd-restore +etcd --data-dir ./etcd-restore \ + --listen-client-urls http://127.0.0.1:32379 \ + --advertise-client-urls http://127.0.0.1:32379 \ + --listen-peer-urls http://127.0.0.1:32380 & + +etcdctl --endpoints http://127.0.0.1:32379 \ + get /registry/secrets/default/kms-check --print-value-only | head -c 64 | hexdump -C +``` + +The value must start with `k8s:enc:kms:v2:vault:`. A value that starts with `k8s:enc:secretbox:v1:` was not re-encrypted. + +To list every Secret and HelmRelease that is not yet encrypted with KMS: + +```bash +for prefix in /registry/secrets/ /registry/helm.toolkit.fluxcd.io/helmreleases/; do + etcdctl --endpoints http://127.0.0.1:32379 \ + get "$prefix" --prefix --write-out json \ + | jq -r '.kvs[]? | select((.value | @base64d | startswith("k8s:enc:kms:v2:")) | not) | .key | @base64d' +done +``` + +The output must be empty. Then stop the local etcd and delete the copy: + +```bash +kill %1 +rm -rf ./etcd-restore db.snapshot +kubectl --namespace default delete secret kms-check +``` + +{{% alert color="warning" %}} +An etcd snapshot contains the whole cluster state, including every resource that is not encrypted. Handle it as sensitive data and delete it right after the check. +{{% /alert %}} + +## 7. Remove the Old Key + +When the check above shows no Secrets left on `secretbox`, remove the `secretbox` entry from the providers in every node file and apply the node files one at a time: + +```yaml + providers: + - kms: + apiVersion: v2 + name: vault + endpoint: unix:///var/kms/vaultkms.socket + timeout: 3s + - identity: {} +``` + +Until you do this, the old key in the machine configuration can still decrypt any Secret written before the migration. + +## Aggregated API Servers + +The Cozystack API server (`apps.cozystack.io`, `core.cozystack.io`, `sdn.cozystack.io`) has no storage of its own: everything it serves is stored by the management kube-apiserver, so the configuration above covers it. + +An aggregated API server that runs its own etcd, for example one you deploy on top of Cozystack, needs its own encryption configuration. Any API server built on `k8s.io/apiserver` with etcd storage accepts the same `--encryption-provider-config` flag: + +- Run the KMS plugin as a sidecar container in the API server pod and share the socket through an `emptyDir` volume. +- Run the plugin with the same UID as the API server, so that the server can connect to the socket the plugin creates. +- Use a separate Transit key for every API server, so that a credential leaked from one of them cannot decrypt the data of the others. +- List the API server's own resources as `.`, the same way as `helmreleases.helm.toolkit.fluxcd.io` above. +- Do not point a liveness probe with a short timeout at the plugin's `/live` endpoint: it calls Vault on every check. The KMS health check of the API server itself is part of `/readyz`, not `/livez`. + +## Tenant Kubernetes Clusters + +KMS encryption for tenant Kubernetes clusters is not supported yet: their control planes run as pods managed by Kamaji, and the `kubernetes` application does not let you add a sidecar next to kube-apiserver. By default, Secrets of tenant clusters are stored in their etcd unencrypted. + +What works today is encryption with a local key, set through the values of the `Kubernetes` application. The key is stored in a Secret in the tenant namespace of the management cluster. This protects the data in the tenant etcd and in its backups, but anyone who can read Secrets in that namespace can read the key. If the management cluster encrypts `secrets` with KMS, the key itself is encrypted at rest. + +Create the Secret in the namespace of the `Kubernetes` application: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: mycluster-encryption-config + namespace: tenant-example +stringData: + config.yaml: | + apiVersion: apiserver.config.k8s.io/v1 + kind: EncryptionConfiguration + resources: + - resources: + - secrets + providers: + - secretbox: + keys: + - name: key1 + secret: + - identity: {} +``` + +Generate the key with `head -c 32 /dev/urandom | base64`. Then add to the values of the `Kubernetes` application: + +```yaml +controlPlane: + apiServer: + extraArgs: + - --encryption-provider-config=/etc/kubernetes/encryption/config.yaml + - --encryption-provider-config-automatic-reload=true + extraVolumes: + - name: encryption-config + secret: + secretName: mycluster-encryption-config + extraVolumeMounts: + - name: encryption-config + mountPath: /etc/kubernetes/encryption + readOnly: true +``` + +Once the control plane has restarted, rewrite the existing Secrets with the kubeconfig of the tenant cluster, using the first command from [step 5](#5-re-encrypt-existing-data). With automatic reload enabled, later changes to the Secret, such as a new key, are picked up without a restart. + +## Rotating the Key + +The Kubernetes project [recommends](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/) rotating the key encryption key at least every 90 days. With Vault Transit, a rotation does not require touching the nodes: + +```bash +vault write -f transit/keys/mgmt-etcd/rotate +``` + +The plugin reports the new key version to kube-apiserver, which polls it about once a minute and starts using the new version for new writes without a restart. Existing Secrets stay readable, because Vault keeps the old key versions. To move them to the new version, wait a few minutes after the rotation and run the rewrite from [step 5](#5-re-encrypt-existing-data) again. + +Only after that, and after checking that no Secret is still encrypted with an old version, you may raise `min_decryption_version` on the Transit key. Doing it earlier makes the remaining old Secrets unreadable. + +## Monitoring and Failure Modes + +If the plugin or Vault becomes unavailable, kube-apiserver keeps writing for up to about three minutes and keeps reading from its cache, then Secret reads and writes start to fail. A kube-apiserver that starts while the plugin is down comes up, but reports not ready and cannot serve Secrets. + +Watch for it with: + +- the `kms-providers` check in `kubectl get --raw '/readyz?verbose'`; +- `apiserver_envelope_encryption_kms_operations_latency_seconds`, labelled with the gRPC status code of each call to the plugin; +- `apiserver_envelope_encryption_invalid_key_id_from_status_total`, which grows when the plugin reports a broken key version; +- `apiserver_storage_transformation_operations_total` with a non-OK `status`, which counts failed encryption and decryption operations. + +A ready API server does not always mean Secrets can be read. The most reliable signal is a periodic check that reads a canary Secret. + +## Rolling Back + +To stop using KMS, put `identity: {}` first while keeping the `kms` provider and the plugin running, apply every node, rewrite all Secrets with the command from [step 5](#5-re-encrypt-existing-data), and only then remove the `kms` provider and the plugin. Removing the plugin before the rewrite makes every KMS-encrypted Secret unreadable. + +## Troubleshooting + +### `etcd encryption config is already set in v1alpha1 config` + +The rendered configuration still carries `cluster.secretboxEncryptionSecret`. Check that the `$patch: delete` directive is in the node file body and is indented under `cluster`. As a fallback, remove the `secretboxencryptionsecret` field from `secrets.yaml`; keep its value, because the `secretbox` provider still needs it. + +### `[-]kms-providers failed` + +kube-apiserver cannot reach the plugin. Check the plugin logs: + +```bash +kubectl --namespace kube-system logs vault-kubernetes-kms- +``` + +Typical causes: the socket is not owned by UID 65534 (the init container did not run, or the plugin runs as another user), the node cannot reach Vault, or the AppRole binding does not include the node address. From 5c52ecd0ca62ab5c66b944fb64ab82c44ff78303 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Wed, 7 Oct 2026 20:58:46 +0200 Subject: [PATCH 2/3] docs(operations): add guide to encrypt Keycloak user data Describe how to enable keycloak-kms-proxy on the platform Keycloak with HashiCorp Vault Transit: Vault setup, creating the shared DEK set, encrypting existing rows in a maintenance window, enabling the proxy through the cozystack.keycloak package, verification and limits. The guide makes deksetSecretName mandatory: without it the proxy mints a new key on every start and cannot read data written before a restart. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- .../next/operations/encryption/keycloak.md | 214 ++++++++++++++++++ 1 file changed, 214 insertions(+) create mode 100644 content/en/docs/next/operations/encryption/keycloak.md diff --git a/content/en/docs/next/operations/encryption/keycloak.md b/content/en/docs/next/operations/encryption/keycloak.md new file mode 100644 index 00000000..ca8b74c0 --- /dev/null +++ b/content/en/docs/next/operations/encryption/keycloak.md @@ -0,0 +1,214 @@ +--- +title: "Encrypting Keycloak User Data" +linkTitle: "Keycloak" +description: "Encrypt usernames, emails, names and credentials of the platform Keycloak in PostgreSQL with keycloak-kms-proxy and HashiCorp Vault" +weight: 20 +--- + +The platform Keycloak keeps its users in PostgreSQL in plain text: anyone with access to the database, a dump or a backup can read every username, email and name. [keycloak-kms-proxy](https://github.com/cozystack/keycloak-kms-proxy) sits between Keycloak and PostgreSQL and encrypts these columns on the way in and decrypts them on the way out, so the database holds only ciphertext. + +The proxy encrypts the columns with data encryption keys (DEKs). The DEKs are stored wrapped by a key in [HashiCorp Vault](https://developer.hashicorp.com/vault/docs/secrets/transit) Transit and are unwrapped in memory when the proxy starts. + +The platform Keycloak exists only when OIDC is enabled; see [Enable OIDC](/docs/next/operations/oidc/enable_oidc/). + +## What Is Encrypted + +| Column | Encryption | Effect | +| --- | --- | --- | +| `USER_ENTITY.USERNAME`, `USER_ENTITY.EMAIL` | Deterministic, lower-cased first | Login by username or email keeps working | +| `USER_ENTITY.FIRST_NAME`, `USER_ENTITY.LAST_NAME` | Randomized | Cannot be searched | +| `USER_ATTRIBUTE.VALUE`, `USER_ATTRIBUTE.LONG_VALUE` of attributes whose name starts with `pii-` | Randomized | Cannot be searched | +| `CREDENTIAL.SECRET_DATA`, `CREDENTIAL.CREDENTIAL_DATA` | Randomized | Adds a layer on top of password hashing | + +Everything else is stored as before. In the admin console, a search by username or email works only for an exact value: a substring search does not find encrypted users. + +Encrypted values start with the `$KKP$` marker. + +{{% alert color="warning" %}} +Always set `encryption.deksetSecretName`. Without it, the proxy generates a new DEK every time it starts and keeps it only in memory, so after a restart of the proxy pod it can no longer decrypt anything it has encrypted before. +{{% /alert %}} + +## Prerequisites + +- OIDC enabled, so that the `cozystack.keycloak` package is installed. +- A HashiCorp Vault server reachable from the management cluster, and a Vault token that can create a Transit key and use it once to encrypt. +- Go, to run the proxy's backfill tool from source; it is not part of the proxy image. +- `kubectl` and the `flux` CLI. No local `psql` is needed: the check below runs it inside the database pod. +- A maintenance window. Keycloak is stopped while the existing rows are encrypted. + +## 1. Prepare Vault + +Create a Transit key for Keycloak: + +```bash +vault secrets enable transit +vault write -f transit/keys/keycloak +``` + +The proxy only encrypts and decrypts with this key: + +```hcl +# keycloak-kms-proxy.hcl +path "transit/encrypt/keycloak" { + capabilities = ["update"] +} +path "transit/decrypt/keycloak" { + capabilities = ["update"] +} +``` + +```bash +vault policy write keycloak-kms-proxy keycloak-kms-proxy.hcl +``` + +The recommended way for the proxy to log in is the [Kubernetes auth method](https://developer.hashicorp.com/vault/docs/auth/kubernetes), so that no Vault credential is stored in the cluster. Configure the auth method for the management cluster as described in the Vault documentation, then bind a role to the proxy's ServiceAccount: + +```bash +vault write auth/kubernetes/role/keycloak-kms-proxy \ + bound_service_account_names=keycloak-kms-proxy \ + bound_service_account_namespaces=cozy-keycloak \ + policies=keycloak-kms-proxy \ + ttl=1h +``` + +AppRole and a static token are supported too; see [Other Vault Login Methods](#other-vault-login-methods). + +## 2. Create the DEK Set + +Generate the DEKs, wrapped by the Transit key, with the backfill tool of the proxy version your Cozystack ships (`0.2.3` in this release): + +```bash +git clone --branch v0.2.3 https://github.com/cozystack/keycloak-kms-proxy +cd keycloak-kms-proxy +go run ./cmd/backfill generate-dekset \ + -vault-addr https://vault.example.com:8200 \ + -vault-token "$VAULT_TOKEN" \ + -vault-mount transit \ + -vault-key keycloak \ + -out dekset.json +``` + +Store the result in a Secret next to Keycloak: + +```bash +kubectl --namespace cozy-keycloak create secret generic keycloak-dekset \ + --from-file=dekset.json=dekset.json +``` + +`dekset.json` is useless without the Vault key, but the Vault key is useless without it too: keep a copy of this file in your backups. Losing either one makes the encrypted data unreadable. + +## 3. Encrypt the Existing Rows + +Enabling the proxy also switches Keycloak to it, so the rows that are already in the database must be encrypted first. Otherwise logins by username or email stop working. + +Take a backup of the `keycloak-db` database, then stop Keycloak. Suspend its HelmRelease first, or Flux scales it back up: + +```bash +flux suspend helmrelease keycloak --namespace cozy-keycloak +REPLICAS=$(kubectl --namespace cozy-keycloak get statefulset keycloak --output jsonpath='{.spec.replicas}') +kubectl --namespace cozy-keycloak scale statefulset keycloak --replicas=0 +kubectl --namespace cozy-keycloak rollout status statefulset keycloak --timeout=5m +``` + +Open a connection to the database and run the backfill from the same checkout: + +```bash +kubectl --namespace cozy-keycloak port-forward service/keycloak-db-rw 5432:5432 & + +DB_USER=$(kubectl --namespace cozy-keycloak get secret keycloak-db-app --output jsonpath='{.data.username}' | base64 -d) +DB_PASS=$(kubectl --namespace cozy-keycloak get secret keycloak-db-app --output jsonpath='{.data.password}' | base64 -d) +DB_NAME=$(kubectl --namespace cozy-keycloak get secret keycloak-db-app --output jsonpath='{.data.dbname}' | base64 -d) + +go run ./cmd/backfill encrypt-rows \ + -dekset dekset.json \ + -vault-addr https://vault.example.com:8200 \ + -vault-token "$VAULT_TOKEN" \ + -vault-mount transit \ + -vault-key keycloak \ + -dsn "postgres://${DB_USER}:${DB_PASS}@127.0.0.1:5432/${DB_NAME}" +``` + +The backfill skips values that already carry the `$KKP$` marker, so running it again is safe. + +## 4. Enable the Proxy + +Set the encryption values on the `cozystack.keycloak` package: + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: Package +metadata: + name: cozystack.keycloak +spec: + variant: default + components: + keycloak: + values: + encryption: + enabled: true + deksetSecretName: keycloak-dekset + replicas: 2 + kms: + backend: vault-transit + vault: + address: https://vault.example.com:8200 + mount: transit + keyName: keycloak + auth: kubernetes + kubernetes: + role: keycloak-kms-proxy +``` + +```bash +kubectl apply --server-side --filename keycloak-package.yaml +flux resume helmrelease keycloak --namespace cozy-keycloak +``` + +Flux deploys the proxy and updates Keycloak to point at it. If the `keycloak` StatefulSet stays at zero replicas afterwards, scale it back with `kubectl --namespace cozy-keycloak scale statefulset keycloak --replicas="$REPLICAS"`. With a shared DEK set the proxy can run more than one replica; without `deksetSecretName`, `replicas` above 1 is refused. + +If Vault uses a private CA, add it as `encryption.kms.vault.caBundle` (PEM), or reference an existing Secret with `encryption.kms.vault.caSecretName` and `caSecretKey`. + +## 5. Verify + +Check that the proxy and Keycloak are running and that a known user can log in: + +```bash +kubectl --namespace cozy-keycloak get pods +``` + +Then look at the raw data in PostgreSQL. No email must be left without the `$KKP$` marker: + +```bash +DB_NAME=$(kubectl --namespace cozy-keycloak get secret keycloak-db-app --output jsonpath='{.data.dbname}' | base64 -d) +kubectl --namespace cozy-keycloak exec -i keycloak-db-1 --container postgres -- \ + psql --dbname "$DB_NAME" --tuples-only <<'SQL' +SELECT count(*) FROM user_entity +WHERE email IS NOT NULL AND email NOT LIKE '$KKP$%'; +SQL +``` + +The proxy exposes Prometheus metrics on port 9090 of the `keycloak-kms-proxy` Service. `kkp_decrypt_failures_total` must stay at zero. + +## Other Vault Login Methods + +Set `encryption.kms.vault.auth` to one of: + +- `kubernetes`: the proxy logs in with its ServiceAccount `keycloak-kms-proxy`. Set `kubernetes.role`, and `kubernetes.mount` if the auth method is not mounted at `kubernetes`. +- `approle`: create a Secret in `cozy-keycloak` with the keys `role-id` and `secret-id`, and set `appRole.secretName` (and `appRole.mount` if it is not `approle`). The proxy logs in again with the same secret ID when its token expires, so the secret ID must not expire or run out of uses. +- `token`: create a Secret in `cozy-keycloak` with the key `token` and set `tokenSecretName`. The proxy never renews this token, so an expired token surfaces on the next proxy restart. + +## Key Rotation + +Rotating the Transit key needs no change in the cluster: the DEK set stays wrapped with the old key version, which Vault keeps. + +```bash +vault write -f transit/keys/keycloak/rotate +``` + +Rotating the DEKs themselves is not supported by the current proxy tools. + +## Limitations + +- There is no way back. The tools cannot decrypt the data, and switching `encryption.enabled` off makes Keycloak read ciphertext. To undo the encryption, restore the database backup taken before step 3. +- If Vault is unreachable when the proxy starts, the proxy does not start and Keycloak answers logins with an internal error. A running proxy keeps its DEKs in memory and is not affected by a Vault outage. +- Writes to the database that bypass the proxy, such as manual SQL, are stored in plain text. From 8da73b36a061a3264d23111c2e678f46cb6d76f6 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 18:16:25 +0200 Subject: [PATCH 3/3] docs(operations): match the Keycloak guide to the required DEK set The keycloak chart now refuses to enable encryption without encryption.deksetSecretName, so describe that instead of a default to avoid. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- content/en/docs/next/operations/encryption/keycloak.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/next/operations/encryption/keycloak.md b/content/en/docs/next/operations/encryption/keycloak.md index ca8b74c0..6ed2acd5 100644 --- a/content/en/docs/next/operations/encryption/keycloak.md +++ b/content/en/docs/next/operations/encryption/keycloak.md @@ -25,7 +25,7 @@ Everything else is stored as before. In the admin console, a search by username Encrypted values start with the `$KKP$` marker. {{% alert color="warning" %}} -Always set `encryption.deksetSecretName`. Without it, the proxy generates a new DEK every time it starts and keeps it only in memory, so after a restart of the proxy pod it can no longer decrypt anything it has encrypted before. +The proxy needs a persistent DEK set, `encryption.deksetSecretName`. Without one it would generate a new DEK every time it starts and keep it only in memory, so after a restart of the proxy pod it could no longer decrypt anything it had encrypted before. The chart refuses to enable encryption without it. {{% /alert %}} ## Prerequisites @@ -164,7 +164,7 @@ kubectl apply --server-side --filename keycloak-package.yaml flux resume helmrelease keycloak --namespace cozy-keycloak ``` -Flux deploys the proxy and updates Keycloak to point at it. If the `keycloak` StatefulSet stays at zero replicas afterwards, scale it back with `kubectl --namespace cozy-keycloak scale statefulset keycloak --replicas="$REPLICAS"`. With a shared DEK set the proxy can run more than one replica; without `deksetSecretName`, `replicas` above 1 is refused. +Flux deploys the proxy and updates Keycloak to point at it. If the `keycloak` StatefulSet stays at zero replicas afterwards, scale it back with `kubectl --namespace cozy-keycloak scale statefulset keycloak --replicas="$REPLICAS"`. Because the DEK set is shared, the proxy can run more than one replica. If Vault uses a private CA, add it as `encryption.kms.vault.caBundle` (PEM), or reference an existing Secret with `encryption.kms.vault.caSecretName` and `caSecretKey`.