From 844dcb7c4da1d512abcfe6f2fc9dd56bc96a0b9d Mon Sep 17 00:00:00 2001 From: Gaurav Trivedi Date: Tue, 28 Jul 2026 00:36:06 +0530 Subject: [PATCH] feat(optimize): add etcd pressure reduction procedures Add 3 procedure pages to reduce etcd storage usage on large clusters: - Configure automatic cleanup of inactive workspaces (DevWorkspace Pruner) - Disable workspace CA bundle mount (ca-certs-merged ConfigMaps) - Disable Copied CSVs (OLM namespace-scoped CSV copies) Each procedure includes prerequisites, verification steps, and cross-module xrefs to plan:etcd-storage-limits for context. Co-authored-by: Cursor --- modules/optimize/nav.adoc | 4 ++ ...omatic-cleanup-of-inactive-workspaces.adoc | 56 +++++++++++++++++++ .../optimize/pages/disabling-copied-csvs.adoc | 49 ++++++++++++++++ .../disabling-workspace-ca-bundle-mount.adoc | 49 ++++++++++++++++ 4 files changed, 158 insertions(+) create mode 100644 modules/optimize/pages/configuring-automatic-cleanup-of-inactive-workspaces.adoc create mode 100644 modules/optimize/pages/disabling-copied-csvs.adoc create mode 100644 modules/optimize/pages/disabling-workspace-ca-bundle-mount.adoc diff --git a/modules/optimize/nav.adoc b/modules/optimize/nav.adoc index 8ee0ee21ee..eede30b0c1 100644 --- a/modules/optimize/nav.adoc +++ b/modules/optimize/nav.adoc @@ -12,3 +12,7 @@ * xref:configuring-autoscaling.adoc[] ** xref:configuring-number-of-replicas.adoc[] ** xref:configuring-machine-autoscaling.adoc[] +* Reduce etcd pressure at scale +** xref:configuring-automatic-cleanup-of-inactive-workspaces.adoc[] +** xref:disabling-workspace-ca-bundle-mount.adoc[] +** xref:disabling-copied-csvs.adoc[] diff --git a/modules/optimize/pages/configuring-automatic-cleanup-of-inactive-workspaces.adoc b/modules/optimize/pages/configuring-automatic-cleanup-of-inactive-workspaces.adoc new file mode 100644 index 0000000000..e4e2d1205c --- /dev/null +++ b/modules/optimize/pages/configuring-automatic-cleanup-of-inactive-workspaces.adoc @@ -0,0 +1,56 @@ +:_content-type: PROCEDURE +:description: Configure the DevWorkspace Operator pruner to automatically delete workspaces that have not been started within a configurable retention period. +:keywords: optimize, etcd, cleanup, pruner, DevWorkspace, cron +:navtitle: Configure automatic cleanup of inactive workspaces + +[id="configuring-automatic-cleanup-of-inactive-workspaces"] += Configure automatic cleanup of inactive workspaces + +[role="_abstract"] +As `DevWorkspace` objects accumulate on large clusters, etcd storage grows and cluster performance degrades. Configure the {devworkspace} Operator pruner to automatically delete workspaces that have not been started within a configurable retention period. + +.Prerequisites + +* An active `{orch-cli}` session with administrative permissions to the destination {orch-name} cluster. See {orch-cli-link}. + +* {devworkspace} Operator version 0.34.0 or later is installed on the cluster. + +.Procedure + +. Configure the `DevWorkspaceOperatorConfig` object to enable the cleanup cron job: ++ +[source,yaml,subs="+attributes"] +---- +apiVersion: controller.devfile.io/v1alpha1 +kind: DevWorkspaceOperatorConfig +metadata: + name: devworkspace-operator-config + namespace: {prod-namespace} +config: + workspace: + cleanupCronJob: + enabled: true + dryRun: false + retainTime: 2592000 + schedule: "0 0 1 * *" +---- ++ +retainTime:: The number of seconds a workspace must be inactive before it is marked for deletion. The default value `2592000` equals 30 days. ++ +schedule:: A cron expression that defines how often the pruner runs. The default value `"0 0 1 * *"` runs the pruner once per month. + +. Optional: To preview which workspaces would be deleted without removing them, set `dryRun` to `true`. + +.Verification + +* Verify that the cleanup cron job is scheduled: ++ +[source,bash,subs="+attributes"] +---- +$ {orch-cli} get cronjobs -n {prod-namespace} +---- + +[role="_additional-resources"] +.Additional resources + +* xref:plan:etcd-storage-limits.adoc[] diff --git a/modules/optimize/pages/disabling-copied-csvs.adoc b/modules/optimize/pages/disabling-copied-csvs.adoc new file mode 100644 index 0000000000..7cf0ee53c9 --- /dev/null +++ b/modules/optimize/pages/disabling-copied-csvs.adoc @@ -0,0 +1,49 @@ +:_content-type: PROCEDURE +:description: Disable Copied CSVs to reduce etcd storage usage and OLM memory consumption on large clusters. +:keywords: optimize, etcd, OLM, CSV, namespace +:navtitle: Disable Copied CSVs to reduce etcd usage + +[id="disabling-copied-csvs"] += Disable Copied CSVs to reduce etcd usage + +[role="_abstract"] +When an Operator is installed by the Operator Lifecycle Manager (OLM), a stripped-down copy of its ClusterServiceVersion (CSV) is created in every namespace the Operator watches. On large clusters with hundreds or thousands of namespaces, Copied CSVs consume significant etcd storage, OLM memory, and network bandwidth. + +.Prerequisites + +* An active `{orch-cli}` session with administrative permissions to the destination {orch-name} cluster. See {orch-cli-link}. + +.Procedure + +. Configure the `OLMConfig` object to disable Copied CSVs: ++ +[source,yaml] +---- +apiVersion: operators.coreos.com/v1 +kind: OLMConfig +metadata: + name: cluster +spec: + features: + disableCopiedCSVs: true +---- ++ +Disabling Copied CSVs reduces etcd storage usage and the memory footprint of OLM. + +.Verification + +* Verify that Copied CSVs are no longer created in user namespaces: ++ +[source,bash,subs="+attributes"] +---- +$ {orch-cli} get csv -n +---- ++ +The command should return no Copied CSVs for namespaces that do not have their own Operator subscriptions. + +[role="_additional-resources"] +.Additional resources + +* link:https://github.com/operator-framework/enhancements/blob/master/enhancements/olm-toggle-copied-csvs.md[OLM toggle Copied CSVs enhancement proposal] +* link:https://olm.operatorframework.io/docs/advanced-tasks/configuring-olm/#disabling-copied-csvs[Disabling Copied CSVs in OLM] +* xref:plan:etcd-storage-limits.adoc[] diff --git a/modules/optimize/pages/disabling-workspace-ca-bundle-mount.adoc b/modules/optimize/pages/disabling-workspace-ca-bundle-mount.adoc new file mode 100644 index 0000000000..9cd15debb8 --- /dev/null +++ b/modules/optimize/pages/disabling-workspace-ca-bundle-mount.adoc @@ -0,0 +1,49 @@ +:_content-type: PROCEDURE +:description: Disable the full CA bundle mount to reduce etcd pressure from ca-certs-merged ConfigMaps in every user namespace. +:keywords: optimize, etcd, CA-bundle, ConfigMap, TLS +:navtitle: Disable the workspace CA bundle mount + +[id="disabling-workspace-ca-bundle-mount"] += Disable the workspace CA bundle mount + +[role="_abstract"] +By default, the Operator creates a `ca-certs-merged` ConfigMap in every user namespace. This ConfigMap contains the full Certificate Authorities (CAs) bundle. On clusters with many TLS certificates, these ConfigMaps consume significant etcd storage. Disable the full CA bundle mount to reduce etcd pressure. + +.Prerequisites + +* An active `{orch-cli}` session with administrative permissions to the destination {orch-name} cluster. See {orch-cli-link}. + +.Procedure + +. Configure the `CheCluster` Custom Resource to disable the workspace CA bundle mount: ++ +[source,bash,subs="+attributes"] +---- +$ {orch-cli} edit checluster/{prod-checluster} -n {prod-namespace} +---- ++ +[source,yaml] +---- +spec: + devEnvironments: + trustedCerts: + disableWorkspaceCaBundleMount: true +---- ++ +With this configuration, {prod-short} no longer mounts the full CA bundle under `/etc/pki/ca-trust/extracted/pem`. Only custom certificates are mounted under `/public-certs`. + +.Verification + +* Verify that new workspaces no longer mount the full CA bundle: ++ +[source,bash,subs="+attributes"] +---- +$ {orch-cli} get configmap ca-certs-merged -n +---- ++ +The ConfigMap should not exist in namespaces for newly created workspaces. + +[role="_additional-resources"] +.Additional resources + +* xref:plan:etcd-storage-limits.adoc[]