helm: configurable liveness, readiness and startup probes + named container ports

docs/setup-robusta/health-probes.rst

@@ -0,0 +1,91 @@
+Health Probes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Robusta ships without Kubernetes health probes. If your platform requires them - for example a policy that
+rejects containers with no ``livenessProbe`` - you can define them yourself on the runner and the forwarder.
+
+Nothing is rendered unless you set it, so the default install is unaffected.
+
+Available Endpoints
+-------------------------------------
+
+The runner serves HTTP on port ``5000``, named ``http``:
+
+* ``/metrics`` - available as soon as the web server starts, independent of any external service.
+* ``/healthz`` - ``200`` only when every configured sink reports healthy, ``500`` otherwise.
+
+The forwarder serves ``/metrics`` on port ``2112``, named ``metrics``. It has no ``/healthz``; any other path
+returns ``404``. The port is bound at process start, before the watch loop syncs.
+
+Recommended Configuration
+-------------------------------------
+
+.. code-block:: yaml
+
+    runner:
+      startupProbe:
+        httpGet:
+          path: /metrics
+          port: http
+        failureThreshold: 30
+        periodSeconds: 10
+      livenessProbe:
+        httpGet:
+          path: /metrics
+          port: http
+        periodSeconds: 30
+      readinessProbe:
+        httpGet:
+          path: /healthz
+          port: http
+        periodSeconds: 15
+
+    kubewatch:
+      livenessProbe:
+        httpGet:
+          path: /metrics
+          port: metrics
+        periodSeconds: 15
+
+Any probe field Kubernetes accepts can be used, including ``tcpSocket`` and ``exec``.
+
+Choosing an Endpoint
+-------------------------------------
+
+Use ``/metrics`` for liveness. It restarts the pod only when the process is genuinely wedged.
+
+Avoid ``/healthz`` for liveness. Because it tracks sink health, a transient outage at Slack or another sink
+would restart the runner repeatedly while nothing is wrong with the runner itself. Robusta shipped a liveness
+probe on ``/healthz`` in early 2023 and removed it again for this reason.
+
+``/healthz`` is a reasonable readiness endpoint if you want a sink-aware signal - it takes the pod out of the
+Service and blocks a rollout from proceeding, without restarting anything.
+
+Note that probes are not how you detect a misconfigured install. A runner that cannot reach the Kubernetes API,
+or a forwarder without a usable kubeconfig, exits at startup and lands in ``CrashLoopBackOff``, which Kubernetes
+already reports on its own.
+
+Why Reference Ports By Name
+-------------------------------------
+
+Both containers declare their port with a name, and the examples above use the name rather than ``5000`` or
+``2112``. Prefer the name in anything you write yourself:
+
+* **The number stops being repeated.** A probe that says ``port: http`` keeps working if the listen port
+  changes. A probe hardcoding ``5000`` has to be found and updated, and silently probes the wrong port until
+  someone notices.
+* **It says what the port is for.** ``port: http`` is readable in a review; ``port: 5000`` needs a lookup.
+* **Other Kubernetes objects resolve names too.** NetworkPolicy rules and Prometheus Operator scrape configs
+  accept a container port name and resolve it per pod, so one rename does not have to be chased across objects.
+* **A typo says so.** A probe naming a port no container declares reports
+  ``port "..." not found``. A probe with the wrong number reports a generic connection refused, which looks
+  the same as an application that is genuinely down.
+
+Port names are more constrained than Service port names: at most 15 characters, lowercase alphanumeric and
+dashes, and unique within the pod.
+
+One place to keep using the number is a Service ``targetPort``, which is why both Services still say
+``targetPort: 5000`` and ``targetPort: 2112``. A numeric ``targetPort`` resolves whether or not the pod
+declares the name, while a named one resolves per pod - so during an upgrade from a chart version whose pods
+had no named port, pods stay ``Ready`` but their endpoints are published with no port at all, and the Service
+black-holes traffic until the rollout finishes.

docs/setup-robusta/index.rst

@@ -22,6 +22,7 @@
    read-only-service-account
    rbac-namespace-scoping
    node-selector
+   health-probes
    proxies
    privacy-and-security
    installation-faq

helm/robusta/templates/forwarder.yaml

@@ -49,6 +49,10 @@ spec:
         image: {{ .Values.image.registry }}/{{ .Values.kubewatch.imageName }}
         {{- end }}
         imagePullPolicy: {{ .Values.kubewatch.imagePullPolicy }}
+        ports:
+          - name: metrics
+            containerPort: 2112
+            protocol: TCP
         env:
           - name: KW_CONFIG
             value: /config
@@ -67,6 +71,18 @@ spec:
         securityContext:
         {{- toYaml . | nindent 12 }}
         {{- end }}
+        {{- with .Values.kubewatch.startupProbe }}
+        startupProbe:
+        {{- toYaml . | nindent 10 }}
+        {{- end }}
+        {{- with .Values.kubewatch.livenessProbe }}
+        livenessProbe:
+        {{- toYaml . | nindent 10 }}
+        {{- end }}
+        {{- with .Values.kubewatch.readinessProbe }}
+        readinessProbe:
+        {{- toYaml . | nindent 10 }}
+        {{- end }}
         resources:
           requests:
             cpu: {{ .Values.kubewatch.resources.requests.cpu }}

helm/robusta/templates/runner.yaml

@@ -104,6 +104,10 @@ spec:
           {{- toYaml . | nindent 12 }}
         {{- end }}
         {{- end }}
+        ports:
+          - name: http
+            containerPort: 5000
+            protocol: TCP
         env:
           - name: PLAYBOOKS_CONFIG_FILE_PATH
             value: /etc/robusta/config/active_playbooks.yaml
@@ -197,6 +201,18 @@ spec:
           {{- with .Values.runner.extraVolumeMounts }}
           {{- toYaml . | nindent 10 }}
           {{- end }}
+        {{- with .Values.runner.startupProbe }}
+        startupProbe:
+        {{- toYaml . | nindent 10 }}
+        {{- end }}
+        {{- with .Values.runner.livenessProbe }}
+        livenessProbe:
+        {{- toYaml . | nindent 10 }}
+        {{- end }}
+        {{- with .Values.runner.readinessProbe }}
+        readinessProbe:
+        {{- toYaml . | nindent 10 }}
+        {{- end }}
         lifecycle:
           preStop:
             exec:

helm/robusta/values.yaml

@@ -651,6 +651,17 @@ kubewatch:
       memory: 512Mi
     limits:
       cpu: ~
+  # health probes, rendered only when set. reference the container port by name
+  # /metrics is served as soon as the process is up, before the watch loop syncs
+  # e.g.
+  # livenessProbe:
+  #   httpGet:
+  #     path: /metrics
+  #     port: metrics
+  #   periodSeconds: 30
+  startupProbe: {}
+  livenessProbe: {}
+  readinessProbe: {}
   additional_env_vars:
   - name: ADVANCED_FILTERS
     value: "true"
@@ -741,6 +752,18 @@ runner:
       memory: 1024Mi
     limits:
       cpu: ~
+  # health probes, rendered only when set. reference the container port by name
+  # /metrics is up whenever the web server is; /healthz additionally reflects sink health, so it
+  # suits readiness but will restart the pod on a transient sink outage if used for liveness
+  # e.g.
+  # livenessProbe:
+  #   httpGet:
+  #     path: /metrics
+  #     port: http
+  #   periodSeconds: 30
+  startupProbe: {}
+  livenessProbe: {}
+  readinessProbe: {}
   additional_env_vars: []
   additional_env_froms: []
   customCRD: []