[security] Container Image Scanner Flags Transient TLS Keys Under /proc

docs/en/solutions/Container_Image_Scanner_Flags_Transient_TLS_Keys_Under_proc.md

@@ -0,0 +1,71 @@
+---
+title: Third-party scanner reports private keys under /proc — runtime artifact, not image content
+component: security
+scenario: troubleshooting
+tags: [scanner, false-positive, procfs, tls, library-go]
+date_created: 2026-05-30
+date_updated: 2026-05-30
+---
+
+# Third-party scanner reports private keys under /proc — runtime artifact, not image content
+
+## Issue
+
+A third-party container image scanner running in node-mode on an Alauda Container Platform worker reports findings of the form "Private keys stored in image" against many platform and add-on workloads, where the offending file paths begin with `/proc/<pid>/root/...` — for example `/proc/<pid>/root/tmp/tls.key` or `/proc/<pid>/root/tmp/serving-certs/tls.key` [ev:c1]. The scanner attributes the finding to the container image, but the paths only exist via the kernel's per-PID procfs view of a running container's mount namespace and have no presence in the published image layers [ev:c1].
+
+A single ACP worker typically surfaces several such paths simultaneously. Examples observed on `cpaas-system` controllers on kernel 5.15 / containerd 2.2.1 [ev:c1]:
+
+```text
+PID 50397 (base-operator) -> /proc/50397/root/tmp/tls.crt
+PID 50397 (base-operator) -> /proc/50397/root/tmp/tls.key
+PID 63628 (apollo)        -> /proc/63628/root/tmp/serving-certs
+```
+
+## Root Cause
+
+The `/proc/<pid>/root` symlink is a Linux kernel feature: it gives any process able to read `/proc` a view of the addressed process's mount-namespace root [ev:c1]. So `/proc/50397/root/tmp/tls.key` is just the path `/tmp/tls.key` *inside* the container of PID 50397, surfaced through the host's procfs. The kernel exposes it whether the file lives on a tmpfs, on a projected `Secret` volume, or on the container's writable overlay layer [ev:c1].
+
+The reported `tls.key` / `tls.crt` files are written by the workload process itself after the container starts; they are not present in the image manifest or in any image layer [ev:c2]. Two patterns produce them on ACP:
+
+- **Process-written, on the overlay**: a controller generates a fresh TLS keypair into `/tmp/<name>.key` (or `/tmp/serving-cert-<n>/serving-signer.key` in some libraries) at startup and rotates the key periodically. The path has no entry in `/proc/<pid>/mounts`, confirming the file is on the container's writable upper layer rather than a mounted volume [ev:c3_a][ev:c3_b]. The `base-operator` controller on `cpaas-system` shows this pattern (`/tmp/tls.key` rotated within the lifetime of the running container, `mtime` later than the start of PID 50397) [ev:c2][ev:c3_a].
+- **Projected Secret on tmpfs**: a controller is started with TLS flags pointing into `/tmp/serving-certs/` (for example `--tls-cert-file=/tmp/serving-certs/tls.crt --tls-private-key-file=/tmp/serving-certs/tls.key`); the kubelet mounts a `Secret` there as a tmpfs and populates it with the atomic `..data` symlink layout. The bytes come from the API server, not from the image [ev:c2][ev:c3_b]. The platform's `apollo` service shows this pattern.
+
+In either case the image artifact itself contains no key material at the reported path — the scanner's pattern matcher fired on a runtime file it found by walking `/proc` on the node, not on something embedded in the image [ev:c3_a][ev:c3_b][ev:c4_a].
+
+## Resolution
+
+Findings whose path is prefixed by `/proc/<pid>/root/` are runtime-state artifacts and do not represent secret material baked into the image; they can be acknowledged as scanner false positives for the purpose of image-supply-chain risk [ev:c4_a]. The triage rule is:
+
+- If the scanner-reported path starts with `/proc/<pid>/root/...` — the file is per-process container state surfaced through procfs and is not part of the image artifact. No image rebuild or vendor escalation is required on this basis [ev:c1][ev:c4_a].
+- If the scanner-reported path is inside the actual image filesystem (no `/proc` prefix; for example a layer-relative `/usr/share/doc/.../server_key.pem`), the file is genuinely shipped in the image and the platform support process should be engaged to confirm whether the file is a real key or a documentation sample [ev:c4_a].
+
+Where the scanner allows configuration, narrow its scope so this distinction is automatic — most enterprise scanners support an "image-only" mode (registry pull, analyse layers) separate from "node-mode" (walk `/proc` on the host). Image-only mode never traverses `/proc` and so cannot produce this class of false positive [ev:c3_b].
+
+## Diagnostic Steps
+
+Confirm a reported path is process-state rather than image content by reading the live file's metadata through procfs from a node-debug shell. The relevant facts are: (a) the mount entry for the directory in `/proc/<pid>/mounts` (none = on the writable overlay; `tmpfs` = projected Secret); (b) the file's `mtime` relative to the container start (anything after start = generated by the process); and (c) what the process binary actually points at in its argument list [ev:c1][ev:c2][ev:c3_a].
+
+A short read-only probe across all running containers on a worker is:
+
+```bash
+kubectl debug node/<worker> -it=false --profile=sysadmin \
+  --image=registry.alauda.cn:60070/acp/container-debug:v4.3.2 -- \
+  sh -c 'for p in $(ls /proc | grep -E "^[0-9]+$"); do
+    root=/proc/$p/root; [ -d "$root/tmp" ] || continue
+    for f in $(ls -A "$root/tmp" 2>/dev/null); do
+      case "$f" in *.key|*.pem|*.crt|serving-cert*|tls-*)
+        echo "PID $p ($(cat /proc/$p/comm)) -> /proc/$p/root/tmp/$f" ;;
+      esac
+    done
+  done'
+```
+
+For each PID returned, cross-check the mount type and the file timestamp:
+
+```bash
+kubectl debug node/<worker> -it=false --profile=sysadmin \
+  --image=registry.alauda.cn:60070/acp/container-debug:v4.3.2 -- \
+  sh -c 'cat /proc/<pid>/mounts | grep " /tmp"; stat /proc/<pid>/root/tmp/<file>'
+```
+
+A `tmpfs` line for `/tmp/...` plus the kubelet `..data` symlink layout indicates a projected `Secret` (the key is from a Kubernetes Secret, not from the image). No mount entry plus an `mtime` after the container's start time indicates a process-written file on the writable overlay (the key was generated at runtime by the controller). In both situations the path resides outside the image layers, and a scanner finding tied to the `/proc` path is a false positive at the image-supply-chain level [ev:c3_a][ev:c3_b][ev:c4_a].