feat: opt-in rootless Docker + kind in the workspace

CLAUDE.md

@@ -80,6 +80,7 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [
 - The Dockerfile writes shared env vars to `/etc/environment` rather than using `ENV`, because `PATH` needs to be extended by a script running after the image is built, not fixed at build time.
 - `parameters.tf`'s `local.validated_*` regex allowlist is the only thing stopping `system_packages`/`preferred_nodes` from injecting shell metacharacters into the init container — any new list-type parameter must go through the same decode-then-validate step.
 - Adding a package/tool has three possible homes, and picking the wrong one is a real mistake, not a style choice — route by the rule in [DESIGN.md](DESIGN.md#where-the-workspace-environment-comes-from): universal + stable → image (`Dockerfile`); occasionally-needed + apt-only + too heavy to bake in → the template's `system_packages` parameter; personal, fast-moving, or not an apt package → the operator's dotfiles (a *different* repo — see below), never this one.
+- The rootless-Docker feature (`enable_docker` parameter) spans this repo *and* the apps repo, and has non-obvious load-bearing details — read [DESIGN-DIND.md](DESIGN-DIND.md) before touching it. Key traps: (1) the `kubernetes` provider 3.2.1 has **no `host_users` field**, so `hostUsers: false` is set by a Kyverno policy in the apps repo keyed off the `com.coder.workspace.docker-enabled` marker label — renaming that label breaks the contract silently; (2) the docker-data PVC is gated on `enable_docker` but **not** on `start_count` (like the home PVC) so it survives workspace stop — gating it on `start_count` would wipe every pulled image on each stop; (3) docker's plumbing is baked into the image but its *behaviour* is parameter-gated, because the setuid `newuidmap`/`newgidmap` + subuid/subgid can only be set at build time as root; those setuid bits must survive the init container's `rsync` into the `system` volume.
 
 ## Neighbouring repos
 

DESIGN-DIND.md

@@ -0,0 +1,120 @@
+# DESIGN-DIND.md — nested containers in the workspace
+
+Why the workspace can run Docker and `kind` while staying an unprivileged pod, and how the
+pieces fit. This is a focused companion to [DESIGN.md](DESIGN.md); read that first for the
+repo-wide intent and the image/template split.
+
+## Intent
+
+A lot of the work done in these workspaces is Kubernetes/GitOps: building container images and
+standing up throwaway `kind` clusters (for example, to run another repo's `chainsaw` tests
+locally, the same way CI does). That needs a container runtime *inside* the workspace.
+
+The constraint that shapes everything: **the workspace must stay unprivileged**. No `privileged`
+container, no `CAP_SYS_ADMIN`, no host Docker socket mounted in. The workspace runs as a fixed
+non-root UID and should not be a foothold onto the node. So "just run Docker" is exactly what we
+can't do — the whole design is about getting a usable container runtime *without* relaxing that.
+
+## Options considered
+
+The obvious answers all fail the unprivileged/low-blast-radius bar in different ways:
+
+| Approach | Why not |
+| --- | --- |
+| Privileged Docker sidecar / mount the node's docker socket | Gives the workspace root on the node. Rejected outright. |
+| **Sysbox** runtime | Solid isolation, but it's a **node-level** runtime install + `RuntimeClass` — a platform component with high blast radius, and its maintenance/OS-support story is shaky. |
+| **Envbox** | Wraps Sysbox but requires a **privileged outer pod**. Rejects on the same privilege bar. |
+| **Kata Containers** | Real VM isolation, but a node-level runtime + `RuntimeClass` dependency and needs KVM/nested-virt. High blast radius for a homelab. |
+| **Rootless Docker/Podman inside the pod** | Runtime is **pod-local** — nothing installed on the node. This is the one that fits. |
+
+Within the rootless family, **Docker** was chosen over Podman/nerdctl for one practical reason:
+the consuming repo's CI runs `kind` on its **default Docker provider**, and matching that keeps
+"what I run locally" identical to "what CI runs" — no experimental-provider divergence. The cost
+is that rootless `dockerd` is the fiddliest of the three to bring up in a bare pod (no user
+systemd session); accepted deliberately for the local-==-CI payoff.
+
+### The strong-isolation piece: user namespaces
+
+Rootless alone still benefits enormously from a **user namespace** that maps the workspace's
+in-container root to an unprivileged host UID — that's what makes a container breakout land as
+"nobody" on the node rather than something with reach. Kubernetes exposes this as
+`pod.spec.hostUsers: false` (GA on the cluster we target).
+
+The catch, and the reason this design has a moving part that looks odd: **the Terraform
+`kubernetes` provider we're pinned to has no field to set `hostUsers`**. Rather than abandon the
+typed `kubernetes_deployment_v1` resource (rewriting the whole pod as a raw manifest loses
+validation and forces unrelated refactors), we set the field **out of band with a Kyverno mutate
+policy** that lives with the platform, keyed off a marker label the template stamps only when the
+user opts in. The template stays typed and simple; the platform grants the privilege-reducing
+mapping. When the provider gains the field, the policy is deleted and the template sets it
+directly — a one-line change, not a redesign.
+
+## How the pieces compose
+
+```mermaid
+flowchart TB
+    subgraph image [Image · baked plumbing]
+      I["rootless dockerd + CLI<br/>uidmap · slirp4netns · fuse-overlayfs<br/>subuid/subgid for coder"]
+    end
+    subgraph template [Template · gated on enable_docker]
+      P["enable_docker parameter"]
+      M["marker label<br/>com.coder.workspace.docker-enabled=true"]
+      V["docker-data PVC + XDG_RUNTIME_DIR<br/>DOCKER_HOST env"]
+      S["agent startup:<br/>start rootless dockerd"]
+    end
+    subgraph platform [Platform · apps repo]
+      K["Kyverno mutate policy<br/>injects spec.hostUsers=false"]
+    end
+    Pod["Unprivileged workspace pod<br/>+ user namespace"]
+    Kind["docker → kind clusters"]
+
+    I --> Pod
+    P --> M --> Pod
+    P --> V --> Pod
+    P --> S --> Pod
+    M -. matched by .-> K -. mutates .-> Pod
+    Pod --> Kind
+```
+
+The division of labour: the **image** carries the universal, stable plumbing (baked so startup is
+fast and reproducible — the [DESIGN.md](DESIGN.md#where-the-workspace-environment-comes-from)
+layering rule); the **template** turns it on per-workspace and provides the pod-local storage and
+runtime wiring; the **platform policy** supplies the one thing the template can't express. `kind`
+itself is *not* baked into the image — it's day-to-day tooling and comes from the operator's
+dotfiles (aqua), per the same layering rule.
+
+## Trade-offs and things to know
+
+- **Opt-in, not baseline.** The plumbing is baked into every image (it must be — the setuid
+  `newuidmap`/`newgidmap` and subuid/subgid ranges can only be set at build time as root), but the
+  *behaviour* (daemon start, the PVC, `hostUsers` mutation) is gated behind `enable_docker`. A
+  workspace that doesn't opt in is byte-for-byte the same runtime posture as before.
+- **Storage is a dedicated PVC, and survives stop/start.** Docker's image/layer store and `kind`
+  node images are large and worth keeping between sessions, so they live on their own
+  `ReadWriteOnce` block PVC rather than the shared home volume or an `emptyDir`. Crucially it is
+  gated on `enable_docker` but **not** on `start_count`, mirroring the external home PVC — tying it
+  to `start_count` would wipe every pulled image on each workspace *stop*.
+- **Storage driver.** With a user namespace and a modern kernel, native `overlay2` works and needs
+  no `/dev/fuse` (which the unprivileged pod lacks); `fuse-overlayfs` is the fallback, `vfs` the
+  last resort. The daemon auto-detects.
+- **Degrades, doesn't break.** If the Kyverno policy is absent or fails open, the pod still runs —
+  just without the `hostUsers` mapping (weaker isolation). The marker label and the policy are a
+  cross-repo contract; neither should be renamed in isolation.
+- **No nested cgroup resource limits.** Rootless dockerd in a pod with no user systemd session
+  runs without delegated cgroup limits; the workspace pod's own Kubernetes limits still cap
+  everything, which is fine for a single operator.
+
+## Future: migrating the cluster runtime
+
+This design is deliberately runtime-agnostic and should *ease* a future migration to a different
+Kubernetes distro rather than block it:
+
+- The image and startup scripts are not tied to the current distro; they carry over unchanged.
+- User namespaces (`hostUsers: false`) are a standard Kubernetes feature; the target distro
+  already enables the necessary kubelet/runtime support, so the same mechanism applies.
+- The Kyverno policy is platform-level and portable as-is.
+- The one thing to re-check at migration time is that the docker-data PVC's storage class exists on
+  the new cluster (or swap it for the equivalent).
+
+The `hostUsers`-via-policy indirection is the only piece expected to be temporary: once the
+Terraform provider exposes the field, the template sets it directly and the policy retires.

DESIGN.md

@@ -63,6 +63,8 @@ The rule that ties the layers together: a package or tool belongs in the *lowest
 
 **Unprivileged by default.** The workspace itself runs as an unprivileged, non-root, fixed-identity container. Anything that genuinely needs elevated privilege (installing packages, preparing shared volume state) is scoped to a narrow, short-lived setup step that runs before the workspace shell exists, not to something the workspace user can reach into.
 
+**Nested containers without giving up that posture.** Running Docker and `kind` *inside* the workspace would normally mean a privileged pod or a node-level runtime — both rejected here. Instead an opt-in rootless-Docker setup keeps the pod unprivileged and socket-free, with user-namespace isolation supplied by a platform Kyverno policy. This is a large enough topic to have its own design doc — see [DESIGN-DIND.md](DESIGN-DIND.md), which also covers the cluster-runtime migration angle.
+
 ## Outcomes targeted
 
 - One operator can keep dependencies current and ship template/image changes at low ongoing effort, without a fleet of environments to maintain.

README.md

@@ -14,6 +14,7 @@ This repo is the middle of a larger stack: the Coder control plane is deployed s
 ## Docs
 
 - **[DESIGN.md](DESIGN.md)** — why the template and image are built the way they are: trade-offs considered, decisions made, targeted outcomes.
+- **[DESIGN-DIND.md](DESIGN-DIND.md)** — how the workspace runs Docker and `kind` while staying an unprivileged pod.
 - **[TESTING.md](TESTING.md)** — how to validate a change, including exercising it against the real cluster without touching production workspace data.
 - **[CLAUDE.md](CLAUDE.md)** — commands and conventions for working in this repo with Claude Code.
 - **[CHANGELOG.md](CHANGELOG.md)** — generated release history (semantic-release).

TESTING.md

@@ -23,6 +23,17 @@ Both modes run the identical sequence of stages; only what each stage is permitt
 
 Because all three stages run for real in dry-run — just scoped away from production — a passing PR is a meaningful signal that a live release would also succeed, not a guess based on static checks alone.
 
+## Exercising the rootless Docker feature
+
+The `enable_docker` feature (see [DESIGN-DIND.md](DESIGN-DIND.md)) can't be judged by lint alone — it needs a live workspace. After the dry-run pipeline builds the image and pushes the disposable template, provision a test workspace from that template with **`enable_docker = true`** and check, from a terminal in the workspace:
+
+- `docker info` — the daemon is up and rootless (`Cgroup Driver` / `rootless: true`), storage driver is `overlay2` (or the `fuse-overlayfs`/`vfs` fallback).
+- `docker run --rm hello-world` — a container actually runs.
+- `kind create cluster` (kind comes from dotfiles) then `kubectl get nodes` — a nested cluster comes up; tear it down with `kind delete cluster`.
+- `kubectl get pod <this-workspace-pod> -o jsonpath='{.spec.hostUsers}'` (from a client with cluster access) returns `false` — confirming the platform Kyverno policy mutated the pod. If it's empty, the policy side isn't in effect and Docker is running in the weaker posture.
+
+The Kyverno policy half lives in the platform repo and has its own chainsaw coverage there (it asserts a marked pod is mutated and an unmarked one is left alone) — a template change here that renames the `com.coder.workspace.docker-enabled` marker must be matched there.
+
 ## After merge
 
 Merging to `main` is what flips the pipeline into live mode — there's no separate promotion step afterward. The dry-run pass on the PR is the actual release gate.

images/homelab-workspace/Dockerfile

@@ -113,6 +113,53 @@ RUN --mount=type=cache,target=/var/cache/apt,id=cache-apt-${TARGETARCH},sharing=
     # Clean pycache created during apt-get install (as apt stills retains some crud in spite of PYTHONPYCACHEPREFIX)
     find /usr -name __pycache__ -exec rm -rf {} +
 
+# ========================================================================================================
+# Rootless Docker support (the runtime kind/Kubernetes-in-Docker relies on; kind itself is installed by the
+# user via aqua in their dotfiles, not baked into the image).
+# Kept as a self-contained block (separate from the general toolset above) so it can later be lifted into a
+# dedicated variant image without untangling it from the base package list.
+# - uidmap:         provides newuidmap/newgidmap (setuid-root) needed to map subordinate uid/gid ranges
+# - slirp4netns:    userspace network stack for rootless dockerd (no CAP_NET_ADMIN required)
+# - fuse-overlayfs: fallback storage driver; native overlay2 (userns + kernel>=5.11) is the primary path,
+#                   used when /dev/fuse is unavailable (the unprivileged pod has no /dev/fuse)
+# - iptables:       required by dockerd/kind for bridge + port-mapping rules
+# We use Docker's official apt repo (not Ubuntu's docker.io) so the rootless entrypoints ship in the
+# docker-ce-rootless-extras package (dockerd-rootless.sh / dockerd-rootless-setuptool.sh).
+RUN --mount=type=cache,target=/var/cache/apt,id=cache-apt-${TARGETARCH},sharing=locked \
+    --mount=type=cache,target=/var/cache/debconf,id=cache-debconf-${TARGETARCH},sharing=locked \
+    --mount=type=cache,target=/var/lib/apt,id=lib-apt-${TARGETARCH},sharing=locked \
+    --mount=type=tmpfs,target=/var/cache/python \
+    --mount=type=tmpfs,target=/tmp \
+    --mount=type=tmpfs,target=/var/log \
+    --mount=type=tmpfs,target=/var/tmp \
+    # rootless-docker runtime dependencies (from Ubuntu repos)
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -yq \
+      fuse-overlayfs \
+      iptables \
+      slirp4netns \
+      uidmap \
+      && \
+    # add Docker's official apt repository + signing key (arch matched to the build platform)
+    install -m 0755 -d /etc/apt/keyrings && \
+    curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc && \
+    chmod a+r /etc/apt/keyrings/docker.asc && \
+    echo "deb [arch=${TARGETARCH} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "${VERSION_CODENAME}") stable" \
+      > /etc/apt/sources.list.d/docker.list && \
+    # Docker engine + CLI + buildx + containerd. docker-ce-rootless-extras is a
+    # SEPARATE package (not pulled in by docker-ce) that ships dockerd-rootless.sh
+    # and dockerd-rootless-setuptool.sh -- the rootless entrypoints this image needs.
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -yq \
+      containerd.io \
+      docker-buildx-plugin \
+      docker-ce \
+      docker-ce-cli \
+      docker-ce-rootless-extras \
+      && \
+    # Clean pycache created during apt-get install (as apt stills retains some crud in spite of PYTHONPYCACHEPREFIX)
+    find /usr -name __pycache__ -exec rm -rf {} +
+
 # ========================================================================================================
 FROM system-base
 SHELL ["/bin/bash", "-o", "pipefail", "-c"]
@@ -142,5 +189,16 @@ RUN groupadd --gid "${CODER_GID}" coder && \
     # package installation is complete, users of resulting image need not cache downloaded packages
     rm -f /etc/apt/apt.conf.d/keep-cache
 
+# Subordinate uid/gid ranges for 'coder', sized to fit INSIDE a user-namespaced pod.
+# When the workspace runs with hostUsers=false (see DESIGN-DIND.md) the pod's userns is only
+# 65536 IDs wide (0-65535), and newuidmap maps subordinate IDs relative to that namespace.
+# useradd's default range (coder:165536:65536) points outside it, so rootlesskit fails with
+# "newuidmap: write to uid_map failed: Operation not permitted". We therefore OVERWRITE (not
+# append) with a range that fits above coder's own UID 10001: 10002..65535 -> 55534 subordinate
+# IDs, enough for nested rootless containers. newuidmap/newgidmap must stay setuid-root.
+RUN echo 'coder:10002:55534' > /etc/subuid && \
+    echo 'coder:10002:55534' > /etc/subgid && \
+    test -u /usr/bin/newuidmap && test -u /usr/bin/newgidmap
+
 USER coder
 WORKDIR /home/coder

templates/kubernetes/homelab-workspace/configmap.tf

@@ -8,8 +8,8 @@ resource "kubernetes_config_map_v1" "workspace_scripts" {
   }
 
   data = {
-    agent_startup_script     = file("${path.cwd}/script-agent-startup.sh")
-    prepare_workspace_script = file("${path.cwd}/script-prepare-workspace.sh")
-    workspace_init_script    = coder_agent.main.init_script
+    agent_startup_script        = file("${path.cwd}/script-agent-startup.sh")
+    workspace_entrypoint_script = file("${path.cwd}/script-workspace-entrypoint.sh")
+    workspace_init_script       = coder_agent.main.init_script
   }
 }