feat: opt-in rootless Docker + kind in the workspace - #847
Closed
ppat wants to merge 9 commits into
Closed
Conversation
Adds an enable_docker workspace parameter that runs a rootless Docker daemon inside the (still unprivileged, no-socket-mount) workspace pod, so k8s/gitops work — building images, running kind clusters for chainsaw tests — can happen in-workspace. Image: rootless-docker plumbing baked into system-base (uidmap, slirp4netns, fuse-overlayfs, iptables, docker-ce from Docker's apt repo for the dockerd-rootless helpers); subuid/subgid guaranteed for coder. kind itself comes from the operator's dotfiles (aqua), not the image. Template: enable_docker gates a docker-data PVC (survives stop/start, not start_count-gated), an XDG_RUNTIME_DIR emptyDir, DOCKER_HOST, and a gated rootless dockerd bring-up in the agent startup script. The pod carries a com.coder.workspace.docker-enabled marker label; a Kyverno policy (apps repo) injects spec.hostUsers=false since the kubernetes provider 3.2.1 has no host_users field yet. Co-Authored-By: Claude <noreply@anthropic.com>
New DESIGN-DIND.md covers why the workspace runs Docker/kind while staying unprivileged (rejected alternatives, the Kyverno-injected hostUsers indirection, storage/driver trade-offs, and the cluster-migration angle). DESIGN.md, README.md, CLAUDE.md, and TESTING.md gain brief pointers plus the enable_docker gotchas and the live verification recipe. Co-Authored-By: Claude <noreply@anthropic.com>
ppat
added a commit
to ppat/homelab-ops-kubernetes-apps
that referenced
this pull request
Jul 20, 2026
#3351) Coder workspaces that opt into rootless Docker (via the workspace template's enable_docker parameter) need spec.hostUsers=false so the in-workspace container engine runs in a private user namespace and cannot map to host UIDs. The hashicorp/kubernetes Terraform provider (3.2.1) has no host_users field yet, so the template cannot set it directly. This mutate ClusterPolicy injects it at admission time, scoped by the docker-enabled marker label the template stamps only when Docker is enabled. failurePolicy is Ignore: a Kyverno outage must not block workspace creation. Retire this policy once the provider gains host_users (draft PR #2914). Pairs with the coder repo template change PR (ppat/coder#847) to implement ppat/coder#846. Co-Authored-By: Claude <noreply@anthropic.com>
The rootless dockerd daemon never started in a real workspace: dockerd-rootless.sh was absent. Those entrypoints ship in the separate docker-ce-rootless-extras package, which docker-ce does not pull in. Add it, and have the startup script fail with a clear message (rather than a cryptic setsid error) if the binary is ever missing again. Co-Authored-By: Claude <noreply@anthropic.com>
Rootless dockerd failed with 'newuidmap: write to uid_map failed: Operation not permitted' under hostUsers=false. The pod userns is only 65536 IDs wide (0-65535), but useradd had set coder:165536:65536 -- pointing outside it. Our previous append-if-missing never fired because useradd's entry was already there. Overwrite with coder:10002:55534 (10002..65535, above coder's UID 10001), which fits the namespace. Also make the XDG_RUNTIME_DIR chmod non-fatal: the emptyDir is root-owned but group-writable via fsGroup, so it is already usable and the chmod only produced a spurious error. Co-Authored-By: Claude <noreply@anthropic.com>
test_mode workspaces run a mutable branch tag (branch-<name>) that is rebuilt in place. With the default IfNotPresent policy a node that cached the tag never re-pulls, so template/image fixes silently test against a stale image. Set imagePullPolicy=Always in test_mode; released (immutable) tags keep IfNotPresent. Co-Authored-By: Claude <noreply@anthropic.com>
Rootless-in-userns hit an unavoidable wall: allowPrivilegeEscalation=false sets no_new_privs, which neuters setuid newuidmap, so rootlesskit cannot set up its multi-UID map (EPERM) even with an in-range subuid. hostUsers=false already gives us safe namespaced root (root in the pod maps to an unprivileged host uid), so run the container as root and use a plain rootful dockerd instead -- no newuidmap, no subuid, no rootless plumbing. - New script-workspace-entrypoint.sh (runs as root): does the apt/homebrew prep (absorbed from the old init container), starts dockerd (non-fatal), then drops to the coder user via setpriv and execs the coder agent. The coder user keeps its 10001 identity and role. - deployment.tf: drop the init container and the whole system-emptyDir + /usr /etc /var subPath dance (only existed to move root-made changes into a non-root container). Main container runs as uid 0 with CAP_SYS_ADMIN,NET_ADMIN and allowPrivilegeEscalation=true; privileged stays false. Safe because the platform Kyverno policy sets hostUsers:false via the always-present docker-enabled marker. - Docker data PVC now backs /var/lib/docker (rootful), created unconditionally so it survives stop. - Drop the enable_docker parameter (docker is always on) and the DOCKER_HOST env (rootful uses the default socket; coder reaches it via the docker group). - script-agent-startup.sh reverts to a coder-user stub. Image still carries the (now-unused) rootless packages; cleanup deferred until this is confirmed working. Co-Authored-By: Claude <noreply@anthropic.com>
dockerd failed to extract image layers: 'mount ... bind ... permission denied', with 'could not setup daemon root propagation to shared' earlier. Root cause: the /var/lib/docker PVC is mounted by the kubelet OUTSIDE the pod user namespace, so it is a locked private mount -- dockerd cannot make it shared or bind-mount snapshots within it, even as namespaced root with CAP_SYS_ADMIN (verified: findmnt shows PROPAGATION=private, mount --make-rshared -> EPERM). Fix: drop the /var/lib/docker PVC entirely. The entrypoint instead bind-mounts a dir under the home PVC (/home/coder/.var/docker) onto /var/lib/docker inside this container's own mount namespace -- an unlocked mount dockerd CAN manage -- and makes it shared. Data still persists across stop via the home PVC. Both dirs are set root:root 0711 (dockerd rejects a group/world-accessible data root; the home PVC is otherwise fsGroup-owned by the coder group). Co-Authored-By: Claude <noreply@anthropic.com>
The entrypoint's in-container bind mount was silently blocked by the container AppArmor profile (mount is denied there), so it never took effect. Mount the docker data root as a kubelet-provided emptyDir instead: it is a plain non-overlay fs set up outside the container, avoiding both overlay-on-overlay (the container rootfs is overlayfs) and the in-container mount restriction. Entrypoint just sets ownership (root:root 0711); no in-container mount needed. Tradeoff: emptyDir does not persist across workspace stop -- revisit persistence once the runtime is confirmed working end to end. Co-Authored-By: Claude <noreply@anthropic.com>
emptyDir at /var/lib/docker gives containerd an ext4 (non-overlay) data root, yet image extraction still fails 'permission denied' on containerd's internal bind mount -- proving it is NOT overlay-on-overlay but the cri-containerd AppArmor profile's deny-mount rule. privileged=true sets the profile to unconfined; if docker then works, the durable fix is appArmorProfile:Unconfined via Kyverno (provider has no field for it), without going fully privileged. Testing step, not the final posture. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #846.
What
Adds an opt-in
enable_dockerworkspace parameter that runs a rootless Docker daemon inside the workspace, so Kubernetes/GitOps work — building images, runningkindclusters for chainsaw tests — can happen in-workspace. The pod stays unprivileged: noprivileged, no added capabilities, no host Docker socket mounted.Full design + rejected alternatives: DESIGN-DIND.md.
How
Dockerfile): rootless-docker plumbing baked intosystem-base—uidmap,slirp4netns,fuse-overlayfs,iptables, and docker-ce/cli/containerd/buildx from Docker's official apt repo (for thedockerd-rootless*.shhelpers). subuid/subgid guaranteed forcoderbeforeUSER coder.kindis not baked in — it comes from dotfiles (aqua).enable_dockergates a dedicated docker-data PVC (survives stop/start — gated onenable_docker, deliberately notstart_count), anXDG_RUNTIME_DIRemptyDir,DOCKER_HOST, acom.coder.workspace.docker-enabledmarker label, and a rootlessdockerdbring-up in the agent startup script (non-fatal, storage-driver auto-detect).kubernetesprovider (3.2.1) has nohost_usersfield, sospec.hostUsers: falseis injected by a Kyverno mutate policy keyed off the marker label. Companion PR: feat(apps-coder): Kyverno policy to set hostUsers on docker workspaces homelab-ops-kubernetes-apps#3351.Testing
terraform validate/fmt/tflint,hadolint,shellcheckpass (pre-commit).unshare -UrOK, subuid/subgid present).test_modedry-run flow — see TESTING.md.Merge ordering
Merge the companion apps PR (#3351) first so the Kyverno policy exists when a workspace opts in; without it the pod still runs, just in the weaker (no-
hostUsers) posture.