diff --git a/CLAUDE.md b/CLAUDE.md index 8b5d6760..687a6d6d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -80,6 +80,7 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [ - The Dockerfile writes shared env vars to `/etc/environment` rather than using `ENV`, because `PATH` needs to be extended by a script running after the image is built, not fixed at build time. - `parameters.tf`'s `local.validated_*` regex allowlist is the only thing stopping `system_packages`/`preferred_nodes` from injecting shell metacharacters into the init container — any new list-type parameter must go through the same decode-then-validate step. - Adding a package/tool has three possible homes, and picking the wrong one is a real mistake, not a style choice — route by the rule in [DESIGN.md](DESIGN.md#where-the-workspace-environment-comes-from): universal + stable → image (`Dockerfile`); occasionally-needed + apt-only + too heavy to bake in → the template's `system_packages` parameter; personal, fast-moving, or not an apt package → the operator's dotfiles (a *different* repo — see below), never this one. +- The rootless-Docker feature (`enable_docker` parameter) spans this repo *and* the apps repo, and has non-obvious load-bearing details — read [DESIGN-DIND.md](DESIGN-DIND.md) before touching it. Key traps: (1) the `kubernetes` provider 3.2.1 has **no `host_users` field**, so `hostUsers: false` is set by a Kyverno policy in the apps repo keyed off the `com.coder.workspace.docker-enabled` marker label — renaming that label breaks the contract silently; (2) the docker-data PVC is gated on `enable_docker` but **not** on `start_count` (like the home PVC) so it survives workspace stop — gating it on `start_count` would wipe every pulled image on each stop; (3) docker's plumbing is baked into the image but its *behaviour* is parameter-gated, because the setuid `newuidmap`/`newgidmap` + subuid/subgid can only be set at build time as root; those setuid bits must survive the init container's `rsync` into the `system` volume. ## Neighbouring repos diff --git a/DESIGN-DIND.md b/DESIGN-DIND.md new file mode 100644 index 00000000..c2338bac --- /dev/null +++ b/DESIGN-DIND.md @@ -0,0 +1,120 @@ +# DESIGN-DIND.md — nested containers in the workspace + +Why the workspace can run Docker and `kind` while staying an unprivileged pod, and how the +pieces fit. This is a focused companion to [DESIGN.md](DESIGN.md); read that first for the +repo-wide intent and the image/template split. + +## Intent + +A lot of the work done in these workspaces is Kubernetes/GitOps: building container images and +standing up throwaway `kind` clusters (for example, to run another repo's `chainsaw` tests +locally, the same way CI does). That needs a container runtime *inside* the workspace. + +The constraint that shapes everything: **the workspace must stay unprivileged**. No `privileged` +container, no `CAP_SYS_ADMIN`, no host Docker socket mounted in. The workspace runs as a fixed +non-root UID and should not be a foothold onto the node. So "just run Docker" is exactly what we +can't do — the whole design is about getting a usable container runtime *without* relaxing that. + +## Options considered + +The obvious answers all fail the unprivileged/low-blast-radius bar in different ways: + +| Approach | Why not | +| --- | --- | +| Privileged Docker sidecar / mount the node's docker socket | Gives the workspace root on the node. Rejected outright. | +| **Sysbox** runtime | Solid isolation, but it's a **node-level** runtime install + `RuntimeClass` — a platform component with high blast radius, and its maintenance/OS-support story is shaky. | +| **Envbox** | Wraps Sysbox but requires a **privileged outer pod**. Rejects on the same privilege bar. | +| **Kata Containers** | Real VM isolation, but a node-level runtime + `RuntimeClass` dependency and needs KVM/nested-virt. High blast radius for a homelab. | +| **Rootless Docker/Podman inside the pod** | Runtime is **pod-local** — nothing installed on the node. This is the one that fits. | + +Within the rootless family, **Docker** was chosen over Podman/nerdctl for one practical reason: +the consuming repo's CI runs `kind` on its **default Docker provider**, and matching that keeps +"what I run locally" identical to "what CI runs" — no experimental-provider divergence. The cost +is that rootless `dockerd` is the fiddliest of the three to bring up in a bare pod (no user +systemd session); accepted deliberately for the local-==-CI payoff. + +### The strong-isolation piece: user namespaces + +Rootless alone still benefits enormously from a **user namespace** that maps the workspace's +in-container root to an unprivileged host UID — that's what makes a container breakout land as +"nobody" on the node rather than something with reach. Kubernetes exposes this as +`pod.spec.hostUsers: false` (GA on the cluster we target). + +The catch, and the reason this design has a moving part that looks odd: **the Terraform +`kubernetes` provider we're pinned to has no field to set `hostUsers`**. Rather than abandon the +typed `kubernetes_deployment_v1` resource (rewriting the whole pod as a raw manifest loses +validation and forces unrelated refactors), we set the field **out of band with a Kyverno mutate +policy** that lives with the platform, keyed off a marker label the template stamps only when the +user opts in. The template stays typed and simple; the platform grants the privilege-reducing +mapping. When the provider gains the field, the policy is deleted and the template sets it +directly — a one-line change, not a redesign. + +## How the pieces compose + +```mermaid +flowchart TB + subgraph image [Image · baked plumbing] + I["rootless dockerd + CLI
uidmap · slirp4netns · fuse-overlayfs
subuid/subgid for coder"] + end + subgraph template [Template · gated on enable_docker] + P["enable_docker parameter"] + M["marker label
com.coder.workspace.docker-enabled=true"] + V["docker-data PVC + XDG_RUNTIME_DIR
DOCKER_HOST env"] + S["agent startup:
start rootless dockerd"] + end + subgraph platform [Platform · apps repo] + K["Kyverno mutate policy
injects spec.hostUsers=false"] + end + Pod["Unprivileged workspace pod
+ user namespace"] + Kind["docker → kind clusters"] + + I --> Pod + P --> M --> Pod + P --> V --> Pod + P --> S --> Pod + M -. matched by .-> K -. mutates .-> Pod + Pod --> Kind +``` + +The division of labour: the **image** carries the universal, stable plumbing (baked so startup is +fast and reproducible — the [DESIGN.md](DESIGN.md#where-the-workspace-environment-comes-from) +layering rule); the **template** turns it on per-workspace and provides the pod-local storage and +runtime wiring; the **platform policy** supplies the one thing the template can't express. `kind` +itself is *not* baked into the image — it's day-to-day tooling and comes from the operator's +dotfiles (aqua), per the same layering rule. + +## Trade-offs and things to know + +- **Opt-in, not baseline.** The plumbing is baked into every image (it must be — the setuid + `newuidmap`/`newgidmap` and subuid/subgid ranges can only be set at build time as root), but the + *behaviour* (daemon start, the PVC, `hostUsers` mutation) is gated behind `enable_docker`. A + workspace that doesn't opt in is byte-for-byte the same runtime posture as before. +- **Storage is a dedicated PVC, and survives stop/start.** Docker's image/layer store and `kind` + node images are large and worth keeping between sessions, so they live on their own + `ReadWriteOnce` block PVC rather than the shared home volume or an `emptyDir`. Crucially it is + gated on `enable_docker` but **not** on `start_count`, mirroring the external home PVC — tying it + to `start_count` would wipe every pulled image on each workspace *stop*. +- **Storage driver.** With a user namespace and a modern kernel, native `overlay2` works and needs + no `/dev/fuse` (which the unprivileged pod lacks); `fuse-overlayfs` is the fallback, `vfs` the + last resort. The daemon auto-detects. +- **Degrades, doesn't break.** If the Kyverno policy is absent or fails open, the pod still runs — + just without the `hostUsers` mapping (weaker isolation). The marker label and the policy are a + cross-repo contract; neither should be renamed in isolation. +- **No nested cgroup resource limits.** Rootless dockerd in a pod with no user systemd session + runs without delegated cgroup limits; the workspace pod's own Kubernetes limits still cap + everything, which is fine for a single operator. + +## Future: migrating the cluster runtime + +This design is deliberately runtime-agnostic and should *ease* a future migration to a different +Kubernetes distro rather than block it: + +- The image and startup scripts are not tied to the current distro; they carry over unchanged. +- User namespaces (`hostUsers: false`) are a standard Kubernetes feature; the target distro + already enables the necessary kubelet/runtime support, so the same mechanism applies. +- The Kyverno policy is platform-level and portable as-is. +- The one thing to re-check at migration time is that the docker-data PVC's storage class exists on + the new cluster (or swap it for the equivalent). + +The `hostUsers`-via-policy indirection is the only piece expected to be temporary: once the +Terraform provider exposes the field, the template sets it directly and the policy retires. diff --git a/DESIGN.md b/DESIGN.md index 4a384204..979bd268 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -63,6 +63,8 @@ The rule that ties the layers together: a package or tool belongs in the *lowest **Unprivileged by default.** The workspace itself runs as an unprivileged, non-root, fixed-identity container. Anything that genuinely needs elevated privilege (installing packages, preparing shared volume state) is scoped to a narrow, short-lived setup step that runs before the workspace shell exists, not to something the workspace user can reach into. +**Nested containers without giving up that posture.** Running Docker and `kind` *inside* the workspace would normally mean a privileged pod or a node-level runtime — both rejected here. Instead an opt-in rootless-Docker setup keeps the pod unprivileged and socket-free, with user-namespace isolation supplied by a platform Kyverno policy. This is a large enough topic to have its own design doc — see [DESIGN-DIND.md](DESIGN-DIND.md), which also covers the cluster-runtime migration angle. + ## Outcomes targeted - One operator can keep dependencies current and ship template/image changes at low ongoing effort, without a fleet of environments to maintain. diff --git a/README.md b/README.md index bf3808ed..4b60c2c9 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ This repo is the middle of a larger stack: the Coder control plane is deployed s ## Docs - **[DESIGN.md](DESIGN.md)** — why the template and image are built the way they are: trade-offs considered, decisions made, targeted outcomes. +- **[DESIGN-DIND.md](DESIGN-DIND.md)** — how the workspace runs Docker and `kind` while staying an unprivileged pod. - **[TESTING.md](TESTING.md)** — how to validate a change, including exercising it against the real cluster without touching production workspace data. - **[CLAUDE.md](CLAUDE.md)** — commands and conventions for working in this repo with Claude Code. - **[CHANGELOG.md](CHANGELOG.md)** — generated release history (semantic-release). diff --git a/TESTING.md b/TESTING.md index 00d369cd..c838e126 100644 --- a/TESTING.md +++ b/TESTING.md @@ -23,6 +23,17 @@ Both modes run the identical sequence of stages; only what each stage is permitt Because all three stages run for real in dry-run — just scoped away from production — a passing PR is a meaningful signal that a live release would also succeed, not a guess based on static checks alone. +## Exercising the rootless Docker feature + +The `enable_docker` feature (see [DESIGN-DIND.md](DESIGN-DIND.md)) can't be judged by lint alone — it needs a live workspace. After the dry-run pipeline builds the image and pushes the disposable template, provision a test workspace from that template with **`enable_docker = true`** and check, from a terminal in the workspace: + +- `docker info` — the daemon is up and rootless (`Cgroup Driver` / `rootless: true`), storage driver is `overlay2` (or the `fuse-overlayfs`/`vfs` fallback). +- `docker run --rm hello-world` — a container actually runs. +- `kind create cluster` (kind comes from dotfiles) then `kubectl get nodes` — a nested cluster comes up; tear it down with `kind delete cluster`. +- `kubectl get pod -o jsonpath='{.spec.hostUsers}'` (from a client with cluster access) returns `false` — confirming the platform Kyverno policy mutated the pod. If it's empty, the policy side isn't in effect and Docker is running in the weaker posture. + +The Kyverno policy half lives in the platform repo and has its own chainsaw coverage there (it asserts a marked pod is mutated and an unmarked one is left alone) — a template change here that renames the `com.coder.workspace.docker-enabled` marker must be matched there. + ## After merge Merging to `main` is what flips the pipeline into live mode — there's no separate promotion step afterward. The dry-run pass on the PR is the actual release gate. diff --git a/images/homelab-workspace/Dockerfile b/images/homelab-workspace/Dockerfile index 04acfc3b..c47a4e73 100644 --- a/images/homelab-workspace/Dockerfile +++ b/images/homelab-workspace/Dockerfile @@ -113,6 +113,53 @@ RUN --mount=type=cache,target=/var/cache/apt,id=cache-apt-${TARGETARCH},sharing= # Clean pycache created during apt-get install (as apt stills retains some crud in spite of PYTHONPYCACHEPREFIX) find /usr -name __pycache__ -exec rm -rf {} + +# ======================================================================================================== +# Rootless Docker support (the runtime kind/Kubernetes-in-Docker relies on; kind itself is installed by the +# user via aqua in their dotfiles, not baked into the image). +# Kept as a self-contained block (separate from the general toolset above) so it can later be lifted into a +# dedicated variant image without untangling it from the base package list. +# - uidmap: provides newuidmap/newgidmap (setuid-root) needed to map subordinate uid/gid ranges +# - slirp4netns: userspace network stack for rootless dockerd (no CAP_NET_ADMIN required) +# - fuse-overlayfs: fallback storage driver; native overlay2 (userns + kernel>=5.11) is the primary path, +# used when /dev/fuse is unavailable (the unprivileged pod has no /dev/fuse) +# - iptables: required by dockerd/kind for bridge + port-mapping rules +# We use Docker's official apt repo (not Ubuntu's docker.io) so the rootless entrypoints ship in the +# docker-ce-rootless-extras package (dockerd-rootless.sh / dockerd-rootless-setuptool.sh). +RUN --mount=type=cache,target=/var/cache/apt,id=cache-apt-${TARGETARCH},sharing=locked \ + --mount=type=cache,target=/var/cache/debconf,id=cache-debconf-${TARGETARCH},sharing=locked \ + --mount=type=cache,target=/var/lib/apt,id=lib-apt-${TARGETARCH},sharing=locked \ + --mount=type=tmpfs,target=/var/cache/python \ + --mount=type=tmpfs,target=/tmp \ + --mount=type=tmpfs,target=/var/log \ + --mount=type=tmpfs,target=/var/tmp \ + # rootless-docker runtime dependencies (from Ubuntu repos) + apt-get update && \ + DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -yq \ + fuse-overlayfs \ + iptables \ + slirp4netns \ + uidmap \ + && \ + # add Docker's official apt repository + signing key (arch matched to the build platform) + install -m 0755 -d /etc/apt/keyrings && \ + curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc && \ + chmod a+r /etc/apt/keyrings/docker.asc && \ + echo "deb [arch=${TARGETARCH} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "${VERSION_CODENAME}") stable" \ + > /etc/apt/sources.list.d/docker.list && \ + # Docker engine + CLI + buildx + containerd. docker-ce-rootless-extras is a + # SEPARATE package (not pulled in by docker-ce) that ships dockerd-rootless.sh + # and dockerd-rootless-setuptool.sh -- the rootless entrypoints this image needs. + apt-get update && \ + DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -yq \ + containerd.io \ + docker-buildx-plugin \ + docker-ce \ + docker-ce-cli \ + docker-ce-rootless-extras \ + && \ + # Clean pycache created during apt-get install (as apt stills retains some crud in spite of PYTHONPYCACHEPREFIX) + find /usr -name __pycache__ -exec rm -rf {} + + # ======================================================================================================== FROM system-base SHELL ["/bin/bash", "-o", "pipefail", "-c"] @@ -142,5 +189,16 @@ RUN groupadd --gid "${CODER_GID}" coder && \ # package installation is complete, users of resulting image need not cache downloaded packages rm -f /etc/apt/apt.conf.d/keep-cache +# Subordinate uid/gid ranges for 'coder', sized to fit INSIDE a user-namespaced pod. +# When the workspace runs with hostUsers=false (see DESIGN-DIND.md) the pod's userns is only +# 65536 IDs wide (0-65535), and newuidmap maps subordinate IDs relative to that namespace. +# useradd's default range (coder:165536:65536) points outside it, so rootlesskit fails with +# "newuidmap: write to uid_map failed: Operation not permitted". We therefore OVERWRITE (not +# append) with a range that fits above coder's own UID 10001: 10002..65535 -> 55534 subordinate +# IDs, enough for nested rootless containers. newuidmap/newgidmap must stay setuid-root. +RUN echo 'coder:10002:55534' > /etc/subuid && \ + echo 'coder:10002:55534' > /etc/subgid && \ + test -u /usr/bin/newuidmap && test -u /usr/bin/newgidmap + USER coder WORKDIR /home/coder diff --git a/templates/kubernetes/homelab-workspace/configmap.tf b/templates/kubernetes/homelab-workspace/configmap.tf index b17c0f49..2c07f2c4 100644 --- a/templates/kubernetes/homelab-workspace/configmap.tf +++ b/templates/kubernetes/homelab-workspace/configmap.tf @@ -8,8 +8,8 @@ resource "kubernetes_config_map_v1" "workspace_scripts" { } data = { - agent_startup_script = file("${path.cwd}/script-agent-startup.sh") - prepare_workspace_script = file("${path.cwd}/script-prepare-workspace.sh") - workspace_init_script = coder_agent.main.init_script + agent_startup_script = file("${path.cwd}/script-agent-startup.sh") + workspace_entrypoint_script = file("${path.cwd}/script-workspace-entrypoint.sh") + workspace_init_script = coder_agent.main.init_script } } diff --git a/templates/kubernetes/homelab-workspace/deployment.tf b/templates/kubernetes/homelab-workspace/deployment.tf index f4e22cd3..7e5260b2 100644 --- a/templates/kubernetes/homelab-workspace/deployment.tf +++ b/templates/kubernetes/homelab-workspace/deployment.tf @@ -22,7 +22,11 @@ resource "kubernetes_deployment_v1" "deployment" { template { metadata { - labels = merge(local.common_labels, local.pod_labels) + # The docker-enabled marker is what the platform Kyverno policy matches to + # inject `hostUsers: false` (the kubernetes provider has no host_users field). + # Docker is always enabled now, so the label is always present. hostUsers:false + # is what makes the root/CAP_SYS_ADMIN container below safe (namespaced root). + labels = merge(local.common_labels, local.pod_labels, { "com.coder.workspace.docker-enabled" = "true" }) } spec { dynamic "affinity" { @@ -43,10 +47,22 @@ resource "kubernetes_deployment_v1" "deployment" { } } automount_service_account_token = false - init_container { - name = "prepare-workspace" - command = ["/bin/bash", "/prepare-workspace-script.sh"] + container { + name = "workspace" + # Entrypoint runs as root: prepares the workspace (apt/homebrew), starts + # dockerd, then drops to the coder user to exec the agent. This replaces the + # old init-container + system-volume dance, which only existed to shuttle + # root-made changes into a non-root container -- unnecessary now. + command = ["/bin/bash", "/workspace-entrypoint.sh"] image = var.workspace_image + # In test_mode the image is a mutable branch tag that gets rebuilt in place, + # so a cached layer must not shadow a fresh push -> Always. Released versions + # use immutable tags where IfNotPresent is correct (and avoids needless pulls). + image_pull_policy = var.test_mode ? "Always" : "IfNotPresent" + env { + name = "CODER_AGENT_TOKEN" + value = coder_agent.main.token + } env { name = "SYSTEM_PACKAGES" value = length(local.validated_system_packages) > 0 ? join(" ", local.validated_system_packages) : "NONE" @@ -55,37 +71,6 @@ resource "kubernetes_deployment_v1" "deployment" { name = "HOMEBREW_PREFIX" value = local.homebrew_directory } - volume_mount { - mount_path = local.home_directory - name = "home" - sub_path = data.coder_workspace.me.name - } - volume_mount { - mount_path = local.homebrew_directory - name = "home" - sub_path = "${data.coder_workspace.me.name}/.linuxbrew" - } - volume_mount { - mount_path = "/prepare-workspace-script.sh" - name = "coder-scripts" - sub_path = "prepare_workspace_script" - } - volume_mount { - name = "system" - mount_path = "/updated" - } - security_context { - run_as_user = 0 - } - } - container { - name = "workspace" - command = ["/bin/bash", "/workspace-init.sh"] - image = var.workspace_image - env { - name = "CODER_AGENT_TOKEN" - value = coder_agent.main.token - } liveness_probe { exec { command = ["/bin/sh", "-c", "pgrep -f \"coder agent\" || exit 1"] @@ -105,12 +90,23 @@ resource "kubernetes_deployment_v1" "deployment" { } } security_context { - allow_privilege_escalation = false + # Root INSIDE the pod user namespace (hostUsers:false, injected by the + # platform Kyverno policy via the marker label above). Namespaced root maps + # to an unprivileged host uid, so even privileged is void on the host. + # + # privileged=true (TEST): the cri-containerd AppArmor profile default-denies + # the mount() syscall, which blocks containerd's overlayfs snapshotter from + # bind/overlay-mounting image layers (confirmed: fails on an ext4 emptyDir data + # root, so it is NOT overlay-on-overlay -- it is AppArmor). privileged sets the + # profile to unconfined (and grants all caps), which should unblock it. If this + # works, the durable fix is appArmorProfile:Unconfined via the Kyverno policy + # (the provider has no app_armor_profile field), keeping caps narrow. + allow_privilege_escalation = true read_only_root_filesystem = false - privileged = false - run_as_user = 10001 - run_as_group = 10001 - run_as_non_root = true + privileged = true + run_as_user = 0 + run_as_group = 0 + run_as_non_root = false } volume_mount { mount_path = local.home_directory @@ -132,24 +128,23 @@ resource "kubernetes_deployment_v1" "deployment" { sub_path = "agent_startup_script" } volume_mount { - mount_path = "/workspace-init.sh" + mount_path = "/workspace-entrypoint.sh" name = "coder-scripts" - sub_path = "workspace_init_script" + sub_path = "workspace_entrypoint_script" } volume_mount { - mount_path = "/usr" - name = "system" - sub_path = "usr" - } - volume_mount { - mount_path = "/etc" - name = "system" - sub_path = "etc" + mount_path = "/workspace-init.sh" + name = "coder-scripts" + sub_path = "workspace_init_script" } volume_mount { - mount_path = "/var" - name = "system" - sub_path = "var" + # Docker data root on a kubelet-mounted emptyDir: a plain (non-overlay) + # filesystem, so containerd's overlayfs snapshotter isn't stacking overlay + # on overlay (which the container rootfs would otherwise force). The kubelet + # sets this mount up outside the container, so it does not hit the in-container + # AppArmor mount restrictions. NOTE: emptyDir does not persist across stop. + mount_path = "/var/lib/docker" + name = "docker-data" } } enable_service_links = false @@ -159,9 +154,11 @@ resource "kubernetes_deployment_v1" "deployment" { "kubernetes.io/arch" = "amd64" } security_context { - run_as_user = 10001 - run_as_group = 10001 - run_as_non_root = true + # Pod-level root, matching the container. Namespaced by hostUsers:false. + # fs_group stays 10001 so the coder user owns its home PVC contents. + run_as_user = 0 + run_as_group = 0 + run_as_non_root = false fs_group = 10001 fs_group_change_policy = "OnRootMismatch" } @@ -192,9 +189,13 @@ resource "kubernetes_deployment_v1" "deployment" { } } volume { - name = "system" + # Docker data root. emptyDir (not PVC) so it is a plain fs the kubelet + # mounts outside the container -- avoids overlay-on-overlay and the locked + # -mount / in-container AppArmor problems. Persistence across stop is a + # follow-up once the runtime is confirmed working. + name = "docker-data" empty_dir { - size_limit = "10Gi" + size_limit = "40Gi" } } } diff --git a/templates/kubernetes/homelab-workspace/env.tf b/templates/kubernetes/homelab-workspace/env.tf index 4b7449db..0fe783db 100644 --- a/templates/kubernetes/homelab-workspace/env.tf +++ b/templates/kubernetes/homelab-workspace/env.tf @@ -3,3 +3,7 @@ resource "coder_env" "welcome_message" { name = "HOMEBREW_PREFIX" value = local.homebrew_directory } + +# Rootful dockerd listens on the default /var/run/docker.sock; the coder user +# reaches it via the `docker` group (see script-workspace-entrypoint.sh), so no +# DOCKER_HOST override is needed. diff --git a/templates/kubernetes/homelab-workspace/script-agent-startup.sh b/templates/kubernetes/homelab-workspace/script-agent-startup.sh index 957175b2..bdf3e7a2 100644 --- a/templates/kubernetes/homelab-workspace/script-agent-startup.sh +++ b/templates/kubernetes/homelab-workspace/script-agent-startup.sh @@ -1,7 +1,9 @@ #!/bin/bash +# Runs as the `coder` user via the Coder agent's startup_script hook, after the +# agent is up. This is the place for per-user setup that must run unprivileged. +# (Docker + system prep now happen earlier, as root, in script-workspace-entrypoint.sh.) set -eo pipefail - main() { if [[ ! -s ~/.bashrc ]]; then echo "Setting up starter bash rc scripts from /etc/skel..." diff --git a/templates/kubernetes/homelab-workspace/script-prepare-workspace.sh b/templates/kubernetes/homelab-workspace/script-prepare-workspace.sh deleted file mode 100755 index 3140d1ee..00000000 --- a/templates/kubernetes/homelab-workspace/script-prepare-workspace.sh +++ /dev/null @@ -1,117 +0,0 @@ -#!/bin/bash -set -euo pipefail - -install_homebrew() { - local brew_user="$1" - export HOMEBREW_CELLAR="${HOMEBREW_PREFIX}/Cellar" - export HOMEBREW_NO_ANALYTICS=1 - export PATH="${HOMEBREW_PREFIX}/bin:${HOMEBREW_PREFIX}/sbin:$PATH" - # init container runs as root but homebrew doesn't support running as root - # we don't want to give any users within the coder workspace the ability to sudo, so this is a workaround - # see: https://github.com/Homebrew/install/blob/7e3a5202cd6d783a2464e387433c4c72acdb0f49/install.sh#L366 - touch /.dockerenv - HOME="/home/${brew_user}" NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" -} - -cleanup_homebrew() { - echo "Cleaning up..." - echo " Cache..." - find "${HOMEBREW_CACHE}/" -mindepth 1 -maxdepth 1 -exec rm -rf {} \; || true - echo " Prefix..." - find "${HOMEBREW_PREFIX}/" -mindepth 1 -maxdepth 1 -exec rm -rf {} \; || true -} - -prepare_homebrew() { - echo "Creating directories and setting permissions..." - if [[ ! -d "${HOMEBREW_CACHE}" ]]; then - echo "Creating cache directory..." - mkdir -p "${HOMEBREW_CACHE}" - fi - chown -R ${brew_user}:root "${HOMEBREW_CACHE}" - chown -R ${brew_user}:root $(dirname "${HOMEBREW_PREFIX}") -} - -ensure_permissions() { - local brew_user="$1" - echo "Making brew installation available to both ${brew_user} user and root..." - chown -R ${brew_user}:root $(dirname "${HOMEBREW_PREFIX}") - chown -R ${brew_user}:root "${HOMEBREW_CACHE}" - echo "Ensuring ${brew_user} user retains ownership of top level directories within their home..." - chown ${brew_user}:${brew_user} /home/${brew_user} - chown ${brew_user}:${brew_user} /home/${brew_user}/.cache -} - -setup_homebrew() { - local brew_user="coder" - export HOMEBREW_CACHE="/home/${brew_user}/.cache/Homebrew" - - echo '------------------------------------------------------------' - echo 'Checking for brew installation...' - local brew_file_count=$(find "${HOMEBREW_PREFIX}" -maxdepth 3 -type f | wc -l) - if [[ "${brew_file_count}" -gt "0" && -f "${HOMEBREW_PREFIX}/bin/brew" ]]; then - echo 'Brew installation already exists... skipping.' - else - echo 'No existing brew installation, proceeding w/ installation...' - echo '------------------------------------------------------------' - echo 'Preparing for homebrew...' - cleanup_homebrew 2>&1 | sed -E -n 's|^| |p' - prepare_homebrew 2>&1 | sed -E -n 's|^| |p' - echo '------------------------------------------------------------' - echo 'Starting homebrew installation...' - install_homebrew "${brew_user}" 2>&1 | sed -E -n 's|^| |p' - fi - echo '------------------------------------------------------------' - echo "Ensuring directory permissions..." - ensure_permissions "${brew_user}" 2>&1 | sed -E -n 's|^| |p' - echo '------------------------------------------------------------' - echo 'Done' -} - -setup_system_packages() { - echo '------------------------------------------------------------' - echo 'Running apt-get update...' - apt-get update 2>&1 | sed -E -n 's|^| |p' - echo 'Running apt-file update...' - apt-file update 2>&1 | sed -E -n 's|^| |p' - echo '------------------------------------------------------------' - echo 'Installing additinal apt packages...' - echo " Packages: $SYSTEM_PACKAGES" - echo - if [[ "$SYSTEM_PACKAGES" != "NONE" ]]; then - DEBIAN_FRONTEND="noninteractive" apt-get install --no-install-recommends -yq $SYSTEM_PACKAGES 2>&1 | sed -E -n 's|^| |p' - fi - echo '------------------------------------------------------------' - echo 'Rsyncing (etc, usr, var) to /updated...' - rsync -aH --stats /etc /usr /var /updated 2>&1 | sed -E -n 's|^| |p' - echo '------------------------------------------------------------' - echo 'Updated system size...' - du -h -d 1 /updated 2>&1 | sed -E -n 's|^| |p' - echo '------------------------------------------------------------' - echo 'Done' -} - -prepare_environment() { - echo '------------------------------------------------------------' - grep -v '^PATH=' /etc/environment > /tmp/environment.bak - local existing_system_path="$(grep '^PATH=' /etc/environment | cut -d'=' -f2)" - local updated_system_path="${HOMEBREW_PREFIX}/bin:${HOMEBREW_PREFIX}/sbin:${existing_system_path}" - echo PATH=${updated_system_path} >> /tmp/environment.bak - sort /tmp/environment.bak > /updated/etc/environment - rm /tmp/environment.bak - cat /updated/etc/environment - echo '------------------------------------------------------------' - echo 'Done' -} - -main() { - echo "Setting up system packages..." - setup_system_packages | sed -E -n 's|^| |p' - echo - echo "Setting up homebrew..." - setup_homebrew | sed -E -n 's|^| |p' - echo - echo "Setting up workspace environment variables..." - prepare_environment | sed -E -n 's|^| |p' -} - -main diff --git a/templates/kubernetes/homelab-workspace/script-workspace-entrypoint.sh b/templates/kubernetes/homelab-workspace/script-workspace-entrypoint.sh new file mode 100644 index 00000000..ba9fe080 --- /dev/null +++ b/templates/kubernetes/homelab-workspace/script-workspace-entrypoint.sh @@ -0,0 +1,135 @@ +#!/bin/bash +# Main container entrypoint. Runs as root (uid 0) INSIDE the pod's user +# namespace (hostUsers=false), so "root" here maps to an unprivileged uid on the +# host -- see DESIGN-DIND.md. Responsibilities, in order: +# 1. prepare the workspace (extra apt packages, Homebrew, PATH) -- previously an +# init container; now done here directly because this container is already root. +# 2. start a rootful dockerd (non-fatal: the workspace must come up even if it fails). +# 3. drop to the unprivileged `coder` user and exec the Coder agent, so everything +# the user interacts with runs as `coder` exactly as before. +set -eo pipefail + +CODER_USER="coder" +CODER_UID="10001" +CODER_GID="10001" +HOMEBREW_PREFIX="${HOMEBREW_PREFIX:-/home/linuxbrew/.linuxbrew}" + +# -------------------------------------------------------------------------------- +install_homebrew() { + export HOMEBREW_CELLAR="${HOMEBREW_PREFIX}/Cellar" + export HOMEBREW_NO_ANALYTICS=1 + export PATH="${HOMEBREW_PREFIX}/bin:${HOMEBREW_PREFIX}/sbin:$PATH" + # Homebrew refuses to install as root; run its installer as the coder user. + # (/.dockerenv makes the installer treat this as a container and skip sudo prompts.) + touch /.dockerenv + setpriv --reuid "${CODER_UID}" --regid "${CODER_GID}" --init-groups \ + env HOME="/home/${CODER_USER}" NONINTERACTIVE=1 \ + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" +} + +setup_homebrew() { + export HOMEBREW_CACHE="/home/${CODER_USER}/.cache/Homebrew" + echo 'Checking for brew installation...' + local brew_file_count + brew_file_count=$(find "${HOMEBREW_PREFIX}" -maxdepth 3 -type f 2> /dev/null | wc -l) + if [[ "${brew_file_count}" -gt "0" && -f "${HOMEBREW_PREFIX}/bin/brew" ]]; then + echo 'Brew installation already exists... skipping.' + else + echo 'No existing brew installation, proceeding w/ installation...' + mkdir -p "${HOMEBREW_CACHE}" + install_homebrew 2>&1 | sed -E -n 's|^| |p' + fi + echo 'Ensuring ownership of brew + home top-level dirs by coder...' + chown -R "${CODER_USER}:root" "$(dirname "${HOMEBREW_PREFIX}")" "${HOMEBREW_CACHE}" + chown "${CODER_USER}:${CODER_USER}" "/home/${CODER_USER}" "/home/${CODER_USER}/.cache" +} + +setup_system_packages() { + echo "Installing additional apt packages: ${SYSTEM_PACKAGES:-NONE}" + if [[ "${SYSTEM_PACKAGES:-NONE}" != "NONE" ]]; then + apt-get update 2>&1 | sed -E -n 's|^| |p' + apt-file update 2>&1 | sed -E -n 's|^| |p' || true + # shellcheck disable=SC2086 + DEBIAN_FRONTEND="noninteractive" apt-get install --no-install-recommends -yq ${SYSTEM_PACKAGES} 2>&1 | sed -E -n 's|^| |p' + fi +} + +prepare_environment() { + # Extend the system PATH (in /etc/environment) with Homebrew, in place -- no + # /updated staging volume any more since this container owns its own rootfs. + local existing updated + existing="$(grep '^PATH=' /etc/environment | cut -d'=' -f2)" + updated="${HOMEBREW_PREFIX}/bin:${HOMEBREW_PREFIX}/sbin:${existing}" + grep -v '^PATH=' /etc/environment > /tmp/environment.new + echo "PATH=${updated}" >> /tmp/environment.new + sort /tmp/environment.new > /etc/environment + rm -f /tmp/environment.new +} + +# -------------------------------------------------------------------------------- +setup_docker() { + if ! command -v dockerd > /dev/null 2>&1; then + echo 'dockerd not found in image; skipping docker setup.' + return 1 + fi + # Let the coder user reach the daemon socket. + groupadd --force docker + usermod --append --groups docker "${CODER_USER}" + + # /var/lib/docker is a kubelet-mounted emptyDir (see deployment.tf): a plain, + # non-overlay filesystem, so containerd's overlayfs snapshotter is not trying to + # stack overlay-on-overlay (which is what the container rootfs would force). The + # dir must be root:root 0711 -- dockerd rejects a group/world-accessible data root. + chown root:root /var/lib/docker + chmod 0711 /var/lib/docker + + echo 'Starting dockerd...' + # Rootful dockerd. CAP_SYS_ADMIN/NET_ADMIN (granted on the container, valid only + # inside the userns) cover what it needs. overlay2 is the default/expected driver. + setsid dockerd > /var/log/dockerd.log 2>&1 & + + echo 'Waiting up to ~30s for docker to become ready...' + local waited=0 + while [[ "${waited}" -lt 30 ]]; do + waited=$((waited + 1)) + if docker info > /dev/null 2>&1; then + echo 'Docker is ready.' + # socket is created 0660 root:docker; ensure the group can use it + chgrp docker /var/run/docker.sock 2> /dev/null || true + chmod 660 /var/run/docker.sock 2> /dev/null || true + return 0 + fi + sleep 1 + done + echo 'Docker did not become ready; see /var/log/dockerd.log' + return 1 +} + +# -------------------------------------------------------------------------------- +main() { + echo '=== workspace entrypoint (root) ===' + + echo '--- system packages ---' + setup_system_packages | sed -E -n 's|^| |p' || echo ' (system package setup failed; continuing)' + + echo '--- homebrew ---' + setup_homebrew 2>&1 | sed -E -n 's|^| |p' || echo ' (homebrew setup failed; continuing)' + + echo '--- environment ---' + prepare_environment || echo ' (environment prep failed; continuing)' + + echo '--- docker ---' + # Non-fatal by design: a docker failure must never stop the workspace coming up. + setup_docker 2>&1 | sed -E -n 's|^| |p' || echo ' (docker setup failed; workspace continues without docker)' + + echo '--- starting coder agent as coder ---' + # Drop from root to the coder user and hand off to the agent. exec so the agent + # becomes the container's main process (as it was when the container ran as coder). + # --init-groups picks up the docker group added above. Preserve the agent env. + exec setpriv --reuid "${CODER_UID}" --regid "${CODER_GID}" --init-groups \ + env HOME="/home/${CODER_USER}" USER="${CODER_USER}" \ + CODER_AGENT_TOKEN="${CODER_AGENT_TOKEN}" \ + /bin/bash /workspace-init.sh +} + +main