diff --git a/deploy/no-envbox-kvm/NOTES.md b/deploy/no-envbox-kvm/NOTES.md new file mode 100644 index 0000000..ab7730f --- /dev/null +++ b/deploy/no-envbox-kvm/NOTES.md @@ -0,0 +1,90 @@ +# Notes: open questions and mapping to existing envbox code + +## Make-or-break tests (run these before committing to the approach) + +1. **Does `/dev/kvm` exist and work in the target node group?** + On the node: `ls -l /dev/kvm` and a `cloud-hypervisor`/`qemu -enable-kvm` + smoke boot. On AWS this means `*.metal` or `c8i/m8i/r8i` + nested virt. + +2. **Does the device-plugin path avoid `privileged` for the workspace pod?** + Confirm a pod requesting `devices.kubevirt.io/kvm` can open `/dev/kvm` and + boot a guest with NO `privileged: true` and NO `hostPID/hostNetwork`. + +3. **Bottlerocket: does `super_t` + device plugin cover the VMM's syscalls?** + Boot the VMM pod on a Bottlerocket node and watch `dmesg`/journal for + SELinux AVC denials on the KVM ioctls, `mmap`, and tun ops. If `super_t` + plus the device plugin covers them -> first-class on stock Bottlerocket. + If a specific op is still denied -> a custom Bottlerocket variant with a + purpose-built SELinux type is required (heavier; reintroduces OS-image + maintenance). + +## Mapping to today's envbox code (what changes / goes away) + +| Envbox today | No-envbox + KVM | +| --- | --- | +| `cli/docker.go` starts sysbox + inner dockerd via `sysbox-runc` | Replaced by an in-pod supervisor that launches the VMM and boots the guest. | +| `xunix/device.go` `mknod`s `/dev/net/tun`, `/dev/fuse` under privileged | Devices come from the device plugin as pod resource requests. | +| `envboxPrivateMounts`: `/lib/modules`, `/usr/src`, `/var/lib/sysbox` | Not needed. The guest ships its own kernel/modules; no host kernel matrix. | +| `--privileged` outer pod | Unprivileged pod + scoped device request. | +| Inner dockerd image store (cold cache) | Node pull-through mirror (new pods warm) + CSI-mounted inner image; ephemeral guest data dir. | +| `xunix/gpu.go` host-lib bind mounts + `CODER_ADD_GPU` | GPU passthrough into the guest (open question; `*.metal` likely first). | +| `CODER_INNER_*` env contract | Preserved at the supervisor boundary so templates barely change. | + +## Preserve the Coder-facing contract + +Keep `CODER_INNER_IMAGE`, `CODER_INNER_USERNAME`, `CODER_INNER_ENVS`, +`CODER_AGENT_TOKEN`, `CODER_MOUNTS`, etc. at the supervisor boundary so +existing envbox templates migrate with minimal edits. The isolation mechanism +changes underneath; the template surface should not. + +## Biggest unknowns (ranked) + +1. Guest rootfs build + boot time vs. envbox cold start (target: comparable or + better once the docker cache is warm). +2. Networking: virtio-net + NAT vs. the pod's CNI; making NetworkPolicy / + service mesh apply to guest traffic. +3. Passing PVC-backed volumes (home dir) into the guest with correct + ownership (virtiofs). +4. GPU passthrough into a nested-virt guest on non-metal families. +5. Graceful shutdown / signal + agent-token lifecycle into the guest. + +## Cache design (first-boot warmth for new pods) + +Goal: a brand-new workspace pod on a node that already has the layers must +**not** hit the remote registry. We do not care about fast restarts, so no +persistence. The blocker is that the inner dockerd's store is private and +format-incompatible with the node's containerd store, so you cannot bind-mount +the node store into dockerd. Two levers instead: + +**Lever 1 -- node-local pull-through mirror (arbitrary in-workspace pulls).** +Run a registry mirror as a DaemonSet with node-local storage and point every +guest dockerd at it (`CODER_REGISTRY_MIRROR`). The first pod on the node to +pull image X populates the mirror; every new pod after that pulls X from the +node at loopback/LAN speed. Pre-seed the mirror with a pre-pull Job/DaemonSet +if the very first pod on a cold node must also be warm. Format-agnostic and +safe (no node socket, no shared graph). + +**Lever 2 -- CSI image-mount for the inner workspace image.** +The biggest single first-boot cost is `CODER_INNER_IMAGE`, which envbox today +pulls with the inner dockerd (never touching the node cache). Instead, mount it +from the node's containerd store via a CSI image driver (e.g. warm-metal +`csi-image`), which pulls via CRI and mounts the image snapshot. A new pod gets +the already-present rootfs with no pull. This is the one clean way to actually +reuse the kubelet/containerd cache. + +**Why not just mount the store:** dockerd has no shared/read-only "additional +image store" concept, so mounting the node's layers read-only does nothing for +a fresh dockerd. Podman *does* have this (`additionalimagestores`); if the +inner runtime were podman/buildah, "mount a node-local read-only store and new +pods reuse it" would be first-class. Bigger change than the mirror; noted as an +option, not the plan. + +**Applies to both paths.** The guest in this KVM sketch points its dockerd at +the same node mirror (Lever 1) and consumes the CSI-mounted inner image +(Lever 2); the guest docker data dir is deliberately ephemeral. The identical +pattern works for the sysbox/vanilla outer pod. + +## Not doing (yet) + +No code wired into the envbox build. This branch is a deployment-surface +sketch to explore options and evaluate feasibility and node/OS constraints. diff --git a/deploy/no-envbox-kvm/README.md b/deploy/no-envbox-kvm/README.md new file mode 100644 index 0000000..2425e52 --- /dev/null +++ b/deploy/no-envbox-kvm/README.md @@ -0,0 +1,97 @@ +# No-envbox workspaces: mounted devices + KVM (exploration) + +> Status: **exploratory proof-of-concept**. Nothing here is wired into the +> envbox build or tested end to end. It exists to sketch what a workspace +> deployment that drops the envbox/sysbox wrapper in favor of a KVM-backed +> microVM (using only device requests, not a privileged outer pod) would +> look like, and to surface the gaps that still need filling. + +## Why + +Today envbox is a **privileged** outer pod that: + +- bundles and supervises the sysbox runtime (`sysbox-mgr`, `sysbox-fs`, + `sysbox-runc`), +- `mknod`s `/dev/net/tun` and `/dev/fuse` for the inner container from inside + the privileged container (see `xunix/device.go`), +- bind-mounts host `/lib/modules` and `/usr/src` to satisfy sysbox's kernel + requirements, +- runs an inner `dockerd` under `sysbox-runc` as the user's workspace. + +That model has four structural problems (see the gap analysis that motivated +this branch): + +1. **Cold image cache.** The inner dockerd has its own image store, isolated + from the node's containerd cache, so every inner pull is cold. +2. **Requires `privileged`.** Blocked by Pod Security Admission, Gatekeeper, + and most hardened clusters. +3. **Coupled to sysbox** (now community-best-effort) and a host kernel matrix + (`/lib/modules`, `/usr/src`). +4. **Not node-OS portable.** Cannot run on immutable/SELinux-locked node OSes + like Bottlerocket that forbid the host mounts and module loading it needs. + +## The idea + +Replace the "shared-kernel + sysbox" isolation model with a **microVM booted +inside an ordinary pod**. Instead of the pod being privileged and manufacturing +its own devices, the pod: + +- **requests `/dev/kvm`** (and `/dev/net/tun`) as scheduled resources from a + device plugin, rather than `mknod`ing them under `privileged`, and +- boots a lightweight VMM (Cloud Hypervisor / Firecracker) that runs a real + guest kernel; the user's `dockerd`/`systemd` runs **inside the guest**, where + overlayfs, DinD, and non-root images "just work" against a normal kernel. + +This is not a Kubernetes RuntimeClass (customers can't install those). The VMM +runs as a normal workload process inside the pod, so no cluster-runtime install +is required. The only node-level requirement is that `/dev/kvm` exists and is +exposed to the pod. + +``` + Before (envbox) After (no-envbox + KVM) + ┌───────────────────────────┐ ┌───────────────────────────┐ + │ privileged outer pod │ │ unprivileged pod │ + │ sysbox daemons │ │ requests devices.../kvm │ + │ ┌──────────────────────┐ │ │ ┌──────────────────────┐ │ + │ │ inner dockerd (sysbox│ │ │ │ VMM (cloud-hypervisor│ │ + │ │ -runc), shared kernel│ │ │ │ / firecracker) │ │ + │ │ user workspace │ │ │ │ ┌────────────────┐ │ │ + │ └──────────────────────┘ │ │ │ │ guest kernel │ │ │ + └───────────────────────────┘ │ │ │ dockerd/systemd│ │ │ + needs: privileged, │ │ │ user workspace │ │ │ + /lib/modules, /usr/src, │ │ └────────────────┘ │ │ + sysbox, host kernel matrix │ └──────────────────────┘ │ + └───────────────────────────┘ + needs: /dev/kvm exposed +``` + +## Node requirements + +`/dev/kvm` must exist on the node: + +- **AWS:** `*.metal` instances, or `C8i` / `M8i` / `R8i` with EC2 nested + virtualization enabled (GA 2026-02-16). Older virtualized families do not + expose VT-x/AMD-V and cannot host KVM. +- **GCP / Azure:** nested virtualization on supported machine types. +- **Bare metal:** native. + +## What's in this directory + +| File | Purpose | +| --- | --- | +| `manifests/kvm-device-plugin.yaml` | DaemonSet exposing `/dev/kvm` (+ `/dev/net/tun`) as schedulable resources, pinned to KVM-capable nodes. Not privileged for workloads; the plugin itself needs host device access. | +| `manifests/registry-mirror.yaml` | Node-local pull-through registry mirror (cache Lever 1): new pods pull already-fetched layers from the node instead of the remote registry. | +| `manifests/workspace-pod.yaml` | A no-envbox workspace pod: unprivileged, requests the device resources, boots a microVM, points its dockerd at the node mirror, and CSI-mounts the inner image (ephemeral docker data dir). | +| `manifests/coder-template.tf` | Terraform `kubernetes_pod` sketch for the equivalent Coder template. | +| `NOTES.md` | Open questions, the make-or-break tests, and mapping to the existing envbox code that would need to change. | + +## Explicitly out of scope for this sketch + +- The guest rootfs image (kernel + init + agent + dockerd) build. +- The in-pod supervisor that launches the VMM, forwards the Coder agent token, + wires vsock/virtio-net, and streams the build log (the analog of today's + `cli/docker.go` flow). +- GPU passthrough into the guest. + +These are the real engineering; this branch only sketches the deployment +surface so we can evaluate feasibility and node/OS constraints. diff --git a/deploy/no-envbox-kvm/manifests/coder-template.tf b/deploy/no-envbox-kvm/manifests/coder-template.tf new file mode 100644 index 0000000..4720322 --- /dev/null +++ b/deploy/no-envbox-kvm/manifests/coder-template.tf @@ -0,0 +1,87 @@ +// Coder template sketch: KVM-backed workspace pod (no envbox). +// +// This is the Terraform analog of manifests/workspace-pod.yaml, showing how a +// Coder template would provision the pod. It is intentionally trimmed to the +// bits that differ from the existing kubernetes-envbox template: no +// runtime_class_name, no privileged, device resources instead. +// +// SKETCH ONLY -- not runnable as-is. + +resource "kubernetes_pod" "workspace" { + count = data.coder_workspace.me.start_count + + metadata { + name = "coder-${data.coder_workspace_owner.me.name}-${data.coder_workspace.me.name}" + namespace = var.namespace + } + + spec { + # Land on KVM-capable nodes (*.metal or c8i/m8i/r8i + nested virt). + node_selector = { + "coder.com/kvm" = "true" + } + + toleration { + key = "coder.com/workspace-kvm" + operator = "Exists" + effect = "NoSchedule" + } + + # No runtime_class_name (customers can't install one). + # No privileged security_context. + security_context { + seccomp_profile { + type = "RuntimeDefault" + } + } + + container { + name = "workspace" + image = "REPLACE_ME/coder-workspace-vmm:latest" + + # KVM requested as a device, not via privileged. + resources { + requests = { + "devices.kubevirt.io/kvm" = "1" + "devices.kubevirt.io/tun" = "1" + "cpu" = "2" + "memory" = "4Gi" + } + limits = { + "devices.kubevirt.io/kvm" = "1" + "devices.kubevirt.io/tun" = "1" + "cpu" = "4" + "memory" = "8Gi" + } + } + + env { + name = "CODER_INNER_IMAGE" + value = data.coder_parameter.inner_image.value + } + # Point the guest dockerd at the node-local pull-through mirror so new + # pods pull from the node (warm), not the remote registry. See NOTES.md. + env { + name = "CODER_REGISTRY_MIRROR" + value = "http://kube-registry-mirror.kube-system.svc:5000" + } + env { + name = "CODER_AGENT_TOKEN" + value = coder_agent.main.token + } + + volume_mount { + # Ephemeral guest docker data dir. Warmth comes from the mirror + + # CSI-mounted inner image, not from persistence (see NOTES.md). + name = "docker-data" + mount_path = "/var/lib/coder/docker" + } + } + + # Ephemeral: we do not need fast restarts. + volume { + name = "docker-data" + empty_dir {} + } + } +} diff --git a/deploy/no-envbox-kvm/manifests/kvm-device-plugin.yaml b/deploy/no-envbox-kvm/manifests/kvm-device-plugin.yaml new file mode 100644 index 0000000..c90318d --- /dev/null +++ b/deploy/no-envbox-kvm/manifests/kvm-device-plugin.yaml @@ -0,0 +1,73 @@ +# KVM device plugin +# +# Exposes /dev/kvm (and /dev/net/tun) as *schedulable* resources so that +# workspace pods can request them WITHOUT running as privileged and without +# the pod having to mknod its own devices (as envbox does today in +# xunix/device.go). +# +# This is the piece that lets us drop `privileged: true`: a device plugin adds +# a specific device to the pod's cgroup allow-list and bind-mounts it, which is +# exactly the "one device, not all devices" grant that a Kubernetes +# securityContext cannot express on its own. +# +# Reference implementation: kubevirt/kubernetes-device-plugins (kvm, tun). +# Replace the image below with a built/pinned image. +# +# NOTE: the device plugin DaemonSet itself needs host device access (and, on +# Bottlerocket, the super_t SELinux label). That privilege is confined to this +# small, auditable component -- not to every workspace. +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: kvm-device-plugin + namespace: kube-system + labels: + app: kvm-device-plugin +spec: + selector: + matchLabels: + app: kvm-device-plugin + template: + metadata: + labels: + app: kvm-device-plugin + spec: + # Only schedule onto nodes that actually expose /dev/kvm. Label your + # KVM-capable node group (e.g. *.metal or c8i/m8i/r8i w/ nested virt) + # with `coder.com/kvm=true`. + nodeSelector: + coder.com/kvm: "true" + tolerations: + - key: coder.com/workspace-kvm + operator: Exists + effect: NoSchedule + containers: + - name: kvm-device-plugin + image: REPLACE_ME/kvm-device-plugin:latest + args: + - --devices=kvm,tun + securityContext: + # Device plugins need host device access. On Bottlerocket this also + # requires the super_t SELinux label (see NOTES.md); on general + # node OSes privileged is the pragmatic setting for the plugin. + privileged: true + # Bottlerocket equivalent (instead of privileged) -- uncomment and + # test which is sufficient: + # seLinuxOptions: + # user: system_u + # role: system_r + # type: super_t + # level: s0 + volumeMounts: + - name: device-plugin + mountPath: /var/lib/kubelet/device-plugins + - name: dev + mountPath: /dev + volumes: + - name: device-plugin + hostPath: + path: /var/lib/kubelet/device-plugins + - name: dev + hostPath: + path: /dev diff --git a/deploy/no-envbox-kvm/manifests/registry-mirror.yaml b/deploy/no-envbox-kvm/manifests/registry-mirror.yaml new file mode 100644 index 0000000..da015bf --- /dev/null +++ b/deploy/no-envbox-kvm/manifests/registry-mirror.yaml @@ -0,0 +1,70 @@ +# Lever 1: node-local pull-through registry mirror. +# +# Gives new workspace pods first-boot warmth: the first pod on a node to pull +# an image populates this mirror; every new pod after that pulls from the node +# (loopback/LAN) instead of the remote registry. Guest dockerds are pointed +# here via CODER_REGISTRY_MIRROR. See NOTES.md "Cache design". +# +# Storage is node-local hostPath -> the cache is per-node and shared across all +# pods on that node. This is NOT for restart persistence; it is a shared cache. +# +# Pre-seed with a pre-pull Job/DaemonSet if the very first pod on a cold node +# must also be warm. +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: kube-registry-mirror + namespace: kube-system + labels: + app: kube-registry-mirror +spec: + selector: + matchLabels: + app: kube-registry-mirror + template: + metadata: + labels: + app: kube-registry-mirror + spec: + # Only run where workspaces run. + nodeSelector: + coder.com/kvm: "true" + tolerations: + - key: coder.com/workspace-kvm + operator: Exists + effect: NoSchedule + hostNetwork: true # serve on a stable node-local address + containers: + - name: registry + image: registry:2 + env: + - name: REGISTRY_PROXY_REMOTEURL + value: https://registry-1.docker.io + - name: REGISTRY_HTTP_ADDR + value: 0.0.0.0:5000 + ports: + - containerPort: 5000 + hostPort: 5000 + volumeMounts: + - name: cache + mountPath: /var/lib/registry + volumes: + - name: cache + hostPath: + path: /var/cache/coder-registry-mirror + type: DirectoryOrCreate +--- +# Stable in-cluster name for CODER_REGISTRY_MIRROR. With hostNetwork the mirror +# also answers on :5000; a per-node address avoids cross-node hops. +apiVersion: v1 +kind: Service +metadata: + name: kube-registry-mirror + namespace: kube-system +spec: + selector: + app: kube-registry-mirror + ports: + - port: 5000 + targetPort: 5000 diff --git a/deploy/no-envbox-kvm/manifests/workspace-pod.yaml b/deploy/no-envbox-kvm/manifests/workspace-pod.yaml new file mode 100644 index 0000000..360d46c --- /dev/null +++ b/deploy/no-envbox-kvm/manifests/workspace-pod.yaml @@ -0,0 +1,111 @@ +# No-envbox KVM workspace pod: no envbox, no sysbox, no privileged. +# +# The pod is unprivileged. It requests /dev/kvm (and /dev/net/tun) from the +# device plugin and boots a microVM whose guest kernel runs the user's +# dockerd/systemd workspace. Contrast with today's envbox, which requires +# `privileged: true`, bind-mounts host /lib/modules + /usr/src, and runs an +# inner dockerd under sysbox-runc on the shared host kernel. +# +# This is a SKETCH. The `coder/workspace-vmm` image referenced below does not +# exist yet -- it stands in for the in-pod supervisor + guest rootfs described +# in README.md / NOTES.md. +--- +apiVersion: v1 +kind: Pod +metadata: + name: workspace-vmm-example + labels: + app.kubernetes.io/name: coder-workspace +spec: + # Land on a KVM-capable node and tolerate its taint. + nodeSelector: + coder.com/kvm: "true" + tolerations: + - key: coder.com/workspace-kvm + operator: Exists + effect: NoSchedule + restartPolicy: Never + # No hostPID/hostNetwork/hostIPC. No privileged. Compatible with the + # Pod Security "baseline" profile (the device request is the only elevated + # bit, and it is mediated by the device plugin). + securityContext: + runAsNonRoot: false # the VMM currently needs uid 0 in-pod; revisit. + seccompProfile: + type: RuntimeDefault + containers: + - name: workspace + # In-pod supervisor: launches cloud-hypervisor/firecracker, wires the + # guest's virtio-net + vsock, injects CODER_AGENT_TOKEN into the guest, + # and streams the boot/build log. Analog of today's cli/docker.go. + image: REPLACE_ME/coder-workspace-vmm:latest + args: + - --vmm=cloud-hypervisor + - --guest-image=/var/lib/coder/guest/rootfs.img + - --inner-image=$(CODER_INNER_IMAGE) + env: + - name: CODER_INNER_IMAGE + value: codercom/enterprise-base:ubuntu + - name: CODER_INNER_USERNAME + value: coder + # Lever 1: point the guest's dockerd at the node-local pull-through + # mirror so a NEW pod pulls layers from the node (warm) instead of the + # remote registry. See NOTES.md "Cache design". + - name: CODER_REGISTRY_MIRROR + value: http://kube-registry-mirror.kube-system.svc:5000 + - name: CODER_AGENT_TOKEN + valueFrom: + secretKeyRef: + name: workspace-agent-token + key: token + resources: + # KVM is requested here instead of `privileged: true`. + requests: + devices.kubevirt.io/kvm: "1" + devices.kubevirt.io/tun: "1" + cpu: "2" + memory: 4Gi + limits: + devices.kubevirt.io/kvm: "1" + devices.kubevirt.io/tun: "1" + cpu: "4" + memory: 8Gi + volumeMounts: + # The guest rootfs (kernel + init + agent + dockerd), shipped read-only. + - name: guest-image + mountPath: /var/lib/coder/guest + readOnly: true + # Lever 2: the inner workspace image (CODER_INNER_IMAGE) is mounted + # from the node's containerd store via a CSI image driver instead of + # being pulled by the guest's dockerd -> warm on first boot by reusing + # what the kubelet already cached. See NOTES.md "Cache design". + - name: inner-image + mountPath: /var/lib/coder/inner + readOnly: true + # Guest docker data dir. Intentionally EPHEMERAL: we do not need fast + # restarts. First-boot warmth comes from the mirror (Lever 1) and the + # CSI-mounted inner image (Lever 2), not from persisting this dir. + - name: docker-data + mountPath: /var/lib/coder/docker + # User's home / workspace PVC, passed through to the guest. + - name: home + mountPath: /home/coder + volumes: + - name: guest-image + # Ship the guest rootfs as an OCI image mounted read-only via an image + # volume / CSI image driver so the kubelet's node cache is reused. + emptyDir: {} # placeholder; real impl uses an image/CSI volume. + - name: inner-image + # CSI image-mount (e.g. warm-metal csi-image): pulls via CRI and mounts + # the image snapshot, sharing the node containerd store. A new pod gets + # the already-present rootfs with no pull. + csi: + driver: csi-image.warm-metal.tech + readOnly: true + volumeAttributes: + image: codercom/enterprise-base:ubuntu + - name: docker-data + # Ephemeral on purpose (see volumeMount comment). No PVC. + emptyDir: {} + - name: home + persistentVolumeClaim: + claimName: workspace-home