feat(deploy/no-envbox-kvm): sketch no-envbox KVM microVM deployment

deploy/no-envbox-kvm/NOTES.md

@@ -0,0 +1,90 @@
+# Notes: open questions and mapping to existing envbox code
+
+## Make-or-break tests (run these before committing to the approach)
+
+1. **Does `/dev/kvm` exist and work in the target node group?**
+   On the node: `ls -l /dev/kvm` and a `cloud-hypervisor`/`qemu -enable-kvm`
+   smoke boot. On AWS this means `*.metal` or `c8i/m8i/r8i` + nested virt.
+
+2. **Does the device-plugin path avoid `privileged` for the workspace pod?**
+   Confirm a pod requesting `devices.kubevirt.io/kvm` can open `/dev/kvm` and
+   boot a guest with NO `privileged: true` and NO `hostPID/hostNetwork`.
+
+3. **Bottlerocket: does `super_t` + device plugin cover the VMM's syscalls?**
+   Boot the VMM pod on a Bottlerocket node and watch `dmesg`/journal for
+   SELinux AVC denials on the KVM ioctls, `mmap`, and tun ops. If `super_t`
+   plus the device plugin covers them -> first-class on stock Bottlerocket.
+   If a specific op is still denied -> a custom Bottlerocket variant with a
+   purpose-built SELinux type is required (heavier; reintroduces OS-image
+   maintenance).
+
+## Mapping to today's envbox code (what changes / goes away)
+
+| Envbox today | No-envbox + KVM |
+| --- | --- |
+| `cli/docker.go` starts sysbox + inner dockerd via `sysbox-runc` | Replaced by an in-pod supervisor that launches the VMM and boots the guest. |
+| `xunix/device.go` `mknod`s `/dev/net/tun`, `/dev/fuse` under privileged | Devices come from the device plugin as pod resource requests. |
+| `envboxPrivateMounts`: `/lib/modules`, `/usr/src`, `/var/lib/sysbox` | Not needed. The guest ships its own kernel/modules; no host kernel matrix. |
+| `--privileged` outer pod | Unprivileged pod + scoped device request. |
+| Inner dockerd image store (cold cache) | Node pull-through mirror (new pods warm) + CSI-mounted inner image; ephemeral guest data dir. |
+| `xunix/gpu.go` host-lib bind mounts + `CODER_ADD_GPU` | GPU passthrough into the guest (open question; `*.metal` likely first). |
+| `CODER_INNER_*` env contract | Preserved at the supervisor boundary so templates barely change. |
+
+## Preserve the Coder-facing contract
+
+Keep `CODER_INNER_IMAGE`, `CODER_INNER_USERNAME`, `CODER_INNER_ENVS`,
+`CODER_AGENT_TOKEN`, `CODER_MOUNTS`, etc. at the supervisor boundary so
+existing envbox templates migrate with minimal edits. The isolation mechanism
+changes underneath; the template surface should not.
+
+## Biggest unknowns (ranked)
+
+1. Guest rootfs build + boot time vs. envbox cold start (target: comparable or
+   better once the docker cache is warm).
+2. Networking: virtio-net + NAT vs. the pod's CNI; making NetworkPolicy /
+   service mesh apply to guest traffic.
+3. Passing PVC-backed volumes (home dir) into the guest with correct
+   ownership (virtiofs).
+4. GPU passthrough into a nested-virt guest on non-metal families.
+5. Graceful shutdown / signal + agent-token lifecycle into the guest.
+
+## Cache design (first-boot warmth for new pods)
+
+Goal: a brand-new workspace pod on a node that already has the layers must
+**not** hit the remote registry. We do not care about fast restarts, so no
+persistence. The blocker is that the inner dockerd's store is private and
+format-incompatible with the node's containerd store, so you cannot bind-mount
+the node store into dockerd. Two levers instead:
+
+**Lever 1 -- node-local pull-through mirror (arbitrary in-workspace pulls).**
+Run a registry mirror as a DaemonSet with node-local storage and point every
+guest dockerd at it (`CODER_REGISTRY_MIRROR`). The first pod on the node to
+pull image X populates the mirror; every new pod after that pulls X from the
+node at loopback/LAN speed. Pre-seed the mirror with a pre-pull Job/DaemonSet
+if the very first pod on a cold node must also be warm. Format-agnostic and
+safe (no node socket, no shared graph).
+
+**Lever 2 -- CSI image-mount for the inner workspace image.**
+The biggest single first-boot cost is `CODER_INNER_IMAGE`, which envbox today
+pulls with the inner dockerd (never touching the node cache). Instead, mount it
+from the node's containerd store via a CSI image driver (e.g. warm-metal
+`csi-image`), which pulls via CRI and mounts the image snapshot. A new pod gets
+the already-present rootfs with no pull. This is the one clean way to actually
+reuse the kubelet/containerd cache.
+
+**Why not just mount the store:** dockerd has no shared/read-only "additional
+image store" concept, so mounting the node's layers read-only does nothing for
+a fresh dockerd. Podman *does* have this (`additionalimagestores`); if the
+inner runtime were podman/buildah, "mount a node-local read-only store and new
+pods reuse it" would be first-class. Bigger change than the mirror; noted as an
+option, not the plan.
+
+**Applies to both paths.** The guest in this KVM sketch points its dockerd at
+the same node mirror (Lever 1) and consumes the CSI-mounted inner image
+(Lever 2); the guest docker data dir is deliberately ephemeral. The identical
+pattern works for the sysbox/vanilla outer pod.
+
+## Not doing (yet)
+
+No code wired into the envbox build. This branch is a deployment-surface
+sketch to explore options and evaluate feasibility and node/OS constraints.

deploy/no-envbox-kvm/README.md

@@ -0,0 +1,97 @@
+# No-envbox workspaces: mounted devices + KVM (exploration)
+
+> Status: **exploratory proof-of-concept**. Nothing here is wired into the
+> envbox build or tested end to end. It exists to sketch what a workspace
+> deployment that drops the envbox/sysbox wrapper in favor of a KVM-backed
+> microVM (using only device requests, not a privileged outer pod) would
+> look like, and to surface the gaps that still need filling.
+
+## Why
+
+Today envbox is a **privileged** outer pod that:
+
+- bundles and supervises the sysbox runtime (`sysbox-mgr`, `sysbox-fs`,
+  `sysbox-runc`),
+- `mknod`s `/dev/net/tun` and `/dev/fuse` for the inner container from inside
+  the privileged container (see `xunix/device.go`),
+- bind-mounts host `/lib/modules` and `/usr/src` to satisfy sysbox's kernel
+  requirements,
+- runs an inner `dockerd` under `sysbox-runc` as the user's workspace.
+
+That model has four structural problems (see the gap analysis that motivated
+this branch):
+
+1. **Cold image cache.** The inner dockerd has its own image store, isolated
+   from the node's containerd cache, so every inner pull is cold.
+2. **Requires `privileged`.** Blocked by Pod Security Admission, Gatekeeper,
+   and most hardened clusters.
+3. **Coupled to sysbox** (now community-best-effort) and a host kernel matrix
+   (`/lib/modules`, `/usr/src`).
+4. **Not node-OS portable.** Cannot run on immutable/SELinux-locked node OSes
+   like Bottlerocket that forbid the host mounts and module loading it needs.
+
+## The idea
+
+Replace the "shared-kernel + sysbox" isolation model with a **microVM booted
+inside an ordinary pod**. Instead of the pod being privileged and manufacturing
+its own devices, the pod:
+
+- **requests `/dev/kvm`** (and `/dev/net/tun`) as scheduled resources from a
+  device plugin, rather than `mknod`ing them under `privileged`, and
+- boots a lightweight VMM (Cloud Hypervisor / Firecracker) that runs a real
+  guest kernel; the user's `dockerd`/`systemd` runs **inside the guest**, where
+  overlayfs, DinD, and non-root images "just work" against a normal kernel.
+
+This is not a Kubernetes RuntimeClass (customers can't install those). The VMM
+runs as a normal workload process inside the pod, so no cluster-runtime install
+is required. The only node-level requirement is that `/dev/kvm` exists and is
+exposed to the pod.
+
+```
+ Before (envbox)                      After (no-envbox + KVM)
+ ┌───────────────────────────┐        ┌───────────────────────────┐
+ │ privileged outer pod      │        │ unprivileged pod          │
+ │  sysbox daemons           │        │  requests devices.../kvm  │
+ │  ┌──────────────────────┐ │        │  ┌──────────────────────┐ │
+ │  │ inner dockerd (sysbox│ │        │  │ VMM (cloud-hypervisor│ │
+ │  │ -runc), shared kernel│ │        │  │  / firecracker)      │ │
+ │  │  user workspace      │ │        │  │  ┌────────────────┐  │ │
+ │  └──────────────────────┘ │        │  │  │ guest kernel   │  │ │
+ └───────────────────────────┘        │  │  │  dockerd/systemd│ │ │
+    needs: privileged,                │  │  │  user workspace │ │ │
+    /lib/modules, /usr/src,           │  │  └────────────────┘  │ │
+    sysbox, host kernel matrix        │  └──────────────────────┘ │
+                                      └───────────────────────────┘
+                                         needs: /dev/kvm exposed
+```
+
+## Node requirements
+
+`/dev/kvm` must exist on the node:
+
+- **AWS:** `*.metal` instances, or `C8i` / `M8i` / `R8i` with EC2 nested
+  virtualization enabled (GA 2026-02-16). Older virtualized families do not
+  expose VT-x/AMD-V and cannot host KVM.
+- **GCP / Azure:** nested virtualization on supported machine types.
+- **Bare metal:** native.
+
+## What's in this directory
+
+| File | Purpose |
+| --- | --- |
+| `manifests/kvm-device-plugin.yaml` | DaemonSet exposing `/dev/kvm` (+ `/dev/net/tun`) as schedulable resources, pinned to KVM-capable nodes. Not privileged for workloads; the plugin itself needs host device access. |
+| `manifests/registry-mirror.yaml` | Node-local pull-through registry mirror (cache Lever 1): new pods pull already-fetched layers from the node instead of the remote registry. |
+| `manifests/workspace-pod.yaml` | A no-envbox workspace pod: unprivileged, requests the device resources, boots a microVM, points its dockerd at the node mirror, and CSI-mounts the inner image (ephemeral docker data dir). |
+| `manifests/coder-template.tf` | Terraform `kubernetes_pod` sketch for the equivalent Coder template. |
+| `NOTES.md` | Open questions, the make-or-break tests, and mapping to the existing envbox code that would need to change. |
+
+## Explicitly out of scope for this sketch
+
+- The guest rootfs image (kernel + init + agent + dockerd) build.
+- The in-pod supervisor that launches the VMM, forwards the Coder agent token,
+  wires vsock/virtio-net, and streams the build log (the analog of today's
+  `cli/docker.go` flow).
+- GPU passthrough into the guest.
+
+These are the real engineering; this branch only sketches the deployment
+surface so we can evaluate feasibility and node/OS constraints.

deploy/no-envbox-kvm/manifests/coder-template.tf

@@ -0,0 +1,87 @@
+// Coder template sketch: KVM-backed workspace pod (no envbox).
+//
+// This is the Terraform analog of manifests/workspace-pod.yaml, showing how a
+// Coder template would provision the pod. It is intentionally trimmed to the
+// bits that differ from the existing kubernetes-envbox template: no
+// runtime_class_name, no privileged, device resources instead.
+//
+// SKETCH ONLY -- not runnable as-is.
+
+resource "kubernetes_pod" "workspace" {
+  count = data.coder_workspace.me.start_count
+
+  metadata {
+    name      = "coder-${data.coder_workspace_owner.me.name}-${data.coder_workspace.me.name}"
+    namespace = var.namespace
+  }
+
+  spec {
+    # Land on KVM-capable nodes (*.metal or c8i/m8i/r8i + nested virt).
+    node_selector = {
+      "coder.com/kvm" = "true"
+    }
+
+    toleration {
+      key      = "coder.com/workspace-kvm"
+      operator = "Exists"
+      effect   = "NoSchedule"
+    }
+
+    # No runtime_class_name (customers can't install one).
+    # No privileged security_context.
+    security_context {
+      seccomp_profile {
+        type = "RuntimeDefault"
+      }
+    }
+
+    container {
+      name  = "workspace"
+      image = "REPLACE_ME/coder-workspace-vmm:latest"
+
+      # KVM requested as a device, not via privileged.
+      resources {
+        requests = {
+          "devices.kubevirt.io/kvm" = "1"
+          "devices.kubevirt.io/tun" = "1"
+          "cpu"                     = "2"
+          "memory"                  = "4Gi"
+        }
+        limits = {
+          "devices.kubevirt.io/kvm" = "1"
+          "devices.kubevirt.io/tun" = "1"
+          "cpu"                     = "4"
+          "memory"                  = "8Gi"
+        }
+      }
+
+      env {
+        name  = "CODER_INNER_IMAGE"
+        value = data.coder_parameter.inner_image.value
+      }
+      # Point the guest dockerd at the node-local pull-through mirror so new
+      # pods pull from the node (warm), not the remote registry. See NOTES.md.
+      env {
+        name  = "CODER_REGISTRY_MIRROR"
+        value = "http://kube-registry-mirror.kube-system.svc:5000"
+      }
+      env {
+        name  = "CODER_AGENT_TOKEN"
+        value = coder_agent.main.token
+      }
+
+      volume_mount {
+        # Ephemeral guest docker data dir. Warmth comes from the mirror +
+        # CSI-mounted inner image, not from persistence (see NOTES.md).
+        name       = "docker-data"
+        mount_path = "/var/lib/coder/docker"
+      }
+    }
+
+    # Ephemeral: we do not need fast restarts.
+    volume {
+      name = "docker-data"
+      empty_dir {}
+    }
+  }
+}

deploy/no-envbox-kvm/manifests/kvm-device-plugin.yaml

@@ -0,0 +1,73 @@
+# KVM device plugin
+#
+# Exposes /dev/kvm (and /dev/net/tun) as *schedulable* resources so that
+# workspace pods can request them WITHOUT running as privileged and without
+# the pod having to mknod its own devices (as envbox does today in
+# xunix/device.go).
+#
+# This is the piece that lets us drop `privileged: true`: a device plugin adds
+# a specific device to the pod's cgroup allow-list and bind-mounts it, which is
+# exactly the "one device, not all devices" grant that a Kubernetes
+# securityContext cannot express on its own.
+#
+# Reference implementation: kubevirt/kubernetes-device-plugins (kvm, tun).
+# Replace the image below with a built/pinned image.
+#
+# NOTE: the device plugin DaemonSet itself needs host device access (and, on
+# Bottlerocket, the super_t SELinux label). That privilege is confined to this
+# small, auditable component -- not to every workspace.
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kvm-device-plugin
+  namespace: kube-system
+  labels:
+    app: kvm-device-plugin
+spec:
+  selector:
+    matchLabels:
+      app: kvm-device-plugin
+  template:
+    metadata:
+      labels:
+        app: kvm-device-plugin
+    spec:
+      # Only schedule onto nodes that actually expose /dev/kvm. Label your
+      # KVM-capable node group (e.g. *.metal or c8i/m8i/r8i w/ nested virt)
+      # with `coder.com/kvm=true`.
+      nodeSelector:
+        coder.com/kvm: "true"
+      tolerations:
+        - key: coder.com/workspace-kvm
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kvm-device-plugin
+          image: REPLACE_ME/kvm-device-plugin:latest
+          args:
+            - --devices=kvm,tun
+          securityContext:
+            # Device plugins need host device access. On Bottlerocket this also
+            # requires the super_t SELinux label (see NOTES.md); on general
+            # node OSes privileged is the pragmatic setting for the plugin.
+            privileged: true
+            # Bottlerocket equivalent (instead of privileged) -- uncomment and
+            # test which is sufficient:
+            # seLinuxOptions:
+            #   user: system_u
+            #   role: system_r
+            #   type: super_t
+            #   level: s0
+          volumeMounts:
+            - name: device-plugin
+              mountPath: /var/lib/kubelet/device-plugins
+            - name: dev
+              mountPath: /dev
+      volumes:
+        - name: device-plugin
+          hostPath:
+            path: /var/lib/kubelet/device-plugins
+        - name: dev
+          hostPath:
+            path: /dev