hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode)

[bussyjd]

What happened

Ran obolup.sh (v0.9.0 release path) on Linux ARM64 with a fresh ~/.config/obol/. The Hermes init-hermes-data init container immediately crash-looped:

mkdir: cannot create directory '/data/.hermes/home': Permission denied
mkdir: cannot create directory '/data/.hermes/workspace': Permission denied

The PVC backing dir ~/.local/share/obol/hermes-obol-agent/hermes-data/ was owned by root:root (mode 755). Pod runs as securityContext: {fsGroup: 10000, runAsGroup: 10000, runAsUser: 10000}. Same failure recurred in dev-mode (OBOL_DEVELOPMENT=true) against ~/obol-stack-src/.workspace/data/hermes-obol-agent.

What we expected

Commit c066baa fix: hermes PVC ownership on Linux k3d + bootstrap ingress URL (#446) (merged 2026-04-29, included in v0.9.0 and tip-of-main) should have prevented this.

What we found

#446 inserts an init-hermes-perms init container that runs chown -R 10000:10000 /data as runAsUser: 0 before init-hermes-data. The rendered template is correct — see internal/hermes/hermes.go:773-786:

initContainers:
  - name: init-hermes-perms
    securityContext: { runAsUser: 0, runAsGroup: 0 }
    command: [sh, -c, "chown -R 10000:10000 /data"]
    volumeMounts: [{ name: data, mountPath: /data }]
  - name: init-hermes-data
    # mkdir -p /data/.hermes/{home,workspace} as uid 10000

Test internal/hermes/hermes_test.go:145-147 asserts the init container is present. So the fix is in the manifest, but it isn't actually fixing the host-side ownership in this environment.

Two collaborating root causes:

1. Local-path-provisioner pre-creates the dir as root. internal/embed/infrastructure/base/templates/local-path.yaml:28-32 runs the helper-pod with no securityContext, so it mkdir -m 0755 and then chown -R 1000:1000 "${VOL_DIR}" — but the helper-pod is short-lived and the mkdir happens as root (so the host bind-mount sees a root-owned dir; the chown to 1000:1000 sometimes does not survive depending on UID-mapping).

2. KubeletInUserNamespace=true is set in internal/embed/k3d-config.yaml:31-37. With Linux user-namespaced kubelet, init-hermes-perms's runAsUser: 0 is namespace-root, which maps to a host subuid outside the range that owns the bind-mounted dir. The chown inside the pod silently no-ops at the host bind-mount layer, so init-hermes-data (uid 10000 → host subuid 10000+offset) still sees a root-owned dir and mkdir fails with EACCES.

No code path runs a host-side chown of the per-PVC subdir before pod start. internal/stack/backend_k3d.go:140 does os.MkdirAll(absDataDir, 0o755) for the parent only. internal/hermes/hermes.go:1276-1310 (ensureVolumeWritable / fixRuntimeVolumeOwnership) are called only from wallet/skills sync paths, not from Setup / Onboard / Sync for the agent PVC itself. obolup.sh does no chown of the data dir.

Hypothesis

#446 fixes only the in-pod path (init container does the chown), which works on macOS Docker Desktop and on Linux when the helper-pod and init container both run as host-root. On Linux k3d with KubeletInUserNamespace=true, UID translation through the user namespace prevents the in-pod chown from affecting the host bind-mount, so the host dir remains root-owned and the unprivileged main init container can't write to it.

Manual unblock

sudo chown -R 10000:10000 ~/.local/share/obol/hermes-obol-agent
sudo chmod  -R u+rwX,g+rwX  ~/.local/share/obol/hermes-obol-agent
obol kubectl -n hermes-obol-agent delete pod -l app.kubernetes.io/name=hermes

Suggested fix shape

Pick one:

1. Host-side: in hermes.Onboard/Sync (or stack.Up right after PVC apply), ensureVolumeWritable against agentruntime.HomePath(cfg, Hermes, id) and then fixRuntimeVolumeOwnership to 10000:10000. Those helpers already exist at internal/hermes/hermes.go:1276 — they're just not invoked on the agent's data dir.
2. Pod-spec: add spec.securityContext.fsGroupChangePolicy: OnRootMismatch and rely on kubelet's fsGroup recursive chown — but this is also subject to user-namespace mapping, so (1) is more robust.
3. Drop KubeletInUserNamespace=true from internal/embed/k3d-config.yaml if it isn't strictly required for rootful Docker k3d (it's a rootless-k3s feature).

Repro environment

• OS: Ubuntu 24.04 ARM64 (NVIDIA GB10)
• Docker: 29.2.1
• k3d: 5.8.3
• k3s: 1.35.1
• Obol Stack: v0.9.0 (and tip-of-main with OBOL_DEVELOPMENT=true)

[bussyjd]

Resolved — fix(hermes): host-side chown PVC backing dirs after sync (commit 41653c4, verified with git log before posting). Validated on Linux ARM64.