fix: hermes PVC ownership on Linux k3d + bootstrap ingress URL - #446
Merged
Conversation
bussyjd
approved these changes
May 8, 2026
bussyjd
left a comment
bussyjd
left a comment
Contributor
There was a problem hiding this comment.
Validated against latest origin/main in an isolated worktree.
Checks run locally:
- git merge --no-commit --no-ff refs/remotes/pr/446/head onto origin/main
- git diff --check --cached
- go test ./internal/hermes ./internal/stack ./cmd/obol
- go build ./cmd/obol
- go test ./...
All passed. The Hermes PVC ownership fix and bootstrap LocalIngressURL wiring look legitimate.
This was referenced May 12, 2026
bussyjd
added a commit
that referenced
this pull request
May 15, 2026
The in-pod `init-hermes-perms` init container from #446 (c066baa) is neutered on Linux k3d because the embedded k3d config sets `KubeletInUserNamespace=true` (internal/embed/k3d-config.yaml). With user-namespacing, the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path, so the in-pod `chown -R 10000:10000 /data` silently no-ops. The next init container (`init-hermes-data`) then fails with `mkdir /data/.hermes/home: Permission denied` and the pod CrashLoopBackOffs. local-path-provisioner's helper-pod sets the volume to 1000:1000 (internal/embed/infrastructure/base/templates/local-path.yaml), which happens to suit OpenClaw but not Hermes (containerUID = 10000). Fix: after `helmfile sync`, host-side chown the PVC backing dirs to containerUID:containerGID by exec-ing into the k3d server container via `docker exec`. That runs at the Docker daemon's real root and is not subject to the user-namespacing that silently breaks the in-pod attempt. The existing `fixRuntimeVolumeOwnership` helper already does exactly this for wallet keystore paths; we wire it into the agent deploy path via a new `ensureHermesPVCOwnership` that: 1. Waits up to 60s for each PVC (`hermes-data`, `remote-signer- keystores`) to be Bound — local-path is WaitForFirstConsumer so the host dir doesn't exist until the pod is scheduled. 2. Chowns each backing dir. 3. If a Hermes pod is currently stuck in Init:CrashLoopBackOff, deletes it so kubelet recreates immediately rather than after exponential backoff (~5 min worst case). Skips the delete when no pod is stuck so routine syncs (e.g. `obol model sync` after `obol model prefer`) do not gratuitously restart a healthy agent. Called from `hermes.Sync` after `helmfile sync` succeeds, so every Onboard / Setup / Sync call exercises it. Validated on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10) by reverting the PVC dirs to 1000:1000, deleting the Hermes pod, and running `obol model sync`. Before: pod stuck in Init:CrashLoopBackOff with "Permission denied" in init-hermes-data logs. After: PVC dirs flip to 10000:10000 within the sync, pod reaches `Running 2/2` in ~30s with zero CrashLoopBackOff cycles. Test: - `TestHermesPVCPaths` pins the two host paths the helper chowns, so renaming `hermes-data`/`remote-signer-keystores` or relocating the namespace prefix can't silently regress the fix. - Full chown side-effect needs a live k3d cluster and is exercised by the spark2 validation above plus all existing integration flows; no unit-mocked k3d test added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two Linux-specific bugs in fresh
obol stack upflows that don't surface on macOS.1. Hermes pod CrashLoopBackOff on fresh PVC (
internal/hermes/hermes.go)The Stack's local-path-provisioner setup script chowns each new PV to
1000:1000, andKubeletInUserNamespace=true(set ininternal/embed/k3d-config.yaml) silently skips thefsGrouprecursive-chown that would otherwise correct it. Hermes pods run as UID 10000 and crashloop trying tomkdir /data/.hermes/homewithPermission denied.macOS Docker Desktop hides this — its bind-mount filesystem driver fakes file ownership to whoever's asking, so the mismatch never materializes. OpenClaw doesn't hit it either: it pins
fsGroup: 1000, accidentally matching the host UID.Fix: prepend an
init-hermes-permscontainer that runs as root and chowns/datato 10000:10000. Idempotent — also self-heals existing broken PVCs on upgrade.2. Bootstrap probe + browser URL hardcoded to
:8080(cmd/obol/bootstrap.go)When ports 80/8080 are in use at start time, k3d remaps the loadbalancer to a random high port and the rest of
obol stack upcorrectly surfaces it viastack.LocalIngressURL(cfg). Three other places incmd/obol/bootstrap.gowere hardcoded to:8080: the readiness probe (hung until timeout), the browser-open URL, and the "view stack interface at" hint.Fix: reuse
stack.LocalIngressURL(cfg)in all three places.Test plan
go test ./internal/hermes/... ./cmd/obol/...passesgo build ./...cleanobol agent initlands the agent inRunningwithout manual chownobol bootstrapwith port 8080 occupied completes without hang; all four URL surfaces (warning / visit / browser-open / next-steps) agree on the alternate port