eRPC operator overlays are overwritten on stack up / helm sync (HyperEVM basket)

[bussyjd]

Summary

Operator-applied eRPC ConfigMap overlays (multi-upstream baskets with scoring, rate-limit budgets, cache policies, and network failsafe) are not durable. They are wiped or incomplete after obol stack up / helmfile re-render of the erpc release. We hit this live while wiring HyperEVM (chain 999 / alias hyperevm) for first-party Hyperliquid intelligence (hl-intel).

This is the same class of problem Hermes already solved for operator-edited runtime config (mergePreservedHermesConfigKeys), and partially solved for simple RPCs via host-side record-on-write ($CONFIG_DIR/rpc/recorded-upstreams.yaml + ReconcileRecordedRPCs). eRPC still lacks a durable overlay layer for rich baskets.

Context / reproduction (live)

Cluster: long-lived k3s stack (control plane + co-located hl-node host).

Desired state: in-cluster consumers call:

http://erpc.erpc.svc.cluster.local/rpc/hyperevm
# or
http://erpc.erpc.svc.cluster.local/rpc/evm/999

with a basket that prefers a local non-validator HyperEVM endpoint and fails over to public no-key RPCs.

What we did:

1. Patched erpc/erpc-config ConfigMap to add:
   • network { alias: hyperevm, chainId: 999 } + failsafe
   • upstreams: local hl-node (http://<node-ip>:3001/evm) + public fallbacks
   • rateLimiters.budgets for public peers
   • finality: realtime cache policy for eth_call (precompile reads)
2. Annotated obol.hl/hyperevm-basket=local-overlay so operators know it is hand-applied
3. Verified live: /rpc/hyperevm → eth_chainId = 0x3e7 (999), blockNumber advances

What will clobber it:

• obol stack up / helmfile sync of the embedded eRPC chart values (internal/embed/infrastructure/values/erpc.yaml.gotmpl) re-materializes the base ConfigMap (mainnet / hoodi / base / base-sepolia only)
• Our overlay is not in the chart and is not replayed by ReconcileRecordedRPCs
• Even using obol network add --endpoint … only supports one custom-<chainId>-0 upstream, no scoring / budgets / multi-peer baskets, and local-node-style IPs are not first-class

Operator doc for the basket lives out-of-tree today:
obol-exex-indexer/services/hyperliquid/erpc-hyperevm-basket.patch.yaml (post-render patch runbook). That is a footgun, not a product surface.

Why obol network add is not enough

Capability needed for HyperEVM intel	network add --endpoint today	Full ConfigMap overlay
Multiple public fallbacks	❌ single custom-<id>-0	✅
Prefer local node (score / order)	⚠️ endpoint first only	✅ scoreMultipliers / order
Per-upstream rateLimitBudget + auto-tune	❌	✅
Network-level hedge/retry tuned for eth_call	❌ generic failsafe	✅
finality: realtime eth_call cache (2s)	❌	✅
Survive stack up / helm sync	✅ via recorded-upstreams.yaml (simple form only)	❌ today

So operators who need a real basket are forced into manual ConfigMap surgery, which silently dies on the next stack lifecycle event — same failure mode as pre-record RPC adds.

Analogy: how Hermes already handles “don’t overwrite my edits”

Hermes re-renders config.yaml on obol agent sync, but merges operator intent back in:

// internal/hermes/hermes.go — mergePreservedHermesConfigKeys
// carries operator-edited Hermes keys across obol agent sync.
// generateConfig only knows stack-managed defaults; security knobs
// like command_allowlist set via `hermes config` must survive re-render.

And eRPC already has a narrower version of the same idea for simple remotes:

// internal/network/record.go
// Record-on-write for remote RPC upstreams.
// AddPublicRPCs / AddCustomRPC update $CONFIG_DIR/rpc/recorded-upstreams.yaml
// ReconcileRecordedRPCs replays after obol stack up

Gap: there is no equivalent of “merge preserved operator overlay onto generated eRPC config” for structured multi-upstream baskets (or general YAML overlays).

Proposed solution

Goal

Make operator eRPC customizations first-class, durable, and re-applied after every stack lifecycle, without forking the embedded chart for every chain.

Design (recommended): host-side eRPC overlay + merge-on-reconcile

Mirror Hermes + recorded-RPCs, but for a mergeable overlay document.

1. Host-side source of truth

$OBOL_CONFIG_DIR/rpc/erpc-overlay.yaml   # 0600, like recorded-upstreams.yaml

Schema sketch:

version: 1
# Optional: full fragment(s) deep-merged into the rendered eRPC config after
# chart/base generation and after recorded-upstreams replay.
networks:
  - alias: hyperevm
    architecture: evm
    evm: { chainId: 999 }
    failsafe:
      timeout: { duration: 10s }
      retry: { maxAttempts: 3, delay: 100ms }
      hedge: { delay: 150ms, maxCount: 1 }
upstreams:
  - id: local-hl-node
    endpoint: http://192.168.50.21:3001/evm   # or host.k3d.internal / Service DNS
    evm: { chainId: 999 }
    routing:
      scoreMultipliers:
        - { network: "*", method: "*", overall: 5.0 }
  - id: hyperevm-official
    endpoint: https://rpc.hyperliquid.xyz/evm
    evm: { chainId: 999 }
    rateLimitBudget: hyperevm-official
    tags: ["tier:fallback"]
rateLimiters:
  budgets:
    - id: hyperevm-official
      rules: [{ method: "*", maxCount: 100, period: second }]
cachePoliciesAdd:
  - { network: "*", method: eth_call, finality: realtime, connector: memory-cache, ttl: 2s }

2. Apply path (single choke point)

Extend the post-stack up resume sequence (already ordered in cmd/obol/main.go):

helmfile render/apply eRPC base
  → ReconcileRecordedRPCs()          # existing simple remotes
  → ReconcileERPCOverlay()           # NEW: deep-merge overlay + writeERPCConfig + rollout

Also call ReconcileERPCOverlay from:

• obol network sync (if that rewrites eRPC)
• any path that rewrites the ConfigMap via writeERPCConfig

Implementation sketch:

1. readERPCConfig
2. Deep-merge overlay (networks by alias/chainId, upstreams by id, budgets by id, append unique cache policies)
3. writeERPCConfig (existing patch + rollout)

Idempotent; safe to re-run.

3. CLI surface (operator ergonomics)

# Install / replace overlay from a file (the out-of-tree basket becomes in-tree intent)
obol network overlay apply -f services/hyperliquid/erpc-hyperevm-basket.patch.yaml

# Show effective vs overlay vs base
obol network overlay status

# Remove overlay and re-render base + recorded remotes only
obol network overlay clear

Alternatively (smaller API surface): extend obol network add with:

obol network add hyperevm --chain-id 999 \
  --endpoint http://192.168.50.21:3001/evm --primary \
  --endpoint https://rpc.hyperliquid.xyz/evm \
  --endpoint https://hyperliquid.drpc.org \
  --rate-limit-budget ...   # optional later

…and persist the multi-endpoint intent into the overlay/record format (not a single custom-999-0).

Prefer the overlay file approach for v1: it covers scoring, budgets, and cache policies without exploding CLI flags.

4. Helm ownership rules

• Chart/base remains the generated floor (Obol-hosted remotes, base/sepolia, etc.).
• Overlay is additive and never deleted by helm three-way merge of the chart alone, because we re-apply after helm.
• Document: do not hand-edit the live ConfigMap; edit $CONFIG_DIR/rpc/erpc-overlay.yaml or use obol network overlay apply.

Optional hardening: set ConfigMap annotation
obol.stack/erpc-overlay-hash: <sha> and warn on stack up if live CM hash ≠ expected (detect manual drift).

5. What not to do

Anti-pattern	Why
Keep teaching kubectl ConfigMap patches	Breaks on every stack up / upgrade
Fork erpc.yaml.gotmpl in every consumer repo	Diverges from release; no reconcile path
Stuff full baskets only into recorded-upstreams.yaml as-is	Schema only supports one custom endpoint + chainlist list
Put secrets in the overlay without 0600 + redaction	Same rules as paid RPC URLs in recorded remotes

Acceptance criteria

• Operator can declare a multi-upstream HyperEVM basket that survives obol stack up and helmfile eRPC re-sync
• /rpc/hyperevm and /rpc/evm/999 keep working after a full stack recycle without manual kubectl
• Overlay merge is idempotent and does not drop base upstreams (mainnet/base/…)
• obol network overlay status (or equivalent) shows overlay presence + last apply
• Unit tests: merge by chainId/id; reconcile called from stack-up resume guard (mirror stackup_resume_guard_test.go for ReconcileERPCOverlay)
• Docs: obol network help + short guide; migrate note from out-of-tree erpc-hyperevm-basket.patch.yaml

Suggested implementation slices

1. P0 — erpc-overlay.yaml + ReconcileERPCOverlay after ReconcileRecordedRPCs; file-based apply via obol network overlay apply -f
2. P1 — Multi-endpoint network add that writes overlay (not only single custom)
3. P2 — First-party preset: obol network add hyperevm --preset public+local --local-endpoint … shipping the curated public basket in-tree (optional; can stay in consumer repos as overlay files)

Related code

• internal/network/rpc.go — readERPCConfig / writeERPCConfig / AddCustomRPC
• internal/network/record.go — recorded-upstreams.yaml, ReconcileRecordedRPCs
• internal/hermes/hermes.go — mergePreservedHermesConfigKeys (preserve operator intent across re-render)
• cmd/obol/main.go — stack-up resume order + stackup_resume_guard_test.go
• internal/embed/infrastructure/values/erpc.yaml.gotmpl — base eRPC generation
• Consumer basket (today): obol-exex-indexer/services/hyperliquid/erpc-hyperevm-basket.patch.yaml

Motivation

First-party services (Hyperliquid hl-intel, future chain-local nodes) need LAN-primary + public failover baskets with real rate-limit / cache tuning. Without durable overlays, every stack upgrade silently breaks chain-oracle / HyperEVM reads and forces tribal knowledge kubectl patches. Hermes already taught us: generated config + merge preserved operator intent is the right product shape.