release: v0.14.0-rc2

CLAUDE.md

@@ -62,6 +62,7 @@ Integration tests use `//go:build integration`; skip when prerequisites missing.
 | Public (tunnel) | none | `/.well-known/agent-registration.json` | ERC-8004 httpd |
 | Public (tunnel) | none | `/skill.md` | service catalog |
 | Public (tunnel) | none | `/api/services.json` | service catalog JSON feed (`displayName`, `tagline`, `logoUrl`, `theme`, `themeVars`, `faviconUrl`, `ogImageUrl`, `description(+Html)`, `customCss`, `services[]`) |
+| Public (tunnel) | none | `/.well-known/x402` | aggregate x402 discovery fallback (AgentCash/x402scan-style crawlers that don't parse `/openapi.json`'s `x-payment-info`) |
 | Public (tunnel) | tunnel hostname only | `/` | storefront landing page (Next.js) |
 
 **NEVER remove hostname restrictions from frontend or eRPC HTTPRoutes** — exposing the frontend/RPC to the public internet is a critical security flaw.
@@ -457,7 +458,7 @@ A registry digest pin instead of `:latest` on the verifier means your dev rewrit
 20. **402 page silently falls back to JSON when the template errors** — `sendPaymentRequiredHTML` swallows template-exec errors and re-sends the JSON body, so referencing a field in `payment_required.html` that isn't in the render's data struct doesn't crash anything: browsers just start getting JSON. Symptom: `Content-Type: application/json` on an `Accept: text/html` request. Any template-field addition needs the struct field added in `paymentrequired.go` AND a test asserting the HTML branch still renders (the existing branding tests check Content-Type/markup).
 21. **`html/template` rejects `data:` URIs in URL contexts (`#ZgotmplZ`)** — inline logos/favicons from `sell info set --logo-file` are `data:image/...;base64` URIs; interpolating them into `src=`/`href=` via a plain string yields the literal `#ZgotmplZ` (broken image). Branding asset URLs must go through `storefront.SafeAssetURL` (validates http(s)/`data:image` then returns `template.URL`); regression test `TestPaymentRequiredHTML_InlineDataURILogo`.
 22. **Poisoned LiteLLM model group — intermittent 404 `{'detail': 'Not Found'}` on ~50% of requests** — two `model_list` deployments share a `model_name` but disagree on `api_base` (one with `/v1`, one without); LiteLLM shuffles between them and does not retry a 404. Historic cause: auto-discovery registering the same host endpoint that `obol model setup custom` later added correctly (#745). Diagnosis: compare the router's live view (`GET /model/info`) against the `litellm-config` CM — the drift checker compares model NAMES only and cannot see divergent `api_base` (#746). The vLLM access log is decisive: alternating `POST /chat/completions 404` / `POST /v1/chat/completions 200`.
-23. **Buyer disconnect ≠ no charge (zombie settlement)** — a client abort routinely does NOT propagate past cloudflared; the upstream agent finishes the turn, the handler returns 2xx, and settlement fires → buyer debited for a response nobody received (proven on-chain 2026-07-14, the Bankr incident). The verifier skips settlement when the request context is already canceled (#743), but that only covers propagated cancels — the real protection is keeping paid agent runs SHORT (`Agent.spec.maxTurns`; runs must finish inside the ~100s tunnel window and typical client timeouts). Concurrency: Hermes caps simultaneous runs in-process (`gateway.api_server.max_concurrent_runs`, internal default 10, 429 "Too many concurrent runs"; exposed as `Agent.spec.maxConcurrentRuns`, 0 = disabled) — keep it coherent with the edge `spec.limits.maxInFlight` Traefik gate; 4xx responses are never settled, so both gates are financially safe.
+23. **Buyer disconnect ≠ no charge (zombie settlement)** — a client abort routinely does NOT propagate past cloudflared; the upstream agent can finish 2xx and older seller builds still called `/settle` → buyer debited while the UI showed failure (Bankr ~30s retries ×3 on-chain, 2026-08-05). This deferred-to-`finalize()` protection applies ONLY to `text/event-stream` (SSE) responses — `settlementInterceptor.WriteHeader` still settles eagerly, before committing the status, for every non-streaming response (plain `sell http`, non-streaming `sell agent`), exactly as before this fix. For SSE, `finalize()` skips `/settle` on canceled context, **client Write errors** (broken pipe), or zero body bytes — see `docs/observability.md` ("External buyers (Bankr)"). Separately, Bankr auto-pay often fails verify with `facilitator_error` because it signs EIP-3009 `validAfter=now` (use past buffer + ≥180s timeout). Still keep paid agent runs SHORT (`Agent.spec.maxTurns`) inside the tunnel window. Concurrency: Hermes `max_concurrent_runs` / `Agent.spec.maxConcurrentRuns` must stay coherent with edge `spec.limits.maxInFlight`; 4xx is never settled.
 
 For a fuller debug catalog with symptom->fix mapping, see `.agents/skills/obol-stack-dev/references/release-smoke-debugging.md`.
 
@@ -478,6 +479,7 @@ The Cloudflare tunnel exposes the cluster to the public internet. Only x402-gate
 - `/.well-known/agent-registration.json` — ERC-8004 discovery
 - `/skill.md` — machine-readable service catalog
 - `/api/services.json` — service catalog envelope (`displayName`, `tagline`, `logoUrl`, theme/branding fields, `services[]`)
+- `/.well-known/x402` — aggregate x402 discovery fallback (AgentCash/x402scan-style discovery convention; no secret material)
 - `/` on tunnel hostname — public storefront landing page (Next.js)
 
 ## Dependencies

cmd/obol/sell_info.go

@@ -841,7 +841,22 @@ func applySellerProfile(cfg *config.Config, profile schemas.StorefrontProfile) e
 	if err != nil {
 		return err
 	}
-	if err := kubectlApply(cfg, manifest); err != nil {
+	raw, err := json.Marshal(manifest)
+	if err != nil {
+		return fmt.Errorf("marshal storefront profile: %w", err)
+	}
+	bin, kubeconfig := kubectl.Paths(cfg)
+	// Inline data:image values can make kubectl's client-side
+	// last-applied-configuration annotation exceed its 256 KiB limit even
+	// though the ConfigMap itself remains below Kubernetes' 1 MiB limit.
+	// The host-side profile record is authoritative, so use server-side apply
+	// for both initial writes and updates.
+	if err := kubectl.ApplyServerSideForceConflicts(
+		bin,
+		kubeconfig,
+		raw,
+		"obol-storefront-profile",
+	); err != nil {
 		return fmt.Errorf("apply storefront profile: %w", err)
 	}
 	return nil

cmd/obol/sell_info_test.go

@@ -1,12 +1,49 @@
 package main
 
 import (
+	"fmt"
+	"os"
+	"path/filepath"
 	"strings"
 	"testing"
 
+	"github.com/ObolNetwork/obol-stack/internal/config"
 	"github.com/ObolNetwork/obol-stack/internal/schemas"
 )
 
+func TestApplySellerProfileUsesServerSideApplyForInlineImages(t *testing.T) {
+	cfg := &config.Config{ConfigDir: t.TempDir(), BinDir: t.TempDir()}
+	argsPath := filepath.Join(t.TempDir(), "kubectl-args")
+	script := fmt.Sprintf("#!/bin/sh\nprintf '%%s\\n' \"$@\" > %q\ncat >/dev/null\n", argsPath)
+	if err := os.WriteFile(filepath.Join(cfg.BinDir, "kubectl"), []byte(script), 0o755); err != nil {
+		t.Fatal(err)
+	}
+
+	profile := schemas.StorefrontProfile{
+		LogoURL: "data:image/png;base64," + strings.Repeat("a", 270_000),
+	}
+	if err := applySellerProfile(cfg, profile); err != nil {
+		t.Fatalf("applySellerProfile: %v", err)
+	}
+
+	args, err := os.ReadFile(argsPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := string(args)
+	for _, want := range []string{
+		"apply\n",
+		"--server-side\n",
+		"--force-conflicts\n",
+		"--field-manager=obol-storefront-profile\n",
+		"-f\n-\n",
+	} {
+		if !strings.Contains(got, want) {
+			t.Fatalf("kubectl args missing %q:\n%s", want, got)
+		}
+	}
+}
+
 func TestClearProfileFields(t *testing.T) {
 	base := schemas.StorefrontProfile{
 		DisplayName:  "Acme",

docs/observability.md

@@ -252,6 +252,64 @@ A `Transfer` to the expected recipient that exists while the buyer reports
 
 ---
 
+## External buyers (Bankr): two failure modes, one canonical ledger
+
+Bankr chat / `bankr x402 call` / Apps against Obol agent offers confused
+operators because the UI error and the on-chain outcome often disagreed.
+Live Base mainnet testing (2026-08-05) against a public Obol tunnel clarified
+what is seller-fixable vs buyer-client limits.
+
+**Context — Bankr surfaces are not one path.** Chat auto-pay, CLI
+`bankr x402 call`, and Apps (`bankr.x402.fetch`) are different clients.
+Bankr docs also describe chat/CLI auto-pay as oriented around endpoints
+deployed via `bankr x402 deploy` / their discovery index
+([docs.bankr.bot/x402-cloud/quick-start](https://docs.bankr.bot/x402-cloud/quick-start/)),
+and Apps require a manifest `allowedHosts` allowlist
+([docs.bankr.bot/apps/sdk](https://docs.bankr.bot/apps/sdk)).
+
+**Allowlisting is not enough for agent offers.** We built a Bankr App with
+`pay:x402`, `allowedHosts: ["<our-tunnel-hostname>"]`, and
+`bankr.x402.fetch` against bounty-radar. Payment **often verified** on Base
+mainnet; the App still failed with **`rpc timeout` ~30s** while the agent was
+still running. So the Apps failure we hit was not “missing allowlist” or
+“wrong network” — it was Bankr’s short client timeout on slow agent SSE.
+HTTP offers through the same App/chat path usually succeed because they
+finish in ~1s.
+
+When Bankr *does* attempt a paid call, there are **two separate wire-level
+failure modes**:
+
+| Mode | What the buyer sees | What happened | Charge? |
+|---|---|---|---|
+| **A — Voucher** | JSON `503` with `reason:facilitator_error`, `detail:unexpected_error` | Buyer signed EIP-3009 with `validAfter=wall-clock now` (or a bad typed-data hash). Base USDC rejects (`not yet valid` / `invalid signature`). | Usually **no** (verify never succeeded). |
+| **B — Timeout / zombie** | `rpc timeout`, 504, or generic failure after ~30s | Verify **succeeded**, agent still running (often 30–120s to first SSE byte). Bankr client aborted. Cloudflare often does **not** cancel seller context, so older seller builds still called `/settle` after upstream finished → BaseScan shows 0.001 USDC Transfers. | **Yes** on older builds. |
+
+**Seller hardenings** (in-process HandleProxy settlement for agent/http
+gateways that settle after upstream — not the Traefik ForwardAuth
+verify-only path):
+
+- Settle SSE only in `finalize()` after the stream completes — never on the
+  first `WriteHeader(200)`.
+- Skip `/settle` when `r.Context()` is canceled, when a body `Write` to the
+  client fails (broken pipe — the reliable signal when cancel does not
+  propagate), or when zero body bytes were written.
+- Upstream proxy errors after verify return structured JSON with
+  `paymentVerified:true`, `paymentSettled:false`, `retriable:false` so buyers
+  do not auto-retry storms.
+
+**Buyer guidance** (storefront Bankr prompts are type-specific):
+- **http** — prefer Bankr chat auto-pay (fast enough for the ~30s window).
+  Do not ask chat to run `bankr wallet sign` (it cannot).
+- **agent / inference** — do not use Bankr chat/Apps auto-pay (`rpc timeout`
+  even with a correct Apps `allowedHosts`); use `bankr wallet sign` with a
+  past `validAfter` buffer and HTTP timeout ≥180s. After any timeout, check
+  BaseScan before retrying.
+
+**Still true:** chain Transfers are canonical. Seller `paymentSettled:false`
+and Bankr UI copy are best-effort signals.
+
+---
+
 ## Recording rule conventions
 
 Naming follows the standard Prometheus pattern:

flows/clients/x402-generic-buyer.go

@@ -0,0 +1,141 @@
+//go:build ignore
+
+// Generic x402 HTTP buyer client for flow-22-external-buyer-compat.
+//
+// Built entirely from the public x402-foundation/x402/go/v2 SDK's
+// documented client pattern (see CLIENT.md "Basic HTTP Client" in the SDK
+// module): wrap a plain *http.Client with x402 payment handling and call it
+// like any other HTTP client. No Obol CLI, no buy.py, no PurchaseRequest CR
+// — this is what a third-party buyer tool's own wallet/agent does under the
+// hood (AgentCash, Bankr, or any other x402-compliant client), so a
+// successful "paid" run here is evidence the seller works with any
+// standards-compliant buyer, not just Obol's own tooling.
+//
+// Run from the repo root (module context):
+//
+//	go run flows/clients/x402-generic-buyer.go \
+//	  -mode <unpaid|paid> -url <resource-url> \
+//	  [-method GET|POST] [-body '<json>']
+//
+// Modes:
+//
+//	unpaid  call the endpoint with no signer registered — expect a 402
+//	paid    call the endpoint with a signer from X402_CLIENT_KEY — the
+//	        wrapped client detects the 402, signs, retries, and returns
+//	        whatever the seller sends back (expect 200 on success)
+//
+// -method/-body cover both the plain HTTP demo (GET, empty body) and the
+// OpenAI-compatible chat-completions shape (POST + JSON body) used by
+// inference/agent offers.
+//
+// X402_CLIENT_KEY carries the buyer's 0x-prefixed private key via env so it
+// never appears on argv. Output is ONE JSON object on stdout; the flow
+// asserts on its fields.
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	x402 "github.com/x402-foundation/x402/go/v2"
+	x402http "github.com/x402-foundation/x402/go/v2/http"
+	exactclient "github.com/x402-foundation/x402/go/v2/mechanisms/evm/exact/client"
+	evmsigner "github.com/x402-foundation/x402/go/v2/signers/evm"
+)
+
+func main() {
+	url := flag.String("url", "", "paid resource URL")
+	mode := flag.String("mode", "paid", "unpaid|paid")
+	network := flag.String("network", "eip155:84532", "CAIP-2 network to register the signer for")
+	method := flag.String("method", "GET", "HTTP method")
+	body := flag.String("body", "", "optional request body (e.g. chat-completions JSON)")
+	timeout := flag.Duration("timeout", 120*time.Second, "per-request timeout")
+	flag.Parse()
+
+	out := map[string]any{"mode": *mode, "method": strings.ToUpper(*method)}
+	if *url == "" {
+		out["error"] = "-url is required"
+		emit(out)
+		os.Exit(1)
+	}
+	if err := run(*url, *mode, *network, strings.ToUpper(*method), *body, *timeout, out); err != nil {
+		out["error"] = err.Error()
+		emit(out)
+		os.Exit(1)
+	}
+	emit(out)
+}
+
+func emit(m map[string]any) {
+	b, _ := json.Marshal(m)
+	fmt.Println(string(b))
+}
+
+func run(url, mode, network, method, body string, timeout time.Duration, out map[string]any) error {
+	ctx, cancel := context.WithTimeout(context.Background(), timeout)
+	defer cancel()
+
+	// mode=unpaid deliberately skips the x402 wrapper entirely — a real
+	// unpaid buyer is just a plain HTTP client with no x402 awareness, and
+	// the wrapper's RoundTrip returns an error (not a 402 response) when no
+	// scheme is registered for the seller's network, since it can't build a
+	// payment payload. Bypassing it here is what lets this mode observe the
+	// seller's raw 402 challenge, exactly like a naive caller would.
+	httpClient := http.DefaultClient
+	if mode == "paid" {
+		key := os.Getenv("X402_CLIENT_KEY")
+		if key == "" {
+			return fmt.Errorf("X402_CLIENT_KEY is required for -mode paid")
+		}
+		signer, err := evmsigner.NewClientSignerFromPrivateKey(key)
+		if err != nil {
+			return fmt.Errorf("signer: %w", err)
+		}
+		out["buyerAddress"] = signer.Address()
+
+		// Following CLIENT.md's quick-start verbatim: create the core
+		// client, register the scheme, wrap a plain http.Client.
+		client := x402.Newx402Client().
+			Register(x402.Network(network), exactclient.NewExactEvmScheme(signer, nil))
+		httpClient = x402http.WrapHTTPClientWithPayment(http.DefaultClient, x402http.Newx402HTTPClient(client))
+	}
+
+	var bodyReader io.Reader
+	if body != "" {
+		bodyReader = strings.NewReader(body)
+	}
+	req, err := http.NewRequestWithContext(ctx, method, url, bodyReader)
+	if err != nil {
+		return fmt.Errorf("build request: %w", err)
+	}
+	if body != "" {
+		req.Header.Set("Content-Type", "application/json")
+	}
+	resp, err := httpClient.Do(req)
+	if err != nil {
+		return fmt.Errorf("do: %w", err)
+	}
+	defer resp.Body.Close()
+
+	respBody, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return fmt.Errorf("read body: %w", err)
+	}
+
+	out["status"] = resp.StatusCode
+	out["body"] = string(respBody)
+	if v := resp.Header.Get("X-PAYMENT-RESPONSE"); v != "" {
+		out["paymentResponseHeader"] = v
+	}
+	if v := resp.Header.Get("PAYMENT-RESPONSE"); v != "" {
+		out["paymentResponseHeaderV2"] = v
+	}
+	return nil
+}