Skip to content

ci(release): auto-repin embedded x402 image pins + release freshness gate - #618

Closed
bussyjd wants to merge 1 commit into
mainfrom
chore/auto-repin-x402-images
Closed

ci(release): auto-repin embedded x402 image pins + release freshness gate#618
bussyjd wants to merge 1 commit into
mainfrom
chore/auto-repin-x402-images

Conversation

@bussyjd

@bussyjd bussyjd commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Automates the release-time image repin so it can never be forgotten again. Two mechanisms, belt and suspenders:

flowchart LR
    P["push to main / release/**<br/>(or workflow_dispatch)"] --> B["docker-publish-x402<br/>build + push images<br/>:shortsha"]
    B --> R["repin-embedded-pins job<br/>rewrite x402.yaml + llm.yaml pins<br/>commit chore(ci): repin … [auto]"]
    R --> T["operator tags vX.Y.Z"]
    T --> G{"release.yml<br/>verify-image-pins gate"}
    G -->|pins fresh| REL["binaries built,<br/>draft release"]
    G -->|stale / digest mismatch /<br/>unresolvable pin| BLOCK["release FAILS<br/>with fix instructions"]
Loading
  1. Auto-repin (repin-embedded-pins job in docker-publish-x402.yml): after every successful branch image build, .github/scripts/repin-x402-images.sh rewrites the embedded x402-verifier / serviceoffer-controller / x402-buyer pins to the multi-arch index digests of the images just built, and the job pushes a chore(ci): repin x402 images to <sha> [auto] commit. Branch refs only — never tags.
  2. Release gate (verify-image-pins in release.yml, release now needs it): .github/scripts/verify-x402-pins.sh fails the tag when any source in the three binaries' live import graph (go list -deps, not a hand-maintained path list) changed after the pinned build commit.

This closes the trap that hit v0.10.0-rc14: the train's pins were its own merge base (04bebbc), so the shipped manifests deployed images containing none of the train's verifier/buyer changes until a manual rebuild+repin (2db429b, the rc11 pattern). With this PR that manual step is automatic, and a tag cut without it cannot release.

Gate properties (each adversarially tested)

  • Fail-closed: a go list failure refuses to pass rather than silently degrading to a partial path set (reproduced: with a broken go, the old draft waved stale pins through; now exits 1).
  • Digest↔tag binding: the embedded digest must match what GHCR serves for the pinned tag — a fresh tag with a hand-edited digest fails (the digest, not the tag, is what Kubernetes pulls). VERIFY_X402_PINS_OFFLINE=true skips for air-gapped runs.
  • No self-staleness: the two pin-carrying templates are hunk-filtered — pin lines are ignored, any other edit to them (RBAC, args, env) still counts as stale. _test.go/testdata churn is ignored (doesn't compile into the binaries).
  • Consistency: all three pins must share one build commit; multiple distinct pins for one image fail extraction.
  • Caught the real thing: run against the pre-repin rc14 state, the gate flags agent_render.go, openapi.go, go.mod, … — exactly the trap.

Workflow security

  • Context values (github.ref_name, github.sha) are env-bound in the repin job, never interpolated into script text (branch names may contain shell metacharacters).
  • The push step verifies the diff touches only the two template files before committing; job has contents: write, the new gate job is contents: read.
  • The bump is committed via the GraphQL createCommitOnBranch API: GitHub signs the commit itself (verified, github-actions bot), so the job is compatible with a required_signatures ruleset — a workflow git push can never produce a verified commit. expectedHeadOid is the live remote head with one retry on race; only the two guarded files are ever sent. No recursion: API commits made with GITHUB_TOKEN don't trigger workflows.
  • Path filters extended to approximate the binaries' real import graph (the gate computes the exact one at tag time, so anything the filter misses fails the release instead of shipping stale); release/** pushes now build+repin like main.

Test refactor

The exact-ref equality tests (TestEmbeddedImages_X402ControllerAndBuyerUseFixPins, TestX402VerifierImage_CarriesAgentAuthFix, TestServiceOfferControllerImage_CarriesSecretCreateOnlyFix) became invariant tests, so the bot can bump pins without editing Go files while the guarantees get stronger:

  • pins must be <repo>:<short-sha>@sha256:<digest> (digest discipline already covered by TestEmbeddedImages_NamedImagesAreDigestPinned),
  • all three must share one build commit (TestEmbeddedImages_X402PinsShareOneBuildCommit),
  • the pinned commit must descend from the named fix commits — b39bcaa (Secret-create-only), abfd55a (agent auth), ab71481, 86b8c9f — ancestry-verified via git (TestEmbeddedImages_X402PinsCarryRequiredFixes), skipped gracefully on shallow clones where the release gate covers it. Only main-reachable commits belong in that list (release-branch SHAs stop being ancestors after a squash-merge; documented in the test).

Review

42-finding adversarial review pass (4 lenses × refutation agents): 1 major fixed (fail-open on go list failure), 2 minors fixed (ref_name injection, digest binding), nits fixed (gate job permissions, multi-pin extraction, diff-header anchor that could hide YAML doc-separator deletions, ambiguous-SHA error hint). Branch-protection findings refuted by measurement (no protection/rulesets on main).

Validation

  • shellcheck clean on all three scripts; both workflows parse.
  • Gate: positive (current pins), negative × 3 (rc13 mixed pins → same-tag failure; pre-repin rc14 pins → staleness with the exact culprit files; corrupted digest → registry mismatch), fail-closed (broken go).
  • Repin script: idempotent no-op at current pin; real bump to 2db429b images verified end-to-end (gate + tests pass on bumped state), then restored.
  • Full go test ./... green (34 packages).

Stacks on #616 (branched from the v0.10.0-rc14 tag commit); merge that first.

Rebased onto current main (post-#616 squash) — single verified commit; the earlier stale release-branch history is gone.

@bussyjd
bussyjd requested a review from OisinKyne June 10, 2026 18:16
…n freshness

Every docker-publish-x402 branch build now lands a pin-bump commit
(repin-embedded-pins job) updating the embedded x402-verifier /
serviceoffer-controller / x402-buyer references to the images just built,
and the release workflow gains a verify-image-pins gate that fails the tag
when any source in the binaries' live import graph (go list -deps) changed
after the pinned build commit. Together they make the rc14 stale-pin trap
— a release whose embedded pins predate its own payment-path changes —
structurally impossible: the bump is automatic, and a tag cut before the
bump lands cannot release.

The bump is committed through the GraphQL createCommitOnBranch API, so
the commit is signed by GitHub itself (verified, github-actions bot) —
compatible with the repo ruleset rejecting unsigned commits, which a
workflow git push could never satisfy. expectedHeadOid is the live
remote head with one retry on race; only the two guarded template files
are ever sent.

The gate is fail-closed (a go-list failure refuses to pass rather than
degrade to a partial path set), binds each embedded digest to what GHCR
serves for the pinned tag (a fresh tag with a hand-edited digest fails),
ignores _test.go/testdata churn, and hunk-filters the two pin-carrying
templates so pin bumps don't self-stale while any other edit to them
still counts. release/** branches get the same build+repin treatment as
main.

The exact-ref pin tests become invariant tests: pins must be digest-
pinned, share one build commit, and descend from the named fix commits
(ancestry-verified via git, skipped on shallow clones) — so the bot can
bump pins without touching Go files while the carries-fix-X guarantees
get stronger.
@bussyjd
bussyjd force-pushed the chore/auto-repin-x402-images branch from 4b9bb7b to 93d8f62 Compare June 11, 2026 03:43
@OisinKyne OisinKyne closed this Jun 11, 2026
@OisinKyne
OisinKyne deleted the chore/auto-repin-x402-images branch July 1, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants