test: Go simnet harness for charon/pluto e2e duty tests

[varex83agent]

What

Adds test-infra/harness/, a Go test harness that runs simnet-style end-to-end attester duty tests over distributed-validator clusters mixing charon nodes (in-process, reusing charon's own app.Run) and pluto nodes (subprocesses of the pluto binary), plus a CI workflow.

Charon v1.7.1 (our parity reference) is consumed as a regular Go module dependency — no fork. The harness reuses its cluster.NewForT deterministic fixtures, testutil/beaconmock, testutil/validatormock and testutil/relay directly.

Design

                relay (in-process libp2p)
                beaconmock (shared chain state, deterministic duties)
                    ▲ Go              ▲ HTTP (per-node gateway + submission capture)
             charon app.Run      pluto run / charon run (subprocess)
             (in-process)             ▲ validator API
                                 validatormock (harness-driven over HTTP)

• Per-node HTTP beacon gateways over one shared beaconmock. Charon's beaconmock implements dynamic behaviour (duties, validators, attestation data) only on its Go client interface; its HTTP server serves static stubs. The gateway fronts the shared mock over real HTTP so external processes get a complete beacon API, and captures each node's submissions for assertions.
• Real p2p partial-signature exchange. Upstream simnet uses an in-memory ParSigEx transport; the harness omits it so out-of-process nodes participate in QBFT consensus and threshold signing.
• Assertions at the beacon boundary: for some slot, every node must submit an attestation, and all payloads must be identical after JSON normalization (group-signed threshold aggregates). The in-process baseline additionally uses upstream-style BroadcastCallback assertions.

Scenarios

Test	Cluster	Status
TestSimnetAttesterCharonInProcess	3× charon, in-process bmock	✅ passes (~5s)
TestSimnetAttesterCharonViaGateway	3× charon via HTTP gateway	✅ passes (~4s)
TestSimnetAttesterPluto	3× pluto	⏭ skips until pluto run lands
TestSimnetAttesterMixed	2× charon + 2× pluto, threshold 3	⏭ skips until pluto run lands

The pluto scenarios probe $PLUTO_BIN run --help and activate automatically once the run command exists. The subprocess path is already validated end-to-end today: pointing PLUTO_BIN at a charon binary (which accepts the same flags) passes the mixed scenario in ~5s, covering the on-disk fixture layout, run flags, readiness polling, HTTP-driven validator mock and capture assertions.

CI

.github/workflows/go-harness.yml:

• charon-simnet: charon-only scenarios, no Rust build — fast signal.
• pluto-simnet: builds pluto-cli (same steps as dkg-runner) and runs the full suite with PLUTO_BIN set.

Notes

• The two replace directives in go.mod are copied from charon's own go.mod (Go does not propagate a dependency's replaces); they must move in lockstep when bumping the pinned charon version.
• pluto run requirements for the scenarios to activate are documented in the README (charon-equivalent flags + validator API on the configured address).
• Follow-ups: proposer/sync-committee scenarios (direct translation of upstream simnet's beaconmock options) and below-threshold fault injection.

🤖 Generated with Claude Code

[varex83]

@varex83agent /review-pr

[varex83agent]

Summary

Adds test-infra/harness/ — a Go simnet harness running end-to-end attester-duty tests over clusters mixing in-process charon nodes and subprocess pluto nodes (charon v1.7.1 consumed as a plain module dependency) — plus four small, well-scoped production fixes: #[serde(default)] on partial_deposit_data, relay circuit-reservation address advertisement, Node::remove_external_address, and RUST_LOG > --log-level precedence.

Overall quality is high. The Rust fixes are correct and unit-tested where it matters; the serde(default) change is parity-safe (partial_deposit_data is part of the SSZ hash-tree-root covered by lock verification, so defaulting it to empty cannot bypass validation and matches charon's omitempty deserializer). The relay fix correctly gates auto-detected listen addresses behind filter_private_addrs.

Bugs (must-fix)

None.

Major

None.

Minor / nit

A handful of minor findings, none blocking:

1. Relay parity divergence — the manually-configured --p2p-external-ip path adds addresses to libp2p without an is-public filter, whereas go-libp2p strips private addrs from circuit reservations. Low impact (only under misconfiguration, and it adds rather than removes reachable addrs), but worth a filter or a documented note.
2. RUST_LOG precedence intentionally inverts charon's flag-beats-env rule — defensible, but the doc comment should call out the divergence.
3. Harness assertion coverage — the gateway/mixed path only asserts attestation equality, not the aggregator duties the in-process baseline checks.
4. Test-flakiness sources — port-allocation TOCTOU + SkipIfBindErr not covering subprocess bind failures; non-deterministic slot selection; t.Logf-after-test race under -race.

Plus assorted nits. Leaving as COMMENT for author discretion — nothing here blocks merge.

🤖 Reviewed by the multi-agent review pipeline.

[varex83agent]

Manually-configured external addresses bypass private-address filtering, and rust-libp2p has no go-libp2p reservation backstop.

This loop calls add_external_address for every entry in external_addrs (built from operator-supplied --p2p-external-ip/--p2p-external-hostname) with no filter_private_addrs/is-private gate. In rust-libp2p, add_external_address → ExternalAddrConfirmed feeds libp2p-relay's AcceptReservationReq { addrs } verbatim; libp2p-relay does not filter non-public addresses out of circuit reservations. go-libp2p does: makeReservationMsg (circuitv2/relay/relay.go) unconditionally skips !manet.IsPublicAddr(addr).

Net effect: a misconfigured private/loopback external addr is advertised inside circuit reservations under Pluto, whereas charon/go-libp2p would silently drop it. The auto-detected AddrUpdate::Add path (line 138) is correctly gated by filter_private_addrs, so only the manual path is affected. Impact is low (it adds addresses rather than breaking connectivity, and only under misconfiguration), but consider mirroring go-libp2p's is-public guard here or documenting the divergence.

[varex83agent]

nit: listeners.push(address) on AddrUpdate::Add is unconditional. If libp2p ever re-emits NewListenAddr for the same address (interface flaps / re-listens), listeners accumulates duplicates. This is masked downstream — web.rs's advertised-addrs dedups the external∪listeners union via a HashSet, and swarm.add_external_address dedups internally — so there's no user-visible drift. Correctness is fine; a dedup-on-push (or a set) would just be tidier for a long-lived relay.

[varex83agent]

RUST_LOG winning over an explicitly-passed --log-level inverts charon's precedence. Charon resolves as explicit --log-level flag > CHARON_LOG_LEVEL env > default (cmd/cmd.go bindFlags short-circuits on f.Changed), so an explicit flag always beats the env var there.

The change is defensible — RUST_LOG is the tracing-subscriber standard and supports per-target directives (info,pluto_p2p=debug) that charon's single-scalar level can't express, and charon has no RUST_LOG. But it's a genuine precedence divergence for that env var; the doc comment should state explicitly that this intentionally differs from charon (where the explicit flag wins over the env).

[varex83agent]

EnvFilter::try_new(&directive).unwrap_or_else(|_| default_env_filter()) swallows a parse error and silently drops to info. This preserves the pre-existing behavior (not a regression), but it's poor observability for exactly the fine-grained escape-hatch path the new doc comment advertises: an operator who typos RUST_LOG=infp,pluto_p2p=debug gets plain info with zero signal. Since tracing isn't initialized yet here, consider an eprintln! to stderr on the error arm before falling back. Not blocking.

[varex83agent]

Asymmetric blank-string handling: rust_log is rejected when trim().is_empty() (so Some(" ") falls through to the override), but override_env_filter is returned verbatim with no equivalent guard. A blank --log-level/CHARON_LOG_LEVEL (Some(" ")) is therefore forwarded to EnvFilter::try_new, fails to parse, and silently falls back to info — instead of cleanly returning None and using the default. The override is CLI-sourced so blank is unlikely, but the asymmetry is surprising given both are Option<&str> directives. Either apply the same trim().is_empty() filter to the override, or document the difference.

[varex83agent]

Port-allocation TOCTOU: testutil.AvailableAddr binds localhost:0, resolves the addr, and immediately closes the listener before the port is handed to the pluto subprocess. Between close and the subprocess's bind, another process/test can grab the port. A subprocess bind failure does not route through SkipIfBindErr — it crashes and surfaces as a WaitReady timeout, turning a flaky port race into a hard failure rather than a skip. Applies to the p2p/monitoring addrs here and in StartPlutoNode (lines 136-137). Consider retrying node startup on bind failure or passing the still-open listener fd.

[varex83agent]

The ServeHTTP switch enumerates known duty/submission endpoints and falls through to the reverse proxy for everything else. A POST to an endpoint the underlying mock doesn't implement (e.g. a future block-proposal or sync-message submission) is silently proxied and returns whatever the mock returns (often 404/405), which a node may treat as a transient error and retry indefinitely rather than failing the test clearly. Acceptable for the curated attester-only flow today, but a default that logs unhandled POSTs would close a latent completeness gap.

[varex83agent]

nit: the default: arm of versionedAttestationJSON sets inner = att.Electra and reports att.Version.String() as the version. For a genuinely unset/unknown att.Version, att.Electra may be nil, producing an inconsistent {version, data} pair (version says X, body is a nil Electra attestation) that the client could mis-decode. Prefer returning an error for unrecognized versions so a spec mismatch fails loudly in the test log.

[varex83agent]

nit: SimnetVMock: true is set unconditionally for every in-process charon node, including the gateway/mixed scenarios. That's intentional (charon drives its own in-process VC while pluto nodes get the harness HTTP vmock in vmock.go), but it means charon and pluto nodes are exercised through different validator-client drivers in the same interop test — charon via its scheduler-wired validatormock, pluto via the harness HTTP vmock with a wall-clock ticker. The interop assertion therefore compares outputs of two different vmock drivers, not one shared driver. Worth a comment here or in the README so the assertion's scope isn't overstated.

[varex83agent]

nit: both jobs share this paths filter including **/*.rs, **/Cargo.toml, and Cargo.lock. The charon-simnet job builds no Rust and only exercises charon in-process, so it re-runs on every Rust-only change that can't affect it. Only pluto-simnet legitimately needs the Rust paths. Splitting the filters per job would avoid redundant runs; a unified filter is a reasonable simplicity tradeoff if you'd rather keep it.