Integration test suite for the `run` command (simnet MVP, CI-runnable)

[emlautarom1-agent]

Summary

Add a minimal, CI-runnable integration test suite that validates the run command works end-to-end under normal conditions. The goal is not feature coverage — it is an MVP that proves a run cluster boots, forms a network, and completes duties, validated by launching the real binary and observing it from the outside, the same way scripts/dkg-runner/run.sh validates DKG.

Charon is the source of truth for what needs to be tested. Its integration suite lives in testutil/integration/ (gated by a runtime -integration flag).

Prerequisites (must exist before any test runs)

1. A run binary with a simnet mode. Charon exposes --simnet-beacon-mock and --simnet-validator-mock on charon run: the node runs its own in-process beacon-node mock (ticks slots, emits deterministic attester/proposer/sync duties) and its own validator-client mock (fetches, signs, submits). Without equivalent flags there is nothing deterministic to black-box.
2. On-host transports for a single-machine cluster. Charon's simnet runs consensus over libp2p via a relay and exchanges partial signatures in-memory. The node must talk to peers without a real beacon chain: a reachable relay + a partial-sig path that works cluster-locally.
3. Cluster artifact generation. A way to lay down N node directories (cluster lock + p2p key + BLS key shares) on disk — via the create cluster / create dkg CLI (already produced by the DKG runner).
4. A local relay the nodes can register with (Charon starts one in-process; a CI script can start a relay binary, or reuse a hosted one as the DKG runner does).
5. An external success signal. In-process, Charon asserts via a broadcast callback. For a black-box binary, the harness needs to read success from outside: a readiness/health/monitoring endpoint (peer count, beacon sync, duties seen) and/or structured logs it can grep for "duty broadcast".
6. A shell runner skeleton mirroring the DKG runner phases (setup → start-nodes → monitor → collect → verify), CI-gated.

The three high-value tests (Charon as source of truth)

Duty workflow (port of TestSimnetDuties)

• What Charon proves: a 3-node cluster runs the full pipeline per duty type — schedule → fetch → consensus → sign → partial-sig exchange → threshold-aggregate → broadcast. The load-bearing assertion is agreement: every node produces the same aggregated signature for a given duty, and each expected duty type fires within a couple of slots.
• MVP scope: the attester path alone proves the whole pipeline. Proposer / sync-committee / builder-registration are the "matrix" — defer to a nightly variant.
• Black-box realization: launch N run nodes in simnet mode against their beacon mocks, wait a few slots, assert each node reports a successfully broadcast attestation and that the signatures agree.

P2P connectivity (port of TestPingCluster)

• What Charon proves: nodes discover each other, connect through the relay, upgrade to direct connections, and reach a full ping mesh (variants: direct, relay, wrong-external-IP fallback).
• MVP scope: just that the cluster forms a fully-connected mesh — a precondition for the duty-workflow test, so it is cheap to assert first.
• Black-box realization: poll each node's monitoring endpoint until connected-peer count reaches N-1 within a timeout.

Black-box CLI (port of the compose TestSmoke)

• What Charon proves: the actual charon run binary, in simnet mode, reaches a healthy state across small topologies.
• Key insight: this is not a separate test so much as the harness that the duty-workflow and connectivity tests run inside. Booting the real binary in simnet mode, confirming readiness, running one full duty cycle, and shutting down cleanly is the smoke test — and it maps directly onto the DKG-runner model.

In Charon these are three separate in-process Go tests. For a CI-first Pluto MVP they collapse into one black-box harness: the black-box CLI harness is the scaffolding (boot + ready + clean exit), the connectivity test is an assertion on the running cluster (mesh formed), and the duty-workflow test is an assertion on the same cluster (duties flow and agree).

CI execution model (mirror scripts/dkg-runner/run.sh)

A sibling runner (e.g. scripts/simnet-runner/) with the same phase structure and ergonomics (PID/pgid tracking, TIMEOUT, CI-aware logging, work dir preserved on failure):

Phase	DKG-runner analog	Simnet runner does
setup	setup.sh	generate N node dirs (lock + p2p key + shares) via CLI
start-nodes	start-nodes.sh	launch N run --simnet-* processes against a local relay
monitor	monitor.sh	poll health API within TIMEOUT: mesh formed → nodes ready → ≥1 duty broadcast
collect	collect.sh	gather per-node logs + a health snapshot
verify	verify-*.sh	assert mesh size, duty broadcast, signature agreement across nodes, clean exit

Gate it behind a CI flag; run the attester-only 3-node case on every PR (fast, deterministic) and the full duty matrix / fault variants nightly.

Suggested sequencing

1. Prereqs — simnet flags on run (beacon + validator mock), cluster-artifact generation, a local relay, and a readiness/health signal to poll.
2. Black-box CLI skeleton — the runner boots a 3-node simnet cluster, confirms all become ready, then shuts down cleanly. Immediate "does run boot" signal and the scaffold everything layers onto.
3. Connectivity assertion — extend the monitor phase to require the full peer mesh before proceeding.
4. Duty workflow (attester) — assert at least the attester duty flows end-to-end and all nodes agree on the aggregated signature. This is the core correctness win.
5. Expand (nightly) — proposer / sync-committee / builder-registration duty matrix, then a t-of-n fault case with one node down.

Charon references

• testutil/integration/simnet_test.go — TestSimnetDuties (duty workflow)
• testutil/integration/ping_test.go — TestPingCluster (P2P connectivity)
• testutil/compose/smoke/smoke_test.go — TestSmoke (black-box CLI)
• app/app.go — TestConfig / SimnetBMock / SimnetVMock seams
• testutil/beaconmock/, testutil/validatormock/, cluster.NewForT, testutil/relay/ — simnet building blocks

[emlautarom1-agent]

Status update — integration testing (simnet MVP)

All code prerequisites are merged. The simnet plumbing the suite depends on is done and manually verified live (3-node cluster + relay completing all duty types, 2-of-3 fault tolerance, clean shutdown):

• Simnet mode on run — feat(app): implement simnet integration #553 (--simnet-beacon-mock/--simnet-validator-mock/--simnet-slot-duration/--simnet-validator-keys-dir/--simnet-beacon-mock-fuzz, in-process mocks, slot-tick seam), plus feat(app): wire the core node and implement the run command #536/feat(cmd): implement run and unsafe run #503 (run cmd), feat(testutil): extend beaconmock for the simnet duty cycle #545 (beaconmock duty cycle), fix(validatormock): submit Electra wire formats for simnet #547 (validatormock Electra), fix(scheduler): emit to subscribers asynchronously #544 (async scheduler).
• Transports — fix(p2p): recover relay boot races so cluster peers stay routed #554 (relay boot-race recovery), feat(parsigex): add eth2-based partial signature verifier #511 (parsigex verifier), fix(peerinfo): serve inbound requests on Inactive connections #542 (peerinfo inbound).
• Health/readiness signal — feat(app): implement health #498, feat(app): add monitoring API for liveness and readiness #500, feat(app): wire the monitoring API into pluto run #539.
• Cluster artifacts — CLI + feat(cluster): add load_cluster_lock (backport Charon #4130) #523 (load_cluster_lock).

In-process test coverage merged (not black-box): crates/app/tests/simnet.rs (single-node boot + readiness) and wiring.rs::multinode_parsig_exchange_reaches_submission (3-node partial-sig → SigAgg → submit oracle).

Remaining deliverable — the CI-runnable black-box harness — is not merged; it's in open PRs:

• test: Go simnet harness for charon/pluto e2e duty tests #526 — test-infra/harness/ Go e2e harness (attester duties, mixes charon in-process + pluto subprocess nodes, reuses charon v1.7.1 test primitives) + go-harness.yml CI workflow.
• test(app): add smoke test #541 — docker-compose smoke tests porting charon's testutil/compose (all_pluto 4-node passing).

Deviation from the plan: the proposed scripts/simnet-runner/ bash runner was not built. The equivalent is being delivered as the Go harness (#526) + compose smoke tests (#541), consuming charon's own mocks/relay directly rather than reimplementing them. No simnet CI workflow is on main yet.

Bottom line: enabling infra complete; the automated black-box CI suite is in review (#526, #541), not yet landed.