Context
A zeam node on devnet-eth2 wedged: head frozen 5+h, every blocks_by_root/status request returning Disconnected ("no conn to peer") and IoError, 0 peer_connected and 0 dial events in 10 minutes despite the metric reporting 15 connected peers, and gossip_silent_ms=36006 with mesh_peers=26 (36s of zero gossip). The transport-facing symptom is that zeam's peer view was decoupled from its actual live connections.
The primary root cause was on the zeam sync side (blockblaz/zeam#1046 — a false "peer does not support blocks_by_range" that degraded catch-up into a per-block storm). But the trigger underneath is here: connections went away and the app kept believing peers were connected, never re-establishing them.
What's already correct (verified)
connection_manager.onConnectionClosed emits peer_disconnected, calls req_resp.onPeerDisconnected, and re-arms a redial for known peers (retry-forever with capped backoff, reconnectDelayMs).- QUIC transport params:
max_idle_timeout = 30s, app-layer /meshsub keepalive = 20s — healthy conns stay up; genuinely idle-dead conns should be reaped within 30s and flow through detectOutboundConnectionClose → onConnectionClosed.
So for outbound / known peers the loop is sound.
Gaps to investigate
-
Inbound-only peers are never re-dialed. onConnectionClosed only re-arms a dial when self.known.getPtr(peer) hits. On a full mesh ~half of every node's peers are inbound-only (they dialed us; we never registered them as known). When such a conn dies, we emit peer_disconnected but never attempt to reconnect — and if the remote doesn't re-dial either, the peer is lost until restart. Consider: capture a dialable addr for inbound peers via identify/signed-peer-record and make them redial-eligible (guarded against NAT-undialable peers and against dial churn).
-
Silent conn loss with no lifecycle event. The 0-peer_connected/0-dial + stale-book + 36s gossip-silence pattern suggests either (a) the idle-timeout close wasn't surfaced for some conns, or (b) the drive loop / event propagation stalled while the by-root retry path kept running. Needs a live repro to distinguish; instrument detectOutboundConnectionClose / the inbound syncSeenFlags poll and the peer_connected/peer_disconnected emit counts vs. the conn_table size.
-
Defense-in-depth (quinn/lighthouse parity): inbound req/resp rate limiting so a storming peer (or a buggy remote sync loop like the one in #1046) is throttled rather than allowed to thrash connections.
Why not patched blind here
The redial/idle/keepalive machinery is correct and this file set is heavily scarred by past dial-churn regressions (see the many "no eager redial / duplicate-connection churn" comments). A speculative change to inbound-peer redial or close-detection without a reproduction is likely to reintroduce churn. Filing to fix with a repro. The sync-layer amplifier (the thing that turned this into a multi-hour wedge) is fixed in blockblaz/zeam#1046.
Context
A zeam node on devnet-eth2 wedged: head frozen 5+h, every
blocks_by_root/statusrequest returningDisconnected("no conn to peer") andIoError, 0peer_connectedand 0dialevents in 10 minutes despite the metric reporting 15 connected peers, andgossip_silent_ms=36006withmesh_peers=26(36s of zero gossip). The transport-facing symptom is that zeam's peer view was decoupled from its actual live connections.The primary root cause was on the zeam sync side (blockblaz/zeam#1046 — a false "peer does not support blocks_by_range" that degraded catch-up into a per-block storm). But the trigger underneath is here: connections went away and the app kept believing peers were connected, never re-establishing them.
What's already correct (verified)
connection_manager.onConnectionClosedemitspeer_disconnected, callsreq_resp.onPeerDisconnected, and re-arms a redial forknownpeers (retry-forever with capped backoff,reconnectDelayMs).max_idle_timeout = 30s, app-layer/meshsubkeepalive =20s— healthy conns stay up; genuinely idle-dead conns should be reaped within 30s and flow throughdetectOutboundConnectionClose→onConnectionClosed.So for outbound / known peers the loop is sound.
Gaps to investigate
Inbound-only peers are never re-dialed.
onConnectionClosedonly re-arms a dial whenself.known.getPtr(peer)hits. On a full mesh ~half of every node's peers are inbound-only (they dialed us; we never registered them asknown). When such a conn dies, we emitpeer_disconnectedbut never attempt to reconnect — and if the remote doesn't re-dial either, the peer is lost until restart. Consider: capture a dialable addr for inbound peers via identify/signed-peer-record and make them redial-eligible (guarded against NAT-undialable peers and against dial churn).Silent conn loss with no lifecycle event. The 0-
peer_connected/0-dial+ stale-book + 36s gossip-silence pattern suggests either (a) the idle-timeout close wasn't surfaced for some conns, or (b) the drive loop / event propagation stalled while the by-root retry path kept running. Needs a live repro to distinguish; instrumentdetectOutboundConnectionClose/ the inboundsyncSeenFlagspoll and thepeer_connected/peer_disconnectedemit counts vs. the conn_table size.Defense-in-depth (quinn/lighthouse parity): inbound req/resp rate limiting so a storming peer (or a buggy remote sync loop like the one in #1046) is throttled rather than allowed to thrash connections.
Why not patched blind here
The redial/idle/keepalive machinery is correct and this file set is heavily scarred by past dial-churn regressions (see the many "no eager redial / duplicate-connection churn" comments). A speculative change to inbound-peer redial or close-detection without a reproduction is likely to reintroduce churn. Filing to fix with a repro. The sync-layer amplifier (the thing that turned this into a multi-hour wedge) is fixed in blockblaz/zeam#1046.