Stable node identity + lifecycle for storeless Solo hosts (#3188)

[jeremydmiller]

Fixes #3188.

Problem

A Solo host with no durable message store (NullMessageStore) never runs StartSoloModeAsync() — it sits behind the storeless early return in startAgentsAsync() (WolverineRuntime.Agents.cs:148). StartSoloModeAsync() is the only place a Solo host gets a stable AssignedNodeNumber = 1 and a NodeStarted() notification, so a storeless Solo host instead keeps the constructor default:

// DurabilitySettings.cs
public int AssignedNodeNumber { get; internal set; } = Guid.NewGuid().ToString().GetDeterministicHashCode();

…a random value that differs on every process start. Its WolverineHeartbeat pings (and any node-keyed signal) carry that churning id, and it never emits NodeStarted()/NodeStopped(). Downstream monitors (CritterWatch JasperFx/CritterWatch#510) see phantom, restart-churning nodes that never reconcile with a lifecycle record. AssignedNodeNumber has an internal set, so this has to be fixed inside Wolverine.

Host shape	AssignedNodeNumber	NodeStarted()/NodeStopped()
Solo + durable store	1 ✅	fired ✅
Solo + NullMessageStore	random per process ❌	never fired ❌
Balanced	assigned via election	fired

Fix

A lightweight, store-independent liveness path for Solo, split so the timing is correct (identity must be set before transports start; the NodeStopped() bookend must fire while transports are still up, since a publish-based observer like CritterWatch emits a record there):

• Identity (runtime). The runtime now assigns AssignedNodeNumber = 1 for a storeless Solo host in startAgentsAsync(), before startMessagingTransportsAsync() — envelope ownership and the heartbeat both read it.
• Lifecycle bookends (SoloHeartbeatService). A new IHostedService owns NodeStarted()/NodeStopped() for storeless Solo only. It is registered after the runtime hosted service, so NodeStarted() fires once transports are up and — because hosted services stop in reverse order — NodeStopped() fires before the runtime tears the transports down. No-ops unless the host is Solo with a NullMessageStore.
• Default observer tolerance. PersistenceWolverineObserver.NodeStarted()/NodeStopped() now catch NotSupportedException (matching LostLeadership), so the default observer doesn't throw when a storeless Solo node fires the bookends against a NullMessageStore.

Stored Solo (StartSoloModeAsync) and Balanced are untouched — the new service no-ops for them, so there is no double-NodeStarted/NodeStopped.

Design note: the issue floated a single host-ordered IHostedService that also sets identity. That can't satisfy both constraints at once — a service registered before the runtime stops after it (transports already gone → NodeStopped publish fails), and one registered after starts after the runtime (identity set too late). Splitting identity (runtime, pre-transport) from the lifecycle bookends (service, registered after the runtime) is what makes the timing correct. Confirmed with the issue author before implementing.

Tests

solo_storeless_node_identity (CoreTests, no infra):

• storeless_solo_gets_node_1_and_fires_lifecycle_bookends — AssignedNodeNumber == 1, NodeStarted() once at boot, NodeStopped() once on clean stop.
• storeless_solo_node_number_is_stable_across_restart — two boot/stop cycles both yield 1 (no GUID-hash churn).
• does_not_fire_for_a_storeless_non_solo_host — a storeless Balanced host gets neither bookend.

The two positive tests fail without the fix (random node number; bookends never fire) and pass with it. Full CoreTests suite green.

🤖 Generated with Claude Code

[jeremydmiller]

Pushed reply-routing regression fixes uncovered while validating this change.

The #3188 "Solo == node 1" change means every storeless Solo host gets AssignedNodeNumber = 1. Transports that key their per-node reply endpoint on the node number with no service discriminator then collide when two Solo services share a broker, cross-delivering each other's replies (request/reply compliance: "Expected one message … but detected 2"). Fixed in Solo mode by keying on the always-unique UniqueNodeId ("N" format — dash-free hex, safe for subject/topic names); Balanced is unchanged (its AssignedNodeNumber is unique via election):

• NATS — NatsTransport (validated against broker: request/reply 12/12, full suite 121/121)
• MQTT — MqttTransport + updated has_response_topic_automatically assertions (validated: request/reply 12/12)
• GCP Pub/Sub — PubsubTransport (reproduced against the emulator: 7 request/reply failures → 16/16 green after fix)

Redis / Azure Service Bus / AWS SQS already namespace their response endpoint by ServiceName, so they were unaffected.

GCP Pub/Sub has no CI job, which is why its collision wasn't caught automatically — tracked in #3191.