Describe the bug
In DoHealthChecksAsync, nodes with outdated LastHealthCheck are filtered before being transferred to tryStartLeadershipAsync. If the current node itself gets into the filter (aggressive StaleNodeTimeout, GC pause, DB lag), then self!.Assign Agents([LeaderUri]) on line 171 of the NodeAgentController.HeartBeat.cs file drops from NRE. By this point, the IsLeader has already been set to true, the lock in the database has been received, the AssumedLeadership has already been sent — the cluster remains in an inconsistent state, agents are not distributed correctly, other nodes are constantly falling off and being re-registered.
[ERR] Wolverine.Runtime.Agents.NodeAgentController: Error trying to attain a leadership lock
System.NullReferenceException: Object reference not set to an instance of an object.
at Wolverine.Runtime.Agents.NodeAgentController.tryStartLeadershipAsync(IReadOnlyList`1 nodes, AgentRestrictions restrictions)
in NodeAgentController.HeartBeat.cs:line 171
at Wolverine.Runtime.Agents.NodeAgentController.DoHealthChecksAsync()
in NodeAgentController.HeartBeat.cs:line 78
To Reproduce
Set opts.Durability.StaleNodeTimeout = TimeSpan.FromSeconds(30) and raise two nodes on the same circuit.
Expected behavior
either stepDownAsync with self == null, or add the current node to the list unconditionally after filtering the stale.
The problem was fixed by doing the following:
opts.Durability.StaleNodeTimeout = TimeSpan.FromSeconds(120);
Describe the bug
In DoHealthChecksAsync, nodes with outdated LastHealthCheck are filtered before being transferred to tryStartLeadershipAsync. If the current node itself gets into the filter (aggressive StaleNodeTimeout, GC pause, DB lag), then self!.Assign Agents([LeaderUri]) on line 171 of the NodeAgentController.HeartBeat.cs file drops from NRE. By this point, the IsLeader has already been set to true, the lock in the database has been received, the AssumedLeadership has already been sent — the cluster remains in an inconsistent state, agents are not distributed correctly, other nodes are constantly falling off and being re-registered.
To Reproduce
Set opts.Durability.StaleNodeTimeout = TimeSpan.FromSeconds(30) and raise two nodes on the same circuit.
Expected behavior
either stepDownAsync with self == null, or add the current node to the list unconditionally after filtering the stale.
The problem was fixed by doing the following:
opts.Durability.StaleNodeTimeout = TimeSpan.FromSeconds(120);