ApplyRestrictionsAsync never dispatched the commands it computed (GH-698)

[jeremydmiller]

Upstream half of CritterWatch#698 (@erdtsieck — the console's "pause projection" acks Succeeded but the agent never pauses). Targeting 6.18.0.

The bug

IAgentRuntime.ApplyRestrictionsAsync persisted the restriction row and then had no immediate effect — the agent kept running.

EvaluateAssignmentsAsync returns the commands that carry a restriction out to the cluster: pausing an agent detaches it from the assignment grid, which produces a StopRemoteAgent. Every other caller pumps that return value through a MessageBus — the health-check loop does exactly this. ApplyRestrictionsAsync threw it away:

await NodeController!.EvaluateAssignmentsAsync(nodes, assignments);   // <-- return value discarded

So the pause only took hold whenever some later leader heartbeat happened to re-read restrictions from the database. Any caller that treated the method returning as "it is paused now" — which is exactly what CritterWatch's pause handler does — was acking a no-op.

Second defect, same method

The in-memory WolverineRuntime.Restrictions was never refreshed. It is otherwise only loaded once at startup (WolverineRuntime.HostService.cs), but ListeningAgent reads it on every start attempt to decide whether a listener is meant to stay paused. So a listener paused through restrictions would keep restarting itself until the node restarted.

The fix

ApplyRestrictionsAsync now drains the returned AgentCommands cascade (commands can yield further commands) and refreshes the in-memory copy. The drain loop is shared with the health-check loop rather than copied a third time.

Test

pausing_an_agent_stops_it_immediately, added to LeadershipElectionCompliance so it runs for every provider.

It disables health checks first, on purpose. The health-check loop re-reads restrictions from the database and would eventually stop the agent anyway — a polling assertion would therefore pass even against the broken code and prove nothing. The defect is the absence of an immediate effect, so the test asserts synchronously once ApplyRestrictionsAsync returns.

• Verified red against the old code (fails), green with the fix.
• Full Postgres leader-election suite 14/14 green — that covers the health-check/agent-distribution path the shared drain loop touches.
• Full wolverine.slnx Release build clean.

ApplyRestrictionsAsync had zero test coverage before this, which is what let it ship.

Does this close CritterWatch#698?

It fixes a real defect that produces exactly the reported symptom, but not necessarily the reporter's wolverine_agent_restrictions = 0 rows. The persist path is unconditional and agent-family-agnostic, so I cannot statically explain zero rows; the restriction should have been written even before this fix. The leading remaining hypothesis is that the command executed against a different Wolverine runtime than the one inspected (the console has its own compiled PauseProjectionHandler), which would put the row in the console's database. That is being followed up on the CritterWatch side, along with verify-before-ack so a green ack can never again mean nothing happened.

🤖 Generated with Claude Code