Node reuse node count includes nodes that can't be reused by the counting instance

[ViktorHofer]

CountActiveNodesWithMode / GetPossibleRunningNodes filter only on process name MSBuild plus NodeMode.OutOfProcNode in the command line. Nothing checks whether a counted node is actually usable by the counting instance, so the count includes nodes that instance can never reuse. That count then feeds DetermineNodesForReuse, which terminates the instance's own nodes.

Measured on a .NET VMR CI agent building up to 8 repos in parallel, 8 cores, threshold 12. Counts logged at build completion:

repo build	system-wide count	owned by this instance	kept
arcade	18	8	2
symreader	30	4	0
xdt	31	4	0
cecil	33	6	0
emsdk	46	4	0
nuget-client	59	8	0
fsharp	69	14	0

Two independent reasons a counted node is unusable, both observed:

1. The node is busy. It is attached to another concurrently running build's host, so the pipe connect times out. In a set of repo builds that all shared one toolset (.dotnet/sdk/11.0.100-rc.1.26420.103), probing produced 147 pipe-connect timeouts and 0 handshake rejections. The probing path already anticipates this:

// A candidate belongs to another build and may exit, be killed, hit its idle timeout,
// or refuse a handshake at any moment, so probing it is inherently racy.

2. The node belongs to a different toolset. Repos that use VS MSBuild for part of their build run two toolsets side by side - C:\Program Files\Microsoft Visual Studio\18\Enterprise\MSBuild\Current\Bin and the bootstrapped SDK - within a single repo build. Because toolsDirectory is part of the handshake salt, those nodes correctly reject each other. In those legs probing produced 60 handshake rejections ("Handshake failed on part 1. Probably the client is a different MSBuild build") against 34 timeouts.

Note this needs no MSBUILDNODEHANDSHAKESALT; the default salt already encodes the tools directory, so the rejections are the handshake working as designed.

Either way the node is not a reuse candidate, and either way it still counts toward the threshold that terminates nodes which are reuse candidates.

The instance already has this information, then discards it

Every probed candidate is recorded in _processesToIgnore (L370), keyed {handshake}|{pid}. ShutdownConnectedNodes clears that dictionary (L117) and calls DetermineNodesForReuse nine lines later (L126).

Suggestions

• Move the _processesToIgnore clear after the reuse decision and subtract those PIDs from the count. Nearly free, and it covers both cases above, since both end up recorded there.
• Count only nodes that are genuinely reuse candidates. A busy node belongs to a live build and will be handled by its own owner at completion.
• Longer term, fold a handshake hash into the pipe name (for example MSBuild.{handshakeHash}.{pid}) so compatible nodes can be counted by enumerating pipe names, with no process enumeration and no connect attempt. That makes case 2 free to detect.

src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs L763-L841 - affects v11.0.0+

Related: #15060

[JanProvaznik]

Isn't that the intent rather than a bug?

If machine has a lot of MSBuild processes online we don't want more to linger.

[ViktorHofer]

@JanProvaznik The goal isn't in dispute - shedding nodes when the machine is crowded is reasonable. The objection is that the count doesn't measure what it's being used for, so the mechanism doesn't achieve that goal.

An instance can only terminate nodes it owns. In the trace above, 17 of the 33 counted nodes belong to other builds; those are the ones that pushed this instance over the threshold, and they are exactly the ones that survive. What gets terminated instead are the 16 nodes this build just finished warming. The machine ends up with the same number of lingering processes, only now they're the stale ones. Repeat across concurrent builds and node reuse stops working entirely while the count stays high.

On lingering specifically: that's already bounded. NodeEndpointOutOfProcBase.WaitForConnection exits after NodeConnectionTimeout, and idle nodes do go away on their own. The default is 15 minutes, with a comment saying it's sized so "a dev will typically do another build in this time" - reasonable interactively, far too long for CI. That value looks like the real lever for the problem you're describing, and it doesn't require one instance to make a global decision from local information. More on that in #15060.

So the ask here is narrower than "don't shed nodes": filter the count to nodes this instance could actually reuse. If the 17 incompatible ones were excluded, this build would have kept 12 rather than 0, which is what the threshold intends.

[ViktorHofer]

Correction to my earlier reply: I attributed the unreusable nodes to handshake salt differences. That was wrong, and I have rewritten the issue body with what the traces actually show.

There are no salt or SDK differences here. Every node on the agent is compatible, and 38 reuse connections did succeed. The nodes that get counted but cannot be used are simply busy - attached to another concurrently running build's host, so the pipe connect times out. 249 timeouts against 69 successful connections across 11 parallel repo builds.

That reframes your question rather than dismissing it. You are right that 69 MSBuild processes on one agent is real load. But they are load because 8 builds are actively running, not because anything is lingering. Terminating my instance's 14 nodes does not reduce that number by one - those 69 keep working for their own hosts. And when each of those builds finishes it runs the same calculation and reaches the same conclusion, so node reuse ends up disabled machine-wide for the duration.

So the metric and the goal have drifted apart: the feature wants to limit lingering idle nodes, and the count measures all nodes including busy ones. During parallel builds those are nearly disjoint sets.

[ViktorHofer]

Data from a second CI leg adds a case I had ruled out too quickly above, so the issue body now covers both.

Repos that build partly with VS MSBuild run two toolsets side by side within a single repo build - C:\Program Files\Microsoft Visual Studio\18\Enterprise\MSBuild\Current\Bin and the bootstrapped SDK. On those legs probing produced 60 handshake rejections against 34 pipe timeouts, all "Handshake failed on part 1", i.e. the salt, which encodes toolsDirectory. On legs where every repo shared one toolset it was the reverse: 147 timeouts and 0 rejections.

So the unusable-but-counted population has two sources - busy nodes, and nodes from a genuinely different toolset - and the proportion depends on which repos happen to overlap. Neither is filtered out of the count, and both push instances that do have reusable nodes down to keeping zero.

Worth noting the rejections are the handshake working correctly, with no MSBUILDNODEHANDSHAKESALT set. The gap is only that the counting path does not apply the same knowledge the connecting path does.

[rainersigwald]

The gap is only that the counting path does not apply the same knowledge the connecting path does.

This is fairly intentional. The goal of the resource-reduction effort is to reduce resource usage after concurrent builds, and "the builds used different SDKs" does not really factor into that decision IMO.

[ViktorHofer]

Fair — if the count is a resource measure rather than a reusability measure, then filtering it by toolset is the wrong ask.

The concern that survives is that the count includes nodes busy serving live builds, which makes the shed unconditional under continuous load. That's the same lever as #15060, so I've moved the data there. Closing this as a duplicate.