Node reuse over-provisioning heuristic keeps zero nodes when builds share a machine

[ViktorHofer]

DetermineNodesForReuse (added in #13220) compares a system-wide node count against 1.5 x logical cores. That assumes one build owns the machine. When several MSBuild builds run concurrently, every build ends up terminating all of its own nodes, so node reuse stops working.

Seen in the .NET VMR, which builds up to 8 repos in parallel, each running its own MSBuild. On an 8-core agent (threshold 12), comm traces show:

System-wide node count: 33, threshold: 12, this instance has: 16 nodes
Keeping 0 of 16 nodes in this instance to help meet threshold of 12

System-wide counts of 25/31/33/43 were normal for that build.

An instance can only terminate its own nodes

nodesToKeepInThisInstance = Math.Max(0, maxNodesToKeep - (systemWideNodeCount - nodeCount)) -> Max(0, 12 - 17) = 0.

The 17 nodes belonging to other builds are what pushed it over the limit, and they survive untouched. So the just-warmed nodes get killed while older idle ones persist, and the next build to finish does the same. It never converges.

An idle-expiry mechanism already exists

NodeEndpointOutOfProcBase.WaitForConnection waits CommunicationsUtilities.NodeConnectionTimeout for a host to reconnect and then exits (NodeEndpointOutOfProcBase.cs L419, "Connection timed out waiting a host to contact us"). The default:

// CommunicationsUtilities.cs L45
private const int DefaultNodeConnectionTimeout = 900 * 1000; // 15 minutes; enough time that a dev will typically do another build in this time

15 minutes, explicitly tuned for an interactive developer. In CI nobody "does another build in this time", so every orphaned node lingers for 15 minutes and the system-wide count climbs. That is most of the over-provisioning this feature is reacting to.

Suggestions

• Shorten the idle timeout of excess nodes rather than terminating them. NodeBuildComplete already carries PrepareForReuse, so it could also carry a timeout. Over-provisioned nodes then expire on their own in seconds instead of 15 minutes, warm nodes stay warm, and no instance has to make a global decision from local information.
• Lower the default timeout when ContinuousIntegrationBuild is set.
• Never let an instance go to zero: Math.Min(nodeCount, maxNodesToKeep) instead of the expression above. Bounded and convergent, and it stops sacrificing warm nodes for orphans the instance does not control.
• Add an escape hatch (e.g. MSBUILDNODEREUSETHRESHOLD, 0 = disabled). There is currently none: disabling change wave 18.5 makes things worse, because GetPossibleRunningNodes then returns every MSBuild.exe process unfiltered.

src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs L697-L745 - affects v11.0.0+ (repro on SDK 11.0.100-rc.1.26420.103)

Related: #15061 covers a second problem with the same feature: the node count is not filtered to handshake-compatible nodes.

[rainersigwald]

I don't think I understand--this sounds like a description of the exact behavior I expect, but phrased as negative.

When the machine is overprovisioned with nodes, we would like to reduce the count of nodes. But we can't take them away from any running build without breaking that build. So we close the nodes that are now idle. We could leave the nodes, but then we're using OS resources for idle nodes that may be used but may not--which was the earlier behavior that led to problems in many-parallel-builds situations.

[ViktorHofer]

That's a fair description of the intent, and the asymmetry argument holds — you can't reclaim another build's nodes without breaking it.

The gap I'd flag is the trigger rather than the action: CountSystemWideActiveNodes counts nodes that are busy serving live builds, not just idle ones. In these traces 249 of 318 connection probes timed out because the node was already attached to another host. So on an agent that builds continuously the count never falls below the threshold and the shed becomes unconditional — every instance computed max(0, 12 - others) => 0, and 2 of 72 nodes survived across 12 repo builds.

The net effect under that load is churn rather than reduction: if A keeps 12 and B reuses them, the machine holds 12; if A sheds and B cold-starts, it also holds 12, plus 12 teardowns and 12 creations. Cold start measured at ~750 ms mean.

For context this is worth ~1% of a 110-minute leg for us, so I'm not asking for a redesign — flagging it in case "overprovisioned" was meant to mean idle rather than total. If it's still intended as-is, happy for this to be closed.

[rainersigwald]

I think total makes more sense than idle, because active nodes can become idle as soon as the build completes. This causes churn in cases where builds are fully continuous, but can still produce reuse if there's ever an idle period:

1. build 1 starts
2. build 2 starts
3. build 2 finishes, closes all nodes
4. build 1 finishes, nodes go idle
5. build 3 starts, reuses 1's nodes

And in the "fully continuous" case there's never a time when idle nodes are consuming OS resources that could be going to the running builds. That's not necessarily optimal--the machine could have enough resources to spend that RAM to reduce the cost of a subsequent build--but it's the tradeoff we opted for to avoid the negative cases we were seeing where idle workers took gigabytes of memory and people were upset about it.