feat: Support pod resources, scheduling, and priority in the worker pool CRD

[han2ni3bal-pixel]

Problem
WorkerPool only exposes spec.replicas and spec.ateomImage. The reconciler hard-codes the worker Pod template with no resources, priorityClassName, nodeSelector, affinity, or tolerations.

Worker pods therefore run as BestEffort / priority 0, are easily evicted under node pressure, and cannot be declaratively pinned to dedicated or larger nodes. This is awkward in mixed-workload or heterogeneous clusters, and it conflicts with docs/architecture.md, which describes WorkerPool as defining hardware shape.

Resume also picks a free worker at random (workflow_resume.go), with no awareness of node capacity.

Impact

1. Worker pods are among the first evicted during resource pressure
2. A terminated worker can leave actors stranded in RUNNING (related: Handle Pod Eviction/Deletion gracefully #23)
3. No supported way to isolate workers or prefer certain nodes via the WorkerPool API
4. replicas means “warm slots”, not CPU capacity

Proposal
Add a template (or equivalent) field to WorkerPoolSpec so operators can configure at least:

• resources (requests/limits)
• priorityClassName
• nodeSelector, tolerations, affinity
• extra labels/annotations
  Propagate these into the managed Deployment Pod template and document an example in docs/api-guide.md.

Workaround today
Manually patch the generated Deployment, or rely on cluster-level node pools/taints. Neither is reflected in the WorkerPool CR.

[juli4n]

Let's dissect the proposal, because I think it touches on different areas. Some of them overlap with other issues that have been reported already.

1. Worker pods always have QoS BestEffort.

We don't have a way to specify resource requirements in the WorkerPod spec today. This means that all worker pods have QoS BestEffort. This makes them first in row for eviction, but it also seems like a feature gap to me. I feel it's a reasonable feature request to add resource requirements to the WorkerPool spec.

2. We don't handle Pod deletion gracefully.

As you pointed out, this is already reported in #23. I would leave it out of scope from this issue.

3. No supported way to isolate workers or prefer certain nodes via the WorkerPool API

Agree. This is a gap today. We have #47 which has a proposal to address it. See the proposal in #47 (comment). Is that expressive enough to accommodate your use cases?

4. No way to propagate labels and annotations to worker pods.

This looks similar to point 1. Sounds like a reasonable feature request to add this to the WorkerPool spec.

As a path forward, I think we could either scope down this issue to handle point 1 and point 4 or we can file separate issues for them (they seem orthogonal to each other).

[han2ni3bal-pixel]

Julian Gutierrez Oschmann (@juli4n) #47 provides a very good direction, and label/selector-based matching seems like a useful starting point.

My thought is that actor-to-worker placement could potentially be modeled in a slightly more expressive way, similar in spirit to how Kubernetes schedules Pods onto Nodes. For example, instead of only relying on a hard label match, the actor scheduler could use lightweight filtering and scoring based on affinity, labels/selectors, worker pool type, resource shape, hardware/spec type, locality, or other placement hints, and then choose the most suitable worker.

This does not need to be as complex as the Kubernetes scheduler. It could remain lightweight enough to preserve Substrate’s fast actor scheduling/resume path, while still providing more flexible placement semantics.

[han2ni3bal-pixel]

Julian Gutierrez Oschmann (@juli4n) Plus, thanks for the clarification. I think it makes sense to keep #47 focused on actor-to-worker matching, while using this issue to address the lower-level WorkerPool-to-Kubernetes-Node scheduling gap.

For an initial PR, I’m thinking about keeping the scope intentionally small and only exposing a controlled subset of scheduling/resource fields for WorkerPool-managed Pods:

• resources
• nodeSelector
• tolerations
• priorityClassName
• nodeAffinity

The goal is not to expose a full PodTemplateSpec, but to provide enough control for WorkerPool Pods to request resources, get proper QoS, run on dedicated/preferred nodes, and tolerate tainted node pools.

I would avoid supporting full affinity in the first version, since podAffinity / podAntiAffinity may add more complexity than needed for the initial WorkerPool placement use case. If this scope sounds reasonable, I’d be happy to work on an initial PR.

[juli4n]

https://docs.google.com/document/d/1_i__JEdmRexKOGP4VJ7bOWrgdS9vPd-BQSR0sxg7GQ4/edit?resourcekey=0-jtYHhxfAGZCNcoknXn2GbA&tab=t.0 summarizes a proposal to address this.