noderesources: exempt annotated pods from max-pod-count cap

[eakman-datadog]

Datadog: NOT FROM UPSTREAM K8s.

The purpose of this draft PR is to display testing evidence for a custom patch on 1.34 I'm working on.

I tested the PR using kind

Get the current number of pods on the kind node and fill it to the max with filler pods

➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ CURRENT=$(kubectl get pods -A --field-selector spec.nodeName=pod-exemption-worker --no-headers | wc -l | tr -d ' ')
➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ REPLICAS=$((10 - CURRENT))
➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: filler
spec:
  replicas: $REPLICAS
  selector:
    matchLabels:
      app: filler
  template:
    metadata:
      labels:
        app: filler
    spec:
      nodeSelector:
        kubernetes.io/hostname: pod-exemption-worker
      tolerations:
        - operator: Exists
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: 1m
              memory: 1Mi
EOF
deployment.apps/filler created
➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ kubectl rollout status deployment/filler --timeout=60s
deployment "filler" successfully rolled out

Test that non-exempt pods are not scheduled

➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ kubectl run canary --image=registry.k8s.io/pause:3.9 \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"pod-exemption-worker"}}}'
pod/canary created
➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ kubectl describe pod canary | grep -A5 Events
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  10s   default-scheduler  0/2 nodes are available: 1 Too many pods, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling.
➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ k delete pod canary
pod "canary" deleted from default namespace

Test that exempt pods get scheduled

➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: exempt
  annotations:
    kubelet.datadoghq.com/exclude-from-max-pods: "true"
spec:
  hostNetwork: true
  nodeSelector:
    kubernetes.io/hostname: pod-exemption-worker
  containers:
    - name: pause
      image: registry.k8s.io/pause:3.9
EOF
pod/exempt created
➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ kubectl wait pod/exempt --for=condition=Ready --timeout=30s
pod/exempt condition met
➜  kubernetes git:(ea/max-pod-count-exemption-v1.34) ✗ k get pod exempt -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE                   NOMINATED NODE   READINESS GATES
exempt   1/1     Running   0          24s   172.18.0.3   pod-exemption-worker   <none>           <none>

[datadog-official]

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/kubernetes | goreleaser-snapshot     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bd3f76b | Docs | Datadog PR Page | Give us feedback!}