Taints as the node's veto and the NoExecute effect that evicts what is already running, affinity as the pod's request, the topologyKey that gives anti-affinity its meaning, and why Pending is always a filtering result you can read verbatim.
Filter, then score. Every Pending pod is a filter that emptied the list, and the scheduler tells you which one.
Scoring never rescues a pod that failed filtering — it only ranks the survivors. That is why "add more nodes" does not fix an affinity rule no node can satisfy.
The diagram above is the high level: what the pieces are. These two are the ones you want when something is wrong — what is inside one of those boxes, and the path a request really takes through them.
One belongs to the node and one to the pod; they are routinely confused for each other.
A taint is on the node and repels pods. A toleration is on
the pod and says "this one is allowed anyway". Note the direction: a toleration does not
attract a pod to a node — it only removes an objection. Wanting a pod to land on
specific nodes needs nodeSelector or affinity as well.
| Effect | New pods | Already-running pods |
|---|---|---|
NoSchedule | Rejected unless tolerating | Left alone |
PreferNoSchedule | Avoided if possible | Left alone |
NoExecute | Rejected unless tolerating | Evicted unless tolerating |
That last row is the one that causes surprise outages: adding a NoExecute taint
to a node evicts everything on it that does not tolerate it, immediately.
Kubernetes taints nodes automatically on conditions — not-ready,
unreachable, memory-pressure, disk-pressure,
pid-pressure, unschedulable. The first two are
NoExecute with a default 300-second toleration injected into every pod, which is
why a node going unreachable takes five minutes to shed its pods. Shortening that
tolerationSeconds for latency-sensitive workloads is a real tuning knob, and one of
the few places where a default is too conservative rather than too aggressive.
Where taints are the node's veto, affinity is the pod's request. Two families:
nodeSelector, with
operators (In, NotIn, Exists, Gt,
Lt).Each comes in two strengths, and the names are long enough that people copy them without
reading: requiredDuringSchedulingIgnoredDuringExecution is a hard filter — unmet
means Pending forever. preferredDuringSchedulingIgnoredDuringExecution is a
scoring hint — unmet just means a lower score. IgnoredDuringExecution in both
names is a promise: once bound, a pod is never moved because the rule stopped holding.
Anti-affinity says "not co-located", and topologyKey defines co-located.
kubernetes.io/hostname means one per node. topology.kubernetes.io/zone
means one per zone — which, with three replicas and three zones, is exactly what you want, and
with four replicas leaves one Pending forever if the rule is required.
"Spread my replicas evenly" is better expressed with
topologySpreadConstraints than with anti-affinity: it takes a
maxSkew, so it degrades gracefully instead of wedging, and
whenUnsatisfiable: ScheduleAnyway gives you best-effort spreading that cannot cause
a Pending pod.
Scheduling rules are evaluated once, at binding. If all three replicas end up in one zone because the other two were briefly full, they stay there after capacity returns — nothing rebalances them. A rolling restart is the rebalance, and descheduler is the tool if you want it continuous.
Pending always means the same thing: filtering produced an empty list.
The scheduler says which predicate failed and on how many nodes, and that sentence is the entire
diagnosis — it is just easy to skim past.
| Message fragment | Cause | Fix |
|---|---|---|
Insufficient cpu / memory | No node has that much free — requests, not usage | Lower requests, or add capacity |
had untolerated taint | Node repels it | Add the toleration, and a selector to attract it |
didn't match Pod's node affinity/selector | No node carries the label | kubectl get nodes --show-labels |
didn't match pod anti-affinity rules | Its own replicas are in the way | Relax to preferred, or use topology spread |
node(s) had volume node affinity conflict | The PV is in another zone | WaitForFirstConsumer on the StorageClass |
node(s) were unschedulable | Cordoned | kubectl uncordon |
exceeded quota | ResourceQuota refused it before scheduling | kubectl describe quota -n <ns> |
A node showing 30% CPU in top can be 100% allocated and refuse new pods,
because the scheduler adds up requests, not actual consumption. That gap is the most
common "we have plenty of capacity, why is it Pending" confusion, and
kubectl describe node shows both numbers side by side.
kubectl describe pod ends with a line of the form 0/12 nodes are available: 8 Insufficient cpu, 4 node(s) had untolerated taint. That accounts for every node in the cluster and why each one was excluded. There is rarely anything to deduce beyond reading it.
| Command | What it answers |
|---|---|
kubectl describe pod <p> | Which predicate failed, on how many nodes |
kubectl get nodes --show-labels | What affinity rules can actually match |
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints | Every taint in one view |
kubectl describe node <n> | grep -A8 Allocated | Requested versus capacity — what scheduling uses |
kubectl top node | Actual usage, for contrast |
kubectl taint node <n> key=value:NoSchedule | Repel new pods |
kubectl taint node <n> key- | Remove a taint (trailing dash) |
kubectl cordon / uncordon | Stop or resume scheduling |
kubectl get pods -o wide --sort-by=.spec.nodeName | Where everything landed |
kubectl get pods --field-selector status.phase=Pending -A | Everything stuck, cluster-wide |
kubectl get priorityclass | What can preempt what |