K8s Error Runbook

Runbook

ERROR REFERENCE

Click any layer to explore errors, causes, diagnostic commands, and step-by-step fixes.

💥

CrashLoopBackOff

Container starts and exits repeatedly. Kubernetes applies exponential backoff (10s → 20s → 40s … max 5min).

CriticalPod LifecycleRuntime

Critical▾

Diagnostic Commands

kubectl

❯kubectl describe pod <pod> -n <ns>

❯kubectl logs <pod> --previous -n <ns>

❯kubectl logs <pod> --tail=100 -n <ns>

❯kubectl get events -n <ns> --sort-by=.lastTimestamp

Root Causes

🔴Application crash: Exit code 1 — check logs for stack trace

🔴OOMKilled (exit 137): Memory limit too low — increase resources.limits.memory

🟠Missing ConfigMap/Secret: Key not found — verify env / volumeMount references

🟠Wrong entrypoint: Image CMD vs pod spec command mismatch

🔵Liveness probe: Probe failing before app is ready — increase initialDelaySeconds

🔵Dependency unreachable: DB / service not ready — use initContainers with wait-for

Fix Steps

Check kubectl logs <pod> --previous — find the last error before crash

Check exit code in kubectl describe pod → 137 = OOM, 1 = app error, 127 = binary not found

If OOM: increase resources.limits.memory and add resources.requests.memory

If config error: verify all env.valueFrom.secretKeyRef and configMapKeyRef keys exist

Add startupProbe with high failureThreshold if app takes long to initialize

🚫

ImagePullBackOff / ErrImagePull

Kubernetes cannot pull the container image from the registry. Pod stays in Waiting state.

CriticalImage Registry

Critical▾

Diagnostic Commands

kubectl

❯kubectl describe pod <pod> | grep -A 10 Events

❯kubectl get events -n <ns> | grep -i image

❯kubectl get pod <pod> -o jsonpath='{.spec.containers[*].image}'

Root Causes

🔴Image not found: manifest unknown — verify image name, tag, registry path

🔴Auth failure: unauthorized: access denied — missing or wrong imagePullSecret

🟠Registry unreachable: Network policy or firewall blocking node → registry

🔵Rate limiting: Docker Hub toomanyrequests — use authenticated pull or mirror

Fix — Create Image Pull Secret

kubectl

❯kubectl create secret docker-registry regcred \

--docker-server=<registry> --docker-username=<user> \

--docker-password=<token> -n <namespace>

❯kubectl patch sa default -n <ns> \

-p '{"imagePullSecrets":[{"name":"regcred"}]}'

💾

OOMKilled (Exit Code 137)

Container exceeded memory limit. Kernel OOM killer terminates the process immediately.

CriticalMemoryResources

Critical▾

Diagnostic Commands

kubectl

❯kubectl describe pod <pod> | grep -E 'OOMKilled|Limits|Requests'

❯kubectl top pod <pod> -n <ns> --containers

❯kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState}'

Fix Steps

Increase resources.limits.memory in pod spec (e.g. 256Mi → 512Mi)

Set requests = limits for Guaranteed QoS class — prevents preemption

Profile app memory with JVM heap flags or heap dump on OOM

Deploy VPA (Vertical Pod Autoscaler) in recommendation mode to right-size

⏳

Pod Stuck in Pending

Scheduler cannot place pod on any node. Pod stays in Pending state indefinitely.

HighSchedulerResources

High▾

Diagnostic Commands

kubectl

❯kubectl describe pod <pod> | grep -A 20 Events

❯kubectl get nodes -o wide

❯kubectl describe nodes | grep -A 5 'Allocated resources'

❯kubectl get pvc -n <ns> # if volume mount

Root Causes

🔴Insufficient CPU/Memory: 0/N nodes insufficient cpu — scale cluster or reduce requests

🟠nodeSelector mismatch: Label on pod doesn't match any node — fix labels

🟠Taint not tolerated: Node has taint, pod missing toleration

🔵PVC unbound: StorageClass provisioner not running or no PV available

🔵Pod affinity unmet: Relax affinity rules or adjust topologySpreadConstraints

🔧

Init Container Failure

InitContainers must complete before main container starts. A failing init blocks pod indefinitely.

HighInit

High▾

Diagnostic Commands

kubectl

❯kubectl logs <pod> -c <init-container-name> -n <ns>

❯kubectl describe pod <pod> | grep -A 5 'Init Containers'

Fix Steps

Read init container logs for the specific failure reason

For DB wait: use until nc -z db 5432; do sleep 1; done as init command

Verify all secrets and configmaps referenced by init container exist

Confirm init container image name and tag are correct

🗑️

Pod Eviction

kubelet evicts pods when node drops below eviction thresholds for memory, disk, or PIDs.

HighNode Pressure

High▾

Diagnostic Commands

kubectl

❯kubectl get events -n <ns> | grep Evicted

❯kubectl describe node <node> | grep -E 'Eviction|pressure'

❯kubectl get pod <pod> -o jsonpath='{.status.reason}'

Eviction Thresholds

💾memory.available < 100Mi: Increase node memory; reduce pod memory requests

💿nodefs.available < 10%: Clean up: crictl rmi --prune, log rotation

🖼️imagefs.available < 15%: Remove unused images, add larger container disk

⚙️pid.available < 1000: Check for fork bombs; increase pid limit

🔴

Node NotReady

Node fails to report healthy status within node-monitor-grace-period (default 40s). All pods on node affected.

CriticalNode Healthkubelet

Critical▾

Diagnostic Commands

kubectl + node

❯kubectl get nodes

❯kubectl describe node <node-name>

❯systemctl status kubelet

❯journalctl -u kubelet -f --since '10 min ago'

❯kubectl get events --field-selector involvedObject.name=<node>

Node Conditions

🔴MemoryPressure=True: dmesg | grep -i oom; reduce pod density

🔴DiskPressure=True: df -h; clean /var/lib/docker or /var/log

🟠PIDPressure=True: ps aux | wc -l; check for runaway processes

🟠NetworkUnavailable=True: CNI plugin not configured; restart CNI daemonset

🔴kubelet not running: All conditions Unknown; systemctl restart kubelet

⚙️

kubelet Failure

kubelet is the node agent managing pod lifecycle. Failures prevent pod creation and health reporting.

Criticalkubelet

Critical▾

Diagnostic Commands

systemd

❯systemctl status kubelet

❯journalctl -u kubelet --since '30 min ago' | tail -200

❯openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

❯cat /var/lib/kubelet/config.yaml

Common Errors

🔴Certificate expired: kubeadm certs renew all

🔴PLEG not healthy: Container runtime down — restart containerd

🟠API server unreachable: Check network/firewall to control plane

🔵Config file error: Validate /var/lib/kubelet/config.yaml syntax

🔵Disk full: Failed to create pod sandbox — check df -i (inodes)

📦

Container Runtime Failure

containerd or CRI-O failure prevents pod creation and container management across the node.

CriticalcontainerdCRI

Critical▾

Diagnostic Commands

crictl

❯systemctl status containerd

❯journalctl -u containerd -n 200

❯crictl ps && crictl pods

❯crictl rm $(crictl ps -a -q --state exited) # cleanup

❯crictl rmi --prune # remove unused images

Fix Steps

Restart runtime: systemctl restart containerd

Check socket: ls -la /var/run/containerd/containerd.sock

Review config: /etc/containerd/config.toml

Free disk: prune stopped containers and unused images

💿

Node Disk Pressure

Node filesystem usage above eviction threshold. Pods are evicted and new pods cannot be scheduled.

HighStorage

High▾

Cleanup Commands

node shell

❯df -h && df -i # check disk + inode usage

❯du -sh /var/lib/containerd/* | sort -rh | head -20

❯du -sh /var/log/* | sort -rh | head -10

❯crictl rmi --prune # remove unused images

❯find /var/log -name '*.log' -size +100M -exec truncate -s 0 {} \;

🔧

Node Maintenance (Cordon / Drain)

Safe procedure to remove a node from service for maintenance without disrupting workloads.

MediumMaintenance

Procedure▾

Maintenance Workflow

kubectl

❯kubectl cordon <node-name> # step 1: prevent new scheduling

❯kubectl drain <node-name> --ignore-daemonsets \

--delete-emptydir-data --grace-period=60 # step 2: evict

# --- perform maintenance ---

❯kubectl uncordon <node-name> # step 3: return to service

🌐

CNI / Networking Failure

CNI plugin failure causes pod-to-pod or pod-to-service communication failures cluster-wide.

CriticalCNICoreDNS

Critical▾

Diagnostic Commands

kubectl

❯kubectl get pods -n kube-system | grep -E 'calico|flannel|cilium|weave'

❯kubectl run nettest --image=busybox --rm -it -- wget -qO- http://<svc>

❯kubectl run dnstest --image=busybox --rm -it -- nslookup kubernetes.default

❯kubectl logs -n kube-system <cni-pod> --tail=100

Fix by Symptom

DNS broken: kubectl rollout restart deploy/coredns -n kube-system

CNI down: Restart CNI daemonset pods in kube-system

kube-proxy: kubectl rollout restart ds/kube-proxy -n kube-system

NetworkPolicy blocking: Audit policies with kubectl get netpol -A

💾

PersistentVolume / Storage Errors

Storage errors prevent pods from mounting volumes. PVC stuck in Pending, Lost, or Terminating state.

HighPVCCSI

High▾

Diagnostic Commands

kubectl

❯kubectl get pv,pvc -A

❯kubectl describe pvc <name> -n <ns>

❯kubectl get events -A | grep -i 'provision\|mount\|volume'

❯kubectl get storageclass

PVC State Reference

🔴Pending: No matching PV or CSI provisioner not running

🔴Lost: Backing PV deleted — recreate PV with same claimRef

🟠Terminating stuck: kubectl patch pvc <name> -p '{"metadata":{"finalizers":null}}'

🔵RWX unsupported: Use NFS, EFS, or CephFS StorageClass

🔐

RBAC 403 Forbidden

Role-Based Access Control misconfigurations cause 403 errors on API calls from pods or users.

HighRBACSecurity

High▾

Diagnostic Commands

kubectl auth

❯kubectl auth can-i list pods --as=system:serviceaccount:<ns>:<sa>

❯kubectl auth can-i create deployments --as=<user> -n <ns>

❯kubectl get roles,rolebindings,clusterroles,clusterrolebindings -n <ns>

❯kubectl create rolebinding <name> --clusterrole=view \

--serviceaccount=<ns>:<sa> -n <ns>

Fix Steps

Use kubectl auth can-i to identify exactly what permission is missing

Create Role or ClusterRole with the required verbs and resources

Bind it to the ServiceAccount via RoleBinding

Verify with kubectl auth can-i using --as flag again

📊

Resource Quota Exceeded

Namespace resource quotas block pod creation when CPU, memory, or object count limits are reached.

MediumQuota

Medium▾

Diagnostic Commands

kubectl

❯kubectl describe resourcequota -n <ns>

❯kubectl describe limitrange -n <ns>

❯kubectl get events -n <ns> | grep -i quota

Error Messages

🟠exceeded quota: requests.cpu — Increase quota or reduce pod CPU requests

🟠must specify limits.memory — LimitRange requires explicit limits in pod spec

🔵exceeded quota: count/pods — Increase pod quota; delete unused pods

📈

HPA Not Scaling / Unknown Metrics

Horizontal Pod Autoscaler unable to read metrics or stuck at min/max replicas.

MediumHPAAutoscaling

Medium▾

Diagnostic Commands

kubectl

❯kubectl get hpa -n <ns>

❯kubectl describe hpa <name> -n <ns>

❯kubectl top pods -n <ns>

❯kubectl get pods -n kube-system | grep metrics-server

HPA Conditions

🔴AbleToScale: False: metrics-server not installed — deploy metrics-server

🟠Unknown metrics: Custom metrics API unavailable — install Prometheus adapter

🔵ScaleDown blocked: PodDisruptionBudget preventing scale-down — review PDB

🚨

API Server Unreachable

kube-apiserver is down or not accepting connections on port 6443. All cluster operations fail.

CriticalAPI Server

Critical▾

Diagnostic Commands

kubectl + node

❯kubectl cluster-info

❯kubectl get --raw='/healthz'

❯kubectl get --raw='/readyz'

❯crictl logs $(crictl ps -q --name=kube-apiserver)

❯cat /etc/kubernetes/manifests/kube-apiserver.yaml

Root Causes

🔴Pod crashed: Check static pod manifest in /etc/kubernetes/manifests/

🔴etcd unreachable: etcd cluster unhealthy — check etcd health

🔴Certificate expired: kubeadm certs renew all; restart API server

🟠OOMKilled: Increase API server memory limits in static pod manifest

🔵Audit log disk full: Rotate audit logs; tune audit policy

🗄️

etcd Failure / Quorum Lost

etcd stores all cluster state. Losing quorum (majority of members) makes the cluster read-only or unavailable.

CriticaletcdQuorum

Critical▾

Diagnostic Commands

etcdctl

❯ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \

--cacert=/etc/kubernetes/pki/etcd/ca.crt \

--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \

--key=/etc/kubernetes/pki/etcd/healthcheck-client.key \

endpoint health --cluster

❯etcdctl endpoint status --cluster -w table

❯etcdctl defrag --cluster # maintenance: defragment

etcd Error Reference

🔴No leader elected: Check network between etcd nodes; restart etcd

🔴DB size limit: mvcc: database space exceeded — defragment + compact history

🟠Slow disk I/O: High latency, leader elections — use SSD for etcd

🟠Peer unreachable: Fix network; check TLS peer certs

🪝

Webhook / Admission Controller Failure

Mutating or Validating webhooks blocking pod creation when unavailable or misconfigured.

HighWebhooksAdmission

High▾

Diagnostic Commands

kubectl

❯kubectl get validatingwebhookconfigurations

❯kubectl get mutatingwebhookconfigurations

❯kubectl get events -A | grep -i webhook

❯kubectl describe validatingwebhookconfiguration <name>

Fix Steps

Verify webhook service and pod are running and healthy

Check webhook TLS certificate validity and CA bundle

Set failurePolicy: Ignore temporarily to restore operations

Review timeout — default 10s. Increase if webhook is slow

⏱️

API Rate Limiting / Throttling

HTTP 429 errors or slow kubectl responses due to too many concurrent requests to the API server.

MediumRate Limit

Medium▾

Diagnostic Commands

kubectl

❯kubectl get events -A | grep -i throttl

# Watch for: "Throttling request took X seconds"

# HTTP 429 Too Many Requests in client logs

❯kubectl get flowschemas,prioritylevelconfigurations # APF config

🔑

Certificate Expired

All Kubernetes components use mTLS. Expired certificates cause widespread authentication failures.

CriticalTLSCerts

Critical▾

Certificate Management

kubeadm

❯kubeadm certs check-expiration # check all cert expiry dates

❯kubeadm certs renew all # renew all certificates

❯kill -s SIGHUP $(pidof kube-apiserver) # reload without restart

❯kill -s SIGHUP $(pidof kube-controller-manager)

❯kill -s SIGHUP $(pidof kube-scheduler)

🛡️

SCC Violation (Security Context Constraints)

Pod rejected because its security context violates the allowed SCC. Most common OpenShift error.

CriticalSCCSecurity

Critical▾

Diagnostic Commands

❯oc get pod <pod> -o yaml | grep scc

❯oc get scc

❯oc get events -n <ns> | grep -i scc

❯oc adm policy who-can use scc restricted

❯oc adm policy add-scc-to-user anyuid -z <sa> -n <ns>

SCC Error Reference

🔴unable to validate against any SCC: Grant appropriate SCC to ServiceAccount

🔴runAsUser not allowed: Container runs as root — grant anyuid SCC

🟠hostPath not allowed: Grant hostmount-anyuid SCC or use PVC

🟠hostNetwork not allowed: Grant hostnetwork SCC to ServiceAccount

📊

Cluster Operator Degraded

OpenShift cluster operators manage platform components. Degraded state blocks upgrades and indicates failures.

CriticalCluster Operator

Critical▾

Diagnostic Commands

❯oc get clusteroperators # check all operators

❯oc describe co <operator-name>

❯oc logs -n openshift-<operator> deploy/<operator>

❯oc adm inspect co/<operator> # full diagnostic bundle

Operator Reference

🔴kube-apiserver: etcd issues, cert expiry, master node failure

🔴authentication: OAuth provider unreachable, CA cert mismatch

🟠image-registry: Storage backend unavailable, PVC unbound

🟠monitoring: Prometheus OOM, PVC full, scrape errors

🔵machine-config: Node stuck in config, MCO loop

🔀

Route / Ingress Failure

OpenShift Routes (HAProxy) not routing traffic. 503 errors or TLS handshake failures.

HighRouteHAProxy

High▾

Diagnostic Commands

❯oc get route -n <ns>

❯oc describe route <name> -n <ns>

❯oc get pods -n openshift-ingress

❯oc describe ingresscontroller default -n openshift-ingress-operator

❯oc get endpoints <svc> -n <ns> # check backend pods

Route Error Reference

🔴503: Backend pods not ready or wrong service port — check endpoints

🟠TLS failure: Cert/key mismatch or expired — update route TLS secret

🟠Route not admitted: Hostname conflict — use unique hostname

🔵Router crashloop: HAProxy config error — check router pod logs

🏗️

Build / ImageStream Failure

OpenShift S2I builds or ImageStream imports failing. CI/CD pipeline cannot produce images.

HighBuildImageStream

High▾

Diagnostic Commands

❯oc get builds -n <ns>

❯oc logs build/<build-name> -n <ns>

❯oc get imagestream -n <ns>

❯oc describe co image-registry

Build Errors

🔴Failed to pull base image: Add pull secret to builder SA

🟠Push to registry failed: Check image-registry operator status and PVC

🟠S2I assemble failed: Application compile error — check build logs

🔵Build pod evicted: Node under resource pressure — free resources

⚙️

Machine Config Pool (MCP) Stuck

MCO manages node OS config. Node stuck in Updating or Degraded state during config rollout.

HighMCONode OS

High▾

Diagnostic Commands

❯oc get mcp

❯oc describe mcp worker

❯oc logs -n openshift-machine-config-operator deploy/machine-config-operator --tail=100

❯oc debug node/<node> # open shell on stuck node

MCP States

🔴Degraded: True: Node failed to apply config — check MCD logs

🟠Updating: stuck: Node unreachable — check node health, drain if needed

🔵Config render failed: Invalid MachineConfig YAML — oc get mc to find bad config

🔑

OAuth / Authentication Failure

OpenShift OAuth server handles all user authentication. Failures prevent any user from logging in.

CriticalOAuthAuth

Critical▾

Diagnostic Commands

❯oc get pods -n openshift-authentication

❯oc logs -n openshift-authentication deploy/oauth-openshift --tail=100

❯oc get oauth cluster -o yaml

❯oc describe co authentication

Auth Errors

🔴502 Bad Gateway: OAuth pod not running — restart oauth-openshift deployment

🔴LDAP: connection refused: Check network; update OAuth CA cert bundle

🟠Invalid client secret: Recreate oauth client secret in openshift-config

🔵Certificate not trusted: Add CA to additionalTrustBundle in OAuth config

Code	Signal	Meaning	Kubernetes Cause
0	—	Graceful exit	Process completed normally. Check if it's expected.
1	—	General error	Application error — check logs for stack trace.
126	—	Permission denied	Command not executable — check file permissions in image.
127	—	Command not found	Missing binary in container image. Check CMD/ENTRYPOINT.
134	SIGABRT	Abnormal termination	Assert failure, heap corruption — check application.
137	SIGKILL	OOMKilled / forced kill	Memory limit exceeded → increase resources.limits.memory
139	SIGSEGV	Segmentation fault	Null pointer or buffer overflow in application.
143	SIGTERM	Graceful termination	Kubernetes graceful shutdown signal. Normal during drain.

K8s ERROR
RUNBOOK

5-LAYER ERROR MODEL

ERROR REFERENCE

EXIT CODE REFERENCE

COMMAND CHEATSHEET

5-LAYER ERROR MODEL

ERROR REFERENCE

EXIT CODE REFERENCE

COMMAND CHEATSHEET

K8s ERROR
RUNBOOK