Pod Security rolled out with warn before enforce, the removed API that takes workloads with it on upgrade, the difference between an etcd snapshot and a real backup, and the three cluster failures that are silent until they are outages.
Harden, upgrade, drain, back up, spread — and prove the restore works before you need it to.
PodDisruptionBudget appears in the drain row for a reason: the same object that protects availability is what stops an upgrade and what pins a node against scale-down.
The diagram above is the high level: what the pieces are. These two are the ones you want when something is wrong — what is inside one of those boxes, and the path a request really takes through them.
The four operational concerns that are nobody's feature work until the day they are the incident.
Cluster hardening is a long subject with a short high-value core. Four controls, each one label or object, each closing a category of problem:
privileged, baseline, restricted — applied as a namespace
label. restricted blocks running as root, privilege escalation, host namespaces and
most capabilities.create pods and get secrets are both effectively privileged.PSA can warn and audit without blocking. Labelling a namespace
warn=restricted first tells you exactly which workloads would break, from real
traffic, before anything is refused. Going straight to enforce on a live namespace
is how you find out during an incident.
Two rules govern every Kubernetes upgrade:
The thing that actually breaks workloads is API removal. A deprecated API is removed on a schedule, and a manifest or controller still calling it starts failing the moment the control plane moves. The cluster knows who is calling what, which turns this from an audit into a query.
Check for removed APIs, take an etcd snapshot, confirm PDBs will not deadlock the drain, upgrade the control plane, then the nodes one pool at a time.
| etcd snapshot | Velero | |
|---|---|---|
| Contains | Every API object | Selected namespaces + PV contents |
| Restores | The whole cluster to that instant | Namespaces, into this or another cluster |
| Granularity | All or nothing | Per namespace, per label |
| PV data | No | Yes — snapshots or file-level copy |
| Right for | Lost quorum, corrupted control plane | Deleted namespace, migration, real DR |
Conflating the two is how a DR test fails. "We back up etcd nightly" does not protect against someone deleting the prod namespace with Delete-policy PVCs — the objects come back from a cluster-wide restore, the volume contents do not, because they were never in etcd.
An etcd restore stops the control plane, restores on one member and rebuilds the others. It is documented, disruptive, and takes as long as it takes. The only way to know that number is to do it once on a cluster you can afford to break — and the first rehearsal always takes longer than anyone predicted.
On a Delete-policy StorageClass, removing a PVC destroys the backing volume immediately — including when the PVC goes as a side effect of deleting a namespace or a Helm release. Retain turns irreversible loss into orphaned volumes you clean up deliberately, which is a far better problem.
Past a certain size the question stops being "how big can this cluster get" and becomes "how many clusters, split how". The honest reasons to run more than one:
And the costs, which are consistently underestimated: every platform component installed N times, N sets of credentials and policy to keep consistent, cross-cluster service discovery to solve, and a fleet-management story (Argo CD ApplicationSets, Cluster API, Fleet) that is now itself production infrastructure.
Not "do we have a second cluster" but "what is our RPO and RTO, and have we measured them". Active-passive with Velero restore is hours. Active-active with replicated data is minutes, and a much larger standing bill. Both are defensible; only one is usually what people have while believing they have the other.
Cluster-level failures differ from workload failures in one way that matters: they are rarely loud. A certificate that expires in 30 days, a backup that has silently failed for a month, an etcd database approaching its quota — each is invisible until it is an outage.
| Symptom | Cause | Check |
|---|---|---|
| API server suddenly refuses everything | Certificate expired | kubeadm certs check-expiration |
| Writes fail, reads work | etcd lost quorum, or hit its DB quota | etcdctl endpoint status -w table |
| Everything slow, nothing down | etcd fsync latency / leader elections | etcd_disk_wal_fsync_duration_seconds p99 > 10 ms |
| Upgrade drain hangs on one node | A PDB that allows zero disruptions | kubectl get pdb -A |
| Workloads vanish after upgrade | A removed API their manifests used | kubectl get apirequestcount — before, not after |
| Restore produces empty volumes | etcd snapshot, not an application backup | Velero with --snapshot-volumes |
| Node NotReady, kubelet fine | CNI or the container runtime | journalctl -u kubelet -u containerd on the node |
Each of these three fails by not happening: the backup does not run, the certificate does not renew, the compaction does not occur. None produces an error to alert on. The alert has to be on the age of the last success — which is a different kind of rule, and the one most clusters are missing.
| Command | What it answers |
|---|---|
kubeadm certs check-expiration | The silent outage 30 days out |
etcdctl endpoint status -w table | Quorum, leader and DB size |
kubectl get apirequestcount | Who still calls an API about to be removed |
kubectl get pdb -A | What will deadlock the next drain |
kubectl drain <n> --ignore-daemonsets --delete-emptydir-data | Evacuate a node properly |
kubectl get ns -o custom-columns=...enforce | Which namespaces enforce Pod Security |
kubectl get networkpolicy -A | Which namespaces are actually isolated |
velero backup get | Whether the backup ran, and when it last succeeded |
velero restore create --from-backup <b> | Bring a namespace back |
kubectl version -o json | jq .serverVersion | Where this cluster sits in the skew |
kubectl get nodes -o wide | kubelet and runtime versions per node |
kubectl diff -f manifests/ | Drift between git and the cluster |