K8s Storage Deep-Dive

Deep-Dive

STORAGE REFERENCE

Click any layer to explore concepts, YAML, and production-tested guidance.

📦

Static vs Dynamic Provisioning

A PV is a cluster resource representing actual storage. A PVC is a namespaced request for that storage. How the PV gets created is the whole story.

Must KnowProvisioning

Must Know▾

Static: admin pre-creates the PV

pv-static.yaml

apiVersion: v1

kind: PersistentVolume

metadata: {name: pv-billing-logs}

spec:

capacity: {storage: 50Gi}

accessModes: [ReadWriteOnce]

persistentVolumeReclaimPolicy: Retain

nfs: {server: 10.20.4.5, path: /export/billing}

Static Provisioning

🔵Admin creates the PV up front (NFS export, pre-carved LUN, existing disk)

🔵A matching PVC binds to it — no provisioner call involved

🟢Good for: NFS shares, pre-existing SAN LUNs, migration scenarios

Dynamic Provisioning

PVC references a storageClassName instead of an existing PV

The CSI external-provisioner sidecar watches for unbound PVCs

It calls the backend API (EBS, Ceph, etc.) and creates the PV automatically

This is the default in almost every production cluster today

🔐

Access Modes

Access modes describe how many nodes can mount the volume simultaneously — not a permissions system. Getting this wrong causes multi-attach errors.

ImportantBinding

Important▾

The Four Access Modes

Mode	Meaning	Typical Backend
RWO	ReadWriteOnce — one node, read-write	EBS, Azure Disk, GCE PD
ROX	ReadOnlyMany — many nodes, read-only	NFS, CephFS
RWX	ReadWriteMany — many nodes, read-write	NFS, CephFS, EFS, Azure Files
RWOP	ReadWriteOncePod — single pod (K8s 1.22+), stricter than RWO	CSI drivers supporting block-mode RWOP

Common Mistake

Requesting RWX on EBS/Azure Disk — these block-storage backends only support RWO. Use EFS/Azure Files/CephFS instead.

Scaling a Deployment with an RWO PVC beyond 1 replica → new pod stuck Pending with a multi-attach error since two nodes can't mount the same RWO volume.

♻️

Reclaim Policy & PVC Lifecycle

What happens to the underlying disk once the PVC is deleted — the single most consequential setting for data safety.

Data Safety

Recommended▾

Reclaim Policies

🔴Delete (default for dynamic): backend volume is destroyed when the PVC is deleted — irreversible

🔵Retain: PV becomes Released, data stays, needs manual admin cleanup/reclaim before reuse

🟢Recycle (deprecated): basic rm -rf scrub — don't use it

PVC Binding Lifecycle

PVC created → Pending

Matching PV found/provisioned → Bound

PVC deleted → PV becomes Released (Retain) or is destroyed (Delete)

Set persistentVolumeReclaimPolicy: Retain on anything holding data you can't regenerate

⚙️

Defining a StorageClass

A StorageClass names a provisioner and passes it backend-specific parameters. PVCs reference the class by name — that's the entire contract.

Must Know

Must Know▾

storageclass-fast-ssd.yaml

yaml

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata: {name: fast-ssd}

provisioner: ebs.csi.aws.com

parameters: {type: gp3, iops: "3000", throughput: "125"}

reclaimPolicy: Delete

allowVolumeExpansion: true

volumeBindingMode: WaitForFirstConsumer

Commands

kubectl

❯kubectl get storageclass

❯kubectl describe storageclass fast-ssd

❯kubectl get sc -o jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}'

⏱️

Volume Binding Mode

Controls when the volume is actually provisioned — immediately at PVC creation, or once a pod is scheduled and its node is known.

ImportantScheduling

Important▾

Immediate

🟠Volume is provisioned as soon as the PVC is created, before any pod is scheduled

🟠Risk: the volume can land in an AZ where no eligible node exists → pod stuck Pending

WaitForFirstConsumer

Scheduler picks a node for the pod first

Provisioner then creates the volume in that node's zone

Eliminates zone-mismatch scheduling failures — the default recommendation for cloud CSI drivers

📐

Volume Expansion & Multi-Class Strategy

Growing a volume without downtime, and running more than one StorageClass for different performance tiers.

Operations

Recommended▾

Expanding a PVC

StorageClass must have allowVolumeExpansion: true

Edit the PVC's spec.resources.requests.storage to a larger value

Most CSI drivers expand online; some filesystems need a pod restart to see the new size

Multi-Class Pattern

🔵fast-ssd — databases, indexing workloads

🟢standard — general application data, default class

🟢archive-hdd — logs, cold backups, cost-optimized

🧩

Controller Plugin vs Node Plugin

Every CSI driver ships as two components running very different jobs, each wrapped with sidecar containers that talk to the Kubernetes API.

Must Know

Must Know▾

Controller Plugin (Deployment, 1-3 replicas)

🔴Runs cluster-wide, calls the storage backend's API

🔴Sidecars: external-provisioner (create/delete), external-attacher (attach/detach), external-resizer (expand), external-snapshotter

Node Plugin (DaemonSet, every node)

Mounts/unmounts the already-attached volume into the pod's filesystem

Sidecar: node-driver-registrar registers the driver with kubelet

Talks to kubelet over a Unix socket at /var/lib/kubelet/plugins/<driver>/csi.sock

🏭

Choosing a CSI Driver

The backend decides your access modes, snapshot support, and expansion story. Pick deliberately, not by default.

Important

Important▾

Driver	Best For	RWX Support
aws-ebs-csi-driver	Single-node block storage on EKS	No (block-mode only)
aws-efs-csi-driver	Shared file storage, config volumes	Yes
ceph-csi (RBD / CephFS)	On-prem / OpenShift, mixed workloads	CephFS: yes · RBD: no
portworx	High-IO databases, replicated on-prem storage	Yes
nfs-subdir-external-provisioner	Simple shared storage on existing NFS	Yes

🩺

Volume Lifecycle Troubleshooting

Where "PVC stuck Pending" and "multi-attach error" actually come from, and which component to check first.

Troubleshooting

Recommended▾

Diagnostic Commands

kubectl

❯kubectl describe pvc <name> -n <ns>

❯kubectl get events -A | grep -i provision

❯kubectl get volumeattachments

❯kubectl -n kube-system logs -l app=ebs-csi-controller -c csi-provisioner

Common Root Causes

🔴Multi-attach error: RWO volume still attached to old node — pod eviction race, wait for detach or force-delete the stuck pod

🟠Stuck Pending: zone mismatch with Immediate binding, or IAM/role permissions missing on the CSI controller

🔵Provisioning quota exceeded: cloud account volume-count limit hit — check provisioner logs for throttling errors

🆔

StatefulSet vs Deployment

A Deployment's pods are interchangeable. A StatefulSet's pods each have a fixed name, fixed network identity, and their own dedicated PVC.

Must Know

Must Know▾

statefulset-postgres.yaml

yaml

apiVersion: apps/v1

kind: StatefulSet

spec:

serviceName: postgres-headless

replicas: 3

volumeClaimTemplates:

- metadata: {name: data}

spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 100Gi}}}

Pods Get Fixed Names

🔵postgres-0, postgres-1, postgres-2 — never renamed, never mixed up

🔵Each gets its own PVC: data-postgres-0, data-postgres-1...

Ordered Lifecycle

Scale up: pods created 0 → N, each waits for the previous to be Ready

Scale down: pods terminated N → 0, reverse order

Matters for quorum-based systems: primary election, replica bootstrap order

🌐

Headless Services & Stable DNS

A StatefulSet needs a headless Service (clusterIP: None) so each pod gets its own resolvable DNS name — required for peer discovery.

ImportantNetworking

Important▾

DNS pattern

resolvable names

# <pod-name>.<service-name>.<namespace>.svc.cluster.local

postgres-0.postgres-headless.billing.svc.cluster.local

postgres-1.postgres-headless.billing.svc.cluster.local

postgres-2.postgres-headless.billing.svc.cluster.local

Why It Matters

🟠Replicas can address each other by a name that survives pod restarts — clustered databases (etcd, Cassandra, Kafka) depend on this for peer discovery

🔄

Update Strategy & PVC Retention

RollingUpdate with partitions for canary-style rollouts, and what happens to volumes when you scale down or delete.

Operations

Recommended▾

Update Strategies

🔵RollingUpdate (default): highest ordinal updated first, one at a time

🔵partition: N: only pods with ordinal ≥ N are updated — canary a single replica before the rest

🟢OnDelete: pods only update when manually deleted — full manual control

PVC Retention (K8s 1.27+)

By default, PVCs from volumeClaimTemplates outlive the StatefulSet — deleting the STS does not delete the data

Set persistentVolumeClaimRetentionPolicy.whenDeleted/whenScaled: Delete to change that behavior explicitly

📸

The VolumeSnapshot API

A native Kubernetes API — backed by the CSI snapshotter sidecar — for point-in-time, storage-level snapshots you can restore straight into a new PVC.

Must Know

Must Know▾

snapshot-and-restore.yaml

yaml

# 1. Take the snapshot

apiVersion: snapshot.storage.k8s.io/v1

kind: VolumeSnapshot

spec: {volumeSnapshotClassName: csi-snap, source: {persistentVolumeClaimName: data-postgres-0}}

# 2. Restore into a brand-new PVC

apiVersion: v1

kind: PersistentVolumeClaim

spec: {dataSource: {name: pg-snap-0930, kind: VolumeSnapshot, apiGroup: snapshot.storage.k8s.io}}

🗄️

Cluster Backup Tools

Snapshots cover the disk. A real backup strategy also captures Kubernetes object state — the two are not the same thing.

Important

Important▾

Tool	Backs Up	Notes
Velero	K8s objects + PV data (via CSI snapshots or restic/kopia)	Most widely adopted, CNCF project
Kasten K10	K8s objects + PV data + app-consistent hooks	Commercial, strong for stateful DBs
Portworx PX-Backup	Portworx-managed volumes + K8s metadata	Tightly coupled to Portworx storage

Velero Quick Commands

velero

❯velero backup create billing-ns-backup --include-namespaces billing

❯velero schedule create nightly --schedule="0 2 * * *" --include-namespaces billing

❯velero restore create --from-backup billing-ns-backup

🎯

RPO / RTO & DR Strategy

Backups without a defined recovery objective are just disk space being consumed. Define the numbers before the incident, not during it.

Planning

Recommended▾

Define These First

🔵RPO (Recovery Point Objective): how much data loss is acceptable — sets your snapshot/backup frequency

🔵RTO (Recovery Time Objective): how fast you must be back up — sets whether you need warm standby or cold restore is fine

Cross-Region Pattern

Replicate snapshots to a second region/bucket on a schedule

Test restore quarterly — an untested backup is a guess, not a plan

Document the exact restore runbook — who runs it, in what order, with what access

Workload	Access Mode	Backend Fit	Notes
Single-replica database	RWO	EBS / Azure Disk / GCE PD	Cheapest, fastest, standard choice
StatefulSet database cluster	RWO per replica	EBS + volumeClaimTemplates	Each replica gets its own dedicated disk
Shared config / static assets	RWX	NFS / EFS / Azure Files	Multiple pods reading the same files
Log aggregation buffer	RWX or RWO	NFS or local emptyDir + sidecar ship	Depends on whether multiple writers exist
CI/CD build cache	RWO	Fast SSD class	Ephemeral-ish, prioritize IOPS over durability

K8s STORAGE
DEEP-DIVE

THE STORAGE STACK

STORAGE REFERENCE

WHICH ACCESS MODE / CLASS DO I NEED?

COMMAND CHEATSHEET

THE STORAGE STACK

STORAGE REFERENCE

WHICH ACCESS MODE / CLASS DO I NEED?

COMMAND CHEATSHEET

K8s STORAGE
DEEP-DIVE