What are StatefulSets, PersistentVolumes/Claims, and how do you run stateful workloads on Kubernetes?

Question

IT Interview Practice · Accepted Answer

Stateful workloads (databases, queues, clustered stores) need **stable identity** and **durable storage** — things a Deployment does not provide. A **StatefulSet** gives Pods stable names and ordered lifecycle; **PersistentVolumes (PV)** and **PersistentVolumeClaims (PVC)** give them storage that outlives the Pod.

## Why Deployments aren't enough

```text
Deployment Pods are INTERCHANGEABLE: random names (web-7f9-abc), any order, share
nothing per-Pod. That's perfect for stateless apps — but a database replica needs:
  → a STABLE network identity (so peers can find it: db-0, db-1, db-2)
  → its OWN durable disk that survives restarts/rescheduling
  → ORDERED, controlled startup/scaling (bring up db-0 before db-1)
StatefulSet provides exactly these.
```

## StatefulSet guarantees

```text
Stable NAMES     → pod-0, pod-1, pod-2 (ordinal, not random)
Stable DNS       → db-0.db.default.svc... via a headless Service
Stable STORAGE   → each Pod gets its OWN PVC (db-0 keeps its disk across restarts)
Ordered ops      → create/scale up in order 0,1,2…; terminate in reverse
```

## Persistent storage: PV, PVC, StorageClass

```text
StorageClass → the "how to provision" template (e.g. AWS gp3, fast-ssd)
PVC          → a REQUEST for storage ("I need 20Gi, ReadWriteOnce")   ← app-facing
PV           → the ACTUAL provisioned volume bound to the PVC          ← cluster-facing
Dynamic provisioning: a PVC referencing a StorageClass auto-creates a matching PV.

App → PVC → (StorageClass provisions) → PV → real disk (EBS, PD, Ceph, NFS…)
```

## A StatefulSet with per-Pod storage

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db            # headless Service for stable DNS
  replicas: 3
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: db
          image: postgres:16
          ports: [{ containerPort: 5432 }]
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:      # each Pod gets its OWN PVC/PV from this template
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: fast-ssd
        resources:
          requests: { storage: 20Gi }
```

```yaml
# The headless Service that gives stable per-Pod DNS (db-0.db, db-1.db, …)
apiVersion: v1
kind: Service
metadata: { name: db }
spec:
  clusterIP: None            # headless
  selector: { app: db }
  ports: [{ port: 5432 }]
```

## Access modes & caveats

```text
ReadWriteOnce (RWO)  → mounted read-write by ONE node (most block storage: EBS/PD)
ReadOnlyMany (ROX)   → read-only by many nodes
ReadWriteMany (RWX)  → read-write by many nodes (needs NFS/CephFS/EFS)
⚠ Deleting a StatefulSet does NOT delete its PVCs by default (data is preserved).
⚠ Reclaim policy (Retain/Delete) decides what happens to the PV when the PVC is removed.
⚠ Kubernetes gives identity+storage, but app-level clustering/replication is YOUR job
  (often via an Operator, e.g. for Postgres/Kafka).
```

```bash
kubectl get statefulset,pvc,pv
kubectl get pods -l app=db      # db-0, db-1, db-2 (stable, ordered)
```

## Why it matters

Running stateful workloads correctly is one of the harder parts of Kubernetes, so understanding StatefulSets and persistent storage is senior-level knowledge with real production consequences. The core insight is *why* Deployments fall short: their Pods are interchangeable (random names, any order, no per-Pod state), which is ideal for stateless services but wrong for databases and clustered stores that need **stable identity** (so peers can find `db-0`, `db-1`, `db-2` by DNS), **their own durable disk** that survives restarts and rescheduling, and **ordered, controlled** startup and scaling. A **StatefulSet** provides exactly these guarantees, and it pairs with the storage abstractions: a **PersistentVolumeClaim** is the application-facing request for storage, a **PersistentVolume** is the actual provisioned volume, and a **StorageClass** enables dynamic provisioning — with `volumeClaimTemplates` giving each Pod its own PVC/PV. The caveats are where senior engineers earn their keep: **access modes** (RWO vs RWX and which backends support them), the fact that deleting a StatefulSet **preserves its PVCs** by default, **reclaim policy** governing the PV's fate, and — critically — that Kubernetes provides identity and storage but **application-level replication and clustering remain your responsibility**, often delegated to an Operator. Because data durability and correct stateful behavior are unforgiving (mistakes mean data loss or split-brain), mastering StatefulSets, PVs/PVCs, and their caveats is fundamental to running databases and other stateful systems reliably on Kubernetes.

Why Deployments aren't enough

StatefulSet guarantees

Persistent storage: PV, PVC, StorageClass

A StatefulSet with per-Pod storage

Access modes & caveats

Why it matters

Related questions