Skip to content

StatefulSet Stuck During Rolling Update

Symptom

The StatefulSet update stops on one ordinal, one or more pods remain unavailable, or the rollout never progresses to the next pod.

Possible Causes

  • PodDisruptionBudget rules prevent eviction or replacement.
  • A prior ordinal never becomes Ready, so later ordinals never roll.
  • Headless Service or DNS dependencies are broken.
  • Storage attach or mount recovery for the updated pod is incomplete.

Diagnosis Steps

flowchart TD
    A[StatefulSet rollout stalls] --> B[Check rollout status and blocked ordinal]
    B --> C[Inspect PDBs readiness and events]
    C --> D[Check headless Service DNS and storage recovery]
    D --> E[Resolve the blocked pod before retrying]

  1. Inspect StatefulSet rollout status.

    kubectl rollout status statefulset "$STATEFULSET_NAME" \
    --namespace "$NAMESPACE"

  2. Inspect the StatefulSet and blocked pod events.

    kubectl describe statefulset "$STATEFULSET_NAME" \
    --namespace "$NAMESPACE"

kubectl describe pod "$POD_NAME" \
    --namespace "$NAMESPACE"

  3. Inspect PodDisruptionBudgets in the namespace.

    kubectl get pdb \
    --namespace "$NAMESPACE"

  4. Verify the headless Service and endpoint records.

    kubectl get service "$SERVICE_NAME" \
    --namespace "$NAMESPACE" \
    --output yaml

Resolution

  • Fix the first blocked ordinal before expecting later pods to update.
  • Relax or temporarily adjust the PDB if it blocks the required voluntary disruption.
  • Repair headless Service, DNS, or readiness conditions that keep the pod from becoming Ready.
  • Resolve underlying attach or mount failures, then re-run or resume the rollout.

Prevention

  • Test StatefulSet rollout behavior in nonproduction with the same PDB and readiness settings.
  • Keep storage changes and image changes in separate rollout windows when possible.
  • Document whether the workload depends on strict ordered rollout or can tolerate parallelism.

See Also

Sources