Skip to content

Ready Capacity Drops Below Desired

Symptom

deployment.spec.replicas still shows the desired count, but readyReplicas or availableReplicas dips during node loss, zonal impairment, replacement image pull failure, unschedulable recovery, or autoscaler lag.

Possible Causes

  • A node crash or node pool impairment removed running pods faster than replacement pods could start.
  • A zonal incident reduced available nodes in one failure domain and the workload could not rebalance immediately.
  • Replacement pods exist but cannot become Ready because of image pull failure, probe failure, or missing capacity.
  • The cluster autoscaler is enabled, but recovery takes longer than the application's tolerance window.
  • The workload relies on PDBs or graceful termination to avoid a gap, even though the actual incident path is involuntary.

Diagnosis Steps

flowchart TD
    A[Desired replicas unchanged] --> B[Compare desired ready and available counts]
    B --> C[Check node loss and zonal health]
    C --> D{Replacement pods created?}
    D -->|No| E[Inspect scheduling and autoscaler path]
    D -->|Yes| F[Inspect image pull and readiness path]
    E --> G[Restore capacity or relax constraints]
    F --> G

  1. Confirm the deployment-level gap.

    kubectl get deployment <deployment-name> \
    --namespace <namespace> \
    --output wide

  2. Inspect pod placement and current readiness state.

    kubectl get pods \
    --namespace <namespace> \
    --selector app=<label> \
    --output wide

  3. Check whether a node or zone event preceded the dip.

    kubectl get nodes \
    --label-columns=topology.kubernetes.io/zone,kubernetes.azure.com/agentpool

    kubectl get events \
    --all-namespaces \
    --sort-by=.lastTimestamp

  4. If replacement pods are missing, inspect scheduling failure and autoscaler bounds.

    az aks nodepool list \
    --resource-group "$RG" \
    --cluster-name "$CLUSTER_NAME" \
    --query "[].{name:name,zones:availabilityZones,count:count,enableAutoScaling:enableAutoScaling,minCount:minCount,maxCount:maxCount}" \
    --output table
    Command Purpose
    az aks nodepool list List node pools with zones and scaling.
    --resource-group Resource group that contains the AKS cluster.
    --cluster-name Name of the AKS cluster.
    --query Selects name, zones, count, and autoscaling.
    --output Output format for the result.

  5. If replacement pods exist but are not Ready, inspect their exact failure path.

    kubectl describe pod <replacement-pod-name> \
    --namespace <namespace>

    Common pivots:

  6. Check whether the workload is expecting PDBs or graceful termination to solve an involuntary failure path.

    kubectl get pdb \
    --namespace <namespace>

    PDBs and preStop can reduce some voluntary disruption paths, but they do not eliminate the gap after crashes, zone loss, image pull failure, or unschedulable replacements.

Resolution

  • Restore lost node capacity first when the incident began with node or zone impairment.
  • Fix image pull or readiness failures on replacement pods before tuning disruption settings.
  • Increase replica count and spread replicas across zones if the current steady-state count cannot absorb one-node or one-zone loss.
  • Reduce scheduler strictness or rebalance per-zone capacity if replacements stay pending.
  • Tune autoscaler bounds and headroom so the cluster can recover fast enough for the workload's recovery objective.

Prevention

  • Run enough replicas to survive at least one node loss without dropping below the minimum usable capacity.
  • Combine spread rules, autoscaler bounds, and image delivery design so replacement pods can actually start in another zone.
  • Use PDBs, preStop, and terminationGracePeriodSeconds for voluntary drain paths, while documenting that involuntary failure still needs spare capacity and fast replacement.
  • Rehearse node-loss, zonal-pressure, and image-registry failure scenarios separately.

See Also

Sources