Skip to content

Upgrade Failure

Symptom

An AKS upgrade does not start, stalls mid-rollout, or completes on the control plane while one or more node pools remain unhealthy.

Possible Causes

  • PodDisruptionBudgets block drain and surge progression.
  • Deprecated APIs or incompatible controllers block the target Kubernetes version.
  • Node image rollout is stuck after the version move is accepted.
  • Surge nodes cannot be created because subnet or IP capacity is exhausted.

Diagnosis Steps

flowchart TD
    A[Upgrade failed or stalled] --> B{Where is the failure?}
    B -->|Drain blocked| C[Upgrade Blocked by Pod Disruption Budget]
    B -->|Target rejected| D[Upgrade Blocked by Deprecated API]
    B -->|Node image rollout stuck| E[Node Image Upgrade Stuck]
    B -->|Surge nodes cannot start| F[Surge Upgrade IP Exhaustion]

  1. Review the control-plane result and node-pool state.

    az aks show \
    --resource-group "$RG" \
    --name "$CLUSTER_NAME" \
    --query "{version:currentKubernetesVersion,autoUpgradeProfile:autoUpgradeProfile,provisioningState:provisioningState}" \
    --output yaml
    Command Purpose
    az aks show Show version, auto-upgrade profile, and state.
    --resource-group Resource group that contains the AKS cluster.
    --name Name of the AKS cluster.
    --query Selects version, auto-upgrade profile, and provisioning state.
    --output Output format for the result.

  2. Review cluster events and the current node and PDB picture.

    kubectl get events --all-namespaces --sort-by=.lastTimestamp
kubectl get nodes
kubectl get pdb --all-namespaces

  3. Route to the focused playbook that matches the first hard failure:

  4. Upgrade Blocked by Pod Disruption Budget

  5. Upgrade Blocked by Deprecated API
  6. Node Image Upgrade Stuck
  7. Surge Upgrade IP Exhaustion

Resolution

  • Use the focused playbook that matches the actual blocker.
  • Do not keep retrying the same upgrade path until the blocking condition is removed.

Prevention

  • Keep support-window tracking continuous, not quarterly.
  • Rehearse upgrade and validation in non-production with the same PDB and surge assumptions.
  • Prefer blue/green for the highest-risk production upgrades.

See Also

Sources