Skip to content

Availability-Zone-Imbalanced Node Pools and Spread Failures

Symptom

The workload's placement rules are reasonable, but AZ recovery still fails because one zone has fewer nodes, zero nodes, a lower max-count, or a zone-specific quota or SKU constraint that prevents AKS from restoring balanced capacity.

Possible Causes

  • One zone has no dedicated node pool or its pool scaled down much further than the others.
  • max-count is lower in the zone that must absorb the missing replicas.
  • The requested VM SKU is unavailable in one of the target zones.
  • Cluster autoscaler is enabled, but zonal scale behavior is misaligned with the workload's strict spread constraints.
  • The cluster relies on a multi-zone pool when the workload actually needs zone-aligned scaling control.

Diagnosis Steps

flowchart TD
    A[Spread contract looks correct] --> B[Inspect per-zone node-pool counts and bounds]
    B --> C{Zone missing capacity or max headroom?}
    C -->|Yes| D[Check autoscaler profile and per-zone pool symmetry]
    C -->|No| E[Check SKU support quota and regional capacity]
    D --> F[Rebuild balanced zonal pool model]
    E --> F

  1. Map the current node inventory by zone and pool.

    kubectl get nodes \
    --label-columns=topology.kubernetes.io/zone,kubernetes.azure.com/agentpool

  2. Compare Azure-side pool counts, zones, and autoscaler bounds.

    az aks nodepool list \
    --resource-group "$RG" \
    --cluster-name "$CLUSTER_NAME" \
    --query "[].{name:name,zones:availabilityZones,count:count,enableAutoScaling:enableAutoScaling,minCount:minCount,maxCount:maxCount,vmSize:vmSize}" \
    --output table
    Command Purpose
    az aks nodepool list List node pools with zones and scaling.
    --resource-group Resource group that contains the AKS cluster.
    --cluster-name Name of the AKS cluster.
    --query Selects name, zones, count, autoscaling, and VM size.
    --output Output format for the result.

  3. If a zone looks empty or permanently smaller, check whether the pool design itself is asymmetric.

    Look for patterns such as:

    • zone 1 pool max-count 6, zone 2 pool max-count 2, zone 3 pool absent
    • one shared multi-zone pool plus one extra zone-aligned pool that biases recovery

  4. Verify that the requested VM family is actually accepted in the needed zones.

    az aks list-vm-skus \
    --location "$LOCATION" \
    --output table
    Command Purpose
    az aks list-vm-skus List VM SKUs supported for AKS node pools.
    --location Azure region to query.
    --output Output format for the result.

  5. Check the cluster autoscaler design if the workload depends on zone-aligned recovery.

    az aks show \
    --resource-group "$RG" \
    --name "$CLUSTER_NAME" \
    --query "autoScalerProfile"
    Command Purpose
    az aks show Show the cluster autoscaler profile.
    --resource-group Resource group that contains the AKS cluster.
    --name Name of the AKS cluster.
    --query Selects the autoscaler profile.

  6. If a zone cannot scale because of quota or broader capacity, pivot to Azure-side capacity evidence.

    az vm list-usage \
    --location "$LOCATION" \
    --output table
    Command Purpose
    az vm list-usage Show regional vCPU quota usage.
    --location Azure region to query.
    --output Output format for the result.

    If the cluster uses node auto-provisioning for some workloads, also cross-check NAP Fails to Provision.

Resolution

  • Rebuild the zonal pool model so each required zone has explicit, reachable headroom.
  • Raise max-count symmetrically across per-zone pools when the workload requires even recovery behavior.
  • Enable balance-similar-node-groups when multiple similar zonal pools back the same workload.
  • Move from one multi-zone pool to one pool per zone when strict zonal behavior matters more than simplified pool management.
  • Relax the accepted VM family or request additional quota when one zone cannot provision the current SKU.

Prevention

  • Treat zonal node-pool symmetry as part of the workload contract, not as an incidental scaling detail.
  • Review node-pool counts and bounds after every cost-optimization or autoscaler-policy change.
  • Validate the chosen VM size in every target zone before standardizing on it for strict spread workloads.
  • Keep Topology Spread Skew Under Capacity linked in incident runbooks so responders distinguish manifest intent from cluster-shape defects.

See Also

Sources