Cluster Autoscaler Issues¶

1. Summary¶

Use this playbook when pending pods do not trigger scale-out, empty nodes do not scale in, or autoscaler events show repeated failures. In AKS, the most common causes are tight node pool min and max limits, unsupported scheduling constraints, exhausted subnet or quota limits, or incorrect expectations about daemonsets and disruption budgets.

Typical incident window: 10-20 minutes to establish whether the issue is workload-specific, node-specific, or cluster-wide. Time to resolution: 30 minutes to several hours depending on whether the fix is manifest-level, node-level, or Azure control-plane level.

Symptoms¶

• Pods remain Pending with scheduling messages even though autoscaler is enabled.
• kubectl describe pod shows CPU, memory, or node affinity constraints that no current node satisfies.
• AKS activity or autoscaler logs mention scale-up rejection, quota issues, or subnet capacity exhaustion.
• Scale-in never happens because PDBs, local storage, or daemonset overhead keep nodes non-empty.

Diagnostic flowchart¶

flowchart TD
    A[Reported symptom] --> B{Can the object be reproduced now?}
    B -->|No| C[Use recent events, Container Insights, and rollout history]
    B -->|Yes| D[Capture current state with kubectl and Azure CLI]
    D --> E{Is the issue isolated to one workload or node pool?}
    E -->|Workload| F[Check image, probes, config, and service wiring]
    E -->|Node pool| G[Check node health, autoscaler, subnet, and VMSS state]
    F --> H[Validate with KQL and controller logs]
    G --> H
    H --> I[Apply targeted fix and verify telemetry returns to baseline]

2. Common Misreadings¶

3. Competing Hypotheses¶

4. What to Check First¶

1. Confirm the current object state from Kubernetes

   kubectl get pods \
       --all-namespaces \
       --output wide
   

2. Describe the affected object to capture recent events

   kubectl describe pod <pod-name> \
       --namespace <namespace>
   

3. Check AKS cluster and node pool configuration from Azure

   az aks show \
       --resource-group "$RG" \
       --name "$CLUSTER_NAME" \
       --query "{name:name,provisioningState:provisioningState,kubernetesVersion:kubernetesVersion,nodeResourceGroup:nodeResourceGroup}" \
       --output json
   

   Command	Purpose
   az aks show	Show core cluster properties.
   --resource-group	Resource group that contains the AKS cluster.
   --name	Name of the AKS cluster.
   --query	Selects name, provisioning state, version, and node resource group.
   --output	Output format for the result.

4. List node pools and autoscaler settings

   az aks nodepool list \
       --resource-group "$RG" \
       --cluster-name "$CLUSTER_NAME" \
       --output table
   

   Command	Purpose
   az aks nodepool list	List the node pools in the cluster.
   --resource-group	Resource group that contains the AKS cluster.
   --cluster-name	Name of the AKS cluster.
   --output	Output format for the result.

5. Run a fast Container Insights control query

   az monitor log-analytics query \
       --workspace "$WORKSPACE_ID" \
       --analytics-query "KubePodInventory | where TimeGenerated > ago(15m) | summarize Restarts=sum(ContainerRestartCount) by Namespace | order by Restarts desc" \
       --timespan "PT15M"
   

   Command	Purpose
   az monitor log-analytics query	Query per-namespace container restarts.
   --workspace	Log Analytics workspace to query.
   --analytics-query	KQL query text to execute.
   --timespan	Time range for the query.

5. Evidence to Collect¶

5.1 KQL Queries¶

KubePodInventory
| where TimeGenerated > ago(30m)
| summarize Restarts=max(ContainerRestartCount), LastSeen=max(TimeGenerated) by ClusterName, Namespace, PodName, ContainerName
| order by Restarts desc

ContainerLogV2
| where TimeGenerated > ago(30m)
| summarize LogLines=count(), LastSeen=max(TimeGenerated) by Namespace, PodName
| order by LastSeen desc

KubeEvents
| where TimeGenerated > ago(30m)
| where Reason in ("Failed", "BackOff", "Unhealthy", "NodeNotReady", "FailedScheduling")
| project TimeGenerated, Namespace, Name, Reason, Message
| order by TimeGenerated desc

5.2 CLI Investigation¶

kubectl logs <pod-name> \
    --namespace <namespace> \
    --previous

Interpretation: previous logs are usually more valuable than current logs during restart loops because they contain the container exit path.

kubectl get events \
    --all-namespaces \
    --sort-by=.lastTimestamp

Interpretation: look for probe failures, image pull errors, FailedScheduling, NodeNotReady, or backend controller warnings near the incident start time.

az vmss list-instances \
    --resource-group "$NODE_RESOURCE_GROUP" \
    --name "$VMSS_NAME" \
    --query "[].{instanceId:instanceId,provisioningState:provisioningState,latestModelApplied:latestModelApplied}" \
    --output table

Interpretation: when the problem is node- or ingress-related, VMSS state and model drift provide important Azure-side evidence.

6. Validation and Disproof by Hypothesis¶

Node pool limits or autoscaler profile settings prevent scale-out¶

Proves if: Node pool max count, surge headroom, or profile settings block new nodes.

Disproves if: Node pool max count, surge headroom, and profile settings do not block new nodes.

kubectl get configmap cluster-autoscaler-status \
    --namespace kube-system \
    --output yaml

Pod constraints cannot be satisfied by any node pool¶

Proves if: Node selectors, taints, zones, or resource requests mismatch the available pools.

Disproves if: Node selectors, taints, zones, and resource requests do not mismatch the available pools.

kubectl describe pod <pod-name> \
    --namespace <namespace>

Azure quota or subnet capacity prevents new nodes¶

Proves if: Azure CLI shows VMSS, vCPU, or subnet address space limits reached.

Disproves if: Azure CLI does not show VMSS, vCPU, or subnet address space limits reached.

az vm list-usage \
    --location "$LOCATION" \
    --output table

Scale-in blocked by workload protections¶

Proves if: PDBs, local storage, or daemonsets keep nodes from becoming removable.

Disproves if: PDBs, local storage, and daemonsets do not keep nodes from becoming removable.

kubectl get poddisruptionbudgets \
    --all-namespaces

7. Likely Root Cause Patterns¶

8. Immediate Mitigations and Step-by-Step Resolution¶

1. Inspect pending pod events and the target node pool autoscaler settings together.
2. Increase max counts or add a suitable pool only after proving that workload constraints are legitimate.
3. Resolve quota or subnet exhaustion before retrying scale actions.
4. Tune PDBs, requests, and daemonset placement so scale-in can happen safely.
5. Review cost and reliability implications after every autoscaler policy change.

If the incident pattern keeps recurring because workload shapes no longer fit a fixed-pool model, stop treating it as a one-off autoscaler tuning issue and evaluate Node Autoprovisioning.

Example resolution commands:

kubectl rollout restart deployment/<deployment-name> \
    --namespace <namespace>

az aks nodepool update \
    --resource-group "$RG" \
    --cluster-name "$CLUSTER_NAME" \
    --name "$NODEPOOL_NAME" \
    --max-count 10

9. Prevention Checklist¶

• [ ] Create saved Container Insights queries for the symptom family and link them in the team runbook.
• [ ] Require long-flag CLI examples and standardized evidence capture in incident response docs.
• [ ] Review ingress, autoscaler, probes, and node pool settings during every production readiness review.
• [ ] Alert on restart spikes, NotReady nodes, and FailedScheduling events before customers report impact.
• [ ] Document which changes require platform-team approval, especially around networking, ingress, and security policy.

See Also¶

• Scaling Failure
• Node Autoprovisioning
• Best Practices: Autoscaling
• Cost Optimization
• Tutorial 05: AKS Disaster Recovery

Sources¶

• Troubleshoot / Azure / Azure Kubernetes / Welcome Azure Kubernetes
• Azure / Aks / Cluster Autoscaler
• Azure / Azure Monitor / Containers / Container Insights Overview
• Azure / Aks / Concepts Network