Scaling Operations¶

Scaling is operationally safe only when you understand which layer is changing: replicas, requests, or node capacity. This runbook focuses on operating those controls during normal growth and incident response.

Prerequisites¶

Metrics pipeline is available for pods and nodes.
Workloads have requests and limits defined.
Autoscaler min/max boundaries are documented.

When to Use¶

Traffic growth requires more replicas or nodes.
Pending pods suggest cluster capacity constraints.
Cost review requires tuning autoscaler boundaries.

Procedure¶

flowchart TD
    A[Observe pressure] --> B[Check HPA and requests]
    B --> C[Check node capacity]
    C --> D[Adjust autoscaler or pool]
    D --> E[Verify readiness and latency]

kubectl get hpa \
    --all-namespaces
kubectl top nodes
kubectl top pods \
    --all-namespaces
az aks update \
    --resource-group "$RG" \
    --name "$CLUSTER_NAME" \
    --enable-cluster-autoscaler \
    --min-count 3 \
    --max-count 10

Command	Purpose
`kubectl get hpa`	List HorizontalPodAutoscalers across namespaces.
`kubectl top nodes`	Show current node CPU and memory usage.
`kubectl top pods`	Show current pod CPU and memory usage.
`az aks update`	Enable the cluster autoscaler bounds.
`--resource-group`	Resource group that contains the AKS cluster.
`--name`	Name of the AKS cluster.
`--enable-cluster-autoscaler`	Turn on the cluster autoscaler.
`--min-count`	Minimum node count for autoscaling.
`--max-count`	Maximum node count for autoscaling.

Verification¶

kubectl describe hpa <hpa-name> \
    --namespace <namespace>
kubectl get pods \
    --all-namespaces \
    --field-selector=status.phase=Pending
az aks show \
    --resource-group "$RG" \
    --name "$CLUSTER_NAME" \
    --query "agentPoolProfiles[].{name:name,min:minCount,max:maxCount,count:count}" \
    --output table

Command	Purpose
`kubectl describe hpa`	Show HorizontalPodAutoscaler status and events.
`kubectl get pods`	List pending pods awaiting scheduling.
`az aks show`	Show autoscaler bounds per node pool.
`--resource-group`	Resource group that contains the AKS cluster.
`--name`	Name of the AKS cluster.
`--query`	Selects per-pool min, max, and count.
`--output`	Output format for the result.

Rollback / Troubleshooting¶

Reduce aggressive HPA targets if scale churn causes instability.
If node growth stalls, inspect quota and subnet IP capacity.
If scaling works but latency remains high, investigate application bottlenecks rather than adding more nodes blindly.
If queue-driven workers are not scaling, troubleshoot the KEDA path instead of only the HPA path.
If fixed node pools are becoming hard to manage, reassess whether node autoprovisioning is the better operating model.