Skip to content

Spot Eviction Storm

Symptom

Many spot-backed nodes are evicted in a short period, causing a sudden wave of pod rescheduling, backlog growth, or degraded service quality.

Possible Causes

  • The workload mix depends too heavily on spot capacity without on-demand fallback.
  • The region or VM family experiences a temporary capacity squeeze.
  • Retry or checkpoint logic is weak, so normal spot churn becomes application failure.
  • Autoscaler minimums and fallback pools are too small for the resulting displacement.

Diagnosis Steps

flowchart TD
    A[Spot eviction spike] --> B[Confirm affected pool and displaced workloads]
    B --> C[Check autoscaler min and fallback on-demand capacity]
    C --> D{Workloads rescheduling cleanly?}
    D -->|No| E[Fix tolerations affinity and retry path]
    D -->|Yes| F[Assess whether mixed pool ratio is too aggressive]
    F --> G[Increase regular pool or reduce spot dependence]

  1. Identify the impacted spot pool and the workloads running there.

  2. Confirm whether the workloads have the expected spot toleration and affinity, rather than accidentally landing on spot through broad selectors.

  3. Check autoscaler settings for both the spot pool and any regular fallback pool.

    az aks nodepool show \
    --resource-group "$RG" \
    --cluster-name "$CLUSTER_NAME" \
    --name "$NODEPOOL_NAME" \
    --query "{priority:scaleSetPriority,min:minCount,max:maxCount,count:count,eviction:scaleSetEvictionPolicy}" \
    --output yaml
    Command Purpose
    az aks nodepool show Show a spot node pool's priority and scaling.
    --resource-group Resource group that contains the AKS cluster.
    --cluster-name Name of the AKS cluster.
    --name Name of the node pool to show.
    --query Selects priority, min, max, count, and eviction policy.
    --output Output format for the result.

  4. Review pod retry and reschedule behavior.

Resolution

  • Shift critical or customer-facing workloads back to on-demand pools.
  • Increase on-demand fallback capacity or autoscaler minimums.
  • Improve queue checkpointing, retries, and graceful shutdown behavior for interruptible workers.
  • Reduce spot dependence for VM families with repeated eviction pressure.

Prevention

  • Keep spot workloads explicitly isolated and interruption tolerant.
  • Use mixed pools so evictions degrade throughput before they degrade availability.
  • Rehearse eviction scenarios during load testing, not during incidents.
  • Review whether the chosen spot ratio still matches business tolerance for delayed work.

See Also

Sources