Skip to content

HPA Flapping

Symptom

An HPA-managed workload repeatedly scales up and down over short intervals, often with latency spikes, pod churn, or customer-visible instability.

Possible Causes

  • behavior.scaleDown.stabilizationWindowSeconds is too small or omitted for a bursty workload.
  • Scale-up or scale-down policies are too aggressive for pod warm-up time.
  • The metric source is delayed, coarse, or noisy.
  • The metric being used measures stress after the fact instead of actual demand.

Diagnosis Steps

flowchart TD
    A[HPA replica churn] --> B[Inspect HPA events and target metrics]
    B --> C[Check metrics freshness and scrape delay]
    C --> D{Scale-down damping too weak?}
    D -->|Yes| E[Increase stabilization window]
    D -->|No| F{Scale policies too aggressive?}
    F -->|Yes| G[Reduce percent or pod-step changes]
    F -->|No| H[Revisit metric choice and minimum replicas]

  1. Inspect the HPA state.

    kubectl describe hpa <hpa-name> \
    --namespace <namespace>

  2. Compare replica changes with the underlying metric freshness.

  3. Look for patterns where pods are removed before the previous scale event has fully converged.

  4. If using Prometheus-backed metrics, verify the metric path is updating at the expected interval.

Resolution

  • Increase behavior.scaleDown.stabilizationWindowSeconds.
  • Use smaller scale steps for Percent or Pods policies.
  • Raise minimum replicas for latency-sensitive services.
  • Replace a lagging or ambiguous metric with one that reflects demand earlier.

Prevention

  • Treat HPA tuning as a control-loop problem, not only a threshold problem.
  • Test scaling policies under burst and cooldown patterns before production rollout.
  • Prefer managed Prometheus or another well-understood metrics path with known freshness.
  • Record accepted stabilization defaults per workload archetype.

See Also

Sources