Memory Scale Rule Not Triggering Despite High Memory Percentage¶

1. Summary¶

Symptom¶

The Azure Portal metric chart for a Container Apps revision shows Memory Percentage (Preview) sustaining 60-70% for tens of minutes, while a memory scale rule configured with Utilization=50 does not increase replicas above minReplicas. The customer concludes "the memory scale rule is broken".

Why this scenario is confusing¶

There are two independent contributors to the same surface symptom, and they are easy to confuse:

1. HPA ceiling math. KEDA uses the standard Kubernetes HPA formula desiredReplicas = ceil(currentReplicas * currentMetric / targetMetric). With currentReplicas = 2 and targetMetric = 50, the formula returns 2 for any per-replica metric value at or below 50 and only rounds up to 3 once the per-replica value strictly exceeds 50. A 70% Portal reading does not by itself say what KEDA sees per replica.
2. Metric source mismatch. Portal Memory Percentage (Preview) comes from Azure Monitor and is reported as a percentage of the container memory limit; it reflects the cgroup working set, which can include reclaimable page cache. KEDA's memory scaler instead reads container memory from the Kubernetes Metrics Server and evaluates Utilization as a percentage of the container's requested memory. For cache-heavy workloads these two values can diverge by tens of percentage points.

The Portal chart is intended for OOM-risk situational awareness, not as a ground-truth view of what KEDA evaluates.

What to look at for which question¶

Troubleshooting decision flow¶

flowchart TD
    A[Portal MemoryPercentage high, replicas flat] --> B{Apply split-by-replica<br/>in Portal metric chart}
    B --> C{Per-replica<br/>average value M}
    C --> D{Compute<br/>ceil currentReplicas times M over target}
    D -->|result equals currentReplicas| H1[H1 HPA ceiling math<br/>working as designed]
    D -->|result greater than currentReplicas| E{Did replicas actually rise<br/>after polling interval}
    E -->|Yes| H2[H2 Working as designed<br/>customer missed the event]
    E -->|No| F{cgroup memory stat<br/>file dominates anon}
    F -->|Yes| H3[H3 Page cache inflating<br/>Portal metric only]
    F -->|No| H4[H4 Real scaler problem<br/>check KEDA logs]

2. Common Misreadings¶

• "Portal shows 70%, so KEDA must see 70%." Portal and KEDA use different metric pipelines. The Portal value is informational; KEDA's number is what drives scale decisions.
• "Any value above the target should scale out." The HPA formula divides by targetMetric and multiplies by currentReplicas before rounding up. With two replicas at target 50, the per-replica average has to exceed ~50% (i.e., produce ceil(2 * 51/50) = 3) before scale-out occurs.
• "KEDA is broken if it does not react to a single spike." KEDA polls on a configurable interval (default 30 seconds for the memory scaler) and applies a cooldown (default 300 seconds) before scaling in.
• "The chart average reflects each replica." It does not unless you apply splitting. A two-replica app with one heavy and one idle replica shows the mean, not the per-replica peak.

3. Competing Hypotheses¶

4. What to Check First¶

Per-replica metric splitting¶

In the Azure Portal, open the Container App's Memory Percentage chart and apply Apply splitting -> Replica. If one replica is far above the others, the average misrepresents the per-replica scaler input.

HPA ceiling math (quick calculator)¶

TARGET=50
CURRENT_REPLICAS=2
CURRENT_METRIC=40      # observed average per-replica utilization
python3 -c "import math; print(math.ceil($CURRENT_REPLICAS * $CURRENT_METRIC / $TARGET))"

If the printed value equals currentReplicas, no scale-out is expected.

cgroup memory composition (anon vs file)¶

APP_NAME="ca-myapp"
RG="rg-myapp"
ACTIVE_REV="$(az containerapp revision list --name "$APP_NAME" --resource-group "$RG" \
  --query '[?properties.active]|[0].name' --output tsv)"
REPLICA="$(az containerapp replica list --name "$APP_NAME" --resource-group "$RG" \
  --revision "$ACTIVE_REV" --query '[0].name' --output tsv)"

az containerapp exec --name "$APP_NAME" --resource-group "$RG" \
  --replica "$REPLICA" --container "$APP_NAME" \
  --command "/bin/sh -c '(cat /sys/fs/cgroup/memory.stat 2>/dev/null || cat /sys/fs/cgroup/memory/memory.stat) | head -10'"

The path varies by cgroup version: cgroup v2 exposes /sys/fs/cgroup/memory.stat; cgroup v1 exposes /sys/fs/cgroup/memory/memory.stat. The field names also differ — cgroup v1 uses rss / cache, while cgroup v2 uses anon / file (and splits file further into inactive_file / active_file). Treat rss ≈ anon and cache ≈ file when comparing the two layouts.

If file is much larger than anon (or cache ≫ rss on v1), the working set is dominated by page cache. In that situation Azure Monitor's Memory Percentage reads high while the Kubernetes resource-memory value KEDA evaluates often reads materially lower, because cgroup totals and the scaler input do not map one-to-one.

KEDA polling and cooldown defaults¶

5. Resolution¶

6. Prevention¶

• Document the HPA ceiling math in the scaling design review so target and minReplicas are chosen with awareness of the rounding boundary.
• When designing memory-based scale rules, prefer working-set values that reflect actual demand (anonymous memory, active page cache) over raw cgroup totals. The Portal metric is an OOM-risk signal, not a scaler proxy.
• Add an alert on Replicas (Max) plateau combined with `MemoryPercentage

  60%` for more than 15 minutes; investigate per the decision flow above rather than assuming the scaler is broken.

See Also¶

• Lab guide: Memory Percentage vs KEDA Utilization
• Playbook: HTTP Scaling Not Triggering
• Playbook: Replica Load Imbalance
• Platform: CPU and Memory Scaler

Sources¶

• Set scaling rules - Azure Container Apps
• Available metrics - Azure Container Apps
• Horizontal Pod Autoscaler algorithm details - Kubernetes
• KEDA memory scaler