KEDA "No Metrics Returned" Reproduction Lab¶

Reproduce the KEDA/HPA log messages no metrics returned from resource metrics API and invalid metrics by creating containers that are intentionally Not Ready or crash-looping, then compare against a healthy baseline to confirm the errors are caused by container lifecycle events.

Lab Metadata¶

1) Background¶

KEDA's CPU and memory scalers query the Kubernetes Resource Metrics API for per-container utilization data. The Metrics Server can only return data for containers that are Running and Ready. When a container is:

• Still initializing (readiness probe not yet passing)
• Restarting after a crash or OOMKill
• Being rescheduled during platform maintenance

…the Metrics API returns an empty response, and the HPA controller logs:

failed to get memory usage: unable to get metrics for resource memory: no metrics returned from resource metrics API

If the app has multiple scale rules and only some metrics fail:

invalid metrics (1 invalid out of 5), first error is: failed to get <app> container metric value: ...

Additionally, scale rules using --scale-rule-metadata "type=Utilization" trigger a deprecation warning because KEDA v2.7+ replaced metadata.type with the trigger-level metricType field (removed in v2.18):

scaler memory info: The 'type' setting is DEPRECATED and will be removed in v2.18 - Use 'metricType' instead.

2) Hypothesis¶

IF three Container Apps share the same memory scale rule but run workloads with different startup/crash behaviors, THEN:

• Scenario A (slow-start): System logs show "no metrics returned" during the first ~2 minutes (while the container is Not Ready), then the errors stop once the readiness probe passes.
• Scenario B (crash-loop): System logs show recurring "no metrics returned" and "invalid metrics" that correlate with container restart timestamps. The pattern repeats with exponential backoff intervals.
• Scenario C (healthy): System logs show a brief burst of metric error entries (~30-60s) immediately after deployment due to the Kubernetes Metrics Server warm-up period, then no further errors. The DEPRECATED warning also appears (it is independent of container health).

Architecture¶

flowchart TD
    A["Container Apps environment"] --> B["Scenario A: ca-nometrics-slow"]
    A --> C["Scenario B: ca-nometrics-crash"]
    A --> D["Scenario C: ca-nometrics-healthy"]
    B --> B1["MODE=slow-start<br/>DELAY_SECONDS=120"]
    C --> C1["MODE=crash-loop<br/>DELAY_SECONDS=30"]
    D --> D1["MODE=healthy"]
    B1 --> S["Same scale rule:<br/>memory Utilization=50<br/>min 1-2 / max 10"]
    C1 --> S
    D1 --> S
    S --> R["Observe:<br/>ContainerAppSystemLogs_CL<br/>RestartCount<br/>Replica status"]

3) Runbook¶

Deploy infrastructure¶

export RG="rg-aca-no-metrics-lab"
export LOCATION="koreacentral"
export BASE_NAME="nometrics"

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
    --resource-group "$RG" --name main \
    --template-file labs/keda-no-metrics-returned/infra/main.bicep \
    --parameters baseName="$BASE_NAME"

export ACR_NAME="$(az deployment group show --resource-group "$RG" --name main \
    --query properties.outputs.containerRegistryName.value --output tsv)"
export ENV_NAME="$(az deployment group show --resource-group "$RG" --name main \
    --query properties.outputs.environmentName.value --output tsv)"

Create three scenarios¶

bash labs/keda-no-metrics-returned/trigger-scenario-a.sh
bash labs/keda-no-metrics-returned/trigger-scenario-b.sh
bash labs/keda-no-metrics-returned/trigger-scenario-c.sh

Observe (wait at least 10 minutes for Log Analytics ingestion)¶

sleep 600

for APP in ca-nometrics-slow ca-nometrics-crash ca-nometrics-healthy; do
    APP_NAME=$APP bash labs/keda-no-metrics-returned/verify.sh
done

verify.sh queries ContainerAppSystemLogs_CL for "no metrics returned", "invalid metrics", and "DEPRECATED" messages, then checks console logs, replica status, and restart counts.

4) Expected Evidence¶

The hypothesis is confirmed when all of the following hold:

4a) Experiment Log¶

Tested in Azure region Korea Central, 2026-06-05, az CLI 2.73.0 (containerapp extension preview).

Metric error summary [Measured]¶

App                   ErrorCount  FirstError (UTC)       LastError (UTC)
--------------------  ----------  ---------------------  ---------------------
ca-nometrics-slow     12          04:05:14               04:06:30  (~76s window)
ca-nometrics-crash    29          04:06:07               04:26:15  (20+ min, ongoing)
ca-nometrics-healthy  10          04:10:26               04:11:27  (~61s window)

Key observations¶

1. Scenario A (slow-start): 12 error entries in 76 seconds. The container slept 120s before starting the HTTP server, so the readiness probe (StartUp probe) failed continuously — confirmed by ProbeFailed system log entries. Once the server started at 04:07:15, metric errors stopped.

2. Scenario B (crash-loop): 29 error entries over 20+ minutes. The container exited every 30s, triggering restarts with exponential backoff. Errors appeared in ~15s intervals during active restarts, then spaced out to ~5min as CrashLoopBackOff lengthened the restart delay.

3. Scenario C (healthy): Unexpected finding — 10 error entries in 61 seconds immediately after deployment. The container started within seconds ([app] listening on :8000 logged instantly), but the Kubernetes Metrics Server needed ~60s to begin returning data for the new pod. This proves that even a perfectly healthy container produces "no metrics returned" logs during initial provisioning.

4. DEPRECATED warning: All three apps produced exactly one instance of:

   scaler memory info: The 'type' setting is DEPRECATED and will be removed in v2.18 - Use 'metricType' instead.
   

   This confirms the warning is configuration-driven, not health-dependent.

Error log samples (PII masked)¶

# Scenario A — transient during startup probe failure
[04:05:14] ca-nometrics-slow | FailedGetContainerResourceMetric | failed to get memory usage: unable to get metrics for resource memory: no metrics returned from resource metrics API
[04:05:14] ca-nometrics-slow | FailedComputeMetricsReplicas     | invalid metrics (1 invalid out of 5), first error is: failed to get ca-nometrics-slow container metric value: ...

# Scenario B — recurring with crash-loop
[04:06:07] ca-nometrics-crash | FailedGetContainerResourceMetric | failed to get memory usage: ...
[04:08:38] ca-nometrics-crash | FailedGetContainerResourceMetric | failed to get memory usage: ...
[04:16:12] ca-nometrics-crash | FailedGetContainerResourceMetric | failed to get memory usage: ...  (interval lengthened due to backoff)
[04:21:13] ca-nometrics-crash | FailedGetContainerResourceMetric | failed to get memory usage: ...
[04:26:15] ca-nometrics-crash | FailedGetContainerResourceMetric | failed to get memory usage: ...

# Scenario C — brief deployment gap even for healthy container
[04:10:26] ca-nometrics-healthy | FailedGetContainerResourceMetric | failed to get memory usage: ...
[04:11:27] ca-nometrics-healthy | FailedComputeMetricsReplicas     | invalid metrics (1 invalid out of 5), ...
(no further errors after 04:11:27)

# All scenarios — DEPRECATED warning (once per app)
[04:04:59] ca-nometrics-slow    | scaler memory info: The 'type' setting is DEPRECATED ...
[04:05:52] ca-nometrics-crash   | scaler memory info: The 'type' setting is DEPRECATED ...
[04:10:12] ca-nometrics-healthy | scaler memory info: The 'type' setting is DEPRECATED ...

Operator takeaway from experiment¶

The Scenario C result is the most important takeaway: expect 30-60 seconds of "no metrics returned" logs after every deployment, even when the application starts instantly. This is a normal Kubernetes Metrics Server warm-up period, not a defect.

4b) Phase B Falsification Gates¶

verify.sh is a pure file processor against the 18 historical-capture artifacts (6 files × 3 scenarios under evidence/ca-nometrics-*/) and emits four falsifiable gate JSONs under evidence/. Each gate is evaluated against a strict 2-path predicate per sub-gate — a Strong path that matches the exact lab specification (e.g. exact field match in a specific JSON file or column-aligned section parse), and a Fallback path that tolerates the same controlling behavior under minor numeric drift (e.g. substring search or weaker ordering check). All twelve sub-gates (4 gates × 3 sub-gates each) emitted on 2026-06-20 pass on both the Strong and Fallback paths.

The Gate 13 cross-scenario ratio threshold is intentionally lower than the observed 4.44× (set to ≥ 3.0×) to absorb run-to-run variance while still falsifying the case where crash-loop duration converges with the healthy/slow baselines. The Gate 11 sub-gate (b) intentionally keys off the §2 inline JSON runningState and provisioningState fields rather than the sidecar revisions-*.json (which carries only healthState, name, replicas, trafficWeight) because runningState and provisioningState are the authoritative platform-observed state for a failing revision. See labs/keda-no-metrics-returned/evidence/README.md "Honest disclosure" section for the full empirical platform behavior documented during the 2026-06-20 live run, including the sidecar replicas semantics (DESIRED replica count from the revision spec, NOT the count of replicas that successfully reached Running).

The H1 gates (10/11) together prove that "no metrics returned from resource metrics API" is not a generic platform noise signal — its count, span, and duration scale with container unreadiness severity. The H2 gate (12) proves that the same signal still appears (in bounded form) on a healthy container during Metrics Server warm-up. The H3 cross-scenario gate (13) proves the differential — crash-loop produces materially more errors over a materially longer window than either the slow-startup or healthy baseline. Together these four gates yield the operator guidance in the Operator takeaway from experiment subsection above: a transient burst of "no metrics returned" during the first 30–90 seconds after revision creation is expected (Metrics Server warm-up); the same signal spanning multiple 5-minute bins with §2 reporting Unhealthy + Failed indicates a container-health defect that requires application-level diagnosis (crash-loop, OOMKill, readiness probe misconfiguration), not a platform escalation.

5) Verification Queries¶

KQL: metric error log pattern¶

ContainerAppSystemLogs_CL
| where ContainerAppName_s in ("ca-nometrics-slow", "ca-nometrics-crash", "ca-nometrics-healthy")
| where Log_s has_any ("no metrics returned", "invalid metrics", "failed to get")
| summarize ErrorCount=count() by ContainerAppName_s, bin(TimeGenerated, 5m)
| order by TimeGenerated asc

Expected: ca-nometrics-crash shows sustained error counts across multiple 5-min bins. ca-nometrics-slow shows errors only in the first 1-2 bins. ca-nometrics-healthy shows errors only in the first bin (deployment warm-up), then zero.

KQL: DEPRECATED warning¶

ContainerAppSystemLogs_CL
| where ContainerAppName_s in ("ca-nometrics-slow", "ca-nometrics-crash", "ca-nometrics-healthy")
| where Log_s has "DEPRECATED"
| summarize count() by ContainerAppName_s

Expected: All three apps show the warning (it is configuration-based, not health-based).

6) Portal Evidence (to capture after reproduction)¶

Azure Portal screenshots to collect for each scenario. Save to docs/assets/troubleshooting/keda-no-metrics-returned/.

Scenario A — ca-nometrics-slow (transient metric gap during startup)¶

Scenario B — ca-nometrics-crash (recurring metric gaps from CrashLoopBackOff)¶

Scenario C — ca-nometrics-healthy (brief deployment gap)¶

All scenarios — DEPRECATED warning¶

All scenarios — Error timeline (KQL)¶

Screenshot capture checklist¶

When re-running the lab, capture the following screenshots and save to docs/assets/troubleshooting/keda-no-metrics-returned/:

Clean Up¶

bash labs/keda-no-metrics-returned/cleanup.sh

Related Playbook¶

• KEDA "No Metrics Returned from Resource Metrics API"

See Also¶

• Memory Percentage vs KEDA Utilization Lab
• CPU and Memory Scaler
• Scale Rule Mismatch Lab

Sources¶

• Set scaling rules - Azure Container Apps
• KEDA memory scaler
• HPA with container metrics fails when pod is not ready - kubernetes#127169
• Deprecating parameter 'type' in CPU/Memory scaler - kedacore/keda#6348
• Troubleshooting HPA metric retrieval - SigNoz