Skip to content

Evidence, Captures, and Lab Environment

How these numbers were produced

The numeric samples above came from a deliberate load test environment to demonstrate that each metric publishes the expected shape of data. The topology was:

Reproduce these captures

See labs/metrics-load-test/README.md for the full reproduction guide — provisioning script, load generator, dual-environment topology diagram, metric → app mapping, prerequisites, estimated cost, capture-refresh checklist, and known limitations.

Evidence baseline status

The lab now ships a committed Option C special-case evidence baseline at labs/metrics-load-test/evidence/README.md. It is intentionally not a standard Phase B falsification pack: no gate JSONs, no verify.sh, and no H1/H2/falsification claim. Repository framing: 27/28 falsification labs + 1 metrics evidence baseline = 28 total.

App Role Configuration Drives metric(s)
ca-loadtest-d38538 Public ingress test target (in cae-basics-d38538) --cpu 0.5 --memory 1Gi --min-replicas 2 --max-replicas 10, HTTP scaler at 20 concurrency UsageNanoCores, WorkingSetBytes, RxBytes, TxBytes, Replicas, Requests, ResponseTime, CpuPercentage, MemoryPercentage, CoresQuotaUsed, TotalCoresQuotaUsed
ca-crashloop-d38538 Memory-leak loop, no ingress --cpu 0.25 --memory 0.5Gi, container allocates 16 MiB every 200 ms Drove RestartCount after the process exceeded its memory limit and the kernel OOM-killed it
ca-res-503 Resiliency target serving constant 503 2 replicas, policy-503 with retries on 503/500 and outlier detection ResiliencyRequestRetries, ResiliencyEjectedHosts, ResiliencyEjectionsAborted
ca-res-slow Resiliency target with 4-second response policy-slow with timeoutPolicy.responseTimeoutInSeconds: 1 (per-request response budget of 1 second) ResiliencyRequestTimeouts
ca-res-pool Resiliency target with tight pool policy-pool with http1MaxPendingRequests: 1 ResiliencyRequestsPendingConnectionPool
ca-res-blackhole TCP listen without accept TCP transport, no HTTP server Negative-example target — produces baseline-zero captures for ResiliencyConnectTimeouts and ResiliencyEjectionsAborted (see the "Why ResiliencyConnectTimeouts did not move" admonition below)
ca-res-caller Intra-environment caller 100 threads x 5 targets via service discovery (http://target-name) Drives traffic into the four resiliency targets above
cae-wp-d38538 Dedicated workload profile environment (separate from the load-test env) D4 profile pinned at min-nodes=2, max-nodes=2, one app (ca-node-anchor) deployed with internal ingress NodeCount, baseline IngressUsageNanoCores, IngressUsageBytes, IngressCpuPercentage, IngressMemoryPercentage

Load was generated by 5 parallel hey processes against the public endpoint of ca-loadtest-d38538 (one each hitting /health, /cpu, /payload, /error, /slow) and by ca-res-caller issuing intra-environment calls to the four resiliency targets on port 80 (Container Apps service discovery routes hostname-only URLs through the env-internal ingress proxy on port 80, regardless of the target's --target-port).

Why ResiliencyConnectTimeouts did not move

ca-res-blackhole calls socket.listen() without accept(), so the kernel's SYN backlog completes the TCP handshake on its own. Envoy's connection to the upstream succeeds; the subsequent HTTP request hang shows up as ResiliencyRequestsPendingConnectionPool and ResiliencyRequestTimeouts, not as a connect-phase timeout. Triggering ResiliencyConnectTimeouts reliably requires dropping SYN packets at L3 (a firewall DROP rule) which a Container Apps replica cannot do without the NET_ADMIN capability. In production, a non-zero ResiliencyConnectTimeouts typically means the destination replica died and the data plane has not yet removed it from the endpoint slice.

Query metrics with az CLI

az monitor metrics list returns the same data as the Portal Metrics blade. Use it for scripted dashboards, CI checks, or alert validation.

CPU usage percentage

# CPU usage percentage, last 1 hour, 5-minute granularity
az monitor metrics list \
    --resource "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG/providers/Microsoft.App/containerApps/$APP_NAME" \
    --metric CpuPercentage \
    --aggregation Average \
    --interval PT5M \
    --output table
Command flag What it does
--resource Full Azure resource ID for the Container App or Container Apps Environment
--metric Metric ID from the metrics tables (case sensitive)
--aggregation One of Average, Minimum, Maximum, Total, Count (must be supported by the metric)
--interval ISO 8601 duration for the aggregation bucket (for example PT1M, PT5M, PT1H)
--filter Dimension filter, for example podName eq '*' to split by replica (use podName, not replicaName -- see the dimension mapping)
--output Output format such as table, json, or tsv

Memory working set bytes split by replica

# Memory working set bytes split by replica (pod)
az monitor metrics list \
    --resource "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG/providers/Microsoft.App/containerApps/$APP_NAME" \
    --metric WorkingSetBytes \
    --aggregation Average \
    --interval PT1M \
    --filter "podName eq '*'" \
    --output table
Command flag What it does
--metric WorkingSetBytes Absolute memory numerator
--interval PT1M 1-minute aggregation buckets for higher resolution
--filter "podName eq '*'" Splits the series across every replica reported in the interval. The actual API dimension key is podName, not replicaName -- passing "replicaName eq '*'" returns BadRequest. See the dimension mapping.

HTTP request count split by status code category

# HTTP request count split by status code category, last 24 hours
az monitor metrics list \
    --resource "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG/providers/Microsoft.App/containerApps/$APP_NAME" \
    --metric Requests \
    --aggregation Total \
    --interval PT15M \
    --filter "statusCodeCategory eq '*'" \
    --output table
Command flag What it does
--metric Requests HTTP request counter
--aggregation Total Sums requests inside each interval rather than averaging
--filter "statusCodeCategory eq '*'" Splits the series across 2xx, 3xx, 4xx, 5xx categories

Environment-level node count

# Environment-level node count
az monitor metrics list \
    --resource "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG/providers/Microsoft.App/managedEnvironments/$ACA_ENV_NAME" \
    --metric NodeCount \
    --aggregation Average \
    --interval PT5M \
    --output table
Command flag What it does
--resource Full Azure resource ID for the Container Apps Environment in this query
--metric Environment metric ID from the environment metrics table
--aggregation Average returns the per-interval mean node count
--interval PT5M aligns the bucket with the default Portal Metrics blade granularity

See Also

Sources