Metric alerts by incident question¶
When an on-call SRE asks "what's the right alert for incident X?", this page maps the question to the exact metric, aggregation, split dimension, and a starting threshold. The page is a selection aid, not an alert-rule tutorial — for CLI syntax, action groups, and Portal capture of alert blades, see Alerting for Container Apps. For metric definitions, denominators, and dimensions, see the Container Apps metrics reference.
Decision path for incident-question-driven metric alerts:
flowchart TD
Q["Incident question from on-call"] --> M{"Answerable by Container Apps metrics?"}
M -- yes --> MAP["Map question to metric: replicas, execution, http, resource usage"]
MAP --> ALERT["Create metric alert with action group"]
M -- "no — behavioral or trace data" --> LOGS["Use logs or App Insights instead"]
ALERT --> V["Verification: alert fires with threshold and dimensions as intended"]
LOGS --> V
V --> RB["Rollback / Troubleshooting if noisy or silent"] Prerequisites¶
You already know how to create an Azure Monitor metric alert (az monitor metrics alert create) — that lives in Alerting for Container Apps. This page assumes you are looking for the correct metric and threshold, not the CLI shape.
When to Use¶
Use this page in three situations:
- Designing a new metric alert — you have an incident question ("what should page me when CPU saturates?") and need to pick the metric, aggregation, and split before writing the
az monitor metrics alert createcommand. - Evaluating an existing alert's signal-to-noise — your current alert fires too often or never fires, and you want to confirm you chose the right metric and aggregation, or check whether a metric alert is even the right vehicle.
- Incident triage — an alert just fired and you want to confirm what the metric actually measures, what its denominator is, and whether a companion signal (e.g.,
RestartCountconfirmingMemoryPercentage) should also have fired.
Skip this page if you need: CLI syntax (see Alerting for Container Apps), threshold-tuning methodology (see Recommended Threshold Baseline), or metric definitions (see the metrics reference).
Procedure¶
- Scan the Incident question column for the symptom you want to alert on.
- Check the Use when / Preconditions column — many rows are only valid under specific workload conditions (always-on apps, profile-pinned envs, attached resiliency policies, etc.). Skip rows whose preconditions do not match your app.
- Click the metric link to confirm the exact dimension names, denominators, and observed-data sample in the Metrics reference.
- Use the starting threshold as a first draft, then tune from 2-4 weeks of baseline data per the methodology in Alerting for Container Apps → Recommended Threshold Baseline.
- Write the alert using the CLI shape documented in Azure Monitor alert rules.
Incident question to metric mapping¶
Dimension display name vs API filter key
The Portal Metrics blade chip says "Replica", but the API filter key (and the value you pass to az monitor metrics list --filter) is podName. Calling the API with replicaName eq '*' returns BadRequest. The Split by column below uses the API key. See Dimension display name vs API filter name for the full per-metric matrix.
| Incident question | Use when / Preconditions | Metric or signal | Aggregation | Split by | Starting threshold | Severity | Notes |
|---|---|---|---|---|---|---|---|
| CPU saturated on a replica? | App is provisioned with --cpu and is CPU-bound (web, encoders, ML inference, etc.) | CpuPercentage | Avg, Max | podName | > 85% Avg for 10m | sev2 | Preview metric. Denominator is the configured --cpu value, not host capacity. CpuPercentage supports podName only — revisionName returns BadRequest. |
| Memory pressure / OOM risk? | App is provisioned with --memory | MemoryPercentage + WorkingSetBytes | Avg | podName | > 85% Avg for 10m | sev2 | Preview metric. Pair with the next row as a confirming companion alert. MemoryPercentage supports podName only. |
| OOM kills are recurring? | A memory alert has fired, or you want an independent confirming signal | RestartCount | Total | podName | > 0 over 5m, repeating | sev1 | Confirms the memory pressure converted into kills. Use podName to find the failing replica; add a second alert split by revisionName if you also want rollout attribution. |
| 5xx errors are climbing? | App has external ingress; you want platform-metric alerting (no ratio math) | Requests | Total | statusCodeCategory | > N requests / 5m where statusCodeCategory=5xx (tune per app) | sev2 | Azure Monitor metric alerts do not natively compute "5xx as percent of total". For true error-rate / SLO alerting see Use logs or App Insights instead below. |
| Average latency degraded? | You accept an Average latency signal (not percentile) | ResponseTime | Avg | statusCodeCategory=2xx (so 5xx fast-fail does not mask 2xx slowdown) | > 1500ms for 10m | sev2 | Preview metric. Average only — for p95/p99 use App Insights percentile alerts (see below). |
| Replicas dropped to zero (always-on app)? | min-replicas > 0 is the intentional design AND the app must stay warm (low-latency external ingress, etc.) | Replicas | Min | revisionName | == 0 for 2m | sev1 | Skip this row entirely if min-replicas=0 is intentional (event-driven, idle-tolerant, batch). |
| Replicas pinned at max (capacity saturation)? | Paired with rising Requests, ResponseTime, or CPU/memory saturation — alone it can be legitimate load | Replicas AND a saturation signal | Max | revisionName | Replicas == max-replicas for 15m | sev3 alone, sev2 if paired | A single-metric alert on Replicas at max produces false positives during legitimate load spikes; the pairing matters. |
| Retry storm (intra-env)? | You have a retry resiliency policy attached | ResiliencyRequestRetries | Total | revisionName (or none) | Large delta over 7-day rolling baseline | sev2 | Pairs with ResiliencyEjectionsAborted (circuit breaker engaged). Tune from observed baseline, not absolute count. |
| Connection pool exhausted (downstream slow)? | You have a pool policy (http1MaxPendingRequests) attached | ResiliencyRequestsPendingConnectionPool | Maximum | revisionName | > N pending (tune from baseline) for 5m | sev2 | This is a queue-depth gauge; use Maximum to catch saturation spikes. Avg can hide short bursts. |
| Per-app vCPU reservation approaching its scaling ceiling? | App uses HTTP scaler or KEDA with max-replicas set, AND reserved cores matter for cost/quota planning | TotalCoresQuotaUsed | Max | (none — this metric has no dimensions) | > 80% of max-replicas × --cpu for 5m | sev3 | Per-container-app metric, not per-environment. For environment-level quota saturation see Rollback / Troubleshooting. |
| Workload-profile node fleet undersized? | Env uses workload profiles (not Consumption-only) AND the profile has min-nodes < max-nodes AND apps are sensitive to scheduling delays | NodeCount | Min | workloadProfileName | < expected baseline for 5m | sev2 | Preview metric. Does not apply to Consumption-only environments. Pair with replica-scheduling-failure log queries. |
| Which KEDA scaler caused the last replica change? | You need diagnostic attribution after a scale event, NOT proactive alerting | Not a metric alert — use system logs + KQL | N/A | N/A | N/A | N/A | See KEDA scaler observability and the KEDA scaler metrics KQL pack. The Metrics blade alone cannot attribute scale events to specific scalers — Replicas only splits by revisionName. |
Use logs or App Insights instead¶
Azure Monitor metric alerts on Container Apps platform metrics are the fastest signal (1-2 minute evaluation), but they have hard limits. Use these alternatives when the metric-alert column above cannot express what you need.
| Need | Why metric alerts fall short | Use instead |
|---|---|---|
| 5xx as percent of total requests (SLO-style) | Requests metric alerts threshold absolute counts; no native ratio math between two metric splits | Azure Monitor log alert against the Application Insights requests table (requests \| summarize errorRate=...) |
| p95 / p99 latency | ResponseTime is Average only | Azure Monitor log alert against the Application Insights requests table (requests \| summarize percentile(duration, 95) ...) |
| Semantic error patterns from app stdout (e.g., specific exception class, retry-after-503) | Platform metrics have no log content | Log search alert on ContainerAppConsoleLogs |
| Environment-wide quota saturation | TotalCoresQuotaUsed is per-container-app; no env-wide quota metric exists | az containerapp env list-usages poll + threshold logic in a Logic App or scheduled job |
| KEDA scaler attribution for a specific replica change | Replicas metric does not carry scaler/trigger identity | KEDA scaler observability + KQL pack |
Verification¶
Before you commit a new metric-alert rule to production, confirm the metric is actually being emitted for your specific app and that your aggregation produces non-empty values over your window.
-
Confirm the metric is published for your resource (catches metric-typo and preview-metric-not-yet-available errors):
az monitor metrics list-definitions \ --resource "/subscriptions/<subscription-id>/resourceGroups/$RG/providers/Microsoft.App/containerApps/$APP_NAME" \ --query "[?name.value=='CpuPercentage']"Flag Purpose --resourceFull Azure resource ID of the target container app; metrics list-definitionsis resource-scoped, so the shorter--nameform is not accepted here.--queryJMESPath filter on the response to verify the specific metric ID ( CpuPercentage) is actually published for this resource and not just listed in Azure-wide reference docs. -
Dry-run the aggregation over a recent window to confirm non-empty data and a realistic baseline. The
--offsetflag accepts portable##d##hnotation (default1h) and avoids OS-specificdatesubstitutions:az monitor metrics list \ --resource "/subscriptions/<subscription-id>/resourceGroups/$RG/providers/Microsoft.App/containerApps/$APP_NAME" \ --metric "CpuPercentage" \ --aggregation Average \ --interval PT5M \ --offset 2hFlag Purpose --resourceTargets the exact Container App resource whose metric stream you plan to alert on, so the dry run reflects the same scope the future alert rule will evaluate. --metric "CpuPercentage"Chooses the specific metric signal under review, which lets the operator confirm that this incident question maps to a real Container Apps metric instead of a guessed name. --aggregation AverageTests the same rollup style an alert would use, which matters because averages can look healthy even when single spikes would have triggered a Maximum-based rule.--interval PT5MBuckets the series into 5-minute windows so you can judge whether the alerting granularity is too coarse or too noisy for incident response. --offset 2hPulls a recent two-hour sample window, which is long enough to confirm the metric is emitting and to estimate a realistic threshold before you create the alert. -
After creating the rule, confirm it is firing as expected against historical data by reviewing the rule's Fired alerts tab in the Portal, per Alerting for Container Apps → Portal view.
Rollback / Troubleshooting¶
If your alert misfires (too noisy or silent), the rollback is usually not to disable the rule but to retune the threshold or change the metric. Use this decision order:
- Alert fires too often (false positives) → Re-check the Use when / Preconditions column for your row. Common cause: alerting
Replicas == max-replicaswithout a saturation-signal pairing produces false positives during legitimate load spikes. Adjust by adding the paired signal or by widening the window. - Alert never fires when you expect it to → Confirm the metric is published for your app via the
list-definitionscommand in the Verification section above. Preview metrics (CpuPercentage,MemoryPercentage,ResponseTime,NodeCount, etc.) may not be available in all regions/SKUs. - You picked a metric alert but actually need ratio math or log content → Switch to the Use logs or App Insights instead table above and migrate the rule to a log-based or Application Insights alert.
- Environment-wide quota saturation →
TotalCoresQuotaUsedis per-app, not per-environment. For env-wide quota useaz containerapp env list-usagesin a scheduled job; there is no metric alert for this dimension. - Threshold tuning methodology (collecting 2-4 weeks of baseline before locking thresholds) → Alerting for Container Apps → Recommended Threshold Baseline.
To disable a rule temporarily without deleting it:
| Flag | Purpose |
|---|---|
--name | The exact name of the metric alert rule to update (case-sensitive). |
--resource-group | The resource group that owns the alert rule (not the resource group of the monitored container app). |
--enabled false | Disables evaluation without deleting the rule. Re-enable with --enabled true to resume; severity, conditions, and action groups are preserved. |
What this page does not cover¶
This page is scoped to metric-alert selection. The following are owned by other pages — do not duplicate the configuration here:
- Metric alert CLI syntax (
az monitor metrics alert create), action groups, notification channels, severity-to-channel routing → Alerting for Container Apps. - Log-based and Activity-log alerts (the full KQL alert lifecycle) → Alerting for Container Apps → Log-Based Alerts with KQL and Activity Log Alert Example.
- Portal screenshots of the Alerts blade, Alert rules blade, and Alert rule detail → Alerting for Container Apps → Portal view sections.
- Metric definitions (what each metric measures, denominators, dimensions, observed sample data, capture screenshots) → Container Apps metrics reference.
See Also¶
- Alerting for Container Apps — alert-rule CLI, action groups, baseline thresholds, Portal screenshots
- Container Apps metrics reference — metric IDs, aggregations, dimensions, denominators, observed sample data
- KEDA scaler observability — system-logs-only attribution for scale events
- KEDA scaler metrics (KQL pack)
- Scaling events (KQL pack)
- Observability Operations (monitoring index)
- Recovery and Incident Readiness