Evidence Map for App Service Troubleshooting¶

This page maps common investigation questions to the best evidence source, the CLI command to run, and the KQL table/query to use.

Use it when you know what you need to answer but not where to collect proof.

flowchart TD
    Q[Investigation Question] --> S[Evidence Source]
    S --> CLI[CLI Command]
    S --> KQL[KQL Query]
    CLI --> P[Proof Artifact]
    KQL --> P
    P --> H[Hypothesis Validation]

Why an evidence map¶

During incidents, teams lose time by checking the wrong signal first.

• HTTP errors are checked in CPU charts
• startup issues are checked only in app code
• DNS/SNAT issues are diagnosed without outbound evidence

An evidence map reduces this by pairing each question with a reproducible command and query.

Set the blade-level Time range to match each query's | where TimeGenerated > ago(...) clause, otherwise a stale time filter returns empty results. All queries on this page assume KQL mode rather than Simple mode, and the Query history pane is worth preserving in the incident ticket alongside CLI output.

Quick Map (Question → Source → Command → Table)¶

Detailed Evidence Recipes¶

1) Was the app restarting?¶

CLI¶

az monitor activity-log list --resource-group <resource-group> --offset 24h

KQL¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where ResultDescription has_any ("restart", "recycle", "container", "stopped", "started")
| project TimeGenerated, OperationName, ResultDescription, Host
| order by TimeGenerated desc

2) Were requests failing?¶

CLI¶

az monitor metrics list --resource <app-resource-id> --metric "Http5xx,Requests" --interval PT1M

KQL¶

AppServiceHTTPLogs
| where TimeGenerated > ago(6h)
| summarize total=count(), err5xx=countif(ScStatus >= 500 and ScStatus < 600) by bin(TimeGenerated, 5m)
| extend errPct=todouble(err5xx)*100.0/iif(total==0,1,total)
| order by TimeGenerated asc

3) Was startup failing?¶

Log stream can reveal startup errors in real time, but an empty stream is not a negative observation about the process. [Observed] Record whether the selected instance and enabled log type produced lines. [Inferred] Matching tracebacks or bind errors can prioritize a crash or listener hypothesis. [Not Proven] No lines do not show that startup failed before logging initialized. Verify log configuration, application output destination, query window, collection delay, selected instance, and stream health; then compare platform events with process/listener state. If those checks or the telemetry source are unavailable, record INCONCLUSIVE.

CLI¶

az webapp log tail --resource-group <resource-group> --name <app-name>

KQL¶

AppServiceConsoleLogs
| where TimeGenerated > ago(6h)
| where ResultDescription has_any ("failed to start", "could not bind", "listen", "startup", "didn't respond")
| project TimeGenerated, ResultDescription, Host
| order by TimeGenerated desc

4) Was a dependency slow?¶

Portal view: Application Insights Overview dashboard¶

Purpose: Show that request telemetry is reaching Application Insights.

Look for: The Failed requests, Server response time, and Server requests charts.

Expected result: Charts with data confirm the app is sending telemetry; charts that stay empty after traffic point to instrumentation or connection-string problems.

The Application Insights Overview can establish that response time changed, but a value above baseline does not identify downstream latency. Treat dependency slowness as a hypothesis until same-window AppDependencies duration and failure data isolate the affected target and operation; compare those rows with request timing and worker pressure. The Logs button in the command bar opens the KQL editor used for that correlation, while Application Dashboard provides a curated dependency view. Use the incident window first, then widen it only to test whether the pattern recurs. Logs workspace identifies a workspace-based Application Insights resource where the AppDependencies and AppRequests tables can be queried.

CLI¶

az monitor metrics list --resource <app-resource-id> --metric "AverageResponseTime" --interval PT1M

KQL¶

AppServiceAppLogs
| where TimeGenerated > ago(6h)
| where ResultDescription has_any ("timeout", "dependency", "upstream", "database", "redis", "key vault")
| summarize hits=count() by bin(TimeGenerated, 5m)
| order by TimeGenerated asc

5) Was DNS failing?¶

CLI¶

az webapp ssh --resource-group <resource-group> --name <app-name>

Inside the session, run:

nslookup <dependency-hostname>

KQL¶

AppServiceConsoleLogs
| where TimeGenerated > ago(6h)
| where ResultDescription has_any ("Name or service not known", "Temporary failure in name resolution", "getaddrinfo", "DNS")
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

6) Was scale involved?¶

Portal view: Metrics blade empty state with metric selector¶

Purpose: Show the Metrics starting point before a metric is chosen.

Look for: Scope, Metric Namespace, Metric, and Aggregation.

Expected result: The chart stays empty until you select a metric and an aggregation.

The Metrics blade is the visual companion to the az monitor metrics list CLI command below. The required fields are the same: Scope (the App Service resource), Metric Namespace (App Service standard metrics), Metric (for example, CpuPercentage, MemoryWorkingSet, or Http5xx), and Aggregation (Avg, Max, Sum, or Count). A CPU spike before a request drop makes capacity or scaling delay a hypothesis, not a conclusion. Test it against autoscale events, instance-count and configured-capacity changes, queueing or request-start delay, and per-instance utilization in the same window. Use Apply splitting by Instance to determine whether pressure is broad or isolated, and use Drill into Logs to correlate the chart with platform events and request evidence.

CLI¶

az monitor metrics list --resource <app-resource-id> --metric "CpuPercentage,MemoryWorkingSet,Http5xx,AverageResponseTime" --interval PT1M

KQL¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where ResultDescription has_any ("scale", "instance", "restart", "recycle")
| project TimeGenerated, OperationName, ResultDescription
| order by TimeGenerated asc

7) Was disk full?¶

CLI¶

az webapp log tail --resource-group <resource-group> --name <app-name>

Use SSH to confirm with:

df -h

KQL¶

AppServiceConsoleLogs
| where TimeGenerated > ago(24h)
| where ResultDescription has_any ("No space left on device", "ENOSPC", "disk full")
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

8) Was memory exhausted?¶

CLI¶

az monitor metrics list --resource <app-resource-id> --metric "MemoryWorkingSet" --interval PT1M

KQL¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where ResultDescription has_any ("OOM", "killed", "memory", "SIGKILL")
| project TimeGenerated, OperationName, ResultDescription, Host
| order by TimeGenerated desc

9) Was SNAT exhausted?¶

Inbound 499/503 status and repeated duration boundaries are routing clues, not SNAT measurements. Compare client cancellation timestamps, worker queue depth or saturation, per-target dependency failures, and the SNAT Port Exhaustion detector for the same instance and window. [Inferred] SNAT pressure becomes the leading hypothesis only when outbound failures align with measured detector pressure. [Not Proven] If the detector or dependency telemetry is unavailable, keep the result Unknown; do not infer exhaustion from HTTP timing alone.

CLI¶

az monitor metrics list --resource <app-resource-id> --metric "Http5xx,Requests,AverageResponseTime" --interval PT1M

KQL¶

AppServiceConsoleLogs
| where TimeGenerated > ago(6h)
| where ResultDescription has_any ("connect timed out", "ReadTimeout", "ConnectTimeout", "socket", "ECONNRESET")
| summarize errors=count() by bin(TimeGenerated, 5m)
| order by TimeGenerated asc

10) Was there a deployment?¶

Portal view: Activity log blade for the Web App¶

Purpose: Show where control-plane operations on the app are audited.

Look for: The Timespan and Resource filters and the Operation name, Status, and Event initiated by columns.

Expected result: Each management operation appears with its status and who started it; widen the timespan to cover the incident window.

The Activity log is the authoritative record of every control-plane operation against this Web App — deployments, configuration changes, slot swaps, and scale operations all appear as discrete Operation name entries with the initiating principal. For "was there a deployment?" the operations to grep are Microsoft.Web/sites/publish/Action, Update Site, Update Configuration, and Update App Settings. The Event initiated by column distinguishes user-driven deployments from automated CI/CD service principals — critical for separating intentional deploys from third-party automation. The Timespan: Last 6 hours chip should match the incident window; the Add Filter button lets you scope further by operation name. Use Export Activity Logs to set up continuous export to Log Analytics so the same data is available as the KQL query below.

CLI¶

az monitor activity-log list --resource-group <resource-group> --offset 24h --status Succeeded

KQL¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where OperationName has_any ("Deploy", "Publish", "SiteConfig", "Container")
| project TimeGenerated, OperationName, ResultDescription
| order by TimeGenerated desc

11) Was health check failing?¶

CLI¶

az webapp show --resource-group <resource-group> --name <app-name>

KQL¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where ResultDescription has_any ("health check", "unhealthy", "warmup", "probe")
| project TimeGenerated, OperationName, ResultDescription, Host
| order by TimeGenerated desc

12) Was slot swap involved?¶

CLI¶

az webapp deployment slot list --resource-group <resource-group> --name <app-name>

KQL¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where ResultDescription has_any ("swap", "slot", "warm-up", "warmed up")
| project TimeGenerated, OperationName, ResultDescription
| order by TimeGenerated desc

13) Was there a config change?¶

CLI¶

az webapp config appsettings list --resource-group <resource-group> --name <app-name>
az monitor activity-log list --resource-group <resource-group> --offset 24h

KQL¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where OperationName has_any ("Update Site", "Update App Settings", "Update Configuration")
| project TimeGenerated, OperationName, ResultDescription
| order by TimeGenerated desc

14) Was the container killed?¶

CLI¶

az webapp log tail --resource-group <resource-group> --name <app-name>

KQL¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where ResultDescription has_any ("killed", "SIGKILL", "exit code", "container stopped", "OOM")
| project TimeGenerated, OperationName, ResultDescription, Host
| order by TimeGenerated desc

15) Were there network errors?¶

CLI¶

az webapp log tail --resource-group <resource-group> --name <app-name>

KQL¶

AppServiceConsoleLogs
| where TimeGenerated > ago(6h)
| where ResultDescription has_any ("connection refused", "connection reset", "ENETUNREACH", "EHOSTUNREACH", "timed out")
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

16) Did warm-up fail during slot swap?¶

CLI¶

az monitor activity-log list --resource-group <resource-group> --offset 6h

KQL¶

AppServicePlatformLogs
| where TimeGenerated > ago(6h)
| where ResultDescription has_any ("swap", "warm-up", "did not respond", "health check")
| project TimeGenerated, OperationName, ResultDescription
| order by TimeGenerated desc

Evidence Quality Checklist¶

• Keep all evidence in one incident time window.
• Correlate HTTP, console, and platform signals before selecting a root cause.
• Preserve query text used during the incident for post-incident review.
• Capture command outputs in ticket notes with sensitive identifiers removed.

See Also¶

• Troubleshooting Method
• Detector Map
• Architecture Overview
• Decision Tree
• Troubleshooting Mental Model
• 5xx Trend Over Time
• Restart Timing Correlation
• Startup Errors

Sources¶

• Monitor Azure App Service
• Enable diagnostic logging for apps in Azure App Service
• Azure Monitor metrics overview
• Azure Activity Log overview
• Troubleshoot intermittent outbound connection errors in Azure App Service
• App Service diagnostics overview