Performance Degradation¶

1. Summary¶

This playbook applies when an Azure App Service app becomes slow, latency rises, throughput drops, or CPU and memory behavior degrade under load or after a release. Use it when the site still responds but users experience timeouts, slow APIs, queueing, or intermittent 5xx errors tied to resource pressure.

Symptoms¶

• Requests that normally finish in milliseconds now take seconds.
• P95 or P99 latency spikes while request volume stays similar.
• CPU or memory rises and the app becomes unstable or restarts.
• The app serves intermittent 500, 502, 503, or 504 responses during busy periods.

Common error messages¶

• The request timed out.
• 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout.
• OutOfMemory, worker recycle, or repeated restart messages.
• SNAT exhaustion-style dependency failures that look like app slowness.

flowchart TD
    A[Performance degradation reported] --> B{Is latency high for all routes?}
    B -->|No| H1[H1: One endpoint or dependency regressed]
    B -->|Yes| C{Is CPU saturated?}
    C -->|Yes| H2[H2: CPU bottleneck or insufficient scale]
    C -->|No| D{Is memory growing or workers restarting?}
    D -->|Yes| H3[H3: Memory pressure or garbage-collection stress]
    D -->|No| E{Are dependencies failing or slow?}
    E -->|Yes| H4[H4: Downstream dependency or outbound issue]
    E -->|No| F[Check cold start, disk pressure, and recent release changes]

2. Common Misreadings¶

3. Competing Hypotheses¶

4. What to Check First¶

1. Confirm app state and plan relationship

   az webapp show \
       --resource-group $RG \
       --name $APP_NAME \
       --query "{state:state,serverFarmId:serverFarmId,enabled:enabled}" \
       --output json
   

   Command	Purpose
   az webapp show --resource-group $RG --name $APP_NAME --query "{state:state,serverFarmId:serverFarmId,enabled:enabled}" --output json	Shows the app state, enabled flag, and hosting-plan resource ID so you can confirm the app is attached to the expected plan.
   --resource-group $RG	Selects the resource group that contains the target web app or related resource.
   --name $APP_NAME	Specifies the target web app name for this command.
   --query "{state:state,serverFarmId:serverFarmId,enabled:enabled}"	Projects only these named top-level properties into a smaller object before formatting the output.
   --output json	Formats the command output as JSON for full-fidelity inspection.

2. Inspect autoscale or instance count context if applicable

   az appservice plan show \
       --resource-group $RG \
       --name $PLAN_NAME \
       --query "{sku:sku.name,workerSize:workerSize,numberOfWorkers:numberOfWorkers,maximumElasticWorkerCount:maximumElasticWorkerCount}" \
       --output json
   

   Command	Purpose
   az appservice plan show --resource-group $RG --name $PLAN_NAME --query "{sku:sku.name,workerSize:workerSize,numberOfWorkers:numberOfWorkers,maximumElasticWorkerCount:maximumElasticWorkerCount}" --output json	Shows the App Service plan capacity and worker settings so you can judge scale context.
   --resource-group $RG	Selects the resource group that contains the App Service plan you are inspecting.
   --name $PLAN_NAME	Specifies the App Service plan name to inspect.
   --query "{sku:sku.name,workerSize:workerSize,numberOfWorkers:numberOfWorkers,maximumElasticWorkerCount:maximumElasticWorkerCount}"	Projects the selected plan fields, including nested sku.name, into a smaller object before formatting the output.
   --output json	Formats the command output as JSON for full-fidelity inspection.

3. Review startup and runtime settings that influence performance behavior

   az webapp config show \
       --resource-group $RG \
       --name $APP_NAME \
       --query "{alwaysOn:alwaysOn,http20Enabled:http20Enabled,minimumTlsVersion:minTlsVersion,appCommandLine:appCommandLine}" \
       --output json
   

   Command	Purpose
   az webapp config show --resource-group $RG --name $APP_NAME --query "{alwaysOn:alwaysOn,http20Enabled:http20Enabled,minimumTlsVersion:minTlsVersion,appCommandLine:appCommandLine}" --output json	Shows the effective site configuration so you can inspect the runtime or startup-related properties.
   --resource-group $RG	Selects the resource group that contains the target web app or related resource.
   --name $APP_NAME	Specifies the target web app name for this command.
   --query "{alwaysOn:alwaysOn,http20Enabled:http20Enabled,minimumTlsVersion:minTlsVersion,appCommandLine:appCommandLine}"	Projects only these named top-level properties into a smaller object before formatting the output.
   --output json	Formats the command output as JSON for full-fidelity inspection.

4. If a release just occurred, inspect recent deployment history

   az webapp log deployment list \
       --resource-group $RG \
       --name $APP_NAME \
       --output table
   

   Command	Purpose
   az webapp log deployment list --resource-group $RG --name $APP_NAME --output table	Lists deployment history records so you can verify whether the latest deployment completed.
   --resource-group $RG	Selects the resource group that contains the target web app or related resource.
   --name $APP_NAME	Specifies the target web app name for this command.
   --output table	Formats the command output as a readable table for quick triage.

5. Evidence to Collect¶

Use one shared incident window across HTTP logs, console/platform logs, and Application Insights requests or dependencies if available.

5.1 KQL Queries¶

Query 1: Request latency and error trend¶

AppServiceHTTPLogs
| where TimeGenerated > ago(24h)
| summarize Requests=count(), Errors=countif(ScStatus >= 500), P95=percentile(TimeTaken, 95), P99=percentile(TimeTaken, 99) by bin(TimeGenerated, 5m)
| order by TimeGenerated asc

Query 2: Slowest request paths¶

AppServiceHTTPLogs
| where TimeGenerated > ago(24h)
| summarize Requests=count(), P95=percentile(TimeTaken, 95), MaxTime=max(TimeTaken) by CsUriStem
| order by P95 desc
| take 15

Query 3: Restart and pressure correlation¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where Message has_any ("Restarting", "ContainerTimeout", "OutOfMemory", "Stopping container", "recycle")
| project TimeGenerated, Level, Message
| order by TimeGenerated asc

Query 4: Application Insights dependency latency (if enabled)¶

dependencies
| where timestamp > ago(24h)
| summarize Calls=count(), Failures=countif(success == false), P95=percentile(duration, 95) by target, type, bin(timestamp, 5m)
| order by timestamp asc

Application Insights is the most efficient surface for the dependency hypothesis because it correlates request latency with dependency duration in a single trace. A high failure count with low server response time points to fast-failing dependencies, while a normal failure count with high server response time points to local processing. Click Logs to reach the query editor where the dependencies table lives, or open Application Map under Investigate for a visual dependency graph that surfaces failing downstream targets without writing KQL.

5.2 CLI Investigation¶

# Check plan capacity context
az appservice plan show \
    --resource-group $RG \
    --name $PLAN_NAME \
    --query "{sku:sku.name,numberOfWorkers:numberOfWorkers,maximumElasticWorkerCount:maximumElasticWorkerCount}" \
    --output json

Sample output:

{
  "maximumElasticWorkerCount": 1,
  "numberOfWorkers": 2,
  "sku": "P1v3"
}

Interpretation:

• Low worker count during sustained traffic supports H2, but only when logs also show load-driven degradation.
• Scale context alone never proves the app is the bottleneck.

# Show app configuration relevant to warm and steady-state performance
az webapp config show \
    --resource-group $RG \
    --name $APP_NAME \
    --query "{alwaysOn:alwaysOn,http20Enabled:http20Enabled,appCommandLine:appCommandLine}" \
    --output json

Sample output:

{
  "alwaysOn": true,
  "appCommandLine": "gunicorn --workers 4 --bind 0.0.0.0:8000 src.app:app",
  "http20Enabled": true
}

Interpretation:

• This output helps distinguish configuration regressions from capacity issues.
• Compare worker/process configuration to the observed CPU and latency profile.

6. Validation and Disproof by Hypothesis¶

H1: One endpoint, query, or code path regressed¶

Proves if only a small set of URIs dominates P95/P99 latency or error increases.

Disproves if latency is uniform across nearly all routes and no path stands out.

Validation steps:

1. Sort slowest paths by P95 and request count.
2. Compare incident path set to the last known good release.
3. Check whether one dependency or database call aligns only to those endpoints.

H2: CPU saturation or insufficient capacity¶

Proves if latency and errors rise with demand and scale context is tight.

Disproves if CPU remains modest while dependency latency or memory pressure explains the symptom better.

Validation steps:

1. Correlate workload growth with the latency trend.
2. Confirm the app is not simply blocked on external dependencies.
3. Scale temporarily only to test the hypothesis, not as permanent proof.

H3: Memory pressure or worker degradation¶

Proves if restarts, recycling, or out-of-memory signatures align with latency growth.

Disproves if workers stay stable and no pressure signatures appear.

Validation steps:

1. Review platform log restarts during the incident.
2. Compare request latency before and after each restart.
3. Use existing memory-focused playbooks if the pattern is confirmed.

H4: Downstream dependency or outbound issue¶

Proves if dependency latency/failure rises before or during app latency while local CPU remains lower.

Disproves if no dependency trend is visible and local resource pressure dominates.

Validation steps:

1. Check Application Insights dependencies or route-specific downstream errors.
2. If outbound networking is suspected, compare with SNAT/DNS playbooks.
3. Validate whether retries amplify latency.

7. Likely Root Cause Patterns¶

8. Immediate Mitigations¶

1. If a release triggered the issue, consider rollback before broad tuning changes.
2. Reduce pressure by scaling out temporarily if the app is still serving traffic.
3. Disable or rate-limit the worst offender endpoint if one route is dominating the outage.
4. Short-circuit expensive startup or background tasks that run on every worker recycle.
5. If dependency latency is dominant, reduce retries and timeouts that amplify queueing.
6. Re-collect latency evidence after each change and stop once user-facing latency returns to baseline.

9. Prevention¶

Prevention checklist¶

• [ ] Track P50, P95, P99, error rate, and dependency latency for every production app.
• [ ] Load-test hot endpoints before major releases.
• [ ] Keep staging or canary validation for new builds and config changes.
• [ ] Monitor memory and restart patterns, not just average CPU.
• [ ] Maintain clear SLO-based alerts for latency and failure rate.

See Also¶

• Playbooks
• Slow Response but Low CPU
• Memory Pressure and Worker Degradation
• Intermittent 5xx Under Load

Sources¶

• Monitor instances in Azure App Service (Microsoft Learn)
• Enable Application Insights for Azure App Service (Microsoft Learn)
• Troubleshoot performance issues in App Service (Microsoft Learn)