Lab Guides¶
Hands-on troubleshooting labs for Azure Container Apps with deployable infrastructure and scripted failure/recovery flows.
All sample outputs in lab guides are PII-scrubbed and use ca-myapp, cae-myapp, and job-myapp naming.
Looking for a specific symptom or error string?
This page is a browse-oriented catalog — labs listed by importance with difficulty and duration. If you already know the error string, exit code, or symptom keyword (for example ImagePullBackOff, AADSTS70021, exit code 137), use the Lab Finder instead. The finder groups all 54 labs by symptom cluster with the first-evidence signal per row.
Available Labs¶
| Lab | Description | Difficulty | Duration | Guide | Lab Files |
|---|---|---|---|---|---|
| ACR Image Pull Failure | Reproduces ImagePullBackOff from a non-existent image tag, then fixes image publishing/update. | Beginner | 20-30 min | Guide | Directory |
| Revision Failover and Rollback | Deploys a healthy revision, then breaks ingress port on a new revision and restores traffic. | Intermediate | 20-30 min | Guide | Directory |
| Scale Rule Mismatch | Uses unrealistic HTTP scaling thresholds to show non-scaling under load, then corrects KEDA settings. | Intermediate | 25-35 min | Guide | Directory |
| Probe and Port Mismatch | App listens on port 3000 while ingress targets 8000, causing probe failures until target port is fixed. | Beginner | 20-25 min | Guide | Directory |
| Managed Identity Key Vault Failure | App uses managed identity to read Key Vault secret but fails without Key Vault Secrets User role assignment. | Intermediate | 25-35 min | Guide | Directory |
| ACA Secret KV Ref MI Network Path | Reproduces az containerapp secret set --identity ... --key-vault-url ... failing with Unable to get value using Managed identity and a transport-level EOF on login.microsoftonline.com/<tenant>/.well-known/openid-configuration when a UDR routes the workload subnet through Azure Firewall without an Application Rule for the Entra authority FQDNs; falsifies with a reader-generated 17-gate offline verifier and 8 explicit drops (stderr wording, log ingestion latency, retry cadence, component identity, response body shape, token caching, SKU generality, region generality). | Advanced | 60-90 minutes | Guide | Directory |
| Revision Provisioning Failure | Revision fails because container env var references a missing secret; fixed by setting secret and deploying new revision. | Intermediate | 20-30 min | Guide | Directory |
| Ingress Target Port Mismatch | Diagnose and fix ingress failures caused by target port misconfiguration. | Beginner | 15-20 min | Guide | Directory |
| Traffic Routing Canary Failure | Diagnose traffic splitting failures when a bad revision receives production traffic. | Intermediate | 20-30 min | Guide | Directory |
| Dapr Integration | Troubleshoot Dapr sidecar and component configuration issues. | Intermediate | 35-45 min | Guide | Directory |
| Observability and Tracing | Set up OpenTelemetry and Application Insights, troubleshoot missing traces and metrics. | Intermediate | 35-45 min | Guide | Directory |
| CD Reconnect RBAC Conflict | Reproduces AppRbacDeployment: The role assignment already exists after a previous CD disconnect left RBAC role assignments behind. | Intermediate | 25-35 min | Guide | Directory |
| Subnet CIDR Exhaustion | Demonstrates Azure Container Apps environment creation failure when subnet is too small (/29) and resolves by resizing to /27. | Intermediate | 20-30 min | Guide | Inline guide only |
| UDR and NSG Egress Blocked | Shows replica startup failure when required outbound FQDNs are blocked by a UDR/NVA; resolves by allowing required rules. | Advanced | 30-45 min | Guide | Inline guide only |
| Private Endpoint DNS Failure | Reproduces DNS NXDOMAIN when Private DNS Zone is not linked to Container Apps VNet; resolves by adding VNet link. | Intermediate | 25-35 min | Guide | Inline guide only |
| ACR Network Path A — Firewall Allowlist | Reproduces ACR HTTP 403 from networkRuleSet.ipRules allowlist mismatch when Container Apps egresses through Azure Firewall; toggling the firewall PIP entry in ipRules deterministically flips fresh-pull behavior while the already-running revision keeps serving from cached layers. | Advanced | 2-3 hours | Guide | Directory |
| ACR Network Path B — PE Direct | Reproduces ImagePullUnauthorized on fresh pull when the workload VNet → privatelink.azurecr.io zone link is removed; the already-running revision keeps serving from cached layers and the pull recovers when the link is restored. | Advanced | 2-3 hours | Guide | Directory |
| ACR Network Path C — PE with Forced Inspection | Demonstrates the silent inspection-bypass class: pull succeeds in both states, but Azure Firewall application-rule log goes silent when the /32 UDR routes per PE NIC IP are removed (system-injected /32 PE route beats 0.0.0.0/0 UDR by longest-prefix match). | Advanced | 2-3 hours | Guide | Directory |
| ACR Network Path D — Record-Level Zone Authority | Reproduces workload-side NXDOMAIN on the per-region ACR data endpoint when its A record is deleted from the linked privatelink.azurecr.io zone (the linked zone is authoritative for that namespace, so Azure DNS returns NXDOMAIN rather than falling back to public DNS). | Advanced | 2-3 hours | Guide | Directory |
| ACR Network Path E — DNS Forwarder Bypass | Reproduces workload-side public-IP resolution after a custom DNS forwarder (dnsmasq on a B1s VM) upstream is swapped from 168.63.129.16 to 8.8.8.8, bypassing the Azure DNS path that resolves the linked privatelink.azurecr.io zone. | Advanced | 2-3 hours | Guide | Directory |
| Egress IP Change | Documents egress IP shift when environment is recreated and shows how to update downstream firewall allowlists. | Intermediate | 20-30 min | Guide | Inline guide only |
| Custom Domain TLS Renewal | Reproduces managed certificate stuck in Pending when CNAME/asuid TXT records are missing or stale. | Intermediate | 20-30 min | Guide | Inline guide only |
| WebSocket and gRPC Ingress | Demonstrates broken WebSocket connection when session affinity is off; resolves by enabling sticky sessions. | Intermediate | 25-35 min | Guide | Inline guide only |
| Session Affinity Failure | Shows state loss across replicas when sticky sessions are disabled; resolves by enabling ingress affinity. | Intermediate | 20-30 min | Guide | Inline guide only |
| Azure Files Mount Failure | Reproduces SMB mount error when storage account key or share name is wrong; resolves by correcting environment storage config. | Intermediate | 25-35 min | Guide | Inline guide only |
| EmptyDir Disk Full | Shows OOMKill-like restart when ephemeral storage is exhausted by log accumulation; resolves by increasing ephemeralStorage limit. | Intermediate | 20-30 min | Guide | Inline guide only |
| Volume Permission Denied | Reproduces permission denied when container UID does not match volume mount ownership; resolves by setting mountOptions. | Intermediate | 25-35 min | Guide | Inline guide only |
| CPU Throttling | Uses a CPU-intensive workload to trigger throttling; shows metrics and resolves by increasing CPU allocation or scaling out. | Intermediate | 25-35 min | Guide | Inline guide only |
| Memory Leak OOMKilled | Injects a memory leak to trigger OOMKilled restarts; resolves by profiling and patching the leak plus setting memory limits. | Advanced | 35-45 min | Guide | Inline guide only |
| Replica Load Imbalance | Demonstrates uneven replica utilization under steady load; resolves by tuning KEDA scale rules and concurrency. | Advanced | 30-40 min | Guide | Inline guide only |
| Memory Percentage vs KEDA Utilization | Reproduces the Portal-vs-KEDA divergence: high MemoryPercentage chart with no scale-out, isolating HPA ceiling math and cache-inflation effects. | Intermediate | 30-40 min | Guide | Directory |
| Docker Hub Rate Limit | Reproduces toomanyrequests pull error from Docker Hub anonymous pulls; resolves by adding authenticated registry credentials. | Beginner | 15-25 min | Guide | Inline guide only |
| Image Size Startup Delay | Shows cold-start latency from a large image (>1 GB); resolves by multi-stage build and layer caching. | Intermediate | 25-35 min | Guide | Inline guide only |
| Multi-Arch Image Mismatch | Reproduces exec format error when an ARM64-only image is pulled on AMD64 Container Apps host; resolves by building a multi-arch manifest. | Intermediate | 20-30 min | Guide | Inline guide only |
| Log Analytics Ingestion Gap | Demonstrates missing logs when diagnostic settings are not configured; resolves by enabling and linking Log Analytics workspace. | Beginner | 15-25 min | Guide | Inline guide only |
| App Insights Connection String Missing | Shows No telemetry when APPLICATIONINSIGHTS_CONNECTION_STRING env var is absent; resolves by injecting the secret. | Beginner | 15-20 min | Guide | Inline guide only |
| Diagnostic Settings Missing | Reproduces silent Log Analytics ingestion when a Container Apps environment is provisioned without appLogsConfiguration (destination: null); resolves by setting the env-level destination via az containerapp env update --logs-destination log-analytics. | Beginner | 20-25 min | Guide | Directory |
| GitHub Actions OIDC Failure | Reproduces AADSTS70021 when federated credential subject does not match repo/branch; resolves by correcting subject claim. | Intermediate | 25-35 min | Guide | Inline guide only |
| Bicep Deployment Timeout | Shows revision stuck in Provisioning during IaC deploy; resolves by reducing container startup time and tuning probe settings. | Intermediate | 25-35 min | Guide | Inline guide only |
| Revision History Limit | Demonstrates RevisionCountLimitReached (100-revision cap); resolves by deactivating and deleting old revisions. | Beginner | 15-20 min | Guide | Inline guide only |
| Subscription Quota Exceeded | Reproduces QuotaExceeded when core quota is exhausted; resolves by requesting quota increase or moving to another region. | Intermediate | 20-30 min | Guide | Inline guide only |
| Workload Profile Mismatch | Shows cost and performance issues from selecting wrong profile (Consumption vs Dedicated); resolves by switching profile. | Intermediate | 25-35 min | Guide | Inline guide only |
| Min Replicas Cost Surprise | Demonstrates unexpected billing from minReplicas: 2 during off-hours; resolves by setting minReplicas: 0 with cold-start mitigation. | Beginner | 15-20 min | Guide | Inline guide only |
| Scheduled Job Missed | Reproduces missed cron job execution due to UTC timezone mismatch; resolves by correcting cron expression. | Beginner | 15-20 min | Guide | Inline guide only |
| Event Job Storm | Demonstrates queue-backed job storm from low maxExecutions; resolves by tuning KEDA scale rules. | Advanced | 30-40 min | Guide | Inline guide only |
| Dapr State Store Failure | Reproduces Dapr state-store component failure from wrong component name or missing scope; resolves by correcting YAML. | Intermediate | 25-35 min | Guide | Inline guide only |
| Dapr Pub/Sub Failure | Shows messages not delivered when Dapr pub/sub component has wrong topic or missing consumer app scope. | Intermediate | 25-35 min | Guide | Inline guide only |
| EasyAuth Entra ID Failure | Reproduces AADSTS50011 redirect URI mismatch after enabling built-in auth; resolves by updating app registration reply URLs. | Intermediate | 20-30 min | Guide | Inline guide only |
| Multi-Region Failover | Demonstrates traffic failing to shift to secondary region when Front Door health probe is misconfigured. | Advanced | 35-50 min | Guide | Inline guide only |
ACR Network Path Series¶
A focused 5-lab series that reproduces the five distinct network paths a Container App can take to reach Azure Container Registry. See the ACR Network Path Selection platform doc for the conceptual taxonomy that names and orders all five paths.
| Order | Lab | Topology | Failure mode class |
|---|---|---|---|
| 1 | Path A — Firewall Allowlist | Public ACR via Azure Firewall SNAT; ACR networkRuleSet.ipRules allowlist toggling on the firewall public IP | Pull-fails (HTTP 403 from ACR firewall; the DENIED message names the firewall PIP as the rejected source) |
| 2 | Path B — PE Direct | ACR Premium PE in workload VNet + privatelink.azurecr.io linked DNS zone; no firewall on the image-data path | Pull-fails on fresh pull (ImagePullUnauthorized after the VNet → zone link is removed) |
| 3 | Path C — PE with Forced Inspection | ACR Premium PE + 0.0.0.0/0 UDR to Azure Firewall + explicit /32 UDR routes per PE NIC IP | Silent inspection bypass (pull succeeds in both states; the firewall log goes silent when the /32 routes are missing) |
| 4 | Path D — Record-Level Zone Authority | ACR Premium PE + default Azure DNS + linked privatelink.azurecr.io zone with a missing per-region data record | Workload-side NXDOMAIN on the data FQDN (the linked zone is authoritative for that namespace) |
| 5 | Path E — DNS Forwarder Bypass | ACR Premium PE + custom VNet DNS server (dnsmasq on a B1s VM) with the upstream swapped from 168.63.129.16 to 8.8.8.8 | Workload-side public-IP resolution after the upstream swap (the linked zone is bypassed entirely) |
Each lab is independently runnable and documents its specific topology, auth mode (ACR admin credentials for A and C; managed identity for B, D, E), and reproduction date in its own Lab position table and Observed in this lab admonition. Labs A and C cleanly prove fresh-pull behavior because admin credentials remove the control-plane token-exchange confound that the managed-identity labs (B, D, E) call out as [Not Proven] for the broken-window fresh pull.
Suggested Learning Path¶
- ACR Image Pull Failure
- Probe and Port Mismatch
- Revision Failover and Rollback
- Revision Provisioning Failure
- Scale Rule Mismatch
- Managed Identity Key Vault Failure
- Ingress Target Port Mismatch Lab
- Traffic Routing and Canary Failure Lab
- Dapr Integration
- Observability and Tracing
- CD Reconnect RBAC Conflict
How to Use These Labs Effectively¶
Use this section when you want a repeatable learning loop (reproduce → observe → fix → verify).
flowchart TD
A[Choose Lab by Symptom] --> B[Deploy Lab Infrastructure]
B --> C[Trigger Failure]
C --> D[Collect Evidence]
D --> E[Apply Targeted Fix]
E --> F[Verify Recovery]
F --> G[Capture Lessons Learned] Run labs like incident drills
Treat each lab as an on-call simulation. Time-box your investigation and record which signal (revision state, system log, console log, metrics) gave you the fastest root-cause clue.
Reuse one naming convention across all labs
Keep variable names consistent between labs ($RG, $APP_NAME, $ACA_ENV_NAME, $ACR_NAME, $LOCATION) so your troubleshooting muscle memory transfers cleanly.
Lab Selection Matrix¶
| Lab | Primary Symptom | First Signal to Check | Typical Root Cause | Fastest Recovery |
|---|---|---|---|---|
| ACR Image Pull Failure | Revision never starts | ContainerAppSystemLogs_CL pull errors | Bad image tag / registry auth | Push valid image + update app image |
| Revision Failover and Rollback | New revision unhealthy | az containerapp revision list | Risky config change in latest revision | Shift traffic back to healthy revision |
| Scale Rule Mismatch | Load increases, replicas do not | Replica count + KEDA events | Threshold too high / max replicas too low | Tune scale rule and retry load |
| Probe and Port Mismatch | Probe failures, no stable ready state | Probe failure warnings | App bind port != ingress target port | Align target port and rollout new revision |
| Managed Identity Key Vault Failure | Route returns 500/403 | App logs with identity errors | Missing role assignment on Key Vault scope | Assign RBAC role and re-verify |
| Revision Provisioning Failure | Revision stuck/failed provisioning | Revision lifecycle events | secretRef points to missing secret | Add secret and redeploy revision |
| Ingress Target Port Mismatch | External endpoint unreachable | Ingress target port config | Target port doesn't match app listen port | Fix target port to match app |
| Traffic Routing Canary Failure | Intermittent failures (~50%) | Traffic weight and revision health | Bad revision receiving traffic | Rollback traffic to healthy revision |
| Dapr Integration | Dapr calls fail | System logs with Dapr errors | Sidecar not enabled or component misconfigured | Enable Dapr and fix component YAML |
| Observability and Tracing | No traces in App Insights | Application Insights query | Connection string not set | Configure OTel and connection string |
| CD Reconnect RBAC Conflict | AppRbacDeployment failure on reconnect | Role assignment ID in deployment error | Orphaned role assignment from previous CD | Delete conflicting assignment, then reconnect |
Step-by-Step: Standard Lab Execution Pattern¶
-
Prepare shell variables
export RG="rg-aca-lab-shared" export LOCATION="koreacentral" export ACA_ENV_NAME="cae-myapp" export APP_NAME="ca-myapp" export ACR_NAME="acrmyapp"Expected output: no output (environment variables set in your shell).
-
Validate CLI context
Command Why it is used az account show ...Confirms the active Azure subscription context before changes are made. Expected output: active subscription metadata and extension upgrade confirmation.
-
Deploy the chosen lab infrastructure
az deployment group create \ --name "lab-run" \ --resource-group "$RG" \ --template-file "./labs/<lab-name>/infra/main.bicep" \ --parameters baseName="labrun"Command Why it is used az deployment group create ...Deploys the Bicep or ARM template into the target resource group. Expected output pattern:
-
Trigger failure and collect signals
Expected output: one or more failure indicators (for example
ImagePullBackOff,ProbeFailed,403 Forbidden, or non-scaling replica count). -
Apply targeted fix and verify recovery
# Use the specific fix command from each lab guide az containerapp revision list --name "$APP_NAME" --resource-group "$RG" --output tableCommand Purpose az containerapp revision list --name "$APP_NAME" --resource-group "$RG" --output tableLists revisions, so you can see whether rollout state matches the failure pattern. Expected output pattern: at least one
Healthyrevision with intended traffic weight. -
Clean up resources
Expected output: deletion completed or a
Succeededstate for cleanup actions.
Expected vs Actual Investigation Template¶
| Checkpoint | Expected State | Typical Failure State | Action |
|---|---|---|---|
| Revision health | Healthy and active | Failed or stuck provisioning | Inspect system logs and revision events |
| Replica status | Running replicas under load | 0 replicas or repeated restart | Check probes, scale settings, and runtime logs |
| Route behavior | HTTP 200 with expected payload | 5xx, timeout, or connection refused | Validate ingress + target port + dependencies |
| Identity access | Token retrieval and authorized resource call | 401/403 in console logs | Verify managed identity and RBAC scope |