Lab Finder¶

Symptom-first index of every reproduction lab in this guide. Use this page when you already know the error string, exit code, or user-visible symptom and want to jump straight to the matching lab.

If you do not yet know which symptom cluster applies, start with the Quick Decision Tree or the First 10 Minutes runbook instead — this page assumes the cluster is already narrowed.

How this page is organized¶

This finder complements the two existing indexes rather than replacing them:

Page	Question it answers	Coverage
Lab Guides catalog	"What labs exist? What difficulty and duration?"	All 54 labs, browse-oriented, alphabetical-by-importance
Lab Guides Selection Matrix	"Which curated lab should I run first?"	11 curated labs, learning-path oriented
This page	"I am seeing X — which lab reproduces X?"	All 54 labs, symptom-first, grouped by 12 clusters

Every row below cites the first evidence signal an on-caller would observe (log excerpt, CLI output, or metric shape) so the finder is usable from an alert or paging context, not just from a browse context.

Labs reproduce, playbooks diagnose

Every cluster below has matching multi-hypothesis playbooks under Playbooks. During an active incident, open the playbook first to enumerate hypotheses and gather evidence. Use this finder to reproduce the failure in a lab environment when you want to build muscle memory or validate a fix strategy against a known-good repro.

flowchart TD
    A[Symptom Observed] --> B{Cluster known?}
    B -->|No| C[Quick Decision Tree]
    B -->|Yes| D[This page: pick cluster below]
    D --> E[Match error string or metric shape]
    E --> F[Open lab guide]
    F --> G[Deploy lab infra + trigger + verify + fix]

Cluster 1 — Startup and provisioning failures¶

Revision never reaches Healthy. az containerapp revision list shows ProvisioningState=Failed or Provisioning.

Symptom / signal	First evidence to check	Lab guide	Backing infra
`ImagePullBackOff` / `MANIFEST_UNKNOWN`	`ContainerAppSystemLogs_CL` pull errors; `az deployment operation group list` `statusMessage.error.message`	ACR Image Pull Failure	labs/acr-pull-failure/
Revision stuck / `secretRef` unresolved	Revision lifecycle events; `az containerapp revision show`	Revision Provisioning Failure	labs/revision-provisioning-failure/
Bicep deploy hangs / times out in Provisioning	`az deployment group show` status + revision provisioning events	Bicep Deployment Timeout	Inline guide only
Cold start latency > 30 s from image size	Startup probe timings; container image size	Image Size Startup Delay	Inline guide only
`exec format error` on startup	System logs; image manifest platform	Multi-Arch Image Mismatch	Inline guide only
`toomanyrequests` from Docker Hub	Pull error text mentioning `docker.io` rate	Docker Hub Rate Limit	Inline guide only
`QuotaExceeded` on environment or app create	Deployment error text with quota name	Subscription Quota Exceeded	Inline guide only
Wrong workload profile chosen (perf/cost)	`az containerapp env workload-profile list`; metrics divergence	Workload Profile Mismatch	Inline guide only
Startup degraded but recovers	Transient probe failures during warm-up window	Startup Degraded Transient Failure	Inline guide only

Cluster 2 — Networking and ingress¶

External endpoint unreachable, or app-to-registry / app-to-KeyVault paths fail on network layer.

Start with the ACR Network Path Selection platform doc if the symptom is ACR-pull-adjacent — it names and orders all five distinct ACR paths.

Symptom / signal	First evidence to check	Lab guide	Backing infra
ACR HTTP 403 on pull, firewall SNAT path	ACR firewall logs; DENIED message names firewall PIP	ACR Network Path A — Firewall Allowlist	labs/acr-network-path-firewall-allowlist/
`ImagePullUnauthorized` on fresh pull, PE path	PE zone link state; fresh vs cached pull behavior	ACR Network Path B — PE Direct	labs/acr-network-path-pe-direct/
Pull succeeds but firewall log goes silent	Missing `/32` UDR routes per PE NIC IP	ACR Network Path C — PE Forced Inspection	labs/acr-network-path-pe-forced-inspection/
Workload NXDOMAIN on ACR data FQDN	Linked `privatelink.azurecr.io` zone missing per-region A record	ACR Network Path D — Record-Level Zone Authority	labs/acr-network-path-record-split-brain/
Workload resolves ACR to public IP	Custom DNS forwarder upstream not `168.63.129.16`	ACR Network Path E — DNS Forwarder Bypass	labs/acr-network-path-dns-forwarder-bypass/
Application Gateway → internal ACA returns 502	NSG effective rules; AppGW backend health	AppGW to Internal ACA NSG Mismatch	Inline guide only
External endpoint unreachable, wrong target port	`az containerapp ingress show`; app listen port	Ingress Target Port Mismatch	labs/ingress-target-port-mismatch/
Probe failing, port mismatch	Probe warnings in system logs	Probe and Port Mismatch	labs/probe-and-port-mismatch/
Environment create fails with subnet too small	`az deployment group show` error text mentioning subnet size	Subnet CIDR Exhaustion	Inline guide only
Replica startup fails from blocked egress FQDNs	UDR to NVA/firewall; egress deny logs	UDR and NSG Egress Blocked	Inline guide only
Private endpoint DNS NXDOMAIN	Private DNS Zone → VNet link state	Private Endpoint DNS Failure	Inline guide only
Downstream firewall dropping traffic after env recreate	Env outbound IP list before vs after recreate	Egress IP Change	Inline guide only
Managed certificate stuck `Pending`	`az containerapp hostname list`; CNAME + `asuid` TXT records	Custom Domain TLS Renewal	Inline guide only
WebSocket disconnects mid-session	Session affinity setting on ingress	WebSocket and gRPC Ingress	Inline guide only
Session state lost across replicas	Ingress `stickySessions.affinity` setting	Session Affinity Failure	Inline guide only

Cluster 3 — Scaling and runtime behavior¶

Replica count does not track load, or replicas scale but do not converge on healthy steady-state.

Symptom / signal	First evidence to check	Lab guide	Backing infra
Load rises, replicas do not	Replica count + KEDA `ScaledObject` events	Scale Rule Mismatch	labs/scale-rule-mismatch/
KEDA scaler returns 0 metrics	KEDA operator logs + `ScaledObject` status	KEDA No Metrics Returned	labs/keda-no-metrics-returned/
First request after idle takes 5-30 s	Replica count timeline; `minReplicas=0` setting	Cold Start Scale to Zero	Inline guide only
Replicas concentrate on one node	Replica distribution across availability zones	Replica Node Spread	labs/replica-node-spread/
Uneven load across replicas	Per-replica request rate metric	Replica Load Imbalance	Inline guide only
High `MemoryPercentage` but no scale-out	HPA denominator + cache inflation math	Memory Percentage vs KEDA Utilization	labs/memory-percentage-vs-keda-utilization/

Cluster 4 — Storage and volumes¶

Mount fails or fills, or permissions deny access to persistent volumes.

Symptom / signal	First evidence to check	Lab guide	Backing infra
SMB mount error on Azure Files	Env storage config; storage account key + share name	Azure Files Mount Failure	Inline guide only
OOM-like restart from ephemeral disk full	Container writes to `/tmp`; `ephemeralStorage` limit	EmptyDir Disk Full	Inline guide only
`permission denied` on volume path	Container UID vs mount ownership	Volume Permission Denied	Inline guide only

Cluster 5 — Identity and security¶

Managed identity token flow, RBAC role assignments, federated credentials, or built-in auth fails.

Symptom / signal	First evidence to check	Lab guide	Backing infra
500/403 from route calling Key Vault	App logs for `ManagedIdentityCredential`; Key Vault RBAC scope	Managed Identity Key Vault Failure	labs/managed-identity-key-vault-failure/
AADSTS50011 redirect URI mismatch	EasyAuth logs; app registration reply URLs	EasyAuth Entra ID Failure	Inline guide only
AADSTS70021 no matching federated credential	GH Actions OIDC token subject claim	GitHub Actions OIDC Failure	Inline guide only
`AppRbacDeployment: role assignment already exists`	Existing role assignments on CD identity	CD Reconnect RBAC Conflict	labs/cd-reconnect-rbac-conflict/

Cluster 6 — Observability and logging¶

Logs missing from Log Analytics, or Application Insights returns no telemetry.

Symptom / signal	First evidence to check	Lab guide	Backing infra
Silent Log Analytics ingestion	Env `appLogsConfiguration.destination` is `null`	Diagnostic Settings Missing	labs/diagnostic-settings-missing/
Logs missing entirely from workspace	Diagnostic setting on env resource	Log Analytics Ingestion Gap	Inline guide only
No traces in Application Insights	`APPLICATIONINSIGHTS_CONNECTION_STRING` env var; OTel config	App Insights Connection String Missing	Inline guide only
Distributed traces missing spans	OTel exporter config; sampling rate	Observability and Tracing	labs/observability-tracing/

Cluster 7 — Deployment and revisions¶

Traffic routes to wrong revision, revision cap hit, or transient revision failures during rollout.

Symptom / signal	First evidence to check	Lab guide	Backing infra
New revision unhealthy after risky change	`az containerapp revision list`; ingress port config	Revision Failover and Rollback	labs/revision-failover/
Intermittent 5xx (~50%) after canary	Traffic weight + revision health per revision	Traffic Routing Canary Failure	labs/traffic-routing-canary/
`RevisionCountLimitReached` (100-revision cap)	`az containerapp revision list --all` count	Revision History Limit	labs/revision-history-limit/

Cluster 8 — Jobs and cron¶

Container Apps Jobs miss schedules or queue-backed jobs storm.

Symptom / signal	First evidence to check	Lab guide	Backing infra
Cron job did not run at expected time	Job schedule cron expression; UTC vs local timezone	Scheduled Job Missed	Inline guide only
Queue backs up despite job running	Job `maxExecutions` + KEDA scale rule	Event Job Storm	Inline guide only

Cluster 9 — Dapr integration¶

Dapr sidecar not injected, or component YAML resolves to wrong resource.

Symptom / signal	First evidence to check	Lab guide	Backing infra
Dapr sidecar calls fail	System logs with Dapr sidecar errors; `dapr.enabled`	Dapr Integration	labs/dapr-integration/
Dapr state store errors	Component YAML `metadata.name` + `spec.scopes`	Dapr State Store Failure	Inline guide only
Messages not delivered on pub/sub	Component topic + consumer scope	Dapr Pub/Sub Failure	Inline guide only

Cluster 10 — Resource pressure (CPU and memory)¶

Container hits CPU throttling or is killed on memory limit.

Symptom / signal	First evidence to check	Lab guide	Backing infra
Response latency spikes under load	`UsageNanoCores` vs CPU limit; throttling metric	CPU Throttling	Inline guide only
Exit code 137 (SIGKILL) / OOMKilled restart loop	`WorkingSetBytes` vs memory limit; D&S Container Exit Events detector	Memory Leak OOMKilled	Inline guide only

Cluster 11 — Cost and profile¶

Unexpected billing or wrong-profile selection.

Symptom / signal	First evidence to check	Lab guide	Backing infra
Unexpected off-hours billing	`minReplicas` setting; per-replica active-second billing	Min Replicas Cost Surprise	Inline guide only

Cluster 12 — Multi-region and failover¶

Front Door or Traffic Manager does not shift traffic to the healthy region.

Symptom / signal	First evidence to check	Lab guide	Backing infra
Traffic does not fail over to secondary	Front Door health probe config; backend pool health	Multi-Region Failover	Inline guide only

When no lab matches your symptom¶

If none of the 54 labs above match:

Confirm the symptom cluster with the Quick Decision Tree.
Check Representative Log Patterns for KQL fragments that may narrow the cluster.
Search the KQL Query Library by service (ContainerAppSystemLogs_CL, ContainerAppConsoleLogs_CL) — a query pack may already match the signal.
If the symptom is genuinely new, open an issue against this repo describing the signal and suggested cluster.