AppGW to Internal Azure Container Apps NSG Mismatch Reproduction Lab¶

Reproduce H1 from the AppGW to Internal Container Apps: NSG Destination Pinned to staticIp Fails playbook: an Application Gateway backend health goes Unhealthy against an internal Container Apps environment because the container app subnet NSG inbound rule uses Destination = staticIp (a single IP) instead of the container app's subnet CIDR. NSGs behind an internal load balancer evaluate the destination NIC, not the load balancer frontend, so staticIp as a destination is by-design broken on workload profiles environments.

Lab Metadata¶

1) Background¶

An internal Azure Container Apps environment (vnetConfiguration.internal = true) exposes container apps only through the environment's internal load balancer (ILB). The ILB frontend IP is surfaced as environment.properties.staticIp and is inside the container app's infrastructure subnet CIDR. When an operator fronts the internal environment with an Application Gateway (AppGW), the AppGW must be able to reach the container app via the ILB, which in turn forwards traffic to the edge-proxy NIC inside the container app subnet (on ports 31443 for HTTPS and 31080 for HTTP).

For the subnet NSG to allow this traffic, the Microsoft Learn firewall-integration NSG table for workload profiles prescribes:

• Source = the Application Gateway subnet CIDR (or the client IP range).
• Destination = the container app's subnet CIDR (not staticIp).
• Destination ports = 443 and 31443 (add 80 and 31080 for HTTP).

The Destination requirement is the failure surface this lab reproduces. Operators frequently pin the NSG rule Destination to staticIp because that is the address AppGW ends up sending traffic to (after resolving the container app FQDN through the linked Private DNS Zone) and the address written into the Private DNS Zone A record. But NSGs behind a load-balanced pool do not evaluate destination against the load balancer frontend — they evaluate against the destination NIC (the edge-proxy NIC in snet-cae, whose IP is inside the subnet CIDR but is not equal to staticIp). A rule with Destination = <staticIp>/32 therefore drops every packet that lands on the edge-proxy NIC, and AppGW backend health goes Unhealthy.

This class of misconfiguration is especially common:

• After migrating from a Consumption-only environment to workload profiles (the old NSG table listed staticIp as an acceptable Destination, but the workload-profiles table lists only the subnet CIDR).
• When operators copy-paste NSG rules from unrelated AppGW tutorials that target public backends where staticIp is the on-wire address.
• When infrastructure-as-code templates hard-code env.properties.staticIp as the Destination because it is a stable, environment-scoped value that resolves at deployment time.

This lab isolates the failure to a single controlled variable: rule 100's Destination address. Ports (443+31443) are left correct, the ILB probe path (rule 200) is left correct, and no other rule changes between the broken and fixed states. If the lab shows AppGW backend health flipping Healthy → Unhealthy → Healthy in lock-step with the Destination toggle, H1 is falsifiably confirmed.

2) Hypothesis¶

IF the container app subnet NSG inbound rule 100 has Destination = <staticIp>/32 while all other rule properties (Source, ports, protocol, priority) are left correct AND the NSG also carries a Deny-all rule at priority 4096 so that the default AllowVnetInBound (priority 65000) cannot mask rule 100, THEN:

• Baseline state (before trigger.sh, NSG has only Azure default rules): Application Gateway backend health reports every backend server as Healthy; a curl against the AppGW public IP returns HTTP 200.
• Broken state (after trigger.sh applies rules 100 + 200 + 4096): Application Gateway backend health transitions to Unhealthy on every backend server; a curl against the AppGW public IP returns HTTP 502; the container app subnet NSG rule 100 has destinationAddressPrefix = <staticIp>/32 (or destinationAddressPrefixes = [<staticIp>/32]).
• Fixed state (after fix.sh rewrites rule 100 Destination from <staticIp>/32 to the CAE subnet CIDR, all other properties unchanged): Application Gateway backend health returns to Healthy on every backend server; a curl against the AppGW public IP returns HTTP 200; rule 100 now reports destinationAddressPrefix = <caeSubnetPrefix>.

Falsification requires the fix to restore Healthy without changing any other variable — no port changes, no ILB probe rule changes, no AppGW backend or probe changes. If restoring Healthy requires any other edit, the H1 attribution to rule 100's Destination is not clean.

Architecture¶

flowchart TD
    Client["Operator workstation<br/>(public internet)"] -->|HTTP :80| AppGW["Application Gateway Standard_v2<br/>snet-appgw 10.0.1.0/24"]
    AppGW -->|"backend pool = app FQDN<br/>HTTPS :443, pickHostNameFromBackendAddress"| PDZ["Private DNS Zone<br/>= env.defaultDomain<br/>* and @ A records -> staticIp"]
    PDZ -->|resolves to staticIp| ILB["Internal ILB frontend<br/>= env.properties.staticIp"]
    ILB -->|forwards to :31443 on backend NIC| NSG{"NSG on snet-cae<br/>rule 100: Allow, Dst = ???, ports 443+31443<br/>rule 200: Allow, Src = AzureLoadBalancer, ports 30000-32767<br/>rule 4096: Deny all"}
    NSG -->|"Dst = staticIp/32 (BROKEN)"| Drop["Packet dropped:<br/>edge-proxy NIC IP != staticIp<br/>-> AppGW backend Unhealthy"]
    NSG -->|"Dst = snet-cae CIDR (FIXED)"| Edge["Edge-proxy NIC in snet-cae<br/>-> Container App replica<br/>-> AppGW backend Healthy"]

3) Runbook¶

Deploy infrastructure¶

export RG="rg-appgw-aca-nsg-lab"
export LOCATION="koreacentral"
export BASE_NAME="appgwnsg"

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
    --resource-group "$RG" --name appgw-aca-nsg-mismatch \
    --template-file labs/appgw-to-internal-aca-nsg-mismatch/infra/main.bicep \
    --parameters baseName="$BASE_NAME" location="$LOCATION"

Reproduce the failure¶

bash labs/appgw-to-internal-aca-nsg-mismatch/trigger.sh
bash labs/appgw-to-internal-aca-nsg-mismatch/verify.sh

Apply the fix¶

bash labs/appgw-to-internal-aca-nsg-mismatch/fix.sh
bash labs/appgw-to-internal-aca-nsg-mismatch/verify.sh

Clean up¶

bash labs/appgw-to-internal-aca-nsg-mismatch/cleanup.sh

4) Experiment Log¶

The per-scenario observation log from the 2026-07-04 re-executed live run in Korea Central (the currently committed evidence pack). The initial 2026-07-03 live run surfaced the CAE FQDN routing quirk documented in the success admonition above; the 2026-07-04 re-execution captured the artifact set now in-tree. Every [Observed] bullet cites the raw artifact under labs/appgw-to-internal-aca-nsg-mismatch/evidence/ captured during the 2026-07-04 re-execution.

Baseline scenario (before trigger.sh)¶

• [Observed] evidence/baseline-backend-health.json: .backendAddressPools[0].backendHttpSettingsCollection[0].servers[0].health = "Healthy".
• [Observed] evidence/baseline-nsg-rules.json: empty custom rule set (only Azure default rules are in effect — AllowVnetInBound priority 65000, AllowAzureLoadBalancerInBound priority 65001, DenyAllInBound priority 65500 — implicit and not enumerated in az network nsg rule list output).
• [Observed] evidence/baseline-curl.txt: HTTP 200 in 0.275887s.

Broken scenario (after trigger.sh)¶

• [Observed] evidence/broken-backend-health.json: .backendAddressPools[0].backendHttpSettingsCollection[0].servers[0].health = "Unhealthy" with probe error text Cannot connect to backend server. Check whether any NSG/UDR/Firewall is blocking access to the server. Check if application is running on correct port. To learn more visit - https://aka.ms/servernotreachable. This is the strongest single-string signal that the failure is at the NSG/UDR/Firewall layer, not at the container-app-runtime layer.
• [Observed] evidence/broken-nsg-rules.json: rule 100 (allow-appgw-inbound-broken) with sourceAddressPrefix = "10.0.1.0/24", destinationAddressPrefix = "10.0.2.243/32" (the staticIp/32 misconfig — 10.0.2.243 is the environmentStaticIp from deploy-outputs.json, inside the CAE subnet CIDR 10.0.2.0/23), destinationPortRanges = ["443","31443"]; rule 200 (allow-azure-lb-probes) with sourceAddressPrefix = "AzureLoadBalancer", destinationAddressPrefix = "10.0.2.0/23", destinationPortRange = "30000-32767"; rule 4096 (deny-all-inbound) with all *.
• [Observed] evidence/broken-curl.txt: HTTP 502 in 0.325699s — AppGW returned HTTP 502 within ~0.3 s of the client request, matching the original hypothesis prediction. AppGW Standard_v2 short-circuited to a Cannot connect to backend server response because its backend health had already transitioned to Unhealthy before this request arrived (rule 100 misconfig had been applied 150 s earlier by trigger.sh and probe re-convergence had already flipped every backend server to Unhealthy). In an alternative shape — for example if AppGW's request timeout exceeded the client's --max-time, or if backend health had not yet re-converged at the moment of the request — the manifestation could shift to HTTP 000 (client timeout with curl: (28) Operation timed out); both HTTP 502 and HTTP 000 are consistent with the same underlying H1 root cause because both mean the client did not receive a valid successful HTTP response.
• [Correlated] The transition from Healthy (baseline) to Unhealthy (broken) was observed within one AppGW probe re-convergence window after trigger.sh applied rule 100 (the trigger.sh script waits 150 s for probe reconvergence and then captures evidence, and the health-status flip was already stable at that capture point). No other variable changed between the two scenarios.

Fixed scenario (after fix.sh)¶

• [Observed] evidence/fixed-backend-health.json: .backendAddressPools[0].backendHttpSettingsCollection[0].servers[0].health = "Healthy".
• [Observed] evidence/fixed-nsg-rules.json: rule 100 with destinationAddressPrefix = "10.0.2.0/23" (the CAE subnet CIDR from deploy-outputs.json .caeSubnetPrefix.value); all other rule 100 properties (sourceAddressPrefix = "10.0.1.0/24", destinationPortRanges = ["443","31443"], priority = 100, access = Allow, protocol) unchanged from the broken scenario.
• [Observed] evidence/fixed-curl.txt: HTTP 200 in 0.376139s — restored to the same latency band as baseline (~0.3-0.4 s).
• [Correlated] Falsification (post-fix evidence): the fix restored Healthy without changing any other variable. Ports (443+31443) were left correct in both broken and fixed states (gate F), and rule 200 kept the AzureLoadBalancer → snet-cae, 30000-32767 allow path present in both states (gate G). These two invariants exclude H2 (missing edge-proxy ports) and H3 (AllowAzureLoadBalancerInBound shadowed by a higher-priority Deny) as compounding causes.

Summary¶

• [Inferred] If gates A + B + C are all true at post-trigger.sh verification, and gates D + E + F + G are all true at post-fix.sh verification, then H1 (NSG rule 100 Destination pinned to staticIp/32) is the sole cause of the AppGW backend Unhealthy state. Both verdict = HYPOTHESIS_CONFIRMED and falsification = FIX_VERIFIED should appear in evidence/verify-result.json.
• [Observed] On the 2026-07-04 re-executed live run all seven gates evaluated true in the currently committed evidence/verify-result.json: verdict = HYPOTHESIS_CONFIRMED, falsification = FIX_VERIFIED, verify.sh exit 0. H1 (NSG rule 100 Destination pinned to staticIp/32) is confirmed as the sole cause of the AppGW backend Unhealthy state on this deployment shape. The 2026-07-03 initial run also had all seven gates true after the CAE FQDN routing quirk fix was applied; the 2026-07-04 re-execution reproduces the same result against a fresh deployment.

Expected Evidence¶

verify.sh emits evidence/verify-result.json with the following seven gates. The hypothesis is confirmed when gates A, B, and C are all true. Falsification of the hypothesis is closed when gates D and E are also true after fix.sh. Gates F and G actively exclude H2 (missing edge-proxy ports) and H3 (AllowAzureLoadBalancerInBound shadowed by a higher-priority Deny) as compounding causes; if F or G is false, the H1-only attribution weakens even if D and E are true.

Verdict transitions:

• Before trigger.sh completes: verdict = HYPOTHESIS_NOT_CONFIRMED, falsification = NOT_YET_TESTED.
• After trigger.sh (gates A, B, C all true): verdict = HYPOTHESIS_CONFIRMED, falsification = NOT_YET_TESTED. Gates F and G also evaluate at this point and MUST be true — if either is false, verify.sh exits non-zero because the H1-only attribution is compromised by an H2/H3 compounding cause.
• After fix.sh (gates D, E also true, F and G still true): verdict = HYPOTHESIS_CONFIRMED, falsification = FIX_VERIFIED.
• If fix.sh runs but backend does not recover: verdict = HYPOTHESIS_CONFIRMED, falsification = FIX_DID_NOT_RECOVER, and verify.sh exits non-zero — this outcome would invalidate the single-variable claim and require investigating H2 (missing edge-proxy ports, ruled out by gate F on the broken snapshot) or H3 (Deny rule shadowing AllowAzureLoadBalancerInBound, ruled out by gate G on the broken snapshot) as compounding causes.

Client-side evidence¶

trigger.sh and fix.sh also capture curl -w '%{http_code}' results against the AppGW public IP into baseline-curl.txt, broken-curl.txt, and fixed-curl.txt. Observed on the 2026-07-04 re-executed live run (the currently committed evidence/*-curl.txt):

• baseline-curl.txt: HTTP 200 in 0.275887s
• broken-curl.txt: HTTP 502 in 0.325699s
• fixed-curl.txt: HTTP 200 in 0.376139s

The observed broken-state manifestation on the currently committed run is HTTP 502 in ~0.3 s, matching the original hypothesis prediction. The lab treats HTTP 502 and HTTP 000 (client timeout) as equivalent evidence of the same underlying H1 failure (Cannot connect to backend server from AppGW backend health), because both mean the client did not receive a valid successful HTTP response. The exact status code depends on AppGW's request-timeout configuration relative to the client's --max-time and on the timing of the request relative to backend-health probe re-convergence; if AppGW's request timeout exceeds the client's --max-time, or if the request arrives before probe re-convergence has fully flipped every backend server to Unhealthy, the manifestation can shift to HTTP 000 (client timeout with curl: (28) Operation timed out), and both are consistent with H1.

The curl evidence is redundant with the AppGW backend health gates but provides an operator-friendly end-to-end signal that the failure is observable from an unprivileged client.

5) Verification Queries¶

The lab does not require KQL because the primary signals are AppGW control-plane state (backend health) and NSG rule state — both are read via az CLI, not Log Analytics. However, once the lab is deployed the following KQL queries against the AppGW diagnostic settings can supplement the CLI evidence.

KQL: AppGW access log 502 traces during the broken window¶

Replace the two datetime() placeholders with the wall-clock window between trigger.sh completing step 6 (broken evidence captured) and fix.sh starting step 1. trigger.sh prints its step-6 completion line to stdout; the operator can capture the start and end timestamps manually or from the deployment activity log. The TimeGenerated filter is required — without it, a shared Log Analytics workspace would return 502 rows from unrelated AppGW deployments and rows from a previous run of this lab, which would contradict the "zero such rows before trigger.sh and after fix.sh" claim below.

let brokenStart = datetime(2026-07-04T10:30:00Z);
let brokenEnd   = datetime(2026-07-04T10:35:00Z);
AzureDiagnostics
| where TimeGenerated between (brokenStart .. brokenEnd)
| where ResourceType == "APPLICATIONGATEWAYS"
| where Category == "ApplicationGatewayAccessLog"
| where httpStatus_d == 502
| project TimeGenerated, clientIP_s, requestUri_s, httpStatus_d, backendPoolName_s, backendSettingName_s, serverStatus_s, serverResponseLatency_s
| order by TimeGenerated desc

Expected during the broken window: rows where httpStatus_d = 502, backendPoolName_s = aca-backend-pool, and serverStatus_s indicates the backend was unreachable. Re-run the same query with brokenStart and brokenEnd bracketing the baseline window (before trigger.sh) and the fixed window (after fix.sh) and expect zero rows in both.

KQL: AppGW backend health transitions¶

Application Gateway backend health transitions are not written to AzureDiagnostics on every probe cycle (only ApplicationGatewayAccessLog and ApplicationGatewayFirewallLog are surfaced). To trend backend health over time, use the az network application-gateway show-backend-health command on a schedule and archive the JSON, or use the HealthyHostCount and UnhealthyHostCount Application Gateway metrics via az monitor metrics list. The lab does not automate this — the three JSON snapshots (baseline / broken / fixed) captured by trigger.sh and fix.sh are sufficient for the H1 falsification argument.

6) Portal Evidence¶

Azure Portal screenshots for the five key states of this lab. Saved under docs/assets/troubleshooting/appgw-to-internal-aca-nsg-mismatch/. Captured 2026-07-04 against a live re-deployment in Korea Central (agw-appgwnsg-kl4kt7, nsg-snet-cae-kl4kt7) via a Playwright MCP session on ms.portal.azure.com. Each PNG was sanitized per the AGENTS.md PII rules and visually verified before commit; the az CLI JSON files under evidence/ (baseline-backend-health.json, broken-backend-health.json, fixed-backend-health.json, broken-nsg-rules.json, fixed-nsg-rules.json) remain the authoritative machine-readable evidence for gates A-G and are what verify.sh reads. The Portal screenshots below are the operator-facing complement — they show the same failure and recovery in the shape a support engineer would see when triaging the incident from the Portal.

Baseline — Application Gateway Backend health¶

Broken — Application Gateway Backend health¶

Broken — NSG rule 100 Destination¶

Fixed — Application Gateway Backend health¶

Fixed — NSG rule 100 Destination¶

Recapture reference (for future re-runs)¶

If the lab is re-deployed and the Portal screenshots need to be re-captured (for example after a Portal UX change that alters the blade layout), use the table below. The five PNGs above were captured against these exact blades on 2026-07-04.

Follow the Portal Screenshot Capture (PII Replacement Rules) documented in AGENTS.md when re-capturing: use the reusable PII helper, verify each PNG with the Read/look_at tool, and confirm no MICROSOFT NON-PRODUCTION badge, no real subscription IDs, and no employee emails or aliases are visible before referencing the PNG from this file.

Clean Up¶

bash labs/appgw-to-internal-aca-nsg-mismatch/cleanup.sh

Related Playbook¶

• AppGW to Internal Container Apps: NSG Destination Pinned to staticIp Fails

See Also¶

• Application Gateway Integration with an Internal Container Apps Environment
• UDR and NSG Egress Blocked Lab
• Private Endpoint DNS Failure Lab
• Ingress Not Reachable Playbook
• Networking Best Practices

Sources¶

• Securing a Virtual Network in Azure Container Apps (workload profiles) (Microsoft Learn)
• Protect Azure Container Apps with Application Gateway and Web Application Firewall (WAF) (Microsoft Learn)
• Provide an internal environment for Azure Container Apps (Microsoft Learn)
• Networking in Azure Container Apps environment (Microsoft Learn)
• Network security groups (Microsoft Learn)
• Application Gateway backend health monitoring (Microsoft Learn)
• Application Gateway health probes overview (Microsoft Learn)
• Bicep BCP120 — value must be a compile-time constant (Microsoft Learn)