ACR Network Path A — Firewall Allowlist Lab¶

Reproduce Scenario A from ACR Network Path Selection: the Container Apps replica reaches ACR over ACR's public FQDN, but that egress is forced through an Azure Firewall whose SNAT public IP is the only entry in ACR's networkRuleSet.ipRules allowlist. ACR's selected-networks IP rule is therefore keyed on the firewall's outbound public IP, not on any replica IP — replicas have no public IP of their own, and their RFC1918 internal IPs are invisible to ACR's firewall layer once SNAT'd at the firewall.

This lab makes a non-obvious Azure Container Apps behavior falsifiable: toggling a single entry in networkRuleSet.ipRules on the ACR resource deterministically flips fresh-pull behavior between success and failure on the replica side, while the already-running revision keeps serving traffic from cached image layers throughout the broken window. The failure surfaces as an HTTP 403 from ACR's firewall layer with a response body that literally names the firewall's public IP as the rejected source — which is the strongest possible smoking-gun evidence that the IP allowlist on the registry resource controls fresh pulls.

Lab Metadata¶

Lab position¶

This lab is part of the 5-lab ACR network path series that reproduces the five distinct network paths a Container App can take to reach ACR. See ACR Network Path Selection for the conceptual taxonomy that names and orders all five paths.

This behavior was reproduced in Korea Central on 2026-06-29 for the canonical Phase B evidence pack with the specific topology described above (Azure Firewall Basic, ACR Premium with defaultAction=Deny + networkRuleBypassOptions=None, Container Apps Consumption profile, ACR admin-credential auth via az containerapp registry set --username --password). Treat it as validated for this lab's specific topology, auth mode, and timing — not as a universal statement for every Azure Container Apps + ACR deployment. Different ACR SKUs (Basic/Standard lack networkRuleSet), different auth modes (managed identity adds a control-plane token-exchange step on a different network path), different egress designs (NAT Gateway, App Gateway, no firewall at all), and different Container Apps platform versions can change the observed behavior. The phrases below (e.g. "the firewall public IP is the only relevant source IP") refer to this specific topology, not to every deployment that uses Container Apps with ACR.

1) Background¶

Azure Container Apps can reach ACR through several network paths — public via firewall, Private Endpoint direct, Private Endpoint with forced inspection, or one of two DNS failure scenarios. The ACR Network Path Selection page documents all of them.

Scenario A (Public ACR via Firewall) is the canonical "Path A" topology: ACR is left publicly accessible, but the Container Apps environment is constrained by a 0.0.0.0/0 UDR that forces every replica's outbound traffic through an Azure Firewall in the same VNet. The firewall SNATs the outbound flow to its own public IP, which becomes the source IP that ACR sees on the inbound TCP connection. ACR is then locked down with networkRuleSet.defaultAction=Deny and a single entry in ipRules — the firewall's public IP. The result: replicas pull successfully because their flows arrive at ACR with the allow-listed source IP, but any other client (a developer laptop, a non-firewalled VM, an attacker on the public internet) is rejected at the ACR firewall layer with HTTP 403.

This topology is operationally common in regulated environments where the security team requires registry traffic to traverse a stateful firewall (so it can be logged and inspected at the application-rule layer) but the cost or complexity of an ACR Private Endpoint is not justified. It is not the recommended default for production Container Apps on ACR — Path B (Private Endpoint with default routing) is — but it is a legitimate production pattern when the operator wants registry-side access control to live on the registry resource itself, keyed on a single well-known public IP, and reflected in Azure Firewall's application-rule logs.

Three properties make Scenario A worth reproducing as a hands-on lab:

• The selected-networks IP allowlist on ACR is keyed on the egress firewall's SNAT public IP, not the replica's internal IP. Replicas in a Container Apps workload-profile subnet have RFC1918 IPs from snet-aca (e.g., 10.80.0.0/23). Those internal IPs are invisible to ACR's firewall layer because Azure Firewall performs SNAT on every outbound flow, replacing the source IP with its own public IP before the flow leaves the VNet. ACR's networkRuleSet.ipRules therefore must contain the firewall's public IP for the allow-list to match. Misunderstanding this relationship is the most common reason "I added my Container Apps subnet to the ACR allowlist and it still gets denied" — there is no Container Apps source IP that ACR ever sees; the firewall's SNAT public IP is the only relevant address.
• In Azure Container Apps, removing the firewall PIP from ACR's allowlist breaks fresh pulls but does NOT degrade the already-running revision. This is the central operational asymmetry of the lab. When the v1 revision is already running on cached image layers, removing the firewall PIP from ACR's ipRules produces no immediate health-state change — the replica continues to serve / with build_tag=v1 because it never needs to re-pull. Only a new revision deployment (in this lab, the deliberate v-broken deploy) actually exercises the broken pull path, and that is when the failure surfaces as provisioningState lingering at Provisioned with healthState=None while ACR returns HTTP 403 to the firewall's SNAT IP.
• The HTTP 403 response from ACR contains the rejected source IP verbatim, and Container Apps captures it in system logs. This is the strongest evidence in the lab. ACR's firewall-layer response body is {"errors":[{"code":"DENIED","message":"client with IP '<observed-source-ip>' is not allowed access, refer https://aka.ms/acr/firewall to grant access"}]}. Azure Container Apps captures this entire payload into ContainerAppSystemLogs_CL when the image pull fails, with Reason_s == "ContainerTerminated" and the full DENIED message in Log_s. The IP that appears in the message is provably the firewall's public IP — not the replica IP, not the Container Apps control-plane egress IP, not any other Azure-managed IP. This makes the SNAT public IP directly observable from Container Apps system logs, which is unusual: most Azure firewall-layer rejections require correlation across multiple log tables to attribute the rejected source IP, but ACR's response body shortcuts this by including it inline.

The lab's outcome taxonomy enumerates the observable states across the baseline → broken → recovery cycle:

Architecture¶

flowchart TD
    REPLICA[Container App replica<br/>snet-aca 10.80.0.0/23<br/>RFC1918 internal IP]
    DOCKER[docker login + image pull<br/>over public ACR FQDN<br/>auth: ACR admin creds]
    UDR[UDR 0.0.0.0/0<br/>next-hop = AFW privateIPAddress<br/>10.80.2.4 in this layout]
    AFW[Azure Firewall Basic<br/>private IP 10.80.2.4<br/>application rules<br/>Allow: ACR login + data + MCR + acs-mirror]
    SNAT[SNAT to firewall public IP<br/>20.196.208.15]
    ACRFW[ACR firewall layer<br/>networkRuleSet.defaultAction=Deny<br/>networkRuleBypassOptions=None<br/>ipRules=AFW PIP <- controlled variable]
    ACR[ACR Premium<br/>publicNetworkAccess=Enabled<br/>adminUserEnabled=true<br/>dataEndpointEnabled=true]
    DENIED[HTTP 403 with body:<br/>client with IP AFW-PIP<br/>is not allowed access]
    CACHED[Already-cached image layers<br/>v1 revision stays Healthy<br/>during broken window]

    REPLICA --> DOCKER
    REPLICA --> CACHED
    DOCKER -->|0.0.0.0/0 default route| UDR
    UDR --> AFW
    AFW -->|application rule allow<br/>HTTPS to ACR FQDN| SNAT
    SNAT --> ACRFW
    ACRFW -->|HEALTHY: AFW PIP in ipRules| ACR
    ACRFW -.->|BROKEN: AFW PIP NOT in ipRules<br/>403 with PIP named in body| DENIED

    classDef workload fill:#e1f5ff,stroke:#0078d4
    classDef brokenpath fill:#fde7e9,stroke:#a4262c
    classDef registry fill:#e5f5e5,stroke:#107c10
    classDef neutral fill:#f3f2f1,stroke:#605e5c

    class REPLICA,DOCKER workload
    class DENIED brokenpath
    class ACR,ACRFW registry
    class UDR,AFW,SNAT,CACHED neutral

The solid arrows are the image-pull data path in the HEALTHY baseline state: replica → UDR-forced default route → Azure Firewall → SNAT to firewall public IP → ACR firewall layer (allow because AFW PIP is in ipRules) → ACR backend serves the layer. The dotted arrow is what happens in the BROKEN state after az acr network-rule remove removes the firewall PIP from ipRules: ACR's firewall layer rejects the inbound flow with HTTP 403 before the request reaches the registry backend, and the response body contains the literal text client with IP '<AFW-PIP>' is not allowed access. Note that DNS is NOT the controlled variable in this lab — the VNet uses default Azure DNS, both ACR FQDNs (login and data) resolve to public IPs, and the firewall's application rule allows them by FQDN. The neutral-colored "Already-cached image layers" node captures the lab's central Container Apps finding: during the broken window, the v1 revision continues to serve / from already-cached layers without re-pulling, and revision healthState stays Healthy. Only fresh pulls (the v-broken revision deployment) actually exercise the broken path.

2) Hypothesis¶

IF the Container Apps environment is integrated into a VNet with a 0.0.0.0/0 UDR forcing all snet-aca egress through Azure Firewall, the firewall has an application rule allowing HTTPS to both ACR FQDNs (login + regional data endpoint), ACR is configured with publicNetworkAccess=Enabled + adminUserEnabled=true + dataEndpointEnabled=true, and the Container App is wired to ACR via az containerapp registry set --username --password (admin-credential auth, NOT managed identity), THEN:

• with the firewall public IP in ACR's networkRuleSet.ipRules, deploying v1 succeeds: the v1 revision becomes Healthy and / returns build_tag=v1, proving a fresh pull of v1 traversed the firewall and was admitted by ACR's firewall layer;
• removing the firewall public IP from ACR's ipRules and deploying v-broken causes the v-broken pull to fail at ACR's firewall layer with HTTP 403; Container Apps captures the rejection in ContainerAppSystemLogs_CL with Reason_s == "ContainerTerminated" and a Log_s body containing the literal string client with IP '<firewall-public-ip>' is not allowed access; the v-broken revision lingers at provisioningState=Provisioned with healthState=None;
• AND the v1 revision stays Healthy and continues to serve / with build_tag=v1 throughout the broken window from already-cached image layers — falsifying the alternative hypothesis "removing the firewall PIP from ACR's allowlist will degrade the already-running revision's health";
• re-adding the firewall public IP to ACR's ipRules and deploying v-recover causes the v-recover pull to succeed: the v-recover revision becomes Healthy and / returns build_tag=v-recover, closing the loop on the falsification.

3) Runbook¶

Deploy baseline infrastructure¶

export RG="rg-acr-firewall-allowlist-lab"
export LOCATION="koreacentral"
export BASE_NAME="acrfwallow"

az extension add --name containerapp --upgrade

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
    --resource-group "$RG" \
    --name acr-firewall-allowlist \
    --template-file labs/acr-network-path-firewall-allowlist/infra/main.bicep \
    --parameters baseName="$BASE_NAME"

Expected output pattern:

"provisioningState": "Succeeded"

The deployment takes 10-15 minutes; Azure Firewall Basic provisioning is the dominant tail.

Bicep gotcha: UDR next-hop must be read from the firewall resource property¶

The UDR's nextHopIpAddress is computed from firewall.properties.ipConfigurations[0].properties.privateIPAddress, which is the firewall instance's actual private IP. Do not compute it from the subnet CIDR (e.g., cidrHost(firewallSubnetPrefix, 0)) — that returns the Azure-reserved gateway address (.1), not the firewall instance address. Azure reserves the first four IPs of every subnet (.0 through .3), so in a /26 AzureFirewallSubnet the firewall instance lands at .4 (e.g., 10.80.2.4 if the subnet is 10.80.2.0/26). Pointing the UDR at .1 (the gateway, which is not a virtual appliance) silently breaks default-route egress: every replica's outbound packet is dropped at the next-hop because the target does not exist as a virtual appliance, the new revision lingers in PullingImage with no ImagePullFailure (because the packet never reaches ACR to get an error response), and the symptom is "v1 revision stuck pulling forever, no useful error in any log table." Reading the firewall resource property is correct in all cases regardless of subnet size or address family.

Build images and lock down ACR¶

bash labs/acr-network-path-firewall-allowlist/trigger.sh

The trigger script runs:

az acr build --registry "$ACR_NAME" --image "${IMAGE_REPO}:v1"        --build-arg "BUILD_TAG=v1"        --file workload/Dockerfile workload
az acr build --registry "$ACR_NAME" --image "${IMAGE_REPO}:v-broken"  --build-arg "BUILD_TAG=v-broken"  --file workload/Dockerfile workload
az acr build --registry "$ACR_NAME" --image "${IMAGE_REPO}:v-recover" --build-arg "BUILD_TAG=v-recover" --file workload/Dockerfile workload

az acr network-rule add --name "$ACR_NAME" --ip-address "$FW_PIP"
az acr update --name "$ACR_NAME" --default-action Deny --allow-trusted-services false

az containerapp registry set --name "$APP_NAME" --resource-group "$RG" \
    --server "$ACR_LOGIN_SERVER" --username "$ACR_USERNAME" --password "$ACR_PASSWORD"

az containerapp update --name "$APP_NAME" --resource-group "$RG" \
    --image "${ACR_LOGIN_SERVER}/${IMAGE_REPO}:v1"

Expected output (from the live run): three az acr build jobs complete in sequence (~30 seconds each, depending on layer cache), az acr show --query networkRuleSet --output json shows defaultAction=Deny + ipRules=[{firewall-public-ip}], and the v1 revision becomes Healthy within ~60 seconds of the az containerapp update call.

Verify the healthy baseline is live¶

bash labs/acr-network-path-firewall-allowlist/verify.sh

The verify script proves three independent signals simultaneously hold:

1. The latest revision's healthState is Healthy and provisioningState is Provisioned.
2. az acr show --query networkRuleSet confirms defaultAction=Deny + ipRules contains the firewall public IP.
3. GET https://<app>.<env>.azurecontainerapps.io/ returns JSON with build_tag=v1, proving a fresh pull of v1 traversed the firewall and was admitted by ACR's firewall layer. The workload code reads BUILD_TAG from the image's ENV (set by the Dockerfile's ARG/ENV BUILD_TAG=${BUILD_TAG}), and trigger.sh deliberately does not override this via --set-env-vars. Image identity is therefore the proof: if the v1 image was not actually pulled and started, the workload cannot return build_tag=v1.

All three must hold for the topology to be Scenario A baseline at the workload layer. The script exits non-zero on any mismatch.

Expected pattern (from the live run on 2026-06-06):

[verify] latest revision: ca-acrfwallow-t2itdu--0000001 healthState=Healthy provisioningState=Provisioned
[verify] inspecting ACR network rule set on acracrfwallowt2itdu
[verify]   defaultAction          = Deny
[verify]   networkRuleBypassOptions = None
[verify]   ipRules                  = 20.196.208.15
[verify] / response:
{
    "build_tag": "v1",
    "message": "ACR firewall allowlist lab workload"
}
[verify] PASS: revision ca-acrfwallow-t2itdu--0000001 is Healthy
[verify] PASS: ACR locked down (Deny default + ipRules=[20.196.208.15])
[verify] PASS: / returns build_tag=v1 (proves fresh pull of v1 through the firewall)
[verify] PASS: Scenario A BASELINE confirmed.

Falsify the hypothesis¶

bash labs/acr-network-path-firewall-allowlist/falsify.sh

The falsification script proves the workload-layer signal flips with the IP allowlist's contents while the already-running revision continues to report Healthy. Every check is a hard fail — the script exits non-zero (and prints why) at the first sign that any link in the chain did not behave as predicted, so a PASS from falsify.sh is itself the proof that all of the following held:

1. Calls / and asserts build_tag=v1 (baseline).
2. Removes the firewall public IP from ACR's ipRules via az acr network-rule remove --ip-address $FW_PIP.
3. Waits 60 seconds for ACR's firewall changes to propagate.
4. Deploys the v-broken image: az containerapp update --image ${ACR}/${IMAGE}:v-broken.
5. Polls for up to 5 minutes for the new revision to surface as provisioningState=Failed, healthState=Unhealthy, or healthState=None. Hard fail if the broken revision provisions successfully — that would mean the controlled variable is not what we claim.
6. Confirms the OLD v1 revision is still Healthy and / still returns build_tag=v1. Hard fail if v1 stopped serving — that would mean the cached-image guarantee Container Apps depends on is broken.
7. Hard fail if ContainerAppSystemLogs_CL does not record the smoking-gun DENIED message naming the firewall PIP as the rejected source within ~2.5 minutes (5 × 30s retry loop). Without this row, the script can only claim "the pull failed", not "the pull failed BECAUSE ACR rejected the firewall PIP" — the latter is the entire point of Scenario A.
8. Re-adds the firewall public IP to ACR's ipRules via az acr network-rule add --ip-address $FW_PIP.
9. Waits 60 seconds for ACR's firewall changes to propagate.
10. Deploys the v-recover image: az containerapp update --image ${ACR}/${IMAGE}:v-recover.
11. Polls for up to 5 minutes for the recovery revision to reach healthState=Healthy and asserts / returns build_tag=v-recover. Hard fail on either condition.

If falsify.sh exits 0, then by construction (a) the broken-window pull failed, (b) the v1 revision kept serving from cached layers, (c) the failure logs in ContainerAppSystemLogs_CL name the firewall PIP as the rejected source, and (d) the recovery pull succeeded after the firewall PIP was re-added. That four-way evidence — failure + cached-serve + named-IP smoking gun + symmetric recovery — is the complete falsification of any alternative hypothesis (e.g., DNS, managed-identity confound, PE-NIC IP) for fresh-pull behavior on this topology.

Subtle wrinkle: v-broken eventually pulls successfully after step 8¶

The platform image puller has a retry loop with backoff. During the 60-second wait between az acr network-rule add (step 8) and az containerapp update --image ${ACR}/${IMAGE}:v-recover (step 10), the puller's most recent retry attempt for v-broken runs against an ACR with the firewall PIP back in ipRules — and succeeds. This means the v-broken revision's terminal state at the end of falsify.sh is technically Healthy, not Failed. The broken-window evidence is nevertheless intact because the script's hard-fail gates fired before step 8:

1. Step 5 hard-fail — the script will not proceed past step 5 unless the v-broken revision is observed in a Failed/Unhealthy/None state at least once during the broken window. By the time step 8 re-adds the IP, the broken state has already been recorded.
2. Step 7 hard-fail — the smoking-gun query against ContainerAppSystemLogs_CL runs before step 8 and requires the DENIED row naming the firewall PIP as the rejected source. That row is durable in LAW regardless of what the v-broken revision does afterward.
3. Cached-serve hard-fail (step 6) — the v1 revision stayed Healthy throughout the broken window. If v1 had stopped serving (e.g., the cached layers were evicted, or the broken revision somehow took traffic), the script would have exited 1 there.
4. Recovery proof (step 11) — the v-recover revision is the unambiguous recovery proof: a freshly tagged image that the platform had never seen before, deployed after step 8's IP re-add, that pulls cleanly and serves build_tag=v-recover.

This is an honest reflection of how Container Apps' image puller actually behaves in the field: it does not give up after one failure; it keeps retrying. The lab proof relies on the time-windowed failure logs (the durable evidence — the DENIED row pinned to the broken window timestamp) and the v-recover deploy (the unambiguous fresh-pull recovery), not on a permanent failure of the v-broken revision. A "stronger" version of the lab that guarantees v-broken stays Failed permanently would require either (a) a non-existent image tag (which is a different failure mode — ManifestUnknown, not DENIED) or (b) tearing down the broken-window window so quickly the puller cannot retry, which is logistically fragile. The current design — durable broken-window logs + unambiguous recovery deploy + hard-fail gates that exit before the retry would mask the broken state — is the cleanest honest reproduction.

Inspect system evidence¶

az containerapp logs show \
    --name "$APP_NAME" \
    --resource-group "$RG" \
    --type system \
    --tail 50

A more direct query, useful when reproducing the experiment, is the KQL pack pattern for ACR pull-path observability with the broken-window timestamp range:

ContainerAppSystemLogs_CL
| where ContainerAppName_s == "<your-app-name>"
| where TimeGenerated between (datetime(<broken-window-start>) .. datetime(<broken-window-end>))
| where Reason_s in ("PullingImage", "PulledImage", "ContainerTerminated", "ImagePullFailed", "ImagePullUnauthorized", "BackOff")
| project TimeGenerated, Reason_s, Type_s, Log_s
| order by TimeGenerated asc

In the live run on 2026-06-06, this query returned Reason_s=ContainerTerminated rows whose Log_s field begins with Container 'app' was terminated with exit code '' and reason 'ImagePullFailure'. Image acracrfwallowt2itdu.azurecr.io/firewall-allowlist-lab:v-broken not found in registry. ErrorMessage=Image not found or access denied. Error: {"errors":[{"code":"DENIED","message":"client with IP '20.196.208.15' is not allowed access, refer https://aka.ms/acr/firewall to grant access"}]} — the value 20.196.208.15 in the DENIED message is the firewall's public IP captured in pip-afw-acrfwallow-t2itdu's properties.ipAddress. The smoking gun is the DENIED message itself: ACR's firewall layer rejects the inbound flow, names the source IP it observed, and that source IP is provably the firewall's SNAT public IP — not the replica IP, not any other Azure-managed IP.

4) Experiment Log¶

Pre-fix evidence (H1 confirmed)¶

• [Observed] labs/acr-network-path-firewall-allowlist/evidence/03-acr-network-rules-pre-fix.json shows publicNetworkAccess=Enabled, networkRuleBypassOptions=None, networkRuleSet.defaultAction=Deny, and an empty ipRules list after the firewall public IP was removed.
• [Observed] labs/acr-network-path-firewall-allowlist/evidence/06-system-logs-pre-fix.json and labs/acr-network-path-firewall-allowlist/evidence/08-h1-failure-window.json capture the broken v-broken pull window, including the DENIED/403 payload that names the rejected firewall public IP directly in the ACR error message.
• [Observed] labs/acr-network-path-firewall-allowlist/evidence/02-revision-list-pre-fix.json records the failed v-broken revision state while the cached v1 revision remains present.
• [Measured] labs/acr-network-path-firewall-allowlist/evidence/05-baseline-success-window.json proves the baseline path was non-vacuous by showing successful PullingImage and PulledImage markers before the H1 toggle.
• [Inferred] Taken together, these files isolate the ACR allowlist entry as the broken-window trigger: the registry stayed public-but-locked-down, the firewall IP was absent from ipRules, and the next fresh pull failed with the firewall IP named in the DENIED response.

Post-fix evidence (H2 confirmed — falsification step)¶

• [Observed] labs/acr-network-path-firewall-allowlist/evidence/09-acr-network-rules-post-fix.json shows the firewall public IP restored as the sole ACR allowlist entry while defaultAction=Deny and networkRuleBypassOptions=None stay constant.
• [Observed] labs/acr-network-path-firewall-allowlist/evidence/10-revision-list-post-fix.json records a healthy v-recover revision and preserves the cached v1 continuity anchor needed for the workload-silence proof.
• [Measured] labs/acr-network-path-firewall-allowlist/evidence/12-h2-recovery-window.json contains both PullingImage and PulledImage for v-recover and the post-fix / response reports build_tag=v-recover.
• [Observed] labs/acr-network-path-firewall-allowlist/evidence/11-app-spec-post-fix.json captures the explicit post-deactivation v-broken snapshot. After deactivation, Azure can normalize the old broken revision to an inactive control-plane object (active=false, replicas=0, runningState=Stopped, sometimes healthState=Healthy, provisioningState=Provisioned) or omit it from later revision listings. That is expected and does not indicate retroactive repair. The non-retroactive-repair proof is that H2 success markers belong only to v-recover and no H2 PulledImage row references v-broken.
• [Inferred] This closes the falsification loop required by the 16-concept lab methodology: removing one allowlist entry caused the H1 failure, re-adding the same entry restored H2 recovery, and the cached healthy revision stayed silent throughout the broken window.

Status: Canonical Phase B evidence pack refreshed live in koreacentral on 2026-06-29 with Azure CLI 2.79.0. The committed evidence pack and gate JSONs reflect this 2026-06-29 run. The detailed step table below remains a historical 2026-06-06 narrative reference.

Expected Evidence¶

Observed Evidence (Live Azure Test — historical 2026-06-06 narrative)¶

Live reproduction in koreacentral on 2026-06-06 with Azure CLI 2.79.0. RG rg-acr-firewall-allowlist-lab, ACR acracrfwallowt2itdu (publicNetworkAccess=Enabled, defaultAction=Deny, networkRuleBypassOptions=None), Container App ca-acrfwallow-t2itdu, firewall public IP 20.196.208.15, firewall private IP 10.80.2.4, image tags firewall-allowlist-lab:v1, :v-broken, :v-recover (each with a distinct BUILD_TAG build-arg producing distinct digests). All three states below were captured by verify.sh and falsify.sh in a single uninterrupted run.

Baseline (verify.sh and falsify.sh step 1)¶

The Container App is running revision --0000001 on the freshly pulled v1 image:

{
    "build_tag": "v1",
    "message": "ACR firewall allowlist lab workload"
}

ACR's network rule set is locked down to allow only the firewall's public IP:

{
  "defaultAction": "Deny",
  "ipRules": [
    {
      "action": "Allow",
      "ipAddressOrRange": "20.196.208.15"
    }
  ]
}

The fact that v1 is serving traffic proves a fresh pull traversed the firewall (egress → SNAT to 20.196.208.15 → ACR firewall layer admits because 20.196.208.15 is in ipRules → image manifest + layers download).

Broken window (falsify.sh steps 2-6, after az acr network-rule remove --ip-address 20.196.208.15)¶

ACR's ipRules is now empty:

{
  "defaultAction": "Deny",
  "ipRules": []
}

Deploying the v-broken image surfaces revision --0000002 with provisioningState=Provisioned healthState=None. The platform image puller's failed pull is captured in ContainerAppSystemLogs_CL with this verbatim payload (timestamps in UTC, Type_s shown for severity):

TimeGenerated:             2026-06-05T23:44:32.875Z
Reason_s:                  ContainerTerminated
Type_s:                    Warning
Log_s:                     Container 'app' was terminated with exit code '' and reason 'ImagePullFailure'.
                           Image acracrfwallowt2itdu.azurecr.io/firewall-allowlist-lab:v-broken not found in registry.
                           ErrorMessage=Image not found or access denied.
                           Error: {"errors":[{"code":"DENIED","message":"client with IP '20.196.208.15' is not allowed access, refer https://aka.ms/acr/firewall to grant access"}]},
                           GenevaTraceId=c94333c7d6f4d04ea14fcfa848b95aa0.

The client with IP '20.196.208.15' is not allowed access substring is the smoking gun. The IP 20.196.208.15 in the DENIED message is provably the firewall's public IP — az network public-ip show --name pip-afw-acrfwallow-t2itdu --resource-group rg-acr-firewall-allowlist-lab --query ipAddress --output tsv returns the same value. The chain of inference is direct:

1. The image-pull HTTPS connection from the Container Apps workload subnet to acracrfwallowt2itdu.azurecr.io was forced through Azure Firewall by the 0.0.0.0/0 UDR on snet-aca. (The Container Apps workload subnet hosts both user replicas and platform-managed components, including the component that performs image pulls; Container Apps does not expose which specific component owns a given source IP, so this lab uses "Container Apps workload-subnet source IP" rather than "replica IP" for the pull-path source.)
2. Azure Firewall SNAT'd the source IP from a 10.80.0.0/23 workload-subnet RFC1918 IP (the pre-SNAT source the firewall observed on the inbound flow) to the firewall's public IP 20.196.208.15.
3. ACR's firewall layer received the inbound TCP connection with source IP 20.196.208.15.
4. ACR's networkRuleSet evaluated defaultAction=Deny + ipRules=[] and rejected the connection with HTTP 403.
5. ACR's HTTP 403 response body contained the literal text client with IP '20.196.208.15' is not allowed access.
6. Container Apps captured the 403 response and surfaced it verbatim in ContainerAppSystemLogs_CL.

The IP that appears in the message is therefore directly observable evidence that the SNAT mechanism worked correctly (without it, ACR would have observed a different IP and the message would have named that IP instead) AND that ACR's ipRules is the single controlled variable for fresh-pull behavior.

During the same broken window, falsify.sh step 6 confirmed the v1 revision was still serving traffic:

[falsify]   v1 revision ca-acrfwallow-t2itdu--0000001: healthState=Healthy active=true
[falsify] (broken-window) / response:
{
    "build_tag": "v1",
    "message": "ACR firewall allowlist lab workload"
}

This is the central Container Apps finding: removing the firewall PIP from ACR's allowlist immediately breaks fresh pulls but does not affect the already-running revision, which continues serving from cached image layers. Pull-path failure is a new-revision failure mode, not an existing-revision failure mode.

Broken-window pull retry timeline (subtle wrinkle)¶

The full system-log timeline for revision --0000002 (v-broken) during the broken window is below. Note the alternating PullingImage (retry) and ContainerTerminated (DENIED) cycles, and the eventual successful pull at 23:45:45 after falsify.sh step 7 re-added the firewall PIP:

2026-06-05T23:44:17.886Z  KEDAScalersStarted   Scaler external-push is built
2026-06-05T23:44:17.886Z  KEDAScalersStarted   KEDA is starting a watch for revision 'ca-acrfwallow-t2itdu--0000002'
2026-06-05T23:44:18.849Z  AssigningReplica     Replica '--0000002-58bc4b94d5-dtmhq' has been scheduled
2026-06-05T23:44:18.849Z  RevisionReady        Successfully provisioned revision 'ca-acrfwallow-t2itdu--0000002'
2026-06-05T23:44:32.875Z  ContainerTerminated  ImagePullFailure ... DENIED, client with IP '20.196.208.15' is not allowed access
2026-06-05T23:44:32.875Z  PullingImage         Pulling image 'acracrfwallowt2itdu.azurecr.io/firewall-allowlist-lab:v-broken'
2026-06-05T23:44:34.842Z  PullingImage         (retry)
2026-06-05T23:44:34.842Z  ContainerTerminated  ImagePullFailure ... DENIED, client with IP '20.196.208.15' is not allowed access
2026-06-05T23:44:44.846Z  ContainerTerminated  ImagePullFailure ... DENIED, client with IP '20.196.208.15' is not allowed access
2026-06-05T23:44:44.846Z  PullingImage         (retry)
2026-06-05T23:45:04.674Z  PullingImage         (retry)
2026-06-05T23:45:04.674Z  ContainerTerminated  ImagePullFailure ... DENIED, client with IP '20.196.208.15' is not allowed access
2026-06-05T23:45:45.084Z  PullingImage         (retry)
2026-06-05T23:45:45.706Z  PulledImage          Successfully pulled image ... in 1.53s. Image size: 49283072 bytes.
2026-06-05T23:45:45.706Z  ContainerStarted     Started container 'app'
2026-06-05T23:45:59.825Z  RevisionCreation     Creating a new revision: ca-acrfwallow-t2itdu--0000003
2026-06-05T23:46:55.107Z  ContainerTerminated  ManuallyStopped  (after v-recover deploy promoted --0000003)

The platform image puller retries the pull every ~10-30 seconds with exponential backoff. The four DENIED rows between 23:44:32 and 23:45:04 are the durable broken-window evidence — each contains the smoking-gun message naming the firewall public IP. The successful pull at 23:45:45 happened after falsify.sh step 7 re-added the firewall PIP to ipRules (which executes around 23:45:30 based on the script's timing), so the v-broken revision's terminal state is Healthy rather than permanently Failed. The lab proof relies on the time-windowed DENIED rows being durably captured in Log Analytics — they are, with full timestamps and verbatim body text — not on v-broken's terminal state.

Recovery (falsify.sh step 10, after az containerapp update --image ${ACR}/${IMAGE}:v-recover)¶

The Container App is running revision --0000003 on the freshly pulled v-recover image:

{
    "build_tag": "v-recover",
    "message": "ACR firewall allowlist lab workload"
}

The v-recover image is a different image (different BUILD_TAG build-arg → different content → different digest) that the platform had never pulled before. A fresh pull of it succeeded, with ACR's ipRules containing the firewall PIP again. The bidirectional falsification is closed: removing the firewall PIP breaks fresh pulls; re-adding it restores them.

Observed Evidence (Portal Captures — historical 2026-06-06)¶

A live reproduction on 2026-06-06 captured the full Scenario A topology and the workload-path falsification surface from the Azure Portal.

Environment

[Observed] Container Apps environment Networking blade: workload-profile environment is attached to VNet vnet-acrfwallow-t2itdu, infrastructure subnet snet-aca, Virtual IP External. This is the foundation of the workload-path lab — the Container App's outbound traffic leaves from this subnet and hits the UDR forcing it through the firewall.

[Observed] VNet vnet-acrfwallow-t2itdu Overview blade: address space 10.80.0.0/16, DNS servers: Azure provided DNS service (no custom DNS server is configured). DNS is not the controlled variable in this lab — both ACR FQDNs (login and data) resolve to public IPs via Azure DNS, and the firewall's application rule allows them by FQDN. The capture also enumerates the three subnets — snet-aca (10.80.0.0/23), AzureFirewallSubnet (10.80.2.0/26), AzureFirewallManagementSubnet (10.80.2.64/26) — which together implement the Path A topology.

[Observed] ACR acracrfwallowt2itdu → Networking → Public access tab: Public network access = Selected networks, with a single firewall IP entry 20.196.208.15 in the Firewall section. This is the central controlled variable of the lab. Removing this single IP from the allowlist (the broken-window state) caused ACR's firewall layer to reject all inbound flows from the Container Apps replica with HTTP 403; re-adding it restored fresh-pull behavior. The capture was taken in the recovered state (firewall PIP re-added).

[Observed] Azure Firewall afw-acrfwallow-t2itdu Overview blade: SKU = Basic, public IP 20.196.208.15, private IP 10.80.2.4, status Succeeded. The public IP value here is the same value that must appear in ACR's ipRules (above) — proving that the firewall PIP is the single shared variable across the two resources. The private IP 10.80.2.4 is what snet-aca's UDR points at as the next-hop; the value comes from firewall.properties.ipConfigurations[0].properties.privateIPAddress, NOT from subnet arithmetic.

[Observed] Firewall policy afwp-acrfwallow-t2itdu → Application rules blade: rule collection acr-and-platform-allow containing four application rules — allow-acr-login (HTTPS to acracrfwallowt2itdu.azurecr.io), allow-acr-data (HTTPS to acracrfwallowt2itdu.koreacentral.data.azurecr.io), allow-mcr (HTTPS to mcr.microsoft.com and *.data.mcr.microsoft.com for system images), and allow-acs-mirror (HTTPS to acs-mirror.azureedge.net and packages.aks.azure.com for Container Apps platform components). Source addresses are restricted to 10.80.0.0/23 (the snet-aca workload subnet). This is the firewall's outbound allowlist; combined with ACR's inbound allowlist (capture 03), the two together implement Scenario A's symmetric pair: outbound from the firewall to ACR is allowed by FQDN; inbound to ACR from the firewall's PIP is allowed by IP.

[Observed] Route table rt-acrfwallow-t2itdu → Routes blade: single route default-via-afw with address prefix 0.0.0.0/0, next-hop type Virtual appliance, next-hop IP 10.80.2.4. This is the UDR that forces every snet-aca outbound flow through the firewall. The next-hop IP 10.80.2.4 matches the firewall's private IP from capture 04 (proving the route table and firewall agree on the rendezvous point). Note the next-hop value is .4, not .1: in a /26 AzureFirewallSubnet, Azure reserves .0 through .3, so the firewall instance lands at .4. Pointing the UDR at .1 (the gateway, which is not a virtual appliance) would silently break egress — which is exactly what the original Bicep did before the fix.

[Observed] Container App ca-acrfwallow-t2itdu → Revisions and replicas blade, Active revisions tab: a single revision --0000003 (v-recover) is shown with Running (at max) status, 100% traffic, and 1 replica. This is the recovery revision that succeeded after az acr network-rule add re-added the firewall PIP — proving the fresh-pull path works when ACR's ipRules contains the firewall's SNAT public IP.

[Observed] Same blade, Inactive revisions tab: three deactivated revisions are listed — --0000002 (v-broken, deactivated when v-recover took 100% traffic), --0000001 (v1, deactivated when v-broken was promoted), and --y5eac1h (the bootstrap revision created automatically by az containerapp create from the mcr.microsoft.com/k8se/quickstart placeholder image, deactivated when our first user image v1 was deployed). The four revisions shown here — bootstrap --y5eac1h, then user-driven --0000001, --0000002, and --0000003 — match the create-time placeholder plus the three deploy actions in trigger.sh and falsify.sh. The presence of independently-named revisions for each tag is itself evidence of the experimental design: each user-driven revision corresponds to a distinct image digest, and each digest's deploy outcome was determined entirely by the firewall PIP's presence in ACR's ipRules at deploy time.

[Inferred] The seven captures (counting 07a + 07b as a single revision-blade pair), taken in time order from the Portal, document Scenario A's required topology (Container Apps environment integrated into the VNet — capture 01; VNet with three subnets including AzureFirewallSubnet and AzureFirewallManagementSubnet — capture 02; ACR public surface locked down with the firewall PIP as the only allow-listed source IP — capture 03; Azure Firewall Basic with the same PIP that appears in ACR's ipRules — capture 04; firewall policy with application rules allowing ACR login + data + MCR + acs-mirror by FQDN — capture 05; UDR forcing snet-aca egress through the firewall's private IP — capture 06) and prove the lab's central finding via the revision history (captures 07a + 07b): the recovery revision --0000003 is the only Active revision and is Healthy after the firewall PIP was re-added, while the three earlier revisions sit on the Inactive tab as historical evidence of the experimental sequence. The Portal captures complement the CLI evidence above — together they show that Scenario A's controlled variable lives on the ACR resource (networkRuleSet.ipRules), the value that variable must hold lives on the firewall resource (publicIPAddresses[0].properties.ipAddress), and the symptom of removing the value lives in ContainerAppSystemLogs_CL (the DENIED message naming the firewall PIP). This is the cleanest single-controlled-variable falsification in the 5-lab ACR network path series — no DNS confound, no managed-identity token-exchange confound, no PE-NIC IP confound.

Clean Up¶

bash labs/acr-network-path-firewall-allowlist/cleanup.sh

Which runs:

az group delete --name "$RG" --yes --no-wait

Azure Firewall Basic + its two public IPs is the dominant cost (~$24/day) — do not leave the lab running between sessions. ACR Premium is the second cost (~$1.67/day). Total run cost for a 2-3 hour reproduction is ~$3-4 USD.

Related Playbook¶

• Private Endpoint DNS Failure — covers the Private Endpoint DNS failure family (Path B / Scenarios D and E); this Path A lab is the public-with-firewall alternative to those private-path topologies.
• UDR and NSG Egress Blocked — covers UDR/NSG misconfigurations that break Container Apps egress; the Bicep gotcha in this lab (UDR next-hop pointing at the subnet gateway instead of the firewall instance) is a specific instance of that family.

See Also¶

• ACR Network Path Selection — the platform decision page this lab implements; see Path A: Public ACR via Firewall / NVA
• ACR Network Path B Lab — PE Direct — the Private Endpoint topology that is the realistic alternative to this lab's public-with-firewall topology. Path B removes the firewall from the image-pull data path entirely (PE traffic does not traverse the firewall by default), at the cost of requiring privatelink.azurecr.io zone management and a PE NIC subnet.
• ACR Network Path D Lab — Record-Level Zone Authority — Scenario D, the record-CONTENT failure on the Path B private path. Lab 3 used managed identity for ACR auth, which prevented a clean broken-window fresh-pull proof; this Path A lab is the first in the series to cleanly prove fresh-pull behavior because admin-credential auth removes the control-plane token-exchange confound.
• ACR Network Path E Lab — DNS Forwarder Bypass — Scenario E, the resolver-topology failure on the Path B private path. Same managed-identity-confound caveat as Lab 3.
• Use Azure Firewall with Azure Container Apps (Microsoft Learn) — official reference for the firewall-integration pattern this lab implements.
• Configure public network access for an Azure container registry (Microsoft Learn) — official reference for ACR's networkRuleSet selected-networks model.

Sources¶

• Use Azure Firewall with Azure Container Apps (Microsoft Learn) — official guidance for the Path A pattern, including the UDR + firewall private-IP-as-next-hop requirement.
• Configure public network access for an Azure container registry (Microsoft Learn) — ACR's networkRuleSet model, including defaultAction, ipRules, and networkRuleBypassOptions.
• Configure rules to access an Azure Container Registry behind a firewall (Microsoft Learn) — the FQDN list a firewall must allow for ACR pulls (login server, regional data endpoint).
• Azure Firewall SNAT private IP address ranges (Microsoft Learn) — explains Azure Firewall's SNAT behavior, why every outbound flow's source IP becomes the firewall's public IP from ACR's perspective.
• Deploy and configure Azure Firewall Basic and policy (Microsoft Learn) — the Basic SKU's AzureFirewallManagementSubnet requirement and management-NIC behavior.
• Networking in Azure Container Apps (Microsoft Learn) — Container Apps VNet integration, workload-profile subnet requirements (delegation, /23 minimum), and inheritance of UDRs.
• Revisions in Azure Container Apps (Microsoft Learn) — revision lifecycle, including the cached-image-layer behavior that lets an already-running revision survive an ACR firewall change.
• Authenticate with an Azure container registry (Microsoft Learn) — admin user vs. managed identity; the auth choice that makes this lab's single-controlled-variable design possible.