ACR Network Path D — Record-Level Zone Authority Lab¶

Reproduce Scenario D from ACR Network Path Selection: the Container Apps environment uses default Azure DNS (168.63.129.16) for resolution, the ACR Private Endpoint is healthy, and privatelink.azurecr.io is correctly linked to the VNet — but the per-region data A record (<registry>.<region>.data) has been deleted from the zone. The registry login FQDN still resolves to the PE NIC's RFC1918 IP (the registry-group A record is intact); the data FQDN, however, does not silently flip to a public IP in this topology — Azure DNS returns NXDOMAIN because the linked Private DNS Zone is authoritative for its namespace.

This lab makes a non-obvious Azure Container Apps behavior falsifiable: deleting one A record from the linked private DNS zone produces no immediate revision-health impact on the already-running revision. The already-running revision stays Healthy and continues to serve traffic unchanged from cached image layers, while a four-layer probe (DNS → TCP → TLS → HTTP) run from inside the replica clearly shows the asymmetry — topology_class=data_nxdomain, with the data endpoint returning socket.gaierror: [Errno -2] Name or service not known at the DNS layer (before any TCP/TLS/HTTP layer can run), while the registry endpoint still returns DNS=private/HTTP 401 from the ACR backend. Pull-path observability is therefore not a reliable early warning for Scenario D in Container Apps in this scenario. The /probe endpoint in this lab is the workload-side signal that clearly moves.

Lab Metadata¶

Lab position¶

This lab is part of the 5-lab ACR network path series that reproduces the five distinct network paths a Container App can take to reach ACR. See ACR Network Path Selection for the conceptual taxonomy that names and orders all five paths.

1) Background¶

Azure Container Apps can reach ACR through several network paths — public via firewall, Private Endpoint direct, Private Endpoint with forced inspection, or one of two DNS failure scenarios. The ACR Network Path Selection page documents all of them.

Scenario D (record-level zone authority) is the record-CONTENT failure class. Unlike Scenario E (which breaks the resolver path itself), Scenario D leaves the resolver path entirely correct: the Container Apps VNet sends queries to Azure DNS at 168.63.129.16 (the default when no custom DNS server is configured on the VNet), Azure DNS sees that the VNet is linked to privatelink.azurecr.io, and Azure DNS treats the linked zone as authoritative for that namespace. The failure is in the zone's CONTENT: one or more A records are missing, so Azure DNS answers the query with NXDOMAIN.

For ACR specifically, the zone needs BOTH the <registry> A record (for the registry login endpoint) AND the <registry>.<region>.data A record (for the regional data endpoint). The Private Endpoint's privateDnsZoneGroup auto-populates both records on baseline deploy, so this asymmetry only appears when someone (a misconfigured automation pipeline, a careless manual cleanup, a stale IaC template that only created one record, etc.) deletes the data record. The result in the default Azure DNS topology this lab reproduces: registry private, data NXDOMAIN.

Three properties make Scenario D worth reproducing as a hands-on lab:

• Azure Private DNS Zones linked to a VNet are authoritative for their namespace. The Private DNS Zone IP-substitution mechanism documented in Azure Private Endpoint DNS configuration works at the zone level: once a VNet is linked to a private zone, Azure DNS treats every query for a name in that namespace as a query against the zone, and answers with the zone's record if one exists or with NXDOMAIN if no record exists. There is no automatic fallthrough to the public CNAME chain from Azure DNS itself. This means a missing <registry>.<region>.data record produces a hard DNS failure for the application (the data FQDN is unaddressable), not a silent flip to a public IP. The platform doc's older "split-brain" framing — login private, data public — is accurate only for a custom-DNS topology where a customer-managed resolver is explicitly configured to fall back to public DNS on NXDOMAIN; that is a different topology than this lab.
• In Azure Container Apps in this reproduction, deleting the data A record produces no immediate revision-health impact on the already-running revision. This is the central finding of the lab, and it mirrors the Scenario E lab's central finding for a different root cause. Empirically, when the <registry>.<region>.data record is deleted from privatelink.azurecr.io, the already-running revision continues to report healthState=Healthy and serves traffic from the already-cached image layers. The workload, however, immediately sees the broken record — the 4-layer probe's data-FQDN DNS layer flips from class=private (PE NIC IP) to class=null with error="gaierror: [Errno -2] Name or service not known", and all downstream layers (TCP, TLS, HTTP) are skipped because there is no IP to connect to.
• Scenario D is distinct from Scenario E in a way that matters operationally. Scenario E breaks the resolver path so the ENTIRE ACR namespace resolves publicly; Scenario D leaves the resolver path correct but breaks a SINGLE record so only PART of the namespace fails to resolve. Both can produce workload-layer failures without revision-health impact, but they require DIFFERENT FIXES — fix the forwarder for E, fix the zone record for D. A lab that can distinguish them on the wire (this lab vs. the sibling Scenario E lab) is operationally useful because the diagnostic signal differs: Scenario E shows topology_class=both_public (custom DNS bypasses Azure DNS entirely), Scenario D in default Azure DNS shows topology_class=data_nxdomain (Azure DNS is authoritative and answers NXDOMAIN for the missing record).

The lab's topology_class taxonomy enumerates the observable outcomes:

Architecture¶

flowchart TD
    REPLICA[Container App replica<br/>workload profile subnet 10.70.0.0/23]
    PROBE[Workload /probe endpoint<br/>4-layer probe DNS/TCP/TLS/HTTP<br/>for BOTH FQDNs]
    CACHED[Already-cached image layers<br/>no pull during broken window]
    AZDNS[Azure DNS<br/>168.63.129.16<br/>default VNet DNS - no custom forwarder]
    ZONE[Private DNS Zone<br/>privatelink.azurecr.io<br/>linked to VNet - AUTHORITATIVE]
    ZONEREG["A record: registry<br/>HELD CONSTANT"]
    ZONEDATA["A record: registry.region.data<br/>HEALTHY: present<br/>BROKEN: DELETED"]
    PEREG[PE NIC registry IP<br/>10.70.4.x]
    PEDATA[PE NIC data IP<br/>10.70.4.x]
    ACR[Azure Container Registry<br/>Premium<br/>publicNetworkAccess=Disabled]
    NXDOMAIN[NXDOMAIN<br/>socket.gaierror in replica<br/>data FQDN unaddressable]
    ACRREGBE[ACR registry backend<br/>HTTP 401 auth challenge]
    ACRDATABE[ACR data backend<br/>HTTP 403 default for unauth /v2/]

    REPLICA --> PROBE
    REPLICA --> CACHED
    PROBE -->|registry FQDN| AZDNS
    PROBE -->|data FQDN| AZDNS
    AZDNS --> ZONE
    ZONE --> ZONEREG
    ZONE --> ZONEDATA
    ZONEREG -->|returns PE NIC IP| PEREG
    ZONEDATA -->|HEALTHY: returns PE NIC IP| PEDATA
    ZONEDATA -.->|BROKEN: record gone, zone authoritative| NXDOMAIN
    PEREG --> ACRREGBE
    PEDATA --> ACRDATABE
    ACRREGBE --> ACR
    ACRDATABE --> ACR

    classDef workload fill:#e1f5ff,stroke:#0078d4
    classDef brokenpath fill:#fde7e9,stroke:#a4262c
    classDef registry fill:#e5f5e5,stroke:#107c10
    classDef neutral fill:#f3f2f1,stroke:#605e5c

    class REPLICA,PROBE workload
    class NXDOMAIN brokenpath
    class ACR,PEREG,PEDATA,ACRREGBE,ACRDATABE registry
    class CACHED,ZONEREG,ZONEDATA neutral

The solid arrows are the workload DNS path and image data path in the HEALTHY baseline state. The dotted arrow is what the workload sees for the data FQDN only in the BROKEN state after the data A record is deleted — Azure DNS returns NXDOMAIN because the linked Private DNS Zone is authoritative for that namespace, and the application code surfaces this as socket.gaierror. The registry login FQDN is unchanged across both states. Note that the registry backend returns HTTP 401 (auth challenge) and the data backend returns HTTP 403 (default response to an unauthenticated /v2/ probe) even in the BASELINE state — the HTTP status difference between registry and data is the data endpoint's default behavior, not a discriminator for the broken vs. healthy state. The reliable broken-state discriminator is the DNS layer's NXDOMAIN/gaierror on the data FQDN. The neutral-colored Already-cached image layers node captures the lab's central Container Apps finding: during the broken window, the lab does not trigger a fresh pull, so the replica continues to run from the already-cached layers without re-pulling, and revision healthState stays Healthy. See §"Why we do not script a broken-window fresh pull" for the scope boundary.

2) Hypothesis¶

IF the Container Apps environment uses default Azure DNS (no custom VNet DNS server), the ACR Private Endpoint exists with publicNetworkAccess: Disabled, and privatelink.azurecr.io is linked to the VNet with BOTH the <registry> and <registry>.<region>.data A records correctly populated by the PE's privateDnsZoneGroup, THEN:

• with both records present, a 4-layer probe from inside the replica returns topology_class=both_private, with both registry and data resolving to PE NIC RFC1918 IPs; HTTP layer returns 401 from the ACR registry backend (auth challenge for /v2/) and 403 from the ACR data backend (default response to an unauthenticated /v2/ probe — this 403 is the data endpoint's normal behavior and is independent of the broken state);
• deleting the <registry>.<region>.data A record from privatelink.azurecr.io causes the data FQDN's probe to flip — DNS layer returns class=null with error="gaierror: [Errno -2] Name or service not known" because Azure DNS is authoritative for the linked zone and returns NXDOMAIN, and TCP/TLS/HTTP layers are all skipped because there is no IP to connect to — while the registry FQDN's probe is unchanged (still private, still HTTP 401). The composite signal is topology_class=data_nxdomain;
• AND the already-running revision stays Healthy throughout the broken window — falsifying the alternative hypothesis "a missing record in privatelink.azurecr.io will be detected through ImagePullFailed or revision health degradation in Container Apps."

Re-creating the data A record with the captured PE NIC IP must flip the data FQDN's probe back to DNS class private (PE NIC IP) and the composite signal back to topology_class=both_private, closing the loop on the record-content causation.

3) Runbook¶

Deploy baseline infrastructure¶

export RG="rg-acr-record-split-brain-lab"
export LOCATION="koreacentral"
export BASE_NAME="acrrecsplitbrain"

az extension add --name containerapp --upgrade

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
    --resource-group "$RG" \
    --name acr-record-split-brain \
    --template-file labs/acr-network-path-record-split-brain/infra/main.bicep \
    --parameters baseName="$BASE_NAME"

Expected output pattern:

"provisioningState": "Succeeded"

The initial Container App boots from a public placeholder image (mcr.microsoft.com/k8se/quickstart:latest) so the deployment succeeds before there is anything in the private ACR.

Switch the app to the private ACR image¶

bash labs/acr-network-path-record-split-brain/trigger.sh

The trigger script runs:

az acr update --name "$ACR_NAME" --public-network-enabled true --default-action Allow
az acr build --registry "$ACR_NAME" --image "record-split-brain-lab:v1" \
    --file labs/acr-network-path-record-split-brain/workload/Dockerfile \
    labs/acr-network-path-record-split-brain/workload
az acr update --name "$ACR_NAME" --public-network-enabled false

az containerapp registry set --name "$APP_NAME" --resource-group "$RG" \
    --identity system --server "$ACR_LOGIN_SERVER"

az containerapp update --name "$APP_NAME" --resource-group "$RG" \
    --image "${ACR_LOGIN_SERVER}/record-split-brain-lab:v1" \
    --set-env-vars "BUILD_TAG=v1"

Expected output: provisioningState: Succeeded. A new revision is created and begins pulling.

Verify the healthy baseline is live¶

bash labs/acr-network-path-record-split-brain/verify.sh

The verify script proves four independent signals simultaneously hold:

1. The latest revision's healthState is Healthy.
2. The ACR Private Endpoint NIC holds RFC1918 IPs for BOTH the registry FQDN and the data FQDN inside snet-pe.
3. GET https://<app>.<env>.azurecontainerapps.io/probe returns JSON with topology_class=both_private.
4. The registry probe and data probe both return DNS class private with IPs matching the PE NIC IPs. HTTP status is 401 on the registry FQDN (auth challenge from ACR backend) and 403 on the data FQDN (default response of the ACR data backend to an unauthenticated /v2/ probe — both prove the PE path is alive end-to-end).

All four must hold for the topology to be Scenario B (Path B) end-to-end at the workload layer. The script exits non-zero on any mismatch.

Expected pattern:

[verify] latest revision: healthState=Healthy provisioningState=Provisioned
[verify] PE NIC IPs: registry=10.70.4.5, data=10.70.4.4
[verify] /probe response includes topology_class=both_private
[verify] PASS: revision is Healthy
[verify] PASS: workload /probe topology_class=both_private
[verify] PASS: registry <acr>.azurecr.io -> 10.70.4.5 (private, matches PE NIC IP 10.70.4.5), HTTP 401
[verify] PASS: data     <acr>.koreacentral.data.azurecr.io -> 10.70.4.4 (private, matches PE NIC IP 10.70.4.4), HTTP 403
[verify] PASS: Scenario D BASELINE confirmed. Both records present in the private DNS zone.

Falsify the hypothesis¶

bash labs/acr-network-path-record-split-brain/falsify.sh

The falsification script proves the workload-layer signal flips with the zone record's presence while the already-running revision continues to report Healthy:

1. Calls /probe and asserts topology_class=both_private (baseline).
2. Captures the data PE NIC IP from the live zone record (so step 7 can re-create the record byte-for-byte).
3. Deletes the <registry>.<region>.data A record from privatelink.azurecr.io via az network private-dns record-set a delete.
4. Waits 90 seconds for client-side DNS cache TTL to expire.
5. Calls /probe and asserts topology_class=data_nxdomain, with data DNS class null and error="gaierror: [Errno -2] Name or service not known" (the broken-state workload signal — Azure DNS is authoritative for the linked zone and returns NXDOMAIN for the missing record).
6. Reads az containerapp revision list and asserts the already-running revision is still healthState=Healthy (the Container Apps finding — the workload's view of the data endpoint has changed but the platform has not torn down the revision or attempted a new pull during the broken window).
7. Re-creates the <registry>.<region>.data A record with the captured PE NIC IP via az network private-dns record-set a create + add-record.
8. Waits 90 seconds for the recreated record to become visible to clients.
9. Calls /probe and asserts topology_class=both_private again (recovery).

If step 5 returns topology_class=data_nxdomain with data DNS gaierror NXDOMAIN, step 6 shows the already-running revision still Healthy, and step 9 returns topology_class=both_private, then the <registry>.<region>.data A record is provably the controlled variable for workload data-endpoint reachability while the already-running revision's health is invariant under the same window — the workload's view of the data endpoint has decoupled from the platform's revision-health signal for the duration of the broken window.

Why we do not script a broken-window fresh pull¶

A stronger version of this lab would also deploy a brand-new image tag during the broken window and assert that the resulting fresh pull succeeds — proving the platform pull path is independent of the missing zone record, not just that an already-running revision continues to run. We do not script that variant because it cannot cleanly isolate the data-record variable in this topology.

[Inferred from prior lab — Scenario E reproduction, identical control-plane behavior expected here] When the sibling Scenario E lab attempted az containerapp update --image <new-tag> while the resolver path was broken AND ACR publicNetworkAccess=Disabled, the new revision failed to provision with Reason_s == "FetchingKeyVaultSecretFailed". The control-plane error was identical to what would happen in this lab during the broken window: the Container Apps control plane performs the ACR token exchange step from outside the customer VNet, against the public ACR endpoint, from an Azure-managed egress IP (e.g. 20.249.167.56). Under publicNetworkAccess=Disabled that egress IP is not in any allow-list, so ACR rejects the token exchange with HTTP 403 before the data-plane pull ever begins. This failure is unrelated to the zone record state — even if the data record were present, the token exchange would still be denied at the ACR firewall in this topology.

[Inferred] A scripted broken-window fresh-pull test therefore cannot cleanly isolate the missing-record variable. Under publicNetworkAccess=Disabled the test fails for an unrelated reason (ACR firewall on the control-plane token exchange path); under publicNetworkAccess=Enabled a successful fresh pull would not prove "the puller is independent of the missing record" — only that "ACR is reachable via either path." The lab therefore restricts its claim to what it can falsifiably demonstrate: no immediate revision-health impact on the already-running revision during the broken window, plus the in-replica 4-layer probe (HTTP 403 from the data endpoint) as a direct, ACR-specific signal of the asymmetry. The broken-window fresh-pull behavior of the platform image puller in Azure Container Apps with ACR publicNetworkAccess=Disabled is [Not Proven] by this lab.

Inspect system evidence¶

az containerapp logs show \
    --name "$APP_NAME" \
    --resource-group "$RG" \
    --type system \
    --tail 50

In the post-break window the system logs should keep showing PulledImage for the most recent image tag from the baseline pull, with no new pull events during the broken window — this is the surprising signal:

Reason_s        Log_s
--------------  -----------------------------------------------------------------------
PullingImage    Pulling image '<acr>.azurecr.io/record-split-brain-lab:<tag>'
PulledImage     Successfully pulled image '<acr>.azurecr.io/record-split-brain-lab:<tag>' in 2.x s

A more direct query, useful when reproducing the experiment, is the KQL pack pattern for ACR pull-path observability:

ContainerAppSystemLogs_CL
| where ContainerAppName_s == "<your-app-name>"
| where Reason_s in ("PullingImage", "PulledImage", "ImagePullFailed", "ImagePullUnauthorized", "BackOff")
| order by TimeGenerated asc

Even with the data record deleted, this query should return no new pull events during the broken window (the already-running revision is not re-pulling) and no ImagePullFailed / ImagePullUnauthorized. That is the lab's central pedagogical signal: pull-path observability is silent for the already-running revision in this scenario.

4) Experiment Log¶

Status: Reproduced live in koreacentral on 2026-06-06 with Azure CLI 2.79.0. All nine falsification steps PASSED. The actual values from the live run are below.

Expected Evidence¶

Observed Evidence (Live Azure Test)¶

Live reproduction in koreacentral on 2026-06-06 with Azure CLI 2.79.0. RG rg-acr-record-split-brain-lab, ACR acracrrecsplitbrainb2jo7q (publicNetworkAccess=Disabled), Container App ca-acrrecsplitbrain-b2jo7q, image record-split-brain-lab:v2 (digest sha256:aae2111345eba56bcec4a25fbcd4f54e686adb397f0afc49bb85e03187532b9e), active revision ca-acrrecsplitbrain-b2jo7q--0000002. PE NIC IPs: registry=10.70.4.5, data=10.70.4.4. All three states below were captured by falsify.sh in a single uninterrupted run.

Baseline (falsify.sh step 1, before record deletion)¶

The 4-layer probe shows both FQDNs resolving to PE NIC private IPs. Registry returns HTTP 401 (auth challenge from the ACR registry backend); data returns HTTP 403 (the ACR data endpoint's default response to an unauthenticated /v2/ probe — this 403 is not a firewall reject, it is the data backend's normal behavior on the private path):

{
    "build": "v2",
    "registry": {
        "fqdn": "acracrrecsplitbrainb2jo7q.azurecr.io",
        "dns":  { "ip": "10.70.4.5", "class": "private", "error": null },
        "tcp":  { "connected": true, "error": null },
        "tls":  { "handshake": "ok", "error": null },
        "http": { "status": 401, "status_line": "HTTP/1.1 401 Unauthorized", "error": null }
    },
    "data": {
        "fqdn": "acracrrecsplitbrainb2jo7q.koreacentral.data.azurecr.io",
        "dns":  { "ip": "10.70.4.4", "class": "private", "error": null },
        "tcp":  { "connected": true, "error": null },
        "tls":  { "handshake": "ok", "error": null },
        "http": { "status": 403, "status_line": "HTTP/1.1 403 Forbidden", "error": null }
    },
    "topology_class": "both_private"
}

Broken window (falsify.sh step 5, 90 seconds after az network private-dns record-set a delete)¶

The data record was deleted from the linked privatelink.azurecr.io zone. Subsequent az network private-dns record-set a show returned (NotFound) The resource record 'acracrrecsplitbrainb2jo7q.koreacentral.data' does not exist in resource group 'rg-acr-record-split-brain-lab'. The 4-layer probe shows the registry FQDN unchanged (still private, still HTTP 401) and the data FQDN's DNS layer failing with socket.gaierror NXDOMAIN — Azure DNS is authoritative for the linked zone and answers with NXDOMAIN because the record is gone. All downstream layers (TCP, TLS, HTTP) are skipped because there is no IP to connect to:

{
    "build": "v2",
    "registry": {
        "fqdn": "acracrrecsplitbrainb2jo7q.azurecr.io",
        "dns":  { "ip": "10.70.4.5", "class": "private", "error": null },
        "tcp":  { "connected": true, "error": null },
        "tls":  { "handshake": "ok", "error": null },
        "http": { "status": 401, "status_line": "HTTP/1.1 401 Unauthorized", "error": null }
    },
    "data": {
        "fqdn": "acracrrecsplitbrainb2jo7q.koreacentral.data.azurecr.io",
        "dns":  { "ip": null, "class": null, "error": "gaierror: [Errno -2] Name or service not known" },
        "tcp":  { "connected": null, "error": "skipped (no DNS)" },
        "tls":  { "handshake": null, "error": "skipped (no DNS)" },
        "http": { "status": null, "status_line": null, "error": "skipped (no DNS)" }
    },
    "topology_class": "data_nxdomain"
}

falsify.sh step 6 then queried az containerapp revision list --query "[0].properties.healthState" and confirmed the already-running revision ca-acrrecsplitbrain-b2jo7q--0000002 was still Healthy during the broken window. This is the central Container Apps finding: the workload's view of the data endpoint has changed (the application would see DNS errors on any code path that tries to call the data FQDN), but the platform's revision-health signal has not changed — the platform is not re-pulling the image, so it has no reason to discover the broken record. Pull-path observability is silent.

Recovery (falsify.sh step 9, 90 seconds after az network private-dns record-set a create + add-record --ipv4-address 10.70.4.4)¶

The data record was re-created with the captured PE NIC IP. The 4-layer probe returns to the baseline state — identical JSON to the baseline section above, with topology_class=both_private, data.dns.ip=10.70.4.4, data.http.status=403. The record-content variable has been falsified bidirectionally: deleting the record produces data_nxdomain, restoring the record returns to both_private.

{
    "build": "v2",
    "registry": {
        "fqdn": "acracrrecsplitbrainb2jo7q.azurecr.io",
        "dns":  { "ip": "10.70.4.5", "class": "private", "error": null },
        "tcp":  { "connected": true, "error": null },
        "tls":  { "handshake": "ok", "error": null },
        "http": { "status": 401, "status_line": "HTTP/1.1 401 Unauthorized", "error": null }
    },
    "data": {
        "fqdn": "acracrrecsplitbrainb2jo7q.koreacentral.data.azurecr.io",
        "dns":  { "ip": "10.70.4.4", "class": "private", "error": null },
        "tcp":  { "connected": true, "error": null },
        "tls":  { "handshake": "ok", "error": null },
        "http": { "status": 403, "status_line": "HTTP/1.1 403 Forbidden", "error": null }
    },
    "topology_class": "both_private"
}

Pull-event KQL silence assertion¶

Across the entire baseline → broken → recovery window (~4 minutes wall-clock), ContainerAppSystemLogs_CL for ContainerAppName_s == "ca-acrrecsplitbrain-b2jo7q" produced no new PullingImage / PulledImage / ImagePullFailed / ImagePullUnauthorized / BackOff events for the already-running revision. The most recent pull-event row in the workspace was the baseline PulledImage from the trigger.sh deploy, hours earlier. This silence is the operational signal the lab is designed to teach: a missing data record produces an immediate, hard DNS error at the workload layer, but is invisible to the platform's image-puller observability while the existing revision continues to run on cached layers.

Observed Evidence (Portal Captures — 2026-06-06)¶

A live reproduction on 2026-06-06 captured the full Scenario D topology and the workload-path falsification surface from the Azure Portal.

Environment

[Observed] Container Apps environment → Networking blade: workload-profile environment is attached to VNet vnet-acrrecsplitbrain-b2jo7q, infrastructure subnet snet-aca, Virtual IP External. This is the foundation of the workload-path lab — the Container App's outbound DNS queries leave from this subnet and hit whatever DNS server the VNet advertises.

[Observed] VNet vnet-acrrecsplitbrain-b2jo7q Overview blade: address space 10.70.0.0/16, DNS servers: Azure provided DNS service (no custom DNS server is configured). This is the single most important distinguishing topology of Scenario D vs. Scenario E. Because Azure DNS is the resolver, when the workload queries acracrrecsplitbrainb2jo7q.koreacentral.data.azurecr.io, the query arrives at Azure DNS, which checks the VNet→privatelink.azurecr.io link and consults the private zone as authoritative. With Azure DNS as the resolver and the zone linked, the answer is whatever the linked private zone returns — there is no upstream resolver to fall back to.

[Observed] ACR → Networking → Public access tab: Public network access = Disabled. The registry surface that any public-fallthrough resolution would reach is closed at the firewall. The lab's NXDOMAIN finding (below) means workload DNS for the data subdomain never resolves to any IP in the broken state, so this firewall state is not what the lab tests — but it ensures the lab's claim that the workload-side failure is a DNS resolution error (not a network reachability error) is not confounded by an accidentally-open public path.

[Observed] ACR → Networking → Private access tab: private endpoint connection pe-acracrrecsplitbrainb2jo7q Approved with connection state Approved, Auto-Approved, Provisioning state Succeeded. The blade also shows the registry vs. data endpoint split explicitly — Global endpoint acracrrecsplitbrainb2jo7q.azurecr.io and Data endpoint (Korea Central) acracrrecsplitbrainb2jo7q.koreacentral.data.azurecr.io, with "Use dedicated data endpoint" enabled. These are the two distinct FQDNs whose DNS records can independently drift, which is the topological precondition for Scenario D — a single PE provides both sub-resources but exposes two separately-resolvable hostnames.

[Observed] Private DNS zone privatelink.azurecr.io Recordsets blade — baseline state: 3 record sets present — the SOA record (@), the registry A record (acracrrecsplitbrainb2jo7q → 10.70.4.5, TTL 10s), and the data A record (acracrrecsplitbrainb2jo7q.koreacentral.data → 10.70.4.4, TTL 3600s). Both A records map to PE NIC IPs inside snet-pe (10.70.4.0/24). In this baseline state, the workload resolves both FQDNs to PE NIC IPs and HTTP probes succeed against both (registry returns HTTP 401 from /v2/, data returns HTTP 403 from /v2/ — both are expected non-error responses indicating the private endpoint is reachable).

[Observed] Private DNS zone privatelink.azurecr.io Recordsets blade — broken state (after falsify.sh step 3 deleted the data record): 2 record sets remain — the SOA record and the registry A record. The data A record (acracrrecsplitbrainb2jo7q.koreacentral.data) is missing. This is the entire change between baseline and broken — a single A record in a single private DNS zone, deleted via one az network private-dns record-set a delete call. Combined with 02-vnet-default-azure-dns.png (Azure DNS is the resolver) and 04-acr-private-endpoint-approved.png (PE is still Approved), this capture proves the lab's central finding: with Azure DNS resolving against a linked, authoritative private zone that lacks the queried record, the answer is NXDOMAIN, not a public-IP fallthrough — and this is reproducible from the Portal in one delete-record click.

[Observed] Container App Revisions and replicas blade: revision ca-acrrecsplitbrain-b2jo7q--0000002 Running (at max), 100% traffic, 1 replica, created 6/6/2026 6:52:48 AM KST. This is the already-running revision that stays Healthy throughout the broken window per falsify.sh step 6 — including during the window between 05a-private-dns-records-baseline.png and 05b-private-dns-records-broken.png above. The lab's central asymmetry is observable here: even though application-layer DNS resolution for the data subdomain has already flipped to NXDOMAIN (per the /probe response topology_class=data_nxdomain captured in the live test log above), the operator-facing revision health surface in the Portal stayed green and did not signal the underlying zone-record deletion.

[Inferred] The seven captures, taken in time order from the Portal, document Scenario D's required topology (Container Apps environment with default Azure VNet DNS — 02-vnet-default-azure-dns.png, ACR with publicNetworkAccess=Disabled + approved PE that exposes split registry/data FQDNs — 03/04, private DNS zone linked to the VNet — implied by the records existing at all) and prove the lab's central finding via direct A/B comparison: 05a-private-dns-records-baseline.png shows 3 records and the workload sees topology_class=both_private; 05b-private-dns-records-broken.png shows 2 records (data record missing) and the workload sees topology_class=data_nxdomain. Throughout both states, 06-revisions-healthy.png shows the revision --0000002 stays Running (at max) / Healthy. The Portal captures complement the CLI evidence above — together they show that, in default Azure DNS topology, deleting a single record from an authoritative linked private zone is the entire necessary and sufficient input to flip workload DNS to NXDOMAIN for that specific FQDN while every other operator-facing surface (PE Approved, revision Healthy, public-access still Disabled) stays unchanged. This is precisely the silent-failure asymmetry that makes Scenario D dangerous: every Portal blade outside the Recordsets list shows a healthy system, yet workload DNS is broken in a hard, immediate, namespace-authoritative way.

Clean Up¶

bash labs/acr-network-path-record-split-brain/cleanup.sh

Which runs:

az group delete --name "$RG" --yes --no-wait

ACR Premium is the dominant cost (~$1.67/day), so do not leave the lab running between sessions.

Related Playbook¶

• Private Endpoint DNS Failure — covers the broader Private Endpoint DNS failure family; this lab is the Scenario D (record-level split-brain) reproduction inside that family, featuring the workload-vs-pull-path distinction specific to Container Apps.

See Also¶

• ACR Network Path Selection — the platform decision page this lab implements; see the DNS Patterns That Decide the Path table and the In Azure Container Apps, Scenarios D and E may surface at the workload layer first bullet in ACR-Specific Gotchas
• ACR Network Path B Lab — the happy-path PE topology this lab silently degrades from at the zone-record layer
• ACR Network Path E Lab — the resolver-topology variant of the same outcome class; same operator-facing silence (revision stays Healthy, no platform pull events) but different root cause (broken forwarder upstream vs. missing zone record) and different fix (re-point the forwarder vs. restore the zone record). The 4-layer probe distinguishes them: Scenario E shows topology_class=both_public (the resolver bypasses Azure DNS for the entire ACR namespace), Scenario D in default Azure DNS shows topology_class=data_nxdomain (Azure DNS is authoritative for the linked zone and returns NXDOMAIN for the missing record)
• Egress Control — explains why the platform pull path itself has its own outbound dependencies even when the workload's view of DNS is broken
• Private Endpoints — Private Endpoint fundamentals, including the privatelink.* zone substitution requirement

Sources¶

• Azure Private Endpoint DNS configuration (Microsoft Learn) — why Azure DNS performs IP substitution for privatelink.* zones and how linked private zones behave authoritatively for names in that namespace
• Configure a private link for an Azure Container Registry (Microsoft Learn) — ACR PE topology, sub-resource model, and the registry+data FQDN split
• Networking in Azure Container Apps (Microsoft Learn) — Container Apps VNet DNS behavior (defaults to Azure DNS when no custom DNS server is configured on the VNet)
• Use a private endpoint with Azure Container Apps (Microsoft Learn)
• Configure public network access for an Azure container registry (Microsoft Learn) — the ACR firewall behavior that produces HTTP 403 from publicNetworkAccess=Disabled
• Configure rules to access an Azure Container Registry behind a firewall (Microsoft Learn)
• Authenticate with an Azure container registry (Microsoft Learn)