ACA Secret Key Vault Reference — Custom DNS Override Variant (H4e) Lab¶

Reproduce the custom DNS override failure surface where az containerapp secret set --identity system --key-vault-url ... fails with Unable to get value using Managed identity → Get https://login.microsoftonline.com/<tenant>/.well-known/openid-configuration: EOF, without any Azure Firewall or UDR in path. This lab proves that a custom linked Private DNS override for login.microsoftonline.com and login.microsoft.com can sink the Entra authority lookup to 192.0.2.1, and that removing the override restores success with the same Key Vault, identity, and RBAC state.

This lab is a reader-generated 17-gate Phase B falsification workflow. You run trigger.sh and falsify.sh against your own Azure subscription to capture one live H0 → H1 → H2 cohort (files 01-13) into labs/aca-secret-kv-ref-mi-network-path-h4e/evidence/. You then run verify.sh, which reads only those local files (no Azure API calls) and deterministically emits the four Phase B gate JSONs (14-17) that validate the narrow claim: the custom DNS override is the sole controlled variable, and flipping it changes whether the managed-identity OIDC discovery step can reach the public Entra authority.

Bounded-scope disclosure: the workflow does not prove exact stderr wording, exact DNS propagation latency, or that the ACA control-plane secret resolver shares an identical DNS view with az containerapp exec. The replica DNS result is a direct data-plane observation; the control-plane implication is an inference justified only by the H0/H1/H2 secret-set behavior. Those confounders are carried explicitly in Gate 17 explicit_drops.

Lab Metadata¶

1) Background¶

Azure Container Apps supports Key Vault references in the secret manifest: the app declares a reference of the form --key-vault-url https://<vault>.vault.azure.net/secrets/<name> and the platform resolves it using a managed identity. Before the platform can call Key Vault it must acquire a token, and managed identity token acquisition requires an OIDC discovery step against the Entra authority — a plain HTTPS GET https://login.microsoftonline.com/<tenant>/.well-known/openid-configuration (the client may use login.microsoft.com instead — it picks one host at runtime).

H4e reproduces a DNS-level failure before any outbound firewall discussion is relevant. Azure-provided DNS for a VNet consults linked Private DNS zones for matching names. If the workload VNet is linked to a custom zone named login.microsoftonline.com or login.microsoft.com, and that zone's apex A record points to a sink address such as 192.0.2.1, callers using that DNS view resolve the Entra authority host to the sink instead of to a public Microsoft IP. The subsequent TCP connection attempt times out or resets on the wrong destination, and the user only sees the same generic managed-identity OIDC discovery error surface.

This is why the evidence discipline matters:

• The replica nslookup result is a direct data-plane DNS observation. It shows what a running workload replica resolves.
• The failing caller in this playbook is the ACA-managed control-plane secret resolver, not the workload container itself.
• Azure does not formally document that these two components share an identical DNS view.
• Therefore the control-plane implication must stay bounded: the replica DNS result is [Observed]; the claim that the control-plane resolver saw the same override is only [Inferred] or [Strongly Suggested], justified by the H0/H1/H2 secret-set flip.

Architecture¶

flowchart TD
    A["Operator: az containerapp secret set --identity system --key-vault-url"] --> B["ACA control-plane worker resolves KV URL"]
    B --> C["MI token acquisition begins:<br/>GET https://login.microsoftonline.com/{tenant}/.well-known/openid-configuration"]
    C --> D["Azure-provided DNS for VNet vnet-{baseName}-{suffix}"]
    D --> E{"Custom linked Private DNS zone for<br/>login.microsoftonline.com or login.microsoft.com?"}
    E -->|H0 / H2: No| F["Public Microsoft IP returned<br/>OIDC discovery succeeds<br/>secret set succeeds<br/>revision unchanged"]
    E -->|H1: Yes| G["Apex A record -> 192.0.2.1<br/>TCP connection goes to sink IP<br/>OIDC discovery fails before Key Vault read<br/>secret set fails<br/>revision unchanged, ingress HTTP 200"]
    G --> H["Replica nslookup login.microsoftonline.com<br/>returns 192.0.2.1"]

2) Hypothesis¶

IF an Azure Container Apps environment uses Azure-provided DNS on its infrastructure VNet, and the app has a system-assigned managed identity granted Key Vault Secrets User at the target Key Vault scope, and there is no Azure Firewall and no UDR on the workload path, THEN:

• H0 baseline (no override): az containerapp secret set --identity system --key-vault-url ... succeeds with exit code 0. The named secret kvref-h0 appears in properties.configuration.secrets with a populated keyVaultUrl field. latestReadyRevisionName is unchanged.
• H1 (custom linked Private DNS override present): After creating Private DNS zones login.microsoftonline.com and login.microsoft.com, linking both to the ACA VNet, and setting each apex A record to 192.0.2.1 with TTL 10 seconds, the command fails with exit code non-zero. stderr carries the marker phrases Failed to update secrets, Unable to get value using Managed identity, and includes the substring openid-configuration. configuration.secrets does not contain kvref-h1. latestReadyRevisionName is still unchanged. Ingress still returns HTTP 200. [Observed] Replica nslookup login.microsoftonline.com returns 192.0.2.1.
• H2 (override removed, wait > TTL): After unlinking/deleting the custom override and waiting beyond TTL, a new secret-set attempt succeeds with exit code 0. kvref-h2 appears in configuration.secrets. latestReadyRevisionName is still unchanged from baseline. Ingress still returns HTTP 200. [Observed] Replica nslookup login.microsoftonline.com no longer returns 192.0.2.1 and instead shows a public Microsoft IP. [Inferred] The control-plane resolver is no longer trapped by the custom override because the secret-set operation succeeds again without any other topology change.

3) Runbook¶

Prerequisites¶

• Azure CLI 2.80+ with the containerapp extension.
• Azure subscription permissions for: resource group deploy, role assignment (Microsoft.Authorization/roleAssignments/write), Container Apps management, Key Vault management, and Private DNS zone/link management.
• jq, curl, and a local shell that can run the lab scripts.

Deploy infrastructure¶

export RG="rg-aca-secret-kv-ref-mi-network-path-h4e"
export LOCATION="koreacentral"
export BASE_NAME="acasech4e01"

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
    --resource-group "$RG" \
    --name aca-secret-kv-ref-mi-network-path-h4e \
    --template-file labs/aca-secret-kv-ref-mi-network-path-h4e/infra/main.bicep \
    --parameters baseName="$BASE_NAME" \
    --parameters deploymentPrincipalId="$(az ad signed-in-user show --query id --output tsv)"

Expected output:

• Resource group creation succeeds.
• Deployment provisioningState is Succeeded.
• Bicep outputs include appName, environmentName, keyVaultName, keyVaultUri, vnetName, acaSubnetPrefix, and logAnalyticsCustomerId.
• trigger.sh later records that the VNet uses Azure-provided DNS and the ACA subnet has no route table.

Run the H0 baseline (trigger.sh)¶

bash labs/aca-secret-kv-ref-mi-network-path-h4e/trigger.sh

Expected output:

• 04-h0-secret-set-outcome.json contains exit_code: 0.
• 05-h0-app-state-after.json shows the secret kvref-h0 present in configuration.secrets with a populated keyVaultUrl field.
• 05-h0-app-state-after.json latestReadyRevisionName matches the value in 02-h0-app-state-before.json (secret set does not create a new revision).
• 01-deployment-outputs.json shows azure_firewall_present: false, route_table_attached: false, and vnet_dns_servers: [].

Run the H1 → H2 falsification (falsify.sh)¶

bash labs/aca-secret-kv-ref-mi-network-path-h4e/falsify.sh

Expected output:

• 06-h1-dns-override-created.json confirms both custom Private DNS zones exist, both apex A records point to 192.0.2.1, TTL is 10 seconds, and the zones are linked to the ACA VNet.
• 07-h1-secret-set-outcome.json contains exit_code non-zero and stderr with Failed to update secrets, Unable to get value using Managed identity, and openid-configuration.
• 08-h1-app-state.json shows revision name unchanged, ingress HTTP 200, and kvref-h1 absent from configuration.secrets.
• 09-h1-replica-dns-view.json shows observed_sink_presence: true for login.microsoftonline.com.
• 10-h2-dns-override-removed.json confirms the override is removed and post_removal_wait_seconds > ttl_seconds.
• 11-h2-secret-set-outcome.json contains exit_code: 0.
• 12-h2-app-state.json shows revision name still unchanged from baseline, ingress HTTP 200, and kvref-h2 present in configuration.secrets.
• 13-h2-replica-dns-view.json shows observed_sink_presence: false.

Run the offline verifier over your locally generated pack¶

bash labs/aca-secret-kv-ref-mi-network-path-h4e/verify.sh

Expected output:

• 17/17 gate passes on a valid cohort.
• evidence/14-cohort-integrity-gate.json shows the revision silence invariant and the non-H4a topology anchors.
• evidence/15-h1-dns-override-produces-failure-gate.json shows the H1 sub-gates pass, including the replica sink-IP observation.
• evidence/16-h2-override-removal-restores-success-gate.json shows the H2 sub-gates pass, including the post-removal wait > TTL predicate and kvref-h2 presence.
• evidence/17-bounded-falsification-gate.json enumerates the explicit drops for control-plane-vs-data-plane DNS view equivalence, exact stderr wording, DNS cache timing, and the lack of a supported privatelink.microsoftonline.com zone.

Optional: inspect the custom override manually during H1¶

az network private-dns zone list --resource-group "$RG"

az containerapp exec \
    --name "$(jq -r .app_name labs/aca-secret-kv-ref-mi-network-path-h4e/evidence/01-deployment-outputs.json)" \
    --resource-group "$RG" \
    --command "sh -c 'nslookup login.microsoftonline.com || true'"

Expected output:

• [Observed] During H1, az network private-dns zone list includes login.microsoftonline.com and login.microsoft.com.
• [Observed] During H1, the replica nslookup output includes 192.0.2.1.
• [Inferred] Because H0 succeeds, H1 fails, and H2 succeeds with no firewall or UDR change, the control-plane secret resolver is strongly suggested to be affected by the same custom override. This is not claimed as proven identity of DNS view.

4) Experiment Log¶

Evidence discipline for H4e¶

This lab intentionally separates what is directly observed from what is only inferred:

• [Observed] 09-h1-replica-dns-view.json and 13-h2-replica-dns-view.json capture the data-plane DNS view from a running replica.
• [Observed] 07-h1-secret-set-outcome.json fails, while 11-h2-secret-set-outcome.json succeeds.
• [Inferred] The ACA control-plane secret resolver was affected by the same DNS override because the secret-set result flips exactly with the override and no firewall/UDR/RBAC variable changes.
• [Not Proven] Azure does not document that the control-plane secret resolver and az containerapp exec share an identical DNS path or cache. That equivalence is intentionally left as an explicit drop in Gate 17.

Why this is not a firewall lab¶

The baseline topology is deliberately silent on firewall evidence because there is no firewall to inspect. 01-deployment-outputs.json captures three direct anchors:

1. azure_firewall_present: false
2. route_table_attached: false
3. vnet_dns_servers: []

That means the H1 failure cannot be explained by H4a/H4b-style firewall denial. The packet never needs a firewall theory if the Entra authority host is already resolving to 192.0.2.1 inside the workload DNS view. This is the point of the H4e variant: the same customer-facing OIDC discovery error can originate from a DNS override even when the usual AzFW-deny hypotheses are structurally impossible.

5) Verification Queries¶

CLI: inspect the replica DNS view¶

az containerapp exec \
    --name "$(jq -r .app_name labs/aca-secret-kv-ref-mi-network-path-h4e/evidence/01-deployment-outputs.json)" \
    --resource-group "$RG" \
    --command "sh -c 'nslookup login.microsoftonline.com || true'"

Expected interpretation:

• H1 window: [Observed] the output includes 192.0.2.1. This is the data-plane DNS view.
• H2 window: [Observed] the output no longer includes 192.0.2.1 and instead shows a public Microsoft IP.
• [Inferred] The control-plane secret resolver likely followed a compatible DNS outcome because the secret-set result flips in lockstep with the override.
• [Not Proven] This query alone does not prove control-plane/data-plane DNS-view equivalence.

CLI: inspect the custom Private DNS zones and VNet DNS settings¶

az network private-dns zone list --resource-group "$RG"

az network vnet show \
    --resource-group "$RG" \
    --name "$(jq -r .vnet_name labs/aca-secret-kv-ref-mi-network-path-h4e/evidence/01-deployment-outputs.json)" \
    --query "dhcpOptions.dnsServers"

Expected interpretation:

• H1 window: [Observed] az network private-dns zone list includes login.microsoftonline.com and login.microsoft.com.
• H0/H2 windows: [Observed] those zones are absent.
• All phases: [Observed] dhcpOptions.dnsServers is [] (Azure-provided DNS), so the override is coming from linked Private DNS zones rather than from custom VNet DNS servers.
• [Not Proven] The absence of dhcpOptions.dnsServers does not rule out every external enterprise DNS influence in other topologies; it only bounds this lab's topology.

6) Portal Evidence (reader-generated captures)¶

Azure Portal screenshots to collect after running the H0 → H1 → H2 sequence. Save to docs/assets/troubleshooting/aca-secret-kv-ref-mi-network-path-h4e/ with the exact filenames listed in the Screenshot capture checklist below.

Container App — Secrets blade (H0 baseline)¶

Private DNS zone — H1 apex A record¶

Private DNS zone — VNet link present in H1¶

Container App — Revisions blade (silence gate)¶

H2 zone removed / H2 success state¶

Screenshot capture checklist¶

Clean Up¶

bash labs/aca-secret-kv-ref-mi-network-path-h4e/cleanup.sh

Related Playbook¶

• Secret and Key Vault Reference Failure — H4 Variants: When Base H4 KQL Returns Zero Rows (H4e row)

See Also¶

• ACA Secret Key Vault Reference — Managed Identity Network Path Lab (H4a base variant)
• ACA Secret Key Vault Reference — Logging Gap Variant (H4b)
• Managed Identity Auth Failure Playbook
• Egress Control

Sources¶

• Manage secrets in Azure Container Apps
• Managed identities in Azure Container Apps
• Custom VNet in Azure Container Apps
• Azure Private DNS virtual network links
• What is Azure Private DNS?