ACA Secret Key Vault Reference — Virtual WAN + Routing Intent Variant (H4d) Lab¶

Cost warning Virtual WAN secured-hub deployments are expensive. Prefer existing secured-hub mode whenever possible. Use full synthetic mode only with explicit opt-in (DEPLOY_VIRTUAL_WAN=true) and delete the resource group immediately after the run.

Reproduce the Virtual WAN + Routing Intent failure surface where az containerapp secret set --identity system --key-vault-url ... fails with Unable to get value using Managed identity and an openid-configuration error only while Routing Intent forces the ACA infrastructure VNet's egress through a secured-hub Azure Firewall. The firewall policy stays intentionally restrictive throughout: it allows *.vault.azure.net, but it does not allow the Entra authority FQDNs login.microsoftonline.com and login.microsoft.com.

This lab is a reader-generated 17-gate Phase B falsification workflow. You run trigger.sh and falsify.sh against your own Azure subscription to capture one live H0 → H1 → H2 cohort (files 01-13) into labs/aca-secret-kv-ref-mi-network-path-h4d/evidence/. You then run verify.sh, which reads only those local files (no Azure API calls) and deterministically emits the four Phase B gate JSONs (14-17) that validate the narrow claim: Routing Intent is the controlled route-state variable while workload, identity, Key Vault, and firewall policy stay constant.

Bounded-scope disclosure: this workflow does not prove packet capture, dataplane bypass, regional-hub propagation bugs, or perfect route-table/dataplane synchronization at the failure instant. It also does not use az containerapp exec as the primary egress-path proof; the reproducer uses the same az containerapp secret set symptom in H0, H1, and H2. Those confounders are carried explicitly in Gate 17 explicit_drops.

Lab Metadata¶

1) Background¶

Azure Container Apps supports Key Vault references in the secret manifest: the app declares a reference of the form --key-vault-url https://<vault>.vault.azure.net/secrets/<name> and the platform resolves it using a managed identity. Before the platform can read Key Vault, it must acquire a token, and managed identity token acquisition requires an OIDC discovery step against the Entra authority — an HTTPS request to https://login.microsoftonline.com/<tenant>/.well-known/openid-configuration (or the related login.microsoft.com authority host).

H4d reproduces a route-state failure. The ACA environment itself stays simple: Azure-provided DNS, no custom DNS servers, no Key Vault private endpoint, and no UDR attached to the ACA subnet. The secured-hub firewall policy is also stable: Key Vault public FQDNs stay allowed, while the Entra authority FQDNs stay blocked. What changes is whether Routing Intent on the Virtual Hub redirects internet/private traffic through that firewall.

That evidence discipline matters:

• [Observed] az network vhub routing-intent show reports whether the Routing Intent policy exists and where it points.
• [Observed] az network vhub get-effective-routes --resource-type HubVirtualNetworkConnection --resource-id $ACA_VNET_CONNECTION_ID ... reports whether 0.0.0.0/0 targets the hub firewall from the Virtual Hub control-plane perspective.
• [Observed] H0 succeeds, H1 fails, and H2 succeeds again using the same az containerapp secret set symptom.
• [Not Proven] Effective routes are control-plane evidence, not packet capture.
• [Inferred] Routing Intent is the causal variable only because the symptom flips while the app, identity, Key Vault, and firewall policy stay constant.

Architecture¶

flowchart TD
    A["Operator: az containerapp secret set --identity system --key-vault-url"] --> B["ACA control-plane worker resolves the Key Vault reference"]
    B --> C["Managed identity OIDC discovery must reach login.microsoftonline.com / login.microsoft.com"]
    C --> D["ACA infrastructure VNet"]
    D --> E{"Routing Intent active on the Virtual Hub?"}
    E -->|H0 / H2: No| F["No active secured-hub internet route on this path\nOIDC discovery reaches public Entra authority\nsecret set succeeds\nrevision unchanged"]
    E -->|H1: Yes| G["InternetTraffic + PrivateTraffic -> secured-hub Azure Firewall\nFirewall policy still allows *.vault.azure.net only\nEntra authority remains blocked\nsecret set fails before Key Vault read\nrevision unchanged, ingress HTTP 200"]
    G --> H["Virtual Hub effective routes show 0.0.0.0/0 -> hub firewall"]

2) Hypothesis¶

IF an Azure Container Apps environment uses Azure-provided DNS on its infrastructure VNet, the app has a system-assigned managed identity granted Key Vault Secrets User at the target Key Vault scope, Key Vault remains public, and the secured-hub firewall policy continues to allow *.vault.azure.net but continues to block login.microsoftonline.com and login.microsoft.com, THEN:

• H0 baseline (no active Routing Intent path): az containerapp secret set --identity system --key-vault-url ... succeeds with exit code 0. The named secret kvref-h0 appears in properties.configuration.secrets. latestReadyRevisionName is unchanged.
• H1 (Routing Intent ON): after connecting the ACTUAL ACA infrastructure VNet to the Virtual Hub and enabling Routing Intent so both InternetTraffic and PrivateTraffic target the hub Azure Firewall, the command fails with exit code non-zero. stderr carries Unable to get value using Managed identity and includes openid-configuration. configuration.secrets does not contain kvref-h1. latestReadyRevisionName is still unchanged. Ingress still returns HTTP 200. [Observed] az network vhub get-effective-routes --resource-type HubVirtualNetworkConnection --resource-id $ACA_VNET_CONNECTION_ID ... shows 0.0.0.0/0 targeting the hub firewall.
• H2 (Routing Intent OFF): after removing Routing Intent while leaving the firewall policy unchanged, the effective routes no longer show 0.0.0.0/0 targeting the hub firewall, and a new secret-set attempt succeeds with exit code 0. kvref-h2 appears in configuration.secrets. latestReadyRevisionName is still unchanged from baseline. Ingress still returns HTTP 200.

3) Runbook¶

Prerequisites¶

• Azure CLI 2.80+ with the containerapp extension and the Virtual WAN extension commands available.
• Azure subscription permissions for: resource group deploy, role assignment (Microsoft.Authorization/roleAssignments/write), Container Apps management, Key Vault management, and Virtual WAN / Virtual Hub / Routing Intent management.
• jq, curl, and a local shell that can run the lab scripts.
• Either an existing secured hub (preferred) or explicit willingness to pay for a synthetic secured hub.

Deploy infrastructure¶

Use one of the two supported modes below.

export RG="rg-aca-secret-kv-ref-mi-network-path-h4d"
export LOCATION="koreacentral"
export BASE_NAME="acasech4d01"

export EXISTING_VIRTUAL_HUB_RESOURCE_ID="/subscriptions/<subscription-id>/resourceGroups/<vwan-rg>/providers/Microsoft.Network/virtualHubs/<vhub-name>"
export EXISTING_AZURE_FIREWALL_RESOURCE_ID="/subscriptions/<subscription-id>/resourceGroups/<vwan-rg>/providers/Microsoft.Network/azureFirewalls/<azfw-name>"
export EXISTING_FIREWALL_POLICY_RESOURCE_ID="/subscriptions/<subscription-id>/resourceGroups/<vwan-rg>/providers/Microsoft.Network/firewallPolicies/<policy-name>"
export EXISTING_FIREWALL_LOG_ANALYTICS_CUSTOMER_ID="00000000-0000-0000-0000-000000000000"

bash labs/aca-secret-kv-ref-mi-network-path-h4d/trigger.sh

If you need the fully synthetic mode instead, add export DEPLOY_VIRTUAL_WAN="true" before running trigger.sh and omit the three EXISTING_*_RESOURCE_ID variables.

Run the H1 → H2 falsification (falsify.sh)¶

bash labs/aca-secret-kv-ref-mi-network-path-h4d/falsify.sh

Expected output:

• 06-h1-routing-intent-enabled.json confirms the ACTUAL ACA infrastructure VNet is connected to the Virtual Hub, Routing Intent converged, and effective routes show 0.0.0.0/0 targeting the hub firewall.
• 07-h1-secret-set-outcome.json contains a non-zero exit_code and stderr with Unable to get value using Managed identity plus openid-configuration.
• 08-h1-app-state.json shows revision name unchanged, ingress HTTP 200, and kvref-h1 absent.
• 10-h2-routing-intent-removed.json confirms effective routes no longer show 0.0.0.0/0 targeting the hub firewall.
• 11-h2-secret-set-outcome.json contains exit_code: 0.
• 12-h2-app-state.json shows revision name still unchanged from baseline, ingress HTTP 200, and kvref-h2 present.

Run the offline verifier over your locally generated pack¶

bash labs/aca-secret-kv-ref-mi-network-path-h4d/verify.sh

Expected output:

• 17/17 gate passes on a valid cohort.
• evidence/14-cohort-integrity-gate.json shows the HubVirtualNetworkConnection anchor and revision silence invariant.
• evidence/15-h1-routing-intent-produces-failure-gate.json proves H1 route-state plus failure markers.
• evidence/16-h2-routing-intent-removal-restores-success-gate.json proves H2 route-state removal plus recovery.
• evidence/17-bounded-falsification-gate.json enumerates the explicit drops for Azure Firewall log silence, effective-route limits, DNS-override non-coverage, Key Vault private endpoint non-coverage, and lack of packet-capture proof.

Optional: inspect the Virtual WAN route state manually¶

az network vhub routing-intent show \
    --name "defaultRoutingIntent" \
    --resource-group "<vwan-rg>" \
    --vhub "<vhub-name>"

az network vhub get-effective-routes \
    --resource-type HubVirtualNetworkConnection \
    --resource-id "/subscriptions/<subscription-id>/resourceGroups/<vwan-rg>/providers/Microsoft.Network/virtualHubs/<vhub-name>/hubVirtualNetworkConnections/<connection-name>" \
    --name "<vhub-name>" \
    --resource-group "<vwan-rg>"

Expected interpretation:

• H1 window: [Observed] the Routing Intent policy exists and the effective route table includes 0.0.0.0/0 targeting the hub firewall.
• H2 window: [Observed] the effective route table no longer shows 0.0.0.0/0 targeting the hub firewall.
• [Not Proven] This query alone does not prove dataplane packet forwarding, only Virtual Hub control-plane route state.

4) Experiment Log¶

Evidence discipline for H4d¶

• [Observed] 06-h1-routing-intent-enabled.json and 10-h2-routing-intent-removed.json capture the Virtual Hub control-plane route state.
• [Observed] 07-h1-secret-set-outcome.json fails, while 11-h2-secret-set-outcome.json succeeds.
• [Observed] 08-h1-app-state.json and 12-h2-app-state.json prove the silence gates: the same revision keeps serving ingress HTTP 200.
• [Inferred] Routing Intent is the causal variable because the symptom flips while the app, identity, Key Vault, and firewall policy stay constant.
• [Not Proven] Effective routes do not prove packet forwarding or that Azure Firewall never received the traffic.

5) Verification Queries¶

CLI: inspect the Virtual WAN route state¶

az network vhub routing-intent show \
    --name "defaultRoutingIntent" \
    --resource-group "<vwan-rg>" \
    --vhub "<vhub-name>"

az network vhub get-effective-routes \
    --resource-type HubVirtualNetworkConnection \
    --resource-id "/subscriptions/<subscription-id>/resourceGroups/<vwan-rg>/providers/Microsoft.Network/virtualHubs/<vhub-name>/hubVirtualNetworkConnections/<connection-name>" \
    --name "<vhub-name>" \
    --resource-group "<vwan-rg>"

Expected interpretation:

• H1 window: [Observed] 0.0.0.0/0 targets the hub firewall.
• H2 window: [Observed] 0.0.0.0/0 no longer targets the hub firewall.
• [Not Proven] This is control-plane route evidence, not packet capture.

CLI: optional Azure Firewall clue (never the pass condition)¶

az monitor log-analytics query \
    --workspace "00000000-0000-0000-0000-000000000000" \
    --analytics-query "AZFWApplicationRule | where Fqdn has 'login.microsoftonline.com' or Fqdn has 'login.microsoft.com' | order by TimeGenerated desc | take 20"

Expected interpretation:

• [Observed] If diagnostics are enabled and the flow reaches Azure Firewall, rows may appear for the Entra authority FQDNs.
• [Not Proven] A zero-row result does not prove bypass; it may reflect diagnostics latency, disabled categories, the wrong workspace, or the wrong firewall.
• [Inferred] Use this only as a clue in real investigations. The reproducer itself gates on Routing Intent + effective-route state, not on log silence.

6) Portal Evidence¶

Reader-generated deferral note: Portal captures for this lab are tracked in issue #366.

Clean Up¶

bash labs/aca-secret-kv-ref-mi-network-path-h4d/cleanup.sh

Related Playbook¶

• Secret and Key Vault Reference Failure — H4 Variants: When Base H4 KQL Returns Zero Rows (H4d row)

See Also¶

• ACA Secret Key Vault Reference — Managed Identity Network Path Lab (H4a base variant)
• ACA Secret Key Vault Reference — Logging Gap Variant (H4b)
• ACA Secret Key Vault Reference — Custom DNS Override Variant (H4e)
• Managed Identity Auth Failure Playbook
• Egress Control

Sources¶

• Manage secrets in Azure Container Apps
• Managed identities in Azure Container Apps
• Custom VNet in Azure Container Apps
• How to configure Virtual WAN Hub routing policies
• View effective routes in a Virtual WAN hub